[OE-core] [PATCH] gpg_sign: perform rpm signing serially

[Markus Lehtonen]

Hi,

I quickly run some tests on a Xeon server, using glibc-locale as the recipe to build.
100: 154s 
10: 162s (+5%)
1: 234s (+51%)

Even if signing is not parallel, the difference may be explained by the number of rpm processes that get spawned. I would drop the factor to 10 or use BB_NUMBER_THREADS as Andre suggested in another email.
  - Markus



On 16/08/2017, 19.00, "Leonardo Sandoval" <leonardo.sandoval.gonzalez at linux.intel.com> wrote:

    On Wed, 2017-08-16 at 15:28 +0300, Markus Lehtonen wrote:
    > I agree. I don't see reason for dropping parallelism completely. There is a real gain when running on beefier machines. Making it configurable would probably be best. Or just drop it to a saner value, like 20 or 10.
    >    - Markus
    > 
    
    I ran some tests with 100, 20 and 1 and I saw (I can rerun and provide
    times) no difference on times. gpg may be intrinsically serial so
    passing 1 or N files wont make much difference in type. The only gain
    when using file chunks is that one one process is launched.
    
    I the other hand, I tried using multiprocessing.Pool, but failed
    miserably due to file looking reasons.
    
    
    
    > On 16/08/2017, 2.53, "Mark Hatle" <openembedded-core-bounces at lists.openembedded.org on behalf of mark.hatle at windriver.com> wrote:
    > 
    >     It would probably be better if this was configurable with a 'safe' default.
    >     
    >     Moving from parallel to single will greatly affect the overall performance on
    >     larger build machines (lots of memory and cores) that can handle the load vs a
    >     typical development machine.
    >     
    >     --Mark
    >     
    >     On 8/15/17 4:40 PM, leonardo.sandoval.gonzalez at linux.intel.com wrote:
    >     > From: Leonardo Sandoval <leonardo.sandoval.gonzalez at linux.intel.com>
    >     > 
    >     > gpg signing in file batches (which was default to 100) is a memory expensive
    >     > computation, causing trouble in some host machines (even on production AB
    >     > as seen on the bugzilla ID). Also, in terms of performance, there is no real
    >     > gain when rpm signing is done in batches. Considering the latter issues, perform the
    >     > rpm signing serially.
    >     > 
    >     > Log showing errors observed recently at AB workers:
    >     > 
    >     >     | gpg: signing failed: Cannot allocate memory
    >     >     | gpg: signing failed: Cannot allocate memory
    >     >     | error: gpg exec failed (2)
    >     >     | /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/core2-64-poky-linux/base-passwd/3.5.29-r0/deploy-rpms/core2_64/base-passwd-dev-3.5.29-r0.core2_64.rpm:
    >     > 
    >     > [YOCTO #11914]
    >     > 
    >     > Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez at linux.intel.com>
    >     > ---
    >     >  meta/lib/oe/gpg_sign.py | 6 +++---
    >     >  1 file changed, 3 insertions(+), 3 deletions(-)
    >     > 
    >     > diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py
    >     > index f4d8b10e4b..5c7985a856 100644
    >     > --- a/meta/lib/oe/gpg_sign.py
    >     > +++ b/meta/lib/oe/gpg_sign.py
    >     > @@ -45,9 +45,9 @@ class LocalSigner(object):
    >     >              if fsk_password:
    >     >                  cmd += "--define '_file_signing_key_password %s' " % fsk_password
    >     >  
    >     > -        # Sign in chunks of 100 packages
    >     > -        for i in range(0, len(files), 100):
    >     > -            status, output = oe.utils.getstatusoutput(cmd + ' '.join(files[i:i+100]))
    >     > +        # Sign packages
    >     > +        for f in files:
    >     > +            status, output = oe.utils.getstatusoutput(cmd + ' ' + f)
    >     >              if status:
    >     >                  raise bb.build.FuncFailed("Failed to sign RPM packages: %s" % output)
    >     >  
    >     > 
    >     
    >     -- 
    >     _______________________________________________
    >     Openembedded-core mailing list
    >     Openembedded-core at lists.openembedded.org
    >     http://lists.openembedded.org/mailman/listinfo/openembedded-core
    >     
    > 
    >