[oe-commits] [openembedded-core] 04/06: bitbake.conf: omit XZ threads and RAM from sstate signatures

[git at git.openembedded.org]

This is an automated email from the git hooks/post-receive script.

rpurdie pushed a commit to branch master-next
in repository openembedded-core.

commit 7689182b132d97c583b648643b21c1699c328321
Author: André Draszik <git at andred.net>
AuthorDate: Tue Mar 3 16:05:12 2020 +0000

    bitbake.conf: omit XZ threads and RAM from sstate signatures
    
    The number of threads used, and the amount of memory allowed
    to be used, should not affect sstate signatures, as they
    don't affect the outcome of the compression if xz operates
    in multi-threaded mode [1].
    
    Otherwise, it becomes impossible to re-use sstate from
    automated builders on developer's machines (as the former
    might execute bitbake with certain constraints different
    compared to developer's machines).
    
    This is in particular a problem with the opkg package writing
    backend, as the OPKGBUILDCMD depends on XZ_DEFAULTS. Without
    the vardepexclude, there is no re-use possible of the
    package_write_ipk sstate.
    
    Whitelist the maximum number of threads and the memory limit
    given assumptions outlined in [2] below.
    
    Signed-off-by: André Draszik <git at andred.net>
    
    [1] When starting out in multi-threaded mode, the output is always
    deterministic, as even if xz scales down to single-threaded later,
    the archives are still split into blocks and size information is
    still added, thus keeping them compatible with multi-threaded mode.
    Also, when starting out in multi-threaded mode, xz never scales
    down the compression level to accomodate memory usage restrictions,
    it just scales down the number of threads and errors out if it
    can not accomodate the memory limit.
    
    [2] Assumptions
    * We only support multi-threaded mode (threads >= 2), builds
      should not try to use xz in single-threaded mode
    * The thread limit should be set via XZ_THREADS, not via
      modifying XZ_DEFAULTS or XZ_OPTS, or any other way
    * The thread limit should not be set to xz's magic value
      zero (0), as that will lead to single-threaded mode on
      single-core systems.
    
    Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
---
 meta/conf/bitbake.conf | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 131ba29..4b544a2 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -795,7 +795,10 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
 PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
 
 # Default parallelism and resource usage for xz
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count(at_least=2)}"
+XZ_MEMLIMIT ?= "50%"
+XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
+XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
+XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 
 ##################################################################
 # Magic Cookie for SANITY CHECK

-- 
To stop receiving notification emails like this one, please contact
the administrator of this repository.