[OE-core] xz threads / memlimit behaviour (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)

[André Draszik]

On Mon, 2020-02-24 at 16:44 +0000, Richard Purdie wrote:
> On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> > On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > > The number of threads used, and the amount of memory allowed
> > > to be used, should not affect sstate signatures, as they
> > > don't affect the result.
> > 
> > Unfortunately they can affect the result.
> 
> I looked into this a bit and its complicated. The threads are used to
> compress chunks and their compression should be deterministic whether
> done serially or in parallel.
> 
> I did some tests and:
> 
> xz <file>
> gave equivalent output to:
> xz <file> --threads=1
> 
> and
> 
> xz <file> --threads=2
> xz <file> --threads=5
> xz <file> --threads=50
> 
> all give different identical output.
> 
> So if we force --threads >=2 we should have determinism?

I did another test...

Even single threaded compression gives the same result as multi-threaded,
if single threaded is a result of xz scaling down the memory usage due
to --memlimit, ie:

xz -f -c -9 --threads=2 --memlimit=2000M --keep --verbose --verbose xztest > xztest2_down_to_1.xz
  xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
  xz: Using up to 2 threads.
  xz: 2499 MiB of memory is required. The limit is 2000 MiB.
  xz: Decompression will need 65 MiB of memory.
  xz: Adjusted the number of threads from 2 to 1 to not exceed the memory usage limit of 2000 MiB

is different from

xz -f -c -9 --threads=1 --memlimit=2000M --verbose --verbose xztest > xztest1.xz
  xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
  xz: 674 MiB of memory is required. The limit is 2000 MiB.
  xz: Decompression will need 65 MiB of memory.


There even is a comment in the xz sources about this. This simplifies things a bit...


> > > Otherwise, it becomes impossible to re-use sstate from
> > > automated builders on developer's machines (as the former
> > > might execute bitbake with certain constraints different
> > > compared to developer's machines).
> > > ...
> > > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
> > > ...
> > 
> > Threaded compression can result in slightly worse compression
> > than single-threaded compression.
> > 
> > With memlimit the problem is actually the opposite way,
> > and worse than what you were trying to fix:
> > 
> > When a developer hits memlimit during compression, the documented
> > behavour of xz is to scale down the compression level.

This is not what is actually happening - it appears the documentation only
talks about single-threaded mode.
If you look at the code and look at command output, xz only ever
tries to reduce the number of threads if multi-threading was enabled. In
that case, it will *never* even attempt to change the compression level.

If it can't satisfy the memory limit by reducing the number of threads,
it just errors out.

So as long as all builders have at least 2 cores, i.e. --threads >= 2, memlimit
doesn't ever affect the outcome (except that it can make the operation fail as
a whole).


Cheers,
Andre'