[OE-core] [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[André Draszik]

On Mon, 2020-02-24 at 16:44 +0000, Richard Purdie wrote:
> On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> > On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > > The number of threads used, and the amount of memory allowed
> > > to be used, should not affect sstate signatures, as they
> > > don't affect the result.
> > 
> > Unfortunately they can affect the result.
> 
> I looked into this a bit and its complicated. The threads are used to
> compress chunks and their compression should be deterministic whether
> done serially or in parallel.
> 
> I did some tests and:
> 
> xz <file>
> gave equivalent output to:
> xz <file> --threads=1
> 
> and
> 
> xz <file> --threads=2
> xz <file> --threads=5
> xz <file> --threads=50
> 
> all give different identical output.
> 
> So if we force --threads >=2 we should have determinism?

How large were your files? Given threaded operation works by operating in
'blocks', you should have a different number of blocks in your output.
They're all compressed independently, so I don't see how the result could
be identical, unless the block size is large enough for xz to not create
as many blocks as you allowed via setting the upper limit on the # of
threads.


> > > Otherwise, it becomes impossible to re-use sstate from
> > > automated builders on developer's machines (as the former
> > > might execute bitbake with certain constraints different
> > > compared to developer's machines).
> > > ...
> > > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
> > > ...
> > 
> > Threaded compression can result in slightly worse compression
> > than single-threaded compression.
> > 
> > With memlimit the problem is actually the opposite way,
> > and worse than what you were trying to fix:
> > 
> > When a developer hits memlimit during compression, the documented
> > behavour of xz is to scale down the compression level.
> > 
> > I assume 50% wrongly gives the same sstate signature no matter how
> > much RAM is installed on the local machine?
> 
> I did some tests locally and I could see different output checksums
> depending on how much memory I gave xz.
> 
> Perhaps we should specify a specific high amount like 1GB?

If I understand the man page right, at compression level 9 (default in OE),
each thread / block will use up to 674MiB of RAM, so a limit of 1GiB
effectively reduces parallelism to 1.


> Does anyone know more about the internals and how to have this behave
> "nicely" for our needs?

What are the needs specifically?

In my case, I have some defined builds that run on their own, so they
are allowed to use all memory and a certain (lowish) number of threads.

I also have other builds where several can run in parallel, and xz would
just be killed if all of them were allowed to use all RAM, so I have to
reduce threads and memory consumption.

This of course means slower builds...

I don't want to penalise myself or anybody else by generally forcing a
low number of threads either.

> FWIW we haven't seen variation on the autobuilder due to this as far as
> I know.

BTW, pigz and pbzip should have a similar thread related problem, according
to the man pages, if you read between the lines.

Files are split into chunks, and chunks are compressed individually in the
pbzip2 case.
pigz terminates each chunk with an empty block in the final .gz file.


Cheers,
Andre'