[OE-core] [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Richard Purdie]

On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
> On Mon, 2020-02-24 at 16:44 +0000, Richard Purdie wrote:
> > On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> > > On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > > > The number of threads used, and the amount of memory allowed
> > > > to be used, should not affect sstate signatures, as they
> > > > don't affect the result.
> > > 
> > > Unfortunately they can affect the result.
> > 
> > I looked into this a bit and its complicated. The threads are used
> > to
> > compress chunks and their compression should be deterministic
> > whether
> > done serially or in parallel.
> > 
> > I did some tests and:
> > 
> > xz <file>
> > gave equivalent output to:
> > xz <file> --threads=1
> > 
> > and
> > 
> > xz <file> --threads=2
> > xz <file> --threads=5
> > xz <file> --threads=50
> > 
> > all give different identical output.
> > 
> > So if we force --threads >=2 we should have determinism?
> 
> How large were your files?

105MB total which should be enough as the largest xz block size is
32MB?

>  Given threaded operation works by operating in 'blocks', you should
> have a different number of blocks in your output.
> They're all compressed independently, so I don't see how the result
> could be identical, unless the block size is large enough for xz to
> not create as many blocks as you allowed via setting the upper limit
> on the # of threads.

I don't follow. The algorithm xz uses appears to be designed to give
consistent results regardless of numbers of threads. It gets split into
the same number of chunks. Each chunk is compressed independently, but
deterministically and then they're assembled in order. It doesn't
matter if the chunk is compressed in parallel or serially.

> > > > Otherwise, it becomes impossible to re-use sstate from
> > > > automated builders on developer's machines (as the former
> > > > might execute bitbake with certain constraints different
> > > > compared to developer's machines).
> > > > ...
> > > > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count
> > > > ()}"
> > > > > Files are split into chunks, and chunks are compressed
> > > > individually
> > > > > in the
> > > > > pbzip2 case.
> > > > > pigz terminates each chunk with an empty block in the final
> > > > .gz file.
> > > > > 
> > > > > 
> > > > > Cheers,
> > > > > Andre'
> > > > > 
> > > > > 
> > > > ...
> > > 
> > > Threaded compression can result in slightly worse compression
> > > than single-threaded compression.
> > > 
> > > With memlimit the problem is actually the opposite way,
> > > and worse than what you were trying to fix:
> > > 
> > > When a developer hits memlimit during compression, the documented
> > > behavour of xz is to scale down the compression level.
> > > 
> > > I assume 50% wrongly gives the same sstate signature no matter
> > > how
> > > much RAM is installed on the local machine?
> > 
> > I did some tests locally and I could see different output checksums
> > depending on how much memory I gave xz.
> > 
> > Perhaps we should specify a specific high amount like 1GB?
> 
> If I understand the man page right, at compression level 9 (default
> in OE),
> each thread / block will use up to 674MiB of RAM, so a limit of 1GiB
> effectively reduces parallelism to 1.

Right, I agree that isn't feasible.

> > Does anyone know more about the internals and how to have this
> > behave "nicely" for our needs?
> 
> What are the needs specifically?

The project is aiming to have deterministic builds, our output for
given input metadata is binary identical.

> In my case, I have some defined builds that run on their own, so they
> are allowed to use all memory and a certain (lowish) number of
> threads.
> 
> I also have other builds where several can run in parallel, and xz
> would just be killed if all of them were allowed to use all RAM, so I
> have to reduce threads and memory consumption.
> 
> This of course means slower builds...
> 
> I don't want to penalise myself or anybody else by generally forcing
> a low number of threads either.

We didn't have resource issues on the autobuilder with xz but others
did report it and its why the memlimit was set. It would be better to
remove the memlimit for determinism really as far as I can tell and if
necessary throttle the threads count.

Maybe setting an upper limit of say 10 threads and a minimum of 2 might
given us what we need. Perhaps we just allow the number of xz threads
to be set independently?

> > FWIW we haven't seen variation on the autobuilder due to this as
> > far as
> > I know.
> 
> BTW, pigz and pbzip should have a similar thread related problem,
> according to the man pages, if you read between the lines.

Agreed. We have less of an issue with these as their use is much less
frequent. xz is used by deb and ipk for packaging.

Cheers,

Richard