[OE-core] [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[André Draszik]

On Tue, 2020-02-25 at 00:00 +0200, Adrian Bunk wrote:
> On Mon, Feb 24, 2020 at 05:32:29PM +0000, Richard Purdie wrote:
> > On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
> > ...
> > > I don't want to penalise myself or anybody else by generally forcing
> > > a low number of threads either.
> > 
> > We didn't have resource issues on the autobuilder with xz but others
> > did report it and its why the memlimit was set. It would be better to
> > remove the memlimit for determinism really as far as I can tell and if
> > necessary throttle the threads count.
> > 
> > Maybe setting an upper limit of say 10 threads and a minimum of 2 might
> > given us what we need.
> 
> On a Threadripper with 128 cores and 256 GB RAM it would not be a
> problem to use all cores.
> 
> A laptop with 8 cores and 8 GB RAM is problematic.
> 
> > Perhaps we just allow the number of xz threads
> > to be set independently?
> 
> dpkg manually reduces the number of threads until less than half
> of the RAM is used:
> https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574
> 
> In a script it would be possible to use --no-adjust to achieve the same:
> $ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
> xz: Memory usage limit is too low for the given filter setup.
> xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
> $
> 

The problem with --no-adjust is that it also prevents xz from reducing the
number of threads. It will do that in preference to changing compression
parameters, so as long as you have more than 2499MiB of memory to support at
least 2 threads (with default block size), compression will be identical.

> > > > FWIW we haven't seen variation on the autobuilder due to this as
> > > > far as
> > > > I know.
> > > 
> > > BTW, pigz and pbzip should have a similar thread related problem,
> > > according to the man pages, if you read between the lines.
> > 
> > Agreed. We have less of an issue with these as their use is much less
> > frequent. xz is used by deb and ipk for packaging.

They both actually don't. It's reproducible no matter how many threads.
Interesting...

> 
> Good point, dpkg does parallel xz compression and I am not aware of any
> reproducibility problems this causes.

Debian's dpkg tries to be clever by simply reducing the number of threads until
compression uses at most half of physical
ram.

I guess this typically works out as at least two threads in most if not all
environments.

Cheers,
Andre'