[OE-core] [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures
Adrian Bunk
bunk at stusta.de
Mon Feb 24 22:00:13 UTC 2020
On Mon, Feb 24, 2020 at 05:32:29PM +0000, Richard Purdie wrote:
> On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
>...
> > I don't want to penalise myself or anybody else by generally forcing
> > a low number of threads either.
>
> We didn't have resource issues on the autobuilder with xz but others
> did report it and its why the memlimit was set. It would be better to
> remove the memlimit for determinism really as far as I can tell and if
> necessary throttle the threads count.
>
> Maybe setting an upper limit of say 10 threads and a minimum of 2 might
> given us what we need.
On a Threadripper with 128 cores and 256 GB RAM it would not be a
problem to use all cores.
A laptop with 8 cores and 8 GB RAM is problematic.
> Perhaps we just allow the number of xz threads
> to be set independently?
dpkg manually reduces the number of threads until less than half
of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574
In a script it would be possible to use --no-adjust to achieve the same:
$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null
xz: Memory usage limit is too low for the given filter setup.
xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
$
> > > FWIW we haven't seen variation on the autobuilder due to this as
> > > far as
> > > I know.
> >
> > BTW, pigz and pbzip should have a similar thread related problem,
> > according to the man pages, if you read between the lines.
>
> Agreed. We have less of an issue with these as their use is much less
> frequent. xz is used by deb and ipk for packaging.
Good point, dpkg does parallel xz compression and I am not aware of any
reproducibility problems this causes.
> Cheers,
>
> Richard
cu
Adrian
More information about the Openembedded-core
mailing list