[OE-core] [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[André Draszik]

On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > The number of threads used, and the amount of memory allowed
> > to be used, should not affect sstate signatures, as they
> > don't affect the result.
> 
> Unfortunately they can affect the result.

I'm actually aware of that, and was contemplating on expanding on
that topic ... I guess I should have

> 
> > Otherwise, it becomes impossible to re-use sstate from
> > automated builders on developer's machines (as the former
> > might execute bitbake with certain constraints different
> > compared to developer's machines).
> > ...
> > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
> > ...
> 
> Threaded compression can result in slightly worse compression
> than single-threaded compression.
> 
> With memlimit the problem is actually the opposite way,
> and worse than what you were trying to fix:
> 
> When a developer hits memlimit during compression, the documented
> behavour of xz is to scale down the compression level.

I believe none of the above is relevant in the context here, though.
The whole point is to avoid running xz again (within
a different
environment) if artefacts are available already.

threads and memlimit in particular are meant to enable compressing on
resource constrained systems. Allowing sstate re-use no matter what
threading and memory limit was set seems sensible, even if a build-
machine has more ram than a deverloper's machine (or vice versa), as
the point here is to allow sstate re-use to speed up builds.
The point is not to change the compression, or to invalidate upon
slight variations due to different amount of RAM.

It's good in particular to re-use the sstate artefacts, rather than
creating new ones, just because the build happens to run on a different
machine.

Once the artefact has been created, it doesn't matter anymore how it
was created, as long as the high-level options aren't being changed (like
an explicit compression level, checksum method, etc.)


> I assume 50% wrongly gives the same sstate signature no matter how
> much RAM is installed on the local machine?

Likely, but a different problem...

Even before this patch, you will get differing results, but this patch
doesn't prevent you from being more strict and specifying --no-adjust
and it doesn't prevent you from changing the compression level as a whole
either.


Cheers,
Andre'