[OE-core] reproducible builds involving xz (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)

[Richard Purdie]

On Tue, 2020-02-25 at 11:16 +0000, André Draszik wrote:
> On Mon, 2020-02-24 at 17:10 +0200, Adrian Bunk wrote:
> > On Mon, Feb 24, 2020 at 02:58:21PM +0000, André Draszik wrote:
> > > On Mon, 2020-02-24 at 16:31 +0200, Adrian Bunk wrote:
> > > > On Mon, Feb 24, 2020 at 02:21:57PM +0000, André Draszik wrote:
> > > > > ...
> > > > > Once the artefact has been created, it doesn't matter anymore
> > > > > how it
> > > > > was created, as long as the high-level options aren't being
> > > > > changed (like
> > > > > an explicit compression level, checksum method, etc.)
> > > > > ...
> > > > 
> > > > meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"
> > > 
> > > I still maintain these are two different problems. I am not
> > > trying to fix
> > > reproducible builds.
> > 
> > But won't fixing reproducible builds require reverting the patch
> > you 
> > suggest for fixing your problem?
> 
> No, I don't think so.
> 
> To fix reproducible, you need to either patch xz to always behave
> like in
> multi-threaded mode, even when --threads=1 was given in the
> arguments.
> Additionally, a patch to xz to allow it to scale down the number of
> threads, but not the compression level (--no-adjust prevents both),
> could be useful.
> 
> Alternatively, you need to decide if you want to support it with
> --threads=1 or --threads >= 2, and issue at least a warning in
> reproducible_build.bbclass if somebody deviates.
> 
> 
> You should also calculate the memory requirements, reduce the number
> of threads to fit, and again give at least a warning if the
> compression
> can not be done without changing compression level after that.
> 
> I still maintain that none of that is related to my original patch.

If I merge your patch, what will happen is that builds on the
autobuilder can potentially start failing as it will potentially detect
reproduciblity issues which it wouldn't previously see.

To address this, I'd ike to see a new parameter added to
oe.utils.cpu_count() which allows us to have a minimum of 2. I'd also
like to reparametise the memlimit so we can drop it entirely by
default.

This should get us some defaults which won't break things and give
reproducibile builds. They may not work for everyone but its then
something which we can build upon.

I do want to take a patch as I agree there is a serious issue here but
I'd prefer we get it right and try and stop it causing me headaches on
the testing infrastructure.

Cheers,

Richard