[OE-core] [PATCH 0/1] Initial QA test for reproducible builds

[Joshua Watt]

On Mon, 2019-05-20 at 10:47 -0700, Khem Raj wrote:
> Hi Joshua
> 
> Thanks for contributing this will provide some teeth to reproducible 
> builds QA
> 
> On 5/20/19 9:57 AM, Joshua Watt wrote:
> > Implements an initial QA check for reproducible builds. This check
> > is
> > sufficient for an initial implementation, and will catch a wide
> > variety
> > of reproducible problems, but it does have the following problems:
> > 
> >   1) It doesn't pass. Currently, about 800 packages fail to build
> >      in a reproducible manner for core-image-minimal. I've found
> > two
> >      major sources of non-reproducibility so far:
> >       a) The perl-module packages don't have a consistent
> >          SOURCE_DATE_EPOCH which means when they are packaged the
> >          timestamps on all the files are different. Thankfully,
> > this
> >          accounts for several hundred of the packages, so fixing
> > this
> >          should remove a lot of the failures
> 
> maybe we can start with inhriting reproducible_build_simple which
> has 
> hardcoded values for SOURCE_DATE_EPOCH

reproducible_build.bbclass automatically inherits
reproducible_build_simple if BUILD_REPRODUCIBLE_BINARIES is set, and it
sets that by default.

I'm not actually sure how the value of SOURCE_DATE_EPOCH set in
reproducible_build_simple.bbclass matters at all. AFAIK,
reproducible_build.bbclass tries *really* hard to get a discernible
date from the source code repository itself after do_unpack, and
failing that uses 0. Near as I can tell, the value set in the class or
by the user only matters until the end of do_unpack.

I think something happens to be broken with this heuristic in the perl
recipe. I haven't tracked it down just yet.

> 
> >       b) Debug package strings aren't consistent. It appears that
> > in some
> >          of the -dbg packages, the linker changes the order of the
> > merged
> >          .debug_strings section. This trickles down into the
> > packages
> >          that contain the executables because it changes the hash
> > the
> >          executable contains to ensure the debug symbols match up.
> > 
> 
> try adding -fno-merge-debug-strings to linker and see if that fixes
> this 
> problem. If that happens then we know its an option to add when
> doing 
> reproducible builds.

Excellent. I will try that.

> 
> >   2) It's not easy to debug issues when there are reproducibility
> >      problems. I had initially intended to run diffoscope on the
> >      resulting files but this takes much longer than I think we are
> >      willing to run on the autobuilder and also generates far too
> > much
> >      output to be really useful. I think a better long term route
> > is to
> >      have the test dump the list of non-reproducible packages and
> > then
> >      write a helper script that can consumer this list, allow the
> > user to
> >      select a package, then run diffoscope to examine it.
> 
> I think that might be needed to wrap diffoscope.

I'm not sure I quite follow what you are saying here?

> 
> >   3) This test currently is incomplete and won't catch all classes
> > of
> >      reproducibility problems. At the least, I know that it won't
> >      consistently catch the use of the __DATE__ macro in source
> > code,
> >      since that requires the builds to be done on two separate
> > dates (on
> >      the other hand, use of __TIME__ will be caught pretty reliably
> > since
> >      the builds are done serially). I suspect the correct solution
> > to
> >      this is to borrow from Debian and use something like faketime
> > to
> >      fake out the system time to some suitable future date when
> > doing the
> >      test build, but this will require some though to how it should
> > be
> >      implemented.
> > 
> >   4) It currently only tests Debian packages and core-image-
> > minimal. The
> >      test case has support for building the other package formats
> > and
> >      other images at the same time, the idea being that the long
> > step in
> >      this test is building everything from scratch, and building
> > multiple
> >      package formats and images at the same time will be much
> > faster
> >      overall than having multiple tests that have to do from-
> > scratch
> >      builds (although, there might be a way to serialize multiple
> > tests
> >      and have them share the test build TMPDIR). Until at least 1
> > package
> >      format and image are passing, I don't see a huge motivation to
> >      enable more.
> 
> why does it have to depend on packaging backend ?

It doesn't particularly. Comparing the end packages is just the easiest
way to check for reproducibility (I think?). I originally tried
comparing sstate tarballs, but I quickly realized that was going to be
very difficult without making a lot of changes to the way sstate
tarballs are generated (e.g. timestamps and such).

On the other end of the spectrum would be to compare the final image
files themselves; I like that idea and I think we *should* do that
after we get the packages to be reproducible, but it's not very useful
right now because it's hard(er) to track a difference in the image back
to a recipe than it is to track a difference in a package back.

You could leave the test up to whatever package classes are enabled on
the autobuilder, but I think this is less than ideal right now because:

 1) IPKs aren't reproducible *at all*... they are basically all
different. Perhaps because of timestamps? Maybe that's easy fix?

 2) The long pull on this QA is the test build from scratch... it takes
quite a while and will take even longer with a bigger image.
Comparatively, generating multiple packages classes doesn't add that
much more time to the build, so I *think* it might make sense in the
long run to have the test build all the package formats in one go, or
at least being able to share the test build TMPDIR between multiple
serial tests.

> 
> > Joshua Watt (1):
> >    oeqa: Add reproducible build selftest
> > 
> >   meta/lib/oeqa/selftest/cases/reproducible.py | 159
> > +++++++++++++++++++
> >   1 file changed, 159 insertions(+)
> >   create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py
> > 
-- 
Joshua Watt <JPEWhacker at gmail.com>