[OE-core] [PATCH 0/1] Initial QA test for reproducible builds

[Khem Raj]

Hi Joshua

Thanks for contributing this will provide some teeth to reproducible 
builds QA

On 5/20/19 9:57 AM, Joshua Watt wrote:
> Implements an initial QA check for reproducible builds. This check is
> sufficient for an initial implementation, and will catch a wide variety
> of reproducible problems, but it does have the following problems:
> 
>   1) It doesn't pass. Currently, about 800 packages fail to build
>      in a reproducible manner for core-image-minimal. I've found two
>      major sources of non-reproducibility so far:
>       a) The perl-module packages don't have a consistent
>          SOURCE_DATE_EPOCH which means when they are packaged the
>          timestamps on all the files are different. Thankfully, this
>          accounts for several hundred of the packages, so fixing this
>          should remove a lot of the failures

maybe we can start with inhriting reproducible_build_simple which has 
hardcoded values for SOURCE_DATE_EPOCH

>       b) Debug package strings aren't consistent. It appears that in some
>          of the -dbg packages, the linker changes the order of the merged
>          .debug_strings section. This trickles down into the packages
>          that contain the executables because it changes the hash the
>          executable contains to ensure the debug symbols match up.
> 

try adding -fno-merge-debug-strings to linker and see if that fixes this 
problem. If that happens then we know its an option to add when doing 
reproducible builds.

>   2) It's not easy to debug issues when there are reproducibility
>      problems. I had initially intended to run diffoscope on the
>      resulting files but this takes much longer than I think we are
>      willing to run on the autobuilder and also generates far too much
>      output to be really useful. I think a better long term route is to
>      have the test dump the list of non-reproducible packages and then
>      write a helper script that can consumer this list, allow the user to
>      select a package, then run diffoscope to examine it.

I think that might be needed to wrap diffoscope.

> 
>   3) This test currently is incomplete and won't catch all classes of
>      reproducibility problems. At the least, I know that it won't
>      consistently catch the use of the __DATE__ macro in source code,
>      since that requires the builds to be done on two separate dates (on
>      the other hand, use of __TIME__ will be caught pretty reliably since
>      the builds are done serially). I suspect the correct solution to
>      this is to borrow from Debian and use something like faketime to
>      fake out the system time to some suitable future date when doing the
>      test build, but this will require some though to how it should be
>      implemented.
> 
>   4) It currently only tests Debian packages and core-image-minimal. The
>      test case has support for building the other package formats and
>      other images at the same time, the idea being that the long step in
>      this test is building everything from scratch, and building multiple
>      package formats and images at the same time will be much faster
>      overall than having multiple tests that have to do from-scratch
>      builds (although, there might be a way to serialize multiple tests
>      and have them share the test build TMPDIR). Until at least 1 package
>      format and image are passing, I don't see a huge motivation to
>      enable more.

why does it have to depend on packaging backend ?

> 
> Joshua Watt (1):
>    oeqa: Add reproducible build selftest
> 
>   meta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++
>   1 file changed, 159 insertions(+)
>   create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py
>