[OE-core] Reproducible Build Status

[Richard Purdie]

I think we're now down to:

sudo path issues (Ross is poking)
ncurses-doc issue (assigned to JPEW)
perl host compiler flags issue 
sysvinit missing group problem (assigned to Mingli Yu at present)

(thanks Alex for spotting the obvious I couldn't see with one of the
perl issues!)

I wanted to document some of the interesting quirks we've seen with
reproducibile build failures in case anyone else runs into these kinds
of issues.

First is the "embedded git hash" problem where a source tree embeds the
output of "git describe" or the hash from a git command, even though
the source has come from an upstream tarball.

These commands travel back up the directory structure and find the 
hash or description (e.g. 3.1M1-XXXX) for our own metadata (poky or OE-
Core). These are consistent as long as the same revision of the
metadata is built. As soon as there are new commits or a rebase, they
change and trigger the reproducible builds failure. This means they
usually pass for master-next and break on master.

Second is setcap issues. A number of recipes look for setcap on the
host system and then change behaviour depending on whether they find
it. Some of our workers have it as part of their default install some
don't. Some recipes fall back to suid bits if setcap isn't present too
just to confuse things.

Third is the weird group problem for sysvinit. In this case its because
the do_package_write_ipk was running against do_package from sstate and
that gave different output to a full build due to the differing sysroot
contents (missing etc/group file). The reproducer is simple once you
realise what is breaking:

bitbake sysvinit
bitbake sysvinit -c clean
bitbake sysvinit -c package_write_ipk -f

Were also seeing a smattering of more well known issues like hardcoded
hosttool paths (/bin/mv vs /usr/bin/mv), dash vs bash, sorting issues
in generated header/C files and some host variable infection problems.

Cheers,

Richard