[OE-core] The state of reproducible Builds

[Douglas Royds]

On 2/07/19 3:58 AM, Joshua Watt wrote:

> 1. Testing RPM and IPK package formats. I think RPMs will be pretty 
> easy; IPKs might be more challenging since AFAIK the tools that make 
> them don't generate reproducible output to begin with.


This has not been my experience. I have been building reproducible ipks, 
indeed, it is the hashsums of the ipks that I've been examining. In most 
cases, the correct SOURCE_DATE_EPOCH is enough, but there have been 
cases where I've had to correct upstream projects to cope with the 
SOURCE_DATE_EPOCH or avoid the effect of differing uname settings.


> 1. HOSTTOOLS differences. There are a lot of tools listed in 
> HOSTTOOLS, and unfortunately some of them have version dependent 
> output and are used for target builds (the one I've currently stumbled 
> upon is pod2man, but I'm sure there are others). Unfortunately, one 
> could probably argue that HOSTTOOLS is somewhat antithetical to the 
> above statement, at least in regard to target builds. Any host tool 
> output that "leaks" into the target build output can result in a 
> non-reproducible build across hosts, and possibly should be avoided; 
> the alternative is to use (or mandate) the corresponding -native 
> recipe that provides that tool as a DEPENDS so that the controlled 
> internally built version is used instead. Note that this only really 
> applies target builds, not -native (or nativesdk right now). -native 
> recipes would obviously need more HOSTTOOLS to help bootstrap the 
> system. I suspect this would require reworking how HOSTOOLS works so 
> that they can be split into two categories somehow; the tools that 
> have "ubiquitous and stable" interfaces and are fine for all recipes 
> (e.g. cat, sed, true, rm, etc.) and those that are variable and should 
> only be used for -native builds (e.g. pod2man, rpcgen(?), chrpath(?), 
> tar(?)... others?). Anyone have thoughts on this?


Perhaps reproducibility is the decision-point for adding a tool to the 
HOSTTOOLS: If the precise version of the tool has no impact on 
reproducibility (eg. cat, sed, and even gawk), it is a good candidate 
for the HOSTTOOLS. pod2man shouldn't be in the HOSTTOOLS, because we 
need to control the version.