<div dir="ltr"><div>I've seen that some of these patches were already merged to master.</div><div><br></div><div>Can you please resend remaining patches for oe-core? I'm testing some slightly older version of your patches and I see couple recipes failing e.g. like this one:</div><a href="http://errors.yoctoproject.org/Errors/Details/151939/">http://errors.yoctoproject.org/Errors/Details/151939/</a></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 9, 2017 at 7:48 PM, Juro Bystricky <span dir="ltr"><<a href="mailto:juro.bystricky@intel.com" target="_blank">juro.bystricky@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This patch-set contains basic changes needed in order to support building of<br>
reproducible bianries. The set containes the following patches:<br>
<br>
0001-reproducible_build.<wbr>bbclass-initial-support-for-<wbr>binar.patch<br>
0002-image-prelink.bbclass-<wbr>support-binary-<wbr>reproducibility.patch<br>
0003-rootfs-postcommands.<wbr>bbclass-support-binary-<wbr>reproduci.patch<br>
0004-busybox.inc-improve-<wbr>reproducibility.patch<br>
0005-image.bbclass-support-<wbr>binary-reproducibility.patch<br>
0006-cpio-provide-cpio-<wbr>replacement-native.patch<br>
0007-image_types.bbclass-<wbr>improve-cpio-image-<wbr>reproducibili.patch<br>
0008-python2.7-improve-<wbr>reproducibility.patch<br>
0009-python3-improve-<wbr>reproducibility.patch<br>
0010-kernel.bbclass-improve-<wbr>reproducibility.patch<br>
0011-poky-reproducible.conf-<wbr>Initial-version.patch<br>
<br>
Using this patch set while building core-image minimal (two clean builds, same<br>
machine/OS, same date, two different folders, at two different times) I got the<br>
following results:<br>
<br>
Same:<br>
<br>
core-image-minimal-initramfs-<wbr>qemux86<br>
bzImage-qemux86.bin<br>
vmlinux.gz-qemux86.bin<br>
(Some binaries i.e. ext4 differ, but the differnce is due to conversion to<br>
.ext4)<br>
<br>
Comparing Debian packages in tmp/deploy/deb:<br>
<br>
Same: 4005<br>
Different: 38<br>
Total: 4043<br>
<br>
(The remaining packages that still differ can be dealt with on an individual basis)<br>
<br>
<br>
Although the patches contain commit messages explaining the purpose and implementation,<br>
a somewhat more detailed description of selected patches seems prudent:<br>
<br>
0001-reproducible_build.<wbr>bbclass-initial-support-for-<wbr>binar.patch<br>
==============================<wbr>==============================<wbr>===<br>
<br>
This patch creates a new class "reproducible_build.bbclass",<br>
introducing two new variables:<br>
<br>
BUILD_REPRODUCIBLE_BINARIES: "0" (default) business as usual, "1" turn on various pieces of<br>
codes to improve reproducible builds<br>
<br>
REPRODUCIBLE_TIMESTAMP_ROOTFS: only used if BUILD_REPRODUCIBLE_BINARIES="<wbr>1".<br>
Catch-all timestamp for various rootfs files, pre-linker, etc. If needed, timestamps can<br>
be better granulated later on, right now we use a single value.<br>
<br>
Having a new variable BUILD_REPRODUCIBLE_BINARIES serves two purposes:<br>
1. Lets user decide (there are minor trade-offs)<br>
2. Setting to "0" will guarantee to cause zero regressions.<br>
3. Setting to "1" will force the the environment to contain SOURCE_DATE_EPOCH<br>
<br>
BUILD_REPRODUCIBLE_BINARIES is globally exported, as this will initially force all kinds<br>
of rebuilds. I know no simple way around this, though. This variable is needed in numerous<br>
places: configuration, compilation, rootfs creation, packaging etc.<br>
REPRODUCIBLE_TIMESTAMP_ROOTFS does not need to be globally exported, it is exported locally<br>
based on the need.<br>
Once these variables are "official", various classes and recipes can be modified to conditionally<br>
support binary reproducibility.<br>
<br>
Setting SOURCE_DATE_EPOCH is essential for binary reproducibility.<br>
We need to set a recipe specific SOURCE_DATE_EPOCH in each recipe environment for various tasks.<br>
One way would be to modify all recipes one-by-one, but that is not realistic. So determining<br>
SOURCE_DATE_EPOCH is done in this class automatically: After sources are unpacked (but<br>
before they are patched), we try to determine the value for SOURCE_DATE_EPOCH.<br>
<br>
There are 4 ways to determine SOURCE_DATE_EPOCH:<br>
1. Use value from src-data-epoch.txt file if this file exists. This file was most likely created<br>
in the previous build by one of the following methods 2,3,4.<br>
(But it could be actually provided by a recipe via SRC_URI)<br>
<br>
If the file does not exist:<br>
2. Use .git last commit date timestamp (git does not allow checking out files and preserving their<br>
timestamps)<br>
3. Use "known" files such as NEWS, CHANGLELOG, ...<br>
4. Use the youngest file of the source tree.<br>
<br>
Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe source tree in<br>
a text file "src-date-epoch.txt'.<br>
<br>
If this file is found by other recipe task, the value is placed in the SOURCE_DATE_EPOCH var in<br>
the task environment. This is done in an anonymous python function, so SOURCE_DATE_EPOCH is<br>
guaranteed to exist for all tasks. (If the file is not found SOURCE_DATE_EPOCH is set to 0)<br>
This can optimized in the future, as some tasks (all tasks before fetch, tasks such as package QA,<br>
rm_work, ...) do not need SOURCE_DATE_EPOCH in the environment.<br>
<br>
<br>
0008-python2.7-improve-<wbr>reproducibility.patch<br>
0009-python3-improve-<wbr>reproducibility.patch<br>
==============================<wbr>==============<br>
These are back ports of existing patches. They ensure the compiled .pyc files<br>
contain timestamp based on SOURCE_DATE_EPOCH (if defined in the environment).<br>
(May not be needed in the future, my understanding is support for SOURCE_DATE_EPOCH is already<br>
upstreamed in master)<br>
<br>
<br>
0010-kernel.bbclass-improve-<wbr>reproducibility.patch<br>
==============================<wbr>===================<br>
<br>
This patch contains several changes, was created by squashing several commits.<br>
Several tweaks to improve reproducibility:<br>
<br>
We want to set KBUILD_BUILD_TIMESTAMP to some reproducible value. Normally,<br>
we would use the value for SOURCE_DATE_EPOCH. However, to accommodate local kernel sources,<br>
these are not obtained the usual way via do_unpack and hHence we end up with<br>
SOURCE_DATE_EPOCH set to 0. In this case we obtain the timestamp from top entry of GIT repo,<br>
or (if there is no GIT repo) fallback to REPRODUCIBLE_TIMESTAMP_ROOTFS as the last resort.<br>
<br>
Kernel and kernel modules contain hard coded paths referencing the host<br>
build system. This is usually because the source code contains __FILE__<br>
at some place. This prevents binary reproducibility. However, some compilers<br>
allow remapping of the __FILE__ value. If we detect the compiler is capable<br>
of doing this, we replace the source path $(S) part of __FILE__ by a string "/kernel-source".<br>
This works very well for oe-embedded cross-compilers, but it is not guaranteed to work for<br>
external toolchains. Hence, the check for the option being supported. Note that this<br>
is done regardless of the value od BUILD_REPRODUCIBLE_BINARIES.<br>
<br>
When compressing vmlinux.gz, use gzip "-n" option as recommended in all guidelines to achieve<br>
binary reproducibility.<br>
<br>
<br>
0011-poky-reproducible.conf-<wbr>Initial-version.patch<br>
==============================<wbr>===================<br>
Support building of reproducible images by setting<br>
DISTRO="poky-reproducible"<br>
<br>
This is mostly for convenience so the user does not have to modify<br>
local.conf.<br>
<br>
Please note setting LDCONFIGDEPEND = ""<br>
This prevents building of ldconfig cache, which (when built) breaks binary<br>
reproducibility.<br>
<br>
Also, it should avoid reproducibility issue with etc/passwd, where for example<br>
two different builds can lead to two different values i.e:<br>
<br>
build 1:<br>
distcc:x:993:65534::/dev/null:<wbr>/bin/sh<br>
pulse:x:994:1001::/var/run/<wbr>pulse:/bin/false<br>
<br>
build 2:<br>
pulse:x:993:1001::/var/run/<wbr>pulse:/bin/false<br>
distcc:x:994:65534::/dev/null:<wbr>/bin/sh<br>
<br>
<br>
<br>
Juro Bystricky (11):<br>
reproducible_build.bbclass: initial support for binary reproducibility<br>
image-prelink.bbclass: support binary reproducibility<br>
rootfs-postcommands.bbclass: support binary reproducibility<br>
busybox.inc: improve reproducibility<br>
image.bbclass: support binary reproducibility<br>
cpio: provide cpio-replacement-native<br>
image_types.bbclass: improve cpio image reproducibility<br>
python2.7: improve reproducibility<br>
python3: improve reproducibility<br>
kernel.bbclass: improve reproducibility<br>
poky-reproducible.conf: Initial version<br>
<br>
meta-poky/conf/distro/include/<wbr>reproducible-group | 50 ++++++++++<br>
meta-poky/conf/distro/include/<wbr>reproducible-passwd | 25 +++++<br>
meta-poky/conf/distro/poky-<wbr>reproducible.conf | 38 ++++++++<br>
meta/classes/base.bbclass | 4 +<br>
meta/classes/image-prelink.<wbr>bbclass | 12 ++-<br>
meta/classes/image.bbclass | 16 ++-<br>
meta/classes/image_types.<wbr>bbclass | 14 ++-<br>
meta/classes/kernel.bbclass | 39 +++++++-<br>
meta/classes/reproducible_<wbr>build.bbclass | 108 +++++++++++++++++++++<br>
meta/classes/rootfs-<wbr>postcommands.bbclass | 27 +++++-<br>
meta/recipes-core/busybox/<wbr>busybox.inc | 7 ++<br>
.../python/<a href="http://python-native_2.7.13.bb" rel="noreferrer" target="_blank">python-native_2.7.<wbr>13.bb</a> | 1 +<br>
.../python/python/<wbr>reproducible.patch | 34 +++++++<br>
.../python/<a href="http://python3-native_3.5.3.bb" rel="noreferrer" target="_blank">python3-native_3.5.<wbr>3.bb</a> | 1 +<br>
.../support_SOURCE_DATE_EPOCH_<wbr>in_py_compile.patch | 97 ++++++++++++++++++<br>
meta/recipes-devtools/python/<a href="http://python3_3.5.3.bb" rel="noreferrer" target="_blank">p<wbr>ython3_3.5.3.bb</a> | 1 +<br>
meta/recipes-devtools/python/<a href="http://python_2.7.13.bb" rel="noreferrer" target="_blank">p<wbr>ython_2.7.13.bb</a> | 1 +<br>
meta/recipes-extended/cpio/<wbr>cpio_v2.inc | 2 +<br>
18 files changed, 467 insertions(+), 10 deletions(-)<br>
create mode 100644 meta-poky/conf/distro/include/<wbr>reproducible-group<br>
create mode 100644 meta-poky/conf/distro/include/<wbr>reproducible-passwd<br>
create mode 100644 meta-poky/conf/distro/poky-<wbr>reproducible.conf<br>
create mode 100644 meta/classes/reproducible_<wbr>build.bbclass<br>
create mode 100644 meta/recipes-devtools/python/<wbr>python/reproducible.patch<br>
create mode 100644 meta/recipes-devtools/python/<wbr>python3/support_SOURCE_DATE_<wbr>EPOCH_in_py_compile.patch<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
2.7.4<br>
<br>
</font></span></blockquote></div><br></div>