[OE-core] [PATCH v3 01/11] reproducible_build.bbclass: initial support for binary reproducibility
Martin Jansa
martin.jansa at gmail.com
Sat Aug 19 08:25:28 UTC 2017
Would it make sense to create src_date_epoch.txt outside ${S} directory?
In some recipes (which weren't picky what they install from ${S} to ${D})
I've noticed that src_date_epoch.txt is now included in their package (I've
noticed, because 2 different packages were trying to install this file on
the same path in rootfs).
Can we use temp or ${WORKDIR} for this file? ${S} might be also be
completely archived with archiver.bbclass, not sure if you want
src_date_epoch.txt part of that or not.
Regards,
On Wed, Aug 9, 2017 at 7:48 PM, Juro Bystricky <juro.bystricky at intel.com>
wrote:
> Conditionally set some environment variables in order to achieve
> improved binary reproducibility. Providing BUILD_REPRODUCIBLE_BINARIES is
> set to "1", we set the following environment variables:
>
> export PYTHONHASHSEED=0
> export PERL_HASH_SEED=0
> export TZ="UTC"
>
> Additionally, we export and set SOURCE_DATE_EPOCH. This is the most
> crucial step to
> achieve binary reproducibility. The value for this variable (timestamp) is
> obtained after source code for a recipe has been unpacked, but before it
> is patched.
> If the code sources come from a GIT repo, we get the timestamp from the top
> commit. (GIT repo does not preserve file mktime timestamps). It is not
> safe to assume
> folders named "git" contain git repositories, so we check for presence of
> .git folder in ${S}.
> Otherwise, if GIT repo is not present, we get mtime from known files suche
> as NEWS,
> ChangeLog, etc. If this fails, we go through all files and get the
> timestamp
> from the youngest one. We create an individual timestamp for each recipe.
> The timestamp is stored in the file 'src_date_epoch.txt'. Later on, each
> task
> reads this file and sets SOURCE_DATE_EPOCH based on the value found in the
> file.
>
> The file src_date_epoch.txt file is re-used if found. This can be the file
> we previously created ourselves, or it can be a file provided by a user via
> a recipe.
>
> [YOCTO#11178]
> [YOCTO#11179]
>
> Signed-off-by: Juro Bystricky <juro.bystricky at intel.com>
> ---
> meta/classes/base.bbclass | 4 ++
> meta/classes/reproducible_build.bbclass | 108
> ++++++++++++++++++++++++++++++++
> 2 files changed, 112 insertions(+)
> create mode 100644 meta/classes/reproducible_build.bbclass
>
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 8c86977..bf79eb9 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -9,6 +9,7 @@ inherit utils
> inherit utility-tasks
> inherit metadata_scm
> inherit logging
> +inherit reproducible_build
>
> OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package
> oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license"
> OE_IMPORTS[type] = "list"
> @@ -166,6 +167,7 @@ python base_do_unpack() {
> try:
> fetcher = bb.fetch2.Fetch(src_uri, d)
> fetcher.unpack(d.getVar('WORKDIR'))
> + create_src_date_epoch_stamp(d)
> except bb.fetch2.BBFetchException as e:
> bb.fatal(str(e))
> }
> @@ -386,6 +388,7 @@ def set_packagetriplet(d):
>
> settriplet(d, "PKGMLTRIPLETS", archs, tos, tvs)
>
> +
> python () {
> import string, re
>
> @@ -685,6 +688,7 @@ python () {
> bb.warn("Recipe %s is marked as only being architecture
> specific but seems to have machine specific packages?! The recipe may as
> well mark itself as machine specific directly." % d.getVar("PN"))
> }
>
> +
> addtask cleansstate after do_clean
> python do_cleansstate() {
> sstate_clean_cachefiles(d)
> diff --git a/meta/classes/reproducible_build.bbclass
> b/meta/classes/reproducible_build.bbclass
> new file mode 100644
> index 0000000..af8db95
> --- /dev/null
> +++ b/meta/classes/reproducible_build.bbclass
> @@ -0,0 +1,108 @@
> +
> +BUILD_REPRODUCIBLE_BINARIES ??= "0"
> +BUILD_REPRODUCIBLE_BINARIES[export] = "1"
> +
> +# Unix timestamp
> +REPRODUCIBLE_TIMESTAMP_ROOTFS ??= ""
> +
> +def get_src_date_epoch_quick(d, path):
> + import subprocess
> + src_date_epoch = 0
> + saved_cwd = os.getcwd()
> + os.chdir(path)
> + if os.path.isdir(".git"):
> + try:
> + src_date_epoch = int(subprocess.check_output(['
> git','log','-1','--pretty=%ct']))
> + except subprocess.CalledProcessError as grepexc:
> + bb.warn("Not a git repository in .git folder? error:%d" %
> (grepexc.returncode))
> + else:
> + known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
> +
> + for file in known_files:
> + if os.path.isfile(file):
> + mtime = int(os.path.getmtime(file))
> +
> + # There may be more than one "known_file" present.
> + # If so, use the youngest one
> + if mtime > src_date_epoch:
> + src_date_epoch = mtime
> +
> + os.chdir(saved_cwd)
> + return src_date_epoch
> +
> +
> +def create_src_date_epoch_stamp(d):
> + if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
> + path = d.getVar('S')
> +
> + epochfile = os.path.join(path,'src_date_epoch.txt')
> + if os.path.isfile(epochfile):
> + bb.debug(1, " path: %s reusing src_date_epoch.txt" %
> epochfile)
> + return
> +
> + filename_dbg = None
> + src_date_epoch = get_src_date_epoch_quick(d, path)
> +
> + if src_date_epoch == 0:
> + exclude = set(["temp", "licenses", "patches",
> "recipe-sysroot-native", "recipe-sysroot", "pseudo"])
> + for root, dirs, files in os.walk(path, topdown=True):
> + files = [f for f in files if not f[0] == '.']
> + dirs[:] = [d for d in dirs if d not in exclude]
> +
> + for fname in files:
> + filename = os.path.join(root, fname)
> + try:
> + mtime = int(os.path.getmtime(filename))
> + except:
> + mtime = 0
> + if mtime > src_date_epoch:
> + src_date_epoch = mtime
> + filename_dbg = filename
> +
> + # Most likely an empty folder
> + if src_date_epoch == 0:
> + bb.warn("Unable to determine src_date_epoch! path:%s" % path)
> +
> + f = open(os.path.join(path,'src_date_epoch.txt'), 'w')
> + f.write(str(src_date_epoch))
> + f.close()
> +
> + if filename_dbg != None:
> + bb.debug(1," src_date_epoch %d derived from: %s" %
> (src_date_epoch, filename_dbg))
> + #bb.warn(" src_date_epoch %d derived from: %s" %
> (src_date_epoch, filename_dbg))
> +
> +export PYTHONHASHSEED
> +export PERL_HASH_SEED
> +export SOURCE_DATE_EPOCH
> +
> +BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH PYTHONHASHSEED PERL_HASH_SEED
> "
> +
> +python () {
> + import string, re
> +
> + # Create reproducible_environment
> +
> + if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
> + import subprocess
> + d.setVar('PYTHONHASHSEED', '0')
> + d.setVar('PERL_HASH_SEED', '0')
> + d.setVar('TZ', 'UTC')
> + path = d.getVar('S')
> + epochfile = os.path.join(path,'src_date_epoch.txt')
> + if os.path.isfile(epochfile):
> + f = open(epochfile, 'r')
> + src_date_epoch = f.read()
> + f.close()
> + bb.debug(1, "src_date_epoch stamp found ---> stamp %s" %
> src_date_epoch)
> + d.setVar('SOURCE_DATE_EPOCH', src_date_epoch)
> + else:
> + bb.debug(1, "src_date_epoch stamp not found.")
> + d.setVar('SOURCE_DATE_EPOCH', '0')
> + else:
> + if 'PYTHONHASHSEED' in os.environ:
> + del os.environ['PYTHONHASHSEED']
> + if 'PERL_HASH_SEED' in os.environ:
> + del os.environ['PERL_HASH_SEED']
> + if 'SOURCE_DATE_EPOCH' in os.environ:
> + del os.environ['SOURCE_DATE_EPOCH']
> +}
> --
> 2.7.4
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20170819/af892ba5/attachment-0002.html>
More information about the Openembedded-core
mailing list