[OE-core] [PATCH v2] classes/reproducible_build: Move SDE deploy to another directory

[Alex Kiernan]

On Thu, Sep 26, 2019 at 6:58 PM Joshua Watt <jpewhacker at gmail.com> wrote:
>
> The deployment of the source date epoch file had a race condition where
> any task attempting to read from the file would race with creation of
> the sstate archive for the do_deploy_source_date_epoch task. The
> creation of the sstate archive requires moving the directory to a
> temporary location, then moving it back. This means that the file
> disappears for a short period of time, which will cause a failure if any
> other task is running and trying to open the file to get the current
> source date epoch.
>
> The solution is to copy the source date epoch file to a separate
> directory when deploying so the file never disappears. When the file is
> restored from sstate, it is moved to the correct location after being
> extracted.
>
> [YOCTO #13501]
>
> Signed-off-by: Joshua Watt <JPEWhacker at gmail.com>
> ---
>  meta/classes/reproducible_build.bbclass | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
> index 8788ad7145c..99b749a9ee2 100644
> --- a/meta/classes/reproducible_build.bbclass
> +++ b/meta/classes/reproducible_build.bbclass
> @@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr
>
>  SDE_DIR ="${WORKDIR}/source-date-epoch"
>  SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
> +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
>
>  SSTATETASKS += "do_deploy_source_date_epoch"
>
>  do_deploy_source_date_epoch () {
>      echo "Deploying SDE to ${SDE_DIR}."
> +    mkdir -p ${SDE_DEPLOYDIR}
> +    if [ -e ${SDE_FILE} ]; then
> +        cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt
> +    fi
>  }
>
>  python do_deploy_source_date_epoch_setscene () {
>      sstate_setscene(d)
> +    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
> +    if os.path.exists(sde_file):
> +        os.rename(sde_file, d.getVar('SDE_FILE'))
>  }
>
> -do_deploy_source_date_epoch[dirs] = "${SDE_DIR}"
> -do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
> +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
> +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
>  addtask do_deploy_source_date_epoch_setscene
>  addtask do_deploy_source_date_epoch before do_configure after do_patch
>

This seems to have broken builds with reproducible builds enabled and
sstate downloaded from a mirror - I just get a blizzard of:

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
WARNING: Setscene task
(virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene)
failed with exit code '1' - real task will be run instead
ERROR: autoconf-native-2.69-r11 do_deploy_source_date_epoch_setscene:
Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_deploy_source_date_epoch_setscene(d)
     0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
     0054:python do_deploy_source_date_epoch_setscene () {
     0055:    sstate_setscene(d)
     0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
     0057:    if os.path.exists(sde_file):
 *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
     0059:}
     0060:
     0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
     0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt'
-> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/temp/log.do_deploy_source_date_epoch_setscene.113869
WARNING: Setscene task
(virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/autoconf/autoconf_2.69.bb:do_deploy_source_date_epoch_setscene)
failed with exit code '1' - real task will be run instead
ERROR: m4-native-1.4.18-r0 do_deploy_source_date_epoch_setscene: Error
executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_deploy_source_date_epoch_setscene(d)
     0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
     0054:python do_deploy_source_date_epoch_setscene () {
     0055:    sstate_setscene(d)
     0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
     0057:    if os.path.exists(sde_file):
 *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
     0059:}
     0060:
     0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
     0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/deploy-source-date-epoch/__source_date_epoch.txt'
-> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/source-date-epoch/__source_date_epoch.txt'

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/temp/log.do_deploy_source_date_epoch_setscene.113897


-- 
Alex Kiernan