[OE-core] [PATCH v2] classes/reproducible_build: Move SDE deploy to another directory

[Alex Kiernan]

On Fri, Sep 27, 2019 at 7:50 PM Joshua Watt <jpewhacker at gmail.com> wrote:
>
>
> On 9/27/19 1:03 PM, Alex Kiernan wrote:
> > On Thu, Sep 26, 2019 at 6:58 PM Joshua Watt <jpewhacker at gmail.com> wrote:
> >> The deployment of the source date epoch file had a race condition where
> >> any task attempting to read from the file would race with creation of
> >> the sstate archive for the do_deploy_source_date_epoch task. The
> >> creation of the sstate archive requires moving the directory to a
> >> temporary location, then moving it back. This means that the file
> >> disappears for a short period of time, which will cause a failure if any
> >> other task is running and trying to open the file to get the current
> >> source date epoch.
> >>
> >> The solution is to copy the source date epoch file to a separate
> >> directory when deploying so the file never disappears. When the file is
> >> restored from sstate, it is moved to the correct location after being
> >> extracted.
> >>
> >> [YOCTO #13501]
> >>
> >> Signed-off-by: Joshua Watt <JPEWhacker at gmail.com>
> >> ---
> >>   meta/classes/reproducible_build.bbclass | 12 ++++++++++--
> >>   1 file changed, 10 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
> >> index 8788ad7145c..99b749a9ee2 100644
> >> --- a/meta/classes/reproducible_build.bbclass
> >> +++ b/meta/classes/reproducible_build.bbclass
> >> @@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr
> >>
> >>   SDE_DIR ="${WORKDIR}/source-date-epoch"
> >>   SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
> >> +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
> >>
> >>   SSTATETASKS += "do_deploy_source_date_epoch"
> >>
> >>   do_deploy_source_date_epoch () {
> >>       echo "Deploying SDE to ${SDE_DIR}."
> >> +    mkdir -p ${SDE_DEPLOYDIR}
> >> +    if [ -e ${SDE_FILE} ]; then
> >> +        cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt
> >> +    fi
> >>   }
> >>
> >>   python do_deploy_source_date_epoch_setscene () {
> >>       sstate_setscene(d)
> >> +    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
> >> +    if os.path.exists(sde_file):
> >> +        os.rename(sde_file, d.getVar('SDE_FILE'))
> >>   }
> >>
> >> -do_deploy_source_date_epoch[dirs] = "${SDE_DIR}"
> >> -do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
> >> +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
> >> +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
> >>   addtask do_deploy_source_date_epoch_setscene
> >>   addtask do_deploy_source_date_epoch before do_configure after do_patch
> >>
> > This seems to have broken builds with reproducible builds enabled and
> > sstate downloaded from a mirror - I just get a blizzard of:
> >
> > ERROR: Logfile of failure stored in:
> > /home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
> > WARNING: Setscene task
> > (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene)
> > failed with exit code '1' - real task will be run instead
> > ERROR: autoconf-native-2.69-r11 do_deploy_source_date_epoch_setscene:
> > Error executing a python function in exec_python_func() autogenerated:
> >
> > The stack trace of python calls that resulted in this exception/failure was:
> > File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
> >       0001:
> >   *** 0002:do_deploy_source_date_epoch_setscene(d)
> >       0003:
> > File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
> > lineno: 58, function: do_deploy_source_date_epoch_setscene
> >       0054:python do_deploy_source_date_epoch_setscene () {
> >       0055:    sstate_setscene(d)
> >       0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
> > '__source_date_epoch.txt')
> >       0057:    if os.path.exists(sde_file):
> >   *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
> >       0059:}
> >       0060:
> >       0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
> >       0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
> > Exception: FileNotFoundError: [Errno 2] No such file or directory:
> > '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt'
> > -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'
>
> Sorry about that. It looks like destination directory isn't getting
> created. You can try giving
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/hash-equivalence&id=7177bb4c4c174f0852b984b9b2d2d23de92e70cd
> a try to see if it resolves the issue.
>

LGTM.

-- 
Alex Kiernan