[OE-core] [PATCH v2] classes/reproducible_build: Move SDE deploy to another directory

[Joshua Watt]

On 9/27/19 1:03 PM, Alex Kiernan wrote:
> On Thu, Sep 26, 2019 at 6:58 PM Joshua Watt <jpewhacker at gmail.com> wrote:
>> The deployment of the source date epoch file had a race condition where
>> any task attempting to read from the file would race with creation of
>> the sstate archive for the do_deploy_source_date_epoch task. The
>> creation of the sstate archive requires moving the directory to a
>> temporary location, then moving it back. This means that the file
>> disappears for a short period of time, which will cause a failure if any
>> other task is running and trying to open the file to get the current
>> source date epoch.
>>
>> The solution is to copy the source date epoch file to a separate
>> directory when deploying so the file never disappears. When the file is
>> restored from sstate, it is moved to the correct location after being
>> extracted.
>>
>> [YOCTO #13501]
>>
>> Signed-off-by: Joshua Watt <JPEWhacker at gmail.com>
>> ---
>>   meta/classes/reproducible_build.bbclass | 12 ++++++++++--
>>   1 file changed, 10 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
>> index 8788ad7145c..99b749a9ee2 100644
>> --- a/meta/classes/reproducible_build.bbclass
>> +++ b/meta/classes/reproducible_build.bbclass
>> @@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr
>>
>>   SDE_DIR ="${WORKDIR}/source-date-epoch"
>>   SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
>> +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
>>
>>   SSTATETASKS += "do_deploy_source_date_epoch"
>>
>>   do_deploy_source_date_epoch () {
>>       echo "Deploying SDE to ${SDE_DIR}."
>> +    mkdir -p ${SDE_DEPLOYDIR}
>> +    if [ -e ${SDE_FILE} ]; then
>> +        cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt
>> +    fi
>>   }
>>
>>   python do_deploy_source_date_epoch_setscene () {
>>       sstate_setscene(d)
>> +    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
>> +    if os.path.exists(sde_file):
>> +        os.rename(sde_file, d.getVar('SDE_FILE'))
>>   }
>>
>> -do_deploy_source_date_epoch[dirs] = "${SDE_DIR}"
>> -do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
>> +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
>> +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
>>   addtask do_deploy_source_date_epoch_setscene
>>   addtask do_deploy_source_date_epoch before do_configure after do_patch
>>
> This seems to have broken builds with reproducible builds enabled and
> sstate downloaded from a mirror - I just get a blizzard of:
>
> ERROR: Logfile of failure stored in:
> /home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
> WARNING: Setscene task
> (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene)
> failed with exit code '1' - real task will be run instead
> ERROR: autoconf-native-2.69-r11 do_deploy_source_date_epoch_setscene:
> Error executing a python function in exec_python_func() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>       0001:
>   *** 0002:do_deploy_source_date_epoch_setscene(d)
>       0003:
> File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
> lineno: 58, function: do_deploy_source_date_epoch_setscene
>       0054:python do_deploy_source_date_epoch_setscene () {
>       0055:    sstate_setscene(d)
>       0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
> '__source_date_epoch.txt')
>       0057:    if os.path.exists(sde_file):
>   *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
>       0059:}
>       0060:
>       0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
>       0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
> Exception: FileNotFoundError: [Errno 2] No such file or directory:
> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt'
> -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'

Sorry about that. It looks like destination directory isn't getting 
created. You can try giving 
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/hash-equivalence&id=7177bb4c4c174f0852b984b9b2d2d23de92e70cd 
a try to see if it resolves the issue.


>
> ERROR: Logfile of failure stored in:
> /home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/temp/log.do_deploy_source_date_epoch_setscene.113869
> WARNING: Setscene task
> (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/autoconf/autoconf_2.69.bb:do_deploy_source_date_epoch_setscene)
> failed with exit code '1' - real task will be run instead
> ERROR: m4-native-1.4.18-r0 do_deploy_source_date_epoch_setscene: Error
> executing a python function in exec_python_func() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>       0001:
>   *** 0002:do_deploy_source_date_epoch_setscene(d)
>       0003:
> File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
> lineno: 58, function: do_deploy_source_date_epoch_setscene
>       0054:python do_deploy_source_date_epoch_setscene () {
>       0055:    sstate_setscene(d)
>       0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
> '__source_date_epoch.txt')
>       0057:    if os.path.exists(sde_file):
>   *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
>       0059:}
>       0060:
>       0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
>       0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
> Exception: FileNotFoundError: [Errno 2] No such file or directory:
> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/deploy-source-date-epoch/__source_date_epoch.txt'
> -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/source-date-epoch/__source_date_epoch.txt'
>
> ERROR: Logfile of failure stored in:
> /home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/temp/log.do_deploy_source_date_epoch_setscene.113897
>
>