[OE-core] Overriding SDE_FILE

[Jacob Kroon]

On 2/28/20 2:39 PM, Jacob Kroon wrote:
> On 2/27/20 11:35 PM, Douglas Royds via Openembedded-core wrote:
>> On 28/02/20 11:34 am, Joshua Watt wrote:
>>
>>>
>>> On 2/27/20 4:29 PM, Douglas Royds wrote:
>>>>
>>>> On 28/02/20 10:49 am, Joshua Watt wrote:
>>>>
>>>>>
>>>>> On 2/27/20 3:22 PM, Douglas Royds wrote:
>>>>>>
>>>>>> On 28/02/20 5:45 am, Joshua Watt wrote:
>>>>>>
>>>>>>> On 2/27/20 9:01 AM, Joshua Watt wrote:
>>>>>>>> On 2/26/20 11:46 PM, Douglas Royds wrote:
>>>>>>>>>
>>>>>>>>> On 26/02/20 4:53 am, Jacob Kroon wrote:
>>>>>>>>>
>>>>>>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote:
>>>>>>>>>>> Hi Douglas,
>>>>>>>>>>>
>>>>>>>>>>> You updated a comment in reproducible_build.bbclass, commit 
>>>>>>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d:
>>>>>>>>>>>
>>>>>>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is 
>>>>>>>>>>>> stored in the recipe's ${SDE_FILE}.
>>>>>>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is 
>>>>>>>>>>>> stored in the recipe's SDE_FILE.
>>>>>>>>>>>> +# If none of these mechanisms are suitable, replace the 
>>>>>>>>>>>> do_deploy_source_date_epoch task
>>>>>>>>>>>> +# with recipe-specific functionality to write the 
>>>>>>>>>>>> appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
>>>>>>>>>>>> +#
>>>>>>>>>>>
>>>>>>>>>>> But I can't really get this to work. What did work for me was 
>>>>>>>>>>> to replace "do_create_source_date_epoch_stamp()" in my recipe:
>>>>>>>>>>>
>>>>>>>>>>> do_create_source_date_epoch_stamp() {
>>>>>>>>>>>      mkdir -p ${SDE_DIR}
>>>>>>>>>>>      date -d "1981-03-03" "+%s" > ${SDE_FILE}
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> What is the intended way to achieve the thing I'm trying to 
>>>>>>>>>>> do here ?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> FYI, JPEW has a proposed patch here
>>>>>>>>>>
>>>>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce 
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly.
>>>>>>>>>>
>>>>>>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH 
>>>>>>>>>> being in BB_HASHBASE_WHITELIST ?
>>>>>>>>>>
>>>>>>>>>> /Jacob
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On the surface of it, my comment appears to be just wrong: It 
>>>>>>>>> does make sense to replace do_create_source_date_epoch_stamp() 
>>>>>>>>> as you suggest.
>>>>>>>>>
>>>>>>>>> Joshua's proposed patch looks promising:
>>>>>>>>>
>>>>>>>>>   * Should the new function not be called first, so that it
>>>>>>>>>     takes priority over the git, known files, and youngest file
>>>>>>>>>     functions? If someone has explicitly set SOURCE_DATE_EPOCH,
>>>>>>>>>     then they want it to take priority.
>>>>>>>>>
>>>>>>>> Having that be the first option makes sense. The only case in 
>>>>>>>> which that might not work, is if a recipe does something like:
>>>>>>>>
>>>>>>>>  SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
>>>>>>>>
>>>>>>>> e.g. uses a function to get the SDE instead of setting to a 
>>>>>>>> fixed value, but that's probably going to be extremely rare.
>>>>>>>>
>>>>>>>>
>>>>>>>>>  *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>   * As you observe, SOURCE_DATE_EPOCH would need to be removed
>>>>>>>>>     from BB_HASHBASE_WHITELIST. I'm not sure why it was in the
>>>>>>>>>     whitelist in the first place.
>>>>>>>>>
>>>>>>>> I'm not sure why exactly it is whitelisted; I didn't write the 
>>>>>>>> original code that whitelisted it, but I've CC'd Juro in case he 
>>>>>>>> happens to remember.
>>>>>>>>
>>>>>>> After a discussion with Richard, we figured out why 
>>>>>>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the 
>>>>>>> variable *must* be calculable at parse time before any task is 
>>>>>>> ran, but in practice it's value is only available once the 
>>>>>>> __source_date_epoch.txt file is present, which is after parsing. 
>>>>>>> This causes the taskhash to be calculated differently during 
>>>>>>> parsing and task execution which causes taskhash mismatch errors.
>>>>>>>
>>>>>>
>>>>>> True.
>>>>>>
>>>>>> What to do? Would it work to use a different non-whitelisted 
>>>>>> variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?
>>>>>>
>>>>>
>>>>> Ya, that would work. You'd have to figure out how to get the 
>>>>> variable to be included in each taskhash even though it's not 
>>>>> directly referenced, but I'm sure that's possible.
>>>>>
>>>>>
>>>>> Another option that's at lot more "magic" would be something like 
>>>>> this: 
>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd 
>>>>>
>>>>>
>>>>> That's really weird, and I'm not sure I like it, but worth a thought.
>>>>>
>>>> If it works, it's good. I like the fact that the user can just set 
>>>> SOURCE_DATE_EPOCH directly in their recipe.
>>>>
>>>> How about ...
>>>>
>>>>     BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not
>>>>     source_date_epoch_var(d) else ''}"
>>>>
>>> Ya, that works. The only reason I chose "is None" was in case a user 
>>> really wanted to do
>>>
>>>   SOURCE_DATE_EPOCH = "0"
>>>
>> True, best to allow for that.
>>
> 
> Joshua, I just tried the two top patches in your reproducible branch; I 
> can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, however, 
> changing its value in my recipe doesn't trigger rebuilding for some 
> reason..

In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} in 
my do_install() task.