[OE-core] Overriding SDE_FILE

Fri Feb 28 14:32:04 UTC 2020

On 2/28/20 3:18 PM, Joshua Watt wrote:
> 
> On 2/28/20 7:41 AM, Jacob Kroon wrote:
>> On 2/28/20 2:39 PM, Jacob Kroon wrote:
>>> On 2/27/20 11:35 PM, Douglas Royds via Openembedded-core wrote:
>>>> On 28/02/20 11:34 am, Joshua Watt wrote:
>>>>
>>>>>
>>>>> On 2/27/20 4:29 PM, Douglas Royds wrote:
>>>>>>
>>>>>> On 28/02/20 10:49 am, Joshua Watt wrote:
>>>>>>
>>>>>>>
>>>>>>> On 2/27/20 3:22 PM, Douglas Royds wrote:
>>>>>>>>
>>>>>>>> On 28/02/20 5:45 am, Joshua Watt wrote:
>>>>>>>>
>>>>>>>>> On 2/27/20 9:01 AM, Joshua Watt wrote:
>>>>>>>>>> On 2/26/20 11:46 PM, Douglas Royds wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 26/02/20 4:53 am, Jacob Kroon wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote:
>>>>>>>>>>>>> Hi Douglas,
>>>>>>>>>>>>>
>>>>>>>>>>>>> You updated a comment in reproducible_build.bbclass, commit 
>>>>>>>>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it 
>>>>>>>>>>>>>> is stored in the recipe's ${SDE_FILE}.
>>>>>>>>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it 
>>>>>>>>>>>>>> is stored in the recipe's SDE_FILE.
>>>>>>>>>>>>>> +# If none of these mechanisms are suitable, replace the 
>>>>>>>>>>>>>> do_deploy_source_date_epoch task
>>>>>>>>>>>>>> +# with recipe-specific functionality to write the 
>>>>>>>>>>>>>> appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
>>>>>>>>>>>>>> +#
>>>>>>>>>>>>>
>>>>>>>>>>>>> But I can't really get this to work. What did work for me 
>>>>>>>>>>>>> was to replace "do_create_source_date_epoch_stamp()" in my 
>>>>>>>>>>>>> recipe:
>>>>>>>>>>>>>
>>>>>>>>>>>>> do_create_source_date_epoch_stamp() {
>>>>>>>>>>>>>      mkdir -p ${SDE_DIR}
>>>>>>>>>>>>>      date -d "1981-03-03" "+%s" > ${SDE_FILE}
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is the intended way to achieve the thing I'm trying to 
>>>>>>>>>>>>> do here ?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> FYI, JPEW has a proposed patch here
>>>>>>>>>>>>
>>>>>>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce 
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly.
>>>>>>>>>>>>
>>>>>>>>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH 
>>>>>>>>>>>> being in BB_HASHBASE_WHITELIST ?
>>>>>>>>>>>>
>>>>>>>>>>>> /Jacob
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On the surface of it, my comment appears to be just wrong: It 
>>>>>>>>>>> does make sense to replace 
>>>>>>>>>>> do_create_source_date_epoch_stamp() as you suggest.
>>>>>>>>>>>
>>>>>>>>>>> Joshua's proposed patch looks promising:
>>>>>>>>>>>
>>>>>>>>>>>   * Should the new function not be called first, so that it
>>>>>>>>>>>     takes priority over the git, known files, and youngest file
>>>>>>>>>>>     functions? If someone has explicitly set SOURCE_DATE_EPOCH,
>>>>>>>>>>>     then they want it to take priority.
>>>>>>>>>>>
>>>>>>>>>> Having that be the first option makes sense. The only case in 
>>>>>>>>>> which that might not work, is if a recipe does something like:
>>>>>>>>>>
>>>>>>>>>>  SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
>>>>>>>>>>
>>>>>>>>>> e.g. uses a function to get the SDE instead of setting to a 
>>>>>>>>>> fixed value, but that's probably going to be extremely rare.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>  *
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>   * As you observe, SOURCE_DATE_EPOCH would need to be removed
>>>>>>>>>>>     from BB_HASHBASE_WHITELIST. I'm not sure why it was in the
>>>>>>>>>>>     whitelist in the first place.
>>>>>>>>>>>
>>>>>>>>>> I'm not sure why exactly it is whitelisted; I didn't write the 
>>>>>>>>>> original code that whitelisted it, but I've CC'd Juro in case 
>>>>>>>>>> he happens to remember.
>>>>>>>>>>
>>>>>>>>> After a discussion with Richard, we figured out why 
>>>>>>>>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the 
>>>>>>>>> variable *must* be calculable at parse time before any task is 
>>>>>>>>> ran, but in practice it's value is only available once the 
>>>>>>>>> __source_date_epoch.txt file is present, which is after 
>>>>>>>>> parsing. This causes the taskhash to be calculated differently 
>>>>>>>>> during parsing and task execution which causes taskhash 
>>>>>>>>> mismatch errors.
>>>>>>>>>
>>>>>>>>
>>>>>>>> True.
>>>>>>>>
>>>>>>>> What to do? Would it work to use a different non-whitelisted 
>>>>>>>> variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?
>>>>>>>>
>>>>>>>
>>>>>>> Ya, that would work. You'd have to figure out how to get the 
>>>>>>> variable to be included in each taskhash even though it's not 
>>>>>>> directly referenced, but I'm sure that's possible.
>>>>>>>
>>>>>>>
>>>>>>> Another option that's at lot more "magic" would be something like 
>>>>>>> this: 
>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd 
>>>>>>>
>>>>>>>
>>>>>>> That's really weird, and I'm not sure I like it, but worth a 
>>>>>>> thought.
>>>>>>>
>>>>>> If it works, it's good. I like the fact that the user can just set 
>>>>>> SOURCE_DATE_EPOCH directly in their recipe.
>>>>>>
>>>>>> How about ...
>>>>>>
>>>>>>     BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not
>>>>>>     source_date_epoch_var(d) else ''}"
>>>>>>
>>>>> Ya, that works. The only reason I chose "is None" was in case a 
>>>>> user really wanted to do
>>>>>
>>>>>   SOURCE_DATE_EPOCH = "0"
>>>>>
>>>> True, best to allow for that.
>>>>
>>>
>>> Joshua, I just tried the two top patches in your reproducible branch; 
>>> I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, 
>>> however, changing its value in my recipe doesn't trigger rebuilding 
>>> for some reason..
>>
>> In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} 
>> in my do_install() task.
> 
> 
> I don't think it matters. My understanding is that an export variable 
> (like SOURCE_DATE_EPOCH) is included in the taskhash of all tasks unless 
> it is specifically whitelisted. It's possible there is some bug I 
> haven't throughly tested it yet.... or it just can't work that way at 
> all for some reason. You might be able to dump the task signature for 
> your do_install task to see if SOURCE_DATE_EPOCH is being included.
> 

Using bitbake-dumpsig I can see that SOURCE_DATE_EPOCH is still in 
"basewhitelist:"