[bitbake-devel] (EXT) Re: Forcing the rebuild of dependent tasks across recipes

[Mark Hatle]

On 11/25/19 3:58 AM, Matthias Schiffer wrote:
> On Fri, 2019-11-22 at 11:15 -0600, Mark Hatle wrote:
>>
>> On 11/22/19 9:09 AM, Matthias Schiffer wrote:
>>> Hi all,
>>>
>>> is there a robust way to enforce the automatic rebuild of dependent
>>> tasks across recipes? I'm having the following issue:
>>>
>>> - Recipe my-core-image has an output file that is copied to the
>>> deploy
>>> dir. The output file is not fully reproducible, so cleaning the
>>> recipe
>>> and rebuilding it will yield a different file even through the
>>> sstate
>>> signature of the recipe is unchanged (the file I'm interested is
>>> DMVerity hash data that I generate as an additional filesystem
>>> type)
>>>
>>> - Task do_compile of recipe my-initramfs-scripts depends on my-
>>> core-
>>> image:do_image_complete. I need to enforce rebuilding of this task
>>> whenever the deployed files of my-core-image have changed (the
>>> recipe
>>> uses the "root hash" generated by DMVerity as input, so that the
>>> filesystem can be verified on mount) - and of course also of all
>>> dependent tasks like the initramfs image and FIT kernel image
>>>
>>> I've experimented with things like file-checksums and
>>> BB_DONT_CACHE,
>>> but everything I've tried fails when the rebuilds of my-core-image
>>> and
>>> my-initramfs-scripts are supposed to happen in the same run of
>>> Bitbake:
>>>
>>> - file-checksums only seems to be taken into account at initial
>>> recipe
>>> parse time, not when the task I'm interested in is actually
>>> executed? I
>>> need to run bitbake twice to get the correct results.
>>> - When I use an anonymous Python function to put the root hash I'm
>>> interested in into a variable, Bitbake complains about
>>> nondeterministic
>>> metadata.
>>
>> There is a way of doing this, I've had to do it in the past.
>>
>> The anonymous python function needs to product the same output on
>> each and every
>> run.  If it doesn't this will cause the non-deterministic
>> behavior.  (It can't
>> rely on another items to finish building for instance.)
>>
>> So you end up having to have a recipe (with a file-checksum and/or
>> anon python)
>> that sets up variables that will keep the hash up-to-date for THAT
>> recipe.. then
>> your image (or other) recipes need to depend on that one.
> 
> Hello Mark,
> 
> I'm afraid I don't follow. How does doing something in another recipe
> fix the non-deterministic behaviour?

You must always compute your internal variables in -exactly- the same way.  You
were saying you were passing in an external object and needed to have that
influence the values.  (Unless I am understanding incorrectly.)

> I'd think that any solution based on the task signatures will have the
> same issue - that a hash correctly reflecting the inputs that go into
> my initramfs task would depend on the build results of my core image
> task, which is not available at recipe parse time.

All signatures are computed at parse time.  They either need to be computed
based on external (non-changing) input, or other recipes (which themselves would
have an static hash, but can change from build to build.)

> I think I'll have to go with nostamp after all - the unnessesary
> rebuilds it causes in this case are ugly, but rather negligible in
> terms of build time.

So is the input changing out from under the system?  I.e. during the build
process did it change, or are you trying to use the output of one recipe to
create input from another without standard recipe dependency behavior?

--Mark

> 
> Thanks,
> 
> Matthias
> 
> 
> 
>>
>> Unfortunately I no longer have access to the example where I had to
>> do this in
>> the past, or I could try to give you something more concrete to look
>> at.
>>
>> --Mark
>>
>>>
>>> Using nostamp would be an option, but obviously an ugly one.
>>>
>>> I'm currently on Yocto Thud / Bitbake 1.40.
>>>
>>>
>>> Thanks in advance,
>>>
>>> Matthias
>>>