[bitbake-devel] (EXT) Re: (EXT) Re: Forcing the rebuild of dependent tasks across recipes

[Mark Hatle]

On 11/26/19 2:43 AM, Matthias Schiffer wrote:
> On Mon, 2019-11-25 at 08:54 -0600, Mark Hatle wrote:
>>
>> On 11/25/19 3:58 AM, Matthias Schiffer wrote:
>>> On Fri, 2019-11-22 at 11:15 -0600, Mark Hatle wrote:
>>>>
>>>> On 11/22/19 9:09 AM, Matthias Schiffer wrote:
>>>>> Hi all,
>>>>>
>>>>> is there a robust way to enforce the automatic rebuild of
>>>>> dependent
>>>>> tasks across recipes? I'm having the following issue:
>>>>>
>>>>> - Recipe my-core-image has an output file that is copied to the
>>>>> deploy
>>>>> dir. The output file is not fully reproducible, so cleaning the
>>>>> recipe
>>>>> and rebuilding it will yield a different file even through the
>>>>> sstate
>>>>> signature of the recipe is unchanged (the file I'm interested
>>>>> is
>>>>> DMVerity hash data that I generate as an additional filesystem
>>>>> type)
>>>>>
>>>>> - Task do_compile of recipe my-initramfs-scripts depends on my-
>>>>> core-
>>>>> image:do_image_complete. I need to enforce rebuilding of this
>>>>> task
>>>>> whenever the deployed files of my-core-image have changed (the
>>>>> recipe
>>>>> uses the "root hash" generated by DMVerity as input, so that
>>>>> the
>>>>> filesystem can be verified on mount) - and of course also of
>>>>> all
>>>>> dependent tasks like the initramfs image and FIT kernel image
>>>>>
>>>>> I've experimented with things like file-checksums and
>>>>> BB_DONT_CACHE,
>>>>> but everything I've tried fails when the rebuilds of my-core-
>>>>> image
>>>>> and
>>>>> my-initramfs-scripts are supposed to happen in the same run of
>>>>> Bitbake:
>>>>>
>>>>> - file-checksums only seems to be taken into account at initial
>>>>> recipe
>>>>> parse time, not when the task I'm interested in is actually
>>>>> executed? I
>>>>> need to run bitbake twice to get the correct results.
>>>>> - When I use an anonymous Python function to put the root hash
>>>>> I'm
>>>>> interested in into a variable, Bitbake complains about
>>>>> nondeterministic
>>>>> metadata.
>>>>
>>>> There is a way of doing this, I've had to do it in the past.
>>>>
>>>> The anonymous python function needs to product the same output on
>>>> each and every
>>>> run.  If it doesn't this will cause the non-deterministic
>>>> behavior.  (It can't
>>>> rely on another items to finish building for instance.)
>>>>
>>>> So you end up having to have a recipe (with a file-checksum
>>>> and/or
>>>> anon python)
>>>> that sets up variables that will keep the hash up-to-date for
>>>> THAT
>>>> recipe.. then
>>>> your image (or other) recipes need to depend on that one.
>>>
>>> Hello Mark,
>>>
>>> I'm afraid I don't follow. How does doing something in another
>>> recipe
>>> fix the non-deterministic behaviour?
>>
>> You must always compute your internal variables in -exactly- the same
>> way.  You
>> were saying you were passing in an external object and needed to have
>> that
>> influence the values.  (Unless I am understanding incorrectly.)
>>
>>> I'd think that any solution based on the task signatures will have
>>> the
>>> same issue - that a hash correctly reflecting the inputs that go
>>> into
>>> my initramfs task would depend on the build results of my core
>>> image
>>> task, which is not available at recipe parse time.
>>
>> All signatures are computed at parse time.  They either need to be
>> computed
>> based on external (non-changing) input, or other recipes (which
>> themselves would
>> have an static hash, but can change from build to build.)
>>
>>> I think I'll have to go with nostamp after all - the unnessesary
>>> rebuilds it causes in this case are ugly, but rather negligible in
>>> terms of build time.
>>
>> So is the input changing out from under the system?  I.e. during the
>> build
>> process did it change, or are you trying to use the output of one
>> recipe to
>> create input from another without standard recipe dependency
>> behavior?
> 
> The latter. I have a regular core image built by Yocto, and I don't
> want to care about full reproducibility - so rebuilding the core image
> will give me a different image regarding timestamps etc. The second
> recipe (and its dependents) needs to rerun when the hash of the built
> image changes. This does not happen without nostamp, because no
> variable known to Bitbake can contain this hash at parse time.

Bitbake DOES know the hash of the produced image -recipe-, and that should be
enough to know if you need to rebuild.

So you'd end up with a recipe dependency set that looks something like:

final_recipe -> image_recipe -> glibc, bash, etc...

(with modern OE, you can do a multiconfig build if your final_recipe and
image_recipe are different architectures..)


or you build as two separate steps (two different bitbake commands)..


--Mark

> Matthias
> 
> 
>>
>> --Mark
>>
>>>
>>> Thanks,
>>>
>>> Matthias
>>>
>>>
>>>
>>>>
>>>> Unfortunately I no longer have access to the example where I had
>>>> to
>>>> do this in
>>>> the past, or I could try to give you something more concrete to
>>>> look
>>>> at.
>>>>
>>>> --Mark
>>>>
>>>>>
>>>>> Using nostamp would be an option, but obviously an ugly one.
>>>>>
>>>>> I'm currently on Yocto Thud / Bitbake 1.40.
>>>>>
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Matthias
>>>>>