[bitbake-devel] (EXT) Re: (EXT) Re: Forcing the rebuild of dependent tasks across recipes

[Matthias Schiffer]

On Mon, 2019-11-25 at 08:54 -0600, Mark Hatle wrote:
> 
> On 11/25/19 3:58 AM, Matthias Schiffer wrote:
> > On Fri, 2019-11-22 at 11:15 -0600, Mark Hatle wrote:
> > > 
> > > On 11/22/19 9:09 AM, Matthias Schiffer wrote:
> > > > Hi all,
> > > > 
> > > > is there a robust way to enforce the automatic rebuild of
> > > > dependent
> > > > tasks across recipes? I'm having the following issue:
> > > > 
> > > > - Recipe my-core-image has an output file that is copied to the
> > > > deploy
> > > > dir. The output file is not fully reproducible, so cleaning the
> > > > recipe
> > > > and rebuilding it will yield a different file even through the
> > > > sstate
> > > > signature of the recipe is unchanged (the file I'm interested
> > > > is
> > > > DMVerity hash data that I generate as an additional filesystem
> > > > type)
> > > > 
> > > > - Task do_compile of recipe my-initramfs-scripts depends on my-
> > > > core-
> > > > image:do_image_complete. I need to enforce rebuilding of this
> > > > task
> > > > whenever the deployed files of my-core-image have changed (the
> > > > recipe
> > > > uses the "root hash" generated by DMVerity as input, so that
> > > > the
> > > > filesystem can be verified on mount) - and of course also of
> > > > all
> > > > dependent tasks like the initramfs image and FIT kernel image
> > > > 
> > > > I've experimented with things like file-checksums and
> > > > BB_DONT_CACHE,
> > > > but everything I've tried fails when the rebuilds of my-core-
> > > > image
> > > > and
> > > > my-initramfs-scripts are supposed to happen in the same run of
> > > > Bitbake:
> > > > 
> > > > - file-checksums only seems to be taken into account at initial
> > > > recipe
> > > > parse time, not when the task I'm interested in is actually
> > > > executed? I
> > > > need to run bitbake twice to get the correct results.
> > > > - When I use an anonymous Python function to put the root hash
> > > > I'm
> > > > interested in into a variable, Bitbake complains about
> > > > nondeterministic
> > > > metadata.
> > > 
> > > There is a way of doing this, I've had to do it in the past.
> > > 
> > > The anonymous python function needs to product the same output on
> > > each and every
> > > run.  If it doesn't this will cause the non-deterministic
> > > behavior.  (It can't
> > > rely on another items to finish building for instance.)
> > > 
> > > So you end up having to have a recipe (with a file-checksum
> > > and/or
> > > anon python)
> > > that sets up variables that will keep the hash up-to-date for
> > > THAT
> > > recipe.. then
> > > your image (or other) recipes need to depend on that one.
> > 
> > Hello Mark,
> > 
> > I'm afraid I don't follow. How does doing something in another
> > recipe
> > fix the non-deterministic behaviour?
> 
> You must always compute your internal variables in -exactly- the same
> way.  You
> were saying you were passing in an external object and needed to have
> that
> influence the values.  (Unless I am understanding incorrectly.)
> 
> > I'd think that any solution based on the task signatures will have
> > the
> > same issue - that a hash correctly reflecting the inputs that go
> > into
> > my initramfs task would depend on the build results of my core
> > image
> > task, which is not available at recipe parse time.
> 
> All signatures are computed at parse time.  They either need to be
> computed
> based on external (non-changing) input, or other recipes (which
> themselves would
> have an static hash, but can change from build to build.)
> 
> > I think I'll have to go with nostamp after all - the unnessesary
> > rebuilds it causes in this case are ugly, but rather negligible in
> > terms of build time.
> 
> So is the input changing out from under the system?  I.e. during the
> build
> process did it change, or are you trying to use the output of one
> recipe to
> create input from another without standard recipe dependency
> behavior?

The latter. I have a regular core image built by Yocto, and I don't
want to care about full reproducibility - so rebuilding the core image
will give me a different image regarding timestamps etc. The second
recipe (and its dependents) needs to rerun when the hash of the built
image changes. This does not happen without nostamp, because no
variable known to Bitbake can contain this hash at parse time.

Matthias


> 
> --Mark
> 
> > 
> > Thanks,
> > 
> > Matthias
> > 
> > 
> > 
> > > 
> > > Unfortunately I no longer have access to the example where I had
> > > to
> > > do this in
> > > the past, or I could try to give you something more concrete to
> > > look
> > > at.
> > > 
> > > --Mark
> > > 
> > > > 
> > > > Using nostamp would be an option, but obviously an ugly one.
> > > > 
> > > > I'm currently on Yocto Thud / Bitbake 1.40.
> > > > 
> > > > 
> > > > Thanks in advance,
> > > > 
> > > > Matthias
> > > >