[bitbake-devel] (EXT) Re: Forcing the rebuild of dependent tasks across recipes

[Matthias Schiffer]

On Fri, 2019-11-22 at 16:34 +0000, Richard Purdie wrote:
> On Fri, 2019-11-22 at 16:09 +0100, Matthias Schiffer wrote:
> > is there a robust way to enforce the automatic rebuild of dependent
> > tasks across recipes? I'm having the following issue:
> > 
> > - Recipe my-core-image has an output file that is copied to the
> > deploy
> > dir. The output file is not fully reproducible, so cleaning the
> > recipe
> > and rebuilding it will yield a different file even through the
> > sstate
> > signature of the recipe is unchanged (the file I'm interested is
> > DMVerity hash data that I generate as an additional filesystem
> > type)
> > 
> > - Task do_compile of recipe my-initramfs-scripts depends on my-
> > core-
> > image:do_image_complete. I need to enforce rebuilding of this task
> > whenever the deployed files of my-core-image have changed (the
> > recipe
> > uses the "root hash" generated by DMVerity as input, so that the
> > filesystem can be verified on mount) - and of course also of all
> > dependent tasks like the initramfs image and FIT kernel image
> > 
> > I've experimented with things like file-checksums and
> > BB_DONT_CACHE,
> > but everything I've tried fails when the rebuilds of my-core-image
> > and
> > my-initramfs-scripts are supposed to happen in the same run of
> > Bitbake:
> > 
> > - file-checksums only seems to be taken into account at initial
> > recipe
> > parse time, not when the task I'm interested in is actually
> > executed?
> > I
> > need to run bitbake twice to get the correct results.
> > - When I use an anonymous Python function to put the root hash I'm
> > interested in into a variable, Bitbake complains about
> > nondeterministic
> > metadata.
> > 
> > Using nostamp would be an option, but obviously an ugly one.
> > 
> > I'm currently on Yocto Thud / Bitbake 1.40.
> 
> If you look behind the scenes at how "bitbake x -f" works, it writes
> .taint files into ${STAMPS}.
> 
> If A depends on B and B taints A's stamp each time it runs, it will
> rerun that task and all tasks that depend upon it. See
> bb.build.write_taint(). That function is not designed to be called
> externally so you may need your own version. 

Hello Richard,

I don't think this will help me. If I'm interpreting the Bitbake code
correctly, the taint will only be interpreted at initial parse time
(which it must, as it would lead to non-deterministic task signatures
otherwise). So I can either set the taint early (always invalidating
the signature, leading to nostamp behaviour), or during my core image
creation (leading to the file-checksums behaviour, requiring a second
bitbake run before the taint has any effect). Or am I overlooking
something?


Thanks,

Matthias


> 
> Ugly, but your problem is ugly. We don't have a standard mechanism as
> we don't really want to encourage this.
> 
> Cheers,
> 
> Richard
> 
>