[bitbake-devel] [yocto] Ordering of anonymous Python functions and task signature calculations

[Matt Hoosier]

Thanks for the reply.

On Wed, Jul 25, 2018 at 4:58 PM Richard Purdie <
richard.purdie at linuxfoundation.org> wrote:

> On Wed, 2018-07-25 at 08:47 -0500, Matt Hoosier wrote:
> > Quite a while ago I wrote the following message:
> > On Tue, Jan 24, 2017 at 11:55 AM Matt Hoosier <matt.hoosier at gmail.com
> > > wrote:
> > > In order to support a use-case that embeds information about the
> > > Git revision of Yocto itself that was used to make a build, I would
> > > like to run some arbitrary Python code and dump the results (Git
> > > SHA1's, tag names, etc) into a Bitbake variable.
> > >
> > > The idea is that the variable would get consumed in a do_install()
> > > task to populate some cookie-crumb file in the package payload.
> > >
> > > Something like:
> > >
> > >   DEPENDS = "git-native"
> > >
> > >   python () {
> > >     # This pseudocode isn't strictly functional for invoking Git,
> > > but you get the idea
> > >     d.setVar('GIT_INFO', subprocess.Popen(['git', 'rev-list',
> > > ...]).communicate().stdout)
> > >   }
> > >
> > >   do_install () {
> > >     install -d ${D}/etc
> > >     echo "${GIT_INFO}" > ${D}/etc/git-info.txt
> > >   }
> > >
> > > This all works on a fresh run, but Bitbake appears not to be
> > > evaluating the anonymous Python function on each execution. This
> > > leads it to have out-of-date information about the Git working copy
> > > when global state changes but happens not to impact the particular
> > > recipe holding this logic.
> > >
> > >
> >
> > The replies given at the time were very helpful in letting me
> > accomplish my immediate goals, but I never really did get to the
> > bottom of the real question:
> >
> > > Is there any trick available to cause the Python function to
> > > execute (and hence update the value of ${GIT_INFO} each Bitbake
> > > execution so that the up-to-the-moment contents of it filter into
> > > the calculation of the signature for do_install()? Or am I just
> > > trying a wrong-headed approach and doing something illegitimate?
> > >
> >
> > Some build systems I know of [1] that are based on hashes rather than
> > timestamps have a way that you can inform the task execution engine
> > to always poison the inputs to a certain task's up-to-dateness check.
> > Is there anything in Bitbake that would allow a declaration to force
> > such-and-such variable in a recipe always to be re-evaluated:
> >
> >   EXTERNALLY_INFLUENCED_VARIABLE = "${@int(random.random() * 10)}"
> >   do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "
> >
> >   do_install() {
> >     echo '${ EXTERNALLY_INFLUENCED_VARIABLE }' > ${D}/etc/stuff.txt
> >   }
> >
> > Don't read too much into the example calling random() here. That's
> > not really what I want to do; the point is just that the variable's
> > value (though deterministic) isn't computed strictly from Bitbake
> > metadata. So I'm searching for a way to get the value of
> > EXTERNALLY_INFLUENCED_VARIABLE to be re-computed each time Bitbake
> > starts. If the computed value ends up matching what was seen on the
> > last execution, then any dependent tasks wouldn't need re-run.
> >
> > Any ideas?
> >
> > [1] I'm thinking particularly of Waf's hash-based task graph, which
> > supports annotations that can be used to always cause a task to be
> > re-run even if all the inputs feeding into it appear up-to-date.
>
> From a bitbake perspective there are two things you need to ensure
> here, that the recipe gets reparsed at every execution


Your advice about BB_DONT_CACHE does cause reparsing to get triggered. I
can see that in logs.


> and that the
> task dependency exists. You have the dependency above and that is the
> easier piece to fix.
>

But this part I'm not seeing. The variable's value is always recalculated
(sometimes more than once during a Bitbake run, but that's okay because
it's idempotent), but the task only re-executes if the payload of the .bb
file itself has changed. The blurb I wrote above:

  do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "

is just some nonsense syntax I wrote there to illustrate my point. I don't
think there's any var flag called "vardepsalwaysdirty". Did you have this
line in mind when you were saying that I'd already gotten the task made
dependent on the variable? Or were you just noting that the syntactic use
of the variable inside of the task should engage the normal hash-based
task-dependency calculation logic?


>
> If the recipe is reparsed, the anon python reruns and the task hash
> recalculated.
>

Is there anything extra I need to do to cause the reparsed variable's hash
to cascade to the task?


>
> If a recipe sets BB_DONT_CACHE, it will be reparsed upon each run and
> the hashes will dynamically change as you want. BB_SRCREV_POLICY !=
> "cache" uses that behind the scenes so that it can dynamically update
> srcrevs.
>
> Hope that helps...
>
> Cheers,
>
> Richard
>
>

Thanks! -Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20180726/73b7fb0d/attachment-0002.html>