[bitbake-devel] [PATCH 4/4] runqueue: Rewrite and optimize recrdepends handling

[Paul Barker]

On Mon, Feb 26, 2018 at 4:53 PM, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> On Sun, 2018-02-25 at 22:20 +0000, Paul Barker wrote:
>> On Sun, Feb 25, 2018 at 10:17 PM, Paul Barker <pbarker at toganlabs.com>
>> wrote:
>> >
>> > On Sun, Feb 25, 2018 at 10:11 PM, Richard Purdie
>> > <richard.purdie at linuxfoundation.org> wrote:
>> > >
>> > > On Sun, 2018-02-25 at 19:25 +0000, Paul Barker wrote:
>> > > >
>> > > > I've been having build failures recently which I've tracked
>> > > > down to
>> > > > this commit in bitbake via git bisect.
>> > > >
>> > > > I have an image recipe, oryx-image, and a publish recipe,
>> > > > oryx-publish. In oryx-publish I have a do_publish task that
>> > > > copies
>> > > > files out of "tmp/deploy/images/*" to their final places and so
>> > > > it
>> > > > needs to run after the tasks in oryx-image that create the
>> > > > relevant
>> > > > files. So I've explicitly added oryx-image:do_build to
>> > > > do_publish[depends].
>> > > Is do_publish added before do_build?
>> > > Does oryx-publish inherit any of the image/meta classes?
>> > >
>> > > I suspect I need to better understand the problem as you've
>> > > described
>> > > what you think the problem is but not what problem you're
>> > > seeing...
>> > >
>> > The recipe is here:
>> > https://gitlab.com/oryx/meta-oryx/blob/master/recipes-core/publish/
>> > oryx-publish.bb.
>> > ORYX_SYSTEM_PROFILE_PUBLISH_DEPENDS expands to "oryx-
>> > image:do_build".
>> >
>> > The problem I'm seeing is that the files don't exist in
>> > "tmp/deploy/images/*" when do_publish runs. Investigation via
>> > "bitbake
>> > -g" shows that the dependency on "oryx-image:do_build" has been
>> > dropped from do_publish:
>> >
>> Somehow trying to paste resulted in the email being sent.... stupid
>> gmail!
>>
>> The problem I'm seeing is that the files don't exist in
>> "tmp/deploy/images/*" when do_publish runs. Investigation via
>> "bitbake
>> -g" shows that the dependency on "oryx-image:do_build" has been
>> dropped from do_publish:
>>
>> "oryx-publish.do_publish" [label="oryx-publish
>> do_publish\n:1.0-r0\n/home/pbarker/oryx/meta-oryx/recipes-
>> core/publish/oryx-publish.bb"]
>> "oryx-publish.do_publish" -> "image-json-file.do_build"
>>
>> Commenting out the following lines in lib/bb/runqueue.py seems to
>> improve things for me but I know this is just a hack:
>>
>>         for tid in self.runtaskentries:
>>             self.runtaskentries[tid].depends.difference_update(recurs
>> ivetasksselfref)
>>
>> With those commented out, the dependencies are correct for
>> do_publish:
>>
>> "oryx-publish.do_publish" [label="oryx-publish
>> do_publish\n:1.0-r0\n/home/pbarker/oryx/meta-oryx/recipes-
>> core/publish/oryx-publish.bb"]
>> "oryx-publish.do_publish" -> "image-json-file.do_build"
>> "oryx-publish.do_publish" -> "oryx-image.do_build"
>
> Thanks, more information helped. I think you have found a bug. I'm
> wondering about:
>
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index 48df0131556..f2e52cf758c 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -765,7 +765,7 @@ class RunQueueData:
>              bb.debug(1, "Added %s recursive dependencies in this loop" % extradeps)
>
>          # Remove recrdeptask circular references so that do_a[recrdeptask] = "do_a do_b" can work
> -        for tid in self.runtaskentries:
> +        for tid in recursivetasksselfref:
>              self.runtaskentries[tid].depends.difference_update(recursivetasksselfref)
>
>          self.init_progress_reporter.next_stage()
>
> as a fix for it...
>

That makes sense. I'll apply that change and see if it works.

-- 
Paul Barker
Togán Labs Ltd