[bitbake-devel] [PATCH 4/4] runqueue: Rewrite and optimize recrdepends handling

[Richard Purdie]

On Sun, 2018-02-25 at 22:20 +0000, Paul Barker wrote:
> On Sun, Feb 25, 2018 at 10:17 PM, Paul Barker <pbarker at toganlabs.com>
> wrote:
> > 
> > On Sun, Feb 25, 2018 at 10:11 PM, Richard Purdie
> > <richard.purdie at linuxfoundation.org> wrote:
> > > 
> > > On Sun, 2018-02-25 at 19:25 +0000, Paul Barker wrote:
> > > > 
> > > > I've been having build failures recently which I've tracked
> > > > down to
> > > > this commit in bitbake via git bisect.
> > > > 
> > > > I have an image recipe, oryx-image, and a publish recipe,
> > > > oryx-publish. In oryx-publish I have a do_publish task that
> > > > copies
> > > > files out of "tmp/deploy/images/*" to their final places and so
> > > > it
> > > > needs to run after the tasks in oryx-image that create the
> > > > relevant
> > > > files. So I've explicitly added oryx-image:do_build to
> > > > do_publish[depends].
> > > Is do_publish added before do_build?
> > > Does oryx-publish inherit any of the image/meta classes?
> > > 
> > > I suspect I need to better understand the problem as you've
> > > described
> > > what you think the problem is but not what problem you're
> > > seeing...
> > > 
> > The recipe is here:
> > https://gitlab.com/oryx/meta-oryx/blob/master/recipes-core/publish/
> > oryx-publish.bb.
> > ORYX_SYSTEM_PROFILE_PUBLISH_DEPENDS expands to "oryx-
> > image:do_build".
> > 
> > The problem I'm seeing is that the files don't exist in
> > "tmp/deploy/images/*" when do_publish runs. Investigation via
> > "bitbake
> > -g" shows that the dependency on "oryx-image:do_build" has been
> > dropped from do_publish:
> > 
> Somehow trying to paste resulted in the email being sent.... stupid
> gmail!
> 
> The problem I'm seeing is that the files don't exist in
> "tmp/deploy/images/*" when do_publish runs. Investigation via
> "bitbake
> -g" shows that the dependency on "oryx-image:do_build" has been
> dropped from do_publish:
> 
> "oryx-publish.do_publish" [label="oryx-publish
> do_publish\n:1.0-r0\n/home/pbarker/oryx/meta-oryx/recipes-
> core/publish/oryx-publish.bb"]
> "oryx-publish.do_publish" -> "image-json-file.do_build"
> 
> Commenting out the following lines in lib/bb/runqueue.py seems to
> improve things for me but I know this is just a hack:
> 
>         for tid in self.runtaskentries:
>             self.runtaskentries[tid].depends.difference_update(recurs
> ivetasksselfref)
> 
> With those commented out, the dependencies are correct for
> do_publish:
> 
> "oryx-publish.do_publish" [label="oryx-publish
> do_publish\n:1.0-r0\n/home/pbarker/oryx/meta-oryx/recipes-
> core/publish/oryx-publish.bb"]
> "oryx-publish.do_publish" -> "image-json-file.do_build"
> "oryx-publish.do_publish" -> "oryx-image.do_build"

Thanks, more information helped. I think you have found a bug. I'm
wondering about:

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 48df0131556..f2e52cf758c 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -765,7 +765,7 @@ class RunQueueData:
             bb.debug(1, "Added %s recursive dependencies in this loop" % extradeps)
 
         # Remove recrdeptask circular references so that do_a[recrdeptask] = "do_a do_b" can work
-        for tid in self.runtaskentries:
+        for tid in recursivetasksselfref:
             self.runtaskentries[tid].depends.difference_update(recursivetasksselfref)
 
         self.init_progress_reporter.next_stage()

as a fix for it...

Cheers,

Richard