[oe] [Bitbake-dev] Bitbake runqueue performance improvement

[Michael Smith]

Richard Purdie wrote:
> On Tue, 2009-07-21 at 19:32 +0100, Richard Purdie wrote:
>> The attached patch rewrites the problematic function entirely with the
>> following benefits:
> 
> Of course one of my last minute cosmetic cleanups broke it :(.
> 
> http://git.pokylinux.org/cgit.cgi/poky/commit/?id=502bd2ef92f6cddf74ac85279dab621372791dc0 is the key fix with a couple of other things I noticed too.

Hi Richard,

I've been playing around with this and it seems to be creating some 
problems, unless it's just finding problems in my overlay that the old 
code didn't see.

With the latest fix, it finds dependency loops, but I'm not sure how to 
track them down as the dependencies are identified only by numbers:

NOTE: Preparing runqueue
ERROR: Unbuildable tasks were found.
These are usually caused by circular dependencies and any circular 
dependency chains found will be printed below. Increase the debug level 
to see a list of unbuildable tasks.

Identifying dependency loops (this may take a short while)...

ERROR:
Dependency loop #1 found:
   Task 7 
(/home/michael/startitup/public/recipes/images/toastix-image.bb, 
do_install) (depends: set([9, 6]))
   Task 8 
(/home/michael/startitup/public/recipes/images/toastix-image.bb, 
do_populate_staging) (depends: set([7]))
   Task 9 
(/home/michael/startitup/public/recipes/images/toastix-image.bb, 
do_rootfs) (depends: set([1024, 6, 2049, 8, 2057, 2059, 13, 791, 2067, 
21, 23, 2077, 31, 33, 2087, 41, 43, 2097, 51, 53, 2107,

[snip a few hundred numbers]

Before the fix from 502bd2e, it was dropping some do_package_write_deb 
dependencies that are needed to ensure .debs are built for packages that 
are not listed in IMAGE_INSTALL, but are required by other packages. My 
task-depends.dot was missing about 1000 entries. So was my depends.dot.

For example, I've got openssh in my IMAGE_INSTALL and it DEPENDS on zlib 
(and has a run-time dependency via shlibs). If I removed my zlib debs 
and tmp/stamps/blah/zlib-1.2.3-r5.do_package*, building my image should 
have built the zlib debs (and did before applying your patches), but it 
didn't so the rootfs failed.


BTW, is there any chance of getting Bitbake officially moved to a Git 
repo? I think a few people are working out of git-svn trees, or subdirs 
inside larger git trees, but it takes some gymnastics to merge between 
them as Git doesn't know they're related.

Thanks,
Mike