[bitbake-devel] [PATCH] runqueue: Reimplement recrdepends so it works more correctly

[Richard Purdie]

On Wed, 2012-06-27 at 11:06 +0100, Richard Purdie wrote:
> Currently, recrdepends is extremely greedy. For example:
> 
> do_foo[rdepends] = "somedep:sometask"
> addtask foo
> 
> which adds foo with *no* dependencies, will suddenly start appearing
> as a dependency in every task which uses recrdepends. So far this has
> been mildy annoying but we now have use cases where this makes no sense
> at all.
> 
> This reworks the recrdepends code to avoid this problem. To do this we
> can no longer collapse things into lists just based on file ID. The problem
> is this code is extremely performance sensitive. The "preparing runqueue"
> phase spends a lot of time in these recursive dependency calculations so any
> change here could negatively impact the user experience.
> 
> As such, this code has been carefully tested on convoluted dependency trees
> with operations like "time bitbake world -g". The net result of this change
> and the preceding changes combined is a net speed up of these operations in
> all cases measured.
> 
> Tests were made comparing "bitbake world -g" task-depends.dot before and after
> this patch. There *are* differences, particularly that a lot of *-native
> dependencies now disappear from recrdepends tasks like do_build. Also, -nativesdk
> do_build dependencies on -native recipes are no longer present. All removed
> dependencies appear to be sensible improvements to the system. The "rdepends"
> cross contamination issue above is also fixed.
> 
> As a result of this, people are going to notice significant falls in the numbers
> of tasks their standard builds will run, mainly from the removal of many -native
> "noexec" tasks. All the signs are this is a good thing.

I'm not entirely sure what I was thinking earlier on but the
disappearance of these tasks, whilst "nice" since we don't care about
them is incorrect and they should not be disappearing. There is a chunk
of dependencies we're ignoring and there could be something we do care
about getting lost. I'm working on a v2 of this patch.

Cheers,

Richard