[OE-core] [poky] Illustration of task latency

[Richard Purdie]

On Sat, 2011-04-09 at 09:26 -0700, Richard Purdie wrote:
> I've dug into this a bit. When running a given task, it is parsing all
> bbclassextend variants again. I have a simple patch which fixes that
> which I'll cleanup and share.
> 
> What I also noticed which was more odd was that the first finalise()
> call was taking 0.2s, subsequent ones were taking 0.1s. It turns out
> that the parsercache which is used by the siggen code isn't functioning
> the way it should with a lot of cache misses. I think this is related to
> the parallel parsing and only saving out cached data from the core, not
> the individual recipes (i.e. the cache from the parser subthreads). I
> suspect we could get some performance improvement by fixing this.

Fixes for the issues I noticed are in:

http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/bboptimise

It turns out executing a function at process exit time with
multiprocessing is quite tricky and I'm not 100% sure I'm using public
API with the approach I've ended up using.

The speedups have a good feel to them and show the right things with
benchmarks. It will be interesting to see how much affect they have on
real world build time (which I'll test when I get home next week).
They're also showing a significant reduction in parsing time.

I also noticed that skipped recipes were always being reparsed which is
suboptimial and I've included a work in progress fix for that in the
above branch too.

Cheers,

Richard