[bitbake-devel] [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Martin Jansa]

On Fri, Jan 10, 2014 at 03:14:17PM +0100, Martin Jansa wrote:
> On Wed, Jan 01, 2014 at 11:38:01PM +0000, Richard Purdie wrote:
> > So can we find the "least delta" fast? Its not actually that hard
> > computationally or on resources, at least in the experiments I've made.
> > We know the hashes of the current target's tasks and we can quickly tell
> > which are in the cache and which are not (using the same function sstate
> > uses for that purpose).
> 
> I've finally got some numbers to support my claim that it "feels" slower
> with this added functionality.
> 
> My script is calling
> openembedded-core/scripts/sstate-diff-machines.sh --machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/;
> 
> which in turn runs
> bitbake -S world
> three times (once for each machine).
> 
> The script on jenkins doing world builds is completely crazy (I've
> killed it in last run assuming it was stuck, but it wasn't just terribly
> slow).
> http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/
> Running 21 hours already and doing 2nd machine from 3 (maybe because
> longer SSTATE_MIRROR?). There is one python process using 100% cpu.

FWIW: this jenkins job was completed yesterday after 16 days

small part of strace:
http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/log.strace

complete log:
http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/signatures.sstate-diff-machines

Previous runs of the same job took 25-40 minutes

> In local builds without SSTATE_MIRROR it's not so bad, but still
> significantly slower with each release (dylan-dora-master).
> 
> Master builds have that WIP patch included in order to finish.
> 
> It's not very accurate, because it was running on my desktop and each one has
> different metadata, but at least the layers included are similar (only
> systems which got separated in dora and removal
> of meta-webos-backports which isn't needed for dora and newer).
> 
> At least I've executed each test twice.
> 
> dylan:
> real   9m37.271s
> user   6m37.104s
> sys    3m12.625s
> 
> real   10m5.787s
> user   6m37.832s
> sys    3m23.614s
> 
> dora:
> real   26m11.374s
> user   9m7.325s
> sys    3m0.048s
> 
> real   28m10.606s
> user   8m27.895s
> sys    2m57.497s
> 
> master:
> real   38m17.641s
> user   17m46.905s
> sys    5m12.061s
> 
> real    36m12.748s
> user    16m32.961s
> sys     4m43.103s
> 
> master-without-S-changes:
> real    27m34.228s
> user    8m6.811s
> sys     2m53.724s
> 
> real    27m49.752s
> user    8m25.026s
> sys     2m50.921s
> 
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com



-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20140127/f232e161/attachment-0002.sig>