[bitbake-devel] [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Chris Larson]

On Monday, December 30, 2013, Richard Purdie wrote:

> On Mon, 2013-12-30 at 17:35 +0100, Martin Jansa wrote:
> > On Mon, Dec 30, 2013 at 09:29:38AM -0700, Chris Larson wrote:
> > > On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson at kergoth.com<javascript:;>>
> wrote:
> > >
> > > > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <
> martin.jansa at gmail.com <javascript:;>>wrote:
> > > >
> > > >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:
> > > >> > Its useful to understand where the delta starts against an
> existing
> > > >> sstate cache
> > > >> > for a given target. Adding this to the output of the -S option
> seems
> > > >> like a
> > > >> > natural fit.
> > > >> >
> > > >> > We use the hashvalidate function to figure this out and assume it
> can
> > > >> find siginfo
> > > >> > files for more than just the setscene tasks.
> > > >>
> > > >> I haven't tried to find smaller reproducer or exact part of code
> where
> > > >> it happends, but in bigger builds I have it fails with
> > > >>
> > > >> I'll try to reproduce and narrow it a bit, but if you have tip where
> > > >> to look first..
> > > >>
> > > >> IndexError: list index out of range:
> > > >
> > > >
> > > > I’m hitting this now as well.
> > >
> > >
> > > To clarify further, I’m hitting this with stock poky master, qemux86,
> > > bitbake -S core-image-base.
> > >
> > > Actually, this is getting in my way trying to do something I don’t
> want it
> > > doing anyway. I want to dump the signature data from one config, and
> use
> > > bitbake-whatchanged from another build. If I wanted to compare
> something
> > > right here, I’d have run bitbake-whatchanged. This change confuses me a
> > > bit. -S emits signature data, according to the bitbake help and the
> > > semantics of the command. Are we trying to kill bitbake-whatchanged in
> > > favor of making -S do both jobs, emit and compare?
> >
> > Same here, I'm using bitbake -S from
> > oe-core/scripts/sstate-diff-machines.sh
> >
> > and this added analysis and output is IMHO very useful, but it would be
> > nice to control it with some new option (and keep default -S behavior
> > fast and simple).
> >
> > It looks like the error is triggered for every task which wasn't built
> > before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or
> > bitbake -S world after bitbake some-image).
> >
> > I see it for many do_fetch and do_rm_work_all tasks.
>
> To put things in context, I was trying to work with the existing
> "whatchanged" type tools or more specifically I wanted to know "why
> isn't my sstate cache being used". Having to do a two step process for
> any of these enquires seems a little odd. There are also two specific
> and as it turns our quite different work flows:
>
> a) What changed since my last build?
> b) What is the least delta between my current build and whatever I have
> cached?


> Those experiments revealed some big holes in the sstate metadata and
> some good improvements there. It also highlights that there are two
> different questions which have different answers.
>
> In the process, I found bitbake does have some state in runqueue which
> can help pinpoint the "whatchanged" part of the problem much more
> accurately/quickly. Obviously that state is lost after bitbake -S
> finishes so I started printing out some extra data.
>
> Clearly bitbake shouldn't crash, that is a bug and I think Martin is
> heading in the right direction with the other patch.
>
> As for bitbake -S being kept fast, I think (but I need to double check
> with measurements) that the parsing part of bitbake -S is the slow part
> and this extra data is effectively "for free".
>
> So basically consider the current state an experiment to see if we can
> make -S more useful/interesting (without much cost) and to try to better
> understand where/what functionality we need going forward.
>

I can see that, but I think you're making the semantics of the command-line
argument confusing by trying to combine the two operations into one. I also
think that trying to find the "least delta" is extremely weird, and not
always likely to produce anything worthwhile. My SSTATE_DIR has weeks if
not months of output from hundreds of builds of different combinations of
machine, distro, layer selection, upstream vs mel, as well as between tons
of build environments with slight local alterations. Unless I'm
misunderstanding something, how exactly is it going to know what I want it
to compare against, and how it is going to keep finding the "least delta"
fast, amongst all that data? With the current tools, I can explicitly use
-S in one env and whatchanged in another (with appropriate STAMPS_DIR) to
compare between two exact build environments, rather than letting bitbake
try to automagically find what I meant. Now that the two are combined into
one, I'm not seeing how to do what I'm currently doing. Can you explain
this?

-Chris


-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20131230/39e16a3f/attachment-0002.html>