<br><br>On Monday, December 30, 2013, Richard Purdie wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Mon, 2013-12-30 at 17:35 +0100, Martin Jansa wrote:<br>
> On Mon, Dec 30, 2013 at 09:29:38AM -0700, Chris Larson wrote:<br>
> > On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <<a href="javascript:;" onclick="_e(event, 'cvml', 'clarson@kergoth.com')">clarson@kergoth.com</a>> wrote:<br>
> ><br>
> > > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <<a href="javascript:;" onclick="_e(event, 'cvml', 'martin.jansa@gmail.com')">martin.jansa@gmail.com</a>>wrote:<br>
> > ><br>
> > >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:<br>
> > >> > Its useful to understand where the delta starts against an existing<br>
> > >> sstate cache<br>
> > >> > for a given target. Adding this to the output of the -S option seems<br>
> > >> like a<br>
> > >> > natural fit.<br>
> > >> ><br>
> > >> > We use the hashvalidate function to figure this out and assume it can<br>
> > >> find siginfo<br>
> > >> > files for more than just the setscene tasks.<br>
> > >><br>
> > >> I haven't tried to find smaller reproducer or exact part of code where<br>
> > >> it happends, but in bigger builds I have it fails with<br>
> > >><br>
> > >> I'll try to reproduce and narrow it a bit, but if you have tip where<br>
> > >> to look first..<br>
> > >><br>
> > >> IndexError: list index out of range:<br>
> > ><br>
> > ><br>
> > > I’m hitting this now as well.<br>
> ><br>
> ><br>
> > To clarify further, I’m hitting this with stock poky master, qemux86,<br>
> > bitbake -S core-image-base.<br>
> ><br>
> > Actually, this is getting in my way trying to do something I don’t want it<br>
> > doing anyway. I want to dump the signature data from one config, and use<br>
> > bitbake-whatchanged from another build. If I wanted to compare something<br>
> > right here, I’d have run bitbake-whatchanged. This change confuses me a<br>
> > bit. -S emits signature data, according to the bitbake help and the<br>
> > semantics of the command. Are we trying to kill bitbake-whatchanged in<br>
> > favor of making -S do both jobs, emit and compare?<br>
><br>
> Same here, I'm using bitbake -S from<br>
> oe-core/scripts/sstate-diff-machines.sh<br>
><br>
> and this added analysis and output is IMHO very useful, but it would be<br>
> nice to control it with some new option (and keep default -S behavior<br>
> fast and simple).<br>
><br>
> It looks like the error is triggered for every task which wasn't built<br>
> before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or<br>
> bitbake -S world after bitbake some-image).<br>
><br>
> I see it for many do_fetch and do_rm_work_all tasks.<br>
<br>
To put things in context, I was trying to work with the existing<br>
"whatchanged" type tools or more specifically I wanted to know "why<br>
isn't my sstate cache being used". Having to do a two step process for<br>
any of these enquires seems a little odd. There are also two specific<br>
and as it turns our quite different work flows:<br>
<br>
a) What changed since my last build?<br>
b) What is the least delta between my current build and whatever I have<br>
cached?</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Those experiments revealed some big holes in the sstate metadata and<br>
some good improvements there. It also highlights that there are two<br>
different questions which have different answers.<br>
<br>
In the process, I found bitbake does have some state in runqueue which<br>
can help pinpoint the "whatchanged" part of the problem much more<br>
accurately/quickly. Obviously that state is lost after bitbake -S<br>
finishes so I started printing out some extra data.<br>
<br>
Clearly bitbake shouldn't crash, that is a bug and I think Martin is<br>
heading in the right direction with the other patch.<br>
<br>
As for bitbake -S being kept fast, I think (but I need to double check<br>
with measurements) that the parsing part of bitbake -S is the slow part<br>
and this extra data is effectively "for free".<br>
<br>
So basically consider the current state an experiment to see if we can<br>
make -S more useful/interesting (without much cost) and to try to better<br>
understand where/what functionality we need going forward.<br></blockquote><div><br></div><div>I can see that, but I think you're making the semantics of the command-line argument confusing by trying to combine the two operations into one. I also think that trying to find the "least delta" is extremely weird, and not always likely to produce anything worthwhile. My SSTATE_DIR has weeks if not months of output from hundreds of builds of different combinations of machine, distro, layer selection, upstream vs mel, as well as between tons of build environments with slight local alterations. Unless I'm misunderstanding something, how exactly is it going to know what I want it to compare against, and how it is going to keep finding the "least delta" fast, amongst all that data? With the current tools, I can explicitly use -S in one env and whatchanged in another (with appropriate STAMPS_DIR) to compare between two exact build environments, rather than letting bitbake try to automagically find what I meant. Now that the two are combined into one, I'm not seeing how to do what I'm currently doing. Can you explain this?</div>
<div><br></div><div>-Chris</div><br><br>-- <br>Christopher Larson<br>clarson at kergoth dot com<br>Founder - BitBake, OpenEmbedded, OpenZaurus<br>Maintainer - Tslib<br>Senior Software Engineer, Mentor Graphics<br>