[OE-core] Sad story about Shared State

[Martin Jansa]

On Fri, May 27, 2011 at 12:13:35AM +0100, Richard Purdie wrote:
> On Fri, 2011-05-27 at 00:08 +0200, Martin Jansa wrote:
> > Good evening,
> > 
> > sorry for long and pesimistic e-mail, but I hope that someone will point
> > me to something I'm doing completely wrong.
> > 
> > First I was surprised to see how many recipes were rebuilt after this
> > change:
> > http://git.openembedded.net/cgit.cgi/openembedded-core/commit/?id=aebb9d6599aac683456adf56dc11f8b9f10f25c3
> > 
> > because [YOCTO #1074] was resolved by bitbake commit:
> > http://git.openembedded.net/cgit.cgi/bitbake/commit/?id=139b8a625818225c358a1b8363518d7ed6913188
> > and I had rm_work change
> > http://patches.openembedded.org/patch/4807/
> > during this build I decided to find out what is changing so many sstate
> > checksums for such small changes
> > 
> > so I took another small dbus change from patchwork
> > http://patches.openembedded.org/patch/4831/
> > and rebuild many recipes again to play with bitbake-diffsig for a while..
> 
> [...]
> 
> > .. end of story ..
> > 
> > .. I agree it's "right (TM)" to rebuild not only changed package but also its dependencies ..
> > .. I agree it's not possible to differentiate between important library change
> >    which can infulence also packages depending on it even if only transitive and
> >    cosmetic change improving ie RDEPENDS_${PN} which doesn't influence even direct dependants
> > .. but impact of such small change on small shr-lite-image?
> >    almost 2 hours on my quadcore AMD Phenom(tm) II X4 965 at 3.4Ghz, 4GB RAM, tmp/work on raid0 (3 SATA2 discs)
> > 
> > guess how long is runqueue when someone bumps binutils or gcc PR.. it's like DISTRO_PR bump :/
> > but almost after every "git pull --rebase".
> > 
> > this was for our minimal image
> > task-shr-feed we have in old OE takes about 2-3 days on our buildhost (dualcore with 2GB ram)
> > for 1 arch, we (SHR) are supporting 3 arm architectures 
> > armv4t (om-gta01, om-gta02), armv6-novfp (htcdream), armv7a (palmpre, palmpre2, nokia900)
> > so currently it takes 10-14 days to rebuild feed after DISTRO_PR change..
> > 
> > I'm not sure if we can afford our feeds to track oe-core/meta-oe HEADs 
> > like we did with old OE :(.
> 
> It should still be possible for OE-Core to behave like "old OE".
> 
> > Is it expected use-case to base layers only on released version of 
> > oe-core/meta-oe and never touch recipes which are higher in dependency tree?
> 
> It is expected that as we get a more stable and fully featured core that
> people should be able to base off released versions of it rather than
> needing to track master the whole time.
> 
> > Is it possible to disable sstate and do DISTRO_PR bumps manually when 
> > distro maintainder decides it's really needed and buildhost is free 
> > for long enough to populate feeds?
> 
> What do you mean by disable sstate as there are several different
> elements to it.

I meant something like switch back to something like pstage was or
disable both (IIRC I had strange error on many places without sstate
inherited by distro).
 
> Which BB_SIGNATURE_HANDLER are you using? It sounds like you're using
> basichash but want the behaviour generated by basic (the current
> default). Nobody is forcing you to use basichash.

AFAIK I'm using default basic..
OE @ ~/shr-core/meta-smartphone $ git grep BB_SIGNATURE_HANDLER
OE @ ~/shr-core/meta-openembedded $ git grep BB_SIGNATURE_HANDLER
OE @ ~/shr-core/openembedded-core $ git grep BB_SIGNATURE_HANDLER
meta/conf/bitbake.conf:BB_SIGNATURE_HANDLER ?= "basic"

OE @ ~/shr-core $ bitbake -e shr-lite-image | tee -a log; grep BB_SIGNATURE_HANDLER log
# BB_SIGNATURE_HANDLER=basic
BB_SIGNATURE_HANDLER="basic"

but even when I set it to not existing basic2
it doesn't show error from
lib/bb/siggen.py
        logger.error("Invalid signature generator '%s', using default 'noop'\n"
                     "Available generators: %s",
                     ', '.join(obj.name for obj in siggens))
but fails like this:
NOTE: Preparing runqueue
ERROR: Running idle function
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/bb/server/process.py", line 151, in idle_commands
    retval = function(self, data, False)
  File "/usr/lib64/python2.7/site-packages/bb/cooker.py", line 792, in buildTargetsIdle
    retval = rq.execute_runqueue()
  File "/usr/lib64/python2.7/site-packages/bb/runqueue.py", line 940, in execute_runqueue
    self.rqexe = RunQueueExecuteScenequeue(self)
  File "/usr/lib64/python2.7/site-packages/bb/runqueue.py", line 1434, in __init__
    valid = bb.utils.better_eval(call, locs)
  File "/usr/lib64/python2.7/site-packages/bb/utils.py", line 390, in better_eval
    return eval(source, _context, locals)
TypeError: expected a character buffer object

I'll add some debugging code to bitbake to see what's really going on here..

> As you mention, there are benefits and drawbacks either way.

Yes I see the benefits and if we find out that basichash is somehow used
instead of configured basic I'll be really happy to continue using it.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com