[OE-core] package.tgz files not touched in sstate-cache

[Mike Crowe]

On Friday 29 January 2016 at 17:42:13 +0000, Richard Purdie wrote:
> On Fri, 2016-01-29 at 15:47 +0000, Mike Crowe wrote:
> > We have a shared sstate-cache that is populated by us building nightly
> > images for various branches using Jenkins. This cache is pruned
> > periodically (with special care of the siginfo files since we don't have
> > 80b3974081c4a8c604e23982a6db8fb32c616058 yet.)
> > 
> > Recently we went back to one of those branches that hadn't been changed for
> > a while and were surprised to find that Bitbake wanted to build loads of
> > packages from source.
> > 
> > It turned out that whilst the populate_sysroot.gz and package_ipk.tgz files
> > were still in the sstate-cache, the package.tgz tarball was not. This meant
> > that the whole package was built again in order to satisfy the build
> > dependencies of other packages that had changed.
> > 
> > I'm suspicious that this behaviour was introduced by the optimisation in
> > a0f584ac3d5a94dec121b684206ecd40c968f7fc so we hadn't seen it on older
> > branches.
> > 
> > Is there a way to cause the package.tgz file to be touched during these
> > nightly builds too? Or, is there a way to get hold of the complete set of
> > sstate files so they can be touched all in one go?
> 
> The "right" answer here is to figure out what in your builds seems to
> be needing the package.tgz files.
> 
> The only time the system should need to pull the package tgzs from
> sstate is if package_qa or package_write_* were going to rerun. This
> doesn't happen often, if ever. I've sometimes wondered if we should in
> fact generate them. They are useful for switching package backends.

Indeed. These files are currently taking up over fifty gigabytes in our
shared sstate-cache!

> So the fact they're not getting touched doesn't surprise me. The more
> concerning thing is that its actually trying to use them.
> 
> We have found a number of bugs in this area in sstate and have some
> patches merged, I know there are a small number pending too. Whether
> we've found all the issues remains unknown.

The branch we saw this on was using dizzy from about
e848484989307ae6826ba0f5217f7702322181e3 so it's quite likely that we're
lacking some more recent fixes.

I've tried to reproduce the problem with the current tip of oe-core and
cannot so I'll go looking for patches.

> For debugging this, you usually need the task graph info and the -DDDv
> output of a build, then investigate what was depending on the package
> sstate file.

Thanks for your help.

Mike.