[OE-core] [PATCH RFC] sstate: Add eventhandler which cleans up stale recipe data

[Richard Purdie]

On Mon, 2015-06-08 at 14:43 +0200, Martin Jansa wrote:
> On Sun, Jun 07, 2015 at 08:20:12AM +0100, Richard Purdie wrote:
> > "Incremental builds do not work well when renaming recipes or changing
> > architecture" is a long standing issue which causes people considerable
> > pain. We've struggled for a long time to come up with a way to
> > generically address the problem.
> > 
> > There are additional issues where removal of a layer caused data to
> > continue to exist and additionally, changing DISTRO_FEATURES also caused
> > problems in an existing TMPDIR.
> > 
> > This patch attempts to address this by adding a mapping between stamp
> > files and manifests. After parsing we can easily tell which stamp files
> > are still reachable, if any manifest has a stamp that can no longer be
> > reached, we can remove it. Since this code ties this to the sstate
> > architecture list, it will not remove data from other than the current
> > MACHINE (and its active architectures). It does not clean the sstate
> > cache so if another build activates something which was cleaned, it
> > should reinstall from sstate.
> > 
> > We can also go one step further, depending on the setting of
> > SSTATE_PRUNE_OBSOLETEWORKDIR, workdirs which are no longer active can
> > also be removed. This avoids the buildup of many old copies of data in
> > WORKDIR for example when versions are upgraded.
> > 
> > The one thing which may surprise people with this change is if you
> > remove a layer, data added by that layer will be "uninstalled" before
> > the next build continues. I believe this is a feature and a good thing
> > to do though.
> > 
> > This code is safe with existing builds. If something isn't in the new
> > index it simply isn't removed. Since changes to the sstate code trigger
> > a rebuild, after this merges, we can assume the code will start to
> > detect changes from that point onwards.
> > 
> > [Right now this is an RFC, it appeared to do the right things in some
> > brief local tests and I am pretty excited that this could solve a long
> > standing usability issue in a clean and effective way. There is a bug
> > related to DISTRO_FEATURES changes right now since even skipped recipes
> > will still show active stamps meaning systemd isn't removed from a
> > sysvinit build and vice versa. I should be able to fix that next week
> > before merging. This patch depends on the patch on the bitbake list for
> > the event it needs. Before merging I will bump version number
> > requirements to ensure people have it. I also want to improve the log
> > output so it tells users what its doing rather than the obtuse
> > bb.error().]
> > 
> > [YOCTO #4102]
> 
> Thanks RP, good work, I'm cherry-picking both changes for my next world
> build.

There were some small but annoying bugs in the first version I posted.
I've shaken them out and posted a version which appears to work much
better.

The sysvinit issue turned out to be the metadata markup, not the skipped
metadata problem I was expecting it to be.

I'm quietly excited with this change as I think it should improve
people's experiences in a number of ways.

Cheers,

Richard