[OE-core] [PATCH] bitbake.conf: point PERSISTENT_DIR outside TMPDIR

[Richard Purdie]

On Fri, 2011-03-04 at 15:53 +0100, Martin Jansa wrote:
> On Fri, Mar 04, 2011 at 02:01:26PM +0000, Richard Purdie wrote:
> > On Fri, 2011-03-04 at 11:40 +0100, Martin Jansa wrote:
> > > * sanity check warning user that he still has peristent cache in old location
> > > * similar patch was sent to OE, to keep PERSISTENT_DIR location consistent
> > >   http://lists.linuxtogo.org/pipermail/openembedded-devel/2011-March/030623.html
> > > 
> > > Signed-off-by: Martin Jansa <Martin.Jansa at gmail.com>
> > > ---
> > >  meta/classes/sanity.bbclass |    8 ++++++++
> > >  meta/conf/bitbake.conf      |    2 +-
> > >  meta/conf/sanity.conf       |    1 +
> > >  3 files changed, 10 insertions(+), 1 deletions(-)
> > 
> > When I saw this on the OE list, I'd assumed you'd done what Poky had
> > done, I've just realised you're moving this file to TOPDIR instead of
> > TMPDIR.
> > 
> > My big concern here is that we really shouldn't be writing to data to
> > the conf directory as its not what the user expects.
> > 
> > I say this knowing the raft of usability problems we've had every time
> > we take something outside of TMPDIR as users have the expectation that
> > deleting tmp wipes out all stored state and rebuilds. Having this file
> > hidden in the conf directory breaks that expectation. Note there are
> > only two things we place outside of TMPDIR, downloads and sstate data.
> > Even things like buildstats which you might want to preserve over
> > different builds are still inside TMPDIR.
> 
> From my experience I always had to say:
> 
> You have to rebuild from scratch. Just wipe out tmp directory, but don't
> forget to save&load tmp/cache/bb_persist_data.sqlite3 because that file is
> really important for upgradable path.

Its really important to a very specific use case albeit a use case that
you use and therefore prioritise.

It will be an annoyance to general user who doesn't use floating source
revisions.

> So that was my motivation to move it outside TMPDIR and no strong
> opinion about keeping it inside conf/toplevel/any other directory.
> 
> > Can we put it in cache as a top level file rather than machine specific?
> 
> "machine specific"?
> 
> something like?
> PERSISTENT_DIR = "${TOPDIR}/persistent"
> or?
> PERSISTENT_DIR = "${TOPDIR}/persistent-cache"
> would be OK?

No, I was meaning to keep it within TMPDIR/cache which is what you want
to change. I think if we want to move this outside TMPDIR, we need to
look to solve the overall/underlying problem better.

The list of things that don't really fit TMPDIR I can think of off the
top of my head are:

* build stats
* downloads
* sstate
* pkghistory

The trouble is the persist cache doesn't fit well against those as the
above can all be shared between machines over NFS, the persist cache
can't. I had been thinking about some kind of shared directory but
persistent data doesn't fit.

Reimplementing it as a flat on disk structure would have the benefit
that it could then be shared amongst a group of autobuilder type
machines over NFS. The other solution we've talked about is a single
daemon talking over the network where the daemon maintains the database.
We do have the problem of how to control the revision numbers for
packages looming in the future with the advent of full sstate enablement
since manual PR bumps will become unneeded which is why I'm thinking
about this very seriously.

Does anyone else have any thoughts on this?

Cheers,

Richard