[oe] [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR

[Martin Jansa]

On Fri, Mar 04, 2011 at 12:11:46AM +0000, Richard Purdie wrote:
> On Thu, 2011-03-03 at 07:38 -0700, Tom Rini wrote:
> > On 03/03/2011 12:27 AM, Martin Jansa wrote:
> > > On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote:
> > >> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900
> > >> * we had CACHE var pointing to machine specific dir since 2006
> > >>    caf077679022f37ce55d758101f130e4e93bd7b5
> > >> * current bitbake is looking for cache dir like this:
> > >>    cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
> > >>                bb.data.getVar("CACHE", d, True))
> > >>    which without PERSISTENT_DIR defined picks machine-specific CACHE dir
> > >>    so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
> > >>    in same builddir :/
> > >
> > > ping?
> > >
> > 
> > I'd like Richards comments here.
> 
> The intent was always to have CACHE be machine specific and
> PERSISTENT_DIR not be machine specific. This means all builds share the

> idea of revision numbers yet the main cache is machine specific so you
> don't hit reparses when switching machines unnecessarily.

BTW: with bitbake master and shared CACHE on one system I noticed that
if I set MACHINE/DISTRO only in shell environment (with correct
BB_ENV_EXTRAWHITE), then cache is not invalidated and metadata reparsed,
resulting in broken builds where parts were respecting ${MACHINE} from
shell and some parts had different machine from previous build.

IIRC I noticed it on different machine listed in build summary and then
another one in filename of machine-specific ipk. Removing cache after
switching MACHINE in shell env worked fine.

Regards,

> So yes, I think the change is valid, I hadn't realised OE didn't have
> that change and I'm happy to ack it...
> 
> Cheers,
> 
> Richard
> 
> 
> > >>
> > >> Signed-off-by: Martin Jansa<Martin.Jansa at gmail.com>
> > >> ---
> > >>   classes/sanity.bbclass |   12 ++++++++++++
> > >>   conf/bitbake.conf      |    2 ++
> > >>   conf/sanity.conf       |    1 +
> > >>   3 files changed, 15 insertions(+), 0 deletions(-)
> > >>
> > >> diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass
> > >> index 1e6b170..ffe3a49 100644
> > >> --- a/classes/sanity.bbclass
> > >> +++ b/classes/sanity.bbclass
> > >> @@ -189,6 +189,18 @@ def check_sanity(e):
> > >>   	if messages != "":
> > >>   		raise_sanity_error(messages)
> > >>
> > >> +	#
> > >> +	# Check there isn't old persistent cache
> > >> +	#
> > >> +	cache = data.getVar('CACHE', e.data, True)
> > >> +	persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True)
> > >> +	persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True)
> > >> +	if os.path.exists(cache + '/' + persistent_cache_filename):
> > >> +		messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir)
> > >> +
> > >> +	if messages != "":
> > >> +		raise_sanity_error(messages)
> > >> +
> > >>   python check_sanity_eventhandler() {
> > >>       if isinstance(e, bb.event.BuildStarted):
> > >>           check_sanity(e)
> > >> diff --git a/conf/bitbake.conf b/conf/bitbake.conf
> > >> index 88ad2ba..0c11cdb 100644
> > >> --- a/conf/bitbake.conf
> > >> +++ b/conf/bitbake.conf
> > >> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
> > >>
> > >>   TMPDIR = "${TOPDIR}/tmp"
> > >>   CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
> > >> +# The persistent cache should be shared by all builds
> > >> +PERSISTENT_DIR = "${TOPDIR}/conf/persistent"
> > >>   CO_DIR = "${DL_DIR}"
> > >>   CVSDIR = "${CO_DIR}/cvs"
> > >>   SVNDIR = "${CO_DIR}/svn"
> > >> diff --git a/conf/sanity.conf b/conf/sanity.conf
> > >> index e9902f6..c1ac332 100644
> > >> --- a/conf/sanity.conf
> > >> +++ b/conf/sanity.conf
> > >> @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2"
> > >>
> > >>   SANITY_ABIFILE = "${TMPDIR}/abi_version"
> > >>   SANITY_PRFILE = "${TMPDIR}/distro_pr"
> > >> +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3"
> > >>
> > >>   INHERIT += "sanity"
> > >> --
> > >> 1.7.4.1
> > >>
> > >
> > >
> > >
> > > _______________________________________________
> > > Openembedded-devel mailing list
> > > Openembedded-devel at lists.openembedded.org
> > > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> > 
> > 
> 
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel at lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.openembedded.org/pipermail/openembedded-devel/attachments/20110304/18019725/attachment-0002.sig>