[oe] [PATCH] bitbake.conf: define shared PERSISTENT_DIR

[Martin Jansa]

On Thu, Feb 24, 2011 at 04:25:56PM +0100, Martin Jansa wrote:
> * import from poky
>   commit 6a11cf7dfe930461a6660e3b783b546fa2634900
>   Author: Richard Purdie <rpurdie at linux.intel.com>
>   Date:   Sun May 31 22:23:30 2009 +0100
> * we had CACHE var pointing to machine specific dir since
>   commit caf077679022f37ce55d758101f130e4e93bd7b5
>   Author: Richard Purdie <rpurdie at rpsys.net>
>   Date:   Mon Nov 20 17:40:20 2006 +0000
> 
> * current bitbake is looking for cache dir like this:
>   cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
>               bb.data.getVar("CACHE", d, True))
>   which without PERSISTENT_DIR defined picks machine-specific CACHE dir
>   so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
>   in same builddir :/
> 
> Signed-off-by: Martin Jansa <Martin.Jansa at gmail.com>
> ---
>  conf/bitbake.conf |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/conf/bitbake.conf b/conf/bitbake.conf
> index dcc5c70..53bb7b7 100644
> --- a/conf/bitbake.conf
> +++ b/conf/bitbake.conf
> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
>  
>  TMPDIR = "${TOPDIR}/tmp"
>  CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
> +# The persistent cache should be shared by all builds
> +PERSISTENT_DIR = "${TMPDIR}/cache"

Or even better we can move it outside ${TMPDIR} and add sanity check to
warn user that he still has ${CACHE}/bb_persist_data.sqlite3 and should
move it to ${PERSISTENT_DIR}/bb_persist_data.sqlite3. (I'll send such
patch if requested).

But while thinking about it I found that there are probably other issues
with that too.

Machine-specific persistent cache was rebuilding packages with
FEED_ARCH, which works ok only as long as all machines started with same
cache and all machines are always built (so _count are incremented
consistently in all caches).

Shared persistent cache is much better, but still fails if some machine
is overriding SRCREV_machine ie in u-boot_git it's quite common with
same git SRC_URI and same branch (so key in persistent cache is the
same, but _rev is different so _count is incremented twice every time
this _machine is built. But overriding SRCREV is less common than
multiple machines building _same_ package with FEED_ARCH or even "all".

It could be maybe improved if we add actual PACKAGE_ARCH to key, but I
think that's not available while bitbake is parsing recipe (ie before
calling utils.is_machine_specific and changing PACKAGE_ARCH to
MACHINE_ARCH.

This improved cache would provide consistent _count, but will still fail
on target. ie when SRCREV for FEED_ARCH was changed twice and for
om-gta02 just once:

Package: frameworkd-config
Version: 1:0.9.5.9+gitr2+e31fa77906981a56be2271246693fd929c973abf-r12.6
Architecture: armv4t

will be preferred instead of

Package: frameworkd-config
Version: 1:0.9.5.9+gitr1+aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-r12.6
Architecture: om-gta02

even with:
SHR root at gjama / $ cat /etc/opkg/arch.conf 
..
arch armv4t 26
arch om-gta02 31

because afaik opkg prefers newer version before higher arch priority.

I don't see how to fix this in all corner cases :/.
But shared PERSISTENT_DIR outside TMPDIR seems much better than machine
specific in CACHE.

BTW: I'm quite frustrated lately to see that OE is great only for 99%,
but last 1% of corner cases like this and ie substrings in packages with
debian.bbclass are quite hard to find/debug/explain/fix :(.

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.openembedded.org/pipermail/openembedded-devel/attachments/20110224/23be73c4/attachment-0002.sig>