[OE-core] [PATCH] sstate: Improve handling of machine specific manifests

Mon Oct 22 13:17:05 UTC 2012

On Mon, Oct 22, 2012 at 02:02:34PM +0200, Martin Jansa wrote:
> On Mon, Oct 22, 2012 at 12:42:51PM +0100, Richard Purdie wrote:
> > On Mon, 2012-10-22 at 13:08 +0200, Martin Jansa wrote:
> > > On Mon, Oct 22, 2012 at 11:58:28AM +0100, Richard Purdie wrote:
> > > > On Mon, 2012-10-22 at 12:44 +0200, Martin Jansa wrote:
> > > > > On Fri, Oct 19, 2012 at 03:48:55PM +0100, Richard Purdie wrote:
> > > > > > Now do_package isn't machine specific, we're only left with do_populate_sysroot as a
> > > > > > machine specific task. This change marks only the machine specific manifests as machine
> > > > > > specific, defaulting to PACKAGE_ARCH for everything else.
> > > > > > 
> > > > > > This means we do less work where there are multiple machines using the same
> > > > > > core package architecture and we can start to clean up the sstate duplicate files
> > > > > > whitelist.
> > > > > > 
> > > > > > Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
> > > > > > ---
> > > > > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > > > > > index d2a120b..dee84bf 100644
> > > > > > --- a/meta/classes/sstate.bbclass
> > > > > > +++ b/meta/classes/sstate.bbclass
> > > > > > @@ -17,10 +17,7 @@ SSTATE_EXTRAPATH   = ""
> > > > > >  SSTATE_EXTRAPATHWILDCARD = ""
> > > > > >  SSTATE_PATHSPEC   = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}*/${SSTATE_PKGSPEC}"
> > > > > >  
> > > > > > -# In theory we should be using:
> > > > > > -# SSTATE_DUPWHITELIST = "${DEPLOY_DIR_IMAGE}/ ${DEPLOY_DIR}/licenses/ ${DEPLOY_DIR_IPK}/all/ ${DEPLOY_DIR_RPM}/all ${DEPLOY_DIR_DEB}/all/ ${TMPDIR}/pkgdata/all${TARGET_VENDOR}-${TARGET_OS}"
> > > > > > -# However until do_package is not machine specific, we'll have to make do with all of deploy/pkgdata.
> > > > > > -SSTATE_DUPWHITELIST = "${DEPLOY_DIR}/ ${TMPDIR}/pkgdata/"
> > > > > > +SSTATE_DUPWHITELIST = "${DEPLOY_DIR_IMAGE}/ ${DEPLOY_DIR}/licenses/"
> > > > > 
> > > > > Looks like warnings are back :/
> > > > > 
> > > > > WARNING: The recipe attr is trying to install files into a shared area when those files already exist. Those files are:
> > > > >    /OE/jansa-test/shr-core/tmp-eglibc/deploy/ipk/armv7a-vfp-neon/attr-locale-de_2.4.46-r4_armv7a-vfp-neon.ipk
> > > > >    /OE/jansa-test/shr-core/tmp-eglibc/deploy/ipk/armv7a-vfp-neon/attr-dbg_2.4.46-r4_armv7a-vfp-neon.ipk
> > > > >    /OE/jansa-test/shr-core/tmp-eglibc/deploy/ipk/armv7a-vfp-neon/attr-locale-sv_2.4.46-r4_armv7a-vfp-neon.ipk
> > > > > ...
> > > > > 
> > > > > and new warnings from pkgdata
> > > > > WARNING: The recipe bison is trying to install files into a shared area when those files already exist. Those files are:
> > > > >    /OE/jansa-test/shr-core/tmp-eglibc/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/bison
> > > > >    /OE/jansa-test/shr-core/tmp-eglibc/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime/bison-locale-nl.packaged
> > > > >    /OE/jansa-test/shr-core/tmp-eglibc/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime/bison-dbg.packaged
> > > > >    /OE/jansa-test/shr-core/tmp-eglibc/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime/bison-doc
> > > > >    /OE/jansa-test/shr-core/tmp-eglibc/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime/bison-locale-th.packaged
> > > > > ...
> > > > 
> > > > The question is why as they shouldn't be, these changes were meant to
> > > > fix this properly. Initially I wondered if this was another
> > > > manifestation of https://bugzilla.yoctoproject.org/show_bug.cgi?id=3219
> > > > but I'm not so sure.
> > > 
> ]> > Probably not as this happens on builder with 2 machines using the same
> > > tune and the same CCARGS.
> > > 
> > > > Can you figure out which two recipes are trying to install these sets of
> > > > files?
> > > 
> > > I'll try to compare them with scripts/sstate-diff.sh again to see if
> > > checksums are the same between those 2 machines, but those warnings are
> > > shown already when building 1 machine.

Hmm, checksums are different not only for target recipes but also for native now:

OE @ ~/jansa-test/shr-core $ ls */*/*/zlib-native*do_package.sigdata*
stamps.1350908965/nokia900/x86_64-linux/zlib-native-1.2.7-r0.do_package.sigdata.f8ca8403df4a25b68553291341717a29
stamps.1350908965/om-gta02/x86_64-linux/zlib-native-1.2.7-r0.do_package.sigdata.af2b16d88415c39245731409c8e15f70
stamps.1350908965/tuna/x86_64-linux/zlib-native-1.2.7-r0.do_package.sigdata.815e9385589d3ffe41099fca79c3d8e7
OE @ ~/jansa-test/shr-core $ ls */*/*/zlib-1*do_package.sigdata*
stamps.1350908965/nokia900/armv7a-vfp-neon-oe-linux-gnueabi/zlib-1.2.7-r0.do_package.sigdata.3c23b95f321e5430b3d21d7288acea89
stamps.1350908965/om-gta02/armv4t-oe-linux-gnueabi/zlib-1.2.7-r0.do_package.sigdata.bcfc3d562d4dc6d85b2d8bf0aa7f0b89
stamps.1350908965/tuna/armv7a-vfp-neon-oe-linux-gnueabi/zlib-1.2.7-r0.do_package.sigdata.0b50fd9e90c6c922877f637378280163

OE om-gta02 at shr ~/jansa-test/shr-core $ bitbake-diffsigs stamps.1350908965/nokia900/armv7a-vfp-neon-oe-linux-gnueabi/zlib-1.2.7-r0.do_package.sigdata.3c23b95f321e5430b3d21d7288acea89 stamps.1350908965/tuna/armv7a-vfp-neon-oe-linux-gnueabi/zlib-1.2.7-r0.do_package.sigdata.0b50fd9e90c6c922877f637378280163
basehash changed from ff270729884d94ba712e579e19e9989f to b01c64c65799d2a3febcf7ec164a8b14
Variable PKGTRIPLETS value changed from nokia900-oe-linux-gnueabi armv7a-vfp-neon-oe-linux-gnueabi armv7a-vfp-oe-linux-gnueabi armv7a-oe-linux-gnueabi armv6-vfp-oe-linux-gnueabi armv5e-vfp-oe-linux-gnueabi armv5e-oe-linux-gnueabi armv5-vfp-oe-linux-gnueabi armv5-oe-linux-gnueabi armv4-oe-linux-gnueabi arm-oe-linux-gnueabi noarch-oe-linux-gnueabi any-oe-linux-gnueabi all-oe-linux-gnueabi to tuna-oe-linux-gnueabi armv7a-vfp-neon-oe-linux-gnueabi armv7a-vfp-oe-linux-gnueabi armv7a-oe-linux-gnueabi armv6-vfp-oe-linux-gnueabi armv5e-vfp-oe-linux-gnueabi armv5e-oe-linux-gnueabi armv5-vfp-oe-linux-gnueabi armv5-oe-linux-gnueabi armv4-oe-linux-gnueabi arm-oe-linux-gnueabi noarch-oe-linux-gnueabi any-oe-linux-gnueabi all-oe-linux-gnueabi
Hash for dependent task eglibc_2.16.bb.do_package changed from 44e54b452f831141f15d7fa61a637997 to 31b9950443659def02296d4b2aed43a5
Hash for dependent task gcc-runtime_4.7.bb.do_package changed from 7db515c51f06c6db5a77e8f089aedf64 to 6e1ee054521f3a41fcc6e17aa0d8acbc

OE om-gta02 at shr ~/jansa-test/shr-core $ bitbake-diffsigs stamps.1350908965/nokia900/x86_64-linux/zlib-native-1.2.7-r0.do_package.sigdata.f8ca8403df4a25b68553291341717a29 stamps.1350908965/tuna/x86_64-linux/zlib-native-1.2.7-r0.do_package.sigdata.815e9385589d3ffe41099fca79c3d8e7
basehash changed from b086f2a2ad21c163d0002edd355103fb to 61aa7f2d5d3ec073cbde0e1df1b0813b
Variable PKGTRIPLETS value changed from nokia900-linux armv7a-vfp-neon-linux armv7a-vfp-linux armv7a-linux armv6-vfp-linux armv5e-vfp-linux armv5e-linux armv5-vfp-linux armv5-linux armv4-linux arm-linux noarch-linux any-linux all-linux to tuna-linux armv7a-vfp-neon-linux armv7a-vfp-linux armv7a-linux armv6-vfp-linux armv5e-vfp-linux armv5e-linux armv5-vfp-linux armv5-linux armv4-linux arm-linux noarch-linux any-linux all-linux

I guess I don't need to build test anything now..

> > > > Or perhaps this is a one off transition issue I didn't see here when
> > > > testing this? Does a build from a clean tmp do this?
> > > 
> > > Yes I've removed tmp-eglibc before starting this build (kept only
> > > sstate-cache) and it's building first machine.
> > 
> > If the warnings are showing up even after building one machine, there is
> > something very strange going on. Did it run the setscene do_package task
> > for these recipes?
> 
> No, only for populate_lic and populate_sysroot:
> 
> NOTE: recipe attr-2.4.46-r4: task do_populate_sysroot_setscene: Started
> NOTE: recipe attr-2.4.46-r4: task do_populate_sysroot_setscene: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_populate_lic_setscene: Started
> NOTE: recipe attr-2.4.46-r4: task do_populate_lic_setscene: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_fetch: Started
> NOTE: recipe attr-2.4.46-r4: task do_fetch: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_unpack: Started
> NOTE: recipe attr-2.4.46-r4: task do_unpack: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_patch: Started
> NOTE: recipe attr-2.4.46-r4: task do_patch: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_configure: Started
> NOTE: recipe attr-2.4.46-r4: task do_configure: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_compile: Started
> NOTE: recipe attr-2.4.46-r4: task do_compile: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_install: Started
> NOTE: recipe attr-2.4.46-r4: task do_install: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_populate_sysroot: Started
> NOTE: recipe attr-2.4.46-r4: task do_populate_sysroot: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_package: Started
> NOTE: recipe attr-2.4.46-r4: task do_package: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_package_write_ipk: Started
> NOTE: recipe attr-2.4.46-r4: task do_package_write_ipk: Succeeded
> NOTE: recipe attr-2.4.46-r4: task do_rm_work: Started
> NOTE: recipe attr-2.4.46-r4: task do_rm_work: Succeeded
> 
> But I see this behavior when setscene tasks are executed and 
> later it rebuilds it again quite often (assuming that some error 
> in setscene task was hidden - e.g. that fetcher error from
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=3250
> wasn't shown until lately, but maybe was there all the time)
> 
> > Its as if it installed from sstate and then decided to overwrite the
> > files. Something isn't making sense though :/. Firstly, the sstate
> > should have been invalidated due to the package class and variable
> > changes. Secondly, if it had installed the files, it should be
> > uninstalled them before installing a second set. So I'm confused as to
> > what is going on here...
> 
> It seems to reinstall a lot of files in sysroot too (which weren't shown before).
> 
> e.g. all files from gst-plugins-good now show warning
> WARNING: The recipe gst-plugins-good is trying to install files into a shared area when those files already exist. Those files are:
>    /OE/jansa-test/shr-core/tmp-eglibc/sysroots/tuna/usr/share/gstreamer-0.10/presets/GstIirEqualizer10Bands.prs
>    /OE/jansa-test/shr-core/tmp-eglibc/sysroots/tuna/usr/share/gstreamer-0.10/presets/GstIirEqualizer3Bands.prs
> ...
> 
> I'll try to wipe tmp-eglibc again after this build finishes to see if it was
> one off or if it will repeat again.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20121022/b8fe5144/attachment-0002.sig>