[OE-core] [RFC] Allarch and packagegroup improvement proposal

[Martin Jansa]

On Mon, Aug 18, 2014 at 09:42:50PM +0800, Robert Yang wrote:
> 
> 
> On 08/18/2014 09:14 PM, Martin Jansa wrote:
> > On Mon, Aug 18, 2014 at 01:46:50PM +0100, Richard Purdie wrote:
> >> As some people are already painfully aware, the current way packagegroup
> >> and allarch interact with the sstate signatures is painful.
> >>
> >> Some example problem cases:
> >>
> >> * An allarch package of scripts with an interpretor dependency (e.g perl)
> >>
> >> * A packagegroup with dependencies on something which is debian renamed
> >>    e.g. build-essentials on libgmp
> >>
> >>
> >> Currently, packagegroups default to allarch and have their sstate
> >> dependencies truncated. This means that when something like gmp
> >> rebuilds, it can change package name thanks to debian renaming but the
> >> packagegroup referencing the old name remains, causing image creation
> >> failures.
> >>
> >> So the "obvious" fix is to stop truncating the dependencies? Well, that
> >> means that the allarch package is written out every time machine changes
> >> to a different arch.
> >>
> >> So we need to make packagegroups PACKAGE_ARCH specific? Well, this still
> >> doesn't solve the problem of allarch packages having ever greater
> >> problems with things like dependencies on perl.
> >
> > Well then maybe that allarch package with perl dependency shouldn't be
> > marked as allarch.
> >
> >> An alternative solution I'm wondering about to address both issues is:
> >>
> >> * the sstatesig packagegroup code for dependencies handling should only
> >> trigger for allarch
> >> * we add a sanity test that checks if renaming has happened for
> >> packagegroup dependencies and warn if it has.
> >>
> >> The trouble is the sstatesig code doesn't have the full data store, only
> >> a limited cross section so it can't check if PACKAGE_ARCH == all, only
> >> if the class is inherited. We therefore probably need to change
> >> packagegroup to do conditional class inclusion (which is possible with
> >> the modern code base at least). The use of the class can be tested from
> >> the sstatesig code and hence indicate allarch.
> >>
> >>
> >> For the second part, we could add something like:
> >>
> >> +    if bb.data.inherits_class('packagegroup', d) and bb.data.inherits_class('allarch', d):
> >> +        if set(deps) != set(new_depends):
> >> +            bb.warn("allarch packagegroup %s has renamed dependency %s" % (pkg, str(set(deps) ^ set(new_depends))))
> >>
> >> to runtime_mapping_rename in package.bbclass which tells us which
> >> packagegroups need to move to be PACKAGE_ARCH. Unfortunately, a quick
> >> run of OE-Core shows:
> >>
> >> WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
> >> WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
> >> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
> >> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
> >> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
> >> WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
> >> WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dbg has renamed dependency set(['libsdl-dbg', 'libsdl-1.2-dbg'])
> >> WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dev has renamed dependency set(['libsdl-dev', 'libsdl-1.2-dev'])
> >> WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
> >> WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
> >> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
> >> WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
> >> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
> >> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
> >> WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])
> >> WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])
> >>
> >> so we do have the problem in a number of places.
> >>
> >> Does anyone have any thoughts?
> >
> > I'm in favor of removing default allarch and setting correct
> > PACKAGE_ARCH manually in the packagegroup recipes like we do elsewhere.
> 
> I did a rough find and grep in oe-core, we have 27 packagegroup recipes:
> $ find meta -name 'packagegroup-*.bb' | wc -l
> 27
> 
> And 7 ones which set the PACKAGE_ARCH:
> $ grep PACKAGE_ARCH meta -r | grep 'packagegroup-.*.bb:' | wc -l
> 7
> 
> It seems that allarch still wins.
> 
> I think that we really need such a WARNING not matter whether remove the
> default allarch or not. (if removed, we have to set the allarch for a few
> packagegroup recipes, then we still need such a check.)

FWIW: there is also scripts/sstate-diff-machines.sh which can easily
show when some packagegroup recipe has incorrect PACKAGE_ARCH

Example output:
http://lists.openembedded.org/pipermail/openembedded-core/2014-August/095195.html

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20140818/9b98db93/attachment-0002.sig>