[OE-core] [RFC] Allarch and packagegroup improvement proposal

[Robert Yang]

On 08/18/2014 09:14 PM, Martin Jansa wrote:
> On Mon, Aug 18, 2014 at 01:46:50PM +0100, Richard Purdie wrote:
>> As some people are already painfully aware, the current way packagegroup
>> and allarch interact with the sstate signatures is painful.
>>
>> Some example problem cases:
>>
>> * An allarch package of scripts with an interpretor dependency (e.g perl)
>>
>> * A packagegroup with dependencies on something which is debian renamed
>>    e.g. build-essentials on libgmp
>>
>>
>> Currently, packagegroups default to allarch and have their sstate
>> dependencies truncated. This means that when something like gmp
>> rebuilds, it can change package name thanks to debian renaming but the
>> packagegroup referencing the old name remains, causing image creation
>> failures.
>>
>> So the "obvious" fix is to stop truncating the dependencies? Well, that
>> means that the allarch package is written out every time machine changes
>> to a different arch.
>>
>> So we need to make packagegroups PACKAGE_ARCH specific? Well, this still
>> doesn't solve the problem of allarch packages having ever greater
>> problems with things like dependencies on perl.
>
> Well then maybe that allarch package with perl dependency shouldn't be
> marked as allarch.
>
>> An alternative solution I'm wondering about to address both issues is:
>>
>> * the sstatesig packagegroup code for dependencies handling should only
>> trigger for allarch
>> * we add a sanity test that checks if renaming has happened for
>> packagegroup dependencies and warn if it has.
>>
>> The trouble is the sstatesig code doesn't have the full data store, only
>> a limited cross section so it can't check if PACKAGE_ARCH == all, only
>> if the class is inherited. We therefore probably need to change
>> packagegroup to do conditional class inclusion (which is possible with
>> the modern code base at least). The use of the class can be tested from
>> the sstatesig code and hence indicate allarch.
>>
>>
>> For the second part, we could add something like:
>>
>> +    if bb.data.inherits_class('packagegroup', d) and bb.data.inherits_class('allarch', d):
>> +        if set(deps) != set(new_depends):
>> +            bb.warn("allarch packagegroup %s has renamed dependency %s" % (pkg, str(set(deps) ^ set(new_depends))))
>>
>> to runtime_mapping_rename in package.bbclass which tells us which
>> packagegroups need to move to be PACKAGE_ARCH. Unfortunately, a quick
>> run of OE-Core shows:
>>
>> WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
>> WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
>> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
>> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
>> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
>> WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
>> WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dbg has renamed dependency set(['libsdl-dbg', 'libsdl-1.2-dbg'])
>> WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dev has renamed dependency set(['libsdl-dev', 'libsdl-1.2-dev'])
>> WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
>> WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
>> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
>> WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
>> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
>> WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
>> WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])
>> WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])
>>
>> so we do have the problem in a number of places.
>>
>> Does anyone have any thoughts?
>
> I'm in favor of removing default allarch and setting correct
> PACKAGE_ARCH manually in the packagegroup recipes like we do elsewhere.

I did a rough find and grep in oe-core, we have 27 packagegroup recipes:
$ find meta -name 'packagegroup-*.bb' | wc -l
27

And 7 ones which set the PACKAGE_ARCH:
$ grep PACKAGE_ARCH meta -r | grep 'packagegroup-.*.bb:' | wc -l
7

It seems that allarch still wins.

I think that we really need such a WARNING not matter whether remove the
default allarch or not. (if removed, we have to set the allarch for a few
packagegroup recipes, then we still need such a check.)

// Robert

>
> packagegroups are small and "rebuilt" quickly, so I don't mind
> "building" them once per TUNE_PKGARCH or even once per MACHINE_ARCH like
> we do for couple of them already.
>
> I can even find few changes from me on ML which do exactly that.
>
>
>