[oe] [RFC] Layers, PRINC and bbappends

[Koen Kooi]

Op 15 mei 2013, om 16:44 heeft Paul Eggleton <paul.eggleton at linux.intel.com> het volgende geschreven:

> Hi Koen,
> 
> On Tuesday 14 May 2013 09:36:02 Koen Kooi wrote:
>> What triggered PR going backwards this time was a BSP removing its netbase
>> bbappend because it wasn't needed anymore. That's great, I hate netbase
>> bbappends since they tend to be ifupdown specific and don't work as
>> intended with connman/NM/netctl/etc. Long story short:
>> 
>> 	ERROR: Package version for package netbase-dbg went backwards which would
>> break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package version
>> for package netbase-staticdev went backwards which would break package
>> feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package version for package
>> netbase-dev went backwards which would break package feeds from (0:5.0-r3.1
>> to 0:5.0-r1.2) ERROR: Package version for package netbase-doc went
>> backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
>> ERROR: Package version for package netbase-locale went backwards which
>> would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package
>> version for package netbase went backwards which would break package feeds
>> from (0:5.0-r3.1 to 0:5.0-r1.2)
>> 
>> To avoid things like that in the future I have a few recommendations I'd
>> like to get feedback on:
>> 
>> 1) For a given BSP split it in multiple layers:
>> 	a) One 'core' BSP layer with only:
>> 		o Bootloader
>> 		o Kernel
>> 		o Tooling to build bootloader/kernel/image
>> 	b) One BSP layer with bbappends for shadow/netbase/xorg-conf/etc
>> 	c) One or more layers with bbappends for libav/clutter/etc[2]
> 
> I don't like this. This will mean effectively 3x the number of BSP layers, with 
> independent testing effectively expected for the different combinations; I don't 
> think this will be practical. It's fine for distros with pre-supplied lists of 
> layers but lots of users are assembling their own layer stacks and this is 
> just going to add pain for them.

The flip side is that adding BSP layers becomes a lot safer. I don't mind adding layers. Adding layers is easy. Removing layers is pretty much impossible, though.

> It's probably time we looked seriously at more effective analysis tools for 
> examining what a layer is actually doing. We now have the variable history in 
> bitbake which means that pinpointing a change to a variable down to the 
> specific line in the file that made the change is now possible; we should be 
> able to build upon this to provide tools that determine when e.g. BSP layers 
> are doing things that affect builds for other machines.
> 
> Of course, getting maintainers to deal with these issues is another problem, 
> but identifying them in the first place would help a lot.

Yes, layer maintainers are proving to be the weak link is the layer system.

>> 2) *Never* use PRINC in type b) bbappends
>> 
>> 3) Avoid PRINC in general
> 
> Do we even need PRINC at all any more? I realise we have PRINC values around 
> in bbappends that we can't easily drop without causing the problem you 
> mentioned.

If PRSERV would work reliably we wouldn't need it. But even with the current flaky PRSERV PRINC is causing more problems than it's supposed to solve.

>> 4) make buildhistory.bbclass mandatory in OE-core
> 
> Buildhistory does have some overhead as you know, which is why it's not on by 
> default. I wonder if we could move the version going backwards check to 
> package.bbclass or package QA, instead making it read the previous values from 
> pkgdata. I'm not sure if sstate would get in the way in that the previous set 
> of files might already be removed by the time it got to do_package; this would 
> need to be investigated. Even if that is the case it might be that we can work 
> around it.

I personally think the overhead is worth it, but as you say, moving bits to package.bbclass (and maybe using a sqlite db) would be very good as well.

>> During the last TSC meeting we discussed that there is currently no way to
>> force maintainers to behave. I think the most we can do is have clear
>> guidelines on the OE side and enforce those during the "Yocto Project
>> Compatible" process. 
> 
> Being more specific in the Yocto Project Compatible process requirements might 
> help; I wasn't involved in developing that process so I can't really comment 
> more than that. That wouldn't help with other layers in the OE community that 
> aren't put through that process however.
> 
> I think we need to improve the documentation we have about what is and what is 
> not appropriate in a BSP layer. Most of the time it's not that maintainers are 
> too lazy or deliberately setting out to force values, it's that they don't 
> know they're doing anything wrong because we haven't documented all of the 
> best practices anywhere. There was recently a question on this very mailing 
> list about what constitute distro policy settings in the context of what 
> shouldn't go into a BSP layer; the response has been a resounding "".
> 
>> And have the layer index orange/red flag layers that
>> do stupid things. Come to think of it, that would actually be the most
>> visible way to go, having wikipedia style warning banners on the offending
>> layers.
> 
> I think that's reasonable as long as maintainers can understand the problem. 
> We'd need to be notifying maintainers directly about specifically what they are 
> doing wrong and how to fix it; just putting a warning in the index isn't enough 
> IMO. The layer index does record maintainer information and regularly reads 
> layer metadata, so we have the infrastructure to do this now, just not the 
> specific tools.

My thinking was that adding such a banner would email the maintainers of that layer. That still assumes that the layer maintainer actually reads email, which I don't think is true for a lot of layers :/

regards,

Koen