[Openembedded-architecture] layer.conf and layerindex change proposal

[Paul Eggleton]

On Wed, 08 Jun 2016 16:15:42 Mark Hatle wrote:
> I've been looking at the layer.conf and layer index and think some minor
> changes to the layer.conf could allow the layer index to be more powerful.
> 
> (background)
> 
> The layer index consists of:
> 
> name
> summary
> description
> URL (git url)
> web URL
> usage url (docs?)
> mailing list url
> 
> layer dependency -> required layer
> 
> recipes
> machines
> branches
> 
> The layer.conf file currently has:
> 
> BBFILE_COLLECTIONS
> 
> LAYERVERSION_<collection>
> LAYERDEPENDS_<collection>
> 
> 
> Assuming the BBFILE_COLLECTIONS is used to specify a unique name for this
> layer...
> 
> 
> (proposal)
> 
> Currently the LAYERVERSION and LAYERDEPENDS can be used to automatically
> construct the dependency/required information.  (It's not actually used this
> way, but could be.)
> 
> Currently a lot of layers are not using these fields, but I would like to
> see them become mandatory.  At a minimum each layer.conf should include
> 'core', even better if they specified the version of core they were
> compatible with...
> 
> I'm thinking LAYERSUMMARY, LAYERDESCRIPTION, LAYERHOMEPAGE could easily be
> added (but optional) so that a layer maintainer could specify values, and
> the layer index could use these for populating the index.
> 
> In addition, many of the layers I work with have a clear set of required
> layers (LAYERDEPENDS), but also have 'recommended' layers...  So I'd love
> to see a LAYERRECOMMENDS as well.
> 
> For this one, in bitbake, we could add the LAYERRECOMMENDS to the priority
> sorting code -- but if the layer is not included it's not an error or
> warning.
> 
> 
> So the work required would be:
> 
> 1) Verify LAYERVERSION and LAYERDEPENDS are defined by a layer.conf
>    - 'core' would have an empty LAYERDEPENDS
> 
> 1.1) Update the layer index to use these, which would require remembering
> the BBFILE_COLLECTIONS name for each layer indexed, so it can be used for
> resolving deps.
> 
> 2) Add support for LAYERRECOMMENDS, require a small change to bitbake
> 
> 3) Add LAYERSUMMARY, LAYERDESCRIPTION and LAYERHOMEPAGE to the
> documentation.conf, and update the layerindex to use their fields if
> present.
>
> Comments?  I know there had been some talk on removing the
> 'BBFILE_COLLECTIONS', but then I don't see any other unique name for a
> layer... I'm not against moving this, but I do think a unique layer name is
> needed... then the rest of these items remain.

The thing about BBFILE_COLLECTIONS is it's a bit of a relic of the past 
support for collections in OE-Classic, before we had layers. Apart from the 
naming it suffers from a few problems:

1) There was the idea that you could have multiple collections defined in a 
single layer, so whilst we might like to rely upon there being a 1-1 
correlation between collection name and layer name, that isn't guaranteed. 
Having said that other than very old branches of meta-intel I don't know of 
any layers that actually do this however.

2) The collection names often don't match the actual layer names. This is why 
I introduced BBLAYERS_LAYERINDEX_NAME, although that variable is likely rarely 
set given that it's only used by bitbake-layers and I haven't really promoted 
it.

3) I don't believe there's actually anything enforcing that layers have unique 
collection names, either in bitbake itself or the layer index. It's not hard 
to imagine that there are layers out there with duplicate collection names and 
due to the lack of enforcement (or adverse consequences other than priority 
values being overwritten) nobody has noticed.

Regarding the plan to use LAYERDEPENDS to drive the layer index, I think 
that's great in theory but I suspect what you'll find is at the moment what we 
have in the layer index is vastly more accurate than what's layer.conf in the 
average layer since the benefits of setting these aren't clear, and you have 
to know what the collection names are - in the layer index it's easier because 
you have the whole list in front of you with names you can recognise.

These are fixable, but particularly renaming the collection names to match the 
layer names is going to cause some upheaval unless we have some kind of smooth 
transition mechanism. My plan for tidying all of this up would be:

1) Have the layer index collect BBFILE_COLLECTIONS from each layer as you 
suggest.

2) Add some validation code into the layer index. This would check that a 
collection was defined by a layer (and only one collection), that the 
collection name matches the layer name in the index, and that it's unique 
(though it would have to be unique to match with the name since the layer 
index doesn't allow duplicate names). We could also validate LAYERDEPENDS 
against the dependencies defined in the index. This would give us an idea of 
the state of the metadata out there, at least for public layers.

3) Add a LAYERPROVIDES or similar so we have a way of avoiding a flag day on 
renaming the collections to match the layer name.

4) After we've cleaned up a good portion of the commonly used public layers, 
have bitbake warn about duplicate collection names and missing LAYERDEPENDS in 
the current configuration.

5) Consider renaming BBFILE_COLLECTIONS, though this is only cosmetic and we'd 
need to read both variables for a time to avoid compatibility issues.


I'm not totally sold on LAYERRECOMMENDS, though I have often been frustrated 
at being forced to add a dependent layer when in fact that dependency exists 
only to satisfy a DEPENDS in one or two recipes that I don't even need to 
build, so it's possible that could help in that situation - but it relies on 
the layer maintainer applying it (and agreeing that it's the right thing to do 
vs. LAYERDEPENDS). Conceivably you could also use LAYERRECOMMENDS to augment 
missing DEPENDS errors e.g. "Nothing PROVIDES xyz - perhaps you need to add 
layer meta-a, meta-b or meta-c to satisfy this?"

As for LAYERSUMMARY, LAYERDESCRIPTION and LAYERHOMEPAGE, we could certainly 
add these although LAYERHOMEPAGE doesn't match up directly with anything we 
have in the layer index (instead we have usage information and the web link to 
the repo, and the latter is mostly set automatically based on the git URL if 
it's a host we know about, e.g. github or git.yoctoproject.org). It's less 
clear to me how useful these are though apart from to save a little time on 
submitting to the layer index (or adding to a private layer index instance).
One problem it will create is that right now entering the summary / 
description are mandatory when submitting to the layer index, but presumably 
if the information was in layer.conf people would want that to be optional in 
the submission form - but then it's optional in the layer.conf as well and we 
won't actually know whether it's set until we parse the layer, which doesn't 
happen until after the layer is manually published (for security and keeping 
some measure of order in the database.)

I have been thinking about some of these shortcomings for a while myself and I 
agree we should do something here. It would be good to hear other opinions as 
well.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre