[Openembedded-architecture] layer.conf and layerindex change proposal

[Mark Hatle]

On 6/9/16 12:04 AM, Paul Eggleton wrote:
> On Wed, 08 Jun 2016 16:15:42 Mark Hatle wrote:
>> I've been looking at the layer.conf and layer index and think some minor
>> changes to the layer.conf could allow the layer index to be more powerful.
>>
>> (background)
>>
>> The layer index consists of:
>>
>> name
>> summary
>> description
>> URL (git url)
>> web URL
>> usage url (docs?)
>> mailing list url
>>
>> layer dependency -> required layer
>>
>> recipes
>> machines
>> branches
>>
>> The layer.conf file currently has:
>>
>> BBFILE_COLLECTIONS
>>
>> LAYERVERSION_<collection>
>> LAYERDEPENDS_<collection>
>>
>>
>> Assuming the BBFILE_COLLECTIONS is used to specify a unique name for this
>> layer...
>>
>>
>> (proposal)
>>
>> Currently the LAYERVERSION and LAYERDEPENDS can be used to automatically
>> construct the dependency/required information.  (It's not actually used this
>> way, but could be.)
>>
>> Currently a lot of layers are not using these fields, but I would like to
>> see them become mandatory.  At a minimum each layer.conf should include
>> 'core', even better if they specified the version of core they were
>> compatible with...
>>
>> I'm thinking LAYERSUMMARY, LAYERDESCRIPTION, LAYERHOMEPAGE could easily be
>> added (but optional) so that a layer maintainer could specify values, and
>> the layer index could use these for populating the index.
>>
>> In addition, many of the layers I work with have a clear set of required
>> layers (LAYERDEPENDS), but also have 'recommended' layers...  So I'd love
>> to see a LAYERRECOMMENDS as well.
>>
>> For this one, in bitbake, we could add the LAYERRECOMMENDS to the priority
>> sorting code -- but if the layer is not included it's not an error or
>> warning.
>>
>>
>> So the work required would be:
>>
>> 1) Verify LAYERVERSION and LAYERDEPENDS are defined by a layer.conf
>>    - 'core' would have an empty LAYERDEPENDS
>>
>> 1.1) Update the layer index to use these, which would require remembering
>> the BBFILE_COLLECTIONS name for each layer indexed, so it can be used for
>> resolving deps.
>>
>> 2) Add support for LAYERRECOMMENDS, require a small change to bitbake
>>
>> 3) Add LAYERSUMMARY, LAYERDESCRIPTION and LAYERHOMEPAGE to the
>> documentation.conf, and update the layerindex to use their fields if
>> present.
>>
>> Comments?  I know there had been some talk on removing the
>> 'BBFILE_COLLECTIONS', but then I don't see any other unique name for a
>> layer... I'm not against moving this, but I do think a unique layer name is
>> needed... then the rest of these items remain.
> 
> The thing about BBFILE_COLLECTIONS is it's a bit of a relic of the past 
> support for collections in OE-Classic, before we had layers. Apart from the 
> naming it suffers from a few problems:
> 
> 1) There was the idea that you could have multiple collections defined in a 
> single layer, so whilst we might like to rely upon there being a 1-1 
> correlation between collection name and layer name, that isn't guaranteed. 
> Having said that other than very old branches of meta-intel I don't know of 
> any layers that actually do this however.
> 
> 2) The collection names often don't match the actual layer names. This is why 
> I introduced BBLAYERS_LAYERINDEX_NAME, although that variable is likely rarely 
> set given that it's only used by bitbake-layers and I haven't really promoted 
> it.

Based on defacto behavior, we could very easily set BBLAYERS_LAYERINDEX_NAME ?=
BBFILE_COLLECTIONS (or the reverse), but I do think we need some variable that
gives each layer a unique namespace.  This namespace should be the one that is
used by the other definition variables...

> 3) I don't believe there's actually anything enforcing that layers have unique 
> collection names, either in bitbake itself or the layer index. It's not hard 
> to imagine that there are layers out there with duplicate collection names and 
> due to the lack of enforcement (or adverse consequences other than priority 
> values being overwritten) nobody has noticed.

Two layers with the same name can cause problems due to the priority, and
dependency validation..  does it cause ACTUAL problems, not sure.

> Regarding the plan to use LAYERDEPENDS to drive the layer index, I think 
> that's great in theory but I suspect what you'll find is at the moment what we 
> have in the layer index is vastly more accurate than what's layer.conf in the 
> average layer since the benefits of setting these aren't clear, and you have 
> to know what the collection names are - in the layer index it's easier because 
> you have the whole list in front of you with names you can recognise.

I know for my own products, I am directly looking at using these components to
help the user construct their own projects.  This is primarily driven by toaster
having the ability to use the layer index, and automatically download
dependencies.  Since the bitbake system already does dependency validation using
the collection naming, it makes sense to me to use it in the index as well..
(but I do agree the index should have 'manual override' ability because some
layer maintainers will never fix these things.)

Then again, maybe we start enforcing this type of stuff where we can, make it
best practice -- and since people DO copy our examples and use our docs..
documenting how to do this as a best practice could help future layers have a
better quality here.

> These are fixable, but particularly renaming the collection names to match the 
> layer names is going to cause some upheaval unless we have some kind of smooth 
> transition mechanism. My plan for tidying all of this up would be:

I see the collection name having nothing to do directly with the layer name.
Layer name is really a repository or directory name.. the collection name is a
"namespace" for that layer.  (I would PREFER they be the same, but I don't see
it as a requirement.)

> 1) Have the layer index collect BBFILE_COLLECTIONS from each layer as you 
> suggest.
> 
> 2) Add some validation code into the layer index. This would check that a 
> collection was defined by a layer (and only one collection), that the 
> collection name matches the layer name in the index, and that it's unique 
> (though it would have to be unique to match with the name since the layer 

I'd love to see the layerindex be able to handle the namespace issues.  Some
kind of way for users to detect collisions (before they create their layers)
would be REALLY useful in this scope.

> index doesn't allow duplicate names). We could also validate LAYERDEPENDS 
> against the dependencies defined in the index. This would give us an idea of 
> the state of the metadata out there, at least for public layers.
> 
> 3) Add a LAYERPROVIDES or similar so we have a way of avoiding a flag day on 
> renaming the collections to match the layer name.

I don't object to a LAYERPROVIDES as the method for LAYERDEPENDS (RECOMMENDS) to
use.

> 4) After we've cleaned up a good portion of the commonly used public layers, 
> have bitbake warn about duplicate collection names and missing LAYERDEPENDS in 
> the current configuration.
> 
> 5) Consider renaming BBFILE_COLLECTIONS, though this is only cosmetic and we'd 
> need to read both variables for a time to avoid compatibility issues.
> 
> 
> I'm not totally sold on LAYERRECOMMENDS, though I have often been frustrated 
> at being forced to add a dependent layer when in fact that dependency exists 
> only to satisfy a DEPENDS in one or two recipes that I don't even need to 
> build, so it's possible that could help in that situation - but it relies on 
> the layer maintainer applying it (and agreeing that it's the right thing to do 
> vs. LAYERDEPENDS). Conceivably you could also use LAYERRECOMMENDS to augment 
> missing DEPENDS errors e.g. "Nothing PROVIDES xyz - perhaps you need to add 
> layer meta-a, meta-b or meta-c to satisfy this?"

In my case, I have some code that checks the included layers(collections) and
will activate specific features if they are available.

> As for LAYERSUMMARY, LAYERDESCRIPTION and LAYERHOMEPAGE, we could certainly 
> add these although LAYERHOMEPAGE doesn't match up directly with anything we 
> have in the layer index (instead we have usage information and the web link to 
> the repo, and the latter is mostly set automatically based on the git URL if 
> it's a host we know about, e.g. github or git.yoctoproject.org). It's less 
> clear to me how useful these are though apart from to save a little time on 
> submitting to the layer index (or adding to a private layer index instance).

I personally want these things so that I can actively index my internal private
layers (things not in the public layer index) without having to constantly
curate the index, but make changes to the layer.conf as necessary.  Also if this
becomes a best practice it also helps my customers to do the same thing.

>From a commercial perspective, I'm currently investigating effectively giving
the commercial user the same kind of tools (layers.openembedded.org, toaster
connection, etc) as we all see in the community, but on a private implementation
basis.  (I've seen re-use problems in large commercial organizations, being
unable to figure out how to 'share' between projects within an organization...
being able to duplicate some of the success of Open Source seems like a first
attempt at fixing this.)

> One problem it will create is that right now entering the summary / 
> description are mandatory when submitting to the layer index, but presumably 
> if the information was in layer.conf people would want that to be optional in 
> the submission form - but then it's optional in the layer.conf as well and we 
> won't actually know whether it's set until we parse the layer, which doesn't 
> happen until after the layer is manually published (for security and keeping 
> some measure of order in the database.)
> 
> I have been thinking about some of these shortcomings for a while myself and I 
> agree we should do something here. It would be good to hear other opinions as 
> well.

Ya, I'm not sure the best way to handle this.... but I'd hope -if set- the
values in the layer.conf are "better" then any manually entered values...  (But
one possibility is a 'manual override' flag.)

I.e. require on entry the user to fill out the information.  We then parse the
layer.conf and use the automated information if present, fall back to manual
entry.  Then if the index curator flips an 'override with manual option' we can
ignore the layer.conf version.  (I know a bit complicated, but really this is
the type of automated curating I've seen elsewhere.. and as long as "someone"
cares about the quality of the data it does help a lot.)

> Cheers,
> Paul
>