[Openembedded-architecture] bblayers.conf - Past, Present and Future?

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-architecture-bounces at lists.openembedded.org
> [mailto:openembedded-architecture-bounces at lists.openembedded.org] On
> Behalf Of Richard Purdie
> Sent: den 11 maj 2016 18:02
> To: Otavio Salvador
> Cc: Christopher Larson; openembedded-architecture
> Subject: Re: [Openembedded-architecture] bblayers.conf - Past, Present
> and Future?
> 
> On Wed, 2016-05-11 at 12:15 -0300, Otavio Salvador wrote:
> > On Wed, May 11, 2016 at 11:44 AM, Richard Purdie
> > <richard.purdie at linuxfoundation.org> wrote:
> > > On Wed, 2016-05-11 at 11:08 -0300, Otavio Salvador wrote:
> > > > On Wed, May 11, 2016 at 8:42 AM, Richard Purdie
> > > > <richard.purdie at linuxfoundation.org> wrote:
> > ...
> > > > I think a mix of our tools, with the Mentor Graphics ones, would
> > > > provide a good basis for it. This is something Christopher and I
> > > > been
> > > > discussing for some time but did not yet started to work on.
> > >
> > > Its easy to say "hey, everyone should copy what I do". Creating
> > > something common/extensible which people can base off and around
> > > would
> > > be very in keeping with what the project tries to do so I think its
> > > worth asking questions to see if its possible.
> > >
> > > If everyone just wants to reply "mine is best" and that we don't
> > > want
> > > to bother even trying, so be it though. I've way too much to do
> > > already
> > > so perhaps this is best ignored.
> >
> > I know English is not my native language but I think I wrote:
> >
> >  "I think a mix of our tools, with the Mentor Graphics ones, would
> >   provide a good basis for it. This is something Christopher and I
> > been
> >   discussing for some time but did not yet started to work on."
> >
> > What I meant is both companies have public repositories which address
> > this problem and we can learn from them. Instead of throw a stone on
> > me, did you read the doc and checked what it can do?
> 
> Otavio and I did talk a little off list. I did respond fairly strongly
> to his email, perhaps too much so. The bit which triggered that was "I
> think adding a setup logic on BitBake violates the KISS principle. It
> does not belongs there and ought to be done externally.". That
> basically kills my proposal dead but I should explain why as perhaps
> that isn't clear.
> 
> If we go down the external tool route, firstly that doesn't change
> things much from the way things are today. There is nothing stopping
> people creating external tools now, or in the future. These obviously
> have features and drawbacks but a clear winning standard is yet to
> emerge. 5 years on since the Yocto Project came into being and we
> started pushing the layer model, I personally think one would have by
> now if it was going to.
> 
> The alternative to that model is to pick and promote a tool. One way to
> do that is to put it into bitbake. This helps encourage
> standardisation, since people then are encouraged to change and focus
> on one rather than just doing their own thing.
> 
> Putting something into bitbake (as a separate tool) has some other
> advantages, in that if bitbake is present, we get its parser and its
> fetcher code "for free". Being a separate tool means we don't need to
> complicate bitbake itself more, means the tool can easily be replaced
> by those who don't like it and it can have its own commandline struture
> (bitbake itself is very difficult to extend at the commandline level).
> 
> As with anything we do in OE, we're not going to force everyone down a
> particular route, differentiation is good and great things can come
> from it. We'd have to make any alternative in the core good enough that
> people can't afford not to switch.
> 
> Cheers,
> 
> Richard

I think an official OE tool for this can be very much beneficial to 
the community. It will probably be a lot of work to unify all the 
use cases from various users, but hopefully it will be worth it in 
the end.

The rest of this mail was written based on a an earlier mail in the 
thread, but it is too much text to rewrite it again... (The thread 
is moving too fast.) ;)

We are currently using repo (as many others). We have one repository 
with repo manifests for each project, function team, etc. They all 
include a base manifest that sets up the base layers and then they 
add any extra layers they need for the specific project. The base 
manifest also uses repo's support for conditional projects, which 
we use to easily add some extra layers on demand (via the --groups 
option to repo).

We have then added a small (~250 lines) shell script wrapper around 
the oe-init-build-env (and oe-init-build-env-memres) scripts which 
creates a templateconf directory where it does the following:
* It copies the bblayers.conf.sample file from meta-poky/meta-yocto 
  and inserts all layers it can find (i.e., what repo retrieved). 
  (For Poky and OpenEmbedded we use repo's ability to create links
  into the repositories for the layers we actually use).
* It gathers local.conf.sample.XX fragments from all layers (where XX 
  is a double digit number used to sort them) and combines them into 
  a local.conf.sample file. It treats the local.conf.sample file from 
  meta-poky/meta-yocto as local.conf.sample.50 which makes it easy to 
  add other fragments before and after it.
* It does the same for conf-notes.txt.XX fragments (maybe not as 
  useful, but it reuses the same code as for local.conf.sample so it 
  was more or less free).

Through this it is easy for one of our development projects to adapt 
the build environment by adding one or more layers to their 
configuration. And in their layers they can then, e.g., add a small 
local.conf.sample fragment that sets MACHINE to a good default for 
the project.

I can see a number of benefits in having a common tool for OE that 
basically does what repo does in addition to what our shell wrapper 
mentioned above does as that tool would be able to take advantage of 
its knowledge of the system and how it is intended to work (which 
should make it possible to avoid some of the quirks we currently 
have with repo).

It may be good to separate the tasks of retrieving and keeping the 
repositories up-to-date from the tasks of managing the templateconf 
directory (or similar). I.e., regardless of how you create your work 
directory (e.g., by checking out Poky, using repo to fetch 
repositories based on a manifest, some new tool), it should then be 
possible get it to build based on what you have.

So, if you want to create a tool to basically do what repo does plus 
what our shell script mentioned above does, here are a couple of 
requirements based on what we use:

* It must be possible to specify any number of repositories to check 
  out, and specify any kind of revision to use (e.g., sha-1, branch, 
  tag, Gerrit reference, whatever).
* It must be possible to include configuration files from each other 
  in a way that you can add, remove and modify repositories specified 
  in earlier files.
* It must be possible to specify conditional repositories, e.g., via 
  an option to the tool that retrieves them.
* For repositories with multiple layers (e.g., Poky and OpenEmbedded) 
  it should be possible to select only the layers one need.
* It should be possible to show the status of all repositories at a 
  glance, to see what has changed and to find anything in any of them 
  (i.e., what corresponds to repo status, repo diff and repo grep).
* It must be possible to lock down an existing configuration file to 
  specify only SHA-1s (i.e., all branches and tags are replaced with 
  their corresponding SHA-1). This is how we take a general manifest 
  that typically specifies "master" as the reference for most
  repositories and turn it into something we can call a release, and 
  recreate at a later date if needed. For convenience this should 
  also combine all included files into one.

Oh, and all references to ${COREBASE}/LICENSE (i.e., the LICENSE file 
in the top of Poky) should be removed from meta, or the file be moved 
into meta. This is because when you combine your own set of layers 
(using repo, combo-layer or whatever), the LICENSE file in Poky 
typically does not cover this combination. However, using 
${COREBASE}/LICENSE in LIC_FILES_CHSUM in multiple recipes in meta 
makes it hard to replace it with an appropriate LICENSE file...

//Peter