[Openembedded-architecture] bblayers.conf - Past, Present and Future?

[Jérémy Rosen]

As a "final user" of Yoco, I would be extremely interested in that sort 
of tool.

My day job is to help various non-computer companies that need to 
develop embedded systems for their product.

I have worked with dozen of yocto-supported board and as many different 
tools from various providers, and this area of the yocto system is 
horrendous. I am not calling them BSP because most board providers do 
not provide yocto BSP, they provide shell scripts that will patch a 
yocto installation. yeah, no bbappend, that's too complicated.

I have seen lots of tools that try to deal with the problem you 
describe, none of which did the job well enough to mention. The options 
you mentionned are the most sensible I have found so far. All sorts of 
comments in your post below

On 11/05/2016 13:42, Richard Purdie wrote:
> We currently have a few different ways people setup and build
> combinations of OE layers. Koen's presentation and ELC covers some of
> this, we've also heard representations at OEDEM/OEDAM and I think
> everyone would agree that its worth seeing if there is some way we
> could improve the situation.
> We have various tools/options:
>
> * TEMPLATECONF
big advantage : it allows to set local.conf

most board makers have a huge readme with a dozen of changes you have to 
do manually to your local.conf
TEMPLATECONF is way less known that it deserves to be.

one of the (minor) drawback limitation of TEMPLATECONF is that it needs 
a .templateconf file in the toplevel directory to
point to an alternative local.conf
The toplevel directory is usually coming from openembedde-core.git so 
you need to fork that repo if you want to automate the process completely
> * combo-layer
combining all git into one big git repo is not very clean, but it 
simplifies the process for the user immensly.
A tool that would do all the git fetch would reduce the appeal of this 
tool, but for the good reason that it provides the
same ease-of-use.

Combo-layer is great for what it produces : a single, self-contained git 
repo for the user, but it's a bit complicated to use for the integrator, 
and the average quality of the yocto providers is... pretty low
> * repo
one of the big advantage (and drawback) of repo is that it is not based 
on a xml file, it's based on a git repo that contains (possibly 
multiple) xml files.
This has the big advantage that you can save extra configuration 
fragments with the xml file and repo will put them at the right places 
when you do the checkout.
If you want to add a site.conf or other file, repo can do that. repo can 
also replace .templateconf for you, which is pretty cool.

I think the big thing to remember here is that a tool like bitbake-setup 
should not just have a .xml as a source, but also the possiblity to 
fetch some files to tune the generated repo (bitbake infrastructure can 
certainly help for that)
> * git submodules
i'm not very fond of that one. Again, I stumble on the fact that my 
toplevel yocto directory is already a git repo (poky.git or 
openembedded-core.git) so I need to fork to handle submodules, which 
complicates quality management quite a bit

> and we all have views, some of them quite strong about different
> elements of these.
>
> At the end of the day, our users want/need something which sets up the
> system for them to use and "works", they don't really care about the
> details.
>
> To summarise the general problem, we need some way of saying "get this
> set of git repositories and check them out in this layout, then setup a
> bblayers.conf file like this".
and a local.conf. You need to at least set MACHINE= in there, and the 
point is to really automate 100% of the setup process
>
> In parallel, we're seeing people trying to use OE in CI systems. To
> make buildbot work for the Yocto Project, we have a ton of
> configuration and "fetching" code and I know Jenkins integration runs
> into the same issues. We'd love to be able to throw away the need to
> have CI specific layer addition code, or the bulk of the custom git
> fetcher we ended up having to add there for example.
>
> The key question is perhaps "how this should work?" and then we could
> work backwards from there.
>
> The starting point is probably to have bitbake (as in the repo) and a
> configuration file available. A command like "bitbake-setup
> <configfile>", perhaps with a way to specify a default configfile would
> perhaps be the starting point. As a minimum this would then need to
> fetch the appropriate git repos (if not already present) and generate a
> bblayers.conf file.
see my comment about why repo forcing the config-file to be a git repo 
is usefull. That might not be the way to go with your idea, but it's 
important to take the use-case into account. a way to fetch files (not 
just meta- ) would be usefull
>
> Having a "bitbake-setup update" command to pull in changes from
> upstream would also seem appropriate. For git cloned layers, this is
> simple, it would also need to have a way to understand how to make
> changes to the original configuration file and pull in changes to
> bitbake itself. This would likely depend on the way the original
> starting point was created and is where some of the complexity would
> start to creep in but it should be manageable.
there are two update path that I can see, both are usefull for different 
use-cases

* update all the meta- to their latest version and test them (later, 
generate a new conffile and push it)
* re-fetch the conffile and update all meta- to that new version, 
because upsttream has provided a new version and we are a user, not a 
yocto dev

since a git url can point to a commit or a branch, it might be possible 
to include the first use-case into the second : if the conffile points 
to a branch, update that branch. if it points to a commit or tag, update 
to that commit or tag
>
> One good thing is we already have fetcher functionality within bitbake
> so we can comparatively easily have a url syntax which would support a
> wide range of layer fetching/checkout needs.
>
> The main dilemma would be the format of the configfile. Do we use a
> bblayers.conf.sample style format, extended to allow for clone
> urls/checkout locations and so on? Do we create a new format .conf file
> which could generate a bblayers.conf in bitbake's "conf" format? or do
> we create something like a json file, similar to what toaster ended up
> doing to partially solve some of this problem? I'm actually torn, part
> of likes the idea of a json file, part of me says we should try and be
> consistent to our current file format.
no strong opinion, I like the yocto format with all its _append and 
OVERRIDES etc... I don't know if it would fit that tool's need well, but 
it avoids inventing a second syntax..