[Openembedded-architecture] bblayers.conf - Past, Present and Future?

[Flanagan, Elizabeth]

On 11 May 2016 at 12:42, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> We currently have a few different ways people setup and build
> combinations of OE layers. Koen's presentation and ELC covers some of
> this, we've also heard representations at OEDEM/OEDAM and I think
> everyone would agree that its worth seeing if there is some way we
> could improve the situation.
>
> We have various tools/options:
>
> * TEMPLATECONF
> * combo-layer
> * repo
> * git submodules
>
> and we all have views, some of them quite strong about different
> elements of these.
>
> At the end of the day, our users want/need something which sets up the
> system for them to use and "works", they don't really care about the
> details.
>
> To summarise the general problem, we need some way of saying "get this
> set of git repositories and check them out in this layout, then setup a
> bblayers.conf file like this".
>
> In parallel, we're seeing people trying to use OE in CI systems. To
> make buildbot work for the Yocto Project, we have a ton of
> configuration and "fetching" code and I know Jenkins integration runs
> into the same issues. We'd love to be able to throw away the need to
> have CI specific layer addition code, or the bulk of the custom git
> fetcher we ended up having to add there for example.
>

Let me talk about the CI set of things to maybe explain what this looks like.

Most CI solutions have fairly simple source fetchers. When we looked
at buildbots source fetcher, in order to get the optimisations we
wanted, we had to hack it up a bit and essentially reimplement bitbake
fetch functionality.

If you are only building out a few different types a build, I don't
think this particular optimisation buys you a lot, but if you're
pushing out what the project does, it saves us close to 45+ minutes of
git fetch/clone time each build run.

> The key question is perhaps "how this should work?" and then we could
> work backwards from there.
>
> The starting point is probably to have bitbake (as in the repo) and a
> configuration file available. A command like "bitbake-setup
> <configfile>", perhaps with a way to specify a default configfile would
> perhaps be the starting point. As a minimum this would then need to
> fetch the appropriate git repos (if not already present) and generate a
> bblayers.conf file.
>
> Having a "bitbake-setup update" command to pull in changes from
> upstream would also seem appropriate. For git cloned layers, this is
> simple, it would also need to have a way to understand how to make
> changes to the original configuration file and pull in changes to
> bitbake itself. This would likely depend on the way the original
> starting point was created and is where some of the complexity would
> start to creep in but it should be manageable.

I would love to be able to checkout a repo with bitbake and maybe a
config with known, working sets of layers and their URLS, that are
tested on a regular basis and actively maintained, uncomment a bunch
of them and run bitbake-setup and have this all clone those layers and
produce a bblayers that I can then go about and modify.

My worry is the "tested on a regular basis and actively maintained"
part. Just looking at the layers in the layer-index and wondering how
many of them are actively working with master makes my stomach knot
up. Yeah, I would love this tool but only if the meta data behind it
was functional.

>
> One good thing is we already have fetcher functionality within bitbake
> so we can comparatively easily have a url syntax which would support a
> wide range of layer fetching/checkout needs.
>
> The main dilemma would be the format of the configfile. Do we use a
> bblayers.conf.sample style format, extended to allow for clone
> urls/checkout locations and so on? Do we create a new format .conf file
> which could generate a bblayers.conf in bitbake's "conf" format? or do
> we create something like a json file, similar to what toaster ended up
> doing to partially solve some of this problem? I'm actually torn, part
> of likes the idea of a json file, part of me says we should try and be
> consistent to our current file format.

Off the cuff, I'd be in favour of maybe using BBLAYERS_LAYERINDEX_URL
and pulling that info from there?

So something like:

bitbake-setup-layerindex <--type=[conf|json]> <-u http://foo/index>
<-o layerindex.conf>
(modify layerindex.conf)
bitbake-setup <-o layerindex.conf>

I don't think this needs to be an either/or, (example, I find great
utility in combo-layer). I think for new things I'm working on, where
I'm not entirely sure what bblayers are all going to look like, having
a tested group of layers right there that can generate my bblayers
would be useful.

-b

>
> I'm going to put this idea out here without going more into the
> details, I'd like to see what people think. Is this a direction we
> should look further into? Would a "bitbake-setup" solve some of our
> problems? Is there a better way to proceed?
>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture



-- 
Elizabeth Flanagan
Yocto Project
Build and Release