[Openembedded-architecture] bblayers.conf - Past, Present and Future?

[Nathan Rossi]

On Wed, May 11, 2016 at 9:42 PM, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> We currently have a few different ways people setup and build
> combinations of OE layers. Koen's presentation and ELC covers some of
> this, we've also heard representations at OEDEM/OEDAM and I think
> everyone would agree that its worth seeing if there is some way we
> could improve the situation.
>
> We have various tools/options:
>
> * TEMPLATECONF
> * combo-layer
> * repo
> * git submodules
>
> and we all have views, some of them quite strong about different
> elements of these.
>
> At the end of the day, our users want/need something which sets up the
> system for them to use and "works", they don't really care about the
> details.
>
> To summarise the general problem, we need some way of saying "get this
> set of git repositories and check them out in this layout, then setup a
> bblayers.conf file like this".
>
> In parallel, we're seeing people trying to use OE in CI systems. To
> make buildbot work for the Yocto Project, we have a ton of
> configuration and "fetching" code and I know Jenkins integration runs
> into the same issues. We'd love to be able to throw away the need to
> have CI specific layer addition code, or the bulk of the custom git
> fetcher we ended up having to add there for example.
>
> The key question is perhaps "how this should work?" and then we could
> work backwards from there.
>
> The starting point is probably to have bitbake (as in the repo) and a
> configuration file available. A command like "bitbake-setup
> <configfile>", perhaps with a way to specify a default configfile would
> perhaps be the starting point. As a minimum this would then need to
> fetch the appropriate git repos (if not already present) and generate a
> bblayers.conf file.
>
> Having a "bitbake-setup update" command to pull in changes from
> upstream would also seem appropriate. For git cloned layers, this is
> simple, it would also need to have a way to understand how to make
> changes to the original configuration file and pull in changes to
> bitbake itself. This would likely depend on the way the original
> starting point was created and is where some of the complexity would
> start to creep in but it should be manageable.
>
> One good thing is we already have fetcher functionality within bitbake
> so we can comparatively easily have a url syntax which would support a
> wide range of layer fetching/checkout needs.
>
> The main dilemma would be the format of the configfile. Do we use a
> bblayers.conf.sample style format, extended to allow for clone
> urls/checkout locations and so on? Do we create a new format .conf file
> which could generate a bblayers.conf in bitbake's "conf" format? or do
> we create something like a json file, similar to what toaster ended up
> doing to partially solve some of this problem? I'm actually torn, part
> of likes the idea of a json file, part of me says we should try and be
> consistent to our current file format.

Just throwing the follow suggestion out there to add to the
discussion. It could be worth having the layer configuration come from
the bitbake-setup args. A while back now, I wrote a tool (I am
referencing it here only as a source of ideas, not recommending it for
use) which does similar to what bitbake-setup would be doing, grabbing
layers and setup bblayers/etc all from short command lines (or long
for more specifics). It works quite well for getting environments
setup with one line execution and avoids the need to copy files
around. Xilinx published the code (python) under MIT here,
https://github.com/Xilinx/hopper.

Some examples of more complex things (like repo branches/commit ids)
were handled with somewhat obscure but relatively straight forward
strings. e.g.

hopper -l:oe-core/meta
-l:meta-openembedded/meta-oe=master:2f3a4997c70cf7c1e28ccf33a31ce1437fe1db27 at https://github.com/openembedded/meta-openembedded.git

The tool does a lot more than layer fetch/setup, if anyone is curious
the "hopper help" command provides a more complete documentation than
the README.

Regards,
Nathan

>
> I'm going to put this idea out here without going more into the
> details, I'd like to see what people think. Is this a direction we
> should look further into? Would a "bitbake-setup" solve some of our
> problems? Is there a better way to proceed?
>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture