[oe] Product platforms based on OE
Peter Kjellerstedt
peter.kjellerstedt at axis.com
Thu Feb 22 17:36:05 UTC 2018
Since the subject of how to manage product platforms based on OE
interests me very much, I took the liberty of giving this discussion
its own subject since it really hasn't got much to do with the
original discussion about splitting meta-oe.
> -----Original Message-----
> From: openembedded-devel-bounces at lists.openembedded.org
> [mailto:openembedded-devel-bounces at lists.openembedded.org] On Behalf Of
> Otavio Salvador
> Sent: den 22 februari 2018 10:41
> To: Patrick Ohly <patrick.ohly at intel.com>
> Cc: OpenEmbedded Devel List <openembedded-devel at lists.openembedded.org>
> Subject: Re: [oe] Splitting meta-oe?
>
> On Thu, Feb 22, 2018 at 6:27 AM, Patrick Ohly <patrick.ohly at intel.com> wrote:
> > On Thu, 2018-02-22 at 07:53 +0100, Jonas Bonn wrote:
> >> On 21 February 2018 at 15:09, Martin Hundebøll <mnhu at prevas.dk> wrote:
> >>
> >> > Now that the discussion branched out a bit...
> >> >
> >> > We would like better support for this too. Our setup uses a "manifest"
> >> > repository with git submodules to setup the layers:
> >> >
> >> > > yocto/
> >> > > meta-poky/
> >> > > meta-qt5/
> >> > > meta-foo/
> >> > > meta-bar/
> >> > > conf/
> >> > > bblayers.conf
> >> > > local.conf
> >> > > .gitmodules
> >> >
> >> > With this setup, customers simply need to clone our yocto repo
> >> > recursively, run `yocto/meta-poky/oe-init-build-env yocto` and then
> >> > `bitbake image-recipe`.
> >> >
> >> > But this is rather inflexible, as it requires the "yocto" folder
> >> > to be the build folder to activate the config files...
> >> >
> >> > We looked into putting the configs in "meta-foo/conf/*.conf.sample" and
> >> > using TEMPLATECONF, but the "oe-init-build-env" script is rather picky
> >> > about poky being the "top" directory.
> >> >
> >> > I guess the oe-init-build-env script can be changed to look for
> >> > .templateconf in any parent folder?
> >>
> >>
> >> Putting together a deliverable setup that's easy for the customer to get
> >> started with is a bit tricky. Here's the approach that's worked well for
> >> me:
> >>
> >> /myproject
> >> /env <-- script
> >> /build
> >> /meta-myproject
> >> /bitbake
> >> /oe-core
> >> /meta-layer1
> >> /meta-layer2
> >>
> >> env, build, meta-myproject are part of the myproject repo, everything
> >> else is a submodule.
> >
> > refkit used the same approach. One thing that I would prefer to do
> > differently is the location of the submodule: having them in their own
> > directory would make it more transparent which code is "external" and
> > which is "internal".
> >
> >> "env" is a script containing just the following:
> >> . ./oe-core/oe-init-build-env build/ bitbake/
> >
> > We ended up with a top-level "oe-init-build-env" wrapper script around
> > the actual oe-core/oe-init-build-env. That way the repo could be used
> > the same way as poky. The script sets TEMPLATECONF, so the usual local
> > build setup happens based on refkit sample files.
>
> We have a script set and a document which describes how we do it:
>
> http://doc.ossystems.com.br/managing-platforms.html
>
> The script sources can be seen at:
>
> https://code.ossystems.com.br/gitweb?p=ossystems-yocto-base-scripts.git
>
> --
> Otavio Salvador O.S. Systems
> http://www.ossystems.com.br http://code.ossystems.com.br
> Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
As I see it there are two major use cases for how you work with layers.
One is the ad hoc method, where you start with some base set of layers,
e.g., Poky and then add layers as you need using, e.g., bitbake-layers.
When you want to build something else you start over. This method is
what BitBake and OE Core currently provides.
The second method is the case where you have a well-defined set of
layers for your project. Typically you want to build a specific product,
and what layers are needed to do that is specified by someone else.
Anyone who builds the same product uses the same set of layers. The
set of layers are typically defined in some kind of manifest (e.g., as
a Git repository with submodules as Martin mentions above or as a
repo manifest as Otavio uses).
Since I work for a company where we have some 80+ products that we can
build based on OE, I will concern myself with the second method.
We too base our platform on repo manifests. We store all our manifests
in a single Git repository. Whenever a new project starts, they add a
new manifest that specifies the layers they need. Typically this is done
by referencing the manifest for our base platform and then add the
project specific layer(s). This makes it very easy to add a new project,
as the manifest file typically just looks something like:
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<include name="cvp/cvp.xml"/>
<project path="meta-foobar" name="layers/projects/meta-foobar" revision="master"/>
</manifest>
cvp.xml is the platform manifest for our Core Video Platform and is
used by all our video products. It is actually a symbolic link to
the CVP manifest for the current version of OE that we currently use,
so right now it is cvp-pyro.xml. This means that when we want to
upgrade to a new version of OE, all we do is redirect that link to,
e.g., cvp-rocko.xml instead, and all products will from then on be
based on Rocko instead.
Once repo has checked out all layers specified in the manifest, we
end up with something like:
.
|-- axis-oe-init
| |-- add_paths
| |-- create_bblayers_conf_sample
| |-- create_sample_file
| |-- find_all_layers
| `-- oe-init-build-env
|-- bitbake -> poky/bitbake
|-- meta -> poky/meta
|-- meta-axis
| `-- conf
| |-- conf-notes.txt.40
| |-- local.conf.sample.40
| `-- local.conf.sample.60
|-- meta-axis-bsp
| `-- conf
| |-- local.conf.sample.10
| `-- local.conf.sample.55
|-- meta-comptools
| `-- conf
| `-- local.conf.sample.65
|-- meta-gplv2
|-- meta-oe -> openembedded/meta-oe
|-- meta-poky -> poky/meta-poky
|-- meta-python -> openembedded/meta-python
|-- meta-webserver -> openembedded/meta-webserver
|-- oe-init-build-env -> axis-oe-init/oe-init-build-env
|-- oe-init-build-env-memres -> axis-oe-init/oe-init-build-env
|-- openembedded
|-- poky
|-- scripts -> poky/scripts
|-- templateconf
| |-- bblayers.conf.sample
| |-- conf-notes.txt
| `-- local.conf.sample
`-- zzz-repo-workaround
|-- openembedded
`-- poky
There are a couple of interesting observations that can be made
from looking at the above.
First, we too use a wrapper script to set up the environment. It is
available to the user as a symbolic link from oe-init-build-env, so
for anyone used to OE Core, it will look and work exactly as usual.
What this script does is basically to gather the set of layers from
the top directory that match "meta*" and create a bblayers.conf.sample
file in a templateconf directory. Our own layers are typically Git
repositories, and the layers from Poky and OpenEmbedded are symbolic
links into those repositories.
The other thing the wrapper script does is to create local.conf.sample
and conf-notes.txt. This is done by looking in all the layers for
files matching conf/local.conf.sample.XX and conf/conf-notest.txt.XX
where XX is a number from 00 to 99. We treat the files from meta as
if they had XX == 50. These files are then concatenated together in
numeric order to form the final files. We came up with this solution
to handle that layers may be combined in different combinations and
it is not possible to have static bblayers.conf.sample and
local.conf.sample files in the layers as it is impossible to know
what other layers may be in use. I think this is similar to the hooks
that Otavio's solution use.
The last thing the wrapper script does is to add paths for scripts
directories found in the various layers, in addition to the scripts
directory provided by OE Core.
The above should be typical examples of what support is needed from
Bitbake to be able to extend the pretty static setup that OE Core
currently provides to something that is flexible and adapts to the
layers that are used, both when it comes to what layers go into
bblayers.conf and the contents of local.conf.
Finally, if anyone wonders about the zzz-repo-workaround directory
above, the rest of this mail is a description of a repo "hack" I came
up with to make "repo init" create links without having to bind them to
a specific repository (normally links created by repo are listed in the
manifest together with the respective repository, which makes it
impossible to make them conditional). However, if one writes them like
this (this example matches the links we set up by default for Poky):
<project path="zzz-repo-workaround/poky" name="layers/repo-workaround" revision="master">
<linkfile src="../../poky/bitbake" dest="bitbake"/>
<linkfile src="../../poky/meta" dest="meta"/>
<linkfile src="../../poky/meta-poky" dest="meta-poky"/>
<linkfile src="../../poky/scripts" dest="scripts"/>
</project>
we can then easily change the repository definition without having to also
change the links (something that is needed if, e.g., a project manifest
wants to override the revision of a layer that is specified in the platform
manifests).
Conditional links can then be specified like:
<project groups="notdefault,meta-selftest" path="zzz-repo-workaround/meta-selftest" name="layers/repo-workaround" revision="master">
<linkfile src="../../poky/meta-selftest" dest="meta-selftest"/>
</project>
<project groups="notdefault,meta-skeleton" path="zzz-repo-workaround/meta-skeleton" name="layers/repo-workaround" revision="master">
<linkfile src="../../poky/meta-skeleton" dest="meta-skeleton"/>
</project>
<project groups="notdefault,meta-yocto-bsp" path="zzz-repo-workaround/meta-yocto-bsp" name="layers/repo-workaround" revision="master">
<linkfile src="../../poky/meta-yocto-bsp" dest="meta-yocto-bsp"/>
</project>
They can then be enabled via the --groups option to repo init, or if
a project needs one of the conditional layers (e.g., meta-filesystems),
it can add something like this in its manifest:
<extend-project groups="default" path="zzz-repo-workaround/meta-filesystems" name="layers/repo-workaround"/>
It is not very pretty, but it works. :)
//Peter
More information about the Openembedded-devel
mailing list