[Openembedded-architecture] Heterogeneous System Proposal

[Richard Purdie]

On Mon, 2019-12-02 at 17:19 -0600, Mark Hatle wrote:
> 
> On 12/2/19 3:58 PM, Richard Purdie wrote:
> > On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
> > > In all of the above examples, the user has manually configured
> > > the
> > > multiconfig within their project.  There is a simple way to move
> > > that
> > > configuration to a layer, simply place it in a conf/multiconfig
> > > directory within that layer.
> > > 
> > > This ability suggests to be that there should be a standard way
> > > to
> > > specify a layer above that of machine, which defines the overall
> > > characteristics of the system.
> > > 
> > > I'm proposing calling this new layer type as a system layer and
> > > it's
> > > configuration variable "SYSTEM".  It will be used instead of
> > > MACHINE
> > > when there is no single machine to quantify the contents of the
> > > produced system image.  When implementing a system, we do not
> > > want to
> > > make major changes to any other components.  Due to the existing
> > > implementation requiring MACHINE and certain TUNE parameters,
> > > this
> > > will require us to provide a special MACHINE value that can be
> > > used
> > > for a heterogeneous system.  I suggest we create a new
> > > 'nomachine'
> > > system that only defines/uses an equivalent noarch style
> > > tune.  This
> > > will instruct the system that this configuration can be used to
> > > noarch software and create images (including with wic), but it is
> > > not
> > > able to compile specific applications.  Each of these
> > > applications or
> > > images must come from a defined MACHINE.
> > 
> > I'm not sure I 'buy' this explanation and I think a "nomachine"
> > target
> > will be horrible in practice. In your example below, why can't
> > mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other
> > bits of
> > linux.conf and BBMULTICONFIG = "bootloader fpga" ? 
> 
> This assumes the linux side of things will load the others.  In the
> systems I am thinking of Linux may not be a primary system.
> 
> I.e.  a baremetal system based on a microblaze may be the core
> system, which then bootstraps and launches the linux system.  The
> core 'machine' should it be defined as baremetal and
> zcu102_microblaze then?

Sure, why not? I picked on 'linux' but I could have equally chosen any
one of the targets, there wasn't any specific reason.

> Problem is baremetal isn't really an operating system, so how do we
> handle that?

I'm not sure it matters, MACHINE doesn't define an operating system.

> Introduce nomachine and notune, and adjust system components to
> > > know
> > > how to deal with these properly.  I.e. recipes that are NOT
> > > noarch
> > > would be be rejected in this configuration.
> > 
> > I definitely do not like this at all. I'm not sure why its needed.
> 
> I want a way to say that this configuration is a sum of it's parts,
> but by itself can't do anything.  It's not arm, it's not ppc, it's
> not x86, etc..

The more I read and think about this, the more I'm concluding it still
*is* a MACHINE. A MACHINE represents a target piece of hardware, not an
OS, or an architecture. Those are configurations of that hardware
combined with software.

> This is common on FPGA systems that may have to load from special
> configurations, the hardware itself may be loading a bitstream (not a
> cpu instruction set) that in turn may be a first stage loader for one
> or multiple on system CPUs, which may in turn instruct other aspects
> of the system to load.
> 
> I don't want the proposal to be FPGA specific, as I expect it will be
> used in other heterogeneous cases.  Primarily ones that have a single
> host CPU (that boot) and multiple plugin cards.  In those cases, it
> -CAN- have both a SYSTEM and a MACHINE defined, or just simply a
> MACHINE with multiconfigs that the MACHINE has a way of
> understanding.

I think you need a convention for MACHINE that works for these systems.
I'm not sure we need a "new" level specially for this.

> > I'd also put the idea out there that in many ways what you're
> > describing is actually a configuration or system configuration. Its
> > configurations like this we actually test on the project
> > autobuilder
> > and its what the code in yocto-autobuilder-helper generates from
> > json
> > files.
> 
> I would expect in my particular use-case that I do need to generate
> configurations based on certain parameters (specifically a system
> device tree).  However, other configurations are going to be very
> static and then predefined 'systems' can be provided by layer that
> give the user the ability to modify (as they would any other part of
> the system today.)
> 
> > There is a little known/used piece of bitbake which is BBTARGETS.
> > If
> > your myssystem.conf sets:
> > 
> > BBTARGETS = "system-images:do_image"
> 
> I wasn't aware this was a thing, but it could very well be used by a
> system configuration to define the right combination of things to
> build.  I didn't mention it here, but we need a better way to build
> an image recipe that collects together pieces from other multiconfigs
> then manually specifying everything on an do_image[mcdepends] ...  I
> figured that would come over time through classes and other future
> proposals.

Well, this is why I'm trying to suggest higher level configuration. If
you implement "SYSTEM", you're stuck with doing this in recipes.

> > (or something like that), it would even then just work from
> > "bitbake"
> > being run without anything else as a parameter.
> > 
> > I do wonder if it would be better to have some higher level
> > "configuration" abstraction in bitbake (layer and configuration
> > setup?)
> > rather than adding in yet more levels to the conf file hierarchy.
> > 'system' configuration would then be one subset of that...?
> 
> In the end my thought was what works now works well.  I don't want to
> change it.

Adding SYSTEM to the hierarchy is a *huge* change to people's mental
models of the project. We already struggle with MACHINE and DISTRO.

>  I want to exploit it, and make it easier to combine those pieces
> together to build new systems that have previously been constructed
> via external (to the yocto project/OE-core) scripting.
> 
> There is another thing I didn't mention which will be an issue.  In a
> large multiconfig build, each config adds more and more parsing
> overhead to the system.  For a heterogeneous system in which some of
> the configurations are baremetal, non-Linux (unix-like) OSes, we
> should be able to disable a large part of the recipe pool easily --
> this should improve parsing time, I think.
>
> (Again, future work -- once the framework is in place to exploit
> this.)

You may want to consider this now. Adding system with the "nomachine"
mode you mention is a nightmare scenario for parsing. Bitbake has a
problem with knowing when to parse or not to parse since to 'skip'
parsing a recipe, it needs to parse enough to know it should skip it.
COMPATIBLE_MACHINE is implemented with the skip mechanism for example.
If parsing is a concern (and I agree it is), you could probably get
much further with a BBMASK approach to just the system-images recipe as
then bitbake would know what it needs to parse without needing to read
the recipes.

I suspect BBMASK may be parsed at the wrong points to quite get this
right today but it can likely be fixed to work well with multiconfig.

Cheers,

Richard