[Openembedded-architecture] Heterogeneous System Proposal

[Mark Hatle]

On 12/2/19 5:53 PM, Richard Purdie wrote:
> On Mon, 2019-12-02 at 17:19 -0600, Mark Hatle wrote:
>>
>> On 12/2/19 3:58 PM, Richard Purdie wrote:
>>> On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
>>>> In all of the above examples, the user has manually configured
>>>> the
>>>> multiconfig within their project.  There is a simple way to move
>>>> that
>>>> configuration to a layer, simply place it in a conf/multiconfig
>>>> directory within that layer.
>>>>
>>>> This ability suggests to be that there should be a standard way
>>>> to
>>>> specify a layer above that of machine, which defines the overall
>>>> characteristics of the system.
>>>>
>>>> I'm proposing calling this new layer type as a system layer and
>>>> it's
>>>> configuration variable "SYSTEM".  It will be used instead of
>>>> MACHINE
>>>> when there is no single machine to quantify the contents of the
>>>> produced system image.  When implementing a system, we do not
>>>> want to
>>>> make major changes to any other components.  Due to the existing
>>>> implementation requiring MACHINE and certain TUNE parameters,
>>>> this
>>>> will require us to provide a special MACHINE value that can be
>>>> used
>>>> for a heterogeneous system.  I suggest we create a new
>>>> 'nomachine'
>>>> system that only defines/uses an equivalent noarch style
>>>> tune.  This
>>>> will instruct the system that this configuration can be used to
>>>> noarch software and create images (including with wic), but it is
>>>> not
>>>> able to compile specific applications.  Each of these
>>>> applications or
>>>> images must come from a defined MACHINE.
>>>
>>> I'm not sure I 'buy' this explanation and I think a "nomachine"
>>> target
>>> will be horrible in practice. In your example below, why can't
>>> mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other
>>> bits of
>>> linux.conf and BBMULTICONFIG = "bootloader fpga" ? 
>>
>> This assumes the linux side of things will load the others.  In the
>> systems I am thinking of Linux may not be a primary system.
>>
>> I.e.  a baremetal system based on a microblaze may be the core
>> system, which then bootstraps and launches the linux system.  The
>> core 'machine' should it be defined as baremetal and
>> zcu102_microblaze then?
> 
> Sure, why not? I picked on 'linux' but I could have equally chosen any
> one of the targets, there wasn't any specific reason.
> 
>> Problem is baremetal isn't really an operating system, so how do we
>> handle that?
> 
> I'm not sure it matters, MACHINE doesn't define an operating system.

Correct distro does, I was confusing my statements.  Both MACHINE and DISTRO
have to work together to define the system.  My assumption was that a system
would define reasonable defaults, like a multiconfig does.  (Maybe this is an
error on my part though.)

>> Introduce nomachine and notune, and adjust system components to
>>>> know
>>>> how to deal with these properly.  I.e. recipes that are NOT
>>>> noarch
>>>> would be be rejected in this configuration.
>>>
>>> I definitely do not like this at all. I'm not sure why its needed.
>>
>> I want a way to say that this configuration is a sum of it's parts,
>> but by itself can't do anything.  It's not arm, it's not ppc, it's
>> not x86, etc..
> 
> The more I read and think about this, the more I'm concluding it still
> *is* a MACHINE. A MACHINE represents a target piece of hardware, not an
> OS, or an architecture. Those are configurations of that hardware
> combined with software.
> 
>> This is common on FPGA systems that may have to load from special
>> configurations, the hardware itself may be loading a bitstream (not a
>> cpu instruction set) that in turn may be a first stage loader for one
>> or multiple on system CPUs, which may in turn instruct other aspects
>> of the system to load.
>>
>> I don't want the proposal to be FPGA specific, as I expect it will be
>> used in other heterogeneous cases.  Primarily ones that have a single
>> host CPU (that boot) and multiple plugin cards.  In those cases, it
>> -CAN- have both a SYSTEM and a MACHINE defined, or just simply a
>> MACHINE with multiconfigs that the MACHINE has a way of
>> understanding.
> 
> I think you need a convention for MACHINE that works for these systems.
> I'm not sure we need a "new" level specially for this.

Agreed.  The only reason to add a SYSTEM would be to distinguish it in the
local.conf, but from a practical reason it and MACHINE are equivalent in
hierarchy in my mind.  SYSTEM is 1..n multiconfigs, while MACHINE requires a
DISTRO to provide a working system.

>>> I'd also put the idea out there that in many ways what you're
>>> describing is actually a configuration or system configuration. Its
>>> configurations like this we actually test on the project
>>> autobuilder
>>> and its what the code in yocto-autobuilder-helper generates from
>>> json
>>> files.
>>
>> I would expect in my particular use-case that I do need to generate
>> configurations based on certain parameters (specifically a system
>> device tree).  However, other configurations are going to be very
>> static and then predefined 'systems' can be provided by layer that
>> give the user the ability to modify (as they would any other part of
>> the system today.)
>>
>>> There is a little known/used piece of bitbake which is BBTARGETS.
>>> If
>>> your myssystem.conf sets:
>>>
>>> BBTARGETS = "system-images:do_image"
>>
>> I wasn't aware this was a thing, but it could very well be used by a
>> system configuration to define the right combination of things to
>> build.  I didn't mention it here, but we need a better way to build
>> an image recipe that collects together pieces from other multiconfigs
>> then manually specifying everything on an do_image[mcdepends] ...  I
>> figured that would come over time through classes and other future
>> proposals.
> 
> Well, this is why I'm trying to suggest higher level configuration. If
> you implement "SYSTEM", you're stuck with doing this in recipes.

I'm not sure how that changes..  Alejandro has more experience working with this
then I do.

What I was expecting was one or more image recipes (and wks files) that together
would assemble the heterogeneous pieces into a configuration that would be bootable.

For instance, some systems the disk may need to be partition with a certain
number of partitions for the (hardware) loader to go to the partition and load
the data into the right CPU.

>>> (or something like that), it would even then just work from
>>> "bitbake"
>>> being run without anything else as a parameter.
>>>
>>> I do wonder if it would be better to have some higher level
>>> "configuration" abstraction in bitbake (layer and configuration
>>> setup?)
>>> rather than adding in yet more levels to the conf file hierarchy.
>>> 'system' configuration would then be one subset of that...?
>>
>> In the end my thought was what works now works well.  I don't want to
>> change it.
> 
> Adding SYSTEM to the hierarchy is a *huge* change to people's mental
> models of the project. We already struggle with MACHINE and DISTRO.

Adding the mental different was intentional in the proposal.  I wanted a way to
be clear that I was intending this to be at a higher level then a traditional
homogeneously configured 'machine'.  With that said, MACHINE = "system-foo" is
fine as well.  I just want a way to make it clear this isn't a regular
homogeneous 'thing'.

But yes, MACHINE & DISTRO need to be considered together in all of these things.

>>  I want to exploit it, and make it easier to combine those pieces
>> together to build new systems that have previously been constructed
>> via external (to the yocto project/OE-core) scripting.
>>
>> There is another thing I didn't mention which will be an issue.  In a
>> large multiconfig build, each config adds more and more parsing
>> overhead to the system.  For a heterogeneous system in which some of
>> the configurations are baremetal, non-Linux (unix-like) OSes, we
>> should be able to disable a large part of the recipe pool easily --
>> this should improve parsing time, I think.
>>
>> (Again, future work -- once the framework is in place to exploit
>> this.)
> 
> You may want to consider this now. Adding system with the "nomachine"
> mode you mention is a nightmare scenario for parsing. Bitbake has a
> problem with knowing when to parse or not to parse since to 'skip'
> parsing a recipe, it needs to parse enough to know it should skip it.
> COMPATIBLE_MACHINE is implemented with the skip mechanism for example.
> If parsing is a concern (and I agree it is), you could probably get
> much further with a BBMASK approach to just the system-images recipe as
> then bitbake would know what it needs to parse without needing to read
> the recipes.

The problem I have with BBMASK is that when a user tries to do something with
it, they don't get a reasonable error message.

For instance if I BBMASK out bash, and the user does bitbake bash they get an
error.  Instead of a 'bash is unavailable because ...'.

Also does BBMASK work across multiconfigs, I didn't think that it would from
looking at the code I have.  The masking looked to be global.

> I suspect BBMASK may be parsed at the wrong points to quite get this
> right today but it can likely be fixed to work well with multiconfig.

Ya, I think you would have to mask when parsing vs when choosing what to load.
I don't understand the multiconfig way things are loaded so maybe it already (or
almost already does this..)  But I thought things were loaded once, then parsed
multiple times.

--Mark

> Cheers,
> 
> Richard
> 
> 
> 
> 
> 
> 
> 
> 
> 
>