[Openembedded-architecture] Heterogeneous System Proposal

[Mark Hatle]

On 12/2/19 3:58 PM, Richard Purdie wrote:
> On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
>> In all of the above examples, the user has manually configured the
>> multiconfig within their project.  There is a simple way to move that
>> configuration to a layer, simply place it in a conf/multiconfig
>> directory within that layer.
>>
>> This ability suggests to be that there should be a standard way to
>> specify a layer above that of machine, which defines the overall
>> characteristics of the system.
>>
>> I'm proposing calling this new layer type as a system layer and it's
>> configuration variable "SYSTEM".  It will be used instead of MACHINE
>> when there is no single machine to quantify the contents of the
>> produced system image.  When implementing a system, we do not want to
>> make major changes to any other components.  Due to the existing
>> implementation requiring MACHINE and certain TUNE parameters, this
>> will require us to provide a special MACHINE value that can be used
>> for a heterogeneous system.  I suggest we create a new 'nomachine'
>> system that only defines/uses an equivalent noarch style tune.  This
>> will instruct the system that this configuration can be used to
>> noarch software and create images (including with wic), but it is not
>> able to compile specific applications.  Each of these applications or
>> images must come from a defined MACHINE.
> 
> I'm not sure I 'buy' this explanation and I think a "nomachine" target
> will be horrible in practice. In your example below, why can't
> mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other bits of
> linux.conf and BBMULTICONFIG = "bootloader fpga" ? 

This assumes the linux side of things will load the others.  In the systems I am
thinking of Linux may not be a primary system.

I.e.  a baremetal system based on a microblaze may be the core system, which
then bootstraps and launches the linux system.  The core 'machine' should it be
defined as baremetal and zcu102_microblaze then?

Problem is baremetal isn't really an operating system, so how do we handle that?

>> The SYSTEM level multiconfig could be used to combine any homogeneous
>> or heterogeneous configuration.  For example:
>>
>> build
>>   conf
>>    local.conf:
>>     SYSTEM = "mysystem"
>> layers
>>   meta-<system>
>>    conf
>>     system
>>      mysystem.conf
>>       MACHINE = "nomachine"
>>       BBMULTICONFIG = "bootloader fpga linux"
>>      mysystem.wks
>>     multiconfig
>>      bootloader.conf:
>>       MACHINE = "zcu102_microblaze"
>>       DISTRO = "baremetal"
>>       TMPDIR = "${TOPDIR}/tmp/multi/bootloader"
>>      fpga.conf:
>>       MACHINE = "zcu_fpga"
>>       DISTRO = "baremetal"
>>       TMPDIR = "${TOPDIR}/tmp/multi/fpga"
>>      linux.conf:
>>       MACHINE = "zcu_cortex-a72"
>>       DISTRO = "poky"
>>       TMPDIR = "${TOPDIR}/tmp/multi/linux"
>>    recipes
>>     images
>>      system-images.bb
>>       do_image[mcdepends] = "mc:bootloader:application:do_deploy \
>>                              mc:fpga:application:do_deploy \
>>                              mc:linux:core-image-minimal:do_rootfs"
>>       do_image() { ... instructions for combining stuff ... }
>>   meta-<machine>
>>    conf
>>     machine
>>      zcu_microblaze.conf
>>      zcu_fpga.conf
>>      zcu_cortext-a72.conf
>>   meta-mybootloader
>>    conf
>>     distro
>>      baremetal.conf
>>    recipes
>>     newlib
>>      newlib_ver.bb
>>     first_stage
>>      first_stage.bb
>>     second_stage
>>      second_stage.bb
>>     bootloader
>>      bootloader.bb
>>       DEPENDS = "newlib first_stage second_stage"
>>   meta-fpga
>>    conf
>>     distro
>>      baremetal.conf
>>    recipes-baremetal
>>     newlib
>>      newlib_ver.bb
>>     library
>>      library.bb
>>     application
>>      application.bb
>>       DEPENDS = "newlib library application"
>>
>> So proposal:
>>
>> Introduce a new OPTIONAL "SYSTEM" variable.  Which, like machine,
>> would automatically "include conf/${SYSTEM}.conf", if defined.
> 
> I know we talked and I think I mentioned SYSTEM. I'm still not entirely
> sold on that name, however the name is only one piece of the puzzle so
> I'll defer that for now.
> 
>> Introduce nomachine and notune, and adjust system components to know
>> how to deal with these properly.  I.e. recipes that are NOT noarch
>> would be be rejected in this configuration.
> 
> I definitely do not like this at all. I'm not sure why its needed.

I want a way to say that this configuration is a sum of it's parts, but by
itself can't do anything.  It's not arm, it's not ppc, it's not x86, etc..

This is common on FPGA systems that may have to load from special
configurations, the hardware itself may be loading a bitstream (not a cpu
instruction set) that in turn may be a first stage loader for one or multiple on
system CPUs, which may in turn instruct other aspects of the system to load.

I don't want the proposal to be FPGA specific, as I expect it will be used in
other heterogeneous cases.  Primarily ones that have a single host CPU (that
boot) and multiple plugin cards.  In those cases, it -CAN- have both a SYSTEM
and a MACHINE defined, or just simply a MACHINE with multiconfigs that the
MACHINE has a way of understanding.

>> Introduce system layers to the yocto check script, and layer index.
>>
>> Add documentation explaining the new SYSTEM variable and it's usage
>> for
>> building complex heterogeneous systems.
>>
>> Option:
>>
>> If SYSTEM (as described above) does not make sense, then we could re-
>> use
>> MACHINE instead, but I would still like a notune option for a
>> 'machine'
>> that shouldn't have software compiled for it.
> 
> I'd also put the idea out there that in many ways what you're
> describing is actually a configuration or system configuration. Its
> configurations like this we actually test on the project autobuilder
> and its what the code in yocto-autobuilder-helper generates from json
> files.

I would expect in my particular use-case that I do need to generate
configurations based on certain parameters (specifically a system device tree).
 However, other configurations are going to be very static and then predefined
'systems' can be provided by layer that give the user the ability to modify (as
they would any other part of the system today.)

> There is a little known/used piece of bitbake which is BBTARGETS. If
> your myssystem.conf sets:
> 
> BBTARGETS = "system-images:do_image"

I wasn't aware this was a thing, but it could very well be used by a system
configuration to define the right combination of things to build.  I didn't
mention it here, but we need a better way to build an image recipe that collects
together pieces from other multiconfigs then manually specifying everything on
an do_image[mcdepends] ...  I figured that would come over time through classes
and other future proposals.

> (or something like that), it would even then just work from "bitbake"
> being run without anything else as a parameter.
> 
> I do wonder if it would be better to have some higher level
> "configuration" abstraction in bitbake (layer and configuration setup?)
> rather than adding in yet more levels to the conf file hierarchy.
> 'system' configuration would then be one subset of that...?

In the end my thought was what works now works well.  I don't want to change it.
 I want to exploit it, and make it easier to combine those pieces together to
build new systems that have previously been constructed via external (to the
yocto project/OE-core) scripting.

There is another thing I didn't mention which will be an issue.  In a large
multiconfig build, each config adds more and more parsing overhead to the
system.  For a heterogeneous system in which some of the configurations are
baremetal, non-Linux (unix-like) OSes, we should be able to disable a large part
of the recipe pool easily -- this should improve parsing time, I think.

(Again, future work -- once the framework is in place to exploit this.)

--Mark

> Cheers,
> 
> Richard
>