[Openembedded-architecture] Heterogeneous System Proposal

[Richard Purdie]

On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
> In all of the above examples, the user has manually configured the
> multiconfig within their project.  There is a simple way to move that
> configuration to a layer, simply place it in a conf/multiconfig
> directory within that layer.
> 
> This ability suggests to be that there should be a standard way to
> specify a layer above that of machine, which defines the overall
> characteristics of the system.
> 
> I'm proposing calling this new layer type as a system layer and it's
> configuration variable "SYSTEM".  It will be used instead of MACHINE
> when there is no single machine to quantify the contents of the
> produced system image.  When implementing a system, we do not want to
> make major changes to any other components.  Due to the existing
> implementation requiring MACHINE and certain TUNE parameters, this
> will require us to provide a special MACHINE value that can be used
> for a heterogeneous system.  I suggest we create a new 'nomachine'
> system that only defines/uses an equivalent noarch style tune.  This
> will instruct the system that this configuration can be used to
> noarch software and create images (including with wic), but it is not
> able to compile specific applications.  Each of these applications or
> images must come from a defined MACHINE.

I'm not sure I 'buy' this explanation and I think a "nomachine" target
will be horrible in practice. In your example below, why can't
mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other bits of
linux.conf and BBMULTICONFIG = "bootloader fpga" ? 

> The SYSTEM level multiconfig could be used to combine any homogeneous
> or heterogeneous configuration.  For example:
> 
> build
>   conf
>    local.conf:
>     SYSTEM = "mysystem"
> layers
>   meta-<system>
>    conf
>     system
>      mysystem.conf
>       MACHINE = "nomachine"
>       BBMULTICONFIG = "bootloader fpga linux"
>      mysystem.wks
>     multiconfig
>      bootloader.conf:
>       MACHINE = "zcu102_microblaze"
>       DISTRO = "baremetal"
>       TMPDIR = "${TOPDIR}/tmp/multi/bootloader"
>      fpga.conf:
>       MACHINE = "zcu_fpga"
>       DISTRO = "baremetal"
>       TMPDIR = "${TOPDIR}/tmp/multi/fpga"
>      linux.conf:
>       MACHINE = "zcu_cortex-a72"
>       DISTRO = "poky"
>       TMPDIR = "${TOPDIR}/tmp/multi/linux"
>    recipes
>     images
>      system-images.bb
>       do_image[mcdepends] = "mc:bootloader:application:do_deploy \
>                              mc:fpga:application:do_deploy \
>                              mc:linux:core-image-minimal:do_rootfs"
>       do_image() { ... instructions for combining stuff ... }
>   meta-<machine>
>    conf
>     machine
>      zcu_microblaze.conf
>      zcu_fpga.conf
>      zcu_cortext-a72.conf
>   meta-mybootloader
>    conf
>     distro
>      baremetal.conf
>    recipes
>     newlib
>      newlib_ver.bb
>     first_stage
>      first_stage.bb
>     second_stage
>      second_stage.bb
>     bootloader
>      bootloader.bb
>       DEPENDS = "newlib first_stage second_stage"
>   meta-fpga
>    conf
>     distro
>      baremetal.conf
>    recipes-baremetal
>     newlib
>      newlib_ver.bb
>     library
>      library.bb
>     application
>      application.bb
>       DEPENDS = "newlib library application"
> 
> So proposal:
> 
> Introduce a new OPTIONAL "SYSTEM" variable.  Which, like machine,
> would automatically "include conf/${SYSTEM}.conf", if defined.

I know we talked and I think I mentioned SYSTEM. I'm still not entirely
sold on that name, however the name is only one piece of the puzzle so
I'll defer that for now.

> Introduce nomachine and notune, and adjust system components to know
> how to deal with these properly.  I.e. recipes that are NOT noarch
> would be be rejected in this configuration.

I definitely do not like this at all. I'm not sure why its needed.

> Introduce system layers to the yocto check script, and layer index.
> 
> Add documentation explaining the new SYSTEM variable and it's usage
> for
> building complex heterogeneous systems.
> 
> Option:
> 
> If SYSTEM (as described above) does not make sense, then we could re-
> use
> MACHINE instead, but I would still like a notune option for a
> 'machine'
> that shouldn't have software compiled for it.

I'd also put the idea out there that in many ways what you're
describing is actually a configuration or system configuration. Its
configurations like this we actually test on the project autobuilder
and its what the code in yocto-autobuilder-helper generates from json
files.

There is a little known/used piece of bitbake which is BBTARGETS. If
your myssystem.conf sets:

BBTARGETS = "system-images:do_image"

(or something like that), it would even then just work from "bitbake"
being run without anything else as a parameter.

I do wonder if it would be better to have some higher level
"configuration" abstraction in bitbake (layer and configuration setup?)
rather than adding in yet more levels to the conf file hierarchy.
'system' configuration would then be one subset of that...?

Cheers,

Richard