[Openembedded-architecture] Heterogeneous System Proposal

[Mark Hatle]

Problem Statement
-----------------
In the current world there is an increasing number of heterogeneous
systems being developed.  Currently these components can be built
independently of each other, and then combined later.  For ease of use,
it would be nice to be able to build these systems with a single build,
including generating a final bootable image.  A heterogeneous system may
simply be different configurations for different components, different
operating systems for different components, or a system made up of
diverse processor architectures.  Recently the Yocto Project has added
multiconfig in order to enable these types of configurations, but
suggested workflows for various configurations are needed to avoid
confusion and to avoid developers implementing their own schemes.

Proposal
--------
During the course of discussing this with other people a number of terms
have been involved, and after numerous discussions it is clear that
people have a different definition for the various components.  The
purpose of this document is to explain the various pieces of this
proposal, so that we can all be using the same items for the various
components necessary to build a heterogeneous system.

For a homogeneous build, the traditional Open Embedded/Yocto Project
components that are used include:

* Build Configuration (local.conf)
* MACHINE
   * Specifies target device information, including hardware
     capabilities, console settings, boot image configurations, etc.
     These settings are used by MACHINE packages, as well as image
     generation.
   * Define MACHINE in the conf/local.conf file in the Build Directory.
* SOC_FAMILY (optional, but implied by the machine)
   * Way to group together machines based upon common System On Chip
     components.
   * A SOC_FAMILY by itself is not a fully configured and bootable
     machine, but may be used by a series of machines to specify the
     common components.
* TUNE (typically specified implicitly by the machine)
   * Specifies the CPU (instruction set), and ABI settings available to
     the user
   * DEFAULTTUNE then selects which of the available tunes is to be used.
     This is usually set by the machine.
* DISTRO (Distribution Configuration)
   * Specifies cross recipe configurations that together result in an
     overall distribution configuration.
   * Define DISTRO in the conf/local.conf file in the Build Directory.
     If not specified, a default "nodistro" distribution is used.
* Recipes (.bb files)
   * Specify how to generically build individual items.  These recipes
     will inherit the system wide settings from the distribution
     configuration, and tune.  For MACHINE specific packages they can
     also inherit machine specific settings as well.
* Image Recipes
   * Specify a series of dependencies that cause recipes to be built and
     a list of resulting packages to be installed into a target image.
   * Image recipes are responsible for constructing a filesystem image.
     Further the system can extend these into a bootable disc image
     format.

In any sort of heterogeneous configuration we want to use and build upon
the existing homogeneous components.  A heterogeneous solution is really
comprised of a number of homogeneous configurations that when deployed
together result in a fully functional device.  In other words, each of
individual parts of the heterogeneous build are standalone and not tied
to the assembly of a specific system.

Based on this, we want to avoid any changes that complicate the existing
homogeneous build components or even adding additional levels of
configuration as this will complicate existing and future uses.  For
example, in the past there have been suggestions for SUBMACHINE or other
levels of hierarchy between SOC_FAMILY and MACHINE.  Adding this level
of indirection can make it more difficult to combine different
configurations into new heterogeneous solutions.  For example, if
someone has already defined a heterogeneous solution using a MACHINE,
SUBMACHINE, and SOC_FAMILY heirachy and you wish to extend it the
existing MACHINE / SUBMACHINE may interfear with your own systems unique
configuration.

There are a few types of heterogenous systems that I have seen.  Each of
them can be constructed by combining the output of homogeneous
configurations.  The most basic heterogenous systems I have seen include
either a collection of containers or different OSes, a primary CPU +
co-processors, or all of the CPUs are independent of each other but
share resources.  It is also possible to combine these heterogenous
configurations such as a multiple CPU system, with some CPUs having
co-processors with one or more CPU running containers.

In the case of the container based system, you really want a master
homogeneous machine configuration along with a few additional
configurations that can be incorporated into that image.  Using a
multilib configuration you would have something like:

build (build directory)
  conf
   local.conf:
    MACHINE = "genericx86_64"
    DISTRO = "poky"
    BBMULTICONFIG = "container1 containter2"
   multiconfig
    container1.conf:
     MACHINE = "genericx86_64"
     DISTRO = "mydistro1"
     TMPDIR = "${TOPDIR}/tmp/multi/container1"
    container2.conf:
     MACHINE = "genericx86_64"
     DISTRO = "mydistro2"
     TMPDIR = "${TOPDIR}/tmp/multi/container2"
layers
  meta-<custom_layer>
    conf
     distro
      mydistro1.conf
      mydistro2.conf
    recipes-images
     microservices
      service-image-1.bb
      service-image-2.bb
     other
      my-custom-image-recipe.bb:
       do_image[mcdepends] = "mc:container1:service-image1:do_rootfs \
                       mc:container2:service-image2:do_rootfs"
       do_image() { ... instructions for combining stuff ... }


This configuration allows for individual configurations for each
container and changes to the multiconfigs, but general re-use of much of
the system.  Especially if the distros in each one are same or similar.

Similar to the above, you could use a multiconfig system to combine
different operating systems.  For instance, instead of building a
standard-alone bare-metal style bootloader as part of the OS
configuration, you could think of it as an external non-OS application.
A configuration might look like:

build
  conf
   local.conf:
    MACHINE = "genericx86_64"
    DISTRO = "poky"
    BBMULTICONFIG = "bootloader"
   multiconfig
    bootloader.conf:
     MACHINE = "genericx86_64"
     DISTRO = "baremetal"
     TMPDIR = "${TOPDIR}/tmp/multi/bootloader"
layers
  meta-mybootloader
   conf
    distro
     baremetal.conf
   recipes
    newlib
     newlib_ver.bb
    first_stage
     first_stage.bb
    second_stage
     second_stage.bb
    bootloader
     bootloader.bb
      DEPENDS = "newlib first_stage second_stage"
  meta-<custom_layer>
   recipes
    bootloader
     mybootloader.bb
      depend on mc:bootloader:second_stage:....


For a simple heterogeneous solution, where the main CPU may need to load
software for co-processors, the configuration would be similar to the
above bare-metal example.  The difference would be that the MACHINE and
DISTRO settings for a DSP would need to have the necessary
configurations to properly build applications for the DSP.  Then the
Linux side of things can take this baremetal software and package it up
independently or in conjunction with software to actually configure and
load that software:

build
  conf
   local.conf:
    MACHINE = "myarmccpu"
    DISTRO = "poky"
    BBMULTICONFIG = "dsp"
   multiconfig
    dsp.conf:
     MACHINE = "magic-dsp"
     DISTRO = "baremetal"
     TMPDIR = "${TOPDIR}/tmp/multi/dsp"
layers
  meta-dsp
   conf
    distro
     baremetal.conf
    machine
     dsp.conf
   recipes-dsp
    newlib
     newlib_ver.bb
    library
     library.bb
    application
     application.bb
      DEPENDS = "newlib library application"
   recipes-linux
    dsp-application
     ...depends on application...

Note: in the above, separating the distro, machine and application
components into individual layers may be needed for Yocto Project
compliance.


In all of the above examples, the user has manually configured the
multiconfig within their project.  There is a simple way to move that
configuration to a layer, simply place it in a conf/multiconfig
directory within that layer.

This ability suggests to be that there should be a standard way to
specify a layer above that of machine, which defines the overall
characteristics of the system.

I'm proposing calling this new layer type as a system layer and it's
configuration variable "SYSTEM".  It will be used instead of MACHINE
when there is no single machine to quantify the contents of the produced
system image.  When implementing a system, we do not want to make major
changes to any other components.  Due to the existing implementation
requiring MACHINE and certain TUNE parameters, this will require us to
provide a special MACHINE value that can be used for a heterogeneous
system.  I suggest we create a new 'nomachine' system that only
defines/uses an equivalent noarch style tune.  This will instruct the
system that this configuration can be used to noarch software and create
images (including with wic), but it is not able to compile specific
applications.  Each of these applications or images must come from a
defined MACHINE.

The SYSTEM level multiconfig could be used to combine any homogeneous or
heterogeneous configuration.  For example:

build
  conf
   local.conf:
    SYSTEM = "mysystem"
layers
  meta-<system>
   conf
    system
     mysystem.conf
      MACHINE = "nomachine"
      BBMULTICONFIG = "bootloader fpga linux"
     mysystem.wks
    multiconfig
     bootloader.conf:
      MACHINE = "zcu102_microblaze"
      DISTRO = "baremetal"
      TMPDIR = "${TOPDIR}/tmp/multi/bootloader"
     fpga.conf:
      MACHINE = "zcu_fpga"
      DISTRO = "baremetal"
      TMPDIR = "${TOPDIR}/tmp/multi/fpga"
     linux.conf:
      MACHINE = "zcu_cortex-a72"
      DISTRO = "poky"
      TMPDIR = "${TOPDIR}/tmp/multi/linux"
   recipes
    images
     system-images.bb
      do_image[mcdepends] = "mc:bootloader:application:do_deploy \
                             mc:fpga:application:do_deploy \
                             mc:linux:core-image-minimal:do_rootfs"
      do_image() { ... instructions for combining stuff ... }
  meta-<machine>
   conf
    machine
     zcu_microblaze.conf
     zcu_fpga.conf
     zcu_cortext-a72.conf
  meta-mybootloader
   conf
    distro
     baremetal.conf
   recipes
    newlib
     newlib_ver.bb
    first_stage
     first_stage.bb
    second_stage
     second_stage.bb
    bootloader
     bootloader.bb
      DEPENDS = "newlib first_stage second_stage"
  meta-fpga
   conf
    distro
     baremetal.conf
   recipes-baremetal
    newlib
     newlib_ver.bb
    library
     library.bb
    application
     application.bb
      DEPENDS = "newlib library application"

So proposal:

Introduce a new OPTIONAL "SYSTEM" variable.  Which, like machine, would
automatically "include conf/${SYSTEM}.conf", if defined.

Introduce nomachine and notune, and adjust system components to know how
to deal with these properly.  I.e. recipes that are NOT noarch would be
be rejected in this configuration.

Introduce system layers to the yocto check script, and layer index.

Add documentation explaining the new SYSTEM variable and it's usage for
building complex heterogeneous systems.

Option:

If SYSTEM (as described above) does not make sense, then we could re-use
MACHINE instead, but I would still like a notune option for a 'machine'
that shouldn't have software compiled for it.

--Mark
This email and any attachments are intended for the sole use of the named
recipient(s) and contain(s) confidential information that may be proprietary,
privileged or copyrighted under applicable law. If you are not the intended
recipient, do not read, copy, or forward this email message or any attachments.
Delete this email message and any attachments immediately.