[bitbake-devel] [PATCH 0/5] Implement a new remote layer module

[Mark Hatle]

The code for this work is available at:

git://git.openembedde.org/poky-contrib  mgh/bitbake-setup


A new module 'layers' is being added to bitbake.  This module contains
two parts that work together.  A 'layerindex' piece and a 'manager' piece.

The layerindex piece is designed to connect to layerindex content, such as
layers.openembedded.org, pull down layerindex information and provide basic
functionality for using the index.  It is capable of managing information
from multiple layerindex sources and resolving dependencies across indexes.

The connection to the external items is handled through plugins.  This permits
us to implement both a restapi format (connect to the web), but also an
internal 'cooker' implementation.  The cooker implementation will allow us to
create a mini-layerindex from the currently parsed system contents.

The internal data representation always uses the web style restapi, but is
wrapped with a number of layerindex provided object classes.

The layerindex provides object classes for each exported layerindex item, such
as Branch, LayerItem, LayerBranches, LayerDependency, etc...  Each of these
objects contain various accessor functions that simply return the data or
further process the data to make it easier to access related objects.  For
example a LayerBranch object refers to both a Branch-Id and a LayerItem-Id.
There are accessor functions designed to return back the Branch or LayerItem
without the user needing to know how to process the data structures.


The manager provides connection information to get to cooker instances for
some plugins, such as the layerindex's cooker plugin.  It also can interpret
the basic BBLAYERS and COLLECTIONS information to determine what the user
already has.  (The layerindex cooker plugin relies on this behavior.)

However, it's primary purpose is to manage downloads of layers, based on
layerindex generated dependency information.  This download mechanism also
uses a plugin architecture.  Currently only a 'fetch2' plugin has been
implemented. (I expect in the future additional plugins for 'git-repo', and
others to be implemented as needed.)

The download portion of the manager takes a 4 step approach:
  * setup(dependencies) - takes a dependency list of what to download/install
  * fetch() - fetch the layers
  * unpack() - unpack the fetched layers
  * update_bblayers() - update bblayers based on the previous activities


The code includes a set of bitbake-selftests for the described functionality.
Some items are not testable, such as the cooker integration, as cooker is not
available and the test data may be too limited to be useful.

The bitbake-layers has been updated to use these new modules for the
layerindex-fetch and layerindex-show-depends information.

The 'lsupdate.py' for the toaster has also been updated to use this item.
However, the longer term expectation is Toaster will be modified to no longer
handle an internal 'snapshot' of the layer index, but will use this new
functionality instead of process layer information.  That work will begin once
these items have been reviewed and agreed upon by the community.

Once toaster has been updated, the intention is to create a new
'bitbake-setup' program, that will have similar functionality to the
'wr-lx-setup' project on github (https://github.com/Wind-River/wr-lx-setup).


Mark Hatle (4):
  lib/layers: Initial layer and layer index implementeation
  bitbake-selftest: Add layers module tests
  lib/bblayers: Add support for the new layer modules
  toaster/orm/management/commands/lsupdates.py: Use new layerindex
    module

Paul Eggleton (1):
  bitbake-layers: disable parsing for layerindex commands

 bin/bitbake-selftest                             |   5 +-
 lib/bblayers/layerindex.py                       | 317 +++-----
 lib/layers/__init__.py                           |   0
 lib/layers/layerindex/__init__.py                | 974 +++++++++++++++++++++++
 lib/layers/layerindex/common.py                  | 146 ++++
 lib/layers/layerindex/cooker.py                  | 223 ++++++
 lib/layers/layerindex/restapi.py                 | 375 +++++++++
 lib/layers/manager/__init__.py                   | 253 ++++++
 lib/layers/manager/common.py                     |  60 ++
 lib/layers/manager/fetch2.py                     | 210 +++++
 lib/layers/tests/__init__.py                     |   0
 lib/layers/tests/layerindex.py                   | 372 +++++++++
 lib/layers/tests/manager.py                      | 155 ++++
 lib/toaster/orm/management/commands/lsupdates.py | 215 ++---
 14 files changed, 2954 insertions(+), 351 deletions(-)
 create mode 100644 lib/layers/__init__.py
 create mode 100644 lib/layers/layerindex/__init__.py
 create mode 100644 lib/layers/layerindex/common.py
 create mode 100644 lib/layers/layerindex/cooker.py
 create mode 100644 lib/layers/layerindex/restapi.py
 create mode 100644 lib/layers/manager/__init__.py
 create mode 100644 lib/layers/manager/common.py
 create mode 100644 lib/layers/manager/fetch2.py
 create mode 100644 lib/layers/tests/__init__.py
 create mode 100644 lib/layers/tests/layerindex.py
 create mode 100644 lib/layers/tests/manager.py

-- 
1.8.3.1