[oe] Really package based staging

[Esben Haabendal]

Hi

As part of some work we have been doing here at DoréDevelopment since
spring 2010, I would like to share with the OpenEmbedded community my
idea on how build dependencies could/should be handled.

I hope that some will take the time to consider the underlying
concepts and think of where it might be lead to.

The basic idea is:

For each recipe being build (ie. each ${WORKDIR}), a complete stage
directory is being setup completely from scratch, built entirely from
"normal" packages.  A typical stage directory will thus contain a mix
of native packages, cross packages and machine/target packages (the
sysroot part of this staging), unpacked to a defined directory layout.

For recipe developers, this means that ${DEPENDS} must be specified in
the form of the packages needed, and not the recipes needed.  You
cannot depend on a recipe, only the packages it provides.  This is
obviously fundamentally different to how ${DEPENDS} is specified in
existing OpenEmbedded recipes.

This way we don't have to have a lot of special handling of staging
and build dependencies.  It is mostly "business as usual" as we know
it when building fx. a target filesystem. KISS.

With this per-recipe staging, there are no more race conditions
related to shared staging/sysroot.

There is no need to care about packaging for both build and run-time
usage, reducing both the amount of code needed to handle this, and the
diskspace consumed in tmp/.

The concept is fully working, with code available at
http://www.gitorious.org/oe-lite

A lot of existing OpenEmbedded features are not supported by OE-lite,
as the focus has been exclusively on building "deeply-embedded" Linux
system images for industrial use.  But the concept above is not the
root to any of these omissions.

Also, if you dig into the OE-lite code, you will find other
differentiators in OE-lite, but they are not directly related to the
subject of this mail.

I am fully aware that it will require significant effort to extend
this concept to OpenEmbedded as a whole, but I strongly believe that
it would be worth the effort in the long run.  Also, I don't see a way
to do this in "microsteps", as it will require a specific point in
time where the metadata should be changed over to this new way of
defining build dependencies. Alternatively, something like OE-lite
that is significantly smaller in scope than OpenEmbedded will at least
be here.

So what do you think?

Best regards,
Esben Haabendal