[oe] [RFC] Initial Proposal for Packaged Staging Revamp (was [RFC] Make some big changes right after next stable)

[Chris Larson]

On Wed, Mar 3, 2010 at 9:30 AM, Tom Rini <tom_rini at mentor.com> wrote:

> As many people know, there's a lot of "odd" internal things that OE
> does, that if we had it to do over, we would do differently.  What I
> would like to propose is that in time for the next stable branch we:
> 1: Define a set of DISTROs/MACHINEs/build targets that need to stay
> working
> 2: In a separate branch (per big change), get one of these big, going to
> leave some stuff broken changes
> 3: Define / document what needs to be done before these branches can be
> merged back (something like #1 is working still, and if applicable a
> guide to the common problems/how to fix people are going to run into).
>
> What I'm getting at is that this would let us do things like rework the
> "this is where we place things that we build other recipes with" concept
> so that sysroot just works (and otherwise makes more sense again).  Or
> "let us have more consistency in build with compared to what could end
> up on device".  And so on.
>
> What do people think?  And what would people work on?
>

I think this is a great idea overall, and have many ideas for large changes
to be considered.  Here's my first (note, have mercy, I haven't had much
need to write anything longer than a paragraph in some time, will need to
work on my writing more :):

Proposal for the Revamp of "Packaged Staging"

Goals:
- Simple implementation
- Managed staging area
- "Build" from cached/prebuilt binaries
- Reduce behavioral differences between the prebuilt and from scratch cases
- Intrinsic to the system, no longer opt-in

For those who aren't familiar with it, "packaged staging" is what it sounds
like.  It utilizes package management to manage the OpenEmbedded staging
area.  This provides a number of benefits, such as ease of uninstallation of
staging files, switch to different versions of staging files without leaving
remnants around, etc.  I believe that these benefits are obvious, and good
enough on their own merits (given people have hit the issues it fixes on
multiple occasions, even today) to argue for its inclusion as default
OpenEmbedded behavior, but there are other features, including caching of
build output to speed up builds and ease of checks for missing dependencies
via using this build output.

Currently, one opts into its usage by setting INHERIT += "packaged-staging".
 This results in a number of behavioral changes in a typical build.  An
early task for all recipes, 'do_setscene' starts to do more work, but note
that this component is not for the package management of staging, but was
done to take advantage of the packaged staging files instead as a cache of
previous build output, to speed up build time.  The setscene task checks to
see if the packaged staging package for this recipe already exists, and if
so, it extracts it, including the stamps for the recipe, so that most of the
subsequent tasks will be skipped if the package is valid.  It does some
sanity checks to determine this validity.

In addition to the do_setscene behavioral change are the pieces of code
which produce the packaged staging package, and which installs/uninstalls
pstage packages from the staging area.

I would like to propose an alternative to the current implementation, which
I believe will aleviate some headaches (for example, those caused by the
stagefile bits, which is more functionality that slips beyond the original
intent of package managed staging), make it easier to add more traceability
to the builds, reduce behavioral differences between the use of cached
binaries and building from scratch, and should help to prepare for some
possible moves in the future.

To summarize, I propose the creation of an archive/package which acts as the
primary artifact to come out of the build of a recipe.  By capturing *all*
output of a recipe into a single place, we reduce confusion and make things
easier to track.  Every subsequent task by the recipe (or other recipes)
will go based on this archive (or cached, extracted contents of this
archive, for performance).  Builds from cached binaries would operate based
on this archive, so that the execution of the subsequent tasks would be
identical between the prebuilt and from scratch cases, and it makes it clear
that this is not just "packaged staging" in concept or intent.

I intend this archive to essentially contain the output of do_install, and
to add the necessary bits into the build process to skip all tasks that
do_install depends upon if the archive already exists, without capturing the
stamps themselves into the archive, to reduce implementation headaches and
complexity.  Staging will be populated based on this archive (and new style
staging is already based upon do_install, so this is no great stretch),
likely using a generated package with a package manager under the hood for
staging alone, not all of TMPDIR, and not in a way that's at all user
visible.  Note that by doing it this way, it also moves the packaging
process, use of prebuilts will no longer mean that packaging has already
been completed.  This makes it easier for us to change around the packaging
of a recipe without having to rebuild the thing, and makes it easier for us
to move in a direction of separation of responsibilities at some point
(packaging is more distro policy than recipe responsibility), if we choose
to do so.

While I haven't fully worked out the logistics of the implementation of
this, I did, in the past, create a prototype of a "private staging areas"
implementation which also implemented this primary archive as a side effect,
and utilized it for the prototype.  Does anyone have any thoughts on this,
ideas for improvement, arguments either for or against a pstage revamp, or
alternative ideas for a revamp?  I'd love to hear what people think about
this possibility.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics