[Openembedded-architecture] Recipe Specific Sysroots

[Richard Purdie]

We've talked about these for a long time.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=1560

To summarise, we have a problem with the existing sysroots since
everything gets put there and due to that, recipe can find something
which wasn't in their DEPENDS. This gives rise to race and determinism
problems. The solution would be to have a sysroot for each recipe.

I've made a proof of concept available on this branch:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/wip-rs
s

I'd like to discuss what I've done here a bit and dive into the
technical details.

The good news is I was able to build core-image-sato using this. The
downside is that its fairly invasive (no real surprise), there are
potentially some performance problems and in the course of making this
work, I've run into some issues I hadn't thought about/realised.

The first is that there is big difference between a sysroot per task
and a sysroot per recipe. A sysroot per task would be fully
deterministic, but very heavy. A recipe specific sysroot is not
entirely deterministic since each task requires different dependencies.
For example, do_fetch might need subversion-native, do_patch, quilt-
native and then do_configure, DEPENDS.

For performance, its always been envisaged that the recipe specific
sysroots would consist of hardlinks to minimise disk usage and IO and
to try and minimise the performance impact of this change. Currently
sstate is relocated into position once. With more sysroots, we need to
relocate the code into each sysroot. This is particularly try with
things that contain large numbers of references to their installed
location. quilt-native is particularly bad, as are native .la, .pc and
-config files.

The sstate code is modified to be able to defer the "relocation"
command. It does this by extracting the sstate to a directory structure
but leaving the path replacement files listed in a file.

The paths are replaced using a file list stored in a file and a sed
expression. I was running these once per dependency installed into the
sysroot but it was a bottleneck so I found a way to run it all as part
of one larger command.

This also meant a bigger change to the way sstate works. Currently it
does:

<generate sstate install directory>
<copy this to sstate package build directory>
<replace install directory references with tokens>
<create sstate package>
<move install directory into position>

Note that no relocation happens in this mode since the original install
directory is re-used and relocation only happens upon installation from
sstate. In the updated model it does:

<generate sstate install directory>
<copy this to sstate package build
directory>
<replace install directory references with tokens>
<create
sstate package>
<move build directory into install position>
<move
install directory into position>
<* replace tokens with final paths>

(* means the step is deferred)

This seems slightly convoluted but its complicated by "from sstate" and
"from build" codepath variants and the fact that one sstate package may
control multiple directories. It means we gain access to the "un-
relocated" version of the directory rather than the relocated version.
The downside is that I'd once hoped we could "defer" sstate objection
creation/compression. That is no longer looking possible, or at the
very least more difficult (and needing more directory copies).

I also needed to change the way sstate POSTFUNCS worked, these are used
by pixbufcache.bbclass, useradd.bbclass and a small number of recipes.
In some cases its possible to write better code which avoids their need
but its unavoidable for the classes. I'm handling that by installing
"postinst" scripts into $bindir which get run in the final sysroots.

Due to multiple tasks running in parallel against the same recipe
sysroot, I had to put in locking for the installation code path
portions.

I added a do_recipe_prepare_sysroot task which explicitly runs before
do_configure and handles DEPENDS.

The same function is executed as a prefunc for other tasks, if they
have DEPENDS on do_populate_sysroot items. This handles quilt-native
for do_patch for example but there are many other examples (do_rootfs,
do_image*, do_package, do_package_write_*).

We could drop the m4 macro mess in autotools.bbclass (the logic in
staging.bbclass is actually based on it). This does mean we have a
native and target m4 macro directory though which is problematic
(rather than one m4 macro directory like we had).

Some new variables are added:

RECIPE_SYSROOT - the sysroot representing the target system
RECIPE_SYSROOT_NATIVE - the sysroot representing the native tools

Also, pseudo is a special case and runs directly out of its own staging
directory. Its statically linked so this isn't a big deal for it.

Compiler bootstrap now doesn't need a special sysroot. The -initial
versions are blocked by the depvalid code in sstate from making it into
 recipe specific sysroots outside those needed for compiler bootstrap.

Known issues/Todo:
* Once a dependency is installed into a recipe specific sysroot, it 
  can't be removed. In particular, if you rebuild that dependency (and 
  its sstate checksum changes), it isn't reinstalled.
* We don't detect broken recipes. We should probably rename the 
  PRE/POSTFUNC variables and then error if they're set so users get 
  better feedback about the changes.
* There are problems with anonymous python execution ordering. If   
  depends get added after the staging.bbclass anon python,
dependencies 
  are missed and they don't get installed into the sysroot. We could 
  run the population funciton on every task but that seems wasteful.
* The autotools.bbclass code I based the code in staging.bbclass on is 
  likely buggy as I found problems with it. Those fixes should be 
  merged to the autotools version before we remove it for backporting 
  to older releases.
* Performance needs analysing. My last tests showed 
  do_recipe_prepare_sysroot tasks taking mostly under a second with 
  around 5 under 5s and one under 10s for core-image-sato. It was much 
  worse, it remains to be seen if it can be better.
* There are some workarounds with PATH manipulation which would no 
  longer be needed with recipe specific sysroots.
* Dependencies for packages which have native tool requirements (e.g.  
  gdkpixbuf-native) at rootfs time are problematic and ill defined in 
  the metadata.
* Need to cleanup/remove/fix the debug in staging.bbclass, its very 
  verbose).

Sorry if this is all a bit of a stream of consciousness, there is a lot
to think about with all this.

I am interested in people's comments/feedback. I'll work on teasing out
the fixes I can merge outside this patch and addressing the
clean/rebuild problem which means adding some kind of manifest handling
to the sysroots, I'll then repost it for further comments.

Cheers,

Richard