[Openembedded-architecture] Recipe Specific Sysroots

[Richard Purdie]

On Tue, 2016-11-29 at 13:13 -0600, Richard Tollerton wrote:
> Richard Purdie <richard.purdie at linuxfoundation.org> writes:
> > The first is that there is big difference between a sysroot per
> > task
> > and a sysroot per recipe. A sysroot per task would be fully
> > deterministic, but very heavy. A recipe specific sysroot is not
> > entirely deterministic since each task requires different
> > dependencies. For example, do_fetch might need subversion-native,
> > do_patch, quilt- native and then do_configure, DEPENDS.
> OTOH, I would expect do_fetch and do_patch specifically to be
> extremely common across all recipes. It almost makes more sense to
> have a separate sysroot for do_fetch/do_patch/etc than to use the
> same sysroot as used in the build.

Kind of. Otavio also proposed we could use the checksum data to decide
if a given sysroot matched and then have a pool of sysroots and make it
all task specific.

Right now, I think this would complicate things more than it solves,
particularly where I'm at with implementation but its worth giving
thought to in due course.

> > For performance, its always been envisaged that the recipe specific
> > sysroots would consist of hardlinks to minimise disk usage and IO
> > and
> > to try and minimise the performance impact of this change.
> > Currently
> > sstate is relocated into position once. With more sysroots, we need
> > to
> > relocate the code into each sysroot. This is particularly try with
> > things that contain large numbers of references to their installed
> > location. quilt-native is particularly bad, as are native .la, .pc
> > and
> > -config files.
>
> Possibly stupid asking-for-a-pony question: are we approaching a
> level of complexity here where switching to a chroot-like approach
> might be more maintainable than relocations? i.e., create a new root
> dir for each task/recipe, bind-mount the sysroot and relevant
> source/build components into it, then chroot into that to do the
> build. Or fake a chroot out via LD_PRELOAD.

We're not approaching that, no.

The nice thing about the current approach is that it uses approaches
and code we've already had around for a while. BB_TASKDEPS and the code
in autotools for finding m4 macros, our sysroot structure and sstate
code.

Taken a different way, http://git.openembedded.org/openembedded-core-co
ntrib/log/?h=rpurdie/wip-rss shows:

27 files changed, 438 insertions, 288 deletions

which given the scale of the change, isn't so bad :)

The scenario you describe is several leaps into much more unknown
waters. The build tools would have to be relocated below the head of
the sysroot for example which I think would cause our tools to scream
amongst other things.


> This also faintly reminds me of Google's cloud build tools [1],
> wherein a large amount of development complexity winds up getting
> hidden behind a FUSE fs. One could avoid creating sysroots on disk in
> the first place, instead populating and relocating them on-the-fly
> from sstate or something. That could also provide a way to solve
> dependency rebuild problems.
> 
> [1] http://google-engtools.blogspot.com/2011/10/build-in-cloud-distri
> buting-build.html

The trouble is its very alien compared to the way the system works and
expects to work today so I'd imagine a lot more work and problems
trying to move to a model like this.

> > * There are problems with anonymous python execution ordering. If
> > depends get added after the staging.bbclass anon python,
> > dependencies
> > are missed and they don't get installed into the sysroot. We could
> > run
> > the population funciton on every task but that seems wasteful.
> Doing that would be on the critical path to per-task sysroots, no?

We still need to ensure that any task's dependencies are in the
sysroot, regardless of whether its a recipe specific sysroot or a task
specific sysroot so we need to solve this one either way.

Cheers,

Richard