[oe] Eliminating dependency race-conditions

[Frans Meulenbroeks]

Dear all,

pb and I had a discussion about this on irc earlier today.

Actually we feel there are three different issues in the proposal:
1. only stage those recipes that are in DEPENDS.
2. stage based on packages instead of recipes (so e.g. you can stage
only a lib package)
3. use tar (or a different tool) to stage things; or in other words,
decouple the storage format and the package format.

I think each of these has its merits. I also understand that
incremental introduction is desired above a big bang approach.

Let us look at the issues in detail:

1. only stage those recipes that are in DEPENDS.

This seems pretty useful as it gives consistent builds not depending
on whatever you have build before (and so making things deterministic)
This is especially useful when doing incremental builds. It can also
solve races.
Introducing it would mean changes in staging; incremental introduction
is possible if e.g. we add a flag PACKAGE_BASED_STAGING or so to the
recipes we want this to have (or add a special class or so).
Once all recipes are moved, the flag can be removed again.

Question is how efficient this can be. It mostly depends on how we
want to achieve the staging.
I'm definitely no expert, but we could e.g.
- create the staging area from earlier package images each time we
build it. Does not seem overly efficient
- stage each package in its own dir and create symlinks in the
required staging area.
- stage each pacakge in its own dir and add to -I and -L to only stage
the packages that are needed.

The staging area for a package to be build should probably be within
the work dir of the recipe otherwise we'll loose the capability to
build things in parallel

2. stage based on packages instead of recipes (so e.g. you can stage
only a lib package)

This gives more fine grained control. Not sure how much it brings and
what it will cost.
It might be possible to do it incrementally by e.g. having a keyword
PACKAGE_DEPENDS.
Question is if it is worth the effort

3. use tar (or a different tool) to stage things; or in other words,
decouple the storage format and the package format.

To me that is a completely different question only loosely related to
the above.
If staging is from packaged images, it would help though to have a
format that is fast to unpack.

Another, perhaps more interesting reason, would be to use a package
manager agnostic format and use that to build packages from (for the
package manager you want to have it).
I don't oversee the impact of this one though.


Your opinion on this proposal, improvement suggestions, discussion on
the best way forward etc is appreciated.

Frans