[oe] [RFC] Initial Proposal for Packaged Staging Revamp (was [RFC] Make some big changes right after next stable)

[Frans Meulenbroeks]

Tom: +1 for the initiative

Having had my share of trouble with packaged staging and as a result
from that having fixed a bug or what, I do welcome the idea of
improving it.

Wrt to the ideas & discussion above:
I consider the output from do-install as being the most important.
That is the deliverable of the package. Staging intermediate steps to
me seems less useful.
However if it is decided to do that, please consider making it
optional. I don't think I appreciate having staged packages with all
the intermediate object files.
Also if something changes typically all of the package needs to be
rebuild. E.g. if a provider changes it could be that configure gives a
different result, so there is no point in keeping that.

Wrt the stamps: I think the current stamps are useful when it comes to
deciding what is build and what has to be rebuild (e.g. if do_compile
fails).
For packaged staging it would be nice if one could also put them on a
feed so others could build prebuild packages (if they desire).
(this could also be internal in e.g. a company).
The current time-based stamps are not too handy here as they might
cause problems due to clock skew between systems and people mixing up
local time and GMT etc.

Instead of the current stamps a signature could be better.
A very simplistic proposal would be to use the chain of the child
signatures followed by the PR of the package (only the digit part, not
the r)
Basically in the end this delivers a sequence of all PR values of all
underlying recipes. If an underlying recipe is bumped the signature
generated while parsing does not match any more and dependent packages
need to be rebuild.

Unfortunately things are probably not that simple. We might need to
decide on ordering of packaged (the way they appear in DEPENDS?
Alphabetically by package name?
Also I can imagine that signatures (especially for images become very long).
We might be able to use a hash function to reduce the size of the
signatures and only store the hash.

What I am not sure of is whether it is desirable to build a package
that you do not want to be staged (or not yet).
One of the issues I am facing every once in a while is that I modified
a recipe, bumped PR, but in the end the change turns out not to be
good and I want to revert my work.
Guess it might be nice to be able to inhibit the staging of recipes
that are not yet commited (maybe user configable in e.g. local.conf)

BTW in the past I've been pondering about another scenario: give each
package its own subdir in staging and control which packages are
actually used through -I and -L flags.
That would avoid that one has to unpack a lot of packages.
(I'm assuming that we want to start with an empty staging when a build
starts; and that it is being popluated as needed; otherwise you can
still inadvertedly pick up remains from another package, but I am
fearing the performance penalty; I can imagine that it makes sense to
have some core packages always present (e.g. libc).

Thinking of it, do we also want to apply this for native, or should we
assume that native always contains all native packages (so implictly
making all recipes depend on all -native recipes).
After all native is the build environment.

guess there are plenty of things I overlooked and forgot. I'll add if
new things pop up.

Frans

PS: if desired I am willing to contribute, but note that my pythonese
is marginal.