[OE-core] [RFC] One shared state reuse solution

[Koen Kooi]

Op 30 mrt. 2012, om 14:54 heeft Chris Larson het volgende geschreven:

> Greetings,
> 
> Over the past day, I've implemented a solution for the shared state
> reuse issues Mentor has seen with our poky-based product. This
> solution is similar in concept to what we had in our Mentor Embedded
> Linux 4 (non-yocto-based) product.
> 
> This implementation restructures SSTATE_DIR such that non-target
> sstate archives are placed in a directory specific to the host we're
> running on, and allows fallback to sstates from compatible hosts. In
> addition, there is a hook in place for modifying the returned build
> host identifier string. Using these capabilities, configured as you'll
> see below at the gist, I can populate sstate-cache from a Centos5
> machine and fully reuse that shared state on a u1004 machine, but if I
> take the u1004 sstate-cache and pull it over to Centos5, all the
> non-target recipes will be rebuilt.
> 
> This has a list of "compatible hosts", which are used as fallback
> regardless of what distro you're running on, so assumes you won't be
> running on a host older than the ones you're using as your
> compatibility baseline. I think this will satisfy the needs of most,
> and as you'll see when you look at the implementation, is entirely
> opt-in currently, so does no harm to anyone who chooses not to utilize
> it.
> 
> https://gist.github.com/2253903 - shows an example of how to make use
> of this functionality
> https://github.com/kergoth/oe-core/compare/sstate-structure - the implementation
> 
> Regarding the implementation, I realize it isn't as clean as it could
> be, but the only way to resolve it in a cleaner way would be to modify
> bitbake, which I wasn't prepared to do at this juncture. The
> fundamental issue adding complexity is that SSTATE_DIR is pulled from
> the configuration metadata, not cached per-recipe, so one can't
> manipulate where individual recipes get their archives stored. As a
> workaround, I set the global SSTATE_DIR to the host-bound location,
> then when writing the archives for target recipes, moves the archive
> up to the parent directory (to the root of the original SSTATE_DIR)
> and symlinks it back.
> 
> Richard had proposed modifying the filename rather than the directory
> structure (e.g. via the sstate package arch), but this would make
> things far more complex. In order to implement fallback, one would
> have to mangle the filename, and one wouldn't be able to simply
> leverage SSTATE_MIRRORS to fetch the variants in a simple way, as it
> would have to attempt to fetch multiple filenames, which would require
> invasive changes to sstate.bbclass. I think using the directory
> structure is the easiest and cleanest route to the goal without
> invasive changes, and given it's opt-in nature, I'd like to see this
> go upstream, at a minimum as a temporary measure until/if a longer
> term more invasive solution occurs.
> 
> I'm looking for questions, comments, testers, and in particular,
> thoughts on whether this will meet the needs of others with similar
> requirements (e.g. others shipping metadata with associated shared
> state).

This is not strictly related to your patchset, but has anyone thought about license based blacklisting of sstate? I can imagine that an autobuilder will build everything, includes things like evil 3d drivers, but no want anyone to access the sstate for those builds.

regards,

Koen