[OE-core] [RFC] One shared state reuse solution

[Koen Kooi]

Op 30 mrt. 2012, om 16:23 heeft Chris Larson het volgende geschreven:

> On Fri, Mar 30, 2012 at 3:02 PM, Koen Kooi <koen at dominion.thruhere.net> wrote:
>> Op 30 mrt. 2012, om 14:54 heeft Chris Larson het volgende geschreven:
>> 
>>> Greetings,
>>> 
>>> Over the past day, I've implemented a solution for the shared state
>>> reuse issues Mentor has seen with our poky-based product. This
>>> solution is similar in concept to what we had in our Mentor Embedded
>>> Linux 4 (non-yocto-based) product.
>>> 
>>> This implementation restructures SSTATE_DIR such that non-target
>>> sstate archives are placed in a directory specific to the host we're
>>> running on, and allows fallback to sstates from compatible hosts. In
>>> addition, there is a hook in place for modifying the returned build
>>> host identifier string. Using these capabilities, configured as you'll
>>> see below at the gist, I can populate sstate-cache from a Centos5
>>> machine and fully reuse that shared state on a u1004 machine, but if I
>>> take the u1004 sstate-cache and pull it over to Centos5, all the
>>> non-target recipes will be rebuilt.
>>> 
>>> This has a list of "compatible hosts", which are used as fallback
>>> regardless of what distro you're running on, so assumes you won't be
>>> running on a host older than the ones you're using as your
>>> compatibility baseline. I think this will satisfy the needs of most,
>>> and as you'll see when you look at the implementation, is entirely
>>> opt-in currently, so does no harm to anyone who chooses not to utilize
>>> it.
>>> 
>>> https://gist.github.com/2253903 - shows an example of how to make use
>>> of this functionality
>>> https://github.com/kergoth/oe-core/compare/sstate-structure - the implementation
>>> 
>>> Regarding the implementation, I realize it isn't as clean as it could
>>> be, but the only way to resolve it in a cleaner way would be to modify
>>> bitbake, which I wasn't prepared to do at this juncture. The
>>> fundamental issue adding complexity is that SSTATE_DIR is pulled from
>>> the configuration metadata, not cached per-recipe, so one can't
>>> manipulate where individual recipes get their archives stored. As a
>>> workaround, I set the global SSTATE_DIR to the host-bound location,
>>> then when writing the archives for target recipes, moves the archive
>>> up to the parent directory (to the root of the original SSTATE_DIR)
>>> and symlinks it back.
>>> 
>>> Richard had proposed modifying the filename rather than the directory
>>> structure (e.g. via the sstate package arch), but this would make
>>> things far more complex. In order to implement fallback, one would
>>> have to mangle the filename, and one wouldn't be able to simply
>>> leverage SSTATE_MIRRORS to fetch the variants in a simple way, as it
>>> would have to attempt to fetch multiple filenames, which would require
>>> invasive changes to sstate.bbclass. I think using the directory
>>> structure is the easiest and cleanest route to the goal without
>>> invasive changes, and given it's opt-in nature, I'd like to see this
>>> go upstream, at a minimum as a temporary measure until/if a longer
>>> term more invasive solution occurs.
>>> 
>>> I'm looking for questions, comments, testers, and in particular,
>>> thoughts on whether this will meet the needs of others with similar
>>> requirements (e.g. others shipping metadata with associated shared
>>> state).
>> 
>> This is not strictly related to your patchset, but has anyone thought about license based blacklisting of sstate? I can imagine that an autobuilder will build everything, includes things like evil 3d drivers, but no want anyone to access the sstate for those builds.
> 
> I'd think that this would best be handled as a part of population of
> the shared state mirror. That is, we could create a class like
> copyleft_compliance, but for population of a shared state repository,
> obeying licensing for distribution constraints.

That was my idea as well, but I worded my question too vague :)

regards,

Koen