[Openembedded-architecture] [openembedded-architecture] sstate equivalency

[Mark Hatle]

On 5/1/18 3:28 PM, Joshua Watt wrote:
> All,
> 
> I have some interest it looking into the sstate equivalency as
> discussed in the Yocto Project Technical Meeting. I know there have
> been a few discussions about how to handle this problem, so please
> chime in with any information or background you may have.
> 
> To start things off, I figured I would try to briefly summarize the
> problem so that we have a clear idea what we are trying to solve.
> 
> My understanding is that there are many changes that can be made to a
> recipe that don't actually have any affect on what the recipe
> generates (comments in shell tasks(?), that sort of thing), but still

Note: Occasionally we've used shell comment changes specifically to force a
rebuild.  So if we can get smart enough to avoid certain things like this -- we
will still need a simply way to force a rebuild in the few cases where the
developers knows it's needed, but it's not automated.

> figure into the sstate task hashes. When such changes are detected, it
> would be convenient to users if the build could detect that these task
> hashes are actually equivalent, and allow one sstate cache object to
> be used in place of the other (e.g. If you have sstate object A and
> your build calculates the hash as B, you can ask somewhere, "is A
> equivalent to B?", and if so, use A in lieu of rebuilding to produce
> B).
> 
> I think one of the biggest open questions I have is the mechanism for
> detecting if some new task hash is equivalent to an old one (known)
> one. I think there are two basic options:
> 
> 1) Do an actual build and determine equivalence afterward. This seems
> the easiest option in my mind, but it does require at least one build
> of the new hash to determine that it is functionally equivalent to an
> old hash, and then a mechanism to publish this result so that
> hopefully no one else has to do it also.

Determining the post build equivalency needs a rule though -- many of the
existing tools we have are likely adequate.

My next question then is, how do I publish equivalencies to other people (inside
and outside of my organization.)

> 2) Try to calculate if the new hash is going to be equivalent without
> doing a build. This seems *much* harder, IMHO... but also eliminates
> the need for a "central" database of equivalent hashes (because
> everyone can determine it themselves).
> 
> 
> 
> I *think* based on what I have heard, everyone is leaning toward
> option #1, which makes a lot of sense. In that regard, I think that
> this is very doable. As a simplistic first approach, I think it should
> be entirely possible to piggyback on the reproducable build effort
> that has been on-going to solve the question of "when are two tasks
> identical". Essentially, if the outputs from two tasks are equivalent,
> the two input task hashes can be considered to be equivalent. You
> could of course perhaps get more intelligent than that, but I don't
> have a clear idea of what that would look like.
> 
> After that, the other question I can think of right now is "how are
> results published". I'm a little less clear on what ideas people had
> there, so please chime in. I think you could go a simple as "symlinks
> in the sstate cache", and as complicated as a dedicated hierarchy of
> server with authentication and signing.
> 
> I'm sure I'm glossing over a lot here....

Ya, unfortunately I have no clear answer, but a lot of possibilities.  A text
representation like the locked down hashes..  or a semi-database (think PR w/ an
sqlite DB)... or even something like a 'DNS' like solution.  Where your query is
on the hash, and your answer is on equivalence... I say DNS 'like' since if the
local query doesn't know it could pass it to more 'authoritative' services until
they are exhausted...  While complex, it could allow for the YP to publish
equivalencies, as well as OSV, development groups, etc... and have them all work
together.   (Of course we run into network access issues, is there any
identifying info in the hashes, etc etc etc..)

--Mark

> Thanks,
> Joshua Watt
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>