[Openembedded-architecture] [openembedded-architecture] sstate equivalency

[Joshua Watt]

All,

I have some interest it looking into the sstate equivalency as
discussed in the Yocto Project Technical Meeting. I know there have
been a few discussions about how to handle this problem, so please
chime in with any information or background you may have.

To start things off, I figured I would try to briefly summarize the
problem so that we have a clear idea what we are trying to solve.

My understanding is that there are many changes that can be made to a
recipe that don't actually have any affect on what the recipe
generates (comments in shell tasks(?), that sort of thing), but still
figure into the sstate task hashes. When such changes are detected, it
would be convenient to users if the build could detect that these task
hashes are actually equivalent, and allow one sstate cache object to
be used in place of the other (e.g. If you have sstate object A and
your build calculates the hash as B, you can ask somewhere, "is A
equivalent to B?", and if so, use A in lieu of rebuilding to produce
B).

I think one of the biggest open questions I have is the mechanism for
detecting if some new task hash is equivalent to an old one (known)
one. I think there are two basic options:

1) Do an actual build and determine equivalence afterward. This seems
the easiest option in my mind, but it does require at least one build
of the new hash to determine that it is functionally equivalent to an
old hash, and then a mechanism to publish this result so that
hopefully no one else has to do it also.

2) Try to calculate if the new hash is going to be equivalent without
doing a build. This seems *much* harder, IMHO... but also eliminates
the need for a "central" database of equivalent hashes (because
everyone can determine it themselves).



I *think* based on what I have heard, everyone is leaning toward
option #1, which makes a lot of sense. In that regard, I think that
this is very doable. As a simplistic first approach, I think it should
be entirely possible to piggyback on the reproducable build effort
that has been on-going to solve the question of "when are two tasks
identical". Essentially, if the outputs from two tasks are equivalent,
the two input task hashes can be considered to be equivalent. You
could of course perhaps get more intelligent than that, but I don't
have a clear idea of what that would look like.

After that, the other question I can think of right now is "how are
results published". I'm a little less clear on what ideas people had
there, so please chime in. I think you could go a simple as "symlinks
in the sstate cache", and as complicated as a dedicated hierarchy of
server with authentication and signing.

I'm sure I'm glossing over a lot here....

Thanks,
Joshua Watt