[Openembedded-architecture] [openembedded-architecture] sstate equivalency

[Richard Purdie]

On Wed, 2018-05-02 at 10:41 +0800, ChenQi wrote:
> My focus is on user experience.
> sstate is a successful mechanism because it has simple and clear
> rules. 
>  From the user's view, it's easy to understand why something is
> rebuilt.
> Adding such complex equivalency checking will surprise users, and
> then 
> confuse them.
> So regardless of the solution, the goal itself is trying to break
> these 
> simple and clear rules.

I certainly agree that we want to keep things simple. I believe the
equivalency mechanism can be kept very simple though and shouldn't
cause too much added confusion (compared to everything else we do).

> > I think one of the biggest open questions I have is the mechanism
> > for
> > detecting if some new task hash is equivalent to an old one (known)
> > one. I think there are two basic options:
> > 
> > 1) Do an actual build and determine equivalence afterward. This
> > seems
> > the easiest option in my mind, but it does require at least one
> > build
> > of the new hash to determine that it is functionally equivalent to
> > an
> > old hash, and then a mechanism to publish this result so that
> > hopefully no one else has to do it also.
> This solution doesn't converge.
> Say we have in database stating that recipe A's do_install would 
> consider the following changes to not affect the final output.
> 1) adding comment "# this is 1"
> 2) adding comment "# this is 2"
> 3) adding comment "# this is 3"
> Then, when user adds a comment "# this is 1", A's do_install is not 
> rerun; when the user adds a comment "# this is 4", and the do_install
> is  rerun. It's not reasonable.
> Basically we have infinite numbers of comments, so this solution does
> not converge.

do_install is not an sstate task but it could be in theory so lets
pretend it is.

The result of do_install is the ${D} directory containing the set of
installed files which sstate would turn into a tarball.

Imagine we have three builds with each of your different comments, I'll
call them builds 1, 2 and 3.

We would create that sstate tarball and take a checksum of it in each
case. If the checksums are the same, we'd deem the build to be
equivalent since the binary output is the same. I appreciate we'd have
to be careful about timestamps etc. but that is a solvable issue.

All the sstate server would need to know is that the sstate checksums
corresponding to builds 1 to 3 are equivalent.

The setup implies that new configurations have to get built at least
once. The hope is that if something early in the build chain is found
to be the same, large parts of previous builds can be reused from
sstate.

Does that help?

Cheers,

Richard