[Openembedded-architecture] [openembedded-architecture] sstate equivalency

[Martin Jansa]

I think the important part is that equivalency mechanism should cover the
output of the tasks, but mostly from the POV of the other tasks which
depend on it.

Adding or changing comments inside do_install shell functions is something
which signature handler (or rather bitbake parser) can skip to prevent the
do_install task to be re-executed in first place.

But what I understand as "sstate equivalency" issue is that if you apply
CVE fix for e.g. zlib (includes zlib-native, nativesdk-zlib), then all the
zlib tasks of course need to be re-executed, new packages created and
included in new images and so on, but the equivalency checker will compare
the output of zlib (e.g. from do_install) and discover that zlib is
different the checksum changed from ABC to DEF (in this case because
security bug was fixed), but the ABI is the same as from the previous
build, so this is "equivalent" and DEF will marked as equivalent to ABC
on equivalence server. Then when e.g. libxml2 tasks are being considered
for execution, then libxml2.do_configure which depends on
zlib.do_populate_sysroot will see that zlib.do_populate_sysroot changed
checksum from ABC to DEF, but after consulting with equivalence server
bitbake knows that this is OK and this change of dependency can be ignored
and libxml2.do_configure checksum 456 is again marked as equivalent to 123
and libxml2 build doesn't need to be re-executed (and whole hierarchy
depending on it also doesn't need to be).

It gets much more complicated for components like gcc-cross. When some bug
is fixed in gcc-cross then the output might have the same ABI, but for
cross we don't care about ABI, we care about rebuilding the binaries with
fixed toolchain, so for gcc-cross we might still need to rebuild
everything, but still if the above is implemented, then gcc-cross won't
really change the checksum so often, because e.g. ABI safe change in zlib
won't cause gcc-cross to change.

I think that the best attempt on implementing this was from BMW Car IT.

For reference, couple related posts from thread:
[yocto] RFC: Backwards compatibility checking sstate-cache
it is spread across couple months (so it's shown disconnected when looking
through pipermail):

https://lists.yoctoproject.org/pipermail/yocto/2017-June/036919.html
https://lists.yoctoproject.org/pipermail/yocto/2017-July/036931.html
https://lists.yoctoproject.org/pipermail/yocto/2017-September/038058.html
https://lists.yoctoproject.org/pipermail/yocto/2017-September/038074.html

Cheers,

On Fri, May 4, 2018 at 5:53 PM, Richard Purdie <
richard.purdie at linuxfoundation.org> wrote:

> On Wed, 2018-05-02 at 10:41 +0800, ChenQi wrote:
> > My focus is on user experience.
> > sstate is a successful mechanism because it has simple and clear
> > rules.
> >  From the user's view, it's easy to understand why something is
> > rebuilt.
> > Adding such complex equivalency checking will surprise users, and
> > then
> > confuse them.
> > So regardless of the solution, the goal itself is trying to break
> > these
> > simple and clear rules.
>
> I certainly agree that we want to keep things simple. I believe the
> equivalency mechanism can be kept very simple though and shouldn't
> cause too much added confusion (compared to everything else we do).
>
> > > I think one of the biggest open questions I have is the mechanism
> > > for
> > > detecting if some new task hash is equivalent to an old one (known)
> > > one. I think there are two basic options:
> > >
> > > 1) Do an actual build and determine equivalence afterward. This
> > > seems
> > > the easiest option in my mind, but it does require at least one
> > > build
> > > of the new hash to determine that it is functionally equivalent to
> > > an
> > > old hash, and then a mechanism to publish this result so that
> > > hopefully no one else has to do it also.
> > This solution doesn't converge.
> > Say we have in database stating that recipe A's do_install would
> > consider the following changes to not affect the final output.
> > 1) adding comment "# this is 1"
> > 2) adding comment "# this is 2"
> > 3) adding comment "# this is 3"
> > Then, when user adds a comment "# this is 1", A's do_install is not
> > rerun; when the user adds a comment "# this is 4", and the do_install
> > is  rerun. It's not reasonable.
> > Basically we have infinite numbers of comments, so this solution does
> > not converge.
>
> do_install is not an sstate task but it could be in theory so lets
> pretend it is.
>
> The result of do_install is the ${D} directory containing the set of
> installed files which sstate would turn into a tarball.
>
> Imagine we have three builds with each of your different comments, I'll
> call them builds 1, 2 and 3.
>
> We would create that sstate tarball and take a checksum of it in each
> case. If the checksums are the same, we'd deem the build to be
> equivalent since the binary output is the same. I appreciate we'd have
> to be careful about timestamps etc. but that is a solvable issue.
>
> All the sstate server would need to know is that the sstate checksums
> corresponding to builds 1 to 3 are equivalent.
>
> The setup implies that new configurations have to get built at least
> once. The hope is that if something early in the build chain is found
> to be the same, large parts of previous builds can be reused from
> sstate.
>
> Does that help?
>
> Cheers,
>
> Richard
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-architecture/attachments/20180504/f27303c5/attachment-0002.html>