[Openembedded-architecture] [openembedded-architecture] sstate equivalency

[Richard Purdie]

On Fri, 2018-05-04 at 18:38 +0200, Martin Jansa wrote:
> I think the important part is that equivalency mechanism should cover
> the output of the tasks, but mostly from the POV of the other tasks
> which depend on it.
> 
> Adding or changing comments inside do_install shell functions is
> something which signature handler (or rather bitbake parser) can skip
> to prevent the do_install task to be re-executed in first place.

I think comments are a false lead here. In that case there are things
we could do to remove the rebuild, however, its also something we
comparatively rarely do and not the real problem.

> 
> But what I understand as "sstate equivalency" issue is that if you
> apply CVE fix for e.g. zlib (includes zlib-native, nativesdk-zlib),
> then all the zlib tasks of course need to be re-executed, new
> packages created and included in new images and so on, but the
> equivalency checker will compare the output of zlib (e.g. from
> do_install) and discover that zlib is different the checksum changed
> from ABC to DEF (in this case because security bug was fixed), but
> the ABI is the same as from the previous build, so this is
> "equivalent" and DEF will marked as equivalent to ABC on equivalence
> server. Then when e.g. libxml2 tasks are being considered for
> execution, then libxml2.do_configure which depends on
> zlib.do_populate_sysroot will see that zlib.do_populate_sysroot
> changed checksum from ABC to DEF, but after consulting with
> equivalence server bitbake knows that this is OK and this change of
> dependency can be ignored and libxml2.do_configure checksum 456 is
> again marked as equivalent to 123 and libxml2 build doesn't need to
> be re-executed (and whole hierarchy depending on it also doesn't need
> to be).

This is where a plugable "equivalence" mechanism could be important.

Note that your scenario above is actually potentially dangerous to the
system as you could end up with two different sstate objects, one with
a security fix and one without. They are different and should not
really ever be associated with the same hash as you want people to pick
up the new different one with the fix.

In this case you really want the original object to be not equivalent
but all dependencies to be equivalent.

I think to start with I'd prefer to see a general identical tarball
approach taken as the first pass. People would then be free to
experiment with more interesting tests for equivalence and how the
dependencies may "ripple" (or not).

> It gets much more complicated for components like gcc-cross. When
> some bug is fixed in gcc-cross then the output might have the same
> ABI, but for cross we don't care about ABI, we care about rebuilding
> the binaries with fixed toolchain, so for gcc-cross we might still
> need to rebuild everything, but still if the above is implemented,
> then gcc-cross won't really change the checksum so often, because
> e.g. ABI safe change in zlib won't cause gcc-cross to change.
> 
> I think that the best attempt on implementing this was from BMW Car
> IT.
> 
> For reference, couple related posts from thread:
> [yocto] RFC: Backwards compatibility checking sstate-cache
> it is spread across couple months (so it's shown disconnected when
> looking through pipermail):

It really depends on how deeply we dive into ABI comparisons. To start
with I think we need to look at solving some of the simpler problems.
The mechanism can scale to such ABI checks but if we try and start
there I think people would run into issues and quickly lose faith in
the whole idea.

It is good to think about whether the simple approach can scale into
these scenarios though.

Cheers,

Richard