[OE-core] Hash Equiv rehash problems

[Mark Hatle]

On 11/24/19 9:58 AM, Richard Purdie wrote:
> There may be two possible fixes:
> 
> a) We force setscene tasks which have already run to rerun if this kind
> of rehash happens. We did previously do this but runqueue *really*
> doesn't like rerunning setscene tasks.
> b) We report these 'additional' equivalences to the server
> 
> We are seeing other symptoms of this "rehashing" in builds where
> "bitbake X; bitbake X" will suddenly rebuild a load of stuff in the
> second build as it didn't build it originally due to the "ignoring
> rehash" messages. This is very counter-intuitive and effectively a
> different representation of the same bug. Its less problematic since we
> just rebuild things (eSDK can't).
> 
> If we decide b) is correct, it also raises an interesting scope
> question. Should we:
> 
> a) only report things we've run into in real builds
> b) report all hashes (there'd be loads)
> c) report all hashes which are present in sstate

With some of the things I've been thinking of, specifically a combination PR,
sstate-cache server and hash equivalency server, (C) is what I was thinking of.

I've already talked to you about this, but for others who might not know why.

PR Service and sstate-cache files need to be in sync otherwise the packages that
are produced (new builds) and re-used together otherwise image installs can and
will fail.

With the new hash equivalency, determining which of the hashes to use for any
particular part of the system, all three really need to be in sync.

This is also useful for a binary distribution -> eSDK (DevTool) -> full Yocto
Project supported system.  You need all three sets of artifacts collected
together to make a transition from one to another in order to change the
configuration or add more stuff.

(Additionally you could actually validate the contents of the sstate-cache and
hash/hash-equivalency.  Multiple builds can feed into it, and the system can
validate the artifacts are actually the same or blacklist certain hashes as
non-deterministic.)

-Mark

> ?
> 
> I think I need to sit and think about this for a while. Ccing Joshua in
> case he has any insights into this...
> 
> We can't enable hashequiv by default until we get this fixed somehow.
> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
>