[OE-core] Cache unihash ... doesn't match BB_UNIHASH ...

[Richard Purdie]

On Fri, 2020-02-07 at 08:16 +0000, Alex Kiernan wrote:
> Hi
> 
> We've got a couple of multiconfig builds with different
> configurations in one bitbake run (whether it needs to be like this a
> different question - I suspect we had something that worked and
> stopped there!)
> 
> However, for sometime we've been seeing errors like this on master
> (the recipes vary):
> 
> ERROR: mc:forensic:ca-certificates-20190110-r0
> do_deploy_source_date_epoch: Cache unihash
> 37d2546973a2b7c16fbcb1a8dd4939e4a1bba8c3d6d2dfb6e7204ca08d437ace
> doesn't match BB_UNIHASH
> 09baf78bd9316acdd308980c42ea4ed1eacea61bcc0d840c36293d1a5a22db69

This is an internal sanity test. It means the hash that it had in the
task/worker context didn't match what the server thought the unihash
should be. There have been changes in this area to try and fix various
things, I'm wondering if we introduced a bug with multiconfig handling.

There are some patches in master-next related to this area so it would
be interesting to know if that made things better/worse.

> Coupled with deadlock messages:
> 
> WARNING: Runqeueue deadlocked on deferred tasks, forcing task
> mc:factory:virtual:native:/var/lib/jenkins/workspace/nanohub_master/b
> uild/../poky/meta/recipes-
> support/sqlite/sqlite3_3.31.1.bb:do_populate_sysroot

This happens if a build has multiple different components with matching
hashes. It builds the "first" one and defers the others since they
could become available in sstate after the first one builds.

You should see messages from runqueue.py:

bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h]))

The behaviour should be to therefore prioritise one multiconfig. I
worry that the sorting isn't quite right and its building bits of
several multiconfigs and deadlocking?

Do you have the logs with the Deferring XXX after YYY messages in you
could share? I suspect its a sorting problem.

Cheers,

Richard