[OE-core] Debug from failing hashequiv builds - server side problem?

[Richard Purdie]

Hi Joshua,

I did a little digging on the weird hash equiv failure. I've attached
two logs, one is the simplified console output, the other is the reams
of debug.

What is really odd is this is totally reproducible. If I clean out tmp
and cache from the build, then "bitbake nspr-native", it breaks, every
time and strange as it sounds, I'm not sure it should break
reproducibly.

What is bothering me is it never reuses m4-native from the cache.

At the start of the build we see:

DEBUG: Found unihash 6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f in place of 6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f for /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/m4/m4-native_1.4.18.bb:do_populate_sysroot from typhoon.yocto.io:8686

i.e. it looked it up and found the hash didn't change. Fine.

Later the task runs:

DEBUG: m4-native-1.4.18-r0 do_populate_sysroot: Task 6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f unihash changed 6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f -> a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd by server typhoon.yocto.io:8686

so it was changed to a new hash by the server.

So why when I clean tmp/cache and re-run the build do I not get this
new hash back?

If I don't clean cache, the hash is cached and is used as remapped.

In case its interesting, the "short" console output is:

Sstate summary: Wanted 17 Found 15 Missed 2 Current 0 (88% match, 0% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
NOTE: Task /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/m4/m4-native_1.4.18.bb:do_populate_sysroot unihash changed to a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd
NOTE: Task /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb:do_populate_sysroot unihash changed to 82d9ac234a96a673b706be0b7ee7651cbca379c41dae0bd8eb8ce2f52da5f67d
NOTE: Reported task virtual:native:/home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-support/nspr/nspr_4.24.bb:do_populate_sysroot as unihash 89b717131d37f2cf8313f0cadc8c0c7ebe9a2229c55e7effcbefe7f0a8013f33 to typhoon.yocto.io:8686 ({'unihash': '3418b83888e28fa9660977f995414241c9c47a1b963ceca8d2a281483be52df9', 'taskhash': 'f0cca9fbe2764fffc26f60e41de01d01d0e60abd447167eb0ef61fb3cef40391', 'method': 'oe.sstatesig.OEOuthashBasic'})
NOTE: Task virtual:native:/home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-support/nspr/nspr_4.24.bb:do_populate_sysroot unihash 3418b83888e28fa9660977f995414241c9c47a1b963ceca8d2a281483be52df9 unchanged by server
NOTE: Already covered setscene for virtual:native:/home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-support/nspr/nspr_4.24.bb:do_populate_sysroot so ignoring rehash (remap)
NOTE: Setscene tasks completed
ERROR: nspr-native-4.24-r0 do_compile: oe_runmake failed

so there is a second error somewhere in the runqueue code where the
failed remap causes other problems. Not sure why the remap fails but
this first bug is making the second one reproducible...

Thought I'd send in case anything obvious jumps out to you. I've cc'd
the list just so people know where we are with this.

I plan not to disturb the autobuilder too much until we get to the
bottom of these issues whilst we can reproduce them. This will delay
other patches but I think this is important.

Cheers,

Richard


-------------- next part --------------
A non-text attachment was scrubbed...
Name: failing-log.txt.gz
Type: application/gzip
Size: 2611 bytes
Desc: not available
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20191222/d7d627ea/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: failing-log-full.txt.gz
Type: application/gzip
Size: 14652 bytes
Desc: not available
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20191222/d7d627ea/attachment-0003.bin>