[OE-core] Help with intermittent autobuilder exposed bugs

[Richard Purdie]

We have a situation brewing on the autobuilder. Each run there are some
bugs which sometimes can appear. Law of averages says one or more of
the issues below will now appear as we have so many of them. The
project's standard processes aren't helping to capture the issues or
track them down. 

With the release mostly sorted out from my perspective I can look at
some of the other health aspects of the project.

In theory SWAT should have filed bugs for these. In most cases they
haven't. If they did, we'd discuss them at triage, decide they looked
hard and then there wouldn't be anyone to "assign" to them so they'd
most likely end up on my plate anyway. I find it hard to know what to
do with these. I already have a load in my bug backlog which I've not
gotten sorted out so adding more probably won't help.

To summarise the issues [just from the last few builds]:

https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/756
oe-selftest recipetool.RecipetoolTests.test_recipetool_load_plugin
RP has a long standing bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13070
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1177
qa-extras2 step2c testimage logrotate.LogrotateTest.test_2_logrotate
for core-image-sato
[no open bug?]

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/441
oe-selftest devtool.DevtoolExtractTests.test_devtool_deploy_target
[no open bug, unhelpful backtrace from tinfoil]

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/440
oe-sefltest signing.LockedSignatures.test_locked_signatures
[no open bug or any idea why it failed]

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/440
oe-selftest distrodata.Distrodata.test_maintainer
[no open bug, same unhelpful backtrace from tinfoil]

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/440
oe-selftest reproducible.ReproducibleTests.test_reproducible_builds
[no open bug?, fedora specific perldoc reproducibility issue]

https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/52
qemuarm systemd timeout parselogs error
"""Central error: Oct 19 11:21:48 qemuarm login[228]:
pam_systemd(login:session): Failed to create session: Connection timed
out"""
[no open bug]

Some of these are with master-next but I don't believe are related to
any issue specific to -next, its just because -next has had more test
runs.

I'm not sure what we do about this. The SWAT process really needs to
pay more attention, equally the people are volunteers and therefore I
try to be grateful for any help.

These issues are becoming more frequent and mean we lose confidence in
the builds which delays patch merging. People often ask me how they can
help, ideas on how we can move forward with this would be welcome...

I also know Armin has a difficult to debug selftest issue with warrior
which is blocking its release build right now.

Cheers,

Richard