[OE-core] Help needed with ptest

[Richard Purdie]

We now have better data than ever before on the status of the project
from a test results perspective. In particular we can now easily
collect ptest data.

I triggered the tests for qemux86-64 on the autobuilder yesterday, the
results are published and you can see them with:

wget https://autobuilder.yocto.io/pub/non-release/20190127-9/testresults/qemux86-64-ptest/testresults.json 

To show the raw logs you can quickly hack to make then more readable:

cat testresults.json  | sed 's/\\n/\n/g'  > testresults1.json 

and there are numerous worrying things when you read them such as if
you grep for "TIMEOUT:".

I also have a suspicion that some ptest runners are exiting without
cleaning up all processes resulting in corrupted output later in the
logs too, I saw something like that locally, I think for perl. More
investigation is needed for that.

There are multiple areas that could do with help:

a) investigate/fix the timeouts
b) improve the test results tool Ee Peng has posted to include ptest
processing tools:
 i) view the logs
 ii) compare the results
 iii) highlight timeouts 
 iv) summarise durations
c) Split logs into individual ptest runs instead of one block
d) record the timings for the individual ptest blocks
e) Add timeout status for tests into the json so we can explicitly test
for it.

I'd welcome help from anyone in any of the above areas, we probably
need to create feature enhancement bugs for them.

Cheers,

Richard