<div dir="ltr"><div dir="ltr">Hmm... do we want to expand on the "running ptests" section in the documentation to include some stuff about viewing and analyzing results? We currently have this section here ... <a href="https://yoctoproject.org/docs/2.7/mega-manual/mega-manual.html#testing-packages-with-ptest">https://yoctoproject.org/docs/2.7/mega-manual/mega-manual.html#testing-packages-with-ptest</a></div><div dir="ltr"><br></div><div>Scott<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 28, 2019 at 3:39 AM Richard Purdie <<a href="mailto:richard.purdie@linuxfoundation.org">richard.purdie@linuxfoundation.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">We now have better data than ever before on the status of the project<br>
from a test results perspective. In particular we can now easily<br>
collect ptest data.<br>
<br>
I triggered the tests for qemux86-64 on the autobuilder yesterday, the<br>
results are published and you can see them with:<br>
<br>
wget <a href="https://autobuilder.yocto.io/pub/non-release/20190127-9/testresults/qemux86-64-ptest/testresults.json" rel="noreferrer" target="_blank">https://autobuilder.yocto.io/pub/non-release/20190127-9/testresults/qemux86-64-ptest/testresults.json</a> <br>
<br>
To show the raw logs you can quickly hack to make then more readable:<br>
<br>
cat testresults.json | sed 's/\\n/\n/g' > testresults1.json <br>
<br>
and there are numerous worrying things when you read them such as if<br>
you grep for "TIMEOUT:".<br>
<br>
I also have a suspicion that some ptest runners are exiting without<br>
cleaning up all processes resulting in corrupted output later in the<br>
logs too, I saw something like that locally, I think for perl. More<br>
investigation is needed for that.<br>
<br>
There are multiple areas that could do with help:<br>
<br>
a) investigate/fix the timeouts<br>
b) improve the test results tool Ee Peng has posted to include ptest<br>
processing tools:<br>
i) view the logs<br>
ii) compare the results<br>
iii) highlight timeouts <br>
iv) summarise durations<br>
c) Split logs into individual ptest runs instead of one block<br>
d) record the timings for the individual ptest blocks<br>
e) Add timeout status for tests into the json so we can explicitly test<br>
for it.<br>
<br>
I'd welcome help from anyone in any of the above areas, we probably<br>
need to create feature enhancement bugs for them.<br>
<br>
Cheers,<br>
<br>
Richard<br>
<br>
<br>
-- <br>
_______________________________________________<br>
Openembedded-core mailing list<br>
<a href="mailto:Openembedded-core@lists.openembedded.org" target="_blank">Openembedded-core@lists.openembedded.org</a><br>
<a href="http://lists.openembedded.org/mailman/listinfo/openembedded-core" rel="noreferrer" target="_blank">http://lists.openembedded.org/mailman/listinfo/openembedded-core</a><br>
</blockquote></div>