[OE-core] Help needed with ptest

[Richard Purdie]

On Mon, 2019-01-28 at 11:39 +0000, Richard Purdie wrote:
> There are multiple areas that could do with help:
> 
> a) investigate/fix the timeouts
> b) improve the test results tool Ee Peng has posted to include ptest
> processing tools:
>  i) view the logs
>  ii) compare the results
>  iii) highlight timeouts 
>  iv) summarise durations
> c) Split logs into individual ptest runs instead of one block
> d) record the timings for the individual ptest blocks
> e) Add timeout status for tests into the json so we can explicitly
> test for it.

I've now got code to at least help with c/d/e above.

Just to update and document a few things:

a) openssl ptest is broken, it runs but doesn't report the results in
any form that works for us

b) there is a bug in ptest-runner which intermixes stdout and stderr.
Anibal has a patch which should merge shortly

c) lines can get corrupted when logs didn't end with a newline meaning
you'd see DURATION: at the end of a log line. Anibal knows how to fix
and should merge shortly

d) running with 256B memory is too low. This shows as the OOM killer
activating in dmesg. It doesn't show in the ptest runner logs. We
should add checking the logs to the ptest runtime test and increase the
memory for sato-sdk-ptest

e) adding more memory meant the test run took longer but completed more
tests

f) timing data for ptest for qemux86-64 "core-image-sato-sdk-ptest -c
testimage":
5656s for 256MB no SMP
6386s for 1024MB and -smp 4
6803s for 1024MB and no SMP

Cheers,

Richard