[OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis

[Richard Purdie]

On Thu, 2019-01-31 at 05:23 +0000, Yeoh, Ee Peng wrote:
> Hi RP,
> 
> I looked into ptest and regression. The existing "resultstool
> regression" can be used to perform regression on ptest, since the
> testresults.json capture ptest status. I had executed regression
> script for the below 2 ptest testresults.json. Attached was the
> regression report for ptest. 
> 
> https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86-64-ptest/testresults.json
> https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86-64-ptest/testresults.json
> 
> The only challenges now was since ptest result set was relatively
> large, it was taking some time for computing the regression. Also
> there was this "ptestresult.rawlogs" testcase that does not contain
> status but the large rawlog. 
> 
> I did an experiment where I run the regression on testresults.json
> with and without the ptest rawlog. It shows the time taken for
> regression was significantly larger when it contain the rawlog. I
> will try to improve the regression time by throw away the rawlog at
> runtime when perform computing. 
> testresults.json with rawlog
> Regression start time: 20190131122805
> Regression end time:   20190131124425
> Time taken: 16 mins 20 sec
> 
> testresults.json without rawlog
> Regression start time: 20190131124512
> Regression end time:   20190131124529
> Time taken: 17 sec

Analysing the rawlog makes no sense so the tool needs to simply ignore
that. 16 minutes is far too long! 

I've just merged some changes which mean there are probably some other
sections it will need to ignore now too since the logs are now being
split out per ptest (section). I've left rawlogs in as its useful for
debugging but once the section splits are working we could remove it.

This adds in timing data so we know how long each ptest took to run (in
seconds), it also adds in exit code and timeout data. These all
complicate the regression analysis but the fact that lttng has been
timing out (for example) has been overlooked until now and shows we
need to analyse these things.

I'm considering whether we should have a command in resulttool which
takes json data and writes it out in a "filesystem" form.

The code in logparser.py already has a rudimentary version of this for
ptest data. It could be extended to write out a X.log for each ptest
based on the split out data and maybe duration and timeout information
in some form too.

The idea behind flat filesystem representations of the data is that a
user can more easily explore or compare them, they also show up well in
git.

Its also worth thinking about how we'll end up using this. testresult
will get called at the end of builds (particularly) release builds and
we'll want it to generate a QA report for the automated test data. The
autobuilder will likely put an http link in the "release build ready"
email to an html like report stored alongside the testresults json
files.

I'm still trying to figure out how to make this all fit together and
allow automated comparisons but the build performance data would also
fit into this (and already has html reports).

Cheers,

Richard