[OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis

[Yeoh, Ee Peng]

Hi RP,

I had executed the runtime/ptest using the latest master with the latest changes to understand the new improvement. 

For now, the resulttool regression will ignore both 'ptestresult.rawlogs' & 'ptestresult.sections' as current regression operation focuses on comparing the "status" differences and it does not need the log as well as the new section information.  By ignoring both 'ptestresult.rawlogs' & 'ptestresult.sections' , the regression time was optimized to seconds instead of minutes for ptest.  

For additional information inside 'ptestresult.sections', do we need similar regression? Any idea which data inside 'ptestresult.sections' will be useful for regression? 

Currently, resulttool regression will only print text based report, if html report was needed, it can be extend by using jinja2 framework.  Do we need html report for this regression? Any requirement for the html report? 

http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278971.html
http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278972.html
http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278973.html

Thanks,
Yeoh Ee Peng 

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie at linuxfoundation.org] 
Sent: Friday, February 1, 2019 7:40 AM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; 'openembedded-core at lists.openembedded.org' <openembedded-core at lists.openembedded.org>
Cc: Eggleton, Paul <paul.eggleton at intel.com>; Burton, Ross <ross.burton at intel.com>
Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis

On Thu, 2019-01-31 at 05:23 +0000, Yeoh, Ee Peng wrote:
> Hi RP,
> 
> I looked into ptest and regression. The existing "resultstool 
> regression" can be used to perform regression on ptest, since the 
> testresults.json capture ptest status. I had executed regression 
> script for the below 2 ptest testresults.json. Attached was the 
> regression report for ptest.
> 
> https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults
> /qemux86-64-ptest/testresults.json
> https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults
> /qemux86-64-ptest/testresults.json
> 
> The only challenges now was since ptest result set was relatively 
> large, it was taking some time for computing the regression. Also 
> there was this "ptestresult.rawlogs" testcase that does not contain 
> status but the large rawlog.
> 
> I did an experiment where I run the regression on testresults.json 
> with and without the ptest rawlog. It shows the time taken for 
> regression was significantly larger when it contain the rawlog. I will 
> try to improve the regression time by throw away the rawlog at runtime 
> when perform computing.
> testresults.json with rawlog
> Regression start time: 20190131122805
> Regression end time:   20190131124425
> Time taken: 16 mins 20 sec
> 
> testresults.json without rawlog
> Regression start time: 20190131124512
> Regression end time:   20190131124529
> Time taken: 17 sec

Analysing the rawlog makes no sense so the tool needs to simply ignore that. 16 minutes is far too long! 

I've just merged some changes which mean there are probably some other sections it will need to ignore now too since the logs are now being split out per ptest (section). I've left rawlogs in as its useful for debugging but once the section splits are working we could remove it.

This adds in timing data so we know how long each ptest took to run (in seconds), it also adds in exit code and timeout data. These all complicate the regression analysis but the fact that lttng has been timing out (for example) has been overlooked until now and shows we need to analyse these things.

I'm considering whether we should have a command in resulttool which takes json data and writes it out in a "filesystem" form.

The code in logparser.py already has a rudimentary version of this for ptest data. It could be extended to write out a X.log for each ptest based on the split out data and maybe duration and timeout information in some form too.

The idea behind flat filesystem representations of the data is that a user can more easily explore or compare them, they also show up well in git.

Its also worth thinking about how we'll end up using this. testresult will get called at the end of builds (particularly) release builds and we'll want it to generate a QA report for the automated test data. The autobuilder will likely put an http link in the "release build ready"
email to an html like report stored alongside the testresults json files.

I'm still trying to figure out how to make this all fit together and allow automated comparisons but the build performance data would also fit into this (and already has html reports).

Cheers,

Richard