[OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis

[Richard Purdie]

On Tue, 2019-01-22 at 17:42 +0800, Yeoh Ee Peng wrote:
> OEQA outputs test results into json files and these files were
> archived by Autobuilder during QA releases. Example: each oe-selftest
> run by Autobuilder for different host distro generate a
> testresults.json file.
> 
> These scripts were developed as a test result tools to manage
> these testresults.json file.
> 
> Using the "store" operation, user can store multiple testresults.json
> files as well as the pre-configured directories used to hold those
> files.
> 
> Using the "merge" operation, user can merge multiple testresults.json
> files to a target file.
> 
> Using the "report" operation, user can view the test result summary
> for all available testresults.json files inside a ordinary directory
> or a git repository.
> 
> Using the "regression" operation, user can perform regression
> analysis
> on testresults.json files specified.

Thanks Ee Peng, this version is much improved!

As an experiment I had a local test results file and I was able to run:

$ resultstool regression /tmp/repo/testresults.json /tmp/repo/testresults.json -b sdk_core-image-sato_x86_64_qemumips_20181219111311 -t sdk_core-image-sato_x86_64_qemumips_20181219200052
Successfully loaded base test results from: /tmp/repo/testresults.json
Successfully loaded target test results from: /tmp/repo/testresults.json
Getting base test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219111311
Getting target test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219200052
============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
assimp.BuildAssimp.test_assimp : ERROR -> PASSED
==============================End Regression==============================

I was able to clearly see that my failing test case went from ERROR to
PASSED which is good. The interface and the way the information is
being presented and stored are now the things we need to work on.

What is odd about the current tool/behaviour is that it sometimes
expects files that are stored in a repository and sometimes expects
standalone files. It doesn't feel like the work flow and the way the
user would interact with this is quite correct.

You can see from my test case above that the results file I wanted to
compare were both in the same repo file which I'd already
merged/stored. The tool can do it by passing the file twice and
specifying the IDs but its awkward and not obvious.

I'm still a little unsure how we expect to use the tool and whether the
layout of the files in the git repository is how we're going to
ultimately want to do this.

For example, combining all results into a single json file means we
can't really show useful comparison information directly from "git
diff". Would we be better with a file per results ID for example with a
layout allowing "git diff" to be useful? A directory structure like:

selftest/<HOST_DISTRO>/
runtime/<DISTRO>/<MACHINE>
sdk/<MACHINE>/<SDKMACHINE>
esdk/<MACHINE>/<SDKMACHINE>
build-perf/<HOST_DISTRO>/

for example? I don't have the "right" answer here as I've not
experimented with this but its something we need to think about.

As another example, the regression command can't compare the file from
two different git commits easily, you have to copy one file out the git
repo and then checkout the other version to compare them. This isn't
easy for the user.

In summary, I think the base underlying tools here are heading the
right way but the layout of the git repo and the way people are
expected to use and interact with it needs a little bit more work...

Cheers,

Richard