[OE-core] [PATCH 0/2 v7] test-case-mgmt
Yeoh, Ee Peng
ee.peng.yeoh at intel.com
Mon Feb 18 08:33:54 UTC 2019
Hi RP,
I have a question for "TESTSERIES".
* Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name.
May I know why was "TESTSERIES" was added as one of the key configuration for regression comparison selection inside regression_map?
regression_map = {
"oeselftest": ['TEST_TYPE', 'MACHINE'],
"runtime": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE', 'IMAGE_PKGTYPE', 'DISTRO'],
"sdk": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE', 'SDKMACHINE'],
"sdkext": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE', 'SDKMACHINE']
}
Firstly, from the current yocto-testresults repository, I noticed that "TESTSERIES" was mostly duplicated with "MACHINE", or "MACHINE" & "DISTRO", or "TEST_TYPE" for selftest case.
Secondly, since "TESTSERIES" was created based on directory name from the source directory being used, will this introduce unexpected complication to regression comparison in the future if directory name for the source was changed? If directory name was changed even slightly, example for runtime_core-image-lsb, if the source directory name changed from "qemuarm-lsb" to "qemuarm_lsb", I believe the regression comparison will not able to compare the result id set even though they were having same configurations and they were meant to be compare directly.
Examples:
"runtime_core-image-minimal_qemuarm_20190215014628": {
"configuration": {
"DISTRO": "poky",
"HOST_DISTRO": "ubuntu-18.04",
"IMAGE_BASENAME": "core-image-minimal",
"IMAGE_PKGTYPE": "rpm",
"LAYERS": {
"meta": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-poky": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-yocto-bsp": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
}
},
"MACHINE": "qemuarm",
"STARTTIME": "20190215014628",
"TESTSERIES": "qemuarm",
"TEST_TYPE": "runtime"
},
"runtime_core-image-lsb_qemuarm_20190215014624": {
"configuration": {
"DISTRO": "poky-lsb",
"HOST_DISTRO": "ubuntu-18.04",
"IMAGE_BASENAME": "core-image-lsb",
"IMAGE_PKGTYPE": "rpm",
"LAYERS": {
"meta": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-poky": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-yocto-bsp": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
}
},
"MACHINE": "qemuarm",
"STARTTIME": "20190215014624",
"TESTSERIES": "qemuarm-lsb",
"TEST_TYPE": "runtime"
},
"oeselftest_debian-9_qemux86-64_20190215010815": {
"configuration": {
"HOST_DISTRO": "debian-9",
"HOST_NAME": "debian9-ty-2.yocto.io",
"LAYERS": {
"meta": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-poky": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-selftest": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-yocto-bsp": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
}
},
"MACHINE": "qemux86-64",
"STARTTIME": "20190215010815",
"TESTSERIES": "oe-selftest",
"TEST_TYPE": "oeselftest"
},
Please let me know if you have any question for the above.
Best regards,
Yeoh Ee Peng
-----Original Message-----
From: Richard Purdie [mailto:richard.purdie at linuxfoundation.org]
Sent: Monday, February 18, 2019 12:10 AM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
On Thu, 2019-02-14 at 13:50 +0800, Yeoh Ee Peng wrote:
> v1:
> Face key error from oe-git-archive
> Undesirable behavior when storing to multiple git branch
>
> v2:
> Include fix for oe-git-archive
> Include fix for store result to multiple git branch
> Improve git commit message
>
> v3:
> Enhance fix for oe-git-archive by using exception catch to
> improve code readability and easy to understand
>
> v4:
> Add new features, merge result files & regression analysis
> Add selftest to merge, store, report and regression functionalities
> Revise codebase for pythonic
>
> v5:
> Add required files for selftest testing store
>
> v6:
> Add regression for directory and git repository
> Enable regression pairing base set to multiple target sets
> Revise selftest testing for regression
>
> v7:
> Optimize regression computation for ptest results
> Rename entry point script to resulttool
>
> Mazliana (1):
> scripts/resulttool: enable manual execution and result creation
>
> Yeoh Ee Peng (1):
> resulttool: enable merge, store, report and regression analysis
Hi Ee Peng,
Thanks for working on this, it does get better each iteration. I've been struggling a little to explain what we need to do to finish this off. Firstly I wanted to give some feedback on some general python
tips:
a) We can't use subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. This lead to failures like:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/242
We can use check_call or other functions instead.
b) I'd not recommend using "file" as a variable name in python as its a keyword, similarly "dict" (in resultutils.py).
c) get_dict_value() is something we can probably avoid needing if we use the .get() methods of dicts (you can specify a value to return if a value isn't present).
I started to experiment with the tool to try and get it to follow the workflow we need with the autobuilder QA process. Right now I'm heavily focusing on what we need it to do to generate reports from the autobuilder, to the extent that I'm ignoring most other workflows.
The reason for this is that I want to get it merged and use this to run
2.7 M3 testing on the autobuilder. The other workflows can be added if/as/when we find we have need of them.
I ended up making a few changes to alter the tool to do the things I think we need it to and to improve its output/usability. I'll send out a separate patch with my changes so far. I've tried to summarise some of the reasoning here:
* Rename resultsutils -> resultutils to match the resultstool -> resulttool rename
* Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name.
* When we don't have failed test cases, print something saying so instead of an empty table
* Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id)
* Improve/simplify the max string length handling
* Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user
* Removed the confusing header in the regression report
* Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output
* Try harder to "pair" up matching configurations to reduce noise in the regressions report
* Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils
* Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge
* Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter
* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings
* Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes
* Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults)
Despite my changes there are things that still need to be done.
Essential things which need to happen before this code merges:
* oe-git-archive is importing using the commit/branch of the current
repo, not the data in the results file.
* Fix the -t option to merge command
* Audit the command option help
* Revisit and redo the way the git branch handling is happening. We
really want to model how oe-build-perf-report handles git repos for
comparisons:
- Its able to query data from git repos without changing the current
working branch,
- it can search on tag formats to find comparison data
* Add ptest summary to the report command
Things which may be "nice to have" which can come in the future:
* Make the percentage vs. count in the report a commandline option?
(not sure but I wondered if that would be better)
* Add ptest sub-command to extract log data
* Generate HTML report
* Generate graphical ptest result charts
I'd be interested in your feedback on my changes and hope you agree with them! I'll continue to work on some of the above items as I'd like to get this merged sooner than later. If you're going to work on any of them let me know first. I'll try and keep
http://git.yoctoproject.org/clean/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222
up to date with my changes.
Cheers,
Richard
More information about the Openembedded-core
mailing list