[OE-core] [PATCH] test-result-log: testcase management tool to store test result

[Yeoh, Ee Peng]

Hi Richard,

To address the maintainable concern, I had restructured the codes in order to enable oeqa framework to write out a json file directly for testresult.  

Attached were the patches to enble oeqa framework to write testresult into json files, where these files will later be used by the future QA test case management tools (eg. store testresult, test reporting, execute manual test case and write testresult to json). 

This patch include enable oe-selftest to write json testresult. I had tested these patches on our local server where it will write out json files for testresult as expected.

Please let me know if you have any more feedback. Thank you very much!

Best regards,
Yeoh Ee Peng

-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org] 
Sent: Tuesday, September 11, 2018 11:09 PM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH] test-result-log: testcase management tool to store test result

Hi Ee Peng,

I've been having a look at this code and whilst some of it is good, I also have some concerns. With it coming in so late in the cycle, its made it hard to have time to review it and allow time to get it right.
With something as important as this to the way the future QA work is done, we do need to ensure we use the right approach and that its maintainable.

The patches are ok as a first attempt at this. My biggest concern is that its currently parsing log files which we control and generate within out own codebase and that parsing is likely to break. In particular, these lines worry me from the qalogparser:

regex = ".*RESULTS - (?P<case_name>.*) - Testcase .*: (?P<status>PASSED|FAILED|SKIPPED|ERROR|UNKNOWN).*$"
regex = "core-image.*().*Ran.*tests in .*s"
regex = "DEBUG: launchcmd=runqemu*
qemu_list = ['qemuarm', 'qemuarm64', 'qemumips', 'qemumips64', 'qemuppc', 'qemux86', 'qemux86-64']

since here we're hardcoding the list of qemu's we support, we're also only allowing core-image-* and we're relying upon the results format not changing. That makes it hard for anyone to extend/reuse this or to use it with real hardware?

For example the recent oe-selftest parallelisation code did change the output of the tests slightly. I'm not sure if this broke the parsing or not but it is an example of the kind of fragility this code has.

What would probably work better for us is if the oeqa framework wrote out a json file directly containing the information we need in it, then this code would just need to collect up the json files.

I'm also a little concerned at the way unittest discovery is being done, grepping for *.py files as far as I understand it. We should probably use the list options to the various current test pieces? Also, this is something we probably only ever need to do once to seed the QA results store?

Finally, much of the code is using "internal" methods prefixed with "_". I can understand why but it seems the code doesn't have a good well structured public API as a result.

As such there may me a little too much work needed on this to get it in for 2.6 :(

Cheers,

Richard
-------------- next part --------------
An embedded message was scrubbed...
From: "Yeoh, Ee Peng" <ee.peng.yeoh at intel.com>
Subject: [PATCH 2/2] oeqa/selftest/context: enable selftest to write json
 testresult
Date: Wed, 19 Sep 2018 08:31:00 +0000
Size: 3531
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20180919/c6859e85/attachment-0004.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: "Yeoh, Ee Peng" <ee.peng.yeoh at intel.com>
Subject: [PATCH 1/2] oeqa/core/runner: enable oeqa to write testresult into
 json file
Date: Wed, 19 Sep 2018 08:30:59 +0000
Size: 9527
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20180919/c6859e85/attachment-0005.mht>