[OE-core] [PATCH 1/4] oeqa/core/runner: write testresult to json files
Yeoh, Ee Peng
ee.peng.yeoh at intel.com
Tue Oct 23 06:39:13 UTC 2018
Hi Richard,
I submitted the revised patches below. Sorry for the missing "version #" in the patch title. After this, I will add the "version #" into the patch title.
Please let me know if any question or inputs. Thank you very much for your attention & sharing!
Best regards,
Yeoh Ee Peng
A) oeqa/core/runner:
- removed no longer used function
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156990.html
B) oeqa/selftest/context:
- concatenate list of element from DISTRO to single element, replace empty character
Eg. 'HOST_DISTRO': ('-'.join(platform.linux_distribution())).replace(' ', '-')
- change result_id to separator character from "-" to "_"
Eg. '%s_%s_%s' % (configuration['TEST_TYPE'], configuration['HOST_DISTRO'], configuration['MACHINE'])
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156991.html
C) testimage.bbclass
- removed no longer used "configuration" argument inside function
Eg. def _get_testimage_json_result_dir(d):
- concatenate list of element from DISTRO to single element, replace empty character
- change result_id to separator character from "-" to "_"
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156992.html
D) testsdk.bbclass
- removed no longer used "configuration" argument inside function
Eg. def _get_testimage_json_result_dir(d):
- concatenate list of element from DISTRO to single element, replace empty character
- change result_id to separator character from "-" to "_"
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156993.html
-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org]
Sent: Tuesday, October 23, 2018 6:54 AM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 1/4] oeqa/core/runner: write testresult to json files
On Mon, 2018-10-22 at 18:34 +0800, Yeoh Ee Peng wrote:
> As part of the solution to replace Testopia to store testresult, OEQA
> need to output testresult into single json file, where json testresult
> file will be stored in git repository by the future
> test-case-management tools.
>
> The json testresult file will store more than one set of results,
> where each set of results was uniquely identified by the result_id.
> The result_id would be like "runtime-qemux86-core-image-sato", where
> it was a runtime test with target machine equal to qemux86 and running
> on core-image-sato image. The json testresult file will only store the
> latest test content for a given result_id. The json testresult file
> contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
> result (eg. PASSED, FAILED, ERROR), test log, and result_id.
>
> Based on the destination json testresult file directory provided, it
> could have multiple instances of bitbake trying to write json
> testresult to a single testresult file, using locking a lockfile
> alongside the results file directory to prevent races.
>
> Also the library class inside this patch will be reused by the future
> test-case-management tools to write json testresult for manual test
> case executed.
>
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh at intel.com>
> ---
> meta/lib/oeqa/core/runner.py | 39
> ++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 38 insertions(+), 1 deletion(-)
>
> diff --git a/meta/lib/oeqa/core/runner.py
> b/meta/lib/oeqa/core/runner.py index f1dd080..2243a10 100644
> --- a/meta/lib/oeqa/core/runner.py
> +++ b/meta/lib/oeqa/core/runner.py
> @@ -6,6 +6,7 @@ import time
> import unittest
> import logging
> import re
> +import json
>
> from unittest import TextTestResult as _TestResult from unittest
> import TextTestRunner as _TestRunner @@ -119,8 +120,9 @@ class
> OETestResult(_TestResult):
> self.successes.append((test, None))
> super(OETestResult, self).addSuccess(test)
>
> - def logDetails(self):
> + def logDetails(self, json_file_dir=None, configuration=None, result_id=None):
> self.tc.logger.info("RESULTS:")
> + result = {}
> for case_name in self.tc._registry['cases']:
> case = self.tc._registry['cases'][case_name]
>
> @@ -137,6 +139,11 @@ class OETestResult(_TestResult):
> t = " (" + "{0:.2f}".format(self.endtime[case.id()] - self.starttime[case.id()]) + "s)"
>
> self.tc.logger.info("RESULTS - %s - Testcase %s: %s%s" %
> (case.id(), oeid, status, t))
> + result[case.id()] = {'status': status, 'log': log}
> +
> + if json_file_dir:
> + tresultjsonhelper = OETestResultJSONHelper()
> + tresultjsonhelper.dump_testresult_file(result_id, result,
> + configuration, json_file_dir)
>
> class OEListTestsResult(object):
> def wasSuccessful(self):
> @@ -249,3 +256,33 @@ class OETestRunner(_TestRunner):
> self._list_tests_module(suite)
>
> return OEListTestsResult()
> +
> +class OETestResultJSONHelper(object):
> +
> + testresult_filename = 'testresults.json'
> +
> + def _get_existing_testresults_if_available(self, write_dir):
> + testresults = {}
> + file = os.path.join(write_dir, self.testresult_filename)
> + if os.path.exists(file):
> + with open(file, "r") as f:
> + testresults = json.load(f)
> + return testresults
> +
> + def _create_json_testresults_string(self, test_results, result_id, test_result, configuration):
> + test_results[result_id] = {'configuration': configuration, 'result': test_result}
> + return json.dumps(test_results, sort_keys=True, indent=4)
I think the above function can be removed as its no longer used anywhere.
I've queued these patches for testing on the main autobuilder as I think we're close and wanted to check it doesn't show issues. I still wonder if we couldn't write out the files to a specific directory more reliably but I think I'd need to look at the code for longer to come up with any proposal. Right now I guess we can at least configure the autobuilder to put the files in a common location (so it can collect them up).
Cheers,
Richard
> + def _write_file(self, write_dir, file_name, file_content):
> + file_path = os.path.join(write_dir, file_name)
> + with open(file_path, 'w') as the_file:
> + the_file.write(file_content)
> +
> + def dump_testresult_file(self, result_id, test_result, configuration, write_dir):
> + bb.utils.mkdirhier(write_dir)
> + lf = bb.utils.lockfile(os.path.join(write_dir, 'jsontestresult.lock'))
> + test_results = self._get_existing_testresults_if_available(write_dir)
> + test_results[result_id] = {'configuration': configuration, 'result': test_result}
> + json_testresults = json.dumps(test_results, sort_keys=True, indent=4)
> + self._write_file(write_dir, self.testresult_filename, json_testresults)
> + bb.utils.unlockfile(lf)
> --
> 2.7.4
>
More information about the Openembedded-core
mailing list