[OE-core] [PATCH 2/5] oeqa/core/runner: write testresult to json files

[Yeoh, Ee Peng]

Hi Richard,

Previous test result data format contain the hierarchy format (eg. testsuite and testcase) for better human reading, as discussed and agreed that the human readability was not needed, thus I will removed the hierarchy format with testsuite and testcase. 

New data format that I am thinking now. 
Attached sample data format
Two section :
1. Test environments: storing all key-value pair information related to test environments
2. Test cases: storing test result and log
Please let me know your inputs and thoughts. 

Thanks,
Yeoh Ee Peng 

-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org] 
Sent: Monday, October 15, 2018 4:54 PM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 2/5] oeqa/core/runner: write testresult to json files

On Mon, 2018-10-15 at 15:24 +0800, Yeoh Ee Peng wrote:
> As part of the solution to replace Testopia to store testresult, OEQA 
> need to output testresult into json file, where these json testresult 
> file will be stored in git repository by the future 
> test-case-management tools.
> 
> Both the testresult (eg. PASSED, FAILED, ERROR) and  the test log (eg. 
> message from unit test assertion) will be created for storing.
> 
> Also the library class inside this patch will be reused by the future 
> test-case-management tools to write json testresult for manual test 
> case executed.
> 
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh at intel.com>
> ---
>  meta/lib/oeqa/core/runner.py | 64 
> +++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 63 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oeqa/core/runner.py 
> b/meta/lib/oeqa/core/runner.py index 56666ee..56e8526 100644
> --- a/meta/lib/oeqa/core/runner.py
> +++ b/meta/lib/oeqa/core/runner.py
> @@ -6,6 +6,8 @@ import time
>  import unittest
>  import logging
>  import re
> +import json
> +import pathlib
>  
>  from unittest import TextTestResult as _TestResult  from unittest 
> import TextTestRunner as _TestRunner @@ -119,8 +121,9 @@ class 
> OETestResult(_TestResult):
>          self.successes.append((test, None))
>          super(OETestResult, self).addSuccess(test)
>  
> -    def logDetails(self):
> +    def logDetails(self, json_file_dir=''):

json_file_dir=None

>          self.tc.logger.info("RESULTS:")
> +        results = {}
>          for case_name in self.tc._registry['cases']:
>              case = self.tc._registry['cases'][case_name]
>  
> @@ -137,6 +140,12 @@ class OETestResult(_TestResult):
>                  t = " (" + "{0:.2f}".format(self.endtime[case.id()] - self.starttime[case.id()]) + "s)"
>  
>              self.tc.logger.info("RESULTS - %s - Testcase %s: %s%s" % 
> (case.id(), oeid, status, t))
> +            results[case.id()] = (status, log)
> +
> +        if len(json_file_dir) > 0:

if json_file_dir:

> +            tresultjsonhelper = OETestResultJSONHelper()
> +            tresultjsonhelper.dump_testresult_file(results, json_file_dir)
> +            tresultjsonhelper.dump_log_files(results, 
> + os.path.join(json_file_dir, 'logs'))
>  
>  class OEListTestsResult(object):
>      def wasSuccessful(self):
> @@ -249,3 +258,56 @@ class OETestRunner(_TestRunner):
>              self._list_tests_module(suite)
>  
>          return OEListTestsResult()
> +
> +class OETestResultJSONHelper(object):
> +
> +    def get_testsuite_from_testcase(self, testcase):
> +        testsuite = testcase[0:testcase.rfind(".")]
> +        return testsuite
> +
> +    def get_testsuite_testcase_dictionary(self, testresults):
> +        testsuite_testcase_dict = {}
> +        for testcase in testresults.keys():
> +            testsuite = self.get_testsuite_from_testcase(testcase)
> +            if testsuite in testsuite_testcase_dict:
> +                testsuite_testcase_dict[testsuite].append(testcase)
> +            else:
> +                testsuite_testcase_dict[testsuite] = [testcase]
> +        return testsuite_testcase_dict
> +
> +    def _create_testcase_testresult_object(self, testcase_list, testresults):
> +        testcase_dict = {}
> +        for testcase in sorted(testcase_list):
> +            testcase_dict[testcase] = {"testresult": testresults[testcase][0]}
> +        return testcase_dict
> +
> +    def _create_json_testsuite_string(self, testresults):
> +        testsuite_testcase = self.get_testsuite_testcase_dictionary(testresults)
> +        testsuite_object = {'testsuite': {}}
> +        testsuite_dict = testsuite_object['testsuite']
> +        for testsuite in sorted(testsuite_testcase.keys()):
> +            testsuite_dict[testsuite] = {'testcase': {}}
> +            testsuite_dict[testsuite]['testcase'] = self._create_testcase_testresult_object(
> +                                                        testsuite_testcase[testsuite],
> +                                                        testresults)
> +        return json.dumps(testsuite_object, sort_keys=True, indent=4)

This patch itself is better but I want to make sure we're using the right data format for this results file. Can you describe the format you're planning to use for the results?

I think what we may need to do is:

a) Combine the results and log data into a single file

b) Stop trying to split things into testsuites here and do that during the results processing. Here, we should just be writing the test case names as they are in the system.

c) Don't rely on the filename to pass information. For example,
logDetails() needs to take a parameter which says whether these are "runtime", "selftest", "manual" or whatever and that information also needs to be stored in the json file.

I think if you describe the results format and/or provide a sample results entry in json format, this will become clearer.



> +    def _write_file(self, write_dir, file_name, file_content):
> +        file_path = os.path.join(write_dir, file_name)
> +        with open(file_path, 'w') as the_file:
> +            the_file.write(file_content)
> +
> +    def dump_testresult_file(self, testresults, write_dir):
> +        if not os.path.exists(write_dir):
> +            pathlib.Path(write_dir).mkdir(parents=True, 
> + exist_ok=True)

The above can be moved into _write_file() rather than being duplicated.
Also, can't we use bb.utils.mkdirhier() like the rest of the code? That can be called unconditionally as it tests internally if it needs to create the directory.

> +        json_testsuite = self._create_json_testsuite_string(testresults)
> +        self._write_file(write_dir, 'testresults.json', 
> + json_testsuite)
> +
> +    def dump_log_files(self, testresults, write_dir):
> +        if not os.path.exists(write_dir):
> +            pathlib.Path(write_dir).mkdir(parents=True, exist_ok=True)
> +        for testcase in testresults.keys():
> +            test_log = testresults[testcase][1]
> +            if test_log is not None:

Just "if test_log:" should be fine.

> +                file_name = '%s.log' % testcase
> +                self._write_file(write_dir, file_name, test_log)


Cheers,

Richard

-------------- next part --------------
A non-text attachment was scrubbed...
Name: testresults.json
Type: application/octet-stream
Size: 1446 bytes
Desc: testresults.json
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20181015/5dbbe094/attachment-0002.obj>