[OE-core] [PATCH 2/5] oeqa/core/runner: write testresult to json files

[Yeoh, Ee Peng]

Hi Richard,

Please ignore the two patches below for testimage.bbclass and testsdk.bbclass, initially testing successfully when running these patch separately but I am finding one issue with method namespace collision after testing both patches together.  I am in progress of resolving it. Sorry for the inconvenience.

http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156928.html
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156929.html

Best regards,
Yeoh Ee Peng 

-----Original Message-----
From: Yeoh, Ee Peng 
Sent: Monday, October 22, 2018 3:23 PM
To: 'richard.purdie at linuxfoundation.org' <richard.purdie at linuxfoundation.org>; openembedded-core at lists.openembedded.org
Subject: RE: [OE-core] [PATCH 2/5] oeqa/core/runner: write testresult to json files

Hi Richard,

I had done making the changes to match your inputs and suggestions.
Below was the patches resubmitted. 

Please let me know if you have any more inputs or questions.

Best regards,
Yeoh Ee Peng 

http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156926.html
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156927.html
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156928.html
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156929.html


-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org] 
Sent: Friday, October 19, 2018 5:56 AM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 2/5] oeqa/core/runner: write testresult to json files

On Thu, 2018-10-18 at 09:47 +0000, Yeoh, Ee Peng wrote:
> Thank you very much for your great feedbacks and inputs!
> I had completed the refactor and resubmitted the patches as below. 
> Please let me know if any more feedbacks and inputs!

Thanks for the revised series, its looking very much improved! I've queued 1/5 onto the autobuilder and its tested ok so that will likely merge.

I still have a few concerns about the data format for 2/5.

@@ -249,3 +256,24 @@ class OETestRunner(_TestRunner):
>              self._list_tests_module(suite)
>  
>          return OEListTestsResult()
> +
> +class OETestResultJSONHelper(object):
> +
> +    def _create_json_testresult_string(self, test_results, test_environments):
> +        testcase_dict = {}
> +        for testcase in sorted(test_results):
> +            testcase_dict[testcase] = {"testresult": test_results[testcase][0], "log": test_results[testcase][1]}
> +        testresult_object = {'testenvironment': test_environments,
> +                             'testcase': testcase_dict}

I'd like to be able to store more than one set of results in a single json file. In order to make that work, the above format will need tweaking slightly. Something like:

testresult_object = {resultid : {'configuration': test_environments,
                                  'results': testcase_dict}}

where resultid would be something like "imagetest-qemux86-core-image-sato-YYMMDDHHMMSS" 
and passed in to the function by the call site. It'd be similar to the other log file the tests generate. We could also perhaps use pid instead of a timestamp in there. A timestamp would be good in the configuration section.

Also, we need to record which revision of the configured layers this test result was with. The get_layers_branch_rev() function in base.bbclass should give an idea of how to get this information using existing functions.

> +        return json.dumps(testresult_object, sort_keys=True, 
> + indent=4)
> +
> +    def _write_file(self, write_dir, file_name, file_content):
> +        file_path = os.path.join(write_dir, file_name)
> +        with open(file_path, 'w') as the_file:
> +            the_file.write(file_content)
> +
> +    def dump_testresult_file(self, test_results, test_environments, write_dir):
> +        if not os.path.exists(write_dir):

No need for this if test here since mkdirhier handles that.

> +            bb.utils.mkdirhier(write_dir)
> +        json_testresult = self._create_json_testresult_string(test_results, test_environments)
> +        self._write_file(write_dir, 'testresults.json', 
> + json_testresult)

As mentioned above, I'd really like if if we could put all the test results into one json file as this would save the autobuilder having to combine them all. That should be straight forward to do but we need to be careful about races.

This means locking a lockfile alongside the results file which we can do with bb.utils.lockfile() call, then writing the results, then calling unlockfile().

With regard to the other patches, for selftest we do need to log which MACHINE/DISTRO was used.

DISTRO and IMAGE_PKGTYPE also needs adding to testimage and SDKMACHINE and IMAGE_PKGTYPE to testsdk.

Cheers,

Richard