[OE-core] [PATCH 2/5] oeqa/core/runner: write testresult to json files

[richard.purdie at linuxfoundation.org]

On Thu, 2018-10-18 at 09:47 +0000, Yeoh, Ee Peng wrote:
> Thank you very much for your great feedbacks and inputs!
> I had completed the refactor and resubmitted the patches as below. 
> Please let me know if any more feedbacks and inputs!

Thanks for the revised series, its looking very much improved! I've
queued 1/5 onto the autobuilder and its tested ok so that will likely
merge.

I still have a few concerns about the data format for 2/5.

@@ -249,3 +256,24 @@ class OETestRunner(_TestRunner):
>              self._list_tests_module(suite)
>  
>          return OEListTestsResult()
> +
> +class OETestResultJSONHelper(object):
> +
> +    def _create_json_testresult_string(self, test_results, test_environments):
> +        testcase_dict = {}
> +        for testcase in sorted(test_results):
> +            testcase_dict[testcase] = {"testresult": test_results[testcase][0], "log": test_results[testcase][1]}
> +        testresult_object = {'testenvironment': test_environments,
> +                             'testcase': testcase_dict}

I'd like to be able to store more than one set of results in a single
json file. In order to make that work, the above format will need
tweaking slightly. Something like:

testresult_object = {resultid : {'configuration': test_environments,
                                  'results': testcase_dict}}

where resultid would be something like "imagetest-qemux86-core-image-sato-YYMMDDHHMMSS" 
and passed in to the function by the call site. It'd be similar to the
other log file the tests generate. We could also perhaps use pid
instead of a timestamp in there. A timestamp would be good in the
configuration section.

Also, we need to record which revision of the configured layers this
test result was with. The get_layers_branch_rev() function in
base.bbclass should give an idea of how to get this information using
existing functions.

> +        return json.dumps(testresult_object, sort_keys=True, indent=4)
> +
> +    def _write_file(self, write_dir, file_name, file_content):
> +        file_path = os.path.join(write_dir, file_name)
> +        with open(file_path, 'w') as the_file:
> +            the_file.write(file_content)
> +
> +    def dump_testresult_file(self, test_results, test_environments, write_dir):
> +        if not os.path.exists(write_dir):

No need for this if test here since mkdirhier handles that.

> +            bb.utils.mkdirhier(write_dir)
> +        json_testresult = self._create_json_testresult_string(test_results, test_environments)
> +        self._write_file(write_dir, 'testresults.json', json_testresult)

As mentioned above, I'd really like if if we could put all the test
results into one json file as this would save the autobuilder having to
combine them all. That should be straight forward to do but we need to
be careful about races.

This means locking a lockfile alongside the results file which we can
do with bb.utils.lockfile() call, then writing the results, then
calling unlockfile().

With regard to the other patches, for selftest we do need to log which
MACHINE/DISTRO was used.

DISTRO and IMAGE_PKGTYPE also needs adding to testimage and SDKMACHINE
and IMAGE_PKGTYPE to testsdk.

Cheers,

Richard