[OE-core] [PATCH 1/4] oeqa/core/runner: write testresult to json files

[richard.purdie at linuxfoundation.org]

Hi Ee Peng,

On Mon, 2018-10-15 at 08:42 +0000, Yeoh, Ee Peng wrote:
> Thank you very much for your inputs!
> I had completed making most of the enhancements following your
> inputs, except the 1 input to put test result logs alongside the test
> result data in the json file. 
> The concerns I have over putting test result logs into test result
> json file:
> 1. Test result logs will be arbitrary long depending on the type of
> failure/error and tetecase. With test result logs adding to the test
> result json file, it will potentially making it hard to read the json
> file, where the test logs will potentially span over multiple lines. 
> 2. By having test logs per each test case in separate file, it will
> allow quick and easy regression of test logs per specific test
> case.  By putting all the test logs into the one testresult json
> file, I am worry that it will make regression on test logs per
> specific test case harder. 
> 
> I hope the concerns above justify keeping test logs separate from
> testresult json file. Please let me know your thoughts and inputs. 

The log data and the test results are connected and belong together.
The intent here is to make files which capture the results information
and direct user readability is a secondary issue, these files are not
intended to be directly read by humans.

If we need a clean readable output, I'm imagining we'd have a simple
processing tool which could for example just filter out the test
results.

Having a single results file also makes it easier for the automated
systems to collect up the results files from different autobuilders and
also makes it easier to potentially merge results together.

One piece I suspect may also be missing is that we may need to record
some extra information into these files too, such as the MACHINE,
DISTRO, hostname of the builder that ran them and the revision of the
codebase (layers) used to run the test. There is probably other
information we need to record in order to make these results useful in
the wider context of the project but I do believe it can and should be
recorded in a single file.

Does that help explain why I'm asking for this?

Cheers,

Richard