[OE-core] [PATCH 1/4] oeqa/core/runner: write testresult to json files

[Yeoh, Ee Peng]

Hi Richard,

Thanks for explaining this in great depth, I fully understood them now. 
Currently, the environments information (eg. MACHINE, DISTRO) discovered inside OEQA were used to create the filesystem directories for each testresult json file. 
Example: MACHINE:qemux86, IMAGE: core-image-sato-sdk
└── runtime
    └── qemux86
        └── core-image-sato-sdk
            ├── logs
            ├── ping.json
            └── ssh.json

These filesystem directories that representing the test environments have 2 use cases:

Use case#1: 
During testresult json storing phase, one can input additional environments information and the extra filesystem directories will be created and appended on existing directories, then the overall filesystem directories that represent the entire environments will be store in git repository.  With these environments stored in filesystem, one will be able to view the overall test environments for all test components (eg. runtime, selftest, sdk, manual tests, etc) by walk through the filesystem directories. 

Use case#2:
These filesystem directories that represent the test environment were used during result storing in git repository.  The store program will check if the git repository already have the environments used by the testresult json file, if git repository already have the environments (filesystem directories exists), it will ask user to provide overwrite argument in order to overwrite the testresult that was stored previously.

I agreed that these environments need to be store in the testresult json file as well. Let me work on the adding these environments information into the testresult json file. 

Thanks,
Yeoh Ee Peng 

-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org] 
Sent: Monday, October 15, 2018 5:00 PM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 1/4] oeqa/core/runner: write testresult to json files

Hi Ee Peng,

On Mon, 2018-10-15 at 08:42 +0000, Yeoh, Ee Peng wrote:
> Thank you very much for your inputs!
> I had completed making most of the enhancements following your inputs, 
> except the 1 input to put test result logs alongside the test result 
> data in the json file.
> The concerns I have over putting test result logs into test result 
> json file:
> 1. Test result logs will be arbitrary long depending on the type of 
> failure/error and tetecase. With test result logs adding to the test 
> result json file, it will potentially making it hard to read the json 
> file, where the test logs will potentially span over multiple lines.
> 2. By having test logs per each test case in separate file, it will 
> allow quick and easy regression of test logs per specific test case.  
> By putting all the test logs into the one testresult json file, I am 
> worry that it will make regression on test logs per specific test case 
> harder.
> 
> I hope the concerns above justify keeping test logs separate from 
> testresult json file. Please let me know your thoughts and inputs.

The log data and the test results are connected and belong together.
The intent here is to make files which capture the results information and direct user readability is a secondary issue, these files are not intended to be directly read by humans.

If we need a clean readable output, I'm imagining we'd have a simple processing tool which could for example just filter out the test results.

Having a single results file also makes it easier for the automated systems to collect up the results files from different autobuilders and also makes it easier to potentially merge results together.

One piece I suspect may also be missing is that we may need to record some extra information into these files too, such as the MACHINE, DISTRO, hostname of the builder that ran them and the revision of the codebase (layers) used to run the test. There is probably other information we need to record in order to make these results useful in the wider context of the project but I do believe it can and should be recorded in a single file.

Does that help explain why I'm asking for this?

Cheers,

Richard