[OE-core] [PATCH 0/2 v7] test-case-mgmt

[Yeoh, Ee Peng]

Hi Richard,

Thank you for sharing on the selftest comparison consideration! 

I agreed with you that in the high level, selftest should be independent of which HOST_DISTRO, it shall compared 2 selftest even when the host distro are different. 

But in the case that the build have multiple set of selftest each with slightly different environments (eg. host distro), in that case, will it better to compare selftest more closely if possible with same host distro used? 

Cheers,
Ee Peng 

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie at linuxfoundation.org] 
Sent: Monday, February 18, 2019 5:08 PM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt

Hi Ee Peng,

On Mon, 2019-02-18 at 08:09 +0000, Yeoh, Ee Peng wrote: 
> I did some testing with the latest from resulttool: Update to use 
> gitarchive library function.
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2
> 22&id=b9eecaabe56db5bcafff31e67cdabadc42e2d2e4
> 
> I had 2 questions. 
> 1. For "resulttool regression", currently it was comparing result id 
> set without comprehending the difference in the host distro used to 
> executed the oeselftest. Example: it was matching oeselftest run with
> fedora28 host distro with oeselftest run with ubuntu18 host distro, is 
> this the expected behavior?
> Match: oeselftest_fedora-28_qemux86-64_20190201181656
>        oeselftest_ubuntu-18.04_qemux86-64_20190201175023
> Match: oeselftest_fedora-26_qemux86-64_20190131144317
>        oeselftest_fedora-26_qemux86-64_20190131144317
> Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
>        oeselftest_fedora-28_qemux86-64_20190201181656
> Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
>        oeselftest_opensuse-42.3_qemux86-64_20190126152612

There were two reasons for this:

a) the results of the selftest should be independent of which HOST_DISTRO they're run on so they can be compared.

b) some builds only have one oe-selftest (a-quick) and some have four (a-full). In an a-quick build, the HOST_DISTRO would likely therefore be different between two builds but we still would like the tool to compare them.

> 2. For "resulttool store", I had noticed that it will now generally 
> stored testresults.json in a meaningful file directory structure based 
> on the store_map except oeselftest. oeselftest currently store 
> multiple result id set inside oselftest file directory without 
> comprehend the host distro.
> 
> For example runtime, store testresult.json with the configured 
> store_map.
> ├── oeselftest
> │   └── testresults.json
> ├── runtime
> │   ├── poky
> │   │   ├── qemuarm
> │   │   │   ├── core-image-minimal
> │   │   │   │   └── testresults.json
> │   │   │   ├── core-image-sato
> │   │   │   │   └── testresults.json
> │   │   │   └── core-image-sato-sdk
> │   │   │       └── testresults.json
> │   │   ├── qemuarm64
> │   │   │   ├── core-image-minimal
> │   │   │   │   └── testresults.json
> │   │   │   ├── core-image-sato
> │   │   │   │   └── testresults.json
> │   │   │   └── core-image-sato-sdk
> │   │   │       └── testresults.json
> 
> I believe that we shall again comprehend the 'HOST_DISTRO'
> configuration inside the store_map.  
> store_map = {
> -    "oeselftest": ['TEST_TYPE'],
> +    "oeselftest": ['TEST_TYPE','HOST_DISTRO'],
>      "runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'],
>      "sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
>      "sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 
> 'IMAGE_BASENAME']
> 
> Doing so, it will store oeselftest in a more useful file directory 
> structure with host distro comprehended.
> └── oeselftest
>     ├── fedora-26
>     │   └── testresults.json
>     ├── fedora-28
>     │   └── testresults.json
>     ├── opensuse-42.3
>     │   └── testresults.json
>     └── ubuntu-18.04
>         └── testresults.json

The reasoning is the same as the above, its more useful to allow the files to be directly compared between different host distros.

Cheers,

Richard