[OE-core] [PATCH 0/2 v7] test-case-mgmt

[Richard Purdie]

Hi Ee Peng,

On Mon, 2019-02-18 at 08:09 +0000, Yeoh, Ee Peng wrote: 
> I did some testing with the latest from resulttool: Update to use
> gitarchive library function. 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=b9eecaabe56db5bcafff31e67cdabadc42e2d2e4
> 
> I had 2 questions. 
> 1. For "resulttool regression", currently it was comparing result id
> set without comprehending the difference in the host distro used to
> executed the oeselftest. Example: it was matching oeselftest run with
> fedora28 host distro with oeselftest run with ubuntu18 host distro,
> is this the expected behavior? 
> Match: oeselftest_fedora-28_qemux86-64_20190201181656
>        oeselftest_ubuntu-18.04_qemux86-64_20190201175023
> Match: oeselftest_fedora-26_qemux86-64_20190131144317
>        oeselftest_fedora-26_qemux86-64_20190131144317
> Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
>        oeselftest_fedora-28_qemux86-64_20190201181656
> Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
>        oeselftest_opensuse-42.3_qemux86-64_20190126152612

There were two reasons for this:

a) the results of the selftest should be independent of which
HOST_DISTRO they're run on so they can be compared.

b) some builds only have one oe-selftest (a-quick) and some have four
(a-full). In an a-quick build, the HOST_DISTRO would likely therefore
be different between two builds but we still would like the tool to
compare them.

> 2. For "resulttool store", I had noticed that it will now generally
> stored testresults.json in a meaningful file directory structure
> based on the store_map except oeselftest. oeselftest currently store
> multiple result id set inside oselftest file directory without
> comprehend the host distro. 
> 
> For example runtime, store testresult.json with the configured
> store_map. 
> ├── oeselftest
> │   └── testresults.json
> ├── runtime
> │   ├── poky
> │   │   ├── qemuarm
> │   │   │   ├── core-image-minimal
> │   │   │   │   └── testresults.json
> │   │   │   ├── core-image-sato
> │   │   │   │   └── testresults.json
> │   │   │   └── core-image-sato-sdk
> │   │   │       └── testresults.json
> │   │   ├── qemuarm64
> │   │   │   ├── core-image-minimal
> │   │   │   │   └── testresults.json
> │   │   │   ├── core-image-sato
> │   │   │   │   └── testresults.json
> │   │   │   └── core-image-sato-sdk
> │   │   │       └── testresults.json
> 
> I believe that we shall again comprehend the 'HOST_DISTRO'
> configuration inside the store_map.  
> store_map = {
> -    "oeselftest": ['TEST_TYPE'],
> +    "oeselftest": ['TEST_TYPE','HOST_DISTRO'],
>      "runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'],
>      "sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
>      "sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE',
> 'IMAGE_BASENAME']
> 
> Doing so, it will store oeselftest in a more useful file directory
> structure with host distro comprehended. 
> └── oeselftest
>     ├── fedora-26
>     │   └── testresults.json
>     ├── fedora-28
>     │   └── testresults.json
>     ├── opensuse-42.3
>     │   └── testresults.json
>     └── ubuntu-18.04
>         └── testresults.json

The reasoning is the same as the above, its more useful to allow the
files to be directly compared between different host distros.

Cheers,

Richard