[OE-core] [PATCH 0/2 v7] test-case-mgmt

[Yeoh, Ee Peng]

Hi RP,

Thank you very much again for continuously providing your precious feedbacks to me.
Also thank you very much for spending great amount of time to improve this patchset siginificantly. 
 
I did some testing with the latest from resulttool: Update to use gitarchive library function. 
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=b9eecaabe56db5bcafff31e67cdabadc42e2d2e4

I had 2 questions. 
1. For "resulttool regression", currently it was comparing result id set without comprehending the difference in the host distro used to executed the oeselftest. Example: it was matching oeselftest run with fedora28 host distro with oeselftest run with ubuntu18 host distro, is this the expected behavior? 
Match: oeselftest_fedora-28_qemux86-64_20190201181656
       oeselftest_ubuntu-18.04_qemux86-64_20190201175023
Match: oeselftest_fedora-26_qemux86-64_20190131144317
       oeselftest_fedora-26_qemux86-64_20190131144317
Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
       oeselftest_fedora-28_qemux86-64_20190201181656
Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
       oeselftest_opensuse-42.3_qemux86-64_20190126152612

I believe that we shall comprehend the 'HOST_DISTRO' configuration inside the regression_map.  
regression_map = {
-    "oeselftest": ['TEST_TYPE', 'MACHINE'],
+    "oeselftest": ['TEST_TYPE', 'HOST_DISTRO', 'MACHINE'],
     "runtime": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE', 'IMAGE_PKGTYPE', 'DISTRO'],
     "sdk": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE', 'SDKMACHINE'],
     "sdkext": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE', 'SDKMACHINE']
 }

After comprehending this 'HOST_DISTRO', it was able to perform regression for oeselftest with the matching host distro.
Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
       oeselftest_ubuntu-18.04_qemux86-64_20190201175023
Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
       oeselftest_opensuse-42.3_qemux86-64_20190126152612
Match: oeselftest_fedora-26_qemux86-64_20190131144317
       oeselftest_fedora-26_qemux86-64_20190131144317
Match: oeselftest_fedora-28_qemux86-64_20190201181656
       oeselftest_fedora-28_qemux86-64_20190201181656

2. For "resulttool store", I had noticed that it will now generally stored testresults.json in a meaningful file directory structure based on the store_map except oeselftest. oeselftest currently store multiple result id set inside oselftest file directory without comprehend the host distro. 

For example runtime, store testresult.json with the configured store_map. 
├── oeselftest
│   └── testresults.json
├── runtime
│   ├── poky
│   │   ├── qemuarm
│   │   │   ├── core-image-minimal
│   │   │   │   └── testresults.json
│   │   │   ├── core-image-sato
│   │   │   │   └── testresults.json
│   │   │   └── core-image-sato-sdk
│   │   │       └── testresults.json
│   │   ├── qemuarm64
│   │   │   ├── core-image-minimal
│   │   │   │   └── testresults.json
│   │   │   ├── core-image-sato
│   │   │   │   └── testresults.json
│   │   │   └── core-image-sato-sdk
│   │   │       └── testresults.json

I believe that we shall again comprehend the 'HOST_DISTRO' configuration inside the store_map.  
store_map = {
-    "oeselftest": ['TEST_TYPE'],
+    "oeselftest": ['TEST_TYPE','HOST_DISTRO'],
     "runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'],
     "sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
     "sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME']

Doing so, it will store oeselftest in a more useful file directory structure with host distro comprehended. 
└── oeselftest
    ├── fedora-26
    │   └── testresults.json
    ├── fedora-28
    │   └── testresults.json
    ├── opensuse-42.3
    │   └── testresults.json
    └── ubuntu-18.04
        └── testresults.json

Please let me know if you have any question related to above. 

Best regards,
Yeoh Ee Peng 

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie at linuxfoundation.org] 
Sent: Monday, February 18, 2019 6:46 AM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt

On Sun, 2019-02-17 at 17:54 +0000, Richard Purdie wrote:
> > Despite my changes there are things that still need to be done.
> > Essential things which need to happen before this code merges:
> > 
> > * oe-git-archive is importing using the commit/branch of the current
> >   repo, not the data in the results file.

Also now fixed. I put my patches into master-next too.

With this working, I was able to run something along the lines of:

for D in $1/*; do
    resulttool store $D $2 --allow-empty done

on the autobuilder's recent results which lead to the creation of this
repository:

http://git.yoctoproject.org/cgit.cgi/yocto-testresults/


> > * Revisit and redo the way the git branch handling is happening.
> > We 
> >   really want to model how oe-build-perf-report handles git repos 
> > for
> >   comparisons:
> >   - Its able to query data from git repos without changing the 
> > current
> >     working branch, 
> >   - it can search on tag formats to find comparison data

Which means we now need to make the git branch functionality of the report and regression commands compare with the above repo, so we're a step closer to getting thie merged.

Ultimately we'll auto-populate the above repo by having the autobuilder run a "store" command at the end of its runs.

I have a feeling I may have broken the resulttool selftests so that is something else which will need to be fixed before anything merges. Time for me to step away from the keyboard for a bit too.

Cheers,

Richard