[OE-core] [PATCH 0/2 v7] test-case-mgmt

[Richard Purdie]

On Mon, 2019-02-18 at 08:33 +0000, Yeoh, Ee Peng wrote:
> Hi RP,
> 
> I have a question for "TESTSERIES".
> * Formalised the handling of "file_name" to "TESTSERIES" which the
> code will now add into the json configuration data if its not
> present, based on the directory name.
> 
> May I know why was "TESTSERIES" was added as one of the key
> configuration for regression comparison selection inside
> regression_map? 
> regression_map = {
>     "oeselftest": ['TEST_TYPE', 'MACHINE'],
>     "runtime": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME',
> 'MACHINE', 'IMAGE_PKGTYPE', 'DISTRO'],
>     "sdk": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE',
> 'SDKMACHINE'],
>     "sdkext": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME',
> 'MACHINE', 'SDKMACHINE']
> }
> 
> Firstly, from the current yocto-testresults repository, I noticed
> that "TESTSERIES" was mostly duplicated with "MACHINE", or "MACHINE"
> & "DISTRO", or "TEST_TYPE" for selftest case.

The store_map doesn't use TESTSERIES as a key since I didn't think it
was needed for the git file layout. Particuarly for the runtime tests,
it does mean we end up with a larger number of results under the
qemux86* files in particular.

When performing regression analysis it seemed like a useful way to know
which results to compare and lowered the number of inexact matches we
had to make.

> Secondly, since "TESTSERIES" was created based on directory name from
> the source directory being used, will this introduce unexpected
> complication to regression comparison in the future if directory name
> for the source was changed? If directory name was changed even
> slightly, example for runtime_core-image-lsb, if the source directory
> name changed from "qemuarm-lsb" to "qemuarm_lsb", I believe the
> regression comparison will not able to compare the result id set even
> though they were having same configurations and they were meant to be
> compare directly. 

For Yocto QA usage, the directory names are going to come directly from
the autobuilder target names so they should be consistent. I'd be fine
with adding a parameter to control it, right now it does add useful
information we need for the autobuilder though as it makes the results
more accurate (and hints to the user which autobuilder target the
change came from too).

Cheers,

Richard