[OE-core] [PATCH 01/14] oeqa/core/loader.py: support the 'auto' keyword

[Richard Purdie]

On Fri, 2018-06-01 at 13:58 +0800, ChenQi wrote:
> On 06/01/2018 01:41 PM, Alexander Kanavin wrote:
> > 2018-06-01 7:42 GMT+03:00 ChenQi <Qi.Chen at windriver.com>:
> > > I agree with you that we need to re-evaluate the settings of
> > > DEFAULT_TEST_SUITES.
> > > 
> > > I plan to do this work from next week. And I plan to put such
> > > changes in
> > > another patchset.
> > > 
> > > Looking at the codes, I can see that the DEFAULT_TEST_SUITES are
> > > set with
> > > consideration of qemu machines and libc.
> > > e.g.
> > > # aarch64 has no graphics
> > > DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
> > > # musl doesn't support systemtap
> > > DEFAULT_TEST_SUITES_remove_libc-musl = "stap"
> > 
> > I suppose we can introduce TEST_SUITES_EXCLUDE for such cases,
> > which
> > would be considered when 'auto' is expanded to a specific list?
> 
> I don't know yet. I guess if test case is written with more 
> sophisticated considerations, we don't need such variable. But I'm
> not sure about it. I'll start to investigate it from next week.

This patchset looks good, thanks for working on it.

The original intent of some of these values was to indicate which tests
were expected to pass for which images. I think in 2.6 we have some
people working on collecting up the test results for a given build,
then having a comparison tool which compares that set of results with a
list of tests we expect to pass.

The desire here is to detect when something that should get tested is
no longer getting tested because the test isn't running.

I'm pretty sure the existing mechanism isn't fit for purpose though.

I also want to change some things like getting rid of testimage_auto
and changing that to be configurable with a variable. I'd noticed it
was totally broken so I doubt anyone can have been using it...

Cheers,

Richard