[OE-core] [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES

[richard.purdie at linuxfoundation.org]

On Sun, 2018-11-11 at 13:34 -0800, akuster808 wrote:
> On 11/11/18 1:18 PM, richard.purdie at linuxfoundation.org wrote:
> > On Sun, 2018-11-11 at 12:52 -0800, akuster808 wrote:
> > > On 11/9/18 6:29 AM, Richard Purdie wrote:
> > > > Now the test markup of the development tools tests is complete,
> > > > this
> > > > can be further tweaked to auto run the correct tests.
> > > > 
> > > > Signed-off-by: Richard Purdie <
> > > > richard.purdie at linuxfoundation.org>
> > > > ---
> > > >  meta/classes/testimage.bbclass | 8 ++++----
> > > >  1 file changed, 4 insertions(+), 4 deletions(-)
> > > > 
> > > > diff --git a/meta/classes/testimage.bbclass
> > > > b/meta/classes/testimage.bbclass
> > > > index b00d47441b2..4e9494296e1 100644
> > > > --- a/meta/classes/testimage.bbclass
> > > > +++ b/meta/classes/testimage.bbclass
> > > > @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?=
> > > > "${WORKDIR}/testimage/packages"
> > > >  TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
> > > >  TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
> > > >  
> > > > -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi
> > > > ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam
> > > > stap ldd xorg"
> > > > -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip
> > > > buildgalculator "
> > > > +BASICTESTSUITE = "\
> > > > +    ping date df ssh scp python perl apt opkg gi ptest dnf rpm
> > > > parselogs \
> > > > +    logrotate connman systemd oe_syslog pam stap ldd xorg
> > > > kernelmodule gcc \
> > > > +    buildcpio buildlzip buildgalculator"
> > > 
> > > Again Nak. You are forcing folks to have x11, systemd etc in the
> > > "Basic". 
> > 
> > No, I'm not. If the image doesn't have x11 in it, the test will
> > show
> > "skipped". If the image doesn't have systemd in it, it will show
> > those
> > tests as "skipped". If you just want to test ptest, you can specify
> > that, which is what you had to do before this change anyway *just*
> > to
> > run ptest.
> > 
> > So I'm not really seeing the problem?
> 
> So for meta-security, since it needs python and perl, will it re-run
> those tests every time I run the meta-security ptests?  if it does,
> one may interpret those changes as a negative to the overall testing
> experience.

It depends. It will probably continue to do what it did before which is
to "auto" run tests, unless you're building one of the core-image-XXX
images which had specific tests defined.

If you want to just run the ptest tests, just set:

TEST_SUITES = "ptest"

There are likely some "testing experience" issues but I think these
changes are a key step in improving it, by making the tests much more
automatic without having to define specific test suites for specific
images.

You can still specify specific test subsets just as you've always been
able to.

I agree the output from testing is confusing and I have some ideas on
how to improve that too, but one step (patch) at a time.

Cheers,

Richard