[Openembedded-architecture] Recipe testing

[Richard Purdie]

I've been asked what our strategy is around recipe testing and we
probably need a bit of discussion around it.

To recap, roughly speaking with our automated QA we currently have:

* oe-selftest and bitbake-selftest
* <image> -c testimage
* <sdk> -c testsdk
* Autobuilder coverage of a matrix of options

These let us test various scenarios but recipe tests aren't well
covered and don't really fit into the above.

There are some scenarios where we might want to make recipe specific
tests:

* As part of the AUH (automatic upgrade helper) - Does this recipe we
just upgraded work?
* General "Is this recipe well written and behave as expected under a
variety of circumstances?"
* As part of devtool - Is the recipe that has been generated good?

There are also probably other cases.

Testing a recipe is a process like oe-selftest, it requires multiple
executions of bitbake against the recipe under test.

We do have an open bug for this, 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=6370

Having written the above, I do think we have a need for such a
component of the test framework. Daniel has been looking at this and
has some code: https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/
commit/?h=distrate/bug_6370_testrecipe&id=9a74de200c9cc4a87b7b21f8be41f
3a532f0e611

I am however a little concerned that we don't want to duplicate too
much of oe-selftest if we can help it.

There are some specific recipe tests would it would be good to validate
in the metadata in general, for example does the recipe work when
installed from sstate, does it work if its individual tasks are rerun?
We've had issues in all these areas in the past and occasionally see
regressions.

In the bug, there are also questions being raised about whether we need
this at all.

What I don't want to see is AUH and devtool for example both developing
"recipe test code" independently of each other, I think we need one
good way of doing this.

So I do see a need for it, equally, I don;t want to see too much code
duplication if we can help it. So where to we go from here?

Cheers,

Richard