[Openembedded-architecture] Recipe testing

[Esquivel, Benjamin]

sent again but now subscribed to the openembedded-architecture list
________________________________________
From: Esquivel, Benjamin [benjamin.esquivel at intel.com]
Sent: Monday, February 15, 2016 1:04 PM
To: Richard Purdie; openembedded-architecture
Cc: Istrate, Daniel AlexandruX; Georgescu, Alexandru C; Randle, William C; Eggleton, Paul
Subject: RE: Recipe testing

Some comments below

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie at linuxfoundation.org]
> Sent: Friday, February 12, 2016 6:22 AM
> To: openembedded-architecture <openembedded-
> architecture at lists.openembedded.org>
> Cc: Istrate, Daniel AlexandruX <daniel.alexandrux.istrate at intel.com>;
> Esquivel, Benjamin <benjamin.esquivel at intel.com>; Georgescu, Alexandru
> C <alexandru.c.georgescu at intel.com>; Randle, William C
> <william.c.randle at intel.com>; Eggleton, Paul <paul.eggleton at intel.com>
> Subject: Recipe testing
>
> I've been asked what our strategy is around recipe testing and we probably
> need a bit of discussion around it.
>
> To recap, roughly speaking with our automated QA we currently have:
>
> * oe-selftest and bitbake-selftest
> * <image> -c testimage
> * <sdk> -c testsdk
> * Autobuilder coverage of a matrix of options
>
> These let us test various scenarios but recipe tests aren't well covered and
> don't really fit into the above.
>
> There are some scenarios where we might want to make recipe specific
> tests:
>
> * As part of the AUH (automatic upgrade helper) - Does this recipe we just
> upgraded work?
> * General "Is this recipe well written and behave as expected under a variety
> of circumstances?"
> * As part of devtool - Is the recipe that has been generated good?
>
> There are also probably other cases.
>
> Testing a recipe is a process like oe-selftest, it requires multiple executions of
> bitbake against the recipe under test.
>
> We do have an open bug for this,
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6370
>
> Having written the above, I do think we have a need for such a component of
> the test framework. Daniel has been looking at this and has some code:
> https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/
> commit/?h=distrate/bug_6370_testrecipe&id=9a74de200c9cc4a87b7b21f8be
> 41f
> 3a532f0e611
>
> I am however a little concerned that we don't want to duplicate too much of
> oe-selftest if we can help it.

I understand that Daniel's work extends from oe-selftest. Did you spot some
duplicity there?

>
> There are some specific recipe tests would it would be good to validate in the
> metadata in general, for example does the recipe work when installed from
> sstate, does it work if its individual tasks are rerun?
> We've had issues in all these areas in the past and occasionally see
> regressions.
>
> In the bug, there are also questions being raised about whether we need this
> at all.
>

It took me a couple of days to see that this recipe testing efforts are
scattered in more than a couple of places and it is easy to see them overlapping
at some point but also providing unique value in some areas too.

> What I don't want to see is AUH and devtool for example both developing
> "recipe test code" independently of each other, I think we need one good
> way of doing this.
>
If recipe testing has enough critical mass to consolidate then we need to
gather it. Document what is the main purpose of the recipe testing, include
it into the CI as appropriate and also document what will not be tested by the
main recipe testing component so the other areas (AUH, devtool) can test the
gaps that are left on purpose.
> So I do see a need for it, equally, I don;t want to see too much code
> duplication if we can help it. So where to we go from here?
>
I'm suggesting:
- creating a recipe test plan that lists objective, areas and tests.
- receive  the recipe testing that makes sense into the recipe test tool
- integrate the testing into the CI

> Cheers,
>
> Richard