[Openembedded-architecture] Recipe testing

[Randy Witt]

On 02/12/2016 04:22 AM, Richard Purdie wrote:
> I've been asked what our strategy is around recipe testing and we
> probably need a bit of discussion around it.
>
> To recap, roughly speaking with our automated QA we currently have:
>
> * oe-selftest and bitbake-selftest
> * <image> -c testimage
> * <sdk> -c testsdk
> * Autobuilder coverage of a matrix of options
>
> These let us test various scenarios but recipe tests aren't well
> covered and don't really fit into the above.
>
> There are some scenarios where we might want to make recipe specific
> tests:
>
> * As part of the AUH (automatic upgrade helper) - Does this recipe we
> just upgraded work?
> * General "Is this recipe well written and behave as expected under a
> variety of circumstances?"
> * As part of devtool - Is the recipe that has been generated good?
>
> There are also probably other cases.
>
> Testing a recipe is a process like oe-selftest, it requires multiple
> executions of bitbake against the recipe under test.
>

Do we want to do some analysis of the recipe as well? For instance to determine 
what variations of PACKAGECONFIG or DISTRO_FEATURES it cares about. Or do we 
only care about the variations for the builds for which we currently run testimage?

> We do have an open bug for this,
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6370
>
> Having written the above, I do think we have a need for such a
> component of the test framework. Daniel has been looking at this and
> has some code: https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/
> commit/?h=distrate/bug_6370_testrecipe&id=9a74de200c9cc4a87b7b21f8be41f
> 3a532f0e611
>
> I am however a little concerned that we don't want to duplicate too
> much of oe-selftest if we can help it.
>
> There are some specific recipe tests would it would be good to validate
> in the metadata in general, for example does the recipe work when
> installed from sstate, does it work if its individual tasks are rerun?
> We've had issues in all these areas in the past and occasionally see
> regressions.
>
> In the bug, there are also questions being raised about whether we need
> this at all.
>
> What I don't want to see is AUH and devtool for example both developing
> "recipe test code" independently of each other, I think we need one
> good way of doing this.
>
> So I do see a need for it, equally, I don;t want to see too much code
> duplication if we can help it. So where to we go from here?
>
> Cheers,
>
> Richard
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>