[Openembedded-architecture] Recipe testing

[Aníbal Limón]

Hi,

Comments below,

Cheers,
	alimon

On 02/15/2016 01:19 PM, Esquivel, Benjamin wrote:
> sent again but now subscribed to the openembedded-architecture list
> ________________________________________
> From: Esquivel, Benjamin [benjamin.esquivel at intel.com]
> Sent: Monday, February 15, 2016 1:04 PM
> To: Richard Purdie; openembedded-architecture
> Cc: Istrate, Daniel AlexandruX; Georgescu, Alexandru C; Randle, William C; Eggleton, Paul
> Subject: RE: Recipe testing
> 
> Some comments below
> 
>> -----Original Message-----
>> From: Richard Purdie [mailto:richard.purdie at linuxfoundation.org]
>> Sent: Friday, February 12, 2016 6:22 AM
>> To: openembedded-architecture <openembedded-
>> architecture at lists.openembedded.org>
>> Cc: Istrate, Daniel AlexandruX <daniel.alexandrux.istrate at intel.com>;
>> Esquivel, Benjamin <benjamin.esquivel at intel.com>; Georgescu, Alexandru
>> C <alexandru.c.georgescu at intel.com>; Randle, William C
>> <william.c.randle at intel.com>; Eggleton, Paul <paul.eggleton at intel.com>
>> Subject: Recipe testing
>>
>> I've been asked what our strategy is around recipe testing and we probably
>> need a bit of discussion around it.
>>
>> To recap, roughly speaking with our automated QA we currently have:
>>
>> * oe-selftest and bitbake-selftest
>> * <image> -c testimage
>> * <sdk> -c testsdk
>> * Autobuilder coverage of a matrix of options
>>
>> These let us test various scenarios but recipe tests aren't well covered and
>> don't really fit into the above.
>>
>> There are some scenarios where we might want to make recipe specific
>> tests:
>>
>> * As part of the AUH (automatic upgrade helper) - Does this recipe we just
>> upgraded work?

Yes it only update certain parts of the recipe and then builds the
recipe using bitbake and generate buildhistory-diff.

>> * General "Is this recipe well written and behave as expected under a variety
>> of circumstances?"
>> * As part of devtool - Is the recipe that has been generated good?
>>
>> There are also probably other cases.
>>
>> Testing a recipe is a process like oe-selftest, it requires multiple executions of
>> bitbake against the recipe under test.
>>
>> We do have an open bug for this,
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6370
>>
>> Having written the above, I do think we have a need for such a component of
>> the test framework. Daniel has been looking at this and has some code:
>> https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/
>> commit/?h=distrate/bug_6370_testrecipe&id=9a74de200c9cc4a87b7b21f8be
>> 41f
>> 3a532f0e611
>>
>> I am however a little concerned that we don't want to duplicate too much of
>> oe-selftest if we can help it.

I agree about add the recipe (format/ouput) tests to the oe-selftest,
because there are two things for tests in a recipe the recipe it-self
and the recipe output (package, binaries, etc).

> 
> I understand that Daniel's work extends from oe-selftest. Did you spot some
> duplicity there?
> 
>>
>> There are some specific recipe tests would it would be good to validate in the
>> metadata in general, for example does the recipe work when installed from
>> sstate, does it work if its individual tasks are rerun?
>> We've had issues in all these areas in the past and occasionally see
>> regressions.
>>
>> In the bug, there are also questions being raised about whether we need this
>> at all.
>>
> 
> It took me a couple of days to see that this recipe testing efforts are
> scattered in more than a couple of places and it is easy to see them overlapping
> at some point but also providing unique value in some areas too.
> 
>> What I don't want to see is AUH and devtool for example both developing
>> "recipe test code" independently of each other, I think we need one good
>> way of doing this.

I think we need to use oe-selftest with certain parameters for test
recipe format. With this way the AUH and devtool could call oe-selftest
for test recipe format and build output.

>>
> If recipe testing has enough critical mass to consolidate then we need to
> gather it. Document what is the main purpose of the recipe testing, include
> it into the CI as appropriate and also document what will not be tested by the
> main recipe testing component so the other areas (AUH, devtool) can test the
> gaps that are left on purpose.
>> So I do see a need for it, equally, I don;t want to see too much code
>> duplication if we can help it. So where to we go from here?
>>
> I'm suggesting:
> - creating a recipe test plan that lists objective, areas and tests.
> - receive  the recipe testing that makes sense into the recipe test tool
> - integrate the testing into the CI
> 
>> Cheers,
>>
>> Richard
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-architecture/attachments/20160216/7aa61c5e/attachment.sig>