[OE-core] oe-selftest developer oriented improvements

[Paul Barker]

On 23 April 2014 10:26, Stoicescu, CorneliuX
<corneliux.stoicescu at intel.com> wrote:
> Hello,
>
> During the 1.6 development/testing cycle we introduced and used oe-selftest more and more for QA purposes to identify various issues in the master branch. For those who don't know yet what is oe-selftest, it is a python unit test based testing framework designed to simulate poky external usage patterns. What we basically do is translate test case steps into python classes and methods. This is useful both in reducing QA workload and the coverage of our tests. For more information please visit the oe-selftest wiki page: https://wiki.yoctoproject.org/wiki/Oe-selftest
>
> What we would like oe-selftest to be useful with as well is helping developers quickly test their patches before submission. With this in mind, we came up with some ideas but we also encourage everyone to contribute with theirs.
>
> Test suites can be created from existing automated tests and accessed using command-line options. For example, using 'oe-selftest --test-suite type=recipe target=man' would test the recipe man and 'oe-selftest --test-suite type=class target=buildhistory' would test the buildhistory class.
>
> 1)   Testing recipes updates or new recipes
>
> Even though we cannot test every single scenario or the functionality of a recipe, we could create a test suite that would:
> - build the recipe with all major architectures(qemux86, qemux86-64, qemuarm, qemuppc, qemumips)
> - rebuild the recipe from sstate(with or without a sstate file for the recipe)
> - perform cleaning operations on the recipe(cleansstate)
> - force all major tasks on the recipe (bitbake -C <task> <recipe>)
> - selectively use each combination of .bbappend files with the recipe; all the combinations should not break the recipe build.
> - we could also create mini test suites just for some of these tests like testing only the rebuild from sstate.
> (any experience from common recipe build fails can be helpful here)
>

I'd really like to see this, it could cut down on cases where
something works for one developer but then hits an edge case when sent
to the autobuilder.

I can also see 'type=package' being useful, to check that a generated
package installs cleanly in an image. For example, if package x
contains a file which clashes with something already installed on
core-image-minimal, 'bitbake x' won't show this but 'bitbake
core-image-minimal' with 'IMAGE_INSTALL += x' set will do.

Taking that a step further, that image could then be booted in qemu
and the relevant ptest suite ran. I'm not sure how that would be
automated myself or how the ptest log and results would be pulled off
the qemu image but if it is possible I think it would be a very useful
option.

I also think it would be helpful to have options for a 'quick' test
and a 'full' test. For some of my projects I have recipes which aren't
pushed upstream to oe-core, meta-oe, etc and have a fairly restricted
use-case. A quick test that just targets the current value of MACHINE
and maybe skips some tests which would be expected to take a longer
time could be very helpful.

Thanks,

-- 
Paul Barker

Email: paul at paulbarker.me.uk
http://www.paulbarker.me.uk