[OE-core] oe-selftest developer oriented improvements

[Stoicescu, CorneliuX]

Hello,

During the 1.6 development/testing cycle we introduced and used oe-selftest more and more for QA purposes to identify various issues in the master branch. For those who don't know yet what is oe-selftest, it is a python unit test based testing framework designed to simulate poky external usage patterns. What we basically do is translate test case steps into python classes and methods. This is useful both in reducing QA workload and the coverage of our tests. For more information please visit the oe-selftest wiki page: https://wiki.yoctoproject.org/wiki/Oe-selftest 

What we would like oe-selftest to be useful with as well is helping developers quickly test their patches before submission. With this in mind, we came up with some ideas but we also encourage everyone to contribute with theirs.

Test suites can be created from existing automated tests and accessed using command-line options. For example, using 'oe-selftest --test-suite type=recipe target=man' would test the recipe man and 'oe-selftest --test-suite type=class target=buildhistory' would test the buildhistory class.

1)   Testing recipes updates or new recipes

Even though we cannot test every single scenario or the functionality of a recipe, we could create a test suite that would:
- build the recipe with all major architectures(qemux86, qemux86-64, qemuarm, qemuppc, qemumips)
- rebuild the recipe from sstate(with or without a sstate file for the recipe)
- perform cleaning operations on the recipe(cleansstate)
- force all major tasks on the recipe (bitbake -C <task> <recipe>)
- selectively use each combination of .bbappend files with the recipe; all the combinations should not break the recipe build.
- we could also create mini test suites just for some of these tests like testing only the rebuild from sstate.
(any experience from common recipe build fails can be helpful here)

2)  Testing classes

This is tricky. There is no general testing pattern to test all classes as they provide different functionality. Because of this we should have specific tests for each class or set of classes.  Also when updating a class, the developer could add new tests or update the old ones. The tests used here should be straightforward and well documented.
Also I believe it's desired to have tests(even rudimentary ones) for all oe-core features and this could help push this forward.

3)  Stress testing 

There were scenarios where sometimes a recipe would fail to build due to a race condition. We could create tests that run in a configurable number of build directories at a time with a specific number of threads and parallel make. For example, on a 8 threaded machine we could have 3 builds with 4 threads and parallel make each run in parallel building the same recipe oven and over until it fails. 

I hope many of you have made it to this line and hope to hear your feedback(comments, additions or new ideas). Based on it we will introduce the mentioned functionality or not.

Thank you again for the time!
Regards,
Corneliu
Yocto QA