[Openembedded-architecture] OEQA Framework Refactor & Improvements

[Aníbal Limón]

After some time working in the Open-embedded project, i noticed that we
don't have a unified manner to create tests for different components in
the project. Currently we have duplicated test code around the oe-core
repository in our 2 major areas of testing one about what we produce
Runtime (Image), SDK and Extensible SDK and the other related to the
build system itself, Selftest.

This RFC has the purpose a new tool called oe-test, the oe-test
in conjunction with a new set of classes will abstract/standardize the
Test component implementations, this doesn't mean that the current
workflows for testing testimage, testsdk are supported.

Before enter in details, next to define the meaning of some words:

Test component: An area of testing in the Project, for example: Runtime,
SDK, eSDK, Selftest.
Test data: Data associated with some Test component, currently we use
bitbake datastore as a Test data input.
Test context: A context of what tests needs to be run and how to do it,
additionally provides access to the Test data and could have custom
methods, attrs.

The new tool oe-test has the ability to scan the code base for Test
components and provide a unified way to run test cases. Internally it
scans folders inside oeqa module in order to find specific classes that
implements a Test component.

The main design consideration was to implement the needed functionality
in top of Python unittest framework following it a core Test component
was create as described next:

- oeqa/core/runner.py: Provides OETestResult and OETestRunner base
classes extending the unittest ones, this class has the support for
export results to different formats currently RAW and XML.
- oeqa/core/loader.py: Provides OETestLoader extending unittest ones,
also implements unified way for decorator support and for filtering test
cases.
- oeqa/core/case.py: Provides OETestCase base class extending
unittest.TestCase and provides access to the Test data (td), Test context
and Logger functionality.
- oeqa/core/decorator: Provides OETestDecorator a new class to implement
decorators for Test cases.
- oeqa/core/context: Provides OETestContext a HIGH level API for
loadTests and runTests of certain Test component and
OETestContextExecutor a base class for enable oe-test to discover/use
the Test component.

In terms of how to implement a new Test component, the developer only
needs to extend OETestContext and OETestContextExecutor with custom
functionality.

The current functionality was take into account and at this time we are
in the step of migrating the current suites to Test components, the core
module code could be found at [1] and for review the SDK migration at
[2] this gives you a clear vision on how it will work.

Also we have a README [3] for the core module explaining how to run the
core example test cases and how to test the OEQA framework code.

Next steps,

- Migrate the runtime testing, testimage class and qemu runner .
- Decouple test export functionality embedded into runtime testing.
- Migrate Selftest and split the suite into small pieces.
- Add support for Test coverage, at this time only available in Selftest.


This is an invasive set of changes and comes with some risks like lost
of functionality and broke the system in the way that couldn't execute
automated testing to address this kind of issues a local Autobuilder
cluster was used for test the current change set.

I sent the patches to the oe-core mailing list for review [4].

Cheers,
	alimon


[1]
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/oeqa_core
[2]
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/oeqa_sdk_migration
[3]
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/core/README?h=alimon/oeqa_core
[4]
http://lists.openembedded.org/pipermail/openembedded-core/2016-December/129725.html

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-architecture/attachments/20161206/8783bd7f/attachment-0002.sig>