[OE-core] [PATCH 0/3] RFC: Package testing

[Björn Stenberg]

(An earlier version of this was posted to the yocto list last week:
https://lists.yoctoproject.org/pipermail/yocto/2012-August/011166.html)

This is a first set of patches implementing the package testing concept
I proposed in
https://lists.yoctoproject.org/pipermail/yocto/2012-June/009443.html

In summary, the proposal creates a new package group called -ptest that
contains the test suites included with many source packages, compiled to
run on target.

Packaging the test suites sometimes requires patching makefiles to allow
test tools to be cross-compiled. And it sometimes requires patching
and/or translation of the test output to produce a generic output format
that can be automatically parsed. I have included an example patch for
bash that does just that.

I have also included "ptest-runnner", a simple script to find and run
all installed ptests.

The idea is to eventually upstream these changes, both for the benefit
of the entire ecosystem and of course to reduce the number of patches.

However I won't downplay the effort required to get hundreds of upstream
projects to agree on a common concept for test cross compilation and
result formatting.

I see two main options: Implement first or discuss first.

a) Implement first: We design and implement a good system for yocto/oe
and then, once it has proven itself, engage the larger open source
community with our already tested concept.

Pro: It's easier to discuss an already working system
Con: We'll have to manage patches for all packages until agreement

b) Discuss first: We invite the whole open source community to discuss
and try to agree on how to handle cross-compiled test cases and test
result formatting before we implement anything.

Pro: No increased patch load until agreement
Con: No result until agreement, which could potentially take a long time

My suggestion is a). I'm open for all suggestions that would make the
patch load easier to manage.

Björn Stenberg (3):
  Add -ptest package group
  Enable bash-ptest
  New recipe: ptest-runner

 meta/classes/distutils-common-base.bbclass         |    5 ++-
 meta/classes/image.bbclass                         |    6 ++-
 meta/classes/task.bbclass                          |    2 +-
 meta/conf/bitbake.conf                             |   12 ++++++-
 .../bash/bash-4.2/build-tests.patch                |   34 ++++++++++++++++++++
 meta/recipes-extended/bash/bash-4.2/run-ptest      |    2 +
 .../bash/bash-4.2/test-output.patch                |   19 +++++++++++
 meta/recipes-extended/bash/bash.inc                |   23 +++++++++++++
 meta/recipes-extended/bash/bash_4.2.bb             |    5 ++-
 .../ptest-runner/files/ptest-runner                |   16 +++++++++
 .../ptest-runner/ptest-runner_1.0.bb               |   17 ++++++++++
 11 files changed, 135 insertions(+), 6 deletions(-)
 create mode 100644 meta/recipes-extended/bash/bash-4.2/build-tests.patch
 create mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest
 create mode 100644 meta/recipes-extended/bash/bash-4.2/test-output.patch
 create mode 100644 meta/recipes-support/ptest-runner/files/ptest-runner
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_1.0.bb

-- 
1.7.5.4