[OE-core] [PATCH v2 00/12] Add gnu testsuite execution for OEQA

[Nathan Rossi]

This series adds support to execute the gnu test suites for binutils,
gcc and glibc. With the intention for enabling automated test running of
these test suites within the OEQA framework such that they can be
executed by the Yocto Autobuilder.

The test suites covered need significant resources or build artifacts
such that running them on the target is undesirable which rules out the
use of ptest. Because of this the test suites can be run on the build
host and call out to the target for execution (via ssh or using qemu
usermode).

The following implementation adds a do_check task to gcc-runtime, and
creates recipes for binutils and glibc in order to execute the test
suite on the build host. For binutils the reason for a separate recipe
is due to test suite dependence on target libraries (libgcc, glibc) and
the issues associated with a -cross recipe depending on target
libraries. For glibc the reason for a recipe is due to the dependency
chain (libgcc -> glibc -> libgcc-initial) and the test suite dependence
on libgcc.

Target execution is another important issue specifically since target
execution is slow to extremely slow depending on the physical or
emulated target performance. In order to provide faster execution
performance qemu linux-user is implemented alongside qemu system
emulation (via ssh+nfs). In initial testing qemu linux user vs qemu
system emulation provided performance gains of between 10x to 60x whilst
initially having small pass/fail differences. Further work as covered in
this series reduces the pass/fail differences for gcc/gcc-runtime close
to 0, which can be further reduced by marking known failures with test
result filtering.

However glibc is more strict with its expectations of CPU implementation
as well as syscall behaviour. Additionally glibc expects to be able to
execute OS tools such as 'sh' and 'echo' of which is not easy to provide
in the recipe-sysroot (bindir is not populated into the sysroot). As
such correct test results for glibc rely on execution with qemu system
emulation or on a physical target. This series however still includes
qemu user execution for glibc which can be useful (especially on slow
targets) assuming the known failing tests are excluded.

This series also introduces some OEQA test cases which cover running the
test suites. The test cases are split into binutils, gcc and glibc.
Individual test cases provide execution of the sub-suites of tests
within each target. For example binutils has binutils, gas and ld and
suites which can each be executed independently.

The test cases populate the individual test suite test cases into
testresults similar to ptestresults so that resulttool can analyse them.
Filtering of the failed test cases where expected failures occur is not
included in this series.

The OEQA test cases implement execution on qemu linux user by default.
Subclasses implement qemu system emulation setup and configuration for
running the tests.

Changes in v2:
- Dropped merged patches from series:
  - scripts/lib/resulttool/report.py: Add more result types
  - oeqa/utils/nfs: Add unfs_server function to setup a userspace NFS server
  - binutils: Fix mips patch which changes default emulation
- Added dejagnu recipe to oe-core, imported from meta-oe
- Changes from binutils-cross to binutils-cross-testsuite recipe
- Dropped gold and libiberty test suite execution
- Changes from gcc-cross to only gcc-runtime for gcc/g++ tests
- BUILD_TEST_* to TOOLCHAIN_TEST_*
- New changes to oeqa/core for tagging tests, and patches to fix up
  broken tests in the oeqa/core testsuite
- New changes to oeqa/selftest for '--run-only-tags' and
  '--run-exclude-tags'
- Changes to new selftest tests to populate results like ptestresults
- Mark new selftest tests with 'machine' tag
---

Nathan Rossi (12):
  dejagnu: Add dejagnu for binutils/gcc test suites
  binutils-cross-testsuite: Create recipe for test suite execution
  gcc-runtime: Add do_check task for executing gcc test suites
  gcc-common.inc: Process staging fixme with correct target/native
    sysroot
  glibc-testsuite: Create a recipe to implement glibc test suite
  oeqa/core/tests: Skip test_fail_duplicated_module
  oeqa/core/tests: Fix test_data module tests
  oeqa/core: Rework OETestTag and remove unused OETestFilter
  oeqa/selftest: Add test run filtering based on test tags
  oeqa/selftest/binutils: Create selftest case for binutils test suite
  oeqa/selftest/gcc: Create selftest case for gcc test suite
  oeqa/selftest/glibc: Create selftest case for glibc test suite

 meta/lib/oeqa/core/context.py                 |   4 +-
 meta/lib/oeqa/core/decorator/__init__.py      |  20 ++--
 meta/lib/oeqa/core/decorator/oetag.py         |  27 -----
 meta/lib/oeqa/core/loader.py                  |  61 +++-------
 meta/lib/oeqa/core/tests/cases/data.py        |   2 +-
 meta/lib/oeqa/core/tests/cases/oetag.py       |  21 +++-
 meta/lib/oeqa/core/tests/common.py            |   4 +-
 meta/lib/oeqa/core/tests/test_data.py         |  10 +-
 meta/lib/oeqa/core/tests/test_decorators.py   |  77 +++++++-----
 meta/lib/oeqa/core/tests/test_loader.py       |  26 +---
 meta/lib/oeqa/selftest/cases/binutils.py      |  64 ++++++++++
 meta/lib/oeqa/selftest/cases/gcc.py           | 111 ++++++++++++++++++
 meta/lib/oeqa/selftest/cases/glibc.py         |  97 +++++++++++++++
 meta/lib/oeqa/selftest/context.py             |  11 ++
 .../glibc/glibc-testsuite_2.30.bb             |  51 ++++++++
 .../glibc/glibc/check-test-wrapper            |  71 +++++++++++
 .../binutils/binutils-cross-testsuite_2.32.bb |  83 +++++++++++++
 .../recipes-devtools/dejagnu/dejagnu_1.6.2.bb |  15 +++
 meta/recipes-devtools/gcc/gcc-common.inc      |   2 +-
 meta/recipes-devtools/gcc/gcc-runtime.inc     |  42 +++++++
 meta/recipes-devtools/gcc/gcc-testsuite.inc   | 106 +++++++++++++++++
 21 files changed, 762 insertions(+), 143 deletions(-)
 delete mode 100644 meta/lib/oeqa/core/decorator/oetag.py
 create mode 100644 meta/lib/oeqa/selftest/cases/binutils.py
 create mode 100644 meta/lib/oeqa/selftest/cases/gcc.py
 create mode 100644 meta/lib/oeqa/selftest/cases/glibc.py
 create mode 100644 meta/recipes-core/glibc/glibc-testsuite_2.30.bb
 create mode 100644 meta/recipes-core/glibc/glibc/check-test-wrapper
 create mode 100644 meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb
 create mode 100644 meta/recipes-devtools/dejagnu/dejagnu_1.6.2.bb
 create mode 100644 meta/recipes-devtools/gcc/gcc-testsuite.inc
---
2.23.0.rc1