[OE-core] [RFC PATCH] Add gnu testsuite execution for OEQA

[Nathan Rossi]

On Tue, 9 Jul 2019 at 21:16, Burton, Ross <ross.burton at intel.com> wrote:
>
> This is awesome!
>
> On Sat, 6 Jul 2019 at 12:39, Nathan Rossi <nathan at nathanrossi.com> wrote:
> >         | gcc          | g++          | libstdc++   | binutils    | gas         | ld          | glibc
> > x86-64  |   589/135169 |   457/131913 |     1/13008 |     0/  236 |     0/ 1256 |   166/ 1975 |  1423/ 5991
> > arm     |   469/123905 |   365/128416 |    19/12788 |     0/  191 |     0/  872 |   155/ 1479 |    64/ 5130
> > aarch64 |   460/130904 |   364/128977 |     1/12789 |     0/  190 |     0/  442 |   157/ 1474 |    76/ 5882
> > powerpc | 18336/116624 |  6747/128636 |    33/12996 |     0/  187 |     1/  265 |   157/ 1352 |  1218/ 5110
> > mips64  |  1174/134744 |   401/130195 |    22/12780 |     0/  213 |    43/ 7245 |   803/ 1634 |  2032/ 5847
> > riscv64 |   456/106399 |   376/128427 |     1/12748 |     0/  185 |     0/  257 |   152/ 1062 |    88/ 5847
>
> So what I'm really interested in is what those numbers look like for
> e.g. x86-64 on real hardware: are the test suites always failing a
> bit, or are these indicative of problems we've introduced?

Just to clarify, by real hardware are you interested in the tests
running against a system running with a oe distro + kernel. Or just
running on e.g. the build host?

As for the failing tests, do not put to much emphasis on the values
provided. I am sure there are a number of tests that are failing due
to minor configuration issues that simply need a deeper look at the
test result and output to fix the issue. And the large number of
failures for powerpc are due to qemu user failing due to illegal
instruction.

The goal would be to get the results similar to what other
users/distro get, by reviewing results posted here:
https://gcc.gnu.org/ml/gcc-testresults/.
And for glibc the results from: https://sourceware.org/glibc/wiki/Release/2.29

Thanks,
Nathan