[OE-core] [RFC PATCH] Add gnu testsuite execution for OEQA

[Mark Hatle]

On 7/24/19 8:55 AM, richard.purdie at linuxfoundation.org wrote:
> On Wed, 2019-07-24 at 22:30 +1000, Nathan Rossi wrote:
>> So I only hit one major issue with the oe-core qemu* machines I
>> tested on when running under system emulation, specifically with
>> aarch64/qemuarm64.
>>
>> One (or more) test in the "c" suite of gcc causes qemu to crash due
>> to what appears to be an internal qemu issue.
>>
>> qemu outputted:
>> qemu-4.0.0/tcg/tcg.c:3952: tcg_gen_code: Assertion
>> `s->gen_insn_end_off[num_insns] == off' failed.
>>
>> The test that caused this was:
>> gcc.target/aarch64/advsimd-intrinsics/vldX.c   -O2  execution test
> 
> Ok, interesting. I'd hope that issue could be resolved if it started
> causing problems for us, or we could skip that test as a workaround I
> guess.

When we have done similar testing like this, we have skipped these tests.  Much
of the time they are in corner case instructions and not something that really
gives relevant results.

But with that said, we should check if QEMU is already aware of this failure and
if not report it to them.

(The GNU tests also test various instructions that may not be available on all
processors, we've seen this many times on IA32 machines.  They just assume all
instructions are available, even if running on an older CPU.  So this is also
something to be aware of when interpreting the results. -- but QEMU shouldn't
crash!)

--Mark

>> Just an update here, managed to get the results for this. As you will
>> see below, running some of these tests is very slow on qemu system
>> emulation. Though kvm did give a decent boost in performance.
>>
>> Note: qemuarm64 (sys) is missing gcc results because one of the gcc
>> tests crashes qemu.
> 
> The results were wrapped and hard to read so I unwrapped them (added
> here for others):
> 
>                            | g++          | gcc          | glibc       | libatomic   | libgomp     | libitm      | libstdc++-v3
> qemuarm (usr)              |   365/128416 |   469/123905 |    65/ 5130 |     0/   49 |     0/ 2515 |     0/   46 |     23/12782
> qemuarm (sys)              |   365/128416 |   468/123874 |    47/ 5130 |     0/   49 |     0/ 2515 |    18/   46 |     48/12790
> qemux86-64 (usr)           |   457/131913 |   589/135169 |  1423/ 5991 |     0/   54 |     0/ 2522 |     0/   46 |      1/13008
> qemux86-64 (sys)           |   418/131913 |   519/135221 |  1418/ 5991 |     0/   54 |     1/ 2522 |    18/   46 |     51/13010
> qemux86-64 (sys+kvm)       |   418/131913 |   519/135415 |    40/ 5991 |     0/   54 |     1/ 2522 |    18/   46 |     46/13010
> qemuarm64 (usr)            |   364/128977 |   459/130904 |    75/ 5882 |     0/   54 |     0/ 2515 |     0/   46 |      1/12789
> qemuarm64 (sys)            |   364/128977 |              |    43/ 5882 |     0/   54 |     0/ 2515 |    18/   46 |     62/12791
> qemuppc (usr)              |  6747/128636 | 18336/116624 |  1220/ 5110 |     0/   49 |     2/ 2515 |     0/   46 |     33/12996
> qemuppc (sys)              |   383/129056 |   800/119824 |  1188/ 5110 |     0/   49 |     2/ 2515 |    18/   46 |     34/12998
> qemuriscv64 (usr)          |   376/128427 |   460/106399 |    86/ 5847 |     0/   54 |     4/ 2508 |             |      1/12748
> qemuriscv64 (sys)          |   376/128427 |   458/106451 |    53/ 5847 |     0/   54 |     0/ 2508 |             |     52/12750
> 
>                            | g++          | gcc          | glibc       | libatomic   | libgomp     | libitm      | libstdc++-v3
> qemuarm (usr)              |       9m 24s |       15m 3s |     37m 10s |          8s |      6m 52s |          8s |       1h 24s
> qemuarm (sys)              |   3h 58m 30s |  12h 21m 44s |  5h 36m 53s |         55s |     45m 57s |         53s |  12h 16m 11s
> qemux86-64 (usr)           |       8m 22s |      15m 48s |     36m 52s |          8s |       6m 1s |          8s |      34m 42s
> qemux86-64 (sys)           |   5h 38m 27s |  15h 15m 40s |  5h 54m 42s |       1m 8s |     45m 52s |         55s |   3h 26m 11s
> qemux86-64 (sys+kvm)       |      16m 22s |      56m 44s |  2h 29m 45s |         25s |     16m 58s |         21s |      19m 20s
> qemuarm64 (usr)            |       8m 34s |      16m 15s |     44m 25s |          8s |      6m 23s |          8s |      35m 38s
> qemuarm64 (sys)            |    4h 2m 53s |              |   6h 2m 39s |       1m 7s |     44m 47s |         53s |    3h 9m 37s
> qemuppc (usr)              |       6m 54s |      10m 47s |     32m 50s |          6s |      6m 22s |          7s |      34m 25s
> qemuppc (sys)              |   5h 46m 23s |  16h 16m 10s |   4h 10m 6s |      1m 16s |   1h 3m 11s |      1m 12s |   4h 32m 45s
> qemuriscv64 (usr)          |       6m 54s |      10m 23s |     36m 50s |          7s |      9m 38s |             |      33m 13s
> qemuriscv64 (sys)          |   2h 19m 24s |   6h 27m 37s |  4h 23m 43s |         47s |     31m 47s |             |   1h 52m 18s
> 
> This makes very interesting reading, thanks!
> 
> I'm quite amazed how much faster user mode qemu is at running the tests
> against a system kvm qemu. The accuracy of sys, usr and sys+kvm looks
> questionable in different places.
> 
> There isn't a clear answer here although its obvious qemuppc user mode
> emulation is bad. The usermode testing is clearly the winner speed wise
> by a long margin. I would like to understand why though as KVM should
> be reasonable...
> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
>