[Openembedded-architecture] New QA process/tooling

[Richard Purdie]

I wanted to mention we now have a few developments working in our QA
process/tooling workflow. 

-----------------------------------------------------------------------
In summary:

We have buildhistory data comparing -next/mut builds to master:

https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/qemux86/buildhistory.txt

We have a summary of the test results (including ptest):

https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/testresult-report.txt

We have build performance reports:

https://autobuilder.yocto.io/pub/non-release/20190224-3/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_master_20190224030033_47eb3d0.html

All this data is stored into git repos:
http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/
http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/
http://git.yoctoproject.org/cgit.cgi/yocto-testresults/

-----------------------------------------------------------------------
In detail:

Non-release builds publish some data to 
https://autobuilder.yocto.io/pub/non-release/YYYYMMDD-NN

e.g. https://autobuilder.yocto.io/pub/non-release/20190223-15

where NN is an incremented number for each build.

For the qemu machines, buildhistory is generated, stored in a git repo
and buildhistory-diff is run against it. For -next/mut builds this is
against master:

https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/qemux86/buildhistory.txt
https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/qemuarm/buildhistory.txt

(for master it would be against the last master build)

Historical data is saved into a git repo:

http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/

(master-next and mut branches there will be force pushed)

The structure also contains various testresults.json files. resulttool
is used to find them all and import them into 
http://git.yoctoproject.org/cgit.cgi/yocto-testresults/ 

We then run resulttool report to generate:

https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/testresult-report.txt

and we also generate the regression report:

https://autobuilder.yocto.io/pub/non-release/20190223-15/testresults/testresult-regressions-report.txt

although right now this isn't working as well as I'd like (since the
previous build was a-quick, not an a-full and therefore there are missing components.

The test results report includes ptest data and includes data on how
long the ptests take to run and whether they timed out (a T in the time
column).

The build performance measurements are being run every 6 hours on two
performance measurement workers (ubuntu1604 and centos7) triggered from
the autobuilder. These generate the reports mentioned above as well as
pushing the results data into:

http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/

build performance tests can be run against custom branches to test code
changes and then the report will be against the nearest commit in
master.

In a release build, the same data is generated but its published into:

https://autobuilder.yocto.io/pub/releases/

The hope is that these reports and data storage allow us to much more
accurately and easily figure out what was tested and whether/where we
have regressions.

It sounds relatively simple when put like this but there is a lot of
work that has gone into this by many people both past (Markus' work on
build performance or Paul on buildhistory) and present (Ee Peng on
resulttool). It is great to finally see it being pulled together into
something which is part of every build!

There are no doubt bugs and things to improve but believe this to be in
a good state for our 2.7 feature freeze.

Cheers,

Richard