[Openembedded-architecture] A journey into parallelisation of oe-selftest

[Richard Purdie]

I thought I'd write down my findings as I looked into parallelising oe-
selftest, both so there is a record of them somewhere I can point
people at if they have questions and so that others may be able to
build from this in future.

There are basically three ways we could parallelise it:

a) High level wrapper which lists the modules, then runs each module as
"oe-selftest -r X" in its own process. Leo did this using GNU parallel.
On the plus side its simple, on the downside the logging and results
reporting is poor.

http://lists.openembedded.org/pipermail/openembedded-core/2017-December/145743.html

b) Use threads. Anibal tried this and whilst it works to a point and we
have some code in oeqa for it, it was starting to need special handling
in many places. It was found to be problematic for some tests which had
to run in the main thread and doesn't work with things like tinfoil or
potentially memory resident bitbake. The more we went down that rabbit
hole, the more I worried it was getting too complex and wasn't scaling
well.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/oe_selftest_threaded

c) Use multiple processes but connect them to a central unittest
instance. There is prior art for this "upstream" in python-testtools
and its conncurrenttestsuite. This allows an existing unittest setup to
be retrofitted with multiple process execution. Sounds great!

Others have tried a/b and we'd found issues so I decided to look at c
which we hadn't tried.

One key piece of parallelising this is that the different tests need to
run in different build directories. This means we need a mechanism to
copy the default build config into a new build directory. There are
some implementation decisions to be made here. We could have a
directory for every test however there is a lot of setup that goes into
these directories with extraction of things from sstate. In summary
I've chosen to have a build directory for each process and each shares
a common DL_DIR and SSTATE_DIR. The build directory is reused for each
test in a given process.

One factor in making this decision was realising how hard the machine
gets hammered when we initially launch the tests and 15 build
directories all parse the metadata in parallel. Parsing hits 100% CPU
usage with one build directory so you can imagine what 15 builds does.
We can mitigate the problem slightly by copying in the build/cache
directory which contains the codeparser cache. Not all the cache will
be reused due to the path differences but a large chunk can. The main
cache directory is pointless to copy as the path/config differences
mean it would reparse anyway. Having a builddir per process at least
means the cache can be reused there rather than having a reparse per
test.

Looking more closely at concurrenytestsuite (cts), it has advantages in
that it wraps around standard python unittest but this also brings some
drawbacks as some of the games it plays to ensure that compatibility is
rather ugly. There are alternative "streaming" variants in testtools
but those mean moving off standard unittest so I've avoided them and
decided to see how far we can get with cts.

One big win implementation detail wise is that when a parallisation
option isn't passed, the oeqa code can behave as it does today and
doesn't need to import testtools (or subunit which it depends upon).
This means we can have these as optional OE dependencies which you only
need if you specify the -j option. I was able to implement most of what
we need in concurrencytest.py without many changes to the core. Some of
that "retrofitting" is ugly and some needs to move into the core as a
first class citizen but its good for testing and a sign the abstraction
is good.

The way cts works, it has to create a dummy "startTest" event later
once the result is available. It stores its own internal timestamps
about the test duration and accessing them is basically a pain. Its
useful to mark up the test results with start/end times so that we can
debug which tests are taking the most time.

Initially when I implemented this, I was shocked to see oe-selftest
still taking several hours despite the signs being good that it was
running in parallel.

I've since discovered that one of the systemd_boot tests took around 5
hours and the rest of the tests finished within about an hour (the
sstate cache was hot). I strongly suspect this is due to its use "-c
cleansstate" which on the autobuilder with its huge sstate store on NFS
is very very slow. There is a patch pending to remove that as tests
should never be deleting from sstate anyway.

The "scheduling" algorithm used by the tests is also not ideal, it just
 puts things into N buckets, then passes the list of tests to the
subprocess. This means if one process burns through its list quickly,
it will just finish. The argument is that the test runtimes should
average out in general and to an extent they do.

I ended up choosing to schedule things based on groups at the test
'class' level (tests are of the form module.class.testname) since this
let us split things up yet keep similar things in the same build
directory for efficiency. This will likely mean we need to tweak some
of the test groupings, in particular I have a patch which splits
devtool up since it was taking by far the longest of the test modules
to complete. Wic will likely need splitting into two but that meant
fixing its hardcoded racy reference to /var/tmp/wic-selftest.

I found some tests are making changes to metadata as part of the tests.
For this reason meta-selftest is copied into each build directory. I
believe there may still be issues in devtool in particular with changes
to the meta/ directory being temporarily made.

There are a raft of other tweaks I've ended up making such as forcing
all the tests to run in "buffer" mode to capture stdout/stderr and only
show that upon failure and fixes for various races these changes are
highlighting.

I also found subunit assumes sys.stdout.buffer exists which it doesn't
for unittest's buffer mode which uses io.StringIO, I ended up hacking
around that.

My main issues now are how to neatly log to the console, the timings
output I've developed for debugging is noisy but also very effective
for debugging and I'm considering leaving it on all the time.

An exmaple 57 minute selftest run with a hot sstate cache with the
confusing output:

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/1189/steps/Running%20oe-selftest/logs/stdio

In builds in parallel with other things, it seems this time rises to
around 2 hours which is still much better than it used to be. In that
output you can see entries like "X:Y/Z (time taken) (testname)" where X
is the process number (it had 15 processes) and Y/Z is test number Y of
a total of Z in that process.

In summary, I think this approach is going to be the way to make oe-
selftest work better on our infrastructure and I'll propose a patch
series accordingly. As part of that I'll propose we remove the existing
threading code as this new approach should replace it.

Cheers,

Richard