[bitbake-devel] [PATCH 2/2] runqueue: Introduce load balanced task spawning

[Alexander Kanavin]

2018-08-14 13:03 GMT+02:00 Alexander Kanavin <alex.kanavin at gmail.com>:
> Both make and ninja have -l option:
>
>       -l [load], --load-average[=load]
>             Specifies  that  no  new  jobs (commands) should be
> started if there are others jobs running and the load average is at
> least load (a floating-point number).  With no argument, removes a
>             previous load limit.
>
>        -l N   do not start new jobs if the load average is greater than N
>
> Maybe that could be appended to PARALLEL_MAKE, which is far less
> invasive than any other approach?

I've done some tests, and yes -l does help. We currently have a nasty
quadratic growth rate with cpu cores as input, which hits especially
badly when the amount of cores is high, and a lot of long, heavy (e.g.
c++) do_compile tasks run at once. This potentially means n*n compiler
instances, where n is how many cpu cores are available. '-l' option
does neatly limit that to a constant amount of compilers per core.

However, this does not solve the other resource problem: running out
of available RAM and going into swap thrashing. Neither make nor ninja
can currently watch the RAM, even though it is not complicated:
>>> import psutil
>>> psutil.virtual_memory()
svmem(total=16536903680, available=7968600064, percent=51.8,
used=16347615232, free=189288448, active=11750494208,
inactive=2882383872, buffers=3158528000, cached=4620783616)

I think we should teach both to do that, and then replace a static
'number of jobs' limit in PARALLEL_MAKE with limits on CPU load and
RAM usage.

Alex