[bitbake-devel] [PATCH 2/2] runqueue: Introduce load balanced task spawning

[Richard Purdie]

On Tue, 2018-08-14 at 10:07 +0200, Alexander Kanavin wrote:
> 2018-08-14 8:32 GMT+02:00  <Mikko.Rapeli at bmw.de>:
> > Just my 2 € cents to the discussion:
> > 
> > we had to limit number of threads because heavy C++ projects were
> > using
> > all of RAM and causing heavy swapping. Single g++ processes were
> > eating
> > up to 20 gigabytes of physical ram. It's not just the CPU which is
> > the
> > limiting factor to parallel task execution.
> 
> I do believe some kind of clever dynamic limiter would be useful
> here.
> Obviously it's an absurd situation when on a 32 core processor there
> are 32 do_compile c++ tasks, each running 32 instances of g++ - which
> is the default configuration. On the other hand running things like
> do_configure or do_install should happen in parallel. I like Andre's
> idea, but it should also watch the available RAM.

If people want to play, I did experiment with "proper" thread pooling:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237

This implements a make job server within bitbake, then connects make to
it. The net result is that you can then put a limit on the number of
processes across all tasks.

I seem to remember there were some bugs with it and not all are listed
in the commit message, I don't remember what the other issues were...

Bonus marks for connecting in the other parallel "pool" tasks we have
in do_package_* and friends but even a common pool for compile would be
nice!

Cheers,

Richard