[bitbake-devel] [PATCH 2/2] runqueue: Introduce load balanced task spawning

[Richard Purdie]

On Tue, 2018-08-14 at 10:43 +0100, Richard Purdie wrote:
> On Tue, 2018-08-14 at 10:07 +0200, Alexander Kanavin wrote:
> > 2018-08-14 8:32 GMT+02:00  <Mikko.Rapeli at bmw.de>:
> > > Just my 2 € cents to the discussion:
> > > 
> > > we had to limit number of threads because heavy C++ projects were
> > > using
> > > all of RAM and causing heavy swapping. Single g++ processes were
> > > eating
> > > up to 20 gigabytes of physical ram. It's not just the CPU which
> > > is
> > > the
> > > limiting factor to parallel task execution.
> > 
> > I do believe some kind of clever dynamic limiter would be useful
> > here.
> > Obviously it's an absurd situation when on a 32 core processor
> > there
> > are 32 do_compile c++ tasks, each running 32 instances of g++ -
> > which
> > is the default configuration. On the other hand running things like
> > do_configure or do_install should happen in parallel. I like
> > Andre's
> > idea, but it should also watch the available RAM.
> 
> If people want to play, I did experiment with "proper" thread
> pooling:
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/w
> ipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237

More recent version of the patch:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue7&id=236ca8be128ba7a4edb9fb2c9e512d181679eee8

Cheers,

Richard