[bitbake-devel] [PATCH 2/2] runqueue: Introduce load balanced task spawning

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces at lists.openembedded.org <bitbake-devel-
> bounces at lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 14 augusti 2018 11:45
> To: Alexander Kanavin <alex.kanavin at gmail.com>; Mikko.Rapeli at bmw.de;
> Andreas Müller <schnitzeltony at gmail.com>; Martin Jansa
> <martin.jansa at gmail.com>
> Cc: bitbake-devel at lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH 2/2] runqueue: Introduce load
> balanced task spawning
> 
> On Tue, 2018-08-14 at 10:43 +0100, Richard Purdie wrote:
> > On Tue, 2018-08-14 at 10:07 +0200, Alexander Kanavin wrote:
> > > 2018-08-14 8:32 GMT+02:00  <Mikko.Rapeli at bmw.de>:
> > > > Just my 2 € cents to the discussion:
> > > >
> > > > we had to limit number of threads because heavy C++ projects 
> > > > were using all of RAM and causing heavy swapping. Single g++ 
> > > > processes were eating up to 20 gigabytes of physical ram. 
> > > > It's not just the CPU which is the limiting factor to 
> > > > parallel task execution.
> > >
> > > I do believe some kind of clever dynamic limiter would be
> > > useful here. Obviously it's an absurd situation when on a 32 
> > > core processor there are 32 do_compile c++ tasks, each running 
> > > 32 instances of g++ - which is the default configuration. On 
> > > the other hand running things like do_configure or do_install 
> > > should happen in parallel. I like Andre's idea, but it should 
> > > also watch the available RAM.
> >
> > If people want to play, I did experiment with "proper" thread
> > pooling:
> >
> > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237
> 
> More recent version of the patch:
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue7&id=236ca8be128ba7a4edb9fb2c9e512d181679eee8
> 
> Cheers,
> 
> Richard

Even though make is the prevalent tool used to build code, we have 
to consider others such as cmake and meson... Any idea if their 
parallelism can be controlled in some similar way?

//Peter