[bitbake-devel] [PATCH 2/2] runqueue: Introduce load balanced task spawning

[Andreas Müller]

On Tue, Aug 14, 2018 at 12:28 PM, Peter Kjellerstedt
<peter.kjellerstedt at axis.com> wrote:
>> -----Original Message-----
>> From: bitbake-devel-bounces at lists.openembedded.org <bitbake-devel-
>> bounces at lists.openembedded.org> On Behalf Of Richard Purdie
>> Sent: den 14 augusti 2018 11:45
>> To: Alexander Kanavin <alex.kanavin at gmail.com>; Mikko.Rapeli at bmw.de;
>> Andreas Müller <schnitzeltony at gmail.com>; Martin Jansa
>> <martin.jansa at gmail.com>
>> Cc: bitbake-devel at lists.openembedded.org
>> Subject: Re: [bitbake-devel] [PATCH 2/2] runqueue: Introduce load
>> balanced task spawning
>>
>> On Tue, 2018-08-14 at 10:43 +0100, Richard Purdie wrote:
>> > On Tue, 2018-08-14 at 10:07 +0200, Alexander Kanavin wrote:
>> > > 2018-08-14 8:32 GMT+02:00  <Mikko.Rapeli at bmw.de>:
>> > > > Just my 2 € cents to the discussion:
>> > > >
>> > > > we had to limit number of threads because heavy C++ projects
>> > > > were using all of RAM and causing heavy swapping. Single g++
>> > > > processes were eating up to 20 gigabytes of physical ram.
>> > > > It's not just the CPU which is the limiting factor to
>> > > > parallel task execution.
>> > >
>> > > I do believe some kind of clever dynamic limiter would be
>> > > useful here. Obviously it's an absurd situation when on a 32
>> > > core processor there are 32 do_compile c++ tasks, each running
>> > > 32 instances of g++ - which is the default configuration. On
>> > > the other hand running things like do_configure or do_install
>> > > should happen in parallel. I like Andre's idea, but it should
>> > > also watch the available RAM.
>> >
>> > If people want to play, I did experiment with "proper" thread
>> > pooling:
>> >
>> > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237
>>
>> More recent version of the patch:
>>
>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue7&id=236ca8be128ba7a4edb9fb2c9e512d181679eee8
>>
>> Cheers,
>>
>> Richard
>
> Even though make is the prevalent tool used to build code, we have
> to consider others such as cmake and meson... Any idea if their
> parallelism can be controlled in some similar way?
>
BTW: do_package_ipk is also a CPU eater these days although these task
do not last as long as compile.

Andreas