[oe] PARALLEL_MAKE

[Denys Dmytriyenko]

On Fri, Mar 13, 2009 at 03:19:37PM +0100, Ulf Samuelsson wrote:
>
>> On Thu, Mar 12, 2009 at 11:49:30PM -0400, Denys Dmytriyenko wrote:
>>> On Thu, Mar 12, 2009 at 06:38:54PM -0700, Khem Raj wrote:
>>> > On Thursday 12 March 2009 16:32:16 Ulf Samuelsson wrote:
>>> > > Is there any activity to make this more useful?
>>> > > Just managed to install a Core i7 machine, which speeded
>>> > > up my Buildroot build from about 4 hours to less than 1 hour.
>>> > > (There was an error in the build right at the end,
>>> > > so it could have been downto 40 minutes or so)
>>> > > Building the cross compiler took 10 minutes with "make -j 16"
>>> > > compared to 1-2 hours on a P4 @ 3GHz.
>>> > >
>>> > > Openembedded does not seem to use PARALLEL_MAKE extensively.
>>> > > I have the frequency meter activated for all 8 CPUs in the Core i7,
>>> > > and most of the time the CPUs seems to idle at 1.6 GHz,
>>> > > with one CPU (or rarely 2) are running at the full 2.67 GHz.
>>> > >
>>> > > Last time I built OE on a P4@ 3 GHz it took ~40 hours
>>> > > for the SAM9263, so some speedup would be nice.
>>> > > Maybe we should make an effort?
>>> >
>>> > I use
>>> >
>>> > PARALLEL_MAKE = "-j4"
>>> > BB_NUMBER_THREADS = "2"
>>> >
>>> > in my local.conf. I see my core 2 duo machine spawning parallel tasks.
>>>
>>> I'm setting 4/4 to the above two variables on my Quad Core and it loads 
>>> all 4
>>> cores at 100% almost all the time. Sometimes it slows down due to the
>>> bottleneck in the harddrive access...
>>
>> On the dual dual-core (4 total) boxes I have, I do -j6/4 threads.  On my
>> new quad-core box I haven't yet had a chance to figure out what's best.
>> I also use -j6/4 threads on an older dual-core amd64 box we've got
>> around and that seems best for my tasks at least (I tried a few
>> combinations of more jobs/threads and less and both were slower).
>>
>
> I looked at a few recipies which by a stanrge coincidence had 
> PARALLEL_MAKE="",

That's not a strange coincidence, that means a specific package has issues 
building in parallel - usually some dependencies go out of order.

> When, I rechecked, I had
> PARALLEL_MAKE ="-j 16"
> as well and this should be
> PARALLEL_MAKE ="-j16"

Having only 4 cores and setting number of parallel makes to 16 kills your 
performance, as system spends more time switching contexts instead of actually 
doing something useful - i.e. compiling.

> Changed to
> PARALLEL_MAKE ="-j8"
> BB_NUMBER_THREADS = "2"
>
> I see some more cores beeing used.
> The initial setup after you changed local.conf seems to run on a single 
> core though.

After you change your configuration, bitbake rebuilds the cache - that is a 
single threaded operation, hence uses only one core.

-- 
Denys