[oe] PARALLEL_MAKE

[Ulf Samuelsson]

Best Regards
Ulf Samuelsson                ulf at atmel.com
Atmel Nordic AB
Mail:  Box 2033, 174 02 Sundbyberg, Sweden
Visit:  Kavallerivägen 24, 174 58 Sundbyberg, Sweden
Phone +46 (8) 441 54 22     Fax +46 (8) 441 54 29
GSM    +46 (706) 22 44 57



> On Tue, Mar 17, 2009 at 05:28:27PM +0100, Ulf Samuelsson wrote:
>>> Yes, I know,
>>> Maybe it is a good idea if the result of a build is published somewhere,
>>> so that others do not have to go through this process.
>>>
>>> Right now, openembedded does not even build for me,
>>> since the native perl build fails :-(
>>>
>>>> [1] http://en.wikipedia.org/wiki/Hyper-threading
>>
>> I now managed to build the Angstrom image for the AT91SAM9263 by applying
>> a few fixes present in the ulf/linux branch.
>> (Did not load on H/W yet though)
>>
>> perl-native was "fixed" by not building the library which crashed.
>> libtheora was fixd, not to use docs/examples.
>> busybox updated to 1.13.3
>
> Any specific changes between 1.13.2 and 1.13.3 you need? Would be nice to
> cherry-pick into .dev branch...

The reason I did this was that I had an error on downloading one
of the patches for 1.13.2. (md5 Checksum was not the same as in recipe)
I did not investigate further.
Just generally thought it to be a good idea to use the latest.

>
>> u-boot, gpsd was set to PARALLEL_MAKE = ""
>> u-boot updated to 2009.01
>> linux-2.6.28
>> at91bootstrap-2.10
>>
>> Building with PARALLEL_MAKE & Multiple threads was not smooth.
>> The build crashed several times, due to erros
>> caused by parallellism. "gettext", "gpsd" and "u-boot" were culprits.
>
> Would be nice to fix those. Tom Rini was working on fixing some of those
> issues.
>
>> u-boot is NOT built cleanly, even after PARALLEL_MAKE was set to "".
>> The actual compile completes and generates an u-boot.bin
>> but this is not deployed to the result directory.
>> There are logs for different stages, but after the compile, they exist, 
>> but
>> are empty.
>>
>> If I remove all the stamps and "bitbake -b <u-boot>" afterwards
>> then the build completes correctly, and there is an
>> u-boot-at91sam9263ek.bin
>> in the result directory.
>>
>> Can it be so, that a stage is started, before the previous stage has been
>> completed???
>
> If there are no dependencies between packages, their stage tasks can run 
> in
> parallel.

I know that, but the u-boot problem seems to be that a stage
is started before an earlier stage is completed.
I have not verified this, but doing bitbake gpe-image
will cause the u-boot-2009.01 (in my branch) to fail.
The u-boot.bin is not copied to the result.
while doing "bitbake -b <u-boot-2009.01>.bb" will work.

The remainder of the stages will run, when I do bitbake gpe-image
but the log is empty in the first case.


>
>> ---------------------
>> BTW: The total buildtime was around 2 hours. There is some overhead
>> since the build was restarted 3-4 times, and local.conf was changed once.
>> I started the build with PARALLEL_MAKE = "-j2" and BB_NUMBER_THREADS=8.
>>
>> I noticed that having a lot of threads helps most of the time,
>> since the CPUs were executing at max frequency.
>>
>> When the cross compiler, was built,it was different,
>> most of the cores were not used with this configuration.
>> 2-3 max while compiling gcc.
>
> Since gcc/cross is quite a big package and lots of other packages depend 
> on
> it, it is being built for you only using 2 cores, as specified by
> PARALLEL_MAKE. Everything else is waiting for it to finish.

Yes, I realize that so I have to increase PARALLEL_MAKE
to speed up the  build of the tools,
but I would also like to have a large number of threads otherwise.
But I do not want to have a large number of threads and
and a large number of make jobs at the same time
if I want to run at optimum speed throughout the build.



>
>> Maybe the best approach is to build the Angstrom in several stages
>> with different settings for parallelism.
>
> -- 
> Denys
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel at lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>