[OE-core] Fetch time optimization (svn : gcc/eglibc - git : linux-yocto)

[Samuel Stirtzel]

2012/3/30 Richard Purdie <richard.purdie at linuxfoundation.org>:
> On Fri, 2012-03-30 at 08:44 +0200, Samuel Stirtzel wrote:
>> this might be a bit off-topic, but another idea would be to add a
>> separate threading mechanism for fetching.
>>
>> Current threading can help to use the CPU and memory load to it's optimum,
>> but sometimes you have to wait for a download to finish..
>> Instead there could be a separate set of threads that only download
>> the sources and make optimal use of the bandwidth too.
>>
>> This would also allow to fetch files when the normal threads are busy
>> with configuring/building/packaging recipes.
>>
>>
>> The downside would be that it requires some sort of inter process
>> communication.
>> Or it could be regulated with a simple check if the download is finished..
>>
>> How does this idea sound to you?
>
> Its easier than you think to do this, bitbake has a plugable scheduler
> implementation so you'd just have to write one which ignores "fetch"
> operations from the total thread count.
>
> Sadly this isn't really the place most people have a bottleneck in day
> to day usage of the system. People have tried various algorithms for
> enhancing the scheduler and as far as I know never found anything that
> makes a significant difference, much to everyone's surprise :/.
>
> Cheers,
>
> Richard
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core at lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Of course this will only reduce the time of recipes if they are build
for the first time,
or when the version/URL changes.
It is not that important, I agree,
but it would improve the situation for first time users, or new installations.


Example for 2 threads:
http://pastebin.com/kviwQZJ3
It is very likely that the current situation also uses cpu and network
resources at the same time,
but it might occur that the build-task has to wait for a download to
finish or vice versa.

Ignoring fetch tasks from the thread count would only do half of the
job and _could_ cause network bottlenecks ;)
Fetching should be "independent" from the dependency chain.
E.g.: it should not wait with the downloads for dependencies to finish building,
the download sequence should still match the dependency chain sequence.



If it is really that easy, then I will look into it.

-- 
Regards
Samuel