[OE-core] Fetch time optimization (svn : gcc/eglibc - git : linux-yocto)

[Richard Purdie]

On Fri, 2012-03-30 at 08:44 +0200, Samuel Stirtzel wrote:
> this might be a bit off-topic, but another idea would be to add a
> separate threading mechanism for fetching.
> 
> Current threading can help to use the CPU and memory load to it's optimum,
> but sometimes you have to wait for a download to finish..
> Instead there could be a separate set of threads that only download
> the sources and make optimal use of the bandwidth too.
> 
> This would also allow to fetch files when the normal threads are busy
> with configuring/building/packaging recipes.
> 
> 
> The downside would be that it requires some sort of inter process
> communication.
> Or it could be regulated with a simple check if the download is finished..
> 
> How does this idea sound to you?

Its easier than you think to do this, bitbake has a plugable scheduler
implementation so you'd just have to write one which ignores "fetch"
operations from the total thread count.

Sadly this isn't really the place most people have a bottleneck in day
to day usage of the system. People have tried various algorithms for
enhancing the scheduler and as far as I know never found anything that
makes a significant difference, much to everyone's surprise :/.

Cheers,

Richard