[OE-core] Fetch time optimization (svn : gcc/eglibc - git : linux-yocto)

[Richard Purdie]

On Fri, 2012-03-30 at 10:50 +0200, Eric Bénard wrote:
> Le Thu, 29 Mar 2012 23:03:13 +0100,
> Richard Purdie <richard.purdie at linuxfoundation.org> a écrit :
> 
> > On Thu, 2012-03-29 at 22:53 +0200, Eric Bénard wrote:
> > > I noticed in from scratch builds for qemuarm that the longest time is
> > > taken in fetching sources, especially those fetched using git
> > > (linux-yocto for example) & svn (gcc, eglibc & co).
> > 
> > Are you timing these as fetches from the source control systems or from
> > the mirror tarballs of the repositories. The tarballs should be
> > faster...
> > 
> the default configuration seems to fetch from source control systems
> as I always see very long time to fetch gcc/eglibc/linux-yocto
> (despite having a 2.2 MBytes/s downlink DSL line).

If you're hitting the SCMs I can understand the frustration.

> I don't think that's a size problem but that fetching through svn or
> git is far less efficient than http or ftp especially from gnu's svn
> which may be overloaded.

Agreed.

> Morover in a pure OE context we have no interest of all the source
> history provided by svn or git and that makes a very big volume to
> download.

The fetcher will deal with this well in the svn case. In the git case,
we made a decision to include history since its not that more expensive.
Both these assumptions are based on a working up to date mirror.

> . ./openembedded-core/oe-init-build-env 
> edit local.conf to select qemuarm & BBTHREAD to 8
> bitbake core-image-minimal -c fetchall
> 
> and then I see bitbake stops at around 209 or 214 tasks waiting and
> I see that in ps :
> /home/ebenard/OE-CORE/build/tmp-eglibc/sysroots/x86_64-linux/usr/bin/git.real
> clone --bare --mirror
> git://git.yoctoproject.org/linux-yocto-3.2 /home/ebenard/OE-CORE/build/downloads/git2/git.yoctoproject.org.linux-yocto-3.2
> and
> svn co -r 184847
> http://gcc.gnu.org/svn/gcc/branches/gcc-4_6-branch@184847
> 
> ... and both are actually fetching at only around 200 KiB/s which last
> for quite a long time as (from an other downloads dir) the final
> tree size are huge :
> du -s git.yoctoproject.org.linux-yocto-3.2/
> 610824	git.yoctoproject.org.linux-yocto-3.2/
> du -s gcc.gnu.org/
> 1602496	gcc.gnu.org/
> du -s www.eglibc.org/
> 625048	www.eglibc.org
> If I launch at the same time :
> wget ftp://ftp.gnu.org/gnu/gcc/gcc-4.6.3/gcc-4.6.3.tar.bz2
> I get a download speed close to 1MB/s and the file to download is only
> 64MB which would save bandwidth.

Try adding this to your configuration:

PREMIRRORS = "\
git://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
svn://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n"

and see if that helps the performance. It might be we consider making
this the default for OE-Core although some people are nervous about
doing this...

Cheers,

Richard