[OE-core] Fetch time optimization (svn : gcc/eglibc - git : linux-yocto)

[Eric Bénard]

Le Thu, 29 Mar 2012 23:03:13 +0100,
Richard Purdie <richard.purdie at linuxfoundation.org> a écrit :

> On Thu, 2012-03-29 at 22:53 +0200, Eric Bénard wrote:
> > I noticed in from scratch builds for qemuarm that the longest time is
> > taken in fetching sources, especially those fetched using git
> > (linux-yocto for example) & svn (gcc, eglibc & co).
> 
> Are you timing these as fetches from the source control systems or from
> the mirror tarballs of the repositories. The tarballs should be
> faster...
> 
the default configuration seems to fetch from source control systems
as I always see very long time to fetch gcc/eglibc/linux-yocto
(despite having a 2.2 MBytes/s downlink DSL line).

> > To reduce the fetch time would that make sense to 
> > - fetch gcc/glibc & co from the archive of a stable version and then
> >   apply patches on top of it (maybe patches stored in an archive
> >   fetched from oe's website and applied in bulk or patches stored in OE)
> 
> Unfortunately the patches tend to get unwieldy. The tarballs of the svn
> repos on the mirror should be about equal in size to the upstream
> archive in this case. 
> 
I don't think that's a size problem but that fetching through svn or
git is far less efficient than http or ftp especially from gnu's svn
which may be overloaded.
Morover in a pure OE context we have no interest of all the source
history provided by svn or git and that makes a very big volume to
download.

> > - do the same thing for the linux-yocto kernel or add a --reference
> >   option to the git fetcher so that we can provide a local tree as a
> >   reference ?
> 
> This is effectively how the repositories in DL_DIR are used. If you
> place a tree in the right place there, it should reuse references...
> 
> > Do you have other ideas (appart from using a local mirror) to optimize
> > the fetch time ?
> 
> I'd be interested firstly to understand if you're using the SCM directly
> or using the mirror tarballs as that should make a big difference. In
> the standard configuration it should be using mirror tarballs...
> 
that doesn't seems to be the case :
from a clean oe-core + bitbake clone :

. ./openembedded-core/oe-init-build-env 
edit local.conf to select qemuarm & BBTHREAD to 8
bitbake core-image-minimal -c fetchall

and then I see bitbake stops at around 209 or 214 tasks waiting and
I see that in ps :
/home/ebenard/OE-CORE/build/tmp-eglibc/sysroots/x86_64-linux/usr/bin/git.real
clone --bare --mirror
git://git.yoctoproject.org/linux-yocto-3.2 /home/ebenard/OE-CORE/build/downloads/git2/git.yoctoproject.org.linux-yocto-3.2
and
svn co -r 184847
http://gcc.gnu.org/svn/gcc/branches/gcc-4_6-branch@184847

... and both are actually fetching at only around 200 KiB/s which last
for quite a long time as (from an other downloads dir) the final
tree size are huge :
du -s git.yoctoproject.org.linux-yocto-3.2/
610824	git.yoctoproject.org.linux-yocto-3.2/
du -s gcc.gnu.org/
1602496	gcc.gnu.org/
du -s www.eglibc.org/
625048	www.eglibc.org
If I launch at the same time :
wget ftp://ftp.gnu.org/gnu/gcc/gcc-4.6.3/gcc-4.6.3.tar.bz2
I get a download speed close to 1MB/s and the file to download is only
64MB which would save bandwidth.

Eric