[OE-core] Fetch time optimization (svn : gcc/eglibc - git : linux-yocto)

[Richard Purdie]

On Thu, 2012-03-29 at 22:53 +0200, Eric Bénard wrote:
> I noticed in from scratch builds for qemuarm that the longest time is
> taken in fetching sources, especially those fetched using git
> (linux-yocto for example) & svn (gcc, eglibc & co).

Are you timing these as fetches from the source control systems or from
the mirror tarballs of the repositories. The tarballs should be
faster...

> To reduce the fetch time would that make sense to 
> - fetch gcc/glibc & co from the archive of a stable version and then
>   apply patches on top of it (maybe patches stored in an archive
>   fetched from oe's website and applied in bulk or patches stored in OE)

Unfortunately the patches tend to get unwieldy. The tarballs of the svn
repos on the mirror should be about equal in size to the upstream
archive in this case. 

> - do the same thing for the linux-yocto kernel or add a --reference
>   option to the git fetcher so that we can provide a local tree as a
>   reference ?

This is effectively how the repositories in DL_DIR are used. If you
place a tree in the right place there, it should reuse references...

> Do you have other ideas (appart from using a local mirror) to optimize
> the fetch time ?

I'd be interested firstly to understand if you're using the SCM directly
or using the mirror tarballs as that should make a big difference. In
the standard configuration it should be using mirror tarballs...

Cheers,

Richard