[OE-core] Fetch time optimization (svn : gcc/eglibc - git : linux-yocto)

[Richard Purdie]

On Fri, 2012-03-30 at 17:24 +0200, Eric Bénard wrote:
> Le Fri, 30 Mar 2012 16:12:44 +0100,
> Richard Purdie <richard.purdie at linuxfoundation.org> a écrit :
> > On Fri, 2012-03-30 at 10:50 +0200, Eric Bénard wrote:
> > > the default configuration seems to fetch from source control systems
> > > as I always see very long time to fetch gcc/eglibc/linux-yocto
> > > (despite having a 2.2 MBytes/s downlink DSL line).
> > 
> > If you're hitting the SCMs I can understand the frustration.
> > 
> that's not a frustration, that's a feedback on the default
> behaviour. But I agree with you that could be a frustration for someone
> trying OE-core for the first time ;-)
> 
> > Try adding this to your configuration:
> > 
> > PREMIRRORS = "\
> > git://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> > svn://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n"
> > 
> > and see if that helps the performance. It might be we consider making
> > this the default for OE-Core although some people are nervous about
> > doing this...
> > 
> sure that will help : in my work setup I have my own mirrors configured
> but here again, that's not what a new user will have and in that
> case, I'm testing the plain default configuration to help finding bugs
> or things to improve the release.
> 
> I think fetching from git or svn should not be the first thing to do in
> recipes like gcc, eglibc, linux & co where we are based on a
> stable released version : this doesn't bring real added value to the
> user in OE context and this wastes bandwidth (a tbz2 kernel is around
> 75MB, a git one is around 600MB).

We've gone around in circles on this. We did use tarballs for gcc,
people complained. We switched to svn, you're not happy and probably
others aren't. We can't win.

Adding the PREMIRRORS makes the situation better, I agree its not
perfect. The original question was how can we speed it up and this is an
easy way to do so for the default user case without changing anything
major.

I hear what you're saying on the tarball vs. SCM issue but using
tarballs does break use cases some users do use, the opposite is not
true. 

Its also ultimately down to the maintainers of recipes. The gcc issue is
more maintainable the way its set up now compared to large numbers of
patches (which took an age to apply) and doesn't have much of an
additional bandwidth cost.

The linux-yocto kernel recipe heavily uses the SCM to do things so
whilst it does have a higher download cost, it as adds value and is
ultimately a maintainers choice too.

So whilst I hear what you're saying, I don't think we can change
anything other than the PREMIRROR...

Cheers,

Richard