[bitbake-devel] [PATCH] fetch2/git: always use premirror first if update is required

[Richard Purdie]

On Thu, 2016-09-22 at 11:17 +0100, Richard Purdie wrote:
> On Thu, 2016-09-22 at 09:45 +0200, Pascal Bach wrote:
> > 
> > > 
> > > 
> > > > 
> > > > 
> > > >     The tarball should only be fetched if the local ud.clonedir
> > > > is out of date, this means the revision that is requested via
> > > > SRCREV is not in the local repository.
> > > >     In the case the revision is present the tarball should not
> > > > be
> > > > fetched again, at least that was the intention and my local
> > > > tests
> > > > didn't indicate otherwise.
> > > > 
> > > > 
> > > > Yes, but it’s better to run ‘git fetch’ in an out of date clone
> > > > than to re-download a 1gb mirror tarball which may well be out
> > > > of
> > > > date too.
> > > Agreed, but this doesn't work in the case where the machine
> > > doesn't
> > > have access to the upstream git repository. For example in the
> > > case
> > > where BB_ALLOWED_NETWORKS = "*.my.company". The fetcher would
> > > still
> > > got to to github.com which is not allowed, in that case it should
> > > fall back to fetch the tarball from "mirror.my.company".
> > > 
> > > I'm open for suggestions how to make this more efficient.
> > Does anyone have a suggestion how this should behave in the ideal
> > case? In my opinion if there is an internal mirror defined it
> > should
> > always have precedence even if it might be less efficient.
> "less efficient" in this case translates a few seconds of git fetch
> operation on something like linux-yocto into a hundreds of megabytes
> download.
> 
> Much as you might not like it, I really don't think we can change the
> code as your patch does as it would badly affect standard usage for
> many users.
> 
> We need to find an alternative which allows your use case to work but
> I'm not sure what that is. Much as I hate it, we may just have to
> have
> a setting which you can set which changes the behaviour. If we do
> that,
> I will want to see test cases added for it though as the fetcher code
> is a mass of different configurations and its very very difficult to
> maintain as it is without yet more different code paths :(.


FWIW there is an open bug which I believe is related to this:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9061

It might be worth making the code aware of git aware premirror sources
compared to other things like mirror tarballs as a start at fixing
this.

Cheers,

Richard