[OE-core] Fetch failure for source at kernel.org

[Richard Purdie]

On Thu, 2011-09-22 at 13:27 +0200, Koen Kooi wrote:
> Op 22 sep 2011, om 12:20 heeft Richard Purdie het volgende geschreven:
> 
> > On Thu, 2011-09-22 at 10:28 +0200, Koen Kooi wrote:
> >> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende  
> > This just means there aren't appropriate git:// -> git:// mappings in
> > people's mirrors files (including Yocto). If git:// -> git:// mappings
> > don't work, we should fix that but they should.
> 
> You mean something like git.kernel.org/scm/torvalds/linux-2.6.git ->  
> github.com/torvalds/linux.git ?

Yes. Thinking about this, the current fetcher might not be optimised in
making this as efficient as possible but it should work and if it
doesn't, we should make it.

> > Now someone has explained the issue, yes, I can see why it is causing
> > problems. I do think some of them are of people's own choosing but I'm
> > trying to be flexible and not make judgement on that.
> >
> > So we can do something along the lines of what you after but there are
> > the following considerations:
> >
> > a) The versioned tarballs do need to be different to what we had  
> > before
> > in that the tarball will be of a .git directory containing the git
> > metadata, not just a checkout. Why? This is so we can incrementally
> > update a git repo if the user changes the source revision as one
> > example. We're dealing with the git fetcher and limiting ourselves  
> > to a
> > flat one dimension doesn't make sense.
> 
> Agreed on that. The bitbake patch I sent does at least do that :)

Right :)

I just want to be clear we're doing something different to what was
there before (and why).

> > b) We need to agree what order the tarballs on the server should be
> > searched for. The "obvious" order would be:
> >
> > 1. Specific Revision X tarball
> > 2. generic mirror tarball
> > 3. git clone
> >
> > The problems come about when we have an existing git checkout which
> > doesn't contain the revision we want.
> 
> There currently is a bug in the fetcher that makes it hard to use now  
> that kernel.org is down. The situation:
> 
> I have a recipe that pulls from git (e.g. cpufrequtils.git from  
> kernel.org) with SRCREV fixed to e.g. deadbeef. DL_DIR/git/ 
> git.kernel.org.utils.cpufrequtils has revision deadbeef. When I do a - 
> c fetch -f it will try to do an update and fail since kernel.org is  
> down. I would have expected it to check the existing bare clone for  
> deadbeef first and not try updating if it is present.

Agreed, it shouldn't be hitting the network unless the revision is
floating. That is bad :/.

> Anyway, I agree on the search order above.
> 
> >  Currently we can just update and
> > assume that will bring in the revision. If we now support people doing
> > rebases and other things, we now at least in theory have to:
> >
> > 1. See if the revision exists locally
> > 2. Try pulling
> > 3. If that doesn't work look for a versioned tarball
> > 4. Blow away all fetcher git status from DL_DIR
> > 5. Extract the tarball
> 
> Since git supports local fetches the above can be:
> 
> 1) see if rev exists
> 2) try pulling
> 3) try versioned tarball
> 4) extract tarball into tmpdir, fetch objects into DL_DIR/git2 repo
> 
> That would stop things from being blown away. It does allow the  
> versioned snapshots to differ, but they will always have the rev they  
> were versioned with.

This is a much better approach. Its quite a big difference from anything
we've done before but the approach should fit in better. I need to think
on it more...

> > My big concern with this is rather than having some general
> > archictecture and standard way of doing this, the fetcher once again
> > becomes a set of special case if A do B but if C do D first then do E
> > but don't forget to do A if Z happens type code.
> 
> I know virtually nothing of the fetcher code, but I think treating  
> every tarball as a local mirror that gets fetched into DL_DIR/git2/ 
> repodir would avoid a lot of special casing.

Yes, its nicer. We keep falling into the trap of thinking about the
non-distributed SCM cases but we have more capability with git and it
makes sense to use it.

> > So what I'm saying is I don't want the fetcher to become the
> > unmaintainable mess the that the code paths in fetch1 were. I don't
> > honestly know how to add a code path like the above and avoid this :(
> >
> > The patch on the bitbake list doesn't consider any of this  
> > complexity or
> > the corner cases
> 
> Sadly it's the best thing I can accomplish with my python skills :( It  
> does solve my #1 need, which is an easy way (ls | grep) to check for  
> GPL compliance.

Ok. Can you at least file a bug in the yocto bugzilla with a summary of
this discussion please? :) That should ensure the issue gets more
attention since with the best intentions, I can't remember everything.

Cheers,

Richard