[bitbake-devel] git fetcher and github pull requests

[Richard Purdie]

On Thu, 2018-02-15 at 12:19 +0100, Nicolas Dechesne wrote:
> On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie
> <richard.purdie at linuxfoundation.org> wrote:
> > 
> > On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote:
> > > 
> > > we've been debugging an issue these days on our builder which
> > > ended
> > > up
> > >  do_fetch failing with
> > > 
> > > LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
> > > git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with
> > > exit
> > > code 128, output:
> > > error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
> > > error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73
> > > 
> > > It happens that these refs correspond to pull requests done on
> > > github
> > > which existed at some point, but have been updated with newer
> > > commits,
> > > and won't exist anymore.
> > > 
> > > the bitbake fetcher seems to be greedy and fetches refs/* which
> > > ends
> > > up fetching pull request when fetching from github, e.g. in my
> > > workspace:
> > > 
> > > in $DL_DIR/git2/
> > > 
> > > github.com.docker.containerd.git/refs/pull/66
> > > github.com.docker.containerd.git/refs/pull/459
> > > github.com.docker.containerd.git/refs/pull/551
> > > github.com.docker.containerd.git/refs/pull/321
> > > github.com.docker.containerd.git/refs/pull/60
> > > github.com.docker.containerd.git/refs/pull/523
> > > github.com.docker.containerd.git/refs/pull/40
> > > github.com.docker.containerd.git/refs/pull/561
> > > 
> > > It looks inefficient to fetch and store on each builder pull
> > > requests.
> > > I understand this is just because how PR are implemented in
> > > github,
> > > but github is quite central, so many we should/could do something
> > > about it?
> > > 
> > > Beyond the inefficiencies, we now are seeing unrelated build
> > > issues
> > > as well.
> > > 
> > > What do you think? Should we try to avoid fetching refs/pull/*
> > > from
> > > github? or is it our git fetch command that needs to be improve
> > > to
> > > work in this situation?
> > The git fetcher deals with a complicated set of problems since it
> > needs
> > to handle offline mirroring (with mirror tarballs) and a variety of
> > different scenarios. The reason its pulling a complete set of
> > references is likely to because of the mirroring and other demands
> > placed upon it from the different scenarios. This isn't bad as
> > such,
> > github really shouldn't be sharing repos with invalid references in
> > them.
> the repos don't have any invalid references. But when pull requests
> are updated by users on github, new references are pushed, and we
> seem
> to be keeping old (invalid) and new references in our mirrors...

This isn't making sense to me :/.

Can you share a tarball of a repo where the above command gives the
reference errors you mention? I'd like to have a closer look at what is
happening...

Or is there another way to reproduce the error?

Cheers,

Richard