[bitbake-devel] git fetcher and github pull requests

[Nicolas Dechesne]

On Thu, Feb 15, 2018 at 1:02 PM, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> On Thu, 2018-02-15 at 12:19 +0100, Nicolas Dechesne wrote:
>> On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie
>> <richard.purdie at linuxfoundation.org> wrote:
>> >
>> > On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote:
>> > >
>> > > we've been debugging an issue these days on our builder which
>> > > ended
>> > > up
>> > >  do_fetch failing with
>> > >
>> > > LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
>> > > git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with
>> > > exit
>> > > code 128, output:
>> > > error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
>> > > error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73
>> > >
>> > > It happens that these refs correspond to pull requests done on
>> > > github
>> > > which existed at some point, but have been updated with newer
>> > > commits,
>> > > and won't exist anymore.
>> > >
>> > > the bitbake fetcher seems to be greedy and fetches refs/* which
>> > > ends
>> > > up fetching pull request when fetching from github, e.g. in my
>> > > workspace:
>> > >
>> > > in $DL_DIR/git2/
>> > >
>> > > github.com.docker.containerd.git/refs/pull/66
>> > > github.com.docker.containerd.git/refs/pull/459
>> > > github.com.docker.containerd.git/refs/pull/551
>> > > github.com.docker.containerd.git/refs/pull/321
>> > > github.com.docker.containerd.git/refs/pull/60
>> > > github.com.docker.containerd.git/refs/pull/523
>> > > github.com.docker.containerd.git/refs/pull/40
>> > > github.com.docker.containerd.git/refs/pull/561
>> > >
>> > > It looks inefficient to fetch and store on each builder pull
>> > > requests.
>> > > I understand this is just because how PR are implemented in
>> > > github,
>> > > but github is quite central, so many we should/could do something
>> > > about it?
>> > >
>> > > Beyond the inefficiencies, we now are seeing unrelated build
>> > > issues
>> > > as well.
>> > >
>> > > What do you think? Should we try to avoid fetching refs/pull/*
>> > > from
>> > > github? or is it our git fetch command that needs to be improve
>> > > to
>> > > work in this situation?
>> > The git fetcher deals with a complicated set of problems since it
>> > needs
>> > to handle offline mirroring (with mirror tarballs) and a variety of
>> > different scenarios. The reason its pulling a complete set of
>> > references is likely to because of the mirroring and other demands
>> > placed upon it from the different scenarios. This isn't bad as
>> > such,
>> > github really shouldn't be sharing repos with invalid references in
>> > them.
>> the repos don't have any invalid references. But when pull requests
>> are updated by users on github, new references are pushed, and we
>> seem
>> to be keeping old (invalid) and new references in our mirrors...
>
> This isn't making sense to me :/.

well. that's the reason why i sent this email in the first place, it
doesn't seem to make any sense!

>
> Can you share a tarball of a repo where the above command gives the
> reference errors you mention? I'd like to have a closer look at what is
> happening...
>
> Or is there another way to reproduce the error?

right now, i can't, it was on our builders, and the fix was to delete
the git tarballs in DL_DIR...

the commit mentioned in the do_fetch() error correspond to github pull
request that have been submitted and eventually merged using github
interface with 'rebase and merge' option. the sha1 that bitbake
complains about corresponds to the sha1 of the initial pull request
before it was rebased and merged in the tree.

I will try to reproduce with a fake github project.

>
> Cheers,
>
> Richard