[bitbake-devel] git fetcher and github pull requests

[Nicolas Dechesne]

On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote:
>> we've been debugging an issue these days on our builder which ended
>> up
>>  do_fetch failing with
>>
>> LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
>> git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit
>> code 128, output:
>> error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
>> error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73
>>
>> It happens that these refs correspond to pull requests done on github
>> which existed at some point, but have been updated with newer
>> commits,
>> and won't exist anymore.
>>
>> the bitbake fetcher seems to be greedy and fetches refs/* which ends
>> up fetching pull request when fetching from github, e.g. in my
>> workspace:
>>
>> in $DL_DIR/git2/
>>
>> github.com.docker.containerd.git/refs/pull/66
>> github.com.docker.containerd.git/refs/pull/459
>> github.com.docker.containerd.git/refs/pull/551
>> github.com.docker.containerd.git/refs/pull/321
>> github.com.docker.containerd.git/refs/pull/60
>> github.com.docker.containerd.git/refs/pull/523
>> github.com.docker.containerd.git/refs/pull/40
>> github.com.docker.containerd.git/refs/pull/561
>>
>> It looks inefficient to fetch and store on each builder pull
>> requests.
>> I understand this is just because how PR are implemented in github,
>> but github is quite central, so many we should/could do something
>> about it?
>>
>> Beyond the inefficiencies, we now are seeing unrelated build issues
>> as well.
>>
>> What do you think? Should we try to avoid fetching refs/pull/* from
>> github? or is it our git fetch command that needs to be improve to
>> work in this situation?
>
> The git fetcher deals with a complicated set of problems since it needs
> to handle offline mirroring (with mirror tarballs) and a variety of
> different scenarios. The reason its pulling a complete set of
> references is likely to because of the mirroring and other demands
> placed upon it from the different scenarios. This isn't bad as such,
> github really shouldn't be sharing repos with invalid references in
> them.

the repos don't have any invalid references. But when pull requests
are updated by users on github, new references are pushed, and we seem
to be keeping old (invalid) and new references in our mirrors...

>
> I did read this when you sent it, I just don't have any good
> recommendation for you :(

the email is not a request for someone to fix.. just to get some
feedback, and at least mention this issue on the list in case someone
else hits it too...

>
> I dread "special casing" in the git fetcher as has burnt us a lot in
> the past, equally, that may be the only way to avoid this kind of
> problem...

I will try to reproduce the issue with a simpler build/config. It
could be that git is confused by github using refs as refs/pull..

>
> Cheers,
>
> Richard
>
>
>