[bitbake-devel] git fetcher and github pull requests

[Richard Purdie]

On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote:
> we've been debugging an issue these days on our builder which ended
> up
>  do_fetch failing with
> 
> LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
> git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit
> code 128, output:
> error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
> error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73
> 
> It happens that these refs correspond to pull requests done on github
> which existed at some point, but have been updated with newer
> commits,
> and won't exist anymore.
> 
> the bitbake fetcher seems to be greedy and fetches refs/* which ends
> up fetching pull request when fetching from github, e.g. in my
> workspace:
> 
> in $DL_DIR/git2/
> 
> github.com.docker.containerd.git/refs/pull/66
> github.com.docker.containerd.git/refs/pull/459
> github.com.docker.containerd.git/refs/pull/551
> github.com.docker.containerd.git/refs/pull/321
> github.com.docker.containerd.git/refs/pull/60
> github.com.docker.containerd.git/refs/pull/523
> github.com.docker.containerd.git/refs/pull/40
> github.com.docker.containerd.git/refs/pull/561
> 
> It looks inefficient to fetch and store on each builder pull
> requests.
> I understand this is just because how PR are implemented in github,
> but github is quite central, so many we should/could do something
> about it?
> 
> Beyond the inefficiencies, we now are seeing unrelated build issues
> as well.
> 
> What do you think? Should we try to avoid fetching refs/pull/* from
> github? or is it our git fetch command that needs to be improve to
> work in this situation?

The git fetcher deals with a complicated set of problems since it needs
to handle offline mirroring (with mirror tarballs) and a variety of
different scenarios. The reason its pulling a complete set of
references is likely to because of the mirroring and other demands
placed upon it from the different scenarios. This isn't bad as such,
github really shouldn't be sharing repos with invalid references in
them.

I did read this when you sent it, I just don't have any good
recommendation for you :(

I dread "special casing" in the git fetcher as has burnt us a lot in
the past, equally, that may be the only way to avoid this kind of
problem...

Cheers,

Richard