[bitbake-devel] [PATCH 2/8] git-make-shallow: add script to make a git repo shallow

[Richard Purdie]

On Sat, 2017-05-13 at 02:46 +0500, Christopher Larson wrote:
> This script will be used by the git fetcher to create shallow mirror
> tarballs.
> 
>     usage: git-make-shallow [-h] [--ref REF] [--shrink] REVISION
> [REVISION ...]
> 
>     Remove the history of the specified revisions, then optionally
> filter the
>     available refs to those specified.
> 
>     positional arguments:
>       REVISION           a git revision/commit
> 
>     optional arguments:
>       -h, --help         show this help message and exit
>       --ref REF, -r REF  remove all but the specified refs
> (cumulative)
>       --shrink, -s       shrink the git repository by repacking and
> pruning
> 
> While git does provide the ability to clone at a specific depth, and
> fetch all
> remote refs at a particular depth, the depth is across all
> branches/tags, and
> doesn't provide the flexibility we need, hence this script.
> 
> Refs (branches+tags) can be filtered, as the process of history
> removal scales
> up rapidly with the number of refs. Even the existing `git fetch --
> depth=` is
> extremely slow on an upstream kernel repository with all the branches
> and tags
> kept.
> 
> This uses the same underlying mechanism to implement the history
> removal which
> git itself uses (.git/shallow), and the results, when configured
> similarly, are
> in line with the results git itself produces with `fetch --depth`.

Sorry about the delay in getting to this. In general the series looks
great. One minor thing that bothers me a bit is the name of this
script. If git itself ever creates something similar it may cause us
issues. I wondered if we wanted to namespace this more closely to
bitbake?

I'm leaning towards merging this and then perhaps tweaking the naming.

The other thing I'd really like to see is the documentation collected
out the commit messages and sent to Scott Rifenbark (cc'd) as a summary
of the changes so he can update the manual, particularly the section on
the git fetcher. If you could review it once Scott has done that, that
would be awesome :).

I do have my worries about how much complexity this adds to the git
fetcher but I guess its inevitable and at least we have the tests
(thanks!).

In the back of my mind there is another way I've been wondering about
assisting fetches, namely a "common reference repo" which urls may name
(e.g. git://xxx;refrepo=linux-kernel). If this was specified, any clone
would happen into the reference name as a subset namespace before a
clone by reference into the main repo name. With kernels, this would
mean one core repo and then all the subset kernels would clone by
reference from it. That should mean all kernels would share a common
repo so once you have one things should all be fast. Obviously shallow
clones solve a different problem in some ways but there is overlap.

Cheers,

Richard