[bitbake-devel] [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

Some more notes on this implementation:

On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth at gmail.com>
wrote:

> This only implements support for shallow mirror tarballs, not shallow
> clones.
>

Shallow clones from upstream are not implemented, as, first, you can't
shallow clone from a commit, only the HEAD of a branch, and second, we'd
have to hardcode the exact depth from branch HEAD to SRCREV (this is in
addition to clone depth beyond SRCREV which is covered by this patch
series).

The mirror tarball filename includes branch, revision, and depth. To enable,
> use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific
> URLs. This variable can hold either a clone depth (e.g. `1` to just get
> SRCREV), or a ref or commit, in which case we keep history up to that
> commit.
>
> The shallow support will not, at this time, function correctly for
> linux-yocto, due to its branching scheme & validation.
>

It'd be possible to support linux-yocto, but only if all the dependent
branches were called out in the URI, so they all get retained.

Example:
>
>     BB_GIT_SHALLOW ?= "1"
>     BB_GIT_SHALLOW_pn-linux-yocto = ""
>     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
>     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
>
> This implementation will attempt to fetch a full mirror tarball if it was
> unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow
> tarball will be emitted.
>

I'm not 100% happy with this implementation, at the very least it needs a
little refactoring, to split out a new function or two, and make it a bit
more pythonic, but it might not even be the right approach, which is why
it's an RFC.

The shallow depth control is a variable rather than a URL parameter because
I did the latter initially before realizing that obviously making it a url
parameter would change checksums and cause pointless rebuilds, so the
variable is the best approach there. Opinions on the name of the variable
are welcome. Also open to thoughts on BB_GIT_SHALLOW_<name> vs
BB_GIT_SHALLOW[<name>].

I did attempt to prototype a new fetcher which acted as a premirror for the
git fetcher to handle the shallow logic, but ran into a number of issues. I
don't think any of them are unsurmountable, but it was enough to push me
into trying it this way first:

- The premirror fetcher would need to handle the fetching from the mirrors
itself, since the mirror tarball logic isn't nested, it's only used against
the main urldata, not the mirror urldata.
    - I think this would be fairly straightforward, just a matter of
calling try_mirrors() in download(), with appropriate arguments and
metadata.
- The wrong unpack method gets called. We'd need the premirror fetcher's
unpack method to be called, not the main fetcher's unpack method.
    - I think this could be addressed by passing in the mirror urldata into
the main fetcher's download/unpack methods, so it could call the mirror
unpack() to do the real work instead of handling it itself for cases like
this.
- The try_mirror_uri mirrortarball logic would have to remain, to ensure
that the main fetcher's download method doesn't get called in this context,
which is okay, but then we hit an issue with the symlinking that happens
otherwise, which is used to cover file:// mirrors -- we end up with the
shallow tarball symlinked to ${DL_DIR}/git2/<path>.
    - If we pass the mirror urldata into the main fetcher methods,
presumably we could ditch the try_mirror_uri mirrortarball logic the way I
do in this series, and then the main fetcher download method could simply
punt if it doesn't know how to handle the mirror in question (e.g. not a
non-shallow tarball)
    - Regarding the symlinking, I think we could trivially address that by
making sure the localpath basenames are the same for both urls, and if not,
don't symlink.

So we'll need to consider whether it'd be better to try a prototype of that
approach, addressing the issues, or continue along this path.

All that said, this does do the job, and seemingly does it quite well, so
it's possible we could use this and then improve the underlying
implementation as a secondary step (though, admittedly, doing that tends to
result in never coming back to do it better :).
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20150814/aa5104cc/attachment-0002.html>