<div dir="ltr"><div class="gmail_extra"><div class="gmail_extra">Some more notes on this implementation:</div><div class="gmail_extra"><br></div><div class="gmail_quote">On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <span dir="ltr"><<a href="mailto:kergoth@gmail.com" target="_blank">kergoth@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div id=":1jd" class="a3s" style="overflow:hidden">This only implements support for shallow mirror tarballs, not shallow clones.<br></div></blockquote><div><br></div><div>Shallow clones from upstream are not implemented, as, first, you can't shallow clone from a commit, only the HEAD of a branch, and second, we'd have to hardcode the exact depth from branch HEAD to SRCREV (this is in addition to clone depth beyond SRCREV which is covered by this patch series).</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div id=":1jd" class="a3s" style="overflow:hidden">
The mirror tarball filename includes branch, revision, and depth. To enable,<br>
use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific<br>
URLs. This variable can hold either a clone depth (e.g. `1` to just get<br>
SRCREV), or a ref or commit, in which case we keep history up to that commit.<br>
<br>
The shallow support will not, at this time, function correctly for<br>
linux-yocto, due to its branching scheme & validation.<br></div></blockquote><div><br></div><div>It'd be possible to support linux-yocto, but only if all the dependent branches were called out in the URI, so they all get retained.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div id=":1jd" class="a3s" style="overflow:hidden">
Example:<br>
<br>
BB_GIT_SHALLOW ?= "1"<br>
BB_GIT_SHALLOW_pn-linux-yocto = ""<br>
BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"<br>
BB_GIT_SHALLOW_pn-testrepo = "testbranch"<br>
<br>
This implementation will attempt to fetch a full mirror tarball if it was<br>
unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and<br>
`BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow<br>
tarball will be emitted.<br></div></blockquote></div><div class="gmail_extra"><br></div>I'm not 100% happy with this implementation, at the very least it needs a little refactoring, to split out a new function or two, and make it a bit more pythonic, but it might not even be the right approach, which is why it's an RFC.<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">The shallow depth control is a variable rather than a URL parameter because I did the latter initially before realizing that obviously making it a url parameter would change checksums and cause pointless rebuilds, so the variable is the best approach there. Opinions on the name of the variable are welcome. Also open to thoughts on BB_GIT_SHALLOW_<name> vs BB_GIT_SHALLOW[<name>].</div><div class="gmail_extra"><br></div><div class="gmail_extra">I did attempt to prototype a new fetcher which acted as a premirror for the git fetcher to handle the shallow logic, but ran into a number of issues. I don't think any of them are unsurmountable, but it was enough to push me into trying it this way first:</div><div class="gmail_extra"><br></div><div class="gmail_extra">- The premirror fetcher would need to handle the fetching from the mirrors itself, since the mirror tarball logic isn't nested, it's only used against the main urldata, not the mirror urldata.</div><div class="gmail_extra"> - I think this would be fairly straightforward, just a matter of calling try_mirrors() in download(), with appropriate arguments and metadata.</div><div class="gmail_extra">- The wrong unpack method gets called. We'd need the premirror fetcher's unpack method to be called, not the main fetcher's unpack method.</div><div class="gmail_extra"> - I think this could be addressed by passing in the mirror urldata into the main fetcher's download/unpack methods, so it could call the mirror unpack() to do the real work instead of handling it itself for cases like this.</div><div class="gmail_extra">- The try_mirror_uri mirrortarball logic would have to remain, to ensure that the main fetcher's download method doesn't get called in this context, which is okay, but then we hit an issue with the symlinking that happens otherwise, which is used to cover file:// mirrors -- we end up with the shallow tarball symlinked to ${DL_DIR}/git2/<path>.</div><div class="gmail_extra"> - If we pass the mirror urldata into the main fetcher methods, presumably we could ditch the try_mirror_uri mirrortarball logic the way I do in this series, and then the main fetcher download method could simply punt if it doesn't know how to handle the mirror in question (e.g. not a non-shallow tarball)</div><div class="gmail_extra"> - Regarding the symlinking, I think we could trivially address that by making sure the localpath basenames are the same for both urls, and if not, don't symlink.</div><div class="gmail_extra"><br></div><div class="gmail_extra">So we'll need to consider whether it'd be better to try a prototype of that approach, addressing the issues, or continue along this path.</div><div class="gmail_extra"><br></div><div class="gmail_extra">All that said, this does do the job, and seemingly does it quite well, so it's possible we could use this and then improve the underlying implementation as a secondary step (though, admittedly, doing that tends to result in never coming back to do it better :).</div><div class="gmail_extra">-- <br><div class="gmail_signature"><div dir="ltr">Christopher Larson<br>kergoth at gmail dot com<br>Founder - BitBake, OpenEmbedded, OpenZaurus<br>Maintainer - Tslib<br>Senior Software Engineer, Mentor Graphics</div></div>
</div></div>