<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 6, 2017 at 8:45 AM, <span dir="ltr"><<a href="mailto:Mikko.Rapeli@bmw.de" target="_blank">Mikko.Rapeli@bmw.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<span class=""><br>
On Wed, Sep 06, 2017 at 08:19:55AM -0700, Christopher Larson wrote:<br>
> On Wed, Sep 6, 2017 at 5:33 AM, Mikko Rapeli <<a href="mailto:mikko.rapeli@bmw.de">mikko.rapeli@bmw.de</a>> wrote:<br>
><br>
> > Currently download caches for recipes using gitsm are filled also with<br>
> > the submodule repositories but when the git submodule tree is reused from<br>
> > cache, only the base repository is cloned. After a source URL rewrite,<br>
> > 'git submodule update --init --recursive' is called in the tree<br>
> > which downloads the needed submodules from upstream URL's.<br>
> > This wastes bandwidth since download cache already had the needed<br>
> > commits for the submodule repositories.<br>
> ><br>
> > With this change, also the submodule repositories are copied to the<br>
> > work space from download cache and the download from upstream can<br>
> > be avoided. Using plain 'cp -a' since git cloning doesn't work for the<br>
> > submodule directories.<br>
> ><br>
><br>
> This behavior is both already tracked by a yocto bug, and blindly copying<br>
> will result in breakage when the branch changes in such a way that the<br>
> submodules change. I already submitted an equivalent patch a week or two<br>
> ago and then decided against its inclusion, as it just runs into other<br>
> failures, specifically there was an oe-selftest that fails if you had<br>
> previously fetched a different branch of the sources in question, as the<br>
> upstream url of one of the submodules changes between branches, and we<br>
> don’t re-fetch existing submodules when their urls change. We definitely<br>
> need to fix this, but this fixes one problem and introduces others.<br>
<br>
</span>In short: git submodules suck.<br>
<br>
git exposes all this yuckyness to users who are confused too if their local<br>
submodule repositories stop working between commits.<br></blockquote><div><br></div><div>Yeah, submodules are a mess all around, for a number of reasons, but most of the alternatives have their own idiosyncrasies :)</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
> See <a href="https://bugzilla.yoctoproject.org/show_bug.cgi?id=11830" rel="noreferrer" target="_blank">https://bugzilla.yoctoproject.<wbr>org/show_bug.cgi?id=11830</a> and<br>
> <a href="https://bugzilla.yoctoproject.org/show_bug.cgi?id=11831" rel="noreferrer" target="_blank">https://bugzilla.yoctoproject.<wbr>org/show_bug.cgi?id=11831</a> for reference.<br>
<br>
</span>Ok. I'll use my patch for now until something better comes along<br>
which avoids cloning tens of repositories on every build.<br>
<br>
As for testing, it's possible to setup fake http_proxy environment variable<br>
and break e.g. http connections to <a href="http://github.com" rel="noreferrer" target="_blank">github.com</a>, or in containers to break<br>
DNS resolution using /etc/hosts or all network connectivity.<br>
<br>
Thanks for pushing to fix this issue.<br></blockquote><div><br></div><div>No problem. Thinking about it again, I’d support merging your patches as a workaround for now. It introduces other issues, but those other issues are likely more minor than re-fetching everything in do_unpack. So we can merge this and use it until the rewritten fetcher gets merged. </div></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Christopher Larson<br>kergoth at gmail dot com<br>Founder - BitBake, OpenEmbedded, OpenZaurus<br>Senior Software Engineer, Mentor Graphics</div></div></div></div>
</div></div>