[bitbake-devel] [PATCH v2 1/2] gitsm.py: use download cache for git submodules too

[Christopher Larson]

On Wed, Sep 6, 2017 at 8:45 AM, <Mikko.Rapeli at bmw.de> wrote:

> Hi,
>
> On Wed, Sep 06, 2017 at 08:19:55AM -0700, Christopher Larson wrote:
> > On Wed, Sep 6, 2017 at 5:33 AM, Mikko Rapeli <mikko.rapeli at bmw.de>
> wrote:
> >
> > > Currently download caches for recipes using gitsm are filled also with
> > > the submodule repositories but when the git submodule tree is reused
> from
> > > cache, only the base repository is cloned. After a source URL rewrite,
> > > 'git submodule update --init --recursive' is called in the tree
> > > which downloads the needed submodules from upstream URL's.
> > > This wastes bandwidth since download cache already had the needed
> > > commits for the submodule repositories.
> > >
> > > With this change, also the submodule repositories are copied to the
> > > work space from download cache and the download from upstream can
> > > be avoided. Using plain 'cp -a' since git cloning doesn't work for the
> > > submodule directories.
> > >
> >
> > This behavior is both already tracked by a yocto bug, and blindly copying
> > will result in breakage when the branch changes in such a way that the
> > submodules change. I already submitted an equivalent patch a week or two
> > ago and then decided against its inclusion, as it just runs into other
> > failures, specifically there was an oe-selftest that fails if you had
> > previously fetched a different branch of the sources in question, as the
> > upstream url of one of the submodules changes between branches, and we
> > don’t re-fetch existing submodules when their urls change. We definitely
> > need to fix this, but this fixes one problem and introduces others.
>
> In short: git submodules suck.
>
> git exposes all this yuckyness to users who are confused too if their local
> submodule repositories stop working between commits.
>

Yeah, submodules are a mess all around, for a number of reasons, but most
of the alternatives have their own idiosyncrasies :)


> > See https://bugzilla.yoctoproject.org/show_bug.cgi?id=11830 and
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=11831 for reference.
>
> Ok. I'll use my patch for now until something better comes along
> which avoids cloning tens of repositories on every build.
>
> As for testing, it's possible to setup fake http_proxy environment variable
> and break e.g. http connections to github.com, or in containers to break
> DNS resolution using /etc/hosts or all network connectivity.
>
> Thanks for pushing to fix this issue.
>

No problem. Thinking about it again, I’d support merging your patches as a
workaround for now. It introduces other issues, but those other issues are
likely more minor than re-fetching everything in do_unpack. So we can merge
this and use it until the rewritten fetcher gets merged.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20170906/f208771b/attachment-0002.html>