[bitbake-devel] [PATCH v2 1/2] gitsm.py: use download cache for git submodules too

[Mikko.Rapeli at bmw.de]

Hi,

On Wed, Sep 06, 2017 at 08:19:55AM -0700, Christopher Larson wrote:
> On Wed, Sep 6, 2017 at 5:33 AM, Mikko Rapeli <mikko.rapeli at bmw.de> wrote:
> 
> > Currently download caches for recipes using gitsm are filled also with
> > the submodule repositories but when the git submodule tree is reused from
> > cache, only the base repository is cloned. After a source URL rewrite,
> > 'git submodule update --init --recursive' is called in the tree
> > which downloads the needed submodules from upstream URL's.
> > This wastes bandwidth since download cache already had the needed
> > commits for the submodule repositories.
> >
> > With this change, also the submodule repositories are copied to the
> > work space from download cache and the download from upstream can
> > be avoided. Using plain 'cp -a' since git cloning doesn't work for the
> > submodule directories.
> >
> 
> This behavior is both already tracked by a yocto bug, and blindly copying
> will result in breakage when the branch changes in such a way that the
> submodules change. I already submitted an equivalent patch a week or two
> ago and then decided against its inclusion, as it just runs into other
> failures, specifically there was an oe-selftest that fails if you had
> previously fetched a different branch of the sources in question, as the
> upstream url of one of the submodules changes between branches, and we
> don’t re-fetch existing submodules when their urls change. We definitely
> need to fix this, but this fixes one problem and introduces others.

In short: git submodules suck.

git exposes all this yuckyness to users who are confused too if their local
submodule repositories stop working between commits.

> See https://bugzilla.yoctoproject.org/show_bug.cgi?id=11830 and
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=11831 for reference.

Ok. I'll use my patch for now until something better comes along
which avoids cloning tens of repositories on every build.

As for testing, it's possible to setup fake http_proxy environment variable
and break e.g. http connections to github.com, or in containers to break
DNS resolution using /etc/hosts or all network connectivity.

Thanks for pushing to fix this issue.

-Mikko