[bitbake-devel] [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces at lists.openembedded.org [mailto:bitbake-
> devel-bounces at lists.openembedded.org] On Behalf Of Matt Hoosier
> Sent: den 11 maj 2018 21:27
> To: bitbake-devel at lists.openembedded.org
> Subject: [bitbake-devel] [PATCH] fetch/gitsm: avoid live submodule
> fetching during unpack()
> 
> Although the submodules' histories have been fetched during the
> do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
> 
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
> 
> Rev3 changes:
> 
> - Really fix the shallow-clone issue claimed to be addressed in rev2.
> 
> Rev2 changes:
> 
> - Fixed a failure when unpacking a shallow cloned mirror repository.

The Rev2 and Rev3 changes information should go below the --- below 
as it should not be part of the commit message as it relates to the 
patch submission process rather than the actual change. If any of it 
is relevant for the commit message, then the commit message should 
be changed as well.

> [YOCTO #12739]
> 
> Signed-off-by: Matt Hoosier <matt.hoosier at gmail.com>
> ---
>  lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
>  1 file changed, 27 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..3969f2f5 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,30 @@ class GitSM(Git):
> 
>          if self.uses_submodules(ud, d, ud.destdir):
>              runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> -            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
> +
> +            updateflags = ""
> +
> +            # Copy over the submodules' fetched histories too.
> +            if ud.bareclone:
> +                repo_conf = ud.destdir
> +            else:
> +                repo_conf = os.path.join(ud.destdir, '.git')
> +
> +            if os.path.exists(ud.clonedir):
> +                # This is not a copy unpacked from a shallow mirror clone. So
> +                # the manual intervention to populate the .git/modules done
> +                # in clone_shallow_local() won't have been done yet.
> +                runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> +                updateflags += " --no-fetch"
> +            elif os.path.exists(os.path.join(repo_conf, 'modules')):
> +                # Unpacked from a shallow mirror clone. Manual population of
> +                # .git/modules is already done.
> +                updateflags += " --no-fetch"
> +            else:
> +                # This isn't fatal; git-submodule will just fetch it
> +                # during do_unpack().
> +                bb.error("submodule history not retrieved during do_fetch()")
> +
> +            # Careful not to hit the network during unpacking; all history should already
> +            # be fetched.
> +            runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir)
> --
> 2.13.6

//Peter