[bitbake-devel] [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()

[Burton, Ross]

Sorry, but:

2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
test_devtool_add_fetch_git (devtool.DevtoolTests)
2018-05-14 18:57:44,968 - oe-selftest - INFO -
----------------------------------------------------------------------
2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
line 403, in test_devtool_add_fetch_git
    result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
srcdir, url_rev))
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
line 191, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status
%d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
-f "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d"
-V 1.5' returned non-zero exit status 1:
...

ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools";
export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
submodule update --init --recursive  --no-fetch failed with exit code
1, output:
Submodule 'doxygen2jsdoc'
(https://github.com/arfoll/doxygen2jsdoc.git) registered for path
'doxygen2jsdoc'
fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008
Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
submodule path 'doxygen2jsdoc'

Ross

On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier at gmail.com> wrote:
> Although the submodules' histories have been fetched during the
> do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
>
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
>
> [YOCTO #12739]
>
> Signed-off-by: Matt Hoosier <matt.hoosier at gmail.com>
> ---
>  lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
>  1 file changed, 27 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..c5019e98 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,30 @@ class GitSM(Git):
>
>          if self.uses_submodules(ud, d, ud.destdir):
>              runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> -            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
> +
> +            updateflags = ""
> +
> +            # Copy over the submodules' fetched histories too.
> +            if ud.bareclone:
> +                repo_conf = ud.destdir
> +            else:
> +                repo_conf = os.path.join(ud.destdir, '.git')
> +
> +            if os.path.exists(ud.clonedir):
> +                # This is not a copy unpacked from a shallow mirror clone. So
> +                # the manual intervention to populate the .git/modules done
> +                # in clone_shallow_local() won't have been done yet.
> +                runfetchcmd("cp -fpPRH %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> +                updateflags += " --no-fetch"
> +            elif os.path.exists(os.path.join(repo_conf, 'modules')):
> +                # Unpacked from a shallow mirror clone. Manual population of
> +                # .git/modules is already done.
> +                updateflags += " --no-fetch"
> +            else:
> +                # This isn't fatal; git-submodule will just fetch it
> +                # during do_unpack().
> +                bb.error("submodule history not retrieved during do_fetch()")
> +
> +            # Careful not to hit the network during unpacking; all history should already
> +            # be fetched.
> +            runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir)
> --
> 2.13.6
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel