[bitbake-devel] [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()
Matt Hoosier
matt.hoosier at gmail.com
Mon May 14 21:59:26 UTC 2018
On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton at intel.com> wrote:
> Sorry, but:
>
> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
> test_devtool_add_fetch_git (devtool.DevtoolTests)
> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
> ----------------------------------------------------------------------
> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
> call last):
> File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
> line 32, in wrapped_f
> return func(*args, **kwargs)
> File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
> line 403, in test_devtool_add_fetch_git
> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
> srcdir, url_rev))
> File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
> line 191, in runCmd
> raise AssertionError("Command '%s' returned non-zero exit status
> %d:\n%s" % (command, result.status, exc_output))
> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
> -f "gitsm://
> git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d"
> -V 1.5' returned non-zero exit status 1:
> ...
>
> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
>
> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools";
> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
> submodule update --init --recursive --no-fetch failed with exit code
> 1, output:
> Submodule 'doxygen2jsdoc'
> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path
> 'doxygen2jsdoc'
> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008
> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
> submodule path 'doxygen2jsdoc'
>
> Ross
>
> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier at gmail.com> wrote:
> > Although the submodules' histories have been fetched during the
> > do_fetch() phase, the mechanics used to clone the workdir copy
> > of the repo haven't been transferring the actual .git/modules
> > directory from the repo fetched into downloads/ during the
> > fetch task.
> >
> > Fix that, and for good measure also explicitly tell Git to avoid
> > hitting the network during do_unpack() of the submodules.
> >
> > [YOCTO #12739]
> >
> > Signed-off-by: Matt Hoosier <matt.hoosier at gmail.com>
> > ---
> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
> > 1 file changed, 27 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > index 0aff1008..c5019e98 100644
> > --- a/lib/bb/fetch2/gitsm.py
> > +++ b/lib/bb/fetch2/gitsm.py
> > @@ -132,4 +132,30 @@ class GitSM(Git):
> >
> > if self.uses_submodules(ud, d, ud.destdir):
> > runfetchcmd(ud.basecmd + " checkout " +
> ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> > - runfetchcmd(ud.basecmd + " submodule update --init
> --recursive", d, workdir=ud.destdir)
> > +
> > + updateflags = ""
> > +
> > + # Copy over the submodules' fetched histories too.
> > + if ud.bareclone:
> > + repo_conf = ud.destdir
> > + else:
> > + repo_conf = os.path.join(ud.destdir, '.git')
> > +
> > + if os.path.exists(ud.clonedir):
> > + # This is not a copy unpacked from a shallow mirror
> clone. So
> > + # the manual intervention to populate the .git/modules
> done
> > + # in clone_shallow_local() won't have been done yet.
> > + runfetchcmd("cp -fpPRH %s %s" %
> (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> > + updateflags += " --no-fetch"
> > + elif os.path.exists(os.path.join(repo_conf, 'modules')):
> > + # Unpacked from a shallow mirror clone. Manual
> population of
> > + # .git/modules is already done.
> > + updateflags += " --no-fetch"
> > + else:
> > + # This isn't fatal; git-submodule will just fetch it
> > + # during do_unpack().
> > + bb.error("submodule history not retrieved during
> do_fetch()")
> > +
> > + # Careful not to hit the network during unpacking; all
> history should already
> > + # be fetched.
> > + runfetchcmd(ud.basecmd + " submodule update --init
> --recursive %s" % updateflags, d, workdir=ud.destdir)
> > --
> > 2.13.6
> >
> > --
> > _______________________________________________
> > bitbake-devel mailing list
> > bitbake-devel at lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Okay, I'll have a look at that. Are there any other test suites I should
know about?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20180514/b56fb390/attachment-0002.html>
More information about the bitbake-devel
mailing list