[bitbake-devel] [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()

Matt Hoosier matt.hoosier at gmail.com
Thu May 17 19:17:04 UTC 2018 

Right; I got enough context from your copied/pasted failure report to
duplicate it. I'm just wondering if there are other test batteries besides
bitbake-selftest and oe-selftest that are gates to clear?

-Matt

On Thu, May 17, 2018 at 11:28 AM, Burton, Ross <ross.burton at intel.com>
wrote:

> Hi Matt,
>
> If you run oe-selftest -r devtool then it will run the tests which are
> failing.
>
> Ross
>
> On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier at gmail.com> wrote:
> > On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton at intel.com> wrote:
> >>
> >> Sorry, but:
> >>
> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
> >> test_devtool_add_fetch_git (devtool.DevtoolTests)
> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
> >> ----------------------------------------------------------------------
> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
> >> call last):
> >>   File
> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
> >> line 32, in wrapped_f
> >>     return func(*args, **kwargs)
> >>   File
> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
> >> line 403, in test_devtool_add_fetch_git
> >>     result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
> >> srcdir, url_rev))
> >>   File
> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
> >> line 191, in runCmd
> >>     raise AssertionError("Command '%s' returned non-zero exit status
> >> %d:\n%s" % (command, result.status, exc_output))
> >> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
> >> -f
> >> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a
> 5d058e581d"
> >> -V 1.5' returned non-zero exit status 1:
> >> ...
> >>
> >> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
> >>
> >> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/build/tmp/sysroots-uninative/
> x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/
> yocto-worker/nightly-oe-selftest/build/scripts:/home/
> pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
> selftest/build/build/tmp/work/recipetool-utucnynk/work/
> recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/
> pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
> selftest/build/build/tmp/work/recipetool-utucnynk/work/
> recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/
> yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/
> recipetool-utucnynk/work/recipe-sysroot-native/usr/
> sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/build/tmp/work/recipetool-
> utucnynk/work/recipe-sysroot-native/usr/bin:/home/
> pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
> selftest/build/build/tmp/work/recipetool-utucnynk/work/
> recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/build/tmp/work/recipetool-
> utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/
> yocto-autobuilder/yocto-worker/nightly-oe-selftest/
> build/bitbake/bin:/home/pokybuild/yocto-autobuilder/
> yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools";
> >> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
> >> submodule update --init --recursive  --no-fetch failed with exit code
> >> 1, output:
> >> Submodule 'doxygen2jsdoc'
> >> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path
> >> 'doxygen2jsdoc'
> >> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c587
> 0097e1f008
> >> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
> >> submodule path 'doxygen2jsdoc'
> >>
> >> Ross
> >>
> >> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier at gmail.com> wrote:
> >> > Although the submodules' histories have been fetched during the
> >> > do_fetch() phase, the mechanics used to clone the workdir copy
> >> > of the repo haven't been transferring the actual .git/modules
> >> > directory from the repo fetched into downloads/ during the
> >> > fetch task.
> >> >
> >> > Fix that, and for good measure also explicitly tell Git to avoid
> >> > hitting the network during do_unpack() of the submodules.
> >> >
> >> > [YOCTO #12739]
> >> >
> >> > Signed-off-by: Matt Hoosier <matt.hoosier at gmail.com>
> >> > ---
> >> >  lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
> >> >  1 file changed, 27 insertions(+), 1 deletion(-)
> >> >
> >> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> >> > index 0aff1008..c5019e98 100644
> >> > --- a/lib/bb/fetch2/gitsm.py
> >> > +++ b/lib/bb/fetch2/gitsm.py
> >> > @@ -132,4 +132,30 @@ class GitSM(Git):
> >> >
> >> >          if self.uses_submodules(ud, d, ud.destdir):
> >> >              runfetchcmd(ud.basecmd + " checkout " +
> >> > ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> >> > -            runfetchcmd(ud.basecmd + " submodule update --init
> >> > --recursive", d, workdir=ud.destdir)
> >> > +
> >> > +            updateflags = ""
> >> > +
> >> > +            # Copy over the submodules' fetched histories too.
> >> > +            if ud.bareclone:
> >> > +                repo_conf = ud.destdir
> >> > +            else:
> >> > +                repo_conf = os.path.join(ud.destdir, '.git')
> >> > +
> >> > +            if os.path.exists(ud.clonedir):
> >> > +                # This is not a copy unpacked from a shallow mirror
> >> > clone. So
> >> > +                # the manual intervention to populate the
> .git/modules
> >> > done
> >> > +                # in clone_shallow_local() won't have been done yet.
> >> > +                runfetchcmd("cp -fpPRH %s %s" %
> >> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> >> > +                updateflags += " --no-fetch"
> >> > +            elif os.path.exists(os.path.join(repo_conf, 'modules')):
> >> > +                # Unpacked from a shallow mirror clone. Manual
> >> > population of
> >> > +                # .git/modules is already done.
> >> > +                updateflags += " --no-fetch"
> >> > +            else:
> >> > +                # This isn't fatal; git-submodule will just fetch it
> >> > +                # during do_unpack().
> >> > +                bb.error("submodule history not retrieved during
> >> > do_fetch()")
> >> > +
> >> > +            # Careful not to hit the network during unpacking; all
> >> > history should already
> >> > +            # be fetched.
> >> > +            runfetchcmd(ud.basecmd + " submodule update --init
> >> > --recursive %s" % updateflags, d, workdir=ud.destdir)
> >> > --
> >> > 2.13.6
> >> >
> >> > --
> >> > _______________________________________________
> >> > bitbake-devel mailing list
> >> > bitbake-devel at lists.openembedded.org
> >> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
> >
> >
> > Okay, I'll have a look at that. Are there any other test suites I should
> > know about?
> >>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20180517/c4bcb87b/attachment-0002.html>

More information about the bitbake-devel mailing list