[bitbake-devel] [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()
Matt Hoosier
matt.hoosier at gmail.com
Fri May 25 13:47:27 UTC 2018
These test cases failures are fixed up in the v5 submission:
http://lists.openembedded.org/pipermail/bitbake-devel/2018-May/009350.html.
On Thu, May 17, 2018 at 2:17 PM, Matt Hoosier <matt.hoosier at gmail.com>
wrote:
> Right; I got enough context from your copied/pasted failure report to
> duplicate it. I'm just wondering if there are other test batteries besides
> bitbake-selftest and oe-selftest that are gates to clear?
>
> -Matt
>
>
> On Thu, May 17, 2018 at 11:28 AM, Burton, Ross <ross.burton at intel.com>
> wrote:
>
>> Hi Matt,
>>
>> If you run oe-selftest -r devtool then it will run the tests which are
>> failing.
>>
>> Ross
>>
>> On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier at gmail.com> wrote:
>> > On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton at intel.com> wrote:
>> >>
>> >> Sorry, but:
>> >>
>> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
>> >> test_devtool_add_fetch_git (devtool.DevtoolTests)
>> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
>> >> ----------------------------------------------------------------------
>> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
>> >> call last):
>> >> File
>> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
>> selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
>> >> line 32, in wrapped_f
>> >> return func(*args, **kwargs)
>> >> File
>> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
>> selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
>> >> line 403, in test_devtool_add_fetch_git
>> >> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
>> >> srcdir, url_rev))
>> >> File
>> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
>> selftest/build/meta/lib/oeqa/utils/commands.py",
>> >> line 191, in runCmd
>> >> raise AssertionError("Command '%s' returned non-zero exit status
>> >> %d:\n%s" % (command, result.status, exc_output))
>> >> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
>> >> -f
>> >> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e43
>> 30ccfdd9a5d058e581d"
>> >> -V 1.5' returned non-zero exit status 1:
>> >> ...
>> >>
>> >> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
>> >>
>> >> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly
>> -oe-selftest/build/build/tmp/sysroots-uninative/x86_64-
>> linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-
>> worker/nightly-oe-selftest/build/scripts:/home/pokybuild/
>> yocto-autobuilder/yocto-worker/nightly-oe-selftest/
>> build/build/tmp/work/recipetool-utucnynk/work/recipe-
>> sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/
>> yocto-autobuilder/yocto-worker/nightly-oe-selftest/
>> build/build/tmp/work/recipetool-utucnynk/work/recipe-
>> sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-
>> autobuilder/yocto-worker/nightly-oe-selftest/build/
>> build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-
>> native/usr/sbin:/home/pokybuild/yocto-autobuilder/
>> yocto-worker/nightly-oe-selftest/build/build/tmp/work/
>> recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:
>> /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/
>> build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-
>> native/sbin:/home/pokybuild/yocto-autobuilder/yocto-
>> worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/
>> recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/
>> nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/
>> yocto-autobuilder/yocto-worker/nightly-oe-selftest/
>> build/build/tmp/hosttools";
>> >> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
>> >> submodule update --init --recursive --no-fetch failed with exit code
>> >> 1, output:
>> >> Submodule 'doxygen2jsdoc'
>> >> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path
>> >> 'doxygen2jsdoc'
>> >> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c587
>> 0097e1f008
>> >> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
>> >> submodule path 'doxygen2jsdoc'
>> >>
>> >> Ross
>> >>
>> >> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier at gmail.com> wrote:
>> >> > Although the submodules' histories have been fetched during the
>> >> > do_fetch() phase, the mechanics used to clone the workdir copy
>> >> > of the repo haven't been transferring the actual .git/modules
>> >> > directory from the repo fetched into downloads/ during the
>> >> > fetch task.
>> >> >
>> >> > Fix that, and for good measure also explicitly tell Git to avoid
>> >> > hitting the network during do_unpack() of the submodules.
>> >> >
>> >> > [YOCTO #12739]
>> >> >
>> >> > Signed-off-by: Matt Hoosier <matt.hoosier at gmail.com>
>> >> > ---
>> >> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
>> >> > 1 file changed, 27 insertions(+), 1 deletion(-)
>> >> >
>> >> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
>> >> > index 0aff1008..c5019e98 100644
>> >> > --- a/lib/bb/fetch2/gitsm.py
>> >> > +++ b/lib/bb/fetch2/gitsm.py
>> >> > @@ -132,4 +132,30 @@ class GitSM(Git):
>> >> >
>> >> > if self.uses_submodules(ud, d, ud.destdir):
>> >> > runfetchcmd(ud.basecmd + " checkout " +
>> >> > ud.revisions[ud.names[0]], d, workdir=ud.destdir)
>> >> > - runfetchcmd(ud.basecmd + " submodule update --init
>> >> > --recursive", d, workdir=ud.destdir)
>> >> > +
>> >> > + updateflags = ""
>> >> > +
>> >> > + # Copy over the submodules' fetched histories too.
>> >> > + if ud.bareclone:
>> >> > + repo_conf = ud.destdir
>> >> > + else:
>> >> > + repo_conf = os.path.join(ud.destdir, '.git')
>> >> > +
>> >> > + if os.path.exists(ud.clonedir):
>> >> > + # This is not a copy unpacked from a shallow mirror
>> >> > clone. So
>> >> > + # the manual intervention to populate the
>> .git/modules
>> >> > done
>> >> > + # in clone_shallow_local() won't have been done yet.
>> >> > + runfetchcmd("cp -fpPRH %s %s" %
>> >> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
>> >> > + updateflags += " --no-fetch"
>> >> > + elif os.path.exists(os.path.join(repo_conf,
>> 'modules')):
>> >> > + # Unpacked from a shallow mirror clone. Manual
>> >> > population of
>> >> > + # .git/modules is already done.
>> >> > + updateflags += " --no-fetch"
>> >> > + else:
>> >> > + # This isn't fatal; git-submodule will just fetch it
>> >> > + # during do_unpack().
>> >> > + bb.error("submodule history not retrieved during
>> >> > do_fetch()")
>> >> > +
>> >> > + # Careful not to hit the network during unpacking; all
>> >> > history should already
>> >> > + # be fetched.
>> >> > + runfetchcmd(ud.basecmd + " submodule update --init
>> >> > --recursive %s" % updateflags, d, workdir=ud.destdir)
>> >> > --
>> >> > 2.13.6
>> >> >
>> >> > --
>> >> > _______________________________________________
>> >> > bitbake-devel mailing list
>> >> > bitbake-devel at lists.openembedded.org
>> >> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>> >
>> >
>> > Okay, I'll have a look at that. Are there any other test suites I should
>> > know about?
>> >>
>> >>
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20180525/542936b4/attachment-0002.html>
More information about the bitbake-devel
mailing list