[bitbake-devel] [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()

[Matt Hoosier]

On Fri, May 11, 2018 at 7:28 AM, Burton, Ross <ross.burton at intel.com> wrote:

> This breaks bitbake-selftest:
>
> $ bitbake-selftest  bb.tests.fetch.GitShallowTest
> ........................E
>
> ERROR: test_shallow_submodules (bb.tests.fetch.GitShallowTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
> 1353, in test_shallow_submodules
>     fetcher, ud = self.fetch_shallow(uri)
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
> 1198, in fetch_shallow
>     fetcher, ud = self.fetch_and_unpack(uri)
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
> 1179, in fetch_and_unpack
>     fetcher.unpack(self.d.getVar('WORKDIR'))
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/__init__.py", line
> 1762, in unpack
>     ud.method.unpack(ud, root, self.d)
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/gitsm.py", line
> 141, in unpack
>     runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
> 'modules'), repo_conf), d)
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/__init__.py", line
> 885, in runfetchcmd
>     raise FetchError(error_message)
> bb.fetch2.FetchError: Fetcher failure: Fetch command export
> PSEUDO_DISABLED=1; cp -pr
> /tmp/tmpxi2ydemw/download/git2/tmp.tmpxi2ydemw.gitsource/modules
> /tmp/tmpxi2ydemw/git/.git failed with exit code 1, output:
> cp: cannot stat
> '/tmp/tmpxi2ydemw/download/git2/tmp.tmpxi2ydemw.gitsource/modules': No
> such file or directory
>
> Ross
>

Thanks; I'd not considered shallow mirrored clones; rev2 fixes this.


>
> On 10 May 2018 at 03:33, Matt Hoosier <matt.hoosier at gmail.com> wrote:
> > Although the submodules' histories have been fetched during the
> > do_fetch() phase, the mechanics used to clone the workdir copy
> > of the repo haven't been transferring the actual .git/modules
> > directory from the repo fetched into downloads/ during the
> > fetch task.
> >
> > Fix that, and for good measure also explicitly tell Git to avoid
> > hitting the network during do_unpack() of the submodules.
> >
> > [YOCTO #12739]
> >
> > Signed-off-by: Matt Hoosier <matt.hoosier at gmail.com>
> > ---
> >  lib/bb/fetch2/gitsm.py | 12 +++++++++++-
> >  1 file changed, 11 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > index 0aff1008..1f3fc443 100644
> > --- a/lib/bb/fetch2/gitsm.py
> > +++ b/lib/bb/fetch2/gitsm.py
> > @@ -132,4 +132,14 @@ class GitSM(Git):
> >
> >          if self.uses_submodules(ud, d, ud.destdir):
> >              runfetchcmd(ud.basecmd + " checkout " +
> ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> > -            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive", d, workdir=ud.destdir)
> > +
> > +            # Copy over the submodules' fetched histories too.
> > +            if ud.bareclone:
> > +                repo_conf = ud.destdir
> > +            else:
> > +                repo_conf = os.path.join(ud.destdir, '.git')
> > +            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
> 'modules'), repo_conf), d)
> > +
> > +            # Careful not to hit the network during unpacking; all
> history should already
> > +            # be fetched.
> > +            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive --no-fetch", d, workdir=ud.destdir)
> > --
> > 2.13.6
> >
> > --
> > _______________________________________________
> > bitbake-devel mailing list
> > bitbake-devel at lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20180511/0fd3cc00/attachment-0002.html>