[bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs

[Paul Barker]

On Mon, 24 Feb 2020 at 22:10, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
>
> On Mon, 2020-02-24 at 21:46 +0000, Paul Barker wrote:
> > On Mon, 24 Feb 2020 at 21:32, Richard Purdie
> > <richard.purdie at linuxfoundation.org> wrote:
> > > On Mon, 2020-02-24 at 20:16 +0000, Paul Barker wrote:
> > > > When a shallow mirror tarball is used to satisfy a gitsm URI it
> > > > needs to
> > > > be unpacked so that the .gitmodules file can be examined.
> > > >
> > > > Signed-off-by: Paul Barker <pbarker at konsulko.com>
> > > > ---
> > > >  lib/bb/fetch2/gitsm.py | 6 ++++++
> > > >  1 file changed, 6 insertions(+)
> > > >
> > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > > > index aa121cbe..98f5dae5 100644
> > > > --- a/lib/bb/fetch2/gitsm.py
> > > > +++ b/lib/bb/fetch2/gitsm.py
> > > > @@ -169,6 +169,12 @@ class GitSM(Git):
> > > >                  raise
> > > >
> > > >          Git.download(self, ud, d)
> > > > +
> > > > +        # If we're using a shallow mirror tarball it needs to be
> > > > unpacked so
> > > > +        # that we can examine the .gitmodules file
> > > > +        if ud.shallow and os.path.exists(ud.fullshallow) and
> > > > Git.need_update(self, ud, d):
> > > > +            bb.utils.mkdirhier(ud.clonedir)
> > > > +            runfetchcmd("tar -xzf %s" % ud.fullshallow, d,
> > > > workdir=ud.clonedir)
> > > >          self.process_submodules(ud, ud.clonedir,
> > > > download_submodule, d)
> > > >
> > > >      def unpack(self, ud, destdir, d):
> > >
> > > Does this mean we're missing a test?
> >
> > Possibly. Maybe a two step test with a recipe using a `gitsm://` URL:
> > 1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS,
> > BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror
> > tarballs out of the downloads directory to a local mirror directory.
> > Throw away DL_DIR, SSTATE_DIR and TMPDIR.
> > 2) Using the mirror generated in step 1 try to fetch and unpack the
> > recipe.
> >
> > Does that sound like a sensible test? I think it would have to go in
> > oeqa rather than the bitbake tests if we do it that way.
>
> It sounds like it might be reasonable, I'm not sure why we can't do
> that in lib/bb/tests/fetch.py though? There are other tests which do
> similar sequences of commands.
>
> You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a
> datastore and files in between but I can't see why it would need to be
> in oeqa?

Ok, I've looked more closely at GitMakeShallowTest in
lib/bb/tests/fetch.py. We could write a new
test_shallow_submodule_mirrors function based on
test_shallow_submodules and test_shallow_mirrors.

The case I would want to catch is do_unpack failing. I guess we could
check that fetcher.unpack() works as a proxy for that.