[bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs

[Richard Purdie]

On Mon, 2020-02-24 at 21:46 +0000, Paul Barker wrote:
> On Mon, 24 Feb 2020 at 21:32, Richard Purdie
> <richard.purdie at linuxfoundation.org> wrote:
> > On Mon, 2020-02-24 at 20:16 +0000, Paul Barker wrote:
> > > When a shallow mirror tarball is used to satisfy a gitsm URI it
> > > needs to
> > > be unpacked so that the .gitmodules file can be examined.
> > > 
> > > Signed-off-by: Paul Barker <pbarker at konsulko.com>
> > > ---
> > >  lib/bb/fetch2/gitsm.py | 6 ++++++
> > >  1 file changed, 6 insertions(+)
> > > 
> > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > > index aa121cbe..98f5dae5 100644
> > > --- a/lib/bb/fetch2/gitsm.py
> > > +++ b/lib/bb/fetch2/gitsm.py
> > > @@ -169,6 +169,12 @@ class GitSM(Git):
> > >                  raise
> > > 
> > >          Git.download(self, ud, d)
> > > +
> > > +        # If we're using a shallow mirror tarball it needs to be
> > > unpacked so
> > > +        # that we can examine the .gitmodules file
> > > +        if ud.shallow and os.path.exists(ud.fullshallow) and
> > > Git.need_update(self, ud, d):
> > > +            bb.utils.mkdirhier(ud.clonedir)
> > > +            runfetchcmd("tar -xzf %s" % ud.fullshallow, d,
> > > workdir=ud.clonedir)
> > >          self.process_submodules(ud, ud.clonedir,
> > > download_submodule, d)
> > > 
> > >      def unpack(self, ud, destdir, d):
> > 
> > Does this mean we're missing a test?
> 
> Possibly. Maybe a two step test with a recipe using a `gitsm://` URL:
> 1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS,
> BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror
> tarballs out of the downloads directory to a local mirror directory.
> Throw away DL_DIR, SSTATE_DIR and TMPDIR.
> 2) Using the mirror generated in step 1 try to fetch and unpack the
> recipe.
> 
> Does that sound like a sensible test? I think it would have to go in
> oeqa rather than the bitbake tests if we do it that way.

It sounds like it might be reasonable, I'm not sure why we can't do
that in lib/bb/tests/fetch.py though? There are other tests which do
similar sequences of commands.

You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a
datastore and files in between but I can't see why it would need to be
in oeqa?

Sorry to push testing btw but the git fetcher is one of our better
tested pieces of code so I do want to try and maintain good test
coverage!

Cheers,

Richard