[bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs

[Christopher Larson]

I don't think this is right. This unpacks the shallow tarball into
ud.clonedir. The fetcher is specifically done the way it is to avoid
ud.clonedir ever being shallow so as to avoid having to maintain that state
and convert it when necessary.

On Mon, Feb 24, 2020 at 3:29 PM Paul Barker <pbarker at konsulko.com> wrote:

> On Mon, 24 Feb 2020 at 22:10, Richard Purdie
> <richard.purdie at linuxfoundation.org> wrote:
> >
> > On Mon, 2020-02-24 at 21:46 +0000, Paul Barker wrote:
> > > On Mon, 24 Feb 2020 at 21:32, Richard Purdie
> > > <richard.purdie at linuxfoundation.org> wrote:
> > > > On Mon, 2020-02-24 at 20:16 +0000, Paul Barker wrote:
> > > > > When a shallow mirror tarball is used to satisfy a gitsm URI it
> > > > > needs to
> > > > > be unpacked so that the .gitmodules file can be examined.
> > > > >
> > > > > Signed-off-by: Paul Barker <pbarker at konsulko.com>
> > > > > ---
> > > > >  lib/bb/fetch2/gitsm.py | 6 ++++++
> > > > >  1 file changed, 6 insertions(+)
> > > > >
> > > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > > > > index aa121cbe..98f5dae5 100644
> > > > > --- a/lib/bb/fetch2/gitsm.py
> > > > > +++ b/lib/bb/fetch2/gitsm.py
> > > > > @@ -169,6 +169,12 @@ class GitSM(Git):
> > > > >                  raise
> > > > >
> > > > >          Git.download(self, ud, d)
> > > > > +
> > > > > +        # If we're using a shallow mirror tarball it needs to be
> > > > > unpacked so
> > > > > +        # that we can examine the .gitmodules file
> > > > > +        if ud.shallow and os.path.exists(ud.fullshallow) and
> > > > > Git.need_update(self, ud, d):
> > > > > +            bb.utils.mkdirhier(ud.clonedir)
> > > > > +            runfetchcmd("tar -xzf %s" % ud.fullshallow, d,
> > > > > workdir=ud.clonedir)
> > > > >          self.process_submodules(ud, ud.clonedir,
> > > > > download_submodule, d)
> > > > >
> > > > >      def unpack(self, ud, destdir, d):
> > > >
> > > > Does this mean we're missing a test?
> > >
> > > Possibly. Maybe a two step test with a recipe using a `gitsm://` URL:
> > > 1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS,
> > > BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror
> > > tarballs out of the downloads directory to a local mirror directory.
> > > Throw away DL_DIR, SSTATE_DIR and TMPDIR.
> > > 2) Using the mirror generated in step 1 try to fetch and unpack the
> > > recipe.
> > >
> > > Does that sound like a sensible test? I think it would have to go in
> > > oeqa rather than the bitbake tests if we do it that way.
> >
> > It sounds like it might be reasonable, I'm not sure why we can't do
> > that in lib/bb/tests/fetch.py though? There are other tests which do
> > similar sequences of commands.
> >
> > You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a
> > datastore and files in between but I can't see why it would need to be
> > in oeqa?
>
> Ok, I've looked more closely at GitMakeShallowTest in
> lib/bb/tests/fetch.py. We could write a new
> test_shallow_submodule_mirrors function based on
> test_shallow_submodules and test_shallow_mirrors.
>
> The case I would want to catch is do_unpack failing. I guess we could
> check that fetcher.unpack() works as a proxy for that.
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>


-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20200224/3b397171/attachment.html>