[bitbake-devel] gitsm fetching fails

[Stefan Bigler]

Hi Mark

Thank you for the explanation and sorry for the late response from my side.
With your help I was successful.

> On 11/11/18 3:04 PM, Stefan Bigler wrote:
> > Hi
> > I'm updating from rocko to thud and recognized, that my projects with
> > git-submodules cannot be fetched anymore.
> > I tried to debug the fetch task, but I failed. I did a small example
you should
> > have access and it should be easy to reproduce, the repo should be
public.
> > The recipe works fine with rocko but fails with the latest master
branch of poky.
> > I'm using gitlab for my repos. When I use https it works also with
thud, only
> > when *ssh* is used then it fails.
> > Below you find the recipe, the console output and the important part of
the
> > log.do_fetch
> > I hope somebody can give me a hint.

> The previous fetcher handed off everything to git and did not do all of
the
> validation and network caching/fetching like it should have.  This hid a
lot of
> problems and prevented the system from being able to use declared mirror
resources.
>
> The new fetcher iterates over the items and calls the regular git fetcher
to
> pull down the various components.  This ensures that the download
directory and
> mirrors are used appropriately.
>
> Below the message:

> > ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: The URL:
> > 'gitsm:Bigler/gitsm-test-submodule.git;protocol=git at gitlab.com

> indicates that the submodule is not defined properly.  If this is legal,
then it
> may require some kind of a translation that does not exist.

> See: http://git.openembedded.org/bitbake/tree/lib/bb/fetch2/gitsm.py

> Look at lines 86 though 96.  This is where the submodule URLs are
constructed.
> The current code is expected the submodules to be defined with URIs, NOT
SSH
> 'strings'.

> i.e.:
> http://user@host/path/file
> ssh://user@host/path/file
> NOT
> user at host:path/file

> It appears that this submodule reference is using the second format,
instead of
> the actual URI format.  gitlab must be doing something different if it
works on
> http but not ssh, since typically the .gitmodules are declared in a
singular
> format by the author of the repository and don't change, even if the fetch
> method of the initial repository changes.

If I change the .gitmodules to

[submodule "lib"]
    path = lib
-    url =       git at gitlab.com:gitsm/gitsm-test-submodule.git
+    url = ssh://git@gitlab.com/gitsm/gitsm-test-submodule.git

then it works as expected

> Since you have a way to reproduce this, I'd suggest working through that
small
> section of code and trying to determine if it's an invalid URI (SSH
style) and
> then translating it to a URI for this processing.  (If you can make a
reproducer
> publicly available, then I can try to fix it instead.)

The repos are available here
https://gitlab.com/gitsm/gitsm-test/tree/master

> Looking at the git.py, it says the following protocols are supported:
> "git", "http", "https", "file", "ssh" and "rsync"

> So checking the protocol field for one of those, failing it find it
should go
> into a translation step most likely.  The result (I would assume) will
either be
> file or ssh.  If there is a ':', then most likely it's an ssh.. and the
rules to
> make this into a proper URI should be fairly simple.

With the patch below I *hardcoded* the gitlab format and it works as well,
but of course all other will fail.
I do not know if there is a good way to detect if secondary format is used
and then do the conversion.
Anyway, I can live with the standard format and I can change it in my repos.

bitbake/lib/bb/fetch2/gitsm.py
@@ -88,8 +88,17 @@ class GitSM(Git):
             module_hash = module_hash.split()[2]

             # Build new SRC_URI
-            proto = uris[module].split(':', 1)[0]
-            url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
+            proto = "ssh"
+            url   = "gitsm://"
+            url  += uris[module].replace(':', '/', 1)
+
             url += ';protocol=%s' % proto

-- Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20181115/c94c7849/attachment-0001.html>