[bitbake-devel] gitsm Fetcher update

[Mark Hatle]

On 10/1/18 3:52 AM, Jens Rehsack | External Miltenyi Biotec wrote:
> Hi Mark,
> 
> after your last update to gitsm fetcher, it seems it cannot handle submodules not using the master branch correctly:

I'm not sure I understand below what is going wrong.

Does it not checkout the primary repository in the correct branch, or does this
get checked out -- and it's the submodules that get 'confused' between the
pointed to commit and the branches?

The system recursively processes the git submodules, while interrogating it's
way through the system.  If the failure is on a submode (not the main
repository) it would be something in this block...

http://git.openembedded.org/bitbake/tree/lib/bb/fetch2/gitsm.py

47 - 108

Looking at the code, line 81.

            # Build new SRC_URI
            proto = uris[module].split(':', 1)[0]
            url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
            url += ';protocol=%s' % proto
            url += ";name=%s" % module
            url += ";qbareclone=1;nocheckout=1"


It looks like a 'q' accidentally made it into the patch.  Can you remove this
and try it?   This is where the system constructs SRC_URI for the recursive
(submodule) download.  The next block defines the SRCREV and related items and
re-calls fetch.

Nowhere in there do we call or set anything related to the branches.  Maybe
"nobranch=1" needs to be added to the generated SRC_URI?

If you can give me a recipe that used to work through 'do_unpack', then I should
be able to attempt to replicate it locally and work through the problems if the
above isn't enough of a place to look.

(It also shows a missing test case, one that we need to add -- once we
understand what part and why it's failing.)

--Mark

> --- BEGIN INITIAL ERROR
> WARNING: automacs-ui-git-r0 do_fetch: Failed to fetch URL gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8, attempting MIRRORS if available
> ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://git@mgegit.miltenyibiotec.de:7999/cpsm/cmake-modules.git;protocol=ssh;name=cmake/modules;qbareclone=1;nocheckout=1'. Unable to fetch URL from any source.
> ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8'. Unable to fetch URL from any source.
> ERROR: automacs-ui-git-r0 do_fetch: Function failed: base_do_fetch
> ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_fetch.5452
> ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_fetch) failed with exit code '1'
> 
> DEBUG: Running export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*.miltenyibiotec.de,*.miltenyibiotec.com"; export https_proxy="http://proxy2:8080/"; export http_proxy="http://proxy2:8080/"; export PATH="/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/sno/gpw-community-bsp/sources/poky/scripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot/opt/miltenyi/bin/crossscripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/bin:/home/sno/gpw-community-bsp/sources/poky/bitbake/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/hosttools"; export HOME="/home/sno"; git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master 2> /dev/null | wc -l
> --- END INITIAL ERROR
> 
> I cloned the appropriate project locally to see what's going on:
> 
> sno at yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list
> * (HEAD detached at c6c0526)
>   develop
> sno at yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master
> sno at yoctopoc:~/prj/automacs-ui/cmake/modules$
> 
> The I tried to reverse engineer the gitsm fetcher and added:
> 
> poky$ git diff
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
> index 51f8d0e68c..221af934c3 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -79,6 +79,7 @@ class GitSM(Git):
>              url += ';protocol=%s' % proto
>              url += ";name=%s" % module
>              url += ";qbareclone=1;nocheckout=1"
> +            url += ";nobranch=1"
> 
>              ld = d.createCopy()
>              # Not necessary to set SRC_URI, since we're passing the URI to
> 
> which results into 
> 
> --- BEGIN FOLLOW UP ERROR
> ERROR: automacs-ui-git-r0 do_unpack: Fetcher failure: Submodule cmake/modules does not exist in /home/sno/gpw-community-bsp/mops-yocto-platform/downloads/git2/mgegit.miltenyibiotec.de.7999.umsa.automacs-ui.git/modules/cmake/modules or /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/git/.git/modules/cmake/modules.
> ERROR: automacs-ui-git-r0 do_unpack: Function failed: base_do_unpack
> ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_unpack.6551
> ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_unpack) failed with exit code '1'
> --- END FOLLOW UP ERROR
> 
> because it uses the submodule names as archive name:
> 
> $ tar xzf downloads/git2_mgegit.miltenyibiotec.de.7999.<TAB>
> git2_mgegit.miltenyibiotec.de.7999.cpsm.cmake-modules.git.tar.gz
> git2_mgegit.miltenyibiotec.de.7999.cpsm.mcs-lib.git.tar.gz
> git2_mgegit.miltenyibiotec.de.7999.umsa.corert.git.tar.gz
> git2_mgegit.miltenyibiotec.de.7999.umsa.protos.git.tar.gz
> 
> Cheers,
> Jens
>