[bitbake-devel] gitsm Fetcher update

Mon Oct 1 17:18:19 UTC 2018

> Am 01.10.2018 um 15:34 schrieb Mark Hatle <mark.hatle at windriver.com>:
> 
> On 10/1/18 3:52 AM, Jens Rehsack | External Miltenyi Biotec wrote:
>> Hi Mark,
>> 
>> after your last update to gitsm fetcher, it seems it cannot handle submodules not using the master branch correctly:
> 
> I'm not sure I understand below what is going wrong.
> 
> Does it not checkout the primary repository in the correct branch, or does this
> get checked out -- and it's the submodules that get 'confused' between the
> pointed to commit and the branches?

The primary repository is - as far as I can see from the logs - checked out correctly.
It just tries to apply the branch 'master' to the submodule. The primary repository has
the branch 'develop' - what makes me a bit curios.

> The system recursively processes the git submodules, while interrogating it's
> way through the system.  If the failure is on a submode (not the main
> repository) it would be something in this block...
> 
> http://git.openembedded.org/bitbake/tree/lib/bb/fetch2/gitsm.py
> 
> 47 - 108
> 
> Looking at the code, line 81.
> 
>            # Build new SRC_URI
>            proto = uris[module].split(':', 1)[0]
>            url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
>            url += ';protocol=%s' % proto
>            url += ";name=%s" % module
>            url += ";qbareclone=1;nocheckout=1"
> 
> 
> It looks like a 'q' accidentally made it into the patch.  Can you remove this
> and try it?   This is where the system constructs SRC_URI for the recursive
> (submodule) download.  The next block defines the SRCREV and related items and
> re-calls fetch.

Will do tomorrow when I'm back in company and report back.

> Nowhere in there do we call or set anything related to the branches.  Maybe
> "nobranch=1" needs to be added to the generated SRC_URI?

I tried that and this ended up in wrong names for the archive files created
for the download cache (see below).

> If you can give me a recipe that used to work through 'do_unpack', then I should
> be able to attempt to replicate it locally and work through the problems if the
> above isn't enough of a place to look.

Unfortunately we have only proprietary recipes using that. I can try create
one for a perl module I maintain, if you like.
You will need meta-cpan for that - can you confirm that meta-cpan works
for you in general (https://github.com/meta-cpan <https://github.com/meta-cpan>)?

> (It also shows a missing test case, one that we need to add -- once we
> understand what part and why it's failing.)
> 
> --Mark

Best regards,
Jens

>> --- BEGIN INITIAL ERROR
>> WARNING: automacs-ui-git-r0 do_fetch: Failed to fetch URL gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8, attempting MIRRORS if available
>> ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://git@mgegit.miltenyibiotec.de:7999/cpsm/cmake-modules.git;protocol=ssh;name=cmake/modules;qbareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>> ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8'. Unable to fetch URL from any source.
>> ERROR: automacs-ui-git-r0 do_fetch: Function failed: base_do_fetch
>> ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_fetch.5452
>> ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_fetch) failed with exit code '1'
>> 
>> DEBUG: Running export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*.miltenyibiotec.de,*.miltenyibiotec.com"; export https_proxy="http://proxy2:8080/"; export http_proxy="http://proxy2:8080/"; export PATH="/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/sno/gpw-community-bsp/sources/poky/scripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot/opt/miltenyi/bin/crossscripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/bin:/home/sno/gpw-community-bsp/sources/poky/bitbake/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/hosttools"; export HOME="/home/sno"; git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master 2> /dev/null | wc -l
>> --- END INITIAL ERROR
>> 
>> I cloned the appropriate project locally to see what's going on:
>> 
>> sno at yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list
>> * (HEAD detached at c6c0526)
>>  develop
>> sno at yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master
>> sno at yoctopoc:~/prj/automacs-ui/cmake/modules$
>> 
>> The I tried to reverse engineer the gitsm fetcher and added:
>> 
>> poky$ git diff
>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>> index 51f8d0e68c..221af934c3 100644
>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>> @@ -79,6 +79,7 @@ class GitSM(Git):
>>             url += ';protocol=%s' % proto
>>             url += ";name=%s" % module
>>             url += ";qbareclone=1;nocheckout=1"
>> +            url += ";nobranch=1"
>> 
>>             ld = d.createCopy()
>>             # Not necessary to set SRC_URI, since we're passing the URI to
>> 
>> which results into
>> 
>> --- BEGIN FOLLOW UP ERROR
>> ERROR: automacs-ui-git-r0 do_unpack: Fetcher failure: Submodule cmake/modules does not exist in /home/sno/gpw-community-bsp/mops-yocto-platform/downloads/git2/mgegit.miltenyibiotec.de.7999.umsa.automacs-ui.git/modules/cmake/modules or /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/git/.git/modules/cmake/modules.
>> ERROR: automacs-ui-git-r0 do_unpack: Function failed: base_do_unpack
>> ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_unpack.6551
>> ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_unpack) failed with exit code '1'
>> --- END FOLLOW UP ERROR
>> 
>> because it uses the submodule names as archive name:
>> 
>> $ tar xzf downloads/git2_mgegit.miltenyibiotec.de.7999.<TAB>
>> git2_mgegit.miltenyibiotec.de.7999.cpsm.cmake-modules.git.tar.gz
>> git2_mgegit.miltenyibiotec.de.7999.cpsm.mcs-lib.git.tar.gz
>> git2_mgegit.miltenyibiotec.de.7999.umsa.corert.git.tar.gz
>> git2_mgegit.miltenyibiotec.de.7999.umsa.protos.git.tar.gz
>> 
>> Cheers,
>> Jens
>> 
> 

--
Jens Rehsack - rehsack at gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20181001/488ee583/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20181001/488ee583/attachment-0002.sig>