[bitbake-devel] git submodule fetcher regression

[Mark Hatle]

On 10/4/18 11:45 AM, Stefan Agner wrote:
> Hi Laurent,
> 
> On 04.10.2018 18:28, Bonnans, Laurent wrote:
>> Hello Stefan,
>>
>> On 10/04/2018 06:08 PM, Stefan Agner wrote:
>>> Hi,
>>>
>>> It seems that 9c6b39adf978 ("fetch2/gitsm.py: Rework the git submodule fetcher") broke git submodules in some cases. My particular case is aktualizr from meta-updater (https://github.com/advancedtelematic/meta-updater). Fetching aktualizr reliably fails for me now:
>>>
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Fetcher failure: Submodule tests/tuf-test-vectors does not exist in /home/ags/oe/downloads/git2/github.com.advancedtelematic.aktualizr/modules/tests/tuf-test-vectors or
>>> /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/git/.git/modules/tests/tuf-test-vectors.
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Function failed: base_do_unpack
>>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_unpack.35143
>>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_unpack) failed with exit code '1'
>>>
>>> Any idea what that could be?
>>>
>>> --
>>> Stefan
>>
>> We also noticed the problem and supplied a fix to the bitbake-devel 
>> mailing list (actually, there were two separate issues in our case). The 
>> commits are currently sitting in master-next branches of bitbake and 
>> poky and should, in all likelihood, move to master soon.
> 
> I guess you refer to:
> fetch2/gitsm: checkout submodules recursively
> fetch2/gitsm: handle submodules nested inside subdirs
> 
> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
> 
> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
> 
> I did use -c cleanall before, which hopefully should be sufficient?

If you are still having issues from master-next (bitbake), can you distill down
a reproducer recipe?

All it takes is a minimal recipe with a SRC_URI line that used to work for you
through the do_unpack stage.  (I don't care about anything after do_unpack...)

typical reproducer may look like YP Bug 12941:

SUMMARY  = "Library to access Blu-Ray disk"
SECTION = "misc"
HOMEPAGE = "http://videolan.org"
LICENSE = "GPLv2"
LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"

DEPENDS = "libxml2"

SRC_URI = "gitsm://git.videolan.org/libbluray.git"

SRCREV = "${AUTOREV}"


--Mark

> --
> Stefan
>