[bitbake-devel] git submodule fetcher regression

[Bonnans, Laurent]

Hello Stefan,


On 10/04/2018 06:58 PM, Mark Hatle wrote:
> On 10/4/18 11:45 AM, Stefan Agner wrote:
>> I guess you refer to:
>> fetch2/gitsm: checkout submodules recursively
>> fetch2/gitsm: handle submodules nested inside subdirs
>>
>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
>>
>> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
>>
>> I did use -c cleanall before, which hopefully should be sufficient?
This should be solved in meta-updater 
8a44c8a26966d0cb9b57ea1ef4c6f25c2a944dd8 and beyond. I believe it's the 
same issue reported by Jens Rehsack on Oct 1.
We had a simple workaround in this case, so we did not investigate the 
core bitbake issue at that time.
> If you are still having issues from master-next (bitbake), can you distill down
> a reproducer recipe?
>
> All it takes is a minimal recipe with a SRC_URI line that used to work for you
> through the do_unpack stage.  (I don't care about anything after do_unpack...)
>
> typical reproducer may look like YP Bug 12941:
>
> SUMMARY  = "Library to access Blu-Ray disk"
> SECTION = "misc"
> HOMEPAGE = "http://videolan.org"
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"
>
> DEPENDS = "libxml2"
>
> SRC_URI = "gitsm://git.videolan.org/libbluray.git"
>
> SRCREV = "${AUTOREV}"
>
>
> --Mark
>
>
Hello Mark, here is a short reproducer (aktualizr.bb)


SUMMARY = "Aktualizr SOTA Client"
DESCRIPTION = "SOTA Client application written in C++"
HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
SECTION = "base"
LICENSE = "MPL-2.0"
LIC_FILES_CHKSUM = 
"file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"

PV = "1.0+git${SRCPV}"

SRC_URI = " \
   gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
   "
SRCREV = "79246b930a1b5b6f956b69f74f1f0c62060061df"
BRANCH ?= "master"


Then, `aktualizr -c unpack` fails with the same error Stefan described.

The root cause is this submodule declaration:

[submodule "third_party/HdrHistogram_c"]
     path = third_party/HdrHistogram_c
     url = https://github.com/cajun-rat/HdrHistogram_c

Which is set to the commit 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae, 
current HEAD of the branch `fix-const-cast` but not merged in 
cajun-rat's master.

Please let me know if you need more details.


Also, while investigating these issues, I noticed that `cleanall` now 
does not remove the fetched submodules like it would be expected. So, 
running `fetch` again only download the main project and will reuse 
cached submodules repositories if up to date.


Laurent