[bitbake-devel] git submodule fetcher regression

[Mark Hatle]

On 10/12/18 8:50 AM, Stefan Agner wrote:
> Hi Mark, Hi Laurent,
> 
> On 05.10.2018 10:38, Bonnans, Laurent wrote:
>> Hello Stefan,
>>
>>
>> On 10/04/2018 06:58 PM, Mark Hatle wrote:
>>> On 10/4/18 11:45 AM, Stefan Agner wrote:
>>>> I guess you refer to:
>>>> fetch2/gitsm: checkout submodules recursively
>>>> fetch2/gitsm: handle submodules nested inside subdirs
>>>>
>>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
>>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
>>>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
>>>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
>>>>
>>>> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
>>>>
>>>> I did use -c cleanall before, which hopefully should be sufficient?
>> This should be solved in meta-updater 
>> 8a44c8a26966d0cb9b57ea1ef4c6f25c2a944dd8 and beyond. I believe it's the 
>> same issue reported by Jens Rehsack on Oct 1.
>> We had a simple workaround in this case, so we did not investigate the 
>> core bitbake issue at that time.
>>> If you are still having issues from master-next (bitbake), can you distill down
>>> a reproducer recipe?
>>>
>>> All it takes is a minimal recipe with a SRC_URI line that used to work for you
>>> through the do_unpack stage.  (I don't care about anything after do_unpack...)
>>>
>>> typical reproducer may look like YP Bug 12941:
>>>
>>> SUMMARY  = "Library to access Blu-Ray disk"
>>> SECTION = "misc"
>>> HOMEPAGE = "http://videolan.org"
>>> LICENSE = "GPLv2"
>>> LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"
>>>
>>> DEPENDS = "libxml2"
>>>
>>> SRC_URI = "gitsm://git.videolan.org/libbluray.git"
>>>
>>> SRCREV = "${AUTOREV}"
>>>
>>>
>>> --Mark
>>>
>>>
>> Hello Mark, here is a short reproducer (aktualizr.bb)
>>
>>
>> SUMMARY = "Aktualizr SOTA Client"
>> DESCRIPTION = "SOTA Client application written in C++"
>> HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
>> SECTION = "base"
>> LICENSE = "MPL-2.0"
>> LIC_FILES_CHKSUM = 
>> "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"
>>
>> PV = "1.0+git${SRCPV}"
>>
>> SRC_URI = " \
>>    gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
>>    "
>> SRCREV = "79246b930a1b5b6f956b69f74f1f0c62060061df"
>> BRANCH ?= "master"
>>
>>
>> Then, `aktualizr -c unpack` fails with the same error Stefan described.
>>
>> The root cause is this submodule declaration:
>>
>> [submodule "third_party/HdrHistogram_c"]
>>      path = third_party/HdrHistogram_c
>>      url = https://github.com/cajun-rat/HdrHistogram_c
>>
>> Which is set to the commit 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae, 
>> current HEAD of the branch `fix-const-cast` but not merged in 
>> cajun-rat's master.
>>
>> Please let me know if you need more details.
>>
>>
>> Also, while investigating these issues, I noticed that `cleanall` now 
>> does not remove the fetched submodules like it would be expected. So, 
>> running `fetch` again only download the main project and will reuse 
>> cached submodules repositories if up to date.
> 
> Any update on this?

I don't have an immediate solution for the clear issue.  I'm hoping someone else
might have an idea there.

Since there repository is up-to-date (it has the commit id of what the submodule
wants) it should be safe for most users, unless they truly need to refetch for
another reason.

I'd consider this a low priority issue, but definitely a bug.

--Mark

> Still the case in bitbake master-next...
> 
> --
> Stefan
>