[bitbake-devel] git submodule fetcher regression

[Stefan Agner]

On 12.10.2018 16:50, Mark Hatle wrote:
> On 10/12/18 8:50 AM, Stefan Agner wrote:
>> Hi Mark, Hi Laurent,
>>
>> On 05.10.2018 10:38, Bonnans, Laurent wrote:
>>> Hello Stefan,
>>>
>>>
>>> On 10/04/2018 06:58 PM, Mark Hatle wrote:
>>>> On 10/4/18 11:45 AM, Stefan Agner wrote:
>>>>> I guess you refer to:
>>>>> fetch2/gitsm: checkout submodules recursively
>>>>> fetch2/gitsm: handle submodules nested inside subdirs
>>>>>
>>>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
>>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
>>>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
>>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
>>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
>>>>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
>>>>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
>>>>>
>>>>> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
>>>>>
>>>>> I did use -c cleanall before, which hopefully should be sufficient?
>>> This should be solved in meta-updater
>>> 8a44c8a26966d0cb9b57ea1ef4c6f25c2a944dd8 and beyond. I believe it's the
>>> same issue reported by Jens Rehsack on Oct 1.
>>> We had a simple workaround in this case, so we did not investigate the
>>> core bitbake issue at that time.
>>>> If you are still having issues from master-next (bitbake), can you distill down
>>>> a reproducer recipe?
>>>>
>>>> All it takes is a minimal recipe with a SRC_URI line that used to work for you
>>>> through the do_unpack stage.  (I don't care about anything after do_unpack...)
>>>>
>>>> typical reproducer may look like YP Bug 12941:
>>>>
>>>> SUMMARY  = "Library to access Blu-Ray disk"
>>>> SECTION = "misc"
>>>> HOMEPAGE = "http://videolan.org"
>>>> LICENSE = "GPLv2"
>>>> LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"
>>>>
>>>> DEPENDS = "libxml2"
>>>>
>>>> SRC_URI = "gitsm://git.videolan.org/libbluray.git"
>>>>
>>>> SRCREV = "${AUTOREV}"
>>>>
>>>>
>>>> --Mark
>>>>
>>>>
>>> Hello Mark, here is a short reproducer (aktualizr.bb)
>>>
>>>
>>> SUMMARY = "Aktualizr SOTA Client"
>>> DESCRIPTION = "SOTA Client application written in C++"
>>> HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
>>> SECTION = "base"
>>> LICENSE = "MPL-2.0"
>>> LIC_FILES_CHKSUM =
>>> "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"
>>>
>>> PV = "1.0+git${SRCPV}"
>>>
>>> SRC_URI = " \
>>>    gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
>>>    "
>>> SRCREV = "79246b930a1b5b6f956b69f74f1f0c62060061df"
>>> BRANCH ?= "master"
>>>
>>>
>>> Then, `aktualizr -c unpack` fails with the same error Stefan described.
>>>
>>> The root cause is this submodule declaration:
>>>
>>> [submodule "third_party/HdrHistogram_c"]
>>>      path = third_party/HdrHistogram_c
>>>      url = https://github.com/cajun-rat/HdrHistogram_c
>>>
>>> Which is set to the commit 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae,
>>> current HEAD of the branch `fix-const-cast` but not merged in
>>> cajun-rat's master.
>>>
>>> Please let me know if you need more details.
>>>
>>>
>>> Also, while investigating these issues, I noticed that `cleanall` now
>>> does not remove the fetched submodules like it would be expected. So,
>>> running `fetch` again only download the main project and will reuse
>>> cached submodules repositories if up to date.
>>
>> Any update on this?
> 
> I don't have an immediate solution for the clear issue.  I'm hoping someone else
> might have an idea there.

Sorry, I meant the issue above, where fetching fails...

Fetching that hash used to work before the gitsm rework, so this is a
regression...

--
Stefan

> 
> Since there repository is up-to-date (it has the commit id of what the submodule
> wants) it should be safe for most users, unless they truly need to refetch for
> another reason.
> 
> I'd consider this a low priority issue, but definitely a bug.
> 
> --Mark
> 
>> Still the case in bitbake master-next...
>>
>> --
>> Stefan
>>