<div dir="ltr">Hi, Mark:<div><br></div><div>Thanks for the explanation, my understanding was wrong, sorry the noise.</div><div><br></div><div>//Ming Liu</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Stefan Agner <<a href="mailto:stefan@agner.ch">stefan@agner.ch</a>> 於 2019年1月26日 週六 上午11:15寫道：<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[adding Patrick, he is often working on the recipe]<br>
<br>
On 26.01.2019 00:19, Mark Hatle wrote:<br>
> On 1/25/19 1:53 PM, Mark Hatle wrote:<br>
>> Is the repository you are using public? If so, give me the URL and commit ID<br>
>> for the main repository and I'll get it added to the test suite.<br>
> <br>
> I reread this and went back into the prior info from you. It appears the<br>
> repository you are talking about is:<br>
> <br>
> gitsm://<a href="http://github.com/advancedtelematic/aktualizr;branch=master" rel="noreferrer" target="_blank">github.com/advancedtelematic/aktualizr;branch=master</a><br>
<br>
Yes that is the repository.<br>
<br>
> <br>
> SRCREV = d00d1a04cc2366d1a5f143b84b9f507f8bd32c44<br>
<br>
And the hash I was using.<br>
<br>
> <br>
> <br>
> I manually checked this out doing the following:<br>
> <br>
> git clone git://<a href="http://github.com/advancedtelematic/aktualizr" rel="noreferrer" target="_blank">github.com/advancedtelematic/aktualizr</a><br>
> git checkout d00d1a04cc2366d1a5f143b84b9f507f8bd32c44<br>
> git submodule init<br>
> git submodule update --recursive<br>
> <br>
> <br>
> I then ran the same thing using the gitsm fetcher. Using:<br>
> <br>
> <br>
> gitsm://<a href="http://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" rel="noreferrer" target="_blank">github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44</a><br>
> <br>
> Comparing the resulting system, they are nearly identical. All of the checked<br>
> out sources are -exactly- the same.<br>
> <br>
> There are some minor differences in the .git directory, but none of them affect<br>
> what is actually checked out.<br>
> <br>
> Output of:<br>
> <br>
> diff -urN 'gitsm version' aktualizr -x .git<br>
> <br>
> is identical<br>
> <br>
> comparing the git directory is slightly different, but all within the<br>
> expectations.<br>
> <br>
> So I do not see anything that the gitsm fetcher is doing incorrectly<br>
> at this point.<br>
> <br>
> You should run the same type of comparison and see if you are getting the same<br>
> results.<br>
> <br>
> <br>
> BTW I went back to bitbake commit c1fcc46e2498, and repeated the experiment.<br>
> <br>
> This shows that a significant part of was checked out incorrectly.<br>
> <br>
> For instance, the submodule partial/extern/isotp-c:<br>
> <br>
> In the current version is checked out on<br>
> 4c5904bff3ed83ec471c8586b5faed62898d0224. (Which matches the manual<br>
> git behavior.)<br>
> <br>
> While the gitsm fetcher in the older system checkedout commit<br>
> 614fe48fe55aeea98b49804b7e31ef2d304f62de. Which is incorrect.<br>
> <br>
<br>
Yeah this would explain the behavior I am seeing.<br>
<br>
> <br>
> So it seems to be that something in the recipe was expecting semi-broken<br>
> behavior.. and its actually a recipe problem.<br>
<br>
Agreed, in this case we need to fix the recipe.<br>
<br>
Thanks for looking into it and sorry for the noise!<br>
<br>
--<br>
Stefan<br>
<br>
> <br>
> --Mark<br>
> <br>
>> (For testing what I've been doing is checking out with raw git and then with the<br>
>> gitsm fetcher.. comparing the difference and then adding those specific things<br>
>> as regressions checks.)<br>
>><br>
>> If it's not public, then you'll likely need to do that yourself. Compare the<br>
>> results and try to figure out what the difference is. Once you have that, it<br>
>> may be more obvious what the problems. (Comparison should include the .git<br>
>> contents as well..)<br>
>><br>
>> --Mark<br>
>><br>
>> On 1/25/19 1:24 PM, Stefan Agner wrote:<br>
>>> On 25.01.2019 16:08, Richard Purdie wrote:<br>
>>>> On Fri, 2019-01-25 at 15:41 +0100, <a href="mailto:liu.ming50@gmail.com" target="_blank">liu.ming50@gmail.com</a> wrote:<br>
>>>>> From: Ming Liu <<a href="mailto:liu.ming50@gmail.com" target="_blank">liu.ming50@gmail.com</a>><br>
>>>>><br>
>>>>> A following issue was observed with gitsm:<br>
>>>>>> git -c core.fsyncobjectfiles=0 config submodule... failed with exit code 255, output:<br>
>>>>>> error: could not lock config file config: File exists<br>
>>>>><br>
>>>>> Since all git submodules share a config file and Git creates a lock<br>
>>>>> file (.git/config.lock) to synchronize access to it, but Git only tries<br>
>>>>> exactly once and returns a error if it's already locked. This leads to<br>
>>>>> a race condition if there are many 'git config submodule' run in<br>
>>>>> different processes.<br>
>>>>><br>
>>>>> It could be fixed by introducing a bitbake file lock to protect config<br>
>>>>> file from concurrent update from submodules.<br>
>>>>><br>
>>>>> Reported-by: Stefan Agner <<a href="mailto:stefan.agner@toradex.com" target="_blank">stefan.agner@toradex.com</a>><br>
>>>>> Signed-off-by: Ming Liu <<a href="mailto:liu.ming50@gmail.com" target="_blank">liu.ming50@gmail.com</a>><br>
>>>>> ---<br>
>>>>> lib/bb/fetch2/gitsm.py | 16 ++++++++++++----<br>
>>>>> 1 file changed, 12 insertions(+), 4 deletions(-)<br>
>>>>><br>
>>>>> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py<br>
>>>>> index f45546b..9465acb 100644<br>
>>>>> --- a/lib/bb/fetch2/gitsm.py<br>
>>>>> +++ b/lib/bb/fetch2/gitsm.py<br>
>>>>> @@ -183,11 +183,19 @@ class GitSM(Git):<br>
>>>>><br>
>>>>> local_path = newfetch.localpath(url)<br>
>>>>><br>
>>>>> - # Correct the submodule references to the local download version...<br>
>>>>> - runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_path}, d, workdir=ud.destdir)<br>
>>>>> + # Protects the config file from concurrent updates by submodules<br>
>>>>> + # e.g. 'git config submodule' run in different processes may lead<br>
>>>>> + # to a race condition.<br>
>>>>> + lf = bb.utils.lockfile(os.path.join(ud.destdir, "gitsm-config.lock"))<br>
>>>>><br>
>>>>> - if ud.shallow:<br>
>>>>> - runfetchcmd("%(basecmd)s config submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd, 'module': module}, d, workdir=ud.destdir)<br>
>>>>> + try:<br>
>>>>> + # Correct the submodule references to the local download version...<br>
>>>>> + runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_path}, d, workdir=ud.destdir)<br>
>>>>> +<br>
>>>>> + if ud.shallow:<br>
>>>>> + runfetchcmd("%(basecmd)s config submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd, 'module': module}, d, workdir=ud.destdir)<br>
>>>>> + finally:<br>
>>>>> + bb.utils.unlockfile(lf)<br>
>>>>><br>
>>>>> # Ensure the submodule repository is NOT set to bare, since we're checking it out...<br>
>>>>> try:<br>
>>>><br>
>>>> I still don't see why this is necessary after Mark's recent changes.<br>
>>>> These files are in ud.destdir which in OE terms is WORKDIR. Only one<br>
>>>> fetch happens against WORKDIR at a time? Therefore how is this racing?<br>
>>><br>
>>> The git config race we only observed before the recent changes by Mark.<br>
>>><br>
>>> If those changes have the potential to fix this issue, we should test<br>
>>> those first and hold on with this patch.<br>
>>><br>
>>>><br>
>>>> Have you seen this problem against the current master code?<br>
>>><br>
>>> Master as of yesterday (or Wednesday) was not able to fetch the project<br>
>>> (aktualizr).<br>
>>><br>
>>> Even with the latest 2 fixes I am not able to build the (very same)<br>
>>> recipe anymore, it ends up in an error during do_configure, presumably<br>
>>> due to missing submodules:<br>
>>><br>
>>><br>
>>> | -- Configuring done<br>
>>> | CMake Error at partial/CMakeLists.txt:116 (add_library):<br>
>>> | Cannot find source file:<br>
>>> |<br>
>>><br>
>>> | extern/isotp-c/deps/bitfield-c/src/bitfield/8byte.c<br>
>>><br>
>>> |<br>
>>><br>
>>> | Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++<br>
>>> .hm<br>
>>> | .hpp .hxx .in .txx<br>
>>> |<br>
>>> |<br>
>>> | CMake Error at partial/CMakeLists.txt:116 (add_library):<br>
>>> | No SOURCES given to target: isotp<br>
>>> |<br>
>>><br>
>>> Using c1fcc46e2498 ("bitbake: Fix Deprecated warnings from regexs")<br>
>>> works fine.<br>
>>><br>
>>> --<br>
>>> Stefan<br>
>>><br>
>>><br>
>><br>
</blockquote></div>