[bitbake-devel] [PATCH] fetch2/gitsm: fix config file concurrent update race

[Mark Hatle]

On 1/25/19 5:35 AM, Ming Liu wrote:
> Hi, Stefan:
> 
> Could you help confirm this? 
> 
> Hi, Richard:
> 
> May I know how has the issue been fixed?

http://git.openembedded.org/bitbake/commit/?id=346338667edca1f58ace769ad417548da2b8d981

http://git.openembedded.org/bitbake/commit/?id=610dbee5634677f5055e2b36a3043cd197fb8c51

Those two commits will eliminate the place in the code where the error was
coming from.

There are only three remaining calls to git config, and they ONLY happen during
unpack.  unpack is safe since there can not be two unpack tasks running against
the same workdir at the same time without other errors occurring as well.  (That
is invalid.)

--Mark

> I updated the tip to:
> ```
> commit 8c8ecec2a722bc2885e2648d41ac8df07bdf660d
>     gitsmy.py: Fix unpack of submodules of submodules
>     
>     If the submodule is in a subdirectory, it needs to have that structure
>     preserved.  This means the unpack path needs to be in the 'dirname' of the
>     final path -- since the unpack directory name is specified in the URI.
>     
>     Additional specific test cases were added to ensure this is working properly
>     based on two recent error reports.
>     
>     Signed-off-by: Mark Hatle <mark.hatle at windriver.com
> <mailto:mark.hatle at windriver.com>>
>     Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org
> <mailto:richard.purdie at linuxfoundation.org>>
> ```
> on top of that commit, I could see there are "git config submodule" being called
> concurrently in different processes, which does have a race condition, and will
> lead to the error observed by Stefan, although it's not easy to be reproduced.
> 
> //Ming Liu
> 
> 
> 
> Richard Purdie <richard.purdie at linuxfoundation.org
> <mailto:richard.purdie at linuxfoundation.org>> 於 2019年1月25日 週五 下午12:18寫道：
> 
>     On Fri, 2019-01-25 at 11:57 +0100, liu.ming50 at gmail.com
>     <mailto:liu.ming50 at gmail.com> wrote:
>     > From: Ming Liu <liu.ming50 at gmail.com <mailto:liu.ming50 at gmail.com>>
>     >
>     > A following issue was observed with gitsm:
>     > > git -c core.fsyncobjectfiles=0 config submodule... failed with exit code
>     255, output:
>     > > error: could not lock config file config: File exists
>     >
>     > Since all git submodules share a config file and Git creates a lock
>     > file (.git/config.lock) to synchronize access to it, but Git only tries
>     > exactly once and returns a error if it's already locked. This leads to
>     > a race condition if there are many 'git config submodule' run in
>     > different processes.
>     >
>     > It could be fixed by introducing a bitbake file lock to protect config
>     > file from concurrent update from submodules.
>     >
>     > Reported-by: Stefan Agner <stefan.agner at toradex.com
>     <mailto:stefan.agner at toradex.com>>
>     > Signed-off-by: Ming Liu <liu.ming50 at gmail.com <mailto:liu.ming50 at gmail.com>>
>     > ---
>     >  lib/bb/fetch2/gitsm.py | 19 +++++++++++++++----
>     >  1 file changed, 15 insertions(+), 4 deletions(-)
> 
>     This issue should already have been fixed with recent changes to the
>     gitsm fetcher. Could you confirm which version of the code you were
>     seeing this problem with?
> 
>     Cheers,
> 
>     Richard
> 
>