[bitbake-devel] [PATCH] Fix gitsm networking and mirroring

[Mark Hatle]

On 9/21/18 9:15 AM, Mikko.Rapeli at bmw.de wrote:
> On Fri, Sep 21, 2018 at 09:04:10AM -0500, Mark Hatle wrote:
>> On 9/21/18 2:02 AM, Mikko.Rapeli at bmw.de wrote:
>>> Looks good from what I understand of the patch, but I have a question about
>>> git/gitsm fetchers:
>>>
>>> Is it possible that two recipes end up modifying the same file e.g. via
>>> hard links when they use git submodules and share one of the repositories?
>>
>> The symlinks (not hard links in this case) are between the components from the
>> submodule and whatever the fetcher returned.
>>
>> The only modified element(s) are the references to the submodules within the
>> bare repositories.
>>
>> Assuming all users use the same bare repositories, it's my
>> understanding/expectation that the code in the 'downloads' will be the same.
>> The only difference will then be in working dir.
>>
>>> I'm seeing very odd build failures on sumo and it looks like
>>> recipes are either mixing up files in their sysroots or seeing each others
>>> patches on top of the shared git submodule tree. These happen when building
>>> the whole system and rebuilding affected recipes alone makes the problem
>>> go away.
>>
>> Before or after my patch?  The gitsm changes here should ONLY affect the
>> repository and any submodules affected.  The code affects the fetch and unpack
>> behaviors of the system.  So no patches would be applied to the tree.. (well
>> they are not supposed to be!)
> 
> Before, on sumo and I was wondering if your patch could fix it.
> 
> But with help from #yocto I've now debugged this into openssl do_configure
> and am testing a patch. With luck my problem had nothing to do with git
> submodules.
> 
>> The previous (current) gitsm fetcher makes it's submodules present using either
>> copies or git submodule updates.  In the case of a copy, it might be possible
>> for one workdir to have a symlink back to the download or other shared location.
>>  (I never encountered this, so this is purely speculation.)
>>
>> If you can get me a gitsm reproducer for the problems you were seeing, I can run
>> them through this code and see if I can make it work.
>>
>> (As an aside, I think git submodules are one of the worst things ever.  I really
>> hate them...)
>>
>>> With svn fetcher we see the same, but that's one of the problems we knew about
>>> from past. svn fetcher does not lock the download cached tree correctly
>>> and if multiple recipes use the same repo, one of them can see the repo
>>> in an inconsistent state and fail with various errors. Our workaround
>>> is to switch to http/s fetcher instead.
>>
>> With my patch set, I'm not doing an explicit locking.  The download process
>> itself, should be locked by the existing fetch2 system, and git fetcher.  The
>> module setup/symlinking process might be slightly different.
>>
>> One place there COULD be an issue is if two different recipes refer to the same
>> gitsm 'source', but different srcrev, and the different srcrev point to
>> different submodules.  I don't know if this is really a concern in practice though.
> 
> Given my luck, we will hit this sooner or later. I think with svn fetcher we
> have already seen cases like this. In the best case build fails and in the
> worst case wrong things end up in the build...

I looked through my code.  There is an iteration step that happens in the
unpack.  If the module being asked for (in a given revision) is not present it
will raise an exception.  This should prevent the 'wrong things' at least.

--Mark

> -Mikko
>