[bitbake-devel] [PATCH] Fix gitsm networking and mirroring

[Mark Hatle]

On 9/21/18 2:02 AM, Mikko.Rapeli at bmw.de wrote:
> Looks good from what I understand of the patch, but I have a question about
> git/gitsm fetchers:
> 
> Is it possible that two recipes end up modifying the same file e.g. via
> hard links when they use git submodules and share one of the repositories?

The symlinks (not hard links in this case) are between the components from the
submodule and whatever the fetcher returned.

The only modified element(s) are the references to the submodules within the
bare repositories.

Assuming all users use the same bare repositories, it's my
understanding/expectation that the code in the 'downloads' will be the same.
The only difference will then be in working dir.

> I'm seeing very odd build failures on sumo and it looks like
> recipes are either mixing up files in their sysroots or seeing each others
> patches on top of the shared git submodule tree. These happen when building
> the whole system and rebuilding affected recipes alone makes the problem
> go away.

Before or after my patch?  The gitsm changes here should ONLY affect the
repository and any submodules affected.  The code affects the fetch and unpack
behaviors of the system.  So no patches would be applied to the tree.. (well
they are not supposed to be!)

The previous (current) gitsm fetcher makes it's submodules present using either
copies or git submodule updates.  In the case of a copy, it might be possible
for one workdir to have a symlink back to the download or other shared location.
 (I never encountered this, so this is purely speculation.)

If you can get me a gitsm reproducer for the problems you were seeing, I can run
them through this code and see if I can make it work.

(As an aside, I think git submodules are one of the worst things ever.  I really
hate them...)

> With svn fetcher we see the same, but that's one of the problems we knew about
> from past. svn fetcher does not lock the download cached tree correctly
> and if multiple recipes use the same repo, one of them can see the repo
> in an inconsistent state and fail with various errors. Our workaround
> is to switch to http/s fetcher instead.

With my patch set, I'm not doing an explicit locking.  The download process
itself, should be locked by the existing fetch2 system, and git fetcher.  The
module setup/symlinking process might be slightly different.

One place there COULD be an issue is if two different recipes refer to the same
gitsm 'source', but different srcrev, and the different srcrev point to
different submodules.  I don't know if this is really a concern in practice though.

--Mark

> -Mikko
>