[bitbake-devel] [PATCH] Fix gitsm networking and mirroring

[Mikko.Rapeli at bmw.de]

On Fri, Sep 21, 2018 at 09:04:10AM -0500, Mark Hatle wrote:
> On 9/21/18 2:02 AM, Mikko.Rapeli at bmw.de wrote:
> > Looks good from what I understand of the patch, but I have a question about
> > git/gitsm fetchers:
> > 
> > Is it possible that two recipes end up modifying the same file e.g. via
> > hard links when they use git submodules and share one of the repositories?
> 
> The symlinks (not hard links in this case) are between the components from the
> submodule and whatever the fetcher returned.
> 
> The only modified element(s) are the references to the submodules within the
> bare repositories.
> 
> Assuming all users use the same bare repositories, it's my
> understanding/expectation that the code in the 'downloads' will be the same.
> The only difference will then be in working dir.
> 
> > I'm seeing very odd build failures on sumo and it looks like
> > recipes are either mixing up files in their sysroots or seeing each others
> > patches on top of the shared git submodule tree. These happen when building
> > the whole system and rebuilding affected recipes alone makes the problem
> > go away.
> 
> Before or after my patch?  The gitsm changes here should ONLY affect the
> repository and any submodules affected.  The code affects the fetch and unpack
> behaviors of the system.  So no patches would be applied to the tree.. (well
> they are not supposed to be!)

Before, on sumo and I was wondering if your patch could fix it.

But with help from #yocto I've now debugged this into openssl do_configure
and am testing a patch. With luck my problem had nothing to do with git
submodules.

> The previous (current) gitsm fetcher makes it's submodules present using either
> copies or git submodule updates.  In the case of a copy, it might be possible
> for one workdir to have a symlink back to the download or other shared location.
>  (I never encountered this, so this is purely speculation.)
> 
> If you can get me a gitsm reproducer for the problems you were seeing, I can run
> them through this code and see if I can make it work.
> 
> (As an aside, I think git submodules are one of the worst things ever.  I really
> hate them...)
>
> > With svn fetcher we see the same, but that's one of the problems we knew about
> > from past. svn fetcher does not lock the download cached tree correctly
> > and if multiple recipes use the same repo, one of them can see the repo
> > in an inconsistent state and fail with various errors. Our workaround
> > is to switch to http/s fetcher instead.
> 
> With my patch set, I'm not doing an explicit locking.  The download process
> itself, should be locked by the existing fetch2 system, and git fetcher.  The
> module setup/symlinking process might be slightly different.
> 
> One place there COULD be an issue is if two different recipes refer to the same
> gitsm 'source', but different srcrev, and the different srcrev point to
> different submodules.  I don't know if this is really a concern in practice though.

Given my luck, we will hit this sooner or later. I think with svn fetcher we
have already seen cases like this. In the best case build fails and in the
worst case wrong things end up in the build...

-Mikko