[bitbake-devel] Issue with bitbake fetch/unpack when using MIRRORS rewrite

[Urs Fässler]

On Tue, 2018-07-17 at 15:00 +0100, richard.purdie at linuxfoundation.org
wrote:
> On Tue, 2018-07-17 at 15:39 +0200, Urs Fässler wrote:
> > You can reproduce the problem with a current (rocko) Yocto. Add the
> > following lines in local.conf:
> >   BB_GIT_SHALLOW = "1"
> >   BB_GENERATE_MIRROR_TARBALLS = "1"
> >   PREMIRRORS += "git://git.yoctoproject.org/.*
> > git://git.yoctoproject.org/git/PATH;protocol=https \n"
> > 
> > and execute
> >   bitbake fstests -c unpack
> > 
> > You should get something like:
> >   tar -xzf
> > .../build/downloads/gitshallow_git.yoctoproject.org.fstests_e5939ff
> > -
> > 1_master.tar.gz failed with exit code 2, output:
> >   tar (child):
> > .../build/downloads/gitshallow_git.yoctoproject.org.fstests_e5939ff
> > -
> > 1_master.tar.gz: Cannot open: No such file or directory
> > 
> > What happens is that the download step generates the tarball:
> >   gitshallow_git.yoctoproject.org.git.fstests_e5939ff-
> > 1_master.tar.gz
> > but the unpack step expects the tarball:
> >   gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz
> > 
> > The difference in the tarball names come from the different url
> > used
> > in the recipe and when rewritten according to PREMIRRORS.
> 
> Is this just a problem with shallow clones or with git recipes in
> general and the above was just a simple/fast example?

With shallow clones the problem exists always when using PREMIRROS.

The problem exists with full clones and MIRRORS, but is difficult to
reproduce. We think that the problem may depend the circumstances of
the first download (i.e. if the upstream or mirrored server is
available). It may also depend on the order of execution of the steps.
I will further investigate into this.

> > The symlink solution would add a symlink from
> > gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz to
> > gitshallow_git.yoctoproject.org.git.fstests_e5939ff-
> > 1_master.tar.gz.
> > 
> > The recipe-url solution would name the tarball
> > gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz.
> > 
> > > > Now the question is which solution we should implement. For us,
> > > > it
> > > > is
> > > > the second one (tarball naming after recipe-url). It comes with
> > > > the
> > > > downside that the one mentioned test fails and has to be
> > > > removed.
> > > > In
> > > > a
> > > > real scenario this results in downloading a repository twice
> > > > and
> > > > having
> > > > 2 tarballs with the same content. But I expect this to be
> > > > unlikely
> > > > in
> > > > a
> > > > real world scenario.
> > > > 
> > > > A third solution may be that we add a link between the download
> > > > and
> > > > unpack task. But this would be the most intrusive solution for
> > > > Bitbake.
> > > 
> > > I'm more than a little concerned about the symlink comment since
> > > the
> > > fetcher assumes that symlinks work in other places too.
> > 
> > Sorry for concerning you. I think it is no issue. We generate the
> > tarballs and archive them on a system without symlinks. Then we get
> > the tarballs over http with the help of a premirror rule. We do it
> > as
> > described in the Bitbake manual chapter "The Download (Fetch)". I
> > expect this to be a fairly common use case.
> > 
> > Another rationale for the recipe-url solution is that the mirrors
> > are
> > used when the server from the recipe-url is not available. When we
> > generate a tarball, it would be strange that the name of it depends
> > on some local conditions (closed ports, local mirror rewrite rules,
> > ...) rather than the recipe.
> > This probably invalidates my argument that the symlink solution is
> > nice since it has the same method for naming as the git clone
> > naming.
> > This are 2 quite different use cases.
> > 
> > > Also, do you have any new test cases to add which illustrate it?
> > 
> > I have a test but it is a bit tricky since there are 2 issues. The
> > one you see (as mentioned above with Yocto) is actually not the
> > real
> > problem but an issue that is only seen in this scenario.
> 
> Could we have separate test cases for the two issues?
>
> > Since I am now quite sure that the solution where we use the
> > recipe-
> > url for the tarball name is the correct one, I like to reformulate
> > my
> > Question: Do you see a reason why it is a bad solution?
> > 
> > If you don't see a problem I will implements this behavior. The
> > failing test will be replaced with other tests that capture the new
> > behavior.
> 
> For better/worse, the current fetcher behaviour with mirrors was to
> use
> symlinks to represent the path its resolved through. We use them in a
> variety of cirumstances such as when referencing local file://
> mirrors
> we can't write to.
> 
> My instinct is therefore to have all the fetchers behave the same way
> with regard to how DL_DIR works. Having consistency is important and
> multiple code paths with different behaviour has caused us problems
> in
> the past.
> 
> At the back of my mind there is some thinking that the symlink
> approach
> does protect us against some potential data duplication in the
> downloads directory but I'm out of time to work through that thought
> right now.

You are right, it may duplicate some tarballs. This happens when you
have different urls in the recipe that end up with the same url by
mirror rewrite rules. I expect this to be a rare case.

> Can you confirm if the normal git fetcher has the issue or its just
> shallow clones as that is probably important?

git shallow tarball: always a problem with rewrite rules, easy to
reproduce
git full tarball: seldom a problem, difficult to reproduce
git repo clone: no problem seen

> I'd be interested to see your alternative patches too if you have
> them.

I am working on the patches for the shallow tarball.

Regards,
Urs