[bitbake-devel] Issue with bitbake fetch/unpack when using MIRRORS rewrite

[richard.purdie at linuxfoundation.org]

On Tue, 2018-07-17 at 15:39 +0200, Urs Fässler wrote:
> You can reproduce the problem with a current (rocko) Yocto. Add the
> following lines in local.conf:
>   BB_GIT_SHALLOW = "1"
>   BB_GENERATE_MIRROR_TARBALLS = "1"
>   PREMIRRORS += "git://git.yoctoproject.org/.*
> git://git.yoctoproject.org/git/PATH;protocol=https \n"
> 
> and execute
>   bitbake fstests -c unpack
> 
> You should get something like:
>   tar -xzf
> .../build/downloads/gitshallow_git.yoctoproject.org.fstests_e5939ff-
> 1_master.tar.gz failed with exit code 2, output:
>   tar (child):
> .../build/downloads/gitshallow_git.yoctoproject.org.fstests_e5939ff-
> 1_master.tar.gz: Cannot open: No such file or directory
> 
> What happens is that the download step generates the tarball:
>   gitshallow_git.yoctoproject.org.git.fstests_e5939ff-1_master.tar.gz
> but the unpack step expects the tarball:
>   gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz
> 
> The difference in the tarball names come from the different url used
> in the recipe and when rewritten according to PREMIRRORS.

Is this just a problem with shallow clones or with git recipes in
general and the above was just a simple/fast example?

> The symlink solution would add a symlink from
> gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz to
> gitshallow_git.yoctoproject.org.git.fstests_e5939ff-1_master.tar.gz.
>
> The recipe-url solution would name the tarball
> gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz.
>
> > > Now the question is which solution we should implement. For us,
> > > it
> > > is
> > > the second one (tarball naming after recipe-url). It comes with
> > > the
> > > downside that the one mentioned test fails and has to be removed.
> > > In
> > > a
> > > real scenario this results in downloading a repository twice and
> > > having
> > > 2 tarballs with the same content. But I expect this to be
> > > unlikely
> > > in
> > > a
> > > real world scenario.
> > > 
> > > A third solution may be that we add a link between the download
> > > and
> > > unpack task. But this would be the most intrusive solution for
> > > Bitbake.
> > 
> > I'm more than a little concerned about the symlink comment since
> > the
> > fetcher assumes that symlinks work in other places too.
> 
> Sorry for concerning you. I think it is no issue. We generate the
> tarballs and archive them on a system without symlinks. Then we get
> the tarballs over http with the help of a premirror rule. We do it as
> described in the Bitbake manual chapter "The Download (Fetch)". I
> expect this to be a fairly common use case.
> 
> Another rationale for the recipe-url solution is that the mirrors are
> used when the server from the recipe-url is not available. When we
> generate a tarball, it would be strange that the name of it depends
> on some local conditions (closed ports, local mirror rewrite rules,
> ...) rather than the recipe.
> This probably invalidates my argument that the symlink solution is
> nice since it has the same method for naming as the git clone naming.
> This are 2 quite different use cases.
> 
> > Also, do you have any new test cases to add which illustrate it?
> 
> I have a test but it is a bit tricky since there are 2 issues. The
> one you see (as mentioned above with Yocto) is actually not the real
> problem but an issue that is only seen in this scenario.

Could we have separate test cases for the two issues?

> Since I am now quite sure that the solution where we use the recipe-
> url for the tarball name is the correct one, I like to reformulate my
> Question: Do you see a reason why it is a bad solution?
> 
> If you don't see a problem I will implements this behavior. The
> failing test will be replaced with other tests that capture the new
> behavior.

For better/worse, the current fetcher behaviour with mirrors was to use
symlinks to represent the path its resolved through. We use them in a
variety of cirumstances such as when referencing local file:// mirrors
we can't write to.

My instinct is therefore to have all the fetchers behave the same way
with regard to how DL_DIR works. Having consistency is important and
multiple code paths with different behaviour has caused us problems in
the past.

At the back of my mind there is some thinking that the symlink approach
does protect us against some potential data duplication in the
downloads directory but I'm out of time to work through that thought
right now.

Can you confirm if the normal git fetcher has the issue or its just
shallow clones as that is probably important?

I'd be interested to see your alternative patches too if you have them.

Cheers,

Richard