[bitbake-devel] Issue with bitbake fetch/unpack when using MIRRORS rewrite

[Urs Fässler]

On Tue, 2018-07-17 at 12:24 +0100, richard.purdie at linuxfoundation.org
wrote:
> Hi Urs,
> 
> On Tue, 2018-07-17 at 13:15 +0200, Urs Fässler wrote:
> > I investigated the issue that the unpack step fails when using
> > mirror
> > rewrite rules.
> > 
> > The root of the problem is, that the download step uses the
> > mirrored url to create the local filename while the unpack step
> > uses
> > the url from the recipe to create the local filename.
> > 
> > As I understand the code, the link between the filename from
> > download and the filename from unpack is missing. It seems to work
> > because they are usually the same.
> > 
> > I tried both solutions proposed by Richard: Using symlinks and the
> > recipe-url.
> > The symlink solution is nice since it follows the same methods as
> > for
> > git clones. Unfortunately, it is not practical for us. We like to
> > store
> > the tarballs on a SMB share or S3 storage. Both do not support
> > symlinks.
> > The recipe-url naming method is nice since the tarball is named
> > after
> > the url as it is written in the recipe. This is easy
> > understandable.
> > But unfortunately this method breaks the test
> > "FetcherNetworkTest.test_gitfetch_premirror", which tests the
> > following: when 2 different recipe-urls point to the same mirrored-
> > url, the repository is cloned only once.
> 
> Would you be able to provide a kind of worked example of the problem?
> I
> think I understand the problem but some example urls, the mirror
> format
> and the resulting different mirror tarball names would probably make
> it
> easier for me to comment on this.

You can reproduce the problem with a current (rocko) Yocto. Add the
following lines in local.conf:
  BB_GIT_SHALLOW = "1"
  BB_GENERATE_MIRROR_TARBALLS = "1"
  PREMIRRORS += "git://git.yoctoproject.org/.* git://git.yoctoproject.org/git/PATH;protocol=https \n"

and execute
  bitbake fstests -c unpack

You should get something like:
  tar -xzf .../build/downloads/gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz failed with exit code 2, output:
  tar (child): .../build/downloads/gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz: Cannot open: No such file or directory

What happens is that the download step generates the tarball:
  gitshallow_git.yoctoproject.org.git.fstests_e5939ff-1_master.tar.gz
but the unpack step expects the tarball:
  gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz

The difference in the tarball names come from the different url used in
the recipe and when rewritten according to PREMIRRORS.


The symlink solution would add a symlink from
gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz to
gitshallow_git.yoctoproject.org.git.fstests_e5939ff-1_master.tar.gz.

The recipe-url solution would name the tarball
gitshallow_git.yoctoproject.org.fstests_e5939ff-1_master.tar.gz.

> > Now the question is which solution we should implement. For us, it
> > is
> > the second one (tarball naming after recipe-url). It comes with the
> > downside that the one mentioned test fails and has to be removed.
> > In
> > a
> > real scenario this results in downloading a repository twice and
> > having
> > 2 tarballs with the same content. But I expect this to be unlikely
> > in
> > a
> > real world scenario.
> > 
> > A third solution may be that we add a link between the download and
> > unpack task. But this would be the most intrusive solution for
> > Bitbake.
> 
> I'm more than a little concerned about the symlink comment since the
> fetcher assumes that symlinks work in other places too.

Sorry for concerning you. I think it is no issue. We generate the
tarballs and archive them on a system without symlinks. Then we get the
tarballs over http with the help of a premirror rule. We do it as
described in the Bitbake manual chapter "The Download (Fetch)". I
expect this to be a fairly common use case.

Another rationale for the recipe-url solution is that the mirrors are
used when the server from the recipe-url is not available. When we
generate a tarball, it would be strange that the name of it depends on
some local conditions (closed ports, local mirror rewrite rules, ...)
rather than the recipe.
This probably invalidates my argument that the symlink solution is nice
since it has the same method for naming as the git clone naming. This
are 2 quite different use cases.

> Also, do you have any new test cases to add which illustrate it?

I have a test but it is a bit tricky since there are 2 issues. The one
you see (as mentioned above with Yocto) is actually not the real
problem but an issue that is only seen in this scenario.

Since I am now quite sure that the solution where we use the recipe-url 
for the tarball name is the correct one, I like to reformulate my
Question: Do you see a reason why it is a bad solution?

If you don't see a problem I will implements this behavior. The failing
test will be replaced with other tests that capture the new behavior.

> Cheers,
> 
> Richard

Thanks,
Urs