[bitbake-devel] Issue with bitbake fetch/unpack when using MIRRORS rewrite

[richard.purdie at linuxfoundation.org]

Hi Urs,

On Tue, 2018-07-17 at 13:15 +0200, Urs Fässler wrote:
> I investigated the issue that the unpack step fails when using mirror
> rewrite rules.
> 
> The root of the problem is, that the download step uses the
> mirrored url to create the local filename while the unpack step uses
> the url from the recipe to create the local filename.
> 
> As I understand the code, the link between the filename from
> download and the filename from unpack is missing. It seems to work
> because they are usually the same.
> 
> I tried both solutions proposed by Richard: Using symlinks and the
> recipe-url.
> The symlink solution is nice since it follows the same methods as for
> git clones. Unfortunately, it is not practical for us. We like to
> store
> the tarballs on a SMB share or S3 storage. Both do not support
> symlinks.
> The recipe-url naming method is nice since the tarball is named after
> the url as it is written in the recipe. This is easy understandable.
> But unfortunately this method breaks the test
> "FetcherNetworkTest.test_gitfetch_premirror", which tests the
> following: when 2 different recipe-urls point to the same mirrored-
> url, the repository is cloned only once.

Would you be able to provide a kind of worked example of the problem? I
think I understand the problem but some example urls, the mirror format
and the resulting different mirror tarball names would probably make it
easier for me to comment on this.

> Now the question is which solution we should implement. For us, it is
> the second one (tarball naming after recipe-url). It comes with the
> downside that the one mentioned test fails and has to be removed. In
> a
> real scenario this results in downloading a repository twice and
> having
> 2 tarballs with the same content. But I expect this to be unlikely in
> a
> real world scenario.
> 
> A third solution may be that we add a link between the download and
> unpack task. But this would be the most intrusive solution for
> Bitbake.

I'm more than a little concerned about the symlink comment since the
fetcher assumes that symlinks work in other places too.

Also, do you have any new test cases to add which illustrate it?

Cheers,

Richard