[bitbake-devel] Issue with bitbake fetch/unpack when using MIRRORS rewrite

[Bach, Pascal]

Hello Everyone

I finally got to pin down an issue that I occasionally observed in the past:

When using the git fetcher with shallow tarballs (BB_GIT_SHALLOW,BB_GIT_SHALLOW_DEPTH) experience the following issues during the do_unpack task:

ERROR: pseudo-native-1.9.0+gitAUTOINC+d7c31a25e4-r0 do_unpack: Fetcher failure: [...]
tar (child): /home/projects/ccp3-labs/oe/build/../../../downloads2/gitshallow_git.yoctoproject.org.pseudo_d7c31a2-1_master.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

The issue is that the do_unpack is trying to find the following file in the downloads directory:
downloads2/gitshallow_git.yoctoproject.org.pseudo_d7c31a2-1_master.tar.gz

However this file doesn't exist, instead the following file was created by the fetch task:
downloads2/gitshallow_git.yoctoproject.org.git.pseudo_d7c31a2-1_master.tar.gz

After some digging I suspect that the following rewrite rule is causing the issue:
MIRRORS += "git://git.yoctoproject.org/.*                               git://git.yoctoproject.org/git/PATH;protocol=https \n"

The line exists because we don't have access to git.yoctoproject.org via the git protocol. So we use the above rule to make access work via HTTPS.
However due to the way the Yocto git server is structured the rewrite causes the URL path to change from / to /git which causes the additional .git in the
resulting tarball, which then makes the unpack task look for the wrong file.

I think the proper behavior would be to always name the tarball after the original SRC_URI not the rewritten one.

BTW: I was able to reproduce the issue with the shallow git tarballs but I think it also applies to normal git tarballs and possibly other fetchers.

The issue still exists with bitbake 1.37.0 on Poky master.

Regards
Pascal