[bitbake-devel] Issue with bitbake fetch/unpack when using MIRRORS rewrite

[Urs Fässler]

On Sun, 2018-03-11 at 06:36 -0700, richard.purdiea wrote:
> On Fri, 2018-03-09 at 09:40 +0000, Bach, Pascal wrote:
> > Hello Everyone
> > 
> > I finally got to pin down an issue that I occasionally observed in
> > the past:
> > 
> > When using the git fetcher with shallow tarballs
> > (BB_GIT_SHALLOW,BB_GIT_SHALLOW_DEPTH) experience the following
> > issues
> > during the do_unpack task:
> > 
> > ERROR: pseudo-native-1.9.0+gitAUTOINC+d7c31a25e4-r0 do_unpack:
> > Fetcher failure: [...]
> > tar (child): /home/projects/ccp3-
> > labs/oe/build/../../../downloads2/gitshallow_git.yoctoproject.org.p
> > se
> > udo_d7c31a2-1_master.tar.gz: Cannot open: No such file or directory
> > tar (child): Error is not recoverable: exiting now
> > tar: Child returned status 2
> > tar: Error is not recoverable: exiting now
> > 
> > The issue is that the do_unpack is trying to find the following
> > file
> > in the downloads directory:
> > downloads2/gitshallow_git.yoctoproject.org.pseudo_d7c31a2-
> > 1_master.tar.gz
> > 
> > However this file doesn't exist, instead the following file was
> > created by the fetch task:
> > downloads2/gitshallow_git.yoctoproject.org.git.pseudo_d7c31a2-
> > 1_master.tar.gz
> > 
> > After some digging I suspect that the following rewrite rule is
> > causing the issue:
> > MIRRORS +=
> > "git://git.yoctoproject.org/.*???????????????????????????????git://
> > gi
> > t.yoctoproject.org/git/PATH;protocol=https \n"
> > 
> > The line exists because we don't have access to
> > git.yoctoproject.org
> > via the git protocol. So we use the above rule to make access work
> > via HTTPS.
> > However due to the way the Yocto git server is structured the
> > rewrite
> > causes the URL path to change from / to /git which causes the
> > additional .git in the
> > resulting tarball, which then makes the unpack task look for the
> > wrong file.
> > 
> > I think the proper behavior would be to always name the tarball
> > after
> > the original SRC_URI not the rewritten one.
> > 
> > BTW: I was able to reproduce the issue with the shallow git
> > tarballs
> > but I think it also applies to normal git tarballs and possibly
> > other
> > fetchers.
> > 
> > The issue still exists with bitbake 1.37.0 on Poky master.
> 
> I do think there are likely problems in this area.
> 
> This gets tricky as one mirror tarball could in theory match against
> multiple different parent urls. I'm therefore torn on whether it
> should
> use the "parent" naming for the tarball, or create a chain of
> symlinks
> back to the high level name.?
> 
> Its something I'd have to sit and look at a bit further and
> understand
> what the code is doing. I know when Chris added the shallow code, he
> did add multiple mirror tarball support so it could be related to
> that
> too...
> 
> Just thinking out loud, you could have a "full" mirror tarball as
> well
> as a shallow one too so it does get quite complex. Which corner cases
> we care about is a tricky one too.
> 
> Did you have a fix to propose?
> 
> With the fetcher, I do ask when we make changes, we do add unit tests
> so that we can protect use cases going forward.
> 
> Cheers,
> 
> Richard

Hi all,
I got the task to fix this issue.
Can you explain where do you see the other problems and tricky
situations you mentioned, maybe with an example?

Regards
Urs