[bitbake-devel] Issue with bitbake fetch/unpack when using MIRRORS rewrite

[Urs Fässler]

On Mon, 2018-07-02 at 14:29 +0200, Urs Fässler wrote:
> On Sun, 2018-03-11 at 06:36 -0700, richard.purdiea wrote:
> > On Fri, 2018-03-09 at 09:40 +0000, Bach, Pascal wrote:
> > > Hello Everyone
> > > 
> > > I finally got to pin down an issue that I occasionally observed
> > > in
> > > the past:
> > > 
> > > When using the git fetcher with shallow tarballs
> > > (BB_GIT_SHALLOW,BB_GIT_SHALLOW_DEPTH) experience the following
> > > issues
> > > during the do_unpack task:
> > > 
> > > ERROR: pseudo-native-1.9.0+gitAUTOINC+d7c31a25e4-r0 do_unpack:
> > > Fetcher failure: [...]
> > > tar (child): /home/projects/ccp3-
> > > labs/oe/build/../../../downloads2/gitshallow_git.yoctoproject.org
> > > .p
> > > se
> > > udo_d7c31a2-1_master.tar.gz: Cannot open: No such file or
> > > directory
> > > tar (child): Error is not recoverable: exiting now
> > > tar: Child returned status 2
> > > tar: Error is not recoverable: exiting now
> > > 
> > > The issue is that the do_unpack is trying to find the following
> > > file
> > > in the downloads directory:
> > > downloads2/gitshallow_git.yoctoproject.org.pseudo_d7c31a2-
> > > 1_master.tar.gz
> > > 
> > > However this file doesn't exist, instead the following file was
> > > created by the fetch task:
> > > downloads2/gitshallow_git.yoctoproject.org.git.pseudo_d7c31a2-
> > > 1_master.tar.gz
> > > 
> > > After some digging I suspect that the following rewrite rule is
> > > causing the issue:
> > > MIRRORS +=
> > > "git://git.yoctoproject.org/.*???????????????????????????????git:
> > > //
> > > gi
> > > t.yoctoproject.org/git/PATH;protocol=https \n"
> > > 
> > > The line exists because we don't have access to
> > > git.yoctoproject.org
> > > via the git protocol. So we use the above rule to make access
> > > work
> > > via HTTPS.
> > > However due to the way the Yocto git server is structured the
> > > rewrite
> > > causes the URL path to change from / to /git which causes the
> > > additional .git in the
> > > resulting tarball, which then makes the unpack task look for the
> > > wrong file.
> > > 
> > > I think the proper behavior would be to always name the tarball
> > > after
> > > the original SRC_URI not the rewritten one.
> > > 
> > > BTW: I was able to reproduce the issue with the shallow git
> > > tarballs
> > > but I think it also applies to normal git tarballs and possibly
> > > other
> > > fetchers.
> > > 
> > > The issue still exists with bitbake 1.37.0 on Poky master.
> > 
> > I do think there are likely problems in this area.
> > 
> > This gets tricky as one mirror tarball could in theory match
> > against
> > multiple different parent urls. I'm therefore torn on whether it
> > should
> > use the "parent" naming for the tarball, or create a chain of
> > symlinks
> > back to the high level name.?
> > 
> > Its something I'd have to sit and look at a bit further and
> > understand
> > what the code is doing. I know when Chris added the shallow code,
> > he
> > did add multiple mirror tarball support so it could be related to
> > that
> > too...
> > 
> > Just thinking out loud, you could have a "full" mirror tarball as
> > well
> > as a shallow one too so it does get quite complex. Which corner
> > cases
> > we care about is a tricky one too.
> > 
> > Did you have a fix to propose?
> > 
> > With the fetcher, I do ask when we make changes, we do add unit
> > tests
> > so that we can protect use cases going forward.
> > 
> > Cheers,
> > 
> > Richard
> 
> Hi all,
> I got the task to fix this issue.
> Can you explain where do you see the other problems and tricky
> situations you mentioned, maybe with an example?
> 
> Regards
> Urs

I investigated the problem and saw that the download/pack task uses the
mirrored url to create the local filename while the unpack task uses
the original url to create the local filename.

As I understand the code the link between the filename from
download/pack and the filename from unpack is missing. It seems to work
because they are usually the same.

I don't have the full picture to decide what solution is best. But from
my understanding, it should be ok to use the parent/original url. It
could match against multiple urls, but they should contain the same
version of the data. In the git case, this is secured by the commit
hash in the filename of the tarball.
But I like to hear your opinion.