[bitbake-devel] [PATCHv3] git: remove symling before updating mirror tarball

[Martin Jansa]

On Tue, May 07, 2013 at 11:51:08PM +0200, Martin Jansa wrote:
> On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote:
> > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS
> >   and referenced as file:///mnt/premirror) we cannot use
> >   BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz
> >   files later became just symlinks to read-only location in PREMIRROR
> >   (it works fine on first build and for new components, because
> >   at that time there isn't tarball on PREMIRROR yet).
> > 
> >   ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
> >   tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system
> >   tar (child): Error is not recoverable: exiting now
> 
> This one is tested and I would like to get it also in 1.16 and 1.18
> 
> Steps to reproduce the issue (without redownloading anything big)
> 
> # mv downloads downloads-ro
> # mkdir downloads
> # mkdir tmp-eglibc/downloads
> # sudo mount -o ro,bind downloads-ro tmp-eglibc/downloads
> # sudo mount -o remount,ro tmp-eglibc/downloads
> 
> conf/local.conf:
> SOURCE_MIRROR_URL = "file:///OE/oe-core/tmp-eglibc/downloads"
> INHERIT += "own-mirrors"
> 
> # bitbake -c fetch qtbase
> # ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz 
>   should be a symlink
> 
> bump SRCREV in qtbase_git.bb to some revision not included in
> git2_qt.gitorious.org.qt.qtbase.git.tar.gz
> 
> # bitbake -c fetch qtbase
> WARNING: Failed to fetch URL git://qt.gitorious.org/qt/qtbase.git;branch=stable, attempting MIRRORS if available
> DEBUG: Fetcher failure: Fetch command failed with exit code 141, output:
> tar (child): /OE/oe-core/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz: Cannot open: Read-only file system
> tar (child): Error is not recoverable: exiting now
> 
> apply this patch
> # bitbake -c fetch qtbase
> works fine
> # ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
> -rw-r--r-- 1 bitbake bitbake 176M May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
> is now proper file instead of symlink

There is one more interesting behavior with such setup where file://
fetcher is used in PREMIRROR

AFAIK it's completely harmless and also not influenced by this change,
but .done stamps for files downloaded from premirror are stored in
${DL_DIR}<path-to-local-premirror>/file.done, e.g.
downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done

qtbase was recreated locally thanks to this patch, that's why it has 2 .done stamps, one in
downloads, 2nd in downloads/OE/oe-core/tmp-eglibc/downloads/

qtdeclarative has only symlink and .done stamp in downloads/OE/oe-core/tmp-eglibc/downloads/

OE @ ~/oe-core $ ll downloads/*qt{base,declarative}.git*
-rw-r--r-- 1 bitbake bitbake 184015350 May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
-rw-r--r-- 1 bitbake bitbake         0 May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done
lrwxrwxrwx 1 bitbake bitbake        82 May  8 00:00 downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz -> /OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz

OE @ ~/oe-core $ ll downloads/OE/oe-core/tmp-eglibc/downloads/*qt{base,declarative}.git*
-rw-r--r-- 1 bitbake bitbake 0 May  7 23:41 downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done
-rw-r--r-- 1 bitbake bitbake 0 May  8 00:00 downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz.done

> > Signed-off-by: Martin Jansa <Martin.Jansa at gmail.com>
> > ---
> >  lib/bb/fetch2/git.py | 4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> > index 052802e..6175e4c 100644
> > --- a/lib/bb/fetch2/git.py
> > +++ b/lib/bb/fetch2/git.py
> > @@ -217,6 +217,10 @@ class Git(FetchMethod):
> >      def build_mirror_data(self, url, ud, d):
> >          # Generate a mirror tarball if needed
> >          if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
> > +            # it's possible that this symlink points to read-only filesystem with PREMIRROR
> > +            if os.path.islink(ud.fullmirror):
> > +                os.unlink(ud.fullmirror)
> > +
> >              os.chdir(ud.clonedir)
> >              logger.info("Creating tarball of git repository")
> >              runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
> > -- 
> > 1.8.2.1
> > 
> 
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com



-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20130508/492b3c22/attachment-0001.sig>