[bitbake-devel] Using gitsm fetcher and PREMIRROR

[Christopher Larson]

Don’t have time for a full reply at the moment, I’ll take a look in the morning, but I did want to note that git2_boringssl.googlesource.com.boringssl.tar.gz is a full git mirror tarball as generated by BB_GENERATE_MIRROR_TARBALLS, and the premirrors fetching will download and use them if you want to skip the git-http-backend approach entirely. They are *not* shallow in any way. Only gitshallow_*.tar.gz are shallow tarballs.
On Oct 25, 2019, 1:39 PM -0700, Konrad Scherer <Konrad.Scherer at windriver.com>, wrote:
> Hello,
>
> I recently found a strange problem involving the gitsm fetcher and our PREMIRRORS and I was able to
> reproduce it using poky.
>
> > git clone git://git.yoctoproject.org/poky poky1
> > cd poky1
> > . oe-init-build-env
> > bitbake ovmf
>
> The ovmf recipe has git submodules inside git submodules. This step works fine.
>
> I then exposed the downloads directory over http (http://<host>/downloads) and the downloads/git2
> directory over http with git-http-backend enabled (http://<host>/git).
>
> I noticed that there are shallow clone tarballs in downloads that match the git repos in
> downloads/git2. For example:
>
> downloads/git2/boringssl.googlesource.com.boringssl/
> downloads/git2_boringssl.googlesource.com.boringssl.tar.gz
>
> Since these are duplicates of the git repos, I deleted the tar.gz files
>
> > rm -f downloads/git2_*
>
> <new shell>
>
> > git clone git://git.yoctoproject.org/poky poky2
> > cd poky2
> > . oe-init-build-env
> > cat >> conf/local.conf <<EOF
> WRS_MIRROR_HOST = "<host>"
> BB_ALLOWED_NETWORKS = "${WRS_MIRROR_HOST}"
>
> BB_NO_NETWORK = '0'
> BB_FETCH_PREMIRRORONLY = '1'
>
> PREMIRRORS_append = " \
> .*://.*/.* http://${WRS_MIRROR_HOST}/downloads/ \n \
> git://.*/.* git://${WRS_MIRROR_HOST}/git/MIRRORNAME;protocol=http \n \
> gitsm://.*/.* git://${WRS_MIRROR_HOST}/git/MIRRORNAME;protocol=http \n \
> "
> CONNECTIVITY_CHECK_URIS = ""
> EOF
> > bitbake ovmf
>
> This fails with:
>
> ERROR: ovmf-edk2-stable201905-r0 do_unpack: gitsm: submodule unpack failed: UnpackError Unpack
> failure for URL:
> 'gitsm://github.com/openssl/openssl;protocol=https;name=CryptoPkg/Library/OpensslLib/openssl;subpath=CryptoPkg/Library/OpensslLib/openssl;bareclone=1;nobranch=1'.
> No up to date source found: clone directory not available or not up to date:
> /ala-lpggp22/kscherer/gitsm/poky2/build/downloads/git2/github.com.openssl.openssl; shallow clone not
> enabled
>
> If I leave the shallow clone tarballs in the PREMIRROR the build succeeds. The other recipes that
> use git are fetched properly. The logs don't have any warnings before the error and I verified that
> the objects are indeed present in the repo.
>
> Are my PREMIRROR settings correct? Is this a bug in the gitsm fetcher?
>
> Any help and insight appreciated.
>
> --
> Konrad Scherer, MTS, Linux Products Group, Wind River
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20191027/b43646d3/attachment.html>