[bitbake-devel] (PRE)MIRRORS doesn't work with https?$:// anymore Was: [PATCH] fetch2: Add new mirror syntax to simplify mirror specifications
Martin Jansa
martin.jansa at gmail.com
Tue Jun 26 17:31:18 UTC 2012
On Thu, Jun 21, 2012 at 03:34:57PM +0100, Richard Purdie wrote:
> When writing mirror specifications, the current regexp syntax can be awkward
> and hard to get it to do what you want. For example, extracting the 'basename'
> of a repository:
>
> PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"
>
> can now become:
>
> PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"
>
> which is much clearer. A MIRRORNAME substitution is also added which contains
> an encoded form of both host and path. One of the problems with the existing
> regexp syntax is you couldn't access HOST information from PATH and vice-versa
> which is an issue this patch also addresses.
One of this fetch2 patches break https?:// in MIRRORS
It's used e.g in openembedded-core/meta/classes/mirrors.bbclass:
https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/
https?$://.*/.* http://sources.openembedded.org/
I've tested it with older bitbake in PREMIRRORS (to make log.do_fetch
log shorter) on libcap (which doesn't exist on upstream URL).
PREMIRRORS_append () {
cvs://.*/.* http://build.shr-project.org/sources/
svn://.*/.* http://build.shr-project.org/sources/
git://.*/.* http://build.shr-project.org/sources/
hg://.*/.* http://build.shr-project.org/sources/
bzr://.*/.* http://build.shr-project.org/sources/
https?$://.*/.* http://build.shr-project.org/sources/
}
bitbake d316f28ed725ff40daa8771c1aa224ac46d5b224:
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2 returning http://build.shr-project.org/sources/libcap-2.22.tar.bz2
NOTE: fetch http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished
bitbake e6ff1d4bab43fdcd8af1230f1d54615f53c1978e
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
NOTE: fetch http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2
...and after trying Upstream and all MIRRORS (none will match) it fails to fetch it..
Removing '$' doesn't work too:
PREMIRRORS_append () {
https?://.*/.* http://build.shr-project.org/sources/
}
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
NOTE: fetch http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2
Using http$ doesn't work too
EBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['http$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
And without any regexp with different PREMIRRORS for http and https it also does weird things (looks like trying to resolve returned PREMIRROR again)
PREMIRRORS_append () {
http://.*/.* http://build.shr-project.org/sources/
https://.*/.* http://build.shr-project.org/https-sources/
}
changed libcap SRC_URI to https:
-SRC_URI = "${KERNELORG_MIRROR}/linux/libs/security/linux-privs/libcap2/${BPN}-${PV}.tar.bz2"
+SRC_URI = "https://foo.org/linux/libs/security/linux-privs/libcap2/${BPN}-${PV}.tar.bz2"
DEBUG: Trying PREMIRRORS
DEBUG: For url ['https', 'foo.org', '/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['https', 'foo.org', '/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
DEBUG: For url https://foo.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2 returning http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2
DEBUG: For url ['http', 'build.shr-project.org', '/https-sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2 returning http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: For url ['http', 'build.shr-project.org', '/sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'build.shr-project.org', '/sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
DEBUG: For url ['http', 'build.shr-project.org', '/https-sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
NOTE: fetch http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Mirror fetch failure for url http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2 (original url: https://foo.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2)
DEBUG: Fetcher failure: Fetch command export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2' failed with exit code 8, output:
STDOUT:
STDERR: http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2:
2012-06-26 19:23:37 ERROR 404: Not Found.
NOTE: fetch http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished
Cheers,
>
> Tests for the new syntax are also added.
>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 1b99a0f..e6f55f8 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -182,7 +182,7 @@ def encodeurl(decoded):
>
> return url
>
> -def uri_replace(ud, uri_find, uri_replace, d):
> +def uri_replace(ud, uri_find, uri_replace, replacements, d):
> if not ud.url or not uri_find or not uri_replace:
> logger.error("uri_replace: passed an undefined value, not replacing")
> return None
> @@ -213,6 +213,8 @@ def uri_replace(ud, uri_find, uri_replace, d):
> if not uri_replace_decoded[loc]:
> result_decoded[loc] = ""
> else:
> + for k in replacements:
> + uri_replace_decoded[loc] = uri_replace_decoded[loc].replace(k, replacements[k])
> #bb.note("%s %s %s" % (i, uri_replace_decoded[loc], uri_decoded[loc]))
> result_decoded[loc] = re.sub(i, uri_replace_decoded[loc], uri_decoded[loc])
> if loc == 2:
> @@ -485,13 +487,20 @@ def build_mirroruris(origud, mirrors, ld):
> uris = []
> uds = []
>
> + replacements = {}
> + replacements["TYPE"] = origud.type
> + replacements["HOST"] = origud.host
> + replacements["PATH"] = origud.path
> + replacements["BASENAME"] = origud.path.split("/")[-1]
> + replacements["MIRRORNAME"] = origud.host.replace(':','.') + origud.path.replace('/', '.')
> +
> def adduri(uri, ud, uris, uds):
> for line in mirrors:
> try:
> (find, replace) = line
> except ValueError:
> continue
> - newuri = uri_replace(ud, find, replace, ld)
> + newuri = uri_replace(ud, find, replace, replacements, ld)
> if not newuri or newuri in uris or newuri == origud.url:
> continue
> uris.append(newuri)
> diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
> index 9961343..1477fab 100644
> --- a/bitbake/lib/bb/tests/fetch.py
> +++ b/bitbake/lib/bb/tests/fetch.py
> @@ -48,7 +48,14 @@ class FetcherTest(unittest.TestCase):
> ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://www.apache.org/dist", "http://archive.apache.org/dist")
> : "http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2",
> ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://.*/.*", "file:///somepath/downloads/")
> - : "file:///somepath/downloads/subversion-1.7.1.tar.bz2"
> + : "file:///somepath/downloads/subversion-1.7.1.tar.bz2",
> + ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http")
> + : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
> + ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http")
> + : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
> + ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http")
> + : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
> +
> #Renaming files doesn't work
> #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"
> #("file://sstate-xyz.tgz", "file://.*/.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
> @@ -143,8 +150,9 @@ class FetcherTest(unittest.TestCase):
> for k, v in self.replaceuris.items():
> ud = bb.fetch.FetchData(k[0], self.d)
> ud.setup_localpath(self.d)
> - newuris = bb.fetch2.uri_replace(ud, k[1], k[2], self.d)
> - self.assertEqual(newuris, v)
> + mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
> + newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
> + self.assertEqual(newuris, [v])
>
> def test_urilist1(self):
> fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel at lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20120626/ad5a421c/attachment-0001.sig>
More information about the bitbake-devel
mailing list