[bitbake-devel] [PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[Mark Hatle]

On 7/25/17 2:30 PM, Andre McCurdy wrote:
> On Tue, Jul 25, 2017 at 7:43 AM, Mark Hatle <mark.hatle at windriver.com> wrote:
>> On 7/24/17 4:04 PM, Andre McCurdy wrote:
>>> On Tue, May 31, 2016 at 6:50 AM, Richard Purdie
>>> <richard.purdie at linuxfoundation.org> wrote:
>>>> On Tue, 2016-05-31 at 15:18 +0200, fabien.proriolpatch at kazoe.org wrote:
>>>>> From: Fabien Proriol <fabien.proriol at jdsu.com>
>>>>>
>>>>> This patch add the possibility to bitbake to avoid git protocol to
>>>>> fetch sources.
>>>>> This is usefull in some network with firewall blocking git port.
>>>>>
>>>>> When BB_GIT_PROTOCOL_FIREWALL is set, the PROTOCOL_MIRRORS table is
>>>>> used to find the new protocol (http or https) and the new host (if
>>>>> different) to used.
>>>>>
>>>>> BB_GIT_PROTOCOL_FIREWALL can also contains a list of host accepted.
>>>>> This is usefull for exemple, if we use local git repository inside
>>>>> the network.
>>>>>
>>>>> Exemple usage:
>>>>>     # Avoid all git protocol
>>>>>     BB_GIT_PROTOCOL_FIREWALL = "1"
>>>>>
>>>>>     # Avoid git protocol, except for srv1 and srv2 in local network
>>>>>     BB_GIT_PROTOCOL_FIREWALL = "srv1.mydomain.com;srv2.mydomain.com"
>>>>
>>>> Why not use the more generic MIRRORS and PREMIRRORS for this? If they
>>>> can't support this, we likely should figure out a way to make them work
>>>> for it rather than adding new fetcher specific variables.
>>>
>>> Sorry for resurrecting such an old thread, but I'm wondering if
>>> there's been any more work on an upstreamable solution?
>>>
>>> The problem being that recipes which define SRC_URI to git://* with
>>> protocol=git (or with protocol unspecified, since git is the default)
>>> fail during fetching for users working behind a firewall which blocks
>>> the native git protocol.
>>>
>>> Although MIRRORS and PREMIRRORS allow mapping from one host to another
>>> there's no support for mapping one git fetch protocol to another and
>>> it looks like extending the MIRRORS syntax to support that would be a
>>> non-trivial change...
>>
>> Won't the following work:
>>
>> PREMIRRORS_append = " \
>>      git://.*/.* git://HOST/PATH;protocol=https \n \
>>      git://.*/.* git://HOST/PATH;protocol=http \n \
>> "
> 
> Thanks! Yes, that does work. It's the first time I've ever seen HOST
> or PATH being used as mirror replacement patterns. Useful to know.

As far as I know the following are defined:

TYPE, HOST, PATH, BASENAME, MIRRORNAME

The premirror pattern matching is broken up into three parts, type, host, path.
I.e.:

<pattern>://<pattern>/<pattern>

Thus the first three replacements, TYPE, HOST, PATH.

The BASENAME is simply the item the last '/' in the PATH.

MIRRORNAME is more complicated, but matches the name set in the DL_DIR.

(What has never been clear to me is how the mirroring handles the parameters,
items after the ';'.  I assume it just preserves the existing parameters and
then if you specify any additional ones adds them to the end of the list
effectively overriding any prior settings.)

I've not seen it specifically documented this way -- but I've been using this
approach now for a few years so it's possible it has been documented and I've
just missed it.

--Mark

> Would a patch adding something similar to the default MIRRORS (ie try
> http[s] as a fallback if the git protocol fails) be acceptable in
> upstream oe-core?
> 
>> Note: I've not actually tried the above, but I expect it will work.  We do
>> something similar to provide local (on-disk) git caches using:
>>
>> git://.*/.* git://${LAYERDIR}/git/MIRRORNAME;protocol=file
>>
>> --Mark
>>
>>>> Cheers,
>>>>
>>>> Richard
>>>> --
>>>> _______________________________________________
>>>> bitbake-devel mailing list
>>>> bitbake-devel at lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>>