[OE-core] Connectivity check uris

[Jean-Marie LEMETAYER]

On Jan 23, 2020, at 2:26 PM, Richard Purdie richard.purdie at linuxfoundation.org wrote:
> On Thu, 2020-01-23 at 08:14 -0500, Jean-Marie LEMETAYER wrote:
>> Hi folks,
>> 
>> I have noticed some hang-ups at the beginning of my builds on the
>> master branch.
>> I have search a little and discovered that the connectivity check
>> made by poky
>> was the root cause. In fact connectivity check use the
>> CONNECTIVITY_CHECK_URIS
>> variable which is by default set to "https://www.example.com/".
>> 
>> So I have made some tests with a little script which gets the URI 100
>> times:
>> 
>> 	$ cat connectivity_check.sh
>> 	#!/bin/sh
>> 	connectivity_check()
>> 	{
>> 		local ok=0
>> 		for i in $(seq 100)
>> 		do
>> 			$@ > /dev/null && ok=$(($ok + 1))
>> 		done
>> 		echo $ok $@
>> 	}
>> 	connectivity_check wget -qO- --timeout=1 --tries=1 $1
>> 	connectivity_check curl -s --connect-timeout 1 $1
>> 
>> Note that in the script, I use wget and curl to see if there are any
>> differences
>> but I don't think that's a thing.
>> 
>> Here are the results:
>> 	$ time ./connectivity_check.sh https://example.com
>> 	45 wget -qO- --timeout=1 --tries=1 https://example.com
>> 	45 curl -s --connect-timeout 1 https://example.com
>> 
>> 	real	2m23.595s
>> 	user	0m3.352s
>> 	sys	0m1.834s
>> 
>> 	$ time ./connectivity_check.sh https://openembedded.org
>> 	100 wget -qO- --timeout=1 --tries=1 https://openembedded.org
>> 	100 curl -s --connect-timeout 1 https://openembedded.org
>> 
>> 	real	4m36.188s
>> 	user	0m3.853s
>> 	sys	0m1.761s
>> 
>> 	$ time ./connectivity_check.sh https://google.com
>> 	100 wget -qO- --timeout=1 --tries=1 https://google.com
>> 	100 curl -s --connect-timeout 1 https://google.com
>> 
>> 	real	0m36.328s
>> 	user	0m4.265s
>> 	sys	0m2.028s
>> 
>> So "example.com" is clearly not a good domain to do connectivity
>> checks.
>> 
>> The "openembedded.org" domain is good but have a slow response time.
>> 
>> And finally "google.com" which have all sort of speedy networking
>> stuff is very
>> efficient.
>> 
>> ---
>> 
>> I think the CONNECTIVITY_CHECK_URIS variable must be updated and not
>> use the
>> "example.com" domain anymore. Using "google.com" may be the best
>> option. But
>> eetting up a custom domain like "connectivitycheck.openembedded.org"
>> could be
>> another idea.
>> 
>> Any ideas
> 
> I think if you look at the history we did use google.com however that
> caused problems for people in geos where that is blocked (e.g. PRC).
> 
> We may want to move the default back given the performance isse, not
> sure...

I understand the issue with blocked domains this is why I suggest the
custom domain "connectivitycheck.openembedded.org".

But I dont know if it could be an option. Can OE have a sub-domain like
that? maybe using some cdn to handle the load?

Another option will be to provide multiple URIs:
    CONNECTIVITY_CHECK_URIS ?= " \
        https://google.com \
        https://www.bing.com \
        https://www.baidu.com \
    "

Best regards,
Jean-Marie