[bitbake-devel] [PATCH] s3.py: Add support for fetching source mirrors/minor cleanup

[Richard Purdie]

On Tue, 2017-03-28 at 13:39 +0200, Elizabeth 'pidge' Flanagan wrote:
> This commits main purpose is to add support for fetching source
> mirrors. In the current incarnation:
> 
> SOURCE_MIRROR_URL ?= "s3://mybucket/downloads"
> 
> will fail for two reasons. First, download doesn't support it,
> but second, without aws included in HOSTTOOLS you'll end up
> with aws not being found by bitbake (for either source mirrors or
> sstate mirrors).
> 
> Part of this is fixed with this commit. However, this will still
> fail if HOSTTOOLS doesn't include 'aws' in bitbake.conf. I've another
> commit or two to fix that as well.
> 
> I've also DRYed up some of the error handling, removed the hardcoded
> aws and added some logging.
> 
> Signed-off-by: Elizabeth 'pidge' Flanagan <pidge at toganlabs.com>
> ---
>  lib/bb/fetch2/s3.py | 20 ++++++++++++++------
>  1 file changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/lib/bb/fetch2/s3.py b/lib/bb/fetch2/s3.py
> index 27993aa..dbd86f4 100644
> --- a/lib/bb/fetch2/s3.py
> +++ b/lib/bb/fetch2/s3.py
> @@ -34,6 +34,7 @@ import urllib.request, urllib.parse, urllib.error
>  from bb.fetch2 import FetchMethod
>  from bb.fetch2 import FetchError
>  from bb.fetch2 import runfetchcmd
> +from bb.fetch2 import logger
>  
>  class S3(FetchMethod):
>      """Class to fetch urls via 'aws s3'"""
> @@ -60,8 +61,13 @@ class S3(FetchMethod):
>          Fetch urls
>          Assumes localpath was called first
>          """
> -
> -        cmd = 'aws s3 cp s3://%s%s %s' % (ud.host, ud.path,
> ud.localpath)
> +        if 'downloadfilename' in ud.parm:
> +            dldir = d.getVar("DL_DIR", True)
> +            bb.utils.mkdirhier(os.path.dirname(dldir + os.sep +
> ud.localfile))
> +            cmd = '%s cp s3://%s%s %s%s%s' %
> (d.getVar("FETCHCMD_s3"), ud.host, ud.path, dldir, os.sep,
> ud.localpath)
> +        else:
> +            cmd = '%s cp s3://%s%s %s' % (d.getVar("FETCHCMD_s3"),
> ud.host, ud.path, ud.localpath)
> +        logger.debug(2, "Fetching %s using command '%s'" % (ud.url,
> cmd))

Do you actually need/use FETCHCMD_s3 ?

I'd prefer not to add this unless we really do need it.

Cheers,

Richard