[bitbake-devel] [PATCH] fetch2: add ability to skip recipes when get_srcrev fails via BB_FETCH_SKIP_ON_SRCREV_ERROR

[Alexander Kanavin]

I think it's better to rearrange the layer itself, and split it into a
privileged and a non-privileged one.

This patch seems to enable skipping the recipe, regardless of what the
actual issue with fetching was (which may or may not be 'access denied').
Skipping on any error doesn't seem right to me as it may mask other types
of problems.

Alex

On Fri, 26 Jul 2019 at 16:55, Chris Laplante via bitbake-devel <
bitbake-devel at lists.openembedded.org> wrote:

> Sometimes it is desirable to just skip a recipe if the upstream(s) can't be
> accessed by the user, rather than fail the whole build outright.
>
> For example, in a corporate environment, sometimes certain git repositories
> are access controlled. If privileged and non-privileged users share
> the same Yocto layers, it would be nice for the non-privileged users to
> be able to build the recipes they need, without tripping over parse
> failures on the non-accessible recipes.
>
> Signed-off-by: Chris Laplante <chris.laplante at agilent.com>
> ---
>  lib/bb/fetch2/__init__.py | 99
> +++++++++++++++++++++++++----------------------
>  1 file changed, 52 insertions(+), 47 deletions(-)
>
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index f6b5529..a165a77 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -741,55 +741,60 @@ def get_srcrev(d, method_name='sortable_revision'):
>      that fetcher provides a method with the given name and the same
> signature as sortable_revision.
>      """
>
> -    scms = []
> -    fetcher = Fetch(d.getVar('SRC_URI').split(), d)
> -    urldata = fetcher.ud
> -    for u in urldata:
> -        if urldata[u].method.supports_srcrev():
> -            scms.append(u)
> -
> -    if len(scms) == 0:
> -        raise FetchError("SRCREV was used yet no valid SCM was found in
> SRC_URI")
> -
> -    if len(scms) == 1 and len(urldata[scms[0]].names) == 1:
> -        autoinc, rev = getattr(urldata[scms[0]].method,
> method_name)(urldata[scms[0]], d, urldata[scms[0]].names[0])
> -        if len(rev) > 10:
> -            rev = rev[:10]
> -        if autoinc:
> -            return "AUTOINC+" + rev
> -        return rev
> -
> -    #
> -    # Mutiple SCMs are in SRC_URI so we resort to SRCREV_FORMAT
> -    #
> -    format = d.getVar('SRCREV_FORMAT')
> -    if not format:
> -        raise FetchError("The SRCREV_FORMAT variable must be set when
> multiple SCMs are used.\n"\
> -                         "The SCMs are:\n%s" % '\n'.join(scms))
> -
> -    name_to_rev = {}
> -    seenautoinc = False
> -    for scm in scms:
> -        ud = urldata[scm]
> -        for name in ud.names:
> -            autoinc, rev = getattr(ud.method, method_name)(ud, d, name)
> -            seenautoinc = seenautoinc or autoinc
> +    try:
> +        scms = []
> +        fetcher = Fetch(d.getVar('SRC_URI').split(), d)
> +        urldata = fetcher.ud
> +        for u in urldata:
> +            if urldata[u].method.supports_srcrev():
> +                scms.append(u)
> +
> +        if len(scms) == 0:
> +            raise FetchError("SRCREV was used yet no valid SCM was found
> in SRC_URI")
> +
> +        if len(scms) == 1 and len(urldata[scms[0]].names) == 1:
> +            autoinc, rev = getattr(urldata[scms[0]].method,
> method_name)(urldata[scms[0]], d, urldata[scms[0]].names[0])
>              if len(rev) > 10:
>                  rev = rev[:10]
> -            name_to_rev[name] = rev
> -    # Replace names by revisions in the SRCREV_FORMAT string. The
> approach used
> -    # here can handle names being prefixes of other names and names
> appearing
> -    # as substrings in revisions (in which case the name should not be
> -    # expanded). The '|' regular expression operator tries matches from
> left to
> -    # right, so we need to sort the names with the longest ones first.
> -    names_descending_len = sorted(name_to_rev, key=len, reverse=True)
> -    name_to_rev_re = "|".join(re.escape(name) for name in
> names_descending_len)
> -    format = re.sub(name_to_rev_re, lambda match:
> name_to_rev[match.group(0)], format)
> -
> -    if seenautoinc:
> -        format = "AUTOINC+" + format
> -
> -    return format
> +            if autoinc:
> +                return "AUTOINC+" + rev
> +            return rev
> +
> +        #
> +        # Mutiple SCMs are in SRC_URI so we resort to SRCREV_FORMAT
> +        #
> +        format = d.getVar('SRCREV_FORMAT')
> +        if not format:
> +            raise FetchError("The SRCREV_FORMAT variable must be set when
> multiple SCMs are used.\n"\
> +                             "The SCMs are:\n%s" % '\n'.join(scms))
> +
> +        name_to_rev = {}
> +        seenautoinc = False
> +        for scm in scms:
> +            ud = urldata[scm]
> +            for name in ud.names:
> +                autoinc, rev = getattr(ud.method, method_name)(ud, d,
> name)
> +                seenautoinc = seenautoinc or autoinc
> +                if len(rev) > 10:
> +                    rev = rev[:10]
> +                name_to_rev[name] = rev
> +        # Replace names by revisions in the SRCREV_FORMAT string. The
> approach used
> +        # here can handle names being prefixes of other names and names
> appearing
> +        # as substrings in revisions (in which case the name should not be
> +        # expanded). The '|' regular expression operator tries matches
> from left to
> +        # right, so we need to sort the names with the longest ones first.
> +        names_descending_len = sorted(name_to_rev, key=len, reverse=True)
> +        name_to_rev_re = "|".join(re.escape(name) for name in
> names_descending_len)
> +        format = re.sub(name_to_rev_re, lambda match:
> name_to_rev[match.group(0)], format)
> +
> +        if seenautoinc:
> +            format = "AUTOINC+" + format
> +
> +        return format
> +    except FetchError as e:
> +        if d.getVar('BB_WORKERCONTEXT') != '1' and
> bb.utils.to_boolean(d.getVar("BB_FETCH_SKIP_ON_SRCREV_ERROR")):
> +            raise bb.parse.SkipRecipe(e)
> +        raise
>
>  def localpath(url, d):
>      fetcher = bb.fetch2.Fetch([url], d)
> --
> 2.7.4
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20190726/b0083bd3/attachment-0001.html>