[bitbake-devel] [PATCH] fetch2/npm: fix parsing multiple matching packages

[Böszörményi Zoltán]

2018-02-11 23:31 keltezéssel, mpvader írta:
> Replace the self-implemented parser with loading the complete json and then
> taking the last element from it.
> 
> This fixes an error fetching the lex-parser package, as a dependency of, for
> example, jison. Reproduce the error with this command:
> 
> devtool add "npm://registry.npmjs.org;name=jison;version=0.4.18"
> 
> _parse_view() didn't properly parse the npm view result [1], resulting in
> an Exception in the subsequent json.loads() function:
> Exception: ValueError: Extra data: line 57 column 4 - line 296 column 2 (char 5874 - 29698)

I can confirm this problem exists in Bitbake 1.36 and goes away with the patch.

Also tested the patch for the handful of node module recipes
I have and they still build properly.

> 
> [1] npm view lex-parser@'0.1.x' --json --registry http://registry.npmjs.org
> 
> Signed-off-by: mpvader <mvader at victronenergy.com>

Tested-by: Zoltán Böszörményi <zboszor at pr.hu>

> ---
>   lib/bb/fetch2/npm.py | 29 ++++++++---------------------
>   1 file changed, 8 insertions(+), 21 deletions(-)
> 
> diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
> index b5f148c..e257355 100644
> --- a/lib/bb/fetch2/npm.py
> +++ b/lib/bb/fetch2/npm.py
> @@ -146,25 +146,6 @@ class Npm(FetchMethod):
>   
>           self._unpackdep(ud, ud.pkgname, workobj, unpackdir, dldir, d)
>   
> -    def _parse_view(self, output):
> -        '''
> -        Parse the output of npm view --json; the last JSON result
> -        is assumed to be the one that we're interested in.
> -        '''
> -        pdata = None
> -        outdeps = {}
> -        datalines = []
> -        bracelevel = 0
> -        for line in output.splitlines():
> -            if bracelevel:
> -                datalines.append(line)
> -            elif '{' in line:
> -                datalines = []
> -                datalines.append(line)
> -            bracelevel = bracelevel + line.count('{') - line.count('}')
> -        if datalines:
> -            pdata = json.loads('\n'.join(datalines))
> -        return pdata
>   
>       def _getdependencies(self, pkg, data, version, d, ud, optional=False, fetchedlist=None):
>           if fetchedlist is None:
> @@ -175,9 +156,15 @@ class Npm(FetchMethod):
>           logger.debug(2, "Calling getdeps on %s" % pkg)
>           fetchcmd = "npm view %s --json --registry %s" % (pkgfullname, ud.registry)
>           output = runfetchcmd(fetchcmd, d, True)
> -        pdata = self._parse_view(output)
> -        if not pdata:
> +        if not output:
>               raise FetchError("The command '%s' returned no output" % fetchcmd)
> +        pdata = json.loads(output)
> +
> +        # npm registry will return multiple dicts in a list when there is more than one matching
> +        # version. Take the last one.
> +        if isinstance(pdata, list):
> +            pdata = pdata[-1]
> +
>           if optional:
>               pkg_os = pdata.get('os', None)
>               if pkg_os:
>