[bitbake-devel] [PATCH] fetch2/npm: fix parsing multiple matching packages

[Böszörményi Zoltán]

Hi Bitbake devs!

The exception with the below "devtool add ..." command still occurs with
bvitbake 1.42.0 used in Yocto 2.7 and is fixed with the patch.

Please consider applying/backporting it.

Best regards,
Zoltán Böszörményi

2018. 02. 19. 9:55 keltezéssel, Böszörményi Zoltán írta:
> 2018-02-11 23:31 keltezéssel, mpvader írta:
>> Replace the self-implemented parser with loading the complete json and then
>> taking the last element from it.
>>
>> This fixes an error fetching the lex-parser package, as a dependency of, for
>> example, jison. Reproduce the error with this command:
>>
>> devtool add "npm://registry.npmjs.org;name=jison;version=0.4.18"
>>
>> _parse_view() didn't properly parse the npm view result [1], resulting in
>> an Exception in the subsequent json.loads() function:
>> Exception: ValueError: Extra data: line 57 column 4 - line 296 column 2 (char 5874 - 29698)
> 
> I can confirm this problem exists in Bitbake 1.36 and goes away with the patch.
> 
> Also tested the patch for the handful of node module recipes
> I have and they still build properly.
> 
>>
>> [1] npm view lex-parser@'0.1.x' --json --registry http://registry.npmjs.org
>>
>> Signed-off-by: mpvader <mvader at victronenergy.com>
> 
> Tested-by: Zoltán Böszörményi <zboszor at pr.hu>
> 
>> ---
>>   lib/bb/fetch2/npm.py | 29 ++++++++---------------------
>>   1 file changed, 8 insertions(+), 21 deletions(-)
>>
>> diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
>> index b5f148c..e257355 100644
>> --- a/lib/bb/fetch2/npm.py
>> +++ b/lib/bb/fetch2/npm.py
>> @@ -146,25 +146,6 @@ class Npm(FetchMethod):
>>           self._unpackdep(ud, ud.pkgname, workobj, unpackdir, dldir, d)
>> -    def _parse_view(self, output):
>> -        '''
>> -        Parse the output of npm view --json; the last JSON result
>> -        is assumed to be the one that we're interested in.
>> -        '''
>> -        pdata = None
>> -        outdeps = {}
>> -        datalines = []
>> -        bracelevel = 0
>> -        for line in output.splitlines():
>> -            if bracelevel:
>> -                datalines.append(line)
>> -            elif '{' in line:
>> -                datalines = []
>> -                datalines.append(line)
>> -            bracelevel = bracelevel + line.count('{') - line.count('}')
>> -        if datalines:
>> -            pdata = json.loads('\n'.join(datalines))
>> -        return pdata
>>       def _getdependencies(self, pkg, data, version, d, ud, optional=False, 
>> fetchedlist=None):
>>           if fetchedlist is None:
>> @@ -175,9 +156,15 @@ class Npm(FetchMethod):
>>           logger.debug(2, "Calling getdeps on %s" % pkg)
>>           fetchcmd = "npm view %s --json --registry %s" % (pkgfullname, ud.registry)
>>           output = runfetchcmd(fetchcmd, d, True)
>> -        pdata = self._parse_view(output)
>> -        if not pdata:
>> +        if not output:
>>               raise FetchError("The command '%s' returned no output" % fetchcmd)
>> +        pdata = json.loads(output)
>> +
>> +        # npm registry will return multiple dicts in a list when there is more than one 
>> matching
>> +        # version. Take the last one.
>> +        if isinstance(pdata, list):
>> +            pdata = pdata[-1]
>> +
>>           if optional:
>>               pkg_os = pdata.get('os', None)
>>               if pkg_os:
>>
>