[bitbake-devel] [PATCH] fetch2/npm: fix parsing multiple matching packages

[Böszörményi Zoltán]

Hi,

2019. 10. 04. 9:42 keltezéssel, Jean-marie Lemetayer írta:
> Hi Zoltán,
> 
> It seems that this issue have already been fixed by this patch:
> https://git.openembedded.org/bitbake/commit/?id=3d319c79981811d3cfd4732885057db4fd5afcc2
> 
> Not the same implementation as yours, but it fixes the same issue.

The implementation is not mine, I just tested it back then with Yocto 2.5.

Will your patch backported to older major versions of Bitbake?
The issue is present in older Yocto versions.

> 
> Best regards,
> Jean-Marie Lemetayer
> 
> ----- Mail original -----
> De: "bitbake-devel" <bitbake-devel at lists.openembedded.org>
> À: "mpvader" <mvader at victronenergy.com>, "bitbake-devel" <bitbake-devel at lists.openembedded.org>
> Envoyé: Vendredi 4 Octobre 2019 07:58:56
> Objet: Re: [bitbake-devel] [PATCH] fetch2/npm: fix parsing multiple matching packages
> 
> Hi Bitbake devs!
> 
> The exception with the below "devtool add ..." command still occurs with
> bvitbake 1.42.0 used in Yocto 2.7 and is fixed with the patch.
> 
> Please consider applying/backporting it.
> 
> Best regards,
> Zoltán Böszörményi
> 
> 2018. 02. 19. 9:55 keltezéssel, Böszörményi Zoltán írta:
>> 2018-02-11 23:31 keltezéssel, mpvader írta:
>>> Replace the self-implemented parser with loading the complete json and then
>>> taking the last element from it.
>>>
>>> This fixes an error fetching the lex-parser package, as a dependency of, for
>>> example, jison. Reproduce the error with this command:
>>>
>>> devtool add "npm://registry.npmjs.org;name=jison;version=0.4.18"
>>>
>>> _parse_view() didn't properly parse the npm view result [1], resulting in
>>> an Exception in the subsequent json.loads() function:
>>> Exception: ValueError: Extra data: line 57 column 4 - line 296 column 2 (char 5874 - 29698)
>>
>> I can confirm this problem exists in Bitbake 1.36 and goes away with the patch.
>>
>> Also tested the patch for the handful of node module recipes
>> I have and they still build properly.
>>
>>>
>>> [1] npm view lex-parser@'0.1.x' --json --registry http://registry.npmjs.org
>>>
>>> Signed-off-by: mpvader <mvader at victronenergy.com>
>>
>> Tested-by: Zoltán Böszörményi <zboszor at pr.hu>
>>
>>> ---
>>>    lib/bb/fetch2/npm.py | 29 ++++++++---------------------
>>>    1 file changed, 8 insertions(+), 21 deletions(-)
>>>
>>> diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
>>> index b5f148c..e257355 100644
>>> --- a/lib/bb/fetch2/npm.py
>>> +++ b/lib/bb/fetch2/npm.py
>>> @@ -146,25 +146,6 @@ class Npm(FetchMethod):
>>>            self._unpackdep(ud, ud.pkgname, workobj, unpackdir, dldir, d)
>>> -    def _parse_view(self, output):
>>> -        '''
>>> -        Parse the output of npm view --json; the last JSON result
>>> -        is assumed to be the one that we're interested in.
>>> -        '''
>>> -        pdata = None
>>> -        outdeps = {}
>>> -        datalines = []
>>> -        bracelevel = 0
>>> -        for line in output.splitlines():
>>> -            if bracelevel:
>>> -                datalines.append(line)
>>> -            elif '{' in line:
>>> -                datalines = []
>>> -                datalines.append(line)
>>> -            bracelevel = bracelevel + line.count('{') - line.count('}')
>>> -        if datalines:
>>> -            pdata = json.loads('\n'.join(datalines))
>>> -        return pdata
>>>        def _getdependencies(self, pkg, data, version, d, ud, optional=False,
>>> fetchedlist=None):
>>>            if fetchedlist is None:
>>> @@ -175,9 +156,15 @@ class Npm(FetchMethod):
>>>            logger.debug(2, "Calling getdeps on %s" % pkg)
>>>            fetchcmd = "npm view %s --json --registry %s" % (pkgfullname, ud.registry)
>>>            output = runfetchcmd(fetchcmd, d, True)
>>> -        pdata = self._parse_view(output)
>>> -        if not pdata:
>>> +        if not output:
>>>                raise FetchError("The command '%s' returned no output" % fetchcmd)
>>> +        pdata = json.loads(output)
>>> +
>>> +        # npm registry will return multiple dicts in a list when there is more than one
>>> matching
>>> +        # version. Take the last one.
>>> +        if isinstance(pdata, list):
>>> +            pdata = pdata[-1]
>>> +
>>>            if optional:
>>>                pkg_os = pdata.get('os', None)
>>>                if pkg_os:
>>>
>>
>