[bitbake-devel] [PATCH] fetch2/npm: fix parsing multiple matching packages

[Jean-marie Lemetayer]

Hi Zoltán,

I am working on the master branch for now. I have several fixes which are under development. When this work will be done, I will consider backporting all of these fixes.

Best regards,
Jean-Marie

----- Mail original -----
> De: "Böszörményi Zoltán" <zboszor at pr.hu>
> À: "Jean-marie Lemetayer" <jean-marie.lemetayer at savoirfairelinux.com>
> Cc: "mpvader" <mvader at victronenergy.com>, "bitbake-devel" <bitbake-devel at lists.openembedded.org>
> Envoyé: Samedi 5 Octobre 2019 08:47:09
> Objet: Re: [bitbake-devel] [PATCH] fetch2/npm: fix parsing multiple matching packages

> Hi,
> 
> 2019. 10. 04. 9:42 keltezéssel, Jean-marie Lemetayer írta:
>> Hi Zoltán,
>> 
>> It seems that this issue have already been fixed by this patch:
>> https://git.openembedded.org/bitbake/commit/?id=3d319c79981811d3cfd4732885057db4fd5afcc2
>> 
>> Not the same implementation as yours, but it fixes the same issue.
> 
> The implementation is not mine, I just tested it back then with Yocto 2.5.
> 
> Will your patch backported to older major versions of Bitbake?
> The issue is present in older Yocto versions.
> 
>> 
>> Best regards,
>> Jean-Marie Lemetayer
>> 
>> ----- Mail original -----
>> De: "bitbake-devel" <bitbake-devel at lists.openembedded.org>
>> À: "mpvader" <mvader at victronenergy.com>, "bitbake-devel"
>> <bitbake-devel at lists.openembedded.org>
>> Envoyé: Vendredi 4 Octobre 2019 07:58:56
>> Objet: Re: [bitbake-devel] [PATCH] fetch2/npm: fix parsing multiple matching
>> packages
>> 
>> Hi Bitbake devs!
>> 
>> The exception with the below "devtool add ..." command still occurs with
>> bvitbake 1.42.0 used in Yocto 2.7 and is fixed with the patch.
>> 
>> Please consider applying/backporting it.
>> 
>> Best regards,
>> Zoltán Böszörményi
>> 
>> 2018. 02. 19. 9:55 keltezéssel, Böszörményi Zoltán írta:
>>> 2018-02-11 23:31 keltezéssel, mpvader írta:
>>>> Replace the self-implemented parser with loading the complete json and then
>>>> taking the last element from it.
>>>>
>>>> This fixes an error fetching the lex-parser package, as a dependency of, for
>>>> example, jison. Reproduce the error with this command:
>>>>
>>>> devtool add "npm://registry.npmjs.org;name=jison;version=0.4.18"
>>>>
>>>> _parse_view() didn't properly parse the npm view result [1], resulting in
>>>> an Exception in the subsequent json.loads() function:
>>>> Exception: ValueError: Extra data: line 57 column 4 - line 296 column 2 (char
>>>> 5874 - 29698)
>>>
>>> I can confirm this problem exists in Bitbake 1.36 and goes away with the patch.
>>>
>>> Also tested the patch for the handful of node module recipes
>>> I have and they still build properly.
>>>
>>>>
>>>> [1] npm view lex-parser@'0.1.x' --json --registry http://registry.npmjs.org
>>>>
>>>> Signed-off-by: mpvader <mvader at victronenergy.com>
>>>
>>> Tested-by: Zoltán Böszörményi <zboszor at pr.hu>
>>>
>>>> ---
>>>>    lib/bb/fetch2/npm.py | 29 ++++++++---------------------
>>>>    1 file changed, 8 insertions(+), 21 deletions(-)
>>>>
>>>> diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
>>>> index b5f148c..e257355 100644
>>>> --- a/lib/bb/fetch2/npm.py
>>>> +++ b/lib/bb/fetch2/npm.py
>>>> @@ -146,25 +146,6 @@ class Npm(FetchMethod):
>>>>            self._unpackdep(ud, ud.pkgname, workobj, unpackdir, dldir, d)
>>>> -    def _parse_view(self, output):
>>>> -        '''
>>>> -        Parse the output of npm view --json; the last JSON result
>>>> -        is assumed to be the one that we're interested in.
>>>> -        '''
>>>> -        pdata = None
>>>> -        outdeps = {}
>>>> -        datalines = []
>>>> -        bracelevel = 0
>>>> -        for line in output.splitlines():
>>>> -            if bracelevel:
>>>> -                datalines.append(line)
>>>> -            elif '{' in line:
>>>> -                datalines = []
>>>> -                datalines.append(line)
>>>> -            bracelevel = bracelevel + line.count('{') - line.count('}')
>>>> -        if datalines:
>>>> -            pdata = json.loads('\n'.join(datalines))
>>>> -        return pdata
>>>>        def _getdependencies(self, pkg, data, version, d, ud, optional=False,
>>>> fetchedlist=None):
>>>>            if fetchedlist is None:
>>>> @@ -175,9 +156,15 @@ class Npm(FetchMethod):
>>>>            logger.debug(2, "Calling getdeps on %s" % pkg)
>>>>            fetchcmd = "npm view %s --json --registry %s" % (pkgfullname, ud.registry)
>>>>            output = runfetchcmd(fetchcmd, d, True)
>>>> -        pdata = self._parse_view(output)
>>>> -        if not pdata:
>>>> +        if not output:
>>>>                raise FetchError("The command '%s' returned no output" % fetchcmd)
>>>> +        pdata = json.loads(output)
>>>> +
>>>> +        # npm registry will return multiple dicts in a list when there is more
>>>> than one
>>>> matching
>>>> +        # version. Take the last one.
>>>> +        if isinstance(pdata, list):
>>>> +            pdata = pdata[-1]
>>>> +
>>>>            if optional:
>>>>                pkg_os = pdata.get('os', None)
>>>>                if pkg_os:
>>>>
>>>