[OE-core] [PATCH v3 00/17] NPM refactoring

[Richard Purdie]

On Thu, 2019-11-21 at 14:53 -0500, Jean-Marie LEMETAYER wrote:
> On Nov 21, 2019, at 7:26 PM, Richard Purdie 
> richard.purdie at linuxfoundation.org wrote:
> > On Wed, 2019-11-20 at 10:33 +0100, Jean-Marie LEMETAYER wrote:
> > > The current NPM support have several issues:
> > >  - The current NPM fetcher downloads the dependency tree but not
> > > the other
> > >    fetchers. The 'subdir' parameter was used to fix this issue.
> > >  - They are multiple issues with package names (uppercase, exotic
> > > characters,
> > >    scoped packages) even if they are inside the dependencies.
> > >  - The lockdown file generation have issues. When a package
> > > depends on
> > >    multiple version of the same package (all versions have the
> > > same checksum).
> > > 
> > > This patchset refactors the NPM support in Yocto:
> > >  - As the NPM algorithm for dependency management is hard to
> > > handle, the new
> > >    NPM fetcher downloads only the package source (and not the
> > > dependencies,
> > >    like the other fetchers) (patch submitted in the bitbake-devel 
> > > list).
> > 
> > I'm a little confused by this item.
> > 
> > If the NPM fetcher only downloads a given package's source and it
> > has
> > dependencies, where/when are the dependencies downloaded?
> > 
> > My worry here is that if the fetcher isn't downloading them, DL_DIR
> > can't contain all the needed pieces needed to reproduce a build at
> > a
> > later date?
> > 
> > I suspect I'm missing something obvious...
> 
> The magic is in the bbclass, in the do_fetch_append() function:
> https://github.com/savoirfairelinux/poky/blob/npm-refactoring-v3/meta/classes/npm.bbclass#L51
> 
> Which calls:
> https://github.com/savoirfairelinux/poky/blob/npm-refactoring-v3/bitbake/lib/bb/fetch2/npm.py#L312
> 
> Which runs the npm fetcher for each dependencies described in the
> shrinkwrap file. This is the same behavior for the do_unpack task.
> 
> 
> To be more clear, a recipe like this:
> 
> ```
> SRC_URI = "npm://registry.npmjs.org/;name=my-package;version=1.0.0"
> S = "${WORKDIR}/npm"
> ```
> 
> Will only fetch/unpack the package source without handling the
> dependencies.
> 
> But a recipe like this:
> 
> ```
> SRC_URI = "npm://registry.npmjs.org/;name=my-package;version=1.0.0"
> S = "${WORKDIR}/npm"
> 
> NPM_SHRINKWRAP = "${THISDIR}/${BPN}/npm-shrinkwrap.json"
> inherit npm
> ```
> 
> Will fetch/unpack the package and all the dependencies which are
> described in the shrinkwrap file.

That does remove some of my worries, thanks for explaining!

It does make me wonder if we shouldn't have two fetch classes and then
support a url like:
 
SRC_URI = "npm://registry.npmjs.org/;name=my-package;version=1.0.0 \
           npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json"

where the "npmsw" handler would handle the shrinkwrap file?

Basically I don't like the way the current code has to tag the
shrinkwrap file handling on to the fetcher...

> Bonus tips: you can have your base package SRC_URI using another
> fetcher than npm (e.g. git) without breaking anything. The behavior
> will still be the same.

That does start to make it clearer why this is advantageous.

Cheers,

Richard