[OE-core] [PATCH v3 00/17] NPM refactoring

[Jean-Marie LEMETAYER]

Hi Richard,

On Nov 21, 2019, at 7:26 PM, Richard Purdie richard.purdie at linuxfoundation.org wrote:
> On Wed, 2019-11-20 at 10:33 +0100, Jean-Marie LEMETAYER wrote:
>> The current NPM support have several issues:
>>  - The current NPM fetcher downloads the dependency tree but not the other
>>    fetchers. The 'subdir' parameter was used to fix this issue.
>>  - They are multiple issues with package names (uppercase, exotic characters,
>>    scoped packages) even if they are inside the dependencies.
>>  - The lockdown file generation have issues. When a package depends on
>>    multiple version of the same package (all versions have the same checksum).
>> 
>> This patchset refactors the NPM support in Yocto:
>>  - As the NPM algorithm for dependency management is hard to handle, the new
>>    NPM fetcher downloads only the package source (and not the dependencies,
>>    like the other fetchers) (patch submitted in the bitbake-devel list).
> 
> I'm a little confused by this item.
> 
> If the NPM fetcher only downloads a given package's source and it has
> dependencies, where/when are the dependencies downloaded?
> 
> My worry here is that if the fetcher isn't downloading them, DL_DIR
> can't contain all the needed pieces needed to reproduce a build at a
> later date?
> 
> I suspect I'm missing something obvious...

The magic is in the bbclass, in the do_fetch_append() function:
https://github.com/savoirfairelinux/poky/blob/npm-refactoring-v3/meta/classes/npm.bbclass#L51

Which calls:
https://github.com/savoirfairelinux/poky/blob/npm-refactoring-v3/bitbake/lib/bb/fetch2/npm.py#L312

Which runs the npm fetcher for each dependencies described in the
shrinkwrap file. This is the same behavior for the do_unpack task.


To be more clear, a recipe like this:

```
SRC_URI = "npm://registry.npmjs.org/;name=my-package;version=1.0.0"
S = "${WORKDIR}/npm"
```

Will only fetch/unpack the package source without handling the
dependencies.

But a recipe like this:

```
SRC_URI = "npm://registry.npmjs.org/;name=my-package;version=1.0.0"
S = "${WORKDIR}/npm"

NPM_SHRINKWRAP = "${THISDIR}/${BPN}/npm-shrinkwrap.json"
inherit npm
```

Will fetch/unpack the package and all the dependencies which are
described in the shrinkwrap file.


Bonus tips: you can have your base package SRC_URI using another
fetcher than npm (e.g. git) without breaking anything. The behavior
will still be the same.


Best regards,
Jean-Marie