[OE-core] [RFC][PATCH 0/6] NPM refactoring

[Jean-Marie LEMETAYER]

Hi Stefan,

On Oct 24, 2019, at 2:01 PM, Stefan Herbrechtsmeier stefan at herbrechtsmeier.net wrote:
> Am 22.10.19 um 13:22 schrieb Richard Purdie:
>> On Tue, 2019-10-22 at 11:03 +0200, Jean-Marie LEMETAYER wrote:
>>> The current NPM support have several issues:
>>>   - The current NPM fetcher downloads the dependency tree but not the other
>>>     fetchers. The 'subdir' parameter was used to fix this issue.
>>>   - They are multiple issues with package names (uppercase, exotic characters,
>>>     scoped packages) even if they are inside the dependencies.
>>>   - The lockdown file generation have issues. When a package depends on
>>>     multiple version of the same package (all versions have the same checksum).
>>>
>>> This patchset refactors the NPM support in Yocto:
>>>   - As the NPM algorithm for dependency management is hard to handle, the new
>>>     NPM fetcher downloads only the package source (and not the dependencies,
>>>     like the other fetchers) (patch submitted in the bitbake-devel list).
> 
> What make the new fetcher different from the simple wget fetcher?

The fetcher asks the registry for the tarball url and its integrity. Then it uses wget to download the tarball and then check the integrity given by the registry. Finally it unpacks the tarball according to the tarball format (the root directory is called 'package' which must be renamed).

>>>   - The NPM class handles the dependencies using NPM (and not manually).
> 
> Is this really an improvement? NPM will do the cross compile during
> fetch, loads additionally archives (not packages) from the internet and
> doesn't reuse dependencies.

In the next version of this patchset the dependencies are cached in a npm_cache directory stored in the DL_DIR during the do_fetch_append task (using 'npm cache --cache=${DL_DIR}/npm_cache'). The 'npm install' doesn't require any internet accesses because it is run with '--offline' option.

>>>   - The NPM recipe creation is simplified to avoid issues.
> 
> We create new not obvious issues. How you would handle prebuild binaries?

I have no case of using prebuild binaries. After checking some informations it seems that it is handled the same way as node-gyp (by using scripts in the package.json). A patchset to support that is welcome. 

>>>   - The lockdown file is no more used as it is no longer relevant compared to the
>>>     latest shrinkwrap file format.
>>>
>>> This patchset may remove some features (lockdown file, license management for
>>> dependencies)
> 
> You really remove the license management of the dependencies? I think a
> main feature of OE is the license management.

The previous npm support was creating subpackages to handle the license management of the dependencies. This is a good idea but it fails with package with exotic names. A work will be needed to cleanly remake it.

>> but fixes the majority of the NPM issues. All of these issues
>>> from the bugzilla.yoctoproject.org are resolved by this patchset:
>>> #10237, #10760, #11028, #11728, #11902, #12534
>> 
>> One key requirement which many of our users have from the fetcher is
>> that its deterministic and allows for "offline" builds.
> 
> I think this is impossible with npm because every dependency could run a
> script and download additional files (ex. prebuild).

I have no case of using prebuild binaries. Can you give me an example of a package using prebuild? Do you know about the '--offline' and the '--ignore-scripts' options? A patchset to support that is welcome.

>> What this means is that should I have a populated DL_DIR, the build
>> should not need to touch the network. Also, only do_fetch tasks would
>> make network accesses.
> 
> @Richard: What is your opinion about the per recipe dependency?
> Typically OE use one recipe per project. The NPM based solution handle a
> project and all dependencies via one recipe.
> 
> @Jean-Marie: Do you know PNPM? They use a different node_modules layout
> which allows the reuse of dependencies.

Thanks for the link. What is the value in our scope ?

Regards,
Jean-Marie