[OE-core] [RFC][PATCH 0/6] NPM refactoring

[André Draszik]

Hi,

This has been an interesting discussion so far.

I'd like to throw in something else...

A couple years back I wrote a little python script to automatically
generate all the required dependency recipes given an npm project's
package.json

The generated recipes worked very well, including cross-compilation using
node-gyp.

At least at the time this was all reasonably straight forward, avoided *any* use
of npm, gave me all the benefits of reproducibility, yocto caching, license
management, correct cross-compilation for native code, etc. Also, the
generated yocto packages contained none of the test- or other items that
npm packages usually contain and don't need to be copied onto the target
for image builds. All this by simply inspecting the package.json

This approach also minimised maintenance cost, as all recipes were auto-generated.

The only downside was that bitbake became a bit slow, as the number of
tasks went to about 15000.

I can dig this out and share on Monday. This script could live in the
scripts/ subdirectory, allowing people to create recipes on demand for
projects they care about.


Cheers,
Andre'


On Tue, 2019-10-22 at 11:03 +0200, Jean-Marie LEMETAYER wrote:
> The current NPM support have several issues:
>  - The current NPM fetcher downloads the dependency tree but not the other
>    fetchers. The 'subdir' parameter was used to fix this issue.
>  - They are multiple issues with package names (uppercase, exotic characters,
>    scoped packages) even if they are inside the dependencies.
>  - The lockdown file generation have issues. When a package depends on
>    multiple version of the same package (all versions have the same checksum).
> 
> This patchset refactors the NPM support in Yocto:
>  - As the NPM algorithm for dependency management is hard to handle, the new
>    NPM fetcher downloads only the package source (and not the dependencies,
>    like the other fetchers) (patch submitted in the bitbake-devel list).
>  - The NPM class handles the dependencies using NPM (and not manually).
>  - The NPM recipe creation is simplified to avoid issues.
>  - The lockdown file is no more used as it is no longer relevant compared to the
>    latest shrinkwrap file format.
> 
> This patchset may remove some features (lockdown file, license management for
> dependencies) but fixes the majority of the NPM issues. All of these issues
> from the bugzilla.yoctoproject.org are resolved by this patchset:
> #10237, #10760, #11028, #11728, #11902, #12534
> 
> The fetcher and recipetool are now aware of a 'latest' keyword for the version
> which allow to build the latest version available on the registry. This feature
> fixes the two issues: #10515, #11029
> 
> Moreover the issue #13415 should also be fixed but I cannot test it.
> 
> I have tested the recipe creation and builds using a self made example to
> generate build failures:
>  - https://github.com/savoirfairelinux/node-server-example
>  - https://npmjs.com/package/@savoirfairelinux/node-server-example
> 
> The test steps are these ones:
>   $ source poky/oe-init-build-env
>   $ bitbake-layers add-layer ../meta-openembedded/meta-oe
>   $ devtool add "npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest"
>   $ devtool build savoirfairelinux-node-server-example
>   $ bitbake-layers create-layer ../meta-test
>   $ bitbake-layers add-layer ../meta-test
>   $ devtool finish savoirfairelinux-node-server-example ../meta-test
>   $ echo IMAGE_INSTALL_append = '" savoirfairelinux-node-server-example"' >> conf/local.conf
>   $ bitbake core-image-minimal
> 
> Also the 'devtool add' url could be one of these:
>  - npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest
>    This url uses the new npm fetcher and request the latest version available on
>    the registry.
>  - npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0
>    This url uses the new npm fetcher and request a fixed version.
>  - git://github.com/savoirfairelinux/node-server-example.git;protocol=https
>    This url uses the git fetcher. As the dependencies are managed by the NPM
>    class, any fetcher can be used.
> 
> When this patchset will be merged, I have planned to update the NPM wiki:
>   https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM
> 
> This patchset is also the base work for the full Angular support in Yocto that I
> am preparing. These applications have a huge dependency tree which is ideal to
> test the NPM support.
> 
> Jean-Marie LEMETAYER (6):
>   npm.bbclass: refactor the npm class
>   devtool: update command line options for npm
>   recipetool/create_npm.py: refactor the npm recipe creation handler
>   devtool/standard.py: update the append file for the npm recipes
>   recipetool/create.py: replace 'latest' keyword for npm
>   recipetool/create.py: remove the 'noverify' url parameter
> 
>  meta/classes/npm.bbclass             | 210 +++++++++----
>  scripts/lib/devtool/standard.py      |  22 +-
>  scripts/lib/recipetool/create.py     |  16 +-
>  scripts/lib/recipetool/create_npm.py | 444 ++++++++++-----------------
>  4 files changed, 331 insertions(+), 361 deletions(-)
> 
> --
> 2.20.1
>