[OE-core] [PATCH v3 00/17] NPM refactoring

[Jean-Marie LEMETAYER]

The current NPM support have several issues:
 - The current NPM fetcher downloads the dependency tree but not the other
   fetchers. The 'subdir' parameter was used to fix this issue.
 - They are multiple issues with package names (uppercase, exotic characters,
   scoped packages) even if they are inside the dependencies.
 - The lockdown file generation have issues. When a package depends on
   multiple version of the same package (all versions have the same checksum).

This patchset refactors the NPM support in Yocto:
 - As the NPM algorithm for dependency management is hard to handle, the new
   NPM fetcher downloads only the package source (and not the dependencies,
   like the other fetchers) (patch submitted in the bitbake-devel list).
 - The NPM class handles the dependencies using NPM (and not manually).
 - The NPM recipe creation is simplified to avoid issues.
 - The lockdown file is no more used as it is no longer relevant compared to the
   latest shrinkwrap file format.

This patchset may remove some features (lockdown file, license management for
dependencies) but fixes the majority of the NPM issues. All of these issues
from the bugzilla.yoctoproject.org are resolved by this patchset:
#10237, #10760, #11028, #11728, #11902, #12534

The fetcher and recipetool are now aware of a 'latest' keyword for the version
which allow to build the latest version available on the registry. This feature
fixes the two issues: #10515, #11029

Moreover the issue #13415 should also be fixed but I cannot test it.

I have tested the recipe creation and builds using a self made example to
generate build failures:
 - https://github.com/savoirfairelinux/node-server-example
 - https://npmjs.com/package/@savoirfairelinux/node-server-example

The test steps are these ones:
  $ source poky/oe-init-build-env
  $ bitbake-layers add-layer ../meta-openembedded/meta-oe
  $ devtool add "npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest"
  $ devtool build savoirfairelinux-node-server-example
  $ bitbake-layers create-layer ../meta-test
  $ bitbake-layers add-layer ../meta-test
  $ devtool finish savoirfairelinux-node-server-example ../meta-test
  $ echo IMAGE_INSTALL_append = '" savoirfairelinux-node-server-example"' >> conf/local.conf
  $ bitbake core-image-minimal

Also the 'devtool add' url could be one of these:
 - npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest
   This url uses the new npm fetcher and request the latest version available on
   the registry.
 - npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0
   This url uses the new npm fetcher and request a fixed version.
 - git://github.com/savoirfairelinux/node-server-example.git;protocol=https
   This url uses the git fetcher. As the dependencies are managed by the NPM
   class, any fetcher can be used.

When this patchset will be merged, I have planned to update the NPM wiki:
  https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM

This patchset is also the base work for the full Angular support in Yocto that I
am preparing. These applications have a huge dependency tree which is ideal to
test the NPM support.

--- V2

 - Add the 'check_network_access' function before each network access to check
   for 'BB_NO_NETWORK' and 'BB_ALLOWED_NETWORKS' variables.

 - Add a 'recipetool.RecipetoolTests.test_recipetool_create_npm' test case for
   'oe-selftest' to test the npm recipe creation.

 - Update the 'npm.bbclass' to fetch the dependencies in the 'do_fetch' task.
   The dependencies are cached in a npm cache stored in '${DL_DIR}/npm_cache'
   allowing the dependencies to be saved in the download directory and verify
   on insertion and extraction [1]:
      "All data that passes through the cache is fully verified
       for integrity on both insertion and extraction."

1: https://docs.npmjs.com/cli/cache.html

--- V3

 - Add a test regarding the 'devtool add' and 'devtool build' command:
     - devtool.DevtoolAddTests.test_devtool_add_npm

 - Add the license management for dependencies.

 - Split the npm workflow to use the bitbake default tasks (do_fetch, do_unpack,
   do_compile, do_install). Make sure that only the do_fetch task requires
   network access.

 - Split the commits for better understanding.

 - Force the rebuild of prebuild addons.

 - Use ${nonarch_libdir} instead of ${libdir} in the do_install task.

  - These patches can be found here:
     - https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring-v3
     - https://github.com/savoirfairelinux/poky/tree/npm-refactoring-v3

Jean-Marie LEMETAYER (17):
  devtool: npm: update command line options
  npm.bbclass: refactor the npm class
  recipetool/create_npm.py: refactor the npm recipe creation handler
  lib/oe/package.py: remove unneeded npm_split_package_dirs function
  devtool/standard.py: npm: update the append file
  recipetool/create.py: npm: remove the 'noverify' url parameter
  recipetool/create.py: npm: replace the 'latest' keyword
  npm.bbclass: split the do_compile task
  devtool/standard.py: npm: exclude the node_modules directory
  recipetool/create_npm.py: handle the licenses of the dependencies
  npm.bbclass: restrict the build to be offline
  npm.bbclass: use the local node headers
  recipetool/create_npm.py: convert the shrinkwrap file
  npm.bbclass: force to rebuild the prebuild addons
  devtool/standard.py: npm: configure the NPM_CACHE_DIR to be persistent
  oeqa/selftest/recipetool: add npm recipe creation test
  oeqa/selftest/devtool: add npm recipe build test

 meta/classes/npm.bbclass                   | 224 +++++---
 meta/lib/oe/package.py                     |  33 --
 meta/lib/oeqa/selftest/cases/devtool.py    |  20 +
 meta/lib/oeqa/selftest/cases/recipetool.py |  21 +
 scripts/lib/devtool/standard.py            |  31 +-
 scripts/lib/recipetool/create.py           |  16 +-
 scripts/lib/recipetool/create_npm.py       | 565 +++++++++++----------
 7 files changed, 510 insertions(+), 400 deletions(-)

--
2.20.1