[OE-core] [PATCH v6 00/13] NPM refactoring

[Jean-Marie LEMETAYER]

Hi folks and happy new year,

For readability here is a link if you want the history of this patchset:
http://lists.openembedded.org/pipermail/openembedded-core/2019-December/290298.html

--- V4

The patches can be found here:
https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring-v4

 - Two fetchers are available: npm and npmsw

 - The npm fetcher is used to fetch a single package from a registry.

 - The npmsw fetcher is used to fetch the dependencies of a package. It uses a
   npm shrinkwrap file to list the dependencies. The file can be edited
   manually if some dependencies needs to be updated.

 - Both fetcher are independent.

   For example you can have a main package using npm:
     SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \
                npmsw://${THISDIR}/npm-shrinkwrap.json"

   The main package can also be fetched another way without changing anything:
     SRC_URI = "git://github.com/foo/bar.git;protocol=https \
                npmsw://${THISDIR}/npm-shrinkwrap.json"

 - The npm class is in charge of executing the 'npm install'. This installation
   is executed completely offline. The package sources can be patched safely
   before the build.

 - The npm class populates a local npm cache during the do_configure task. As
   this step can take time the support of a progress bar have been added.

 - The devtool and recipetool scripts have been updated to match the new
   fetchers and class. The licences are handled and split by dependencies.

 - Both fetcher, the recipe creation and the recipe build are covered by tests.

 - Using devtool I have successfully generated and built recipe for:
   - My test application:
       https://github.com/savoirfairelinux/node-server-example
   - The angular cli which will be used to build angular applications:
       https://cli.angular.io
   - The mozilla iot gateway which is a big project with a lot of dependencies:
       https://github.com/mozilla-iot/gateway
     To build this project you have to manually add this line in the recipe:
       DEPENDS = "nanomsg sqlite3 libuv"

--- V5

The patches can be found here:
https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring-v5

 - As suggested by Christopher, the class now use the shlex.quote() function
   to handle arguments to shell functions.

--- V6

The patches can be found here:
https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring-v6

The npm class needs nodejs-native which is included in the
meta-openembedded/meta-oe layer.

recipetool have a check to ensure that nodejs-native is available and ready to
be used. But it cannot add the layer on its own. So to be able to run the npm
tests on the CI the meta-openembedded/meta-oe layer have to be added. There was
no issue before because there was no npm test before.

It may be an argument to add the nodejs recipe into OE-core.

Jean-Marie LEMETAYER (13):
  classes/npm: refactor the npm class
  classes/npm: restrict the build to be offline
  classes/npm: use the local node headers
  classes/npm: use the native python
  classes/npm: force to rebuild the prebuild addons
  devtool: npm: rename npm command line options
  recipetool/create_npm: refactor the npm recipe creation handler
  recipetool/create_npm: handle the licenses of the dependencies
  lib/oe/package: remove unneeded npm_split_package_dirs function
  devtool/standard: npm: update the append file
  recipetool/create: npm: remove the 'noverify' url parameter
  oeqa/selftest/recipetool: add npm recipe creation test
  oeqa/selftest/devtool: add npm recipe build test

 meta/classes/npm.bbclass                   | 363 ++++++++++++---
 meta/lib/oe/package.py                     |  33 --
 meta/lib/oeqa/selftest/cases/devtool.py    |  20 +
 meta/lib/oeqa/selftest/cases/recipetool.py |  25 +
 scripts/lib/devtool/standard.py            |  18 +-
 scripts/lib/recipetool/create.py           |  10 +-
 scripts/lib/recipetool/create_npm.py       | 514 +++++++++------------
 7 files changed, 567 insertions(+), 416 deletions(-)

--
2.20.1