[OE-core] Status of the NPM refactoring

[Jean-Marie LEMETAYER]

Hi folks,

I am currently trying to update/refactor the handling of the NPM packages.

Here is the history of my patchset:

V1:
http://lists.openembedded.org/pipermail/openembedded-core/2019-October/288179.html
http://lists.openembedded.org/pipermail/bitbake-devel/2019-October/020435.html

V2:
http://lists.openembedded.org/pipermail/openembedded-core/2019-October/288352.html
http://lists.openembedded.org/pipermail/bitbake-devel/2019-October/020449.html

V3:
http://lists.openembedded.org/pipermail/openembedded-core/2019-November/289431.html
http://lists.openembedded.org/pipermail/bitbake-devel/2019-November/020532.html

----

I first tried to separate the fetching of a single npm package (with a "npm://"
fetcher) and the fetching of its dependencies (managed in the npm.bbclass
class). The good thing is that the initial package could be fetched by another
fetcher ("git://", "http://" ...) and still get the dependencies by inheriting
the npm class based on a npm-shrinkwrap.json file [1].

1: https://docs.npmjs.com/files/shrinkwrap.json

In the latest version, Richard suggest to create another "npmsw://" fetcher to
get the dependencies. The fetching of the package and its dependencies is still
divided AND the npm class no longer fetches anything, which is much cleaner.

Here is a visual example of what a recipe could be:

    SRC_URI = "npm://registry.npmjs.org;name=my-package;version=${PV} \
               npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json"

    S = "${WORKDIR}/npm"

    inherit npm

----

Thinking about these fetchers I realized that they are in fact "proxy fetchers"
as they just generate new URIs that are already handled by bitbake
(mainly "http://" but also "git://" and "file://").

The first fetcher "npm://" runs the "npm view" command [2] to get the tarball
URI. There is only one URI for this one. The second fetcher have to manage
multiple URIs which are already resolved in the npm-shrinkwrap.json file.

2: https://docs.npmjs.com/cli/view.html

To achieve this I have to correctly configure the localfile / localpath of the
fetch files and manage the donestamp file and the lockfile. I may need to tweak
a little these parts.

----

Another part is about the checksums. To be able to fetch something using
"http://" a checksum must be set (md5 or sha256). On its side NPM provides
shasum (sha1) or integrity (sha256, sha384 or sha512) [3].

3: https://docs.npmjs.com/files/package-lock.json#integrity

For this to work, I intend to add sha1, sha384 and sha512 support for fetchers.
I would like to post these patches before the others because they are not
directly related to NPM.

----

Is it OK ? Any thought ? Any advice ?

Best Regards,

Jean-Marie LEMETAYER
Enthusiast embedded systems engineer
Savoir-faire Linux