<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 20, 2019, at 1:33 AM, Jean-Marie LEMETAYER <<a href="mailto:jean-marie.lemetayer@savoirfairelinux.com" class="">jean-marie.lemetayer@savoirfairelinux.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">The current NPM support have several issues:<br class=""> - The current NPM fetcher downloads the dependency tree but not the other<br class=""> fetchers. The 'subdir' parameter was used to fix this issue.<br class=""> - They are multiple issues with package names (uppercase, exotic characters,<br class=""> scoped packages) even if they are inside the dependencies.<br class=""> - The lockdown file generation have issues. When a package depends on<br class=""> multiple version of the same package (all versions have the same checksum).<br class=""><br class="">This patchset refactors the NPM support in Yocto:<br class=""> - As the NPM algorithm for dependency management is hard to handle, the new<br class=""> NPM fetcher downloads only the package source (and not the dependencies,<br class=""> like the other fetchers) (patch submitted in the bitbake-devel list).<br class=""> - The NPM class handles the dependencies using NPM (and not manually).<br class=""> - The NPM recipe creation is simplified to avoid issues.<br class=""> - The lockdown file is no more used as it is no longer relevant compared to the<br class=""> latest shrinkwrap file format.<br class=""><br class="">This patchset may remove some features (lockdown file, license management for<br class="">dependencies) but fixes the majority of the NPM issues. All of these issues<br class="">from the <a href="http://bugzilla.yoctoproject.org" class="">bugzilla.yoctoproject.org</a> are resolved by this patchset:<br class="">#10237, #10760, #11028, #11728, #11902, #12534<br class=""><br class="">The fetcher and recipetool are now aware of a 'latest' keyword for the version<br class="">which allow to build the latest version available on the registry. This feature<br class="">fixes the two issues: #10515, #11029<br class=""><br class="">Moreover the issue #13415 should also be fixed but I cannot test it.<br class=""><br class="">I have tested the recipe creation and builds using a self made example to<br class="">generate build failures:<br class=""> - <a href="https://github.com/savoirfairelinux/node-server-example" class="">https://github.com/savoirfairelinux/node-server-example</a><br class=""> - <a href="https://npmjs.com/package/@savoirfairelinux/node-server-example" class="">https://npmjs.com/package/@savoirfairelinux/node-server-example</a><br class=""><br class="">The test steps are these ones:<br class=""> $ source poky/oe-init-build-env<br class=""> $ bitbake-layers add-layer ../meta-openembedded/meta-oe<br class=""> $ devtool add "<a href="npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest" class="">npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest</a>”</div></div></blockquote><div><br class=""></div><div>I noticed this before but forgot to report it:</div><div><br class=""></div>ERROR: Nothing PROVIDES 'c-ares-native' (but virtual:native:/build/meta-openembedded/meta-oe/recipes-devtools/nodejs/nodejs_10.17.0.bb DEPENDS on or otherwise requires it). ...</div><div><a href="http://layers.openembedded.org/layerindex/recipe/50318/" class="">http://layers.openembedded.org/layerindex/recipe/50318/</a></div><div>$ bitbake-layers add-layer ../meta-openembedded/meta-networking</div><div>Which requires</div><div>$ bitbake-layers add-layer ../meta-openembedded/meta-python</div><div><br class=""></div><div><br class=""></div><div>Using tip of bitbake with V3, openembedded-core with V3 and meta-openembedded master, somehow nodejs-native do_compile broke (at least on qemux86-64) since the last time I built it shortly after ELC-E…</div><div>So now I’ll have to git bisect that…more to come…</div><div><br class=""></div><div>FWIW I had previously successfully run your instructions and the tests for both bitbake and oe-selftest for V2 (distroless, e.g. not with poky). Something unrelated has broken I suspect.</div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><div class=""> $ devtool build savoirfairelinux-node-server-example<br class=""> $ bitbake-layers create-layer ../meta-test<br class=""> $ bitbake-layers add-layer ../meta-test<br class=""> $ devtool finish savoirfairelinux-node-server-example ../meta-test<br class=""> $ echo IMAGE_INSTALL_append = '" savoirfairelinux-node-server-example"' >> conf/local.conf<br class=""> $ bitbake core-image-minimal<br class=""><br class="">Also the 'devtool add' url could be one of these:<br class=""> - <a href="npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest" class="">npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest</a><br class=""> This url uses the new npm fetcher and request the latest version available on<br class=""> the registry.<br class=""> - <a href="npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0" class="">npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0</a><br class=""> This url uses the new npm fetcher and request a fixed version.<br class=""> - <a href="git://github.com/savoirfairelinux/node-server-example.git;protocol=https" class="">git://github.com/savoirfairelinux/node-server-example.git;protocol=https</a><br class=""> This url uses the git fetcher. As the dependencies are managed by the NPM<br class=""> class, any fetcher can be used.<br class=""><br class="">When this patchset will be merged, I have planned to update the NPM wiki:<br class=""> <a href="https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM" class="">https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM</a><br class=""><br class="">This patchset is also the base work for the full Angular support in Yocto that I<br class="">am preparing. These applications have a huge dependency tree which is ideal to<br class="">test the NPM support.<br class=""><br class="">--- V2<br class=""><br class=""> - Add the 'check_network_access' function before each network access to check<br class=""> for 'BB_NO_NETWORK' and 'BB_ALLOWED_NETWORKS' variables.<br class=""><br class=""> - Add a 'recipetool.RecipetoolTests.test_recipetool_create_npm' test case for<br class=""> 'oe-selftest' to test the npm recipe creation.<br class=""><br class=""> - Update the 'npm.bbclass' to fetch the dependencies in the 'do_fetch' task.<br class=""> The dependencies are cached in a npm cache stored in '${DL_DIR}/npm_cache'<br class=""> allowing the dependencies to be saved in the download directory and verify<br class=""> on insertion and extraction [1]:<br class=""> "All data that passes through the cache is fully verified<br class=""> for integrity on both insertion and extraction."<br class=""><br class="">1: <a href="https://docs.npmjs.com/cli/cache.html" class="">https://docs.npmjs.com/cli/cache.html</a><br class=""><br class="">--- V3<br class=""><br class=""> - Add a test regarding the 'devtool add' and 'devtool build' command:<br class=""> - devtool.DevtoolAddTests.test_devtool_add_npm<br class=""><br class=""> - Add the license management for dependencies.<br class=""><br class=""> - Split the npm workflow to use the bitbake default tasks (do_fetch, do_unpack,<br class=""> do_compile, do_install). Make sure that only the do_fetch task requires<br class=""> network access.<br class=""><br class=""> - Split the commits for better understanding.<br class=""><br class=""> - Force the rebuild of prebuild addons.<br class=""><br class=""> - Use ${nonarch_libdir} instead of ${libdir} in the do_install task.<br class=""><br class=""> - These patches can be found here:<br class=""> - <a href="https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring-v3" class="">https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring-v3</a><br class=""> - <a href="https://github.com/savoirfairelinux/poky/tree/npm-refactoring-v3" class="">https://github.com/savoirfairelinux/poky/tree/npm-refactoring-v3</a><br class=""><br class="">Jean-Marie LEMETAYER (17):<br class=""> devtool: npm: update command line options<br class=""> npm.bbclass: refactor the npm class<br class=""> recipetool/create_npm.py: refactor the npm recipe creation handler<br class=""> lib/oe/package.py: remove unneeded npm_split_package_dirs function<br class=""> devtool/standard.py: npm: update the append file<br class=""> recipetool/create.py: npm: remove the 'noverify' url parameter<br class=""> recipetool/create.py: npm: replace the 'latest' keyword<br class=""> npm.bbclass: split the do_compile task<br class=""> devtool/standard.py: npm: exclude the node_modules directory<br class=""> recipetool/create_npm.py: handle the licenses of the dependencies<br class=""> npm.bbclass: restrict the build to be offline<br class=""> npm.bbclass: use the local node headers<br class=""> recipetool/create_npm.py: convert the shrinkwrap file<br class=""> npm.bbclass: force to rebuild the prebuild addons<br class=""> devtool/standard.py: npm: configure the NPM_CACHE_DIR to be persistent<br class=""> oeqa/selftest/recipetool: add npm recipe creation test<br class=""> oeqa/selftest/devtool: add npm recipe build test<br class=""><br class=""> meta/classes/npm.bbclass | 224 +++++---<br class=""> meta/lib/oe/package.py | 33 --<br class=""> meta/lib/oeqa/selftest/cases/devtool.py | 20 +<br class=""> meta/lib/oeqa/selftest/cases/recipetool.py | 21 +<br class=""> scripts/lib/devtool/standard.py | 31 +-<br class=""> scripts/lib/recipetool/create.py | 16 +-<br class=""> scripts/lib/recipetool/create_npm.py | 565 +++++++++++----------<br class=""> 7 files changed, 510 insertions(+), 400 deletions(-)<br class=""><br class="">--<br class="">2.20.1<br class=""><br class=""></div></div></blockquote></div><br class=""></body></html>