[OE-core] [RFC][PATCH 0/6] NPM refactoring

[Stefan Herbrechtsmeier]

Hi Jean-Marie,

Am 22.10.19 um 13:22 schrieb Richard Purdie:
> On Tue, 2019-10-22 at 11:03 +0200, Jean-Marie LEMETAYER wrote:
>> The current NPM support have several issues:
>>   - The current NPM fetcher downloads the dependency tree but not the other
>>     fetchers. The 'subdir' parameter was used to fix this issue.
>>   - They are multiple issues with package names (uppercase, exotic characters,
>>     scoped packages) even if they are inside the dependencies.
>>   - The lockdown file generation have issues. When a package depends on
>>     multiple version of the same package (all versions have the same checksum).
>>
>> This patchset refactors the NPM support in Yocto:
>>   - As the NPM algorithm for dependency management is hard to handle, the new
>>     NPM fetcher downloads only the package source (and not the dependencies,
>>     like the other fetchers) (patch submitted in the bitbake-devel list).

What make the new fetcher different from the simple wget fetcher?

>>   - The NPM class handles the dependencies using NPM (and not manually).

Is this really an improvement? NPM will do the cross compile during 
fetch, loads additionally archives (not packages) from the internet and 
doesn't reuse dependencies.

>>   - The NPM recipe creation is simplified to avoid issues.

We create new not obvious issues. How you would handle prebuild binaries?

>>   - The lockdown file is no more used as it is no longer relevant compared to the
>>     latest shrinkwrap file format.
>>
>> This patchset may remove some features (lockdown file, license management for
>> dependencies)

You really remove the license management of the dependencies? I think a 
main feature of OE is the license management.

> but fixes the majority of the NPM issues. All of these issues
>> from the bugzilla.yoctoproject.org are resolved by this patchset:
>> #10237, #10760, #11028, #11728, #11902, #12534
> 
> One key requirement which many of our users have from the fetcher is
> that its deterministic and allows for "offline" builds.

I think this is impossible with npm because every dependency could run a 
script and download additional files (ex. prebuild).

> What this means is that should I have a populated DL_DIR, the build
> should not need to touch the network. Also, only do_fetch tasks would
> make network accesses.

@Richard: What is your opinion about the per recipe dependency? 
Typically OE use one recipe per project. The NPM based solution handle a 
project and all dependencies via one recipe.

@Jean-Marie: Do you know PNPM? They use a different node_modules layout 
which allows the reuse of dependencies.

Regards
   Stefan