[OE-core] [RFC][PATCH 0/6] NPM refactoring

[Jean-Marie LEMETAYER]

Hi Richard,

On Oct 24, 2019, at 3:36 PM, Richard Purdie richard.purdie at linuxfoundation.org wrote:
> On Thu, 2019-10-24 at 14:12 +0200, Alexander Kanavin wrote:
>> On Thu, 24 Oct 2019 at 14:02, Stefan Herbrechtsmeier <
>> stefan at herbrechtsmeier.net> wrote:
>> > @Richard: What is your opinion about the per recipe dependency?
>> > Typically OE use one recipe per project. The NPM based solution
>> > handle a
>> > project and all dependencies via one recipe.
>> 
>> I don't think it's at all realistic to stick to the 'one recipe per
>> component' in node.js world. A typical 'npm install' can pull down
>> hundreds, or over a thousand dependencies, it's not feasible to have
>> a recipe for each.
>> 
>> I very much welcome a solution that uses 'npm install' in a way that
>> preserves offline builds, and integrity/reproducibility of downloads.
>> License management should be also handled by npm, and if it isn't,
>> then we need to work with the upstream to address it.
> 
> I understand however keep in mind the way this patch series has been
> going, it could end up simply forceing all processing into the do_fetch
> task.
> 
> We need determinism form the build in that building this today should
> give the same result as a build run in X years time, assuming the same
> host OS and so on, even if DL_DIR isn't populated. The state of the
> internet should not change that.
> 
> I worry about the amount of magic "npm install" has going on which
> would mean we couldn't achieve this.

I have almost finish a new version for this patchset. All network accesses are now done during the do_fetch and verified using the check_network_access function.

The npm-shrinkwrap.json file ensures that the generated tree will be reproducible. It describes the wanted dependency tree and provides an integrity check for each packages. When using this file the build is totally predictable. Of course this file is mandatory.

Regards,
Jean-Marie