Node.js Node.js Live Events, 26 Jul 2016

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Alexander Gugel - Node.js Live London

Description

Alexander Gugel talks about rethinking package management in Node.js in this Node.js Live London presentation.

A

I currently work at sky arm and my talk will be basically about package management in ojs and I feel really bad, because I feel, like fifty percent of the talks, so far has been about how awesome and pms and- and this talk is going to be the exact opposite, I apologize for that in advance. So I think currently a package mentioned and no, it is kind of broken.

A

My talk will be basically about how I think we can fix it so about six months ago really I started getting annoyed with NPM performance and, in fact, I start getting annoyed so much that I tried to tear it apart and and look what it was actually doing and at that point I realized that it's actually a really complex problem, but I thought that the NPM approach overall could be improved upon so richly.

A

The main motivation behind this project of trying to replace NPM was really NPR's performance, which was at that point, was mpm to and still is, with, NPM free arm, pretty bad and also unpredictable in certain cases. For instance, the watch this case of a progress bar where the progress ball was actually slowing down insulation, noticeably, which I think is a good indicator for that.

A

Also, the problem with NPM to was there was installing redundant dependencies and I will go into more details later on and it that it's sometimes I think it tries to do too many things. Npm should really be about installing the penalties and nothing else. Ideally, ideally, you would want to separate true for anything else. I think, and also how to debug and npm to just couldn't seems a lot of dissipation. Don't like that.

A

I do so, as I mentioned originally I tried to replace NPM and that really the first iteration for that was in through heart, was trying to recreate the MPN to installation mechanism, which is essentially like nasty dependency graph, which is problematic under windows, as mentioned before, because the maximum path length is limited on windows and it actually went pretty far along so it was able to install its own dependency, was kind of bootstrapping, more or less.

A

It supported semi circular dependencies when correctly and work with about sixty percent of the packages that we were hosting 9pm roughly speaking, and it was pretty atomic and by that I mean it either worked or it blew up completely and didn't modify or node modules at all, and when doing that, I didn't have an initial benchmarking case really, but I try to a couple of scenarios and it turned out to be about twice as fast this NPM and a point, and at that point I really started wondering because I didn't have any caching at all at this point.

A

So that's why? A couple of months later I turned this project which originally I called NPM, which was a pretty bad name. It's getting worse, I start writing project called ID, so I ed really tries to be a complete NPM, alternative and replacement, and the goal really is to concentrate more on the core functionality that NPM provides, which is installation of packages.

A

So when I use NPM I pretty much all only use NPM install because that's the only thing I'm more or less interested in it also has a working progress indicator and works with NPM together. So of course, they're features in NPM that are awesome that some people use, and so you can use them with ID as well, so that project started getting some traction initially and they're. Currently quite some people using it actually.

A

So that's why I start kind of investigating the whole NPM issue further, so maybe a little bit about the current state of ID and then I will go into detail why it's actually faster I di just point support most of the awesome features that were mentioned previously, that NPM support, so it supports life cycle events such as start post, install pre-install things like that I key in it is also supported users actually, the same mechanism that NPM uses or NT esri also never got.

A

It also has a new command called I dsh, which allows you to actually populate your current path, with the modules that you installed locally in your current package. So, for instance, when you install go locally, you can actually just one ID sh, then populate your path, and you can execute cope as if you would have installed it globally. In that project.

A

ah There is also, of course, ID install, which is the exact equivalent to NPM install it works quite differently, but has pretty much the same effect, there's ID link, which is the exact which pretty much does the same thing that npm link does and there's ID build, which bill to dependencies, which is also more or less than p.m. equivalent. There's ID peeing and pretty much all like majority of other commands, except one which I will go into more detail later.

A

All supports private registry and scope modules, so that regard it's pretty much the same, so um I want to give you a quick demo, really am and I was afraid of not being able to demo it because I don't know, I was just afraid. ah So let me quickly show you how it actually looks like to install express this without any caching and with a pretty good internet connection, but install an express using ID on this machine index, but like two seconds I'll, which is significantly faster than NPM.

A

If you want to try it out, of course, all benchmarks are flawed, but I made one so browserify. It's really a great example for that, because browserify itself is extremely modular, it has a lot of transitive dependencies that depend on each other and they're like circular dependencies, and it's just a great example for really trying out how fast the package manager really is. So this is, of course, a flawed benchmark.

A

It's all benchmarks, but you can try it out your own, so in this case to show you that I didn't cheat I enable to the silly logging levels like a log level on NPM, so IDs already done and p.m. still takes time.

A

And ya see your point, so I want to go into more detail why ID is faster than NPM.

A

More specifically, why I d install is faster and the main reason is primarily because the control flow is quite differently and in order to manage that our X J's, if you use so pretty much everything in ID, is written RX x for those who don't know its or direct, if extension for javascript, which is a project for dealing with reactive streams, observables and that kind of stuff, it's really nice not going to go into detail how that works.

A

Specifically, I just going to show you some code later and that's essentially the core implementation perspective and difference between an p.m. and ID, I think, and also what's quite different, is siblings are being used instead of having some crazy and p.m. node modules, directory structure, so they're flattened by default, and it only supports wobbles. So that's a fair point that npm supports, keep dependencies and and like can install from pretty much everything github gist. Id doesn't do that and annette bodies, I'm pretty much on. It only concentrates on top balls cause.

A

That's really what a package manager, I think a note, should be primarily about at least at this development stage, and also it concentrates on optimizing startup time. um So the way you're organized you note monster actually actually also impacts the start time of your application. So if you have like a craziness, no module structure where you rely on like parent dependency somewhere over there, it actually can impact your sort of time so managing node modules. So, let's look at mp2.

A

If you run npm install NPM to you get a dependence graph looks something like this load modules exist. It's extremely nested arm, which causes problems on the windows with the maximum path length that you might exceed arm, but it's free, predictable right, so you have express which, except which depends on except switch hands on on types or something and so on, and so forth.

A

It's nested, but it's very predictable on empty and free by contrast, is flatus to dependency graph, which, which is like awesome like in a sense, but also caused some problems, because now, at that point, when you're installing Express, you might actually end up requiring in final handler serve static because there's no difference from the perspective very requiring it. Whether or not you know that it is actually installed like installed ethics as in like exclusively installed or or if it's like, just some other dependency that depends on it. That might then remove the dependency.

A

So in that sense, I think it it's problematic so, for instance, or static you can totally part serve static in here. It totally works, but it really shouldn't- and this tree flattening algorithm- that NPM users, also it's like pretty expensive, actually and also complicated and I, don't think we need so what id does instead is it creates siblings?

A

So the entire mechanism of installing dependencies is split into two parts for stretching them: I. Sorry, three parts resolving them into your sub dependencies, fetching actual troubles and linking them so buy exclusively. Creating some links to reference dependencies of dependencies on it pretty much eliminates all the complexity associated with this tree, flattening, which is at this point, no longer need it, because you're going to teach you to always have a completely flat dependency graph.

A

So the way this works is when you look into the node modules directory off to you, ID installed something you just see a symlink. So when I ID installed Express, for instance, I see a symlink from express to are a directory cast dot and then some random check some NN / package. So this is very different from mpm.

A

So, first of all, this has the advantage that under windows, you don't run into the problem of having very long partners, because they're all only like one level deep, really always, and it's also more predictable in the sense that you, if you delete random directories and you just run IE, install IE, will just repair it and it will just work on, and it's also slightly fast, because there's no fallback between node modules, so new place.

A

A nested dependency in NPM to the way like we sold circular dependencies is by falling back to the parent, which is no longer needed. That way. um So I mentioned this cache directory and the way this is organized is dependencies of being referenced by the contents as opposed to the name. So then, you look into noches directory after installing expressed in NPM this year. Directory called Express and what's in there is expressed I ed, does not do this.

A

Instead, you have a symlink which references a package which can be identified by checksum, so friends, Express might be in, is 518 f9, whatever directory, which itself contains the directory called package. That's where the actual extract. The trouble is located. There's no mochis directory. You then have symlinks to transit, stop dependencies so, for instance, Express, which, let's assume that's the first dependency over here, might have a dependency on debug. Debug, then is being referenced using a sibling as opposed to an actual directory, and that actually simplifies installation mechanism quite a bit.

A

Also, it might look unintuitive. If you just look at the rectory content itself, let's reuse modelsim links to express or being created in top-level directory, and this is hidden in like a controversial storage very similar to get in a sense which does pretty much the same thing. um So I mentioned this identification by Charles, um as opposed to their package name.

A

So when you install a package express four-point 14-point, oh um the name, the way ID actually references packages, flight chasm which in this case is c1 e3f whatever, and it's not being referenced by its name, which is Express arm, and there is there's really one main difference on how to speculative being resolved between NPM and ID, so NPM to at least always hits package root, URL, which contained all possible versions of a package or possible package.json versions of a package when resolving that specific version, ID are actually doesn't implement Cimorelli.

A

Instead, it it's the registry and let's register you solve it. So in this case it says. Okay, give me Express gives me the latest version of the flash latest, which is common J's registry spec compliant.

A

So another reason why it's fast, because it does everything at the same time really so essentially it does request to resolve Express to its latest version and then gets like a bunch of dependencies and then she's like does everything. And what might happen is that you end up installing expresses in fetching a tarball and linking it whatsoever, author actually fetching top of its of dependencies, so in that sense to order becomes unpredictable and slightly harder to manage.

A

But it comes with a really significant speed, improvement and so I mentioned how it's actually harder to kind of manage their control flow and how I'm using RX. For that, and so the way this works is really nice, because you can implement the steps of the insulation as custom operators in RX. So your insulation is really just like kind of like a pipeline observable of dependencies that just come in over time and then you like map over them.

A

You resolve them, you you download them, and then you somehow like link their dependencies together and the nice thing about. Rx really is that that becomes really easy to benchmark and tests, and those custom operates can be just plugged in like, for instance, ID can install from github can also install from troubles, as I mentioned earlier, and from just like any private registry and those custom. Installation mechanisms can be really just custom operators that you write on your own and and then you can just plug them in just work. So it's quite modular arm.

A

There is one part about registry about nkm as a system that I want to talk about, which is publishing packages. So one of the most requested features on ID was the ability to publish packages, and that has intentionally not been implemented, because it's it's quite easy to implement. It's just packing a table in and uploading it. So the reason this doesn't access really is because I think there is an overall, larger problem with the way the NPM registry works. I.

A

Think the NPM registry as a whole is just a completely different ecosystem, NP and publish tufted and just hosted, which is nice and make sense right. But on the other hand, almost all dependencies that are hosted on NPM are also hosted on github at least open source ones. So you yeah I, would say so. The overall majority of rock dependence is definitely also don't get up. I checked, I.

A

A

Exactly that's what I'm going to talk about? The main advantage of having this NPM registry is performance, so I want to introduce kind of like a different project. So the first idea behind this whole ID project was making NPM or the insulation mechanism of installing packages faster, but then I realized that actually, there might be maybe an idea of introducing a completely different ecosystem for installing packages all together.

A

So as I mentioned earlier, those like the registry for hosting packages and github living completely different worlds and what I think should happen as opposed to and forcing you to NPM publish packages is a registry should synchronize well, I saw from from github, for instance, your packages and more or less like ingest them. So this is pretty much what the second part is about really. So, when I briefly go over this I started.

A

Writing this project called Nova, which essentially acts as a mirror for github, and the whole idea behind this is to introduce kind of like a proxy for installing dependencies, as opposed to having a completely isolated and qiyam registry. That has nothing to do with github. Never dynamically builds tar balls just as the NPM registry and cassius those from some arbitrary git repository.

A

So let's really handy if you host your own like five packages or something and and that's really what I want to mention me- that's the first part and the second part is our.

A

So what was also mentioned is life management with NPM an NPM enterprise, so one of the future are features of NPM will be the ability to conditionally review dependencies and approve their licenses because after all, you want to have some visibilities of what you are actually installing on your machine and that's what id stormtrooper will be about so currently what what you're able to do at this point is approved dependencies conditionally and just check whether or not you agree with their licenses and review their code, but eventually it is going to be kind of like a replacement for locally reviewing dependencies manually.

A

It's feeding your soul process up, so it makes you two minutes faster than our plan, so maybe just to give a quick summary. You can get ID and if you two stores, of course, on on github and Noah as well and.

A

What's wrong and yeah and NPM is awesome, and and thank you.