Node.js Node.js Interactive North America 2016, 15 Dec 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: KEYNOTE: State of the Union: npm by Ashley Williams, npm

Description

KEYNOTE: State of the Union: npm - Ashley Williams, npm

A

Hey everyone how's. It going got a good conference alright. So, uh as Michael said, my name is Ashley Williams. You may know me as a g-dubs from the internet. If you read my tweets, all I can say is I'm sorry, but I work at this little company called NPM that you may have heard of have. Has anyone in here heard of this? This thing just a couple Wow sounds like no one has, but it does turn out that we are a company and a lot of people are confused about that.

A

So I won't linger too long. But if you do have money you can give it to us if you're interested well, we'll use it to keep the registry running you know, but at NPM we have 23 humans working there and, unfortunately, no wombats. Yet our office manager says no pets but we're working on it um and we we call ourselves the wombat developers Union if you've ever wondered what that wdu sticker is all about. It's just the NPM logo upside-down.

A

So a lot of people know that I work at NPM, but a lot of people have pretty much no idea what I do there and the reason is because actually my job is kind of vague and I. Don't really even have a job title, so what I? Usually try and say? Is you know if you don't understand how NPM works?

A

That's my problem and it turns out that I have a lot of problems um so today, instead of actually explaining to you how NPM works, we're gonna do something a little bit different now I have a weird hobby and it's paint-by-numbers I'm super into it, and because the title of this talk and paint-by-numbers both use, the word number I'm gonna be doing a little painting of a story with numbers for you today.

A

So if you aren't interested in me, excitedly reading very large numbers, there's gonna be some really pretty paint-by-number pictures in the background for you, alright. So the first thing I want to talk about is the number of packages and downloads that we see from the registry. So if you look at the last 28 days, NPM users have installed 18 billion packages. Now, that's just the last 28 days and to install those 18 billion packages they needed to do six billion downloads.

A

Now you might be asking wait hold on if you have to install 18 billion packages, why are there only six billion downloads? And that's because now, around 66% of all of the installs that are happening in the ecosystem are coming from the cache, which is one thing: that's making it significantly more efficient for us to run and it's making it faster for you now, here's the graph and in the business world this graph represents what they call a hockey stick and I.

A

Don't know why they call it that I guess business people just like sports metaphors, but it's clearly an exponential curve, and this is really really amazing. Yeah I, don't know, maybe they listen to math I like math, so that's how many installs and downloads we've seen. But now, let's talk about packages that have been published and some of these numbers are super cool.

A

So if we take a look at publishes per week, you can see we went from around twelve thousand and five hundred at the beginning of I guess, 2015 and last week there were 39 thousand six hundred and ninety two publishes that week. That's basically forty thousand publishes happened just last week, which is pretty amazing all right now. What about new packages? How fast are these new packages happening? Now we have some amazing spikes there and what I'll tell you right now is.

A

We do have quite a bit of data and we actually cannot account for what happened there. Maybe a bunch of developers just got real excited, but in the last week, just last week almost 5,000 brand-new packages were published, which is a huge huge number now. New packages are interesting, but I'm kind of less interested in packages and I'm more interested in people, and the thing that I think is fantastic is around. Now we are averaging a hundred and sixty people publishing their very first anything every week.

A

That's a hundred and sixty people, so probably by next year, we're gonna be seeing two hundred people either publishing their very first package or publishing like a new version of something for the very first time, and this is amazing because it's showing epic growth, particularly of beginners in the node ecosystem, which is something that I feel very very strongly about.

A

Now, these new developers, these new publishers, are joining the ranks, a hundred and two thousand four hundred and sixty active, unique publishers that are already working in the system, which is also amazing. Now these people are just the publishers, but if you take a look at our rolling seven-day registrations, which is just people who will register on npm, we are currently looking at three hundred and fourteen thousand five hundred and eighty two registered users and that's a huge number.

A

Considering that pretty much, the only thing you get when you register is the ability to publish that's it and people, this number of people still publish, which means that we actually, if you take a look at this one right, we have about two thirds of those people that are getting ready to publish their first package, which is extremely extremely exciting. Now we were just talking about publishers, so I thought it might take a little moment and talk about unpublish --is, since it kind of came up in that opening keynote.

A

So there's been a bit of confusion about this and when I talk about it publishes you might think about this left pad or otherwise known as the eleven lines of code that broke the internet. Now we can argue forever at the bar or whatever later about whether or not it really broke the internet and how long the internet was broken for, but what I can tell you is left pet happened because of naive policies at MPM.

A

Now we completely understood this and we were extremely sorry and by the next week we had published a brand-new, unpublished policy, and so this is the blog post that went out that I wrote and the TLDR is that, basically now you can only unpublish your package within 24 hours of publishing it. Now you might ask why 24 hours it turns out that people get a little trigger-happy with that published sometimes, and so you want to sleep on it and then you know, wake up and be like.

A

Oh no I just published all my API keys, it happens, and even though yes, once they're leaked, it's kind of done forever, people still like taking it down, which is reasonable. Now what was also awesome about this is when we came up with our new, unpublished policy. We didn't want to stop there, because one of the things that left pad made us realize is that we are stewards of an extremely elaborate and complex community ecosystem and we need to be listening to that ecosystem and what they need. So we can serve them appropriately.

A

So, for two weeks after that published policy, blog post came out, we hosted a forum on github where we solicited stuff from the community. Now for what it's worth, it did have to be heavily moderated, but we got a lot of amazing things from it and you can see here.

A

We had 115 comments, which was super cool, and you know not that many down votes, which was awesome yay but what's great about opening this up to the community, is that it actually brought to light something that we hadn't ever thought about, which is that most people who are trying to unpublish packages weren't just unpublishing packages, because maybe they had published their API keys or they were desperately angry at someone who might work at NPM know what they were doing was they didn't want that package to be listed on their user page anymore, pretty much?

A

They just didn't want to be associated with it, and so from this community forum. We came up with this new process of what we now call dissociate and deprecated, which means that if you don't want your package on the registry anymore, the answer is kind of like too bad. It's gonna stay there, but you are able to then reassign that package to the NPM user.

A

So you can get your name off of that and then you can then post a deprecation thing saying that the package is no longer maintained now, yes, having packages that are deprecated and unmaintained, and the registry isn't great, but because that package is still there, it means that all of your NPM installs won't not work. They will continue to work and that is what's extremely important for the ecosystem. So the TLDR of this is that the registry is immutable. Alright, it's not gonna change, there's two caveats: DMCA takedowns and like really really epic security violations.

A

But beyond that and look we're hold for held by the US law to those things the registry is otherwise immutable, which means, if you are afraid about another left, pad happening. It's not going to happen now. You might be like Ashley well, it could happen if the entire registry went down right. If, if all of MPM went down, then potentially we would have probably the most epic left bad situation in the world, and they, while everything like that is possible. I also want to mention that the registry is currently extremely extremely reliable.

A

Now you may or may not be aware that there's a website called ping MP MJS comm. You can go there right now and see real-time stats for the availability of all of the sub public services. That NPM runs I've put just a couple up here, including the registry, and so you can see. We have basically that 99.999999 nine uptime, so things have been really excellent. That's for the registry, the registry tarballs and replicate MP MJS, which we'll talk about in just a bit later now also just being immutable reliable.

A

Another super awesome thing about the registry is that it's really really fast. Now you may be frustrated by the speeds of your NPM installs. But what I'm here to tell you is that since the registries inception, as being this little couch app, it has been separated into a kind of semi elaborate, but very, very efficient, set of micro services that have really improved performance. Mpm pad has been followed.

A

Using this thing called a follower pattern and what we've been able to do is shave off a lot of our architecture so that we can serve the vast majority of the data that we need to serve statically and only update that data. When we get something from what we call it changes feed now, I'm sitting. The registry is fast and ok, maybe that's a bit hand wavy, but I wrote a benchmark to check this out, and so what this benchmark does?

A

Is it tests installing an application that was about using angular, 2 generator and Express generator? One uses just the depths from NPM listed in the package.json, like usual, with the DEP name and the December range, and then what I did is I went through and I replaced. All of those ember ranges was the get URLs and the commits the commit shows exactly those reasons. um Those versions that you needed and then I ran them a bunch of times an average date.

A

I ran them about 30 times each, and so what you can see from these numbers is that on an empty cache downloading your packages from the registry as opposed to github, is 75% faster, which is significantly faster, and it's because the registry is specifically designed for serving packages to you, unlike github, which is an excellent service, but not designed for package distribution. It's designed for version control. Now. What did the side things here? I'll say is that if you've ever wondered, why should I ever get private NPM packages when I could just use private repos?

A

This is why, if you care about speed, but at last I checked on github y'all seem to care about speed quite a bit umm alright. So all of these things that being immutable at being fast it being reliable, is amazing, because the registry is huge.

A

This is a web, a little bit of data that lots of people like to look at, but again, we've got that curve. If you check it out, em PM's registry has more than double the next package registry, which is maven. Alright, we're up to I, think I think it's like maybe three hundred and sixty nine thousand packages, if I checked this morning earlier today, that's a lot of packages.

A

All right NPM is the largest package registry in the world, which means that we're dealing on a level of scale that we simply haven't encountered before and it's kind of amazing and and there's a lot of things that we can learn about our ecosystem from looking at this data in NPM now, what's fascinating is we're the largest package registry in the world, and yet we can still grow, and so here it comes a baby, a bit more hand, wavy numbers, but I think that there are pretty fascinating.

A

So in folk you did a study in 2014, where they, they ended up reasoning, that there were around 18 million developers in the world and then, if you plug that, in with the stack overflow developers survey from 2016, it would suggest that around 85% of those developers write some sort of java script, which means that if you do the math there's around 15 million JavaScript developers in the world now there's around 5.7 NPM users and a million NPM users, and this number is actually very difficult to come by Lori.

A

Our CEO now has written a fascinating blog post about how you can actually get to this number. But if you take a look at these numbers compared with how many JavaScript developers there are in the world we're about 50% all JavaScript developers using NPM. Now that means, if you have two JavaScript developers at your company, we can pretty much guarantee that one of them is going to be using NPM. Now that's a lot of developers, but we can do better. What about the other 50%?

A

And so we took a look at some of our customers and say well. What are people doing with NPM right and so fascinatingly enough? All right? It turns out that 80% of the users of NPM are using packages that have dependencies that only are written for doing front-end development all right, and then we took a look and then around 60% of the people using NPM are using packages that are for server-side development. So that's about 60.

A

Now, if you look at those numbers- and you do that math, what this means is that there are 20% of people using NPM that are only using it for front-end code. Now, if you know anything about the history of NPM, NPM was written for people who were writing modules in node. It wasn't for people writing applications and it certainly wasn't for people writing client-side JavaScript, and this is kind of interesting because in fact, NPM is being used for a lot of things that we absolutely could never have anticipated.

A

That is going to be used for, and this is really important, because I think a lot of the times. People tend to be frustrated with NPM, because part of the reason is people who are using NPM for things that it was just not designed for, and it turns out that if you use something for what it's not designed for, sometimes this doesn't work great now, I, don't see this as a bad thing.

A

I actually see this as an opportunity, because, because of this amazing problem and this amount of scale and the wealth of value that you find in the registry, the community has been deciding to write their own tools. So greenkeeper IO is an amazing product. How many people here know greenkeeper all right? If you're not using it, you should it's awesome, but this is going to help you keep your dependencies updated in your applications, NPM s if you've ever been mad at search on NPM.

A

These people were frustrated by it and they're like I'm gonna make a better one. This is an awesome search, they use metrics and all this, it's fantastic yarn is also something that recently came out. An NPM loves it. It's really taking care of a part of our population that we weren't being able to like successfully serve alright, it's for people who are writing front-end applications, and it does a really neat job with that now a lot of people were like. Oh yarn, isn't that, like an existential threat to NPM, but we weren't upset at all.

A

In fact, there's been many clients written for the registry and they're really awesome. In fact, there's one that's been around for a really long time and is extremely popular, and it's called C NPM, which is the NPM in China, and it's a fantastic client as well.

A

So we really celebrate people building new things on top of NPM, and so I want to leave you with the message that NPM wants you to build cool stuff on top of it, and so I'm gonna give you a couple of resources for ways that you can start playing around with the awesome things in the registry and building your own tools to also help you be more effective in your jobs.

A

So one of my favorite things- and this is a pattern that I talked about earlier in regards to the registry architecture- is that you can write a registry follower which means that as changes happen in the registry they have in and they emit events, and you can follow those and do things based on them, and so this is all the code. You read to write a basic follower, it's 22 lines of node, which is super awesome. What are the very first things?

A

I did when I started writing followers is I wrote something that found out all the curse words that people use in their read Me's, which ended up being extremely fascinating. You can find that on my github, it's not safe for the presentation, uh-huh so, and also something that we're currently working on. This is this is my job at NPM. Right now currently is to document the public API partially, because we're about to start working on an API version, 3, which is going to be even faster and we're particularly excited sidenote.

A

We've started writing a couple things in rust, so it's getting really fast, which is awesome so again, if you're looking for Docs or you want to ask questions about how the registry works. This is constantly the issues here are constantly monitored by their registry team at NPM, and so this is an awesome place to get those resources and ask the questions you want alright. So the final thing is this feed, which I've been talking about a lot that changes feed. This is brand new.

A

We had another one, and this is a nice new, faster version, so replicates at MP, MJS comm. If you go to this right now, you can just watch in real time all of the changes that are happening to merge the registry. But when you write a follower, this is what you'll be following, and this is a really cool awesome resource. So I will leave you with this. We are so excited to see what you build.

A

If you build something awesome out of the registry data, please tell us we'll blow it up for you, because we really. We really want to encourage people doing that and finally update your NPM, please the version that you get when you download node is never the most up-to-date one. So just as a small matter of maintenance, please update your NPM and yeah. So that's that's going to stay to the unit of NPM right now. Thank you very much.