GitHub GitHub Universe 2020: Developer - Open Source Communities, 11 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitHub ❤️ npm - GitHub Universe 2020

Description

Presented by Myles Borins, Staff Product Manager, GitHub

Earlier this year, GitHub acquired npm Inc., and since then we have been heads down on improving it and supporting the world’s largest development community, JavaScript. We’ve polished the services you’ve come to know and trust, released version 7 of the npm CLI, and we could not have done it without your involvement. Let’s recap what we’ve done this year and what the JavaScript community has to look forward to in 2021 and beyond.

For more from GitHub Universe 2020, visit https://githubuniverse.com

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

Hey everyone I'm so happy to be here and uh so glad to be joining you all. um Thank you to both of our hosts for doing such an amazing job and kicking things off, and you know, there's actually a massive crew of folks that you may or may not uh know about that are doing everything from checking our mics to looking at the lights on our camera to moving all the things around.

A

So I just want to quickly do a shout out and thank you to you know all the different people who are helping make this happen, but I think, with that in mind, it's probably time to hop right into things um and talk about why uh github hearts, np um and you know I was introduced by the host but I'll- introduce myself again um I'm miles and the staff product manager of open source here at github. um I also do a lot of things related to javascript, I quite like javascript.

A

It's pretty amazing, all the things you can do, even the things you probably shouldn't. You can still do it with that language and, if you use javascript, you're, probably aware of npm, um but if you're not that's cool, I have a little bit of an intro for you all who want to know. You know like what is even npm um and npm is a critical part of the javascript community, which is one of the largest developer ecosystems in the world.

A

Npm is the package manager for node.js, although npm doesn't stand for node package manager, which you know sometimes folks think it does.

A

Npm is a also a public collection of packages of open source code serving the javascript community, so we do have the npm cli, which is the tool that you'll use to install packages. Npm is also the registry of where all that code lives and is managed.

A

Now, npm inc was acquired um by github in 2020. It was originally founded in 2014 um and right now, there's over 1.4 million packages with over a hundred billion downloads a month, which is you know, quite a few bits that were pushing through those internet tubes um and the big thing that's pushing things through. The tube is the npm registry, and so I want to give you a quick update on an exciting new registry feature which is automation tokens.

A

um So if you are working with npm and you're publishing, whenever you do like an npm login on your system, you're getting a token and you're able to use that token to publish repos, um two-factor authentication is something that we introduced a couple years ago as a great way to help protect your account. The only problem is when you turn on two-factor authentication. It's now no longer possible to publish with continuous deployment, because it's asking for your two-factor authentication token.

A

So these new kinds of tokens, automation, tokens, um don't require two-factor authentication in order to publish, and so you can take those tokens generate them put them in like github actions or your favorite continuous integration and deployment platform, and you can now still publish plat packages using continuous integration and deployment.

A

Another point of npm, that's worth talking about that I'm excited about is our documentation and we've done a pretty major overhaul to docsteadmpmjs.com. You can see on the right what the new doc site looks like for the npm install command. This new doc site has cli documentation for both npm v6 and npm v7.

A

It has um it's built using gatsby, which is primers doctor using uh primer stocked cap theme.

A

um It is built with github actions deployed via github pages, I'm a big fan of dog fooding and using our own technology as much as possible and we're working on getting it into a state where it can be open sourced, and we can accept public pull requests in the near future, and you heard me mention npm 7 in the npm cli and I'm pretty excited to talk about the fact that npm 7 is already shipping today with no js 15..

A

So if you install node via whatever way you like to install it and you get version, 15 you'll be able to and get npm 7 on your system today. If you want to install it manually on an lts version or an even older version of node, you can do it with npm install g npm at 7.

A

and there's a lot of big new, exciting features in npm 7, including workspaces, which is something folks have been asking for for a while, pure dependencies being installed by default. um Support for yarn, lock files, which is a great way to minimize churn in projects. We have a new package, lock format called package log v2.

A

We also allow you to install from a github repo, which is pretty cool, but let's take a second and look at workspaces. So if you want to make a workspace, this is kind of a really high level of what it would look like. You would in a folder, make a package json and in that package json you would define workspaces and you define which workspaces exist and workspaces are individual projects that are modules. So you can see here we have module one and module two.

A

You run ncam install at the top level of the package and that will install all the dependencies for both module one and module. Two but module one will reference module two and module two? Will reference module one when they are used on the local file system for development?

A

This makes it way easier when you have lots of interconnecting packages that are all referencing each other, so you can work in larger, mono, repos or potentially take a number of not of different git repos that aren't connected and put them into workspaces so that you can manage multiple projects. At the same time, peer dependencies is one of the larger changes that we've done as they're now installed by default, they weren't prior to npm seven.

A

This can be a breaking change for your projects. It is possible that something that you installed ran npm install with node 6, npm 6 would have worked, but with npm 7 is going to break, but you can try using dash dash force if you run into an error and a good chunk of those errors should be able to be resolved using the dash dash force command, and you can also try using dash dash legacy pure depths. If you want to make sure that you use the old algorithm, we still maintained it in there.

A

This is like a big design philosophy when it comes to breaking changes. We understand that we need to make breaking changes once in a while and that those breaking changes can have repercussions across the entire ecosystem. But what we try to do is leave back doors. Well, maybe backdoor is not the right way to put it, but an escape hatch.

A

That's a better way to refer to it that we leave escape hatches there to make sure that if it's not working for you, that you still have a way to make sure that your repo that your workflow can work while you kind of get up to date and fix the things that are broken, you can also do this as a environment variable, so you could set npm legacy pure depth.

A

True, you can either do that in like your bash profile, so your system will always use legacy pure depth, or you could run that as a single command in front of npm install. This is something that can allow you to script this.

A

If you or like put it in the environment, if you're doing stuff in ci just different ways of making sure that it's there and you know if you're, maybe a little bit more intense, you can use dash dash strip depths which will actually fail if there's any pure depth issues even in transitive dependencies, because we only really cause issues if you're trained. If your top level dependencies have peer depth issues, but not transitive dependencies as your dependencies of dependencies anyways.

A

This may all sound super confusing because it kind of is, but just try to remember, dash dash force and dash dash legacy pure depth as kind of back door as not backdoors, as escape hatches in case you're running into any problems.

A

So um we talked about npm installing from a github repo, which was really cool, um which you could do for a while. But now you can also install from a github pull request. So if you do install user, slash, repo hash, pull 123 or whatever the pull request number is that allows you to install pull requests directly into the package that you're working on you can track these in your package.

A

Json, if you want, although I don't advise doing that, because you know these pull requests, aren't necessarily guaranteed to stick around, they can move under you. It's always good to have your dependencies tracked against, like you know, a particular sember or shaw sum, so that you know that you're getting the same thing, but this is a great way if you want to quickly test something that has a pull request or if you need to quickly patch something in a project. It's a really great usability um feature now.

A

This won't work for all packages, um some packages that, for example, build and then deploy from a subdirectory- may not work with this, um but it does cover a good chunk of the use cases and we're looking into some of those edge cases where this workflow doesn't work.

A

Now. Some of the breaking changes um just to let you know is obviously, as I mentioned, pure dependencies installing by default can be a breaking change for your project.

A

Npm now uses package exports, which is a feature in node. So by defining those exports it means that you can't deeply traverse into the npm tree anymore. You can only use npm as an interface as defined by us, but to be honest, many of the things that people were deeply using npm's non-public interface for are now standalone modules, such as cache or arborist, feel free to ask on discussions if you've been using npm as like an api- and you know I can tell you about some of the alternative ways that you can do.

A

This um npm x npx has been rewritten using the npm exec command, and it also will prompt you before running things for the first time. This is for security reasons, and we realized that you know this makes some folks grumpy and it complicates some workflows, but we genuinely believe that that for the ecosystem, this was the right thing to do.

A

Npm ls only outputs, the top level dependencies now and similarly, the output to npm fund and audit has been changed. So if you were building tools or workflows that depended on the order or how that output looked like, um you may be broken by the changes that we've done in mpm seven, um you may be asking what's next for the npm cli and there's a couple things like when we're writing lots of tests. That's a huge thing!

A

That's on our radar right now, uh we're also going through and updating all the documentation, um we're going through improving stability, making sure it works. We've got bug reports coming in and we're working on them as quickly as we can and improve performance, we're working on a benchmark suite and we're trying to make sure that npm 7 can be the fastest experience for installing packages.

A

Once all of these things, you know are have all lined up we'll be making it ga so that when you end pm, install npm, you get npm 7 by default, but for right now, you'll still get npm 6. When you do an npm install, because we want to get kind of like all of these things um to a certain level before we make it.

A

The default installation- um and the last thing that we're going to be uh working on here is more workspace, features that so that's kind of like we've been talking about all the things that we'll be doing to make it better. But workspace features is a huge one, we're going to be expanding the capabilities of workspaces and making it more of a fully future product. The last feature that we're definitely looking into the super cool is called package overrides, which will allow you to say hey this package of my entire tree, replace it everywhere.

A

So, let's say you're using react and you want to like patch react and make sure that all the dependencies through your whole tree get the same version of react. Something like package overrides will allow you to do that and finally, we're listening to our community and doing what our community asks for, which leads us to our public roadmap and feedback process, something that I personally am extremely excited about, and was one of the first things that I started working on uh when I joined the team in june.

A

The our process was heavily inspired by github's process for their roadmap and their public feedback, and in particular I want to shout out to alex nichols and becca for all the work they did supporting us all of the random messages that they answered of mine and just kind of always being available and awesome, and in the github process that inspired how they did the roadmap and feedback.

A

They talk about four types of feedback that you can get incidents, questions, problems and suggestions. Incidents are like when things are down uh vulnerabilities questions are when people don't really know how to use things or just generally need help.

A

Problems are when there's a bug, and then this last category suggestions is one that npm didn't have a very clear, interactive way of people coming to us with suggestions of improvements and changes that we can make to npm and that's where comes our public roadmap and feedback system which I'll go ahead and show to you right now? So what you can see on my screen at this moment is the npm public roadmap repo, which you can find at github.com npm, slash roadmap, and this roadmap is.

A

We have a guide to the roadmap where we talk about the phases and the feature areas, the different um stages that things go into the future areas and you'll see that this looks quite a bit like the roadmap over at github. But when you go over and look at our roadmap itself, you'll see everything on here is about npm.

A

You could see what we worked on last quarter, what we've been working on this quarter, what we've launched what's still coming out the door and things that we have on deck to start working on next quarter as well. As you know, ideas and things that we've said hey. We want to work on these things, but haven't yet had time to do it. So, for example, I talked about automation tokens. Well, we have something on here for registry tokens for being able to name them.

A

Another thing for package: scoped, automation, tokens so right now, automation, tokens are scoped to the user, but some folks would like them scoped to the individual packages. But this is example of ideas that have come in that we've um brought in and how do those ideas come in? They come in through our public feedback discussions.

A

So this public feedback discussion here um we have a whole bit about how you make discussions and how you give good feedback and kind of like something that tracks the life cycle of a suggested discussion to an actual feature that the npm team makes. So you can see here what our discussions board looks like there's lots of different things in here. We've got a mix of things that have been answered and not answered. We've got different tags ranging from the cli the registry, the website live, streams and meta.

A

You can see from our live streams here that we actually go through and on a bi-weekly basis, review um topics that have come in from discussions.

A

You could see all sorts of different topics: here's our agenda that we went through and you can actually find the youtube video because we record it and make it available, and I wanted to just quickly as we go show what the life cycle of one of these um looks like so um github flavored markdown for readme, rendering was a feature that we had heard requested multiple times through our kind of private uh support system prior to the acquisition there. There was a lot even coming in, and so we started tracking this as a discussion.

A

We kicked off a discussion. You can see some other people who came in had lots of ideas had lots of suggestions about how we do it, and so we said we are going to prioritize this we're going to work on it, and so we made an issue on the roadmap that pointed back to the feedback discussion where it first came from. um We did high-level scoping of it and from there it turned into an actual work item that the team worked on and shipped this quarter.

A

And so, if you look at readme's here, this is a readme that was published using our old readme render renderer marky markdown, which has things that are great about it and and things that don't work as well. So there's certain things, for example like resilient syntax, highlighting or embedded headers- or this is one of my favorites- is mixed media encoding. So you could see that this awesome gif here that's super trippy um was actually encoded pointing to http and so we're getting these kind of mixed content warnings.

A

So there's all these kind of, like small, subtle differences, especially subtle between the differences of an npm, the readme, as it shows up on npm versus the readme, how it shows up on github and since a large portion of the packages that are on npm, actually live on. Github we've gone through and updated so that the readmes are now being rendered using a github service that actually renders the readme for github.

A

So you can see now that the syntax highlighting is working even when there's bugs that the embedded headers work, the way that we're expecting it and if we go into the console here, we're not getting any mixed content warnings and in fact, what's super cool is. If you look here, you can see the image has actually been put onto github's image, server service called camo and we're now. Well, it still has the canonical url there, so we're not losing that content.

A

We're actually hitting github servers to get that image, which means we're not um potentially going to go down if an external image is not available. um It also means that we're not getting those mixed content warnings and everything that we're serving is https, which is pretty cool. So you know this is a great example of a feature that came from feedback that went to a discussion that discussion turned into an issue.

A

That issue then became something that we prioritized internally and shipped, and if you look at the roadmap of the things that we've got going on for the next quarter, or this quarter there's some stuff that I'm excited about that hasn't shipped quite yet, one of which is that we're going to show typescript information for packages that have type declarations and the other one is show download for download usage perversion um and again, you know these are both um examples that came in from discussions from the community that we then turned into work items that were tracking.

A

We have things on the plate for next quarter that you might be excited about, including, but not limited, to open sourcing, docs.npmjs general availability of the npm 7cli, exploring authentication with github across all of our different products. um Workspace changes to workspaces updates to the website, um reviewing our policy for disputes and abandoned packages, which was uh awesome feedback that we receive from folks and also auditing some of the packages that share names to node.js again, something that came in through our general discussions.

A

um So, overall, as you can see, we have a lot of different ideas that are coming in a lot of different things that we're working on. We're really excited to do that work with you, the community and help kind of align where we're going um and and we're trying to experiment with you know new novel ways of being transparent.

A

It's hard to find that balance, because you know we are a company that works on open source, but we are still an organization um and we're trying to find that balance where we can kind of work with tools that the open source community are used to obviously like github, but also um work using patterns that folks in our ecosystems are used to such as live broadcast meetings, open discussions and transparent process, and so that's it for me today on what's going on with npm. um Thank you so much everyone for having me here today.

A

It was an absolute blast. I will be in the discussions for the next little bit to answer any questions that you have and then, even after that, at any time you can always come to github.com npm feedback to let the team know stuff. If you have any, you know stuff you'd like to see us do so. With that I'll hand, it back to our hosts, and thank you again for having me today.