GitHub Demo Days, 19 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: npm workspaces and overrides #DemoDays

Description

Darcy Clarke, Staff Software engineering manager walks us through two significant additions to the npm-cli, workspaces and overrides along with a tour of npm's progress though last year.

0:00 - Start
2:26 - npm helps you manage packages at the pace of Javascript development
3:35 - key features of v7.0.0, progress further into the release line, dropping Node 10, and the latest changes through v8.3.0
7:32 - npm workspaces: https://docs.npmjs.com/cli/v8/using-npm/workspaces
9:51 - workspaces demo from scratch
16:32 - npm pkg
18:20 - scripts
20:45 - filtered installs
25:20 - recap of Workspace Aware Configuration
26:25 - npm overrides: https://docs.npmjs.com/cli/v8/configuring-npm/package-json#overrides
29:12 - overrides demo
32:53 - 2022 roadmap

https://docs.npmjs.com/cli/v8

A

Welcome everyone welcome. Thank you so much for joining us today in this last edition of demolays for 2021. What how is that even possible? My name is andrea griffiths. I am alicolombian twitter and it is my absolute pleasure to be here today to chat with an amazing staff engineering manager here at github, I'm going to bring him up on screen hello, darcy clark. How are you.

B

A

Wonderful, no I'm just happy to be here with you today. I love the way we're gonna close out the gear with your demo for those of you who are not familiar with darcy and his amazing work. uh You can find him on twitter at darcy clark, I'm sorry yeah and um I will be sharing some other links throughout the conversation uh so that you can reach out to darcy and catch up. But darcy came to us uh via mpm acquisition, which happened early 2020, and it's absolutely mind-blowing to think that it's been that long.

A

But a lot has happened since uh a lot has happened in the npm space and also in the github cli space. So today, darcy is going to take us more inside we're going to get to see behind the curtain. Hopefully we're gonna be talking about workspace uh overrides and a lot more, so super thrilled to have you here, darcy without further ado. Let's just get started I'll get out of the shot and bring up your presentation enjoy everyone.

B

There we go. uh Thank you so much um andrea. That was amazing, intro and uh thank you so much if you're joining us live. Are you watching this as a recording? um It's the last demo days of the year, so I feel very special and privileged that you're joining us here today.

B

uh I'm gonna be diving into workspaces and overrides, and these are two big topics and and two huge features for the npm cli team uh that we've been working on for for almost two years um and have been speccing out and working with the community to try to make sure that we're delivering on that uh on these two goals.

B

We're also going to talk a little bit more on what we're doing in the future in the mpmc live space, and uh this is a lot of content, so I'm gonna try uh get through all of it have a couple demos along the way, um and so uh wish me luck in terms of demo gods and and live demos today. Let's hope everything stays up so um quickly. I want to dive into what we've been up to uh npm, uh as andrew said um uh was acquired in uh 2020 by github and npm.

B

You know is trying to help you develop, run, uh secure, distribute and discover packages and the mpmcli as a package manager. You know that's our number one goal is to help you manage your packages and do so in a way. uh You know that keeps up with the pace of javascript, um because if you didn't know, um the javascript eq system is huge and continues to grow. uh There's uh 1.8 million packages. Now on the npm registry and monthly, we see over 150 billion with a b downloads and installations from that registry.

B

So uh this is exponential growth. um A year ago, or just I would say a little over a year ago, we hit that 100 billion uh mark and we've already hit 150..

B

uh I'm I'm really excited about where we're going, and uh so our team is trying to keep pace with this rapid growth and obviously we're trying to ship great features to go along with it uh quickly uh to talk to sort of the ships and how we got to where we are and some of the features that have landed along the way.

B

I want to go back in time here and go to when we launched the initial v7 version 7, which was the first major release since we were acquired and had been in development uh in back going back to 2018 2019, even in terms of the roadmap there.

B

Some key features in npm v7 were installing peer dependencies again uh introducing some new commands that we thought would help with quality of life like npm, explain and pm exec, which is sort of a map to mpx, and then some improvements to npm audit, supporting yarn, lock files, which was a huge uh win for us, and we think for the community as well, and also we tried to really uh you know, clean up uh our house and do some work in terms of performance.

B

But of course, the thing that we're talking about today, which is workspace, is shift and workspace to support support shift in npm v7, and I I put the little term light there, because, when this first released we didn't have many features to really take advantage of that workspace support. So npm the npm cli was made aware of workspaces and we introduced a new key called workspaces inside of package.json that we supported. But really there wasn't a lot of tooling that sort of built. On top of that.

B

So going further in the actual v7 release line, we began shipping a whole bunch of new commands, including adding workspace awareness to a number of the uh legacy commands that we had in place. So two of the interesting new commands that we shipped were npm package and our pkg and mpm diff, but along the biggest, I think, improvements that we made here um in the last year or so we're really shipping workspace and workspaces support.

B

These are the two flags that allow for the traditional npm sub-commands to essentially operate against the workspaces you have defined in your package json. So that's that's sort of how we got here. uh V8 we actually, if you didn't know, we actually did ship another major release of the npm cli just recently back in september, and what this did was essentially the the crux of this was to drop uh support for legacy, unmaintained versions of node uh and that included node 10.

B

And so, if you're wondering what the uh you know, requirements are for using the latest, npm cli um and staying you know evergreen.

B

um Currently, those are the the engines uh that we have defined and that we support in terms of the matrix of node versions, that you'll need on your system to use the mpm cli uh and last but not least because our team is working so hard in the last month or so we've actually shipped three minor versions uh introducing some net new features, one of which we're going to dive into a little bit more here, um but uh adding flags for workspaces um to uh interact with the root and we'll talk a little bit more about that in a second and sort of how we delineate between your roots of your project and sort of the workspaces itself, um also doing some refactoring work, but uh the big ship that we had actually last week.

B

So this is the first time we're really talking about. This is overrides. um This has been developed in, like I said, for for almost a year, or so we had a spec that was written over a year ago now and we're really happy that we finally were able to land it.

B

uh So if you want to get the latest version of the npm cli, it's always it's been. The same way for a number of years now you can run mpmi npm-g if you've already got node installed or you can go grab one of the node distributions and npm the mpmcli comes pre-baked into every node version.

B

So, let's dive into workspaces um the benefits of really adopting workspaces and sort of a monorepo sort of architecture is there's many of them, but the the five here listed are sort of what I consider to be the the biggest wins for for moving to and starting to adopt workspaces so being able to link, build and test your processes and sort of standardizing the tooling across uh those those projects is, is the huge is a huge win um and as a maintenance, uh I would say the other major uh you know win for for me, and our team is as a maintainer of multiple packages.

B

The maintenance burden, obvious increases with every new package that you distribute every new project. You distribute uh based on how many repos that you now have to monitor and how many communication channels you know, have increased in terms of the issues and prs that will land against those projects, so bringing those back into a single project can sometimes make a lot of sense if they are relative or they have shared dependencies. That you'd like to manage and there's a number of huge wins.

B

I think uh for for a lot of folks that migrate to workspaces or end up using some workspace tooling, uh and these are just the the top five that I could think of uh preparing for today.

B

So I talked a little bit before about the idea that there was some relationships uh that we uncovered and some definitions that we want to sort of delineate within our own workspace implementation, and so I have this graph and this chart that I usually share when people ask me about you know what is the difference between the root folder of my project? What is a workspace dependency? What's the difference between that and a bundle dependency, and this little graph really shows that? Well, um it's pretty straightforward.

B

We take an approach. That's more similar, I would say to to you, know, yarn and and learn as sort of implementation here, but having this definition and using this terminology I think, really helps um and of course, on the left hand side there, you can see how simple it is to define workspaces within package.json um and yeah.

B

It's pretty pretty easy to get started so, okay, so cross our fingers, I'm going to try to really showcase some of the amazing uh sort of tooling and sort of functionality of using workspaces starting from scratch, and I've got some commands here, we'll see if I can run them and, let's all cross our fingers that the demo demo gods will be nice to us. So, switching over to my terminal here, I've got a demo folder open. I've also got the docs website here.

B

So, if you're interested more on how to use workspaces, you can go to docs.mpmjs.com and you can find the workspaces section under using npm, and we have a number of examples and also some information on how to define workspaces definitions also support globbing patterns. That's not something I'm going to use today in my demo, but it's something that many people do use just for ease of use in terms of defining.

B

You know the nested projects that live within your workspace, so so, let's jump into our demo, um so I've got two uh projects that I'm going to or two folders here that I'm going to dive into and try to showcase the the power of uh workspaces and I'm gonna quickly just open this up in my editor on my second screen: awesome: okay!

B

So, let's dive in so when you're starting a new project um and I'll go into the workspaces folder here we're starting a new project. Npm provides to you and I'll increase. My screen here just for folks in the recording and pre npm, provides you with a command sub command called uh npm init, and so, when I'm starting a new project, I can run npm init and it can start me through a sort of interactive flow to answer a whole bunch of questions.

B

But usually a lot of people do this, and I do this myself- is you just take whatever the default is?

B

And so here I'm going to pass the y flag to essentially just say, give me all the defaults and create the package.json file, and here I am I've- got now a package.json file, I'm starting to work, I'm starting to build my project. Let's get started so what's really interesting about the workspace uh flags uh and uh is that with npm in it, we've now been able to provide you with a mechanism to quickly scaffold um a number of workspaces that will essentially add to this root project levels, uh package.json file.

B

So let's show you how that works. So let's say I have uh this project that I just started and I want to create both a application, workspace and a server workspace. So the two ends here I want to do some server-side development. I want to have dependencies that are you know only specific to that part, um and I have an application.

B

That's going to have a whole bunch of different dependencies over there, so I can do npm, init dash, w or dash y again, so take all those defaults, and then I can provide the either the full named workspace field or I can provide dash w, which is an alias to workspace and provide the name app, and I can also provide the name server and you can actually add as many workspace definitions here or provide as many workspace args as you'd like to quickly scaffold a number of workspaces.

B

So there you go. I ran that and what's actually happened here is its output npm in it has output that is, run and created two different package, jsons and net new folders within my project. So it's created an app folder and a server folder, and if I look at the package.json at the root level, it's actually created definitions here uh for app and server which direct to or which map to those directories.

B

So great I've already quickly scaffolded um uh my project. Now I'd like to do something right, um so one of the first things people often do is they want to test their software? So we can quickly do that here in the root project and, of course, the default.

B

You know, output for npm in it is to tell you that you have no test specified, which is it errors out, and it tells you hey. Maybe you should uh start to write some tests. Maybe you should, you know, implement some test coverage, so I can actually also run npm test and and all of the npm sub commands against workspaces quite easily.

B

I can use the workspaces flag, which is different from workspace, in which I'm actually defining an explicit workspace to run a command against, but workspaces will run npm test against all of the workspaces defined. So again we have a shorthand alias for that and that's called dash ws. So if I run that you can see that okay now I've got two errors that are outputting, because I'm uh both running uh this command in the app workspace, as well as the server workspace, um so that this is really powerful.

B

This is really interesting because all of the npm commands now support workspaces and the workspace configs. So, let's, let's keep building on this. Let's actually start to see what else we could do so one of the cool commands we introduced in npm 7 was actually npm exec, and this is essentially the exact same tooling and sort of uh user flow, as you would have with mpx, which is a bin that we distributed separately for npm, um but which comes with every node distribution now as well.

B

So if I run npm exec, I can run any arbitrarily uh sort of shell script that I would like against my workspaces, and you can imagine this is like a very powerful very like uh you know, power user feature to be able to do this, and so, if I run, let's say mpm exec c, which says it means command, then I'm going to pass. Let's say the current working directory. I can see when I run it in the root. It's just going to give me the path to my my current workspaces directory.

B

But if I provide the workspaces flag now you can imagine it's going to run and show me hey. I've run it against app and server. So you can imagine the kinds of you know: unbounded uh scripts. You can write with npm exec with workspaces now another new command we actually introduced in npm v 7 was npm pkg, so npm pkg allows you to get and set values in your package, json. So again, thinking about the different types of user flows. We unlock when we bring npm pkg together with workspaces, it's pretty pretty awesome.

B

So uh here's an example of me just getting the name field from the package json in the root and as you can see it's gonna say: oh, the name of this package was workspaces, but we can also run let's say: npm get version and that's going to get 1.0.0 and that's that's great again.

B

We're just getting the root information from the root package: json, providing the workspace flag, uh wakespace's, flag returns, essentially all the versions of all the workspaces which is really interesting, because now you can imagine workflows where you actually get and or set the version or other fields within package json for uh for your workspaces.

B

So here, instead of get, I'm actually going to write, set and I'm going to set the version to let's say 2.0.0 against all the workspaces, and so, if I run get again, I can see that now all of my versions of all my workspaces are now 2.0.0, which is really awesome. You can imagine how this kind of scales, when you're, trying to keep packages up today when you're trying to sort of keep some consistency within your your projects. The workspaces flag and pkg are great tools to be working together.

B

um So in terms of let's get back to sort of that testing narrative, uh let's go and use npm uh install now to actually pull in, uh let's say a testing library. So if I just pull in, let's say just with npm install uh so mbm install just.

B

Ipm is going to go, fetch it and add as a dependency to the root project.

B

There we go now what we can do and I'm sorry I just cleared my.

A

B

Is we can run npm pkg and we're actually going to set uh our scripts, so you can imagine with manipulating any of the package json fields, one of the key things we see. People do is actually add and remove or modify script run script values. So here we're going to change the actual default test script to use just now, and we can apply that again using the workspace flag to all of our workspaces.

B

So here I have if we do get, we can also see that this was applied now.

B

So now you can see that now I've switched the uh scripts uh the test uh script in each one of my workspaces to actually be running just- and I can put this some arbitrary sort of test uh command in here. uh Anything I want uh really powerful so now.

B

If I want to actually run this test, I actually want to do npm run or npm test.

B

I can do that across all my workspaces like I showed before, but what might be nice is, I don't need to let's say I'm a front-end developer and I don't need to really run the test for the back end, because I'm only working in the front end, uh you can actually use again the workspace specific command to delineate between which uh workspaces you want to run that command in.

B

So here I go, run npm test and I've got some and of course just is telling me I don't have any tests written, but you can see that it runs adjust and it only outputs once because it's only running within the app workspace awesome. So now, one of the biggest capabilities that comes with sort of this scope with using workspaces is actually filtered installs. So this is a huge thing that we've been asking been asking has been asked of us for quite a long time.

B

It's very similar to shallow checkouts from like git shallow checkouts, mpm filtered installs. You basically will only reify the or make real and install the packages uh of the workspace that you define so here's an example of that.

B

So when I want to, let's say change the test suite of my server side uh workspace, because you know I- I have a whole team that doesn't like jess, they want to use tap node tap. I can run npm. I tap to install the dependency and in this, in this case, I'm actually going to install this dependency into the server workspace.

B

So I define the the dash w and then the server uh workspace and now tap will be installed specifically into uh the server uh workspace and and not be installed at the root level and not be installed in the application or app workspace. So now, if I run, let's just quickly do a cats of the server.

B

And package.json, so you can see that tap is now defined as dependency there and if I run npm test and just for the server, I can see that now, I'm going to oh still get the jeff's output. So now I need to update my my set scripts again, so I will quick quickly change um my mpmp using npm pkg I'll quickly change the test script to use, tap and I'll. Do that against the server.

B

There we go, and now, if I run npm test again for this server, you should see the output from tap.

B

Again, it's complaining because I haven't actually written any tests. I've just started to add the uh infrastructure for running tests, um so you can see that this is very powerful to be able to install um and and directly add dependencies to various workspaces. But I really want to show you. uh You know the fact that we are only installing um specific dependencies for the specific workspaces you define when you run npm install with one of the workspace configs enabled.

B

So let me quickly blow away my node modules and my package lock file here and you can see now. uh I just have a clean package.json.

B

The app and server workspaces are still there and now, when I run npm install- and I do that with- let's say the w server- I'm only going to get the tap dependency and being installed, I won't get just. I won't get anything that app has defined and I actually won't get anything that the root has defined either. That is one we actually created. uh We created a separate flag for including uh the root workspace into this.

B

uh The set that actually gets installed with npm install so as you can see that we npm install added 242 packages.

B

This actually would have been double it's roughly around 500 if we, uh if we run install without that scope or that filter of the workspace, and we think that this is a great opportunity for people that build tools or have different use cases for different dependencies that they may need to install in different circumstances. They can use workspaces for that great. uh Let me and I'll quickly do mpmls, and it should show you that, um yes, it's gonna fail because it doesn't have all the uh information it needs at the root level.

B

But if I scope that to server mpmls should not error out and show you that tap actually was installed, and so now you can start running and configuring and using anything with that server.

B

um I think that's it for workspaces, I'd love to jump back to uh my slide deck. If I can and awesome so again, uh sort of to recap what I was showcasing there, uh the workspaces and workspace config arguments uh or flags uh were introduced in uh over time in the last major release line, which was npm, v7 and now should be fully available.

B

If you are using npm v8, we introduce also no work spaces, which is a split, an argument and configuration explicitly to opt out of workspaces in case you're, somehow deeply nested and somebody's configured. Something and you want to turn it off and include workspace root, actually will allow you to combine. uh You know execution against uh your workspaces also include that execution in the workspace root as well. So you can do some really interesting and advanced user workflows using the combination of those flags.

B

um I know we're trying to get through a lot. I know this is very dense, so I apologize, but uh one of the biggest features that I think we've launched this past year and actually just last week, is npm overrides and we're going to talk a lot about this in the new year. So this is going to be a very high level sort of demo and talk through this capability, but npm overrides.

B

Essentially, the benefits of this is to be able to swap out and help give you sort of a power user feature to override um when necessary.

B

uh You know dependencies or swap them out or patch them or fork them, or even in the worst case scenario, potentially even or the best case scenario, to help prevent malware or something like that from leaking into your into your projects. uh So it's a really. uh You know sharp tool, sharp uh knife tool um that we imagine people are going to to use quite a bit, and um we also you know, sort of built this tool in mind based on some of the work that had previously been there with yarn's resolutions.

B

Although our implementation is a little bit different, um specifically, the the way in which you write an override is a lot. um It's we consider it to be a lot more. um I I would say flexible. It's a little bit more freeform, it's more composable and very interesting uh sort of overrides that you can write with the way that we've implemented it here.

B

So again, I'm going to cross my fingers and hope the demo gods give us some luck. Switching back into my terminal here, you can see also some documentation in mpmjs.com, uh specifically under the package.json docs we've outlined um the overrides field and how it's implemented.

B

Also, given you a number of examples of how to use it, I'm only going to dive into sort of one of the use cases that we we find people and and think people will be using overrides for, um but yeah feel free to check it out, we'll be adding more docs, more tutorials and and be talking a lot more about overrides um in the new year and, if you're interested in in how this actually came to be and and sort of the design process around overrides, you can go to the mpmrc's repository on github.com, that's npm, slash, rfcs and inside the accepted field.

B

You'll uh or folder you'll actually find the specification for this. This uh you know feature um so if you don't know mpm, the mpmcli is an open source project. We try to do as much product design in the open and and source feedback from the community, and so this was one of those examples that we had been hearing for a long time. People really wanted over ad support, and so we think we think we've developed something really cool so jumping back into my terminal.

B

I'm going to jump over to the overheads folder in this folder actually have a pre-existing package json, which gives just a narrow scope of what we see a lot in a lot of people's projects today, which is they're, defining uh a dependency like react um that happens to have a very large plug-in ecosystem, a very large uh ecosystem of dependent, dependent or peer dependent packages, and here I've added also another project called react select, and in this case you know, there's conflicting versions and dependencies here that when I go to actually in npm install I'm going to get an error, because npm uh now is trying to resolve uh peer deps.

B

So, historically, uh in the last year or two we've been telling people to use uh lego c peered up flags to to basically ignore these warnings uh or to try to use force um to allow npm to to run in a mode that tries to resolve um pure dependency conflicts.

B

um So now, with overrides, we've provided another mechanism for um the end users to essentially self-serve and try to dictate which versions of of the fancies and and try to resolve conflicts themselves. If they feel like they, they know what that really. The version is they want between these two conflicting versions, and this will help also with transitive dependencies, which end users really can't control and don't maintain even the projects that they're depending upon so in my package.json, I'm going to quickly add um an override.

B

I went over it's back here to define the version of react uh and react, dom that I would like to use. So if I count my package here again, um you can see that here I've defined the overrides field. In my in package json and here I've nested the react, select package identifier to say anytime- that I see react select. I would like it to use this version of react, and this version of react-dom and what's really interesting here, is something that we just introduced, which is the reference key.

B

So like the our reference character in terms of the money sign, so that dollar sign will and the name of the package will basically make a reference to the existing dependency. I have defined, which in this case, is react and use that version and that spec in place of the version that might be getting tried to be pulled in by in this case, react select.

B

So if I now try to run npm install you're going to notice.

B

That there's no more, you resolve warnings and if I run npm ls and react so uh list all the versions of react that are installed. You'll notice that they're all pinned now to uh the latest version of 16 16.14 and everything is deduplicated and uh great. Now, I'm I can get on to work um and and move on with with my life and all those dependency uh conflicts.

B

um So that's overrides. There's many use cases to that. If uh I could switch back to my slides that'd be great.

B

um And I guess before I talk about what's next, we think again overrides is a huge, uh very sharp and very awesome tool that we hope people will start to use and give us feedback on. um But I did want to talk about what uh what was coming next. What's coming down the pipeline for the new year, and I should note that um this is all subject to change.

B

But since uh the talk or demo was called uh workspaces overrides and more, I thought I'd give y'all a little bit of a look into what is coming down our pipeline um come in 2022.

B

Our team is really focused on uh shipping and continuing to deliver on uh the the current major uh release that we support, which is v8 um and uh some things that are going to come in the new year or on isolated mode, which are which is very similar to pmpm's uh reification mode, which we think is going to really help uh folks that use workspaces or have mono repos and just have tons and tons and tons of package dependencies that get pulled in when they install. uh And this.

B

This really think we think is going to address a lot of performance issues and then into the later into the year, we're hoping to ship the next major version of uh of the cli b9 with a whole bunch of new capabilities and within that major release line, we're hoping to really ship some interesting features.

B

One I'm very very excited about is the dependency selector, syntax and there's a number of ui and ux improvements, we're hoping to make to commands like install and audit to make things clear and actionable for our users.

B

If you're interested on how we're actually queuing these things up, you can join us. We actually run weekly calls every wednesday we're taking a break for the holidays here and and sending a bit of vacation but we'll be coming back in the new year. I believe it's january 13th is the next call that we have queued up um and sorry, uh it should say wednesdays. It says uh thursdays there, but uh wednesdays at 2 p.m.

B

Eastern is when we run open rfc calls everyone in the community is allowed to join and and come contribute to the design and development of the mpm cli, and I just want to give a huge shout out to my team. um There are some stellar uh developers and engineers that are behind this work and so huge congrats to them, as well as the contributors that we've seen over the last year, or so we saw an extra uh 100 new contributors.

B

I think it was more than 100, maybe close to closer to 150 new contributors start to give back and and and help us uh build on sort of the goals of the mpm cli, and I think that's it. That's that's it for me. If you want to learn more, follow me on on any of the social media platforms. My twitter username is darcy and my github username is darcy clark, so yeah, and I think that's that wraps it up.

A

Thank you so much darcy that was super insightful. I appreciate the demos and today was a good day. The demo gods were in our favor. So thank you so much for everyone who watch and is joining us now, uh keep in mind. This will be published into our official github youtube channel. So if you wanted to go back and take a look at how these things came to be, you can go ahead and watch the content again shout out to all those new contributors super excited that the mpa cli is uh mpftls open source.

A

So if you would like to contribute like to participate catch one of those calls up in the coming new year and get involved what a great project. Thank you thank you for the focus on performance for next year and dependency, selecting that always like music to my ears. So it sounds like a fantastic plan. So if you want to learn more about darcy and his work be sure to follow him on any of the socials here and don't forget, go check out that repo, it's all open source.

A

So if you want to contribute, be a hundred and one you can be one of the ones getting a shout out next year, so thank you so much darcy. I appreciate everyone. Who's watched us here today uh have a great end of the year and we'll catch you again january. 7Th will be our next demo days, we're going to be talking with the one and only os uh zap proxy creator, simon bennett and we'll be talking about cool security things. So we hope to see you then thank you darcy.

A

I hope this was good for you and we'll see you next year. Fantastic. Thank you all have a great uh rest of your day. Everyone take care.