npm npmCamp Presentations (2016), 30 Aug 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: npmCamp 2016 - Registry Design Patterns by CJ Silverio

Description

Registry Design Patterns by CJ Silverio

A

Pangolins are just amazingly weird dude. Do any of you want to move down front at all, and you have any view just holding back. All the way around the back edge is okay, all right great. How are you doing this? Is the last talk and I'm gonna be really pretty technical, so I want you to wake up, shake yourself into your seat, all right. You ready. Okay, we're gonna talk about design patterns and modularity in the NPM registry.

A

This is who I am the title means to me that I have all of MPM registry MPM engineering under my care, not just the registry, I have the CLI and a website under my care. These are things that I worry about every day. The picture is not of me. It is the picture of my operations. Mascot Malcolm Malcolm will be referring to an emotional subtext through the presentation.

A

So human brains are pattern, detecting machines, we unconsciously search for and find patterns, and everything that we do. You will have found a theme in a pattern to this conference. Whether you're aware of the fact that you were like looking for that or not, and whether the organizers of the conference intentionally put a theme there or not. You're gonna find one because that's what our brains do.

A

We see things we match them to other things, we've seen and we find patterns the process of writing software is a process of abstraction and then pattern extraction from the from the pattern from the abstractions that we see. We as software engineers are obsessed with finding patterns, so we do it in software. We do it in code and we do it in the systems that we build on top of the code. So what I'm gonna do is talk to you about emergent patterns in the NPMs registry. I'm gonna descriptively talk about them, I'm, not gonna.

A

We didn't set out to put them there, but I've seen them now that we're there and I'm not gonna just talk about how they scale. I've talked about that a bunch thinks we're scaling you're installing tons of stuff every minute from us, and you don't notice that it's boring. Let's talk about modularity. This is one of the emergent themes of NPM and something practically every speaker has mentioned when they talk about code here, if they're not talking about community, they talk about code and they say the word modularity or modules.

A

So, let's talk about emergent patterns in the NPM registry. What is the registry I here? Isaac doesn't understand what the registry is anymore. So, let's, let's back up and define some terms, the registry to me is the system of services that manage package tower archives and package metadata. That's it simple as that it doesn't talk about billing. It's not talking about websites I'm talking about the things that let you put a package on the registry and then get it back again afterwards.

A

So this is what I have to think of when and serve as fast as possible to you every day and we get some numbers. I ran these numbers on one of our entire Bowl servers just before coming up here. So this is pretty pretty up-to-date. It's a lot of packages, but I would call this medium data. It doesn't fit in memory, but it does fit on a single disk. Still, barely discs are getting large enough fast enough, but that may stay true for a while.

A

Yet, and so it's NPMs largest engineering challenge making the CLI work on all the hosts. All the machines you run it on is is a significant challenge, but then the registry is the largest one by the numbers.

A

So I've learned a lot by doing this, for you all over the last two and a half years, I'm going to tell you some things that I've learned today here are some of the patterns that I've seen in the registry, monoliths microservices, transaction logs and message queues. That's not pretty Herot! It's a mixed list there, so I'm going to go through them and talk about like how has seen them come up and what I've learned about modularity and these monoliths. We all know this one everything is in one big process: it's probably a rock!

A

That's what lift means all right. One Rock our website, most people's websites, are mono lifts. It's really easy to write. A monolithic website at the registry was at one point, a monolith as an app running inside couchdb. It's nothing wrong with monoliths. They are just fine they're easy to write easy to change. This is fantastic. When you don't know what your problem is, you don't know what your users want. You don't know what you want. You want your code to be very squishy, so put it all in one place.

A

The performance is more than good enough when you're measuring your your your usage in requests per minute right, anything is going to work. So choose the pattern that works to. Let you develop your app and then Malcolm shows up one day says: hey, guess what you succeeded now, what you've got a scale?

A

It's not just scaling performance right. You also have to scale the size of the team that works on the code, and these are. These are two things where monoliths are less okay, the very same factors that make them easy to work on and squishable make them hard to work on later. When you have to make them faster and you have to make more people work on them here, for example, is Malcolm yelling at two people at once, he has scaled by using two cell phones.

A

What if he has to yell at a third person, he's gonna have to have somebody come and hold a third cell phone, a fourth person he's in trouble. This is how you scale a monolith. It's not actually gonna work very well. The other problem is that it's really easy to write highly coupled code inside a non modular monolith and the reason is a human reason. There are no barriers to it. No cognitive conceptual barriers. You've got one repo.

A

Probably you can change lots of things all at once, all over the place, because you are as a human understand everything that's going on. You held the whole system in your head. It's probably not very big. It's very the two secretly couple, your code together.

A

Let's stop a moment again and be less vague about what modularity is.

A

Where does this modularity come from? Comes from something really great term information hiding this term comes to us from a paper that I'm gonna recommend to you all on the criteria to be used in decomposing systems into modules, DL, partes 1972. Could one of you look that up on the internet and tweet a link to it on the hashtag and PM camp right now. Somebody did that the example that partes works through in this paper is very, very 1972.

A

You will you'll, be it's classic the kind of programming problem that people had then, but the concept is key. Here's the concept information is data, how you structure that data and the algorithms you choose to operate on that data, because data structures and algorithms are practically the same thing. If you hide information behind an interface, you can change it. Somebody looking at what a module does is looking at what the module exposes to the outside the handles that it gives you to grasp and pull on the module and how it does things.

A

If nothing outside your module can see how you structured things inside it, then you are free to change. What's inside without breaking your users.

A

This is why it's so important when you hide things you can change them and for any long-lived system or any long-lived project you're going to eventually want to change things because circumstances, change Malcolm is extremely interested in this insight, because it is one of the great keys to programming and if you ever understand it and can manage to do it the first time you do anything.

A

Please tell me, because I would like to know how to master this because I can't it is the constant struggle of programming and the constant fun part of programming, it's figuring out how to do this right. Okay, so we have a monolith and it's got code that isn't really all that modular inside. What do you do?

A

A hot trend these days, you'll hear this at all the node conferences rewrite as a micro service and that's exactly what we did, which is great, because we are 100% on trend and that's awesome about half of the current registry is micro services, little teeny services, but do a thing and talk to each other great thing about micro services. Is that when you write them, you are forced to design an API of some kind and you are forced to put your implementation inside a service away from everything else.

A

Somehow so modularity is kinda thrust on us kind of sort of at least an attempt at modularity. They can still mess it up. Unfortunately, as you probably already know people, this is why a human is writing the code. A human knows how things go over here and a human writing code over here is very, very tempted a lot of the time, particularly under time pressure, to take advantage of what they know about that thing. Over there over here is they're doing their job, because the human doing the typing knows it all.

A

The services they develop can know far too much about how data is structured or how the data needs to change when a package gets published or something.

A

You can scatter a task across multiple services in a microservices architecture. You can make retries and failures hard to cope with. You can have somebody start doing something here continue doing the same thing here and then try to clean it up over here and over here. It doesn't know enough. I might be mentioning this problem for a reason, I think it will look at an example here, it's let's look at what happens when you publish a package, here's a a block, diagram, I love, making block diagrams it. Doesn't this look beautifully modular?

A

Look at all these little little like diagrams with service names in them they have arrows connecting them. This there's a secret horror hidden in this diagram, and it's in the horror of two arrows that point to the same thing: when you authorize to publish a package, the off plug-in goes and talks to the user access control, lists server and sets up a bunch of pacts package access stuff, because that was the easy way to do it at the time, and we knew we needed to do that. Sets it up as a side effect.

A

So what happens? If a service crashes midway through publication or if the validation services, nope nope, sorry, this isn't the valid package Shaw sums wrong moot. Well, we have user access control, tombstones sitting in the database that we have to clean up that one service. Doesn't the service further down the chain doesn't know about these access control lists Malcolm is not relaxed.

A

Malcolm is off to yell at some databases. I'll relax, though we'll come back to this topic, but right now I'm going to relax because after package of publication, it kind of gets awesome after package publication. We do this amazing thing called the transaction log you might hear. This is called a transaction log right ahead, log a commit log. This is a great pattern. I love! This pattern. Learn this pattern. Here's another paper, I want you to read: actually it's a blog post! Somebody looked this one up to the log.

A

What every software engineer should know about. Real-Time data is unifying abstraction written by a LinkedIn engineer on LinkedIn, of course, wrote this amazing thing well, LinkedIn is a is an amazed spam as a really very highly scaled service. So we can learn a lot. Timely did.

A

So couchdb is the database at the heart of npm and it has a superpower. You heard this superpower talked about earlier today. It is the changes feed.

A

This is a commit log, pretty much that you can use to get data about the current state of every document in the couch database, which for us is every package in the range registry in the order of least recently modified most recently modified.

A

We write a whole bunch of things called registry file- followers which are consumers of this commit log. It's exactly the same pattern that greenkeeper uses that we use internally to to follow what happens when packages get published, and this is like really cool. This is a very highly accurate diagram of all the consumers. We have that do various things we fan out like whoa after our package gets published, here's an incomplete list of the things we do when you publish a package after the fact before you publish the clock, is ticking.

A

You want to get that publication in as fast as possible so that you get a response as fast as possible, but after publication we can relax it. We have tens of seconds to do things we distribute tarballs. We invalidate our CD ends. Cache we populate Postgres DB, so the website can see the updates. We put the index data in elasticsearch so that we well you it's not very good search, but we have it. There is a search we scan packages for security leaks. You know this is a thing that we do.

A

If you accidentally publish your NPM token, we will invalidate that token. For you, we populate our internal registry, mirror that we rely on for our operations and we fire web hooks and we do a bunch of other things to each. One of these. Followers is a separate process. Each log consumer does one thing. Well, it's really easy to be modular when you're doing this, this is so awesome. The 12th doctor has showed up to say. Yes, this is awesome.

A

Okay, alright, let's come back to publication; fine, let's fix it, but how are we gonna fix it? Now we enter the speculative portion of the talk, because this is like what we're thinking about doing what we're sort of moving on now. I'm, tired of microservices Malcolm is tired of Micra services. We messed up. The module is a ssin unwinding. Failure. Is a pain, got to figure out how to do this? Okay, two words message, keeps message: queues are an inversion of control. Micro services are very imperative.

A

You have code with a bunch of unit mutable data, saying okay, do this next thing: I'm handing it off to the next next bit of code. It's gonna do this next thing to it, and the data is gonna mutate, as it goes through the message queues you do it kind of the other way around. You have an immutable message that says the state of a task and what the next thing that needs to happen is you have a message in your message. Queue is a kind of a request for work.

A

You can have workers consume messages, do a thing retry, if that's appropriate or unwind on failure, because one place encapsulate s' the state of the work that was done or needs to be done. It's a little clearer to figure out what to do. Worker should should.

A

Unfortunately, people do one thing and then put the message on the queue for the next worker to do one thing and you kind of scale it the same way. You scale out, commit log consumers. Here we have for Malcolm's running around. If we needed to yell at more people, we would just make more Malcolm's.

A

The queue has to be reliable workers can't crash it's okay. We have a message sitting in this robust queue them it's it's a database. We hope our databases are reliable. They aren't don't tell anyone. This is a really popular pattern. This is so popular, go to queues, deho and look at all the queues. This isn't all of them, but.

A

Reimagine as I do what publication looks like if we do it in a message: queue it's much more controlled publication is a series of steps in which data enters the registry. We can each step can succeed or fail failure, contributor, a rollback and then tell you sorry, you don't have the right to publish that package queue, disadvantages, I, don't know we don't have them in production. Clearly there are infinity disadvantages or zero disadvantages, I don't know, but there this is a. This is the next big trend.

A

This is often where systems need to go to when they need to scale to the next level, when that first level of distributed systems doesn't work for you anymore. So you know I, don't know Malcolm nine, eight. Neither of us know you can mess up writing a queue work or just as badly as you can mess up writing anything else, because the hard part is in the brain of the human writing. The code always.

A

So, what's the pattern we get from this discussion of patterns.

A

The registry is a pretty small distributed system. Really, all things considered linked in that giant span farm is much much larger. I, don't even want to think what Twitter looks like inside. That's big, we're small. We can learn something from it anyway and the listen that I have it's gonna shock. Nobody, you ready, there is no silver bullet there. Never is a silver bullet. Fred brooks was right. None of these patterns are wrong.

A

None of these patterns are right. It's all about the specifics of where you are with your project.

A

It's trade offs all the way down it always. Is you can't fool me? What problem are you solving? Do you understand the problem you're solving? Yet, if you don't understand the problems you're solving, yet you should probably pick something that lets you squish. It around. I'll switch the code around a lot, while you're experimenting write a monolith.

A

What tools do have to hand micro-services are very attractive, but they actually add a lot of complexity. Right from the start, you have to. You have to think about how you're handling retries and failures right for the start. Otherwise, you're gonna get the big mess. What is your team experienced with? Are they like? Yes, we use message queues all the time we speak rabbitmq like Kafka. We got it in that case, like you, might want to go with the message: queue based architecture from the start, though, again, what problem are you solving?

A

You will need to fight for modularity, no matter what you pick, we messed it up. We know that modularity matters. We live small modules. We still slice them wrong. That's fine! It's! Okay! You live! You! Learn you better write code. It's wrong! You write code, it's right! I! Think you swear it's accident, but the upside is that you can have modularity, no matter what you pick, because it's all in your brain and how you think about the problem that you have.

A

Make your users happy first, because that's the hard part that's where projects actually fail.

A

Guarantee you probably very few projects have failed because they picked one of these patterns. Instead of one of the other patterns they fail because they they don't connect with somebody who wants to use them.

A

Here's the other thing I've learned for working at NPM over the last two and a half years. You can change them.

A

This is this is a surprising thing, you think, of like code as being this big heavyweight that you can't move, but you actually can you can you can modularity I gave a talk about this one, and this is like I just want to say it again. Modularity is a salvation if you've hidden something you can change it, and that includes from yourself.

A

You can put proxies in front of things so that you internally don't care whether it's this service or that service implementing something we're right now in production have like a user, a CL service, that's legacy and user a CL service. That's is that its new and they're both handling requests.

A

We just switch between them depending on like what the request is and which service knows how to do this and we're gradually, shifting more and more load over to the new one, and when the old one is finally not handling any requests anymore, we'll take it out of production, and you won't have noticed anything and most of the engineers and NPM won't have noticed anything either because we hid our implementation from ourselves in a really kind of cool way. You can change everything that you did. You should probably plan on changing everything.

A

You did it's going to be. Ok, you will get it wrong and then you will fix it and get it right.

A

So I'm gonna try message queues for publication, already, taking little baby steps in this direction and check back with me next year to find out how we messed it up. What went right, what went wrong? How much I hate the message queue we picked because, as you know, the way to hate a database or anything is to put it in production. You you love everything before you use it right.

A

So you'll see mission. You'll see me again as I'll talk about like what I've learned from the next few set of steps that we do and you'll see me again tonight. If you want to come to the after party, please I encourage you all to come to the after party, because I'm going to be DJing. Thank you very much.