Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 15 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: High Reliability Infrastructure Migrations - Julia Evans, Software Engineer, Stripe

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Keynote: High Reliability Infrastructure Migrations - Julia Evans, Software Engineer, Stripe

For companies with high availability requirements (99.99% uptime or higher), running new software in production comes with a lot of risks. But it's possible to make significant infrastructure changes while maintaining the availability your customers expect!I'll give you a toolbox for derisking migrations and making infrastructure changes with confidence, with examples from our Kubernetes & Envoy experience at Stripe.

To Learn More: https://sched.co/GsxA

A

Hello cute con I'm very excited to be here with all of you. This talk is called high. Reliability, infrastructure, migrations. All right, I have an objection, Jan stripe! um All you need to know about stripe. For this talk, so we're payments company, we process billions of dollars for a year for our users. What does that mean a better priorities right? What are our challenges in infrastructure? We do not process ten bajillion rigs a second.

A

If we did my equity people to be worth more, we don't have like twenty like we don't need your return results in twenty milliseconds. Well, we care about infrastructures, reliability and we care about security right um I could have like ridden. This whole talk about security and kubernetes, but it's going to be about reliability. That's it. We had time for mom and a lot of the services that we need to operate at stripe.

A

um I need to like have availability of four nines or a bug right, um and if you want to be up 99.99% of time like how good is their house like Alice about a you, can be down to about a minute a week right, including like everything that goes wrong, not just like what you may have done to your infrastructure. So it's not a lot of time right. We I'm gonna, be talking about sort of two changes we've made in this talk. One of them is he lives somewhere close to kubernetes.

A

We moved a bunch of cron jobs, a bunch of machine learning workloads as well as some of our HTTP services, and we also started move teasing envoy for all of our sort of service to service networking, which is like a really big internal networking change and to use mutual TLS and when I started working on this I didn't know bring either these projects anything about Corrine, Eddie's or envoy, and I really wish that I was like okay, I could I wish I had this like list of like everything that could possibly go wrong and then I could write down all the solutions and then I could like make deploy new infrastructure and everything would just work and we wouldn't have any problems, and this was the the reality right is.

A

That was like me and I was like what go wrong. I, don't know, probably a lot of stuff. How are we gonna get there right like? How are we gonna go from like we know nothing about how to operate the software or very little so like we're, confident and critical production software on this right and we're comfortable like promising our users? That this will be fine, um that's really big leap, so the sock is about like what is the process right like?

A

How do you get from here to there and the way I think about this process is that we have sort of like what could go wrong in your infrastructure and these red dots are sort of like things that could possibly go wrong and on the Left we kind of have like normal problems. We're like. If you talk to anyone, a coupon they'll be like oh yeah. We saw that too. That happens all the time.

A

I'm two problems on the right, where you're like gns did what which which are may be much more uncommon, but that you still have like I mean who has had these problems right? Where you're like what happened?

A

Yeah, it's the reality and if you want to run a high availability service right, that's going to be up all the time. You need to deal with like a substantial number of these issues that are more like on, but like what happened and at the spectra main you should have heard for them. um And how do you do that? When you don't know anything at the beginning, um so the goal is like fix all these problems right somehow. So how are we gonna? Do that?

A

um Let's JIT understand the design run gamedays, classify your failures, five incidents, only one time my favorite I'm like make incremental changes. So let's do it step. One is understanding, kubernetes design, um I'm, not gonna, go into this a lot, but I think it is really important to like realize that, like if you're gonna run software in production, you need to know how it's designed right. Look. You need to know how it works in Korea Nettie's case this sort of like basic version of this yeah Betsy D, that's a the storage.

A

You have the API server. Then you have like all these stateless services. The changes to worry about a little bit less make it. We operate chrome manage ourselves right now, um so it's really important for us to understand this right. Another consequence of this is that we try to really focus our efforts on a really small set of software, so we use kubernetes, we use envoy. We don't use any of this other stuff right.

A

I, don't know what it is for the most part um and I think really focusing really helps us build much more high reliability systems, um but okay, so, like let's say you've like learned something about how compromise is designed. You know the basics. You still know how it can break right. You can't Google like how can kubernetes break it. Doesn't it will not help right? You need to learn how to your system breaks in practice, so you set up to burn a tease. I like you, install it and then like the first I.

A

Think one of the first things that we started to do is like okay, let's cause them problems on purpose. Right, like we can come up with failure modes that we know will happen because, like everything goes wrong in reality and what let's make them happen, so we call these game days. Lots of people have other names for them, but what they do is they test? How your system behave behave under like known failure, conditions right um when those failures happen.

A

It lets you learn how to deal with them kind of like not at 4:00 in the morning, which is what I have learned about some failure conditions and there it turns out that they're really good way to like share knowledge on your team. A couple of examples of things we did one of my favorite game days with kubernetes. It's like terminate all the API servers and see what happens and the first time I did. This I was like what will happen? Will it work?

A

They claimed that my possible keep running, will they and they did and I was like? Oh wow I terminated all the API servers. The pods are still running before we get too excited. We did this again recently we were like. Oh there are now some problems. Cuz they've made our system more complicated I'm. So it's good to run these things repeatedly, even when you think you know how your systems works, cuz, sometimes it's changed. um You do all kinds of things to make your system right and it's great to do.

A

Lots of them, some of them we do in KB, also sometimes do them in production I'm, usually when we're more sure that they'll work, but it it makes sense right. You're these things are gonna happen in production anyway. So might as well happen when you're there it's better. They also let us sort of like test our fixes. We had this issue early on in our use of kubernetes when kubernetes terminated every running pod in our cluster.

A

um This is something called like pod eviction anyway and we're like oh okay, cool good to know that can happen, um and so we like based on kubernetes configuration changes, and then we tested the pics right. We're like we reproduce the failure mode where we like terminated just one API server and we're like okay. It does more like the right thing now. Good.

A

Alright, the next thing I want to talk about is like how to sort of classify your failure modes, um because, when I started out using kubernetes, I kind of had this feeling, like anything, could happen right like I am so terrified. I mean this is not true.

A

Most of the failures that we see with kubernetes in practice are one like pods, don't start for various reasons on there, sometimes for missions, errors where we don't configure communities correctly to like, like their pods, have access to the resources we need, and we seen that we're paying issues and like pods, don't start for so many reasons right like if there could be a bug in the scheduler, but something that happened there could be a program like the current dive controller, which is something that having that Dede it could be down, could have like I am problems we get AB like problems in our application.

A

There are so many reasons, um but we won't have monitoring for all of them. Alright, so having this kind of classification like pods, don't start is a problem right for like who knows why we can develop monitoring. So we built this thing called the what we call the heartbeat job and, basically, what it does is. It tries to start a pod every minute and if the pod doesn't search someone's, get paged, I'm, I'm technical right now of it.

A

After this talk, I will be on call again I'm so like if the heartbeat job doesn't run, I'll get paid and I can go, do something about it right. um That's really comforting to know that um that I'm sort of protected from like this, the cause of issues or at least double know about it.

A

um The next thing I want to talk about is my very favorite thing, which is like have every incident only one time, because, like we have this problem space right like these things that could go wrong and like as like you getting like test kubernetes right, you learn how it works. You can run game days but, like ultimately, you're gonna run into some problems in production like it happens, and ideally, you've been just sort of doing a slow rollout.

A

So when this problems happen to you initially in production, like kubernetes terminates all the pods, your cluster you're, like oh good, we were running anything too important, which is the case which is great, but you need to fix these issues right um and, if you don't fix like it'd, be issued, but you want to fix every issue so that sort of never comes back, because if you don't do that, you could never focus on the stuff which is like really hard and weird to fix, um and then you can never sort of get to the reliability that you need.

A

So it's kind of like if you just like, want to have the time on your team to deal with your issues. You need to have incidents all they want time in the kind of like process, but it's pretty simple in principle. Right, like you, have a problem you find there causes you plummets from remediations. Usually the problem never comes back in general I'm, not that's superstitious, like I'm, a really big believer and like something went wrong.

A

We will figure it out and then it will never come back like we will fly logic systems are logical, like there are no like gremlins and my kubernetes cluster I. Don't believe in that, but something that is important, I think there's two fix like categories of incidents right and not just like sort of individual things that happened. So one example of this was the Envoy. Is it what you started? Seeing we started moving to envoy now, a lot of weird issues like there would be a request. Timeout, it's like flow requests or connection.

A

Timeouts are like because weird like thundering herd issues, we're like a ton of connections would get created and it would use all the CPU and I like we were able to come up with their mediations for most of them, um and oh and I could kind of understand them. But I was like why, like fundamentally like, why is this happening to us right like the Envoy is supposed to be good? It is good but like why it so why is this happening right like why?

A

They're all these weird issues and the way we were running on boy as you ran this connection, pool HTTP, one connections between every pair of pond boys in our cluster, and these were using TLS, and it turns out that, like every single one of these, like weird issues, we're all issues with the HT one connection pool- and it turns out that envoy is really designed for these connection is between two envoys to use HTTP two, and so we switch to using HD two just like pretty recently for like one of when one of our whenever one of our like our biggest cluster, because we're like we need to do something to improve things here, and everything just got better and I was like.

A

Oh wow. Thank God like like we've I, think we found like the the reason that all this weird stuff was happening and I think it's important. It's like well, if it feels like you're sort of like putting a bandaid on a bunch of problems, I'm to try to be like. Is there like another like more root cause to this right? Because often there is yeah. That's right!

A

All right, um the next thing is very important for making sure that your incidents happen only once, there's a tell your co-workers what you learn from incidents on, we had another Siddhant early on we're really put @cd uneasy. Yes and then we're on it's like right throttling we have leader elections, but he fixed it. We moved it off ups and then we did the same exact same thing again: cuz, look. It didn't get communicated that this happened. Right, I was just looking at this.

A

uh This thing that happened was like all the pods getting killed and my coworker wrote this like beautiful three-page report last December and I was like. Oh that's exactly what happened, and it's so nice to go Bo and like go and read like all of the details about this incident and like what exactly caused it and how you investigated it and helped so much and then like you're, not the only one who knows, and if you go on vacation, it's fine, um so something that someone asked me when I was writing.

A

This talk was like what, if I can't figure out what caused this incident right, which is a real concern. There have been incidents that I, couldn't figure out, I. Think like it's hard, but I. Think it's important to think of this is like an opportunity to learn like incidents. Teach you how to build a reliable system right um and I feel, like everything I know, is from investigating incidents. I mean like oh, okay, that's why this happened.

A

um If you want to know I, don't know if you want to know more about like how did you this maybe see every other talk I've ever given. That's the only other thing. I talk about um alright next thing make incremental changes.

A

This is this is obviously key right um like if you're starting out running kubernetes, you don't want to start running it on your most like critical thing, um you want to start running it somewhere, like a little less scary. There are a lot of really common ways to do this. You can be like okay, let's do a 1% of traffic 0.1% of traffic by percent of traffic, a less critical service just host.

A

um This is all extremely important, I think it's pretty well understood and another way to make a real change to a system in general. I. Think that's really nice! It's to establish an interface boundary. So we had this a ploy system kind of circles, 2017, which is a little like a scary. um There may be some ghosts in it, but you didn't really feel like and I said, I, don't believe in gremlins, but I think we kind of felt a little bit.

A

It was a little grab money, and so my coworker John took on a rewrite of the system and what he did was he just sort of like set up an interface boundary in the middle of the system right? That was the type that was really clear, and this was a huge improvement right. This is much better cuz. Then you can start to gradually work on your system and it can be like okay, new client server is still the same, and then we give her the new server um and it was beautiful. It's much now.

A

We understand it. I mean here's. This really great post about this called no haunted forest which I really recommend. Reading it's wonderful um I won't talk about little tiny bit of interfaces.

A

This is this is just this is not a normative statement.

A

This is just our approach to be clear: I don't want to start like a flame war, but the reason that we do this is that we feel it reduces cognitive load I'm for developers who are busy like trying to understand the global payment system, and, frankly, I, don't think, have time to understand kubernetes and to reduce our support burden right like if we define clear interfaces for how people interact with kubernetes it's much easier for us to kind of like support that and operate that um I want to show.

A

So we started out by using gamal to sort of establishment interface between us and our users, so we would be like okay feel like he was like basically how people would configure their cron jobs on kubernetes, um maybe define a sam'l file. It was really nice I'm there, a couple of sort of key issues with this I'm. The main one is that it turns out that it was pretty difficult for developers to understand like what other attributes were supported. They were like what can I put in the sam'l file. Is their schema like?

A

How does it work and like are people you could have documented it better, but I feel like this is kind of like a fundamental issue with the animal. um It's also sort of hard for people to tell like from the outside, like what everybody's configure. This is really gonna generate right. It's it's not very transparent. That's it. We started taking the approach. Instead of letting people call functions, cuz we're like oh, what's an interface. The function is an interface.

A

Everyone understands functions, and so the new way that we're having people define like kubernetes services is vibrating functions. I'm in this language called skylark, which I'm gonna talk about a little more in a second um but you're. Just like okay, like stripe service, here's the commands, here's, how many replicas here's about how much CPU and then this generates kubernetes configuration right, I'm gonna. This is in a subset of Python. Its type check its sandbox. You can run it anywhere. You can run on the web. I'm gonna show you a demo right now.

A

Actually, if we can have, can we have a demo I'm? Not will my computer wake up? Will I get oh yeah, okay cool.

A

So this is a really simple, like webassembly demo demo of this thing, um so basically I've defined like a job function which generates a kubernetes dog and takes like a name at a command as arguments and then I can just call that function and be like maybe I could rename it's like cool job and then like give it a name and then, if it's type checked right, so if I'm, like Oh name equals to it look like no name does not equal two, that's not a name and so I think it's really nice, but like infrastructure software to have this kind of interface, um if you're interested in sort of like configuring kubernetes this way, this is very much like it and early like it's not like big and professional.

A

It's a early-stage thing. Please come work with us on it. If you're interested I'm, you gonna, try it out at Sky, config dot, fun. Alright, the last thing, I wouldn't say, is like always have a rollback plan. um I think this maybe has happened to a lot of us where you're, like oh I, thought the screw.

A

That was going to work oops, and it's really really good I'm when, when this happens, you're like oh good I, had just rolled this up to 1% and I'm gonna roll it back down to zero for said, and it's fine right, I'm going into sort of like every change with a plan like how are we gonna just act if this goes wrong and like here's, how I'm gonna roll it back in five minutes?

A

If it does um makes me sleep a lot better and makes my deploys feel a lot safer, um all right, this was their playbook. This is what we know about what I've learned so far about how to like operate kubernetes and, as a result, I think my team has had no like very serious, like kubernetes incidents, even through making some pretty major changes.

A

It's worked pretty well, um which was surprising to me because we started out being like what is this stuff, and so that's nice I want to talk a little bit about kind of like team culture, because reliability is not like something that I do by myself. Right. Reliability is a team sport. We all do it together and I think on our team. It's okay to start out an expert, but like the people on the team needs to become experts right. We need to really deeply understand the software. We need to know how it works.

A

Beus know what its failure modes are and before to get everyone on the team. There I mean soap. You just kind of build this like engine of learning on our team, where everybody is learning um they're learning about how the cluster can fail. Maybe we do gaming, they should get there and were like hey. Let's go I'll see how this thing bales together. Let me write incident reports amuse me to share them and make sure that other people understand them and don't just kind of like throw them out there and like talk them through right.

A

um If you find something weird explain to other people in your team, what happened and building this extra piece takes a ton of time we've been using kubernetes for a year and a half. We know a lot about how to use it now, but I. Don't think that we're done right like we're not running our most critical stuff about kubernetes, still um I. Think that learning a lot of this software is a really long process um and it's a big investment if you're a manager.

A

um One thing that my manager does, that has made a huge difference, is every time I say: don't let him like hey J. This change isn't gonna be happening this week. This weird thing happened and I want to investigate to make sure that I understand what happened and that we have a plan to make sure it doesn't happen again. He's like great Thank, You Julia.

A

That's what we need to do right so, like here's, an HR space for your team to learn how to fix these issues, and then you can do amazing things together. That's all I have to say thank you.