Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Observing Kubernetes Without Losing Your Mind - Vicki Cheung

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Observing Kubernetes Without Losing Your Mind - Vicki Cheung, KubeCon + CloudNativeCon Europe 2020 Co-Chair & Engineering Manager, Lyft

This is a talk about managing complexity. A lot of teams find operating and monitoring larger scale distributed systems like Kubernetes daunting. When there are so many moving parts, how do you make sure everything is healthy? How do you make sure they're working together properly? Vicki will tell a story of how Infra teams can quickly get a handle on how their systems are running by monitoring from the end user's perspective.

https://sched.co/ZfGv

A

Hi um so I'm vicki and I'm the co-chair of this conference along with constance and today I want to talk to you about how you can observe kubernetes without losing your mind. So I'm a software engineer at lyft. uh If you were paying attention, I was previously a manager. So at some points in this talk I might put on my manager hat, and um this is following a theme from my talk last year.

A

So I want to talk about complexity again, because this is really a topic that comes up uh all the time from different angles. So last time I talked about complexity to your user and how did we simplify the user experience this time? I want to talk about complexity from the operator's point of view and how you can simplify things for your info team.

A

Okay, as an operator step, zero is to deploy your kubernetes. um That sounds simple enough.

A

Okay, so let's deploy all of these things, um I actually had to go into my team's repo to look at like um what things I missed and there were actually quite a bit, but I think you get the point that um there are a lot of pieces to the system.

A

uh You don't need all of it to work like to get a vanilla, kubernetes deployment. You don't need, you know even half of this stuff, but I think for a typical enterprise installation.

A

This is really what you're looking at.

A

So you're, looking at like a lot of pancakes um okay, so you got everything installed and running um and it seems, like things, are green and ready to go so now, you're ready to go on your cloud native journey and on board all your users and your applications.

A

And we all know things go wrong. Sometimes, and you know it's: okay, we're good engineers and we do post-mortems and we, you know, take whatever lessons we can, so we don't ever have that incident again and so a lot of times, uh at least for me. I think a lot of post-mortem uh action items are: how do we catch this faster? How do we monitor better?

A

So now we have to add monitoring and there were a lot of different components in the system. So you know where do you add monitoring? Do you add it everywhere? Now you have to know what it means for each thing to be running well and healthy.

A

So what you end up with, maybe is that you have a ton of alarms, but somehow things are still going wrong.

A

So you know, I think it's a pretty uh routine thing where you have an alarm and then you tune it because you know you didn't have a very good baseline before and maybe you have more data now, so you get paged at 3 a.m and then you grumpily tune your alarms.

A

So now you have all these very artisanally crafted alarms.

A

And you have more postmortems and then you add more artisanally. Crafted alarms.

A

And then you're like what it's even happening anymore, um when you, when you get paged, you know you're, maybe spending a cycle or two being like. Is this real? um What is the impact? Why does it matter.

A

So your team is slowly losing their mind and this is definitely loosely based on some real stories.

A

uh So the moral of the story is that distributing systems are really hard and they have a lot of moving parts, and even if you get each moving part right, you still have to get the interactions between the parts right. So it's not just getting all the individual components running correctly. It's also that how do you make sure they're behaving correctly together.

A

So can you really define the right behavior for each part, as well as their interfaces to other parts?

A

What do you actually care about- and this is me putting my manager hat on? What is the impact.

A

So I think, at the end of the day, um what I care about is actually that my users can do things that my cluster's working correctly for them. um So let's go into an example.

A

For example, my my users are running jobs on my cluster, so let's just say that they can run a job on my cluster. In this example, I chose you know a pod that can make some networking requests.

A

So how do I test for that? Well, it's actually pretty straightforward. um It turns out you can more easily define what the right behavior is to create a good user experience. Then maybe you can define all the individual little uh behaviors from each component of your system to create an end result. That is correct. So here's a very simple test that I can write. It's like a three-step test. I create a pod.

A

The pot does basic potty things like maybe download a private file from s3, and then I clean up after myself, and then I just run this over and over again.

A

That was a very simple test, maybe like a super tiny script but you're getting so much out of it. Here are the signals that maybe you're getting. You know that the api is responding because you can create and delete your pod. You know that scheduling works because the pod did a thing.

A

You know that networking works because the pod was able to download a file, and maybe, throughout this process you log a stat at each point of the at each step of the cycle, and so you can say how long the weight was between each step, and so you can say even that latency is acceptable and so on and so on.

A

So now you're aligning what you care about with what works for your user, um and this makes actually updating your status pages a lot easier, as is you get paged on this particular test like pods can launch now you can just automatically update your status page and say: pods can't launch um so you're really making it so that what you care most about is your users.

A

Okay, let's say you get paged what happens next.

A

This is when you dig into your observability toolbox. um These are just some of the core tools that I rely on that basically come out of the box from my kubernetes installation, and I'm sure people have a lot fancier tools that they've built that they rely on. um But I find that these are the things I reach for most often.

A

And I also want to talk about uh in the same theme uh as the rest of the talk. You know, I focus a lot on simplicity, so I want to talk about a couple of simple tools that I've written, that I got a lot of value out of um it's probably not as fancy as you know, some of your enterprise installations, but uh just bear with me, sir, for a second so tiny tool. One is an event watcher all it does.

A

Is it watches for events on the cluster and it locks them, and then I have a log forwarder that slips up the logs and throws it into the last elastic search with the rest of my locks, tiny tool. 2 is a container watcher. I watch all the containers, events and changes on the hosts and I log those updates.

A

Similarly, my log forwarder reads all those logs and forwards them to where my logs go okay, so those were two super tiny scripts again that are in my toolbox, but combined.

A

What I get is one timeline for what happened to my jobs, so here's a hypothetical timeline for creating a job and then eventually a container crashing.

A

um I think, typically, what you might do if you get paged for um something like this is you'll have to cuddle describe maybe like 20 things, you have to look at logs of like different objects.

A

Here's just my attempt at simplifying that workflow and giving you one unified view.

A

um And then another good thing about the the single timeline is that it also empowers your users to debug their own, um their own issues as well, because because it's simpler because they don't have to know about all the system internals to do cucuto describes um to know where what events are. Even so, I think it's a win-win.

A

And again, I know they're super simple, but sometimes simple is best.

A

And talking about the users, do you have users who build platforms on top of your platform.

A

So now they can use the same trick. Actually they can run their workflow tests continuously on the cluster as well.

A

I think the key takeaway is that when you write these end-to-end monitors, you don't have to be an expert in the underlying system. You just have to be able to define what behaviors you want and how you want them consistently.

A

So you're really only dealing with the highest level of abstraction and then everything that's happening underneath you know the systems uh components and how they interact with each other. You don't really care about.

A

That so again, if I could encourage one thing, is to try to rely on simple things, really think about what what is the simplest thing you can do to get the signals you need that you care about, and just do that, and it makes your customer communications a lot better. Also because now, what you care about is what your customers care about. You care that they can do their thing.

A

You have a monitor that constantly checks that they can do it, and so you can discover errors or even like minor issues or inconsistencies before your customers do.

A

So I hope maybe this will simplify some of your life. If not, please feel free to reach out to me I'd love to talk to you about that. Thank you so much for having me again.