Cloud Native Computing Foundation PrometheusDay NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building a Runbook Automation System for Prometheus and Kubernetes - Natan Yellin, Robusta.dev

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Building a Runbook Automation System for Prometheus and Kubernetes - Natan Yellin, Robusta.dev

Natan will describe Robusta.dev, an open source runbook automation platform that was built for Kubernetes and Prometheus. You will learn about the motivation and challenges of building a production-ready runbook automation system. The talk will also include hands-on demos and examples. By the end of this talk, you will be comfortable writing YAML that automates away your job. You will be able to sleep at night and let your Prometheus alerts investigate themselves and tell you in Slack what the problems are.

A

uh Runbook automation, systems for Prometheus and kubernetes and we have nathanielen from robustard F uh talking to us.

B

Everyone hear me: okay, thank you! So I have some light. Demos planned, but the internet here is a little bit iffy, so I'm going to do my best, but uh bear with me and I've tried to cut them back to the minimum I made to make it work.

B

Okay, so a little bit about myself, so I'm, not tan and I, wear many different hats and I guess this thing doesn't necessarily work: okay, so I'm going to stay a little bit closer, yeah, okay, so I, wear a bunch of different hats and I was a developer for many years. Austin open source contributor uh for 15 years now, starting with gnome Linux I'm, co-founder of Robusta, which does Prometheus based monitoring for kubernetes and I. Do a live, devrel stuff!

B

So here's an example: video I did with my grandmother um devops with Grandma Sue, where we talk about kubernetes and explain in very non-technical terms, why it matters um so Guinea explain to your spouse or to other people. What you do, then that's often a good reference now outside of work, I'm, also a crazy fat guy. So I grow a live different vegetables, mostly tomatoes, grow a whole bunch of different stuff. With my wife and today, I'm going to be talking about run book animation, so I'm going to be talking about what run book.

B

Automation is why you need it and can make your life better, how it works under the hood and assuming that the internet is okay, then you'll see live demos and, if not then you'll see screenshots so wait before I go on I'm just curious is anyone? Is anyone here using some form of runbook Automation in production today?

B

Okay, so I'd love to hear from you guys after as I'm, also curious about what you guys do and I also just want to see on the aspect ratio that this isn't cut off. Can you see the bottom of this okay yeah? So one? Second, let me see if I can fix this. The aspect ratio is wrong.

B

Is that better okay? Perfect? So? Who here knows this other? Has anyone seen this before.

B

Oh yikes, okay uh yeah, you hit the button on this. Has anyone here seen this alert before this is uh an alert called qpod crash looping, it's part of the default set of alerts for cube, Prometheus stack, so super common in there and the runbook automation is, is taking the dirt like this and then adding on context so connecting either to dogs to graphs, adding on knowledge about why another like that is firing and sometimes even being able to fix another like that automatically.

B

Now, when we speak about context, I just want to give a non-technical example to show why context matters so much so. Can anyone guess what this object is.

B

Come on guys give it a go.

B

A security camera. What do you say?

B

Wally, oh yeah, looks a little bit like a robot like Wally, some kind of cute robot or something from The Terminator, okay. So to explain what this is now I'm going to add on context, and this just shows how powerful and how important context is.

B

So this is an object that you use when you work from home and your cat starts jumping all over your desk and you want to occupy your cat so that you could actually get some work done and as soon as you see this object with the context, then it suddenly becomes apparent what this is and why it matters. When you see another out of context or when you see an object like this out of context, then you really don't know what it is and if it means something or not.

B

So that's the goal of runbook automation to add. On that context, now I recently asked on LinkedIn what context we should gather for that I showed you before so I asked I'm putting this in like terms that no more humans use now not about context and lyrics. I said: what's the first thing you do when you investigate this other and that's really like the same question but, for instance, the way people like actually think and.

A

B

Got a whole lot of different answers, so being the daily driven guy that I am I once and I went over all those answers and they summarized it and I counted up how many different times I got each answer and let me go through those answers. So two people say you should name the devops team.

B

One person recommended uh one person recommended running this command.

B

Does anyone know what that command does.

B

Okay, so for those of you who aren't in on the joke that uninstalls kubernetes, but then we got a bunch of serious answers too. So four people said you should run Cube cow, get events or look at the kubernetes events, which are events emitted by the API server. That could be useful diagnostic information.

B

Four people said you should run Cube Cal describe, keep Cal, describe poison events and some other stuff and the overwhelming majority, so that what you want to do when you have that alert, is you want to look at the application Knox? So you should go and fetch the power dogs, and you know why the Pod is crashing.

B

So what we're going to do today is we're going to take the other and we're going to automate that process, so we're going to go and we're going to take better and automatically pull in those pad logs and you'll, get it right there in stock. So as soon as you get better you'll have the context on why that other is firing. Now this is a simple example, the one that we encounter every day, but you can also apply this concept to far more advanced things that we'll see later on.

B

So let me give you a teaser for what this will look like at the end, you're going to get another like that in stock and there. If you look carefully, you can see they're at the bottom. If I know how to use this pointer thing, maybe not, but you can see there at the bottom, like the regular labels and everything essentially we're taking the metadata from either and we're taking the labels and we're mapping it into kubernetes object and then automatically we're pulling in all that context.

B

So the person who gets tired of dirt doesn't get another, and it's like. Okay, here's another good doc, but now you have the context you can see why that either is firing foreign.

B

So now, let's talk a little bit about Prometheus other than architecture, so the way that it's normally work in Prometheus and I'm sure there are people in this room who are far more experienced than experness than I am so, if I'm getting anything wrong here and I'm sure I'm, giving some simplifications here and feel free to come over to me afterwards.

B

um But what we're going to do is sorry. The way that things normally work is Prometheus issues of Earth's. Those are those get forward to their manager and the dirt manager has more advanced Logic. On top of that, for example, it groups the others it has a grouping interval I mean I made noise. Since my notes are resolved, it does various different things. Actually the resolving happens in Prometheus, but the manager has some magic around that too I believe um and then finally, the alert manager takes either or that cellular.

B

So if it grouped it and now it forwards those alerts by webhook to the destination that you receive with earthson, so you get others at the end of the day in stock or in Ms teams or in all these different destinations.

B

um Some recently added support to robust actually for uh Cisco WebEx, so I guess there are people getting there since Cisco WebEx, which kind of surprises me.

B

So you get the alerts there in stock and now we're going to change the architecture a little bit and we're going to add in an additional component. So one way that you can Implement run book automation, which is very popular. That's how we're going to do it is we're going to add in an extra component here in the middle, so you have alerts that come from Prometheus. They go event manager from alert manager. They get sent now not directly to stock retirement teams.

B

They get sent to the wrong book engine the runbook engine takes the other. It pulls an extra context pools in that context about what this other is, what it means. Maybe why it's happening or even an automated fix, and then it sends that other plus the extra context now onto the Final Destination. So that's what we're going to do- and this is a good time to say what Robusta is so Robusta is an open source project on GitHub, of course, and it contains two main parts. So the first is the runbook engine.

B

That's the engine that takes these incoming events and then adds on the context according to a bunch of rules that you define in yaml and then the second part is we've gone and we're going over all the others in Cube Prometheus stock and all the other common alerts that people have, because we all are in kubernetes.

B

So we actually all have very similar alerts to one another and we're taking those alerts and we're adding on these run books out of the box, so that anyone who gets to there it will just get a good alert by default without needing to configure anything. And then that leads to help your developers and people who bother you. That's because now they understand why they're just firing. So that's our goal.

B

Okay, so uh robust is MIT licensed and please take a moment and scan that and give us a star on GitHub that really helps us as a project and as an open source project helps us spread awareness also about what we do and I will also say.

B

We have a bunch of community run books that are like going on mapping and stuff in Cube, Prometheus stack and mapping out other common Prometheus errors, um so we're doing our best to really get wide coverage, and this is one of the reasons why it's so important to us to be here today to really also engage with the brother, Prometheus community and speak about what we're missing and what content we should add on.

B

And let me now show you how you set this up. So the first thing we're going to do is we're going to go to the manager's configuration we're going to add on the web hook receiver.

B

We have full instructions for this online and, of course, the Prometheus docs are excellent and cover this as well, and essentially, what we're telling now Prometheus or to be precise, but we're telling alert manager to do, is we're telling alert manager that, when there's an alert you shouldn't go and send it directly to slack anymore we're going to send that alert to the runbook engine.

B

Foreign next part is like kind of the hard part right now. You actually have to write the wrong book engine and the runbook engine. Is this HTTP server? So it's getting all these alerts by HTTP over webhook and then you have to take those alerts and you have to parse them and you have to pull out the different context about the other, and then you have to like uh contact the API server and pull in the logs and do all this other stuff. So this is the traditional way you would do it.

B

You would write a bunch of code that runs an HTTP server and has like a whole bunch of if else's and different stuff to handle each of the different edge cases, and what we've done with Robusta is we're trying to make this really as simple as possible and to bring run book automation to the masses. So to make that happen, we now turn this into yaml configuration, so another reaches Robusta. Robusta then goes and it parses the CMO file.

B

It looks up the other that arrives in the set of rules, and it says okay for this, given alert. How should I enrich it? What context is missing if this is another about crashing pad or if you get another about a note that ran out of disk space?

B

What do I need to see as a person who's on call in order to solve that issue, and then that's where the brother Community comes in all the community coverage to really out of the box, give you all the stuff that you need to see that everything and to see that kubernetes Meandering.

B

So here's a configuration I'm going to go over the configuration now, so every automation, every runbook in robust, has three parts there's a trigger. So that's the condition. That's triggering this runbook. In this case it's the Prometheus alert from Cube from at the stack, that's called Cube pod crash looping.

B

The action is what we're going to do in that arrives, and the action here is we're going to pull in the logs and the sync is where we send the data so we're sending the data here to slack you can send Thomas teams, you can send it to like I, said Cisco WebEx, you can send tops Genie page or do that you can send it all over the place. I believe you can even send it to data dog um okay. So the last part, then is I just want to say a word, though.

B

That's maybe not obvious when you look at this the first time, but data is flowing through all of these. This is almost like a data pipeline. So if you look at this actually, then we're saying there's an alert here on Prometheus other when the path crashes, but data flows from that trigger into the action. So the action now it says, go fetch the dogs, but it knows who to fetch the dogs for because we're taking the metadata from that Prometheus alert.

B

That's there from the kubernetes Prometheus Discovery and then we're using that in order to map that alert to a kubernetes object, and then we can pull in the logs from that kubernetes object. So this is actually like. uh This is actually like a typed pipeline like the data passing through this knows that this other is related to this pad. It knows the kubernetes object, it has all the kubernetes context and then you can very easily pull in the right data and of course this is customizable okay.

B

So this is the part where we all cross our fingers and I really hope that the Wi-Fi does not fail. Me um I always like to say like especially when we go when we sponsor events like never respond to the Wi-Fi, um especially if you're a company that develops networking equipment. Never sponsor their Wi-Fi, um but we see the same companies doing that again and again: okay, so I'm going to run this I'm now triggering an example there and I'm going to jump over here to stock I.

A

B

I actually have an old like something in there. So let's see I just print a bunch of stuff, so we're not going to cheat too much. Hopefully we won't cheat at all and okay. Here we got it and this other is still loading I think so in just a moment up, oh okay!

B

Here we can see it so what we guys, we got it Prometheus alert and then Robusta automatically pulled in the dogs through the Pod that crashed, and we see that right there, the dogs for the crashing pod and that's right there in the other itself- and that's this and the thing to emphasize that this is all configurable on their rules and we have rules out of the box to really do like the right thing for most of the common alerts.

B

um So it's not just about crashing paths. That just happens to be a very popular use case. Okay, so let's go back to the slides.

B

That's the part where I do the dive demo I want to do more live demos, but I was a little bit worried about the Wi-Fi, so I'm glad we got that done and I want to speak now about three more advanced use cases. So, first of all, let's generalize the concept. Now the constant runbook automation, it's not just about handling events that come from from Prometheus. It's a very popular use case that we see among our users, but it's not the only one you could say.

B

For example, um we had someone say once um I have an Ingress, that's like crucial to the functioning my cluster, and if anyone touches that Ingress I want to get notified or if anyone touches the Ingress go and take some operation.

B

um So you can actually do something like that very easily. It's the same concept. You have a trigger. You have an action.

B

You have a sync, so the trigger is someone modify that in Grass the action is okay, maybe pulling some data or show me exactly like the exact gift that they modified and then the destination is stack or almost teams or wherever so um there's an open source project called cubewatch, that's fairly popular um and the original maintainers left that so we took over as the official maintainers now um and were the official maintenance of Q watch and we use Q watch under the hood as part of this runbook automation engine.

B

So you can actually track any kubernetes change and then you can get notified. You can even run different automated actions when different things happen in your cluster, so this is often very useful. Also, if you want to do the sort of thing, that's not really time series based, but if you want to monitor discrete events, like you might say, I want to monitor the number of Crash of uh like jobs that failed, I, actually care about jobs coming out of these jobs.

B

It's like a discrete event, that's happening a job failed and then you can also do runbook automation or notifications around that sort of thing uh me. So this is very useful and we maintain Cube watch We're actually about to release I think today or tomorrow, a new version of cube watch that fixes a bunch of bugs the first version to come out in a very long time. So that's exciting.

B

um So that's one Advanced use case and then I want to speak about two more advanced use cases. So the second one is deeper insights, so you're not limited to just pulling in data like here of the alerts and connecting to the graphs and other stuff. You can actually apply logic because each one of those automated actions we saw under the hood each one of those automated actions, actually a python function.

B

So it's very easy to extend this yourself and you can actually take all the knowledge like that's in your head or in the head of the best person in the world. That knows a specific application, how to monitor it and how to maintain it, and you can take that knowledge. You can turn that into automated code. So an example that I like to give is like, let's see again another about a problem in elastic search.

B

um You know the problem and you might know. Maybe there are slow queries in elasticsearch, that's great, but the first thing you're going to do is you're going to go and Google that right because you're, not you, you probably aren't an elasticsearch expert, and if you are, then the same thing will happen for mongodb and you'll have to go and Google it, because there's really, we all deploy Stacks nowadays, they're very wide they're very wide stacks, and we don't all have like you can't know everything.

B

So one cool thing you can do is you can take specific issues and then take the knowledge about how to handle that issue, and one time someone from the community really knows that topic turns into an automated run book, and now everyone can benefit from that and one of the great things about kubernetes. Is you have this convergence where everyone is running on kubernetes and many people are deploying the same home charts.

B

Many people are deploying the same software and actually suddenly my errors start looking a lot like your errors, so the potential of runbook automation is the potential to really take the knowledge for how to handle errors that come up in production and to turn into automated Community shared knowledge. So here's an example of that.

B

This is something that we include out of the box if you've ever got to learn about CPU throughout the link in your cluster, and you want to know why you have high CPU throughout the link, then we have coverage out of the box, for example, and we check a few different cases. We check for different known issues. We check whether the issue is uh the kubernetes CPU limit, if so, whether it's safe to remove it.

B

So the potential here is the ability to really capture the understanding of why different errors occur and then to suggest Solutions and even automate the fixes So speaking about automated fixes. That's the third use case and that's use case that I'll look at so we actually support me either complete automations or what we call human in the loop automations.

B

So you can add rules where there will be a button in the slack message that you get that says: okay, here's an issue that we identified and you defined in advance, like here's, the fix I think is the proper fix.

B

You can either run that fix completely automatically or you can give people a button in slack and they just push that button and as soon as they push that button, then it'll go in there around the automated run buck so you're running the automated run book but you're putting a human in the loop and you can automate that 100 as well. It really depends on your philosophy and uh how comfortable you feel with that fix.

B

So we have more demos and demo scenarios like you just run cubecado applied to your cluster and then you can see the stuff in action, um so we have more stuff and get and please also feel free to reach out and suggest stuff, and we have coverage for a bunch of stuff out of the box as well already so like Dean's schedule. The other CPU over commit uh coupon not already different node issues.

B

um Job failures, CPU throttling power system, known disk space image, will back off. We have coverage for a bunch of different things and we're constantly trying to build that out, and this is the part where I say. Please send us to your problems like we like hearing about people's problems and I have the issues here in this list either we can contributed by the community or we contribute because people came to us and they said I have this issue in my cluster I.

B

Don't know why it's happening and I'm busy and I don't have time to look into it. So I'm not going to go and automate myself. So please look into this, and can you out on coverage for that? We do that all the time and we love it so like. Please, give us your issues um and we'll take them, investigate them and trying that on coverage and I want to just end with one more final Story. How am I doing on time.

B

Okay, okay, so I'll be fine, then yeah, so I want to just end up with one more story about why this matters so who here uh likes to rabbit or goes on Reddit regularly.

B

Okay, all of us, yeah I, figured and does anyone here know this subreddit, that's called what is this thing?

B

Okay, so there's a sub that I, like that's, called. What is this thing and people post a picture of something they found they go. You know what is this thing and then the internet tries to tell them what that thing is. So here's a post that I saw I, think it's Monday top all-time posts, and someone said I found this in a crawl space of a house from the 80s, my friend's house, and it was next to these boxes, and so they were hard to read with a picture, but it says radioactive material.

B

No person shall remain within one meter of container unnecessarily and the person who found this thought you know when you find the box, it says radioactive material.

B

What you do is you post it on your edit in on what is this thing and then you like go off and go hiking or something so this book's really bad right, like this, doesn't look like there's a happy ending to this story and I'm going to take this as an analogy for what happens when you get a message in slack and it's like 3am like you, get a message and there's some issue in production, and it could be a really really urgent issue um and if you work in an industry like healthcare, I mean like, depending on industry, that you work in stuff like we deal with software, but the software that we deal with has real world consequences on what people do so moving on with the story like this looks really bad and the advice on Reddit was like go to the hospital immediately and the person updated.

B

um A three-man team from the state of Utah radiation control showed up at my friend's house and that's like the people with the spacesuits like in the movies, and they go around. They took a bunch of like probes and they swap stuff, and it turns out it wasn't really such a big deal.

B

They, the team that came out, found nothing but natural, like Trace Amounts of radon, and it turns out the former owner was just the watchmaker and they used to paint watches they glowed in the dark, with a paint called radium, and most people know about this because, like the people who painted it would like the watches and then they all got tongue cancer, but assuming that you're not picking it and you're not actually ingesting this.

B

This is really not a big deal and I want to take this now as an example back to the learning context, like sometimes there's an issue that looks like a big big deal when you add on a little bit of context, it's really not that bad and we speak a lot of other fatigue right and like not wearing people out with too many issues that look too serious, and then you realize that nothing is serious and then one comes a very serious issue when you ignore it. So that's not.

B

The story is not that you should ignore issues, but that's another story is that you want to get context as fast as possible so that if it's a serious issue you can deal with it and if not, then you can just relax not be under stress.

B

So that's the example of the power of context and what we're out to do with Robusta and the reason that we're here in engaging this community is. We want to add on as much context as possible, with open source, runbook Automation and with the community of all the different issues, so that people can sleep better at night. When you get another in stock, then you can have the context for whatever means and whether it matters or not.

B

So, thank you very much for listening and um we wanted to do something a little bit different in our booth. So we brought the kubernetes ship wheel. We wanted to try and bring kubernetes into the physical world I'm friends over screen of that when I'm not like outside in the garden, but we wanted to like hold kubernetes in our hand, and you can too so. Please come by the booth and please say hi and please tell us about your issues like we'd love, to hear and we'd love to add coverage. Thank you very much.