keptn Community videos, 12 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Andreas Grabner - Don’t GitOps into a Blackhole: Use SLOs and Get DORA a the K8s Native Way!

Description

GitOps has become the default way to manage configuration in cloud native environments with tools like ArgoCD or Flux keeping Git and K8s in sync. But deployment quality hasn't necessarily improved!
Its because GitOps lacks end-2-end traceability when operators make changes on the target environments to identify where things break. GitOps also lacks standardized pre- and post deployment checks against Error Budgets, SLOs or external dependencies.

In this talk we introduce a way to solve those challenges by extending the K8s scheduler. One implementation is the CNCF Incubating project Keptn which provides end-2-end OpenTelemetry-based deployment tracing, introduces deployment lifecycle events and orchestrates pre- and post- deployment checks on Error Budgets, SLOs, dependencies ...
On top of that you get automated DORA metrics via Prometheus - everything in a K8s native fashion!

Andreas Grabner, DevOps Activist at Dynatrace, DevRel for CNCF Keptn, CNCF Ambassador, PurePerformance Podcast host
Andreas on Twitter https://bit.ly/3nLde8Y

This presentation was recorded at WTF is SRE 2023, London. #WTFisSRE

Learn more about Container Solutions events:
https://www.container-solutions.com/events

WTF is Cloud Native Collections:
https://www.container-solutions.com/wtf-is-cloud-native

Links to our social media channels:
LinkedIn - https://bit.ly/3I0NMD2
Twitter - https://bit.ly/3I0NLyY
Youtube SUBSCRIBE TO OUR CHANNEL - https://bit.ly/3VXMUFg

A

All right, thank you for being here and deciding to join my talk. It's really hard to see with all the bright lights, but there's a couple of people, a quick show of hands who is here because you are an S3 and who is here because you want to find out what the is this Siri? Okay, you raise your hand twice, so you honest if I still don't know nobody to do it. Okay, my name is Andy grabner.

A

Thank you for the introduction uh during the day, I work for dynatrace, so in case you're interested in what we do down the trace is my colleagues are downstairs, but really today, I'm representing the cncf, the cloud native Computing Foundation I, am one of the ambassadors and I also am a maintainer and a founder of Captain I'm wearing it with pride. As you can see here, I will everything I show you today is all open source. That means I, don't sell you anything I.

A

Just really think I can give you some ideas on how to solve some of your challenges as an SRE as it comes to observing the stuff that is happening in your environments. The titleists don't get UPS into a black hole using slos and get Dora the kubernetes native way. If you stay with me, I hope you're, not afraid of yaml. There will be yamls. There will be different tools. There will be terminal windows, we adjusted the screen size earlier.

A

We try to make it as big as possible, but for the people that sit in the back, unless you have really good eyes, stay there otherwise feel free to come forward. uh But I understand just before lunch. You may want to sit in the back. There you go there, you go. Let's get all right, I know yeah! Here we go now we have. We have a movement going on here, it's nice all right, so uh we see a lot of live demos and don't be scared. There.

A

You go more there's more seats in the front row second row because I can tell you I'm, 40, plus, and my eyesight is really getting worse and worse, because I was always sitting in the back. That's why I think in the future? Okay, uh quick pulse I want to get a quick pause of the of the audience. Please raise your hand if you're, using or intending to use kubernetes as your standard plan platform.

A

All right, that's good. Second, open Telemetry, who has heard about open, telemeters, use, no pink, Telemetry, also good, pretty much the same hands uh gidops all right. What is kid ups for you, yeah yeah? What does you say? You raised your hand you're interested using it. What is what? How do you use kit UPS?

A

We Point our deployments: okay, perfect, desired State Management if it desired stating kit and you have tools like Argo, City or flux or any others that are then apply in the desired state to the actual State, and you hope that the desired state will become the actual stage State and stays there. Okay, perfect, Dora, metrics.

A

Of course, you have to raise your hand, a guy from Google uh who else raised your hand. I didn't see this one, two three so Dora our four key metrics, that kind of are used to measure the adoption and the I think the majority of devops adoption uh deployment, frequency deployment, failure, rate uh failures of deployment and time to restore service. So these are four of the metrics that are typically used.

A

uh When we uh figure out do we actually do the right things and I'll be moving forward with all the millions and millions we put into the tools and the people we educate right, so Dora metrics another one slos who is using slos understands SLO service level objectives. What is a good SLO? Give me an example.

A

Perfect I'll try to con repeat it and condense it down, but you said the very first thing you said something that is relevant to your customer for your customer. It's not relevant if your CPU hits 85 percent for your customer, it's important that your system is available and that they can do the things that they want to do in a manageable amount of time. So performance is typically key.

A

Latency I think, as you said, also failure rate things should work as expected, so slos and the last thing has anybody heard about Captain before and maybe even using captain. Or is this completely new all right? You get all converted today.

A

um So I have three major agenda items uh first or three major use cases that I think are challenging for sres. These Day first observability tools sprawl what that means. You have many many tools of choices when it comes to observability. The challenge is: how do you connect all of the observability tools to your tools that run in kubernetes that actually make automated decisions? So, if you think about you mentioned Argo you're using Argo CD, but you may be also use Argo.

A

Rollouts for Canary deployments, Argo role, I need to make a decision based on data on good or not good right. But the question is: do you have one observability tool or do you have oops a multiple observability tools? Different teams use different tools and they sit somewhere outside.

A

So I want to talk about how we can solve the problem that, while we have standardized with open Telemetry on how to interest data and obserability, we want to bring a standard back to kubernetes that allows you to consume observability data in kubernetes, regardless of where it's stored and where it's managed.

A

Second, one is unmeasurable change impact we heard about this. The challenge is really it's really hard from the Oracle perspective. I think if you're deploying a complex application- and it fails so if it's slow how to troubleshoot it right, because you can look at logs, you can look at parts and all the events, but um I will give you a demo on how we we are extending how you can extend the kubernetes port scheduler, which is at the heart of kubernetes, and actually get Telemetry and traceability.

A

So, where I'm going to show you how we can create open, Telemetry traces as kubernetes is deploying the changes that Argo CD is pushing out right and then you can use it for troubleshooting, but also you get your durometrics last one is I, call it pipelines, Gone, Wild um I. Think it's really hard coming I have to point What's, Your Name by the way Muhammad it's it's. Actually it's great that we have tools like Argo, but you may not only have argument.

A

You have other tools as well that you're using to deploy and to enforce testing security observability across all of these different pipeline tools is really hard. So I want to show you also a way how we can natively in kubernetes in the Pod scheduler enforce security, observability and tests good. So the first one even already hinted on it. So I have always one slide for the use case or the challenge and then a demo obserability tools brawl. As I mentioned.

A

You may have multiple observability tools on top Prometheus dynatrace data dock new rally, Khan econ whatever it is, and you have tools on the bottom like Argo that need access to this data to make a decision, and the challenge right now is that if you look at HPA or cada right for auto scaling, you have different plugins or also for Argo. That's like two three five for every obstacability tool: there's a plugin and if you're building a pipeline, if you're building a platform, you need to somehow figure out if a new obserability tool comes in.

A

How do we connect this to all the other tools that need to make observability based decisions? And that's why I always call it? You know obso, building tools for all. We need to have a standard for this, and the standard I want to propose is coming out of the cncf project that I'm representing it's the kept metric server. The captain metric server basically specifies a standard on how you can pull in any type of observability data back into kubernetes and make it available through either Prometheus or the kubernetes metrics API.

A

To all of these tools, it solves a major problem. Actually two I think, first of all, all of these tools- vendors on the bottom- they don't need to provide and build plugins for every point to point integration and on the other side it also helps the stability vendors to not have to build and maintain these point-to-point Integrations. So how does this look like a quick demo, um so the My Demo gods and I I got a wired connection, let's see if this works, but my screen just went by all right.

A

um So what do I have here just to show you what I've installed I have installed on my kubernetes cluster I just run case k3s, it's a lightweight kubernetes cluster, on an ec2 machine, so I just have an ec2 machine out. There k3s is installed the whole demo. The link I'll give you later as well. If you install Captain um yo, that's different components that come with it. The one that I focus on right now is the so-called cap metrics operator.

A

You can only also just install this component, and then you get this use case that I'm showing you right now out of the box for free, so the metrics operator, what it does in my kubernetes cluster.

A

It is watching out for things that get deployed in my case with Argo, so this is Argo CD in case you've, never seen it. um What I have in my first use case. I have a git repository here in your environment. I have a git repository with a bunch of files and they are automatically it's my desired state I'm pushed over to kubernetes and to show you how the metrics piece works.

A

The only thing you need to do is you need to tell Captain what is the metrics provider so I have Prometheus as a metrics provider and I also have a second metrics provider. I do work for dynatrace, so I have a dynatrace metrics provider. You could also. We already have a data, doc Matrix provider and AWS Cloud watch somebody's currently working on honeycomb and New Relic. So building a new metrics provider is very easy. It's about 20 lines of code. It's just translating a query to an API call of that Target tool.

A

So you specify what is your data source and then you specify uh what type of metrics do you want to pull in and hopefully again this is big enough, but you can see here. I have a kept metric. That's called available CPUs and I, bring it I, pull it in from Prometheus and here's the prompt ql query: I have a second metric that is called availability, SLO a service level objective.

A

This in my case comes from Dyna Trace, because I use dynators for SLO management, and so here's the query against Dana trace the beauty is Captain. Metrics operator is basically constantly pulling from these from Prometheus from dinotrades from datadog to data in and makes it available to me in kubernetes. So one way to check this out I'm just doing a cube, CTL getmetric get cap metrics you can see here. These are the two metrics I specified with the current value, so we're exposing these values through Prometheus.

A

So if you have most of the tools are out that are out there in kubernetes they have a Prometheus integration. So you can fetch it from there. uh The other thing we do. We also expose it through the um as I mentioned in the beginning, the kubernetes metrics API tools like cada I'm, not sure if you're familiar with Kate and HPA, the horizontal pod, scheduler and cada is also for event-driven uh scaling.

A

um I can now use these metrics to specify how to scale so no longer am I limited to just data that I have available within kubernetes. I can now pull in data from external sources and to Europe what was your name again evgeny. You mentioned what matters. The most are slos that are close to the end user, like availability. So now what I can do I can specify.

A

A where's, my my scaling, this is I, can specify an HPA configuration, horizontal, pods, scheduler and I said I have on our scale my deployment based on the availability as slow and the nice thing about this is I. Don't need to specify here anymore, that this comes from dynatrace and maybe tomorrow, I switch to data dog or to New Relic or some rails, as an SRE I can specify these are the metrics I want to pull back in into kubernetes and give it to the tools that need to make data driven decisions.

A

So we are separating the concerns of what data I need and where the data comes from, and we provide an open standard for this. Okay make sense and you can change it behind the scenes. Yeah yeah and the thing is right: you are your developers, maybe they Define their scaling rules based on whatever metric, but they may not know what your obserability tool of choice is that you are putting into the platform they don't need to learn from ql or dynator is query language, so it separate a separation of concern.

A

Here is nice, so that's the the first demo and the first use case data highlighted the Second Use case that I have is unmeasurable change impact, so I assume uh putting it when you do if Argo Argo makes changes, but maybe you have other tools that make changes. Maybe you have somebody even that dares to remote into a machine and then make changes with Cube CTL apply or something like natively. That means you have multiple tools that make changes to application layer or to even infrastructure layer right.

A

Your increases, your service definitions, your deployments, your workloads and then the question. Is it's really hard to figure out? Why did this deployment all of a sudden with Argo take five times longer than yesterday? What else changed and what uh what else is going on?

A

There's, no real great visibility into this one solution we also bring in with the captain open source project is What's called the captain life cycle operator. The lifecycle operator extends the kubernetes Pod scheduler. Now, if you're not familiar with the parts catalog, the Pod schedule is the component in kubernetes. That says, somebody wants to deploy a pod I'm, trying to figure out where the right resources and then put it there, but we can extend it through plugins, but we can do additional things.

A

The basic thing we're doing here is actually figuring out and monitoring and measuring what the Pod schedule is doing. What is it deploying? How long does it take technically how this looks like right, some technical details here, if you, if you're deploying, let's say three services, then what we can do, we can measure the time from before the deployment actually happens. So as this support scheduler tries to find the right spot and then deploys it, so we can do pre and post, which means we can create metrics timing metrics.

A

But what we also do is we are looking at metadata on your deployments and we can say: hey front-end vacant and storage. They actually belong to a bigger application, so I really want to Monitor and get insights into everything that changes in context of an application that I'm changing and therefore right. So we can also measure in the end, the end to end time. It takes for the whole app to deploy and individual components and the way we are exposing this is through open Telemetry.

A

So I'll give you a demo in my environment, my simple node environment. The only thing you need to do is in order to enable this. The only thing you need to do is on the namespace, where you deploy your app. This is my my namespace is my simple note: it's a simple node app! It's my devspace. You need to enable the cool the lifecycle toolkit.

A

So that means, if I go quickly. Back to my my captain installation in my captain lifecycle toolkit namespace. There is the lifecycle operator right.

A

That's really the thing that is now extending the the Pod scheduler, but it's it's only getting active if you have the namespace annotated with that, you want Captain to become active so by simply doing this, and if I go in now and if I go to my deployment and if I make a change here, I'm just making a simple change: I'm changing my version, I'm just bumping the version here, not deploying anything new, but I'm bumping a version uh v602 commit.

A

So what happens now for those people that know argue or Flex Argo will say: hey, there's a change. I need to sync it there we go right, it's out of sync, so the desired state is is, is, is a is a mismatch right and now it basically says Hey kubernetes I want you to deploy this, as this is happening. Captain detects this namespace is monitored by me.

A

You can see here. Captain actually creates a new object, it's called the hard to see, probably but it's the Captain workload instance. It detects as a new version being deployed.

A

um This way this this runs a little bit of 13 minutes, left and I will add. I will come to the full details in my next in my last demo, but to a quick uh peek forward.

A

What we are doing, what the what Captain does it is creating a trace, a distributed, end-to-end trace of everything that we now see that the Pod schedule does in the context of that deployment of dead. Sync all right. So one of the things we have is we first of all get some nice metrics, the Dora metrics at least some of them. Things like how many deployments happened in my last six hours, that I did time between deployments deployment time perversion average time between deployments, successful, failed and what you also see here on the bottom.

A

uh These are the actual traces behind the scenes. So for those of you that know open, telemetry, open, Telemetry, the the key use cases actually to trace transactions from your from your business, apps from when you're you're, hitting a URL and then tracing it all the way through. But we what we are doing is we are using open, Telemetry to trace a kind of like a business process because deploying your change is kind of like a transaction you're asking kubernetes to do something.

A

So what we are monitoring here, what we're creating is an end-to-end trace of everything we see the Pod scheduler does there's a little bit more in here. That I will explain in my third use case, but we are showing you hey. It took one minute and 25 seconds to the kubernetes to actually deploy this particular workload. There are some additional pre and post deployment checks that I come to in a second, but this allows you to figure out. How long did it take? Why did it fail if it fails?

A

All of the events are also here. Everything will be shown here. It's a nice Trace, giving you observability into your deployments. Okay, good 11 minutes left brings me to my last use case: I call it pipelines, Gone Wild.

A

Why are you smiling now So pipeline's Gone Wild? uh The thing is: we need to enforce different things, different policies in our organization right, you want to run a functional tests.

A

Security tests you want to validate now one option is that you go to all of your development pipelines all of your tools and try to fit this in and enforce this pre and post deployment checks, but it's really hard and so what we bring to the table again with, with the captain life cycle operator that sits in the Pod scheduler, so everything that is deployed, we have an option to actually stop the Pod scheduler before it deploys your pot.

A

We can execute actions and we can do something afterwards, so we can actually natively bring pre and post deployment checks into kubernetes, and then it doesn't matter if you're using Argo Jenkins or if somebody does a cube, CTL apply, we can enforce these checks, natively no, there's no way around it. Nobody can bypass it right. So, if you think about this, this slide again that I showed earlier, where I explained we're using this to measure the end-to-end deployment time and Trace. We now give you the options to execute pre and post deployment.

A

We call them tasks and evaluations. There's two types: we allow you to do and I'll I'll show you in the demo how this works. So, if I go back to My, Demo 10 minutes yeah. Thank you. If I go back to My Demo, this is the the stuff that I may just refresh you quickly.

A

Here we go right. This is the version that I bumped up to version 6 earlier, if I scroll a little further down, you can see here there's some additional annotations called Captain, pre-deploy evaluation, pre-deployed task, post, deploy, evaluation and post deploy tasks. So we allow you to specify tasks and evaluations and I want to show you what these are I start with the evaluation.

A

The evaluation is a metric based evaluation, but again using our kept metrics. That means here, I have a captain. Evaluation definition called evaluate dependencies that are linked for the pre-deployment and I, basically say: hey Captain, whenever whoever deploys I first want to make sure that my available CPUs here's an evaluation Target of bigger than four wherever this metric comes from I, don't care because the SRE team has figured this out how to craft the prompt ql. But this is my pre-deployment uh tasks that I want to evaluate, have evaluated for the Post deployment.

A

I have my evaluate deployment. This is where I say after the deployment I want to again validate that I still have enough. Cpus and I also want to validate that. My availability is more than 90..

A

So now I can combine multiple metrics from multiple different tools but I'm using a single, simple standard and everything is done natively in kubernetes. So that's for the evaluation piece and then the task piece there's also Captain tasks that we allow you to write. Basically what it is.

A

We allow you to specify a typescript function that we execute either pre or post deployment, depending on how you define it, where you want to call it, we call this function and use I'm, not sure if you're familiar with the dino runtime, the dino runtime is a lightweight runtime. So we run this code for you in your namespace pre and post deployment, and then we give it some data. Some secure data I'll show you this in a second, what it is and also the context. The context would be.

A

What's the workload name, what's the version, so basically, this function gets executed in the context of of a current thing and then the uh this the the security piece. My example here is sending a select notification uh and the the slack token right that just gets passed in as a security token and that's the security uh piece on top that comes in.

A

So that means, if I did everything correctly, then, if I go back to my deployment, pre-deployment task was notified and post deployment, so the same function was called twice and if I go in here into my slack, Channel I see that at 1 47 PM I'm on Central European Time version number.

A

Six was pretty like the pre-deployment message and the post deployment message, so that came in whether I do it with Argo, with Jenkins, with flux or cubesatel, apply, Captain, sits there and executes my code and then the other thing that should have happened is first of all our goal. If I do the refresh here to make sure that I see really everything, our goal has also visualized this nicely from here. For me, here's my my equipment, workload instance. That's the post, the pre, the post notify pre-notify.

A

It shows me everything that has been executed and, if I look at the at the post notify uh evaluation, we also show you exactly hey the availability. Slo was successful because it had a value of 99.95 and the CPU was also successful because of the value of 8., so I also get the full trace and trail of what actually was evaluated, why it was successful, why it failed or not, and going back to my dashboard I, have all of my deployments. My deployment time version number six took 116 seconds in total.

A

I have the trace for version number six here as well and in that Trace, and this now explains why the trace is rather extensive. Already you can see here. We also have the pre-tasks and the post tasks showing up in the trace as well, because, if you're putting logic into the deployment- and we execute code for you, then you also want to know how long does it take? Does it fail everything fully traced- and you can look at this data here or your open Telemetry most tools understand open, Telemetry right.

A

You can pick your tool of choice to look at this data so to wrap it up.

A

um I I think there's three challenges for us and history today or tomorrow right, depending on how far along you're on the journey, you're building your new platforms for your organizations, the tools ball. They are measurable change, impact on pipelines, gone wild and with Captain. We show you an open source way to chill and to to counter these challenges right and the nice thing is I know. There's a lot of this is the obserability track.

A

In the end, it's all about a making things observable, because you need the right data in case something fails, but you also want to use observability to make automated decisions. That's exactly what this is all about. So we have uh 4 minutes and 43 seconds left and thank you so much and if there's questions I'm happy to take them.