Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: An SLO-Driven Approach to Enhance Kubernetes Cluster Reliability - Qian Ding & Cong Chen

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

An SLO-Driven Approach to Enhance Kubernetes Cluster Reliability - Qian Ding & Cong Chen, Ant Financial

How to define reliability of a Kubernetes cluster? What are the SLOs? How many 9s is enough to ensure end-users are happy for a Kubernetes cluster with thousands of nodes? Service-level-objective (SLO) is the key to run large-scale production cluster reliably. Defining SLOs for classic web services is simple, since web requests are served synchronously with distinct status code. On the contrast, defining SLOs for Kubernetes services is obscured due to its intent-oriented design and declarative APIs. This talk first briefs the philosophy behind the SLO-driven approach for reliability engineering, followed by a deep dive of how SREs define SLOs for one of the world largest Kubernetes cluster in Ant Financial. Finally this talk shares concrete cases and lessons learned of building SLOs framework from several perspectives, including monitoring, alerting and tracing.

https://sched.co/ekCl

A

Hi everyone thanks for joining our talk today. Today we are going to share our thoughts on maintaining kubernetes cluster reliability with a nasal driven approach. My name is chen and I work at ant group as an infrastructure sre. We have another si here on the cloud joining us today welcome tom all right, so our primary interests include running large-scale kubernetes clusters while maintaining high reliability.

A

So today's talk is going to divide it into three different parts. First, I will briefly talk about our motivations and the problems we encountered when we are doing reliability, engineering on kubernetes clusters and then my colleague tone will do a deep dive on slo designs, specifically for each kubernetes components, and then I will share some thoughts about doing snl management. Specifically, I will introduce our slo-based alerting mechanism to help us to monitor our system better, so motivations.

A

Today, clustering management becomes really increasingly challenging for us because our clusters really grow up really significantly.

A

So, for example, at any group we have thousands of nodes in each clusters with millions of parts, and you know like as an sre, our philosophy includes say our head count say: sre have come, should only grow, say, stop linearly or even remain constant, while our corresponding service grew up right. So we have to deal with the exponential growth of kubernetes part, who is only a small team, so fleet management is really challenging, especially when we are doing releases, I think sre we always do releases right.

A

Sometimes we have to roll out new components for master components in kubernetes, say api server and schedulers, and somehow like when you are doing releases, you have to consider consider reliability right. If you release too frequently, then in the early days, you may not know what happened in the cluster actually right, because all of the components interact with each other and it's really complicated.

A

Another story is about doing node level component upgrade say you are going to release a new version of kubelet.

A

You know when you are doing a release for 10 000 node cluster is a totally different story than doing a release for 10 node customer right, for example, let's say your kublet is trying to list all parts and in a 10 node faster.

A

Your api server may feel nothing because the load is really pretty low right, however, for 000 node cluster, if all the cool blades are going to like say list all the parts for in the entire cluster and send the request to api server. At the same time, trust me, your api server will shout out loudly and you may encounter situations like out of memory or even high cpu loads, which makes your cluster unusable.

A

Well, you may argue that, well, the recommended way to do cool blade update is to first reschedule all the parts from the from the node and then do a code update right. However, this is not possible in real world because we may not have enough capacity okay. So now, let's talk about how we are going to do reliability engineering, so the general approach consists of three different stages.

A

So the first stage is the production stage, which represents your service current status and somehow, if your production hit some issue or have some trouble, because maybe you triggered some alerts or some incident happened, then you are going to do going to the design and improvement stage where you sit down with your dev team and your customer to figure out what happens, and maybe you have to do a post-modern review to figure out what to improve. Maybe it's some bad fix, or maybe it's some alert management for some monitoring changes.

A

Then you are going to the change management state in this stage, so you may follow your company's like change management policies. Maybe you can only do new releases during say business days under business hours right or maybe it's about to be thanksgiving days and you have to do a production freeze, no commit or no release can be pushed, except for, like say, emergency practices.

A

So a national based approach is just a specific implementation of the general reliability approach. So we you just use the slo as a lead to go through all the different stages from the production stage to the improvement stage. You use slo based alerting to trigger your incident response.

A

Your sres will focus more on slo alerts and then use them to drive your dev team to do improvements and new designs.

A

So in the design and change management stage transition, you may say redefine your slos if, after let's say a careful review, you believe your asl doesn't represent or doesn't align with your user interest, then you may go through like say a refinement stage to redesign your slo and then maybe some align with some code changes go into the change management stage. In this stage you will use slo, documentations and agreements.

A

So let's say you daddy agree on like say if the slo is broken, I'm not going to push new releases until like say slo is back to the normal level.

A

So this is the general slo-based approach and the philosophy under it, and then I will hand over to my colleague to do a more device on my server designs for kubernetes itself.

B

Hi everyone, I'm tong, and I'm going to talk about seo design for kubernetes. First, let's do a quick recap on the definition of sli seo and soa. Soi is a mattress that you use to measure your service or health needs, for example, for http server it can be requested error ratio or the average request. The latest sao is the quantitative target that you and your team try to achieve.

B

For example, one can target the 99 percenter success ratio for http requests that you must and finally, seoa is the promise you make to your user like what would the user expect to receive when you broke the slo, for example, if somebody paid to use your http service and you failed to deliver that 99 seo target, you may compensate to your customer with cash sometimes internally. The word sao and soa are interchangeable, so back to our day-to-day job as a infrasre.

B

What we care about kubernetes itself, as chairman mentioned before at the anti-group, we run dozens of clusters globally and we have internal user to using kubernetes for different purposes, for example, provision pavement and transaction system database, as well as running batch job like machine learning when we design our sao the first put is user. First, that means sao should represent the real user, behavior and experience each user may have different privacy when using kubernetes.

B

However, we can summarize them in four categories. The first and the most important sao is about the resource delivery. Resource here often means port or computing resources in a shape of deployment or staff set. We care about delivery, speed and success rate by using asking to create a deployment in our clusters related to this seo. We believe we need to manage port lifecycle precisely this seo targeted on a single port, but during it is a creation, upgrade deletion behavior.

B

We use this seo to figure out our system overall health status and the third sao we believe we should care is about whether all existing port in the cluster are running as expected.

B

If the class is having a network issue and the port are disconnected with other clusters, we may say the port entry or unhealthy state, then we can somehow summa sum up all the health time of the all posts to calculate the class level product uptime.

B

Finally, we know user offers interaction with api server directory for query or other behavior. So we care about api server. Success ratio analytics.

B

Let's take a close look in into the one of the n2 and seo. We would like to come up with a magical to represent the end-to-end port creation. Behavior and the user are happy to see a promise like 99 percent ports should create it successfully to reach a ready state within five minutes in a month.

B

As you can see, the port creation workflow in kubernetes is real conflicts in real production cooperation. Cluster you request. The will pass. Wireless customized web hook may be your secondary, the scheduler to reach conflict. Sometimes you may have multiple user defined controller to register your port to different external dependents.

B

Just like cmdbe network or metadata center was a long journey for a port to create created.

B

Our solution for measuring portal creation seo is to collection, virus support, related audit log and event put them into a centralized log, energetic system which product name as the unicode index.

B

Then the local enterprise system can identify the latency during each product's status transaction, for example. This spends three seconds during webhook admission, 10 seconds during the scheduling and the 13 seconds in imaging pulling also the centralized local enterprise system can identify if the port is in the desired state.

B

Science users may have different requirements for creation, latency some may tolerant to 10 minutes, but some may only tolerance, three minutes. Finally, we produce time series measurements to record the end-to-end seo and let our monitor system to scrap the mattress and produce meaningful sao based alert.

B

So we will talk about a lot later after implementing several end-to-end seo. We are facing new issues, although we can leverage port events for low end science and individual tracing, it is less obvious to identify the case for some component issues and affiliates will sre receive alert about seo. Filliams often, we need to check different component dashboards to see if api server is retaining better response or a customer or customer controller is not working correctly to patch a required label for the port to run. The entire process is like top down approach.

B

You first know something bad happened, and then you start to drill down on each individual component to find the platform. The good thing is about end-to-end sao it is less noise once your matrices is well designed. However, the rule case search may take a longer time and make heavier relay on past experience.

B

To take a step back, let's revisit the flowchart of the product creation. Although we have tens component in this graph, we actually can modern them in two types, namely synchronize the components and asynchronize the components for synchronized components like api server, htc, retreating and the validating web hook. Their interaction model is more stress forward, get requested in process internally and send the response out for asymmetric component, just like scheduler controller or customer operators.

B

Their interaction model are more likely classical leads than the watch pattern, an intern intended driver. They need to watch resource status, chat and then into the reconcile queue and do the we can sell nation repeatedly.

B

So to design seo for synchronous components, we believe we need to pay attention to request the error rate latency and also the component uptime according to the google sre workbook, the other two, the other two golden signal are situation and the traffic, but we believe that these two signals are less important and we can have another way to include them in seo. Design, for example, trafficker actually can be majoried from the client set.

B

Almost every component talks to api server, and each of them has metrics to indicating the success rate to talk to api server, the client-side monetary agreement on the request, the error ratio can somehow reflect the situation and traffic conditions to design the asynchronous component seo.

B

Similarly, we care about the reconcile error, ratio and latency, of course, component uptime. Besides, most component runs as a leader follower pattern, so we can define sao for leader selection time and no leader time.

B

The reconcile query system is another interesting asset accepted of asynchronous components, so we can design sao for the cue depths and all these non-digital items to help us mature component healthness. Okay. Finally, to summarize our overall seo design is a combination to top-down approach and the bottom-up approach. We first separate the seos into two layers.

B

The upper layer is about end to end sao, which we believe both user and sre care, including the resource, deliver port lifecycle, api server, interaction, existing port health needs, the bottom layer focuses more on individual components and we believe sre and dev scales more on this layer.

B

Each individual component seo helps us identify bottlenecks and the limiter in our kubernetes cluster and the word, and then we can take concrete action to improve each of them.

B

Okay, next, I will hand back to chen to talk about sao management. Thank you.

A

All right, thanks tom, for sharing now I'll talk about how we do srm management specifically I'll use an example of doing slo-based, alerting.

A

Suppose we have an slow say. 99.9 percent of the requests to api server are successful every month. So this means a successful request will like say, return http status code less than 500. This is kind of simple and straightforward right.

A

So as an sre, you can actually set up very simple alerts based on the ratio rate, as I listed here in this following prometheus code snippet, you first define two recording rules, one to sum up all the sli error count in a minute and the other recording rules to sum up, or the total counting a minute right. So now we have the per minute like error, account and total count.

A

We simply do a division and compare it with 0.1 and if the error ratio is greater than 0.1 percent, we alert and if you have set up this type of alert in the past, you may already find out that this is very problematic, because you will receive a lot of noise, as I demonstrate here in this hypothetical error ratio graph.

A

So if you use the ratio rate under setup, then every spec here which about this horizontal line, will trigger the alert and it will cost you and the sri team a bunch of operational hours to deal with the noise and maybe to deal with the real outage.

A

And then you may think about say: how can we reduce the noise right? A very straightforward thinking is to add a holding period. Let's say: if this error ratio continues for five minutes, then we trigger the alert.

A

However, this is also problematic, because I can easily bypass this type of alert but still broke my slo, because you can think about this situation. Let's say in the first three minutes: your aps: server receives like 100 narrow rates and then in the next two minutes, because there is no traffic, let's say: there's no traffic, and so the error rate is like, say, 0 and then the traffic comes in again and it becomes 100 error and then 2 minutes, there's no traffic.

A

This repeated pattern will cost you sixty percent, every weight or, let's say a hundred percent. So again, this type of fail to capture this type of like outage. So what can we do? So, let's reconsider our learning philosophy. We know that our solo target should never be 100. Otherwise it's meaningless.

A

That means like the for the total requesting amounts. We can divided them into two parts. We have an expected portion that we want to serve reliably right. It can be 99.9 of the total requests and we always have like say 0.1 error, portions and we think we can manage the risk of it right.

A

So these risks can sometimes come from your dependency failures or you can encounter unexpected network hiccups right. So this red portion. We call it the error budget that we can tolerate.

A

Then we can come up with our learning strategy. So first we set up our slow targets, saying 99.9 the request to aps server are successful every month and then we implement our sli metrics to monitor ourselves right. So typically, we have two metrics the error metric and the total metric, and then we can figure out our maximum tolerable failed requesting amounts. That is our error budget, and then we only alert if a predefined portion of the error budget are consumed in a short time window.

A

So to demonstrate this philosophy, I set up two levels of hourly say. The first level is the page level. That means sres needs to come up with the solution really quickly, because it's really an emergency issue, so the page alert says number of sli failures in an hour if it's greater than three percent of the entire error budget of the month. Zero budget calculation is a very straightforward, simple map, one minus 0.99 times the number of total requests in a month. Right.

A

If that happens, then we alert so that we need to react on this failures and we have another level say the ticket level. This is to deal with those somehow mild situation, says slos sri failures in a day say it's greater than 5 of the entire error budget.

A

So why we choose three percent and five percent here, so three percent in an hour, it means like say in 34 hours we will burn up all of our budget, so we will miss our sro target for the month and for five percent portion in a day. It means in 20 days. If we do nothing, that means cslo will be broken so to implement these alerts in practice. Let's say we use prometheus as an example in this code snippet.

A

Let's say I want to implement the page level alert so other than the per minute sr error count and sli total count which we used in the racial rate based alert. We need to come up with two more recording rules, one to sum up, all the error counts in the past hour, so we some of the time for the permanent error count.

A

This is also very straightforward right and we can come up with another recording rules to get the monthly total count. This can help us calculate the error budget very straightforward.

A

um So then we can come up with the final alert. The left-hand side is apparently the hourly sli error count and the right-hand side either. Three percent of the error budget.

A

Anyway, due to the time limit, I can only share this much about slow, alerting so to do a quick recap on today's session.

A

We basically talk about three different things: first, how we define our fine-grained slos for kubernetes clusters right, we do component level and we do like the user level, so users may care more about n2n slos and we care more about on component level, and we know how to use slo-based metrics to do our monitoring and alerting, because they are less noisy and can be very effective.

A

I don't have enough time to go through how we do slo documentations and how we do discussions with our partners and their teams to reach agreements, but we can discuss them later well in the future, since our kubernetes clusters continues to grow and we need to continuously review our slos to align with our user interests and maybe we will do like say monthly or quarterly slo reports for like the entire fleet.

A

Anyway, we already started another very interesting project on automation. We want to automate the recording process based on our slo-based alerts and other slo-based metrics. Anyway, thanks for listening and now we can take some questions.