Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: API Priority and Fairness: Kube-APIServer Flow-control Protection - Min Jin, Ant Fin

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

API Priority and Fairness: Kube-APIServer Flow-control Protection - Min Jin, Ant Fin

Currently the API Server has no concept of priority or fairness for requests. This means that a buggy webhook or bad actor can potentially DOS an API Server by sending lots of requests. This also means that currently the API Server cannot prioritize more important traffic when subjected to high load. This session will introduce the architecture of the APIPriorityAndFairnesss feature for Kubernetes which aims to solve these issues. Using the feature, administrators can categorize API Server requests into various priority levels and flows within each priority level, allocating concurrency to each priority level and getting fairness between the flows of a priority level. This talk is targeted at Kubernetes developers and SREs who are interested in the new features to help prevent API Server downtime as well as those wanting API Server request limits for multi-tenancy.

https://sched.co/f6S0

A

Hi everybody welcome to my presentation at kubiconcloud.com, north america, 2020 virtual. The topic here today is api. Priority and fairness could be ips overflow control protection before we are getting started. I need to say that, because this presentation is held online virtually so I enrich the contents of this powerpoint service so that you can check out the details after it's finished in the topic is about a new feature.

A

Gate introduced into kubernetes 118 release named api priority and fairness, and earlier this year in april, there is a official blog on the kubernetes sites named api priority and furnace alpha, which basically tells you what's happening in the alpha releases and if you don't got any backgrounds or any other informations about this feature, I highly recommend you to with this vlog. It's only gonna. Take you like five minutes so for for folks who are interested in the design or implementation details.

A

There is also a cap under the kubernetes cvp machinery folder, it's gonna, take you have a deeper insight of this feature um and the things you might be interested um when in terms of further customizing for your cluster, um then I'm gonna have a brief introduction about myself. I mean I'm working at um ounce group as a software engineer, and I've been working as the api machinery for about three years and I'm also a kubernetes c api machinery. Subject: owners covering several subjects, my github analysis, ue 994488q, the same name is also my gmail account.

A

If you got any questions after this presentation, please feel free to contact me by either the email or the names on the kubernetes slack channels, so um the the team, the squad building this feature covers many different developers from different countries in different um companies and my uh so there's mike from ibm daniel from google, david from red hats, and there are also other contributors so far, uh aaron johnson bruce lee monge. Thank you for all your contributions. So far, then, let's take a look at the agenda of our presentation today.

A

um We're gonna show you the background and motivation of this feature. Then there's going to be a visit, virtual inspection of system, design and and next I will also show you the r for the offer station implementation, uh which basically covers the api models in uh a and and the systems the implementation sources.

A

And then we move on to the demo in which I will show you how to customize flow control settings for your own clusters.

A

Mostly, we are using a kmd cluster which is just created by me um in the virtual, um um not not in a virtual website, it's um just a new brand new cluster so that you can easily reproduce that you can easily rerun that in your local environment and at last I will show you there's a few planned enhancements for the data stage.

A

Okay, let's move on to the first part the background and motivation. Basically, there are two higher goals of our future. The first one is the self-protection um which basically covers two points. The first is prioritize cluster critical requests for self-maintenance and the second one is prevent spamming clients or buggy controllers from stunning. The whole cluster.

A

The the first one is basically saying that we are going to sort all the client requests into different priorities and the second one is saying that we are not. um We are not allowing any client request to spoil all the clusters. There will be a vanilla isolation between clients.

A

um As for protection, we are actually protecting cool bfa servers from incoming client requests, so we should start by understanding. There are different kinds of client requests being served by the typical kubernetes servers. There are ips server, loopbacks delegated requests from a aggregated api server or mission reports.

A

There are also controller requests and there are email requests um as for kubernetes server loopbacks, so the coolback server instance will be requesting against itself, even if there are no client requests at all, such as um there's a informal factory instance singleton in each of the google play server process.

A

uh Basically, the cool web server need to know the actual status of each clusters by accessing the object, cache provided by the informal factory. All of these informers will keep raising a list and watch requests against the cool web server until it's down, and there are also several embedded controllers inside equipment server. uh For example, there's cluster ca rotators and there are crd related controllers um and there are also api server application related controllers.

A

All of these controllers should be regarded as first-class citizens in the kubernetes world because they are strongly connected with the house status of whole cluster. If you are failing the cluster loopbacks, then there's gonna be bigger problems in your clusters.

A

um The your clusters is very, very likely to be crashing down and as for the delegated request from aggregate api server and mission notebooks, so basically the google server provided us some extensibilities to allows us to add new results to the cluster and also intercept some of the resources.

A

These extensions will be invoked during the time when the curriculum server serving a incoming request. For example, if you add a mission web box on the powder resource, then when you are start starting a request against kubernetes server on portal resource, then the kubernetes server will raise another request against the emission reports. um The secondary request should be should have a higher priority than the original request. Otherwise there is going to be a deadlock in the request chain, which is going to cause some problems in the clusters, and there are also controller problems.

A

uh We had a situation where a bug in the deployment controller caused it to run a mac under certain certain circumstances, usually issuing requests in the types loop, so we'd, like these um controller parts, not to take the whole system down. um These controllers can also be some custom controllers, which is developed by elaborating some scaffolding tooling, for example, computers playing with the custom resource definitions, and if, if these buggy controllers can have these new behaviors, um it's going to be a harm to recruitment server as well.

A

So the there can be controller singletons and there can be also daemons. It can be a couplet proxy or other per node controllers. um If there are bugs in these per node controllers, the impact the influences on the system is gonna, be multiplied from the uh controller singleton, so we definitely want to avoid them. um On the other hand, these uh the issues from daemons doesn't necessarily connect it with a bug. It can be the cluster reaching its scalability limits.

A

For example, if you have too many nodes in the cluster, then your cluster will be reaching the scalability settings, but we don't. We don't know the ceilings until we actually reach it. So, um but at least at least we don't want the newly added node to be um taking the whole system down.

A

um There should be a way to make the cluster keep running, even if there's too many demands added to the passer there's another higher proposed multi-tenancy, which demands us to provide guaranteed capacity for controllers that are considered less important and the tenants in the same priority band sharing the cluster should get an equal share of their service.

A

um So I believe that kubernetes is designed to be shared by multiple tenants. There are many different kinds of tenants definitions so far uh a tenant can be a namespace, can be a user can be several users sharing a prefix or several users having the same host in the same group, and don't forget that there's another subproject under kubernetes 6 named multi-tenancy, which has a brand new definition of tenants, which is basically a group of name spaces.

A

There are also non-goals of our future background. There will be no coordination between the api server. Now is a external load balancer and will be also not to attempt auto tuning the capacity we will not attempt to reproduce the functionality of the existing event we're limiting emission plugin.

A

The mission plugin is basically intercepting the request against event resource by using a token bucket filter. So this one is basically saying that we are not generalizing the token bucket filters to all the resources or something like that.

A

Let's move on to the second part: the system design retrospective, so uh we need to know their um basics about flow counter. Algorithms. um There are basically two kinds of flow control: algorithms, the first one is that the source or the client side, and the second one is that the gateway or server services.

A

um This client-side re-limiting is already supported because uh could observable about the the kubernetes triangle provides a token bucket re-limiter for muzzling the clients, and there was even a dedicated emission controllers for limiting the re-limiting, the events, but there are still a few known defects. The first one is that user can opt out from the limiting by granting the token buckets, minus or infinite capacity, and also it's um sometimes it's tough to control the granularity if there are multiple controller, multiple clients uh in building in the same components.

A

In the same core processes, for example, we want a higher um a loose, a loose control, limiting for one controller and tighter for another. Then it's going to be a bit tough to configure.

A

um Then, let's take a look at the existing limiters in the kubrick servers. um There are two dimensions to to do the limits. The first one is by configuring max mutating requesting flights max request in flight flex, which allows you to set the limits of required. Concurrency request count on either mutating requests or non-mutating requests, and you can also apply a timeout for non-wrong running requests, which is basically the non-watch requests.

A

Then, let's move on uh learn from the linux qt's systems, because it's going to be really helpful for us to understand how the um the linux networking system really limiting systems which is already proven to be successful. um There are basically two kinds of qdisk in linux: networking the first one says classless and class 4 there are. There are a few outstanding algorithms from robbing codel tbf under the classless q disk and actually there there's a longer list of it, but we're not showing every algorithms just a few uh outstanding algorithms.

A

um Anyway, the in the class list- qdisk every request will be regarded equally, um but in the kubernetes api server um there's already a system of authentication authorization, so there is already a abstract of user identities.

A

um So we need the task force in building the google server there there's a few um class for queue, this algorithms building in the linux networking systems, um there's deficits, wrong, grouping and hierarchical token buckets to mention one thing to mention is that we are learning a lot from the both algorithms.

A

I remember we spent like several meetings discussing um the differences.

A

From that deficit from robin uh and there's a few other algorithms, especially the hierarchical token, buckets it's a bit complicated, but um it's it's proven to be working well for most of the use cases.

A

So um here's a obstruction of a flow control system class for flow control system. We want to build in the google web server they're, basically gonna be three abstract components. The first one is the rule-based classifier, then a queue assigner, then a queue scheduler. The three components is basically working like a lambda functions, classify a maps request through a request class. A keyword signer puts a request in a request class into into certain queues.

A

Then secure scanner applies delays to the queues where the um where the request is waiting, um then let's move on to the to what we can do in the classifier. um We extend the obstruction of class from linux, tc systems to priority levels in the cloud web server in the new flow control system.

A

A priority level is a priority level band that request in higher priorities should have a higher should be executed in prior to lower priorities, and a priority level is a request class in which all of these matching requests will be handled equally and a parameter level is also a pro a request class where we applies the same rejection strategy.

A

So to classify the request into proper priority levels, the informations we can get from the crew backup server request. Contacts is um for the first one is client side identity. uh For now there are two kinds of useful identities: the first one is the user or username. Then the user groups like a list of tags on the users.

A

Then there are requesting targets um which basically covers the requesting name, spaces and other request metadata from example, the works, then the targeted resource types etc.

A

um So so we know that what we are doing in the keyword designer is basically uh mapping a request to a priority level using the identity of this.

A

Then, let's move on to the keyword designer, um so each priority level contains a group of request cues for scheduling uh the pro. The question is: how to map a request to one of the cubes um one way one easiest way or intuitive way is to map one cube per user. um If you have um 10 users, then you're going to have 10 qs in the for one priority levels, but there are going to be problems. If you have tens of thousands of users there.

A

If you have tens of thousand skills in your product level, then there's gonna be a uh significant memory cost um to avoid that we use another technique named the shuffle shading. um The shuffle shading helped us to bounce the memories into um into a fixed number, a constant number uh in the bar. In this picture I showed you a shuffle shouting of hand, size, q and basically, the users from user. One, for example, is started in a round round-robin manner to both q1 and q2. So that's the other users.

A

um The this picture also showed you that when the uh there's some problem with the user user 1, then the impacts will be spread to q1 and q2, so the user 3 number 3 will not be affected at all, and but user, two and user four will be partially impacted.

A

um The higher a higher number of uh handsets you have, the um the impact will be shouted lesser and as for the q scheduler, um we we have. We are we're using a algorithm named fair queueing, which is basically aiming at achieving the following goals, or when s4 scanning requests form kills. The first one is the even distribution of service capacity, and the other is the maximum fairness.

A

There are a few I got a few uh details about the fair killing, but we are not expanding them today because of because the time is limited. um So uh if you got interested in the fair curing algorithms, you can uh revisit this slides afterwards.

A

um So we had a variant of fair querying for server requests to solve these um limitations for google server. The first one is dispatching requests to be served rather than packages to be transmitted. Second, one is multiple. Requests can be served at once. The third limitation is actual service. Time is not known until a request is done being served, um so we made a few mutations variants to the fair queuing algorithm so that we can adapt to the code before server.

A

um I'm still not going to expand it to save our time. So this is um the our flow control system. Gonna be located. uh There's a priority level. Classifier and shuffle sharding can be fair qe.

A

Then, let's take a look at the alpha level. uh Api definition models. uh You can acquire the new feature by enabling the feature gates and add a new flags to the qrp servers starting flex. um This is the exam flow schema where you can see. We we uh it matches. uh uh Then, when a user in the group sees the masters uh accessing everything in the cluster, then it will be matching the exam flow schema.

A

There is also a catch all flow schema. One thing different is that it has a distinguisher method if you've got interested in distinguishable method- and you can, you can take a take- a look at the caps to know how what what what this is doing and uh in the priority level configurations.

A

uh You have places to configure the hand sizes for shuffle sharding, which we just talked about, and then I'm gonna have a demo of customizing. This.

A

So I'm using a cluster, so um you can. You can create a new kmd cluster using these settings and here we got a.

A

Gravity level named workload medium, we are creating this to the cluster and we create a new uh flow schema.

A

Which matches and then the users from demo user.

A

Then we can take a look at this graphene dashboard in this panel. uh It shows you the capacity limit of this priority level and the qps of each proto level how the requests of each protein level are killed. I've been waiting in the actual executing time cost of each party level, or all of this um pans are calculated by using pp50 p90 p99.

A

um In this example, the um medium the workload medium current level configurations only has a concurrency shares of one. uh Now we give it a higher concurrency share, for example 15..

A

Then we apply. Then we apply this to the cluster.

A

um Then you can see the concurrency shares are recalculated, um the uh the medium, the medium workload product level. Configuration now gets a capacity limit of 120.

A

um Yeah and um because the time is limited, so I I can't show you how everything works. um uh If you, if you've got questions, I can give you a detailed demo after or offline.

A

Let's, let's give on move on to the presentation.

A

uh Then it's the planned enhancements for better stage.

A

There are a few. There are blocking items and non-blocking items planned for better stage. As for blocking items.

A

um um It's um improving the observability and robustness. um We added a debug uh we're adding a debug endpoints to the api server, which is already done. I believe it's already published in kubernetes 119 release and.

A

There's also a few new metrics added to the system, which is definitely helpful.

A

But I'm not showing them to you in this presentation, because it's uh basically used for debugging and if you've got interested. I can answer that in the q a section and um the second one is providing approaches to opt-out client-side we're limiting, uh I'm not sure if it's um done so far, but um the ultimate goal is to remove the client side, we're limiting um to move them onto the server side or at the gateway, and then the third one is. We are doing necessary, e2e tests. um This is uh our current progress.

A

We are adding e2e tests, um hopefully before 120 release.

A

And as for non-blocking items, here's a few optional goals, the first one is supports. Concurrency limits on loan loan running request and the second one is allow constant, concurrency and relative shares in priority level api model. But for now it's only a proportion of api shares. Api concurrency shares we hope to allow users to configure a fixed, uh concurrency or preserved concurrency for a priority level.

A

um The third one is automatic, automatically manages versions of mandatory and suggested configurations. um This one is basically saying that we are providing a better user experience um for users to roll out to update upgrade system. Presets flow control, settings um flexibly and the first one is discriminate. Paginated list request. This is um what we are trying to achieve before one twenty release, because um I'm paginated release request is uh very likely to harm the cluster and and it's almost happening every day in our production classes. So uh we hope to prevent this from happening.

A

um So this is what we are going to do: non-blocking items for bigger stage uh for more uh graduating criterias for better stage. You can also read them from the cap and thank you for um visiting my presentation. Then. Let's move on to the q a session.

A