Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro + Deep Dive: SIG Scalability - Marcel Zięba, Google

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Intro + Deep Dive: SIG Scalability - Marcel Zięba, Google

Speakers: Marcel Zięba
This session will focus on the different efforts that SIG Scalability is involved in: defining what scalability means for Kubernetes, driving performance improvements, maintaining infrastructure for scalability testing, guarding Kubernetes against performance regressions. Time for Q&A will be reserved at the end of the session to understand how the SIG can better engage with the community as well as to allow the audience to provide the input about the roadmap.

A

um My name is Marcel genba and I'm software engineer at Google and I'm also one of the two chairs of sex scalability and today, we'll be talking about sex scalability introduction that also Deep dive. So first question that we need to answer is what we do as a sex. Scalability. Six scalability is a bit different from other six, because we do not own. We do not own production code. We only own test code that we use for testing the scalability of kubernetes.

A

So basically, we do have five areas: five areas that we are interested in in terms of scalability, so, first of all, the most important is actually defining what the scalability is and once we know what the scalability means, which will I, which I will explain later, and we also want to set up some goals, so you can think of it.

A

As you know, we could start improving the kubernetes by making some improvements, but it doesn't matter if the end user is not able to see those improvements and actually have some Better scalability Properties, except for defining and driving those goals. We also contribute to kubernetes in a way that we want to make kubernetes more scalable. So what we do is we usually coordinate between six or or ask other states to actually make some improvements in order to make kubernetes more scalable.

A

But okay, let's say that we we do have those goals in mind for kubernetes that we want to achieve, but first of all, we need to measure and monitor how the performance and scalability of kubernetes actually changes over time.

A

um You probably know that every day there are multiple PR's merged into kubernetes and each of these PRS features. These changes can potentially impact the performance of the whole kubernetes. So that's why one of the most important areas of six scalability is actually monitoring the performance of kubernetes to make sure that we do not regress, which is actually the next next Point here.

A

So basically, once we have this monitoring, then we are able to track different metrics and see if there is any regression and and last but not least, we also consult and Coach other six.

A

So probably most of you know that there are caps which cap is kind of like design Dock and when you are when you want to implement new feature, there are some sections that are specific to six scalability and with questions like how does your feature impact um control plane or what kind of calls is making your uh making your feature that are additional to the previous state, and so one more important part is not to confuse wait. A second, oh, not to confuse six scalability with Sig Auto skating. These are two different six.

A

So what is the kubernetes scalability? So sometimes we ask our users okay, what do they want and basically they say, wait? Well, they want scalable clusters, but if we ask them like okay, but what does it mean? Well, they usually don't really know, because scalability is not just a single number. So I can give you one example where in 2015, the kubernetes 1.0 was actually supporting 100 nodes, and then this number changed over a few releases to 1000 nodes.

A

Two thousand five thousand nodes and 5000 nodes was supported in 2017 in kubernetes 1.6, and this number did not change.

A

This number did not change since 2017., so what I want to say is that the scalability of kubernetes is not just the number of nodes. It's much more than that.

A

So now I I just want to introduce this concept of scalability envelope. So if you think about kubernetes and the scalability, what you want to actually think of is multi-dimensional problem. So there are many, many more Dimensions that you want to take into account when thinking about the scalability of kubernetes. So just to give you a few examples like number of nodes is just one of the dimensions that we are interested in, but then you'd have like number of Secrets or like how many pods you can have per node Orchard, for example,.

A

And now take into account all those Dimensions um we have scalability envelope, so the scalability envelope is safe zone. So, if you are within it, your cluster is happy, but we still don't know. What does it mean that the cluster is happy, so you can think of it as like. If you are within those limits, then something happens and now I will try to explain what. So, what does the happiness of the cluster mean.

A

So, in order to think about cluster being happy, we need to introduce two concepts: one is SLI, which is service level indicator and SLO, which is service level objective, so in simple words, I think about SLI as a metric. So let's say, I have metric that says: pod startup latency is X second, and then on top of this metric, what I can do is put some kind of threshold.

A

So, for example, I want to have put startup latency, like 99 percentile, of put startup latency to be below specific threshold, so, for example, in kubernetes it would be five seconds.

A

um So so, basically, the cluster is happy when all the scalability slos are satisfied.

A

um So I I just want to give you a few examples of scalability slos and this this list of scalability slos is actually uh changing over time and we started with a very two simple slos a few years ago, which was API, call latency and put startup latency.

A

You can find all those slos and the description on our six scalability page, but I wanted to give you some kind of idea how well those slots are defined, because just saying that we care about the core latency is not enough and there is actually way way more uh definition to that, and so, let's start with the SLI SLI for the API call latency.

A

So the slide for API call latency for mutating requests only is defined as you take five minutes window and you choose one Resource One verb, so let's say deployments and patch and what you want to do. Is you measure 99th percentile within this window, um so you can think of it as like for each resource for each verb. You will have different metric and then what we do.

A

We say that the SLR is kind of aggregated for each specific resource and verb for the whole day. So in the whole day, you have 200 like 88 Windows of five minutes, and then only only two can like not met this. This kind of threshold, which is one second, so we want to have all the mutating API calls to be within one second, and if you have like 288 Windows of five minutes, then, basically, basically, you can only have two windows that are not meeting this one second threshold.

A

So this is how the uh how the definition of the API equal latency SLO looks like. So that's just one of the things that I showed before and once we have that, and we have this scalability envelope.

A

um What we can do is actually test some limits, because um Computing the whole scalability envelope is impossible. As you can see, there are like multiple dimensions, and you could think of it, as you know, like I could have zero pods, but like one million namespaces, but is it useful? It's it's not really useful for for the user. So what we want to do is kind of model it, how our users use kubernetes and let's say that.

A

Okay, we are interested in some limits, like number of nodes to be up to 5000 number of PODS, like 30 pots per node number of services to 10 000 and as a sixth probability, what we do. We actually run those tests that maximize all those limits and we check for all the all the slos and make sure that they are met and you can find all those thresholds that I mentioned like here.

A

We have three examples, but you can also find multiple other thresholds or limits like for pod, churn or number of Secrets and and so on.

A

So now we will move toward towards um actually our infrastructure like what kind of uh tools we use for, making sure that the scalability envelope and all those slos are are satisfied in kubernetes. So the most important tool that we use is cluster loader too so cluster Loader, 2 2 is actually our our internal um six collability tool used for for testing testing scalability of kubernetes. So what you can think of it is that you provide some kind of states that you want your cluster to be in.

A

So let's say that you are starting with empty cluster and then you want to make sure that the cluster is working when you have 1000 deployments and 100 000 parts, for example. So you are specifying this state and then let's say you are also specifying. How do you want to transition through from empty state of the cluster to to this desired State? And so you let's say you are saying that you want to create one deployment per one.

A

Second, for example, and cluster holder is actually doing that so cluster loader is creating those those deployments as you as you wish, and along the way. The cluster loader is also measuring all those slis and slos to make sure that you know, for example, put startup latency did not exceed five seconds during the whole test, and there is also a bunch of a bunch of other um other other features that you can use for, for example, for debugging, but I will mention a few a bit later.

A

Also, you can imagine that if you want to test 5000 nodes using real clusters is pretty expensive.

A

Even if it's like you know one CPU per per node, then it's still 5000 CPUs so running those tests, it's it's it's pretty expensive and as a six capability, we cannot run them like super often so 5000 nodes tests are on only one once a day, but we also do have Cube Mark and Cube Mark. It's something that we call as a cluster simulation.

A

um So basically, the idea is that we have cubemark master and what we are trying to do is actually test this, this master and we don't want to use 5000 nodes. So what do we do? Is we actually create? Let's say 80 or 100 nodes and on those nodes? What we do is we actually run something that simulates Hollow nodes. So what we do is we Run Hollow nodes and those whole nodes, simulate the traffic that the normal control plane would would see and with that Improvement.

A

uh Basically, instead of having 5000 nodes, we can usually do with like 80 nodes and each let's say four or eight CPUs, which brings down the cost significantly down. But of course the issue is here like Okay. So let's say that we want to have this kind of setup where we have three actual nodes and here 15 whole notes. So how do you actually run those Hollow nodes? So this whole notes, this whole notes actually need to be somehow scheduled onto the real nodes.

A

So this is actually pretty funny, because we use also kubernetes to to do that. So there are actually two clusters. One cluster that is responsible for uh for running those Hollow nodes on actual nodes and those whole nodes are connected to the separate master that we want to scale test and because of that Improvement we are able to actually run scalability tests much more often because it does not consume as much resources as like regular clusters,.

A

Except for that, we have a bunch of monitoring and observability tools, and perfdash is one of our favorite tools. It looks super simple, it's super simple, but it allows us to find multiple regressions um because you know like I mentioned before that um you know we have slos. We have some tests, but there are sometimes regressions that are not found by just as a loss. Let's say that the pot startup latency.

A

Actually this is the startup latency example, and there was some regression some time ago, and we only saw that the number of that the post startup latency increased by like 400 milliseconds and once we found it, we fixed it.

A

And basically here you can see that we bring brought down the latency of pod startup latency by 400 milliseconds, but our tests were not able to actually find it because the SLO says that you know the 99th percentile needs to stay below five seconds, which was still still true, but we still saw the regression due to the curve, Dash and and all those cool charts here.

A

Except for that, if you want to also see um one particular run of our scalability test, then you can use our grafana dashboards and they are super useful and you can just use them for for debugging scalability issues and also like when you're running cluster loader cluster Loader 2 out of box. You get Much, More, Much, More observability, also in terms of profiling.

A

So when running the test, we are also Gathering memory and CPU profiling, and so for example, then later you can see that okay during the load test, there were some parts where the control plane, CPU was burning quite heavily and you can just grab the profiling and see what kind of parts of the kubernetes control plane was actually consuming. The CPU.

A

So now we move to the scalability test, so I showed you all those tools that we use for monitoring and testing the scalability of kubernetes, but now what kind of tests we are actually running to to ensure that the kubernetes is scalable?

A

So there are two types of periodic tests that we run: one are release blocking tests and restocking tests that are actually performance tests for 100 nodes, 5000 nodes, and these are actually run on real clusters and and also correctness, 5000 notes. So the performance is purely about the performance like how fast the pods, startup latency is or like what kind of latencies do we have on the control, plane and correctness is more about that.

A

The features that kubernetes is providing still work when you have like 5000 nodes in your cluster, except for that we we do have non-release blocking tests and they are super informative.

A

So one of the examples- one of my favorite examples- is actually a benchmarking for going so in the past over last I think three or four years, we've noticed that there were multiple regressions coming from from the golang, and so what we did was actually we froze all the dependencies except for the Golan compiler and what we do is we just have Frozen kubernetes version and we run our cluster loader performance test and we just change the go compiler version and based on that, we noticed if there are some regressions in going compiler, and we just inform the golang team that okay, we noticed that there is regression between like, for example, those two comets and they are able to easily detect those regressions as well, because kubernetes is actually one of the actually the biggest project I think in in going.

A

So it's super useful for them as well to validate if they are not introducing any regressions. But except for that, we have also the the cube Mark, some some bunch of storage tests and so on.

A

There is one more thing I wanted to mention, so if you are contributing to kubernetes, then whenever you open PR, there is a bunch of pre-submits and one of the pre subnets is actually uh the performance uh 100 nodes test test. So whenever you open PR, there's actually job running that creates 100 nodes cluster and it's running the load test to check. If there are no any obvious regressions in performance and that's basically it in terms of scalability tests, how we protect the scalability of kubernetes.

A

So I mentioned those CI tests and you can see the our test grid.

A

What's happening basically is like we have those tests running for for the master branch and also we do have for all releases that will be happening, for example, 124 or 125. The releases are still happening, so we are still running those tests and and this test grid is actually available to everyone. If you want to see you can just go to to our test grid and see in what state the current kubernetes scalability is.

A

So I mentioned few few of the standard regressions that we already saw, but scalability is super sensitive One Small Change can break the whole scalability of kubernetes and we've seen regressions pretty much from everywhere, like going I mentioned my favorite, but then also operating system or controllers, API Machinery scheduler at CD cublet, you name it it's it's basically all.

A

um So. What what's happening is that, usually, once we observe that there is some some regression, we kind of try to narrow down where this regression happened and usually contact out or or even revert the whole feature. If, if we think that um the scalability is not as good as it was before,.

A

So there are a few actually um pretty interesting regressions that we solved quite recently um to name it, to name one example.

A

So, for example, there was um recently introduced recently priority and fairness, and probably you heard about it- and um some of the calls in priority and fairness were incorrectly estimated and because they were incorrectly estimated, they were consuming way more. uh We call it seats and what was happening was that in some of the setups that our customers use, the API call latency was significantly higher than it's supposed to be. So that's something that, for example, we recently detected and fixed um last few versions.

A

And there is the also interesting part where we actually drive some improvements um so again like going back to going. uh It's not like going was only introducing uh regressions in kubernetes migration to going one, eight one eighteen uh helped help kubernetes performance quite significantly, but maybe you also heard that the memory footprint also increased significantly. So this is one of the examples where actually the Improvement was was quite huge. uh 99 percentile of API call latencies was basically like 10 time smaller than it was before.

A

So it was quite huge Improvement, but there are also some other some other improvements that we made like page size progression for selectors. So what you can do is you can list some resources from the API server and you can provide limit. So let's say that you provide the limit of one, but you also have some field selector. So what was happening was that whenever, whenever you were making such call there, the elements from the hcd were fetched one by one.

A

So if you have like very rare selector- and let's say you have 1000 items and only the last one matches the selector, what was happening was that there were actually there were actually 1 000 calls to to add CD made, and but we fixed it with a progression of the limit. So the limit that you use for the API calls to API server is no longer bound to the Limit that we used for fetching the results from edcd.

A

So um now, to sum up like, if you want to get involved, um we have home page where you can find all the contact information. If you are interested in scalability, we do have a bi-weekly public meetings. uh We do have also a mailing list and if you want to get involved in, for example, our testing infrastructure, we do have some issues that are marked as help wanted. So you can just pick up one and and get involved in scalability of kubernetes, and now we have um I think a few minutes for Q a.

A

Do we have microphone baby, okay,.

B

Thank you hi. My question is, basically: is it the scale we recommend to which kubernetes operators can go? The one which you are testing.

C

B

If you they go beyond that, it's their own risks of say.

A

That's that's a great question so um actually um what's happening is that there are limits, as I mentioned, that we are testing like, for example, 5000, nodes or 10 000 services. And if you go beyond that, what's usually happening, the performance is the grading kind of like gracefully and in a way that you know the latencies start increasing and everything starts to slow down, but it's usually not like it just breaks, but and that's right that usually it's it's uh the cluster operator who is responsible for that?

A

That needs to monitor it and make sure that those limits um are like the cluster is within those limits. But there is also one more thing that I want to mention that um to make sure that all those slos are met there are. There is also a bunch of requirements that your cluster needs to uh to to be following, for example, one of the requirements. Here we have that you know the control plane, um you have to have 64 cars, for example, or you have to have two instances of hcd and so on.

A

So basically there is the bunch of list of requirements that you need to also have implemented in your cluster to to make it scalable to those limits. But it's all documented there. Basically.

D

Thank you so a follow-up to the question. uh So is that documented? What happens when those latencies increase, Beyond, five thousand and and then it degrades what happens and is it able to recover um when, when you reduce the load.

A

So um there is now document like that, and the reason is that it really depends like. We are sure that you know if you have vanilla set up of kubernetes and you don't have um super fancy controllers, then it will work right. But once you start using some of the components I've seen in the past that there are components that actually do not gracefully kind of like stop working, but they just break the whole cluster.

A

At some point like if you, if you reach the limit of whatever, let's say secrets, and then if you have 20 000 Secrets, it will just stop working, for example, and there are components like that that cluster operators sometimes install. So there's no really simple answer here, because there are so many components you can install in kubernetes that it always depends on what kind of components do you have.

D

uh The second question I had was: what is the significance of that 100 node test that you talked about so.

A

This is kind of like um our first first test that each change is tested on, um because you know in a perfect world what we would be doing is like. Whenever someone makes PR to kubernetes, we would run 5000 nodes test that tests everything, but that's super expensive. So what we are doing is basically we are running this 100 nodes test instead, which is much cheaper, and we can do that.

D

Got it, but why 100 I mean 100 is like.

A

D

Mike yeah just last question: I mean why 100 like 100 is a two less of a number. It's just because of cost you're, saying or.

A

So it's actually both cost, but also it also detects. um It also detects like very basic, like regressions, like significant regressions, so you know it's it's. uh It actually saves us quite a lot of time to to debug it, because once it's past and merged to the master branch of kubernetes, then if we observe regression on larger scale, then it actually involves like human person to to to go through and see what kind of changes were made and what was regressed.

A

So this is kind of like you know, first first test to reduce the number of noise.

E

A

E

For actually for what it's worth, we did have a uh one of our clusters that went above like 5 000 nodes in the control plane just broke, it never recovered, uh but that's actually a great segment of my question. You had a slide on profiling for a cube, API server. Do you need to have access to the control plane because there's a lot of cloud providers that don't give you that access? That's.

A

That's true yeah, so um in open source we are actually testing. You know um like fully on our VMS. We are not testing, you know eks or gke. uh We are only testing uh kubernetes.

A

As you know, the open source version basically yeah. So there we have it.

F

And just two questions: the first one is that um for the we are developing some platform and then part of the kubernetes ecosystem. So what is? Is there any guideline to you know test those operators like custom resource.

A

Yes, so that's a great question: um currently, there is no guidance for it, but um this is this.

A

This topic is actually um coming up more and more often so probably we will need to invest some time into some guidance, how to write controllers and operators in a way that they do not put too much load on the control plane, um but from experience I would say that usually the biggest issues are with the agents that are running on all the nodes, and then there is a chance that if you have some kind of specific bug, then you put too much load on the control plane and you can break basically control plane.

F

A

Yeah, that was your question.

F

Or sort of you know we're trying to see if I evaluate the performance and scalability of the controller in the kubernetes ecosystem, maybe not exactly the kubernetes cluster per se, but controller is part of the you know. You talk the API server and they schedule an LCD all of them loads it there and then, in this kind of context, um yeah I've been trying to still trying to figure out. What's the best way to evaluate this as controllers or the platform boys, but I think that's um yeah, that's fine, I think that we can skip.

F

The second question is that in the cube, Mark there's no part the Creator right. It's like a hollow node, as you mentioned right, and so in that sense. um So this part of creation, time or something. If we want to evaluate that then Cube Mark, is there any other tools and tooling? You know that can um evaluate this um part of the actual part really yeah.

A

So um not really so I think uh I think like yeah in Cube Mark. Basically we are not running those spots, so it's just simulated um but I, don't think there exists anything that you know like if you start running those spots and containers, then basically you're you're close to having like real clustering, I, think yeah.

F

A

Okay, we have time for last question: I, think.

G

Foreign, thank you I think on one of the your slides you list like a three regression.

A

G

One of the three I think if I remember right is about part startup latency so did did. That was that issue ever fixed. Oh.

A

Yeah, actually, actually I can give a bit more context there. So so there was, there was some feature developed um in kubernetes and by accident.

A

What happened was that for each request that was made to API server, there were two Guardians created, and you know by itself like it shouldn't have huge impact, but we observed, because you know in our scalability test we have hundreds of thousands of bird teams in the API server and because we doubled that what was happening that the going scheduler was already kind of slowing down, and because of that we saw the regression input startup latency, but yeah.

A

Then, basically, we just reverted this PR, um because we decided that you know we need to fix it, because the regression was quite significant uh and then um outdoor had to actually implement it in a way that we were not increasing the number of go routines in the API server.

A

Okay, I don't know, can we one more question.

C

Foreign, just a quick one, so basically all of the examples and evil that you gave um were were mostly about uh reactive, um reactive action to to changes in kubernetes.

A

C

Basically, we're doing a lot of taxing and catching the scalability issues and fixing them when they get merged, but how about the trying to improve the actual scalability of the project with the Baseline that we have so, for example, you had that five seconds deadline. There is the sake working on trying to get those thresholds lowered actually doing profiling of. What's taking up those three seconds that you are considering like the normal Baseline now or is that something that's uh completely out, because you don't actually own the production code.

A

um So I think so how we usually operate is we. We really want to set our goals based on our customers expectations. So it's not just you know in improving based on you know like just make just like possibility of improving. So uh there are some some metrics that our users care about, and there are some metrics that user. Don't they do not care about, but we still measure them in order to catch regressions, um but there were some some improvements that actually we, but they still came from like our users.

A

So what we were seeing, for example, is that people use quite a lot of Secrets and what's happening is that when you use secret and you mount the secret to pod, uh this pot or kublet actually opens opens, opens watch and we introduced immutable secrets to kind of resolve this issue to reduce the number of watches and increase scalability, but I think we need to actually finish right now, but it will be still still here. So if you want to talk, then feel free. Thank you.