Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro + DeepDive: SIG Scalability - Wojciech Tyczyński & Marcel Zięba, Google

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Intro + DeepDive: SIG Scalability - Wojciech Tyczyński & Marcel Zięba, Google

This session will focus on the different efforts that SIG Scalability is involved in: defining what scalability means for Kubernetes, driving improvements, infrastructure for scalability testing, tests and guarding Kubernetes against performance regressions. Each of those areas will first be described at the high level, followed up with deeper insight into concrete aspects and summarized with most recent achievements and a roadmap for future work. Time for Q&A will be reserved at the end of the session to understand how the SIG can better engage with the community as well as to allow the audience to provide the input about the roadmap.

A

Welcome everyone: uh this is six capability introduction and the deep dive and um I'm martial jumper, I'm software engineer at google and also a sixth cd member. Unfortunately, white tech won't be presenting with me today, but hopefully he will join for the q, a session he's senior staff software engineer at google and sixth city tech.

A

So, first of all, let's start with what do we do as a sixth capability? There are five different um main areas that we are interested in, starting with defining and driving scalability definition and goals. Once we have those goals, we are interested in monitoring and measuring performance of the kubernetes, so we have goals, we have measurements.

A

So now it's time for improvements.

A

Basically, with those measurements, we can find some bottlenecks and drive those performance improvements in kubernetes and those performance improvements can happen in two different ways: either we just contribute to kubernetes or we coordinate with different, seeks to make it happen.

A

As you can imagine, in kubernetes with each version there is a bunch of new features added, so what we want to do is also protect kubernetes from any scalability regressions.

A

Last but not least, it's consult and coach community members about scalability.

A

You want to add some new feature to kubernetes and basically, before you start implementation, it might be worth to consult it with with us to make sure that there is no any obvious bottlenecks from scalability point of view and the most important. It's, though, not to confuse six capabilities with sick, auto scaling, it actually happens quite often.

A

And yeah we will go through those five different areas uh and during this presentation, so let's start with defining what actually kubernetes scalability means, if you ask average user okay. So what do you want in terms of scalability?

A

Well, they will sell scalable clusters, but if you ask okay, but what does it mean well, most of the users? Unfortunately don't know so. Maybe let's, let's look back at the history of kubernetes and kubernetes 1.0 was released in 2015 and officially it supported 100 nodes, and this number changed over time. It was 1, 000 notes and then in 2017 5 000 nodes.

A

So you might ask okay so how much nodes kubernetes supports right now, and the answer is that this number didn't change at all. It's still five thousand notes, so you may ask okay. um Well then, six calabiti. What were you doing during the last four four years? Well, it turns out that stability is not just the number of nodes that kubernetes supports.

A

In fact, scalability is something that you need to analyze in many more dimensions. To give you a few examples: number of nodes is just one of those dimensions.

A

We have a bunch of other dimensions like number of namespaces number of secrets services and if you have number of services, then for each service, what's important is how many backends do you have per service and so on, so based on those um dimensions. What we would like to do is define save zone scalability envelope, which is a safe zone and by save zone. What I mean is that if your plaster is within this save zone, your cluster will be happy.

A

But again, what does it mean that the cluster is happy? What is the save zone? We will try to define and explain that in more details on the few next slides. So maybe, let's start with what does the mintage cluster is heavy?

A

And here we have two key concepts: sli and slo. Sli is service level indicator.

A

Imagine that you have that you have cluster and you're interested, for example, in pod, startup latency, pod, startup latency is the time how much it takes for pot to be running since it was scheduled to particular node.

A

So you are making those measurements, you have a bunch of them and then you take 99th percentile of of both startup latency measurements and it turns out to be let's say three seconds so this would be actually sli for put startup latency for your for your cluster and on top of that, what we do as a six capability. We add threshold to it and forward startup latency. Let's say it's five seconds, and this is basically slo, so we can think of slo as a sli plus threshold, and that should be satisfied.

A

And in six callability we have bunch of bunch of different slos. A few examples like api call, latency uh already mentioned bot, startup, latency and different types of networking latencies like dns latency. You can find all those slos that we support as six capability on our github page.

A

So maybe, let's see how the definition of api call latency looks like it will be a little bit simplified version. You can find full version on our github page and but basically um api calls. We would like to divide them into two two groups. One will be right, calls and 99 of right calls.

A

We would like to have latency below one second for read calls we would like to have it's a little bit more complex if you are just getting one object like let's say you are getting pod, then the latency should also be within one second, but if you are trying to, for example, list all the secrets in the namespace and then it may, it might take up to five seconds.

A

uh On the other hand, if you want to list all the pods in your cluster all the secrets across all the namespaces, then this type of request might be quite long running up to 30 seconds uh in total, so yeah and remember that it's 99, so you can always be unlucky and in this one percent and basically that your request will take longer than that.

A

Okay, so we have. I think we have pretty much good understanding. What kind of uh sls and six callability uh provides. How do they look like so now we can actually define what does it mean that cluster is happy, so cluster is happy when all of those scalability slos are satisfied and based on that, um we can have this kind of like framework and that if you promise us that your cluster will be within save zone, then we can promise you that all scalability sls will be satisfied, but we still didn't discuss.

A

um We still didn't discuss what's the safe zone.

A

So if we want to compute this scalability envelope, it's actually impossible to compute it precisely because there are multiple different dependencies between dimensions which are quite quite complex and obviously it's changing all the time.

A

But instead of trying to find like this precise scalability envelope that you can put whatever configuration of of cluster, you can imagine and then get output whether you are within or not, and what we can do is actually approximated and we try to approximate it by set of limits, and as mentioned before, we have one one limit at its number of nodes: it's up to 5 000 nodes, then what we have is number of pots number of pots should be less than equal, 30 times number of of nodes, so we can think of it as 30 pots on average per node and, for example, other limit, and that we have is number of services which is 10 000 and all those kind of all those kind of limits you can find on our website.

A

So, at the end, what happens is that if you have cluster, you are probably interested in in those uh in those limits and if they are satisfied, then we promise you. um Our scalability slos will be satisfied as well.

A

So now we need to make sure that actually for for this stability envelope, uh those slos are satisfied, and here we have stability, testing infrastructure. We will start with cluster loader and a few years ago. Actually we didn't have cluster loader and our scalability tests were purely written and going.

A

You can imagine how how hard it was to actually maintain those tests, and since then, we developed cluster loader, which is really great, and you can basically bring your own yam with test description and the test description consists of states in which you want your cluster to be in so you have, you can have multiple states.

A

You can also specify how to transition between those states and, during this whole testing and cluster loader also is gathering a bunch of measurements, so he can make a decision whether whether all scalability slos were satisfied satisfied during this test, except for that um cluster loader also provides a lot of extra observability for debugging, which we will actually cover later. You can find all those features in our perth test repository, which has also implementation of cluster loader 2..

A

The other tool that we commonly use is cubemark. Maybe it's not a tool, it's actually cluster simulation, so in the scalability envelope you saw that we support up to 5000 nodes and testing cluster consisting of 5 000 nodes. It's it's quite quite expensive and time consuming so instead of having actual 5000 vms. But what we are doing is we are creating something we call whole nodes.

A

So here what you can see is: okay, we have three actual vms and each of this vm is running multiple hollow nodes and those hollow nodes, kind of simulate, regular nodes.

A

The difference is that if you, if you schedule some pod on a regular kubernetes cluster, it will be actually running, but here, if you schedule it on hollow note, then kublet, which we call also hollow kublad, will only say: okay, I'm running this, uh this container this spot, but it will not actually run it so, and this allows us to run multiple hole nodes on one node and each of those whole nodes is actually running.

A

Three containers, one is a couplet, the other is cube proxy and a node problem detector, so those whole nodes connect to master, and this master is fully functional.

A

So basically we test this master and whether all the slos are satisfied with cluster loader with our cluster loader. The only problem with this kind of setup is yeah, but how? How do you actually deploy those whole nodes? So this becomes a little bit tricky because um let's say you have hundreds of nodes and then you want to run those hollow nodes on on those hundreds of vms and well running. Multiple containers is something that actually kubernetes solved, so those whole nodes are are scheduled and scheduled, basically by separate master.

A

This master is responsible only for scheduling those whole nodes and running them on physical machines, and that's basically, that's basically it so we have two masters. This master is is not being tested.

A

And to give you some some examples, um I mentioned that yeah we we support 5000 knots. We also have 5 000 nodes tests, but using cubemark uh instead of using 5000 cores, we are actually we can use 700, cores and kind of simulate 5000 cluster, which, which is really great, because it allows us to cut down the cost, but also iterate easier. If, if you can imagine that there is regression that happens only from time to time, let's say one in five runs, uh then with cubemark we can easily run ten tens.

A

Basically of of those runs to find this issue easier.

A

So now we will go uh to observability and debugging, and one of our best tools for visualization is is perv dash and it is really great because just satisfying slo is not always enough here.

A

You can see example of pot startup latency, and we can see, for example, the blue line is 99th percentile of pot startup latency and on the x axis you can see different runs, uh it's around uh 100 points, so I would say it's like three months and you can see that okay, we had around four seconds uh four seconds uh put startup latency and then it decreased. I would say: on average it's around 360, let's say uh 300 300.

A

uh Sorry, three second 3.6 seconds so from four seconds to 3.6 seconds and the question is: was it some kind of scalability improvement? Actually it was a regression that we found and then we fixed, but we will talk about it a little bit later so um peridot, basically analyzes. All those runs and allows us to compare different, runs, different slis but also, let's say cpu usage of vm or memory usage.

A

So it's very useful for finding regressions that are not necessarily violating slo, except for that for each run. Cluster loader has ability to run prometheus within your cluster and gather multiple, multiple metrics and, for example, cube api server, metrics or at cd, scheduler and so on. But also, if you are running, for example, dns latency test, then you can configure it quite easily to also scrape metrics from from some dns spots that are making requests to check. What's the latest and with those metrics, we have pretty pretty nice setup of grafana.

A

That allows you to quite easily uh check what kind of api calls, for example, were evaluated in what what part of tests and so on, except for that cluster loader also automates gathering profiling data, uh it's basically a cpu, but also memory and and mutexes base, and once you are running the test with cluster loader, what happens is then at the end, you also are getting a bunch of profiling data uh which comes in handy when dealing with cpu regression or memory regression on kubernetes master.

A

So now, let's go to scalability tests. What kind of tests we have we have periodic tests? That's our! These are one of the most important for finding regression in kubernetes. We have release blocking tests, but also non-release blocking tests for release blocking tests.

A

We have performance tests, performance test, basically tests, all those slos, and we we for two scales. One is 100 notes and 5 000 nodes, but also we test correctness. So correctness is not only if all the slos were fine. That's the purpose of performance test in correctness. We also check if it's actually functional.

A

So maybe I will tell you a little bit more how the performance test works. So I would say that the performance test you can kind of have three main stages of performance test.

A

Let's say we are starting with empty cluster and the first stage stage is to actually load the cluster. So we have the scalability envelope and we are trying to maximize all those dimensions that we think should be in safe zone and once once we do that. uh What what happens is that the next stage is um scaling up and down different deployments. Updating demon sets so a bunch of different uh stuff, that's generally generating a pot return.

A

And the last stage, basically, is deleting all those things and we test this in two scales: 100 nodes and 5 000 nodes, um because uh 100 nodes sometimes help us to pinpoint exact comet that you know broke. Kubernetes and 5000 nodes are run only once once a day for non-release blocking. We have different tests, starting with cubemark that I mentioned before, but also storage and benchmarks.

A

One of the one of my favorite benchmarks is actually a column compiler. So what we do is we took a performance test. uh We've we kind of fixed all the of the dependencies, except for the golang compiler. So we are taking basically uh one kubernetes version.

A

All the dependencies are the same and what we are changing is we are compiling uh kubernetes with newer and newer golang compiler, and we are checking whether uh whether there's any regression introduced by comic- and this is super interesting because actually in the past we had um we had bunch of different regressions caused by going except for periodic tests. We also do have pre-submit tests, and this is only basically performance test on the scale of 100 nodes and we are running those pre-submits for kubernetes.

A

So if you ever created pr kubernetes to improve something- and you have this bunch of list of pre-submits, one of them is our six capability. Pre-Submit.

A

So how are we protecting?

A

How are we protecting scalability of kubernetes? So, first of all we do have no, we do have test grid uh everyone, it's publicly open. You can go there and and see okay. So what's the status of as performance test at scale or what's the performance.

A

Status of scalability test on scale of 100 nodes, and it's basically one of the first tools that we use for protecting our protecting kubernetes from scalability regressions, and the thing is that scalability is very sensitive, so we've seen regressions coming pretty much from everywhere, as I mentioned before. One of the examples is calling, but then also operating system controllers, api machinery, scheduler and cd couplet, and what we do is we either fix by ourselves or we try hash that to other seeks.

A

So I would like to tell you a little bit more about two regressions that we prevented in 122.. They were pretty interesting. One is spot startup latency regression, so I was actually showing you this regression in perv dash. The difference was around half second, let's say, and this this regression was really interesting because by itself it didn't break our tests, but due to the perv dash, we were able to find it, and what happened was that the number of routines in kubernetes increased twice.

A

Normally we have around five thousand hundred five five hundred thousands of core routines running in api server at scale, but after this change, uh which was related to some improvements to priority and fairness, the number of goal routines jumped to the one million and all of those all of those um slows were still satisfied. But but we saw that uh put startup latency 99th percentile significantly uh increased, so we basically debugged it and and fix the issue.

A

Except for that. We had really interesting other regression, um which was really tricky because one of the features was introduced also to pnf, and it happened to be that bug was found in totally different place and the priority and fairness was only a trigger for this bug.

A

The idea was that you have some periodic calls in api server like updating releases of nodes and by default they happen every 10 seconds.

A

So if you have 5000 nodes cluster, then you can, on average, have around 500 updates per second and if those uh updates are evenly distributed across the whole. Second, then, it's fine, but some changes in priority and fairness caused those 500 or even more calls to kind of like synchronize, and this synchronization is really bad because the load was not spread evenly.

A

So this was very interesting and that's one of the regressions that we also fixed recently, but also we are driving some scalability improvements and let's go through three of them, so the first one will be efficient. Watch resumption, efficient watch resumption is a really great new improvement that helps you with upgrading your your masters.

A

So, as you might know, if you, if, for example, kublaid wants to get secret or config map, then what it does it's making get call, but then also it's making watch call and this watch call basically should be kept alive all the time all the time.

A

But when the api server restarts or you upgrade it, then unfortunately, this watch couldn't be resumed and this was a huge issue, because, um let's imagine that you are upgrading your cluster and then all those watches break and then all uh all 5000 machines are actually trying to to get what they want. At the same time- and this can quite easily overload api servers, so efficient watch resumption actually fix the issue where, where the watch can be can be resumed, except for that we are working with different seeks on priority and fairness.

A

We are constantly improving it also from scalability point of view, like first version, for example, of priority and fairness was not distinguishing between get goals and list calls also, as as you might think, those watch calls are also important in kubernetes. So we also added support for initialization and things like that, and basically, we are working with with other sick teams to to make it better and better to have better reliability of api server and, last but not least, immutable secrets, so imitable secrets actually reduced the load and potential load that api server is receiving.

A

Going back to the same example, if, if you have pod and the spot is using some secret, what happens underneath in kubernetes is that first of all, kubernetes is kublad, specifically, is getting the secret from api server, but then it also watches for any possible changes.

A

In most cases those changes never happen and because they never happen, then then it doesn't make sense to keep this watch. So this introduced immutable secrets that help with reducing number of watches and reducing load on api server.

A

So if you want to get involved here is some links you can find our homepage, but also you can join our slack channel our mailing list and if you want to get involved and then you can just ping us on slack or or just maybe you can check how what kind of issues we have in our repositories with getting started or have wanted labels and yeah looking forward to it if you are interested, so that would be all and thank you for attending and now it's time for q. A thank you.