Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Kubernetes and etcd Features That Unlocked 15k Node Clusters - Wojciech Tyczyński

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Kubernetes and etcd Features That Unlocked 15k Node Clusters - Wojciech Tyczyński, Staff Software Engineer, Google

In this presentation you will learn about improvements made to Kubernetes and etcd that unlocked running massive clusters with 15,000 nodes. We will show how we tackled this challenge from different angles focusing on its technical aspect. You will learn about the most important changes like watch bookmarks, etcd concurrent reads and more, but also our cooperation with other communities (e.g. golang). We will also describe how we approached validating our work and how real Kubernetes users helped us here.

https://sched.co/eoFN

A

Hello cubecon, my name is wytek taczynski. I work in google and I'm part of this amazing kubernetes community for almost six years at this point, amongst others, I'm the tl of six scalability and in connection to that. During this presentation, I'm going to talk about example, features that allow us to run kubernetes clusters with 15 000 nodes.

A

But let's start with making this explicit kubernetes clusters with 15 000 nodes are already a thing, as you might have heard. Earlier this year, buyer crop science and google together published an interesting blog post.

A

You can read there how thanks to running on gk clusters, with 15 000 nodes, with almost quarter million, of course, and over 1.5 petabytes of ram buyer crop science was able to process roughly 15 billion genotypes per hour as part of the pipeline responsible for deciding which seeds, which are the final product should be advanced for further experiments in their r d department.

A

Couple months later, during joint presentation, twitter and google together described how we approached validating that their applications that they are currently running on their mesos aurora clusters on prem can be splitted and run on gk clusters with 15 000 nodes.

A

But the goal of this presentation is to show that this work matters for all of you, even if your clusters are order, or maybe even orders of magnitude smaller.

A

However, we need to start with understanding what scalability really means for kubernetes, even though we tend to use node count or cluster size as a proxy for the overall scale. It's actually much more complex than that.

A

Scalability is a multi-dimensional problem with dozens of dimensions such as number of services, number of volumes, pod, churn and so on, even though they in many use cases they may scale together with cluster size. You need to acknowledge that note. Count is not the only thing that matters, so how did we approach scaling kubernetes to the next level, which 15 000 nodes certainly is the core principle for any scalability or performance work is don't optimize blindly and always solve real life problems.

A

Almost every optimization is making a system a little bit more complex, so it's super important to keep that in mind.

A

According to this rule, we first found users for whom running such a massive clusters would have actual benefits.

A

We started with understanding their use cases, but also their motivations and, as you may suspect, those are the ones that I already mentioned.

A

The first one was buyer crop science, who is using kubernetes to run embarrassingly paralyzable batch computations for them larger clusters immediately translate to having results faster.

A

In addition to that, given that the users are generally data scientists who don't want to understand the underlying infrastructure, they want to make it as simple as possible. Ideally a single cluster.

A

The second one was twitter, who would like to migrate the microservices apps to kubernetes.

A

As I already mentioned, they are currently running them on mesos aurora clusters, which can handle even 40 to 60 000 nodes in a single cluster, so, while migrating to kubernetes, they would like to avoid the need to suddenly manage an order of magnitude, more clusters.

A

In addition to that, they would also like to unify the setup and run applications that they are currently running differently, like some stateful apps, also on kubernetes.

A

At this point, you're- probably thinking that you are wasting your time, I'm saying that we focused on two users: doesn't that sound simple by the way and your workloads probably aren't even similar to what they are running.

A

However, just like landing on the moon pushed the frontiers of technology, it's the same with our scalability work, the improvements we did not only push the scalability limits, but they are primarily making all clusters more reliable and more performant.

A

So now I'm going to describe a couple improvements to show how they make your life better.

A

So, let's start with xcd the main lcd improvement that is worth mentioning is concurrent reads before this change. Every read operation on xcd was blocking all other operations, both read and write for the entire time of its processing.

A

Thanks to this improvement, we are now blocking other operations only for a very short period of time, just to grab a copy and write pointer to the current state. While it's crucial for large clusters, it's not the only place where it helps.

A

Imagine that you have 10 000 custom resources of your crd and I've seen real production clusters like that with just 10 or 20 nodes.

A

Thanks to this improvement, when you are listing custom resources, you will no longer observe spikes in the api called latencies, even if your cluster is small and it's already available in hcd 3.4.

A

So let's now make a step up and talk about api server and api machinery.

A

As you know, watch is the crucial part of our api, but no request runs forever.

A

So what happens when it times out, especially if it's a very selective watch that selects only a small fraction of all objects of a given type to resume the watch. We are using resource version of the last object that we received via it, but many other changes might have happened in the meantime, and now we need to process all of them again, simply because we didn't have a way to signal the client that they have already been processed.

A

So we introduced a concept of watch bookmark, it's basically a new event, type that you can receive via watch that basically tells you we processed everything up to a resource version x. So if you didn't receive anything, it simply means nothing much to your selector.

A

As you can see, there is nothing specific to 15 000 node clusters. It helps for for clusters with thousand or even hundred nodes, and it just happens out of the box, because cubelet that is watching its own parts is a perfect example of very selective watch that benefits a lot from this improvement, and it's already ga in kubernetes 117..

A

However, improvements were needed across the whole stack, so let's take a look into an example from the networking area for those not familiar with endpoints api. The endpoints object contains information about all backends of a given service, so its size is proportional to the number of pods behind that service.

A

That has many consequences, but, let's think about cube proxy, which is an agent running on every single node in the cluster that is responsible for programming in cluster networking.

A

In order to do that, it's watching for changes of every single endpoints object in the cluster, so imagine that you have a service with 5000 pods. The corresponding endpoints object will have optimistically around one megabyte so in 5000 node cluster, and I'm not even talking about 15 000 nodes. At this point, that means that api server has to send five gigabytes of data for a single change of that object and for the rolling upgrade of that service, it would be 25 terabytes of data to mitigate that.

A

We introduced the concept of endpoint size that allows us to share the information about the endpoints of a given service into multiple endpoint slice objects.

A

Thanks to that, we can significantly reduce the load on the control, plane and amount of data that ipa server has to serialize and send the pieces of that solution went better in kubernetes 119..

A

So, let's take a look into one more example, this time from the storage area and let's talk about secrets and config maps, as you probably know, whenever any of those changes cubelets are reflecting those changes to all pods that are mounting them, but in order to do that, they are opening a watch for every single secret and config map that they pods are using.

A

That may translate even to hundreds or hundreds of thousands of additional watches in the system.

A

Optimizing them would bring a lot of complexity to the api machinery and, as I mentioned before, we should always be solving real life problems, so we started talking to users and it appeared that they don't mutate majority of secrets and config maps at all.

A

So we introduced the concept of immutability to secret and config map api when explicitly marked as immutable by the user. Their contents cannot be changed, but cubelets also don't need to watch for them, which is vastly reducing the load on the control plane.

A

As in the previous examples, there is nothing specific to large clusters and you can take advantage of it. Even if your cluster is small and it's already better in kubernetes 119., however, we were also looking outside of car kubernetes.

A

We worked closely with golang community on optimizing its memory locator. It may be surprising to many of you but look contention at the level of go memory. Allocator is actually one of the bottlenecks that we really suffer from, while some optimizations have already landed in newer versions of go even more are coming, and this benefits not just kubernetes.

A

It benefits everyone who is writing their applications in go. I described a couple improvements and all of them as well as tens or maybe even hundreds of others, were done in upstream kubernetes.

A

However, that doesn't immediately mean that every kubernetes distribution will scale to 15 000 node clusters in order to work your ecosystem has to work at that scale too. That includes the underlying infrastructure, both compute and networking. You need 15, 000, vms or machines and networking between them. It includes auto scaling, logging and monitoring, control, plane, upgrades and many other things based on gk experience.

A

I can say that it's a huge effort to make all of them work, so kubernetes improvements are necessary, but they don't solve all the problems for you.

A

There is one more question that we should answer here, which is how do we know when we can stop?

A

Fortunately, the answer for this question is fairly simple. As soon as we meet our slos service level objectives, you can think about them as system level metrics with thresholds, while the concepts they cover in kubernetes are still fairly basic, like api call, latencies or bot startup type, they greatly correlate with user experience.

A

So to summarize, scalability work matters for almost everyone, because scalability is much more than just the cluster size.

A

The improvements we did to push the scalability limits are also, or maybe even primarily, making smaller clusters more reliable and more performant.

A

So if your cluster didn't work because of some scalability or performance issues in the past, with kubernetes, 19118 and higher versions, it's probably time to re-evaluate.

A

Unfortunately, we don't have more time, so I will just mention that, with upcoming releases, you can expect even more improvements and extending the portfolio of use cases supported by clusters with 15 000 nodes, and with that, thank you very much for staying with me.