Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Discontiguous CIDRs for Dynamic Cluster Scaling - Rahul Joshi & Sudeep Modi, Google

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Discontiguous CIDRs for Dynamic Cluster Scaling - Rahul Joshi & Sudeep Modi, Google

Users want more out of their existing clusters. As demand increases, they would like to increase the size of their clusters by adding more nodes and pods (scaling horizontally). One of the key limitations in doing this is immutability of IP addresses. Providing each additional node and pod an IP, quickly exhausts users’ initial IP allocations. They must resort to optimizing IP addresses in their existing clusters, or pre-emptively over-allocating IP addresses during cluster creation time. With Discontiguous Pod and Node CIDRs, IP addresses become a mutable resource that can be added post-cluster creation thereby helping in the growing the cluster dynamically. Adding support for disjoint CIDRs also allows fitting clusters into fragmented IP spaces, a problem commonly faced by enterprise customers. This talk will describe how GKE solves the problem for customers and the lessons learnt along the way. This talk also proposes ideas on how to solve this natively in Kubernetes.

A

Hello, everyone welcome to our talk on discontiguous ciders for dynamic cluster scaling. uh Let's jump into some introductions. My name is rahul zoshi. I am a software engineer working at google on gk networking.

B

Hello, everyone, my name, is sudeep modi and I am a software engineer working in the same team as rahul gk networking and the two of us have been working on this problem for about over a year.

B

So what is the goal of the talk today? We want to present our proposal to support discontiguous ranges for port ips and the reason why we want to support discontiguous ranges is to enable flexible ip address allocation, as well as remove one of the major bottlenecks for scaling a cluster by the end of this talk, our goal is that you understand the problem we are trying to solve.

B

You understand why it's important to solve this problem, we'll introduce some of the difficulties and challenges in solving the problem, and finally, we are going to present a proposal on how to actually solve the problem.

B

So that's the agenda today we I'm going to talk about how kubernetes pod ip allocation is done. One thing to note is I'm talking about vanilla, open source kubernetes, I'm not talking about some of the bells and whistles that cloud providers had and how kubernetes works out of the box. Basically, I'm going to talk about some potential improvements that we can make to the way we do pod ip allocation. Today we are going to talk about cluster cider in kubernetes and cluster cider.

B

Is this one single pod ip range that you provide today and then finally, we're going to talk through our proposal on enhancing and supporting this contiguous spot ip ranges? And finally, we will talk about some of the ongoing work that's going on in the community as at the time of this recording.

B

So this is a typical way. A cluster comes up for a customer. You have on your network ip space and when you bring up a cluster kubernetes ask you to ask you this question. Give me a cluster slider, the cluster slider. Is this one large range from which all pod ips are allocated and in our example cluster over here we have allocated a 17 range for this new cluster, and this gives us roughly 32 000 ips for that cluster.

B

In addition to the cluster, cider kubernetes also asks you to provide a node cider mass size. The way this works is every single time. A new node comes up. The range allocator for kubernetes internally chops up the large cluster slider into blocks of the node zero mass size. So in this case, 24 is the node center mass size. So what is happening is every time a node comes up.

B

The range allocator is going to allocate a slash 24 from the cluster cider range to that node and the way ip allocation works after the pod cider is assigned is independent on each node. There is no global synchronization. Each node is going to look at its own parts hider and pick an ip address from that parts, hider independently of any other node. So there's no coordination.

B

The interesting bit is that you also need to over provision these ips on the nodes. The reason being things like daemon sets being upgraded or parts going up and down. You don't want to reuse your ip addresses, so you want this buffer, and so, even in this case, when you have a slash 24 assigned to the node you're, not going to run anywhere close to 250 parts, you're going to run for your part, so this this exacerbates the use of ip addresses in kubernetes.

B

So what are the problems with these ip allocation? The first problem is that, as soon as you want to create a cluster, the customer needs to go and allocate a large a large ip address range for boards right up front, regardless of how big or small the cluster is going to be. Eventually, so, let's say the the users. The users tend to either overestimate or underestimate the requirements we've seen errors on both sides for with our customers.

B

Sometimes the user thinks that the customer cluster is going to grow to like 250 nodes, but you may not end up putting so many workloads on it. So at that point, if you, if you over project provision your cluster, this ip address wastage.

B

However, the other problem is also true, where you don't know upfront that your cluster is going to grow so large, and then you start sort of start running out of ip addresses. So at the time of cluster creation, you have to decide how many ips to allocate- and this isn't easy.

B

The second problem is that each node is going to get the same range for the part slider.

B

So if you have certain nodes which are of lower capacity, you have to give them a slash, 24 and waste some ip addresses, even though you cannot run close to the number of parts that the larger nodes can run and some larger nodes can may be able to run more parts than what you provisioned for so they may be able to run more than 250 pods in this case, but you can't do that because you run out of ip addresses, so both sides of the problem are in effect and then the most interesting and difficult problem is that once you provision this cluster and your clusters, let's say grows over over time.

B

In this example, if it's grown to 128 nodes, there is absolutely no way to add another node, because you've run out of ip addresses customers. We've seen customers run into this problem all the time, and there is no solution. The only solution is to either migrate certain workloads from this cluster to another cluster, or you recreate the cluster with a larger range and start from scratch all over again and neither is a very palatable solution.

B

So what are the potential improvements we can make to this? The first is to add support for increasing this cluster cider. Somehow, and you want to do this in a manner that doesn't cause any downtime you've run out of ip addresses.

B

There should be a way to allocate more ips and I don't want any workloads to suffer uh and- and if you did this, it addresses the most difficult challenge that I spoke about in the last slide, where you can actually now scale your cluster uh where you earlier could not the second improvement we can make is we can? We can allow specifying discontiguous ranges for pods, and this is true, even at cluster startup time.

B

There are certain customers where we see they don't have a large chunk of a slash 17 lying around, but maybe they have smaller slash 20s lying around and they want to still create a large cluster.

B

So you need to be able to specify these discontiguous power ranges at cluster startup time or this can come when you expand your cluster, and this addresses the problem that a lot of customers run into when they have a fragmented ip space and then, finally, you want to accommodate the different nodes in the different sizes of nodes, as in order to more efficiently use ips across the node, so smaller nodes can get smaller, less fewer ips and larger nodes can get larger ips. You need to be able to specify this.

B

The biggest challenge on this is going to be the dependency on cluster cider. So what kubernetes components are today, depending on this large one cluster cider range, so we have to go and investigate that, and then we wanted to go and remove the dependency on this single contiguous side of block.

B

So rahul is going to talk about the work that we have done and that we are, we are still doing.

A

Thanks sadiq, uh when we started looking through the kubernetes code to find out what components were using cluster cider, we found two uh major components. The first is the node ipam controller. This component runs as part of the cube controller manager and it performs the allocation of the per node ip ranges. This is what we described earlier in the presentation. It chops up your large 17 into smaller 24s.

A

This obviously needs the cluster cider, so it knows what range it's allocating. The second component is the cube proxy. This is kubernetes network proxy that runs on every node. The q proxy is responsible for maintaining the network rules for things like service resolution and handling inter and intra cluster traffic, so pod to pod traffic as well as traffic. That is external to your cluster. That wants to ingress.

A

Let's first discuss how cube proxy uses the cluster sider to make its routing decisions and then we'll talk about how we remove the dependency from cube proxy.

A

One of the rules that q proxy references, cluster cider with is the one for redirecting pod traffic destined for external load, balancer vips. So in this scenario, the user has configured a service um as an external load. Balancer and they've received a load balancer ip that would normally redirect traffic outside of the cluster to back ends inside of the cluster.

A

However, if a pod uses that external load balancer ip to try to access the service, the cube proxy takes note of that, and the cube proxy's goal in this case is to prevent the traffic from unnecessarily egressing your cluster hitting the external load balancer and then just coming back in so instead it short circuits the entire process by redirecting the pods traffic directly to a service backend.

A

In this case, the queue proxy looks at its definition of the service and decides to route the traffic to pod for one of the existing backends and then the clusters network is able to take care of that routing and then on the return path. The q proxy would reverse these uh natting decisions so that the pod thinks it's spoken with a service. In this case, q proxy is using cluster sider to determine whether the traffic is originating inside of the cluster or, if it's coming from outside of the cluster.

A

Another rule where q proxy is using cluster cider, is to masquerade off cluster traffic to service vips within the cluster. So in this case you have some sort of traffic outside of your cluster. That wants to talk to a service fit. Maybe you have a forwarding rule in your network that sends traffic to one of your nodes. In this case, q proxy wants to examine the packet and it realizes that it's destined for a service that it knows about.

A

um However, it can't just forward the packet to one of the back end pods, it's not sure. If the backend pods have connectivity with this external ip, maybe there are firewall rules or other egress or protections in place. However, it does know that the node it's running on can speak with this external ip, so it uses the node as an additional hop. It rewrites the packet using a snap to swap out the external ip for the node ip and then using the standard service resolution.

A

It performs a dnat to reference, one of the backend pods and then q proxy sends the packet on the cluster network towards the back end on the reverse path. It again reverses all of its adding decisions and forwards. The response out to the external ip again in this case cube proxy is using the cluster sider to determine the origin of traffic and whether or not it's coming from outside the cluster or from a pod inside the cluster.

A

The key insight that we had when looking at these rules was that q proxy doesn't need to know about the entire cluster cider about the ips of every single pod. It only needs to know about the pot ips for pods running on that particular node, because every node's cube processes running we can rely on every single q proxy to properly make uh dna and snap decisions at the point that they encounter traffic.

A

If you see traffic for a particular vip that doesn't seem to be coming from your own pods, you can be very confident that that traffic came from outside of the cluster, as every other node would have made a correct routing decision.

A

So in that case, you can rely only on your internal pod cider without worrying about the cluster's global sider. This is part of the proposal that was recently that was added into the open source uh community. uh This kept talks about these examples and a few more and walks through in greater detail exactly the changes that have been made.

A

Thanks to this kep, the q proxy is capable of programming iptables and ipvs, using only the local pod sider, as opposed to the global cluster cider. This cap also added a brand new flag to enable this behavior called detect local mode, which, once used with node cider, will program using just the local siders.

A

This feature has been available since kubernetes release 1.18 this cap and the changes to q proxy removed the single biggest kubernetes assumption around a contiguous cider block. Now that q proxy can function using just its own pod sider.

A

We don't need to rely on a monolithic cluster sider and we can turn our attention to the allocation side of things on the allocation side of things we're back to the node ipam controller that we'd mentioned earlier. This is the component that allocates each node, its actual pod, ip or a pod ip block.

A

When you drill down into it, it has two one has two modes that it can operate. The first of these is the range allocator. This is what kubernetes runs by default. This accepts a cluster sider and a node cider mask, which is how big of a block, how big of a block to assign to each node and it chunks up the cluster sider and then allocates that out. Writing the per node chunks in the node.podcider spec. This is exactly the algorithm that we described earlier in the presentation.

A

The other mode of operation is the cloud allocator. In this case, the cloud allocator relies on your cloud provider to do the per node allocation. It simply queries the cloud provider retrieves the information for a particular node and then writes that back into the node sider, as this model suggests. There's a lot of flexibility with how the node ipam controller behaves. All kubernetes at a core is interested in is that every node gets its pod ip block.

A

So you know end users could come in and write their own custom controller that runs instead of node ipam and performs whatever sort of ipam is relevant for them. However, we wanted a more useful operation right out of the box where users don't need to go configure their own ipam controller, just to expand their cluster slider size. This brings us to the second enhancement proposal, to enhance node ipam to support multiple cluster siders.

A

This is a kept that as if the recording is in active development with the community, it has two major components. The first is a brand new kubernetes resource called the cluster sider config. The details of this object are still in flux, but, broadly speaking, it has an ip cider block, from which every node gets its pot ip chunk. It has a per node mask size, which tells the allocator how big of a chunk to allocate to each node, and then there is a node selector which defines which nodes are targeted by this cidr config.

A

Complementary to this resource is a brand new allocator. This allocator would watch the cluster cider config object and then perform the node ip allocations. Based on that information, the allocator would support, adding and deleting cider configs without restarting your cluster, so you can dynamically resize. It would handle multiple ip families, so ipv4 ipv6 addresses right out of the box and it can allocate variable size ranges to every node. As we mentioned earlier, your smaller nodes can be a slash 24, your larger ones can be a slash 22..

A

This uh slide obviously glosses over a lot of subtle nuance in detail. These discussions are happening in the community and in the cap. If you are interested- and you have thoughts or questions, please, please feel free to. You know, comment on the cap, we'd love to hear your ideas.

B

Now that rahul has gone through the details of our proposal, I'm going to take an example to walk us through whether our proposal actually solves the problem that we were trying to solve. Let's take this example: in this case a user wants to create a new kubernetes cluster. The cluster is going to is expected to grow to about 32 nodes and each node. The user wants to allocate about run about 100 parts, and that means with some buffer. We want to allocate a slash 24 to each node.

B

Now the first thing you want to do when you're creating this cluster is you want to allocate a cloud a cluster sitter range. However, in this particular network, the user does not have a slash 19, which would have been required to run the entire cluster of 32 nodes. Instead, the user has a couple of slash 20s lying around, which are completely discontinuous.

B

Now, with our proposal, what the user does is, creates two different cluster setter. Configs range one and range two: each which comes from a different cidr range, a slash, 20 and each says, allocate a part per node mass size of 24 to the nodes that come up now.

B

Note that in this case, we are not specifying any node selector. This means that our ipam logic, the the new range allocator that we are going to going to be writing, is going to pick a range of range of slash 20 pro from any one of these two ranges range one or range two and allocate it randomly to the nodes that come up.

B

The other thing to note here is that if you were to create a cluster for like a legacy cluster, you can specify the ipc dot block, which is the same as the cluster sitter and the old node seeder mass size becomes the per node mass size and you specify just one range, so this is backward compatible if you wanted to just migrate from the old way of creating things to this new resource.

B

Now, let's say the cluster grows to 32 nodes and then for some reason there are more workloads to be added to this cluster. You are out of the board setter size with our proposal. What you do is add another range. So in this case I have another third range range.

B

Three 10.3.0.0, I'm still allocating a slash 24 to each node, and I create this new range object and I am going to now be able to expand the cluster beyond 32 nodes that were there again, I'm not using any part selectors here, so the ranges are going to be chopped up randomly now. What happens if I want to run a set of nodes where I know that I'm not going to be running so many workloads. Let's say I want to just just run 30 parts on on the new nodes.

B

In this case, I don't want to waste a slash 24 on that node, so the user, what what the user can do is create a yet another cluster set or config. In this case, this is range four and then the user says I want to allocate only a slash 26 to the to these po to these nodes instead of a slash 24.. So in this case the user says part per node. Mass size is 26 and also populates.

B

The node selector logic to say only do this for these nodes, where instance type is small note that the ipc block is the same as range three, so what we are trying to convey here is that the 10.3.0.0 20 range can be chopped up into slash, 24 or slash 26. The back end, ipam controller handles this logic for you, that is the state of this world.

B

When we are recording right now, we are trying to work with sig network and get this proposal vetted out which bring us to brings us to future work with the new cube proxy changes. Anyone can now write an ipam controller and plug it in and do discontinuous parts, uh pod, sitters and certain cloud providers already do this, as well as certain cni also have the ability to do this. Our motivation for this project is to support this feature out of the box from for kubernetes, so that anybody can make use of this.

B

So we are actively working with sig network on refining this proposal. We are iterating on this and figuring out what the final design should be, and then the next step would be to implement, which leaves you with a call to action. Anyone who is interested in this or have ideas around how we should be doing this. Please join us in a sig network meeting. Please comment on the cap or even talk to us, and we will be happy to take your feedback and address any use cases that we may not have addressed.

B

That brings us to the end of our presentation. We thank you for your time. Thank you for taking the time to listening to us and if you have any questions, we'll talk in the q a session. Thank you. Everyone.