Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: What Kind of CPU is it Anyways? Airbnb's Journey to Heterogeneous... - David Morrison & Evan Sheng

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

What Kind of CPU is it Anyways? Airbnb's Journey to Heterogeneous Clusters - David Morrison & Evan Sheng, Airbnb

In this talk we describe the technical and organizational hurdles Airbnb needed to overcome to migrate from running "homogeneous" Kubernetes clusters (i.e., clusters in which the majority of nodes are the same type) to "heterogeneous" clusters (i.e., clusters in which pods can be scheduled on a variety of different node types). Why did we make this change? Two reasons: cost and efficiency. Restructuring our clusters to support multiple different node types unlocked the ability to run workloads on the best machines for that workload, not just whatever our "default" happened to be. However, getting to this point wasn't easy. We'll describe in this presentation changes that were required in almost every part of our infrastructure, from changes to the ways we provision and scale clusters all the way down to changes in the API that our customer teams use. We'll also discuss the organizational hurdles that we had to address to build confidence in this new operating model.

A

um Thanks thomas for the introduction, uh thanks, everyone for being here um hope you all had a good lunch, also uh welcome to everyone who's online, uh happy that you're able to participate as well. um Like thomas said, my name is david. This is my colleague evan we're both software engineers on the compute infrastructure team at airbnb.

A

um We're going to be talking about uh how we moved how our kubernetes clusters have been evolving into heterogeneous clusters over the past several years, quick outline of our presentation, evan's, going to talk a little bit about where we were before, give you some historical context on what our clusters looked like two three four years ago and we're going to do a deep dive into uh three specific problems that we encountered. We encountered a lot more than three, but we picked three that we thought were sort of the most interesting.

A

These are both sort of technical and organizational hurdles that we had to overcome on this journey. um I guess the other thing I'll mention here is this journey is ongoing. Some of the stuff that we're talking about is done. Some of it is in progress. Some of it's not complete yet, but I think we've got a good plan at least, and then lastly, we're going to give you some future plans and lessons learned.

A

The tldr here is that heterogeneous clusters are great they've, already been instrumental for airbnb and for our team. So with that, I'm going to hand it over to evan to talk a little bit about some historical context. That's great.

B

Thanks david yeah, as david mentioned, I'm going to start with a brief history of time of kubernetes at airbnb in 2016 and 2017. We started evaluating kubernetes for production use.

B

Previously we were running chef on aws ec2, where each replica of each service would have its own machine in 2017 and 2018, we started building one touch, which is airbnb's abstraction layer on top of kubernetes for developers.

B

The philosophy here is that all of a services config lives in one place in git, and this config is then converted and applied as kubernetes manifests to the cluster um in 2018 2019. There was a huge effort across our engineering org to migrate uh 90 of our 700 plus services at the time um to kubernetes.

B

uh Initially, our clusters were separated out by environment, so we had a prod cluster, a staging cluster, a dev cluster, etc, but this quickly grew out of hand as we hit kubernetes single node single cluster node size limits and we're forced to split these clusters into different cluster types. So we had uh we, we split our prod cluster into multiple prod clusters, our stage and cluster, into multiple staging clusters, um and these clusters happen to be comprised of a single instance type so single instance type clusters.

B

This is what a single instance type cluster looks like. We have a pod come into a scheduler and the scheduler tries to schedule a pod on a node with enough resources. Initially, we only had c5d 9xls as our node type, because this happened to be the latest one. We were using in ec2.

B

And this was enough at first, but as more and more workloads started migrating, some specialized workloads required different instance types, so uh gpu workloads started migrating, we're forced to create a gpu cluster type workloads requiring a lot of memory.

B

We created a high memory, cluster type for and these cluster types were identical in both setup and config and allowed us to expand them horizontally, um yeah. So why single instance type? um This was before we had cluster autoscaler, which allows you to dynamically scale cluster size based on node capacity.

B

um Without this multi-instance clusters, multiple instance type clusters were not yet feasible. Any scheduling and capacity errors we ran into. We would have to resolve manually, which was quite a pain, because we'd have to look at the different pod requirements and manually determine which node types would fit. Those scheduling constraints.

B

Additionally, no cluster, auto scaler means a lack of automation. We were at the time manually scaling all of our asgs when we got alerted, which was not great yeah. So back to the timeline in 2019 and 2020 as more and more specialized workloads uh migrated, we had a cluster type explosion and we went from around five clusters to over 90 that we have now and uh we have over 30 cluster types.

B

This leads to a lot of painful operational overhead for our team. Here's a screenshot of some 700, almost 700 alerts that we have just for asg sizes, which really isn't great um yeah, and this is our attempted solution to fix this uh multiple instance type clusters um which look like this. um So we have the same thing as before, where a pod comes into the scheduler.

B

But this time the scheduler has many node types to choose from um it can schedule it on any node type that can fit the pod requirements of, in this case, the four that it has.

B

But this time we also have cluster auto scaler, which handles the complexity of managing the asg sizes and it scales up the certain node types uh when required by a pod, but there's no capacity in the cluster um so yeah. Why did we want to migrate? Well?

B

First, it brings our team quite a lot of quality of life, improvements, reducing the number of cluster types to generalize clusters and make it so that we only have one process for cluster creation is really beneficial for our team and it also reduces the number of alerts and alert fatigue.

B

um Secondly, we wanted to be able to control cluster composition at a global level.

B

It unlocks some cost savings for us when we're able to utilize more node types rather than being locked into one gives us some contract flexibility with aws in the future. We are trying to utilize spot instances, so we can run part of our clusters on spot and also new instance generation upgrades we get pretty much for free in multi-class the multi-cluster setting. Historically, this has taken us years to upgrade our entire fleet across new instance. Generations um yeah and the next section is solutions and other problems.

B

um I'm gonna pass it back to david to talk about the first one.

A

Thanks evan, um so in this section we're going to talk about three of the major technical and organizational hurdles that we had to overcome uh to do this migration to heterogeneous clusters, the first one I've entitled in which nobody knows what's going on specifically, this is going to be focusing on cluster autoscaler because it turns out that we didn't know what was going on with cluster autoscaler when we started this process.

A

One of the key questions that all of our stakeholders asked us is: how do we know what's going to be running in our cluster, like? How do you know what mix of instance instance types you have? How do you know where things are going to get scheduled and we thought we had the answer to that.

A

So this first diagram here is not actually what happens, but it's what we thought happened. So a new pod comes in and cube. Scheduler looks at the pod specifications, the resource requests etc, and it's going to do one of two things in the first case, which I'm calling the happy path.

A

There's a node somewhere which meets all of the requirements and cube scheduler just binds the pod. To that note, I'm not going to talk about the happy path anymore. The unhappy path. uh There are no nodes in the cluster where the pod can run and so cube. Scheduler puts the pod into a pending state.

A

It marks it as unschedulable and cluster. Auto scaler then monitors the list of unscheduleable pods and it's going to try to spin up new ec2 instances that match the pod specifications. It looks at things like the resource request. It looks at node, selectors, node labels looks at things like the pod, topology spread, etc.

A

It's going to iterate through all of the different node groups that it has and it's going to select one that matches the pod specifications and it's going to spin up a new host from that node group. So in our setting, node groups correspond exactly to asgs, because in a node group you have to have all of the nodes identical, so they have to have the same cpus. They have to have the same memory.

A

uh This goes down to the labels and everything that are applied to nodes in that group as well, and so we have a different asg that corresponds to each different instance, type that we want available in our cluster, so anyways once a cluster autoscaler spins up that node after it joins the cluster cube scheduler will then see. Oh look, there's a node that can accommodate my pod and it binds the pod to the node, um nice and simple right.

A

Cluster autoscaler is making all these decisions about which node types to spin up, and we thought great, we'll just hook into that process. It's not quite as simple as we thought this diagram here shows. What's actually going on. You can see that I've added a third middle path here, which I've entitled pause, pods evicted.

A

So what is a potus pod? You might be asking well, it turns out that, for a variety of reasons, we want to be able to maintain additional overhead in our clusters.

A

This is useful in the event that we have a spike in traffic or we have some sort of an outage and we need to fail over a service from one cluster to another. Having this overhead gives us the ability to handle those bursts in traffic seamlessly.

A

The way that we maintain this overhead is through something called the cluster proportional autoscaler, which is different than the cluster autoscaler. What cluster proportional autoscaler does? Is it spins up pause, pods, a pause pod is exactly what it sounds like.

A

It just sits there on a node doing nothing, but it reserves that extra capacity, all the pause pods are marked with a lower priority, so we've got our regular priorities and then pause pods have a lower priority so that when a new pod comes in cluster, scheduler will look at that it'll say: oh here's, a node that has a lower priority, it'll evict that one and it'll make room for the new pod that just came in.

A

We did a little bit of data analysis here and discovered that this middle path here is taken approximately 98 of the time in our clusters, which is great, it means that uh the proportional, auto scaler is doing its job. We have enough capacity 98 of the time to handle our bursts and traffic.

A

So what's the problem here well in a single instance, type world. The way that we configured things is we applied a instance type specific node selector to our pause pods. So that means that we're running c5d 9x larges, our pause pods, could only spin up on c5 d9x larges.

A

This is for a variety of historical reasons and when we moved to a multiple instance, type world that started causing us problems, because then we'd create new pause pods for uh m5d, whatever whatever instance types, and I'm sure you can see where this is going. When the new pod comes in it evicts a pause pod and gets bound to the host. Where the pause pod used to be running the pause pod, then transitions into a pending state and cluster auto scaler says: aha, here's a pod that can't be scheduled.

A

Let me try to spin something up to satisfy all of its requests. It has this instance type specific node selector on it, and there's only one node group that can satisfy that.

A

So cube scheduler doesn't know anything about uh different instance types. It doesn't know anything about asgs or anything like that. It's more or less picking pause, pods to evict at random, um and what that means is that cluster autoscaler uh is entirely constrained in its choices by our pause pods. um When a pause pod gets evicted.

A

Cluster auto scaler is forced to launch a new node that exactly matches that pause pod. So this was kind of a problem because it meant that we had all of these interacting control loops. We had cluster autoscaler, we had cluster proportional, auto scaler. We had cube scheduler. All of these things were, in some sense fighting for control over our cluster composition, and this wasn't great. So what do we do about it?

A

On this slide, uh I show a slight modification to the previous slide. I'm going to highlight the differences here. um We ended up identifying two things that we could change. um The first is that we moved away from these instance type specific pause pods to what I call generic pause. Pods I'll, explain that in a moment- and the second is that we built a custom expander plugin for cluster autoscaler, so, let's dive into what each of those are.

A

So, as you might expect, uh the solution for pause pods is pretty straightforward. You just remove the instance type specific node selector. So now we have what we call generic pause pods they can run anywhere. This is great um in terms of sizing. uh Previously we had the pause pod just take up the entire node. uh Now we're going to size the pause pods to be the smallest of the instance types in our cluster.

A

So if we've got node types that have 32 cpus, we've got node types that have 64 cpus, we'll size, the pause pods to request 32 cpus.

A

This means that, well maybe a bigger node might have multiple pause, pods running on it, but that's fine. We can tweak our overhead parameters to accommodate that now.

A

The only sort of remaining concern here is: what do we do about our slas around scheduling, as you recall, before 98 of the time we're evicting these pause pods in order to make room for new workloads that are coming in and what that meant is that, most of the time our services are getting scheduled in one to two seconds, and we really wanted to make sure that, as we move into this multi multiple instance type world that we're able to maintain that, and this becomes a problem.

A

If we start consolidating like our gpu clusters or some high memory workloads, we try to mash all of those different things into a single cluster. While you might have a job that comes along, that's requesting 500, gigs of ram and our pause pods, none of them request 500 gigs of ram, and so it might have to wait that job might have to wait.

A

You know five minutes or 10 minutes or, however long it takes for a new node to come up and join the cluster, and this could be a regression in performance, so the solution that we've identified. This is a piece that we haven't implemented.

A

Yet is that for jobs like gpu workloads or high memory workloads we're going to build in a shadow capacity mechanism so that if you've got a specific service that has a unusual resource request, essentially you're going to get pause, pods that are specific to that service, so that you can maintain those scheduling, guarantees.

A

All right, so, let's move on to the second piece that we are introducing. This is our custom expander plugin. This is in progress. We have a really basic proof of concept right now, but we expect this is going to get a lot more complex in the future.

A

What we observed is that cluster autoscaler is performing a lot of common tasks and we really want to keep all of that functionality in cluster autoscaler. It does things like talk to the cloud provider. It runs this simulation where it determines all of the different places that a pod could go. Does a lot of really nice things for us.

A

What it doesn't do necessarily is capture the business logic that we want to determine what node types to spin up. So how does cluster autoscaler actually pick a node type to spin up? Well, first, it does this filtering step where it identifies all of the node groups that potentially could run the incoming pod. Then it passes those node groups off to what it calls an expander.

A

There are a number of expanders that are built in there's one, that's just the random expander. It's going to pick one of those node groups at random. The one that we use right now is called the priority expander, so you rank all of your node groups by priority and it picks the highest available priority.

A

It's got a few others, but what we predict and what we're already seeing is that we're going to want even more control over the node groups that get selected.

A

What we didn't want to have to happen is that we maintain a fork of cluster autoscaler that has all this business logic and we wanted to be good, uh cncf citizens and be able to contribute some stuff back upstream, and uh we also wanted to make our lives easier, and it turns out that I think, with this custom, expander plugin, we can do all three of those things.

A

So what we're going to do is patch cluster, auto scaler so that it can communicate over grpc with a separate process and that separate process is going to encode all of our expander logic. It's going to be all of the business logic around what instance types we currently want to run. As evan mentioned.

A

We want to start dipping our toes in the spot world, and so this custom expander can do things like look at all of the prices for all of the spot markets that we're in and let's maybe pick one that's cheap right now or that we think has a lot of availability.

A

um It can do things like uh well, we've got a lot of extra. uh You know. We've got a whole bunch of um batch jobs that are going to be requesting a bunch of memory, so maybe for like bin packing reasons. We should request some instances that have more memory or something like that. All of that business logic can get encoded in this expander. That is separate from cluster auto scaler itself. This is nice for a couple of reasons.

A

It means that a we're not maintaining a fork of cluster autoscaler and b. We can upgrade the two out of band um we don't have to if we, if our aws contract changes- and we need to fix some of that business logic, we don't have to deploy a new version of cluster auto scaler. We can just deploy a new version of this service that we're running so. Like I mentioned this is in progress.

A

uh We have a work in progress, pr that I'll link at the end for this expander, but this is going to really be where we control the composition of our clusters, so that I'm going to hand it over to evan to talk a little bit more about some of the organizational issues we've encountered.

A

B

Thanks david yeah, so david covered, some of the technical.

A

B

This section will be a bit more around the organizational issues. This chapter is titled in which customers are concerned so because we're coming from single instance, clusters, uh service owners had the expectation of identical pod performance across different machines. Another issue we thought of was that our our service owners have a lot of migration fatigue. uh We currently have nearly a thousand services and with every infrastructure, config change leads to migration, and every migration causes uh wastes a lot of engineering time.

B

Our first solution here was talking to people we sent out surveys and talked with a lot of customer teams.

B

We realized here there were actually quite a lot of nebulous concerns around performance without the teams, knowing exactly what requirements they had for their service around performance and most of them hadn't really done any performance, testing or performance tuning either. They only came to us with sort of like relative comparisons to their current performance. Whatever that performance may be um out of these talks, we created an extensible api out of which was kind of two-fold.

B

It allows customers to specify specific instance types if their workloads are actually super sensitive to to the different underlying node types, and it is also opt in which solves the um or the the default is opted in meaning service owners would have to opt out. um This solves a lot of the migration issues, because we expect this to work for all almost all services.

B

It also allows for additive requirements when service owners are describing the instance types they want and is therefore extensible for our future work, like I said before, like utilizing spot instances and others um to try to address customer concerns, uh we did a bit of performance testing for these new instance types. We started with running some benchmark tests, so we ran a variety of sith sysbench synthetic tests and found performance uh with uh between the instance types to be within 20, but obviously this isn't good enough for production workloads.

B

We needed to use real workloads to test this, so we took services test environments with production traffic, replay scheduled them onto different uh empty nodes of the uh all the instance types we were testing uh to compare the latency metrics, and we did this for a representative set of service profiles that we have.

B

This was across different languages, so, like java, ruby and node latency requirements, slos and across different workload types. So we had some like cpu intensive services. We had some, I o bounce services.

B

Here's an example of a very latency sensitive service that we have the different colors here. Orange and purple represent the different instance types and each grouping is the p95 and p99 of the latencies. You can see they're like really really similar.

B

We've actually found this to be the case for all of our services and all end points. We we found that performance was within 10, which we deemed acceptable enough to move forward um and yeah. So, as we started roll out, we found quite a number of very very technical issues with cluster autoscaler.

B

We don't have time to cover them all in this presentation, but we'll deep dive on one of them and then link the bug, fixes and issues we filed upstream at the end, and I believe these slides are uploaded. So you all can take a look if you're interested um so yeah. The problem we saw was that during our rollout we observed that certain asgs weren't launching this was kind of strange, because the logs, when we checked them, showed pod topology spread failures.

B

If you're not familiar pod, topology spread, all it does is guarantee that pods are spread evenly across availability zones.

B

Here's a screenshot of the logs you can see if that's not too small, that the m5d12 xl asg's in us east 1b, fails pod topology, spread constraints and then later on in the logs, the c5 ad-12 xl in the same zone ends up succeeding and uh being chosen to scale up. um This is really weird, because it's exactly the opposite of what pod topology spread is uh trying to do, um and then this graph shows our uh gradual introduction of these new instance types to one of our clusters.

B

um You can see these c580 12xls in the top in blue, gradually taking up a larger and larger proportion during our rollout, but c5 ads, 16, xls and m5ds aren't launching at all.

B

So after a while, we realized that the asus that weren't launching were empty asgs.

B

In this case it was the yeah, the m5d ones and the c580 16xls- and this was kind of strange to us, because both of those asgs happen to be the very top priority in our expander ladder and we'd expect them to launch before the other ones.

B

So this was kind of strange. We did quite a lot of code digging and we we learned a couple things about cluster, auto scalar.

B

So what's going on, um we have a cluster here with the four asgs, two of which are populated the c5d 18xls and the c5 ad-12xls, and then the two on the right are empty and we also have cluster auto scaler, which adds nodes to a cluster when incoming pods come in and there's no capacity.

B

um Cluster autoscaler undergoes a scheduling simulation when this happens and looks at possible node types to expand and sees uh if the node types will fit the scheduling requirements of the incoming pod.

B

So how does this scheduling simulator simulation work when evaluating if a pod will fit well, it looks at the available node types it has in this case the four on the bottom.

B

And it creates these fake nodes that populate um node info objects for each type of node, and it sees if this fake node will fit the scheduling constraint.

B

um But that begs the question: how does it populate this nodeinfo object? Well, it turns out there are two paths uh for asgs that have live nodes in the cluster. It copies the node info directly from existing nodes, in this case the two on the left, but for the ones that are empty.

B

We have these things called asg templates, which, whenever cluster autoscaler chooses to launch a new node. It takes this asg template which describes the node config before cubelet starts on a node.

B

This seems fine right, but I mean in most cases it is fine, but kibla actually applies quite a lot of labels and annotations to a node on startup, including architecture, os, the host name, the instance type name, but then, most importantly for the scenario, the topology key topology zone key.

B

This means that asg templates don't have that topology zone key, which explain the behavior. We were seeing because the fake notes that were copied from the asg templates don't have the topology zone key the ones from the existing nodes do have that key, because cube has already started on those existing nodes, and thus we see our behavior in the logs where the existing nodes, even though it's a lower priority than the than the other ones that were empty, is chosen.

B

Our short-term fix for this is, to just manually, add the topology zone key to the asg template. This is pretty counter-intuitive behavior because you wouldn't expect to have to add that, because cubelet does it for you, the long-term fix here is to modify clusterautoscaler to automatically add all of those cubelet tags to the asg templates. When it's evaluating um yeah so david, you want to talk about what we learned sure.

A

So, let's see what did we learn.

A

Testing there we go cool yeah, sorry about that, thanks evan. um What did we learn from this whole process? Well, the short summary is that heterogeneous clusters are really great. uh We should do more of those, um but you might be asking what's so great about them. Well, I think the biggest benefit that we've gotten from all of the work we've done so far, we're still kind of in a half-finished state and we're already seeing this benefit is. It gives us a huge amount of flexibility in our team.

A

We've gone from, as evan mentioned, like taking years to migrate from older generations of ec2 instances onto the most recent ones to being able to do that with a single config change in a couple of hours, even earlier this year it took you know several weeks to even just be able to add a new instance type into our cluster and again, that's now a couple hour task at most. So it gives us a huge amount of flexibility to respond to changes in the in our demands in the cloud provider markets, um you name it.

A

uh We can now handle all of these different uh changes without really overburdening our team, so that's great um second benefit is also. This allows us to future proof our systems a lot more, um and this really boils down to just being able to add new instance types as aws makes those available.

A

The third thing is really around cost. um Historically, we just picked an instance type and went with it, because that seemed like the thing to do now. We can actually like critically analyze. Are we getting the most bang for our buck with the instance types that we have in our cluster um evan, and I have hinted several times that we want to also start incorporating spot instances and all kinds of other stuff we're investigating amd we're investigating graviton. All of these different things.

A

We're now going to have the capability to make these trade-off decisions um where we wouldn't be able to do that before so. What's next, um we still have a number of changes to cluster autoscaler that we're trying to get upstreamed I'll have a list of that a list of those on the next slide, we're still in the process of rolling this out. As well, like I mentioned, some of these things are still in progress.

A

um We only have a handful of our you know: 50 60, odd clusters running uh in a multi-instance type setting right now, so we want to roll this out to the rest of airbnb and then our next big project after that is running on spot instances. This was really a prerequisite to that and yeah. So this is a list of existing prs and issues that we filed against cluster autoscaler.

A

These are on the slides that are uploaded. I think there's going to be like one or two more that are coming, that we just discovered in the last couple weeks, um and with that I think I will conclude. I want to thank everybody again for attending um I'm happy to take any questions or evan any questions that you all have um and just want to put out a shameless plug that we are hiring. So if any of this stuff sounds interesting to work on, uh you can come talk to us after the presentation.

A

So thanks very much.