Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Did Kubernetes Make My p95s Worse? - Jian Cheung & Stephen Chan, Airbnb

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Did Kubernetes Make My p95s Worse? - Jian Cheung & Stephen Chan, Airbnb

When Airbnb first evaluated Kubernetes, they explicitly tested for performance and saw no significant differences. Then in 2019, as Airbnb’s migration of services from EC2/Chef to Kubernetes went into full swing, performance problems started cropping up. Service owners noticed significant latency increases which threatened to halt the overall move to Kubernetes. This talk will share Airbnb’s journey on performance gains and losses in its mass migration to Kubernetes. It will dive into the investigations Airbnb has done, from hardware differences, to cluster settings, to container configurations, to service language problems, and more.

https://sched.co/UaXm

A

All right, hello, everyone, the title of this talk, is a dip. Kubernetes make my p95 worse, quick, show of hands. How many of you think the answer is? Yes, all right cool! That's like half of you who are we hi, I'm, Jen I'm on a service, orchestration team at Airbnb, and we provide tools and guidance for engineers to configure and operate services of scale.

B

B

Here's a quick outline of what we're gonna go through today, so we're gonna start with a brief introduction about.

B

B

So we're gonna start with an introduction of how we migrated kubernetes onto communities. Urban bee then we'll dive into some specific, interesting cases of potential performance for questions on the services that we migrated and finally, we'll go through some of the lessons. Okay. So this is mostly about kubernetes, but also a little bit about containers that Airbnb so to set some context. We had a lot of services that used to be running on ec2 on their own individual instances.

B

So when we migrated onto kubernetes, we migrated onto both containers and kubernetes, so that I'll play out in some of the specific examples.

B

So here's a graph of our migration, so production migration started in earnest in the start of 2018 and as of September 2019. We're at around looks like a thousand services here. So we've we've seen a lot of migrations and got a few interesting cases and here's a little detail.

B

A few details about our environment that will come out later, so we're running on Amazon the next two on our minyan instances, we're running Ubuntu images and we're for our CNI plugin we're using Canal, which is a combination of calico and from that flannel we're making pretty heavy usage of new port services as a integration point. With our in-house service discovery system, smart stack and for our specific application services, we've got many different languages. A lot of them are in Ruby and Java, but we also got Python and go.

A

Alright I'm gonna start talking about the problems, then so during this graph right. There are both that lots of new services created in kubernetes and containers first, but there are also a lot of migrations from ec2 and those are more of the interesting problems because we actually have something for side-by-side comparison, because during that time we would actually start getting questions from our service developers. Asking me hey, why do I have late and she latency issues with like one specific pod, or why is my latency issue getting worse after I?

A

Do a migration and we get more questions about latency and why things are slow and just more latency issues and even more lessee issues, and this makes us sad because, like things are slow and they're asking in front engine areas like hey help us what's happening, this migration is like seems like it's not going so well, but let's actually dive into the actual problems.

A

Let's talk about case where, actually it seems like Layton sees might have improved here is a graph where, during the migration, Layton sees were kind of like went down actually when they were doing a side by side when they have a side by side deployment and actually, when they fully deployed. Let you see he's just like dramatically got better. So what happened here?

A

Well, what do you know about the service in Margaret, from ec2 to kubernetes plus containers? There were no code changes roughly the same amount. Cpu memory was a Java service, latency dramatically improved. Then the service was also spun up in early 2018.

A

For us, the biggest clue was that the fact that this service was spun up in early 2018 when we actually dug into it. This was actually a very common issue or a common. My surprise. The service was actually running on previous generations hardware, so the migration just so happened to have also upgraded to services hardware, and this was like one quote where the guy was like wow, it's just a better box with faster network and it's cheaper.

A

We've also seen cases where things got slower, because they were only super beefy hardware and now we've just kind of like moved them into our cluster, where we have like one type of hardware and for this example, the question is like the kubernetes make. My p95 is better I would say no, but we will tell our customers. Yes, if it got better and I say this, because I think hardware choices have to be made. Anyways and usually instance, types aren't intentionally picked to match the application.

A

Usually, the hardware is just like chosen in the beginning and people just like kind of forget about it later on. The lessons here are like that. The things that can be running on can be different for better or worse. So one example here is hardware right. They could have older hardware or even better hardware, and now this migration just so happens to have also change that other things that might get. You are like the host operating system, because when your service was running on its own box, it might have some OS.

A

But now all your kubernetes host minions are running on, might might be running on a different operating system.

A

All right now, I'm gonna talk about the problem of noisy neighbors. We've seen this case, where sometimes there's only certain pods. So for our service. They have like many many pods and it just so happens like two or three of these pods have extra high latency.

A

We have also seen this issue where latency they were running on kubernetes fine for multiple days and suddenly their Layton sees went up and it just happened. That became the new normal and then sometimes it becomes an incident where this is a sharp spike and all of our pods and they all have latency problems. So what happened here?

A

The hint is it's in the title as noisy, neighbors and explanation of noisy neighbors is a the idea that no you have multiple containers or apps or processes and they're all sharing the resources of a computer and there's only so much CPU to go around in this little example.

A

You know we have like three services like green, blue and orange one, and the green one is like definitely dominating the CPU time, and we can say, like this noisy service, great, this green, noisy services, it's just like just like taking up all the CPU time, so the blue and orange one is just not getting scheduled. Let's call this green service service kill.

A

So in this case we actually learn sometimes we're in certain pods. It was specifically pods that were co-located with a service scale and when it became constant, it was actually the same service service scale that migrated to the same cluster. Just that's also made issues where people are like well, can I get off this cluster cuz? This is no longer a good cluster.

A

We tell them no, and in this case, where it became an incident, it actually was not service killed. This was just us accidentally deploying a staging service to the wrong cluster and it just like got deployed through a production cluster and just took a lot of CPU, and now some of you are thinking. How can this be a problem like? Where is the isolation?

A

Well, in the early days of air beam being kubernetes, we actually decided not to set CPU limits because it seemed to have hurt performance. I would say, that's a bad choice and we don't recommend it out of Crowell study. How many of you do not set limits.

A

Just like one of you, a few of you, okay, I can see there being some use cases if you don't have a latency or performance sensitive applications for any RN cluster, but in general, it's like I would not recommend it because now now it's a problem for us and you think the fix is easy and simple right, like you just add two CPU limits, but that's not the case. So in this example, let's say you have a CPU request, a limit of like ten mill occurs.

A

So the idea is like how do you spend your 10 CPU quota?

A

Well, there's this thing: there's idea, like you have a CPU, say: FCFS quota of 100 milliseconds, and you can use your ten Miller Coors up within this 100 milliseconds and in the first case you could use it all up in like the first 20 milliseconds, but then you'd be throttled for the next 80 780 milliseconds and in second case maybe your CPU is actually like spread out over the whole 100 milliseconds and your application will actually be fine, and this matter is because in multiple use cases our metrics collector would just show similar CPU or low CPU utilization.

A

But we would have really fine-grained hot spots where services would just use up all of their CPU and then just get throttled.

A

So this means that most merchant metrics collectors it's really hard to detect fine-grained hotspots. Some things we've tried we're changing my CFS quota. It did I helped. In our case. We second thing: we tried, whereas a more CPU allocation and using a more finer, grained, CPU metrics collector. That was really useful because we were able to actually see what hey the service would show like really low utilization of a CPU requests. But when we got to find great metrics, we actually see all the spikes happening and three uh we actually set CPU limits.

A

Now other things we were actually looking into is like CPU, pinning or CPU sets or CPU managers there. But for us uh we found some open issues that didn't work with it for us, it's funny. uh Actually most of these open issues got there. There were recent movements like in the last two weeks, so I think that's like a coupon effect. We were just like finishing all uh open issues.

A

So in this case let's say, the question is like the kubernetes make my pee 95s worse and I would say yes because multi-tenancy is awesome, but it's hard to not take some performance hits from it and before applications were running on their own dedicated boxes, but now they're kind of sharing all the resources with other strange applications.

A

The lessons here are, containers should be contained and we recommend setting resource limits. Upfront.

B

Okay, so this is a continuation of noise and neighbors made slightly worse, like kubernetes, so here's a graph of a somewhat large service that we have and one day it decided to go from you know about 600, pods and scaled up to a little over a thousand right. So auto scaling is working, that's great and then on some of our hosts we saw you know no idle CPU CPU contention, but in aggregate the load was fine right, like the overall CPU utilization was just around 50%.

B

So again, a lot of scaling is working great, so we dug into it a little deeper. We found 18 identical service, pods running on a single host and for contexts like this was not a small cluster by any means, so we weren't trying to cram a ton of pods. So this was an unexpected result for us. So why did this happen?

B

So we're gonna go a little bit into the scheduler and talk about some of those features. So at high-level scheduler is broken down into kind of two logical sets of components. One is like filters which explain where your pod can or cannot, run or must or must not run. So some examples of filters are resources so like if your node has like 10 CPUs, it's already been filled up with like 9 requests. It can only fit on max like one additional CPU right, simple and then there's also topology.

B

So if your service pods have a like required affinity to a certain like a Z or a certain node and then other kinds of like anti pod affinity and then after a filtering, there's scoring of the remaining eligible nodes right, which is kind of saying: where should my pod run or where should it not run so some examples of that are preferred affinity spreading by topologies so like if you're running a cluster in multiple ICS, which is what we do, it will try to spread pods evenly across multiple ACS and spread them evenly across multiple nodes.

B

Finally, there's one scoring priority: that's called image locality right so that one basically says if you've already downloaded an image on two one, minion node, it's your pods are more likely to schedule on to that. Minion node and idea is usually to have all these priorities kind of counterbalance each other and offer like a reasonable compromise, but in our case that services had one gigantic image, so image locality was really out weighing all the other priorities.

B

So basically the the tendency was just to pile up on one single node and to make things a little bit worse. This service was also not doing its requests and and limits quite quite correctly, it was using more than its requested CPU and it didn't have any limits so yeah. So back to the the question, did kubernetes make my p95 worse?

B

Definitely yes, in this case, so lessons here are that the scheduler can work against you in some pathological cases, for example like if you have really large damages and something that we're interested in trying is the pod topology spread constraints to limit exactly how many pots we can run per node crazy, but we haven't tried it yet, okay, some more lessons.

B

One thing to mention here is like we didn't quite get into is like. Why was why was CPU and aggregate so good? Why was it like evenly spread at 50% and, like some hosts, were totally starved for CPU, so we'll get into this a little more later, but basically kubernetes services can cause traffic imbalance when you're, using like the iptables, proxy, ER and also another lesson, is that auto scaling uses the average CPU across all pods. So this can even cause some pathological behavior.

B

For example, if some of your pods are let's say unreachable, then the aggregate CPU will drop and then autoscaler will scale down the deployment further and then maybe more pods will become unavailable. But we haven't gotten that one in this presentation.

A

All right now we're going to talk about use case of write once run anywhere.

A

This was a stock or internal Stack, Overflow post, where it's like hey my lady's got worse, but a key points is this: is a java application p95 went from like 30 milliseconds to 100 milliseconds p99 s well through yet 2x, and it was specifically only for DB connections, so not their servers to service connections.

A

Here's a close-up graph of like what happened with it latency, and this is really mysterious for us because, like we're, probably a hundred service service, my questions, deep, all other services seem to be running, steady and fine and we're thinking alright. This is probably just another missing configuration by the application, but this is indeed interesting, because it's only DB connections is it's not like others. It's not service to service connections.

A

When we actually dug into it the problem was we learned that for a specific endpoint when connecting to databases, we have created a new thread, pool per requests and when we actually fix it by reusing a thread pool in a static context, it did fix the performance issues, but the question is still: why did this work before kubernetes?

A

And let's go back to you explaining like how Java services work like if you were running a java application in a in a node all by itself, the JVM might tell your application: hey I have like 36 CPUs, like that's great now, you like put it in a pod. This still works, but I saw this slide for my colleague.

A

But now, if you have like three of these pods all running on the same node, each of them think they have 36 CPUs, and this is a problem because that's how Java kind of Tunes itself. So this is actually a was an open bug. Older versions of Java just were just not container aware, and this is important, because Java Tunes itself, based on how much resources like CPU cores it thinks the system has, and it affects things like how thread pools work.

A

This is fixed in a Java version and there are lots of posts on this. If you just google for it. So for us, when we actually like move to the new JVM performance did dramatically improve. We also tried like this other flag, but it didn't have much of an effect so so here the question is like there could mean that I used to make make my performance worse and after I would say.

A

Yes, because the containers promise of build once run anywhere, isn't 100% accurate languages and applications can have deeper dependencies on the underlying systems that they run on, and some of you might say like well. This is more containers, not kubernetes, but for us we consider it all as like one whole ecosystem during the migration way. Moving from raw application living for applications on raw host nodes to containers and kubernetes.

A

That sort of lessons here are yeah languages and applications have deeper dependencies on the underlying underlying systems. You should upgrade your systems to be more container aware and for us I would say we were very lucky that we had a migration, because having a baseline can be very enlightening. Lots of services were actually created in kubernetes first and the, and this meant like they were blissfully unaware of the performance gains to be have to be like have because we had this migration.

A

We like realize this is a performance problem and then we were able to like tell all of our Java applications. Hey upgrade it and you might just see a dramatic performance improvement.

B

Okay, so now I'm going to get a little bit more into traffic and balance details the horrors of IP tables, okay. So this is one another question from internal stack overflow. It says this service has a lot of pots. Sometimes some pods have markedly different QPS distribution. You can see that on the top graph, some pods seem to have double to keep es of other pods and, of course that means they have higher latency. So let's get a little bit into the details of kubernetes services work.

B

So, like I said earlier, we use node port services as a bridge between our smart stack service, discovery system and kubernetes. So that means that you can talk from surfaces in ec2 to services running in kubernetes. You can talk between different clusters and that usually works great right. So a little bit about no port services.

B

Right like when you your hit the node port on the the instance has IP tables and it will rewrite from the node IP and the node port into the pot IP and the port, and the important note here is that pod, a in this example, is running on an overlay network. So it's a different IP space.

B

Okay, so what happens? If you have two pods of the same service and the traffic comes in again from the new port IP and the port port, so hits IP tables, and then every rights dive you import and then has to choose from two of these pods now. So how does it choose them? Well, there's it uses essentially random allocation and that's a problem for us, because you can't spread load very evenly with random low allocation.

B

Its IP tables is it's not meant to be a load balancer and also, if you have two pods on the same node and one pod on the other node they're likely to have different traffic levels. So in the example of the service that I gave earlier, some knows there were multiple pods and then others. There was a single pod, and so this is kind of the pod colocation problem that ends up with with traffic in balance, so kubernetes make might be 95s worse.

B

Maybe so traffic imbalance caused variable load and latency, but it certainly made latency is more unpredictable.

B

So some of the lessons are that when we first ran kubernetes for started migrating into it, we were running an overlay network. So that provides a lot of flexibility right. You have, you can do infinite IPS. Basically, you can have them running in the same cider range. You can also add like network policy, but that adds a lot of complications and also IP tables load. Balancing. Is it's not ideal right?

B

You don't have like round-robin, you don't have like any other advanced load balancing strategies, so some alternatives that you could use are using envoy for for balancing between poppies. You can also use a cloud provider native IPS to avoid the overlay.

B

Also, you can use IP vs as your through proxy proxy er. If it were, if it works for you, we found some small open issues that caused us to stick with IP tables. For now, but the one that were most strongly considering moving towards is using the cloud provider native IPs.

B

Okay, now we'll talk about keep DNS slowness, so here's another service, latency graph, so on May 10th everything was fine, saw a couple of spikes and latency may be up to like one second, and then it went to five seconds and then 20 seconds so and it stayed there for quite a long time, and this.

A

Was not unique to one.

B

Service right, so these are multiple graphs across multiple services. Right, there's, high latency elevator right. It's like build health checks. So what did these services have in common? They all relied heavily on DNS and made a lot of external DNS queries.

B

So this is once we dug a little deeper. We looked at cube, DNS load right and we saw that some q, p NS pods had pretty high. Keep yes and some pods had rather local. Yes, so traffic imbalance strikes again. So let's talk about how cute DNS works. Now so, let's say you're potty running in your note, you make a DNS request.

B

Again, it's going to hit iptables this time, it's going to use the cluster IP of cheap DNS, and so it's gonna balance across multiple pots and again it's going to do so randomly and so that causes traffic imbalance and then, in this case we hit 80s DNS packet per second limit, so that caused slow fancies and timeouts for multiple services.

B

So did kubernetes make my p95 source. Yes, so the lessons are by default. Dns is discovered through cluster IP and you also by default. Your pods use. Dns policy of cluster first, which reaches out to keep DNS. So even if you're, making like external DNS queries that aren't like the like within the service address URLs, then you're gonna go through ups first. So if you don't need to burn ith DNS resolution, you can set the pod DNS policy to default or not.

B

If you want to do like further customization by like using more fine-grained parameter tuning.

A

Alright, for the recap, so, let's tally up the scores or, let's ease improved yeah I was like kubernetes did not make it better, did not make the performance worse, but take the credit if it improved, because there's so many underlying systems that affect performance like hardware or hose os, cetera and then noisy neighbor cares like yes and some of the lessons are cetera limits and be wary of how CPU is actually counted.

A

Noisy neighbors made worse by a kubernetes like yes, you should try to loop, maybe like look into how priorities and predicates are affecting how your posit getting scheduled and for write once run anywhere. Let's say yes, ish, like obviously more of this was because of the move to containers they're, just like gacho's, when you move applications to a containerized environment, and ideally you can just like fix it by upgrading your applications and languages to be container aware.

B

For traffic and imbalance, maybe be wary of IP tables load balancing again, it's not a little balancer okay and then cube dns sonís kubernetes make my p95 source. Yes, so check your dns policy early and often make sure you're using the right one for use case. A few other takeaways so like the overall message right is that performance includes tuning layers. Have all layers of the stack kubernetes has a lot of layers in the stack there's a cluster like container application language network, a few others right and before you do.

B

Your migration set the expectations that some small performance differences. It can't happen as long as you have a baseline right. You can measure to see exactly how much worse, how much better just keep tracking and run side by side for a little while and then fix any like major regressions that come up.

B

We're all so sorry, so learn more at our medium Airbnb engineering, blog posts or visit airbnb.com slash career thanks.