Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 10 More Weird Ways to Blow Up Your Kubernetes - Jian Cheung & Joseph Kim, Airbnb

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

10 More Weird Ways to Blow Up Your Kubernetes - Jian Cheung & Joseph Kim, Airbnb

Over the past couple of years, Airbnb has standardized microservices, containerization, Kubernetes, and service mesh in our ecosystem, and have built tools around it to make our workloads more robust. We’ve seen great results with thousands of services now aboard Kubernetes, but it wasn’t without mistakes and painful/insightful learnings along the way. Airbnb is back at KubeCon again to share 10 more ways we’ve messed up in just the past year -- come discover what can go wrong as we share some of our experiences running on Kubernetes. Favorites include: - “What do you mean CPU Limits caused OOM-kills?” - “Half the pods are gone and this was broken for 7 days?” - “Yes, things do break when all the masters are down” And more!

https://sched.co/ek9r

A

Hello: everyone, this talk is titled 10, more ways to blow up your kubernetes by airbnb, and this talk is titled 10, more ways, because we have found more ways to blow up since last year's talk that was presented by our colleagues melanie and bruce.

A

So uh who are we? My name is jin, I'm a software engineer at airbnb on the compute infrastructure team. This is a team that manages our kubernetes clusters and manages the abstraction we have over it to make developing services easier.

B

Hi, my name is joseph and I'm on the compute infra team, with gin. I've been on this team and at airbnb for about eight months now. So the outline.

A

Of this talk will be one. We're gonna, give a brief, intro of kubernetes at airbnb, uh two we're gonna dive into ten cases of how we actually uh mess things up and three uh we're gonna go over and do a recap all right, kubernetes and containers at airbnb, so airbnb started our migration to kubernetes from the beginning of 2018, and since then, it has really taken off. uh Kubernetes is actually running in production and we rely on. It is a critical part of our of running airbnb.

A

Our environment is mostly amazon, linux 2 for the actual host. We use ubuntu images we use canal for our cni plug-in. We do use node port services and smart stack for service discovery, and we support many languages, such as ruby java python and go services.

A

Now, let's dive into the real fun stuff. The first case we're going to talk about is called replicating replica sets so for context. We have a job that regularly scrapes our clusters and it runs a command. Something like cubes detail, get pods all name spaces.

A

This job starts getting out of memory errors in one specific cluster, so let's check it out uh and what do we see? We see that in most of our normal clusters, we have maybe on an order of a thousand to two thousand replica, sets any guesses to how many this problematic cluster had it was 56 000..

A

So, let's see what else is up with this cluster when we dive into all these replica sets, we notice that all of these are actually under one deployment, one name space and when we dived deeper into it. We also noticed that the deployment object when we checked its collision count. It was up to a hundred thousand and so right now our theory is likely that the deployment object is creating replica sets that aren't getting adopted by it, so it keeps on creating more as we dive more into this incident.

A

uh We realized that something different about the deployment of this job on this specific cluster that was different from its deployment of other clusters. Was this one? We were actually testing this new feature, called topology, spread constraints and what we noticed it were that these specs were actually not getting picked up by the replica sets, and the theory here is because the replica sets being created by the deployment didn't have these specs. uh The deployment didn't know that it had already created these replicas.

A

That's it didn't know that these replicas belong to it, and so it would just keep on creating more and more replica sets and good thing. We have so many. We had so many test clusters, so we could actually like diff the specific deployments to see what was going on and in this specific incident the actual fix was simply just deleting the whole namespace, which is fine, because this was just a test cluster and good thing.

A

We were just testing this feature out in our test cluster, and so the takeaway here is: do you test new features and test clusters? Because for us? That's what they're, for we were testing this new feature and what we noticed was it kind of blew up our cluster and just kept? Creating infinite replica sets, but not a problem. uh We know to investigate more before we roll this out to production.

A

Our second incident is called mutating time bomb and what we see here is we have the service experiencing high error rate and the service cannot deploy and we get this cryptic kubernetes error, saying uh unable to access invalid index five, so we dive deeper into it and check the status of it. And what do we see? We see that two two pods are running at what hpa thinks there should be nine replicas running and the service is panicking because well, we only have two pods left digging more into it.

A

We notice um there's only one deployment object, but three replica sets which is really weird and the replica sets that actually have pods are 16 days old, as we dug more into it. We noticed uh yeah, there's something really wrong going on with the hp here and the ruffle cassettes and one hour later, what have we tried? Well, we tried deleting the hpa and manually scaling it up, because the theory was perhaps there was something wrong with the hpa uh that did not work. We tried deleting those old bad replica sets.

A

Thinking perhaps it was those old replica sets that were blocking new deploys from happening, but that was not the problem. We also made sure not to delete the replica set that had our last two pods running.

A

We also tried creating a new deployment object just to update it, just to kick it and see what happens that did not solve our problems and we thought about perhaps just recreating the namespace, but that would definitely delete the last two pods running, uh which is no good, because this service is production.

A

So what do we do now? We page a teammate. We call a friend one hour later, and so when we started digging more into it, we come back to the first page where someone had called out this cryptic error message, and we should have done this earlier, where I should have dug more into it, because we realized that this was actually the exact problem. When we googled for the problem, we noticed it was actually a problem with json patching and related to our mutating emission controller, which was recently rolled out in the last month.

A

uh Specifically, the bug was this with our mutating emission controller. Imagine that on the top left is our input, and this is our spec and our desired output is the spec um at the bottom left. uh Here we see we are deleting two items from an array.

A

What the bug was the generator output is this: uh we it generates two operations. uh It removes index two and removes index three. When our first class glance, this patch looks reasonable, but the problem is after the first operation.

A

The array doesn't have an index 3 anymore, so the correct patch is you actually remove index 3, then index 2., and luckily this issue was already well known once we googled for it and we can actually um apply it, and so once we realized it was a mutating emission controller uh to stop the bleed. We did actually deleted the mutant emission controller from the specific cluster which immediately resulted in successful pods coming up, and so our immediate fire is resolved. So we're done right.

A

No, as we dug into it more, we realized our change had actually happened seven days ago. We realized that for some deployments over the last seven days, new code actually wasn't even being deployed. Services were being kept alive by their own replica sets, but over time as pods of the overhead, because slowly died off, uh services were slowly getting degraded yeah, so this problem was way more insidious uh than we had realized, and the takeaway here is uh be aware of the existence of mutating a mission controller.

A

This was something we had recently wrote out and so for many of us, uh when we see an incident, it doesn't immediately come to mind to think hey. It could be the mutating emission controller be willing to ask for help because uh paging our additional team member to help us uh definitely got us moving.

A

Do google the error messages and uh do great alerts when we dug more into it later during the post warning, we realized we actually do have metrics that could have detected this problem earlier. All right. Our next incident is called one to auto scaling, so we're gonna begin with this kubernetes issue.

A

This is still an open issue, the gist of it is. Let's say you have the situation where you have a deployment with three replica later you set up an hpa or horizontal pod, auto scaler, with a much larger number. You then update your deployment and apply it, but this is where the problem happens. The moment you apply that or update that deployment.

A

The number of replica resets down to three, and so the suggested solution outlined by comment, was remove replicas from the last deployment and remove replicas from the current and future deployment to let the hp do its thing and that sounded good to us, and so we ship it great. So our fix right now is, uh if there's a deployment at hpa.

A

We follow the advice of the comment by editing the last employment and deleting the replica account from it, removing the replicas from our current deployment and applying the deployment, but that was not the end of our auto scaling modes, so this next case is called zero to auto scaling.

A

So if you actually look at the hpa algorithm, this is the algorithm where desired. Replicas is a function of your current replicas.

A

Some of you might already see the problem. If current replicas is zero, desired, replicas will always be zero, so this is a sounds like a quick and easy fix and we ship it right, and so now our fix looks like if, if deployment and hpa and if replica is not zero, uh then we apply that fix. Otherwise, if replica was zero, we can actually just leave it alone, and the hp will do the correct thing zero to auto scaling again, but that that was that was not our final problem with the hpa in this incident.

A

Someone alerted us when they, when they did a deploy, their number of replicas actually dropped to zero. So, let's see what happened here, someone followed up and realized. uh Yes, this is actually a known issue which we had already fixed, but we were actually migrating. Our deployment services to spinnaker and spinnaker did not have this standard workaround implemented yet, and so the lesson here is migrations are hard, and so now the fix is oh and remember to copy this logic to spinnaker and once more we have more auto scaling issues.

A

So what do you think happens in this scenario? You have a service that just migrated to kubernetes and still has old, ec2 boxes running. We deployed this to service to kubernetes replicas equals zero.

A

We then scale the service in kubernetes using cube, cube, cuddle scale, deployment and nice. We realize that kubernetes is now taking traffic successfully, and so we delete all of our old ec2 boxes, and so now the service has fully migrated to kubernetes, and now we deploy again with hpa.

A

That's right, an incident happens and the actual bug here is: we had manually scale this deployment with cube, cuddle scale deployment, but this means the last. The most recent proper deployment was actually still zero replica.

A

We didn't go through a proper deployment or what what our infrastructure teams consider a proper deployment, because they had used the mat, the cube, cuddle tool directly to scale up the replicas, and so now our fix is uh add, a warning to a cube cutter scale, deployment tool to warn people that this is like a temporary tool, but they should still go through a normal deploy process and to actually check what the current replica account is, and our takeaway here is a turning on horizontal plot autoscaler for the first time is not trivial.

A

uh You should test your edge cases such as zero, one, etc and remember the non-page paths, because, even though we had fixed this issue in our normal deploy tool, spinnaker was a new, a new path that was still being paved and our next case is called. uh Did we really delete all all our master nodes?

A

Yes, yes, we do, and what happens when you delete um all the master nodes on a cluster? Is it means there's no api server uh cube cuddle commands do not work, there's no new deploys and the cluster is just broken, but luckily for us this was just a test cluster, um and so, as we try to figure out, how can we bring this cluster back up uh our options where one completely delete the cluster and and bring it up all over again or to actually try to fix it in place?

A

And so these are pretty bad situations to be in, but luckily this was still a test. This was just a test cluster and one of our engineers realized. Actually, uh the situation was already documented in our run book, where the suggestion was one delete, our mutation controller as an immediate action via the cube cuddle command and to terminate the cube dns pod to get it rescheduled.

A

The problem with trying to bring up your masters once they have all gone down is the order in terms of how you bring things back up really matters in this specific case. uh Following the run book allowed the necessary services to actually start up again and get unblocked for the masters to actually come up successfully and the takeaway here is: don't drain your masternodes.

A

If you do do check your run books. Our next scenario, our next case is called masters out of memory. So for this specific incident um we noticed our master's cpu usage going up. We noticed that the cube api server is going wild and using a block of cpu and memory.

A

Our xcd resource counts are shooting up and we're getting uh other memory errors on our api server and our alerts are firing. um More alerts are firing and more alerts are firing. So this isn't. This is a real incident. We also realize at least as we dug more into it, is that uh we actually had no assets either. When we tried to ssh into debug, we noticed uh we noticed that the nodes were having real issues and we actually couldn't get in.

A

So what is happening here? So our api server is uh ooming and crashing our control plane. Nodes are severely degraded. We don't have ssh access to them. We wonder is this: a stp problem is std having issue because uh save is having issues. Things would be really bad. We noticed that it was having elevated objects being created, but it's actually still healthy and fine and existing workloads are are also still fine.

A

So our response here was to spin up a bigger instances for the control plane for the masters, and- and this was done just in time, because our old instances actually just completely died, and we noticed that memory was uh still getting eaten up super fast, but good thing, these new instances are much bigger and so for postmortem we realized um this was actually one of our largest clusters that was happening. That was having this issue, so we were wondering. Is this um a new scaling limitation that we've just hit?

A

We also realized the incident lined up with a deploy that had dramatically increased their max surge value and that deployed was having crash loops, uh and the theory here is uh that deployed with a high max surge value.

A

While it was crash looking, it was actually overwhelming. Our api server and the takeaway here is uh uh restricting mac surge, would help protect the cluster and it's a good idea to be ready to scale vertically. We were lucky that we were able to spin up bigger instances for our control plane to make sure that they scaled, but for this incident we actually still have some ongoing unknowns, such as what exactly is a breaking point in terms of a pods or mac surge.

A

We've also had larger clusters in the past before and but we didn't run into this issue. So the question is: what is different now and three: uh why did memory usage actually jump? Suddenly, um if you remember from our graphs, the api servers were actually using up to like 50 90 gigs of memory, uh which is odd because xtv, you know can only have can only use up eight gigs of memory. So what was actually using it all up.

B

Authentication in the world of containerization at airbnb, we rely on aws in roles as a primary way to authenticate our services. This is simple enough in a traditional non-containerized world, where each ec2 instance can have their own. I am role and they can assume that role and they can request the ec2 metadata api to receive temporary credentials.

B

But what happens when you have multiple pods and services running within a single instance?

B

We could let each pod assume any of the other pods rolls or make one big iam roll to encapsulate all the pods I'll be running on it, but we don't want to give each pod more authentication or permissions than it needs to address this issue.

B

We use an open source project, called cube2im q2im im credentials to containers running inside a kubernetes cluster based on annotations. But what does this really mean? Well, cube2im works by running as a daemon set within the cluster. When the pod makes a request to the ec2 api cube2im intercepts the requests, then using a special item role of its own. It assumes the pods I am role.

B

Then it makes a request. The ec2 api itself receives a temporary credentials for the pod and then forwards it back.

B

Now q2im didn't wasn't perfect: it came with some race conditions, so the expected course of events are as follows: with cube2im, first the node should start up, then the cube2impodstartup, then we add an iptable rule to forward request to the ec2 api's ip address to the cube2impod.

B

Then q2im starts watching for new pods, after which an applications pod can start up. Then q2i notices a new pod caches, the pod's ip address to its aws im roll mapping. Then. Finally, the container in the pod can make a request of the ec2 api.

B

However, what we were seeing was that containers would make requests to the ec2 api before cube2im had even noticed the pod come up and because so when cube2im receives requests, it would not be able to find the pod's ip address within its own cache and it would just reject the requests.

B

This was especially problematic with init containers, which would run before all of the other containers start solution.

B

Another init container, that's right. We made a new init container called cube2im weight that would basically just keep pinging the ec2 api until it received a successful response back. This works because we had already set the iptable rule before to forward request to cube2im.

B

So if you got a successful response back, that means cube2im responded successfully, meaning it had already noticed the pod come up now. This is only a temporary solution.

B

We're currently working on a new authentication scheme by which we have the api server, use its own private key to sign a token and inject it into a pod on startup, then, when a pod needs to get authenticated, it uses a different api to get credentials for the pod's imrole.

B

Using the sign token, then the sts verifies the token signature is valid with the public keys, after which, after the validation is complete, sts returns the temporary credentials back to the pod.

B

Our main takeaways here is that ending containers are very versatile, despite their simplicity and they're good solutions, even if they're only temporary.

B

Cpu limit equals number of cpu cores in the node still has throttling so one day we got a message from an engineer asking us: can you explain why we'd still see throttling when limit is set to 36 with only 36 cores? How can we exceed 36 so to kind of give context on this? Let's first talk about how cpu limits work on kubernetes, kubernetes limits, pods, cpu usage, using linux's c groups and the cfs the completely fair scheduler.

B

So, let's take an example of a 36 core machine with a default scheduling, period of 100 milliseconds and let's say we have in our kubernetes manifest limits set to cpu of 1024 millicourse, that's equivalent to one cpu core.

B

This gets translated into linux's cpu.cfs underscore quota of 100 because we have the 100 millisecond default scheduling period times, 1 core, which gives us 100.. Now, let's take a few different scenarios.

B

um Here we have a bunch of different scheduling periods and in our first scenario we have one container running one thread and it can run the 400 milliseconds without any throttling.

B

Now, let's say the container actually has two threads then thread. One and 2 can run simultaneously for 50 milliseconds, after which it will get throttled, because the combined usage within that container would have reached 100 milliseconds.

B

Now, in our final scenario, we have thread 1, 2 and three running simultaneously and then thread one and two either terminates or gets descheduled after 25 milliseconds, then thread 3 can run for an additional 25 milliseconds until it would have reached the 400 milliseconds of this quota, after which the threads from the container will get throttled.

B

So in our previous scenario, we had in our limits a cpu limit of 36 times, 1024 millicourse, which gets translated into a cfs quota of 3600. So in that case we should be able to run.

B

um We should be able to run our threads on all 36 cores for a full 100 milliseconds within that period or within any cfs period, and not get throttled at all. So to figure out what was going on. We ran a bunch of tests. We made one app with a limit of 36 cpu and we made another app with no limits at all and we ran them on different nodes swap nodes, and we found that on average they all ran pretty much with the same performance.

B

But what was surprising was that, despite the similar performance, we actually did see throttling on the app with cpu limits set to 36..

B

So our main takeaways here were that some throttling is inherent and is okay and that we should be be more careful of our metrics docker built-in cpu usage information may not actually be granular enough to capture fully. What's going on, we also learned that percentage of throttle periods is a better metric than absolute value of periods, since the latter is very heavily affected by the number of threads running within the container.

B

Cpu limits cause out of memory kills in kubernetes. We have memory limits and cpu limits, and intuitively memory limits cause out of memory kills if they are exceeded and cpu limits would cause throttling and they do. But we were also seeing that cpu limits were causing out-of-memory kills.

B

So here's an example: we have in the top left our pod usage spike a little bit after 12., after which it was immediately followed by a huge spike in memory usage and eventually an out-of-memory kill.

B

So what's going on here well, the first reason was that these were java processes and the jvm garbage collector is intensive and spawns a lot of threads and the more threads there are running concurrently.

B

The faster each container can get throttled, there's two main flags to be concerned with here: parallel gc threads and concurrent gc threads um within java, and these two parameters are used for different stages of the garbage collection.

B

But the number of threads used for garbage collection, for either of these flags can easily exceed the number of threads in the main application, and when both the main application and the garbage collector is being throttled, we can have a situation where data isn't being processed and any data that is processed is not able to be freed.

B

Our second and third problem was that only certain containers get throttled and the back pressure we had in place to kind of mitigate this wasn't working properly.

B

So since kubernetes relies on linux c groups to implement throttling, this meant that certain containers in the pod can get throttled, while some other containers aren't. In our case, we had the main container processing the data get throttled while another container envoy accepting requests was not. This meant that we kept.

B

This meant that we were continuously accepting requests. While we were unable to process them and we were unable to again because the garbage collector for the main container was also in throttle. We couldn't free the memory properly either we had back pressure in place to mitigate this, to tell the clients to stop sending requests in an event. Something like this happened, but it wasn't working properly at the time.

B

So our main takeaways here be wary of what other processes may be running on the pod and adjust limits accordingly tune. Jvm garbage collector threads, if necessary, and to make sure to have proper back pressure to avoid overloading our services.

B

Rogue ecr, cleaner at airbnb, we make a lot of docker images. This data is from 2019 and at the time we were making over 2 million docker images per day and obviously we can't keep them all. But we also can't do something naive, like deleting all the old images, because we don't know which images might be in use.

B

So we built a service called the cr cleaner that runs as a cron job every hour and, basically all it does. Is it finds all the images in use. Then it loops through each of our ecr repos and deletes all the old images, except for the ones in use fairly simple, and it was working great until it wasn't ecr, cleaner, started, deleting images in production and to figure out what's going on, let's go back and look at what it was doing and specifically, let's look at the find all images and use function.

B

Basically, it would find all the clusters then keep a list of active images. Then loop through each cluster figure out what images are being used by all the pods. Add to that list and return the list of active images.

B

Well, in a recent change, there was an error case and on error it would just return an empty list and you can guess what happens afterwards. Since our list of clusters is empty, it finds no images and returns an empty list of images.

B

Then coming back up, it would loop through each repository and delete all the old images, except for nothing. So it would just start arbitrarily deleting all the old images.

B

And our second mistake was a lack of proper alerting. Ecr, cleaner was failing, but we weren't getting alerts for that and we also weren't getting alerts for image poll back off errors in production across multiple pods, but once we figured out that this was happening, our fix was to first shut down ecr, cleaner, to prevent it from deleting more images.

B

Then look for crash-looking containers, look through ecer, cleaner, logs um and kind of cross-reference them to figure out what images are missing and manually rebuild them all on the right. You can see a teammate rebuilding all these images and keeping track, and this is only a short snippet of all the images that we had to rebuild and it was very tedious.

B

So thank you kind teammate and our takeaways here were that the more critical the service is the more crucial and thorough testing and review should be, and we should have proper error, handling and alerts for infrastructural issues, and we should also try to make sure that our fixes aren't causing problems elsewhere.

B

In this case, because we were rebuilding so many images, we were causing other people's builds to slow down a lot, um and we should have seen foreseen that.

B

Be careful what you break as part of our on-call readiness goals? We started doing fire drills, so this is what they look like. Someone would set something up and send a message in our slack channel. So, for example, fire drill, ci is broken and we cannot do a build for ecr, cleaner. We need to ship a new version to production that displays a new logline on startup, without relying on ci evan, who is on call disclaimer ci is not really broken. Do not panic. This is a drill.

B

This is only a drill and we did a lot of these and we found a lot of bugs a lot of places to improve our code and our documentation, and it was great, but there was one particular fire drill that was pretty interesting, but also pretty insightful, and it had to do with the admission controller.

B

This was actually one of the first fire drills that I did as a new member in the team, so it was meant to be simple. Basically, a change was made in the emission controller to block any creations of a replica set, and it looks something like this.

B

It's a new rule that said, if the resource being applied is a replica set return, an error and the fix was supposed to be simple, just find an old working commit and deploy that back onto the test cluster that we were using and that's exactly what I did, and that was that, and it was all going fine until a few days later, the same person who had made the fire drill sent this in our team chat I deployed on to the test cluster, but it's as if the mission controller didn't get deployed, even though the deployment went through.

B

Has anyone seen that before? Well, I didn't know the time, but as it turned out, I had seen it before. When I went back to deploy an old commit.

B

um I thought the deploy had gone through, but a mission controller didn't actually get deployed. What happened was that there was a bug in our deploy process. Reporting success for failed, deploys specifically to admission controller in this case, but why was a mission controller failing in the first place?

B

Well, it was because the mission controller was not whitelisted by the mission controller. This meant that since the emission controller was blocking all replica sets, it was also blocking itself from being deployed, so we were locked out.

B

So our solution just delete the validating web of configuration for the emission controller, and that way we should be able to bypass the admission controller step, but this didn't work either, and that's because when the request is sent to the api server, it goes through a bunch of steps eventually lands on the validating admission step, at which point it takes a look at the validating web configuration resource, which is why we have deleted to figure out what webhooks it should get confirmation from, but as it turns out on, our mission controller deploys the validating web configuration is applied before the web of the web hook.

B

This meant that by the time it got for us to update our web hook, which is where the bad rule was the validating web of configuration has had already been replaced. So the actual solution, just nuke the whole thing delete the mutating emission controller, webhook, delete the deployment and then redeploy the entire thing from scratch.

B

So our takeaways here double check that your deploys actually went through.

B

Make sure that the thing that controls your deploys can actually fix itself with another deploy, be aware of kubernetes apply ordering and that even the simple fire drill can reveal a lot of insights and bugs and tldr that will never write a rule that blocks all replica sets.

A

All right, to recap, some of our top 10 takeaways, are one: do test new features in test clusters.

A

Two uh be aware of the existence of utility mission controller, three, remember the non-paid paths or uh whatever new pads you might be paving for uh don't drain your master nodes, five be ready to scale vertically.

B

And eight containers are versatile, despite their simplicity and are good solutions, even if they're, only temporary some throttling is inherent and is okay, be wary of what other processes may be running on the pod and adjust limits accordingly. Have proper error, handling and alerts for infrastructural issues and be aware of kubernetes apply ordering.

A

All right, thank you, everyone. This is the end of our presentation. If you would like to learn more check out, our engineering blog um airbnb is also still hiring and a lot of the work we shared today uh were not done by just me or me or joseph. uh It was all done by our wonderful team. So, if you like, if you would like to work with us, uh reach out apply and if you have any more questions, feel free to contact us to follow up. Thank you.