Cloud Native Computing Foundation Kubernetes Community Days Bangalore 2022, 11 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: War Stories with Kubernetes in Production

Description

Rudderstack is an (open source) data pipeline company that helps send data from any source to any destination. In this talk, We want to start by explaining the Rudderstack architecture which is used to send data from any source to any destination that is similar to Kubernetes (ControlPlane/DataPlane), and how we deploy everything on Kubernetes using aws EKS. Following this focus is on giving a few case studies related to our incidents with Kubernetes and what actions we took to prevent the same to happen in the future.
Some examples are mentioned below: 1. How we took down the Kubernetes cluster by deleting config map 'aws-auth' that manages complete access management with EKS. Explain how we prevented this from happening again using the OPA gatekeeper. 2. How the EC2 worker node going down caused some downtime. Explain our runbook and how we improved the time to recover. We like to cover 3-4 such cases with our timeline of recovery and what action we took to prevent this from happening or improve recovery time.

A

uh Yeah hi guys, so today we are going to present a presentation on war stories with on cooper, light cooper. That is uh so I like I before that, like. Let me introduce myself. uh My name is sharma. I work as director of engineering at rudder stack. Traderstack is an open source data pipeline company.

B

A

Yeah and uh with me, we have lukeish.

B

Yeah hi, uh I have lokis and uh I leave the infra team at radastreck and I'll present part of the presentation.

A

Yeah, so let me before we even go like what kind of stories we have. Let me first start with like what kind of setup we have at rather stack because understanding that is going to be important, and it is also fun to know like how we are using the dust uh kubernetes at retrospect.

A

So, first of all, like what is redder stack, why there is a need of such a tool. So one of the things we all know that consumer data is continuously growing uh like we have like data is distributed like across. There is like sas sprawl so that way like we have data a lot of different places and it is really hard to collect the data. So what we are doing is we are building an open source solution uh which, where we don't persist, any data like we simply collect.

A

We simply process the data from take the data from input and then like send it to defined uh destinations, and we we are like processed more than I I would say. 35 billion is a very conservative estimate. It's more than 50 billion events, because, uh like a lot of our customers, are open source. So that way, I I would like. We wouldn't even know like how much data is being sent, I am sure, like it will be more than 100 billion, uh so rather stack come, sits in between like with from well.

A

There are like lot of sources and a lot of destination, and in between there is somewhere under stack, which is processing all the data from like all the sources and sending them to their defined destination, and we are like we are completely cooper that is native. So all our deployments, everything is on kubernetes. Only so, let's go deep into what radius tech is like how? How does real estate architecture look like, because that will give us a better understanding of why we feel cooper? That is a native tool to be used for redder stack.

A

So in data stack, we have divided the complete system into two parts: one is control plane and another is data plane that is more similar to how kubernetes architecture looks like today. So control plane is more like the place where we are storing all the configuration all the observability, uh like everything, whatever is needed to get the complete visibility on how data pipeline is working, but the core aspect like same as kubernetes the core things in kubernetes also run on the data plane on different worker node.

A

Similarly, here we have data plane which process all the events flowing through it and we are using postgres as our storage. uh We have optimized it for like our specific use case. So whenever a event comes from a different source, we store it on a postgres database and then like we do a lot of processing and once all the processing is done, we send it to desired destination and, in our case, destination can be any service or it can be any given warehouse.

A

So now, let's go deep into like how does uh our kubernetes architecture looks like because understanding that will be like very much important to understand like how we have built everything. So our architecture is completely. We have built everything using terraform.

A

We are very much of the belief that infrastructure as code is here to stay, and that is the core like premise, that everything should be code like jit, should be source of truth of all infrastructure. So what we have? We have a layered architecture where what we are doing is initially, we are creating all the infrastructure and then using we are a lot of. You are using help a lot and with the help of arco city, we are doing all our deployments.

A

So what we have is like we have uh ppc networking layer deploy created through terraform and then eks that is also created with terraform. We are using uh application set that comes with argo cd and what it does it helps us create a base layer. So once networking and eks cluster is created, all the clusters register itself to argo cd application set and in that, like argo, cd automatically deploys all the needed initial application on our kubernetes cluster. So layer is like. Initially, there is a networking layer. Then there is the eks layer.

A

Then there is a error for like helm, chart layer which is deployed through our city. Application set and these three layer combine make our base cluster, so in complete error stack if anyone has to create any cluster. This is the base layer which everyone has to create and like. We are also using argo cd with argo city so because it is managing itself. So that way everything is in the code. Argo city is more like comes as a jit offs model, so that way like everything is didn't get like.

A

Basically, we push code there and it immediately get reflected into kubernetes. So now, once the base layer is ready now we have to deploy our own applications. We are also using uh operator to deploy the complex data plane because we already saw we need to. There are like different parts of data plane, so everything is deployed through operator and that and plus there are like some meta services which are written by us, and then there is a control plane and everything is deployed through our city.

A

So if anyone has to like deploy anything, anyone has to do any change, they don't need to go to cluster. They can just go to create a pr, uh and then someone will approve it once it is merged. Everything will get deployed to the cluster.

A

So in argos city it support both manual as well as automatic uh deployments. We are not using automatically everywhere, but still we have like a lot of legal places. uh It is automatic deployment. So once you merge anything into kit, like everything get deployed uh so now we have divided our services into uh different kind, like categories, so one of the initial categories, like external dependency. In our case, because we are very much on aws, we are using eks for uh kubernetes. The main reason is we don't want to manage our own control plane.

A

So in that case, what we are doing is we are using external dns controller, which basically interacts with route 53 and whenever we need to create a new application, we need to create a route 53 entry and that is completely managed by external dns, but external dns also works with different service provider.

A

Second, one is uh ingress, so we need to create an load balancer and in our case we are using alb load balancer, because it helps us create multiple route, multiple route rules in the single alb, so that helps us save cost.

A

We are also using uh nlb with nginx ingress controller, but currently in our production, it's majorly alb uh and the airbnb is controller and take care of all the like uh in rule, traffic rule or interest creation management, and we are also using groups in which we group multiple rules to a single uh alb.

A

Third, one of the most interesting one is aws ebs, csi driver. So previously we were using default uh driver which was coming by default with the ebs, but we started seeing some issues where the volume is not getting detached properly and stateful set gets stuck and yes like we are using stateful sets at a lot of different places so that that's why we migrated to aws vs csi driver which basically helped uh like reduce lot of such issues.

A

So once all the external dependencies are deployed, next thing is: we need to auto scale our cluster. We don't want to do anything manually, so that's why we are using cluster autoscaler which, depending on the requirement like if the cpu cpu is not enough, our memory is not enough. What cpu cluster auto scaler does it automatically interact with aws and it creates more worker nodes, and we are also using cluster over provisional what it does.

A

It creates some dummy parts which take provided resources so that way we always have some extra resource being utilized in our ec2 uh worker nodes and whenever we actually want to deploy any part which is needed by any customer or like our own requirement, we don't need to directly go to cluster autoscaler and ask for a new node. We have some extra space. These parts created by cluster pro overprovisioner are of very low priority.

A

So anything we deploy these parts go down first and then they request cluster, auto scaler, that we need more space because the space which I had was already taken by uh different parts. So that is how where we are using cluster over provisional.

A

So with this, like our base, image is ready so that that is like the core base image which we are using now next, because we need to deploy our data plane for that we have created our own data stack operator, so operator is basically extension of kubernetes apis and anyone can write their own operator and dip.

A

So this also helped us make our complete deployment very easy, because previously we had to deploy like four and five different services and that that was needed to make complete data plan run smoothly and with the help of operator, what we did is we created one crd which basically deploy everything for us, but it's not just deployment. It also helps us scale up and scale down our system in real time, and another interesting question is: why do we need to do write an operator? We can simply run commands by increasing the number of replica.

A

uh So in our case, we provide like ordering guarantee and in case of ordering guarantee just increasing number of nodes will not like number of parts will not solve our problem. We need to create a new deployment and migrate. Everything to the new deployment and like that can only happen when old, like everything gets flushed in the whole deployment. So we have a concept of uh going into degraded mode where we don't process anything but we consume, uh so that is where a district operator becomes a key key player in our infrastructure.

A

Other than that we are using our own application. Services, like the micro control plane in itself, is a big uh micro service like following micro service architecture. So all that thing is deployed through helm and through uh arco, cd, so for monitoring we are like uh now like previously, we used thick stack, so in pick like it was basically based on flux uh db. uh We are not using chronograph for that. We are using grafana for all our monit like dashboarding, uh but now we are very much migrating to prometheus.

A

uh We are even using thanos for uh storing data for longer longer duration and also like multi-cluster kind of set up. Thanos helps a lot and argo cd like, as I already told, is basically that is the key component which help uh us deploy everything uh in our cluster so like. I hope this must have. Given you a brief idea of how our architecture looks like and that that will like help, you understand more so what happens in our case? Is we make a request?

A

So whenever someone a new customer comes, they make a request and kuber asked you. Everyone know kubernetes is very much api driven. So a request comes from kubernetes deployment uh like through the api. It goes to our operator and from there we create all the needed parts. So that's how like, whenever a new customer on board, we create a dedicated environment for them.

A

uh This is more of a pictorial representation of how our architecture looks like. So whenever something is like, whenever there are multiple clusters running on a cluster, we are using aws route, 53 alb ingress for routing all the traffic from external world to our network.

A

We we, we have both shared resources like shared resources, more like shared services, which are which we are using across uh customers, but then we also have dedicated uh services running for each and every customer. Now these dedicated services are running their own pods or like in a lot of enterprise cases. We are also running dedicated databases inside like a namespace which is an which is dedicated to a customer.

A

So that way, like all the data which is coming for the customer is dedicated like uh it stays in the customer. No one else can see that data so like. I hope this complete thing has given you a good understanding of how our architecture looks like and yeah. I think now uh location will take it from here to talk about.

B

So uh our stories are basically uh any large scale, uh outages or incidents that we had when we had to quickly come up with some solution to uh make everything uh bring everything up back and early. So uh we look at like two examples of uh two such examples so and in each case we look at uh what was our setup.

B

That allowed something like this to happen and what are all the confidence involved and uh what exactly happened and how we figured out what happened and what we did while debugging that and and once we debugged, obviously we'll do like an rca and we come up with actual items and so we'll discuss those things as well and how those action items uh helped us in the long term to prevent uh more such instruments.

B

So, uh first, we will look at uh the first of our stories like uh when, when our entire kubernetes cluster went down, uh as smith mentioned earlier, we use uh aws cks for all of masters and we use oadc for our access management and one of the key components. There is something called aws configment that contains uh all the information about all the users that have access to the cluster, and it also contains details.

B

Like I mean it also defines the permissions that all the worker nodes needs to access the kubernetes api so uh like the idea here is to basically use aws's, iam roles and so on, as the authentication for accessing kubernetes cluster for user. So uh there are other ways to do this, but the reason for choosing this is uh this comes by default with aws case, and it also makes sense to keep the authentication in a single spot between aws and kubernetes.

B

Instead of having uh two different ways of doing authentication in case we want to revoke access or, like change, the type of access of some specific user. So that was the reason for going with uh awsr and uh what happened was like when we started out, we had a small, a relatively smaller team, and we were. We had a single roll call. Actually, we had two roles: one was like uh adventure, and other was green.

B

Only and uh everyone in the company was using the using these two saved roles to access everything in the cluster, but uh as we scale up, it becomes important to have like more five-day control over uh who can access what like, what? What life spaces. Each user has access to things like that. So we wanted to uh change our structure of uh uh structure of authentication, and this required lot of changes that we need to make in the aws aws our config map and adding both uh cluster roles and role models and so on.

B

So uh we were trying this thing out on our dev cluster. Obviously, and uh we got everything working but uh then we thought like we should not uh manually apply these things. These changes on production. We should do it through terraform, so uh we were again testing these things out and during this testing, uh like uh I mean we had like two clusters right, we had we had a cluster, we had a production cluster. We were testing this until and in between.

B

We had to switch to production cluster to do some other tasks, and then we came back to testing it and we forgot to switch it back to the dev cluster. So what happened was uh like we were overriding, the uh overwriting, the aws configure, and uh we owe it like. We ordered the production one with the dev one so like they are both in different aws accounts so like what that basically means is uh uh the whole authentication system in production.

B

Cluster is broken right, so no user can actually access the uh actually access the kubernetes api and that's a smaller part of the problem. The bigger part of the problem is, since the worker nodes also use the same authentication system uh like uh none of the nodes, are able to uh communicate with the kubernetes control plane api. So that meant that, uh according to kubernetes, all the nodes are disconnected so because all the nodes are disconnected all the parts are these scheduled, so they all went into pending state and one other bad part about.

B

It is like we didn't even know what was happening, because we didn't have access to look at uh what was happening so basically, how we got about it is like we have like uh multiple levels of alerting setup on our applications, and we came to know about uh know that something uh went wrong uh because of this and, uh like I remember, we didn't even uh connect it to to the aws configure change uh until sometime, because one developer was working on that and uh like a lot of other people, were on the call, but it took some time to realize that it had something to do with the wsr and did not change so um once we realized that uh that was the reason.

B

uh What we uh tried was uh like there is this concept of ownership in uh e case clusters, so the person who created the cluster he'll have like extra privileges over the cluster. So even if uh all these accesses get lost, they can still access the cluster. So we realized that and we brought the owner of the cluster into the call and like um they were able to access things and see the uh see exactly what what happened.

B

uh We saw that the demo account was in the production configman, so luckily we we we had like backups of aws of country maps from earlier which we used today, uh so we used one of those backups and applied it and yeah like pretty quickly after applying it, like all the nodes started connecting to the cluster and all the parts started coming up and everything went back to normal. So I think this whole thing took around like half an hour to 45 minutes, yeah or maybe even more so.

B

Obviously, it's like a very bad bad uh state of affairs so like this required a lot of action items on our side to plan for uh such such things in future and to avoid uh repeating something similar from happening. So uh yeah, like one very obvious thing that we had to do, was like we had an on-call process of one person being on call and uh they look at the uh I mean they they take care of all that.

B

So they look at the alerts for a while, and uh there was not a fixed process of when they should call multiple people and how it should be escalated. It was mostly a judgment call on whoever was on call, so we changed that process to make sure that uh whenever it's a critical alert and like there are multiple errors very like we make sure to uh escalate it to multiple people, and we have a way of calling a lot of high level people into the call at once.

B

So uh we started doing that and another major thing was. uh This was all happening because uh our testing was manual and uh our like our flow, wasn't like very. uh What do you say uh uh doing it on day, one then on production properly, so so so to fix that, like uh we wanted to do even the development and like testing parts also through terraform series terraform, but we also want to.

B

We did not want to do them on the developers machine, because that was the main problem right, so they they were doing something and they contact switch to something else, and uh they missed that part like something similar cannot happen. If you do it uh through a github uh cd type of thing, so we started exploring atlantis for that. So uh and atlantis was like very helpful in that regard.

B

So we quickly moved over everything to uh atlantis so from there on, like any uh any terraform change or any infrastructure that that goes to our uh production. It has to go through atlantis and even uh development. Also it has to go through atlantis like no. No developer can run terraform apply from their local machines that that was the uh main point there and, and another thing that's very specific is like uh aws art config map uh got overridden and then it was deleted and so on. So we had.

B

We set some uh oppa gatekeeper rules so make to make sure like uh the config map is never deleted and yeah and also like uh we did not have a proper uh disaster recovery start season. It was more of like weak strategies during the call and did something but uh yeah we realized. We need we needed to have a plan.

B

uh If something else goes wrong and we don't like the lucky thing was we were able to figure out that uh the country map got messed up there, but if that did not happen, it would have taken uh much longer for us to bring bring it back up. So the disaster recovery part of it is to have a separate cluster running and all the uh helper parts and services type of parts running on the cluster already and whenever we want to uh like, we notice that uh some customer or some cluster is done.

B

It should be easy to uh move them from that cluster to the new cluster, so that was part of the disaster required strategy and- and we also wanted to have um wanted to have all the developers be ready for any such incidents. That can happen so we started doing uh weekly infrareds, where we uh like simulate a scenario and everyone on the call. Everyone in the infrastructure team uh tries to solve it and like if they notice something that's wrong with our run books or there is something that can improve.

B

uh We do that improvements on the call and like this ourselves, a lot uh like incrementally. uh By doing it, we can communicate.

B

So this was the first story. Second story is uh like compared to the first one, slightly minor, but uh again this involved uh kubernetes uh one of the worker vendor so like, as mites mentioned earlier, like we have separate deployments for our enterprise customers.

B

So usually, uh when a cluster goes down, it means like most of our customers go down like who are on the cluster, but even when a node goes on, it can mean like at least like four or five different customers going down. So this is also pretty serious problem for us. So here the context is basically uh eks uh internally uses uh ec2 instances as the worker nodes. So in the case uh the control plane is completely managed by aws.

B

We don't have to do anything but uh easy to workers, ec2 instances act as the actual worker nodes and obviously all the parts were around those worker nodes and because it's easy to instances there is like it's not very often, but uh the instance can go down. Sometimes what that means is like.

B

Sometimes it can be that the instance gets terminated, but it can also mean that there is a network failure on that particular node or like there can be cases where, like uh some of the uh ecg instances in one region, are like completely down and stuff like that.

B

So there is also an another scenario where, like uh aws, does some periodic maintenance when it wants to upgrade some things or change some configuration, so they notify us in advance and uh like they restart the senses, are like they completely kill the instances uh at a given time. They tell us earlier to take care of it, but if you miss it like this can also be a problem so yeah in this case, what happened was.

B

Basically, the cube api server. I it wasn't able to get the heartbeat for one of the worker nodes so uh like when we looked at the logs and all our metrics and network users, and so on, like what we noticed was uh there was nothing coming out from that node. The actual ect instance was running till that time, but uh it looked like basically, the network got uh disconnected on on that note right, so it was it. It went completely dark.

B

There was nothing from there so uh when this happens, like it waits for like a couple of minutes, the controller manager, and then it marks the node in not registering.

B

And so uh again there is again some time out after the node is smart, not ready till uh the part gets marked as not ready, uh so, like the part part will get ready, equal and false as the status, and uh until this point uh from the outset, uh we don't see any uh like uh it's very hard to notice that things are in a bad state but uh like basically, since the network is done, like traffic is actually not reaching the parts but uh like after the uh ready equal to fall status like kubernetes, actually stops routing traffic to those spots.

B

So if you have like a service, that's connected to three parts and one of the parts goes to a ready fall state like uh that's a good thing. Basically, because, like traffic is rotated to other two parts and there isn't any downtime. So this happens, and then after this, uh like the part, doesn't get deleted as soon as it goes goes to not ready states.

B

So uh there is a concept of toleration uh for the parts like where uh you can specify how much time the part has to wait on a unready or not reachable node before it gets deleted. So that is by default five minutes.

B

We never considered to change it before but like yeah, uh because of this, like the part, was still running on uh five minutes and like yeah after five minutes like the stateless parts uh that get deleted completely, and then they move to a new unit like it's more complicated for stateful parts, because uh there is a volume attached. So what happens is uh like the part part gets deleted, so it's like when you're in uh part, delete right like sometimes it doesn't get deleted.

B

Often if there is some fine laser, that's blocking it so here the finalizer is that uh the node is uh the volume attachment is volume is not reachable test, so it took some time to uh detach that partition and yeah. So how like responded to this event was like uh because of some of our like injection services going down.

B

We got alerted right away for this, but the first time it happened uh like we didn't know exactly what happened like we found out that some parts were in like terminating state or deleting state or whatever, but like it took like it, took us some time to figure out that uh it was because of a note going now by the time we figured out and like we did some research on what to do in such scenarios, like all of the previous things happened, and uh everything moved back to a new node on a new node and, like things started coming up, so we didn't have to do anything.

B

But after this happened a couple of times we we did a lot of research and, uh like uh we added alerts on uh when the notes goes into not ready, state or unreachable state. So we realize like uh much sooner.

B

What exactly is happening and, uh as I was saying, there is a concept of toleration, so we change the tolerations of critical parts that needs to be running uh all the time to 20 seconds so like whenever, as node is not reached once for 20 seconds, the part gets deleted from there and moves to in you know, and we also had. We also wrote like uh run books and scripts for deleting all the parts and on the on a single node and also delete all the volume attachments uh from that node.

B

So this helps us in uh like bringing the parts uh like stateful parts up quicker yeah so like this happened multiple times after that and like uh we were more ready and uh things aren't down for longer after we took the uh video description so on and coming to like rca of the things and action items uh that we took here so again like uh here. Also, this meant that uh one alcohol person uh might not be uh very reliable in in such scenarios, where, like multiple customers are affected.

B

So we started writing the alerts to uh multiple people when it's like. uh So, as I was saying like uh we, we have a lots for each customer being done so when there are multiple customers down like it means something went wrong on the intro level.

B

So uh we uh we we started routing it to multiple people and then uh like we, we change the no timeout setting where, uh like we delete the uh node much earlier uh when the node is not reachable, because we observe that once the node goes into uh like once once the heartbeat starts coming like usually doesn't come back up. All that often so like one safe option is to like click much quicker, and we also try to change the max weight for unknown duration.

B

This should help in the detachment of volumes and so on, but uh like this, this felt uh when we tried doing this, it felt a little complicated and uh like we, we faced some blockers, so we didn't do that, but we had some. We have some workaround for this. For this one and one the most, I think the most important part here is like uh we started running uh like all the important parts in hmo. So so, even if one part one node goes down, uh the application is still handy.

B

So we have like an reverse proxy that routes all the requests that takes all the requests and routes it to the relevant service. So uh initially we used to have one per customer. Now we have two per customer with smaller resources so that, like we're not bringing up the costs almost all that much and we run them on different nodes.

B

So so, even if one node goes down, the application is still handy and, uh as I was saying like it's aws uh also does this maintenance type of a thing where the notify is in advance, but it's a notification on aws console.

B

So if you're not paying attention like it's very easy to miss, and we missed this multiple times and like we recently figured out that, uh like we usually get notified of this earlier, so we uh use like event bridge and sms, and we listen to that topic and we listen to that event and send us reminders on slack so so that we are ready to take care of all these things.

B

So yeah thanks for your time.