Cloud Native Computing Foundation Kubernetes Forum Bengaluru 2020, 27 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Resiliency Patterns in Kubernetes - Ravi Hari & Lakshminarayanan S, Intuit

Description

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects - Learn more at https://kubecon.io

Resiliency Patterns in Kubernetes - Ravi Hari & Lakshminarayanan S, Intuit

Kubernetes being a de facto standard for container deployments and orchestration the applications running on kubernetes platform should be highly available and resilient to failures. With Nodes and Pods being ephemeral in kubernetes there are high chances for application users/clients to get 504 or 502 (5xx) errors when a node restarts or a pod gets terminated. This session discusses how Intuit came up with these resiliency patterns and have applied in Kubernetes clusters and applications run in them avoid the failures. This session also talks about Life-Cycle manager an open source project (https://github.com/keikoproj/lifecycle-manager) which uses lifecycle hooks from the autoscaling group to pre-drain the instances and it will also deregister the instance from any discovered ALB target group to avoid in-flight 5xx errors.

https://sched.co/YVxQ

A

Namaskara Bangalore, oh so it's a huge event: Thank You CNCs for organizing it, and thank you all attendees for making it here so I am latch from include with me. I have Ravi, and today we are going to talk on resiliency pattern on in kubernetes, so as enter for today, looks like something like this little bit history on intoed and then what is modern SAS, the platform that we built for our developer, but also why we built it and how do we onboard our micro services onto this platform?

A

And you will be interested to know how long it takes. So we have a slide for that one, and then the meat of the talk is on resiliency, which Tory will take you through.

A

So as a company, it was started on 1983 by our founder when he actually, the idea came up on a kitchen table when he was looking at his wife, how she's managing the book and then he thought there is a better way to do it. So that's where the idea was born. Our first product was Quicken and then, of course, we went public in 93.

A

We have around close to 8k employees, we have a multiple location across the globe, 11 countries, 24 location and all these 8,000 employees or around this place, and we have around 42 million customers, and these are the different customers that we serve the consumer group, which mainly they use our tax product and then small, business and self-employed, and these are our products. The TurboTax is for filing tax.

A

If you are from years, will be very familiarize with that, and we have accounting solution for small business in the form of QuickBooks, and we have a professional tax solution in the form of pro connect and mint is a personal finance solution.

A

So, coming back to the main topic on modern says why we started building this one sec. Since we have around 42 million customer, the recoverability was something we wanted to always improve on right. The really tease it work, it I mean you will end up having some issue some incident always, but it's about how quickly we can recover from me. So we wanted to improve that number by 10 times. So that was one of the goal.

A

While we started working on this platform, the second one was how fast we can release the code to the production. The reason for this is when somebody measures the PR and if it get deployed into the production within a day, then it is pressure on the developer memmer mind that what are the changes that he made so in terms it actually helps in recovery.

A

Just in case we identify some bug or some issue, so you wanted to have a really really fast release cycle and of course, you want to ship all the feature to the customer, so he wanted to have a more frequent release of our product.

A

So the first thing was: we have our everything as a big mahalik product, so the first task was to bring them into or break them into a micro-services which many team did it, so they broke into micro services and then they started hosting it. We mainly use here double years as a hosting platform, so many team went ahead and then provisioned the infrastructure on AWS provision. There I mean come up with their own deployment solution and set up everything on AWS.

A

But what ended up happening was each team had a different way of slightly different way of setting it up and we end up having multiple EA doubly as a counter.

A

The challenge with that was, you have to maintain the security across all the account, and then you want to have some uniform way of pushing some patches and other stuff in so that became our first challenge and second challenge was each team had their own architecture and then, if you have to move around the people between the team, everyone has to learn the new architecture, how the new setup is done, but the most challenging thing was: every developer has to understand how the AWS works.

A

So there is always a learning curve involved, so no sooner they come up with some idea or some product beside working on their own product. They have to spend some sufficient amount of time initially to get on board around terribly as it used to take four or five weeks for them to just on board entire AWS and then start working on the platform. This when we thought that we should do something so, which makes it much easier for the developers, that's where we build up a platform called the modern sass platform.

A

All the developer that you need to do is just come to the platform and request saying that here going to provision a new service on this new platform, so he selects whether it's a Java based or load based service, though we are not limited to these two technologies as long as Nick and authorize, they can run it on this platform, so they come and give their specification or some method or talk about their services, and then they get their git repo where they can put their code, and then we provide something called a deployment repo which I will talk it in our next slide and besides that they get a pipeline Jenkins pipeline to build their artifact.

A

And of course, in this case, artifact is a docker image, and then that gets uploaded to a frog artifactory and it runs on kubernetes. So with that, what we achieved and all it takes is just 15 minutes right at the end of 15 minutes. We get a micro service which runs in production. Not just in free product runs in production, with all the monitoring tool that we use within impudent, whether it's a wavefront or Abdi.

A

Those kind of thing it just runs with all the monitoring tool that he needs and also the security is completely built in so with that. What we achieved is a higher velocity, because the developer can actually focus on the product which they are developing rather than worrying about the infrastructure, so the infrastructure was completely abstracted and the security is actually built into the product say derail. Don't I worry about the security because it comes from a common platform and everything is taken care within the platform.

A

So after provisioning, what happens all the developer? One tool that he needs to know about is gate and which they know really really well right. So he commits the code, then the Junkin pipeline kicks in and it run maven as a build tool. I mean you can use any any of your choice as a build tool.

A

Finally, it converts into their docker image and uploads it to the artifactory, and once the image is uploaded into artifact tree, so we use something called the deployment repo, that's where the net version of the artifact gets updated and then the orga CD, which is our open source solution from in cube, which is used for deploying into the kubernetes. It picks up the latest version of the code and deploys you to the kubernetes and all this happens within few minutes again.

A

So as long as he has a PR which is certified, he can just get into the production pretty quickly. So that's the power of this platform.

A

Of course, there are many tools that makes up the platform today we will focus more on the kubernetes, but there are many tools that we use to build the whole platform and, of course the soft copy is available. Anyone wants to use it, but this is a whole suit of thing that we use to make this platform, and this is the scale at which we operate. So we have for business unit and of course, there are multiple segments. There are around 1200 developers. We have total around 4500, but at least 1200 developers are soft.

A

Today are using this platform and out of all the services we have, 36 percent of the service runs on this platform and all these things we started a year ago, last January we started and then onwards people started migrating to the new platform.

A

We have around 160 clusters and 6k nodes, more than 5,400 namespaces and massive number of ports, 62k watson and every day we have around the 1,300 deployment and it is still growing. So our aim is to make all the services within Intuit run on this platform, so still long way to go 64 percent, but based on the adoption pretty soon we will achieve that number with that said, since we are running it as a massive scale, the problems also plenty. There are many challenges that we faced and, of course, we have solution for that one.

A

But one of the things that we are going to talk about is with respect to the resiliency and with respect to the specific problem, but some of our application faced it in terms of some client end up having 500 errors and to take you through that one. Now we will do it.

B

So we see how the scale at which we are operating. One of the things I've seen is that, no matter how good we design an application system gets what it wants by throwing challenges at us. So these are the challenges that we have run into predominantly in kubernetes platform, which is pod, elimination and no termination.

B

So our clients used to see five zero, two errors whenever there was a pod termination even and they used to see five zero for a red coat on their HTTP client when there is a no termination event that happens so before getting into the details of the termination lifecycle, etcetera. Let us understand the request flow, so, when a client sends a request, we use API gateway in India.

B

So if I give you four words, the request to the alb ingress they'll be ingress, then forwards the request from the node port on any of the node that is registered as a target in the alb targets. From there the node port will look up into the IP tables of the node and then resolve it to the appropriate port and forwards the request to the corresponding port.

B

Now, let's look at the what termination lifecycle right so, whenever the kubernetes deletes a part or V delete a port which is a natural event in the kubernetes platform, API server does a couple of operations in parallel one. It sends a notification to endpoint control. App endpoint controller then removes the pod from the endpoints and then updates back the API server. Api server then forwards a notification to the queue proxy queue. Proxy is run per node, so cue proxies get the notification and they remove the pod entry from their IP tables.

B

While this is happening across the nodes, another parallel event will will happen. The API server will send a notification to cubelet. Cubelet then understands that part needs to be terminated that is running in its node, so it sends a signal signal to the pod and correspondingly to the containers running in the pod. So for fi02 error on the HTTP happens when the Signum signal is not handled gracefully in the application part.

B

It may also happen that when the pod is removed from the IP tables, we still see some key polite connections from alb to the part, because the connections once established they live long until the keep a lifetime mode which is by default, is 60 seconds.

B

So, let's understand what happens specifically when the Signum signal is received by the container. A container shutdown is initiated and then, if we configure a pre stop hook in our pot spec, the pre stop book process is executed and then the signal signal signal is sent to the process and then the process termination starts. If the process is able to terminate within the termination grace period seconds, which is by default 30 seconds, you know it will terminate in the process successfully.

B

If not, cubelet will send a signal which is for stealing the part, and then it will delete the part in the container standing in it. So we need to handle our 5:02 error case by handling the pre, stop books carefully and then gracefully. So when the signal signal is reached to the part, we need to wait in the pre stop hook until the pot IP is removed across all the nodes, such that the no new request will be sent to the part.

B

After that even happens if we are using HTTP keepalive connections, alb maintains a keepalive connection to the pod. So when the Signum signal is received by the application process, it has to handle that shutdown event carefully by closing all the inactive keepalive connections, and if there are connections that are in transit, it has to process them and then, while serving the request back, it has to pass a connection, close header or close the connection in any other way, so that we can proceed for terminating the process, gracefully if we handle in such a fashion.

B

We will never see fi02 error on the client side.

B

So this is a example, pod spec, which is the easiest one where we can configure three stop hook to wait well, say sleep 60 seconds and then termination grace period seconds by default is 30 seconds. If we are increasing the sleep time or any other process that we are doing in the pre stop hook, we need to correspondingly increase the termination grace period seconds.

B

This is one event. The other thing that we have noticed in the infrastructure level is the node termination, and no termination can happen for a variety of reasons. A cloud provider may be doing. Availability zone rebalancing or an auto scaler is scaling in the notes to maximize their socialization, so it may terminate the node so phi0 for error predominantly happens when Elba doesn't respond back to the client in time.

B

That is the timeout configured and the client side and alb may also throw a 5 0 for error when Elba, since the request to the pod and the pod doesn't respond back in time. That is alb timeout. Now there is one more use case when this 5:04 error can happen when the nodes are terminated. Elb ELB has the node in its target group, but the node is terminated. It will not be able to make a connection to the node.

B

So if we, if we learn kubernetes in any of the cloud provider, example, videos que ops to run kubernetes cluster in AWS the cloud provider. That is not aware of the infrastructure of kubernetes that we are running on top of it. So when it terminate instance event happens, it will go and terminate the instance right away, but it doesn't guarantee that it will drain the node. How kubernetes wants it? Not? It doesn't really just up the road from the alb.

B

Those are the events that we need to handle in order to be able to handle that we, we should not directly use the easy to terminate instances function, but when the terminated instance happens, we need to trigger it through terminate instance in order scaling group. The advantage with that is the terminate instance.

B

Auto scaling group provides lifecycle hooks, so we introduced a project called lifecycle manager under the Kiko open-source project lifecycle hooks hooks provide extreme powerful ability to do ec2 terminations synchronously lifecycle manager will intercept such hoops and post an event in the sqs notification engine, and then we pick up the notification and then drain the node as well as drain the target from the alb and then further proceed further note, elimination, which is a graceful event. Now. These are the two pictures where the first one talks about before implementing the lifecycle manager.

B

If you see the lines that are thus landing site where they are in red color, those are the events that we were seeing with the phi0 of four errors when we were running without lifecycle manager and without leveraging the lifecycle hooks. But after implementing it, we have never seen a 5:04 error in our cluster life cycle, so lifecycle manager fits into the reliability component of Kiko project. There are other other other components in the Kiko that you can see like for security, orchestration monitoring, cost efficiency, please feel free to contribute.

B

If you like this project so conclusions in order to be able to be resilient, we need to leverage priests or books and wait for the pod IP to be removed from the IP tables, so that will not receive any new traffic onto the pod. After that, when we send the Signum signal to the pod, it has to be handled gracefully so that it deletes all the keepalive connections. The next thing is on the node termination.

B

We need to use terminate instance in the auto scaling group kind of functions which provide an ability to send a notification event and pause the termination of the ec2 instances, and then we need to leverage these lifecycle hooks to drain the nodes and drain the alb or ELB targets. Before going to the questions, I wanted to stay one statement that kubernetes definitely provides recoverability, but resiliency is something that we need to handle, and we should aim for achieving that.

B

A

A

A

It's not showing up well, we showed a slide on Kiko project feel free to go and contribute it. There are many components that we built address. Different challenges. Resiliency is one such thing we talked about, so there are many components that we use and then we've open sourced it please feel free to use it feel free to contribute it. Thank you.