Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Magic of Kubernetes Self-Healing Capabilities - Saad Ali, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

The Magic of Kubernetes Self-Healing Capabilities - Saad Ali, Google

Kubernetes is used to manage large-scale clusters with hundreds or thousands of nodes. Components will inevitably fail at this scale, but human intervention to detect and correct these failures is unsustainable. This is where Kubernetes shines. With a declarative API and guarantees of eventual consistency, Kubernetes is able to continuously monitor the system and take actions to keep the cluster healthy. This talk provides an architectural overview of the self-healing capabilities of Kubernetes. We will discuss how these capabilities are born from the declarative API, and how Kubernetes components work together to drive to the desired state. We will provide detailed examples from the volume subsystem which automatically makes storage available to containers. We will end by revealing edge cases that Kubernetes currently doesn’t handle well, and explain the plans to address these issues.

https://sched.co/MPcA

A

Hello, everyone we're gonna, get started in a few minutes.

A

My name is Steven Augustus I'm, one of the program committee members for cube con I'm, also one of the sig chairs I'm, just gonna, be introducing Sode Sode is a senior software engineer at Google he's also one of the co-chairs for cig storage and he's gonna be going over a little bit of the self-healing capabilities of kubernetes, so walking through the declarative, API and and how you can get to the point where you can run thousands of clusters on kubernetes and expect the same, consistent behavior out of the clusters.

A

So so, during the question and answer section, we want to make sure that the questions are actually picked up on video. So when you raise your hand, I'm gonna walk over to you and give you the mic just so the questions are picked up and yeah. Take it away perfect.

B

Thank you so much how's everyone doing today.

B

Thank you. So this talk is going to be fairly basic for those of you who are already familiar with kubernetes.

B

The goal here is to talk a little bit about the underpinnings, the design principles behind kubernetes and why it works the way that it does and how that enables you to have kubernetes, basically self-heal, detect and self-heal any issues that that come up within your cluster, so kubernetes is now being used to manage clusters all the way from a single node, all the way up to thousands of nodes, and, as you all know, at that scale, failure is inevitable. Something is going to die, a disk is gonna die, a node is gonna die.

B

Something bad is going to happen at some point and requiring human intervention to detect and to fix that is infeasible. It's also unsustainable, so we have to have automation. The beautiful thing, of course, is that this is where kubernetes shines. This is the purpose of kubernetes. It is a system that monitor detects and rectifies any issues that come up within your cluster, so real, quick.

B

The agenda here is going to talk about we're going to talk about how kubernetes does self-healing then we're gonna move on to an example of that and finally, we're going to talk about some areas for improvement. Kubernetes is obviously not perfect and there are some issues with being able to detect problems and fix them and I'll talk about those at the end and then we'll save cute questions till the very end.

B

So how does kubernetes self-healing work? It's fairly, simple, there's two pieces involved: one is the fact that kubernetes has a declarative API, and the second is that it uses controllers that observe and rectify. So that's, that's! Basically it let's dive into that. Before we talk about declarative, API s, it's important to understand what the alternatives are. The alternative is an imperative API with imperative api's you, you basically specify exactly what you want the server to do. So, if you wanted a specific node to start a container, you would call out to that specific node.

B

There's a server sitting there. That's listening an agent, you tell it to start a container, and then it gives you a result back to say. Yes, I was able to start that container or no I wasn't able to start that container. This seems like a fairly reasonable way to build api's and in fact many many api's are built this way, but there's a problem with building a distributed system off based off of purely imperative. Api s and the challenge is what happens when things go wrong. What happens when there is a failure?

B

So you told the the node to start a container, but maybe the node may be the container crashed, maybe the node itself crashed. Maybe there was a network partition issue that is preventing access to that pod. How do you detect that? And how do you fix that? Do you write your own tooling, so that is the one of the main reasons that we for kubernetes based the entire API off the entire API is a declarative, API.

B

So let's talk a little bit more about declarative api's, so, like I mentioned with an imperative API, the process is very manual. You provide the exact set of instructions, the system executes those instructions and then you monitor the system and then provide corrective instructions. If things deviate with a declarative, API everything is automatic. You declare the desired state. What you want the intended state to be, and then the system works to drive towards that state.

B

So, let's walk through a example: we're gonna create a pod and we're gonna demonstrate how declarative API czar the kubernetes way. So, instead of calling out to the master or calling out to a specific node to create the pod, what you do is you construct a API resource that declares what you want the system to do so in this case, we have a replica set object that says: I want to create a pod that starts this specific container.

B

In this case, engine X and I want there to be one instance of this to be running on my cluster at all times. This is your intention and you're. Creating this API object as yamo against the kubernetes api server. When you do that, that object is persisted on the master, server and you're done now.

B

Everything in kubernetes is going to be monitoring the kubernetes api server and figuring out what to do next. They're gonna react and have the pod scheduled. Let's take a look at that a little bit closely. So when I say everything in the kubernetes world is watching the API server, I mean everything, so the master is broken up into multiple components.

B

You have the API server as a central piece that keeps track of what the desired state is, and then there are multiple other components in the master like the scheduler, the node controller and other controllers that are also watching the API server. In addition to that, all the nodes have an agent called the cubelet that is also watching the API server and reacting. So when you, as a user, declare something on the API server, it gets persisted there. Anybody that's watching. The API server gets a notification to say hey by the way.

B

There's a new object. Are you interested in it? So in this example, the scheduler is going to notice that there's a new pod that hasn't been scheduled and it's going to immediately run its algorithm to figure out where the best place to place that pod on the cluster is going to be, and you would think that the next step would be for the scheduler to then call out to the selected node and tell that node to start the pod or start the container, but that's not the way that it works with the declarative, API.

B

Everything goes through that central API. So, instead of calling out to the node the scheduler once it makes a decision about where that pod should be scheduled, updates the kubernetes api server, the API object for the pod to say this pod should be scheduled on node. Let's say in this case B and now node B has an agent, that's monitoring the API server and it notices. Oh there's a pod that is scheduled to me, but I know that this pod is not running.

B

So let me make sure that that happens and node B is responsible for starting that container and it's responsible for keeping that container running. So, if pod, a crashes for whatever reason, the the node agent is monitoring the API server notices that hey I should be running this pod, the pod is terminated. Let me restart it to make make things happy.

B

And so this system is going to keep your pod running until you delete the intent. The system basically is going to try to achieve eventual consistency until your intent changes and the way that you change your intent. Is you delete the object that you created? You delete your pod object. It disappears off the kubernetes api server and now all the components that are watching it are going to get that deletion event and they're gonna say. Oh for example, in this case, the cube lid on node b is going to notice.

B

This pod does not no longer exists on the api server, so I need to terminate it from terminated from my local instance and it'll go ahead and terminate that pod. So that's the very basics of how the kubernetes api works. It's a declarative, API, and this is the basis of self-healing in kubernetes. This pattern is repeated again and again and again to achieve a system that together is self-healing.

B

So, let's talk about what some of those benefits are. The major benefits of a declarative API are that they give you automatic recovery.

B

There is no single point of failure.

B

The master components are going to be simpler, so if you had basically put an imperative API between the master and the nodes, all the logic to detect, for example, whether a container is failing and what to do about that would now be have to be moved into the master and your master node logic becomes much more complicated. It'll simplify your your nodes, but it'll complicate your masters with a declarative API every component within kubernetes is responsible for itself.

B

So the logic, the complicated logic is distributed to each component and it makes it much more much easier for each component to be revved independently as well.

B

So again, the resulting system is simpler, it's more robust and can easily recover from failure of any component.

B

So I talked about the declarative API. That was the first piece. The second piece of this is controllers. Controllers are just processes that are running monitoring the API server and reacting to it.

B

There are a number of controllers that ship out of the box with kubernetes the controllers you're, probably most familiar with- are, for example, the scheduler itself as a controller, the node has a controller. There is a replica set controller. There is a volume controller, persistent volume controller, as well as a volume attachment controller and many many other controllers that ship out of the box. You can also write controllers of your own. Now. What these controllers do is they maintain an in-memory cache of what the desired state is?

B

They will watch the kubernetes api server and keep a local cache of what the user intends or what the the system wants to happen, and then they also keep a in-memory cache of the actual state, which is what they believe is the real world state that they have observed, and then they have a simple control loop, which is literally just a for loop that will DIF your the the in-memory desired state with the actual state. When these two things differ, the controller kicks off an operation to try to rectify and make sure they match up.

B

So in practice, what this means is that, let's take volumes as an example, the volume attachment controller has an actual and a desired state. The desired state is populated by a component. That's watching the kubernetes api server, when it notices that there is a new pod, that's created with a volume and that pod has been scheduled to a node and that node does not has not been is not already part of the desire State. For that volume, it's going to update the it's gonna update the.

B

B

There we go okay, so the the populate er will update the desired state and the desired state will be updated to say, hey. This volume should be attached to this particular node, and now the reconciler, which is differing between the desired state in the actual state, notices that there there is a volume that should be attached. That is not currently attached. As far as it knows, so, it's going to trigger an operation against that volume type to have that volume attached once that attached operation completes the attached.

B

Detached controller will update its actual state to say. I now have a volume attached to this node. Now the reconciler, which is a dumb loop, is dipping the desired and actual states, and it says: okay, they match, there's nothing else that I need to do now. If the volume, if the pod is moved to a different node, then that volume needs to be attached to the to a different node.

B

The desired state again gets updated by the populate ur and the control loop notices that there's a difference between the desired state and the actual state, and so it's going to trigger another operation to first detach the disk from the old node, because it notices there's no more need for that disk on the old node and then it'll start another operation to attach that disk to the new node.

B

And the reason for that is because that control loop is noticing a difference between the desired and actual state in memory caches, and so these control loops and these controllers themselves are fairly simple. They're, just monitoring the API server they're, keeping track of what the API server is saying: they're, observing the reality on the ground, they're keeping track of that, and when that differs, they start operations to make sure that they are bringing the system back into the desired state. And so each component within kubernetes is doing this.

B

The I talked about the volume attachment controller is doing this to make sure volumes are attached. The scheduler is making doing this to make sure all pods are scheduled. The replica set controller is doing this to make sure that there are always as many number of pods as requested on a replica object. So when you create a replica set object, you say: I want two pods. There is a replica controller which is monitoring the API server and trying to figure out how many pods are actually running in the system.

B

All of these controllers more or less operate. In the same way, there are small control loops that are dipping. What the desired state is against, what the actual state is and then triggering actions to drive towards that state.

B

Ok, so we talked about controllers and, let's walk through another example of automaticall recovery, just to drive the point home, because this is all about self-healing.

B

All the components in kubernetes are using these patterns, basically to implement the the self-healing ability. So we talked about this example where you had a pod that got scheduled. It got started on a node now what if the node dies?

B

So I alluded to this earlier, but here's a picture of it. Basically, the master also has other components, in this case the node controller and the replica controller, the node controller monitors for heartbeats from node from from the nodes so periodically about every 10 seconds, or so every node, every cubelet posts back to the api server on the node api object to say I'm alive, I'm alive, I'm and the node controller basically monitors the API server for node objects, and it looks at when the last keepalive was. If it was more than five minutes ago.

B

It declares that node dead and evicts the pods on it, and so what it does is hey I haven't seen a heartbeat from this particular node for five minutes. So let me go ahead and delete all the pod objects on that node, because, as far as I'm concerned, that node must be dead.

B

Note that this node controller itself is not reaching out to the node to tell it terminate the nodes or reaching out to another node to tell it to start the pods, the node controller just talks to the API server. It just deletes the pod objects, and so once the pod objects are deleted. The replicas cook, the replicas controller notices that there used to be, let's say one instance of the pod running and that's what the user requested. But now there's zero.

B

The user requested one now, there's a zero, so it'll create another pod object automatically on behalf of the user, to ensure that there's always one running when it creates that pod new pod object. The new pod is unscheduled, it's just it's a brand new pod and so then the master scheduler, which is just another controller which is watching the API server, sees a new pod, which is currently unscheduled.

B

It'll run its algorithm to figure out where that, where that, where that pod should be placed and it'll update, the API server object to schedule that volume or schedule that pod and then finally, the new node is going to take over and notice. Oh I have a pod that should be running on me. Let me go ahead and get that running now. The interesting thing is what happens to the old node. It depends on what type of failure happened.

B

If it was a network partition, then the old node doesn't I can't reach out back to the API server. So it doesn't know that the that the old pod was terminated so it'll keep the pod running. If the the node can't reach out to the API server, it maintains the current status. It keeps things running as it lasts odd. The observed the desired state to be as soon as that connection is re-established. It's going to notice. Oh the somebody on the master deleted this pod.

B

This pod should no longer be running and it'll terminate that pod. On the other hand, if the failure is just the node is completely unavailable, then you don't really have to worry about anything. The the node is content and the pod is terminated.

B

So, let's talk about some of the challenges with this kind of model, one of the biggest problems is what, if the actual state cash drifts from the real world- and this is particularly problematic in the storage space. So, for example, the volume attachment controller which is responsible for ensuring that volumes are attached to the correct node could have its actual state skew from what the real world is.

B

And what happens in that case? Basically, what that means is, for example, somebody went behind kubernetes back and detached a volume. Kubernetes doesn't know about it. As far as the attached detached controller is concerned in its in-memory actual state, the volume is attached. So it's happy so to address issues like this. What we do is, we add in logic that will rectify the actual state with observed, real-world state so, for example, the actual state. This sorry, the attached detached controller, has logic to periodically poll which volumes are actually attached.

B

So it goes through the list of everything in its actual state and says: is this attached as this attached is this attached just so that if something becomes unexpectedly detached, it can remove it from its actual state and reissue the attached to make sure everything is okay.

B

Most controllers should be doing this in order to make sure that what they think the reality is doesn't skew from actual reality, but not all controllers do and one example of this is the volume mount code. So there is a controller, a volume manager on the cubelet, which is responsible for ensuring that volumes are mounted into the correct places to make them available inside the container. Now the problem is that the controller can lose its state right if the controller crashes, this is all in memory state.

B

All of that state is gone now when the controller starts again, it needs to rebuild from a source of truth for most controllers the simple ones. The source of truth is the kubernetes api server and that's sufficient, but for some controllers they need additional information and so, for example, for the volume manager it needs to determine which mounts actually exist on the machine and then figure out if those mounts are actually desired and now based on the way that it was initially coded, the it has. The the controller, the volume manager controller has insufficient information.

B

So if a volume, let's say a node, becomes unavailable, there's a network partition. The pod gets deleted and, at the same time, cubelet crashes.

B

The pod gets deleted the cubelet crashes and then everything comes back up. The No hasn't restarted, so the mount still exists, but the pod has been deleted on the API server. So you can't use that as your source of truth, and the only thing that exists is this orphan mount on on the on the host in the past. What used to happen was that those were not cleaned up at all and then starting in about to Burnett he's 1.5. There was logic added to try to clean up those orphan mounts.

B

The problem is that the only information we had was the information in the path for the volume mount and that would include, for example, the volume identifier, but sometimes if there wouldn't be enough information to actually do a clean unmount, and so this code, that does the reconstruction of volume mounts and cleanup if a mount is orphaned, is actually a source of a lot of bugs for us, it's something that we're looking to fix, and so with our new container storage interface implementation.

B

What we did is, instead of just having a volume mount, we also have a file that is persisted in addition to the volume mount and that allows us to do proper reconciliation and clean up any orphan mounts as needed.

B

Now. The last challenge I want to talk about is.

B

Eventual consistency can take a long time and what I mean by this is: let's take a the node shutdown example. If a node shuts down unexpectedly, it will take the node controller, five minutes to declare that node dead and after five minutes it's going to delete the pods.

B

After that, when the pods are deleted, the volume attached controller is going to try to detach the volumes that were on that node used by those pods. But the volume attachment controller tries to be as safe as possible when it is trying to detach. So it waits for a signal from the node to say that it has safely unmounted the volume before it detaches it, but it also doesn't want to get stuck if the node is an unhealthy state. Eventually it will unilaterally detach the disk and that timeout is six minutes.

B

So it'll wait six minutes to hear from the node. If it doesn't, it assumes that the node is dead and then it pulls the disk, and then the volume is free to be attached and the new pod can come up and get started so overall you're, looking at over ten minutes to get to recover from a failure, that's pretty bad, but what you need to understand is that kubernetes gives you automatic recovery, not high availability. This is a very key point that I think a lot of people don't understand.

B

The promise of kubernetes is not high availability. It's the promise is that the system will eventually recover.

B

There are no guarantees on how long it's going to take to recover, and if you have requirements for high availability, then you need to build those into your application layer. You need to make sure that you are not dependent on a single pod as your single point of failure, because when that pod goes down, there is no guarantee for how long it takes to get that pod up and running. If you have an application that desires, high availability make sure you're using multiple pods, and that is my last point.

B

Thank you very much. Any questions.

B