Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubectl Said What? - Christopher Hanson, RX-M, LLC

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubectl Said What? - Christopher Hanson, RX-M, LLC

Pods in a pending state? Containers in a Crashloop? DNS acting dastardly? (okay that last one was a stretch) You’ve definitely deployed your application but it isn’t available. Kubernetes is confirming the conundrum but what exactly is it trying to communicate? The goal of this session is to explore the causes of various phases, conditions, states, and events related to Kubernetes objects that are produced by kubectl so that you debug from an informed position. By understanding the possible causes of these conditions you will learn where to start, what to pay attention to, and what unrelated messages to ignore. Attendees should be interested in saving time, reducing effort, and minimizing frustration. We will interactively examine Kubernetes objects in various error states and the clues being presented by kubectl get, describe, events, and more.

A

Welcome to kubecon on the last friday. I hope you guys had a good week. I know I did it's nice to be in person once again for these talks um so times that I cherish the most is being able to see all you folks and communicate with everybody here at the con. So uh you're here for kube cuddle said what, because uh we need to interpret sometimes what cube cuddle is trying to tell us about our applications.

A

My name's chris christopher, uh I shorten everybody else's name, so I just go by chris, uh I'm a senior cloud native engineer at rxm, I'm an instructor and I'm a consultant. So when I'm not in a class with a bunch of students, I'm usually uh working on a project to help people adopt kubernetes and cloud native applications, and things like that- and um I like the talk.

A

So that's why I'm here and I've done it in the past, so you can check out some of the other talks that I've had the last couple years here at um kubecon, although it was virtual okay. So before we get started on interpreting what kubecuttle is trying to tell us, we need to think about uh what kubernetes is doing uh to deploy our application right, and so this is a resource creation flow.

A

That shows uh often we're the user here in the the upper right hand, corner uh or upper left and uh we're a little bit disconnected from the actual deployed application. That happens many steps down in this flow, and so there are some swim lanes that happen here and so we've delineated those to show that you create the deployment as the user.

A

It's then or the controller, let's say it's, then the controller's job to actually uh submit the api requests to create your applications or create your pods, and then the scheduler has to do its work to make sure that those things get home and then the kubelet has to kick in and communicate with the container runtime and actually you know, create containers and and have running processes and in any one of these swim lanes. Something may go wrong that you may not anticipate and you'll need to figure out.

A

You know what's going on so we're going to look at each one of these swim lanes in turn here so kubernetes resources, all of them have things like conditions phases states that some of which are summaries. That will give you a little bit of information, some clues about the status of your application. Other things are very detailed and we'll give you exactly the message that you need to know about your application and what's happening with it, so they're all give you some sort of clue to help you with that.

A

When you're first learning kubernetes, it can be a little bit difficult to know where to look and what to pay attention to you know I create a pod or I create a deployment, and it says resource created. Oh cool everything should be great right. Well, no, not really a lot of stuff has to happen in the background before that actually becomes true, and so, if you don't know how to dig in and actually find that information, then you're sort of left stranded and wondering how why why can't I connect to my application.

A

Kubernetes said it was created, so we're going to go through those steps here, so things like get with events, get the object itself. Give you some clues. uh It's always a jumping off point right. We get the object, get some sort of summary information and then go from there. We can use modifiers like minus o, so outputting yaml, outputting, json and then getting specific status information uh using something like status.conditions and extracting the actual message from the condition of of the object and, of course, coup ctl describe is very uh helpful as well.

A

So we're going to look at some practical examples, we're going to start with the the first uh top two swim lanes here, so we're going to focus on uh as a user. We create the deployment and then what happens or the controller rather and then what happens when that controller fails to do what we expect it to do, and so in some cases we can get some details from a command like kubectl describe so for deployment.

A

It has some transient states that it will express so, for example, available. This can be a little bit complicated because it requires understanding of the setting which is max unavailable and the other setting, which is the minimum ready seconds.

A

So some number of pods has to be available for a certain amount of time, based on what was expressed in the deployments, essentially rolling update strategy, which determines whether or not the the minimum number of replicas are available to clients all right, and so this will tell us yes or no is the thing actually available to clients, or are there not enough replicas to to make that available?

A

Progressing happens when we have new rss or so a new rs comes around when we have a new deployment or a rolling update, and so in any of those cases we'll see the progressing status, condition show up or if we have a scale up or scale down right. We're making changes to the existing application and then that shares a condition.

A

um Replica failure with the replica set, so both deployments and replica sets will express replica failure if they can't create a pod for any sort of reason. Stateful sets and jobs don't have as much detail so staple sets have a status.conditions.

A

Part of their object, but it's not easy to find, and it's not very detailed, so it's just not as good as the deployment when it comes to being clear about what's going on with pods under a staple set and then job has one that's very transient: that's whether or not something finished with a zero or non-zero, but if it's still running it doesn't it doesn't even present that to you with cube cuddle right, so it's not queryable at that point until it gets it does its work.

A

So we're going to look at a practical example of uh these guys, so I've got a couple of yamls, I'm just going to apply them and then we're going to see what happens all right. So we created a bunch of controllers, all the ones we mentioned, and so I'm just going to cube. Ctl get the deployment the rs the staple set and I'm using the short terms, because uh I'm lazy um and job right, and so this is kind of our jumping off point right.

A

I have a couple deployments. You can see that they're not whole I've got a staple set. It too is not whole. It has a lot less information for me to understand why it's not whole and the same thing with the job.

A

It's not done yet. I don't know if it's in progress or if it can't create a pod or what's going on with it right so um there's a limited amount of information that gets displayed here. But if we go through each one we can get a sense of um what uh what these different columns mean and how they can help us.

A

So, for example, ready this is an important concept in kubernetes readiness, probes, and so at this point we don't know if um two of our pods have passed their or have not passed their readiness probes or if they just don't exist.

A

um So we can look at something like available and up to date to help us with that right. So if the pod doesn't exist, it's not going to be up to date right and we don't have a new revision of this uh deployment, of course, because it's brand new, so we know that we were able to create two pods, uh but two are missing. We still don't know why the rs's actually have a little bit better details, because they've got the desired versus current and ready, and so now I've broken out.

A

You know a little bit more detail information. We know it's not just that the pods aren't ready, but that they really just haven't been created, because the rs has four pods that it wants to create, but only two exists for the one rs and then this one has zero for whatever reason staple set.

A

Again, we don't know they're just zero of three ready. There could be a pod there, but uh the readiness um probe may not have passed or again, there's no pod that exists there. So what we need to do is dig deeper and ask these specific controllers about their statuses. So I can do cube. Ctl describe on one of these controllers, so I'll do deploy and we'll do the dep e x, and so uh our only condition that shows here uh available as false means that again we don't have enough replicas.

A

So this ties into the max unavailable. This is just the default right, so we only have two of four pods. So, of course, we breach this threshold and we're below the max unavailable, and so we expect this to be false. Replica failure is true. Here's our first indication that the controller is failing to create pods, because it has this set to true, and it's still trying, of course, because it's under progressing said as true, but the only event we really get here is that the replica set scaled up to four.

A

So uh what's going on here, we don't have enough clues, so I need to go to the replica set to really get the information.

A

And, of course, I get a huge amount of events right, so the replica set remember in the flow is the actual controller responsible for creating the pods, so the deployment did what was supposed to do. It created the replica set, but the replica set is not being able to create the pods, because I actually ran this in a namespace that has a quota that would be deliberately violated, so we would actually see something that was wrong right. So knowing where to look is key to this.

A

If I'm trying to look at my deployment when what I'm looking for is missing, pods, well, the deployment's not responsible for that right, it's the rs, that's responsible for that, and so, if I go directly to the rs, I can see events like this now when I don't have that two-tiered um relationship between controllers. Of course, I can go directly to the controller like a staple set and get the same message right there there's a quota in place that is preventing that as well and the job's the same way.

A

So if I, if I queried the job, uh you'd, see the quota now, there's one that we didn't see, that is not part of this quota uh scenario that I've set up. So I'm just going to list all again so there's one more deployment here called saex that is failing to create one pod and it looks very similar to the other problems that we're having.

A

Is it a quota issue? They look very similar right if we actually look at that.

A

Again, deployment's not going to tell me much right because it's not what's responsible for for creating pods. So just wanted to reiterate that so I need to do is go to the rs.

A

So now, in this case, has nothing to do with the quota. This is a failed service account, so the pod spec is relied relying upon a service account that doesn't exist again. I made this scenario artificial. Just so we could have something to um to look at in our troubleshooting scenario, but just looking at the git output doesn't give give us these details right being able to go into the actual events of the controller and the correct controller will give us the information that we need in this.

A

In that scenario, so going back here when we look at the deployment staple set with something like coop ctl get a lot of the issues that we're going to have are are going to look identical and we're going to see later on, with some of these other scenarios that they're going to look identical to those scenarios too replica sets give us the the best details about things like ready versus current, uh not not matching desired.

A

uh Pods are going to be missing right, so I can't query pods that don't exist for their status information. I have to rely on what the controller tells me, because I can't query something that hasn't been created yet, and you can actually see the message field that gets populated as a related condition, so those events will populate into the message field of the the controller, so common causes here are things like quotas missing service accounts, as as we saw now, if your controller can actually create pods, then we're moving down a swim lane right.

A

If we see that pods are created, then we need to go and look at pod objects for their status, their phases, and so some of the macro states that exist for pods include things like pending. So this is when the replica set has actually successfully submitted the pod spec to the api, but the pod can actually be created. Remember that pods are represented by the pod sandbox, which is a container, and so the first bullet here that says that uh the containers have not been created.

A

Right includes that infrastructure container that that pod sandbox container running then, is where we actually have scheduled it to a node, but the containers can be in any state right. So they can be restarting, they can be running. They could have gone into a crash in some cases. We'll get that actually, as a as a message succeeded in failed, very straightforward, zero non-zero exit codes and then unknown is not a.

A

Something you'll see very frequently, but it's a lot of times deals with the communication between the host, where the cubelet is running, where that pod is and it's not updating the status if we go a little bit deeper. So this these come with cube, ctl get pod, that's a again good starting point where we jump off.

A

uh But if we get something like coop ctl describe, we can get uh boolean conditions that let us know if something's ready, uh if in knit containers have been launched- or they have failed to finish via initialized, whether or not all the containers in our pod are are actually ready, passing their probes or not, and whether or not it's been scheduled, and this will give us some more details. So if we look at the the next swim lane here, there's.

A

The scheduler responsible for finding pods that have not been bound to nodes once the controller has submitted them and then binding them to nodes. And so, if we look at some examples, I'll.

A

Clear my cluster.

A

And I'm going to need to change namespaces, so I get out of the quota. So we don't deal with that.

A

So we'll switch to a different namespace without a quota we'll apply a new set.

A

Of pods here and I'll just dump everything, because what we're going to need to do is look at the pods themselves, so we'll use get all now. If we again go back to the controllers where we started the first time.

A

These can conditions look fairly similar to what we saw when there was a quota problem or a service account problem. So they're not going to be very helpful right. We have pods but they're, not passing readiness checks and so that column of readiness includes everything up to the point where that probe passes. So a lot of conditions are trapped. We already saw uh upstream that other conditions are trapped by that, so we can't really rely on the controllers to give us that information.

A

We need to go directly to the pods in this case, so I can use get on a pod. We'll call use the res ex example, and if I go to something like a yaml output.

A

I can see that, under the conditions for my pod, I do have a false for the pod scheduled condition, and then I get the actual message as the reason. So this will take the event that traps the reason why the scheduler couldn't schedule it and inject it.

A

So I can extract it directly, so I can actually traverse this output if I wanted to with the json path, outputter go status, conditions and extract this message um and find that each one of my pods have some sort of uh scheduling, issue right, and so, uh if we go through each one, I'll use describe on the other ones,.

A

We can see that nodes aren't available because of things like resources or volumes and.

A

Because we're dealing with the scheduler and pods the messages are going to be related to the pods. I didn't even bother going to the controllers because they would just lead me on a wild goose chase right. So if I go up one level, let's do describe.

A

Right it did what it was supposed to do and that's the event that we see it created the pods that it needed to create it's just the fact that the scheduler can't uh reconcile the use of the resources so uh going back to the pending symptoms right, the controller outputs are going to look identical to what we saw before when the controller was having an issue, but uh the pods going to actually exist as opposed to before when pods just didn't exist.

A

It's going to return pending, of course, in this scenario, and then you'll see pod scheduled as false as part of its status condition. You can get that message field, but there's not going to be any containers. So the difference between something that you need to troubleshoot at the container level versus the pod level is that essentially container status will be no. There are no containers right because they, you can't create containers unless the pod has actually been scheduled.

A

So some common causes here things like insufficient resources right, if you don't have enough cpu or memory things like node selectors.

A

So if I use a node name, a node selector, an infinity that would prevent that and then volumes. These are kind of the primary three things that the scheduler used to schedule. A pod. The last one here uh priority class preemption, is kind of an advanced use case that I didn't go over here, but can look very similar to the first, which is the insufficient resources, because you'll have a bunch of pods that are running.

A

That may be a lower priority class, and so they will then, if a higher priority pod comes along and resources are scarce, those pods will get evicted in favor of one that has a higher priority and needs those resources, in which case the events you'll see, will look like insufficient cpu or insufficient memory. Because the new pending pods that got evicted there are replacements for the old ones that were running at one point.

A

But unless you see that actually happen right in sequence, it's hard to tell that uh it even happened at all, because the new pods I have no relationship to the old pots that were actually running. So that can be a tough one.

A

To look through so uh running so once we have pods that are actually scheduled and running right, we're getting down to the uh bottom swim lane here, containers each have their status as tracked, so this includes in containers. This includes init containers. This also includes ephemeral containers, if that's enabled on your particular cluster, and so you again, you can use things like the container statuses.

A

To group this information and look through it, there are three different types waiting. This is kind of the default state of a container so anytime, it's waiting to actually be created if it's mounting volumes like configmaps or secrets or if there's a crash loop or an error, it will go into a waiting state running is anything. That's of course got to process and if there's any hooks that have been specified, then those have completed successfully. So you can determine that. Okay, my my hook was ran successfully and same thing with terminated.

A

If you have a pre-stop hook, you won't see terminated until that completes uh and then you'll get your zero non-zero. So that sounds pretty straightforward. Right, everything's, fine right you've got a controller that has created a pod. The scheduler has uh scheduled to a node. The kubelet has created the containers.

A

Why are we even talking about this right? Everything's, just fine, correct right. Well, if that's the case, then we actually wouldn't be here talking about it. So, let's clear our cluster again.

A

And we'll apply our new set of pods.

A

And see what sort of things we get into so uh a little bit fast, with the get all we'll run that again right so again, looking at the controllers right, if we go from the top down that we've done before they look very similar to other other cases, right, it's a hard place to start to debug from. But if we look at the pods directly, we get a little bit more details.

A

We know that they have one in one case two containers, but one of them is in peril. So in this case we get an extra status message. This was not included in the official pod statuses.

A

This is actually to help us, so we know there's something causing an error here, causing a container to fail and get restarted, and so just having a running in that case would be a lot harder to troubleshoot.

A

So there are other cases that we'll look at at the very end of this that help us debug by replacing the macro states of just like pending running with these other detailed uh status messages. So, although we haven't really looked at that, yet there are ways that we can populate the status section and give us more clues when we want to uh have an easier time, debugging right.

A

So in this case I can look at my containers in the container status array and the easiest way to do that is getting the pod and we'll do the restart pod.

A

And you can see that we've got two containers in here and we've got a restart count of four and a restart count of zero right, and so this one's not ready and this one's ready. And so we can very easily see that the database here is failing, and then we have the message for the waiting state. So that's in the waiting stay because it's in the crash loop and the related message right in front of us. If I were to instead, let's say describe this pod.

A

I get similar messages from things like the events but back off, restarting failed container, which I don't know right, because it's just a generic message, but by leveraging something like uh minus o yaml with git, I can relate the actual status message to the container that has the problem right and so being able to customize what I see coming back from cube ctl helps me to debug faster. So I know we're running out of time pretty quickly.

A

Now, there's one use case that I want to point out here. This one seems to be healthy and running just fine. uh This is a an example of a liveness probe and so that's easily determined from.

A

Just describe you'll see that as an event, and since we only have one container in there anyway,.

A

You can see that the readiness probe has failed for that one, so easy enough to get from events, but going back here uh nothing seems to be wrong with this container right. It's running. It's ready. The controller is happy, but if I.

A

Show you where it's running so it's running on worker one and I go over here and I stop worker one from functioning in my two node cluster.

A

If I continue to get this pod, I know that pods not running, but kubernetes is telling me that pod is still running because remember the in the uh the swim lanes, the agent responsible for reporting. These things upstream is the kubelet I just disabled, the kubelet.

A

A

I have no indication that this pod is no longer available to clients from this standpoint. But if I again dig in.

A

I can see that the readiness condition has set to false because the node controller.

A

Found that the heartbeat of the kubelet, where this pod was running, has decayed long enough to declare it unready. We don't actually know what's wrong with this pod, because the kubit is no longer communicating that, but to be safe, we stop sending traffic to it. So this will affect things like endpoints controllers, so this pod would no longer receive traffic from clients.

A

But if I go back to my get still running right, so you have to know where to look sometimes now at a certain point. This is going to go to terminating because kubernetes will eventually give up on the pod or give up on the node and terminate all the pods on that node, because the node is not coming back.

A

However, if we just simply bring the node back, kubelet will heartbeat again and then the readiness that we saw on the describe will actually flip back to true, and it will continue to receive traffic, so sometimes running doesn't actually mean running right.

A

Okay, so um turns out. All uh all was not well what's that so uh some of the symptoms and some of the differences between the symptoms, uh so again controllers, not gonna, be very helpful.

A

uh Running is going to be set on your git pod ready is not going to have uh equal values for your current desired schedule will be true because we're past the scheduling, swim lane, things like ready and containers ready are going to be false because those containers are going to be unhealthy, and so you can find messages like containers with unready status will give you container names within things like different outputs, and the different states of the containers, of course, will base change based on cost the difference between something like a probe failure and an actual error is that running will be true for a probe.

A

Failure. Ready, of course, will be false. Something that's crashing will be in a waiting state, whereas they'll both share the ready, equals false state right now, with the node failure. Again controllers are going to look normal because we think the thing's still going once that timestamp decays about 40 seconds you'll, see a mismatch between ready and available show up.

A

In your cube, ctl output right running and ready is going to be there and that's going to be completely misleading, so common cause, as we saw with, but this is a node failure like kubelet, is down can't report in so what happens when we saw one example, this right error versus crashly back off? What happens when we have other examples?

A

Let me clear this cluster again and we'll apply.

A

A

This is kubernetes helping us so that the status condition of these pods is has to do with what errors right, they're actually independent state. So if I look for this, if I use this command json path, where I get the items so all my pods and I look to look at the phase of all my pods they're, actually all an impending state, but kubernetes is helping us by saying well this one, uh which is trying to mount a config map, can't get that config map.

A

This one has some sort of problem with an image right, a mis type and an image or some other issue. This one is not so clear, just container creating forever, so I actually have to dig in on that one.

A

And now, in this case, it's a secret right we're trying to mount a secret for the pod. It's not working so something like where a container gets stuck, creating forever. It's got a dependency and that dependency has not been fulfilled. Now, in other cases like configmaps, uh better details right, you can see it just with kubectl get, but in this case we have to dig in a little bit more okay.

A

So uh right so, even though the status up phase is always going to be one of the standard phases, getting the container status, state.reason is going to give you more clues as to what's actually happening and kubernetes helps us by surfacing these types of messages for- and we didn't cover this here, because it was in the previous section so that you can help help you debug a common, commonly occurring issues that exists. So now you kind of know what coupe cuddle said.

A

So thank you guys.