Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: In Search Of A `kubectl blame` Command - Nick Santos, Tilt

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

In Search Of A `kubectl blame` Command - Nick Santos, Tilt

Developers want understandable tools. Their tools should tell them, “This change here broke that pod there.” But control loops drive the Kubernetes worldview. In a control loop, Kubernetes updates the cluster to make the actual state match desired state. Control loops do not track why the state changed. Nick Santos and the Tilt team tried to build a tool that traced the effects of each apply. He’ll tell stories about several attempts to propagate and assign blame across state changes. Most of them failed! Or broke Kubernetes updates in frustrating ways! Along the way, they learned about labels, informers, UIDs, owner refs, events, and how kubectl apply works internally. If you plan to write a tool that interprets Kubernetes API objects for humans, this talk is for you.

https://sched.co/ekAv

A

Hi, my name is nick. This talk is called in search of a cube, cuddle blend command. No there's, no cube cuddle blame command right now. This talk is about why there should be one what it would do, why it's hard to build and why you should care about it.

A

Let me start with why I care about it. I work on tiltdown. We want to make developing on kubernetes a pleasant experience. You edit your source code tilt, updates your cluster.

A

We wanted you to help, you understand the progress it's making and how that change is affecting your app lots of tools have this problem and they all solve it in a different way.

A

What does this mean? Exactly? Let me give you an example.

A

You apply a deployment you want to check if it worked, one approach to a cube, cuddle, blind command would trace each cause to its effect and tell you how far along the deployment is. Developers often want this tracking in the reverse direction as well.

A

You know a pod is crashing. Is that pod from the deployment you just created, or is it from an older revision of that deployment?

A

Where did the different containers and sidecars of the pod come from when we tilt team started down this path? We thought this would be easy. It turned out not to be so easy. We tried a lot of approaches. Many of them failed. It was pretty frustrating.

A

I don't want to put you through that. I want to explain the problem to you in the way that I wish someone had explained the problem to us, because this problem has old, theoretical roots.

A

We're going to talk about those roots, we're going to take a tour of some of the tools that try to solve this problem and how they solve it, and I'm going to talk about some ways. Future dev tools might make this easier. If anyone here works on kubernetes.

A

Now I don't want this to be just a talk about how to write tube cuddle blame. If I wanted cube cuddle blame I wouldn't have given this talk, I would have used that time to build my own cube, cuddle, plugin, I'm more interested in the abstract problem and its history, because if you understand the history, you can make sense of the different approaches you can reach for non-kubernetes tools, even non-computers, for inspiration on solutions, and we might to see better kubernetes debugging tools in the future.

A

So, let's talk about why this is hard.

A

The fundamental kubernetes architecture is a control loop. The system runs in a loop. You declare a desired state. What the loop does next is a function of the diff between the desired state and the current state.

A

I tried to describe kubernetes to a friend she researches history of science. She says, oh nick, I I know what you're talking about we've been talking about that for decades. Let me send you a paper.

A

She sent me this paper uh origins of feedback control by auto mare, great paper, old control. Loop papers typically have a ton of calculus because they're really about functions on differences. This paper is about history. It's tons of fun very readable, not much math.

A

You probably don't interact with water clocks and windmills day to, but you do probably interact with thermostats and they have this property. That demonstrates the cyclic relationship between cause and effect that we're describing, let's say the current temperature is 60 degrees. Fahrenheit. You turn the thermostat to 70 degrees. Fahrenheit.

A

You wait a bit, then turn it to 90 degrees. Fahrenheit. The heat is on. Did the first setting make the heat turn on or did the second setting make it turn on? What does it even mean for one change to cause the heat to turn on the other important part of a control loop? Is that it's a declarative system?

A

It inherits a lot of the same problems as other declarative systems. You might be familiar with. uh Css is pretty widespread. You may be scared of css. I was a ui engineer for many years and I think css is great. I think it is one of the great declarative programming languages of all time.

A

uh When you look at the initial css proposal. The inventories of css noted that hey it has this nice property. It makes styles very pluggable. It gives you a lot of room to change the implementation and for different contributors to contribute to the style and to optimize the implementation later.

A

The trade-off is, is that css, like other declarative systems, can be very hard to debug unless you have a good mental model about what the underlying system is doing. I remember an old story from the gmail ui team, where someone added a bad selector that looked innocent and that selector killed the performance of the entire page. It took them weeks to figure out why even to figure out which selector caused the problem, modern css debuggers have come a long way displaying calls and effect.

A

You can see the order the rules are applied in and where they're declared, you can see rules cancelling each other out and you can edit them in place.

A

This is frankly what we should aspire to in kubernetes land, but now we're going to take a look at what exists today in the kubernetes ecosystem.

A

The sample app we're going to look at is a stripped-down version of tilt. We build and push an image. We apply a deployment, we track the deployment's progress and wait for the first pod to become ready, quick review of the life cycle of a deployment. So you understand what's going on in these examples, a deployment creates a replica set and the replica set creates a pod.

A

Then the pod is the bit that runs your containers. That runs your code.

A

We're going to look at how these tools differ, the pros and cons and where they fall down. If you want to follow along there's a repo where you can run these examples, it's at the bottom of your screen there.

A

The first example I'm going to show you is what the tilt team did. First again, we thought this would be easy. Deployment would get a label, for example, we might apply. The label deployed at 10 am january 10th.

A

If you see a pod with that label. You know it belongs to this deployment.

A

Here's what the code look like.

A

This is the kubernetes go. Client library, client go a client go gives you an object, called an informer, an informer handles watching for kubernetes objects. Retrying. When there's an error and notifying your code about every change, we can configure it to only watch pods with the label we care about, which is what you see in bold here.

A

Let's watch what this looks like this is an ascii cinema. Video of the tool running our code is going to print an update every time the pod status changes, and we see the deployment was a success.

A

All of our examples have a crash flag that lets us see how it behaves when the pod crashes dash dash crash. So we can watch this deployment come out and eventually we're going to see it hit an error.

A

Great perfect, we saw it so at this point we thought we had solved the problem. We were pretty happy now. This is very efficient. It only watches the pods that matter. We need to inject labels, but that's okay. We can do that for the kind of tool we're building.

A

Then we launched it and everybody hated this and they complained all the time because we had made a mistake.

A

We assumed that kubernetes would see the label on the pod template and would see that that was the only thing that had changed, so it would update the existing pod with the new label. That is not what happens.

A

The? What happens, and the way it's specified to happen is that the deployment controller will notice that the pod template spec has changed and replaced the pod entirely.

A

It was very important to people. We learned that if nothing changes, the pod should stay as is it shouldn't restart it and people hated the behavior where it started every time.

A

So we looked for alternatives. The first alternative we will look at is cube. Cuddle rollout it ships with cube cuddle the example code that we're going to look at uses cube cuddle roll out as a library, let's watch, so it's going to wait for the deployment rollout to finish and it's a little bit repetitive, but good success deployment successfully rolled out now, let's look at what happens when the pod crashes, we start the command with dash dash crash.

A

We build and push the image deploy. The deployment wait for the deployment to finish and waiting.

A

And it times out, okay, what happens when you dig into how cube cuddle raw is implemented? It's very simple: it watches for changes the deployment, it checks, the spec field on the deployment against the status field on the deployment to see. If it's finished,.

A

Here is the code. The full function has a few more if branches, but it's not substantially more complicated than the checks that you see here. The checks in bolt.

A

They check that the number of updated pods is what we expected it to.

A

Be what cube cuddle roll is really good at is understanding the plants, any information about a pod, that's propagated up to the deployment it can find that out, but because of the information we care about, isn't in the deployment status, it can't tell us why things are failing.

A

Let's look at a different tool. Let's look at helm. uh Helm also has the ability to track progress. It has two flags dash dash above and dash dash watch that let you track the progress of a helm upgrade just as an aside helm is pretty awesome. If you want tips on how to use the go client for kubernetes, I highly recommend you look at how helm uses it. The code is very readable. The code is very well organized of the examples that I wrote for this talk. This was the easiest to write.

A

It uses helm as a library to watch progress and it's a couple of lines.

A

Let's watch a video of what it does again we're going to build and push an image, deploy a deployment and wait on the deployment to come up. Deployment is not ready deployed successfully great. That is exactly what we wanted.

A

uh Let's try again with a crashing pod we're going to apply the new deployment with the crashing pod deployment is not ready. Deployment is not ready.

A

The pod is still not.

A

Ready and it times out.

A

Okay, let's, let's try to figure out what's happening when you unpack the helm code, what does it do? The helm code pulls the status of the resource it just deploys. It has a big switch statement of all the different resource types it knows about, it checks their status and it checks their children's status and it does a type specific check to see if it succeeded.

A

Here's what the code looks like uh it's a bit more sophisticated than the cube color rollout command. The important bit is the get new replica set called in bold we're not going to show the implementation, but what it does is. It makes a best guess of what the current replica set is based on the creation timestamp and whether the fields match the deployment.

A

Then it does a similar check to what cute cuddle rod does to see if the deployment and the replica set are finished.

A

This is a fine approach. It's easy to understand and easy to reason about. The downside is that there is no universal definition of done. Each new resource type has its own definition. In that big switch block that helm team maintains those definitions are easy to add, but still you have to add them.

A

The third alternative we're going to look at is cube spy trace. Now we're not going to look at too much of the code. I will warn you. The code is very dense. It does a lot, but it's very comprehensive and you can learn a lot by reading it.

A

Let's take a look at a video of it in action, uh unlike the other ones. We're gonna pause in the middle of this video just uh just to understand it a little bit better because there's a lot q spice trays starts to hint that blame isn't a straight line. It doesn't go straight from deployment to running pod. We can see the current pod and we can also see the pod it's replacing.

A

Let's unpause the video and watch it roll out now. What's going to happen is that we can see the new pond has become ready and it has replaced the old pod great.

A

Now, let's look at what it means to crash again we're going to watch it roll out and we're going to pause at opportune time right here now you can see that the current rollout is crashing and it's crashing with the error message container completed with exit code one, and you can see that because the current raw is crashing, the previous rollout is still up revision. 10 has not yet replaced revision 9..

A

This is awesome. This is this is what we wanted. Let's dig into how cube spy trace works? Now the other tools we looked at tracked: resources top down, they started with a deployment and looked to its children. Cube spy trace. Doesn't do that. If you look at the bold above, it's watching every pod in the name space.

A

Why so cute spy trace? Does both a top-down and bottom-up analysis? First, it creates tables of every type it knows about. It looks at an annotation deployment to find the current replica set.

A

Then it looks at all the pods in the namespace and at the owner ref at each pod to figure out which pods belong to that replica set.

A

This seems like a lot of work, but cube spy's insight is that you need to do both the owner. References tell you definitively. These pods come from this replica set, but deployment is mutable. It has revisions and the owner references can't tell you which revision of the deployment created which pod they simply don't have that information to find out which deployment created which pod we need the deployment controller to pass some information down the object graph, and we need to check that that info has propagated.

A

So this is great. uh We have to build tables of resources up front. We still have to do a lot of custom analysis for each resource type. This approach isn't easily portable to other types or to custom resources.

A

I want to briefly talk about what tilt does today and it's not that much different than what cuba spy does it's less comprehensive than cube spy, which does a lot of different checks.

A

Tilt does use owner references heavily every time tilt deploys a resource, it grabs the uid that cube cuddle apply returns, it watches all pods and every time tilt sees a pod. It climbs the owner references to see if that pod belongs to the thing that we just deployed.

A

Now we still need to do the top down matching, but we wanted this to work for any type, not just for deployments. So we use a label approach, it's a little bit technical uh but I'll. Try to walk you through it. We inject a label into the pod template spec. They are not random labels, they're and they're. Also not auto-incrementing. Revision labels like deployment controller ads, they're, content-based labels, they're a hash of the pod template spec.

A

This ensures that the label changes if and only if, the contents of the pod change- and this gives us the semantics we want- where kubernetes will restart the pod. Only if the contents change now deployments are special in that they can guarantee that this label will make it down to all the pods.

A

This isn't true for all resource types, so till we'll check. If the label made it down to the pod, and if it did, we will only pay attention to pods that match the current uh template. Hash label.

A

Here's what it looks like uh we're going to watch the video and again we're going to wait till the deployment comes out and we're going to pause. Briefly now when we applied the deployment we got the uid of that deployment and we also looked at the content based label on each pod that we're watching, and we saw that. We actually saw three pods two of the pods had a content label from a previous revision of the deployment. So we're going to ignore those, but we are going to keep track of the new pod.

A

Let's wait for it to succeed and we're good now for completeness. Let's look at what this looks like when it's crashing again we're going to build the image push the image apply. The deployment ignore the pause that don't match the current revision and we can see the error of the pod that corresponded to this revision.

A

So this gives us what we need. I am not super happy with this solution and I'll tell you why all the indexing and owner of traversing means this is very complicated and it's hard to make this perform well. uh The the upside is that it does work well with any resource types.

A

It allows us to track the process of custom track the progress of custom resources. It allows us to properly display events and events are super helpful for debugging.

A

So I want to close this talk with a small rant. If you'll indulge me, I don't want you to come away from this talk saying oh cute, cuddle rod is bad. Cubispide trace is better tilt is the best. uh All of these approaches have trade-offs. Some are simple and fast and efficient.

A

Some are more complex and give you more information, but are slower.

A

When I was looking at examples for this talk, I looked at cube cuddle tree a cube, cuddle plugin that visualizes the owner reference graph.

A

I laughed out loud when I read this comment directly from the code. uh Cute cuddle tree just grabs all the resources in the namespace. It sets the qps to a thousand so that it doesn't get throttled, and this is not me, shaming, cube cuddle tree, because this is what it needs to do, and I hope you see how much that stinks and it's not because anyone did a bad job.

A

It's because these are inherently hard problems to solve in a declarative system and to solve it. We need to do some more work, but if the underlying infrastructure gave us a little bit of help and did some things automatically for us, that would make it a lot easier.

A

So what might that look like? Let's start with the bottom up analysis? What can you figure out when you look at a pod right now? Kubernetes gives you owner references which are good, but there is no api to easily traverse them. uh The kubernetes api is very much optimized for querying types, not procuring graphs objects, which is good if you're trying to build a controller like a deployment controller, but not so good.

A

If you're trying to attribute blame uh the other thing about the owner references, is they don't really tell you which revision did what uh they only tell you which resource it came from then there's the other side. What can you figure out if you're, just looking at a deployment?

A

What's weird, is that there are a lot of common patterns uh that you see in different resource types?

A

Most controllers have a way to propagate information from the parent to the child, to see which revision caused, which child change and lots of controllers have ways to propagate status from the child to the parent, but they all do it inconsistently.

A

They all reinvent the wheel every time, even though we've kind of established the patterns by this point uh and we're getting to the point where it might be worth thinking about. What would a generic api look like what are the common idioms that all controllers should be using for this?

A

Lastly, I want to tease you a bit.

A

The kubernetes ecosystem is only going to get more complicated, uh we're seeing more custom resources, we're starting to see mutated mission controllers, which are really cool, uh but they modify, pause and add sidecars and add volumes and other things, but we can't tell they came from those mutated mission controllers.

A

What would a debugging experience look like that was as nice as the css debuggers of today that lets you see a hierarchy of rules that let you see what change each one made and that lets you attribute blame.

A

So, thank you very much for listening to my talk again. My name is nick. I want to give a big shout out to ellen korbs and sasha mumbartz, who helped me put together this deck and gave a lot of feedback on this talk that helped make it a lot better than what it was at the beginning.

A

I also want to thank all the projects I featured in this talk. Client go cube, cuddle, helm and cube spy. I hope none of them think I was criticizing them because I'm not. I think these are all awesome projects, and now we have some time for questions. So, let's go to the questions.