Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Making the Most Out of Kubernetes Audit Logs - Laurent Bernaille & Robert Boll, Datadog

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Making the Most Out of Kubernetes Audit Logs - Laurent Bernaille & Robert Boll, Datadog

The Kubernetes audit logs are a rich source of information: all of the calls made to the API server are stored, along with additional metadata such as usernames, timings, and source IPs. They help to answer questions such as “What is overloading my control plane?” or “Which sequence of events led to this problematic situation?”. These questions are hard to answer otherwise—especially in large clusters. At Datadog, we have been running clusters with 1000+ nodes for more than a year and during that time, the audit logs have proved invaluable. In this talk, we will first introduce the audit logs, explain how they are configured, and review the type of data they store. We will then demo a functioning setup and show a few different types of analysis techniques. Finally, we will describe in detail several scenarios where they have helped us to diagnose complex problems.

https://sched.co/Uab7

A

All right welcome for our first talk of the morning. Here we've got a great talk by Rob and Laura from data dog and they're on the infrastructure team there and they're gonna talk a bit about the Ottawa capability of kubernetes. Give him a big welcome.

B

Hi folks, my name is Rob ball I work on the infrastructure team at beta dog as an engineering manager, hi.

C

I'm long beyond I awoke on the same infrastructure team and we focus on communities clusters. So if you don't know data everything, okay, of course,.

C

C

Well, do W, we don't the dead, a dog, whereas s based company and we do monitoring. So we do infrastructure, monitoring a p.m. and logs, and we had a large customer base and repossess a lot of data points every day we are.

C

You have a few actual figures on the left-hand side of the slide and on the infrastructure chain were actually focused on the infrastructure that is powering the product and we're talking about tens of thousands of hosts and tens of commuters clusters of very different size between 50 and shree thousand nodes for the bigger one, and to give you an idea about, half of them are pretty small, so in the 50-100 node range and the other odd is pretty big between 1500 and 2000, also with multi-cloud and we're growing pretty fast, which comes with quite a bit of challenges, and so a lot of times.

C

We have to understand what's happening, why we have issues and we need to debug deeply what's happening and to do that we're actually using other drugs. This is what the talk is gonna be about.

B

So, first to understand what benefit we're getting out of our logs I think we need to dig in a little bit for some background on the kubernetes api. So let's do that kubernetes api is the set of REST API is provided by the API server. It's backed by a database called that CD and there's a lot of clients. So let's look at what those clients look like.

B

First, we've got the kubernetes control plan, so a bunch of built-in controllers and the scheduler that's in charge of scheduling, workloads on two on two nodes and then some node daemons, like the cubelet that powers most of the kubernetes functionality and cube proxy, which does some networking things and probably a bunch of other no demons like monitoring, demon or CNI or CSI, or things like that and they've got some cluster services of dns and ingress controller is autoscale or external DNS certificate manager, probably a bunch of other kubernetes native applications that you've written in-house and then finally users.

B

So that's might be tools that are using acute control or users that are doing this in an interactive way, so get kind of a complex graph, a lot of different things all reaching the API server, and do it that the API server is even calling itself. So this can be tricky to debug issues, and this you know, sort of a lot of complexity here in this graph, there's even a little bit of hidden complexity in queue control in the way that it works. So, let's take a quick look there as well.

B

Looking at the simplest case here, write a simple get ball and what we see by using a hyper Basit e level on cube control. We can actually see every API call that it's that it's generating so there's some important details exposed here.

B

You see what API server we're reaching the API version of the objects we're looking at what namespace those objects are in as well as the resource type and in this case, since we're just friendly we're getting a single resource, the resource name as well with some more complex, different types of API calls when we do different operations.

B

So in this case, when we do a list, it's actually very similar to the get, but we're missing the resource name, and so we can see how that URL translates it's very similar except missing the resource name at the end and then keep control also does automatic pagination, and so you can see that it's limiting to 500 resources.

B

So if you have a lot of resources that are being returned, you might actually see multiple API calls for this one operation with some more get examples, if you enable watch that actually results in two different API calls. First, the same get that we saw before there returns the list of objects and then a subsequent get. That's actually a watch with a resource version.

B

So we take the resource version from the first list and we look for what this is going to return as all of the changes that have happened, since that first representation we received described is also an interesting use case. So when we look at keep control describe, there's actually a couple different API calls here. The first is what you might expect: we get the resource, we understand the whole representation of it.

B

The second one is a little bit surprising where we're looking at it get on events, and so this is filtered using the involved object of the event, and so the described is actually looking at multiple different types of resources and then stitching them together on the client side, and so again, so you know a lot of different API calls all coming from what looks like a simple operation.

B

There's also some different patterns for mutating requests. So when we do a delete, this is actually three different. Api calls. First, it's a delete, and then it's two subsequent gets first and normal get and get with a watch and what's happening here is this is providing the user like a synchronous interaction?

B

So first we delete the resource and then those two subsequent gets are scoped by some metadata on the object. So we're not getting 404 is when we try to get the object when it's in the path. Instead we're watching for updates on that object and we'll know when they light when the delete is eventually completed, and then there's some higher-level operations that we might want to look at as well. So like keep control create where we're doing a lot on the client side.

B

Keep control is actually generating the payload based on some minimal data that we're passing it in this case, just the image we're building up a whole payload based on the spec that we know for deployment and then using a post request for this and keep control scale does a similar thing where we're doing an update in place. So we first get the representation of the object that we want to scale, and then it generates a minimal request body and uses a patch API call.

B

So why do we go through all of this to understand and I talked about Auto at low and slow they're trying to demonstrate the complexity in the kubernetes api? There's a lot of different components, making calls to kubernetes api and they're doing a whole variety of different API calls. That may not be obvious from the surface. These simple user operations translate to a lot of different calls, and so how can we help? You know with audit log self understand, what's actually going on.

C

So other clothes are actually logs that are going to give you information about all the API calls make sure you've made to your cluster. So just to give you a quick definition, it's like a rich, structured, JSON log output pattern. Api server when you get the API call with additional metadata. You can configure the velocity and what's important, is logging can happen a different stage of the request processing and to give you an idea when a client is doing a request to the API server, so they are room with a one on top of it.

C

The API server is going to process this request and it's going to answer and when the answer is so when the API server receive the request, it can log an object at state request received with all the information from the request, and if you look at state streets of the final, the answer the API server is going to look at state response, completes for a normal operation and respond started for watch, because when you do a watch, it's actually a long live connection to the place.

C

Every winner gets ultimate education to the objects we you're watching. So other clocks contain a lot of very interesting information, and you have a few example of error and we're going to dive into that. So, let's get back to the first gate. Example Rob game earlier when we're just getting a part, so you have the cube CTL come on at the top and the resulting API call and do what you see at the bottom and on the left hand side is the audit log.

C

So of course I'm assuming most of you can't read it because it's very small but we're going to dive into the different part of it. So the first thing you get is what happens so which resource was targeted, so you can see on the first line, the URI. So it's exactly what we got under arrest call and and then you have the verb that was used and in that case it's gets and the data is very structured. So the URI is passed, so you can get a lot of informational happens.

C

So, for instance, you have the API version, the name spare that was used the resource. You were getting and in the name of it and finally, you get the status code, which is here 200, which means things are working. Fine.

C

Another information you get in the audit logs is who did the query so in this example, here I was doing the query, so you have my username and also the API server. It is going to add additional rich information, which is the group you are part of which is interesting, because then you can see why request succeed or wait failed. What's interesting is in very recent version of kubernetes. You can also see the reason why you call was authorized so yeah.

C

This authorization block where you can see that Michael was loads because my user was part of a group. That's bound to a role with the proper permission, and so that's why I can actually do the get call. You also have additional information, so you can know exactly when the request was made and also when it was finished. If you're looking at this stage response comp list I was showing earlier and by using the difference, you can actually add the latency of the call so processing this get worried actually took 14 milliseconds.

C

An additional piece of information you get is where the cargo is made from from which IP address, which can help you troubleshoot issues of where things were coming from. Let's look at another gate code, so you remember from earlier. Rob was in an example of a get pod, which is a global list of all the pods in the namespace, and you can see there that it's very similar except the verb is not get this time, so the API server is clever enough to transform the HTTP method into a via. That makes sense based on context.

C

So here it's not a get call. It's actually a list call on the namespace, which gives you all the above permissions you need. If you want to allow gate and disallow list, for instance, and and so yeah, the gate is mapped to different verbs.

C

We also showed watched before so when you do a watch. Usually what happens? Is you first? Do a first list where you get all the information and then you you do the watch itself and you can see on this example. On the left hand, side the first list call and then the watch call so I like it just a few thing there, so both of them are get calls by dam up to different verbs once again list on the left-hand side and watch on the right-hand side and they appear at different stages.

C

The list call is the call that terminates immediately, so it's captured at response complete and the word code is actually long lasting calls words captured as respond started and, of course the watch go is starting after the list go because the fan first does the least than the watch.

C

If we look at the create call, so the call where work was creating a deployment earlier, we can see exactly the same type of information. This time the verb is create, so it once again, it's not the first verb and there's actually additional information in there, which is the request object. So, depending on the velocity you configure, you can get different types of information and in that case we actually capture the request object and if we zoom on the request object, we can actually see the full spec of the deployment that was created.

C

So early locks contain a lot of very interesting information. The API server is adding very interesting metadata, so I hope. We convinced you that a lot of things we can get out of them and now, let's see how we can get them.

B

So, let's look at how you configure your kubernetes control plan to output audit logs in the most basic configuration. It's just two flags on your API server. The first is a path to the file that you want them output to, and the second is the path to the audit policy file, which is the description of how you want the API server to perform auditing. There's some advanced options that you can set here.

B

There's some stuff around rotation and alternative backends like the web hook back end, which will output to a web hook instead of to a local file and then there's some options around async async' batching. So this is a way that not output an audit record for every API call synchronously with that API call, but instead batch them up for performance reasons.

B

So one thing to be aware of here is that there are some security implications. So if you enable audit logs and for an exposed, API server- and you start getting a lot of unauthenticated- calls to that- API server, you're, getting audit records for every one of them and so you're opening yourself up to potential denial service. There so just be careful about where you're, exposing that API server when you have these enabled the policy file is really where all the magic happens, though. So, let's take a look at that.

B

That policy is basically a list of rules in each of those rules is describing the operations that we want, the API server to generate audit records for and what those audit records should look like. So the first section of each of these is a description of what resources based on the API group and the resource type we want to match and what verbs we perform on those resources. It probably looks familiar to you if you've ever looked at kubernetes our back rules, it's very similar way to describe resources.

B

So in this case you can see we're just we're describing pod resources in the core group and all mutating operations on them, so create update, delete as well as patch will all generate audit records and in the second part of this rule, is describing the level and stage for matching API calls. So that's basically telling you when to log at what point, in the request lifecycle to log and what to log so here we're configuring the request response level and that level is gonna log.

B

Everything about the initial request, including the request body, as well as the response in the response body, so you're gonna get a lot of information.

B

There there's a few gotchas to be aware of when you're writing these policies, though, when you're writing these the set of rules, they're evaluated in order top down in that file, and so we have to be really careful about setting very broad rules at the top of the file, because once it matches once it's not going to match a second time request level will include request, metadata, full request body and request response as the response body as well, and so that can be really powerful.

B

You can understand everything that's going on in those API that the full lifecycle, but you need to be really careful about them for large, large payloads. So if you're doing things like the logs API call, you maybe end up logging all of your logs into your audit logs.

B

In addition to that, you need to be careful about some security, sensitive endpoints, so token review, you're gonna end up with your with your tokens logged out here, other logs as well, and finally, the API group, when, when you leave that empty that references, the kubernetes core API group, so that's not gonna match any third party or custom API services that you have so make sure if you want them to generate audit records there, that you're explicitly putting them into your policy files.

B

I want to provide some recommendations for you to go, write your own policy files. So our first one is to ignore the request receive stage it's typically redundant, where we don't need to log at both the time that we received the request and the time that we send the response we're getting much the same data.

B

We want to use at least the metadata level for almost everything. So we want to ignore potentially health and metrics endpoints, but for everything else. That gives you a really good, a really good way to understand all the operations happening in your cluster and use, request and request response level for critical resources and verbs. This can be super valuable for doing retroactive, debugging and understanding the state of your cluster and how you got there just again be careful for those larger, sensitive request, payloads or responsibilities.

B

You don't want to have that stuff end up in your audit logs inadvertently. There's some great links here, we'll publish the slides later that you can check them out or vacuum fishin on the audit policy, as well as a pretty complete example coming from gke.

B

So some quick takeaway is getting audit logs is pretty simple. You set two flags on your API server and you're, getting them out to a file, and then it's up to you to collect them and put them somewhere, but getting the policies right is harder. You're gonna get a huge volume of logs, so be prepared, don't overwhelm downstream systems that are processing your logs from your cluster and it requires a good solution to analyze them and be able to do some interesting operations that understand them.

B

So, let's take a minute and look at what audit logs look like on a real large cluster. Yes,.

C

Now that we're able to capture audit log- and we know, what's in ZL- it's let's look at what we can take out of them. So the way we approach it is, we took one of our larger cluster cluster with I think 2500 nodes, and we looked at what was happening on this cluster.

C

So this is like a very simple graph, showing the number of Cole we're getting on this cluster on the API server and, as you can see, we're getting about 900 calls per second, which is quite a lot, and the first thing we wanted to do is say well. Who is actually making this call. So we grouped all the call by the user that was making the call and looked at the top 25 users, and you can see they're the first, the first five.

C

So the first one is the API server making call to itself to verify things usually and the two following ones are to proxy and local volunteer vision.

C

Er, which are two daemon, sets running on every node, so the first one you're, probably familiar with its listing and watching endpoints and services to make sure that you can do client-side balancing on your hosts and the second one we use to provide local volumes to our applications, and so this one is actually updating the local volume and getting person, volume, information and the fourth one is the cron job controller, which is in this case very dependent on this tip and this cluster. It's because we have a lot of cron jobs.

C

Of course, the cron job is very active and the fifth one we also wanted to mention, because it's spinnaker, so spinnaker is a tool we use to deploy it to our cluster and spinnaker is doing a lot of queries because it needs to maintain a cache of everything, that's in the cluster. So it's doing a lot of lists and get calls on the cluster and that's why it's so high.

C

So if you do the some of these and you're gonna notice that we're not getting anywhere close to the 900 queries, I was mentioning before I mean well. The total was 900 queries per seconds and we look at the top 25. They only account for about 150 queries, so what's happening to the 85 80 percent of the other calls, so maybe grouping by user. Wasn't such a great idea.

C

So, let's now group by groups, ok, so instead of grouping by users now grouping by the group, the main group used by the better calls- and while we notice that the main one like by far is the system. Note group and the reason that then showed before is because each node has a different user to connect with the API server. But they share the same. You know the same: the same group and they're not making that many calls. They are making a call about one call every three seconds, but we have a lot of notes.

C

So it makes a lot of queries. So, let's yet the DDF. What's this book on so here what we did is we actually took all the call by the system notes group so from the cubelets basically and we're looking at the resources that are targeted and the main three resources targeted are nodes. So note, actually, you can form more than 50% of the total queries on the cluster and then config maps and then secrets.

C

So we've wanted to know why we were getting so many calls by the cubit to the nodes resources, and so what we did is we actually zoomed on a single cubelets and we looked at the call of a single qubit all the time. So here are a few minutes of logs from a single qubit and all the API call it mate and individual. It works, and you can see that this cubelets is making two calls every 10.

C

Second one gets on its own node object and one touch to update its condition so to tell the cluster well I'm still doing fine every, since my aim is the set of my conditions, and here is my status and you actually have, on the right hand, side. The detailed attach call, which is just a very simple call on the under not object to day the status and it's interesting that it's acting every 10 seconds and it's actually a flag and configure on your cubelets and the default is 10.

C

Second, and that's why it's 10 seconds, so we did the same for config, Maps, and so the first thing we did is we'll say: ok, let's look at the single cubelet and what it's doing and config maps, and so it's only doing gates. So it's different from earlier and there is some kind of regularity, but it is not as regular as was before. So what we did next one, the standings is, we actually looked at the actual resource that was retrieved by the cubelets, and here it's different.

C

Color is a different company and if you look at any single config map in single color you're going to see that regular calls are made for this config map and when happens is when you actually mount a volume-based using a config map. The cubit is actually going to synchronize it every few minutes and it's very regular. And if you have butts with a lot of config maps on a lot of butts, it's actually going to be pretty intense on the API server.

C

Because all these calls are gets, it's actually exactly the same secrets and we don't use community secrets at all. Except we use seven seconds, so we have service account tokens that are represented our secrets and downloaded to by the cubit to the nodes.

C

So so far, we've looked at calls and ways to filter them and aggregate them to make sense out of them. But what's interesting too is we can also look at the timing of the queries because, as I was saying before we had, we have the information of when they request started and when it's finished, so we can compute latency and on this graph here. What we did is we plotted latency for list calls by resource ok over time, and so you can see that difference.

C

Resource and enlisting different resources is very different in terms of timing, so some resources are very fast to list. Some resources are slower.

C

So, of course, the more resources you have, a forgiving type, the longer the collagen lasts, but also you're, going to see that it's very interestingly you we have spikes and you can see spikes, for instance, here on the endpoint resource, and we looked into that and when is well q proxy is happy, is watching and bunch of course, but sometimes we restart API servers and when we start API service, all the Q proxies that are connected to this API server are reconnecting to another API server and what happens is when you have hundreds up to proxy during the first initial list to start the watch.

C

Well, it's of course very heavy on the API server and that's why these calls are pretty long. Another interesting thing we can do with latency is compare closer to make sure that all our close you are behaving properly. So in this example. Here, what we're graphing is a single gate, call on the pod objects across different clusters, and you can see that, depending on the cluster, these calls are very different in terms of latency and we actually looked at all this cluster in detail and it turned out.

C

Smaller cluster are much faster and the main reason is: has special get bigger number. The load on the API server is much much more heavy and, of course, it's slower, quick takeaways.

C

The biggest uses of your control plane and of your API server are going to be the components running on each node, so the cubelet and the daemon set such as to proxy, for instance, and there's actually a lot of effort upstream to optimize it and to make sure that the Cuba is going to be more efficient in terms of calls just to enable us to have bigger clusters, because of course it's it's very intense.

C

If you had daemon sets that connect to the API server that do API call be very careful, because if they do operation that are intense on the API server, like least code, for instance, you can easily deduce your API server and it's actually things that happened quite a few times and in our case the structure of the audit logs is very powerful.

C

It's the JSON is very structured, so you can filter and you can group which allows you to do slice and dice and very easily go from a macro view to a very detailed view and that's very powerful reminder again. They are very verbose, so be careful like on this example. Cursor we're about hundred blocks per second. So that's that's. Quite a lot.

B

So, let's see how we can use audit logs to understand the internals of kubernetes and the way that we'll look at this is by looking at a really simple operation right. We're gonna create a deployment, and this is about as minimal as we can make it. It's just a single container in this pot in every pod, we're doing two replicas and just the required labels. So we're not adding anything fancy to this. What happens when we submit it so we're looking at the kind of kubernetes 101 view right now.

B

We did a lot of filtering to remove noise, so we can see just the relevant audit calls here, and this starts with you know the sort of sequence of deploying submitting the deployment and everything that happens until the pods are created and assigned to a node. So we can see first, that the user creates the deployment and that's followed by the deployment controller reacting to that and creating the replica set.

B

And similarly, the replica set controller reacts to that creation by creating the pods associated with it, and then the scheduler eventually binds the pods to a node. Finally, the node updates the status of those pods to container, creating so indicating that the scheduling decision is like accepted and everything's moving forward. We can take a deeper look at one of the audit records to tell us a little bit more about what's happening here. So the one I'm choosing here is the is the bind call the scheduler binding pods to nodes.

B

The call is performed by the scheduler. We can see up at the top here that it uses a create method. It's a 201, so it's successful and then the object reference audit record is pointing at the pod. That's it's being referenced here, and a sub resource called binding, so I actually discovered that binding is a resource while creating this talk. Finally, you can see the target below, which is the node that the pot is being scheduled onto. So we get a sort of whole view of what this operation looks.

B

Like the more advanced view of this whole lifecycle is actually a little bit more complex. Like I mentioned, we did a lot of filtering. So this is what it looks like. This is the full timeline of operations from everything from what we just talked about all the way until the pods are actually running, and it happens across about three seconds on this cluster for this pod. That first section on the left that you're looking at is the graph that we looked at on the last slide, except what you're seeing is in reality.

B

We only showed you ten audit events. There is actually a total of 30, so we filtered out all of the event creation. All the API server calls to validate policies like quotas. They get in list operations from each of these components before they perform their mutations.

B

So there's a lot of different things going on there, as well as the node workflow, so everything that happens once it goes from container, creating to actually running on the node that node workflow is represented by what's in the middle there and that's a sequence of events being published by the cubelet on the node, as it goes through the lifecycle of the pod, going from container creating to being updated through all of the events that happen there, and this is a great way to get an idea of what's actually happening on the node.

B

This is the primary way that all this information for what the cubelet is actually doing is surfaced. Finally, at the end, we see the resulting status updates. So first, the node updates the status of the containers, and then all these controllers in the kubernetes core are updating their the status of the the objects of the replicas setting of the deployment as that cascades to the whole graph. So let's take a closer look at that middle section: the otic records for the node workflow.

B

We can see the audit logs here and they're kind of reversed a little bit because you can see they're in chronological order and by the time stamps on the left, and those first calls are a get call to understand the full resource for the pod updating the container creating which we already talked about and then a get for the service account token, because all of our pods have service accounts, followed by a mountain valium event. So this is one of those node lifecycle events, as it's preparing a volume, that's going to get the service account.

B

Token mounted a couple of events related to pulling the image success of the image poll and the container being created, and then finally, one for starting the container and then another final operation where we get the pod. And then we update its status to running after this. Some other controllers will takeover and cascade that status, update through all the Associated objects.

B

So some takeaways here there's a lot of interactions between all the different components here, a lot of different controllers, and we can use audit logs to give us a really good understanding of it. To sequence, everything and understand how they're all interacting the events can be really spiky and they generate a lot of logs. But events in kubernetes by default have only a one TTL, which means they get purged from the data store after an hour, and you can't see them anymore.

B

So, when you're doing describe on your resources, you're only seeing the view for the last hour if you're archiving your audit logs, though you keep those events for as long as you're keeping your logs. So you have that really nice retroactive view to understand what was going on. So now, let's try to use audit logs to identify some problems in a kubernetes cluster up.

C

To now, we've seen how it can use other folks to have a macro view of what's happening on your treasure and drill down and the way that can be very helpful in understanding the low-level interaction between controllers- and we learned a lot by actually looking at that. We're now going to look at how we can use audit logs to troubleshoot problems.

C

So all these logs help you answer. Questions such as. Why was resource deleted, which is a question? That's happened that we got quite a few times and sometimes it's because another person has deleted it and nobody knew or it, because the controller authority did because it was garbage collected because the underling and underlying node was gone or it's because the HPA decided that the load was low enough. So it can, it could delete one.

C

So all the traffic will reduce information and one of the most interesting thing we've used audit logs for is to debug performance regression or improve performances, because, as I was saying, like all these components may go to the API server and your API servers can become slow. As you have do. A lot of heavy queries and so understanding which applications are doing heavy query, is actually very powerful and also you can actually look at the status codes because its nature tip it's an HTTP call and all the rest.

C

Api calls are HTTP calls and you get a status code and, to be honest, when we started this talk, we didn't figure that we were going to find anything very interesting there. It turns out, we were very wrong, and so that's why we have to have a full section on this. So well, of course.

C

The first thing is, if you, if you graph all these blogs by status codes, you can see a lot of 200 and 201 which make complete sand and that's good, but also you can see 401, for instance, and actually, when you dive into 400 so errors, you can see a variety of different errors and, like that's, where it I mean this is a live huge prediction question: we should I get these errors so first well before one that's a lot of inertia, icicle, that's weird, and so with what trying to understand what was happening because, of course, all our users log before they connect and all the controllers and all the applications.

C

You see this count token. So this should never happen, so we slice and dice we drill down. And finally, what we found out is that all these, for one was actually actually coming to for single IPS and what had happened is for cubelets, had experts at three gates and we're actually trying to connect and do their things and they were getting for once and the reason we missed them is because they'd been removed from the cluster. So they were not part of the presser anymore, but the qubit was still trying to do things.

C

So while we did the talk and we fixed it- well, forestry was kind of a weird. So why are we getting forbidden? I mean it I mean it would happen from time to time, but it shouldn't be that regular, and so it turns out when we plotted it by user. We we found a single trap status account which is tragic, which we use for some ingress that was getting forestries and my first feeling was well.

C

This is probably about a by configuration, so let's look at what's actually failing and what was fairly was a least secret school and I'm. Like that's weird I mean we don't use committed secrets and there's no reason for traffic to do that and of course, traffic doesn't have the proper a back policy to do it because it doesn't need it.

C

It turns out, it's actually built into the code, and it's doing it so yeah traffic is gonna list all your secrets, unless you forbid it, the last one was probably the weirdest we were getting for 22, which are invalid, HTTP queries, and once again this was very weird. So the first thing we did is look at which user were actually getting this for 22 and it tell that it's a built in controller from kubernetes. So how like wow? This is very like a built-in controller should never create invalid queries. I mean it's a built-in controller.

C

Why would that happen? And so we dive deeper and we looked at the at the cogut was actually failing and what getting in for 22- and it was the patch and the pod object. So if you look at the job patch at the at the bottom of the slide, you can see that the garbage controller is trying to remove the owner reference to make the pod open from what it was controlled by enter. Well, that's where this is a very simple patch code: there's no way it should trigger for 22.

C

So what was happening was actually pretty complex. Is these spots were evicted? Okay, so they had been evicted for reasons from from nodes and they were controlled by a replica set controller which had been deleted, and so what should have happened is sponge would have been often by the garbage controller and the way the garbage control is opening. This bus is actually removing the owner reference, and but it was failing so well. We had this replica set that was terminating and it had been terminating for two months as of this morning.

C

I think it's still the case, and so the root cause is, we use mutating webhooks to help application developers and by providing them with sended and Brendon environment variables, DNS configurations- and we configured these web hooks to a both on the create call, a crate operation and update operation, and so they actually applied to touch, and things like these calls are actually modifying immutable field, immutable fields and, of course the API server is failing.

C

When you get a call that is trying to modify an immutable field and that's what the garbage collector wasn't able to garbage collector cuts. So and takeaways, while looking at and using other clocks for debugging, is very helpful and, like just writing this talk, we found three different problem that we fixed. So that was interesting and, let's conclude so.

B

We hope that, with this talk, you guys learned that all in logs can be incredibly valuable. We take advantage of them all the time and they've been instrumental in us understanding our kubernetes clusters, the low-level details and detecting misconfigurations troubleshooting issues. It's it's really been super valuable for us, you know, taking advantage of them does require a little bit of effort. The policy is a little bit complicated and they're hard to get right and they do require a little, an advanced tool train to figure it out, but they can be super valuable.

B

I hope you all go. Enable audit logs in your clusters, if you aren't already and walk away with some value from this, and that's all you can find us at the booth later, where we'll be around at the data dog booth or reach out to us on Twitter or email, if you're interested and we're always hiring. So this is interesting. Come find us.