Cloud Native Computing Foundation Kubernetes Forum Delhi 2020, 27 Feb 2020

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Auditing in Kubernetes 101 - Nikhita Raghunath, Loodse

Description

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects - Learn more at https://kubecon.io

Auditing in Kubernetes 101 - Nikhita Raghunath, Loodse

Have you ever wondered who created particular changes in your cluster, when they created it or what resources were modified? All of such information about “what sequence of events lead to this scenario” can be obtained using the powerful audit logging feature. In this talk, we will first go over what audit logs are and how to leverage them to stay informed with what goes on in your cluster. Keeping both performance impact and accountability in mind, we will then walk through examples of policy configurations to enforce best security practices, detect misuse and make your cluster more compliant. We’ll also do a demo of setting up auditing on a cluster and inspecting the logs. Finally, we will see what future improvements are planned for this feature and how you can provide feedback and get involved.

https://sched.co/YWJ2

A

So, let's get started, Who am I, Who am I talking in front of you, I am a software engineer at a German startup called Luthor I walk remotely from Mumbai I'm, also active in the upstream kubernetes community and I'm on the kubernetes steering committee as well and I'm technical leader, one of the six.

A

So every kubernetes story starts with a cluster. So let's say you have a cluster, it has an app which is running in a pod and you have a secret which contains a password for this app and one day you notice something strange that the password value in the secret got updated and you did not make this change now.

A

That looks really scary and looks like something is fishy and something we are just going on here and since you think, something's weirds going on, you decide to look at logs and, let's just assume you don't know about audit logs, because we need that for this presentation to continue. So you decide to look at logs from the pod. It might tell you that the value got updated, but it won't tell you when it happened, or you can figure it out. Maybe, but you could not.

A

It wouldn't really tell you when it happened or who did it, which is the most important thing that you want to find out. Then you decide to look at events that also doesn't tell you who did it. It might tell you that the secret value got updated when it got updated, but again it doesn't tell you did it. The same goes with API server logs. You still don't find out how this change came along, so you're stuck now you decide to head back to the kubernetes website.

A

You search for what you can use, what feature you can use to make this happen, and then you come across this shiny thing. Called audit logs audit logs, give you a lot of information about what's going on in our cluster, sometimes way more than you need, and you need to filter out what exactly you're looking for in all of that huge blob of data.

A

So this is what an audit log looks like it's a huge JSON blob with lots of information. I, don't expect you to read that it's just way too much we'll be looking into detail about what what each of those represents, and we will try to understand what it's made up of. So the first thing these audit logs tell you is what actually happened. So if you did something like a cube, Cpl get pod or a cube, CTL create secret. So the word in this case is get create, update, delete list.

A

So it tells you about how what happened in the cluster using the work, in this case in your cluster, since, let's say a secret God created the work would be created on what resource did this action take place on a resource and kubernetes is a pod. All kubernetes api objects, a pod, a secret, a replica set a deployment, so the object ref in the whole Jason blob gives you more information about what resource card updated.

A

When did it happen? When did this change happen? So one thing to understand over here and I'll be going into more detail about. This. Is that kubernetes audit logs are basically logs whenever a request hits the API server? So it's always, it always always always happens at the API server. So whenever we talk about when it did happen there, there is something you so. The first thing you see is a request receive timestamp.

A

So this is when the request was received at the API server and you don't need to worry about what stages right now we'll go into that a little bit later. Who did it? So this is the most important information that you are looking for. How did this change come along and that's that's the side that's mentioned by the that's mentioned in the audit logs I used many cube to just get this example going. How many of you all have used many you to understand?

A

Who marries quite a few of you, so we have a mini queue maintainer amongst us, I, don't know if NCR Rohit is on the first row. So if you all want to know more about mini queue or have questions or want to get involved, find Rohit in the session yeah.

A

The next information that it gives is where was this request initiated, so it gives you an IP address for the same.

A

So that's a lot of information that you get I just want to give a brief, like one-liner for folks who are just joining in so ordered lots give a lot of information about what goes on in your cluster, who made a particular change. What change was made when was this change made, and it also answers all of your? What, when how?

A

Where questions and audit logs are dependent on whenever, basically, whenever a request hits the API server, so any requests any requests, even if like I, am making a call or if the system's making a call all of these get logged.

A

If I do a cube serial get pod, the system runs I, think like 40 or 60, more calls, but just one cube see they'll get pod and all of these get locked and you have to find what information you are looking for from this huge blob of data and that's just a lot of data to minimize this data and to control it will control.

A

The verbosity of these logs will tell kubernetes that we care only about certain things and to log only those certain things and we'll do that by telling it what to log and went to log it and we'll do this by no surprises using Yammer. Let's see how to get this running hora del Cuban Eddy is what we what we care about, we'll use something called as an ordered policy.

A

It's a Yaman file. It has the API version of ordered dot, Kate store, I or sash. We won a kind of policy, it has rules which tell kubernetes what we care about and what things we want to login when we want to log it. So you see something called state OCR and we briefly saw that in the whole Jason block thingy as well. So let's see what it is and I don't understand it better.

A

So again, a PS our request comes so the lifecycle of the request through the API server and when the response gets created depending on the time in that whole lifecycle, you can decide when you want to lock things. So the first stage is called request received it's when the ordered Handler and the API server receives the request, but the response is not generated yet so it's just the request receive that this request has been received. The second stage is the response started.

A

The response headers are sent out, but the response body is not sent out yet this is the stage only occurs for long-running requests like watch. The third state is response. Tej res response complete. It means that the response body is also sent out and the fourth stage is panic: whenever a panic is generated in this whole process now this might be a lot to take in I. Let's, let's make this easy and let's visualize it. So you have your API server and the request.

A

Once the request is received into the API server, you get the request received stage. Then the response is generated. If there's a panic in this process, you get the panic stage. If there's no panic and it's the long-running request, so the response headers are sent out like watch may be and then the response body is sent out. So the response is complete and then you finally have the response.

A

So you have these four stages: request received response, started response, complete and panic, and this is very important to understand because you will be using these storms or two in our rules when we tell us, but when we tell kubernetes are these are the things we care about and please log only these things moving on so I talked about went to log, so this was in the lifecycle of the request will define certain stages. Now we can also tell Cuban Eddie's what things we want to log, what sort, what resources we want to log.

A

So we can define the group version resource and works, so we can tell given Eddie's that we only care about secret deletion events, so we will say the group will be course, so we leave it blank resource as the secrets, we can also spread out the work that we want only deletion events, because that's what we care about now. We see some other new term here which says level. Let's look into that levels. Tell kubernetes again what how how much portion of the resource of what's going on that we want to lock.

A

So none means don't log. These requests at all metadata means only log. The request, metadata request, means metadata and the request body and request response means metadata, request, body and response body.

A

So we saw a lot of things right on if you're, just starting with kubernetes. This might look a little overwhelming.

A

This is also a whelming for anyone who, like actually knows kubernetes and has been involved for quite a long time. It's also harder to figure out like hey I, want to log only these resources, because there are system calls happening as well. It's not just the calls if you're making you do just do one cube, CD I'll get pod, you just run one command, but then the system is running a thousand other commands. So how do we? How do we write policies that make it easier for us to get logs?

A

So let's look at some recommendations. First recommendation. That's a very important one is that if you have sensitive resources, like secrets, config maps or token reviews only only only log at the metadata level, because if you don't do that, you'll also get your secret sensitive information in the logs and that kind of destroys the whole point: don't log read-only, URLs or non resource URLs, because you probably don't care about it and it just it's a huge high volume it just it just blows up your logs.

A

The general idea is log it at least the metadata level for most of resources and log at the request response level for critical resources like pods that you care about. Maybe so. In this example, we have two rules. One is a level 4 request response and the other is for metadata. How, when you read two rules, let's see what happens so, rules are evaluated in the top down order, which means at the top rule is considered first, so the first rule is request response and for pods suppose the rondalee critical resources.

A

So you want to log them at the request response level to know if something goes wrong. What actually happened? Sub resources like status and log, a very high volume and generate lots of logs, so you can just log them at the metadata level.

A

There can be very more ways you can define things, so lots of them can be found at this link in the Kuban ID source code yeah. You can check it out here.

A

So, let's find out what so you you realize that ok, we know what order stalks are now. We know that if you want audit logs, we'll have to tell kubernetes what takes to log at so we'll have to write audit policies for it, and we sum we went through some recommendations about how to write, ordered policies, and it's you: where do these logs actually go? Where is it stored? What happens with it?

A

So the place where this logs actually go is called back and- and there are two types of background- a log back in and look back on the log back and writes these logs to a disk and the web hook back and sends it to an external API for the log back, and you will give the audit log path flag to the API server to denote, where the file path to where these locks should be stored and ordered, ordered web app config file flag for the API server denotes the web hook configuration for where these logs should go and the audit policy file flag is used for both backends and it will be used to basically give as the policy file the whole rules that we just learnt about.

A

So let's go a little deeper. We figured out that these logs go to a back-end. Now we can also give more details, or we can also tell Kuban at ease how how we want all of this information to go, that's defined by batching. So do we want it or to go at like in batches or do we want to want them to go individually or do we want something else? So the batch method is, as the name suggests, the logs are. The events go in batches, the blocking method.

A

It sends individual events and the blocking strict method. If there is a failure at the request received state, so there's a failure at the first stage that whenever the request is received at the API server, it leads to the failure of the whole call.

A

You can specify the batching method using the audit webhook mode flag on the API server and the audit log mode at the API server. So it would be audit by book more equals batch or blocking or blocking strict and for the web. App mode batch method is the default method unless specified.

A

So we saw that all these flags are sent to the API are applied whenever you are like starting the API server. So whenever we want to update a policy, let's say you made a typo or you want to change something or you screwed up indentation for yama, which never happens. Then you need to restart the api server each time and that is absolutely no fun at all. So what if to update the ordered policy, we could just update a kubernetes resource. We could just do a cube. Ctl apply eff. That could be helpful.

A

So that's where dynamic audit configuration feature comes in. It is still an alpha feature, so you need to supply the feature gates and runtime config Flatts to the api server and the order dynamic configuration flag as well. Once this feature goes GA, you just need to supply the first flag. So this feature is still alpha. There's a lot of work going into it. We will see how you can what what this work is and how you can get involved as well.

A

So this is why that kubernetes resource looks like the API version is ordered registration at KS rod, IO, it's still alpha and the kind is ordered sync, as in like a kitchen sink, not the sy and C sink it in the spec includes the policy. That is the rules that we talked about it uses. It also includes the webhook configuration where these locks should be sent to.

A

So if you want to update the policy, it's as simple as updating the spec and applying the config and the policy is going to get updated, two things to keep in mind when you're using this feature, because the switches looks dead, simple and well sort of simple. So it feels like you want to use as many of like create as many of these resources as possible, but you need to keep in mind the security and the performance considerations in this case.

A

So regarding security right access to this feature, basically, grants read access to all cluster data, so you want to restrict this to the cluster admin level privilege.

A

We also found that using too many ordered sayings has performance implications for the API server. So you want to make sure not to use too many other things using nominal number is okay, they're still being testing performance testing done on this, and the benchmarks should be published before, like the feature moves to beta or we make any more changes to the feature, and that brings me to the cap. So cap in Cuban IDs is a kubernetes and home hansmann proposal.

A

It's basically a design proposal for any feature that goes into kubernetes, that talks about the motivation behind the future and the whys and how's, and what's what whatever, whatever the design is being proposed in the future. So this cap is related to the dynamic audit policy. We eventually want to make it beta and then finally GA, but right now it's alpha. We realized that we can make this whole process of writing a policy way easier, because we saw that we need to know.

A

What's going on inside the API server, we learned about four stages and four levels, and it's just all all too complicated and a random user of kubernetes should not need to know about so many internals. So to make all of that easy, we have a cap check it out. If this sounds interesting to you, we definitely need more volunteers to help out with this, it's been alpha for quite a while, so you definitely want to make it a ga.

A

Even after this gap, it's still going to be remain in alpha because we're changing lots of things. So this is a very good place to get involved early in the cycle. You can get more influence into the decisions that goes on into this feature as well, so the link is the link to the pull request that adds this cap and if you have, if you want to get involved, if you have feedback or questions, you can check out the sig or slack channel on the kubernetes lock.

A

This was too much information and there's too much logs I just want to give you very, very quick: it's not a demo per se, but a quick intro on how these audit logs look. So let me.

A

Okay, so I will do a mini cube. Start I've already done this before so I'm, not going to run it and wait for it to complete again. If anyone else is interested in many cube, fine Rohit he's a mini Q maintainer and you can ask him any questions about mini Cuban how to get involved.

A

If I want to find out how many logs we have I'm, just gonna run like how many number of lines we have in these audit logs, it's fourteen thousand to one foot and I started this cluster like when the keynotes got started. So it's it's log like fourteen thousand lines in such a short amount of time. So that's a lot of logs and to give a very more information. This is how it looks like resist. We we we way too much information and to analyze.

A

It manually is almost impossible so to make all of this easier and to understand how all of these all of this comes about and how to actually use it. We're gonna be talking a little bit about la collect. The patterns. I'm not gonna, go too deep into this. It's a one-on-one talk and all of like each of these examples can be its own talk.

A

So we look at two brief about how you can collect these logs, how you can make sense out of it, and all of these examples have detailed details about how to set it up in on the kubernetes website itself. So the first thing we're gonna be talking about is using an audit log file and flew Andy. Flew Andy is a CN CF project, which is basically a data collection software.

A

So you can use a log file, you can send it to flu and D and flu Andy will process it and, like you can add filters to it. You can basically get something like audit events for namespace that can help you out too for the filter out. However, you want like, however, you want ordered locks to look like on how you want. However, you want to filter it. The second example is using lock stash. The log stash is a server-side data processing platform, so you can use web hook.

A

Files in the web hook back-end for audit plugs, send it to log stash and you can get something like audit logs per user per file. You can also write policies which which can filter out logs per user. So if you want logs for actions only created by a particular user because I don't know, you think something fishy might happen whenever a particular user, that's something you can use this.

A

Another very popular design pattern is to use a web book file plus Falco Falco is a project for automatic detection of suspicious behavior for cloud native platforms, so you can use a web book file and then send it to Falco to the Cuban Ares ordered endpoint in Falco, and it will automatically detect suspicious behavior and give out results. So it does all of the work for you. So we learned about audit logs and I. Usually I talk about this in the beginning.

A

What are the advantages and go into it, but I wanted to make you understand what audit logs are and to get a brief overview of what it is before. Looking into how it's helpful. So the first thing that not many people use it for is understanding kubernetes internals, so it's user calls and system calls as well. So if you analyze the calls, it can tell you about what's going on behind the scenes and how cuban IDs works internally, it can help you detect miss configurations, as we saw earlier in the example.

A

It can tell you who, who updated your secret with that strange, suspicious value. It can help you answer, questions like who deleted the secret or who deleted this resource. You can also analyze calls and figure out which HTTP status, error code you're getting a lot. So, for example, if you have we're getting lots of photo threes, it probably means we have misconfigured our back. It can help you identify performance issues and answer questions like which app is generating the most number of calls.

A

So we saw a lot of things you are today. We saw what audit logs are. We saw that we need to write audit policies and define rules to tell Kuban. Id is what logs we want. We saw some recommendations for writing these. We saw that we need to send these logs to particular backends. We saw how we can sign these logs to different batching methods, and then we came across this awesome feature called a dynamic audit configuration which helps you define all of this in a simple human. It is resource, it's alpha, Viereck.

A

We want help. Please please get involved, you don't need to know anything about humanities and tunnels or how everything works. But if you want to get involved come find me come find like for mini cube again. I'm going to iterate come find ro head, there are lots of upstream contributors, and this there's lots of upstream are two bidders in this room find any of us and we can help you get started. So thank you and hope you are having a 1-2-3.

A

Do we have time for a Q&A session? No, we don't. So if you have any questions, please find me after the talk. Thank you.