Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Audit in Kubernetes, the Future is Here - Stefan Schimanski & Maciej Szulik, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Audit in Kubernetes, the Future is Here - Stefan Schimanski & Maciej Szulik, Red Hat

Advanced Auditing is now available, and you should learn how to use it. Last year, we proposed a list of improvements to Kubernetes auditing, and this year we are happy to report that we have delivered them! Join us for this session to learn what Advanced Auditing does, how to adapt it to your own requirements, and how to integrate it into your infrastructure. Topics to be covered include: 1. From basic audit to advanced audit 2. Defining a policy (examples) * Deep object logging * Excluding secrets * Logging different objects at different levels 3. Integrating with your infrastructure 4. Performance impact vs. consistency 5. Writing an audit webhook server 6. Which future improvements are planned We are hoping to demystify the complexity of the policy configuration by showing a handful of example rules that will allow attendees to better understand the mechanism.

To learn more: https://sched.co/GrXd

A

My name is ma che I'm, the sexy Li lead and I work for Red, Hat and I'm, one of the core contributors from Red Hat to kubernetes.

B

Working on Murphy, Nui and I did.

A

B

Of so ap I'm just going to be plumbing around auditing, so.

A

Okay, I, apparently I can't be too close to Stefan. Anyway, let's start with a quick history lesson so that you can get a context and a feel how we landed the the current auditing mechanisms that we have in kubernetes. So we started back in 2016 when we introduced the basic auditing mechanism. It was very simplistic. It was like a page-long of code.

A

It was a handler that was responsible for logging, then from basic information about every single request that was hitting API server, that included both the API, the user. That was making the request some basic information about what was being requested, which is the URI and additional information.

A

So aside from the user, is the impersonation information each such request was logging, two lines of code, two lines of data- one was what I just mentioned, and the other one was the response information which, in the simplest code, was just a HTTP code of the request so 200-400 whatever we to reader, we returned from from the API server. Of course, that was very simplistic that allowed us to start further discussions and further work, because a lot of people were actually interested in in auditing and with the help from Google Korres Fujitsu.

A

We started at the implementation that we beforehand drafted with Stefan, so the advanced editing introduced in 1.7 as an alpha allowed us not only to log the basic information from the basic auditing mechanism in a textual form. But additionally, we were. We are now able to lock deep information about the full objects that were both sent over to the server and then we will send back as a response to to the requests, which was probably the most requested feature for auditing that we heard about. Additionally, the textual form was one option for logging, the information.

A

But on top of that, we introduced a JSON format of the JSON format for auditing and the audit event we will be showing in a short while and what was probably the most advanced feature is very sophisticated filtering mechanism for deciding how many information for what kind of information and at what level we are actually logging in the auditing for every request. So the next couple releases all the way up to 112.

A

Let us to stabilizing the mechanism, improving the filtering mechanism, improving the batching requests and entire mech machinery to be able to actually and finally release the advanced editing in version 1.12.

B

We want to go into a bit of the details of auditing to really understand what is locked and what it means. How complete it is, how consistent it is and so on. So we have the event here: it's basically what you, what you get either printed out, runs a lock or sent to a vet book yeah like 20 or something fields we have here so a lot of the metadata, that's basically what we had in basic auditing already.

B

Then we have, of course, when it happens, especially when the request came in and when the event was created. This can be different, so there's a long running request which takes 40 seconds this time stem can be very different, in fact who also on doing this request, so the user, especially which includes also information.

B

This comes from the authentication part of the API server, and also something like user agent and source IP source up. He other guys. The source IP of HTTP, of course, also TCP connection, but also from proxies in between and the user agent, is really what the client claims to be. So it's just a string. You cannot trust and so on. So, but it's it's very helpful in the in the order clock as well and deep logging.

B

What was really requested so weed is object coming into the API server and which went out as a response to understand what those informations mean. Let's take a look at the API server, so he is yeah, so inner working of an API server, so each request dosis follows has got lines whose API server, so it comes in here it goes who authentication impersonation also is Asian?

B

Maybe it's phone I can get it I serve us. So maybe it goes this way out to another process, but usually goes through the implementation here. This long pipeline, where so logic for the API is a test. Logics will be implemented at some point. It goes out and they're sent back to the to control or controller whatever gets requests originally, and of course, we have also admission web hooks today, so so object can be influenced by web hooks as well.

B

We have a number of probes where we collect information and obviously use the information we have already here. So after authentication, we know who this user is usually here also indicated by our token or by a client certificate, and we know user name. We know the groups, maybe uses impersonation, so it tries to do something like a pseudo on a CLI. We can lock that and finally, we check that the user really can access the resource that he requested. That's authorization.

B

Finally, it goes here into the API server and if you imagine, you did to create of a pots, so cou couple create F pot Yama. We have decoding and we record the object as the user. We descend at this position and then we have admission admission hooks one may be some web hooks one and we can also record information in this stage.

B

For example, if you really want to know why could this pot be executed- and it's very important to know to know- support security policy which allows this pot through to one or to do some command like exag or port, for what this is also locked and that's what we record at this position and finally, on the right side, we have subjects and response and we send it back through the wire to the user, and this is a response which we collect there.

B

We can lock different information, different depths of information, so what we had already in in basic auditing long ago was this metadata all those probes here at the top, they just collect metadata. It's very small, easy to collect it's cheap. You can just do it more or less all the time. There's nothing secret about it as well. So it doesn't matter you don't have to forbid metadata for for requests. Usually the request object here, it's much more expensive.

B

If you look at an event later the event as much much bigger because there's a whole object, so it can be a huge pot spec, which takes several pages of JSON text.

B

It's big and it's much more CPU demanding for the API, so I were to do that. So maybe you should think about whether I really need that also for secrets. Maybe you don't want that, because secrets are secret. Obviously, so maybe don't send it over the auditing there poop at this bubble here, it's again metadata some key values, annotations from submission, plugins, also cheap yeah. You can do it more or less every for every request and again the response is similar as a request responses again, usually a big object.

B

They are the same place as for the request.

B

Yeah I mentioned performance. It's mostly CPU demanding what we are doing here, but that's another dimension, um consistency, that's what I called it. So level is mostly about what we watch me record or what we ordered.

B

I've painted one, a timeline here. So this is a request. It comes in its zero milliseconds, so this one starts so there's a first byte going to the to the to control or user other end. There might be a panic. Panic means this may be some bug in the handling, or maybe the connection is destroyed, or something like that. If all this doesn't happen, everything is fine. There's a response complete here and it doesn't help used in certain situations.

B

If you just just have some requests received, so he only knows that there was a request started, but you have no information, whether it's still ongoing. So you maybe want to audit logs, as a request was really finished and there's nothing, no connection anymore, with the cube cutter or also controlled at the other end. So those are two dimensions. First, the depths and then yeah. We call it stages. Each of those stages produces its own event. So you get couple of events. It's like basic editing. You get multiple lines of of output.

B

Here you have different events and.

B

Yeah the same thing in this picture again so request received. Obviously it's he ends beginning and the other events are created here and he has a byte stream or bit stream and when every singer sent response completes assent and before he gets a response started, maybe in between we get a pending event and yeah I I use the word dimension, so those are two dimensions you have to go. You can configure in the policy stages how many events at which positions and levers describes the depths.

A

Ok, I'm good thanks, so we covered how every requests throat flows through the API server, now that we know how to process the requests we want to, and we are currently interested in how to define the policies which basically means how we can decide which requests got locked and at what level at which stages, which is what cell phone was covering a minute ago. Currently you define this by creating a very detailed audit policy file, which you then pass as a flag to the cube API server.

A

There is an option about a dynamic auditing which is currently and now fine once and just release. 113 I'll cover that shortly and towards the end of the slides. But all you need to know at this point in time is that every single policy will always have the API version. Audit kate's, il v1 and its kind will be policy. Then you can define a global stages that you are emitting and there comes the rules and let me show you a couple examples of rules so that you can better understand how you filter every specific request.

A

So, let's start with deep logging objects, because that was the one that was most frequently asked for to define to have an object being deep locked, you specify a request response level, which basically means we are logging full object both coming into this API server and getting out. If you look towards the right, this is pretty big. This is pretty big output of a sample other event for getting pots.

A

This is where the requests you see quickly, because I can't remember this is the response object of the get you can see that it contains full pots, including its spec, including status. It's really big.

A

Now getting back going back to the policy. There are a couple more information that we included in this particular policy. We are logging, specific resources. We are not logging, everything we are very picky and we define that. We only care about the core group and apps group, which basically means every resource that falls in one of those two groups will be logged at the request response level. We also overridden the default. The global ahmud stages policy at this level, but specifying all pretty much I think the same.

A

The same level going further Stephan mentioned that earlier that we don't want to lock auditing. We don't know we don't want secrets to be included in any of the other, linked events, so for that particular purpose. We're going to set the level to metadata and we want to exclude secrets. So in this particular case we are explicitly naming secrets and config maps from the core group as well as touken reviews from the authentication case.

A

Il group you can see in this example how you can apply a rule to multiple groups at once and multiple resources, if you, for example, like from the previous example where we did not specify the resources that meant it applied to every single resource within the group, if you are specifying resources like we are in this example, that rule applies only to those resources in that particular group. Of course, you can include multiple groups and multiple resources in every single in every single policy rule, but they will be.

A

It's an or in that case, okay, going further logging objects at different levels. This is one that is very important and where it comes how the policy overall combines into the actual logging events that you are getting in your audit. So in this particular case we will start with the non resource URLs, which is the example on the right. So, as you might be familiar with, the majority of the resources, are you working with cubes such as jobs deployments? Stateful sets pod services and whatnot. These are so-called resources in the cube API.

A

There are, though, some endpoints in the cube API such as Health's version swagger and a couple more might exist, which are not resources for those. There is a specific rule within every single policy that has named non resource URLs, where you explicitly define partic particular URLs that you want to include in your role. So for this particular case we are setting the level to none, which basically means we are not logging any of those URLs that are similar to this one. The matching is a very simplistic rag axe, so you might see a an asterisk.

A

At the end of the health and swagger end point now, we want to be a little bit more specific how we going to lock resources from the group. Bat, if you compare both of the on the left. You've probably noticed that both of them contain resources from the batch group, but the difference between the two is that the first one logs at the request level, whereas the other one, goes all the way down to the request and response level.

A

What that it means the first rule applies only to particular verbs, and it is important to notice that these verbs are cube, verbs and not a HTTP verbs. So this is a get which gets a particular object from the batch group. This is a list as well as watching watching resources from the batch group and for that particular request, we will only lock the request object for other verbs, which basically means create, update delete, because these are not mentioned.

A

We will be applying the request response, which means not only we will log the phone incoming request, but we will also lock the full response object and at this point in time, it's also important to notice that when you are composing your policy file, the policy matching rules always start at the top, so the first match rule will apply to the will apply to the incoming request, which basically means when you are composing your your policy file, the more the most generic filters should be at the bottom of the file, but the very concrete examples: the very concrete policies there are targeting specific requests that are only, for example, get let's watch from the group a batch has to be at top.

A

So it's very important to note here that we are always processing top to bottom. The policy file. One more example, aside from matching by resource by matching by the actions performed by non URLs. We can also go all the way up to matching by either a group of users or a user, particularly in this particular case. On the slide. We are matching a user that is named, not a user and we are vlogging the entire requests and response of every single action this particular user is performing.

A

Similarly, we can specify a group, a group of users.

B

Yeah after policy and what we collect, the big question is how to integrate that with your infrastructure- everything we are talking about here, it's just in the API server at the moment. So obviously those things are configured as flex and the IPS API server. So I listed them here. I guess you cannot with them. They are like 20 30. Maybe there are so many because we have different backends and each back-end has the same configurations.

B

Basically, what you do is I think we saw it before. Maybe is the first flag, which is important as to give the yeah in the case of logging, gives the pairs of a file or stand out. If we like that, and in the case of a web hook, you can just configure Zaku config and just by doing that, you enable a back-end you enable so auditing. After that you have an audit output, you can configure certain things.

B

Some are common for all of the back and some are not, and one which is crucial is a mode. So we had stages, we had levels before and the mode is a cert I mentioned. We are talking about events and events which are sent somewhere or maybe printed out. This sounds like it magically happens. What happens when an API server dies, for example? So there's a back-end API server and maybe an attacker tries to use a security exploit and he knows how to kill the API server.

B

Maybe he is able to do something malicious and kills a API server. Just after that- and you have no clue about what he did, because there's no auditing being persisted on disk and this mode is exactly about that.

B

We have three modes batch blocking and blocking strict blocking, strict means that we really wait for I/o to happen. We wait until the lock is really out so the output, the sweida finished in the case of the red hook. We really wait until the web hook returns hgp code, so the response is finished.

B

The sounds very, very good, so it's it's pretty sure that your audit event is stored before the request as really handled, but it's slow because it's synchronous, so it slows down the whole yeah. It's a main code, part of the API server. So your request has to wait until all the thing is really done, depending on the use cases, environments. This is what you want, because all thing is so important that it's yeah, this mode is the one you want this to. It means if auditing fails, the request is rejected.

B

So if the audit event cannot be written to disk or set to the web hook, so the request will not succeed.

B

The second one is just blocking it's the synchronous, so there's the blocking behavior, but if the auditing system goes down, just as a system still is working and the last one is betch, betch means that events are collected in memory and sent to the web poke impetuous like a thousand events, so it waits for a thousand events or maybe a time mode. You can configure that it may be. It collects up to ten seconds and then sends it to the audits back book.

B

Those things you can configure so on the on the slide before you will find those options there. If you really look into the details there, obviously, if the APS are precious, you might have lost a couple of events so, depending on your environment, you have to decide what to do all this supports graceful shutdown. So if you have API server holding update this mechanism knows auditing and it's weights on to the web. Who really has gotten all the events in memory? So graceful termination just works. Fine if you use the batch mode for for auditing.

B

So I said it's a certain engine which gives us this nice picture. So again you have stages, which is a point in time when events are sent never the depths, and now we have the backend which gives you something like consistency or something in this direction.

A

Okay, I've mentioned before that I'll shortly describe the dynamic audit configuration. So what are the current problem with the defining the policy? The policy is read only once on the start of the server. So with the current approach of how we allow users to define see, are these or dynamically configure the functionality of the API server? It is always changing the functionality available in the API server. Those the administrators needs to be able to respond to the changes by changing the audit configuration.

A

Unfortunately, currently it is the only possible way to respond to such a change is by updating the file on a disk and restarting the cube, API server, which obviously is a painful process. That's why we come up with the dynamic other configuration option, since this is an alpha feature in 113. You need to be explicit that you want to use this feature.

A

That's why we have the audit dynamic configuration which tells the mecca the audit mechanism that yes, I, am aware of the that this is an alpha feature, but I still want to give it a try.

A

When you do so, you create an object, which is an audit saying our audit registration that cait's that iov one alpha one within which you specify the policy and the source where the data are flowing from the from the dynamic order configuration as you probably notice. The current policy rules for the dynamic audit configuration are very simplistic.

A

Like I said. This is an alpha feature, so very early stage that we landed in for 113. It only allows very simplistic policies. We are hardly working on making it more efficient so that it can have the exact same mechanisms the as the regular audit mechanism that we currently have.

A

Okay, so before we close, there are a couple of references that we want to point you to so the first one is the actual design that the second one is the design proposal that Stefan and I wrote two years ago about how the audit works. You can check it out. It contains. We spent numerous hours trying to cover all the possible use cases. It is a good starting pose also for for understanding how the auditing mechanism work.

A

There is a very good documentation about auditing, including the the description, how to create those policies that we were discussing earlier today. There's the cap, which falls under SiC auth, currently, which describes how they are dynamic. Auditing configuration will work in the future releases if you're interested in helping out feel free to reach out to us feel free to reach out to check off we're more than happy to help people get started to help. You help us with enhancing those features.

A

Finally, as a very interesting point to note, Jordan late wrote a tool which is added to our back. So currently, if you enable Arabic, it is very hard to figure out what you actually want to do. What should be the are back policies, for example, for your app for your new application with auditing. It is very simple in this tool to generate a policy that will be specifically designed for your application. All you need to do is enable the default auditing, which will just log the entire data at the metadata level.

A

If I remember correctly and then run your application, the application will get rejects, which rejects are then processed by Jordan tool and based on the rejects, he's able to create proper Arabic rules. If you do a couple iteration, you will get a very nice very detailed policy, tailored specifically for your application.

A

Takeaways yeah.

B

Takeaways, sweet dimensions: we went into that very much back-end options, logging and web hooks. Those are the two options and hopefully soon we will have certain Eric. One also be epic, of course, but yeah it's dynamically configurable via as a cluster API, and the last thing is a policy.

B

So those are the components you have to understand to really employ auditing yeah and one thing to add: if you are in a environment where you have regulations, for example, and you have requirements for auditing, which we don't fulfill yet feel free to come to us and talk to us yeah this feedback I mean we can block a lot. We can extend event according to requirements, so we need feedback from real applications of auditing, so come to us and speak to us would be very, very welcome. Yeah.

A

Both stefan and I are available on cube slack or on github just ping, either one of us and we're more than happy to help you with the problems that you have with auditing. Thank you very much.