Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Who Shot the Cluster? - Audit Logging in Kubernetes - Marian Lobur & Mik Vyatskov, Google

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Who Shot the Cluster? - Audit Logging in Kubernetes - Marian Lobur & Mik Vyatskov, Google (Intermediate Skill Level)

Did you ever face a case when you did not know who created the resource or modified the security settings. Have you wondered how you can get this information without spending weeks on investigations? Are you required to audit your infrastructure for compliance? In this talk Marian will walk you through the setup and best practices of using Kubernetes Audit Logging along with a demo.

About Marian
Marian is a Kubernetes project contributor since May 2017. He is working on monitoring and logging pipelines. Marian has experience of working on other open source projects, for example he has contributed to Bazel build tool. In his free time Marian makes 3d printings and takes a part in “escape room” quests.

About Mik
Mik is a Kubernetes project contributor since September 2016. He is an active sig-instrumentation member, working on monitoring, logging, and related topics, e.g. audit logging. Mik is especially experienced with logging systems of different kinds, having several years of industry experience deploying and maintaining logging pipelines from the kernel level to the most high-level architecture.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So welcome to the talk which is called, which was the cluster. My name is Marion. I am working on cue branches at Google, exactly on logging and monitoring team of geeky oops.

A

So today we are going to dive into the world of argot locks and how they can help you to answer the question: who was the cluster I? Don't? Does anyone remember that recently there was a case when grants cluster of one big car manufacturer was compromised, the same. Who knows yeah?

A

Okay, so that was done through unsecured kubernetes dashboard, so they didn't have any passwords set on it and I think this issue was discovered by company called red lock, so the companies that perform some security monitoring of public cloud environments so beside stink, some secrets and credentials or from this kubernetes cluster.

A

They have also managed to deploy some crypt monk software. A few important notes from here's a first one as a human age, to hide the mining pool series, so they have used some unknown endpoints. Also mine pulsar was hidden behind some CDN services, so it was very difficult for anti-malware software to detect it also. They have said a small limit or on CPU usage, so it was running clear unnoticeable on their cluster.

A

So, as a question is possible to find something like that, is it possible to track down the problem? What has caused it and a good news? Yes, we can do it and for that you should use, argued logging. So that's anyone knows already. What is the covenants? Are you looking? Please raise your hand if you heard about it and okay and that's everyone already using it on their clusters.

A

Okay, that's not so good! So here you can see offshore definition of covariance I was looking from documentation, but in simpler words, ages we are recording all requests made to the API equivalences API server. Very important to understand that kubernetes is out in logging, doesn't add any additional security level to your kubernetes cluster. So probably you can build some log base matrix and define alerts that will recognize some animals in your cluster, but I would say that it's pretty difficult, it's better to set the correct Arabic rules, correct, firewalls yeah.

A

So when how it looks as the same as a normal logs, they should be used to find a cause of the problem and not find wife Rosa and not to protect your binary from crashing.

A

So, let's jump directly to the demo, let's check if it works. Yeah so I have created a small cluster on google google kubernetes engine. So let's check what we have right here.

A

It may take some time.

A

It takes oh okay, so that's I mean yeah some random names of ports of deployments. Does anyone sees something suspicious here.

A

Yep exactly so that yeah, so that's a very simplified example in your clusters. Something much worse can be happening, but in this case we have two deployments that has almost the same names, but in one we are using normal small all in the second one, someone has put zero inside yeah and if we dive deeply into this deployment, we can see, for example, literal uses some unknown image and so on. Let's try to remove it so.

A

B

A

Not meet Amy miss types and let check phone yeah, so it was removed, and the next thing that we should probably should do is try to find what who has created this deployment. But let's do one more sink so check that everything works as expected. Oh so I have removed it, but it was restarted like 10 seconds ago. Actually that gives me a good hint. What's going on. Probably we have a different job running in our cluster that monitors this deployment and if it's removed, then it recreates it. So, let's go to stackdriver login UI.

A

So, by default in Google Jiki, we have writing all admin activity logs to stackdriver for free, ok, let's call it a little bit down so I hear.

C

A

So, let's jump to advanced filter' yeah.

A

Let's check who has created.

A

Request metadata name and let's check our deployment road builder.

A

Okay, so let's run it yeah. This is that it was recreated like two minutes ago. Let's see what we can find here, so a lot of interesting information, but we interested like in authentication so authentication info. We have say here a service account for default namespace for default.

A

So that's already a bit strange because you shouldn't give such power to different default service account, but let's try to check it. Who has give it like I'm not going to spend your time I have it already in my top.

A

So let's submit a filter yeah. It was created much sooner. So, let's give it seconds yep, let's check it, so we can go inside the payload identification info, oh so that was created by me.

A

Yeah probably I was making some experiments and forget to remove it yeah my my fault yeah, but that doesn't mean that I mining Bitcoin. So let's use some different ways for finding output.

A

Here we have also, in the request, metadata a color 8 IP that we can use actually to figure out on from which node request was executed. Here we are getting node name and now, let's just check what is rank here and if, if you look on it, you can see almost the same part as we have seen before. Someone replaced small, always zero and we have some malicious pot running in our system. You might be, you might wonder like. Is it a lock yeah in this case?

A

It's so I can't simple the demo in real life. It may be not so easy to find it. You need to check every pot the drones on this node, but I have good news. So I was talking with small Claire. They are working on providing such thing as spot identity. So beside having service account, you're also going to have information about what identity, which gives you a pod name from each request, was executed yeah.

A

So let's finish our demo and check who has created this pot as before everything it's written already, let check it, let's give it and who is our attacker identification info? Oh, that was also created by me, which means one of two things: either someone has stolen my credentials or I'm mining Bitcoin on my own Corp resources, yeah and now, let's back to the demo, I had to slice so to be useful.

A

Audit logs should answer next seven questions and now I'm going to show you how, where you can find answers to all of them. So the first one is what happened so that was a pot created. Was it deleted? Has someone got config map or secret? So we store this information in verb field of audit event for resources it contains information like create. They did for non resource objects like healthy.

A

It just contains HTTP method, name like put posts, etc. When did it happen? So it's pretty easy. We have a request, receive timestamp, just check it. Also in all you demand, we have some deprecated fields. The timestamps that you shouldn't use, but also very interested is eh timestamp.

A

You can use both of them like because receive time, step or stage timestamp that depends on your preferences. I'll explain difference between them a bit later. The next question is: who initiated it: yeah we have everything we have user. We have impersonated user, very important notes. If request was sent to unsecured port of API server, we are going to hell like system unsecured here next question on what did it happen for this? Is this you can find in request, URI or object graft field, because you are I can't satisfy us all the time.

A

For example, if you are creating a deployment, then you don't he'll. It's named Enrico CRI. You should look in object graph. Next question: where was it observed? So I'll just logging is a part of API server. It records all requests to the API server, so it was all served on API server, but you might wonder what if we have some multi cluster, it's a regional cluster running and we have multiple instances of a pacer.

A

Unfortunately, we don't have this information in a litigant, but I will tell you how you can get it from where was it initiated? So if we have a list of so source IPS, so it contains like original IP, others and all proxies here. Actually, we have one problem: there is a possible attack vector, so we are not performing any validation of this source IPS. So how we are building. We are just looking on request. Hitter we have. There is a field which called exported from for and I.

A

Actually attacker can put whatever he wants, so you can get in disease some trash actually and where was it going and unfortunately, for this question we also don't have any answer inside of audit event, so you can imagine the some resource changes originated to etcd, but, for example, if you are interested in my Clio cluster matrix, possibly request was originated to metric server, but you can get some information about destination based on the requests. Your I.

A

Sometimes people ask why they can't use events, so there is a big difference between events and audit logs audit logs that something generated by 8a API server. For example. Even if you create a new event, corresponding algid lock is going to be created even at the same time. That's just simple. Kubernetes resources generated by that can be generated by any component running in your cluster, and they I would say, contain information like why one or another thing has happened like why deployment was scaled.

A

Why container was restarted, so you should look on events as something complementary to audit locks.

A

The next thing is kubernetes api server. Has anyone used them to like to find request to have made to API Sarah, okay yep? So probably is that was a good idea when we didn't have audit logs, but now I would say that not needed anymore, so why you shouldn't use commenced a pacer locks. So, first of all, you can't configure information that you are assuming there a second one that lets just API server locks.

A

So it's going to contain all information and if you increase verbosity, that's just going to be a lot of noise inside and the most important point that they just don't have. It doesn't have enough information inside so, for example, that's copied from current API server logs, that's entries that was related to creation of this malicious deployment.

A

Yeah, it doesn't have any name you can play and replace up with some cubelet or something like that. So I wouldn't recommend to use it. And the last point is a bit subjective. I would say, I think that each thing should have only one purpose and in this case Coburn's a PI server logs should be used to debug problems with API server, not with not problems with kubernetes custom.

A

Now we can go a bit and talk a bit about. How are you logging implemented on API, sir.

A

So, from implementation point of view says just a simple filter that we have in liquid chain: it's located exactly after identification. So if audit event says that this action was executed by user a then we can actually be use it. It was executed by user, a.

A

Now how you can stores this locks? Currently, we have two different ways: one is writing to the lock file using lock, back-end and the other one is sending out your locks to remote API using web back-end, and actually you can use both of them at the same time, if you need it.

A

What entered now coming back to the question how we can find out from which API server on which API server we have observed one or another request, so we can configure, for example, flowing year to add this information to the lock or you can get if you set.

A

If you get this information from original IP address of the request on your remote API one another important point about this diagram that it's a blocking way of writing audit logs, which means that if you have some other lock hanging, then you're not going to proceed is a request until it's written to lock file or remote API.

A

If you don't need it, then you can use batching strategy. In this case, we are adding buffer in front of our back ends and we have some co-routines that continuously checking it and, if buffer has, for example, by default as four hundred events, not even added lock events, then they are sent to web hook back-end or to log back end and / shear processed later.

A

Now a few words about stages, not everyone understands that each request to the eight Sara actually can generate multiple Elliott locks so how it works. Once added filter receives requests, then a first added lock is generated, which is- and this stage is called the request received stage the next. We are processing our request, and here we have two different passes, depending on type of your request. So if it's something long crying like watch, then we are going to the stage which is called response started stage.

A

So at this point another algid lock is generated, and once we have sent the last bite to our client, then we are coming to response complete stage, and here we also have a new elite lock and, as you remember, you see, I was mentioning stage time, timestamp so stage, timestamp that time spans temp when request achieved each stage, yeah, so how you can configure audit logging. So the most important point you should pass audit policy file to your API server.

A

So let's check how it's going to look so, for example, we want to record all pots and notes statuses updates made by cubelet.

A

So, on the first line, we are specifying level level is that some kind of verbosity of our audit lock? So we have four types where the first one is none. That means that you are not interested in this at all. A second one is metadata. It puts information such as user timestamp, color, IP, address and cetera. The next one is request, the same as metadata was request body and the last one is request response. So this metadata request body and response body. Next we are specifying users, verbs and resources.

A

If you will meet users, it means that this rule is going to be applied to all of them and in the last line we are specifying which stages we should skip. So if you are not interested in generating on each of those stages, a separate audit log, so then you should specify here which one of them you should skip. You may ask why, in this case, we are skipping request received, so it means that how you block is going to be generated on response complete, considering that we don't want any response beaudion.

A

Thus, anyone has some ideas why we are doing this.

A

Anyone okay, so the problem is that on the last stage we are also making some modifications to metadata. So, for example, we are adding a response code, so you know if you have 200, that your request has succeeded, otherwise it helped and yep. Let's go to the short summary.

A

So, first, the most important point added logging doesn't provide any additional security level to kubernetes cluster you, the second one. Audit logging should be used for tracking down problems in your cluster, so that's not only I some malicious deployments that could be also problem, miss mistakes of developers or some data that you can use later for post more times. So you can improve processes inside of your organization.

A

How do dogs are very easy, configurable and processable, so I, as I have shown in the demo on stackdriver and I think that ugly dogs should be enabled in every cluster by default. So the amount is not as big comparing to all other logs that possibly coming from your applications, but I would say that they bring a lot to heavy one and yep. That's all from my side.

A

The Sangha has some questions related yep. Maybe let's thank you like Mike.

C

It's possible to have also on the response on the a little log. What is changes on the resource so.

B

Yes, deployment.

C

What kind of if the fields are changed so.

A

I, when you're changing something you are sending a request body so by default, so you can get this information from request body. Unfortunately, we are not like processing it and storing a div, a real div, but yeah.

D

I can combine this with yeah.

A

It's imperative for yourself, but maybe in the future, there's going to be done. Any other questions.

A

ah Yes, there's questions.

E

So you say audit log: is that like a special logging, or is that the only logging that takes place- and so could you repeat the questions? This is the only logging that takes place. The audit log I'm fairly new to kubernetes here, I.

A

E

Well, and in a Linux system, I have normal logging and then I have the audit, which is much volume.

A

Knows that this specific blocks are alighted on you, two requests that have been sent to API server. So, of course, you can have also some locks from Linux kernel, but that's a completely different goal of this locks is to give you information what has happened in your cluster. So, for example, let's say someone association to your node and created some binary as a mining Bitcoin. You are not going to get this information from Cobras algae logs, because no request has been sent to API server and.

E

The logs are created on the master, yes,.

A

Iocated on the master and yes, they can be stored to log file or sent to remote API. Okay,.

E

Thank you sure.

A

B

Think you mentioned that we have to enable the on article login by default, for classes recommended I completely agree with that. So do you have any performance number on that, because I mean if I enable I would want to make sure that the performances is not sacrificed.

A

By performance number you mean how much resources are consumed by August filter, so I don't have exact numbers, but I would say like in memory and CPU is negligible compared to everything else that happens on API server. But if you are asking about volume of logs that are generated, that's a bit tricky question because everything depends on the cluster. So we have seen situations when three node cluster was generating much more logs compared to let's say 100 node cluster. So it depends on how dynamic your cluster is and yeah.

A

Actually it's this some ways to generate a lot of them by appending some random information to annotations, let's say, and if you are storing request and response bodies, that's going to be a huge I.

F

Excuse me, I just want you to share a small wall story regarding this. One of our production clusters got infected by this bit mining and what they did was they accessed through the cubelets port, which was unprotected because of my silliness, and they they they checked.

F

The list of running containers found one of the containers with curls in it, and then he called the curl to pull a binary from somewhere and that band drain turned and the mining software containing limits on CPU, limiting to 50% I saw that so we were able to figure all of this out today. I'm telling you all this, because we will figure, we were.

F

We were able to figure all of this out by just the service level, logs of kubernetes api server and cubelets service, the the regular service logs- and we have still not enabled audit log, because we thought it would be a huge tax on the CPU and perhaps the storage. We did fix the security firewall rules to not allow cubelet access from everywhere anymore, but this was this was my war story. Yeah thanks.

A

G

I, thanks for the talk um I was wondering if you can recommend um tooling, that I would use if I want to get notified about some certain events that I'm interested in so I'm not going to actively look at the log files and search for some something but I got notified. Do you have any suggestion of that.

A

Like oh yeah, I, don't have any suggestions, so I know that you can configure it through the stack dryer but as I have thought, building correct, Mattox and alerts that will notice something wrong with based on the information from our logging may be tricky so right now, I, don't have any information about it. Sorry.

H

So the question is about the audit logs for the Linux kernel. For example, someone gets into the perimeter and SSI sirs into the node and that's some pseudo commands so.

B

H

The like best solution to record something like that because currently were using the algae bit from no elasticsearch the container running on each node, and then it exports all the events which happen on a kernel level on the linux and it sends it to elasticsearch. But is there anything better.

A

All right now, it's a bit difficult to answer this question so I.

H

Can ask Google doesn't give this opportunity like out of the box? Would I, sir, could you repeat, I know that the google kubernetes engine does not provide this functionality like out of the box right. So.

A

As far as I know, we are working on providing out. It looks from notes, so not the kubernetes one but like from vm level mm-hmm, but I need to ask colleagues to confirm. What's the current status or size.

A

So if there's no question, thank you for your attention. Have a nice day.