Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 7 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Runtime Security: What Happens if a Container Goes Bad? - Jen Tong & Maya Kaczorowski

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Kubernetes Runtime Security: What Happens if a Container Goes Bad? - Jen Tong & Maya Kaczorowski, Google (Intermediate Skill Level)

Kubernetes security typically focuses on infrastructure: verifying that configuration is safe, and your deployment has no known issues. In this talk Jen and Maya will focus on a different aspect: runtime security. Runtime security is about mitigating damage done when part of your deployment is compromised. They'll start with an overview of common runtime security concerns, and discuss how they change as you move to containers and Kubernetes. Next, they’ll highlight existing features that help you respond to these risks, and explore popular tools to detect and control compromises. They'll finish off with a demo that shows these techniques in action as they protect a web application. You'll leave with an understanding of how to plan ahead to minimize impact, and how to respond to events when they happen.

About Maya
Maya Kaczorowski is a Product Manager at Google, working in container security, and was previously the Product Manager for Google Cloud KMS.

About Jen
Jen is a Security Advocate on Google Cloud Platform. In this role she helps developers and IT professionals stay out of trouble while getting the most out of cloud computing. If she’s away from her laptop, she’s probably skating around a roller derby track, or hanging from aerial silk.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi everybody how's it going having a good conference I. Guess it's coming towards an end soon, so we're gonna be talking about run time. Security today kubernetes run time security, which is about what you do when a container goes bad, but before we get into that, let me tell you a little bit about us. Your presenters, hi I'm, Jen I'm, a security advocate at Google.

A

Excuse me sorry about that I'm, a security advocate at Google, which means that I spend my days doing research on security, related topics and then coming back to events like this to help all of you stay out of trouble, while using cloud stuff and before that I most of my professional experience has been in software development bouncing around building stuff, launching social networks.

A

Things like that and I recently moved towards security topics, because I realized how insecure a lot of the softer I had deployed in the past actually was so I feel like it could make more impact this way and.

B

Hi I'm Maya Katz, röszke I'm, a product manager at Google, I focus on container security prior to computer security. I was working on encryption. Why did I switch to containers because obviously containers are the hot now so now we need to care about this as security professionals, not just encryption, which is old and boring, but necessary and prior to Google. I worked in consulting for security for a large healthcare, financial and insurance companies.

A

Great so enough about us now, I want to learn a little bit about all of you so that we can kind of adjust the stuff. We talked about a little bit to fit your needs the best, so how many of you should forgot? These questions are familiar with the NIFT cybersecurity framework, a good number of hands, more security professionals than you like it so grown.

B

Common common reaction, yeah.

A

Not as much excitement as some questions elicit have you how many of you have ever had that container incident or a container that you have deployed Wow.

B

Come up and talk to me later I want to hear all about it. How.

A

Many of you have our monitoring containers for security issues. Oh good number of hands. Thank you. That's very helpful. I hope it's kind of adjuster content to address all the things you're thinking about. So here's our agenda.

A

What we're gonna talk about today, we're gonna start with what is container security in general and then we're going to pivot to why that's different from virtual machines and then finally, we're going to talk about how to detect bad things that happen during runtime for the code that is already deployed and we'll follow up with a short demo of kind of a silly scenario to kind of show you what this looks like in action using some open source stuff sound like a plan, awesome, okay, let's get started so I'm gonna, pass it over to Mike.

A

B

And so let's talk about what computer security is and why you're even in this room, so why don't you grab our container security, I? Think as as developers or security professionals in this space, we really have two mandates here.

B

One is first figuring out what's possible and going and implementing that and two is building all that chip by default, so that everybody else who doesn't understand it doesn't get hacked and doesn't get screwed and there's this great quote here, which is basically something like nobody understands what the to do and in it it's happening, it's real where we have the Tesla hack, maybe six weeks ago now it's you know one of those kind of non-event hacks, it's not even particularly interesting, but it's one of the few public examples of a kubernetes hack.

B

They were running committees on a cloud provider, I won't name names and they had the community's web web UI dashboard open, not password-protected it had their cloud provider, I am credentials in it. Somebody found it on probably showed an got the credentials and started krypton whining and you're, like the remaining Bitcoin, no other remaining Manero. For all your notes out there, so they have somebody's minding one arrow in your cluster on a cloud provider. That's not a good situation to be in.

B

So how do we think about security in general for containers, the the model that I will champion, at least for the next six months until I get bored of? It is three different areas of container security. The first one is what we call infrastructure security and how and how you deal with with compromises of the communities, API provincial escalation, financial theft, that kind of stuff. This is really familiar to anyone doing in for security work right, you already come up with these risks in how to deal with them.

B

The second one is is what happens if you find a node a in your image tomorrow? How do you even know that it's deployed? How do you patch it? How do you fix it all that kind of stuff? The third one is about finding things that are currently attacking your environment and remediating. Those risks like what, if somebody is currently crypto mining, what if there's somebody's, apparently trying to do something or a particular interesting one for me- is flooding the event pipeline.

B

What, if I just send you so many events that you have no idea what's happening in your environment, and so we've split this into those three buckets infrastructure infrastructure security am I, secure to develop containers software supply chain. Are my secure my secure to build and deploy, containers and run time security, which is about? Am I secure to from containers and the last one? There is a focus for today's talk, so if our make sense great, so why are containers different from VMs?

B

Well, containers are supposed to be dynamic, so we deploy them again and again and again and things change, and they look slightly different than before and for for those of you who are familiar with virtual machines, which is probably anybody who is older than me, or even five years younger than me.

B

You might remember that we have hardware with an OS and a hypervisor, that's the important part right now and then VMs on top and we've packaged the VMS as applications with their binaries and libraries, and why did I focus on hypervisor, because until very recently, like maybe like two days ago, we we thought of a hypervisor is a really strong security boundary and we didn't really have something equivalent in containers right. If we're gonna run two workloads that we think need to be properly isolated from each other, we rely on a VM.

B

We rely on a hypervisor now, what's different in containers, one, we don't have a hypervisor, although please do go. Look at the G visor and cut two containers projects they're doing some really cool stuff, but we also have a much smaller host OS. So we've taken a bunch of the blue that would normally exist there and taking it out and that's that's actually a good thing. We've also have our binaries and library separately, packaged from individual applications, so you're able to manage those separately and which is again making it easier for you.

B

So at a high level. What are the big differences take away? One the surface of attack of a container is different than that of a VM right. We have this smaller host OS, which should be easier to manage on the. On the other hand, you can't necessarily you know you- don't really have something strong to actually contain things- you're not familiar with this surface you're, not familiar with with how to do isolation in containers.

B

So that's a new topic to explore to is resource isolation, containers use namespaces to isolate things like like storage volumes that you might mount, which is great, but they're not meant to be strong isolation.

B

One of my colleagues Tim all Clara gave a great talk this morning on secure pods, which you should go home and watch about how you can think about isolation, containers and what kinds of attacks can happen even if you're running a sandbox, what how can you go through the network, logging etc to escape that sandbox at touch? Another process.

B

Third, is: is root. Permissions information management in general access controls are managed in containers. You know in things in communities like using our back like we're familiar with these types of things. The downside is that containers have much more privileged permissions in your environment than a VM would have had. They have access to a wider range of syscalls, and if you watch the demo from Liz Rice this morning during the keynote she gave a great example about containers running as root way too often, and the last item is lifetime.

B

Containers by defaults are meant to be they're meant to be immutable. They're meant to not change when they're running, so we deploy them again and again and again again, like I was mentioning earlier. What that means is that they have a much shorter lifetime. It should, in theory, be harder for an attacker to persist in your environment.

B

The downside being, if somebody does get into your environment does something bad and they can kill that container or that container is no longer there by the time that you get there, you can't do forensics and that's kind of a scary concept for for a lot of security professionals. So I've pointed out some differences, but what I haven't wanted out is that, basically everything else is the same right. You have the concept of a firewall for VMs. You can have a viral for containers. You have some sort of end point detection thing for the ends.

B

You can have some sort of input, detection thing for containers. You need identities, you need. You need all these things that look very similar you're, not learning brand-new concepts, you're just applying them in a slightly different way than you were doing before. So it's not it's not a complete shift of how you think about security. So with that, let's talk about runtime, specifically Thanks.

A

Okay, so now we're ready to talk about that Brunton part.

A

So one of the first questions, often here is- is why bother because it might seem like you already have everything you need. You already have all these great tools, for you know securing your entire CI CD process and you only deploy vetted containers that have already been you know known, sanitized and clean, so you should be okay right. This is kind of an example of a concept of what's called fence post security, which is when you want to build a fence to keep animals in one place or out of another place.

A

You don't want to build like a narrow tall fence post. You want to build if you only have a certain amount of time, it's much more effective to build a low fence that is wider and covers all those holes and information security is much the same. We don't want to spend all of our time, focusing on the part of our cyber security plan that may be most interesting to us, or that we have the most expertise on. We would much rather have a kind of more smooth coverage across the entire spectrum of potential incidents.

A

This gives us benefits in that things can can check each other. We get different. You know different systems with different techniques that can reinforce each other and make a more secure environment altogether, and in addition, if you, even if you have a secure like container building pipeline and all that, you still have potential issues with incomplete vulnerability scan, so you might have messed up when you did that vulnerability scan on those containers, you might be using something incorrectly or zero-days can come down the pipe any time and you might be able.

A

You know you might have a situation where you get an event before you can get around to patching and redeploying those containers, and this is where that ms/ms nifc framework question comes in it's good to have some kind of framework. Some kind of way of thinking about this problem to help us make sure that we avoid that fence post security scenario, and although this is almost a trope in a lot of security spaces, it's still useful and we can use it now to make sure that we approach container security in kind of a holistic way.

A

So, let's map this to container on the identify and protect part, these are kind of the first two steps. This is the stuff that happens before run time. We want to know that we're deploying stuff that is secure. We want to know that we've configured things correctly. We want to make sure that the right ports are exposed to only the stuff.

A

We should be exposing and we want to make sure that we use the features of kubernetes around security, to make sure that you know we isolate those things correctly, for example in network policies using role-based security, security secrets, encryption and APOD security policy on the next step that detect side of the framework. It's about detecting container behavior that deviates from the norm. So it's visibility into our containers so that we know they're behaving correctly when they're operating right, but when a security event happens, that's when those containers start behaving differently in some significant way.

A

We want some kind of software that will help us differentiate between those two situations. This is definitely in the the philosopher, runtime security and beyond that, once we've detected an event, we need a way to respond.

A

This could be as simple as being alerted when it one of those incidents occurs, but kubernetes gives us a lot of capabilities to make real-time remediation a lot easier, for example like restarting a container and then finally, regardless of what happens with the incident, we want to have some way to recover, and this is a kind of a combination of factors. We won't have the tools in place so that we can dig in and figure out what caused the problem.

A

We're going to be able to do forensics and get meaningful information about the incident, and then we want to be able to take that information and pivot it to change our processes and software. Maybe to make sure that the ax incident doesn't happen again because it's not about if it'll happen it's about when it will happen, and the truth is now that we've all spelled it out much to what Maya mentioned. This is very similar, there's not a lot different from protecting any other kind of resource, any other kind of resource in our infrastructure.

A

So I guess sorry. Containers aren't quite that special well.

B

You're all special still it's okay, so.

A

Let's dig in a little bit more into the parts that are most kind of connected to the runtime security story, which is the detect and response step. There are a lot of different ways to think about these on containers.

A

So, let's talk about monitoring those containers, so how does this work so we're gonna need some kind of hook, some kind of hook into our container so that we can understand. What's going on.

A

We need some mechanism to login that behavior, especially if it's abnormal.

A

We need a way to set the policies around the alerts that come out and the AMA have automatic, readme tree mediation that we do in response to it and then, finally, we need some way to do forensics afterwards, so digging into detect there's a huge volume of different events that are going to come out. No matter what way you look at it, but there's a lot of different ways: there's a lot of different features of our operating system and software that we can use to dig into that, but kind of taking a step back.

A

We're gonna need to look into something some kind of metric that we can use to distinguish those correct behavior from the incorrect behavior of our containers, and this is probably going to go along the lines of looking at system calls or network activity. Something like that and system calls are really useful because they give us access to information about when processes get created, network activity, file, events, devices appearing and disappearing, but it's not foolproof.

A

For example, if there is a flaw in the hardware that can be exploited and like the recently discovered meltdown, that's a situation where it's going to be hard to detect that just by looking at system calls not impossible but difficult, and also there's going to be a huge volume of these events coming. But this is a pretty good place to start.

A

Unfortunately, we have great ways to hook into this Pete Reyes and Kay probes, and you can also access it using features protocols, features of the underlying operating system like EBP, F and X CP, which gives us a kind of way to inspect that stuff. I.

B

Think the only I'll call out there is some of these features only come with newer kernels, so if you're not already running something like a 4.11 or 4.8 or something you're probably gonna want to. If you want to use these look into that the other. The other sad part about this is that I have know a lot about kernel security now, which is very sad. You.

A

Know you don't enjoy reading kernel security details, it's fun, I like it yeahi BPF, is awesome, so just dig into that one.

A

So, given that we want to look at the system cause, there's a few different there's a variety of different ways that a lot of solutions out there kind of approach this.

A

So let's kind of look at the deployment models that are popular so generally, we have. You know a container that we're interested in running on a pod somewhere earning in a node and most of the solutions out there rely in some way on a privileged container. The privileged container is going to kind of act as a broker to a lot of the stuff going on, and it's also where we're going to put all rules or may gee, maybe our magic machine learning stuff to try and infer what behaviors correct versus incorrect behavior.

A

These management containers tend to consume a little bit of CPU time on our node, usually one to five percent overhead. But you know I, guess it's worth it and then getting information to that privileged container varies a little bit. There's a few different solutions out there. Some of them involve putting a kernel extension on the underlying operating system so that we can report that back to the privileged container. Another common approach is to have a special privilege container running on each one of those pods either way.

A

The feed results ends up going back to that management container.

A

Once the analysis is done, we're going to have a giant stream events that are going off to some kind of storage and there's a lot of options here, and it varies by by provider generally they're gonna ship them off to a cloud file system of some sort or maybe putting them in some more local file system or streaming the data into some kind of event, processing system like Apache, beam or Kafka. So you could do real-time analysis and some of them also feature things like a ring buffer. That's running on the host node.

A

That kind of keeps a very high resolution. You know ring buffer of the most recent events at a higher resolution, which could be useful for doing forensics. If you can, if you can get to it quickly,.

A

So, let's take a little bit into the response side. So now we have a system where we can kind of slurp up. All those events do some quick analysis on them and start to distinguish between correct and incorrect behavior.

A

So there's the very first thing you might want to do is you might want to send an alert so when you've detected and took your behavior or something you suspect, this is kind of like the lowest lowest impact on your infrastructure. Just send them to learn, you don't wanna, send too much or people will start to get sad and there's a lot of soccer out there. They can help you further aggregate. This information, like our cloud security command center, for example.

A

If you have something that seems a little more troubling than something you want to send an alert for, you can also isolate the container but I like moving it to a new network. This will help you prevent it like do essing other services, since it has access to your internal network, now could also pause. The container, in other words, suspend it, and this might be useful.

A

There's one of those kind of middle stage: ones like you, detect a crypto mining, it's so it's compromised, but there's not like you know, someone's, probably finished their kill chain and they don't want to go too much further and you might want to pause the container and come back to it later because having it there in its present form is very useful forensics for forensics, because if you destroy it you know a lot of information that might be useful will be gone and if things are getting worse, you might want to restart that container.

A

You know doing a cou control delete pod, get it out of there, and this will. You know, reset the state of the application. It's lower impact, because, hopefully you have you know redone, it's tough out there and no one will notice it and it'll certainly disrupt the hackers ability to step off of adding and cause more trouble. I can also kill the container. You can go, doctor kill it automatically, and this would halt all the rest thing all the processes which will be a little bit less graceful, but it'll.

A

Don't only stop that event in its tracks, just.

B

Say or people take photos if they want to I, think that the thing I'll point out here is also restarting a container is great for getting rid of something that's currently sitting in a container. It doesn't stop someone from doing the exact same thing again, 20 minutes later right, so if you're not sure how they got there in the first place, that's probably not the step that you want to go to. First, you probably want to figure out something else: try to maybe kill the container.

B

If that's, if it's really really bad, rather than take four start as an option and killing is always your last step. Yep.

A

So why are container special again well back in the land of virtual machines? We had long list systems and we could actually try and play the the detect and patch games to find a vulnerability and patch it and keep up.

A

We had a per host software, which allowed intrusion, detection systems that only they really only had to distinguish between running hosts and between specific running hosts with the on that network, whereas in containers we have a little bit more granularity of our services, they go up and down often, which has pros and cons, and the network is a lot different, because there's a like an overlay network. The events that happen might not correspond to a specific host because that container might move around from one node to another.

A

So by the time we see it in our logs, it might have already moved on. So we need something that can kind of figure out the details of that container. That's running what's running on it, so that we can, you actually understand the threat that occurred.

A

So what can you do today? The first thing is: I definitely recommend have this as part of your security plan, so I know a lot of you, especially if you were in the previous talk, are gonna, go back and change the way. You've configured things and I get a security plan in place to make sure that you're well set up to defend against events, make sure that runtime security is at least something to consider. As part of that plan build that fence.

A

You know with actual covers, instead of just some awesome fence posts, oh there's a lot of open source so actually there's a limited amount of open source options out there right now, there's a there's. A few out there I've been playing with, but you're probably gonna want to talk to some commercial providers also and see which options they have because the security space, a lot of the software is, is dominated by commercial providers.

A

It's just a little bit different than we're used to those of us who came from all open-source everything, but there are some open source options. I would recommend giving them a spin, but don't hesitate to actually talk to some of the commercial providers.

A

Also, the other thing I'd encourage you to do is to deploy early so actually start this for fairly before you have something live or if you have something live do sooner than later, because the more time you have to kind of baseline, your data, the better it's going to go, if you don't like, if you try and do this like at the moment of the first event, it's not going to be nearly as useful as if you do it earlier on yeah, they give you a chance to tune those signals and finally rehearse an event- and this is just general security advice, but definitely applies to run him security.

A

Have you know, do a pen test have someone run run some loads that look like they should be suspicious and see what it looks like on your your monitoring system, so that you can make sure that you'll be able to detect that event and also to make sure you don't get a million alerts that make it impossible to figure out. What's actually going on.

B

Okay, so the reason part of the reason were saying: what can you do today? Well, one of the things you can do today and I wouldn't be a product manager if I wasn't talking about a product.

B

Is that on on Google cloud, if you're, using Google, Chrome, Aires engine and you're, using one of one of the five partners that we've integrated, as of as of yesterday aqua security, capsulate stack rock statistics secure twist, lock can now all right findings directly to the cloud security command center about things that are happening in your Google Chrome days, engine clusters. So if you have one of these tools, you're considering one of these tools, you can see it all as a single pane of glass see that information directly in in the product. That's my little.

B

You know 60 second product pitch great.

A

Okay, so we're a little behind so find us accelerate the demo a little bit by playing the pre-recorded video, which will hopefully speed it all on a little bit. Don't know with my Moscow: it's you, yes, I.

B

Was great there.

A

Yeah, the fake mouse tricked me okay, so let me mirror those displays. Cuz technology, you're full displays behind.

B

You yes, yep brilliant.

A

A

Here's an example of a kubernetes cluster I have running it's super simple. Let's take a look at the services on it. We just have a wordpress, my sequel, nothing too exciting, but we're in good shape, because I went ahead. For this event, I went ahead and deployed one of the open source options out there. It's sis dig: has a an open source project called Falco which I arbitrarily chose, but but actually like a lot, and hopefully this will help us detect something if anything were to go wrong.

A

So we can see that this is a this. This particular pod has two nodes on it, which is why there's a special daemon running on both of those like a daemon set- and here is the actual WordPress I have deploy to this cluster. It looks like any other WordPress and it's an example of something that came from a known good image, so, like I, know that this image does not have any more vulnerabilities and WordPress just usually has, but WordPress has some of the unusual stuff about it.

A

Is that even if I do everything right, if one of my users set a terrible for character password someone malicious can cause a lot of trouble because they could come into here and just from a normal account. They can do things like add, not a reverse. So you can add a reverse shell to one of the PHP files up there and, however, she'll works is just by adding that one line of PHP to a known good WordPress instance.

A

Our attacker can go and fire up just a regular VM somewhere else. So this is the attackers VM and they can fire up netcat and wait for incoming connections and then, if the attacker were to go to that 404 page, everyone else that just looks like the 404 page is hanging no big deal. But while that happened, I got a river shell on that, where politics and fortune this WordPress instance is reasonably well built like it's. Actually, WordPress is running as a not root user.

A

So I can't do a ton other than like look around the network and enumerate other potentially unsecure services, but I have already compromised a machine. So I can download like some crypto minor software and try and give it a run. In this case, an ether minor which turns out did they actually run because the system didn't have any GPUs on it.

A

But if you notice what was happening, wow that was getting created is that of course looks a little bit suspicious and falco has a default rule set I didn't write, any rules, I just use the default ones and the default rule set does look for shell creation. Cuz shell creation under containers is something that probably doesn't happen in normal circumstances. So it's totally okay to do an alert based on that.

A

So just an example of us actually detecting something happening. It's awesome, yeah, okay, so back to slides. Where did slides go? Thank you yeah! Oh no, oh there we go. They were hiding on another screen, so real quick. What we discussed gave an overview of container security describes on the path, practical differences between the containers and virtual machines and kind of gave you some advice on how to frame that within the NIS t-- framework avoid fence post security. Then we followed it up with some advice on what you can do today, including on gke.

A

So thanks so much the slides are up here on sketch thanks a lot for coming to our talk.