Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Recent Advancements in Container Isolation - Tim Allclair & Adin Scannell, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Recent Advancements in Container Isolation - Tim Allclair & Adin Scannell, Google

Container orchestration enables higher bin-packing and utilization of machines, but native linux containers do not offer the same degree of isolation between workloads as separate VM instances can. Attackers could abuse this lack of isolation to move through a Kubernetes cluster after gaining a foothold in a container. Fortunately, there are many tools in the defenders’ toolbox that can be applied across multiple levels of the stack.In this survey talk, we will look at several recent or upcoming advancements in container isolation. You will learn about new kernel features, several "sandboxing" approaches, and features being developed in Kubernetes to harden the Pod and Node boundaries. After the talk you will have a better understanding of how to secure your Kubernetes applications and clusters with the latest features.

To Learn More: https://sched.co/GrZl

A

Hello: everyone welcome to recent advancements in container isolation. I think we're gonna get started so hi. My name is Tim all Claire I'm, a software engineer at Google and I'm also co-chair of criminais, cig auth and a member of the product security team and.

B

I'm Aiden scandal I'm also a software engineer. I work on G Weiser.

A

So today, we're going to talk to you about container isolation, but before we dive into that, I want to make sure we're all on the same same page, about what we're actually talking about when we say isolation, a model of security that I really like is the confident she can potentiality integrity and availability model or CIA model. These are three key properties that we want to maintain across any isolation boundary, but there's two other properties that I want to call out. The first is that isolation is really a multi-dimensional problem.

A

It's not enough to just isolate the network when an attacker can potentially escalate privileges through unsecured sis calls and then attack the network from a different angle and security really requires a holistic approach. The second property to call out is that isolation is a directional problem just because a container is well hardened and the node is isolated from processes in that container doesn't mean the inverse is true process running as root on the node. For instance, the cubelet can easily reach into that container, read its data and use its credentials.

A

That sort of thing so, with that I'll hand it off to Aidan, to tell you a story so we're.

B

Gonna tell a story to help frame some of these ideas in more concrete terms. Now we're until this story, with a help of a little bit of time travel, so everyone arriving late might be really confused, but that's okay. Everyone here will know what's going on so almost no one knows this about me, but I don't really care about containers. My true passion is actually meme based cryptocurrency and in 2019 Tim and I. Both leave Google to start an online me I'll coin exchange.

B

Now everyone knows that the past few years have been rough for cryptocurrency generally, but meow coin just takes off like a rocket. Unfortunately, our success is rather short-lived and we're back here in 2020 to present a retrospective of the great meow coin theft.

A

So our story starts with an attacker Eve who manages to penetrate our clusters the way she does. This is by hiding a backdoor in the popular NPM library declawed.

A

We made heavy use of this library in our meme service, which attaches a meme to every meow coin transaction and Eve exploited the backdoor to gain a foothold into our cluster in the memes pod, prod or memes prod pod. So we start here. Eve has code execution in memes prod, and the very first thing she is gonna do is attack the API server because she knows she can get full access to the cluster from there. Fortunately, we're using best practices and since memes prod doesn't need to talk to the API server.

A

We haven't even mounted a service account credential into memes prod. So that's a dead end. Next, she goes after the meow wallet where the goods are fortunately we're using sto to implement a least privileged model of network access. We use fine-grained authorization policies through ISTE, Oh sto is 1.0 since 2018 as I'm sure you all know, and we're not going to say too much more about this here, but there's an entire track on service message meshes on Thursday and I, encourage you to check it out if you're interested in this topic.

B

So here is where we got a little bit lucky. We were running our memes prod service inside a sandbox, and the reason is that it's exposed to the public Internet and we use every random dependency. We can find from every corner of github, so we decided would be prudent to to keep it inside a sandbox, so Eve was was not able to escalate directly now.

B

The reason you might use a sandbox is that you want to mitigate the risk of a kernel owner ability if there's a high chance of running potentially malicious code inside that container, and this may be running user code directly or it may be some front-end that you feel there's a high chance, that's exploited or you're running some media transcoder or operating on some other, potentially malicious user input, or maybe just like us. You have a big pile of third-party dependencies that you don't necessarily trust or can't audit.

B

So the way a sandbox can add protection depends on the type of sandbox. Since this is a very hot topic, we're gonna be talking about a few different variants in general, when you run an application or a container they're interacting directly with a fairly broad, complex kernel, surface and the whole point of a sandbox is that you want to either transform limit or somehow insert an additional boundary in this layer to prevent or mitigate the risk of criminal abilities leading to privilege escalations. So just to reinforce this a little bit.

B

This table shows a list of linux. Kernel owner abilities, actually coda code execution vulnerabilities published in 2017, and this isn't a rag on the kernel at all. Kernel is an incredible piece of software, but its enormous Lee complex and has broad range of responsibilities so, with any large complex piece of software, there's bound to be bugs. People can exploit, including the sandbox layers that I'll talk about, and you just want additional boundaries, additional layers of Defense.

B

So the first thing to consider is that the host kernel itself has a bunch of mechanisms built-in for limiting that surface, assume things like the SATCOM PPF policy for filtering system calls or LSM modules, and these are fantastically useful and you should use it as part of your security best practice even for applications. You trust, because they're very, very high performance, very low overhead and they don't really change the semantics of the container at all.

B

The one challenge with using these for sort of arbitrary workloads is that there's a fundamental trade-off between the the degree of restrictive, nough, C's. So the degree you can lock down that kernel surface and the the diversity of workloads you can run. So it's not necessarily feasible to take an arbitrary container that you know nothing about and run it using a very restrictive set comp policy. So that leads to the first class of sandbox that we'll talk about, which is machine virtualization.

B

The idea here is you take a container or a pod, and you insert you you put that in its own. The so VMs are fantastic, mature technology on bare metal. You typically have very good performance now, there's a stream of sort of cloud native hypervisors that are coming out that are focused on low overhead fastboot times no legacy stuff, but there's a couple of challenges with with VMs in this model in general.

B

The first is that there's a separate guest kernel inside this VM M, that's generally talking physical cores and physical memory, and so you'll have a second scheduler, a second page cache that can reclaim mechanism and the degree to that's, to which that's a problem really depends on the kind of workload that you're running here.

B

The second challenge is that hypervisors, don't natively, run containers right, they're, run machine images, and so you need proxies and various plumbing and infrastructure to actually wire together, containers that are running inside those VMs and that's what projects like cata containers are focused on doing so. The second category of sandbox that I'll talk about I'm calling kernel virtualization, and the idea here is that the sandbox isn't talking directly the kernel anymore, but is talking to some other kernel interface, and you can do this for compatibility reasons.

B

There's a couple of very famous examples, so Alex owns four smart OS allows you to run Linux containers windows subsystem for Linux, another great example where the applications obviously not talking directly to Windows talking to some other component. That is brining Linux ABI or you can do this for isolation, which is what G visor is focused on doing so at the bottom.

B

There is a diagram of how do you guys it works and it's effectively a second user kernel that is interpreting and implementing all of the application system calls, and it does that not talking physical cores in physical memory, but rather speaking threads and memory mappings. So it's trying to preserve the original container semantics, which makes it suitable for small high-density services. It doesn't impose large, fixed overheads, and it has a very fast sort of time. So, but this is a separate kernel implementation.

B

It runs most things today, but it's evolving fast, there's some challenges around compatibility and performance, particularly for cases where we haven't really optimized for those things. So the last class of sandbox that I'll talk about I'm, gonna, call non Linux environments and I get asked about these a lot. So I thought it would be really useful to have a slide about them, but these are things that are not necessarily running containers to running different kinds of workloads. So a good example is nabla containers, which is a runtime that runs so low.

B

Five of you know kernels, and these can be really efficient. They can expose a very, very small host system surface, but you're only able to run certain kinds of workloads that have been built for these environments. You're only able to run you know, kernels you're, not able to like pull something off the shelf and run arbitrary docker containers.

B

The next thing isolates CloudFlare had this famous blog post, where they're talking about doing multi-tenancy using using v8 and isolates- and this is very similar, you're running JavaScript and web assembly programs- you're able achieve high efficiency and performance, but it's not running containers or what we. What we normally talk about is containers. So all of these things are available. There's this new mechanism that you can use, starting in kubernetes, 112 called the runtime class.

B

This allows you to define a distinct set of runtimes and then configure on a per pod basis, which runtime to use and there's a number of improvements that are landing, including accounting for sandbox overhead being a bit smarter about how you can match features how what how to know what can run with what and, of course, a whole bunch of stability performance testing for the runtimes themselves.

A

So our sandbox successfully contained Eve and protected our kernel surface from her. However, we were running another meme service in our cluster, the memes dev service. Since this wasn't exposed to the Internet, we hadn't bothered to sandbox it, but our sto rules allowed for cross communication between meme services, and so she was able to exploit the same backdoor and jump to memes dev, which is non sandboxed so from here we'll talk a little about audit logging and how we were able to piece together these these.

A

This attack, so container ID back in 2018, was being actively discussed for extending the audit subsystem of Linux. This is actually the first native container concept in the kernel. What we think of containers as today are mostly pieced together through different different features of Linux and there's no kind of native container concept within there.

A

So container ID adds that that adds a way to tie audit events to a specific container, but Eve knew that we had a good at audit system, so she decided to attack our to try and obfuscate the logs by changing the system. Time, thanks to the new time namespace that we were running and are super up-to-date colonel when she changed the system time, it only changed the time within her container we're also using best practices.

A

Even though we didn't have a full virtualized machine or kernel, we were still using various host kernel mechanisms to contain that container as well. So docker enables an app armored profile by default on a Parmer enabled nodes, but a new feature called policy namespaces and with that policy stacking lets us layer on an additional application, specific profile to harden that even more.

A

So, thanks to those improvements, she was only had a very small kernel surface exposed to her, but she attacked it anyway.

A

Although we had a lot of vulnerabilities in 2017, we saw much fewer in 2019 and 2020 thanks to the kernel self protection project. This is a project that aims to harden the kernel as opposed to adding new security features. I mean it's doing this through sea hardened c programming techniques, things that aren't necessarily new ideas, but are very difficult to apply to a project as complex and as old as the Linux kernel.

A

So, just to highlight a few improvements, the elimination of variable-length arrays protects against stack, exhaustion type attacks, fixing bugs in switch statements through incorrect fall through z' initializing local variables. Checking downs that sort of thing.

A

Another improvement comes through control, low integrity, so back in, say, 2017 a buffer overflow attack would typically lead to an attacker writing some code on to the Sun to the stack and then changing the the return address to jump to that address. On the stack from there they've gained code execution. We fix this by setting the stack and the heap to non-executable since programs, don't typically run code out of the stack.

A

This led attackers to use a technique called return, oriented programming where, rather than writing the code directly to the stack, they would just modify. The return addresses to call existing functions with arguments that control and by chaining together those function calls they effectively gained code execution again, that's what we fix with control flow integrity and two different mechanisms around this one is to separate the call stack from the data stack, so a buffer overflow on the data stack. Wouldn't it would be hard to go from there to modifying the call stack itself.

A

There's also, some preliminary hardware support coming out through Intel's control, flow enforcement, technology and arms, pointer, authentication.

B

So everyone remembers: 20:18 was a rough year for many reasons, but one of those reasons was that we discovered that Hardware doesn't really work the way we think it should work, and so Eve started scratching at the hardware to figure out if there any secrets or credentials that she could steal from the node itself.

B

And so as a quick background on how spec e to execution works is an attacker will try to trick the processor into executing down some path where it's accessing some data that they wouldn't normally be able to see, and this is a problem because it turns out that processors, don't necessarily respect page table permission, bits, bounds, checks or even EPT translations in the speculative path, which is very bad and then the on that same path.

B

They use that data to generate some side effects in some shared architectural state that they can then observe once the processor realizes. Oh, that's, not the path that I'm taking and unwise that execution. They can still infer the original data that they wanted to get access to. So this could allow someone to steal credentials keys anything on that. On that same host unfortunate, there was a lot of progress through 2018 and 2019 on this front, including a lot of software mitigations. So compilers have incorporated various fences to prevent the bound check, bypass, vector variants.

B

The kernel now has two sets of page tables to ensure that users don't have access to mappings that they shouldn't have access to. The kernel can be compiled with ret killeen to ensure that user space can't poison branch targets or use steering to access gadgets that they want, and the kernel poisons physical addresses for page tables aren't present to defeat the foreshadow attack and, of course, there's no more lazy floating-point, which is actually a performance gain anyways.

B

So our main lesson here, we didn't actually take any action, but our main lesson was to keep all your kernels and tool chains up-to-date, because this stuff is continuing to land. There are also a number of hardware mitigations that landed on 2018-2019.

B

These include mechanisms to basically new control registers that allow the kernel to partition the branch target buffer to prevent using user values from the branch target, buffer or values from a lower privilege level allowed to flush that buffer on privileged transitions and various things to prevent other kinds of attacks here. So the lesson here is also make sure you're keeping your firmware up-to-date, principally because the kernel will select the most appropriate mechanism to use which could be some mix of hardware and software mitigations.

B

So at this point, Eve was pretty frustrated and she decided instead of trying to attack our service. She would just disrupt it by by degrading the service and being a noisy neighbor. So software is rarely bound purely by cycles, it's usually bound by some other mix of different architectural components like cache, bandwidth, memory, bandwidth and so Eve, for example, just started running a random video transcoding service and chewing up all of the memory bandwidth available and degrading the performance of our memes prod service dramatically.

B

So this is an active research area, particularly in light of the last things. I just talked about. There's all these interesting side channels, and so people are looking at how to isolate all of these shared arc four components, but one thing you could try today is Intel rdt, which available in recent kernels and run C, to impose limits on cache bandwidth and memory bandwidth and prevent the noisy, neighbor problem.

B

Unfortunately, we, like everyone else, were taken by surprise in 2019, when some security researchers discovered impossible execution and he was able to escalate privileges right to the node.

A

So this brings us to chapter three escalating privileges: Eve has now gained code execution as a root on the node and we'll see what she can do from there. So the first thing is now: Eve has access to all of the credentials running on the nude. This includes the cubelets credentials as well as any other services that are running on that same note. So now she's going to go back to the API server and let's take a look at a session log and see what what exactly she did so.

A

First she uses the cubelets cube config, which includes its credentials to start making requests to the API server.

A

She looks at the pods that are running, and here we see an abbreviated list of just the ones we care about and now she's going to try and exec into that Miao wallet now coin wallet, but fortunately, since the cubelet doesn't need to exec into pods, especially pods, on on their nodes. We forbid this through least privileged scoping on that. So she takes a closer look at the Miao coin wallet and sees that it uses this wallet key secret. That sounds pretty juicy, so she tries to grab that secret.

A

Fortunately, this is forbidden through something called the nude authorizer, which has been in communities since 1:7, and what that does is it maintains a graph of all the resources that are required by all the pods that are running on a node, and so, in this case there weren't any pods running on node, one that used the wallet key and therefore node. One has no business reading wall at key and it can forbid it. But Eve has another trick up her sleeve. She says what, if I can coerce the wallet to be rescheduled on to node1.

A

So she takes a closer look at the wallet and sees that it's using a node affinity term to schedule the the wallet workload onto a node, that's labeled, with a sensitivity of greater than 10 no problem. She says I'll, just patch node 1 to have that set that sensitivity to 11, but this is forbidden as of 113, because the node restriction, kubernetes dot IO label prefix, is always forbidden.

A

We're also going to work on kind of ramping up the the number of labels that are forbidden as well, but so far we've only been talking about using the cubelet credentials itself. One of the the open areas going into 2019 or I guess we did this in 2019 is to apply these same restrictions to the demon sets and the other workloads that might be running on the node, because those if they need to say edit, the node, then then those workloads would be able to change those labels.

A

So at this point, Eva's starting to starting to get a little worried and she wants to at least maintain persistence on this node. So she tries to leave a rootkit behind, fortunately, we're using a container specific image with a read-only root. Filesystem and we've also heard it protected the kernel via boot attestation to make sure a custom kernel or any unsigned kernel modules can't be loaded, but she has one other idea.

A

There's this metrics pusher service running on node 1 and the metrics pusher receives metrics requests from other workloads in the cluster in order to make sure that those other workloads are pushing metrics for themselves. It delegates authentication to the API server through token review, so the in this case the meow wallet, sends its service count token, along with the request and then metrics pusher delegates that authentication check to the API server.

A

The problem is that metrics server now has a copy of the Miao wallets service count credentials, and it can do anything that that that service count is capable of.

A

Fortunately, we now have enhanced service counts, which add a few nice features. The first is expiration, so those tokens that are stolen can only be used for so long. Also per pod information is attached to the service account so from the audit logs. We can see exactly which pod or token was compromised and, most importantly, in this scenario, we have something called audiences. So with audiences, when now Wallet sends that request to metrics pusher, it's actually going to send a special token.

A

That's intended only to talk to metrics, pusher and cryptographically signed, so that if Eve then uses that same token to try and talk to the API server, the API server will say no. This isn't intended for me.

B

So at this point we thought that Eve must have been pretty frustrated. She tried a lot of paths been pretty successful, but she still did not have her closet any of our Miao coins. So she took a step back and reconsidered her strategy. After a quick google search, she actually found some credentials admin credentials for our cluster on one of our engineers, github files repos. So thankfully, none of that happened and as far as I know, Miao coins are not a real thing.

B

Maybe they are, but we were able to present and talk about a number of recent advancements in container isolation. These are in various degrees of development. The things in black and green are things that you can use today. The things that are yellow are things that you can use today, but are moving fast and improving fast, and the things in red are stuff that will hopefully be usable in the near future.

A

So, just to recap, at the beginning we talked about the multidimensionality of isolation, and here we saw a lot of examples about how our attacker was able to attack from all different angles and slowly kind of weave her way through the system, and so remember that it's really important to kind of consider. Every angle when trying to get achieved, secure isolation and also we saw some examples of directionality where she was able to once.

A

She had escalated to the node access, the credentials of other services running on that node and there's one takeaway that we want to make sure we get across today, which is we talked about all of these really cool technologies that prevent all sorts of advanced attacks. But none of that matters at all. If you don't get your basics right and so kind of just to highlight three important ones: use best practices for credential management, don't post your admin password to github, protect your network and services I.

A

This is something that I see for customers, forgetting occasionally you can isolate your container. All you want, but if you have wide open services on the network, that doesn't matter and finally, of course, keep everything up to date, patch patch and patch, some more so that, thank you.

A

A

Think we have a little time for questions if anyone has anything.

A

The question is, can we talk about overheads for sandboxing and the optimizations that have gone into it so.

B

Did you mean one particular kind in Jimmy, a specific kind of sandbox thing or.

B

4G Visor sure so I mean it. The different types of sandboxing are going to impose different kinds of overheads. So it's really hard to say one number for any given case. For example, the I'm gonna get to a specific answer to your question, but I'm just gonna frame it first, the the machine virtualization case where I mentioned it runs really efficiently on bare metal right. But if you're running in a already virtualized environment you want to carve things up all of a sudden. You don't have bare metal anymore.

B

You're gonna incur massive overhead from nasty virtualization. So that's a particular cost that you'll pay in that environment for gee visor there are. There are really two specific kinds of overhead that you'll pay. One is the for trapping for intercepting the system calls from the application. It really depends on the underlying platform.

B

So similarly, if you have bare metal- and you can use like a KVM based platform, you're doing Hardware interception- that is not that high cost, but if you're running virtualized it's a much higher cost, because you're using P trace or some other interception mechanism, but it again it depends on the exact environmental constraints because it could still be faster than nested virtualization. In many cases, the other kind of overhead for 4G visor that I mentioned there is it's a separate kernel, implementation and so I said we haven't optimized for a lot of use cases.

B

Yet the static file serving and sort of big scalability use cases are two that I mentioned and that's just still really about the the implementation itself. The implementation of kernel and the these sort of things that you've looked at right. We've looked at serve web application servers principally where you're not trying to get as much static content out as you possibly can, and they're. Typically pretty small right. We're focused on the scale to zero problem versus the let's run, one enormous machine- and this is a bunch of scalability work that we have to do.

B

Does that answer your question at all? Maybe not.

A

So the question is docker. The minimum kernel version for docker is much older than what we consider a modern kernel. You can go pretty far back and still run docker and what are the chances of advancing that minimum requirement?

A

I-I-I, don't work on docker directly, so I can't speak exactly for them, but I'm guessing that's more of a technical requirement. So if you run on that kernel, it will work and it sounds like what you're asking about is more of a kind of a compliance thing to say for to comply with whatever standard. We want a minimum kernel version that sounds reasonable, but I don't know off the top of my head. What's what's happening in that area,.

A

Yeah, so the question is about isolation of storage, so we didn't touch on that too much today, because this is really an active research area and we don't know exactly what the plans are here, but you might have remote. You may remember the sibling, vulnerability that we talked about what was it last year at the end of last year, I think or I guess the beginning of this year we came out with the patch, for it there's actually a session.

A

I can't remember exactly when it is, but talking about that vulnerability and the fixes to it if you're interested, just search for assembling comma schedule this morning. Oh it was this morning. Oh sorry, but so that's so we patched that vulnerability by checking very manually for symlinks when volumes are mounted, but that's a very kind of specific fix for that one issue and even if you're using sandboxes the way we do it today is we we basically mount that volume on the host and then sort of pass it through to the sandbox through.

A

Basically, you know the sandbox equivalent of a bind mount, and this means that it's actually exposed to the same types of vulnerabilities, and we really only have one security boundary there, but we want to for sandboxes so we're starting to think about ways to maybe handle some of that mounting directly within the sandbox context. So it doesn't actually have to be mounted to the host. But as I as I mentioned, these are kind of like open, an open research area that we'll be looking into more in 2019.

B

So I mean every distribution, carries security patches right and you want to just make sure that you're using a distribution that you trust to patch appropriately and up to date on all of those trees right, Lee I, don't think. There's a general answer to you know: version X is gonna fix all your problems, but it's a very, very, very important best practice. That's the point. I wanted to hammer home on that. One.

A

Yeah, so the question is: can we give access to multiple service accounts from one container?

A

The technical answer is yes, if you are willing to manage those service accounts yourself, a basic way that you could do, that is to give permission to the main service account that you're attaching to the pod to act as or impersonate the other accounts you can also manually mount. Those service account tokens into the pod, assuming it's running in the same namespace, but also I would kind of caveat that, with once we have audiences for service accounts.

A

Hopefully the need for that for having multiple accounts, isn't as high I think we're just about out of time, but I'll stick around outside. If you have any more questions.