Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Advanced Persistence Threats: The Future of Kubernetes Attacks - Ian Coldwater & Brad Geesaman

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Advanced Persistence Threats: The Future of Kubernetes Attacks - Ian Coldwater, Heroku & Brad Geesaman, Brad Geesaman Consulting

What would happen if your cluster was successfully compromised by an attacker who understands Kubernetes at a deep level? How could they attempt to avoid detection, cover their tracks, achieve full cluster access, obtain persistence, steal credentials, and launch additional attacks in your environment? As Kubernetes grows in popularity, the sophistication of attackers will improve, and security by obscurity will no longer be sufficient. Cluster operators need to be aware of what a skilled and knowledgeable attacker can be capable of. Let’s explore the dark corners of clusters and shine a light on how features such as ephemeral containers and validating webhooks can be used to maliciously mutate pods, exfiltrate data, deploy “shadow” control planes, and more. The audience will learn how to detect these advanced approaches and how to prevent these attacks using practical, proven methods.

https://sched.co/ZesN

A

Hi, I'm ian coldwater.

B

And I'm brad giesemann.

A

And we attack kubernetes we're here to talk about the evolution of the kubernetes project and its attack surface and to look into the future of what attacks on kubernetes can look like.

B

And to make things fun, we'd like to show you a few.

A

Kubernetes turned six years old this year and both of us have been working with kubernetes for a few years. Now, both of us have watched the project, then its rapid rate of adoption grows and change over time, and we, as attackers have evolved along with it when brad and I first started learning kubernetes, it was easier to be able to understand the whole stack.

A

This diagram here was the first kubernetes architecture diagram it's from 2014. Before the project went public even back, then it wasn't entirely accurate. The person who made this diagram now describes the authorization on it as aspirational, because at the time there wasn't any authorization on kubernetes at all.

A

Kubernetes has grown a lot since then, and its security posture has improved a lot. It's still not secure by default, but it's better than it was.

B

We learned kubernetes back when things were simpler and possible to learn enough of everything, but newcomers now have a much steeper hill to climb.

A

Yeah, take a look at this image: it's not just kubernetes anymore, that's growing and evolving. The entire ecosystem around it is growing and evolving confused. It's not just you.

B

A few years ago, kubernetes was the platform. Now it's a platform for building platforms.

A

This cursed image here is the cloud native computing foundation landscape since you're a cloud native con you've probably seen it.

A

This image is so overwhelming that even this screenshot from cncf's webpage asks on the top if you're overwhelmed it's a lot for everyone to keep track of even for people who often work with and pay attention to this stuff.

B

Additional features plugins and configuration on top of kubernetes add whole new layers of complexity.

B

All of these have to be kept up with and secured independently and together, and they don't always play nice with each other, so as complexity grows, defenders and operators have a much harder time understanding the aspects of each component and their interactions in the environment, with more moving parts, there's more to configure and with more complexity per part, they're more likely to be misconfigured.

B

Historically, new versions of kubernetes have been released about every 90 days, and the support for security fixes is only applicable to the most recent three releases.

B

That means given a given version is only supported for approximately nine months due to that rapid cadence managed kubernetes providers, typically only support versions that are one or more minor releases behind the latest and organizations managing their own kubernetes clusters. Often do this too, as it always takes some time to test and validate new releases before they make it to production.

B

For example, the latest generally available kubernetes release is 1.19, but many are running 1.15 or 1.16 in production, simply because those are the latest versions available to them, but that can actually work to our benefit. As defenders, we can use this time to look at new features now before they're actually used in a few months, but as attackers that just means there's plenty of old clusters to break.

A

So everything is growing and evolving really rapidly and changing really rapidly, so as kubernetes evolves, attackers have to evolve with it, as the kubernetes project has implemented more security over time, attackers have had to learn more about kubernetes to be effective than they used to be. It used to be really easy to attack kubernetes.

A

You could just do things like execute commands as root with an unauthenticated curl call not actually kidding. Now you need a deeper understanding of the moving parts on the high level and often the linux primitives that underlie the low levels. So all of us have to level up attackers, defenders, everybody and in that spirit, let's look ahead. A bit we'd like to help you level up.

A

What does the landscape look like coming down the pike? What might attackers be able to do moving forward we'd like to demonstrate some novel attacks to show you some examples of what attacks can look like in the future? What could a sophisticated attacker who knows kubernetes well be capable of we're pretty excited about these attacks. We know you will be too oh yeah before we get into the demos. Let's get into an attacker mindset: what might an attacker bent on foul play generally want to do in kubernetes.

B

Well, here's what we might want to do. First, we want to become cluster admin.

A

This could happen by either compromising the cluster and becoming cluster admin, or maybe you are a malicious, administrator or insider threat. Every attack that we're going to be demonstrating in this talk assumes that you've done this already.

B

Then you probably want to get all the secrets. You don't want to get blocked for sure and establish persistence without getting caught.

A

It's important to remember that different attackers have different goals and motivations cluster admin isn't always the end goal. Not every attacker comes in from the outside, and sometimes the goal can be outside of the cluster itself.

A

So on that, let's say we are maybe a malicious administrator or an attacker who has access to a cluster as cluster admin. We have access to all of the data and secrets in this cluster already and that's pretty cool, but we only have those for right now and they might change over time. So it would be nice to see it in real time and get the newest credentials if they get rolled or something like that as they get updated.

A

There is a component of the api server that we can misuse for this purpose. It's a form of admission control called a validating web.

A

A validating web hook is a native configuration option that lets a cluster administrator check with an external web service for whether or not to allow a resource to get created. If you've heard of projects like open policy, agent, gatekeeper or k-rail, all of these projects use validating web hooks. As a mechanism for enforcing security policy, here's how it works when a user submits a request to do something like create a new pod. After going through authentication and authorization, it gets to the validating admission controllers.

A

If a validating web hook is configured, a full copy of the pod request is sent the admission controller, then parses the request and uses custom logic to give a yes or no response. If the logic allows it by sending a yes response, the pod can be saved in etsy.

A

If not, then not you can have multiple validating web hooks. If you want to, and you can pick which resources you want to trigger on and the best part is that request doesn't have to get sent or run inside the cluster. It can go to any arbitrary url on a system somewhere else.

B

As an attacker or a malicious admin, you might want a web hook to be called only when secrets are created or updated. You could install a web hook like this filter, only on secrets and then send it to an attacking system that just logs what it receives so revisiting this data flow, but with our malicious web hook in line sometime passes and a user might create a new secret via the api server and we'll assume it gets accepted by the first validating web hook in steps two and three in step.

B

Four, the entire secret gets sent to the attacker controlled application via web request. We can configure our malicious application to simply log the secret and always send an approval back to the response it gets saved into etcd, then as normal, but being good evil attackers. We don't want them to notice if our custom web hook service is ever down. So we can tell the api server to wait up to one second for our web service to respond yes and then just fail open as if we send a yes anyway.

B

If we don't send a response, so maybe we add a tiny bit of delay, but not enough for most folks to even notice. Let's demonstrate.

B

So we have a demo cluster and we have cluster admin access, so we can run kube control, get nodes and see that we have a single control, plane, worker, sorry, a single control, plane, node and a single kind worker.

A

We're using kind here, if that wasn't obvious,.

B

We can run a coupe control, get pods in all name spaces and get secrets to give you an idea that this is basically an empty cluster.

B

Okay, so validating web hooks require tls, so we can generate a self-signed certificate.

B

We have a script to do just that and what that's done is created a self-sertin and what that's done is created a certificate signing request and had it automatically approved and that was stored in the secret named validator.

B

So now that that was just generated and stored in the secret, we can fetch it out and save it locally to a file called cert.pem.

B

So next we're going to install the malicious validating web hook, application named validator, which is going to capture all the secrets sent to it by the api server.

B

So we've deployed a config map, the deployment itself and a service called validator that points to it.

B

So, let's look at the validating web hook configuration that we're about to apply.

B

And, as you can see, it's a validating web hook configuration resource and we're pointing it at the validator service in the default namespace and the failure policy is to ignore, which is the fail open setting. We only wait one second in case our application is ever down and we continue on anyway, we've configured the rules to trigger only on create, update or delete of secrets, resources.

B

So, let's install the validating webhook configuration into the api server, okay, so and now, when a regular user creates a new secret such as this one.

B

The damage is already done. We can go. Look at the validating webhook logs with capturing the secret in the clear from our malicious webhook application and.

A

B

See we have honk and that's the base 64 encoded version of that secret awesome awesome. So now we have a permanent method for ensuring. We are always aware of the latest credentials and secrets stored in this cluster in real time.

A

That's pretty sweet and that was pretty cool, but it's not very stealthy. The api server logs everything we do and an observant admin could maybe notice it. The api server also has a lot of authentication authorization options that an administrator could use to. Stop us that's kind of a bummer.

A

We could maybe get around it by doing various things that are kind of convoluted, but I feel like. Maybe we could make it easier for ourselves. Maybe if we bypass the api server, we don't have to conform to its security policy, and maybe we won't be logged in some clusters. You can schedule pods on the same node as where the api server runs.

B

So what if we scheduled a full copy of the real api server on the same control, plane node and used the same keys and network paths to talk to etcd that the real control plane has, but instead of logging activities or blocking anything we just have it set to allow full access as an attacker. That's pretty handy right keeps the logs of the api server clean the real one and maintains a persistent access channel to the underlying state of the cluster, where all the loot is kept.

A

With our own access to xcd, we can control everything in the cluster and, depending on what kind of metadata is in there, we could potentially control access to cloud accounts outside of the cluster as well.

B

Let's see what that would look like yeah.

B

So we've already made an exact copy of the current api server configuration and we've made a few key changes. So, let's take a look, we'll call your attention to us. Setting anonymous auth equals true and authorization mode to always allow this effectively disables authentication and authorization in the api server, and we set the insecure port, meaning the plain http service listing on port 443.

B

The last thing I want to call attention to is because we're running on the same control, plane node. We have access to the certificates that are used to communicate with ftb.

A

We're using tcp 443, because it's a common traffic port that is pretty unlikely to be blocked by firewall rules. Also traffic on 443 is not very likely to be noticed.

B

Indeed, so, let's go ahead and install that all right, so we've installed that on the control, plane, node right now. So let's test our use of this. What we can do is we can run we'll call it an attack pod inside the cluster, and this is a tool that was built by a friend of ours, rory mccune.

A

B

So from this newly created attack pod, we can simply just use curl to hit our api server, not the real one directly and dump all the secrets, as you can see, we're hitting it on the api v1 secrets endpoint, which is saying all secrets all name spaces, and there we go everything coming from at cd directly.

B

Nice so now we can use our own api server to interact with sed directly whenever we want, without fear of being blocked or logged.

A

This diagram explains how what we just did worked and that's pretty cool, but it's not quite a persistent strategy. Let's broaden our horizons a bit. I think we could go bigger.

A

What if we deployed a c2, a c2 is a command and control infrastructure, red teamers or attackers might use it to exfiltrate data to an external place or maybe have a persistent channel that can move data back and forth.

A

You can also command and control the server that you're attacking from your c2 infrastructure and red teamers. Have you know given talks about how you can set up your command and control infrastructure on kubernetes, basically teaching each other? How to deploy things? That's pretty cool, but that's not what I mean here.

A

I feel like what, if we built a c2, not spinning it up on kubernetes what if we built a c2 out of kubernetes itself, we could call it see tubernettis what if the moment we got cluster admin, we could deploy a persistence mechanism really quickly with the least amount of logs tracking that activity and the greatest chance of going undetected for the longest period of time.

A

We could maybe do this by installing an agent on all cluster nodes that joined them to an external cluster that we control. They would still be a part of their own cluster and otherwise behave normally. So probably an admin wouldn't notice, but they'd also just happen to be a part of our cluster too.

A

That would mean that as attackers, we would be using kubernetes as our command and control infrastructure to control multiple compromised clusters in a single place, while staying lurking and undetected in the shadows. How could we do this? I have an idea.

A

We can build it using kubernetes itself. We could build it using k3s. What is k3s and why did we pick k3s to deploy our c2 with kcs is smaller and simpler than a regular kubernetes distribution. It's designed to be lightweight for resource constrained environments such as internet of things and k3s's requirements reflect that k3s has different network requirements than regular kubernetes, which are advantages for the purposes of rc2.

A

Most importantly, it only requires a single tls connection, outbound from knowns to the control plane, which is very likely to be available and also likely to blend in with valid traffic. So it might not be noticed. K3S has nearly all of the same moving parts as a full version of kubernetes on the right here in this slide is the simplified visual of the k3s control, plane, components in green and the k3s nodes in red that we'll overlay into the next slide on the left of this slide is a kubernetes architecture diagram in a normal cluster.

A

So how could we use k3s to build a c2.

B

I think we could install the control plane component on a vm in another cloud, say listening on tcp 443 again who blocks that anyway, but with cluster admin access to the target kubernetes cluster, we can deploy a workload that immediately escapes the container to the underlying host on every worker node from there we can install the k3s node in the background, with a configuration to auto, join our k3s cluster on start and then the container just exits a few seconds after establishing that persistence.

B

The end result would look kind of like this. A couple really cool things that happened with this approach. The k3s agents have full access to the real hosts file system, so we can grab anything we want directly from all the nodes. The best part is the original cluster continues to be fully operational. We didn't break anything that the admins might even notice the next time they run code control, get pods on their cluster. They won't see any k3s pods and they won't see any logs of us hitting their api server.

A

B

So, let's see that in action shall we.

B

So now on our attacker own k3s control play node running on another cloud. Virtual machine we've already set up the k3s control plane and, as you can see, we have just the control plane, node running, there's no worker nodes attached.

B

We also have recently compromised a gke cluster and it has three nodes. We have cluster admin access to this cluster.

B

Let's look closely at our specially crafted c2 daemon set.

B

So we're going to install in the cube system, namespace we're going to run in the host network namespace and it's just a busy box image and there's a lot going on here. But I'll summarize we're basically installing a separate docker network.

B

If we need to and then we're running directly via docker run in the background, the k3s agent and we're supplying the privileged flag and the auto join credentials so that it knows how to phone home to our k3s control plane automatically and we're mounting the root file system on the underlying node in slash root, fs and we're just throwing in some tolerations here. To make sure we get scheduled on any node. That is in a cluster.

B

So let's go ahead and deploy that c2 to our gke cluster.

B

And so when we could control get pods in the cube system. Namespace we can see our honk c2 daemon set is currently running. Okay, so back on k3s on our control plane, uh we can run kube control get nodes again, but this time the gke nodes are also part of our c2 control, plane.

B

So not playing favorites we've also compromised eks.

B

Let's get nodes on that cluster.

B

And there they are just like the gke cluster three nodes: let's deploy rc2 to eks.

B

In a similar process,.

B

Finally, last but not least, we don't want to leave anyone out, we've compromised an aks cluster and it too has three nodes. So, let's deploy rc2 to aks in the exact same way.

B

B

So we don't want to leave any tracks right. So now we can take the opportunity to clean up our daemon sets. We can do that with the kube control delete. So now that's removed from the gke cluster.

B

If we get pods in all name spaces inside the gke cluster, we can see that it's now gone.

B

So let's do that same process to eks cluster, we'll delete that daemon set.

B

And we'll do the same to aks fully cover our tracks, alright, so all of them are now deleted.

B

So back on k3s we can now run code control, get nodes and we can see all three nodes from all three clusters from all three clouds.

A

How's that for a multi-cloud strategy, I.

B

B

So- and you were talking about stealing all the secrets right well now that we have the ability to see the root file system on all the nodes, why don't we make a daemon set to run inside k3s?

B

That steals all the secrets from the nodes sounds.

A

B

A

B

So a simple busy box image doing a find to get all the mounted secrets attached to the kubelet and it's mounting the host file path.

B

And we can deploy that again, we're deploying this to the k3s control plane so that it will run on all the clusters.

B

And now we can review the loot.

B

Oh there, it is so all the certificates, all the keys. All the service count tokens not bad, not bad.

B

You also talked ian about expanding to the rest of the cloud infrastructure right, so why don't we touch the instance metadata credentials.

A

That sounds great, then we can compromise the cloud accounts, not just the clusters.

B

So we can w get on the magic ips for both aws and gcp and for azure. We can go, go ahead and grab that from the file system of the worker nodes again a very similar data set.

B

So we can deploy that data set again to the k3s cluster so that it runs on all the nodes across all the clusters.

A

Across all, the clouds.

B

Oh and there it is so, we've got gcp access tokens, we've got aws credentials and we've got azure principles and client secrets.

A

All your clouds are belong to us.

B

We now have our cluster of clusters.

A

That was pretty sweet yeah. As far as we know. No one has ever done anything like this before we think it's a good example of what cloud native attacks can look like in the future from sophisticated attackers who understand kubernetes at a deep level know how to use it and have the mindset of someone who can figure out how to misuse it.

A

This feature is exciting and if you're a defender, maybe a little bit scary and all of us have to prepare for it because as sure as kubernetes releases come about every 90 days, it's going to keep coming so, while we're looking into the future, let's take a look at what's coming and what's already here here are some upcoming and new kubernetes features to keep an eye on, but we're pretty excited about the possibilities of as attackers, and maybe that you might want to look out for in your environments, if you're a defender, these can do interesting things individually and together.

A

The first one we wanted to point out is a brand new one, as of kubernetes 1.19 coupe ctrl run now has a privileged flag. It's like docker's privileged flag as penetration testers who work in container environments. We know that setting your docker runs to privileged means that basically you're giving us a very short and easy day.

B

In previous versions, kubernetes components were largely configurable via the files on the node's file system or via cli switches. This provided a natural separation between configuration in kubernetes versus configuration of kubernetes components, but it can introduce friction when automating widespread configuration changes. So to help with this, a couple of new feature gates are available that can enable dynamic reconfigurations of kubernetes components via the kubernetes api.

B

So, for example, the dynamic audit logsync configuration can't modify, which api actions to audit log, but it does control where those audit logs should be sent. So we can use this dynamic configuration setting to send all kubernetes audit logs to an application that we control. Instead, we can then filter those logs and forward them on to the original destination, to hide just our malicious log entries and send the rest on and thus covering our tracks.

A

That's pretty sweet but wait. There's two things on this slide: what can we do with dynamic kubla configuration.

B

We can use our privileged access to kubernetes to reconfigure kubernetes itself and change its own security posture.

A

Wow kubernetes security posture really has improved a lot. Remember those attacks. When we first started, I do it was a simpler time. One of the ones I really liked was kublet exploit greets dakires kublat exploit was one of the first kubernetes hacks. It was involved in infamous security peaches. It could be easily used and was exploited in the wild.

A

Since the defaults have improved. You can't really use kublet exploit anymore and personally, I kind of miss its ease of use.

B

Well, actually, the kublet api exploit isn't really an exploit. It's just a configuration on the kubelet's api that let you run arbitrary commands inside any pods container on that node without authenticating.

A

Okay sure, but historically it really was exploited a lot.

B

True true kubla's default configuration has become more secure over time. Disallowing old attacks that used to work like this, but now that reconfiguring the kubelet is so easy via the api. Let's do that and bring the kubelet exploit back all right.

A

Let's see that.

B

Let's see that in action, so back on our demo cluster, we can run another attack pod we're going to use the same image as before.

B

And what we want to do is show you what it looks like to try the kubrick exploit and see it successfully blocked by the coolest configuration.

B

We can do this by executing into that attack pod and just using curl to hit the kubelet's api on port 102.50 against the running pod's endpoint. As you can see it's unauthorized.

B

So before we want to change the kublet's configuration, we probably want to get its current version and just tweak it a little bit. So let's do that. Let's pull the current kubelet's configuration out of the cluster and save it as a local file. So we can edit that we'll call it kubelet, config, z, kind worker, and now, let's modify that.

B

So what we want to do is go to the web hook and disable that and turn on anonymous authentication and then change the authorization mode from web hook to always allow.

B

Okay, we made that modification. Next, we want to create a config map. That's going to hold this kubelet's configuration, we'll call it my node config and save it in the kube system, name space before we deploy that. I just want to show you that we indeed have the web hook off. Anonymous. Auth is true, and the authorization mode is always allow.

B

So let's apply the updated, kubelet configuration.

B

We do this by patching the node object. We say change it to use the config map name, my node config in the cube system- namespace, okay, so that's set. Now. Let's look at the new node configuration that the kubelet has taken on. As you can see, we've got the config source. Is the config map that we just described.

B

So, back on our same attack pod, we can now use curl to hit the local nodes kublet api directly again, using that same mechanism.

B

And this time, when we hit the running pods api, we're going to pipe it into jq to make it a little bit easier to see and there we go. We can see all the running containers pods and what name spaces they're in on the cluster.

B

And what that lets us do.

B

With those bits of information it lets us run a curl command to list the root file system. As an example command, you can run anything against the kubelet api and pointing that to the coupe proxy pod in the kube system. Namespace.

A

Coproxy is a useful one to pick because it always runs privileged and it's available to every node so doing that we're pretty close to root access and as attackers, that's pretty cool.

A

So now with comfort circle. What's old is new again, the increasing complexity and security of kubernetes and its ecosystem leads to demands from developers and administrators for more simplicity and more ease of use. This can lead to opening up new security holes or perhaps reopening old ones as attackers. We're really excited about the possibilities that we see in in the future, and we would like to encourage you, people who are maybe developers or operators or defenders, to take a look at these upcoming features and the systems you build through our eyes.

A

You know how you would use it, but how might an attacker misuse it being able to look at your systems and what you're, building with the perspective of an attacker, can help you understand and building what you're building with this perspective in mind, can help you protect against it and make your systems more secure overall, because, as attackers, we promise we're going to keep finding ways to misuse things. We're pretty excited about this.

A

We also really like challenges so we'd like to encourage you to secure your systems against folks like us, so that we have some challenges.

B

Here are some resources and further reading. If you want to learn more about hacking and hardening kubernetes for yourself and together, you- and I would like to say.

A

Honk the planet thank.

B

A

Much for taking the time with us, we really appreciate it, have a great coupon.