Cloud Native Computing Foundation KCD Italy 2021, 20 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Stranger Danger - hack my misconfigured Kubernetes | Matt Jarvis

Description

KCD Italy 2021* - The video of the talks are now available!

Kubernetes is a powerful and highly configurable tool, but by default it's extremely insecure. In this live hacking session, I'll show how an attacker can expand the blast radius of an application exploit through to control of the entire cluster whilst demonstrating the misconfigurations which allow this to occur and showing how you can avoid these scenarios in the real world.

🤓 SPEAKER: Matt Jarvis, Director of Developer Relation @ Snyk
______________________
* KCD Italy 2021 is the first edition of a community-organized non-profit event that will take place on November 17th and 18th and will gather adopters and technologists from open source and cloud native communities for education, collaboration, and networking.

📯 Follow us to be notified about the upcoming MeetUps: https://community.cncf.io/kubernetes-cloud-native-milano/

A

A

A

A

B

A

A

Thank you for being with us good.

C

A

I'll read you a video bio that is really really incredible and impressive, so I I take the time to read it. So much spent more than 15 years building products and services around open source software on everything from embedded devices- and I was just saying the same- I love the devices to.

D

A

Visibility systems, most recently mata, has been focused on the.

D

A

Cloud infrastructure, space and innovative factors for connected applications. Math is a regular speaker and conference season across the board, including open infrastructure summit before them and openings all things open. A past winner of the open stack, outstanding community contributor award and last, but not.

D

The least in 2031 was.

A

Named one of the top 100 influencers in open technologies in the uk, so since matt does only these things but is also a board director of the uk and the founder of cloud native manchester um okay. So it's really really short on the bio and.

C

I've been around for a while.

A

Yeah yeah just few years.

C

A

So mata, thank you so much for being here with us and next stage.

C

Thank you. Let's uh move this over here, okay, uh good morning, folks uh right, let's get this show on the road so uh yeah. That was a great introduction. Thank you very much. I'm director of devrel at sneak um and and the shortened version of my bio is I've been uh building stuff with open source um for around more than 20 years. Really, um if you want to get in touch with me, feel free to reach out on on twitter or any of the usual places.

C

So what we're going to be talking about today is about how most um exploits in the wild in terms of of people's um infrastructure. Getting uh getting penetrated by hackers is a combination of application, vulnerability together with infrastructure, misconfiguration and the misconfiguration allows the attacker to expand the blast radius and we're going to look at a kind of uh example of this in uh using kubernetes.

C

um So our our starting point for this, for this um kind of uh thought experiment, is that we found a vulnerable web application running on the internet, um and we know that this web application has a remote command execution vulnerability and for the purposes of this demo, um this is a mock-up which will just allow me to run um commands using uh specially crafted urls.

C

But these do exist in the wild. um I was looking at one uh just last week um involving tomcat, so they are um relatively common so before we go and actually start poking around with our application, let's also introduce our timeline of doom, and this is going to uh show us how we over time can expand the scope of our attack and our timeline of doom currently just consists of our application vulnerability, which is going to allow us um to uh run remote commands.

C

So what? What can we do from here? Well, um when we look at web applications in general, you know what we uh often see these days is that things are running. Containers and environment. Variables tend to be the way that we configure containers. So, given that we can run uh commands, let's see if we can run um our uh environment uh the environment command and see what environment variables we have.

C

So, um as we can see, uh we can make an assumption here that we are running in kubernetes has given us a lot of information about um referring to a kubernetes cluster, and so we now know that our application is running in kubernetes, and we also know that, although it's being exposed to us on port 80, um it's it's running on port 5000 inside the cluster, we can see the uh the service, ports and various references to to services here.

C

um So uh we can update our um our graphic now to show that we, uh we know we're in a kubernetes cluster. We know we've got the internal ip of the api server, that was in the environment variables, and we know that we're being um the pod that we, this application is running in, is being exposed on port 5000 internally and it's got a service running in front of it, which is then exposing on port 80 by some unknown mechanism, probably in ingress. But we don't know that at this point.

C

So, uh let's also see what we can find out about the uh network at this point, so if we run the ip command, um we can see the uh ip of the pod that this is running in and this is going to be useful to us later. So uh we're going to take a note of that. That's 1.7 and uh we'll come back to that in a little bit.

C

So we can now again update our our graphic here. We know the ip address of the pod, so let's see what else we can do so by default. um Every pod in a kubernetes cluster um has a service token auto mounted in it, and this service token is associated with service account that was used to launch uh the pod to create the pod, and you can control whether this uh token is automated or not, by uh either on the service account or on the pod level.

C

By setting the uh automatic service attack account token to um false. But let's see whether in our cluster, uh this token is mounted. So we know where the token's going to be within our file system. If it's there and we can see, we've got a token.

C

So um let's see what we can get from the internal iep address of the api server. Using this token, we saw in the environment variables. We know where the internal ip address of the api server is so, let's try and craft a uh a command here um to uh let's just have a look see what we're going to do to start with.

C

So our first, our first um uh further expansion of our exploit here is that we found the um the pod token was available to us inside the pod and now here's what we're going to do. With that token, we're going to try and connect to the internal ip address of the api server.

C

So, let's craft a a command, and we can see we're going to see if we've got curl inside this uh container, which we have because we get a result here and we're going to because we've got the token mounted. We know we're going to have a ca cert there we're going to use the token uh to query the internal address of the api server and we're going to try and get information out of this endpoints api and we've got a return value here. We can see um in a real cluster.

C

This would give us the external ip address of the api server. um This.

D

C

uh Cluster is running on my laptop. I know where the api server is, and it's not actually going to give us that, but in a in a real world default cluster. This would give you uh the external api address of the.

D

C

The api server, so let's update our timeline of doom again, so our pod token, in our case, allowed access to the endpoints api. So it's allowed us to gather more information about how the cluster is configured so um now we have that uh that token, um let's uh we're gonna configure our kubecurl locally to access the external iop address of the api server.

C

D

C

The external ip address is and we're going to configure our um our coupe, so I've got a little helper script here, which is going to help me. Do that. Let me just grab my uh my token again.

C

C

So that's our token and in my case the api server is just going to be on localhost, and this is just going to create me a uh a coupe, config and and configure it for my um for my cue cuttle. So uh now I've got my token configured here. The first thing I'm going to do is I'm going to just see whether I can do cooper to get pods.

C

Hang on a sec, uh let's just do uh make sure my config is actually right. Okay, there we.

D

C

So my cube config is now set up correctly. um So our token from our pod here I'm trying to get pods in the default name space, and we can see that our our um token doesn't let me view pods in the name in the default namespace, but it has given me some more interesting information.

C

It's it's given me the uh name of the service account that uh this token is associated with, and the token is the service account is name space and its name spaced in this secure namespace.

C

So let's see uh whether we can view anything in the secure namespace and we can see that we can see pods in that secure namespace.

C

So, um let's see what other permissions we have um up there, uh google, we can use the cube kettle. Auth can I command and if you do minus minus list, minus minus token equals and our token oh uh where's, my command going wrong.

C

Oh, can I it's nice list is token equals token, but this is in the default namespace and we can see. We don't have very many permissions here. We had permission to view that endpoints api, but we don't really have any more permissions than that. So let's try that command again and look at the secure namespace, and we can see here that we've got a bunch of permissions in the secure namespace um and uh we can also there's another way that we can view our permissions here.

C

So we can use the access matrix plugin um for kubecuttle, which gives us a slightly more um more uh easy to read view. If I run that in the default namespace we can see, we don't have very many permissions at all, but if I run it in the secure namespace um we can see, we've got lots of permissions now.

C

This is a fairly common pattern where people give service accounts um permissions in their own namespace people, assuming that um a namespace is like a security boundary and it really isn't and we'll see how we can uh exploit that um uh throughout this uh this talk, but um if we go now go back to our our slides, we can add a bunch of stuff in here. So we know we've got a default namespace. We know we've got a secure namespace.

C

We know that pod is running in this secure name space, and um this is this is the problem here is that um the service account has been given too many permissions, even in its own namespace. um Someone has made the uh the mistake here of treating the name space like a security boundary and assuming that, uh if we restrict that that uh a service.

D

C

To its own namespace, everything will be fine, so we can update our timeline of doom here. We've got an uh vulnerable role that gives us too many permissions um in the uh in the particular name space.

C

So um now we know uh that we have. We only have a single pod running in our secure namespace, so we can make an assumption that that is the pod running the vulnerable web application. So let's just grab that part and let's see if we can get a shell on it, so um I have got my couple is just uh alias to k for um brevity. So if you see me typing k, uh that's why so, let's uh do k z.

C

I t our pod um in the secure namespace and we're going to try and run a bash shell and let's see what happens.

C

Okay, so we've got a shell there uh and we seem to be uh this web admin users. So uh let's just double check that yeah. So we are this uh web admin user. That's quite a good security posture, we're not running as as root here um the application is running as it as a as a normal user. So, let's try and see if we can elevate our privileges, we're going to run 202 and sudo doesn't.

D

Even exist in the container, so, okay, you.

C

Would think that's a fairly good um security posture here, but let's see whether we can create a file and we can actually create files here- and um this is another potential vulnerability here, because if I can create files, then I can download software. um I know I already have curl, because I found that it was there when I um tested the web application from the outside.

C

I can change configuration potentially, um and this is caused by not setting um read-only root file system equals uh true when, in our security context, when we launch our pod so- um and we can add, this is another potential vulnerability within our uh our container here and, as I said, it allows.

D

An attacker to potentially modify.

C

The container by downloading software or changing configuration.

C

So let's come out of our container for the minute and we're going to try and do something else. So um now that we know we've got permissions in this secure namespace. um Let's try and spawn uh a root pod, perhaps with a with the root user. So um I have some some ammo here prepared. So let's uh have a look. What we've got here so my root pod yaml is going to just try and launch an alpine image out. The box alpine will run as root. So let's let's try and run that.

C

So if we do kubeco apply uh if uh demo yells uh rootpod and we want to do that in the secure namespace and it looks like something's happening there. So let's have a look, what's actually happening in that namespace, so we can see that we've got this, create container config error. So um what.

D

Does that mean well if.

C

We do a k describe pod uh root pod in the secure namespace, we'll get some more information about what that error is, and we can see what the problem is down here, that the kubelet has returned. That container uh has run us. Non-Root and image will run as root. So something has stopped us from.

D

C

Route um pod there, um it's likely something like a pod security policy, but we don't know at this point. So, um let's just uh clear that and what we're gonna try and do now is run a um we'll run, a privileged pod, but we'll run without root, so non-root um privileged.

C

So this is going to run uh privileged as we can see, but it's not going to run as the root user is going to run as a normal user and uh since we're privileged we're going to try and do a host mount um and with the host path here and we're going to try and remount the uh the the root file system of the node that we're running on.

C

And so let's try and apply uh that and see what happens.

C

uh Non-Root uh privileged in the secure namespace- and this has given us a more interesting error message here- we know that there is a pod security policy in place on this particular name space and the pod security policy is set, quite rightly, to not allow us to run um privileged containers.

C

So um that's good. uh Let's go and update our the information that we know here. So we know we've got a pod security policy. It's definitely in place on this secure name space. It may be in place um other places, um and so does that stop us um from uh extending our exploit.

C

Well, let's have a try. So um let's try something else here, so I'm gonna try and run a non-root non-privileged pod uh and let's just have a look at the yaml, we're going to run for that elmo yellows non-root non-privileged.

C

So this is just going to run an image of mine. It's not going to run this route, it's not going to run privileged! So let's try and run that one so k. Prime minus f going animals uh non-root non-privileged in the secure namespace, and this one is now um has now run and this one uh let's just confirm. That is the case.

C

So we can see here. This is the container the pod that we've just launched my sneaky pod. So this is up and running. So um this tells me something else here that this cluster does not restrict me from being able to launch arbitrary images from external repositories and um clearly uh we'll see in a minute why that is a good idea to not let people run random images in your uh cluster, and this is another vulnerability.

C

So I know what's running in this in this pod because I built the image. So what I'm going to do is port forward to this pod, my sneaky pod uh on port 8080 in the secure namespace.

C

Oh, I just felt that wrong uh sneaky there we go and now port forwarding to my new pod and um I'm going to connect to it um so I've just given myself another route into this cluster, because I've now got a uh a pod running a remote terminal. So um in my pod, let's see who I am, we know we launched as non-root.

C

So let's see here what happens when we do two go to, and I've actually managed to elevate my privileges here to root, um and this is um because the pod security policy does not include disallow, allow privilege escalation equals false.

C

So it's allowing me to escalate my privileges in this case using the pseudo command, but I could have a binary that has that um flag set already that allows it to run as root, and you may think uh that this isn't a problem uh because we uh have other controls in place, but we'll see in a minute why that is the case. Let's just um update our our uh problems here. So um this is the setting allow privilege escalation. um False is not set, and I've seen lots of things.

C

uh People writing about secure uh security context, settings uh on the internet and saying well. If I've got, I'm can't run as root and can't run privileged. Then uh this isn't a problem because, ultimately, even if you did escalate to root you're still only getting uh the uh a similar set of of um of kernel controls as you would have, if you're the normal user, but it does matter and I'll show you um I'll show you why so adding updating our timeline of doom. Here, our pod security policy did not disallow privileged escalation.

C

So if we go back to my um my uh uh machine again, my pod and I'm going to come out of root for a second and what I'm going to do um is I'm going to.

C

See what else I can do here? Let's just run, I have config so that we've got our um ip address. 10.244.1.11.

C

And and what I'm gonna try and do is map uh the uh network using this pod. So I'm going to run nmap and I'm going to. uh I know that I had a um a vulnerable pod uh running on port 5000, so I'm going to look from for uh port 5000 within the network and I'm going to do 11, 24. and now.

D

If I run this as a normal user.

C

I'm going to get an error here because I don't have the kernel privileges uh to actually run the nmap command. But if I'm able to um raise my privileges here- and I run that command again- I'm now able to scan the network, and um so I've actually been able to reach out beyond the boundaries of the uh secure namespace, and this is telling me that I have no network controls in place here that are stopping me from further exploring the network, and here we can see that is our original vulnerable pod.

C

We noted the ip address earlier 10.244.1.7 with our vulnerable application running on port 5000.

C

But what I found here is another application, 10.244.1.8 and that's also running in port 5000, and that's very interesting to me, because if I had one vulnerable application running on port 5000, then I may well have another vulnerable application uh running on that port as well. So um let's uh update our um diagram here of what we now know. So we can see I've. I launched my own pod in the secure namespace I connected to it on port 8080. You give myself a remote tty and I've discovered this additional pod.

C

I have no idea which namespace it's in, but it's also running a service on port 5000, so uh back to our um our um custom pod again and what I'm gonna do now is. I am going to construct a tunnel uh from my pod here to this new pod that we've just discovered so I'm gonna run socat and I'm going to uh go uh to 10.244.1.8 and I'm going to that's going to be on port 5000 and I'm going to represent the other end of this tunnel on this pod at port 5001..

C

So if I create that tunnel- and now I go back to my cube kettle and now I go and change my port forward to port 5001, which will then connect via that channel to this new pod that we found so there's my port forward uh getting started. That's now started so, let's see what is actually on uh that um pod. So if I now create a new and lo and behold, we have another instance of our vulnerable web application running on a different pod. So um let's get the token for this one.

C

And let's switch our local cooper to use this new token that we've just discovered. So if I uh just adjust my cube config, let's just point this one out.

C

And let's do our new.

D

C

And let's see whether we can do cucuttle get pods with this new token, and so our new token has allowed me to uh run the getpods command in the default namespace.

C

So, uh let's see what we can do with our token again, let's use the kubeco auth. uh Can I minus minus list? Minus minus token equals our new token in the default namespace, and we can see that this token has permissions in the default namespace.

C

So um if we now have a look what's running in the default namespace, let's see if we can get a shell onto this uh particular pod. So if we do uh okay, exactly so t on that one and we're gonna run bash.

C

D

Now got a shell.

C

On this new pod that we've just discovered and um let's uh see what we can do here,.

C

D

C

Let's not go and answer this one. Actually, let's see what what we can do in the default namespace. So um let's update our uh diagram just so what we know. So we had no network controls in place which allowed me to escape my um escape the name space and discover this pod running in a different name space, and we created a tunnel from our sneaky pod insecure to this new pod, and we now know that that pod exists in the default namespace.

C

So now that I've got a token with privilege with privileges in the default namespace. uh Let's see whether I can now uh run a uh privileged pod here, so I'm gonna run the non-root uh privilege yaml that we looked at earlier and this token actually allows us to run that in the default namespace.

C

So we should see that that uh that pod is now up and running. So, let's connect to this pod.

C

Actually, non-root proof and let's run bimbash again and so we've now uh inside our our um product is running privileged in the default namespace.

C

So if we uh go to root- um and we uh look at the process table here- even though I've escalated my privileges to root, I'm still only seeing the processes that are within that container, but because our um privileged uh container had a host path map that I mounted the um the uh root directory of the node inside the container.

C

What I can now do is change uh my route to the churro folder, where I mounted the um host file system and now, if I run the bs command again, I'm actually seeing the process table of the node here and we can see all of the container d processes.

C

We would see uh any other um any other uh pods that were running on this node. I would now have access to all of those container d um instances.

C

So uh let's update our slides again here, so I was able to launch a privilege pardon using this new token, and I was able to mount uh the host file system so.

A

C

Next uh level of our timeline of doom is that we had um no restrictions in the default namespace. So now we've got um our host file system. Let's see what else we can do so we're going to uh look for the config for the um kubelet, which we know is running on this node.

C

And I have cube kettle on this uh on this node, so we're going to do couple get pods using the kubelet um configuration.

C

We can now uh see the pods that are running in the cube system and in the coop system namespace, which is where all of the control plane runs, and I can also get the name of the nodes. So um this is going to be important to us, because I can now tell by uh doing something like this by describing the xcd cluster um pods, I can see which node this is actually running on and that's going to be important to us um in a second.

C

uh We can see that the node is here kind control plane, um but the other important thing is: let's try and run a actually run a pod using the kubecuttle um uh token from the node. um I'm just gonna try and start busy, but uh busy box and pod there. So I can't actually create pods directly using the cube crystal token like this um because of the permissions that the kubelet has, but it does say that I can create mirror pods, and this is um one other attack vector that I could use.

C

I could put yaml into etcetera kubernetes, manifest on the node and those are gonna run automatically and uh potentially create shadow pods inside the cube system. Namespace. um At this point I probably owned the cluster anyway, I have the kubert token. I know where everything is I'm in the coop system namespace, but I'm going to uh to do uh one more thing here to show um how we could potentially another route that we could potentially use. So I'm gonna. Now I know where that fcd pod cluster is running.

C

I'm gonna come right out of here and I am going to.

C

D

Going to run an xcd.

C

Client pod and I'm going to spawn it on a very specific node, which is the node that I know where the xcd cluster is running, and I know that I'm able to create host path mounts and also to uh to mount those inside my pods. So I'm going to mount or I'm going to uh set all of my xtd configuration to the xcd configuration that exists on the node. So let's uh run that.

D

C

Okay, apply myself demo the animals suv client.

C

That should be starting, that's running now.

C

So, let's see whether xcd client is configured uh correctly, so I'm just going to exec um xtd cuttle on the uh on that pod, and we can see that it is running correctly we're getting information back from from ftd there and that cd contains a lot of interesting information about the cluster, not least of which is secrets. So, let's see whether there are any secrets in this fcd cluster and we can see- we've got a whole bunch of secrets inside fcd and this.

D

C

That we're particularly interested in this cluster roll aggregation control, that's open, because I know that this cluster roll aggregation controller token has um cluster admin rights. So, let's see if we can get that, uh get that token.

C

So we're going to use ftd to get it and we need to put this 2jm4.

C

C

And there we go, we've got the uh the token for the cholesterol aggregation controller there. So, let's just grab that token. We got that straight out of etcd and the k auth. Can I minus minus list minus minus token equals.

C

And what we can see here is that on cluster roles, we have the ability to update all the cluster roles. So I now effectively have a cluster admin token, and I can do whatever I like with the cluster.

C

So um let's have a look. What we did here so we uh created a. um We know where the xcd um cluster pods were running on which host we created an xtd client pod on the same host and mounted the host path, and we got cluster admin rights from xcd.

C

So it's game over. At that point,.

B

Matt sorry interrupt, I'm like the angel of your animal time. Yeah.

C

No worries we're basically done we're basically done.

B

Okay, a couple of minutes.

C

Cool okay, so how could you have prevented this? Well clearly, um you could have scanned your application code um before you deployed it into production so that you didn't have a remote command execution vulnerability. um You could scan your kubernetes files for security issues. It'll pick up a lot of things like um permissions being too wide like security context, settings not set incorrectly, and you can scan your container images before you deploy them to uh to look for vulnerable applications in your container images.

C

You can do all of that um with sneak and you can sign up for a free account at app.sneak.io and uh last, but by no means least, um I want to give big thank yous to uh some folks who who inspired and did lots of the hard work in terms of uh um researching some of these issues, so that uh I could build this demo. um So thanks to mark manning ian caldwell to duffy cooley and many others in the kubernetes security community. So I'm happy to take um any questions.

B

Well, we don't have much time for questions, but um I'm able to only pick one uh it's from fabrizio pandini. Are there instructions available about how to reproduce the this talk uh and play with this locally.

C

Yeah, so if you um this, this talk uh this all of the the code for this, the the uh the setup for for this is just running in kind. So you can um go on my github um repository. uh I will give you the url uh now.

C

uh And you can see all of the uh the setup and and configuration for the for the demo who lives there.

B

Okay, uh you should maybe uh just paste this link into the baby chat or.

C

B

Or um can we can forward it for you, yeah.

C

That'd be great, I'm not currently in the baby chat, so.

B

Yeah yeah yeah no problem uh just one more uh like yeah. Is this the same on all kubernetes platforms.

C

No, I mean that's a good point. I mean your mileage does vary with some of these things. Right, particularly, um you know. If you're uh all of the the vendors will have their own kind of opinionated settings about about security, you know um if you run on openshift a lot of these things are set for you by by default um and if you run in the cloud providers, obviously you have different different security settings depending on which which managed service you're using. um I think that's the important thing when we think about kubernetes security.

C

Lots of people say: oh kubernetes is really insecure, but you know kubernetes. The upstream vanilla, kubernetes is is designed for maximum flexibility right so like if there are no guard rails. So you need to understand how to configure this stuff um uh when you deploy it, if you are deploying uh from scratch, but yes, if you're using a managed service provider, they're gonna have uh their own set of things that they're probably gonna set for you, so you may not be vulnerable to all of the things that I described in the in this talk.

B

Yeah and how can I protect myself.

C

Yeah, like I said I mean you, know scanning, I think getting your mindset into uh into scanning stuff before you deploy it. So you know I work for snake. You can do all these things with with sneak, but you know, other security uh tools are are available in the kubernetes in the cloud native ecosystem, but scanning your uh your kubernetes configuration all your yaml files before you put them into production um scanning your applications to make sure your applications uh are uh are secure.

C

They don't have a vulnerabilities that you've brought in through dependent libraries you're, using or whatever, um but building security into. Your workflows is really the key to this, uh and I, I think, we're getting a lot better at understanding that there's lots of great material out there from uh from uh the kubernetes security community and from the.

D

Cncf security community that you can.

C

uh Read up and uh and learn about about how to go about security in a cloud-native kind of way.

B

Okay, thank you matt. It was really an amazing talk and you also have quite the background behind you. It's almost.

C

B

Virtual background.

C

It's my new office. Actually, I've just moved in. So uh what you can't see is all the boxes over this side. I'm.

B

Still unpacking it's really amazing. uh Well, I I'm I'm trying to buy some time actually to see if some questions come from the chat, but uh nothing we have nothing well. um Well, if you, if you want, you can stay with us and maybe check the chat.

C

Yeah I'll go I'll, go hang out in the chat for a bit if anybody's got any other questions feel free to hit me up there. That's cool.

B

Okay, thank you very much, matt pleasure.

D