Cloud Native Computing Foundation Cloud Native Security Day North America 2020, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Capture the Flag Wrap Up & Summary, Andrew Martin, Control Plane & Magno Logan, Trend Micro

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Capture the Flag Wrap Up & Summary, Andrew Martin, Control Plane & Magno Logan, Trend Micro

A

Hello, everybody and welcome to the capture the flag shakedown. uh We will go through what has occurred today and uh we'll actually replay the scenarios and look at uh what you could have won now, notably, we didn't actually have any prizes today, and that was uh intentional.

A

So let's explain why that was and what has happened. uh Hopefully, my screen should be sharing. We shall see okay, so yes, we we had a ctf today, hello and welcome thanks for your attendance, and hopefully this will leave you mildly entertained. So what were we doing today? uh Today was a day of ambition.

A

We aimed to educate, amuse, beguile and everything in between in a safe and happy environment, and we had the absolute pleasure of introducing people to the joys of container hacking scenarios are each built with a learning outcome in mind and our capable task masters also constructed the scenarios which gave us a little bit more closure on actually helping and giving hints to people as they went through, notably absent from the list of goals, is ranking.

A

uh This is because there's no official scores, there's no prizes, and the idea here is just to take the pressure off and to encourage people to go and hack stuff and to learn and to enjoy themselves breaking things in a safe and importantly, legal environment.

A

Although saying that, if you have captured a flag or more and you'd like some, some secret swag then do dm a physical address to the nice task master and they will ship you something worthy of your efforts.

A

Okay, so today was hopefully an exercise in preparing for the unexpected thinking a little bit outside of the box and uh an encouragement that uh nothing is entirely secure and we should we should pay attention to these things, so we are in the uk time zone on this end. So please bear with us. um It's been five o'clock for quite some time.

A

We had six scenarios, one scenario per talk, so if you did the first scenario, it was concurrently running with the first talk and then detectively throughout the day um and lots of talking but nefarious, small and potentially mild vulnerabilities, unless, of course, uh someone did better than we bargained for and uh the aws build will be significantly higher than we thought.

A

So there are many places to look for vulnerable clusters, but we believe the only way to up skill people is to give them a safe place to practice in so, instead of going on uh cluster safari, we prefer to run ctf-based training but capture. The flags can often be daunting. Big challenging filled with the kind of people who we we look up to and we want to break those barriers down and give everybody the opportunity to be involved.

A

So we hope there's been a strong first user experience, um there's attentive moderators in the channels and teaching assistants and uh yeah. We just want to give people a boost on their cloud native journey, so uh spoiler alert. um If anybody is still playing your time is up and we will uh not be validating any flags after this point. uh So let's go through the scenarios and find all the flags now um excuse me, while I shuffle my.

A

A

Okay, so the first scenario um was uh called node secret breach so back to the screen share. Hopefully,.

A

Okay, so uh what happened here well, first of all, we start we're, starting as the root user, and this is uh this is mixed. Of course, root has a special set of privileges in linux. The root user is a specific and special user.

A

That means it has capabilities, and those capabilities may include things like changing other people's files or opening a network adapter in a low level mode. So we can just either send custom packets or turn it set it into a different mode, so we can sniff stuff on the network.

A

These are not things, we'd want an average user to do, and so the distinction of running in a container without username spaces is important at this point. So, let's see what else is in this container?

A

Let's have a look around well. This is what we expect right. We've got a process namespace, so we can only see the processes that we have available to us. What else might we do? We might have a look around and see what else is mounted. um This is kind of noisy there's not really much here. That's that's useful. There are a few things anytime.

A

We see a docker, uh or perhaps one of these mounts they could be interesting, so uh df the disk free tool is a little bit of a quicker way to get a view on this. In my opinion, and of course we see a service account there, service accounts are juicy and uh we are people that love messing around with service accounts.

A

On this occasion, though, um this was not a service account based challenge and uh okay. So we've had a look. What's running. uh Sorry, we've had a look what's mounted within the container um there's a lot here, but some of the things, notably a device from the host other stuff that doesn't look like it's inside the container.

A

Well, that's because containers are wonderful but they're, not a perfect abstraction, and that concretely means that container runtimes have to put effort in to hide certain things from us and those things may include parts of the proc file system, bits of cis, the way that we interact with the process table and also the way that things are mounted in so uh okay um containers are a child of evolution um rather than intelligent design and like everything else, on the internet, our gaffer tape, together, uh no disrespect to people who've done an excellent job, making them available to us.

A

um Okay. But enough of this there's a lot of noise here we want to get some signal, uh let's think about what we can do with the service counts right, so we can see that dev is here. So what does this mean? So docker will mount in etsy hosts using the mount point from the host that it's on. So wherever you store your container images, those read write layers, that's where this is so already. The abstraction has leaked with the file system. Here.

A

What do we do next then? Well, I guess it's probably worth uh checking our privilege. Now there is a canonical way of doing this, uh which is jesse frazil's am I contained, of course it's not doing anything magic. It's checking states and files that are available to us inside a container, but it makes it very easy for us to do and gives us a unified view. So let's try and do that.

A

One thing to bear in mind is that we are roots, and that means that we have the ability to run package manager commands. We need to be able to write to any location on the file system, so we have to be root to do this generally and uh yeah there. We are so just installed curl, because without curl, I'm gonna have trouble pulling in stuff from remote endpoints, although of course any programming language is an interpreter and most can be used to replicate curl in some way.

A

So, let's pull am I contained from github chamod it and run it happy days? Okay, so we've got some stuff here. What does this say uh we're in kubernetes? Does it have namespaces pid namespace generally? That is always true anyway. Username space is generally always false, because username spaces are not enabled, I suppose, by default in run c or in kubernetes.

A

This is interesting, though, we're unconfined in app armor. Now, if we were running in docker, that would always have a profile kubernetes, of course, disables this, and uh so it's just reinforcing the fact that we're in kubernetes, but it doesn't tell us anything uh too drastic this. However, what is this?

A

This is a a wide bounding set of capabilities so stuff in here this means we can change ownerships override discretionary access control, which is um basically writing on other people's files and all sorts of things in here that we recognize processes being killed, raw input and outputs access to the network adapters. This is pretty likely to be a privileged container okay, so we know what that means.

A

That means that actually, what is in dev is a reflection of what's on the host, so as a reminder, the privileged flag, uh the most dangerous flag in the history of computing, as I will off to repeat disables most namespaces, it turns off app armor and setcomp. If they're enabled it grants all capabilities and it mounts all of the hosts devices into the container, this is a bad day for a system administrator. Why?

A

Because we can say all right, we know which disk is mounted from the host, so let's mount that disk into an empty mount directory.

A

Okay. So this is probably not what we were hoping to see, because if we were blue team um because that's what the root of the containers file system partition looks like, whereas what we've mounted in from the roots, uh let's go back to there, we've mounted it from the host contains extra stuff, and lo and behold it contains a directory called node secrets.

A

This may be obvious oops, let's, uh let's actually yep, go in there uh and there we go. So there is our first flag.

A

The disadvantages of running privileged containers are that they are equivalent to running the process on the host. There is essentially no containerization going on when we run privilege and yes run for the hills. If that's the kind of thing you are trying to defend, because it's more or less indefensible, okay, that was the first one.

A

Next, we have uh escalate and cover secrets and.

A

Let's jump in so this alias will just take me into the um into the next host.

A

Okay, let's spend a bit more time, looking at what's happening here, so um we're in the attack container and escalate and cover secret sre has deployed a postgres demon set and a new pod for monitoring the processes in your cluster following a routine audit of an application, it is believed that the deployment setup allows a compromised process. Audit pod container a mouthful to escalate its attack to other containers on the host.

A

This doesn't sound like it makes sense, there's obviously something a little bit askew here, okay, so the the question, then, is verify this by uncovering both the postgres database password and the secret key. We will start in the process: audit, pod, okay. So, um first of all, we know that it's possible to provision secrets via environment variables.

A

This is um this is something that the 12 factor app has told us historically, but, as we know, the environment of a process leaks to other users of the system, and so we actually prefer to provision passwords with a with a file and an environment variable as a as a pointer containing the path of that file.

A

So I mean the first place to start here, um then, is probably just looking at the environment. Let's just see what is in here: okay, well, there's, actually nothing particularly juicy in here and uh because we know I'll just go up and because we know that the the flag has a specific format. At this point, um we can use that string to uh to just grab through the end.

A

We haven't missed anything, there's, nothing there, okay, um so we're supposed to be, let's just remind ourselves, this audit, this process, audit, pod and we're looking to understand what the database adjacent to us is doing. So, let's just see what we've got running. Okay, this is a red flag for a catalonian festival.

A

We should not be seeing this many processes because, as we saw, let's go back up to where we were earlier, as we saw in the process table.

A

Excuse me it's further than I thought the process table here for the container, which is isolated to itself, we're only seeing a very small number of processors, and notably, we see pid 1, which is not systemd or an init system. It's just a sleep command, so that's what we would expect to see, but actually we can see all sorts of things we can see stuff on the host, uh so this is again a remarkably bad day.

A

um The question here, though, is not how we uh we take over the hosts, although from from here it's um not especially difficult. We still want to exfiltrate those um those crucial bits of data, the secret and the key uh okay. So this is probably how we traversed containers, because we shouldn't be able to see the process list as soon as we can. Let's list it uh in its fullest extent and have a look for postgres okay, so we found postgres and just to be clear.

A

This is post squares, running in a container on the same host.

A

What we're seeing here is a host pid namespace share, and this again is um of questionable repute as a practice that there is some necessity for it, but we should always be careful when disabling a container security feature or container primitive, like host namespaces for networking or pids or even mounting stuff, in that we're breaking the container abstraction by adding stuff or taking stuff away uh really, we want to be locking it down and not sharing stuff further with other workloads on the system.

A

Okay, so um right, we've got a pid for postgres and we can also see the postgres. uh We can also see the pits, so uh let's do some magic, proc diving. So if we go into proc and we put the pid in there- then, let's start off just looking at the command.

A

Okay, so a foible of proc is that everything is. uh Is null terminated so in order to just get a bit of visibility into what's actually happening here, let's uh replace knoll with uh just space this time. Okay, so we can see that it's just been invoked as postgres all right. No problem- that's that's sound enough, but what else is in proc um I'll give you a clue the environment?

A

And at this point, even though that looks like a massive junk, let's see if we've got there's our flag so again using the flag, ctf grep when there's a well-known flag string um can help, because otherwise it's a kind of indeterminate mess of uh of black and white. That's a little bit difficult to pick things out of okay, one down, so that is, uh that is a flag.

A

But if we go back to the beginning of this scenario, we were told uh there is more than one flag, so I'm covering both the database password and the secret key okay. So we know we've got the database password, but there's a secret key.

A

We don't have any access to that adjacent postgres container, except for through the the proc table. Okay. So what else can we do with proc.

A

We can examine the root file system mounted into that container. Let's start with hello and studs.

A

Okay, so it's a sim link. We don't mind too much about that. Here we go again. We have got into the file system of a container running on the same machine as we are so we've got into the mount namespace, essentially without having access to the container itself.

A

If we go into secrets, we can see that there is something that looks like a key there and if I have readline there, we go okay. That is key number two, I'm conscious of time, and I will try and keep on clipping.

A

um So the point here is that there is a real risk with enabling using the um sorry process name spaces and uh yes, while it's necessary. Obviously the feature was shipped for a reason.

A

We should be cognizant and, of course everything should be threat modeled, and then we can balance the impact of that thing actually being breached or negatively affected or or exfiltrated, with the benefit that we get from using it in the way that it's intended. Okay on we go and we're into a ci server, vulnerability.

A

Right, so what are we doing here? uh We're pen testing a cluster. We found the vulnerability. uh The pod is part of the build infrastructure. As we know, uh build infrastructure is a juicy target uh supply chain. Security is a uh a particular interest of uh of of security and the cloud native security day of which we are apart and so build infrastructure. Yeah, that's! Okay! That's of interest to me uh all right what's happening, then we're concerned that a compromise may lead to leaked secrets.

A

Okay, so we want to extract the secret key and look suspiciously like we are in a jenkins, flavored, pod, okay, so so again, we kind of just want to do a bit of recon. Let's just figure out. What's what's going on, process table is okay, uh the amount name space so again, there's a couple of things here that jump straight out. One is the service count.

A

um Of course you can do the same thing again looking here, but nothing really jumps out from there immediately uh that to me again is just a little bit of a mess.

A

The thing that jumps out for me here is not the service count. It is the presence of the hallowed docker socket.

A

A socket is an inter-process communication mechanism, amongst other things, and in this case it means that we can probably talk to the docker demon whose socket is mounted into the pod, which we're probably safe to assume belongs to the jenkins host. um Now, at this point, we would hope the docker is installed.

A

uh We would probably check uh see the kernel is nice and recent. Okay, that's all good see what release we've got. Okay, it's all relatively recent, so we assume probably that we can install docker through the package manager, but we can also do something via backdoor because we have curl.

A

So let's just pull the official docker installer, maybe we'll we'll free ourselves from bugs in docker itself, although those days are much further behind us, um but it's nice to be on the latest version. Isn't it even if we're going to break stuff? Let's do it with uh correct operator practices all right so we're installing the docker client in the expectation that we can use it to abuse the docker socket. We could just send restful commands over the docker socket, but it's a bit more long-winded and right.

A

So what we see here is the docker version command has given us the client and the server happy days. So, let's see what's running here, there is a lot and of course we see kubernetes oops. uh Let's do color in the us, spelling always yeah, and then we see cubenet is all over the place.

A

So it's probably again not a good day for somebody. um In this case, we probably want to look in the nginx container.

A

He says so: let's, um let's have a look in here. If we do a docker inspect, in fact, we can do it more elegantly with the containers. Sure, okay, again, there's a lot of information here we can kind of spool through it, but because we're volume hunting and we uh or flag hunting. I suppose- and we know the flag again sweet there. We go, there's our secret access key.

A

That looks so, let's have a look at some context around that um and colorization is always useful.

A

Okay, so what's happened. The environment of the container has specified this environment variable, so we're back into environment variables. Again, not only do they leak on the host on which they are running, but also they leak from uh metadata about the thing. As you see in this case it is an insecure provisioning method because um it's either set at runtime or set in the image itself, ideally not, of course, so really.

A

What we want here is um is to instead point that secret to a file, and this is, as kubernetes will do with a secret file map for us, and that way we we need to have access not only to to the uh to the containers metadata like this, but also the container itself or, as we saw previously, the process table, um etc.

A

Right at this point, we had pretty good, um I would say, turnout. We had. We had a lot of people we'll uh we'll get to those numbers at the end, but also a lot of people got through these scenarios and uh and at this point most people were still still with us. So let's persist. Next, we have a non-user compromise.

A

Okay and out and back in again.

A

Okay, so what we'll be doing here, um more uh escalation, sideways and so just moving laterally through um through kubernetes or maybe actually on the yep on multiple nodes.

A

So, okay, so secrets have been extracted from the cluster, we're in a post-mortem phase, but it's not clear how the anonymous user managed to escalate sideways after the initial breach. So let's try and replay the intrusion from inside the cluster okay, so we're in the pod that was breached.

A

Excuse me and we're going to replay what the what the, what we expect the attacker did um now again, we've got our our kind of basic recon, which is just saying uh how's our process table.

A

Do we have anything spare or extraneous mounted in here um we'll go to the map points for the whole system uh we can install. Am I contained and see what our bounding set is, but in the interest of time this is this is slightly different. So at this point we are attacking things that are outside our domain as sorry outside our namespace, let's say, and we have got the ip addresses of the nodes.

A

So what runs on the kubernetes worker node? Well, there's the cubelet there's q proxy. There may be things for the cni as well.

A

The cubelet has some configurations that um are less than optimal. Let's say uh such as the uh the read only port, so let's just see if we can find anything. Excuse me if my mouse constantly doing that um in and around here. So uh again I know what I need to use in advance, so I will just install it. I don't need to do an app update.

A

I get curl and jk. So what are we actually doing here?.

A

Let's just see if we can access these host nodes, so let's go up in here and as a reminder, this is a network route from the pod onto the public interface of the host and really there's not a great deal of rationale for um for running like this. uh We should be using network uh network policy and um yeah, frankly constraining our outbound traffic so that we can't hit anything at all start with the default deny and uh and then upgrade okay. That was smooth. Let's just install that too.

A

So, as you can see I'm having to install a lot of software as I go along that's kind of standard, there is uh generally we don't ship curl and that kind of thing to production, because why would we unless we needed it for our application, but as an attacker, I am able to install stuff inside the container, really just by setting a non-root user, uh maybe even removing the package manager if needs be. That makes for a much safer day, let's say the jq and curl. If I installed those, what did I get wrong there?

A

Nothing, okay and let's um skip verifying the certificates. Okay. That means that we can read from the cubelet and we can see lots of things that are running here and what we care about.

A

uh Potentially is something um compromise ish. So let's have a look: uh nope, there's nothing in there. uh Now that probably means that what we're looking for is on the other node. So, let's just remind ourselves which nodes were which and switch over.

A

Happy days that is the pod that we are looking for. Okay, so now we have more information about what we're looking to attack. So what should we do with that? Well, let's see if we can dump the end of this um this pod, so the pod is um what is it going to be the workload pod there and then, if you will just excuse my copy pasta, one second.

A

And this won't quite work because of the node ip. So, let's just place llp is the one we've just used, which is the second one.

A

Okay, so we're posting a command oops, okay, so we're posting to the note ip um force of habits uh on the insecure uh on the read. Only port rather and uh we want we want to go into this pods and then run the print end command.

A

Okay, um joyful, joys, that is.

B

Not coming back with anything.

A

Have I done something wrong.

A

um I guess the pod name should be pod name and not pods.

A

That was a great relief to me and as we go again, there is our flag happy days. Okay, let's uh steamroller on through the rest of these. um We are on to pod breech extract, and for this I will pass over to my worthy companion, magno.

B

C

Everyone think you need to stop sharing andy.

B

Okay, thank you.

B

C

The scenario here that we have for the for the ctf uh is the the pod uh pod breeders tract right. So in this scenario, let me just log in again.

B

C

Verify your suspicion by breaking to the pod and extract the value of user credits, password right so and and your starting point here, you're starting on a visual machine external to the cluster right. So so, how do you get access to to the cluster to the pods of the cluster itself right, so um basic things here in the interest of time.

B

C

Good uh so first thing: if, if I need to access that uh that server or their service right, I need to, uh I don't have the credentials right, so I need to uh first do like.

B

C

Your your server make sure that everything's- okay- I ran this before just before uh the example, just to make sure that uh everything is running smoothly and we don't need to download it again so uh and, and here one of the tools that you can use to to do that to do the network, uh mapping and scanning is nmap.

C

B

Thank you. You will now be placed into.

B

B

Okay, uh can you guys hear me again uh my screen sharing.

B

I'm sharing it.

B

C

B

Can you hear me, can you guys give me some feedback on the chat.

B

Yeah, okay sounds good awesome.

C

uh Yeah so, as I was saying sorry about technical discords here, um so as I was saying, we installed nmap there and another. uh One of the things that we can do is to run any map on this uh specific node id that was provided to us for this challenge. Right: okay and I've selected a few ports here, so the scan doesn't take forever right. So specifically for uh just the demonstration here, uh and we can see that there is a port open on on this higher port here, uh three: zero, zero, two two right.

C

So one of the things uh and it's very famous on on on pen, testing and and like uh application security scenarios is uh brute forcing right. So one of the attacks that you can run, uh you can get a list of username and passwords and try to just brute force the system to guess those credentials right and one of the tools that you can use here or for brute, forcing at this uh the sport and and access the service right. So I don't know exactly from the end map there.

C

I don't know exactly which server it is, but I can. uh I think it can run this. One, let's see if it's gonna give us to us a little bit more information there.

C

If it takes too long, then we can just move on.

B

See here, okay, yeah.

C

Okay, so we can see that this service right so it was showing on the first scan, was showing as unknown right. So I run uh I added the flag dash a here so to scan the services and the versions as well. So we know that this port is running an open, ssh server there and and then- and now I can try to uh to brute force it right passing a list of username and passwords right. So one of the tools that you can easily do that is called hydra and uh the way to install it.

C

Just typing oops.

B

C

Type in this command here.

B

C

Okay, so I have right hydra installed already just to show you guys here and now what I need to run this brute force right. I need a list of username and passwords right. uh One one of the the very common uh list of passwords is the roku list, so you could have download that and use it, but this server this this server here is not very protected. So the the password is not the username and the password are not very hard.

C

um So I created a list uh here already on on this server uh called the list of users, so I have a list of usernames here and I have a list of passwords that I could try, so it doesn't take forever when running right.

C

So here's the command that I'm gonna use for hydra to uh try to brute force this uh this username and password on this server location. Here.

C

C

C

Let me just show you guys so just moving on here, so we don't lose a lot of time, we're almost running out of time here. uh Basically, if you run this command, you should get the uh the results that had it here before. I know.

B

I I really uh okay sounds good.

C

Sure uh almost done uh so yeah, basically just after I find out the username and password I can access.

B

This server oops.

B

C

You should copy and paste here, and so the the user's admin and the passwords password very, very easy for you guys. You know. Even if you don't have like a tool, you could uh try guessing and access that okay, so I'm inside this called jump box right. So what I'm gonna do here is try to uh connect to the api server and make some requests so base base. The the base request here that I can make is check the version of the the kubernetes api server.

C

This is one of the ways to do that right, so I can see that's running our version. 119.4 stuff like that right, so just to wrap up here. uh One of the things that I can do is wha, uh as andy mentioned before. uh There is some um secrets and tokens inside the pods right, so I can grab that and and use to impersonate uh the policy talk to the api server right.

C

So basically, here what I'm gonna do is just create two variables: the namespace uh telling that's uh located at var, run secrets, kubernetes.io service account, namespace and I'll. Do the same thing for the token here: cubetoken, okay, almost done uh and then now I can make uh if I have uh permissions to do that, I can make a request to this namespace and uh through via the api server and ask for the secrets, and since this is a misconfigured cluster, I have the permissions uh so here just just making this api.

C

Let me show that again right. So this is the the request that I made and I can see all the secrets- and here is the flag user creds password and and that's it for this challenge I'll hand it over to andrew again. So thank you guys, bye.

A

Awesome, thank you very much. Okay, let's zoom, through the ends of the slides, there was one more scenario which people didn't quite get to. So um if you would like a go at that, then uh do feel free to dm right, let's get through to the end. So uh this was this. Was the control plane team at uh seven o'clock this morning um we are just about through the other end of it.

A

So honorable mentions thank you to these individuals for uh being with us through most of the journey today, and um we had a great time on dms, it's fair to say, uh there's some really uh people show some real grit and persistence and that's what it's all about so good job. Those people um we have some honorable mentions as well.

A

um Dilshan mathias, michael matthew, walid, steve mohammed and noel, were all there for the whole journey and thank you very much for your efforts in parting, the container defenses um various people enjoyed themselves slightly, and I hope this has been a beneficial learning experience. I'm sorry there was no cake and thank you very much to the control plane uh people at this end for manning everything and to magno for all his assistants and helping us out and testing etc.

A

This is a public service announcement. We don't run administrative endpoints on the public internet. The kubernetes api server is one of them. If you like what you saw today, control plane do this for a living and we'd be happy to stand up a ctf for you. Thank you very much for your attention and thank you to everybody who contributed. Today. We had a total of 327 clusters spun up, uh that is about 15 000 nodes.

A

um We had a peak of 73 users, so thank you for everybody who played, and we welcome all and any feedback have a wonderful day.