Cloud Native Computing Foundation KCD Berlin 2022, 16 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Highjack a Kubernetes cluster - a Walkthrough - Nico Meisenzahl, White Duck

Description

Nico will show how to highjack a Kubernetes cluster based on common attack vectors. You'll also learn why it's important to implement zero-trust to prevent data leaks.

Website: https://whiteduck.de/

Organized by @Microsoft @kubermatic7173 @SysEleven

Thanks to our sponsors @CapgeminiGlobal, @gardenio, @sysdig, @SUSE, @SUSE, @anynines, @redhat, nginx, serve-u

A

Today's talk is mainly um because we had some customer meetings and working with customers and yeah somehow realized that many people are companies out. There are running kubernetes and do not think about any security, um and that is why we spun UPS. Let's talk and yeah would like to tell you and to show some best practices um you could easily Implement into your cluster, so this will be not any Deep dive session. So if you I don't know deep life, security, um one feel free to go into the plan.

A

So so really this is basic basic stuff. Talking your Basics like kubernetes policies, um Network policies, basic security topics, um yeah and Main of the talk will be really a demo. So we will hijack a kubernetes cluster um and see what we can do here um and then maybe if we have time talking about how to prevent this, um but it depends on the on the timing today.

A

um Everything I will show today, including demo, um and the slides are available on GitHub, so you can um just test it or implement it on your own, try again on your own and also slides um I, whatever they are yeah, so um streamer, slides. First of all, uh once again by, we need to care about security. So basically, this is um from the state of kubernetes security report from 2022 they're, pretty new um yeah, and basically, um what we're seeing here is, let me put in the laser.

A

um So in the last 12 months we really get issues related to uh misconfigurations in our clusters to um security issues which may not be fixed um to security issues in runtime, so nothing, let's say really running in our cluster and having the acute issue so and um also failed auditing. um So there is really a need to think about security, to higher security and and the kubernetes world, um and also talking about um yeah customer loses or Revenue losers.

A

We all said in the last 20 uh 12 months, um at least here 31, which had some issues related to where Revenue customer loses due to security issues. So there is need- um and this is basically by um yeah I'm staying here today.

A

Okay, so um what we will do is we basically try to get into a kubernetes cluster where a web application- and we were using log for share for this I guess most of you already know, look for Shell, hopefully already patched it in your clusters, but basically we will use this one to get into our cluster and then try there to get even more information and data. So just uh some some details on luxury shell. So basically we will inject the tndi lookup. Just a string. I will show you in a bit.

A

This will get locked by log for J and then basically, um look for chair will basically talk to an Lop server of our own, which will start in just a sec, um and this one. We basically provide us with some um Java class, which allows us to basically spin up in our case and reverse shell into the running container Running part in our kubernetes cluster and from there we'll. Basically, then uh trying around talking to the kubernetes, API, server and stuff like this.

A

um So, to sum up, um we have our applications very well in check with um which lock for Shell then open up a reversal on a techa machine, basically just a Linux virtual machine um having a public IP to connect to, and then we will try um to talk to the kubernetes API server to see.

A

If we have some access right can do something there trying to hijack the underlying node to maybe see some um yeah workload, which you do not see a kubernetes level, but maybe on a container runtime level, um then trying to produce some data from any kind of uh yeah databases or stuff we're seeing um and furthermore, then trying to also hi check um to the cloud provider and get trying to get um access to non-kubernetes resources or even spinning up um some other workload um on the cloud provider.

A

Side, yeah, but but basically ready for the demo um and, as I said, if it's too small just just give me a hand. um What we'll do and all the markdown is also on the GitHub repo um and we'll also just basically follow up um the markdown and then give you some summer insights, So Below. We have a terminal here, I already connected to the attacker machine, so just virtual machine and they are downloaded um the lock for Shell POC. This is also the link is included in in my repo.

A

um It's just POC. You can use to implement lock for Shell on test stuff against it. It's not developed by me. um It's a nice open source product you can use, and this will be basically the Baseline um we're starting with so first thing we will do is we will spin up um the lap and the HTTP server to be able to inject via log for gel?

A

We'll do this one on the attacker machine here, so it spins up the adap server, it spins up the HTTP server, and then we will furthermore um spin up just an open port. We will use netcut here, um I just decided to go with um 443 because hey. Why not it's an outgoing connection! So if we are able to connect outgoing in the Internet, it's pretty sure when it's allowed it's the area for three, so we're going with server 3 here and open this port.

A

So next thing is: um we need to um try to inject um the chaindi and this we will do inside of the application. So this basically um application implementing log4j and um is here with the security issues here and what we'll do is um we will inject in the username um this string so basically saying Hey would like to get um tndi and please talk to this lap server, which is our attacker machine on the ellipot, and then we are putting in any kind of password here doesn't matter at all.

A

And it's just loading loading loading and here we're getting a connection from an IP address and if we're now here, for example, to an LS we're getting some data, so we now have a reversal open into the port in our kubernetes cluster. So first step we were able to do this because of the existing look for Shell issue.

A

So basically, if you're running under on yeah um workload or code, which is not really secure, you are pretty easy into your kubernetes cluster and into your port um so from there we're now trying to get even further.

A

um So next one would be, of course, to try to talk to the kubernetes API server um and for this basically kubernetes provides us with nearly everything we need. um So, of course, we need to talk to talk to other kubernetes API. The talk gets by default, exposed and mounted into a put so we're just exposing that one here to use it later.

A

um We also need a certificate Authority uh root certificate. This is also exposed um into our workload and then, in this case we already have curl in the container. So once again, it's maybe stuff in the container or in container image. We do not need normally to running the application, but in our case it's even better for us.

A

So what we will do here is we will do a curl providing the certificate adding a header just put put in the bear token we just got from uh already mounted in our workload and then doing a get on HTTP kubernetes service host and kubernetes Service Port https. Also, this is default environment variables. You will get from kubernetes in every part, so you don't need to think about where the API server just use the environment variables and doing a slash API here um and we're getting some data.

A

um So at least we don't get a deny um we're getting some data and not really important. So far we have the API server address here, but it looks like we do, have some access and somehow allowed to talk to the kubernetes API server um yeah. So, let's see um if we can get some more information here, um maybe just grabbing our namespace we are running in this- is also available and and mounted in our Port here.

A

So just exposing that one and then we're doing um nearly the same curl command again just adding um a different content suit here, so we're using slash, API, slash, V1 namespaces, our namespace and trying to get um yeah access to our ports here, executing that one and once again, you'll see we're getting some data, so we're seeing an image image, ID container ID.

A

If it's started or not so it looks like we are able now to to access some kind of data, um and maybe, if you're, where some, for example, environment variables, injecting Secrets or so we would even already got those here. So why did this work um so basically um you're getting the total? This is default, at least until kubernetes 124 with 124. You need to enable it to really get the token, um and this is the first one, the second one. Of course.

A

Normally you do not have any access um to the kubernetes API, so we would have get in deny. But um I've seen this so much that I, don't know somebody developer or admin doesn't matter just grab some some yaml files on GitHub or stick overflow or I, don't know just Google.

A

It copy paste it and just apply it in the cluster and you're getting some role bindings, maybe for a default service account in your default namespace like it's here in our case and then you have end up having a service account which is has access or high access to the kubernetes API server, without even knowing it. So of course you're. Not you!

A

You somehow need to enable this one to make this working, but I have seen it so many times um that there are royal bindings um and excess level set and the cluster and nobody cares or that nobody knows about it um yeah, but let's try it even further. um Curl is fine, but hey. Why not just downloading Cube CDL?

A

um We were able to open up the reverse here, so we have internet access, so nobody cares so next step, I'm just getting the latest Cube CTL um from GitHub, and then it's pretty easy or easier to talk to the API instead of just using Quran and doing this all the stuff manually here.

A

So with this we are back so just doing Cube City area pot and at least seeing here, one pot I. Guess it's our own pot we are in here at the moment, um though, we can now even try.

A

To get an uh minus a to get all namespaces now here we can forbidden. um So it looks like our service account, which I mentioned is in the default. Namespace default service account. So basically this account every workload uses if you do not provide a specific service account here does not have access to other namespaces to see other workload, which is good, um so nothing we can do here or so easily. It says this way. um So, let's see if we can do a cube, City I get notes to get some node information.

A

Also, this does not work um so looks like we have at least access to our um default namespace. So next one is, um we can use cubesatel um and doing and ask can I which basically allow us to use cubectl and ask the kubernetes API. If you're allowed to do something like. If are we allowed to create import? Are we allowed to delete import and stuff like this? So we're just asking hey? Are we allowed to create a pod and we're getting a yes? So it looks like we are not even allowed to just read.

A

We also allowed to um to write some data, at least in our namespace.

A

um This is pretty nice, but with this we now can spin up on container and can try to get even more data and basically leaving the kubernetes API level and trying to get on the Node and see if we can get some data there we can can work with.

A

um This is basically the the next step we will. We will try now.

A

Make it a little bit smaller here, as we will try to spin up a pot um just using the Ubuntu lets here, because hey Ubuntu is a pretty big image with many Tools in there. So we don't need to download some stuff too to get all the things done. So just using um a bigger this through here um and then we're trying two more things.

A

First, one is we're trying to get a privileged container to get some more rides to get some more details, details on the Node or at least to make it our life a little bit easier.

A

If it's allowed to do so, and we also will try to mount um a so-called host pass so we're trying to mount um the disk from the node we are running on um into our pot to then access the data, for example, both of them host pass and privileged are pretty nice um sorry and will help us um yeah to to get some more data from uh from the node.

A

So just copy pasting that one here.

A

And we were able to create that one, so we should also now see it. Our privileged part is running for six seconds, which is pretty nice so and now we can basically just do an exit um into the batch and try to um yeah use that when to to get even further details, um so what we did and what I, what I skipped um is I said we're mounting a host pass here.

A

uh Sorry, um we're trying to two mountain host pass here um on what we are trying to mount it or what what just worked is mounting the Run containerd directory.

A

So basically, now we have the containerdy um socket available in our pod and can now try to use the container dcli, which we will download in a sec um to get access to Containers which are running in the node which we but which we do not see on a kubernetes level, because on a kubernetes level, we just saw our own part here um so to make our lives easier, we're just installing coil and check re once again, because we have internet access. Why not.

A

And also getting um the container dcli from GitHub.

A

This protects second, and then we were trying um to use the CLI. We have to provide an address here because it's not the default uh soccer pass and then saying Hey would like to get everything in the kubernetes namespace of containerdy and doing a container list here.

A

um Thank you and see if we're seeing some some workload here and it worked so as you see on the kubernetes side, we're seeing one part so one container, if you're now on the container D level on the Node we're seeing much more stuff here, um which we now can see and and and see. If we I, don't know, find some data find some Secrets or stuff like this, we can then maybe use to get access to some data.

A

um So next one is, as I said, we are trying to retrieve some some secrets and then trying to get some data um to make my life a little bit easier here. We're just grabbing a specific um ID of a container um can show with that one.

A

um So this is the idea we're working with now and um for this one we will once again use the container dcli with updated or customized address. The namespace and doing container info on this container ID and then doing the jQuery, because we just want to get secrets so, basically, most of the time we will find them the environment variables. So we just go for a spec process and fear and see if we will get some information here.

A

So of course we want to get get the kubernetes informations for our services for AP, API and stuff. Like this, we used in the previous thing um we're getting past information host information looks like it's somehow an nginx here doesn't matter uh but hey. We are also getting a red ski and a rarest host here.

A

um So it looks like we have at least on yeah a host name and key to connect to redis and to see if we will find some data here. um Once again we have internet access. So why not? Just installing the redis tools here to make our life a little bit easier.

A

Exposing um the host and the key name again from the same location- we just just saw it and then using here the ready, CLI, um providing the host information providing the key and just grabbing.

A

um The key here called call data I'm doing a get on that one and see if we can access some data and output is some secret data um even further. We could also do not again, but a set on the data and I don't know set it to.

A

Blah, for example, getting okay, so we are also able to override data. So we could just delete everything override everything. I, don't know, dump it somewhere I'm using all the data we're seeing in the radius here and to sum up, it's one thing: to get access to on radius and to find some secrets and environment stuff um in the environment of a container. But we are really using a different container here, so we are injected into application a and using and accessing data from a radius from a completely different application.

A

So why are we able to access this redness even further? So why are there no network policies or stuff in place? um So we shouldn't need to get this data, but anyway we have it um so next step. Next Step would be to get even further and trying um yeah to access or to exit the kubernetes and getting to the cloud provider. So in this case it's Azure, um so it's a little bit Azure specific, but it basically also works on AWS and on on gcp doesn't matter at all.

A

um So basically, we were using um attempty here and just mounting um the first disk. We will find on the Node and hope it's the right one um and then um you're searching for so configuration file and in this case it's basically within the etcd kubernetes directory and the ass and kubernetes and Azure chasing um and basically every managed kubernetes and identity and the cloud provider um on the Azure side.

A

It's called user-side identity in this case, so we're getting an idea of an identity and we were trapping that one and exposing it and I can also show show it to you.

A

So this is basically the idea of our kubernetes cluster in azure, and we can now go ahead and talking to the Azure metadata, API and retrieving a one-time token here, um so just copy pasting that one and we'll show you in a sec um so we're doing a curl here um on the metadata API of the Azure resource manager and saying hey. I would like to get an os2 token for the clan ID our client and then providing an API version.

A

Some metadata doesn't matter at all um and doing a check really at the end on just grabbing the X token here. So I won't show you this one, otherwise you're able to access my subscription here um in this file. You also can grab some subscription information. Resource of information is azure.

A

Stuff doesn't doesn't matter in this case, um but we'll use this to to get the data from Azure, and now we have everything we need to talk to um yeah the the Azure resource manager, the big Azure API, um and trying to um do and read here.

A

So what we'll do is in curl um we're doing get once again providing the token we just retrieved need to set them other headers here and then talk into management azure.com, um providing our subscription providing our Resource Group and, as I said, doing a get and to get a nicer output just putting that one to check through here and we're getting a Json back um we're seeing some things we're seeing. Okay, the name of our Resource Group we're seeing hey location.

A

So it looks like it's running in rest, Europe and Amsterdam we're seeing some tags um we're seeing who managed this uh this resource here we're seeing the provisional state. So we are now able really to use the identity of the kubernetes less than Azure, to talk to Azure and to read some data but hey. Who would like to read data when you can create something um so because we have a little bit. um We have shorter time today.

A

We do not spin up a nice virtual machine where we can mine some Bitcoins just going with the with the blob storage here um and basically doing similar stuff.

A

um Once again, the token some metadata here and trying to create a so-called storage account, which is basically just blob, search and Azure um and um yeah trying to um create that one um and also this had worked.

A

So we are now not talking about getting access to our kubernetes, so we're now talking about creating resources on Azure and doing I, don't know whatever you want, um which is pretty bad um yeah. So, as you see it's pretty easy. If there are some things bad in your cluster, you can easily get access to other workload. You can easily talk to your cloud provider in this case Azure, but it doesn't matter also works with other ones. So you should really really think about um security here, yeah, so um maybe back two slides.

A

um So how can you now start with introducing security? um So basically I would recommend really think about the whole. Let's say a devops cycle, so not thinking about just the runtime start. Also thinking about yeah securing your application codes, securing your deployment code so really try to start thinking about Security in the whole life cycle and I would recommend, starting with quick fixes so really implementing the best best practice practices.

A

It doesn't matter if you I, don't know doing an internal broad check talking two years about implementing Suzuki and then after two years you talked about it, but you still have um Antico clusters running so starting with the Quick Fix and then iterating and try to get better and better all the time yeah.

A

So what you can think about- um and maybe um I just go a bit to the detail, slides because 10 minutes left here um so once again think about your application costs first, so automate and enforce code checks so doing uh dependency scanning like depend about or random renovate. We had a session on rendered yesterday um think about steady code analysis to really scan your code and to see if you have some security issues in your code.

A

So this is really the first step you should do and um second one is to really make sure you have secure container images. um Well, let's say secure and small, so we started with um the First Command we executed in our container was a curved. So why is quell available in the container image?

A

The application doesn't need car, so really think about what you will need in your container and just put the things you need in your image and don't think about hey, maybe I have to debug the stuff when it's running or so we have now nice options with Cube CTL debug to attach the debugging container and stuff like this.

A

So also debugging tools, you don't need them anymore, and your images, if you ever trust yourself contain binaries, let's go with.net doesn't matter um or digital is um base images to really make sure that you only have the stuff in the container your application needs, and not even more because having less stuff in your image, you have less stuff which might help someone to get access to something or yeah.

A

Introducing any security security issues also make sure to scan your Docker files with static code analyzers to also get Implement all best practices here and scan your container images regularly, not just when building them. Also the whole time in the container registry, and also when you're running them in your kubernetes cluster. So then you can make sure that an image you build two weeks ago um is maybe patched in the container registry or deleted in a container registry, but maybe still running in kubernetes cluster, to also get the information there.

A

When you have a class, an image instance running in your cluster and you're getting security issues there so think about the whole life cycle here, yeah even um secure container image would have make it a way harder to get all the stuff done. We did some minutes ago, um so really important. One um also scanning container images would really help.

A

We would see in just a sec that our application um still has the issue of log for shell and you need to patch it, and you would would have the awareness we could patch it and maybe did not even deploy it to a cluster um once again, also with secure deployment code um so also validate beside your application code.

A

Beside your Docker file is also validate um your development, manifest temp, charts, kubernetes resources, manifest and so on, um and also try 2D to yeah, not Implement, things like running a container as root mounting, host, pass and stuff like that. So basically these things we did um to make sure that if somebody is able to access your container or get access to your container cannot do any any bad things here.

A

um Also this the container we connected to in the first step was running as root, which made it much easier um to get all the things done and get access to the kubernetes API and and to the node even further yeah. Just some links here. um You can check them later uh some tools which can help you with static code analysis um yeah. Furthermore, I think there was also talk on that one, so I will make this one a little bit faster.

A

Think about kubernetes policy, so tell your cluster, which is okay to do like, say. Okay, every container needs to read on the file system, denying host post mounts um denying run, running containers with root and stuff like this.

A

So really you can deny this one and enforce it to on the one side, let's say: give some guidelines for the people deploying to your cluster, but also if somebody is in your cluster and create some pot like we did with privileged and host pass, then kubernetes would say: hey no, it's not possible to host to mount a host pass, it's impossible to run a privileged container, for example.

A

So it's not only about the people using a cluster also, the part, the people that are in your class- and you are not aware of them so really important um as I said, but we have policies in here we could have denied all the stuff uh and we are able to to get access to the node and then to to Azure itself.

A

Tooling, open policy agent forget keeper or Covenant, but we had on session on that one so um skipping some more details on this even more important um kubernetes Network policies, so that you can really think about or make sure that in certain workload is not able to talk to any other container like for example, why does our container we um yeah get into need access to the internet?

A

So why are we able to spin up and reverse here by way we're able to I, don't know, use Quran to download Cube CTL, or so you don't need that one and also the application doesn't. So you can really deny that one um limit access between application name spaces, um so by very able to talk to the redis instance, which was related to a completely different application.

A

um So let's say we have an application a but there's some somewhere running application, B with redis instance. So why does does application a need, access to radius B, for example, and even even more important? The last one um you should deny access to the cloud provider metadata service to, for example, get the one-time token to then talk to azure. So if the application does not integrate with Azure or any other cloud provider, they do not need access to the metadata API to get to be able to retrieve this one-time talk, for example.

A

So with this I don't know, somebody in your cluster can just get a talk and then there's not his access to talk to the kubernetes to the Azure APA, but have the access rights of the kubernetes cluster to do I, don't know something um so really really important.

A

Yeah. Once again, as I already said, Network policies would have denied most of the stuff. We did and even more important container runtime security um to get insights of the stuff which is run in your cluster, so really getting insights when their SNL started or executed in your container, then I don't know.

A

Somebody is executing some commands in a container beside their own application um so that you really really get um yeah alerted or also the requesting nights um when there is somebody doing something in cluster which shouldn't be there um tooling and I think there were talks on both here um FICO, for example, or tetragon, um so both of them are really really great.

A

To make sure that you are aware of stuff running in your cluster that shouldn't be there yeah, also that one, basically, everything we did in the demo would be uh shown up in FICO or tetragon and could have been denied yeah further best practices, as I said, do not share those accounts.

A

If you give a service account higher access, Rights Make sure to be aware of them and not just copy pasting, any other rules from the internet, because it's then ending up with a default account having higher access as we had and then we're able to talk to the kubernetes API server um do not Mount uh the service. You can't talk. If you don't need that one. It's just one line uh in your configuration. You can disable that one.

A

um It just included the link here: um where's kubernetes, 124, it's disabled by default, um yeah and then yeah as I said review all your Snippets um you're applying to your cluster and also maybe think about on web application firewall. If you have some HTTP workload, for example, also that one had would have prevented to even check the lock for shell into the application itself. So also, really really important.

A

Yeah, um as I said, would allow us to talk to the API server and put a free um skip the whole thing so back to uh uh my last slide. So, as I said, all the stuff is available on GitHub all the SlideShare um yeah and with this video 10 seconds over time. But maybe we have some time for questions.

A