Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Insights into Unsecured Kubernetes in the Wild - Jay Chen & Aviv Sasson, Palo Alto Networks

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Insights into Unsecured Kubernetes in the Wild - Jay Chen & Aviv Sasson, Palo Alto Networks

As much as the cloud-native community devoted to building a rock-solid platform, the weakest link has always been the users who inadvertently introduced insecure configurations. Jay and Aviv will share their findings of misconfigured Kubernetes clusters in the wild and malware campaign that exploited these misconfigurations. Their research found 2,100 unsecured Kubernetes clusters that consist of 5,300 nodes, 31,340 CPUs, and 75,270 pods on the internet. Many sensitive data leaked from these clusters, including access credentials, source code, and PII. The researchers will share how they architected their honeypot infrastructure to collect and monitor malicious activities targeting Kubernetes environments. The research also uncovered the first malware that exploited Kubelets to compromise Kubernetes. The malware used sophisticated tactics to evade detection, harvest credentials, move laterally, and perform cryptojacking operations in compromised clusters.

A

Hello, everyone and welcome to our session about insights into unsecured kubernetes in the wild. We are really excited to be here virtually and hope. You'll enjoy and learn from this talk all right. So let's go over the agenda for today. uh First we'll talk about attacks against misconfigured docker, we'll talk about the notorious misconfiguration in docker that allow uh executing uh commands on docker demons, and then we will talk about the trends, the growing trends, trends of attacking those demons.

A

And after that we'll go over the research we did, which is splitted into two parts. The first one is that we thought how kubernetes can be misconfigured and then we actively can scans the internet for such misconfigured kubernetes clusters and the results are going to shock you. At least they shocked me after that. We'll go and talk about the honeypot we deployed, so we deployed those honeypots in order to see how frequent and what the the impact of those attacks on in on misconfigured kubernetes components.

A

So basically, those honeypots are the misconfigured kubernetes components and then we'll show you the results of the the honeypots and then we'll talk about hildegard, which is a crypto jacking malware. We found using one of our honeypots, it's the first ever documented malware that is designed to attack cubelets, and we show you how it takes over all of the cluster and right after it will talk about how to avoid such misconfigurations.

A

So that stuff, like this, won't happen to you all right. So, let's go over, who we are so me and jay. Both are security, researchers at palo alto networks under unit 42, which is the division for uh threat intelligence in palo alto networks.

A

uh Our area of expertise is the cloud we're both really enthusiastic about the cloud and all of our researchers were about anything that involves the cloud that includes vulnerabilities, malware uh campaign attack surfaces and everything that is security, wise interesting and, as you see by the pictures, we are both proud pet owners. I got my dog and jay got his cats, hopefully someday we'll be friends.

A

So, let's start and talk about insecure docker demons, so basically uh docker demon. Listen on a unix socket for api request and only root can acquire this unix socket so docker count on that and that's why they don't have any authentication or authorization mechanism inside so anything anyone that acquired this socket can just run any commands on the docker demon, but there are also another option to configure docker to listen on a tcp port but, as I said earlier, by default, there isn't any authentication or authorization.

A

So that leads to the situation where people just expose their docker demon using the tcp socket and they don't have any authentication or authorization and anyone can just access their daemon and execute commands. They can create new containers, delete existing containers, pull images, execute commands on existing containers; they just got full privileges and actually last year we found that there were over 1400 exposed, unsecured docker demons online, and my hunch is that during the last year, this number just got increased right. So we have this misconfiguration, but is this really problem?

A

Does anyone really exploit those demons? So we wanted to test it and in 2019 unit 42 researchers detected the malware that exploits exploit, dismissed configuration and they call it graboid and it was a warm, a crypto jacking worm.

A

Its purpose was to mine monero, which is a cryptocurrency and it propagated through the misconfiguration of docker and just exploited it and scanned other docker demons and exploited them as well.

A

So we saw that- and this was two years ago- and it was just the tip of the iceberg, because since then we saw numerous other malware designed to attack unsecured docker demons and we actually have a honeypot that been operating for more than one year and a half, and we could see that during this time the frequency of attacks was more than doubled and the number of adversaries was more than tripled. We talked about it on this black hat and the fighting the findings over. There are just nuts.

A

So we have this problem with docker and everyone exploited, but we wanted to to ask ourselves: is there any similar issue like this in kubernetes as well?

A

Are there any common misconfiguration kubernetes that can allow such attacks and if there are, are there any attack surfaces uh for that misconfigurations and if there are any attack surfaces, are they being exploited in the wild?

A

So that's the question that led us in this research and now I'm going to hand it to jay.

B

Thank you, ali, let's dive into kubernetes, so we have seen a lot of attacks against misconfigure, uh dr demons, and we thought that hey. This may also happen to darker bigger brother kubernetes. So we want to look into the attacks against kubernetes in the wild and what are the components?

B

What are the attack surfaces of kubernetes that we are focusing on? So we pick two components in kubernetes, api server and kubila. The reason we pick these two components is that they mirror the functionalities of darker daemons in darker environment.

B

Api server is like the gateway to the entire to entire kubernetes cluster and copy that is like a gateway to uh to a single worker node, so api server can control so api server takes http request and translate into operation to control, nodes parts and deployments in a in a quantity. Cluster could be that text usually takes http request from api server and translate translate those http requests into operations uh in a single worker node. So, let's see how these two components can be uh can can be misconfigured api server misconfiguration.

B

So by default, kubernetes api server listens on two ports: a secure port and a local host port, secured port use, mutual tos to authenticate between clients and servers by default. It runs on port 643, so any clients who need to communicate with the api server need to pass strict mutual authentication in order to invoke any api on the api server and for the local host port, it is by default. It is intended for testing and bootstrap purpose by default. It is exposed to only localhost port 8080.

B

It is not supposed to be exposed to the entire internet. It is, it can only listen to requests coming from the same host of the api server and an identity can authenticate clients accessing this. Particular local host port are mapped to system master groups, meaning that any request sent to this port doesn't need to be authenticated.

B

Any anonymous request sent to this port can actually act as a system assistant academy of the community class. So what can possibly go wrong with this local host port? What if the local host port is exposed to the public internet so on? The right is the this.

B

This is how uh could be a kobe api server can be misconfigured if someone accidentally bind this port to 0.0.0.0 instead of localhost. This port is exposed in to the entire internet, and anyone who knows the address of the iphone of this api server will be able to control the entire cluster.

B

So this is one of the most common misconfiguration that that we have seen in the past year and another commonly seen. Misconfiguration in api server is overly permissive. Row-Based access control so by default kubernetes come with a set of before rule before uh custom rows and before row, binding, for example, by default.

B

What I highlight in the order box here then an authenticated user or an anonymous request sent to the api server is met to authenticate uh an authenticate group, and this group isn't is fine. Is this? This group is found with the public info viewer, so the system, public info viewer role doesn't have much permission. It can only view a list of apis available in this api server or in this kubernetes.

B

Whatever kubernetes complex, each cluster can be shared between tens or hundreds of developers when more and more developers working in the same cluster. What? If someone accidentally finds this system and authenticate group to another more privileged roles, such as cluster admin or admin- and this can be also this again- can be really bad any attackers or cyber criminals who knows the ip address of this api server will be able to gain privilege control to the cluster. Now, let's look at. Let's look at the second component that we focus on copy.

B

Kubilet works similar to api server, but it can control because sync only a single worker node. It can control only the containers and parts in the network node by default. An authenticated request, central copulate, are mapped to system, anonymous user and c our system. uh This user is mapped to the authentic authenticate group, so this user in group doesn't have too much doesn't have a lot of permission in the cluster itself.

B

However, by default, copylet accept anonymous request, so anyone can anyone who can communicate with copylab can send requests to it, and this is a problem by default. This copylet allows anonymous access.

B

What if someone accidentally exposed this kubelet port to the entire internet is again a very common misconfiguration we have seen, although the misconfigured copylab is not as bad as misconfigured api server, however, a misconfigure could be. Let allow an attacker to gain control to a single worker node and from this single worker node.

B

They gain the initial access and can start to uh move later, move laterally to other worker nodes in the cluster, as we will see in the hill that are male 1 that we we found now, let's look at how we can how we find uh real world attacks against kubernetes. We need to find we we at the time we started the research. We have the hypothesis that kubernetes will be attacked. Similarly, in the same way as attacker attack, dr daemon, but we didn't have any proof.

B

We didn't have any data, so we have to find our data or create our data. So we lay out two different research methods. We proactively a project method and the reactive method for the proactive project method. We periodically scan the entire internet for misconfigured kubernetes and for the reactive method uh as we'll talk about in a few. In a few minutes, we've deployed a few is intentionally misconfigure kubernetes honeypot, to attract attackers and to learn what are the?

B

What are the technique, techniques and procedures of of these attacks? So one advantage of having these two methods deployed at the same time is that they can actually cross verify with each other.

B

Our our our kubernetes scanner should be able to find our misconfigured kubernetes honeypot and on our kubernetes honeypot, you should be able to pick up the scanning traffic from our scanner, so this is how we actually make sure these two different methods or work uh as they as we expected for the proactive research as much as we would like to scan the entire ipv4 space for ourselves.

B

It is not that it is not that easy. Actually, as soon as I started, to scan the internet either my isp or the cloud survivor or my service provider send me a warning. They typically don't like any of they don't like their customer to scan the internet. This is scanning traffic. Is usually considered malicious, so I went for easier, easier path instead of scanning the internet. By myself I use the internet iot scanner such as census and children, to help me at least identify the potential misconfigure endpoints.

B

So I use I. I use these two services from census and children to to collect like data from them every week and do some data aggregation and filtering to come up with a list of potential uh kubernetes, node or potential port that run could be api server or quibblet.

B

Then I do the ipad. Then I start to create those nodes to see if they are misconfigured. And after that I did some course analysis. The result is what I'm going to show in the next few slides.

B

Overall, in the period of three months, we identified more than 2000 unsecured, kubernetes cluster that consists of more than 5000 nodes and within this 5000 now there are more than 31 000 cpus that attackers can attend, can potentially uh abuse.

B

And within this 5000 note, there are more than 75 000 active, fast running and one of the biggest clusters that I encounter during the research had more than 500 nodes and more than and more than 2000 active parts.

B

And that said that, that's that's a very shocking finding and also said because I, if an attacker.

B

Found this cluster, they can not only steal data from from that kubernetes cluster. They can also start to deploy a lot of malicious payload in those in in this particular cluster.

B

So the cluster that we so there's no way that I could individually identify the owner of each misconfigured kubernetes cluster. But for a few that I could kick into, I can see that because they belong to e-commerce, finance or even healthcare providers, and the figure here showed the overall findings throughout the research we overall came across more than 37 000 containers and except for this 47 different image and in this cluster they expose more than 83 000 secrets that can be used to access applications or parts or or even the host, and each cluster expose.

B

Each misconfigured kubernetes cluster expose different set of api server, different set of apis so because, due to the misconfigured row based access control, every misconfigured cluster can be misconfigured differently. For example, among all the misconfigured foreign clusters that we encounter. 18 of them expose the the path api. They can list all the paths and read all the parts in those classes, and only third and only 13 of the cluster expose the secret api overall, uh china and united states.

B

These two countries add up to more about 75 of the misconfigured kubernetes cluster. This is kind of expected. These two countries are where most of the network, infrastructure or car infrastructures are hosted, and these are some common. These are some malicious activities that we have seen in those misconfigured kubernetes.

B

Most of the malicious activities started from deploying and official and science doctor image such as ubuntu send always or david. When attackers deploy this type of official and site image, they are generally trusted by the kubernetes platform. However, after deploying this image, the attacker then deployed the malicious payload inside this container.

B

Most of this payload that we have seen involved in jacking operations. Some of the payload may also attempt to uh infect other nodes in the cluster, such as heal the guard that heavy will expand in a few slides. Now I will pass to a bit to talk about the reactive research. What we found in our honeypot right.

A

Thanks jay I'll take it from here. So let's talk about the reactive research, so basically the the reason we did. This reactive research is to see what is the frequency of those attacks and to see if there are any other attacks and if they are, then what are their impacts.

A

So we deployed two types of honeypots for around uh six months and the first one is misconfigured api server, whereas the rbok is misconfigured in a way that any user that will access the api server will have uh admin privileges.

A

So that's really uh critical and the second honeypot is misconfigured cubelet on this. In this honeypot we just deploy cubelet with anonymous access aloud and expose it to the internet so that anyone that will access this cubelet will have permission to do anything. He wants on that specific node right. So, let's go over to their results uh for the api server. It was really kind of disappointing because we saw zero uh exploit attempts.

A

But one interesting thing that we saw is that census uh scanned us really in depth and when I say in depth I mean that they query the api server for nodes for quotes.

A

For uh for a role for roblox for a secret for anything that the api server had to offer and for those those of you who doesn't know what census is, so that's a search engine, they just scan the whole internet, put it in a database and allow users to query this database, so our honeypot for sure is being in senses but other than that. Nothing interesting happened over there, and so, let's go to the second part, to the cubelet honeypot, which was really excited with exciting. uh We got exploited numerous times by different adversaries.

A

uh Most of the attacks were for crypto jacking purposes, and one of the attacks included a really interesting mallard. They did a lot of mess, so we decided to explore it, and then we found out about hildegard. So that's the name of the malware. We decided to call it hildegard uh as there were. There was a spring in the in the binary which was hildegard, and this malware is just crazy. It's the first ever malware that is uh designed to attack cubelets and to propagate across the cluster and we'll talk about how it operates.

A

So, first of all, the first thing that happens is that the attacker exploits an unsecured cubelet and, after that he deployed a container uh that includes first, a teammate client, which is an opers which is a tool that can be used as a reverse shell. And this way the attacker can run commands on the on the node and the second thing they do is they deploy a mass which tools and this tool is ca.

A

This tool scans the internal network of the cluster and try to find other cubelets and when it does, it exploits them as well. So hildegard just tried to propagate to all of the nodes in the cluster and on every node. It deployed first, a crypto miner, which mine monero and a irc client that will connect to an irc server owned by the attackers, and this way the attackers could send commands to the containers and do whatever they want on the cluster.

A

So that's how everything works basically, but we wanted to talk a little bit about the techniques they use because we think it's really worth talking about. So, let's go over and talk about through escalation and hildegard deploy two open source tools.

A

The first one is pirates, so this tool scan for cloud credentials and cloud and in the cloud metadata server whether if it's aws, gcp or azure- and it also looks for uh service account tokens or secrets and when it's found when it finds one of them, they just try to escalate uh to get higher privileges.

A

The second tool is pot b. I hope I pronounce it good. Well, it's break out the box, and, as the name implies it will try to break out of containers. It will try to use known vulnerabilities. uh It will try to find if there are any uh capability or cisco that are. I are in there by mistake and try to exploit them. They'll try to access docker demon on the host, as we talked about in the first slides of the presentation.

A

They also try to find some cloud and kubernetes credentials and even look at the environment variables for credentials and other than that. Hildegard by itself has a script that tries also to find credentials and it will scan the cloud provider metadata server. They scan for ssh keys, they scan for docker credentials and, of course, and foremost they scan for kubernetes service accounts token. So you can see the script uh below that's how it does so all right and to my favorite part, which is defense evasion.

A

Well, hildegard uses two types of techniques of defensive asian, the first one is they use ld preload and we won't get into what is at the preload, because that can take like uh 30 minutes, but we'll just say that hildegard just hijack some lipsy functions and while hijacking them, it replaced them with other functions that basically does the same as the normal functions, except the fact that they uh sanitize the output, and by that I mean that this, how kindergart hide itself.

A

So, let's say, for example, we run ps, so we won't see pierre, let's see hildegard in the ps and if they run ls, we won't see the hildegard in the alice and if we wan run top to see the cpu research, we won't see the uh the superior siege because hilda, because the the ldpro function will show us something else, and the second part is the encrypted elf. Binary and hildegard uses a well-known, rc client in order for for him not to get detected, they encrypt it and decrypt it only when running into the memory.

A

So, in this way, static analysis tools cannot be effective right. That's all for hildegard! For today, uh jay can you take it from here? Thank you.

B

Aviv now, let's look at how individual can protect and defend against uh cyber criminals. This threat exploits its configuration. 99 of the known attacks exploit its configuration, like we only look at the api server, the misconfigured api server and software copyleft, but us, as we mentioned earlier. This is just the tip of the iceberg.

B

To our biggest surprise is that we didn't see any attacks targeting our api server honeypot. We think that attackers haven't really moved from attacking darker demons to kubernetes.

B

Kubernetes is much more complicated than dockers and, however, the potential computational and storage resource, as well as data in the kubernetes should be, can be much bigger and more juicier than this configure darpa daemon. So we think this can be just the beginning, and there we expect to see more and more attacks against kubernetes in the near future. What are the best? What are the uh strategies to protect your kubernetes kubernetes infrastructure?

B

Best practice follow the best practice you should keep you secure, really secure. The layers of defense, built into the kubernetes platform, provide very strong security, such as namespace isolation, network policy and second secure your network and identity.

B

As long as you make sure you don't accidentally expose your kubernetes components such as api server, copyled proxy fcd, you should be safe from 99 of these attacks and also secure your identity. Make sure you follow the principle of this privilege, minimize the number of users in your users and groups in your kubernetes and continuously monitors the permission provision to these users and and applications finally patch frequently patch. Not only your application but patch.

B

Also, your kubernetes well kubernetes is complicated, involving a lot of moving parts and when you have a cluster of hundreds of worker nodes, it's not easy for a single person to oversee them. This. You can never be 100 sure that you don't have any insecure or accidentally introduced any misconfiguration.

B

So if you are not so comfortable to provision and maintain the coordinated clusters from scratch, use many community services, such as aws, aws, eks or azure aks cloud service providers can help you manage and control the control plane, and you only need to manage the application running on top of the kubernetes clusters, and also most of the cloud service provider, have some sort of gun rail to oversee your kubernetes cluster, to make sure that you don't have any misconfiguration next one ephemeral workload design.

B

So if you have an ephemeral, workflow design, meaning that your comp, all the cards and containers in your cluster, can can sailing, they can be killed and restart anytime without losing uh losing anything. So if your container infrastructure or micro service applications are designed this way, then attackers don't really have any way to gain, uh get initial or can any getting the foothold inside your application.

B

As long as you see a containers drift from the from the normal from the normal or benign state, you can simply just kill and restart the container, and you start from a fresh and clean state again and finally, continuous monitoring is also important, use tools to check configurations of your cluster and use tool to check for suspicious workloads or traffic.

B

Again, although your infrastructure is your application, may be vulnerable. Your zero day or new vulnerability may be found in the application deployed in your clusters, and this attacker can can attack those applications. There may be a chance that they they can escape, break out the container and start to attack your uh kinetic, so continuous monitoring to the to a kubernetes cluster is important, and this is the end of our presentation. Thank you for being in our session. If you have any questions, feel free to reach me, jay can call autonetworks.com for habits.

B

This is a as a session, it's his email at colorado network and we also publish the findings in unifor. Tubebugs feel free to go to the university.hellonetworks.com to see more of our research.