Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Prepare to Be Boarded! A Tale of Kubernetes, Plunder, and Cryptobooty - James Condon, Lacework

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Prepare to Be Boarded! A Tale of Kubernetes, Plunder, and Cryptobooty - James Condon, Lacework

How are Kubernetes cluster’s being compromised in the wild? There aren’t a whole lot of public reports detailing successful attacks against Kubernetes clusters. The goal of this talk is to demystify these attacks and provide recommendations to prevent them. In this talk, a successful attack on a Kubernetes honeypot is analyzed. The amount of time it took for this to occur is quite surprising. Next, using this information, the scope of research is widened to survey other clusters that have fallen victim to the same attacks. Multiple cryptojacking campaigns emerge and details behind the methods of the attackers are shared. After providing statistics on these attacks, recommendations for prevention along with indicators of compromise are provided.

https://sched.co/UacH

A

All right well, thanks for joining me today. This has been a really cool conference, a lot of really good security talks, and so I'm really excited to be presenting here as well. So before we start has anyone ever encountered discovering pods in their kubernetes cluster that are cryptocurrency mining?

A

Okay, we got a hand there okay couple hands for those of you didn't put your hands up. That's a good thing! This talk is about looking at some of the thread actors behind some of these crypto jackin campaigns and how they are taking advantage of misconfigured Cabernets clusters, but before we dive into it, we have some terms and definitions to clear up. So this is crypto booty, crypto currency obtained from illicit coin mining and a kubernetes cluster, so I'm gonna need everyone's help in making this a household term a.

A

Little background on myself, that'll give some context for this presentation: I'm, not a software developer. Our DevOps by trade, currently I'm director of research at a company called lacework, we're a start-up in Mountain View, and we do cloud provider account security, Linux workload, security container and cabernet security. I've spent my career in InfoSec started in the Air Force. Then I worked for an instant response, firm, called mandiant and then before lacework I was at another startup called protect wise and there I'd ran our threat.

A

Research team, so primarily I've spent most of my time doing, threat, intelligence and analysis and forensics, particularly focusing on the network side of the house, now kind of pivoting and cloud security. So if these topics are interesting be sure to follow me on Twitter, you can also send me emails check out our blog as well.

A

Alright, so I have a few different goals for this talk. So, first and foremost did some research earlier in the year about major Cabernets components that have been exposed to the Internet. So we're going to dive into a brief overview of that research.

A

Then we're going to look at an active, crypto, jacking campaign that we caught on one of our honey pots, then we'll take a detour and talk about some of the active thread, actor groups that are targeting cloud resources and how that relates to kubernetes, and then we're going to tie this all together with the previous research to kind of get a sense of scale that some of these attacks may be operating.

A

So when I first started doing some research around kubernetes, a common theme and anything in the cloud is misconfigurations. So generally, this occurs from things like expose. S3 buckets, MongoDB open to the world. I feel like you hear about once every week that there's this new database that's been found with a bunch of sensitive information. So when we looked at kubernetes specifically, we wanted to get an idea of what kind of scale are some of the internet facing Cooper neighs components.

A

So to do that, some of the major components that we can see that accidentally exposed to the Internet are kubernetes dashboard, the API server, both the secure port and the insecure port at CB and cubelet. So using some of the mass internet scanning tools like show dan and census, we can gather some stats around how frequently this occurs. Cubelets, the only one we didn't dive into a lot of these scanning services didn't have anything that would pick up on cubelets. So we just focused on the dashboard, API and sed.

A

So if you haven't done this before, you might have some questions on like how do you discover open dashboards, open api's, so I just have a quick little demo, so you can see what it's like so things like census. What they'll do is they crawl the internet? They look at open ports and if they get any information back from crawling, those they'll go ahead and index that and then security, researchers or companies looking to protect their infrastructure can use that to see if they have exposure or other other types of research.

A

So when we look at the communities dashboard, which is kind of a starting point at the end of 2018, there were multiple reports of major companies having exposed dashboards that resulted in things like crypto, jacking and possible access to the account the cloud service provider. We focus our research there, so defined dashboards. One of the ways we can do this is we need something unique that would be returned in the crawling search results. So for our dashboard, its HTML, we can use, there's a unique HTML element and a Canary's dashboard and that's just kubernetes dashboards.

A

So this is what it looks like just searching, coronaries dashboard here.

A

This gives us a result of any IPS that match that search, and then we can go ahead on click on one of these guys and take a closer look, this dives into some of the open ports that they found some of the Whois information, and then we can actually go to the site. So in this instance we end up discovering there's a dashboard here.

A

This was in one of our first clicks, so what we end up doing is we use their api's to go through and verify of those IP results, how many of those are actually dashboards.

A

So after we wrapped up this research, what we ended up finding is on any given day we'd see about 500 dashboards that have been exposed. We use show Dan to look at TD clusters. Showed n actually makes it extremely simple, there's an interesting blog post that was written and it's in the resource section of these slides, but within it they describe how that you showed an look at a bunch these at TD clusters and within the first three found evidence of credentials that were to access like AWS.

A

So just because we see an STD cluster, it doesn't necessarily mean that it's associated with kubernetes, but a lot of times there would be metadata associated with the naming schemes that was kubernetes themed.

A

The other we look at is API server. So specifically, there's the secure port, which is essentially how the API server operates using TLS goes to the regular authentication authorization. This is the lower risk item of these four, but there's still some sort of risk of exposure there and then probably the highest risk item is finding over 600 API servers serving on the insecure port if you're not familiar with that, that's served via HTTP, and that goes past. Your authentication and authorization modules. So anyone with access to this can just compromised your entire cluster.

A

So kind of the next step here that we wanted to take is okay. This is interesting. We've got some stats on these miss configurations. Usually it's surprising because you know you wonder like okay, how do you have the insecure, API server expose the internet like how do these missteps happen, but then the next big question we have is like is anyone actually specifically targeting kubernetes? Are they going after that?

A

As you know, one of the platforms and applications that they commonly target, so we wanted to figure out based on this exposure, if that's being exploited, one of the key questions is you know if we set up a honeypot, that's wide open in the world. How long would it take to actually compromise it? If you set up, say an old version of Redis, that's vulnerable to critical CVEs that a remote code execution, you can probably bet on that being exploited within a couple hours to 24 hours.

A

The other thing we want to know is, you know, what's going to be the best way to actually set up this honeypot, the easiest way that we can do it and also minimize the blast surface in case you have any issues and then didn't find any major reports detailing these type of attacks.

A

But is there anything else we can kind of surface up, so I came across this blog post and it's titled an overview of micro Kate's, the tool to QuickStart kubernetes cluster, and why using it in the cloud, was a terrible idea so background if he owned oku Bernays or if you don't know my croquet eights, it's a single node career nice cluster, they're very clear on the website about it being used for local, local lockdown environment.

A

It's primarily used for development prototyping, these types of things so using it in this way kind of goes against how it's supposed to be used. But the author details how he went ahead and did this and then after some period of time he saw a bunch of pods that we're doing some cryptocurrency mining.

A

So here this is a listing of the pods on their cluster and you'll. See a bunch of these weird named pods and you'll see this as kind of current theme and some of the other areas that we looked at and just so you know this blog.

A

There was a lot of comments about a lot of people work, so my kate's are like hey you're, not supposed to use it like that, like what are you doing, but at the end of the day, there's a PR request to make sure to bind that to a local host I.

A

Found another blog post that was about crypto jacking, but it was, it was focused on Drupal and then someone chimed in in one of the comments and they're like hey I saw something really similar to this, and it was micro, Kate's and I threw it on the cloud and it got compromised right away.

A

So if you take a look here, what we have is there's a couple of curl commands that download some files and then the files are ultimately executed. This is really common theme that you'll kind of see, as we take a look at the thread actors that are actually behind this, then something funny I thought is in conclusion: do not install micro Cades with its default configuration on a server exposed to the Internet. So.

A

Now we have a pretty good idea of how we want to set our honeypot micro. Kate's looks like it's a good one to use so to do this, all we all we need to end up doing is spin up in a bun to server install micro Cades via enable the kubernetes dashboard check, make sure that that insecure API is accessible and expose Internet traffic to it and then put up a few tools to monitor, what's happening on the host and then disclaimer.

A

You know micro Cades isn't supposed to be used like this, so use that at your own risk, so my expectation from reading some of those blogs is that this that there would be an attack. You know occurring somewhat immediately within 24 hours, maybe 48 hours, but all we ended up seeing initially is just a lot of internet scanning. In fact, we would see a lot of exploit attempts for completely unrelated applications.

A

I was about to give up on it and then, on day 31. We finally saw some activity and my hypothesis was you, know insecure API or the dashboard would be the infection vector, but we didn't really see any interaction with the dashboard, so that was kind of interesting.

A

It's right here. This is just a Wireshark screenshot of traffic to the insecure port for the API server. You can see that this is in clear text HTTP and the responses are with the API pass and then on day 31. We see a new user agent and that's cute control. So now we know that someone knows that this cue Bernays and they're actually kind of diving into it in interacting.

A

So what the attacker ends up doing here is they enumerate all the API pass and then eventually we see this post request, and so what this post request is doing is creating a replication controller, which we thought that that was kind of interesting as opposed to using something like a replication set. And then, within this replication controller we see five replicas, we see them use CentOS image, and then we see that same kind of chained, curl commands where here they're downloading the files and then going ahead and executing them.

A

One of the things I thought interesting about this is I assumed that they would probably pull down a backdoored image or an image of a miner said they use a clean image and then install after that.

A

This is a closer look at some of the pods, and now we start to see some overlap with some of the previous reporting. One thing to know is we see this container named. My res do two that comes in a little bit later, but we're starting to see IP addresses that are these same ones that we saw in previous reporting and then something to also point out. Is these files are being pulled down? That's XM, reg and XM. Reg is open source from narrow, minor, that's popularly used by the bad guys.

A

It's also used for good and then that configuration file is just a configuration file to set up the mining pool some enero address, etc. This is what that configuration file looks like so highlighted.

A

There is the Monaro dress and that actually ends up being the same one that we saw in one of our micro, Kate's, blog Commons, and then these IP addresses are some private mining pools that the attackers are using also seen in those comments, so seeing a lot of overlap and a lot of ties here so now that we've captured a successful attack on our honeypot and we're kind of tying a few of the pieces together, we kind of need to take a little bit of step back and talk about some of the attackers.

A

So if you're not part of the info stack world generally, we think of thread actor groups and a number of different categories. These are three of the most predominant ones and usually they're separated by what end objectives are. So in this case we have criminal groups they're after financial monetary gain. It's this things, like you know obtaining credit cards, selling, PII ransomware and what the groups are specifically targeting the cloud. In that case, it's a cryptocurrency mining, specifically Manero, apt, you've, probably heard of apt before that's advanced persistent threat.

A

These generally are groups that are associated with a nation-state and carrying out their objectives. This tends to be economic espionage. This also tends to be stealing proprietary information and then, lastly, we have our hacktivists. These are politically motivated groups and generally, what they're trying to do is disrupt or deny services to their targets.

A

So for the interests of this presentation, what we're actually going to be looking at is within those criminal actor groups, specifically criminal actor groups that are targeting cloud resources and they have a lot of things in common one they're primarily focused on mining Manero right now they have a similar set of attack chains. So generally, this starts off with scanning the internet for vulnerable services. Looking for applications that might be vulnerable to some remote code exploits or brute-forcing passwords for other applications.

A

Generally once they get in a common method that we see is an install script is downloaded. This is usually like on a Linux system. This is just a bash script, and the bash script usually takes care of a lot of work. It does things like downloads further binaries. Usually those binaries will be something like a rootkit that downloads modules, those modules might be things like: a propagation module to move laterally or something like X M ray to actually mine the cryptocurrency.

A

Usually, these scripts will also establish persistence so on Linux OS. This is typically done through cron jobs that repeatedly download the install script. Sometimes this is through setting up services, one of the most fascinating parts and I have another talk on it. Is they actively target other threat actor groups and try and kill off other miners if they're present, so there ends up being a battle, especially on these really easy to exploit hosts where they try and kill their processes sinkhole their infrastructure?

A

And things like that, so there's kind of three main groups that I want to highlight: there's the 8220 mining group: this is a Chinese speaking thread actor they've, also gone by the name. 8220 gang they've been active since 2017 their methods, typically composed of hosting malware and things like pay. Spin and github they've also been known to backdoor docker images. We see a lot of malicious shell scripts.

A

Elf, binaries primarily use XM rig to do their actual Monero mining use this tool called process hider to achieve rootkit functionality on servers that they compromised now their main targets are generally web applications, but something interesting. The note here is: we've also seen them targeted, docker, so open, docker api's. This is something we haven't really seen. A lot of. Some of these other groups do some characteristics. Unique to them is that they're c2s often use TCP port 80 to 28 communicate.

A

Hence where the name of the group came from a lot of their malware, they kind of keep with the same name. So generally, we have a botnet a lot of times. What you might do is continually not really change that naming scheme and that's how some of the other groups can actually target the processes that they want to kill.

A

They do heavy use of a lot of the domains that they use our dot tkt. Oh these, that kind of makes them stick out, but they got their origins from a github repo containing a list of coin mining tools called what miner so apologies for the pixelated image, but this one's kind of funny, because it doesn't disguise what it's used for so it straight up, says: collecting and integrating all different kinds of illicit mining malware, and so one of the other things I'll point out too. Is you see these folders Kay worker, D and sistas?

A

That's generally some of the naming schemes that they use for their malware as well.

A

So next is rock.

A

If you're familiar with CrowdStrike, they have these really cool characters of the different thread: actor groups, since there aren't really ones for the cloud ones right now, I'm using this of the rock. So this is another Chinese speaking threat actor they've also been referred to as iron group or system ten active since about 2018 a lot of similar methods to 8220 mining group.

A

A few different changes there, things like using hf s and am is but a lot of the same kind of methods early on they're, targeting pretty heavily both windows and linux, and they were doing a little bit more ransomware at the beginning. We see them coding their malware in a variety of languages such as Python and go as well.

A

A lot of their targets are the same, but I haven't found any reporting of them targeting docker kubernetes, specifically a couple things that are unique to them is: we've seen a lot of kind of they're adopting new tactics as they go so originally they would use, you know, github to host malware, and then they started using their own infrastructure host malware and then most recently doing new things like using c24 dns, so seeing some changes there.

A

Probably the thing that was most interesting to me is there was someone that reported that they were specifically uninstalling cloud security tools. This came out around March 2019, and this was like a V tools associated with Alibaba and tennyson.

A

Now they have a really interesting beginning because they're reported to a forked the what minor, repo that 8220 mining group did and essentially all they did, is forked it and then I replaced all the infrastructure with their own. So there wasn't a huge deviation there. The name actually comes from a miner Gate wallet and login that rock at live dot. Cn.

A

Last group to highlight don't know as much about them. There's a company called in Azure, that's doing a lot of reporting, so these guys are pretty interesting for a couple reasons: they're another Chinese speaking thread actor first reporting in 2018, pretty similar methods a lot of times. They're malware will do some of the things that normally the other groups. Shell scripts will do similar targets, but something I found pretty interesting. Is these guys heavily target rock? So, for example, one of their pieces of malware?

A

Has this whole list of IP addresses that are associated with rock infrastructure and then what it does? Is it sinkholes those on the host? So we see a lot of targeting of them specifically. The other thing is they also seem to kind of adopt some of the rock tactics pretty quickly, so they also were observed shortly after Rock, actually doing in uninstalling the same cloud. Security tools for Alibaba and Tennison.

A

The malware analysts say in Azure have also reported that they use a little bit more advanced techniques for some of their Linux malware.

A

Okay, so now that we've kind of got a good idea of who some of these threat actors are what some of their objectives are, and we've talked about, how we have these different, miss configurations that are open there, we've seen how our own honeypot has also been compromised. What we want to do now is revisit that original research and kind of see what we can do to tie all this together. So.

A

If we revisit our research looking at the insecure api's, so at the time of research we saw about six hundred and seventy-eight different IPS in our census, search that hit on some of the ways that we're searching for the insecure api's. This was mostly on TCP for 80, 80 and 443. So that's something important to point out using these search engines. If they're. If these were being served on an unusual port, they wouldn't necessarily pick that up. We primarily see these in various clouds like Amazon, GCP, OVH, Tennison and Alibaba.

A

So what we went ahead and did is we took a look at the pods that were running and we found that there's about ten thousand seven 43 different pods and we started doing some analysis around what those pods look like to see if it tied back to anything else that we saw so immediately. We see a lot of these unusual naming schemes this one. In particular. We saw in that screenshot earlier when we were talking about that blog post, that we were reading.

A

We see these common labels and container names, my res d02 and my res DZero one. That's what we saw within the pod specification on our honeypot and then, when we rack and stack these pods, the most popular image, that's being run is CentOS. So we're already starting to see a lot of indicators of compromise there, but kind of the most important piece is looking at.

A

Some of these commands that are used for the containers so found several repeating as five distinct different sets of curl commands and executing a file, and if we kind of take a closer look, we'll see that this IP address that XM rig and its configuration file are being downloaded from was the same IP address that we saw in our honeypot.

A

We see that this one is being downloaded from an IP address over TCP port 80 to 20, and that's a pretty strong TTP associated with the 80 to 20 mining group.

A

Another one is: we see this IP address as well with that same TCP, port 80 to 20, and this was also seen in one a micro Cades blog. These are two other distinct URLs that we saw. Interestingly, most of these were coming from. You know hard-coded IP address, but we did see one that was downloaded straight from github. But if we take another look at this, we see some file naming that's pretty consistent with the 8220 mining group, and then we also see again this IP address.

A

That was also in our honeypot kind of wraps up the research as far as doing a full circle.

A

But what we end up seeing is out of that ten thousand there's four thousand four hundred and fifty of these pods are doing illicit coin mining specific to that description, that we showed we're still doing a little bit more analysis on this, because it looks like there's other and there's multiple campaigns going on here, but just for this one particular group about half of those pods have been in mining Manero for them.

A

So just a couple final thoughts here before we wrap up. You know the real purpose of this research is to illuminate misconfigurations that occur and the attackers taking advantage of them and really the main purpose of this is so we can share this information, so operators can use it to avoid falling victim. So if you ever come across activity like this, you can always reach out to me. We can give you some information on. You know what we might think it's associated with, or if you have concerns that you know.

A

Maybe you have some exposure and you want to learn a little bit more. We can help out there. This also gives us confirmation of criminal threat actors specifically targeting kubernetes, and that was kind of one of the goals for starting out is, you know, are people really targeting coronated, specifically in this research that we did? We have.

A

You know it's pretty high confidence that this is probably attributed to the 8220 mining group, and I think we didn't really talk too much about this, but if any of these actors were actually to do any of this manually as opposed to doing it completely automated, you know it's possible that kubernetes cluster may contain. You know secrets to something like AWS that they could pivot into and then have much more resources as far as we know that hasn't been occurring, but we don't know for sure, because we're just kind of looking at the surface here.

A

So with that, I have a number of resources. These I uploaded this to the schedule also put on our blog, but there's a lot of cool articles that go into depth about some of these thread actors and then from there. We'll do any questions.

A

Yeah and then, if you could come up to the mic, oh.

A

That's a great question: I wish I knew the answer to it. That's pretty good I need to figure that out, so I can work that in yeah, so I got that screen, shot. I think it was I. Think Palo Alto had a report. Maybe a Cisco Talos I was doing like a digests and some of these groups, and they were talking about the origin of it. Yeah I was pretty good any other questions.

A

Okay, so a question is what currencies are they mining? We almost see exclusively Manero I, actually haven't seen any Bitcoin. Occasionally I've come across a few reports of some currencies that I actually haven't heard of some off-brand and weird ones, but one of the other things is when they mined Manero. Manero is a really good one for them to use, because it helps them in their transaction remain a lot more anonymous, but they tend to use their own private mining pools and then those get Reshard a lot. So it's a little bit easier to track.

A

We usually don't see them using public mining pools, which I think is kind of interesting.

A

Anybody else all right cool um feel free to email me anytime. If you do come up with any other questions or want to learn more thanks. Everybody.