Cloud Native Computing Foundation Online Programs, 27 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Defence strategy against Kubernetes attack TTPs (Tactics, Techniques and Procedures)

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Hello, everyone good morning, good evening, uh welcome to the webinar defense strategies against kubernetes attack, ttps.

A

uh My name is manoj and I'm a threat researcher with tigera, I'm currently focusing on kubernetes and cloud attack surfaces.

A

uh My background has been vulnerability: research, malware, reverse engineering and exploit development at a point and my previous roles were at juniper network and intel technologies where I looked at their security posture.

A

So uh today's agenda is straightforward, where we will look at the attack surfaces available to external threat actors and how uh each external threat actor, which is active right now in the wild, is targeting them and what are the mitigation strategies? We can use right now to uh stop them.

A

So uh current uh trends uh from the kubernetes perspective is uh the the external threat actors are interested in uh mining cryptocurrency and this mining activity kind of picks up uh when cryptocurrency uh price comes down uh and and to to mine this cryptocurrency.

A

uh These attackers kind of try to find uh the misconfigured docker apis kubernetes api to access available compute uh on the internet, and for that this can uh entire internet to see if they they can find anything and once they find such misconfigured apis, they do uh do they do get that initial photo. They do privilege escalation and move laterally, and in some cases we found they they do exploit unpatched cves uh so that they they get that initial foothold easily.

A

So uh with that, this is the threat matrix for kubernetes from microsoft and, as you can see, there are almost 45 techniques listed here and each uh technique can be used to compromise. uh Your container uh attacker can uh uh do privilege. Escalations can uh can uh can move laterally within your containers and do the data exfiltration ultimately using this technique. So today we'll be focusing uh on uh these initial access techniques uh from kubernetes perspective.

A

So on the right you can see. This is your typical kubernetes cluster, where you have host or node, which is running on your cloud hardware, and this is particularly instance and inside that host.

A

You have your container runtime running and you have kubernetes vertical, which is kind of managing all these containers that you would deploy and by configuration, uh kubernetes api cumulative docker api can be exposed to internet and attacker can uh really find out these apis and try to get that initial foothold.

A

If you haven't properly secured it, your services are also exposed out of these uh system, so kubernetes can really expose your services, and also there are services like ssh mysql, which can run on host itself and last but not least, attacker can really make use of uh the malicious images that are hosted on public infrastructure or public repositories like docker hub, where you can end up pulling one of the compromised uh or or image uploaded by attacker, and uh you your container can get compromised because of that, and, ultimately your cluster.

A

So this this is the attack. Surface, usually is targeted by uh threat actors currently uh to get that initial foothold.

A

uh And this is just a point in time. Just uh before this webinar, I tried to see how many uh docker apis are available and how many cubelet instances, which is a kubernetes process running on node, is exposed on internet, and I almost filed found 500 entries for each and palo alto did a study a few months back where they almost found. 50 000 containers publicly accessible, and we do agree with that number. As each kubernetes cluster can run multiple containers, even the docker api can run.

A

Multiple containers and numbers can really add up really fast.

A

So, uh let's begin by looking at our first external thread, vector which is team, tnt and team tnt came up with its new malware, which is hildegard, and this particular malware is discovered by palo alto, uh and uh this is targeting a cubelet api to be specific to gain that initial foothold into the kubernetes, and this api has to be unsecured in order to get that initial photo and they did the same thing for docker interfaces in the past and they once they get such apis.

A

They usually drop bunch of shell scripts and do the initial installation and the intent and focus for this uh particular attack vector is a monero coin mining and they have a bunch of scripts which can uh look at your cloud im, aws or ssh keys, so that if there is a chance to expand their presence, they would be able to do that and they have typical cnc capabilities, uh interesting evasion and ddos capabilities.

A

So, as we started looking at this threat actor closely, the first thing we found was their root key and this rootkit was compiled dynamically so that they would they wouldn't be uh targeting specific linux architecture.

A

Rather, the binary should be able to run on as many linux architecture or flavors as possible and they kept the dependencies minimum, as you can see in the picture, only lip c and live dl.

A

So with that uh the the interesting functionality abuse is ld, preload ld preload is a linux ability or linux functionality where, if, uh if you are running any process inside a linux- and you enable this ld preload functionality, uh the binary defined by ld preload will be loaded into that process. Memory uh and- and this will be enabled globally, so any process you run, it will be uh adding the library or binary uh kind of showed by this ldp preload parameter.

A

An attacker is exactly doing that. He's uh loading his he as soon as uh he compromises, or they compromise uh this uh particular container. They set this uh ldp road parameter and, after that, any process that anyone runs.

A

It will load their malicious uh library into it, and what this malicious library does is hooks into read and read. The 64 system calls and what this system calls do is if you gonna do uh ps or ls.

A

It's gonna give you a list of processes or list of files on file system right, so uh attacker will be able to always hide the files or processes it wants. So, if administrator goes into a container and does what are the processes running here, he won't be able to see healthy guard processes, running or crypto mining mining processes run by this particular malware and.

A

And as we looked into this closely, so they kind of have all the symbols already available inside the binary they didn't strip any. So we were able to look at the code pretty easily, and here in the red box, you can see see that they are directly making a call to a particular symbol to locate, read the r64 and and they are able to override the dr-64 after that. Just to show you how it looks like so uh at initially in this.

A

This is a particular container where we are just running a ps command and you can see there is a netcat process which is running here and after that we just used ld, preload and loaded our malicious binary, which kind of hides this netcat process. And after that we did this ps command and, as you can see in the output below, there is no netcat binary.

A

So that's how this particular technique works and also team tnt used already available uh part b uh red teaming two and what this tool does is it can abuse uh c groups if you have a privileged container. So if team tnt ends up breaking into a privileged container, they can basically abuse c group and run the commands on host.

A

Here we use his body framework and just did the hostname command and, as you can see in the red, this is the name of the host. Rather than name of my container and after that, we in in next red box, you can see we were able to read the files on the host escaping the containers, so uh these threat actors kind of here what we saw is they used ready to use third party tools and integrated into their malware scripts. So for command and control.

A

They used teammate and irc as a cd startups uh and as we talked, they use ld preload technique using lead process, hider and uh bot b and p periods for container breakout and credential harvesting and for dns monitoring bypass. They simply override the adc host file and they have a lot of scripts to scan the internet so that they could expand their presence once they compromised uh your infrastructure and the intent seemed to be the same as we discussed earlier, uh a monero coin mining.

A

uh So this is the uh these: are the basic capabilities of this particular thread actors? So, let's move on uh to next one. uh This is skincing, so kinzing targeted uh the docker api and we didn't see that kingston upgraded and targeted kubernetes api, uh but it used another interesting technique where it kind of uploaded its images to public infrastructure that is docker repository and and users. I think at some point there were millions of users which were pulling down uh images uploaded by them, and this this is uh uh in.

A

In our view, the kinsmen keith was kind of advanced compared to team tnts, and but the methods were almost similar and what we found that this skin sing malware just use.

A

The uh uh project which is already available on the getup and focus seems to be similar, uh the coin mining command and control and encryption and the evasion.

A

So uh when we looked at this particular malware's root kit closely, it was similar to tim tnt's. They use dynamically linked uh binary to target as many flavors of linux and architectures, and this is comparatively advanced. As you can see on the right, there are number of functions this particular malware embedded inside their rootkit and all these functions they could do.

A

They could hide process, name, tcp, ports, uh directory names and file names, uh so they were not limited to uh process names, uh as we seen earlier, and uh the techniques were same uh that we discussed and deep reload to hook. This uh particular system calls, and additionally, these guys use encryption to hide their uh hide their uh modules, and this is a simple xr encryption.

A

So uh this is one of the image which was uh uploaded by uh kensing operators, and this is uh this is the this is. This picture is shown uh offered two cord die which can read the container images and each layers and show what were the changes in each layer and, as you can see, it has a monero coin miner, which is being downloaded here in this particular layer, and this particular image had over million hit when we uh kind of uh got this particular image. So almost a million compromises very, very dram.

A

And our another threat, vector from kubernetes perspective, is doki and doki is a back door, and this is a particularly a binary which was uh previously undetected with and broke partner.

A

uh The the reason to mention this because it uses a domain generation algorithm to contact, though this is not a new technique, but they used a really novel djsc in their operation and they use the dodge coin. Api and dodge coin is, uh you may be familiar with this? This is a popular cryptocurrency uh cryptocurrency, uh which is available on the internet.

A

And if you are not familiar with domain generation algorithms, so this is really simple where uh attackers malware, which is running inside your compromise, cluster and attacker, which is sitting somewhere on the internet, has access to same seat, which is uh date and time or currency, pair or temperature of particular city, or even a trending topics on twitter, and they will use that seed and feed it to a pseudo random stream generator and get a random string and append a a particular uh tld to it.

A

Like dot com, uh depending on a day of the week or uh month of the year uh and and query that particular domain and interesting thing is malware, which is running inside your uh compromised container. An attacker would be able to generate same disk and what attacker needs to do is just go ahead and register one of one of the domains uh from there and malware eventually will be able to uh query and contact to that particular domain.

A

And here attackers use the dodge coin wallet, so they pre built a wallet address inside uh inside uh this doki bag door and what they used as a seed was last money uh spent by attacker. So whatever the last transaction amount was that was used as a seat for this particular algorithm and, as you can see on the right, this is the dodge point function that they implemented where they uh queried and called uh the the last money sent by attacker.

A

So with that, let's uh move towards mitigation and how we can mitigate all these threat actors that we discussed right now.

A

So first, first suggestion is use scratch images, because if you use a scratch image, it will contain only dependencies uh which are required by your application and won't have any api to execute or invoke any other binary, which is, uh let's say, drop by attacker, and it may not even have the dependencies required for that particular executable.

A

So this really uh helps where attackers have to be uh very crafty to execute any executable. uh If you can go a step further, you can compile your binary for a specific architecture and link it statically so that there won't be.

A

There will be only one file on your container, which will be running and there is no way attacker can drop another executive bill and invoke it if, if you use this particular strategy- and this really mitigates uh any uh attack that we have seen using uh installation using shell scripts or root keys, that we discussed uh second is use zero trust policies to block access. It is easier to block access, uh increase access to couplet and docker api.

A

uh You can just implement the policy for that, but we would want you to go further and implement zero targets. Trust for uh your not solved traffic. That is, uh that is possible uh with respect to your kubernetes uh kubernetes clusters uh and only white least uh necessary flows going are uh coming inside and you can uh implement that same zero.

A

Trust for your east face traffic, so that if attacker ends up compromising your container or a particular service, he won't be able to move laterally that easily and third thing is block access to external dns. If you are running a kubernetes cluster, the only dns server you should be talking to is a core dns, and uh with that uh on the on the right, you can see that we have particular solutions in blue. That is, if possible, log all your l7 traffic for or l7 data for your service.

A

uh Any processes that is being spawned a container and audit locks for your kubernetes api and have a machine learning algorithms to monitor all these, so that if there is any suspicious activity, machine learning algorithm can notify you of that and last but not least, implement threat beats.

A

They can tell you if attacks the the attacker attacks seen by other members of the community, and you can kind of uh if, if you get compromised in the same way that somebody else has you would you should be able to detect that uh the next thing is, you should be enabling docker contained press uh on your docker apis so that only images signed by you are running on your docker.

A

So it you won't be able to run any publicly available. Docker images, not just you attacker, won't be able to run any publicly available, docker images from unknown sources because of it and uh last suggestion is: have your container isolation policy ready. So when you suspect a particular workload uh behaving suspiciously or there is an issue with that, you can add that particular label of a policy to that container and isolated right away, so that you contain the blast radius for the issue uh and the useful projects are here given by us.

A

Is project calico, of course, to implement all this zero trust that we talked about, and there is a really cool project. Pga intel, if you are logging, dns domains from your core dns uh or within the kubernetes cluster. You can check those uh domains against this lstm based deep learning model to know if it was a dta or not, uh and and that's about uh mitigation uh with respect to uh external threat actors which are currently targeting kubernetes.

A

So if you have any questions related to one of these threat actors or if you have any particular use case, do visit us at tigera, dot, io and let us know so. Thank you and have a good day good.

A

A