Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Bypassing Falco: How to Compromise a Cluster without Tripping the SOC - Shay Berkovich, BlackBerry

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Bypassing Falco: How to Compromise a Cluster without Tripping the SOC - Shay Berkovich, BlackBerry

The explosive growth of Kubernetes has left security professionals scrambling to deploy innovative tools to address the inherent security risks. One such tool is The Falco Project - an incubating CNCF tool for detecting malicious activity at run time. Falco, like many security tools, has some gaps. This talk highlights these gaps by introducing various techniques to silently bypass the default Falco ruleset (based on Falco v0.30.0 release). The attendees will learn 9 different classes of bypasses, 7 of which are novel and have never been presented. The bypasses allow for stealthy target enumeration, privilege escalation and lateral movement. To aid with the bypass automation, Shay will introduce a special container image and multiple code snippets built specifically for Falco bypasses. To wrap up, we will apply the bypass techniques on securekubernetes cluster (presented on KubeCon NA 2019) and demonstrate how an attacker can achieve full cluster compromise without tripping the SOC.

A

Hello, everybody welcome to valencia. Thank you for joining this talk. My name is shay and today we're going to talk about falcon by passing, falco default rule set and perhaps even how to compromise a cluster without tripping the sock. Let's start, let's start with the background. What falco is for and what's container security is about, and this is my view of container security.

A

You can sort of divide it into four areas, so security, cluster security, pre-deployment and post deployment and, of course, how security is important, because if host is compromised, then it's really hard to reason about the workload security that runs on on the on this hubs.

A

Pre-Deployment is about building the base images in an efficient and secure way container scanning, although some people call it a little bit scanning, but I like to be more precise, um scan for exposed secrets, um static compliance, basically everything that we can do with the static image file of the container cluster security, it's all about kubernetes, config and then there's a post deployment and that's where falco is coming into the picture. Runtime detection um there's also runtime prevention, dynamic appliance and prevention is about existing.

A

um Mostly about existing linux security kernel security mechanism that are sometimes given for free in the container context, I've recently listened to the cloud what's the cloud security podcast and they suggested a bit different division uh into areas of container security. So they were talking about build infrastructure and uh and runtime, and that's sort of the way this this maps on to this view, intro, is cluster security and how security built is pre-deployment and the right time is actually post deployment.

A

So if that helps to think about it as well, and we can see that where the falco belongs to runtime detection through the falco architecture, falco is built on top of either kernel module or ebpf sensors running in the kernel space.

A

uh The events mostly syscalls and enriched syscalls are bubbling through the ring buffer up to the user space, where a bunch of library, focal libraries and, more most importantly, rule engine are taking them, processing them and and decide whether the certain rules that are expressed as yaml files, um they, whether the rules should trigger given that event or shouldn't trigger and the if rolls trigger, then one can integrate that with the typical cloud staff like cm or event in some kind of venting engine or grpc, but for our first purpose.

A

We're going to use std out today for the, for the sake of simplicity and you'll, see that in the demo, typical falca rule I took in this case disallowed the ssh connection. There is a concept of macro in falco rule set in this case. It's called inbound outbound. This is really good concept because because it prevents the rule file from bloating, otherwise, if otherwise, it would be way bigger than this, because the macro can be used by different rules.

A

In this case, this lattice stitch connection uses three macros in both outbound ssh port allowed ssh hosts, and one of them is actually has end node condition, which is the opposite of true. So if allowed ssh cost is false, then this rule has a chance of.

A

Triggering and then there's a priority, a mitral attack all kinds, metadata description, so all good now falcals in addition to behavioral detections, they can also be tailored to specific to detect specific exploitation attempts of the cvs if they are constructed properly, and this. This is the example to those those four, um but there are more right now, there's just four uh interesting ones that I thought to mention now. Let's talk about previous work on falco bypasses.

A

This is not the first uh talk about it um before that ncc group did some uh research brett gizman as well. They mostly describe the bypasses around the image. Naming um leonardo um talked about this in one of his talks about the twin cisco's, and I think he also mentioned the simulink evasion is an ongoing issue about the sister calls missing. Sister calls, for example, open opennet, uh exec fee, execute, add duped to this kind of stuff, and then the recent work by go and zen on defcon.

A

They were talking about tacto of file names, mostly um so classic example for the in the openness is called the time of the check of the file name and the time of actual resolution. Of the file name are different so that there's uh that keeps the possibility of the evasion and then there's they talked quickly about the similar incubations all right.

A

So now, with that in mind, let's, uh let's go into the interesting section: um let's start with the read sensitive file and trusted rule, we're going to use it a lot and, let's start with something simple, simulink evasion.

A

The um this is the typical way. The fault is used. We spin container, we do something malicious within the container and falco is um splitting this output or an insensitive file is open because we're trying to read the it's shadow. How can we prevent falca from knowing that we did this? Well, let's create the ceiling to see shadow, and we do see that there's notice this focus still saying something, but it's that's a different rule right, because the previous rule was warning level and right now we're notice.

A

So that means that it's at least degradation of the of the detection um and this by the way this rule the create the monitoring of the creation of the sensitive of the um symlinks over the sensitive files was uh one of the protection mechanism against the tokto attack. We'll see why this is not enough, because, even though we have degradation of detection, which is nice, but we can easier get rid of this rule as well, what can we do? We can create the same link to the unmonitored directory in this case. It's easy to secure.

A

It then we're using the cat to cut the relative path based on the it's, the security link, so one step above and then shadow, and we see the output of the shadow and fireplace silent, because it's confused and the same link is not resolved in focus context. What else can we do? Let's take a look at right below it, see right below root files?

A

In this case, we see the example of how they can be triggered, so we're doing we're creating the symbolic link not to the root directory, but we call it the root link and then using it and, of course, it's that easy to bypass this rule as well, just by using the same link to the directory and then using it for the relative path. Okay, what about hard links? Yes as well, we can, we can use them.

A

Hard link is a direct sort of pointer to the inode and in that case falco until version 0.31, where we actually committed this the new rule for the hard links it wasn't able to detect it as well. um Now this there's a special case of pseudo-potential privilege escalation rule um this case is kind of reversed. So let's, let's see what we're talking about here, there's a the last year there was a infamous cv in sudo which allowed attacker to pretty easily escalate the privileges from the regular user to to root.

A

So when I was trying the typical poc by warwick, I just download it and so and try to see if that works or not, and it worked very very easily on ubuntu, I think that was 1804 and falco was silent, so I got in what happened there and where what stress s trace showed me was that the exactly was actually running the user bin pseudo and not the pseudo edit, that the rule would expect you can see here proclaim equals pseudo-edit rather than pseudo, and what happens here is that ubonto in ubuntu, actually pseudoedit is the symlink to the pseudo.

A

So what the rule? The way it was constructed, what x was expecting? The rule was expecting the symlink to the binary and it was getting the binary and, of course the exploit was still working and the rule wasn't detected. So that was one of the fixes that we that we can make it in the version 0.31 as well, but this this is interesting kind of the other way around of this typical simulink evasion that showed before all right enough of the simulinks. Let's uh talk about proclaims.

A

Franklin is the name of the process, and many many falco rules rely on it a lot um so typical constructs here list, there's a list of login binaries names and management, miners binaries names and these lists are used in the rule, called read, sensitive file and trusted the same rule that we're we're looking at we're using for our examples- and you can see here the construct and not that I was talking about- that's the I call it accepted construct construct where the rule doesn't trigger.

A

So all we need to ensure here is our process that will read. The sensitive file will be named as one of the user management binaries.

A

So what we do here, we we write a very simple uh c utility the treats file and we compile it into system d, login d and call it system systemd, login d um and using it to dump the shadow. And, of course it's fine, because uh falco thinks that systemd login d is the name of that one of the user management binaries and the rule doesn't trigger. What else can we do? The thing is that um this approach is not scalable.

A

We can't write utility for for every um every utility cli command that we want to use. So, oh, but we can't do can't create the symbolic link and so and call it system d, login d like like in this case and use that symbol and click to downplay its shadow and again focus silent in this case. So again, symbolic thing, but with the proc.name perspective, eventually simpler, renaming, also works. If we can rename cat into the systemd login d there we go. We we have a. We have a perfect invasion here.

A

What about parent and ancestors names um yeah? So that's also a problem, because in this case, for example, when we're looking at the condition where proc name should be something that p name should be something what we can do, we can come up with the script um and call it password one of the accepted binary names and the script. The parent name will call the the proc name and that's how we sort of build this condition, and once that works we can.

A

We can easily dump its shadow without being detected, but we can make this evasion even more generic. What we have in the football latest image and you'll see this image a lot. This is the publicly available image that they can that they created with various goodies to bypass the uh the bypass falcon. This image creates one of the faber's fuber fake parents, so that binary that small utility- it knows how to fork um a series of processes with the depth, in this case depth of three, and it calls the ancestor.

A

The first initial process is a google account forks all the rest and then runs this command and what it does. It basically tailors specifically this condition, because the proc a name of three it indeed starts and equal to google accounts, and that's why the file is silent in this case, and we can use this utility um to build. The nesting and name is answer our ancestors, as uh as we want.

A

Reversal detection: uh this is also very interesting. This is a typical reversal, payload that creates initiates reversal, and um indeed it uh it has. It causes for three detections here. uh So, of course, the typical attacker can't can't use it. What can we do? We use? We can try to use, make note and then type it into the ansi. This is also very, very known one liner for initiating the reverse shell and, however, it's still, it still shows the detection.

A

However, this detection is way very easy to do to avoid, because to evade sorry, because we can use either sim link or proc name evasion that we already talked about, but we can. We can do something more interesting. How about we use the msf venom and for people who know msf venom. They know that this is a defect, a payload generator. It's it's a part of um metasploit framework right and it's very easy to generate any type of payload.

A

In this case, I'm generating reverse tcp payload for linux, uh base64 uh copying it into the container marking it as l file and there we go. We see the connection on the hostess name and file place silence. So indeed, this is. This is a tough problem to detect payload execution as an elf, where it doesn't have um a hash, a known hash, but but falco, even not in nothing antivirus right. So it's it's a tough problem to detect this type of behavior. Perhaps this could this should be detected on a network level, though?

A

What else do we have? We can talk about common arguments, manipulation, um netcat, remote code, execution container. You can see that it tries to detect the indication of ansi or and cat along with a bunch of flags.

A

So for ansi, for example, there are two execution flags dash e and dash c that allow you to execute bash, for example or other program upon the connection, and that's what this rule is trying to avoid. However, we know that linux, old and bigly big linux utilities- they have so many flag options that it's really easy, really really hard to come up with the rule that will will block all the potential potential flag combination.

A

That's indeed the case in this case when we we're using ansi with dash e with classic dash ebin bash, it's of course being detected by that not get runs inside container rule. However, if we just add dash v as in verbosity mode, then the syntactical uh comparison fails here, because the the proc rx doesn't contain stash e at this point and then- and there there we go- the netcat runs inside doesn't trigger anymore um all right.

A

What else can we do with the pro qrx?

A

We can take a look at the grab private keys rule, so it tries to block, define to and find when any any search contains, idrsa or id dsa, and for people who know fine know how much, how many options does it happen? What kind of rich functionality does it have and, of course, with dash regex? It's it's very trivial to bypass we're just substituting the r with with the dot, but we can do the same with any other letter and there we and there we go.

A

We don't have syntactical comparison of idea reset at this point um and the last interesting rule that I'm going to show bypass is the sensitive mount bypass. So the launch container with sensitive mount rule relies on this condition, and I don't know if you see the problem here. The problem here is that this, this comparison is very specific.

A

What attacker can do they can mount varan, for example or var, and then they use subdirectory path to refer to the docker, and I will use it during the full simulation attack later.

A

But in this case this is very easy to show we're just running the docker with dash viva run run and there we go and we do have a shell response rule, but not the mounting rule that should have been triggered, and I think, for the sake of time, we I'm going to skip the crypto minor detection bypass. um You can in the privileged container detection bypass.

A

Although this this one is interesting, but you can, you can read about this in the in the github directory of the project later, if you want, if you're interested, uh but let's jump to the toolbar, our toolbox so far contained two images: footer latest and that's the source.

A

You can see that it, it has the docker client, it has cube catal client and it has a bunch of network utilities, g bash run c and they tweak the names of those clients and bash so that falco won't be able to detect them with the static with the traditional rules. And then I have the cv 2021-3156 image just to to simulate that specific rule trigger that.

A

That actually did it didn't work and then, but now it works after our changes um in case somebody wants to play without with this and finally, let's go to the interesting part: let's try to put together all those evasions and see if we can perform the full attack simulation.

A

Okay, so the demo setup is. I was running it within the gke cluster with the custom nodes and silicon load cluster. One word about the cluster that I used is. It was presented in the cubecon 2019 two years ago. It was really cool, uh just a testing cluster. That shows how to just show some attack, techniques and defense technique um and highly recommend it um pretty interesting one, and that works perfectly for our case.

A

So it has two attack scenarios scenario, one and scenario, two in scenario: one is more basic with the more basic attacker and scenario. Two is with more advanced detectors, so let's go and start with the detection of detections of the scenario, one just just we'll see what what we expect from falco to detect within the scenario. So the uh all scenarios start with the attacker. Given the web. Shell assumes that pod is already compromised and, um and then the attacker starts all kind of reconnaissance enumeration from within the pot.

A

It runs the end, it runs the unnamed r and then it says sorry they see that the um the pod, the hust is part of the kubernetes pods and there we go, we have the um kubernetes version, etc. We have all the environment variables. We have some kind of service account mounted here and then at this point, attacker tries to uh go to the typical way of uh compromising the the cluster, so the um they download the cube cattle.

A

They try to use it to get all the resources whatever they can, and there is a bunch of resources. So it takes some time there we go. We have the attackers can see the pods um and then attacker uses auth. Can I to enumerate the the permissions that the current part has, with the amounted service account and apparently attacker can't even create pods right and that's what the attacker does?

A

They create the um the malicious bitcoin airpod that, of course, will um mine bitcoin, and we can see that this bitcoin is an error is created successfully, but all this time we see that falco is successfully detecting various malicious activities right here, so this will, for the socket will be really hard to not to notice any malicious activity in this case- and here I'm using the lint piece just to demonstrate.

A

um If, for those of you who know lean, peace is a very known in linux, enumeration and privileges, escalation tool and and it's very popular so if, if typical attacker will will run it on the host, of course, nothing will happen, but if they run it on the on the faulty monitored, host or container, then then that's a problem, because falco detects a bunch of um sketchy activities here and, of course, sock will not stay silent in this case.

A

Okay, let's go scenario. Two detection scenario: two is a bit more evolved. The attacker is more advanced after the.

A

um So after the scenario one is finished, the the defense um actually detected easily that caught and they killed it, and so cinderella two was able to get the same web shell, but now attacker is more advanced and they want to do something sneakier and they're. Using this one, liner.

A

This one liner is very interesting because it smashes the hospital. Namespace amounts, the same process, namespace and lounge launches the privilege, privilege pod and at this point container, has the access to host resources.

A

You can see that root is the root of cost at this point um so skip ahead, because this right and the attacker has access to the docker circuit um and what attacker does they deploy? The bitcoin error this time on the docker through the direct docker api rather than the kubernetes api, um and that's a bit sneakier because then the defense will for defense. It will be harder to find that bitcoin error um and, in addition, what attacker does they want to?

A

They think about the future so for the persistent they will, they find the cubelet token mounted on the host and use this token to. Actually this is the token and they use the stoking to mount.

A

Sorry, that's the get secrets commands so, yes, attacker can find any get secrets, and this is the amount of the, and this is the creation of the um node port service for the future access. So attacker at this point is able to open the port on the node um and in the future, access this. uh This part, if needed, and we still see that falco is very active at the detection.

A

So again, it will be quite easy for uh for stock to detect that uh something malicious is going on on the on the uh a particular cluster so um put in the attacker's head now we're trying to avoid falco as as hard as possible.

A

We are doing for the same scenario, two actions, but with the mind that we don't want falco rules to trigger.

A

So what we do here, we're still copying the cube cattle, however, we're using our name evasion and we um renaming the cube catalyst, k, cattle and, from now on, the falco rules that detects the cattle inactivation within the container will not work and we can easily use the k kettle from that point, and so the one liner that I am using it's it's a bit trickier here. It's not privileged container at this point. We're still smashing the host pid.

A

If we can um and I'm running the fibre latest, which is the faulty ultimate bypasser image and as you saw already, it has g bash, which is different from the batch, so that rule will not trigger and we're mounting var run to get the access to the docker socket without triggering the mount rule, and we are running the g, docker ps command and, let's see if we have the access to the host level, docker circuit.

A

Yes, we do. We can see that there's a bunch of uh like kubernetes control, plane um images running here, containers running here and we have. We do have access to that and that's great. So at this point we can run commands um I'm what I'm doing right now. I am running the g docker.

A

I'm actually deploying the bitcoin arrow on the docker, and you can see that falca is a bit active so far, just it just had several three detections at this point, but it's all the same rule and this level is notice, an expected connection to kubernetes api and then plowing through using the same wine one liner.

A

This time, I'm looking for the for the cubelet through the cubelet mounted service account token, I'm looking for the one that will be able to to open that to create the nosport service and to open the noteboard by using this token, like we can see, is that it finally can do whatever we want pretty much. So now we have confidence, we can mount the we can create the node part service and that's what we do notebook service.

A

The name is istio management, which is a sneaky way to pretend to be one of the istio services, but for our purpose, what's interesting here is that falco is still only detects. The node is unexpected connection to kubernetes api. So in this case, if the socket is um geared towards detection of the warnings or errors or up, then they will not see the notice plus. They might think that this is some kind of false positive, that's some kind of rule that triggers excessive amounts of false positives and decide to disable this rule.

A

If they, if they do see this rule, but let's see if we can do even better than that.

A

All right so the difference here this we're still talking about the second scenario. However, I'm modifying the command um a bit to.

A

To mount var lib as well, the mount command will still not trigger I'm still using for the latest. I'm still smashing a horse speed.

A

However, I'm performing the g bash command so now I'm supposed to get the best shell into the container- it's not privileged container, but it has the access to the to the doctor and the var leap and the cubelet tokens for the future.

A

We have one detection so far.

A

We have two detections so far, but we are root within the container and we can pull the football latest again.

A

And the more important that we have the access to the docker sock, so the neat trick that we're doing right here- we're tagging the fiber latest as a cystic container and from this point and on falco thinks that cystic that fibrolatus is actually is actually named sysd image.

A

So we will be able to bypass that unexpected connection to kubernetes api server, because the sysdig is one of the containers that allow to make those calls and, as I called it one of the accepted conditions in that rule.

A

So let's see that we indeed do that.

A

So at this point I'm running additional container from within this is the container which is not really cystic. I call it sysdig, but it's not it's still fiber latest. We can see this in the last on the temp folder. It has all those uh utilities that we put there for.

A

um However, right now from this point and on falco thinks that we're in the csd container. So it will excuse this container from accessing the api kubernetes api, and at this point we can do whatever we want. We can rummish through the cubelet tokens get in the tokens.

A

And this command is important.

A

I want to list my permissions whatever I can do from within the nested container, I'm using the insecure skip tls, of course, I'm using the g-cube cattle so that falco will not detect that I'm using actual couple, I'm using this server from the kubernetes server host from the current container and I'm using the cash tier and cash there is important here. I I want to set it a stamp because otherwise the cube cattle will save the cache in the root, slash dot, cube directory which would trigger the file. Otherwise,.

A

And we can see that we do have the the permissions to do whatever we want at this point. So at this point we can create node port and whatever we want, and still we only have two triggers of unexpected connection.

A

um So there we go, we have only two triggers for the full cluster compromise, including the persistent access for the future. Hopefully, um can we do better than that I'll leave it as a homework?

A

So for the fixes, if you, if you're interested in the full project, with all the details, those one-liners you can find them in the blackberry github for the channel changelog, we had a series of commits on january 25th that went into the 0.31 version, discussion and recommendations well uh very quickly, and we can.

A

We can discuss it more in depth during the q, a sections section, I would say that hooking points are very important because we do want the same links, for example, to get resolved as close as possible to possible to the what kernel c to get that the kernel picture no easy way to prevent attack control by passing the rules to land on proc, name and proc, a name and file name. Yes, that thing needs to be minimized. Two menu rule includes construct and not because every exception.

A

I I understand that that's for false pos to minimize false positives, but that's also the the entrance uh entry for the attacker and possibility for the attacker to avoid that evade the true review all priorities in the bypass context period. The check of public exploits for the cve specific rules. On the example of that through the privilege exploit that I showed and then eventually encourage clients to develop their own private rules, because then the attacker needs to know the rules that source before doing any action to understand.

A

How is um how to evade those the customer level rules um before starting that app and that's all I had for today. Thank you so much and I'll be happy to take your questions.