Cloud Native Computing Foundation Cloud Native SecurityCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Eye of Falco: You Can Escape but Not Hide - Stefano Chierici & Lorenzo Susini, Sysdig

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Eye of Falco: You Can Escape but Not Hide - Stefano Chierici & Lorenzo Susini, Sysdig

Container technologies rely on features like namespaces, cgroups, SecComp filters, and capabilities to isolate different services running on the same host. However, SPOILER ALERT: container isolation isn’t bulletproof. Similar to other security environments, isolation is followed by red-teamer questions such as, “How can I de-isolate from this?” Designed with the principle of least privilege in mind, capabilities provide a way to isolate containers, splitting the power of the root user into multiple units. However, having lots of capabilities introduces complexity and a consequent increase of excessively misconfigured permissions and container escape exploits, as we have seen in recently discovered CVEs. Fortunately using Falco, a CNCF container runtime security tool, it’s possible to monitor Linux capabilities, detect misconfigured containers, and proactively respond to secure environments. In this talk, we explain how you can use Falco to detect and monitor container escaping techniques based on capabilities. We walk through show real-world scenarios based on recent CVEs to show where Falco can help in detection and automatically respond to those behaviors

A

So everyone thanks thanks for joining uh today, we are going to talk about how we can use file code to monitor capabilities and all the work that we have done in order to expand FICO to monitor probability.

A

uh So, let's just introduce ourselves I'm, Stefano and I'm, a security researcher at sysdig and with me today is presenting uh Lorenzo suzini, which is a open source engineer at sysdig. We are both from from Italy and we are both a Falco contributor.

A

So, let's start for for for today, and let's start with uh talking about container isolation, so we all know uh container is is a a great way to uh to deploy our our uh our application and transport our code. However, we also know that um containers aren't security boundaries, um so what makes container secure and what make a container resolution are all the layers that are around containers and in error reporters some of them, but we have also uh web capabilities second performer, but we have also uh Linux Linux, namespaces and C group, for instance.

A

So all these layers come together to bring a container container resolution today, for today the main topic will be capabilities, but let's just introduce also a farmer and second so cover bees are a way to restrict uh the Privileges that are permitted in containers.

A

Second, instead, are is Linux feature that can be used to restrict ciscos and the argument that we can pass to to them to Cisco's and also a farmer is a mandatory Access Control framework that can be used to add the tax on programs, and we can enforce the specific programs can have specific access to objects. So, for instance, we can say a specific program, can never access or read access to or to I, don't know ATC past 2D, but not to ATC Shadow.

A

It's also worth mentioning that um by default, when, when you use Docker and we and we deploy a container all these security layers are enabled by default in order to provide the least privileged concept and and that's enabled by default, so we don't need to provide any uh particular Flags or we'll set. Particular Flags is all enabled by default. Today we talk about how we can use fileco to, since fileco is able to to have a visibility over what is going on inside our container.

A

We can also we extended also a fileco to see capabilities and which capabilities are actually available, so I'll, let Lorenzo talk about capabilities and how they works. Yeah.

B

Thanks a lot Stefano for introducing me and let's now jump into the main topic and well, as we all know, traditionally, we used to distinguish Linux processes into two different categories which are Privileges and processes and unprivileged ones. So we can say that this model is binary and it can be risky from a security perspective.

B

This is because, basically, privileged processes are those running with effective user ID equal to zero, and it can basically password kernel permission checks, as we all know, and on the other hand, privileged processes instead are those subject to all current permission, checks and most of the time this is done using their own, effective, user-ready or group ID and from an attacker standpoint. This model is exposable, of course, by attacking directly root processes and once the root process is compromised, the whole system is at risk of being compromised.

B

For this reason to improve its security model, Linux has implemented the capabilities and capabilities. May are a clear example of applying and implementing the principle of list privilege and in fact, as the model States, the power of the super user is splitted into distinct units. That can be independently granted or revoked, and this allow us to run processes with the minimal set of privileges that they need, and hopefully, an attacker getting in control about the beginning. Control on these processes cannot compromise fully the system with the same needs as before, and the classic example about.

B

This is the one over of a web server, for instance, and before the introduction of capabilities, a web server needed to run as root only because it had to open a privileged part. That is a part below 1024..

B

With capabilities. You can map this specific privilege to the capability which is, in this case cabinet Mine Service.

B

We can let it not bypass checks, for instance, on the file system, which is translating into a completely different capabilities, which is in this in this case Capital right. So since you are here to implement proper monitoring of capabilities with the purpose of detecting attacks, our takeovers on this is that not all capabilities are the same in terms of privileges they allow.

B

In fact, some of them can be defined as full root equivalent, if gained by an attacker, and this is because, if an attacker manages to gain one of them, it can possibly gain all of them with other successful attacks and, as we will show later, so, let's go on and to implement to implement monitoring for capabilities. We have to better understand how they work. So, basically, capabilities are usually divided into three main sets and which are defective.

B

The permitted and inheritable set, and you can see that the effective, the effective set in a certain sense reflects a little bit the concept of the effective user ID and in fact this is the set that the kernel uses to perform actual permission check and on the other hand, we have the permitted set, which is basically the super set of all capabilities.

B

The process may have been required and we can move capabilities from the effective set to the permitted set to let the effective set have only the Privileges needed at a specific point in time and last button on list. The inheritable set instead is used to control, which capabilities are inerted upon. Exactly a system call which is a pretty relevant example, so the narratable set is used to perform specific operations by the kernel to compute the new permitted set, as you can see in the next slide and yeah.

B

So this is how capabilities changes upon exact fear and what I want to stress here is that the Cardinal um uh use uh specific, set sort of rules to change capabilities of a process, and, following the previous reasoning, we have to say that the effective set should start to empty when exactly is performed. This is because we want processes to manually move with CIS called capabilities from the permitted sets to the effective one.

B

However, the the kernel also offered backward compatibilities for those binaries that can be defined as capability dumped, meaning that they are not able to move capabilities in between sets and for doing that. The file effective sets and the file effective flag.

B

Sorry it can be used, and this is basically used to move the new permitted set to the effective set so that that the process uh doesn't have to move capabilities using others is called so uh now, let's go back to container security and see our capabilities applied to it, and, in fact, when talking about container security, we basically have two different possibilities: fill out our work with drums. Actually, so the first one is to avoid container running as root.

B

I think everybody here knows about this, because basically, containers like Stefan was saying before: are not the security boundary like VMS and if you're, not using username spaces root in the container is also root on the host, uh but sometimes we want our containers to run privileges or privileged operations, and we may be tempted to run down this route. So uh what can we do? We can do? We can use capabilities, as we said, capabilities.

B

On the other hand, have the possibility to selectively choose the privilege that we want, and anyway you think that since capabilities are a lot, and sometimes it's really difficult to understand which capabilities our workload needs, we may end up in certain cases, running excessively privileged containers as well, and this is for sure not good from a security standpoint and Stefano will now show you why and how capabilities can be used in real attacks.

A

So thanks, um let's now uh deep dive into attacks on uh container escaping I, just reported this slide just to say basically what we have said before right, so containers aren't isolated by default and in this case uh we all know if we deploy a container as privileged. This is far to be isolated right. So, of course we are, we are still using container. We are still using Docker, but this is far to be an isolated environment.

A

That's because if we use capabilities- and we see since we are talking about capabilities with privileged all the capabilities are available and all the capabilities can be used and loaded by a possible attack. So that's just to mention that when we deploy a container as privilege, we need to be careful that this is needed and we we need to be super careful of what we do. So, let's start, let's start now really talk about container skipping attacks and in this case, let's start with the with this technique with the CIS model capability.

A

So this module is used to load and load kernel module and for this technique we have the only requirement here that we have is uh having the assist model capability and how it works is pretty simple. So let's say that an attacker is able to get any uh the initials the initial foothold in a container and find out that the disease model capability is, is available and can be used. So the attacker just need to drop the the module inside the container and since the module is, is loaded in the kernel.

A

All the all the malicious code is going to be executed in the host.

A

So in this, in this example one once we load the malicious module, The Bash reversal is going to be executed, and so the connection will will be open from the container those back to the back to attacker machine, and it's also worth mentioning that uh the attacker might need to um to compile the model inside the container, because just to be sure that the model is gonna is gonna run and and for this the attacker might need the the Linux heater installed in the in the host. Otherwise, it's not possible to run this attack.

A

um Let's just move on with another technique, and in this case we use this. The capability pit race in particular pit race, is used to control other other processes, but also with pit race. We can also modify the the process memory to inject arbitrary code and you can also modify the the pointer extraction in order to run this arbitrary code so in, in other words, with pitrace. We can inject malicious code and actually execute it, so it's pretty powerful capability. So in this case, for this technique we have three main requirements.

A

So first one is of course, having the CSP Trace available. The second one is apartment. We need to have a farmer set as unconfined or or allowing pit race and the third one is the container and those needs to share the dpid address space.

A

uh So, as said before, let's say an attacker has access uh to the to the container and was able to get initial access into the container and find out that see the P Trace capability is is available.

A

So, in this case, the attacker just need to drop its own injector with the with the malicious Shell Code and find a process that you want to inject and and just execute the the Shell Code in this case, as we said before, since the the address space is shared between the container and those, um the the malicious child code is going to be executed in the host and that's why, if we as, as we said before, if we open the reversal, there is the the reversal is going to start the connection from the container source to back to the to the attacker machine and that's another attack, but using a different capability.

A

I'll, just close with with um the last the last example for today. So this attack is a well-known attack for container escaping. uh In this case, it was published on 2019 uh on Twitter. uh In this case. The first example that is reported here on on the left required a privileged run as privileged a container runs privileged after after that, a new, a new exploit came out, and in this case we just need the C submin capability. In order to to be successful.

A

For this specific attack, we have three requirements. So since admin is one a farmer with the unconfined or just allowing the mount operation, and also we need to have a root user inside the container. So to understand how this work, we just need to understand how notifying or release works, and- and this is pretty easy so on if I uh notify. Your release is a feature uh in the C group, V1 versus version one and basically, if it's enabled so it's set to one uh once a container a process terminates.

A

The kernel is going to execute the code inside the release agent file. So whatever code inside is inside the release agent is going to be executed by the kernel and and that's basically the how the exploit works. So, just just going through the code, the exploit just create a directory. A directory is mounting the the C group controller inside this. This director, with the the option RDMA and also is creating the C group directory inside it.

A

Also, as we as we can see, the the Notifier release is set to one and, as we said you just to enable the feature, and then all the other commands is done to create the release agent with the malicious command that we want to execute. In this case, it's just getting the path for from the overlays and and the command that we want to execute. So in this case being sh.

A

So we actually, we are opening a shell in the in the host and the last command is just used to trigger the the exploit uh running, an Eco command um and that's it so once the Eco command is going to to terminate uh notify your releases is, is enabled it's going to trigger uh the workflow and and executing the malicious command that is inside the um the release agent.

A

uh So just to recap what we have said um so far, so we have these three techniques. All that the three techniques that we have presented are using three different uh capabilities, and so now we, let's see um what we did in in FICO, uh in order to detect those uh this malicious behavior and how we can and what we can do with FICO for monitoring abilities.

B

Yeah thanks Stefano for doing that, with such examples, and so now that we are aware about talk of abilities, work and how they can be used in real attacks. We want to show you how to monitor them using Falco. So for those of you who doesn't know about Falco Falco is an open source project for runtime security and it can. It became the de facto standard for kubernetes strategic action.

B

It was originally created by sisdig and then donated to the cncf, and it's currently at incubation level, and we both work for it become better and better and we always try to improve it. Its detection capabilities, so another thing that it's exciting about Falco is its Vibrant, Community and I think that it's really nice to work in such an environment, because people are really active and contributing, and hopefully this all that shows an extension that we made to Falco, we'll encourage you to do the same and improve it and make it become better and better.

B

So before we dig into the details on how we extended Falco to monitor capabilities, let me give everybody a little overview about it and we can say that Falco is powered by its kernel module and it's even if you have probe to collect system, calls data directly from the kernel which is considered the sorts of truths for our purposes, so most of the event, connection components user in buffer to push events that are uninfoliated and used in user space uh in our libraries.

B

What I would like to stress most about this is that there is a part in one of our libraries that is responsible for maintaining the state of the system, and this part is, of course, in limp Sims, as you can see, and um what we you, what what we do with this part of a state is making it accessible by conditions and rules and making it able to write rules and condition able to spot new kind of attacks.

B

So, with this in mind, we realize that Falco was only able to see in black and white, meaning that uh he was only able to discriminate and distinguish root and root processes, and we wanted to put some light in that gray area. That capabilities have created um in between. So to summarize, the part of the work that we have done, we can do it in explaining the two main steps.

B

So, basically, what we did is that when Falco starts, it needs to be to build the state of the world, that is, the state of the the actual System state at a given point in time. This is when Falco starts and most of the information used to do. This comes from the frock file system, and fortunately there is a file in the frock file system which let us retrieve information about all capabilities of process, and we use this to populate the initial state.

B

So once this is built, um Falco starts capturing system calls and keep updating it using system called Data.

B

So what we did is enumerating all the possible system codes that may interact with a process and modifying its own capabilities, and we started with the fork system call which was the easiest one, because basically we only have to copy the capabilities from the parent process to the chart to the child, and almost the same goes for the Clone system calls, except for those clone decks, creates new name spaces and, um moreover, to push to use the space the new capabilities, also in the exact fear, we modified the event of exactly to push information about capabilities.

B

Also, in that case and last button on list, we have also the possibility to move capabilities in between sets, as we already know- and this is usually done with the capset system codes- and to do so- we implemented the monitoring for that for that system called from scratch.

B

So in this way we are able to let rules extract capabilities from the state and we can write some interesting rules. Two spots, attacks based on capabilities and now Stefan will show you what kind of rules we came up with, and hopefully it is interesting to see it together.

A

So thanks and thanks to the Lawrence work on on FICO, we are now able to use capabilities inside filecruz.

A

This is a rule that we created in order to um to control and detect if someone deploy a new container with with excessive capabilities. uh So, as you can see here, just a two word on how how FICO rules work.

A

um So the FICO rules are loaded inside the FICO engine and once a news is called come in, uh FICO engine is able to check and emerge all the uh all the rules and checking if, if I will match with the with the Cisco, uh is able to trigger a security alert and that's how it works.

A

So, let's see the condition here, so the rule is going to trigger when we create a new container with excessive capabilities and in the macro shows all the capabilities that we are monitoring in this case and as you can see, there are assist model and see speed race. There are the two capabilities that we have talked before uh and in the bottom we can see there, the output from from Falcon, and also we have also uh the the other rule that we created.

A

This is specific for the use case, as we talked before about the release agent file. So as we discussed before, uh the the attackers need to modify the release agent file in order to inject the code that you want to execute, and that's mainly where we create the detection. So once the release agent file is open, is open to write a commands and also the the container as the the sysadmin capability and the user is root.

A

We are going to write the security alert and in the bottom we see uh the output mentioning all the capabilities available. So we can see the capsys has been is one of them and in this way we are able to detect with a very low Force positive, um the the potential malicious behavior.

A

So, having said that, the sit from my side I, let Lorenzo close the session and start the the Q a yeah.

B

That was basically it hope you have enjoyed the salt and if you have some questions, feel free to reach out now or in the Falco kubernetes slack, and that's it I hope you have enjoyed the talk and you will use capabilities or try to improve routes and hopefully contribute to the gut to be able to improve Falco. Always thanks a lot for being here.

A

Thank you. So if you have any someone else, any question.

C

B

If you want to use Falcon in your cluster, there are M charts to do that and you can use they are officially provided by the Falco security organization yeah. It is when you wanted to use in kubernetes.

B

It can be used as containers, and you know the arm chart will spawn pots on each node in the cluster as a diamond set, and that's that's it. Basically, you can use this to the play. Fark or Falco is also Deployable on and on, and you can use our packages to to do that.

D

Thank you, uh maybe an obvious question for a Falco. We see all the rule configurations that go in the yaml configuration for the cluster does Falco look also into those type of of aspects. Do we consider that any configuration that occurs is treated as an event and then Falco will see and analyze, or these are two separate, no.

B

No, this is just a different thing because Falco it's runtime security, it only it doesn't look about configuration in the in the yaml. It only looks at runtime if a process is being executed with a specific capability or not and I don't know. If this answer your question well,.

D

These are configuration that can happen at runtime, so it's still part of the runtime, whether I change I do a change in my environment. This is going to be captured or.

B

Not yeah, if you spawn another pod with different capabilities, it will be captured at around Time by Falco.

D

B

E

D

So if, if I'm, able now to capture at runtime, also those type of configuration uh do I have an overlap or do I always need to have both of them, because there are configuration types that cannot be captured by by the uh event triggering and looking, if I, if I'm, allowing Falco to be able to detect all the configurations at runtime that happen. Through this you know, enamel configuration so I have a duplication. Now, I have the rule engines that detect that I have also a file code. That is looking from a logging perspective.

D

To the same thing, do I have a duplication, can I focus only on one of them.

C

D

Okay, so there will be like treat it as different: Falco will have okay or maybe there is an overlap, but not big enough to separate. There is no overlap. Okay, okay. Thank you.

A

Any other questions.

E

C

Sorry I was just looking at the rule. I see that you are actually ignoring Falco because you're running as a privilege. Is it correct the privilege, rule yeah, it says not Falco privileged. Why is that this one right that one? Yes.

E

A condition it says: I'm, not Falco, privileged container.

A

Yeah, the other other markers are used to uh to reduce false positive I haven't reported the the all the macros because it's hard to fit all in in the slide. But there are other macros that are reported to reduce false positive in environments.

A

Sometimes so usually, macros are used to simplify the overall condition to make it more readable, uh and sometimes we use macros to reduce false, positive or or at least some use cases that we know is going to trigger Force positive so and that's one of one of those uh cases, but in general we can use macros for, for those reasons, yeah good.

B

You are saying if it is possible to deploy Falco without privilege, yeah um Falcon, it's a specific and a unique workload because it has to inspect the system. So it has to have some privileges, of course, and we have in a down chart. We we try to run the our BPF probe with these Privileges and there's. There is an option to do that, but in general it has to have some privileges.

F

It's a false positive alert, a raising for the foreign.

F

It needs some privileges to run.

A

Who is securing uh Falco I? Guess.

E

A

Yeah I mean we. This is a security uh tool right, so it needs a visibility over uh over what is going on. Otherwise, it's not it's not I mean it's not going to work, it's a security tool and we need to have visibility over uh um over what is going on. So that's that's the main point of all of this right.

A

So, of course, there are ways to, uh as we said with the with the ebpf, to reduce this kind of um of visibility for FICO, but this is mandatory for the type of tool that we are using and to have visibility. So I guess I mean I.

A

Think most of other tools are basically using the same way uh since I mean it's not possible to do any anyway uh in other ways, so of course we can reduce it, but at the end it's something that we need in order to have visibility, and of course we want to have visibility uh in order to provide the best detection possible. So I think that's the point good.

C

A

A

So thanks, everyone and I wish a nice rest of your day. Thank you.