Cloud Native Computing Foundation Kubernetes Community Days Bangalore 2022, 11 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building Container Defences Executable at a Time

Description

Supply Chain Security issues are on the rise. Supply Chain Security tools will ensure that whatever you are downloading from container registries is not tampered with. But the granularity of these tools is way too coarse. They work at the container image level. How do you ensure the continuous security of the apps shipped inside the container image? How do you secure the app against attacks where an attacker gains access to the container and replaces the app binaries? There are various ways supported in the k8s land, but they give you all or nothing solutions. How do you actively thwart someone from executing anything that is not shipped inside the image? This talk will showcase novel ideas using Linux Kernel technologies to implement solutions to block untrusted binary execution.

A

Hello folks welcome to my talk, building container defense and executable at a time. My name is suraj deshmukh and yeah a bit about me. I work as a senior software engineer at microsoft and I've been managing this kubernetes bangalore meter for a while now, and you can find me at any of these following places so yeah. What is today's talk about?

A

So, let's talk about the threat model where an attacker somehow gets access uh to the container shell and then from there on they try to do some malicious stuff, like uh they use containers identity, to do something on the cloud provider or uh use the containers network to access other stuff or exploit others other other uh ports or machines or whatever they gain access to the volume in there or secret.

A

So this is a threat model we are targeting uh today, and specifically, the attack vector we are trying to see is any attacker when they try to when they get access to the shell. They'll they'll try to download binaries from the internet where they can use that to do other things.

A

uh For example, uh crypto mining is the the most popular one these days. So how do we? How do we mitigate that or what's the defense against it? So our defense is disallowing. Execution of binary is not shipped with the container image. So, if we take care of that here, we are assuming like whatever is shipped with.

A

A container image is secure, or at least when it was built. It was built with an intention of it not being uh used for what it was with a good intention. uh So yeah, let's see so what what existing solutions we have? uh We can. uh We can start a pod or container with read-only file system, uh but the thing with read-only flag on when you start a container in catas, you can still still do stuff in slash dave partition like you can download new files execute uh execute them from there.

A

The other solution is using scratch based image. It is the most secure setup I think, but it is only suitable for apps that have static binaries applications that need interpreter won't be able to run in scratch based images. They have to be shipped with uh with a bunch of package, so yeah those were existing solutions with disadvantages.

A

So the new solution that we are proposing is what, if you knew already what is being executed in the containers root fs like whenever something something is being invoked, you somehow get a notification or a way of defining whether to allow that execution or not.

A

So if we, if we can, if we can do that, then we can design an application that, based on some policies or based on some, whatever user input like we can, we can decide what is allowed and what's not so yeah. How do you get those notifications? How do you know what binary is being run?

A

So that's where fa notify comes in it's a file system, notification framework and linux, so like from the from the ifa notify man page, we get this information like fa, notify api, provides notification and interception of file system events, so this was specifically created for for anti-virus scanners.

A

So yeah we get all all kinds of all kinds of notifications, but the specifically we are looking for is the access permission, decisions where, if a binary this it will will be executed, will will get those notifications so uh like? How does this look like in action?

A

So here are four components that that will interact with each other in some way, so obviously there is container run time, uh creating containers and managing them and then the actual container that will be monitoring the application, as in the the our application, the policy enforcer which run runs alongside and the kernel. Finally, so obviously the application is, is it's continuously looking at what containers are created?

A

So let's say: if there is a new container created, uh we'll we'll poll will pull the container and time to to know that, and once we see that a new container is created, we'll uh will tell kernel that uh you would like to monitor this particular root fs.

A

Now uh this particular the fourth point is not just one step uh in in actual coding. There are two steps here, but for for the sake of simplicity, I've shown it click. It's it's just one step. So what you get from the kernel is a file descriptor, uh it's a queue uh file, descriptor for a queue where you can read events, the events that you run to it and then we we read them and make decisions.

A

So uh let's say there is: there is a new exec file. Binfo is being executed, at which point uh the kernel will send the notification to the that file descriptor we received earlier, and we get to make the decision whether to allow it or not, depending on what your logic is.

A

Until then, that system call the execution of that binary is paused. So we have to respond to those events to to for, for those binaries to go ahead and execute so like you can see here like in this case. We have allowed it, but if, uh if there is some uh malicious binary or whatever we get that notification again and depending on logic, we decide that it should not be allowed execution.

A

So if we send a deny at which point, the the executor gets a uh e-bomb uh or operation, not permitted uh error message. So this this is what this simple workflow looks like. So we register for fa, notify events, and then we will get those events.

A

So, let's, let's take a deeper look here, so these are the two uh system calls that will will make for registering uh for events like fnotify and it creates notification group and returns a file descriptor for the queue like I said earlier, and the the file descriptor is then used in the fnotify mark system call.

A

So if you notify mark is used to place a mark or a way of saying uh this is what I'll, uh what you should monitor so it could be. uh You can place a mark on file system, uh mount point directory file, etc. So it is. It is also used to specify what kind of events you would like to receive. Like uh you want read: write open, delete access, uh there are various various others, so yeah these are. The two system calls and let's see how it looks like with the actual uh parameters.

A

So here we are saying in the: if you're not define it like, we, we want unlimited q and unlimited marks. So if you don't specify that we there is, there is upper limit to how many, uh how long the cube can be and how many marks can be stored there. So we are specifying that uh and then, if f85 mark, when we call it actual, uh we are specifying. We want to monitor this mount point, and specifically, we are looking for open, exact permission events, so the this is.

A

This is the second parameters where we're specifying what kind of events you want and the path. So uh in our case, we'll uh will monitor proc, bedroot and bid being the big one inside the container. So we will have a mechanism of figuring out what the paid one inside the container is on the host.

A

It could be mapped to anything else, so.

A

Once once we do that, we'll start getting those events, so let's look at what the application would look like the various designs that we can go forward with. So the first one is is something where we'll use diff apis uh for verifying something should be allowed or not.

A

uh Let's look at what diff apis is first uh so yeah before um on the on the right, you see like a typical container uh based container image with container layer and it it being run so sorry, so what happens is like, uh let's say when the container has cutter is created and in that layer, if you create a file, slash fubar, it shows up, in the end uh as a final result in the union file system and if you remove certain file that was shipped with container image, so that won't show up.

A

Obviously, so union file system takes care of merging all these uh all these things that happen uh at the container runtime level at the at this level. So uh that's what we're simulating here like. I have started a container called example, uh and here I am creating the fubar file like you see here and then removing the user bend touch like you see here.

A

So uh while this is going on, we can.

A

As long as the container exists, we can talk to docker api and find the diff uh that it has created from its base image. You can see uh a fubar file was added and then user bin touch was removed. So in the process, these two directories were also created in the metadata.

A

So in this way we can see. Okay, like a new file. Foo bar is added, which means it's not something that's shipped with with the with the actual container image, and this is what will help us identify, uh will help us make decision. Let's say an attacker, creates this new file and is trying to execute it, uh and if we see that it's a it's a diff from the base image, we can decide. Okay, like uh this is this could be malicious and should not be allowed so um yeah.

A

Let's see uh the workflow of uh this diff uh tiff model so like before we saw we get a notification that uh binfo is being executed, uh at which point we can reach out to the container on time and uh see what the what the def is, and uh once we get that we can make a decision whether that binary bin foo exists in the diff.

A

uh If it exists in that, if that means it was modified and if it does uh so, we deny that and we allow it if, if the binary path doesn't exist in the def, so plain simple like that, so this is this this. This is one model of enforcing it.

A

The second model is where you have like, uh so there are two components to it. uh We have a runtime verifier like before, but it depends on we getting something called source of truth. Now. What the source of truth is we'll look at uh look at it in a bit.

A

So let's say whatever container image that you have is uh is built with.

A

Like you have all the binaries in there and your your build system gives you extra metadata about those binaries. Let's say your build system is built with secure software supply chain uh in mind, at which point you can have a list of all the binaries and their uh hashes. So it's it's not necessary, but it is possible that your build system takes care of it.

A

uh The other way to build this source of truth is uh you can you can calculate this on the fly before the container starts after the container has been unpacked on the file system before p1 starts? Obviously- and the third possibility is uh is, is you can have another some rest service running so every time a container is started uh or liquid can have a local cache, where you just make a api call, and then it gives you the it gives you this list of hashes and binaries, so this becomes your source of truth.

A

So this is what you'll trust as something that was shipped with the container image, and you can rely on this to make your further decision set off of verification.

A

So similarly, the way uh this this look like this looks like is uh you get again, you get a you get a event that benfo is being executed, at which point you calculate the current hash of that binary as an as it is on the on the on the root fs of the container like so at which point uh so once you once you calculate the current hash, you can match it against the existing hash that you have from the source of truth.

A

So if the hashes match, that means the binary wasn't changed and if it doesn't, then obviously the binary was modified.

A

So like this, you can make make the decision whether to allow certain binary to execute or not um so yeah like uh the two designs have their own pros and cons like design. One is uh is, does not need those precomputed hashes it. uh It removes the whole logistics of creating them and things like that, but uh which which which design2 obviously needs to function.

A

uh Second, is design. One depends. uh It depends on the container runtime to provide those steps uh which contain which design two doesn't it's it's it's a runtime agnostic, so we can. uh uh So what this does is like it adds dependency on the uh runtime to be online and uh like without it the design one one function so, but design two doesn't care about the uptime of uh run times.

A

It will continue to run so ah and that's why, like uh the other thing is almost all run times, have decoupled the container and container runtime manager uh by creating like uh with docker docker, has done container d and then contain addition uh for each container. So using this, this default pi kinda introduces the dependency, thus rendering the uh decoupling useless so uh yeah pros and cons. uh So let's see what policy enforcer app uh like, where it can run even uh depending on that there are uh pros and cons of it like.

A

Let's say we create a standalone application uh which runs as daemon uh using systemd or kubernetes d1 set or whatever the the pros are like. uh We have flexibility, uh especially with kubernetes demon set. We have flexibility and policy change enforcement like we can create a operator which is backed by crd and then users just modify those policies on the go, and then it is enforced accordingly.

A

But the cons are like a single daemon which monitors uh single demon on the node, which is monitoring all the containers. uh The problem with it is, if it goes down then like what is what what would be monitoring the containers, uh so there could be attacks like that, uh taking uh taking advantage of that. What are the other timing? Issues are. uh Let's say, your enforcing application starts after the containers to be enforced starts so uh yeah uh such issues can happen.

A

The other uh place to put your code, or this enforcing app would be in the container edition, so container dsm runs as a daemon for each container.

A

Yeah again, uh this is assuming with content id um so like yeah. So if the code lives there, that means the container uh like this monitoring code like it. It takes birth and dies with the container itself, so yeah. So we don't have all the problem of late start and all that, and uh but the problem is uh in flexibility and policy change enforcement like uh with crds we can.

A

We can enforce that contender lives that lives as a binary on the host, and then it's it's just invoked by container d, so yeah we have that problem of uh getting all that policy translated at the host level and then also there are other issues like uh since this will be a custom content addition uh issues with shipping that uh having a different installation process and things like that anyway, so yeah moving forward.

A

So uh we talked about what the uh earlier before we started. Looking at this new solution, we talked about what the existing solutions were and how this solution, uh this new solution, adds uh benefits over them. Like with read on the file system, we saw like slash dev, it was writable, uh but everything else was not writable.

A

Here everything is writable, but not executable, so even with slashdev or whatever any partition user writes files to, and then uh if we are, since we are monitoring the whole container file system, they cannot execute anything uh and then, on top and this great scratch based image like uh uh like, we saw it was only helpful with static applications.

A

So with this solution, even the interpreter based application can be run without being fear of compromised or even with that baggage. If it is vulnerable and attacker gets access to the shell, they cannot do anything because uh we have this application monitoring what binaries are being executed.

A

So yeah, let's, let's look at a demo, a small demo. So here uh I'm on this node, uh it's a it's a single node, kubernetes cluster. uh I wanted also to show you all the container d uh logs uh what's going on here. uh Let's start, uh let's start a container, uh sorry a board and let's see.

A

And okay, it's running, let's, uh let's say uh well, I'm here, I'm doing I'm doing a cube, ctl exec, but let's say an attacker, gets access to the shell of this particular pod and so now that they have access.

A

So what can they do uh so they can create a new file, for example- and you can see like there are events going on up there. Okay, uh these are all the events that are for for debugging purposes. I have I'm showing like the like. This particular thing is allowed like we ran, uh touch this new file and then we see that this particular binary was executed, and it says that this was allowed um all right.

A

We can see that this new file accessed here new files, so let's try to simulate something like uh an attacker, tries to remove an existing binary and replace it with something new. So let's say our application depends on this binary and then it will eventually run like shell out and accept this binary. So we are trying so as an attacker, we are trying to remove that binary and create a new one in its place.

A

So this new binary is created. Let's give it executable permissions, and now, if I try to run that, like you, can see here up here uh down here, I got our operation not permitted.

A

Obviously it wasn't executed and up here I got a deny message uh as well like uh as a log, so that shows uh that we have successfully averted this attack where an attacker like either to try to replace existing binary or create new one or whatever it doesn't matter, since our application sees that this is something new that wasn't shipped with the image we don't allow it so yeah that was the small demo.

A

So uh what are the disadvantages of this whole life and notify based eventing? First of all, uh it's a user space program. It could slow down the container application due to the kernel mode user mode transition. uh Also, we are calculating in this particular case, since we are calculating those hashes it could it. It is also slower, but uh we have seen uh most of the most of these events or most of the execution for a long running application.

A

Let's say in this case engine x1, the the events are only generated at the start when the application is booting up or the container is booting up once the boot up has uh has been done there are, it is rarely that it really happens that there are events being generated.

A

So uh also again, like I mean it is not a general thing, it depends on the application as well how many events or new ends it will generate at runtime, but in a typical application it should not generate more applications. It will only generate them at the boot time. So with this uh will only will only increase the boot up time of the application.

A

The the second disadvantage is. uh These events are stored in memory, so if they are not processed, it could lead to memory exhaustion on the node, uh although, like uh in in our example code like we saw, we created an unlimited queue with unlimited marks, so yeah we can limit that we can. uh We can have uh different parameters so that the number of events that are stored are less the. The third disadvantage is uh we.

A

We rely on fa notifies uh to get events uh uh like open, exact permission events, but they are generated only when a file is opened using exec cv or exact cv. Ad or use lib system calls.

A

So if someone loads the file into memory and then jumps to it directly uh using mfd system call, then no event will be generated. If not, if I want to know it if notify works at the file system level, so yeah uh that will be skipped and then we won't know something like this has happened, so uh that's possible, but then again uh we can.

A

We can block the system, calls at uh uh second level uh using second profiles designed for uh for these containers, so yeah we can add security from from from using that feature of uh kubernetes.

A

uh The the uh the fourth point is like: uh it is also possible. Let's say: if uh an attacker pipes, uh let's say they do call some url and then pipes, that's scripted to python.

A

It is possible uh that, and and so with python uh the events that will be generated, where will be only for curl and the python interpreter itself. So the problem is, since both were shipped with the container image. They both will be allowed and we won't know it so uh the app so here the application has to be smarter and figure out like such bypasses are being employed.

A

We can even we can even disallow uh stdion for for python interpreter or whatever, and the final problem is like. uh We need to be careful as a as application designer like uh what events we subscribe to, because uh it is easy to create deadlocks with fm notify. Let's say if we subscribe to read events and our enforcer app also is reading files from the container root fs. Then it will generate even more events in the process and yeah. It could quickly spoil it down into deadlocks so yeah.

A

These are a bunch of uh disadvantages of this, uh but yeah going forward uh on the roadmap of what we're working on the we we plan on creating fine grain policies where users can uh provide some deny list, for example, of let's say some interpreters. They don't want uh to have stedia in access and things like that so yeah and uh I think with that. That's uh uh that's the end of my talk.

A

So thanks thanks for attending and listening.

A