Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Armoring Cloud Native Workloads With LSM Superpowers - Barun Acharya, Accuknox

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Armoring Cloud Native Workloads With LSM Superpowers - Barun Acharya, Accuknox

Speakers: Barun Acharya
Containers are not protected by default as the various tools for security into place provides perimeter security at the host, or the network and not necessarily the workload itself. LSMs(Linux Security Modules) provide with security hooks necessary to set up least permissive perimeter for various workloads. KubeArmor is a cloud-native runtime security enforcement system that leverages various LSMs to secure your workloads. LSMs are a really powerful system but they come with a high barrier of entry, steep learning curve and do not provide enough metadata for modern cloud native workloads. This talk will be about how KubeArmor leverages LSM superpowers to abstract away the complexities to help protect modern cloud native workloads, how we leverage eBPF to provide context about what's happening in the containers, how various kernel primitives fair with each to protect modern container workloads and what design considerations/challenges for integrating various LSM into KubeArmor.

A

Hello, everyone thank you for joining me on my session on armoring cloud native workloads with LSM superpowers. I am varunachada. I am a software engineering intern at equinox, working on open source, Cloud Security Solutions, with the various advantages that containers provide organizations worldwide are rapidly adopting containers in one form or another.

A

We have applications running in containers across nodes across clusters, with the rise of adoption of modern Cloud native infrastructure, so has reserved the cyber attacks on the same containers are not really protected by default, as the various tools for security into place provides perimeter security at the host or the network, and not necessarily the workload itself.

A

We have security, focused node images, hardened kernels, strict configuration, but there's a need to enforce security at container level, hence the need for container security.

A

We usually have some static analyzers in place, but they only let us know about what known vulnerabilities are. Containers are vulnerable to, but recent vulnerabilities, like log 4J and pawn kit, could not have been detected. Statically. That's like zero days. Malware meant to evade analyzers and privileges escalation. Attempts manifest at runtime and cannot be detected statically.

A

So how do we deal with vulnerabilities that manifest at runtime? We deal with them at runtime, so container, runtime security? We actively monitor, what's happening with our containers at runtime and try to detect for the malicious in depth. But how do we go about this to better understand? Let's go through an analogy: let's imagine, containers as a private apartment, a private apartment with John and Jane each having their own rooms. They have a kitchen and some maintenance staff around John and Jane value.

A

Their privacy Jane doesn't like if Jon pokes into her room, neither does John if Jane does the same, but they both do like to hang around in the kitchen for functioning.

A

They generally don't want the maintenance staff to roam around in the apartment, but in case something breaks like the shower is broken. The gas is not working or some electrical Appliance is malfunctioning. They need the maintenance staff around.

A

Let's take this analogy back to Containers. We have now a WordPress app, a MySQL app. They each have their own configurations and directory. They need shared libraries to function, and then we have cat and PS and other maintenance. Binaries that help us around in case something is broken but like I mentioned, Apache WordPress should only be able to access the WordPress, config and MySQL app should only be able to access the MySQL directory, but what?

A

If they try to access something else themselves or the files are being accessed by someone else specific other than the binaries? We know that something malicious is happening. They cannot really restrict Jon and Jane to tell them what to do violating their human rights, but we can restrict our containers to tell them what to do and prevent them from from performing actions like these, we have established the fact that containers are not a black box. We need to profile and understand the entities inside the containers as well.

A

Even if we imagine whatever is happening inside containers won't have anything outside our containers have access to the outside configurations as well. Containers have to network with the world and the other peer Network containers. They access capabilities from the kernel, their access to host Define Mount points as well.

A

Now what? If a malicious binary somehow enters the container, it would gain to access it would gain access to all these Mount points, Network and capabilities. Hence, there's a need to understand the entities inside our container. We need to profile our containers beforehand and have fine-grained access control such that, even if a malicious binary sneaks in somehow it won't be able to do anything.

A

So so, let's see what we need to enforce this fine grained Access Control at runtime. There are two ways to go about it. Once we detect the malicious activity, we alert the activity and take action on the activity after we receive the alert or as soon as the malicious activity is detected, the action is taken by a security engine already, and then we receive the alert for further investigation.

A

The prime method is called acing enforcement. Essentially we restart detection engine receives the kernel event, it alerts us and we then decide to terminate the activity, but once this entire cycle happens, the malicious activity could have already sneaked into deeper areas of our container. So the other method is inline enforcement. As soon as we detect that there's a malicious activity, we deny it then and there itself, and then we received the alert for further investigation that what might have triggered this.

A

So what kernel Primitives do we have to enforce this inline inline enforcement of systems? We have LD preload, we have sec.com, that is secure Computing and we have Linux security modules.

A

Let's take a first date: let's take another look at ldp load. First, LD payload is a user space control. It overrides Dynamic libraries and applications go through these libraries, so we have fine-grained access control over them, but attackers can invoke CIS calls without going through the dynamic libraries at all suppose, go Application statically embed their binaries with the library files, so they never need to go through the modified library at all.

A

Next, second, let's say sandbox. Our containers with various system calls what what system calls is accessible to them. It is natively integrated with kubernetes, Docker and other container runtimes, but the issue here is that Dynamic enforcement is not possible and there are limited filtering options. We cannot decide inside a container what application needs to access, what systems response it treats container as a single entity.

A

Finally, we have Linux security modules. Linux security modules include app armor, SL Linux, BPF LS. These are mature ecosystem workbooks, with Decades of development ongoing on them.

A

These do not suffer from a kernel, race condition, called time object and time Optics, but they are not integrated with the modern context and there's a steep learning curve associated with the policy language of these elicits.

A

So how do we integrate these security parameters with our modern Cloud native workloads?

A

We have Port security context. Pod security context defines the privilege and access control settings for a port or a container security content. Settings include, but are not limited to Second capabilities, app armor and SL Linux setcom capable. Let's take a look at SEC comp capabilities and SC Linux. First, they, the issue with them, is that they treat container as a single entity and apply the security posture across the container.

A

But, as we discussed earlier, we needed that fine growth, fine grain control over the entities inside our contactor, then we have a power mode. App armor, app armor provides us with the prime gained access control we needed and it is well integrated with the modern ecosystem. We can apply annotations to link it to our apparent profile, but then we have a learning curve associated with the apartment policy. Language also app armor is not readily available on all the systems.

A

A bigger issue is here that apartment uses ordered logs to generate the Telemetry events, and these Telemetry events do not have enough context. We never know which container triggered this alert and there's a lot of overhead required to to realize that context.

A

So defining a problem statement. We need access control on container entities. We need a declarative way to manage policies for these Access Control. We need inline policy enforcement and importantly, we need the Telemetry events to have the context introducing Cube armor Cube armor is a cloud native runtime security enforcement system that restricts the behavior, such as process execution, file, accesses and network operations of containers and nodes at the system level.

A

Let's see how Q barma does this under the root we have q, Barber listening to various probes in the kernel, listening to the events and acting as a detection, it sends out the log and Telemetry to the users for them to analyze.

A

Cube armor then listens to the security policies that user sends and converts them into the kernel space rules to have inline Access Control inside containers, utilizes ebpf for observability lets us extend kernel in a safe and performance. It has established itself in the cloud native space and is a go-to solution for observability these days and then Q barma utilizes lsms for enforcement. When Q Barber receives the policies it converts them into app.

A

Armor profiles, dpfls and bytecode, or SC Linux labels which are enforced and are responsible for the decision making inside corner, as we established earlier. Lsms are a proven technology for Access, Control and paired with ebpf and leveraging information from container apis. We have a holistic solution per container runtime security system.

A

Let's check out Q barber in action, some context about what we are going to see in the demo. We will have an Apache, WordPress MySQL application, which in which we will showcase how we can allow specific entities and deny the rest how we can restrict our application. Access to certain kernel calls so also kubernetes, auto mode service account tokens, even if we or like set it or set Auto Mount to false, which Bill might need access to those tokens.

A

So we will showcase how we can lock down access to a particular application inside our container to that service account token. And finally, we will showcase how we can secure our Port data for data, including MySQL directory files or configuration files.

A

So, let's get on to the demo.

A

We already have a kubernetes cluster running with Cube armor installed and our MySQL WordPress application installed as well. Let's see how Cube armor is configured to run here.

A

We have a two node setup with container optimized, OS and Cube. Armor is configured to run ppf lsf, PR armoring of two of our workloads here, MySQL and WordPress, and currently they have no policies apply equivalent.

A

Let's start with applying some policies.

A

So this is how a sample Cube armor policy, looks like we Define it as cube armor policy we can specify which selector labeled it should match upon to. We have file Network processes and other entities that we can specify and, finally, our action, which can be allowed to specify whitelisting policies, block 2, specify blacklisting and we can audit things as well. In this policy, we are trying to block access to anything inside the misql directory.

A

Let's try applying this policy.

A

Also, let's, at the same time, let's try to monitor our alerts and telemetries as well.

A

So now that we have applied our policy, let's exit exec into R minus 12 for.

A

As soon as we try to write a file to this directory, The Bash told us that permission is delayed and a relevant Telemetry event was generated for us, the Telemetry event included what was the host? What name space What part? What was the container ID container image all the relevant information that would not have been available in our audit logs is available here, and these it also includes, which was the policy that caused it to be denied, and what was the file process name file?

A

Name all the information is available to us, which can be useful for debugging and tracking down how and where this malicious intent happen. So, similarly, let's try to secure our service. Account opens.

A

We have this policy where we deny access to service account tokens in gender, but allow access to this specific service account token by cat. So now, if we move into our WordPress application,.

A

If I try to access my service account open I can access the service account open, since I mentioned that I should allow access to the service account token. But what, if some other binary in this specific a container tries to access the service account broken.

A

We never applied the policy.

A

So now CAD can still access the service account token, but what, if more, tries to access the service account token we see the permission was denied and a Telemetry event was generated to us that more try to access the service account token. Please look into it.

A

One more thing I mentioned in our analogy, is that maintenance tools will ideally maintenance tools shouldn't be allowed to execute in the production containers, but we might still might need in a developer environment to debug things. So, let's try to restrict access to the.

A

So now this policy includes that we should deny access to apt and app get to be executed.

A

Now, if we try to execute.

A

Either app or Advocate yep, they are denied and we have the Telemetry data with us for that.

A

A final use case was that how we can restrict Network accesses down to the process level. So if we try to apply.

A

This is an allow policy where we have mentioned what what processors can draw be accessed by what processes can TCP be accessed by and what processes? Why can we accessed? Can access UDP? This is generated after profiling, our WordPress application. Now that we have applied this.

A

We see that curl is not part of this.

A

Policy, so if we try to access anything using curl any network using curve, we will see that the permission will be denied. We can see that we could not resolve the host for curl and we had a relevant denial to this event.

A

This this might have been just due to UDP. So let's try to access utcp as well.

A

We couldn't connect to the server, so we have successfully like limited down which bot all can access, TCP and UDP.

A

So to summarize, Q barma provides you with access control on a access control on container entities. We provide you with a declarative way to manage policies for Access Control. We have inline policy enforcement and we have Telemetry data with context.

A

Thank you. You can reach out to us on Cube armor slack. You can check out more details about reformer on qbarma dot. Io I am available on Twitter as demon1024, and you can know more about B at baruch.cc.