Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Uncovering a Sophisticated Kubernetes Attack in Real-Time - Jed Salazar & Natália Réka Ivánkó

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Uncovering a Sophisticated Kubernetes Attack in Real-Time - Jed Salazar & Natália Réka Ivánkó, Isovalent

As Kubernetes adoption continues to explode, the threat actors working on attacks are growing in sophistication. Simple mitigations and security best practices are no longer sufficient alone to protect production workloads. While tools like vulnerability scanning, signed container images, and distroless containers help, constant monitoring must take place in a running environment to ensure it remains safe from compromise. eBPF, an emerging Linux kernel technology, provides us unique visibility directly into any Kubernetes pod. Because pods on a node share a single kernel, a single eBPF program has full visibility to the entire node’s workloads. We’ll show how using such a program gives us the network and process-level visibility to detect and a live sophisticated attack on our cluster. We’ll finish by showcasing how security teams can easily put these same tools to use to protect their critical Kubernetes environments from threats.

A

Welcome to uncovering the sophisticated kubernetes attack in real time, kubernetes has become the de facto cloud operating system and every day more and more critical applications are containerized and moved to kubernetes.

A

This means kubernetes is quickly becoming a crucial component to secure. Today we as defenders are in the dark, because we lack the observability needed to detect and respond to attacks on our workloads.

A

Today, we'll show you how we can build out a detection program to uncover a sophisticated attack against a kubernetes cluster.

A

Hi, I'm jed salazar platform security at tesla. I, like mountains and distributed systems.

B

Hey my name is natalia and I'm a security engineer and I surveillance I really like traveling and flying on planes.

A

Observability in distributed systems is a crucial element in detecting issues and making changes to improve the stability and security of the system. So for sre we use things like prometheus to actually come up with metrics to define how we can actually improve the reliability of a service.

A

What we're advocating for is that security takes the same approach being data driven and using data to drive forward security improvements within our system.

A

So a lot of security today practices they really focus on the pre-fail and post-fail approach to security. Pre-Fail tries to predict the necessary protections that a system is going to require based on threat, modeling or an assumption of risk post fail ensures that you have the necessary logs for intrusion detection to be notified. If you get compromised despite those protections, what we're advocating for is that we actually change that paragon and we move from a pre-fail and post-fail to a pre-data and post data paradigm.

A

Data allows us to continually measure that our hardening and security configurations are suitable to protect against real world threats that are attacked that are detected by observability within our environment with observability. We will open your eyes to the reality of the threats in your environment.

A

So how do we secure this? This is the cncf landscape, basically the set of tools and systems that make up the kubernetes ecosystem. I like to call this the cncf threat landscape.

A

If you're tasked with securing kubernetes, you might look at this landscape and become overwhelmed. It takes a lot of resources and time to successfully build and operate a cloud-native environment, let alone secure it. However, different components of your stack have different levels of risk and prioritizing your focus. Where there's the least trust and the most risk is a good way to get started.

A

For example, it's reasonable to assume that the kubernetes scheduler is less of an attack vector for your cluster than an internet exposed pod that handles untrusted user input.

A

But what, if I'm following cis benchmarks and scanning my container images? Is that enough?

A

Don't do this not to say that vulnerability scanning isn't important? You should definitely not deploy vulnerabilities in the problem.

A

There are a number of different protections that we can implement within a system admission, controls like opa can define security standards and enforce policy for objects. Before they're persisted to etcd.

A

You can protect your containers by removing dangerous debugging tools like interactive shells, package managers, compilers and unneeded run times with distrolus base images. Bonus vulnerability scanning should be integrated to a container image, build stage and runtime sandboxing such as setcomp app, armor, g-visor and firecracker can limit the potential of privilege, escalation or container escapes.

A

Still each of these tools offer varying degrees of locking things down. So how can we verify that these protections give us sufficient protections from the real world attacks that we're actually going to see within our kubernetes environment.

A

So the way that we can enable that is by making security look more like sre. So if I were to ask how secure are you today, could you provide a quantifiable answer to that question? Are you measuring your assumptions on how secure you are?

A

We need to make security more like sre, by sharing a lot of the same sre principles like being.

B

Data driven and.

A

Measuring the data that we actually receive within an environment in this new paradigm, where security shares sre practices, security, collects data, observes the feedback and then makes continuous data-driven decisions to improve the security of our systems.

A

Are you scanning your images? Do you apply network policy? Are you encrypting your data at rest? How are you measuring the efficacy of these protections and can we detect if the service running in our environment is outside of our security.

A

Policy, the way that we do, that is, we trust but verify. This is an old security adage, so you place a lot of trust in your systems at deployment time.

A

For example, if you build up a container image, you should have this container image represented as infrastructure as code, so the configs that we deploy are pretty much well understood.

A

For example, when the cubelet launches our workload, we can assert that the configs have been statically analyzed hardening measures have been applied and we can basically account for every bit in the deployment. However, over time entropy reduces that trust. For example, an internet exposed workload gets attacked, an sre runs, cube, cuddle, exec and installs some debugging tool and fails to clean up after themselves.

A

Over time this entropy becomes a problem and observability verifies the trust that we place in a system throughout its entire life cycle. Until that trust is broken by some type of compromise. That's what we call detection.

A

So we want to test our security of assumptions and measure whether the protections that we're applying are suitable for the threats that we actually detect within our environment, so devsecops integrates security into devops and devops writes tests for code, for example, unit testing, which is ensuring that the behavior of a system is understood.

A

Observability continually validates this behavior. If you have security controls applied, for example, a network policy, do you have the observability, for example, via network flow logs or dns queries to verify that that network policy is suitable for the type of protections that are needed?

A

Capturing system calls and file system access from a container can validate the efficacy of our setcomp and app armor profiles on the left. We actually have unit testing and on the right. We have the observability that can test and verify that unit test.

A

These measurements can continually inform us whether the protections that we're utilizing are sufficient, based on the attacks that we've identified in our environment or if we need better protections.

A

So why bpf bpf is really the optimal tool to provide this observability in kubernetes, because a pod is just a set of linux processes running in the context of a kernel namespace. The pod makes system calls and implements cni network functions within the kernel where ebpf resides.

A

Ebpfx is a virtual machine in the kernel with a generic set of 64-bit registers and evf ebpf programs that are attached to kernel code paths. This means ebpf can intercept system, call events and react in real time with fully programmable logic.

A

Using eppf, we can build out a detection program and the best way to get started is to work with your red team or just attack your systems yourself.

A

Even as a beginner, you can start to collect the knowledge of tools and techniques and use ebpf to detect these these attacks and configure alerting options to detect attacks in real time, but the best way to detect is first knowing how to attack.

B

So in this talk for detection, we are going to use cilium and show how it uses evpf to very efficiently monitor our malicious network and process behavior inside of a kubernetes workload and give you identity over visibility into into those malicious behaviors.

B

So a little bit of background psyllium deploys as a demon set inside of a kubernetes environment. This means that there is a cilium agent running on each one of your communities worker nodes. These agents are communicating with the kubernetes api server to understand, but identities, network policies, services and so on and based on those identities of each one of the workloads deployed inside the kubernetes environment.

B

Celium generates a customized and highly efficient ebpf program to do connectivity, observability and security for those workloads.

B

Serem is able to both enforce what behavior happened inside of linux system, as well as very efficiently observe that behavior it can collect and filter out with observability data directly in the in the kernel and export it to user space as js and events where it can be sent to many external system, for example, cm to do investigation and alerting.

B

In this talk, we are focusing on the security use cases and security themes and how they would like to consume those data.

B

So this is how our json event looks like, which was exported to user space. By an avpf program, you can see the kubernetes identity over information like pods containers, labels and theos level process, visibility, information like binary arguments, docker id and the pid for simplicity. We will just use some python scripts directly consuming the json events that would be otherwise sent to an external system like cm system like elasticsearch or splunk.

B

In this talk, we are going to see two attacks, both happening on a live gk cluster in the first one. We are going to reach the host file system and break out from the container and on the second one we are going to download some sensitive data from an external s3 packet and for detection. We are going to use ceiling and the power of evpf so from an environment perspective.

B

Let's assume that jet has already accessed a gka cluster, that's what we can see on the right terminal and on the same cluster I have psyllium set up and running, which monitors all the kubernetes workloads and provide observability connectivity and security for them. For simplicity, I'm going to use a python script which would parse all the json events coming from the kernel and export it to user space via an ebpf program and then which would look for malicious behaviors.

B

Now the question is: can jed get access to the host file system? So, let's find it out.

A

So the easiest thing to do would be to spin up a pod with the privileged equals true flag. Note that kubernetes allows us by default and disables, and this disables all the sandboxing mechanisms inside of a.

A

Pod: okay: let's apply the privilege pod spec.

A

Now we have a privileged pod, which gives us the same permissions as root on the node. Why is this so powerful privilege enables the capabilities all of the capabilities which the pod, as you can see, has started. This includes capsis admin, which is basically the new route in linux.

B

And on the left terminal we can see that the generated evpf program was able to pick up the new container that has started. We can also see the privileges that include capsis admin.

A

So now, let's try to cue cuddle exec into that pod.

A

Let's see if we have access to the host's file system, I'm going to run cat etsy sorry cat, proc, partitions and see if we have access to the resources on that host.

A

Here we see that the partitions listed are actually partitions from the host.

B

And on the left terminal, the generated ebpf program was able to pick up the sensitive command detected.

A

So now, let's mount the host's file system inside the container, breaking the boundary between the container and the host. This is also known as a container escape.

B

And on the left side we can actually see the name space the source, but then the container.

A

So now that we have access to the host's file system, we can just directly cd into that mount. This gives us direct route privileges over that file system, which is effectively game over and we have full root access to the file system, which means that we can mutate things like you know the roots bash rc. uh We can change, we can add users. We can really do whatever we want. At this point.

B

So the continent escape is not the only thing that you want to monitor. For, sadly, most of the people are only looking for this. In the second demo, we will detect an attack version is able to download sensitive data from an external s3 bucket from an environment perspective. The setup is the same. So let's assume that jet has access to a gk cluster and from a workload perspective, he is currently seeing running application with multiple services like job posting and drag reader and with multiple databases like elasticsearch and zookeeper.

B

So, let's assume that he has already compromised the core api service and got into the core api port. That's what we can see on the right terminal.

B

So now he can run our arbitrary commands to discover the environment, do any lateral movements connect to external services and get access to any sensitive data and on the left terminal, I'm going to run a custom python script which processes the json events exported by the ebpf program directly from the kernel and which monitors all the malicious behavior that he will do.

A

So now that I've compromised the pod, I'm going to run the end command to see if there are any credentials or secrets mounted in the pod that I can take advantage of excellent. We can see that there's an aws access, key and secret key, as well as an s3 bucket name that I can use. So, ideally, secrets are mounted in a volume inside the container to avoid having to restart the container if the secret is rotated. But in this case I'm able to take advantage of the fact that they're environment variables.

B

And on the left left terminal, I can see that that the sensitive command got picked up by the generated ebpf program. I can see the namespace the source, but then the container.

A

So, let's access the s3 bucket and list the contents.

A

So here we see three folders. What's this super secret pdf.

B

And on the left side I can see that the sensitive command get picked up by the generated ebpf program.

A

So, let's download this super secret file and we can see this was successful.

A

B

And on the left terminal, I can see that the sensitive command got picked up by the ebpf program, with all the name speeds the source within the container, and then the network circuit that was initiated was identified as well.

A

So now that we've downloaded this super secret file, let's inspect the contents and see what it contains.

B

A

Okay, I don't know what to say there, I'm just like all right, but how do we translate system calls and network flows into an attack? Not everyone has the detection skills to identify an attack based on these signals alone.

A

The image on the right comes from a blog post called the pyramid of pain, and it identifies the value of detection data progressively from trivial to collect and the relative value that that data provides in a regular environment at the bottom. You have a hash of a file if you flip a single bit in a file, its hash value completely changes, so its value is minimal.

A

However, in the environment of kubernetes we can start to deploy the concept of immutability. Immutability enables us to declare patterns to understand the behavior of an object. If you need to change an object, you don't change it in place or patch it you replace it all together.

A

So we can use this immutability to unlock the detection of tactics, techniques and procedures or ttps, for example. If there's an execution of a binary, that's not part of the container image. We immediately know it's suspicious.

A

Immutability really brings a lot of benefits to security. All the binaries in a container should be packaged into a container image using a hardened base image like distrolus. It makes it easy to identify all the binaries in the container and those that should be running downloading binaries in a production container for purposes other than debugging is an anti-pattern.

A

So if there's a execution of a binary inside of that container image that isn't part of the container image, we immediately know that it's suspect what this further allows is. Those hash values that are trivial in value actually become very critical inside of an immutable environment, because we don't make any changes to any of the files inside of that environment. So a change to a hash in the case of immutability might actually become a tactic, technique or procedure.

A

So wrapping up just wanted to kind of go over. Some of the things that we talked about through this talk so first is that to measure the efficacy of the security in your environment that requires observability.

A

So the key thing here is starting to collect the right data so that you're actually going to be positioned to be able to detect the sophisticated attack that occurs within your environment. um Today, we've talked about a couple of kind of simple attacks, but the key here is that we're actually measuring that data to be able to detect those things.

B

Oh and by the way, cilium is going to open source the ebpf observability tool that was used today for the demos.

A

Awesome so when we actually start with this observability, we can detect the the types of behavior that we expect in a system when things go wrong, for example, whether we have a compromise in the environment, that's called detection and really, if you don't collect this data, if you're actually lacking this data.

A

Hope is your best strategy and if I can take one thing away from the sre book, it's that hope is not a good strategy and, lastly, we just wanted to give a huge shout out to everyone on this slide. Everyone here has given us either feedback directly or helped us somehow throughout our kubernetes security learning journey. This talk wouldn't be what it is without these people. So thank you very much.

A