Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Know Your Enemy: Mapping Security Risks Using Threat Matrix for Kuber... Yossi Weizman & Ram Pliskin

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Know Your Enemy: Mapping Security Risks Using Threat Matrix for Kubernetes - Yossi Weizman & Ram Pliskin, Microsoft

In April, Microsoft released an updated version of the Threat Matrix for Kubernetes which was originally released in 2020. The Threat Matrix is a knowledge base for security threats that target Kubernetes. This matrix was the first attempt to systematically cover the attack landscape of Kubernetes. In this session, we will explain how defenders and SecOps engineers can use the matrix to protect their Kubernetes workloads. We will demonstrate how a real-world attack is mapped to the techniques in the matrix and how organizations can measure their coverage to the attack using the matrix. Inspired by the Threat Matrix for Kubernetes, MITRE expanded their ATT&CK framework to include also containers. In the session, we will examine the differences between the Threat Matrix and MITRE ATT&CK and explain how users can leverage both matrices to gain a better security visibility for their environments.

A

Hello: everyone, I'm josie, weisman, a senior security researcher at microsoft and with me, is ron pliskin, a principal security resource manager at microsoft.

A

We will talk in this session about the threat metrics to kubernetes, which is a knowledge base of the security threats that target kubernetes.

A

This metrics is one of the first attempts to systematically map the threats of kubernetes and we will see how we can use it to improve the security of our environments and measure. Our security coverage to potential attacks.

A

Ram will start with some background of this project so ram. Please. The state is yours.

B

Hi everyone I'm ram, I'm leading the security research team for azure defender at microsoft. Please allow me to outline what you're expected to get from today's talk. First, we will introduce the problem space and what led us to focus on the orchestration layer for containers.

B

Second, the throat metrics for kubernetes. This is the outcome of our research of kubernetes strength, landscape. We will also go over an attack we witnessed and we will unfold it using the metrics.

B

Then we will present an example of how organizations can leverage this knowledge base and honestly to better secure their kubernetes workloads.

B

Lastly, we will discuss the collaboration with folks from mitre to establish mitral support for containers. We will wrap up with highlighting the differences among the two matrixes.

B

To set up the stage we will start by introducing the problem space, and now we got tapped into it so before kubernetes was a thing before docker was a thing. Folks were running distributed systems largely either on bare metal or in virtual machines.

B

When contact when containerization started to take off it, provided a way for consistent and repeatable deployments, with kubernetes being so widely adopted around two years back, we were tasked with building a plan to protect users running kubernetes workloads. That's basically how our journey started with kubernetes, built as extremely robust obstruction layer. We knew we would need to adjust our security perspective.

B

Normally we are, as we are, natively integrated to azure backbone. We tend to leverage any internal signal we could have to strengthen security offerings, but with kubernetes being platform agnostic, we aligned accordingly and shifted our focus towards the kubernetes layers.

B

Let's linger for a second on the diagram in the right. We split it into into three different levels, starting at the bottom. The cloud layer which can often be referenced as the platform layer consists mainly of the control plane of the platform and the node's operating systems.

B

Both these fronts already have an extensive knowledge base, mainly over miter enterprise matrixes, so we knew we would have to dive into into kubernetes apis, both the ones that are being serving the users and also the southward phrasing to the pods themselves or internal benefits. Services.

B

So our goal was to map threats targeting kubernetes and when we started looking into kubernetes landscape, we discovered a variety of attacks with each such as a very different aspect of kubernetes built-in mechanisms we figured. We should map those different areas, so we could keep track with the interface we would like to further explore and with mitre being largely embraced by the community as the go-to place for learning and evaluating security coverages, we figured we should leverage attacked, metrics structure, and so we started defining our own kubernetes-centric ttps.

B

This was later turned out to be jose's great publications from april 2020, as the first threat knowledge base for kubernetes later. In this talk, we will also present the outcome of partnering with mitra, which, which resulted with the release of the attack metrics for containers.

A

Thanks ron, so, as ram said, we saw that there is a gap when it comes to understand the unique security threats, the target orchestration level of kubernetes and that brought us to build the threat metrics to kubernetes, which focus on this specific layer.

A

In this slide, you can see the second version of the matrix that we released earlier this year. Like mitre, we split the metrics into tactics which are the columns in the table in the dark blue. In each tactic there are techniques which are the specific methods that attackers might use.

A

So we won't go over all the techniques right now, of course, because there are many, you can read the full metrics in the link below. But let's see three examples: one is access manager, identity, credential, second, coordinates, poisoning and third images from a private registry.

A

The first one is access, manage identity credential in some cloud providers you can allocate identities to cloud resources like virtual machines, for example in aws you have ec2 roles in azure. You have manage identity or manage service identity in its former name, you can access to the token of the identity. From the virtual machine itself, and also from the containers that run on that virtual machine, so if attackers gain access to a container, they can potentially grab the token of the identity that is attached to the underlying node.

A

With this token, they can later access. Other cloud resources depends on the permissions of this identity. Of course, by the way in some cloud providers, you can prevent containers from accessing this identity token and basically mitigate this threat.

A

Another example is core dns poisoning the technique, talks about poison the clusters, dns not by traditional dns poisoning techniques, but by modifying the core file, which is the configuration file of core dns.

A

This configuration is stored as a config map in kubernetes in the cluster. So if attackers have access to this configmap, they can poison the dns in the cluster.

A

You can see that it's not a traditional dns poisoning that affects any service. This is a unique threat. The target kubernetes specifically.

A

Another example is images from a private register which talks about how attackers might steal images from private registries once they have access to container in the cluster, for example, by leveraging the cloud's managed identities of the working nodes that we talked about earlier.

A

This identity has many times permissions to pull images from the registry that used by the classroom.

A

So, besides being aware to the various threats that are kubernetes, what else can we do with the metrics? So the answer is that we can use it to measure our coverage to real-world attack.

A

Let's see how we can do it.

A

So here is an example of a very simple attack that happened earlier this year and targeted coop flow coop flow. Some of you are probably familiar with. It is a very popular framework for machine learning tasks that run on top of kubernetes.

A

I think that there are also some sessions in this conference about crypto.

A

Some of the functionality of google is exposed via crds and some by a centralized dashboard in some configurations. This dashboard doesn't require any authentication in such configurations. If the dashboard is exposed externally to the internet, it basically allows a free access to a management interface of good flow, which is obviously a bad thing.

A

Earlier this year we saw a large scale campaign that targeted internet, accessible, kubeflow workloads. The attackers used exposed dashboards for deploying a malicious pipeline. A kubflow pipeline is a service of kubeflow that allows users to create machine learning pipelines which is based on algo workflow.

A

Here you can see that by using the dashboard, you can simply create a new pipeline and that's what the attackers did.

A

Using pipelines, the attackers deployed crypto mining containers.

A

Those containers use a legitimate tensorflow image from docker hub. They also leverage the fact that in some cases, cooper workloads use gpus for machine learning in order to increase their gains.

A

Here you can see the attack flow, the attackers accessed the queue flow dashboard using the dashboard. They deployed a new pipeline which created a new container eventually by the pipeline controller, the container which ran a legitimate tensorflow image, fetched a cryptominer from github and ran it.

A

You can read the full report in this link, so we saw an example of an attack. How can we now use the threat metrics to measure our coverage to this attack.

A

So step one is mark the relevant techniques. So here in red, we mark the relevant techniques for this attack.

A

Attackers used exposed sensitive interfaces, which is the keyflow dashboard in this case. So this is the initial access the attackers use containers for execution and persistency.

A

They use legitimate names for hiding their activity, so this is the defensive asian, bipod or container on them. Similarity as we said, they leveraged the pipeline controller permissions for deploying a new container.

A

So this is a container service account and the impact is resource hijacking, which is the cryptocurrency mining.

A

So we mark the relevant techniques step two is evaluate our coverage to those techniques that we just marked for exposed sensitive interfaces. We can monitor exposure of services to the internet, for example, monitor load balance of service creations. Ingress objects are also relevant in openshift, it could be roots, etc.

A

I really want to emphasize this point a very large portion of the attacks that we see against kubernetes clusters start with an exposed, sensitive interface. It affects everyone, literally everyone from small organizations and individuals to huge organizations with fairly large information security departments.

A

Misconfiguration in kubernetes is a serious problem. It is absolutely crucial to monitor your environment for such misconfigurations and when you find one to act immediately, because once it happens, your cluster will become compromised very quickly.

A

So that was initial access for execution and persistency. We should monitor new containers that means to monitoring the images that are used to monitor entry points or arguments to monitor configurations such as privileges, capabilities volume, mounts etc.

A

For name similarity, we can monitor the images in our workload and make sure that only known images and legitimate images are running. We once observed an attack on a kubernetes cluster. Not this attack another one. The attacker used a malicious container as well. We notified the victim and we told them hey. Your cluster is compromised. You should be aware- and they are actually argued, argued with us and told us that this is their end-to-end test in containers and it's fine, it's legitimate.

A

We showed them that the container runs a malicious image and the attackers just used a little bit name from their environment. So it's really important to monitor those things for container service accounts.

A

We should monitor suspicious operations of service accounts. This can be done by using the kubernetes audit log or some people call it keyboard it. This is a very powerful tool that gives you a great visibility to what happens in the kubernetes control plan. You can see basically all the operations in the cluster who did it. When did it happen from which ip and more so it's very useful? When we want to monitor the activity in the cluster for resource hijacking, we can monitor it from two levels.

A

In the orchestration level, we can monitor once again the containers images, entry points, arguments also exact commands.

A

In the node level, we can monitor the running processes, memory, consumption, cpu, consumption, etc. So this was an example of how we can use the threat metrics to measure our coverage to real-world attack.

A

Now ram will speak about mitre's project for of adding coverage for containers in minor attack and the differences between minor attack and the threat metrics ram. Please.

B

So let's talk a bit about micrometrics for containers, so in december 2020 based on the community interest and if I may add, with some inspiration from microsoft, truth metrics might just started to work on attack metrics for containers and in april 2021 the containers matrix was added to attack v9, ok. So if up until recently, we had no solid source of information for learning on possible attacks targeting kubernetes now we have two great sources.

B

So let us pause here for a moment to highlight why these matrixes are different.

B

So first attack is focused on in the wild adversary, behaviors, meaning that might consider a ttp only when it happened in the when it already has been witnessed. Their investigation is focused specifically on gathering intelligence of what adversaries are actually doing, versus what researchers and lab teamers can do.

B

Our motivation is different. We are research oriented, and so we do not have the luxury of waiting for the community to share real world attacks. The second main difference is that the tech metrics is built on existing techniques.

B

Mitel's motivation was to leverage an existing ttp, so in case there was an equivalent technique, for example, from the linux metrics. They prefer to supplement it with a container tag and therefore names and descriptions are often times different as well. Also on some few cases, I think we had some disagreements.

B

Each stakeholder had a different interpretation of a certain activity and that also led to few misalignments with. That being said, it's important to also highlight the core similarities, so please note that each matrix combines adversary, behavioral techniques for both orchestration level and container level.

B

Secondly, both metric cells can be considered as an obstruction layer, for example, if an advisory installs, a crown job within a container that behavior would be covered by the linux metrics instead of the containers matrix to learn more on the evolution of the metrics and how the and the process behind of building those please read. Mitra and microsoft join publication.

B

At this point, let's recap the attack that jose laid out and see how it's been reflected on each of the matrixes. We can use the arrow to map between techniques so from the initial access tactics. The exposed, sensitive interface is mapped to external remote service. On mitros metrics from the execution tactic, the new container is mapped to deploy container on mitral side and the container name. Similarity is mapped to masquerading and the resource ejecting is actually remain. The same.

B

The discrepancy comes from the backdoor container technique and the container service account.

B

Let's look into the vector container technique, so attackers can run their malicious code inside a container within a cluster and by using kubernetes controllers such as demon set or deployments, the attackers can ensure a constant number of containers will be executed will be executed with their malicious container.

B

We chose to represent this technique under both execution and persistent tactics for execution. It's obvious why it's fit there for persistence. We mainly referenced the to the nature of kubernetes controllers that will that will safeguard the attacker's payload remain running.

B

Miter choose to have this technique only under execution.

B

Another discrepancy was the container service account and, according to mitra, lateral movement hasn't been observed in real world attacks for containers, so they have decided to not include it.

B

To wrap up, I will share my key takeaways, so first, I hope we demonstrated the importance of building a knowledge base and always keep challenging challenges. We also showed our organization should measure their existing capabilities against it.

B

Number two defenders should close defenders with close ranks. Microsoft. Collaboration with mitre is only one instance of our defenders and blue teamers across the industry to join hands. It's crucial against the challenge we are facing. Just wanted to call it out with that. Thank you back to you.

A

Thank you ron, so my final words, kubernetes is evolving from all aspects, including security-wise.

A

Kubernetes today is much more secured than it was several years ago or even last year, but as kubernetes gains more and more popularity, it also gets more and more attention from the attackers, so the threats are also evolving.

A

So that's why it's really important to be up to date and we also updated metrics over time to keep the pace with those changes and new threats, and we already released a second version of this metrics this year and we plan to release more versions of this matrix.

A

When we released the first version of the metrics last year, we were very excited to see how the security community and also the kubernetes community, adopted these metrics. It showed us that there was a real need for such knowledge base and we were very happy about it.

A

We really hope you enjoyed the session and understood what is the threat metrics and how you can use it to make your organization more secure. Here are some useful links which are the links that appear in the deck. So thank you very much.