Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Three Surprising K8s Networking “Features” and How to Defend Against Them - James Cleverley-Prance

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Three Surprising K8s Networking “Features” and How to Defend Against Them - James Cleverley-Prance, ControlPlane

Kubernetes' networking model simplifies the user experience, but abstractions can introduce and hide complexity under the hood. This talk challenges perceived trust boundaries in Kubernetes networking and demonstrates some non-obvious and counter-intuitive behaviours. Left unchecked, these issues can mean Kubernetes clusters present a wider attack surface than may be immediately evident. The talk will cover: * The external attack surface of a Kubernetes node * Enumerating externally available cluster information * Exploiting Linux networking to access internal pods and services * Misusing CNI configurations to access internal pods and services You will gain an understanding of these attacks and how to use them, learn mitigation strategies and pragmatic defences, and be able to protect your clusters to avoid compromise.

A

Good afternoon, everyone and welcome to three surprising kubernetes networking features and how to defend against them.

A

So, just a little bit of a background about me, uh I used to be a penetration tester, and these days I work at control plane uh predominantly being a security consultant, but I also help out with training and workshops and contribute to our ctf events, such as the one we ran at cloud native security days.

A

So what are we going to talk about today?

A

We're going to run through the external attack surface of a kubernetes cluster, having a little bit of look at what we can discover, having a little bit of a look at some of the underlying primitives of the cluster and some ways that we can abuse them.

A

We'll take a little bit of a deeper dive into some cni primitives in two separate parts and then we'll run through how we can defend against the attacks that we mentioned.

A

So why do we care about any of this.

A

Typically, in a kubernetes assessment, a security assessment in my experience will start from the perspective of a compromised workload, a compromise pod. Maybe we assume that a developer laptop's been compromised.

A

But there is still some attack service that is not accounted for. Let's have a look at what we can get from an authentic and authenticated perspective in the network, using some more classical techniques.

A

So we're going to start by having a look at open ports on our nodes.

A

So if you use one of our favorite port mapping tools and take a typical worker node, we might expect something like this to show up uh so as we're using linux. Kubernetes nodes, uh it's fairly typical to expect ssh to be open, but port 10256 and 10 2 5 0 are kubernetes components.

A

We can verify this just by jumping on one of the machines and verifying that q proxy owns 10.256, and this exposes a healthy health check api which doesn't really have too much attack surface.

A

So we'll move on port 10250 is a little bit more interesting, as this is the cubelet api in modern versions of kubernetes. This is authenticated, however, and so from a perspective on the network, where we don't have any credentials, there's not much to see here.

A

On that topic, however, if we did find any credentials, um there's an excellent tool, um cybrok called cubelet ctl that actually documents this api. It's a little bit outside of the scope of the talk today.

A

So if we move on to what a control plane load might look like uh in in a self-managed cluster, we can see some familiar ports from the worker node, um so we've still got those cubelet and q, proxy ports and ssh for administration.

A

We also have ports, tcp, 2, 3, 7, 9 and 2 380, which are used by the xcd theme and running on a control plane. If we're running in a configuration um where cds running on our control plane nodes by default, this uses mutual tls and so again not much. We can. We can gain here.

A

The most interesting port here is 6443..

A

This is the default port for the kubernetes api server and we can start learning some information, as the nmap scan indicates uh it. It's detected that there might be some ssl or tls uh service listing on that port.

A

So if we attempt to connect to it with openssl, we can actually dump tls certificate. Listening on that port, although there's a lot of information in there, the most interesting thing to us at this stage is the subject: alternative name field.

A

So this can contain two types of information, the first of which is a dns name and the second of which is an ip address, so by default, kubernetes will assign an ip address to each or sorry assign an ip address in the certificate to each ip address which the ip api server listens.

A

So in this instance, we can see that it contains the ip address of the api server kubernetes service, which is an information leak which could help us determine the kubernetes service network range.

A

This will be important later.

A

The api server also has one or two unauthenticated, endpoints, probably the most common of which is the slash version. Endpoint.

A

It's not a huge amount of information here, but it does at least tell us exactly the exact version of the cluster that we're running, which might be useful if we want to check for already known vulnerabilities or cves.

A

So, just to make this a bit easier, I've written a little march script, that'll, tell us both the subject names and that version information also makes it a little bit easier. If we're enumerating across several hosts.

A

So one area that is sometimes overlooked with network scanning is udp scanning so sometimes because it can be a bit slow or the nature of the thing is just a bit awkward, but in this scenario we have a good idea of what we're expecting to be running on the other side. So we have a higher chance of success, so we can look for ports which run common networking and storage protocols.

A

So, for example, here in the scan on screen, we can see that we have port 4789, which is commonly used for vxlan overlay. Networking, we'll talk a little bit more about that later.

A

So in this first section all these issues have been uh informational. We wouldn't really expect.

A

Them to be, we wouldn't raise them as any kind of exploit really, but in this next section, we'll have a little bit of a look at some of the primitives of the linux nodes that we're running and see. If we can achieve something that we might call an exploit.

A

So, for those of you that have set up a cube, adm cluster or a cluster with a tool like ud cube, adm will be aware that before you're able to run the tool, the node that you're running on has to make to meet certain requirements, and this is based on the features of the linux kernel on the node.

A

So one example of this is the ipv4 forwarding feature and what this means, if we turn it on, is that the the linux machine will root ip packets based on the rules it's configured with. So this will be its ip tables rules or nf tables rules.

A

So in the words of rory mcune kubernetes is a router, and this has some implications that we may not expect.

A

So, if you consider for a moment, we have a uh architecture, something like the above, where we've tried to harden our cluster by not exposing our api server to the public internet. So we implemented a jump host and how we've implemented it is. We've got one network, one subnet with our bastion and that's the same as the all of our kubernetes nodes.

A

Using the previous described routing functionality, we can actually add roots to this, but to this bastion that will allow us to talk to pods and services within within the cluster. So let me show you what I mean.

A

How are we doing everyone at the box see that grand thank you so to start was we used that script? I showed to enumerate uh some basic information about the cluster and we can see that here we're able to reveal the first ip in the service network.

A

So what we'll do now is make a small assumption.

A

That this network is a slash, 24 network.

A

We can increase it at a later stage if we need to, but for the time being, this will allow us to allow our allow us to add a route via one of the nodes that we've previously discovered.

A

A

What we'll now do is look for the commonly deployed kubernetes, dns component or core dns. We know this is going to listen on port tcp 53, so we can scan this service range for all ip addresses. Looking for that port.

A

Great looks like we found it, so we can see that uh core dns is running on the service, ip 10 100, not 10.. This is really powerful, because chordion s contains a lot of information about the state of the cluster.

A

So what we'll do in this next step is we'll use the wildcard functionality of core dns to list all other services within the cluster.

A

And although that's slightly wrapped, hopefully you should be able to see that we have the core dns service, the default kubernetes service and this other dashboard in the default namespace.

A

So we should be able to just connect to that dashboard to see what it presents. Let's try it.

A

And, as we can see, we can connect direct to that service, ip from our bastion without any authentication.

A

We're going to take this a step further and query core dns for all the service endpoints. So these are all the pod ips that the services and the cluster talk to.

A

And what this allows us to see is some of the pod ips in use by the cluster. So we can see down here. We have the 10 naught 102 range ip addresses, and this allows us to make a further educated guess as to what network range the pod network is using.

A

And we're going to guess that it's using the 10, not nor 16 range and again we can add this- add a root to this via one of our nodes.

A

Following this, we can scan directly across this range, and this will allow us to detect all pods, even if they weren't necessarily assigned to a service so over here we're just using three ports, but you could conduct a full port scan if you wanted to.

A

And we can see that we've found a pod ip address 1.8080 and again we can curl it to check. We can reach it for this example we're using the same dashboard, but with this time we're using the pod ip address, we can confirm that we can reach that service from our bastion without any authentication whatsoever.

A

A

Come on slides.

A

So you might think this is a little bit of a little bit dodgy functionality. Why does it exist? But in truth, it's actually used as functionality by some of the existing cni's.

A

So both romana and kubuta make use of this, and it's with good reason, because there are performance and complex complexity benefits to using this kind of routing over, say an overlay network.

A

But I'll leave you to decide whether it's a valid valid feature or is indeed a bug.

A

So so far, we've looked at layer, 2 networks, but in the real world it's likely that we'll have nodes deployed in different subnets different regions, different availability zones- maybe- and so we might want to bridge our traffic over a low 3 network. This is where overlay networks come in many cni's. These days will deploy with some kind of overlay network and under the hood, these will use either vxlan or ipnip technology.

A

So if you could consider for a moment at the following architecture, similar to before, we have a bastion host that we're asking users to pivot through, but this simulation before we have uh our kubernetes nodes are running in one or more other subnets that are not directly the same one as our bastion.

A

So, to give an overview of the the ipip protocol, we'll look at the layers two through two through four of the networking stack ipip encapsulation works by encapsulating a layer 3 ip packet with another layer, 3 ip packet in a standard pod, pod communication, the outer packet will contain the source node ip and the destination node ip and the inner packet will use the source, pod, ipop address and the destination port ip address.

A

However, if an attacker were to have access to a bastion, they may be able to forge or create malicious packets, so they may they would be able to change the content of any of these headers that we care about so for in order for an attacker to root to pods in our cluster, they would need to specify the right destination node ip in the outer ip packet and the right destination polyp in the inner packet.

A

However, they have some control over the source. Ip addresses.

A

The inner ip packet source ip address is important as the attacker.

A

The attacker can set this to influence the return route of the ip packet.

A

Let me explain that a bit so because we're because the kubernetes node has no does not understand that we are sending it ipip packets. We have a single single sided tunnel.

A

It has no route or has no rules to be able to encapsulate for the return journey. So therefore, we need to find a way of returning data to the bastion host.

A

If we set the in an ip packet source address to the attack to the attacker's bastion, when this packet gets unencapsulated.

A

And then a response from whichever pod we're talking to we'll use this source address as the destination in this manner, we can achieve a form of asymmetric routing which allows us to to retrieve a response from the node.

A

The outer ipp packet source address is important, as we can use it to bypass host firewall rules in some scenarios.

A

So I appreciate that a bit to take in. So let me give a little bit of context here.

A

This is a proof of concept script, which demonstrates the concept using the escapee packet, manipulation, library,.

A

We can see that in the outer packet, we're using the 10.123 ip address is the source, so this is just any other node in our cluster.

A

We can see that we're using the 10.123.0.20 ip address as our destination node iv.

A

This is the one where the pod we're targeting is running for the inner packet, we're using the best unip address as the source and the target ip address of the pod, which in this case is called dns as the destination p ip of the inner packet.

A

We then assemble the packet using scapy, targeting the any service cluster.local dns address, hoping to achieve similar results to before.

A

So we can see that this executes and that we send and receive normal dns packets.

A

If we look a bit deeper, we can see that the sent packet isn't is ip and ip encapsulated, and our response is not thus showing we're able to achieve asymmetric routing and communicate with services inside the cluster from the bastion from an authenticated perspective.

A

A

So not all overlay networks use ip in ip.

A

We also have vxlan networks.

A

Again, looking at the structure of layers, two through four.

A

We can see that this time it's much more complicated.

A

In vxlan, a layer, 2 ethernet frame and a vxlan header is encapsulated in a layer, 4 udp datagram.

A

The header fields that we care about in the outer ip packet are again the source node ip and the destination which, in the normal flow, is the node ip in the inner ib packet.

A

The pod ip will be the source. Ip address, sorry will be, the will be the source polyp and the destination ip address will be the destination polyp.

A

However, in vxlan we have some additional headers, so in the vxlan header we have the vni id while it's used in other vxlan implementations in all the kubernetes cni's. I've seen it's hardcoded to one.

A

We also have this concept of vtep, which is the vxlan tunnel endpoint. Unfortunately, for us this is a randomly generated hardware address and so to attack against vxlan in a reasonable time frame. We need to leak list in the cluster.

A

This is annotated on a node resource object, so the permissions we'd need would be get nodes.

A

So thinking back to our network architecture, where we have that bastion in a separate subnet and we have our nodes in adjacent subnets,.

A

Let's have a look at which headers an attacker would want to modify so in order to route to the right, node and right pod. The destination ip addresses in both of the ip packets will need to be set correctly as before and as before.

A

We can set the source ip address in the inner ip packet to achieve asymmetric routing.

A

We can also set the source ip address in the outer ip packet.

A

To that of another node try and bypass any host firewall rules that prevent us from sending packets to that node.

A

An attacker can fairly safely just hard code, the vni the vni to one, although if they know that targeting a cni that uses something different, they can protect, potentially modify it and, as previously mentioned, they would need to leak. The vtep address from the cluster via another method.

A

So, let's have a little bit of a look at how that looks in a proof of concept script.

A

So, in the script we set the outer source ip address to 10123 0.10, which allow, which is the ip address of another node, and we set the outer destination ip address 2.20, which is the target node.

A

We set the vxlan port to 4789, which is well known, and we enumerated during our nmap script earlier and we set the vni to rot one.

A

We include the vtep address that we leaked from the cluster and we set the source ip addresses the inner packet to our bastion and similarly to before we are going to be targeting the core dns service and use the service address of that pod.

A

We then can construct the packet using scapy and send to our cluster.

A

We can see we're able to successfully do so.

A

And that we are able to send a request in in the right format and receive a response in the right format.

A

If we look a little bit deeper, we can see our request is vxlan encapsulated as expected, and our response is not just a standard dns response.

A

So showing this shows that we can, if we have knowledge of the vtep address of a node, we can communicate with pods and services inside our cluster from an adjacent subnet.

A

So some of the the eagle eye amongst you will have noticed that in the proof of concept, scripts they're run as the root user.

A

Although we used it for the proof of concept, we could use the bash dev tcp and devudp sudo devices to achieve similar results. So we don't necessarily need the user to achieve this attack.

A

In order to mitigate some of the level 2 attacks discussed in the first section, I would strongly recommend that you isolate any bastion hosts or any untrusted workloads in different subnets to that of your nodes.

A

If we do, if we do that, it also means that we're able to write firewall rules, strict firewall rules to prevent any kind of other traffic traversing between these machines.

A

So, for example, in the in the second half there were no in in my lab environment, there were no rules between these two subnets to prevent that traffic, which is one of the reasons why we're able to do it in terms of kubernetes native things we can do. Network policies also help us here.

A

We can implement policies between workloads in our cluster, so between individual pods, and this will prevent us uh being able to communicate with pods from outside the cluster. So we wouldn't have been able to curl that unauthenticated dashboard.

A

Additionally, we can also add network policies around cube system if we wanted to prevent access to privileged components. So if we added a network policy to prevent, we can add a network policy to prevent access to core dns from outside the cluster, and it's probably a good idea in terms of general hardening.

A

I mentioned a couple of times that we could set the source ip address of the outer ip packet to avoid host firewall limitations.

A

This is more commonly known as ip spoofing and routers often have protection against this, but it's not always turned on.

A

So in my research, some cni's uh needed this, some didn't, and so you may want to investigate turning this protection on, to prevent these kind of attacks.

A

The linux kernel actually has a feature to prevent some of the uh asymmetric routing that I mentioned earlier called reverse path. Filtering um most distributions, don't actually enable this by default, so it's either off or in a permissive mode.

A

You could experiment with setting this to a strict mode so that these kind of attacks are not possible. Although it's unclear whether this will in fact affect any of the kubernetes functionality.

A

So you should test this well.

A

So I've released the nmap cumulative info script on github and the kubernetes, um the other proof of concept, scripts um that you're welcome, to use and contribute to, and that's my twitter handle. If you want to follow me, but thanks for listening.

A

Think we've got time for questions if anyone has any.

A

Yes, sir down at the front, I think we've got mike.

A

um Could you bypass network policies using the method um if you just used an internal ip address, so you set the package in you, don't expect for blind exploitation? You don't expect an answer, but people trigger something. So the question was. Could I could I use an internal ip address to buy person, network policy, good question.

A

I don't think so, because I think that the in order to buy part network network policy, you would need to be able to set the ip address of the inner packet which, which you well.

A

I guess it would be kind of almost like a csrf kind of situation if you had a workload where, because you'd probably also not be able to make a connection, so you'd need something that accepted udp packets, that you could make a permanent request.

A

It would be quite convoluted, I'm not sure.

A

Cool, if that's it thanks very much.