Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 2 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Picture this! Solving Security Problems Visually with eBPF - Liz Rice, Isovalent

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Picture this! Solving Security Problems Visually with eBPF - Liz Rice, Chief Open Source Officer, Isovalent

eBPF is a wonderful platform for the next generation of security tools, but there can be a big gap between detailed events at the kernel level, and meaningful, understandable information that security and platform teams can act on. Let’s look at some of examples of graphs and visualizations that aggregate information collected through eBPF, that can help us answer security-relevant questions much more easily than wading through logs.

A

Thank you, hello, everyone so, who here has heard of ebpf quite a lot of hands? Excellent, so evpf is a Hot Topic in Cloud native for very good reason. It's a wonderful platform for building Network, observability and security. Tooling ebpf lets us detect and react to events from within the kernel whether those events are triggered by something happening in user space or events that happen in the kernel, like a network packet reaching some point in a network stack and the beauty of this in Cloud native.

A

Is that because ebpf programs run in the kernel, we only have to instrument nodes rather than instrumenting each individual workload.

A

All of the applications that run on a virtual machine share one same kernel and a process whether it's running inside a container in a pod or even directly on the host is using that same kernel and whenever it wants to do anything interesting and potentially security relevant. It's going to have to ask for assistance from the kernel.

A

So if we have eppf programs in the kernel, we can detect and even affect what's happening in any user space application or process, and if an attacker manages to run malicious code on your machine, ebpf tooling will be just as aware of it as any other process.

A

There are several projects in the cncf today and sub-projects that leverage ebpf for connectivity and observability and security. You just heard about Falco, for example, and today, I want to talk about the power of taking the events that these kinds of tools generate and using them to create visualizations that can help us solve security, specific problems.

A

So let's leave Seattle and go and visit an organization. That's actually been using Cloud native tooling for a very long time. The Empire is a huge user of Open Source software adopted kubernetes a long long time ago, and if we look at what's currently running around Endor, we can see there are a few Tie fighters Milling about. There are a couple of instances of well. We run the debtstart with high availability.

A

Obviously the Empire uses psyllium for network connectivity, so we can look at the network flow logs that are generated by psyllium's Hubble component and we can see all the traffic that's flowing in the spacecraft around Endor.

A

The information that goes into these Hubble logs is collected by psyllium's ebpf programs that are inserted into various points in the network stack and because psyllium is aware of kubernetes identities. The network logs, don't show us IP addresses, they show us which pods are involved in each Network flow, and we can also see details of what's happening at layer 7..

A

So this is giving us lots of really great information and lots of really useful details, but the Stormtrooper who's in charge of security on the Death Star knows that it would be a lot easier to understand, what's happening and which spacecraft are communicating with each other. If that was presented in some kind of visual pictorial form.

A

Here's the psyllium Hubble UI, which takes all those individual Network flow logs and builds up a picture of which services are communicating in the end or namespace everything seems okay, we can see some Tie fighters are talking to the Death Star talking to the Star, Wars API totally makes sense, and also to Disney, because I think they're negotiating contracts on some kind of reality. Tv series that they're going to film on the Death Star and on our security Stormtrooper, wants to put some Network policy in place.

A

He wants to know that only Empire traffic is going to be able to flow in this namespace, but Network policy can be a bit fiddly and he doesn't want to get it wrong. He doesn't want to upset the generals, but this Stormtrooper knows that Hubble doesn't just generate Network flows. It also generates Prometheus metrics and in particular, he's interested in the Prometheus metrics that are generated about Network policy verdicts.

A

So whenever a policy decision is made whether to drop or forward or redirect a packet that generates metrics, so our Stormtrooper has set up a grafana security, dashboard and right now, there's nothing on it, because we don't have any policies in place yet. So there are no policy verdicts to show, but let's start by putting in place and allow all policy so that every single packet will generate a policy event.

A

So the allow all policy is going to apply to all the spacecraft in Endor and it's going to allow those craft to receive packets. That's Ingress traffic from anywhere and it's going to allow sending packets that's egress traffic to anywhere and when we put that policy in place once things have settled down, we'll see a pretty steady state of we'll just look at Ingress traffic for now that's being generated by the Thai fighters around Endor.

A

All of that traffic is matching on policy type. So, on the left hand side we can see all the packets are being forwarded and on the right hand, side we can see they're all matching an all policy type.

A

So now our security stormtroop is going to start with the Ingress traffic and introduce a policy. That's going to make sure that traffic can only flow to the Death Star. If it comes from Empire spacecraft and when he applies that policy we can see, traffic continues to be successfully forwarded, so the left-hand side stays the same. But on the right hand, side we can see that the traffic is now being matched by a layer, 3 specific policy rather than that, allow all policy.

A

And if we remove the Ingress rules from that allowable policy, everything is going to basically stay. The same because all our expected traffic is matching that end or specific policy. So that's great.

A

But we're hearing a rumor that there are some Empire State spacecraft in another star system and they're trying to send us some messages from Hoth, and if we check the security dashboard, we can see that those packets are being dropped. This is a problem because some of this traffic from Hoth is coming from an Empire star destroyer and we're going to be in big trouble if that traffic doesn't get through.

A

So our security Stormtrooper adjusts the policy and allows traffic to come from anywhere, provided it comes from an Empire spacecraft, something that's labeled with the organization of Empire and things improve. We see fewer packets being dropped and we can see that the Star Destroyer traffic is being forwarded correctly, but we're still blocking traffic from some X-Wing Fighters that are being flown by the Rebel Alliance and also from some known as the Millennium Falcon whatever that is so.

A

This all seems good, but we have heard another rumor that there's a spy there's some kind of rebel Android thing that stowed away and is trying to exfiltrate the plans to the Death Star.

A

Fortunately, our Stormtrooper team are running with psyllium's tetragon runtime detection tool and the tetragon logs are also hooked up into the security dashboard.

A

And if we check out the executables pane of the security dashboard, we can see some highly suspicious activity right there. At the top highlighted in red. We can see that somebody is running netcat. This is not expected, but we can see exactly which pod did it when it happened, so we're going to be easily able to root out the spy and throw them into the trash compactor.

A

So I've shown here some dashboards set up in grafana that the Empire security team have built using logs coming from psyllium and Hubble and tetragon and Prometheus metrics coming from Hubble and psyllium.

A

Is that you can use to solve other kinds of security problems with Cloud native tooling?

A

Here are a few other examples that I've found so here's a visualization of detecting SQL injections using pixie, it's another ebpf based project in the cncf.

A

You just heard about Falco here's a pie chart showing Falco rule violations.

A

Here's a more complex dashboard, that's been built on tetragon, showing information about access to sensitive files, and all of these tools are able to pull this really detailed information and tie it back to kubernetes identities. Thanks to the power of ebpf and the ability to write tooling, that's both very finely grained and highly performant.

A

So ebpf tools can generate these streams of really rich contextualized events in the form of logs and metrics and traces and as humans we can make much more sense of that data. If we can visualize it and then we can use it to debug Network policy. Like you store our Stormtrooper security team, doing we can use it to spot unexpected access to particularly sensitive files. We can use it to spot data being exfiltrated. We can use it to see unexpected binaries being run anywhere in our cluster, can even identify privilege escalations using ebpf.

A

All of these things can help us solve real security problems.

A

And Darth Vader is so impressed he's actually thinking of having the Empire join the cncf as an end user organization.

A

Thank you very much.