Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Native Superpowers with eBPF - Liz Rice, Isovalent

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Cloud Native Superpowers with eBPF - Liz Rice, Isovalent

Netflix’s Brendan Gregg coined the term “Superpowers for Linux” to describe eBPF back in 2016. Today there are a whole crop of eBPF-based tools for cloud native observability, networking and security, such as Falco, Pixie and Cilium. What makes eBPF such a powerful platform for building this new generation of tools? This talk uses live-coding & demos from across the cloud native ecosystem to illustrate how eBPF programs can - Collect performance & behavioral insights from across an entire system - Relate observed data to Kubernetes objects - Reliably instrument workloads without making any changes to apps or configurations But eBPF isn't a magic wand. We’ll look at its limitations and potential misconceptions that are emerging from the use of this powerful new technology. Even if you’re not a Linux kernel aficionado you’ll leave this talk with an understanding of how eBPF enables high-performance tools that help you manage, debug and secure applications in the cloud.

A

Hi I'm liz rice. I am the chief open source officer at isovalent, we're the company behind the psyllium networking project and I'm also the chair of the technical oversight committee at the cncf, and today I want to talk to you about a technology, I'm very excited about called ebpf and how this is bringing superpowers, particularly for networking and security and observability to the world of cloud native.

A

So, let's start by saying what ebpf is. It stands for extended berkeley packet filter, but I'm not sure that's terribly helpful.

A

What ebpf allows us to do is to run custom programs within the kernel, so you can write custom programs, you don't need to reboot your machine, you can add and remove these programs dynamically and it's incredibly powerful.

A

Over the last few years, there's been a lot of innovation and development within the kernel itself, and also with surrounding tool chains and verifiers and other components that have made this ebpf technology possible, and nowadays, most of the linux kernels that people are using in production, supports sufficient ebpf functionality that we can really start making use of it in the world of cloud native. Today, you may have heard there's been an ebpf foundation created recently, and this is really concentrating on that underlying ebpf technology.

A

The changes to the kernel and the tool chains that enable projects to build on top of ebpf the ebpf foundation is like the cncf, a part of the linux foundation, and so where the ebpf foundation concentrates on that underlying technology.

A

There are several projects within the cncf that make use of that technology and we're going to look at some examples of using these projects to explore some of the power that ebpf gives us, but before we do that, let's just understand what we mean by ebpf programs, so our user space applications can load ebpf programs into the kernel.

A

This can happen dynamically as and when required, and we also have to attach those ebpf programs to an event when that event happens, it triggers the associated ebpf program.

A

What kind of events it could be the arrival of a network packet, it could be a user space application, making a system call. It could be hitting a particular trace point or some dedicated hooks within the kernel. There are a lot of different places we can attach these ebpf programs.

A

So let's make this concrete by looking at a very, very basic example of an ebpf program. This is going to be like a hello world.

A

So this is my ebpf code that I'm going to run within the kernel. It's very straightforward. It's called hello and I'm just going to make it trace out some tracing. Let's say hello cubecon as the message and I'm going to attach this to the event of entering uh the exec ve system call exec ve is used every time we execute a new program so whenever any new program is executed on my virtual machine, it's going to trigger this ebpf program and I've also got some user space code.

A

That's going to read the compiled object created from that c code and it's going to load it into the kernel and attach hello to that. Exec ve system call.

A

When I make this, it's got two steps: it's doing a go, build to build the user space part of the code, and it's also going to use clang to compile the c code into the ebpf code that will run in the kernel.

A

I need to run this as root and that's attached. My tracing function there to exactly system calls, and we can see there's lots going on on this particular virtual machine.

A

You can also see that the tracing function that I've used automatically gives us some additional contextual information about what was happening when this trace was generated. We can see the name of the executable we can see. This is actually the process id there's timestamp information, so ebpf programs, when they run have this additional context.

A

That can be very powerful.

A

So that's my ebpf hello world example, and I hope it illustrates that it was triggered by processes running across the virtual machine. So ebpf programs are aware of everything. That's happening across the entire kernel, we're seeing executables being run in lots of different processes. Here.

A

When we run an ebpf program, it has to not crash the kernel. It has to not leave the kernel in an infinite loop.

A

Those things would be bad and in order to ensure that an ebpf program is safe to run, there's a component called the verifier, and the verifier will check, for example, that every time you have a pointer that you want to dereference, you have to, in your ebpf program, check that that pointer is not nil before you dereference it that ensures that you're not going to be dereferencing a nil, pointer, ebpf programs are limited in what memory they can access and they are.

A

The verifier will check what memory the program is trying to access, uh and it will also check for things like not falling into an infinite loop and because the verifier is ensuring that the ebpf program complies with these rules.

A

Ebpf is sometimes described as a sandbox and that's absolutely true, but for those of us used to the idea of containers, it can be a little bit uh well kind of confusing, because people also describe containers as sandboxes, and these are very different types of sandbox.

A

One misconception that I've heard is people thinking that maybe ebpf is a replacement for containerization, and that is not the case. Both are kind of sandbox but they're for very different purposes, but let's take a look at what it means to be able to run an ebpf program in a kubernetes environment.

A

So in kubernetes all our application code is running in containers that are grouped together into pods, but, however many containers we have. However many pods there are, there is one kernel per node whether that node is running on a bare metal machine or a virtual machine. There is one kernel for that machine and every time our applications running in pods want to do anything interesting.

A

They have to get help from the kernel to do that. So, for example, if your code wants to read or write from a file or send or receive a message, even if even if kubernetes is creating new containers for all of these activities, the kernel has to be involved.

A

So that means the kernel is aware of everything. That's happening in all the pods and uncontainerized processes on that host. The kernel is aware of everything on the host and we can attach ebpf programs to that kernel.

A

Our ebpf programs can be aware of everything that's happening on the host, so we can, for example, redirect or drop network packets from inside ebpf and that's the basis of ebpf networking as used by psyllium.

A

We can bring together the information obtained by ebpf programs. We can share it with user space and pull it together into really powerful observability applications, and we can use it to detect potentially malicious activity for security purposes.

A

One thing I want to be clear about is that I doubt very many of you will need to write ebpf programs yourself. I showed you that hello world to make it concrete, but for most of us you can use the ebpf applications that have been written by projects that make your life easier.

A

Ebpf programming can get really complex. So unless you have a good reason to do it yourself, you will probably be better off using the tools created by different projects.

A

So, let's take a look at some examples of using cncf ebpf projects to illustrate some of the powerful things that we can do with this technology.

A

So I have a kubernetes cluster running a few different sample applications. This has falco and psyllium and pixie all installed, and let's start by taking a look at some network flows.

A

This is using psyllium's hubble ui and we can see a service map here showing us how network traffic is flowing between different services in this particular namespace tenant jobs and because psyllium as a cni is aware of the kubernetes entities making and receiving these network requests.

A

It can map flow information network flow details to the pods and the services that made them receive those requests, so we can build. You know nice ui like this to see, for example, crawler service sent a message to elasticsearch service and we can dig into the details of of that http request yeah. So that's an example of using psyllium for kubernetes, aware network information as another example. Let's take a look at falco, detecting one of its rules being violated so falco.

A

Much like my hello world example attaches to system call events, and this particular rule is like my hello world. Looking for processes being executed using the exec ve system call and this rule is looking for processes called nc netcat, which is often used for malicious activities.

A

So let's try and trigger this rule.

A

So, first of all, I'm going to tail the logs from falco and there are a few things that have been detected in the past already now, I'm going to exec into a pod in this cluster and I'm going to run a shell and we should straight away see another rule being detected there. It is a shell was spawned in a container with an attached terminal and if I run, netcat won't actually work. But the point is that the system call was detected.

A

It triggered the assessment of that rule and falco admitted the fact that what omitted a warning about netcat running inside this container now, let's turn to observability. We already saw in the psyllium hubble ui the ability to observe network traffic pixi allows us to run scripts to generate lots of other kinds of observability data. I quite like this example: let's run this again, just to prove that it's live where it's showing a flame graph of how resources are being used.

A

So we could dig in to see which of the applications are using the most resources and we can see the whole stack trace to understand where our cpu is being used across the cluster.

A

We can also get some interesting data about node resource usage using this pixie ui.

A

So we've seen that ebpf applications can offer very powerful observability because they can access everything that's happening across an entire node.

A

They can also give us some real performance improvements, and I want to touch on how ebpf improves network efficiency.

A

We run applications in a pod. The pod has its own networking stack and that's connected to the networking stack on the host through a virtual ethernet connection.

A

So imagine a pod arrives on the physical interface to this host and it's destined for the pod or a pod on this host. That packet has to traverse the whole networking stack on the host to get sent on through that virtual ethernet connection to the pod.

A

As an ebpf application, psyllium can hook into an event called xdp where the moment the packet is received onto the host before it's hit, the networking stack at all and because cilium was responsible for setting up the networking inside the pod, it can detect that the packet is destined for that pod and it can route it directly to the pod, avoiding having to go through the whole ip stack on the host, and this leads to some genuine performance improvements. We can see this in another flame graph. Here.

A

We can see the what's happening when a packet is received, it spends some time being received in ebpf on the host and then it's sent into the tcp stack within a pod. I think that probably illustrates how that is more efficient than it would be if we had to also run through the tcp stack on the host.

A

But, more importantly, we measured it and there's lots of details in this cni benchmarking blog post on the psyllium site.

A

The blue line on the left here shows how fast requests and responses can happen, with no containerization at all, so sending and receiving between two nodes.

A

The red and the orange bars show that we can achieve very close to the same network performance using ebpf, the the psyllium and calico in ebpf modes in those red and orange bars, and you can see that's significantly better than the yellow and green bars in the middle, where that's psyllium and calico in non-ebpf mode. So the takeaway here is ebpf. Networking can enable faster networking and there's a lot more detail in that benchmark. Blog post, so ebpf allows us to build this huge range of different networking and observability and security tools.

A

But the characteristic of being able to see everything that's happening across an entire node gives us another really significant advantage.

A

That is, we don't need to change the application or the way an app is configured in order to instrument it with ebpf contrast that to the sidecar model that we typically use for a lot of instrumentation in the kubernetes world nathan leclaire. Did this really great cartoon illustrating how ebpf is a really great uh new model for instrumenting applications?

A

Let's explain why that is in a sidecar model. We have a container for the instrumentation that's injected into the application pod.

A

In order to get that sidecar into the pod, there has to be yaml definition.

A

Now you probably don't have to manually configure that yaml, it's probably done as part of an automated process, perhaps in ci cd, perhaps through an admission control web hook, something adds the yaml definition for the sidecar container into the pod specification.

A

But what if that goes wrong or what, if somehow an application, doesn't go through that automated process? If you don't have the sidecar in the pod, it's not instrumented. You have to get that sidecar successfully configured into every application pod that you want to instrument.

A

That isn't the case if we're using evpf, because the ebpf is hooked into the kernel, the ebpf program is hooked into the kernel, so it doesn't matter the the pod yeah it can see what's happening in any pod without any need to change or reconfigure that pod.

A

That makes it much more secure in terms of being able to see every application, that's running whether it got there by fair means or foul. If somebody manages to maliciously run some code on your cluster, they probably won't go through the trouble of adding a sidecar pod into it with ebpf.

A

It will be instrumented because it's running on that shared kernel.

A

If we use ebpf tooling to instrument our kubernetes applications, you've seen how we're able to collect a really wide range of different information, different characteristics about our applications, and we can use that for security forensics.

A

Here's an example of something that we've been experimenting with in psyllium, you've already seen how we can map network flows to kubernetes identities. We know exactly which pods and services are involved in any given networking flow.

A

You've also seen even in the hello world example how we can see: contextual information about the process that was involved when a particular ebpf program was triggered.

A

If we combine those two things, we can get some really deep insight into what happened, perhaps for malicious activity.

A

In this example, we can see a pod, that's got network connections to two completely legitimate uh destinations, twitter, api and an elastic search service.

A

We can also see exactly which pod has those connections, but within that which executable in which process id in which container generated those network connections.

A

We can also see when this happened so imagine that one of those destinations were was a commander control center for malicious activity, or perhaps it's a cryptocurrency mining pool we'd be able to see exactly what process caused that network connection to that dubious endpoint we'd be able to see when it happened.

A

We'd be able to see what pod was presumably compromised, and with this information we would find it much easier to track back whether the pod was compromised at runtime or whether it was already compromised before it was even deployed, perhaps at some point during the the supply chain.

A

This kind of deep forensic insight that we're able to collect using ebpf is hugely powerful, so I hope this has given you some insight into why I believe evpf and the ability to run custom code in the kernel is revolutionizing the kind of tools we can build for networking and security and observability it's creating a whole new generation of amazingly powerful tools to help you operate. Your clusters safely.

A

Thank you so much for your time. I do hope you have questions so come and ask me your questions and if you want to find out more about ebpf, the foundation site is at ebpf.io and of course you can find out more about psyllium and about isovalent on our websites. Thank you.