Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Beyond printf & tcpdump: Debugging Kubernetes Networking with eBPF - Martynas Pumputis & Aditi Ghag

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Beyond printf and tcpdump: Debugging Kubernetes Networking with eBPF - Martynas Pumputis & Aditi Ghag, Isovalent

Debugging Kubernetes networking issues can easily turn into a nightmare. Packets from one pod to another can traverse a couple of dozen Linux kernel functions. Each function execution outcome might depend on an OS state which is not exposed to a user and is not visible with traditional Linux networking tooling. And those tools can’t tell you how network packets relate to K8s pods and services. Luckily, with the help of eBPF we no longer need to treat Linux kernel networking in the context of K8s as a big blackbox. In this talk DevOps who operate K8s clusters will learn: - Packet inspection across layer 2, layer 3 and policy routing, socket, and so on, regardless of the CNI. - How eBPF can be used to efficiently troubleshoot K8s networking issues. - Real-life examples of K8s networking problems and how they were debugged with eBPF. No previous knowledge of kernel internals or deep understanding of networking is required to attend the talk.

A

Well, hi. Everyone uh welcome to my talk. uh I'm aditi, I'm a software engineer working at isovelyn and this talk features a joint work with martinez a bit of context into why debugging kubernetes networking is relevant for us martinez and I work on psyllium, which is an open source cni uh powered by ebpf, and we work with one of the most sophisticated set of users uh and to enable them to network and secure their communities clusters.

A

uh So yeah, let's get started uh in this talk, I'm going to share our debugging experiences and present a new tool that we have developed using ebpf.

A

So let's take a common kubernetes cluster, where our request is coming onto kubernetes node and it's being delivered to a service part. So the request traffic will first get processed at neck. It'll then get routed to linux, kernel, networking stack and then uh it'll traverse from host network namespace to report network namespace via a couple of uh v devices pair.

A

So whenever there are network connect connectivity issues, we start with control plane. Checking all the configurations are correct. Our services, our services are deployed. The service part is up and running and we always make sure that it's not the dns, but today I want to draw attention to a key component that often are treated as a black box and that's the linux kernel network debugging linux. Current networking is hard and let's zoom in to see why that is so.

A

This is a detailed network flow diagram of linux, kernel, network internals and, as you can see, there are overwhelming number of uh packet processing functions. So when you send a single packet, it requires many many hops until it reaches its destination, uh so sure there are like packet, counters and uh stats that you can observe, uh but these are not enough uh to get to the root cause because kernel state can't be observed in real time uh in an easy manner.

A

So we have this internal uh joke uh that whenever we're trying to track a packet in linux kernel, uh it's like finding waldo one can easily feel lost when they uh fall down the uh rabbit hole of linux, kernel networking.

A

So next we are gonna. Look at some of our uh uh most godo tools, um so one of our most favorite tools is tcp dump and you always reach. First, whenever there are like network connectivity issues, and I agree that uh it's a good tool to start your debug routine.

A

But the problem with tcp down is that it only gives you high level information, because uh the tcp tap tcp dump tap points are at the periphery of linux. Kernel networking stack, but in real life situation is much more complex and either the packets are dropped in the neck or in the stack or in the pod network, name, space and so and so forth.

A

So next up moving on to like traditional logging based methods, uh whenever developers don't have access to good uh debugger, uh they always rely on logging based uh debugging. So kernel exposes this print k function, which you can use to add debugging statements in the kernel code, but it requires recompiling the kernel and in many cases reboot. So it's not a viable option in production environments.

A

If you're not careful with your debug statements, it can cause kernel panics.

A

In short, it's it's very slow. uh It has very slow debugging cycles because it requires many, many iterations. uh Let's say you ruled out that dns is not the issue here and you wanna now trace packets going to the cube api server. You can't easily do that so moving on to some some of the generic tracing tools, so a couple of examples are perf or vpf trace.

A

Superf is a performance measuring tool that also allows you to trace calls to some of the network. uh Linux kernel network uh functions and, if you have used for before, then you can identify uh the snippet that I've added uh on the slide.

A

So in this case, I'm using perf record to record all the calls that are going to uh skb uh free uh skb free function that which is which is invoked whenever a packet is dropped in the kernel and first script is then used to dump traces uh that are collected from perf record and, as you can see, uh the output is very limited.

A

uh It doesn't have uh limit. uh It has very limited filtering capabilities. So if you're trying to debug a dns issue, you can't specify hey. I want to trace all the packets that are going to port 53..

A

There is no information about, let's say, for example, source or destination, ip addresses ports, uh and, as I mentioned before, whenever you use sport record, you have to specify what function you're trying to trace calls to, and in many cases uh you don't have that information when you're starting out with debugging uh an issue.

A

So we we approach this debugging uh problem in a very different manner.

A

So not all network connectivity issues involve packet drops and uh if I want to interest introspect kernel uh network state in a uh in a fine-grained manner, then is there any uh way to get a list of all packet processing functions?

A

Next up, uh we want to get callbacks to uh whenever these functions are executed, so that we can analyze kernel state uh uh in a more detailed uh fashion, but, more importantly, uh we're gonna filter this callbacks only for traffic, that's relevant for us.

A

So this is the wish list that martinez and I came up with- and uh this brings to brings me to the second part of my presentation where, uh where we are gonna, find answers to these questions uh using ebpf, so ebpf has captured the imagination of many in recent years, and companies like facebook, netflix isovalene, uh are using ebpf to solve a spectrum of use cases uh to so so to set the context for our new tool.

A

uh Let's look at what ebpf is so ebpf is an internal virtual machine uh that safely executes native code on certain events, it's highly programmable and it's performant, because the code is executed directly in the kernel and because of this reasons it's touted as the javascript for the kernel and let's unpack this information uh with an end to end workflow uh next. So here I have uh a simple uh bpf program where I've added a cr k probe to one of the uh network functions in linux. Kernel and k.

A

Probes is a debug mechanism uh in linux, kernel where uh it allows you to execute bpf programs when, whenever any kernel instruction is executed. So when you attach a k probe uh in your bpf program to a linux runner function, uh your bpf program will be executed whenever uh that function is called. So in this case, as you can see, uh I'm getting call back, uh or rather I hope that I'll get call back whenever ip local deliver function is executed and skb is the linux uh representation of packet.

A

So, in this program, uh I'm going to parse that skb uh dump some fields uh and let's look at what, how we can compile it next, so this program is compiled using clang, which outputs an elf binary that contains bpf byte code.

A

It's it next fed to an ebpf loader, so the loader is going to parse the uh elf file and uh the binary and uh is gonna uh set the context for this uh uh program, which includes like setting the type. So in this case the program type is bpf, prog type, k probe and this type is gonna decide what kernel state this my program is going to have access to next, the loader triggers loading and verification of this program. So in the previous slide I mentioned that bpf.

A

The kernel ensures that bpf programs are run safely. So ebpm verifier is tasked with ensuring that uh the verifier will ensure that your program doesn't have a null pointer, dereferences or it's not trying to access any out of bound memory uh and whether your program terminates or not, and if you have written fair amount of vpf, if you're done a fair amount of bpf programming, then I'm sure you have interesting stories to tell about uh how you fought with the verifier.

A

So once the verifier approves. Your program is just compiled to native code, and I mentioned earlier that ebpf follows an event-driven model. So what does it mean in this case, uh in my program, I've attached a k-probe to iplocaldeliver function, so this function uh delivers packet to a local destination. uh So let's say a packet is delivered on the eth0 interface on your kubernetes node and uh it's supposed to be uh delivered to a local leader in part.

A

Then uh the packet will be uh executed by ip local deliver function where it reassembles ip fragments and uh hands it off to the uh next level or layer. Excuse me, which is the transport layer. So, uh in this case, uh it's gonna generate an event uh and my bpf program is going to be executed and as the bpf program, if you, uh if you can see uh just before it returns, uh it writes this notification to bpf map.

A

So bpf map is a shared data structure between kernel and user space and that's how kernel relays this information into the user space so great yeah, just as we uh traced uh callbacks to the ip local deliver function, can we just get a list of all the uh net kernel uh network packet processing functions and just keep like a hard-coded list, not exactly because kernel function, signature can change across kernel versions, so k probe api is, is considered unstable.

A

So how do we reliably get debug information about different kernel function signatures, so that's where btf comes into play. uh Btf is short for bpf type format and it's a debug format that stores information like function, signatures, uh different data structures that are defined in kernel and so and so forth. But the cool thing about btf is that it's super compact, so in recent kernels it's packaged in kernels by default, so kernel exposes it exposes its own debug information uh like function, signature and different data structure, types via asus, fs interface uh and yeah.

A

Okay, so uh just to get a sense of what btf information looks like uh I've dumped a uh uh btf information for a simple uh function that accesses the skb uh structure and on the right. You can see that it gives you information about various types. So if you start at the top, uh it tells you that there is a pointer to the structure, sk buff and what different members does the escape of structure has and what are their respective offsets.

A

But one important type then I want to highlight here: is the func proto uh func proto defines a function signature. So what kind of arguments uh does a function accept? What what's the written type is so with this? uh Let's revisit our wish list. So we mentioned that hey. We wanted a reliable way to get the list of all the network.

A

uh Processing uh network packet processing functions in the kernel and we can easily get that information from the btf file. Next, we can attach k probes to all of them, so just as k, probes allow you to execute bpa function when a function is executed, k, red probes uh uh are allow. You allow your bpf programs to be executed uh when a function that it's, uh where the function where it the k rate probe is attached, is uh return and, uh more importantly, we can easily uh filter callbacks using with the help of bps maps.

A

So with this background, I'm really excited to present peru. Peru is an ebpf based linux, kernel, debugger and peru is short for packet. Where are you conceptualization of this tool? uh The credit for conceptualization of this tool goes to uh martinez, and next we are going to uh take a walk through of the uh peru internals.

A

So peru consists of a user space agent uh written in go and it interfaces with user to collect, uh filter information. So uh let's say I'm trying to debug traffic. That's going to the cube api server part, so I run peru with this filter parameters where I set the protocol to tcp destination ipsec to the uh cube api uh pod, ip and the destination port is set to 443.

A

Next, peru goes ahead and programs. This information in this filter map, which is a bpf map. It then uses the celium. Excuse me, sorry. It then uses the psyllium evpf loader to collect uh btf information for the underlying kernel. So in this case it's gonna do a call to this sysfs interface. uh It's gonna search, all the kernel uh functions and then it's just gonna get iterate over that list and uh filter them to collect pack uh collect functions that accept skb as one of the parameters and just a reminder.

A

Skb is the packet representation in the linux kernel. It then goes ahead and loads, our ebpf programs to all these functions. uh So, in this particular case, uh I'm interested in getting uh callbacks from the linux kernel whenever my packet goes through, all the uh network functions in the linux kernel, so, uh for example, uh in the previous example, we saw how my bpf program was executed or whenever I p local deliver function was called uh so.

A

The e0, for example, is, is the first event that gets uh called whenever my packet is uh executed by ip local deliver ip local deliver is then gonna hand off that packet to uh parse t uh layer, four, which is like tcp in this case, so that's going to be e1. uh The one thing that I want to highlight here is that uh my bpf program is going to filter all these callbacks based on the filter information that it receives from the bpf map all right.

A

So uh in the beginning of the presentation I mentioned that linux kernel networking is treated as a black box and because of the uh complexity, that's involved, so uh with the help of peru. I hope I can convince you to give it a try and see how easy it is to debug linux kernel, networking issues so in the lower half of uh of the clip, I'm running peru with a set of filters. uh So, for example, I'm filtering traffic destined to 1.1.1.1 our destination port, 80 and protocol is set to tcp.

A

So I have added an iptable rule here, uh which is going to drop packets to this destination and port uh tapu, and whenever I run a call request, it's going to generate output trees and I'm going to know that okay yeah. This is where my packet is getting dropped.

A

So in the next uh few slides we are going to look at real world examples and how peru helped us to debug those scenarios. So the first example I have a kubernetes cluster where I'm running a multi-homing setup. What does it mean? So in this setup a pod is assigned ip addresses from uh two different ip subnets.

A

So let's uh do a request. Walkthrough from right of the screen to the left, where a pod is trying to reach the queue api server part in my cluster, so the port does a curve request where the source ip address is set to the power ip address and the destination is the cube service cluster ip?

A

This request is going to hit the iptable rule, that's installed by q, proxy and the rule says states that hey all the traffic that's destined to uh cube service cluster ip. uh It needs to be denatured to cube api server pod ip address. So after the rule, we can see that the destination ip address has been translated from the cluster ip to the ip address of the cube api server part.

A

This request reaches the destination communities node and it's routed through the eth1 interface, and then it's delivered to the cube api server part.

A

Now the cube api server pod is going to send a reply back the source ip address is uh correctly set to the ip address of the cube api server and the destination ip address is the ip address that was that was received in the request packet and this packet is being routed through e0, not eth1, and there is no surprise that this reply never reaches the source. So, let's see how peru helps and helps us in debugging this uh issue.

A

So here uh I'm right. I'm running peru with a bunch of filters. I'm filtering traffic going to uh destination port 443, which is the uh which is where cube api server. Pod is uh listening on and I've set the protocol to tcp, and I invite you to look at the output of peru here.

A

The nf, so this is the trace of all the kernel, functions that my traffic of interest is uh going through so nf hook. Slow uh is the kernel function, that's executed to implement uh iptable rules and uh we can see that the source ip address in this case is still set to 190 to 168 and not the ip address from uh the the subnet, where that's used to connect to the cube api server.

A

So that's why this packet will uh will be dropped when, when it's sent out of the eth1 interface next up, we frequently have mtu mis configuration issues in our clusters. So I have a simple setup here where uh I have a single node kubernetes cluster. uh My udp pod is getting traffic from an external uh entity and that extra and the size of the udp packet is uh pretty large and uh the sender has said, uh don't fragment bit uh on this request.

A

So this request is gonna traverse through uh the ethernet interface on the kubernetes node, and then uh it's gonna uh traverse through the uh v uh devices pair and then uh it's gonna be delivered to the udp pod and uh based on the mtu information, we can see that there is a mismatch. So, let's see what happens to the traffic so again, I'm running peru, but this time I have changed.

A

I've set the protocol to udp the destination port is 443 and I have I have a new filter here which says: output mirror, which is going to print a bunch of skb members here. So, as you can see, uh the there is a mismatch between packet length and uh mtu. That's configured on the. If index and the if index uh 18 refers to the eth one and seven refers to the v0 and as a result, this packet will be dropped.

A

So, lastly, there is a security mechanism in linux kernel, it's called reverse path. Filter. It essentially ensures that the source address set on packet, a received packet, is routable via the same interface that it came in, and this is to ensure that there are no spoofed uh packets where ip address the source ip address is spooked.

A

So in this case uh I have a. I have an incoming request. uh Coming to my pod. uh Let's say this is my service pod and the request is uh translated uh deniated to my port, so the destination is set to the pod ip address. So when this request is received on the ethernet interface on my kubernetes node, uh there is an rp filter configured on this interface by default and it's set in the strict mode.

A

So this request will be dropped because if you look at the uh ip route table here, uh it says that all the traffic that's uh deleted, destined to 10.0.0 24 subnet. uh It needs to be received on the ve30 interface, but in this case we are receiving it.eth1.

A

So, uh let's see what information peru gives us here so this time I have set the destination ip filter, which is set to my pod ips, and we can see that the packet is being dropped.

A

And, let's see why that is so just before the packet is dropped, we can see that uh there is a callback from fib validate source uh function, so this kernel function essentially executes the rpe reverse path, filter logic uh where it it it will uh check if what the source ip address on this packet is and if it's received on the interface uh that it's expected to so here are the peru highlights.

A

uh Peru is an ebps based open source tool to debug kernel networking, and it does so by abstracting kernel networking details but, more importantly, peru exposes advanced filtering uh capabilities uh and it prints packet level metadata so that you can introspect a kernel in a fine grained manner and it's portable across kernel versions.

A

So some acknowledgements super uses the ebpf kernel functionality uh and based on k-probe. uh So, uh thanks to vpf kernel developers, steven rostedt for k, probes, uh we're using the bpf sn printf btf function, that's exposed in the kernel so that we can print skb uh members in a safe manner and pero uses a psyllium evpf uh go library as an ebpf loader to load our uh to load, ebpf uh functions.

A

And lastly, uh we recently discovered that I there is this uh new function, uh new tool. It's called ipv ipf, ipf trace2, and it has some similar uh functionalities as peru and it was released before peru. So we are acknowledging it here. So yeah thanks for your time and I open it for questions.