Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Understanding and Troubleshooting the eBPF Datapath in Cilium - Nathan Sweet, DigitalOcean

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Understanding and Troubleshooting the eBPF Datapath in Cilium - Nathan Sweet, DigitalOcean

The advent of eBPF (extended Berkeley Packet Filters) has contributed significantly to container networking progress. However, the tooling for diagnosing and troubleshooting eBPF issues is nascent, and most members of the K8s and Linux communities are unfamiliar with it. This talk will help demystify eBPF and cover its history. We'll present the default network datapath of the Linux kernel and contrast it in depth with how various eBPF program types diverge from this datapath. In addition, we'll match up the ways in which cilium implements various CNI and K8s constructs/objects with their eBPF program type, so that you'll be able to identify the right troubleshooting methods easily. Finally, we'll match appropriate methods and tools to the various eBPF program types.

https://sched.co/Uae7

A

Thank you yeah, so as eat I said, I'm need sweet and I'm. A software engineer on the managed kubernetes products for digitalocean and I am also the creator and Co contributor, along with Lorenz Bower, a cloud cloud flare of the new tools, eb PF library, which is now in the process of being slowly ported over to the cilium organization under the same moniker.

A

It's a go only library with no C bindings and it comes with an assembler and debugger CloudFlare has been using it in production for the past year and a half or so and yeah. If you are embarking on an e BPF journey yourself, I would Lorenz and I and Joe stringer, who is also here at the conference for psyllium, would be gratified for any pull requests you might want to give us and yeah, so he and Lorenz actually deserve most of the credit for that library.

A

Being great, so I wanted to plug that just because yeah we're trying to get some some hits on that. So all right, so understanding in troubleshooting the eb PF data path with psyllium. Why should we care the network? Is the bottleneck and I'm assuming most people in here and know that the network is the most time I Oh bound thing you can do, but I think something. That's new that people haven't talked about. So much is that with the advent of the public clouds, writing main writing.

A

Distributed systems has become incredibly mainstream and we are largely running those distributed systems on software that was designed in the 1970s and so getting insights determinism into your network. It's really hard. The the tools that we have now are kind of unwieldy, so I think that EBP f is an interesting step forward. I, don't think it's necessarily the the final solution, but it's I think it's a powerful tool and it's uh I think as we'll see it, it offers quite a bit in terms of flexibility. The other reason this is worth learning about is EB.

A

Pf is kind of spreading everywhere the Linux net devs seem to enjoy putting it everywhere. They can so, whether or not we want to learn eb PF. I think we're gonna see linux kernel's that get built in the coming years, that don't even have netfilter in it anymore, and so Li really rely on ebps I know that that's something psyllium wants to do is offer zero IP tables CNI, which is an interesting proposition, and the third reason is that metrics are money.

A

There was a great talk at the Paris kernel recipes summit this year by a guy named Aurelien Rougerie and from synesthesia, and he had some great debugging more stories, I linked to the to the talk in the slides if you're, if you want to look at it, but he makes this really good point that, like people, don't really want to worry about this stuff right like even if you are interested in systems programming, you want to solve interesting problems. You don't want to understand why the UDP socket buffer maximum got hit in your kernel.

A

Distribution I mean. Maybe you do, but if you do you're kind of weird and so getting this stuff done quickly and efficiently means more money, for you means more money for your organization's makes your products better allows you to focus on problems that actually matter rather than tuning the Linux kernel. So, okay, what is EBP F I'm, hoping that there's some familiarity with it, but I'll do just kind of a quick primer in case you're. Not EVP. F stands for extended berkeley packet filter, and this is an example program of such here.

A

This is an x DP program which stands for expressed data path and express data path. Programs can be attached directly to network devices and specifically XDP programs can do things like drop, pass and redirect packets, and it can do these things very efficiently because it's run almost immediately inside the kernel data path. So in this example that this program example is taking any packet, that's an art packet and dropping it if the source IP of the art packet is contained inside this black list. So we get this data structure.

A

We cast the void pointer to an Ethernet packet. We check the Ethernet protocol if it's ARP or if it's not our, we pass it. So that means continue along the data path. Otherwise, let's cast the offset of the Ethernet packet into an art packet. We look at the source IP of the art packet. If it's in our block blacklist, we drop it.

A

If it's not, we continue on down the data path, so I hope this is kind of a helpful illustration of what EBP F is it's it's programs and maps, and this this visualization hopefully helps a little bit as well. We have a program running inside the kernel, it's doing something and it shares a map structure with user space that map structure contains. The the maps that you BPF are offer are quite extensive.

A

You can have cpu-bound maps, so it's a data structure per CPU, so if you're doing receive channels on multiple CPUs you've, the CPU data structure live or the data structure lives on each CPU, which obviously translates into quite a bit of performance, and then it can also just have configuration data right like I, don't know what a good example would be, but these protocols get dropped or something like that, so ye be PF e b. Pf is fast.

A

It's a very compact amount of logic in a tight space, and that lends itself to being quite quite fast. It's also just in time compiled so I, don't know if how many people know this, but there is a compiler in your Linux kernel that compiles bytecode into raw arm, there's, actually a risk v compiler now for EBP F I, who ever wrote that a brave person and eb PF is flexible. And what do I mean by this eb p? F is just generalized code right, it's.

A

It can be c code, there's a rust compiler, so you can write and rust if you want it wouldn't surprise me if someone writes a web assembly version of it, because web assembly is everywhere now so so yeah. The the flexibility that allow it allows is, is quite enormous and, for example, the XDP program. The linux kernel only cares about the return of past drop and redirects, but within your code you can kind of do whatever you want. You it's a it's.

A

Basically a micro service within the kernel and finally, the eb BPF separates data from functionality. I think that's a really big argument in its favor. I think I hope everyone here appreciates why separating functionality and data is a very powerful constructs.

A

Certainly, we can achieve some of the things that eb PF can achieve with things like net filter and IP tables, and you can even track some amount of state with those things, but I think that it's a lot easier to to create data-driven stateful applications with DB PF rather than a ginormous iptables chain, and I'm just going to touch a brief history of eb PF, because I said I would do that in the abstract.

A

So the the paper that kind of kicked this off was by stephen mccain in 1993. He wrote this paper called the bsd packet filter. It's actually a really good read it's very.

A

What's the word I'm looking for it's a it's an accessible read, so you don't have to be a Linux kernel to have to understand what he's trying to to drive at it in that paper, and he based this idea on the idea of packet filtration, which had been around since the 70s, but he was like well what, if we, what if we put compilers and networks together and that ended up being a pretty bright idea previously packet filtration was more of an abstract concept.

A

You operated on known packet types in their fields rather than having access to the like the robbed of the the robbed bytes. Just as an aside, I was kind of curious people.

A

Talk a lot about the history of this idea. Where did packet filtration come from and it seems like everyone was citing this paper that I just threw up by Geoffrey mogul in 1987, where in the Xerox Alto is, is labeled as being the first packet filtration computer and I was really curious.

A

Oh is that true, and indeed he does cite that the Xerox Alto in 76 was the first computer to have this and I was curious since everyone's citing him what's his source- and you know there- that's kind of an academic faux pas, so I dick I dug a little further, and this turns out to be right because David, Boggs and bends at Ethernet, so indeed the Xerox Alto was the first computer to have packet filtration okay. So what is psyllium?

A

And why should we care EB PF is easy to look at and understand because it's flow control and see or rust or whatever, you're writing. But it can be very difficult to configure, maintain, operate with and so having a hardened wrapper that has all of this stuff built into. It is I, think quite useful and the psyllium is a good general purpose wrapper that gets you motive. What you need when you're dealing with distributed systems and if you're familiar with kubernetes at all it the the object.

A

Names are very familiar and in fact they're pretty much identical and then, as of one sex psyllium, is both a cni and a drop-in replacement for coop proxy. So you can kick coop roxy to the side. If you're running psyllium, one six or greater, which is nice because coop rocks you can ping the API server quite heavily iptables rules, even if they're not running, actually can slow down your data path, and so it it can be very nice to get rid of all that stuff.

A

Alright, so the next six slides I'm going to walk through the kernel data path and we're gonna start at the hardware.

B

A

I'll try to highlight where cilium enters the fray. Hopefully it kind of makes sense as we go along. I'm not gonna, focus too much on the hardware, because we don't really need to know too much about it. Obviously it isn't useless to know this stuff either. So I'm gonna skip some of the steps here and just kind of focus on what happens when the driver is initialized and packets are flowing, so the network card gets assigned direct memory access region in RAM.

A

The Linux kernel gives it that address space and then, when the network card receives the packet, it sends it up to the memory through the PCI bridge into something that effectively is like a ring buffer and then this generalized pull mechanism in the linux kernel called the napi pull mechanism which is used by e-poll and the drivers and lots of basically anything that's doing asynchronous work can use this generalized pull mechanism in the Linux kernel.

A

It's it's listening for these packets and then it's processing them as they come in, and so then we hit the CPU alright. So then we hit layer. Two most drivers are all dry. All network card drivers implement a polemic this this interface into this pull mechanism, and they do lots of different things, but sort of the biggest thing they they do for our purposes is they can call in to our first BPF program, which is xdp. This is where C liam's network policies lived.

A

So this is where we can like drop packets, and this is where we can cut off a lot of work at the pass. If the xdp program passes and says this, this packet is good to go. We move on to the next function that the driver pull function will call, and that is clean Rx, and this is where the Linux kernel allocates a socket buffer for the packet socket buffer, isn't maybe the best name for what it is that makes you think, like. Oh there's a socket, and it has a buffer.

A

It's just a generalized data structure that wraps packets and then finally, we go into the general receipt offload queue. This is a software implementation of a hardware specification called large receive offload, and what this does is, if, if you want, if you haven't enabled you can trade a little bit of latency for a lot of throughput and the packets will aggregate and grow and go into a generalized queue that then will get processed.

A

So if you ever run, TCP dump and you've seen undescribable large packets coming in it's probably because you have gr o turned on and the kernel is giving you fake, huge packets, I'm not going to focus on that too much, because the more important part is getting into receiving the socket buffer and kind of going up the layers. So we call in to receive socket buffer, and this is where we can finally start to differentiate between layer, 2 and layer 3.

A

So when we call receive socket buffer if the network card driver hasn't implemented xdp which a lot of them haven't, although the list is growing quite extensively, then xdp will get called here again. Cilium zhn etwork policies tend to live here. Among other things, the next thing that we do is, if there are some kind of tap interfaces receive socket buffer function, will send the packets into some of the tap interfaces. We don't need to worry about taps for cilium, because there are none. So we move on to the traffic classifier, the traffic classifier.

A

If anyone's worked with queue disciplines and the Linux kernel before the traffic classifier can kind of deal with ease, it can do some routing, it can mangle packets, it can drop them and it drop them in a way in which it may or may not influence the overall statistics of a classifier or queueing discipline, which is really useful. When you're trying to do generalized traffic shaping.

A

Finally, we hit the layer, two net filter again I, don't even think that cool proxy deals with the layer to net filter, cilium doesn't, and so the next thing that we do is we move on to whatever the network device is default. Protocol is 99% of the time. That's IP or ipv6 I, don't know.

A

Maybe some of you were running Apple talk, but I, don't so we have IP receive and the IP receive is I hope that you can see that what we're doing is we're decap, slating right, we're truncating the layers and we're kind of trying to move up. The stack so IP receive gets called, and the first thing that happens is we call net filter, pre routing, so if you've ever written, IP tables pre routing filters before this is where they exist and they can drop a redirect here. We call IP receive finish.

A

We go into the routing package, see Liam will use the routing package if you're not tunneling. So what that helps to do is that helps to short-circuit the data path and move packets down into the beef pairs a lot more quickly, and then we call them that filter post routing these two net filters I think, are the things that could potentially go away in the future.

A

So if you can imagine this flow diagram that I've created without any of the net filter squares, that could be a possibility in the future and finally, we get IP local deliver. This is actually the protocol aware function. So is this TCP or UDP, or maybe it's some other defined protocol that we we don't know about? There are other protocols that the kernel can call into they're, usually kernel modules, though so I don't want to.

A

We don't need to really worry about them, and then we hit the transform policy, and this is where things like: IPSec live, UDP de capsulate things love that nature and then finally, we reach a layer for protocol and I'm gonna focus on UDP, because I don't want to chase the TCP state machine around the Linux kernel, because it's quite unwieldy.

A

So imagine that it is TCP and it's a useful lie. I guess so we receive our UDP packet, and now we have a port number. We finally have something that can map us into user space, and so we're nearing the end of the bottom. Half of our work here UDP receive finds the socket. It goes through one more transform policy, the the layer 4 protocols can go through transform policies for the cilium CNI they do not, and then the sockets have two data structures.

A

They have a receive Q, which is just a list of the socket buffers that are waiting to be processed in user space, and then it has a weight queue. And if you remember that napi pull mechanism- those are the that is the data structure that the different poll mechanisms inside the Linux kernel can use in order to route the packet up into user space. So we push our socket buffer onto the receive queue and we flip this special bit in the weight queue and now we're we're nearing the end of our our packets journey.

A

So this is a socket listening program. It doesn't have any error tracking or whatever, and the e-poll part of it isn't really strictly speaking necessary because receive can do polls for UDP but again pretending that this is like kind of what we need to do. We allocate a socket. We create an e poll instance.

A

We add the socket to to e poll for listening for, receive events, and then we e poll wait.

A

You pull weight and then, when that weight cube, it gets flipped that we talked about the napi mechanism wakes be pulled up, and then it comes back out of user space. Well, first, it populates the events, you're listening for or errors out, because sometimes a pull is wrong and then you come out of user space and you can start processing your packet in this case. We're doing UDP, receive message and so UDP pops the packet off. So this so we're. You know calling an assist.

A

Call now and UDP POC receive message: pops, the packets, that we want to process off of the sockets receive queue, and then it runs them through two more BPF programs. The first one we are concerning ourselves with is the C group EBP F program? This is pretty cool for cilium. This allows for clients, transparent, client-side load-balancing, so if you've ever done, client-side load balancing. You know that your application needs to be aware of it and you're just kind of doing it yourself in the application space. Helium has a configuration thing that you can turn on.

A

That allows you to do client-side load, balancing, that's completely transparent, so you're hitting some load balance service in kubernetes and cilium can load balanced by a policy for you, yeah whoo proxy replacement, there's a lot of things that finally had to happen for coop proxy to get replaced, but I think node. Ports was the biggest one which I think would be a little bit higher up the stack.

A

But there's like there's a lot of stuff that it's doing. If you were following along with that saga, you would look at your your IP tables rules that queue proxy had generated, and you would notice that none of them were being used except for a handful and then as cilium get matured and matured and matured fewer and fewer of them were being used, but yeah trying to think what made you say that way. Oh.

A

Yeah, no, that makes sense I sorry his what the reason he said. That is, because he's that koo proxy does do with iptables. It does some of this internal load balancing on the node. My impression is that koo proxy does a lot of that when it's receiving, though correct okay. So this is more when you're egressing right, you can do a transparent, client-side, egressing load, balancing, which is I. Think this. This is how I think psyllium and envoi work really well together.

A

In this instance, I I should say the psyllium guys told me to say that I don't actually know why they told me to say that I, don't know anything about em boy, so.

A

Okay, so our final, our final BPF program is socket operations and this is really useful for traffic shaping. So if we were dealing with tcp, although there is some stuff it can do with UDP, we can do really fine grained traffic shaping at the socket level. So this client over here for this server gets less bandwidth than this one over here is the kind of thing you can do with sock ops and then, finally, we reach the end of our journey and we have a buffer in user space alright.

A

So this is kind of a an interesting map of kubernetes networking objects to cilium networking objects to that terminal primitives.

A

So there are more networking objects, I realized them in kubernetes than what I've listed here, but these are the primitive objects. So, for example, load balancers are node ports and cluster IPS, which in turn are just service objects, and so those map pretty cleanly on to cilium service objects.

A

Silly mess I mean you see what I was saying about the names being quite similar, so this chart looks good, but unfortunately, the whole story with this charts is more complicated, because psyllium is actually pretty complicated in its implementation, and there are two really big reasons for this that are worth darling upon momentarily and the first one is that the data path is complicated and the data path can change based on the version of the Linux kernel that you're running, so the BPF programs weren't all there in the beginning.

A

So, for example, the C group EPF program is quite new, and so there are lots of different ways of doing traffic shaping maybe in this version of the Linux kernel, there are drops happening in the traffic classifier rather than xdp.

A

I actually don't know if I don't actually think that's a great example, because I don't think that happens, but that's Anna kind of example of what might be happening is that the data path chart can kind of shift and then the other reason this is complicated is that well, what I was kind of talking about is just the the shape of the data path. Changes based on your own, your application logic as well, so, depending on the policies that you implement, they may or may not be implemented in these various programs.

A

So all right, so I'm gonna pull a little bit of a bait-and-switch I hope the title slide kind of indicated this I titled. This talk, understanding and troubleshooting, and my my goal is to kind of my goal and like this, the slides are available to download and you can use them and I think we're gonna try to release them under a Creative Commons license if people want to play with them or add to them or change them. Although they'll be out of date in, like six months, probably.

A

Yeah the goal was was more understanding and the troubleshooting part is is more just understanding what's available, so I think the most useful resource really is the psyllium docker image itself.

A

So if you're running psyllium is and I in the kubernetes cluster exacting into the cilium one of the cilium data set daemon sets, it gives you a lot of power and the two most useful tools in that docker image I think are the cilium CLI itself, which has a lot of functionality in it for debugging and then the second one is BPF tool, which is the tool that the Linux kernel, devs wrote to deal with, BPF and one reason it's so cool that the cilium docker image has.

A

It is that it's not mainline packaged with any of the normal distributions right now. So, if you're running a boom, if you're running whatever you can't app get BPF tool right now, there is a PR to get it into a boom soon, but it's there and psyllium, which is a good thing. So I have a little kind of video to show.

A

Maybe what I can do as a mirror.

A

A

A

Okay, so what we have is we have this service in kubernetes and it's throwing lots of 400s. It's a load balancer with the deployment and I'll shoot. Sorry that I got truncated there see if it and we so we have this service and I'll. Just tell you what's happening since it's getting truncated, I'm about to exec into the cilium pod to see what's going wrong with it, because the service is just throwing the 400s so we're going to exec into it.

A

This is really unfortunate.

A

The person next to who yeah that's a good point, good call.

C

A

Okay, where am I in this thing,.

A

Okay, so basically what happened is I exact into the psyllium pod and I ran a command called psyllium endpoint list, and what that did is that listed all the endpoints on that node? It showed if they had ingress or egress policies attached to them, and then it gave me an endpoint ID and now what I'm about to do is I'm about to run a program called psyllium, monitor and psyllium. Monitor is basically like TCP dump.

A

It allows you to monitor specific sockets, even and within one endpoint or just specific endpoints themselves, so now I'm getting a hex dump of of what's going on in this application, and there's lots of great stuff that you can look at like it's it's protocol aware. So it knows that the connection has been acknowledged and connected and looking at the hex dump I see that proxy protocol is enabled on the client side which that that was our issue, yeah we're using no proxy and we should be using proxy.

A

So, let's fix that and get rid of get rid of. You know make the server proxy protocol aware we redeploy it and I bet. This is gonna work now, okay, it's a pre-recorded demo.

A

So yeah and then the other tool I want to really highlight is object, BPF tool, so it ran a little bit fast, but basically I ran the command BPF tool net and it dumped all the BPF network device programs that are attached. So the if you look at that list, output you'll see that there are network device names and then there are other things which are really useful.

A

Specifically, there is the name of the program that's actually attached to the network device, so we can go in to see Liam's repository go into BPF. Look for the name of the program. If the section name of the program is in there, we can look for the section name and that's actually the the actual BPF code. That's running on that device. That's I think that's really useful, because that first of all, I think that's easier than looking at a giant iptables chain.

A

But if you're ever dealing with something with psyllium, you can easily take a look at the logic just by running that command. So there's a couple of steps.

D

A

And I think that's all I have.

B

A

No I have one more thing that the cilium guys wanted me to say so they just dropped a new open-source tool. Maybe a lot of you saw that this week called Hubble and it actually is a web UI. That does a lot of this stuff. So you can get really fine grain insights into your network, down down to a very, very granular level and.

A

See where why and where drops are happening, you can look at where the policies are taking effect and kind of get better insight into the full data path of of your network, and that's it. Thank you very much.

C

Thank you, Nathan. We do have a couple of minutes for questions. Yeah.

A

If anybody wants to ask questions, I'm fine to stick around otherwise, otherwise, not yeah.

C

D

A question that's kind of more general to BPF, so we're running a lot of logic inside of the kernel or resident to the kernel and I'm curious on you know if there were a bug and a compromise of that bug, you'd effectively be running in ring zero I'm, getting full control of the system. So I'm just wondering like what does the colonel provide? That's doing you know some protection against, say a rogue. You know code, that's that's running in BPM.

A

That's a really great question, so I think everyone probably heard that so there is a verifier in the in the compiler, but I think that's a really good point. I mean that's. Why I said an e BPF is a good step forward. I, don't think it's the end goal, because yeah making your interface Rob by code isn't really secure right.

A

So I am really interested to see if we can come up with better solutions.

A

The network is really hard to get right and it's very understandable why people want this mechanism because it just provides the level of flexibility like an order of magnitude greater than than what we have, but in some ways it is a step back because yeah, it's I mean some of the worst security headaches of the last two years have come from BPF, so it's a trade-off, but the net dev mailing list won't. Have it any other way. So.

C

Yeah, just just.

E

Thinking so you sure that code segment, where the actual code was put into the kernel right, see if it had a problem in the code. What was written boo to crash the kernel.

A

A

The verifier will probably I mean I'm, not gonna, say you couldn't crash. The colonel I think it'd be really hard to do because, for example, it doesn't allow you to execute more like it. This is configurable at compile time of the colonel, but it believe it will allow you to execute more than forty thousand ninety six instructions per program, and then the maps are also severely constrained right. You have to tell before you run the program at compile time.

A

You have to say how big this map can get so it the the programs, are quite deterministic I'm sure that there's a way to mess it up if you really wanted to but I would I would venture to say that that's a pretty that's a I would say it's pretty safe, and it's also why I think it. It's also why it's good to use something like psyllium and good to use open source code right, because it's being used in production by a lot of people, so.

E

A

You already have a what module.

E

A

You already have a netfilter kernel module. Would it be better to use that or EBP eff? Is that the question that I mean I, guess it just depends on your use case.

A

Yeah I would need to know more about what the exact use case is in order to weigh in on that I. Don't know it's a good question, yeah I, to be clear: I, don't want to like stand up here and see that say that EBP F is a panacea right. The right tool for the right job. Definitely don't just throw everything: yeah, don't throw everything out the window. Ip tables is still it's still good and it's been around for a while.

C

One more and then can for the stage.

B

Do you think, if I'm nowhere near needing the performance I'm nowhere near running up into IP tables performance issues, so is the tooling and I couldn't see it I'll go check it out layout is like the tool. Are you saying even independent of performances is the tooling there that it would be compelling for me to do it this way, independent performance, I.

A

Actually do think that that is one of the unsung benefits of psyllium is that it is quite easy to use. It's I think it I personally think it's easier to grok, sometimes than looking at a giant iptables list it's easier to to find problems. Sometimes it is kind of a one-stop shop, whereas sometimes you're messing around with lots of different net tools in traditional Linux.

A

Yeah I will defer to my earlier statement of networking is hard, though.

C

All right I want to thank again Nathan and don't forget to.