Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Better Bandwidth Management with eBPF - Daniel Borkmann & Christopher M. Luciano, Isovalent

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Better Bandwidth Management with eBPF - Daniel Borkmann & Christopher M. Luciano, Isovalent

Kubernetes provides many knobs for managing common system resources such as vCPUs and memory limits per Pod, but often forgotten is the effect of unbounded network communication in a cluster. A large churn of packets from several services can starve bandwidth for other services. Also, out of the box TCP congestion management is not optimal for Internet-facing services. In this talk we will explore how eBPF can be leveraged to dynamically insert logic for flexible, efficient and scalable rate limiting and bandwidth management on a per-Pod basis. This talk details: - The scalability limits of token bucket filters by the bandwidth plugin, and why EDT (Earliest Departure Time) combined with eBPF is a major step forward. - How TCP congestion control with BBR can now be leveraged for Pods thanks to eBPF for significantly improving application latency and throughput. - The benefits of enforcing bandwidth limits at the egress point and considerations when to use ingress enforcement.

A

Hello and welcome everyone to our talk. It's really cool to see a fully packed room. um Today's talk is about better bandwidth management with ebpf, I'm daniel brookman, I'm I co-maintain ebpf in the linux kernel, and I work with the psyllium team and the talkers together with christopher nikolai, who are also from the psyllium team.

A

Before we look into bandwidth management, I wanted to start out with an interesting metric, which you can see here on the right side. The metric is basically the density like the number of uh parts per node and what you can see here, like the more people use kubernetes the the higher the density gets. So, for example, uh it's probably reasonable to assume in 2022 that there's that there is a median of around 50 parts per node and with that increasing density.

A

It also means that there's a competition for resources on a given node, for example, cpu and memory. So basically, operators have to have the issue that they need to tackle, how to allocate and efficiently use resources.

A

One tool that you can use in the parts back, for example, is that you can define resources, requests and limits, for example, resource requests.

A

They define how much cpu and memory a given part requires, and then cubelet will make sure that that part is scheduled in a node which can actually satisfy that constraint limits. For example, if a part overshoots its memory usage, it will get om killed.

A

So now the question is what about networking so tcp in its nature sense as fast as possible. What you can see here in the right picture is a typical tcp congestion control algorithm that will try to send more tcp segments to the network. So there's an exponential growth fast in the beginning, until it experiences a packet loss and then it will back down and then it will try to do the same thing again.

A

So really, the output contract for tcp is to send as fast as possible and shaping is typically done by device output cues. So, basically, the queue limit for those output queues, as well as the remotes receive window size, determine how many packets from tcp can be in flight and but who actually limits a parts network usage in kubernetes.

A

So there's a infrastructure for that in kubernetes, so there's a bandwidth enforcement. So far this has only been experimental. Unfortunately, so if you look into the parts back, there are kubernetes specific annotations, so you have an ingress annotation. You have an egress annotation. Then you can, for example, set something like a part should get equals bandwidth or 50 megabits per second and support for that pod.

A

Annotation, so far has only been implemented with the bandwidth meta plugin, that's a plugin from the cni plugin collection and that basically implements a a basic token bucket filter from the linux traffic control subsystem.

A

So how does it look? um For example, if you have a typical node that you can see here, there's a part in the node and the part is typically connected to the host namespace with two weave devices.

A

One lag in the hostname space, one lakh of the rest of the reef device in the part name, space and token bucket filter cudas are attached there, but the real problem is it's not scalable for production use, and I will show you why, if you, for example, set an ingress bandwidth rate of 50 megabits per second uh in your parts back then, basically that bandwidth meta plugin will then attach a token bucket filter queue disk to the weave device in the host name space that is attached from the tc subsystem in linux on the eco side.

A

But if you look at the logical traffic flow point of view, it's uh like ingressing into the part, the traffic.

A

So if, if there are many clients connecting from the internet to that part, for example, they will all hit that token package, filter qdisk and it has actual design issues, because the token bucket filter queue disk. It's given, it has to track its state in terms of the shaping it there's, basically, a single log that has to be taken across um all cpus and that's a huge contention point if you want to.

A

If, if you receive traffic from multiple cpus at the same time, and that basically completely defeats a multi-cue capability of physical nics, because typically a nic has many receive cues, then once packet arrive on the different receive queues. Cpus will pick that up in parallel. They will process it. They will forward it in parallel, but then they will all hit that single token bucket filter queue disk.

A

The other issue with that is also that queuing on the receive side is actually a no-go, because it it consumes resources it. The packet basically made it over the wire. It consumed your network and then only to receive it on the node itself, where it then goes into that q disk it's waiting in that queue instead of being processed and only then to be dropped, for example, so that really causes buffer plot for your network.

A

If you look at the other direction, for example, if you install egress bandwidth or 50 megabit per second, then that bandwidth, meta plugin will basically set up an additional device. It's called an ifb device type and then all the traffic from the host weave device will be redirected to that ifb device and on that ifb device there is the token bucket filter qdisk installed with the given rate. Why is that workaround?

A

You might ask, but the the issue is logically, that traffic in the host namespace when it egresses the part it arrives in the traffic control subsystem in the linux kernel, on the ingress side and on the ingredient side, the linux kernel cannot shape. So you need to redirect it to a different device only that you're at the dc egress layer and only there you can attach the token bucket filter qdisk.

A

So if you have multiple applications in your part, they can send traffic from multiple cpus and they will all hit that same token. Bucket filter queue disk that also has design issues, because now you're now you're actually queuing twice: you're queuing on the weaved like on the ifb device, but then also you're queuing on the physical device, because there's typically also a q disk attached that is handling all your outgoing traffic, uh for example, on linux by default, that's ftq codal. So that really causes buffer bloat, which is a big issue.

A

Then again you have the single lock contention point across all the cpus, so you cannot actually make it scalable across cpus and then there's also a mechanism in the tcp stack, which is called tcp small queues, tcp small qs is there to reduce buffer plots so that you reduce excessive buffering in queues and tcp. Small queues basically works that works. The way that tcp stack tries to not send too much packets.

A

But now those packets are stuck in queues and when, like once, they are processed there, they go to the upper stack and it will basically fool the tcp stack that the packet might might already be on the wire. Even though it's still on your host.

A

And last but not least, you know you need three net devices instead of just two. So overall I would say it's a latency killer, because you have you have to deal so much with queuing. It doesn't scale across cpus and yeah. It's not really ready for production use. The other thing is like the nature of tcp when you send as fast as possible and what you can see when you look at the cueing theory like so once you get to the point where you uh consume the bottleneck link close to 100.

A

What you can see here in that graph is that that the wait time of packets in the queue they basically skyrocket to infinite.

A

So there has been some interesting research from from google folks and they were asking themselves so can we get rid of queues entirely? So they came up with a really cool idea, which is uh called the earliest departure time model. So they basically said: let's, let's replace, let's replace uh queuing fully and come up with two simple core pieces.

A

One is the earliest departure time, so it's basically a time stamp on the network packet in the host, which will dictate the the time when the packet can be delivered to the network at the earliest possible point in time and the other one is the timing, wheel, scheduler, which will basically hold this constraint and then send packets out.

A

So how can this model be applied to to kubernetes so enter ebpf? So you've probably heard ebpf many times at this conference. So it's basically a way to make the kernel programmable.

A

You have a small programs that get verified for safety and they can be attached to different points in the linux kernel and sodium is an ebpf based cni or you can better call it like networking platform because it does many things. So it does takes care of part connectivity, but also service, load, balancing network policies and so on and so forth. I don't want to go too much into details.

A

The one detail I want to go into is the bandwidth management, so we implemented the sodium bandwidth manager and that bandwidth manager basically allows for lockless earliest departure time based uh part rate limiting with ebpf.

A

So basically what you can see here, the the enforcement point we moved from the reef device to the physical device so that you don't need to queue twice. You can avoid this additional buffer bloat if you have it on the physical device, and you also don't need the upper stack.

A

So we have a mode in psyllium which is called bpf host routing, where all the routing can remain in the tc ebpf layer and it can be forwarded there directly to the physical device and you can use certain functionality of the networking stack such as, for example, the fip lookup. You can just reuse them out of bpf itself, and the good thing with that is that it will keep at a tcp small queue feedback properly.

A

So how does the architecture look like of the bandwidth manager if you set a port, a specification that it can, for example, send 50 megabit or so then, in the end, like the packet arrives at the physical device? There's an ebpf program attached to it uh that was orchestrated by psyllium. It is taking care of the packet departure time stamps and psyllium also sets up a multi-cue qdisk with so-called fair q leaf q discs on the device and those fair q leaf q disks. They basically implement this timing. Wheel, scheduler that I mentioned earlier.

A

If you look at the performance of this uh architecture compared to the token bucket filter approach, it's really interesting because um we run an experiment with multiple concurrent flows in parallel.

A

In this case 256 and those flows, they are basically ping-pong flows, so it will send one byte of data to one direction and one byte back to the other direction. We do that because we want to stress, like the the latency. We want to see the latency of the of the whole system, and each of those concurrent flows has a rate limit of 100 megabit per second.

A

So now what you can see here is the time in microseconds, so the yellow uh bar here is basically the token bucket filter approach and the green one is the earliest departure time model that we implemented with psyllium.

A

So when you look at the median it's around seven times lower latency than the uh traditional approach which is based on cueing, and even if you look at the p99 latency, you get over 4x, better latency and if you look at the actual transaction rate, so when you run net perf, one transaction is basically like a like a ping-pong so one byte of data in one direction and then the other direction you get over seven times better transaction rate. Under this 100 megabit per second constraint.

A

All right so so far we have seen how we can implement scalable bandwidth management with sodium based on the earliest departure time model. But what about thinking even more broadly, can we do like even for the for the internet band, better bandwidth management?

A

So what else does the earliest departure time model enable it enables bbr? So bbr is a tcp congestion control algorithm that has been developed by uh folks from google research and the internet.

A

Today is basically it's basically what you can see on the on the left side of the picture, so you have a loss based loss-based, tcp congestion control algorithm. So if you don't change any defaults on your linux laptop or server, for example, you will have that model. On the left side, it's a so called tcp cubic congestion control algorithm. So, as I mentioned earlier, it tries to ramp up its congestion window until it's until it experiences congestion lost somewhere and then it tries to back down.

A

So you see the sawtooth pattern for a connection on the right side. You have the bbr, so bbr is modeled in a different way. So it's basically modeled around the delivery rate that the traffic can be delivered to the to the receiver as well as the round trip time. So it's not based on packet loss. Basically, so when would it be useful to consider bbr?

A

So if you have a kubernetes cluster, where your clients connect from outside of the internet, uh then it would be um really useful to look into that, because it will significantly improve your latency for low end last mile networks, because typically buffer bloat happens on your on your home routers and it will improve there, but it will also significantly improve the throughput for for high speed networks when you have uh like a like a long delay in this case.

A

So I run an example. For example, I reserved a server on new york from from the packet.net, and I tried to do an iperf measurement to our lab in zurich that we have and what you can see here with the defaults. The default is tcp cubic and fqccaddle as a qdisk.

A

What you can see here, when you look at the bit rate, it's trying it's slowly, trying to ramp up until it reaches a point where it gets to 400 something megabit per second, but then it experienced lost somewhere in the path, and at that point it's you know like it's uh slowing down again and it's trying the same thing until it reaches 400, something megabits. So, overall the average bit rate is 270 megabits per second.

A

And now, if you switch over to bbr for this experiment, um you will see that it will ramp up until until 400 megabit per second, and it will plateau at that point in time.

A

So, just by changing that congestion control algorithm for connections over the internet, you can reach from megabit per second to over 400..

A

So now the question is: can bbr actually be used with kubernetes parts? um So bbr works in conjunction with the with the fqdisk which implements this timing wheel. So you really need a timestamp for your packets, but the problem is: when you have network namespaces, then the kernel actually clears the timestamp when the packet goes from one weave device to another. So, in that case, bbr can actually not work because it will get an instable rate because the timestamps are zeroed.

A

So why is that? It's because of a kernel limitation. So in the kernel we have a packet representation which is called the socket buffer and the socket buffer holds all possible metadata around the packet, including a timestamp, and the timestamp has like for the receive side and for the transmit side it has a different clock base, for example, for the receive side it has the so-called clock tai, it's a it's an atomic clock and for traffic that is going out from the transmit side.

A

It has a clock base of clock monotonic and the problem is the socket buffer. Try is like the most critical data structure in the linux kernel. Networking stack because you have to keep it as small as possible, just to avoid cache misses just to keep your performance high.

A

So there's just a single 64-bit timestamp field in the kernel, and um that is why it had to be cleared and the problem is people tried in the past to standardize just on clock tai, um but it didn't work because the it's like the tai clock when you is typically taken from like like somewhere from the hardware, for example, you have a hardware clock and it will take over the timestamp into the packet.

A

But if there are some, if there's buggy hardware or if you have some clock skus, it will basically mess around with the timestamp too much when, for example, if you get a too high offset for your timestamp, then it will cause a drop in the fair q qdisk because it will go over a certain horizon and that will break tcp.

A

So it has been tried in the past to just standardize on a single clock, but it had to be reverted and then, when you forward from receive to transmit that's the limitation where people had to clear the timestamp back to zero and just as a note like this monotonic clock is not prone to such a clock. Skus.

A

So yeah, that's the reason why it couldn't work, and we have worked together with guys from facebook to fix that problem in the linux kernel. So we presented the issue. I think it was at the last year's bloomberg conference and they run in the end into the exactly the same, and so we fixed this together and now. The timestamp is actually retained for, like the outgoing timestamp for sockets that are in uh that are in parts.

A

So, when packets traverse the network namespaces, it will be retained and then all the way until the physical device. So now with the bandwidth manager that we implemented in cilium with the whole architecture.

A

What basically happens is you can do all the routing in the tc bpf layer, the timestamp will be retained and also the socket association to the packet which is needed for the tcp stack to get the proper feedback and then on the physical device. There's this multi-cue with the fact you leave qrdisks to implement this timing wheel and then it can basically work.

A

Just as a side note bbr. You actually only need to to enable this on the server side, because uh so you don't it's not necessary to have it on the client side, because, typically all the bulk traffic comes goes from the server to the client so now to our demo. So in our demo we are basically want to show you um a streaming service that we implemented like a mini uh netflix in that sense, and we want to compare it for cubic and bbr under different network conditions.

A

This is basically how our setup works. How our setup looks like, so we have a mini cluster that mini cluster uh has one node on that node. There are two parts and yeah there's an ffmpeg part which basically ingests a video and it chunks it up into smaller pieces, and the engine x part is basically the the front end which serves those video chunks to a client, and it is basically behind the kubernetes service that we expose to the internet.

A

And, last but not least, we have an external client. It can be somebody with a phone it can be with the laptop or just the regular workstation. It connects to that kubernetes server that we have over the internet and we we emulate latency and then also later on, latency and drop with the help of a so-called net mq disk, so netemcudus and linux traffic control subsystem is explicitly there to simulate bad network conditions.

A

And yeah, so we want to show this under two different configurations. So one is the bandwidth manager with the cubic tcp congestion, control, algorithm and the other one is the feature that we implemented now with bbr and with that I'm switching over to christopher.

B

A

B

Hello, so what we have here is two clusters: we try to make them as similar as possible, they're both running in madrid in the same data center. The only difference between them is one is enabled with cubic the standard configuration, the other is enabled with bbr, and so what we do is we we scrape in, and we see two different videos.

B

Okay, we got really excited when uh we had like a lot of latency problems, just walking around uh I'm personally data capped on my smartphone, so I can't watch a lot of videos or else they're constantly buffering.

B

So what we have here is one of our colleagues and ponchocena is, you know, doing a little bit of sky surfing here and on the left side? Here we have our cluster with uh cubic set up, and what we see is that cubic is constantly sitting inside of a low resolution and we're loading uh everything.

B

However, we did introduce a little bit of latency, so what we're doing is we're introducing packet loss like every one percent of packets are being dropped, as well as we're introducing our own, um like 100, milliseconds or more of latency, and we're able to do a sustain in the right-hand side with bbr much higher resolution even under distress, so both have exactly the same amount of injected latency being injected into them.

B

But what you see on the left side, if you open up your chrome tools, is that even at 100 milliseconds we're not able to pull down the high resolution ever with a cubic on the right side, same amount of latency. We always get hd quality and we pull that down with the same amount of latency.

B

And so the important piece here is that uh you will sometimes still you know, there's obviously, congestion, there's packets being lost. But the important part is when vbr hits this. It it'll hit the limit. It maybe goes spicy for a bit, but then it keeps right back up at it and it will hit the hd limit under sustained latency cubic will never cross over into hd range. At this sustained latency- and you know, it'll often have a lot more rebuffering.

B

And we can also look into a few of the statistics using the ss utility. So what we see here in cubic is our congestion window being sent a little higher, but it will it will uh kind of sustain around seven or so segments that it's able to send during that window, whereas if we were to do the same thing over in our bbr cluster, we'll get a much higher cap and that'll be sustainable over time. So in here we're operating at around 416 so much much higher than we were able to send before.

A

I should uh mention that the guy in the video was our colleague tom, so he was uh flying in his paraglider at one of our on-site events when we went skiing all right so now the question is, uh you know, is pbr the golden solution. Well, uh bpr has potential uh unfairness issues so, for example, if you use it in a combination with with cubic. So if some nodes are in cubic- and you know in your cluster- some others on bbr um bbr is pretty aggressive, so it could overrun basically cubic.

A

You will see a higher rate of re-transmissions on tcp. Simply it's trying to more aggressively probe uh the google research folks. They are working on bbr v2. um In order to overcome those limitations.

A

uh I just want to call out there's an interesting talk from the netdev conference which, from from someone from dropbox, where they deployed bbr on the edge and also bb rv2, so they did some early measurements. What you can see here, like on the green side, is like this there's like a three percent tcp retransmission rate on the bbr v1, whereas the v2 stays around one percent, so um it is trying to overcome those issues, but on the other hand, bbr has been deployed in production on a large number of fleets.

A

So it is definitely something interesting to consider yeah. So, coming back to the earlier problem statement that we have like, how can you make sure that parts can be rate limited, so the bandwidth enforcement from kubernetes really doesn't have to remain in a poor state. um So with the psyllium bandwidth manager implementation, it will be ga with the next sodium release, which will go out shortly after kubecon.

A

It basically implements efficient and efficient and scalable ebpf, based enforcement with the help of the earliest departure time model and with the work that we did around bbr and to retain the uh timestamps across the network namespace when packets traverse them it's.

A

We are basically like the first cni to support bbr that you can use and deploy your ports with and as a side note, I should mention that realizing this whole architecture around the bandwidth manager is possible thanks to ebpf, simply because with ebpf you can do flexible traffic classification and also set the packet delivery timestamp.

A

If you want to try out the bandwidth manager, there's a getting started guide that we have in our selium documentation the bandwidth manager itself. If you want to use the edt based rate limiting for parts, it needs the kernel 5.1.

A

The support for bbr is a little bit newer because we got that merged around december in the linux kernel, so it requires 518.

A

With that, I would love to thank a couple of folks, especially from google facebook and from our team and the psyllium bpf and netdev kernel, community and yeah. So that's it for the talk thanks thanks a lot and if you have questions I'm happy to answer.

A

Yes, not sure who will give you the mic.

A

Otherwise, uh please shout or I will all right.

A

uh Yes, so I should repeat the question: um the question is: if there's a like a lot of bandwidth available and cubic is trying to ramp up quickly. Is this the same with bbr right? um Yes, it is, it will just uh it. Just has a different uh delivery model around it and it's not prone to packet loss, but it will see when it uh it will see the delivery rate based on the measurements and it will plateau at that point. So, yes,.

A

Also, if there's an increase in available bandwidth, yes exactly, they will both have the slow start, ramp up, yeah cool.

A

Okay, so the question is: if you have a load balancer in the middle, do you also have to set up bbr in the load? Balancer? No, you don't! So you only have to set up. You only have to set it up on the server side, so on your back ends because it's actually transparent, so it you really need to have it on the socket and that's in the back end in the application.

A

Yeah, exactly because the packet is basically load balance based on l4 layer, it you will just redirect it. You will forward it. It's a it's a node in the middle yeah cool all right.

A

If you have further questions, we are still around here, so you can just come to us and ask or ask in slack we will try to look there and yeah with that. Thanks a lot.