Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Measuring K8s Network Performance - Kornilios Kourtis, Isovalent

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Measuring K8s Network Performance - Kornilios Kourtis, Isovalent

Benchmarking is hard. Benchmarking K8s networking doubly so. Measuring the performance of K8s networking is the only reliable means for users to understand the capabilities and limitations of their, often unique, infrastructure. Furthermore, benchmarking allows for informed decisions by quantifying the tradeoffs of different stacks and investigating how performance goals can be met in the most cost-effective way. Yet, it is a hard endeavor. Both the software (from the OS to the application) and the hardware (from the CPU to the NIC) stacks are extremely complicated beasts, rendering results confusing or even misguiding. This talk aims to guide practitioners to properly measure k8s network performance. Specifically we will discuss: - How different workloads and metrics can be used to answer different questions - Setting up and executing benchmarks - Common pitfalls we have encountered in practice, and how to avoid them - Validating and interpreting results

A

Hello, I'm cornelius, and I will be talking about measuring magnetic network performance. I am a software engineer at suvelant, where I work on cilium, which is a network observability and security platform for container environments such as kubernetes.

A

We implement celium by extensively using ebpf, but this talk is not about celium or ebpf. In the ceiling team, we spend a lot of time evaluating network performance, identifying bottlenecks and fixing them. So in this talk, I would like to offer some practical advice for practitioners who also want to evaluate the network performance of the kubernetes cluster, and I will start by discussing potential motivations for doing so.

A

Performance valuation is really useful for asserting or disproving assumptions. For example, it can be used to verify that using more expensive machines with better network cards will actually improve application performance and going even further. It can also check or verify that the benefit the performance benefit that applications will get will actually justify the additional cost of acquiring this new hardware on a related topic. Evaluation can be used to offer concrete and objective arguments for making technical choices.

A

For example, if an organization wants to select a cloud provider and data performance is a factor in this decision, evaluation can be used to actually quantify the different tradeoffs between the different cloud providers and also um try to do the experiment, considering what are the unique needs or workloads for this specific organization and finally, evaluation can help with better understanding your platform.

A

For example, if there are applications that exhibit random delays in communication, you can use evaluation to try to figure out whether this is a network problem or something else is going on and, generally speaking, playing with benchmarks. It's a great way to understand your infrastructure.

A

The problem is that benchmarking is a really hard process and kubernetes network benchmarking, even more so, and the reason for that is that there are many different layers involved in making. For example, two kubernetes boards talk to each other.

A

First, there is a network itself which in many cases, might have an unknown topology and also it might be used by multiple different users having different workloads, and this means that performance might vary between different times or even depending on where the machine is located.

A

Furthermore, uh the nik or the network interface card, which is the part the hardware piece that connects a machine to the network, is a very complicated piece of hardware and it includes a number of optimizations that are responsible for providing good performance just to give some examples.

A

This includes multiple unique hardware cues receive such scaling protocol exam offload, hardware, segmentation, offload, interrupt coalescing and a bunch of others, and all of these optimizations need to properly work in order to get the best performance you can out of your kubernetes network.

A

Another factor is that the host machine might not be a physical machine, but actually a virtual machine. A vm and this vm would share the node hardware with other virtual machines, which introduces additional complications when trying to evaluate performance going further up the stack.

A

There is the host network stack, which includes everything from the nic driver to the protocol processing stack, and this is also a layer which has a lot of different optimizations such as polling such as segmentation of load implemented in software and things like fast packet forwarding between different devices, which is really important for kubernetes networks and again, all of these optimizations are needed in order to guarantee good performance and how they interact with the hardware or with other layers.

A

It's not always obvious, and finally, there is the piece of software that implements the kubernetes network abstractions, and so these are things like pods and services, load, balancing security policies and so on so forth, and this piece of software uses mechanisms such as connection tracking or address translations that complicate packet processing further.

A

So to summarize, the hardware and software stacks that implement a modern kubernetes network are extremely complicated and this makes performance valuation a really challenging process.

A

So now that I've talked about the challenges, I'll talk a bit about the evaluation process from a high level perspective, and I find it useful to think of this as a series of steps, starting with setting the goal for this evaluation.

A

Then defining the setup- and this includes things like the workload or the metrics- that we want to monitor or, generally speaking, the system configuration. While we have defined the setup, we can move forward and do the experiment and get the results that we can evaluate.

A

An important thing here is to try and treat this as iterative process where you evaluate results and then go back into other experiments to kind of refine your understanding and improve your understanding, rather than a thing that you do once and you are done with now I'll talk a bit about the setting and evaluation goal, and while this might be might sound simple, it is really important that each evaluation strives to answer a question and the more specific the question the better and it's even better.

A

If the answer to the question will lead to a specific decision so that you can move forward in whatever your goal is another thing that I find useful is to try and set an expectation of the result for the evaluation, and this might be verified or not, but I find it really useful to think about the sort of thing about sort of things proactively um during this talk I'll mention a couple of pitfalls uh that I think people should keep in mind when trying to evaluate kubernetes network performance and one of them.

A

It's uh is that it's really easy to get booked down in details due to complexity, as I've mentioned before the hardware, software stacks are really complicated and it's easy, and sometimes it's also interesting- try to figure out exactly what's going on on every different player.

A

So it's really important to try to focus your efforts and keep things as simple as possible so that you can get meaningful results and then iterate and refine your understanding. So this is something to watch out for now.

A

um I will try to uh mention some examples of what do I mean by a question that an evaluation should try to answer so an example of such a question might be how does saline perform, but this is not a very useful question because it doesn't really say much about how to design the experiment and what to do with it. Instead, if I say what is the performance overhead introduced by celium for two project, changing tcp messages, that is a much better question that can help the guide design. The experiment.

A

So one big part of benchmarking networks is the workload then specifically the network traffic, that we apply to do the experiment, and this includes different dimensions. For example, it includes the communication pattern. It includes the protocol, which can be either udp or tcp or even layer. Seven protocols such as http, but also what are the different endpoints that talk to each other, for example.

A

Is it portrait pod? Is it brought to host? Is it pod to service, or what have you and um another kind of common pitfall here? That is really important. To keep in mind. Is that while benchmarks with artificial workloads are definitely useful, and I'm going to talk a bit more about this sort of benchmarks in the subsequent slides, it's really important to keep in mind that they do not paint the full picture in terms of production workloads.

A

So in production things are much more complicated, and so, when doing this artificial workload benchmarks, it's really important to keep this in mind, and there is a very kind of delicate balance between kind of keep things simple so that you can make assessments, but also, you know not getting too carried away in terms of having of expectation of having expectations with respect to actual protection workloads.

A

So um I'll start talking a bit about some different communication patterns I'll to to make the com the examples a bit more concrete. I'll talk about multiple communication using tcp, but many of the patterns that I took about here can be expanded into other kind of protocols or other situations as well.

A

So probably the most popular benchmark is the one where one can find point. A pod in this case tries to push as much traffic as possible to another boat and in this case traffic goes over, goes over only one direction, at least the user traffic, because there are also acknowledge packets that go the other way. But this is not part of the user traffic, and so the metric for this benchmark is measuring.

A

How much uh bytes you can push on on the on a time of unit, for example bytes per second, and there are many optimizations that target this particular workload, such as segmentation offload the interrupt coalescing and also things like jumbo frames, and while this is one of the most common benchmarks, people tend to use it's useful to keep in mind that it might not be representative of the workload that your applications have and if your applications frequently exchange large files, then sure that's a very useful workload to keep in mind.

A

But if your applications exchange small messages, as I don't know, many microservice applications would do. Then this is probably not a good workload uh to kind of study or understand the performance of your network instead for applications that exchange messages.

A

This echo reply pattern is much better and in its simplest form, this is a pod, sending a message to another pod and then waiting for the reply and then sending another message and so on so forth, and we do this for a number of times and then we record the round trip latency of these requests and replies, and we can do meaningful statistics on them, such as the median latency or even the tail.

A

Latency, uh such as the 99.9, percentile latency, and one thing to keep in mind uh here in terms of optimizations, is that many of the optimizations that we talked on the previous stream benchmark.

A

Not only they do not help in this particular case, but they can also be detrimental so from a high level uh kind of point of view. The thing to keep in mind is that it's not a common, it's not uncommon for optimizations, for bandwidth and throughput to hurt performance for latency, critical workloads and vice versa. So optimizations are in many cases, depending on the type of workload that you're trying to uh to apply to your network and a generalization of the second reply.

A

Workload is instead of one to have multiple requests in flight, and this uh sometimes is referred as the best as the burst parameter and in this example I'm using five, and this means that that, at any point in time there are five either requests or replies in flight and using this variation.

A

We can now start thinking uh about throughput in terms of requests per. Second, in addition to the latency that we've discussed before and again other optimizations, now that this is a more kind of throughput oriented benchmark coming to play, such as, for example, polling or or other things as well.

A

These particular benchmarks allows you to do a very interesting in my opinion, graph, which is this one. The specific numbers don't really matter that much. This is from an experiment I did I think one year ago or so, or maybe less, but the basic idea here is that on the x-axis you have throughput and on the y-axis, you have latency in this case I'm using medium latency, but you can also do tail latency, for example, and what we do here is that we do different experiments for different values of burst.

A

In this particular example, I'm using burst, which bursts with increasing powers of two and the interesting part here, as we draw the the different points and doing the graph, is that there are different areas that have very different behavior.

A

So in the beginning, we can see, as we increase the burst size, throughput increases without affecting latency, but after a point which is around the 1.8 value, we see that adding more packets in flight, saturates, the system and throughput is not increased that much and latency basically uh rapidly increases.

A

So uh this is uh something interesting because it's really important to know which part of the curve uh your benchmark or your applications are, and what is the point where your system will saturate now so far, I've talked about basic kind of point-to-point communication patterns, but.

A

There are also a lot of reasons why you would want to scale your workloads. One specific reason is that mix are getting faster while cores do not, and this means that for.

A

Say fast network cards such as 100 gigabit, definite devices, a single tcp stream- that is a single core, cannot saturate the link, and this means that stressing the system requires exercising multiple cpus and multiple streams.

A

So in the graph, so here you can see that a single stream can achieve 20 gigabits per second and again, the specific numbers don't really matter so much different configurations or different hardware will produce different numbers, but the basic idea is that if you want to kind of stress the system, this requires using multiple cpus, and this can be done by using either multiple processes, multiple threads or even multiple pods, depending exactly on what you want to do.

A

In addition to that, you can also think about using multiple numbers for evaluating the full cluster rather than just uh two single nodes, and this makes the kind of experiment much more complicated because you have to track uh multiple things, but it's also uh really useful, because it's a much more realistic workload and you can start kind of evaluating things like quality of service and fairness, and things like that.

A

I've already talked a bit about metrics, but I've mostly mentioned basic performance metrics, so things like throughput, which are buckets bytes per second or packets per second and also latency. One thing I would like to note here is that tail latency, that is kind of worst-case scenario. Latency is really important as scale grows and there's an interesting paper that talks about this from google folk.

A

That is called the taylor scale, and you can look at that for more information in addition to basic performance metrics when doing an evaluation, we should also consider overhead metrics- and this includes things like cpu utilization or even memory utilization, and to give you an example of why this is useful. There might be a an optimization or a system, a different system that achieves one percent performance improvement at the cost of doubling the cpu utilization.

A

So if you consider the overhead, this is not necessarily a very attractive optimization, so it's also worth keeping in mind what the overhead of a certain approach is. Another set of metrics are metrics, which have to do with performance, interpretation or identifying botnets and to give some examples here. One thing that you can measure is how the network card interrupts are distributed into different cores, so that you make sure that the traffic is actually balanced.

A

There are also things like bpf flame graphs, which give you a really nice way of looking at where time is spent when kernel processes, packets and also there are many tools using bpf which can help kind of trying to figure out specific things, such as what is the size of the packet buffer processed by nick driver functions.

A

So, with respect to system configuration, the basic goal here is to try to control the environment and to give an example, if you want to do a protopod benchmark, you need to ensure that the pods are scheduled in different nodes, because otherwise it's a very different sort of situation and, generally speaking, the system configuration should try to isolate systems to avoid an unwanted interference.

A

So, for example, when we are benchmarking celium, we try to remove uh all all unwanted interaction, so we run bare metal. We disable preemption in the kernel. We disable core frequency scaling. We pin nic, interrupts specific cores and do a bunch of other things just to focus on the serium performance and not external factors. Now, obviously, the specific decisions might not be appropriate for the goal that you have said.

A

So this is a process where, given the specific goal that you have, you try to figure out what are the most important things that you need to focus on and try to configure the system so that you exclude all other things that might mess your experiment so when it comes to time to actually perform the experiment.

A

I would definitely advise you to in, in addition to the results record system, information- and this includes hardware and these configurations- and this, for example, might be cpus another cpus nics. What is the new configuration memory and things like that, but also the software uh parts as well, so for every software component? That feels it's, that you feel it's relevant.

A

You can record its version or its configuration, and this includes things like the kernel to the kubernetes version or the cni that you are using uh in addition to that running, just one experiment is typically a bad idea, because you don't really know whether the experiment that you did is actually a representative one. So you are strongly advised to increase your confidence in the results by performing multiple experiments.

A

I would argue at least three, if not uh even more, just to make sure that you know what you are seeing is actually something that happens commonly uh and to give you uh some sort of a war story uh on how we do network benchmarking in celium.

A

uh During the 1.9 release cycle, we did a very extensive performance evaluation and one of the things that we figured out was that native ray native excuse me native routing, as a opposed to tunneling had reduced performance, which was unexpected because tunneling, for example, has additional encapsulation.

A

So we would expect uh native native writing to perform at least as well as fundamental, and so the way that we tried to address this issue was to try to figure out whether the problem was in cilium or in something else, and what we did was we replicated the setup, but in this time, without selum.

A

So we did everything, but without the bpf datapath that serium introduces and did the same benchmark and it turned out that the performance problem was still there, which kind of led us to figure out that the issue was on the fifth uh virtual electronic device handling rather than ceiling itself, and once the issue was pinpointed.

A

Members of our team contributed modifications to the linux kernel that actually addressed this issue. So if you have a recent enough version of a kernel and recent enough version of psyllium, then this problem will not be there anymore.

A

So I'll. I want to also talk a bit about tooling. There are many tools that you can use for doing the benchmarking.

A

My personal preference is netperf, which is an old tool, and maybe not very user friendly by today's standards, but it's very powerful and it can implement all the different workload patterns I mentioned before, but also many more. uh We've also developed our own internal tool, which is called uh cubenet benz, which uh uses networth underneath.

A

But it deals with uh some of the things like, for example, uh spawning uh pods into a kubernetes cluster, ensuring that they run at different nodes and some automations, for example, such as gathering automatically gathering configuration data or even automatically gathering the road data for doing ppf flame graphs, and there is a wide variety of bpf tools that you can use to understand performance and I highly suggest looking at brendan gregg's resources for that either the book or the web pages are really great and they can help you really really use bpf to better understand performance.

A

Another utility that we frequently use is perf, which is very common and popular in kernel developers. It's also not very user friendly, but documentation has been improving over the last years.

A

Finally, some additional resources for people that want to dig deeper into performance validation. One of them is a book again by brendan about systems performance, and this discusses all different aspects of systems, not on networking.

A

Another book is a bit older, maybe not sorel advantages, but I also, but I still find it uh useful, at least in terms of the principles it discusses and that's the art of computer systems performance analysis.

A

And finally, if you want to understand how the internals of the kernel work either with respect to networking or something else, I highly recommend the kernel pages on lwn linux linux.

A

So, to conclude, I'd like to summarize with three of takeaway messages that I'm hoping uh people will consider from this presentation. One is design your benchmarks so that they answer a specific question. The other is start simple and iterate, and there is more to neutral performance on string benchmarks.

A

So I would like to thank my colleagues, daniel gilbertopol and the rest of the ceiling team, and also mentioned that you can find us on selum slide on the performance testing on the performance testing channel, where we can talk about kubernetes and google performance.

A

uh Thank you very much for your attention and I'll be online to take your questions.