Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Comprehensive Performance Benchmark on Various Well-Known CNI Plugins - Giri Kuncoro & Vijay Dhama

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Comprehensive Performance Benchmark on Various Well-Known CNI Plugins - Giri Kuncoro & Vijay Dhama, GO-JEK

This session provides comprehensive performance benchmark on various well-known Container Network Interface (CNI) Plugins. Over the past years, there have been plenty of CNI Plugins implemented within the community which comply with CNI specification, enabling standard communication between container runtimes and network resources. However, there appears to be lack of initiative to thoroughly comparing and understanding the performance of each CNI plugin, particularly in a container orchestration system. This research is motivated by a need to identify characteristics of a CNI plugin that might impact container network performance. The studies could inspire further improvement of a particular CNI plugin, and provide insights into it being more accurately selected that adhere with particular system.

To learn more click here: https://sched.co/FuKF

A

To speak out coupon, it's our first cube Khan, and this is our first time in China and we are really excited to be here.

A

Basically, we have been here for last few days, and it's been amazing and obviously not being on call is amazing and it's been a while, though my name is Vijay dama. That means my colleague, kiri Canora, we're.

B

Both engineers from go-jek ride hailing app from Indonesia today, we're gonna be sharing our x-men results on benchmarking, various container network plugins. Here's the agenda for today we'll start off with the overview of various ENF plugins. Just to get you guys a line and we'll be sharing what kind of motivated us on doing these experimentations and what kind of environments we conducted the experiments on and what are the results that we got from these experiments then? Finally, what are the key takeaways that we can learn from our experiments?

B

Vijay will start with brief overview of various E&I plugins that we used to conduct the best. Ok.

A

So a brief overview of humanity's network model before we start with CNI, it's just a humanities network model is meant to be quite simple: every container should be able to talk to every other container without being mounted or traffic bridge through, and all nodes should be able to talk to each every other node without being added, and basically whatever the container IP. The other contingencies is what is the job? What is the continuous IP, so the implementation itself of networking is not done in communities.

A

It just provides an interface through CNI for other plugins to implement and plug in into your humanities, implementation or deployment.

A

Next is just a brief overview of what CNI does CNI two eyes: two things: it provides connectivity and reachability, so connectivity as simple as just provisions or the network interfaces of the continued or pods and second part is which ability which is done by the demon that is basically the cni plugin. It provides you with a way of reaching from one part to another pod or basically, one part to a service. All there are lots of CNI plugins around.

A

This is a link down below where you can find the number of CNI plugins that are actively maintained. There are actually a few moon much more than that, and but these are the listed in our plugins that you can take a look at the cni plugins that we are going to talk about our flannel calico weave psyllium. He brow turd amazon's, here, amazon skate of the cni papaya and romana, starting with the beef production.

A

These are CNI plugins, so I'm, going to just run through these flannel is just a simple as three based on a netbook fabric, which provides a bunch of support for actually support for a bunch of patterns, including VX LAN, which is a default in flannel, and it also suppose other things like you can set up a WSB PC netbook fabric with in a flannel and second, his calico Coleco is a basically three approached for and using you know, it also provides a be a default support for a BGP, and it also provides support needed support for network policies.

A

Not all cni plugin do next is: we've is a product from V box. It is saying I plug-in that provide that works on nowadays works on who is module and which is called something called faster path, and that's provides support for encryption and network policy. Next is helium. Helium is a PPF based CNI plug-in.

A

It has extensive support for network policies, even greater than calico and Romana cube. Router is a simple ipbs based and I plug-in. It does simple networking. It doesn't provide support for overlay network. Then it does cost based routing AWC ni sa' provide support for basically configuring your cuban. It is now clustered within the red grace v pc in that book fabric. So you can provision your given.

A

It is cluster to have IBS from the BBC itself and just directly have a custom, both load running where and you then talk from a vm to directly to a pod without going through service and hot papaya was another excellent based, CNI plugin. Suppose L do with GRE an IPSec for encryption.

A

The last one is l3 basically romana, as it is also a l3 based sinaia plugin, which works with which was PGP or OSP. Next, we're going to go through experiments which we will go through.

B

Let's now see kind of experiments that we did. We started with curiosity on each of this plugin because there's just so many and we started to play around with each of those and conducted some performance study, some simple football study on each of this. When we were conducting this experiment, we had four goals in mind that motivated us in doing this experiment.

B

First, we wanted to see which Siena plug-in that offers the lowest latency and which one has the highest throughput as a typical performance network benchmark started, and then we wanted to evaluate CNF plugins performance with different protocols that were protocols in this case TCP and UDP, as well as how do they perform with various packet sizes? The third one we wanted to see each CNM plug-in consume. How much do they consume and how much time do they take the provision container network, for example?

B

And then lastly, we wanted to evaluate whether there is an impact of communities, Network policies in network performance. These are the environment that we did and conducting the experiment. We spawn eight kubernetes clusters with different CNM plugins. We set this up with a minimal. No sighs, it was two node cluster, the master node we painted so that we could spin up workloads in it. The size is m4x large in Amazon AWS. We chose area because because we cannot use Google Cloud here in.

A

B

You know that we use Debian nine, destroy and kernel 4.9. We use cops to help us spin out the clusters with version 1.9, but when we were doing the experimentations, these are the version of CNI plugins that we leverage to conduct this experiment.

B

An important note here is we threaded each of the Siena plug-in as a black box. So we didn't do any specific tuning or custom configuration with each of the Siena plugins. So we were just using the default configuration that they provided as documented in the communities IO Docs, as well as I, think included in cops.

B

We use three pools to conduct these experiments. First, we were using soft perf to measure throughput and latency, and then we use net perf, particularly there a specific feature called UDP RR to help us measure the CPU utilization that easy and Ciena plugin consume. And lastly, we were using a simple gang based tool implemented by our friends and panel networks that help us in simulating SD workloads and measure response with various network policies. We then conducted five different experiments in eight kubernetes clusters.

B

Our first experiment is on throughput and latency, which is pretty basic. In this case we were running software client pod in note a the first node in our company's cluster and software server, part in the second node or node. In this case, we then configured the packet size to be fixed, 256 bytes for TCP throughput test, as well as 16 bytes for the disabilities in CFS. We we use host network without any C and a plugin to be our baseline, as can be depicted by this picture. This is our dispute result or graph.

B

The red graph, the red bar on the left side, is our baseline, which is also depicted by the line graph in a nice pink color as our baseline. As you can see here, all obviously any plugins suffered dissipate throughput loss, obviously because of their their mechanism right. Some of the plugins do a packet encapsulation encapsulation or over your networking, some does routings, etc.

B

Apart apply for flannel, we've got our Romana. They suffer speed, throughput a bit less around 15 to 20 percent here for overlay networks. Network performance hits are understandable because they need to traverse the network stack back and forth right to do the packet processing. For instance, the packet decapsulation gets invoked at host operating system whenever host OS meets the encapsulated packet at l4, for example, and then the packet will get transmitted to the port through internet interface, which will then traverse the network stack again for weave.

B

The result here is is very interesting because it is pretty close to the baseline I think the closest to the baseline we've came up with I think an interesting feature called fast data path around last year, which is essentially using the data path module of OBS in their implementation, so that they could decrease the number of network. Traversal apartment decrease the the the traversing between user space and kernel space. Basically, this was also documented. In there we've documentation they could improve the result of their performance around three times compared to their old implementation.

B

For this for calico psyllium, coupe router and over CNI. Unfortunately, they suffer significant loss. In our study. We were actually expecting a much better result by calico. However, apparently in the in the default conflict that they have in calico, they still configured IP and IP capsulation as default. So that's why we can see the result here for calico.

B

Here's the result for our tcp latency study. Overall, most of the plugins added around 25% on average latency additional latency here for a diversity on a and Roman. They they suffer latency the lease additional latency the lease, because both NSA Romanic pops to a SS, V PC fabric on the network pack and that's how the packets are relatively faster to travel compared to the others in it. Plugins for cope I owe the the let's see is is doubled at least compared to the baseline.

B

We haven't conducted deeper analysis yet so, if you want to help us, please come meet us. Our second experiment is on different protocols and different packet sizes. So it is important to pick appropriate network protocols to do to get a desired. For example, quality of service in our system and the packet size here plays important role in in a network performance because, for example, given sustained data rate, a larger packet size will give less number of packet being transmitted and, on the other hand, a fix packet being transmitted for large package size.

B

It will give us a larger data rate. We did a similar experiment, what we did in the first experiment. We ran software, fine pot in the first node and then software server apart in the in the second node. However, the difference here is we measure both for TCP and UDP throughput, and then we tweak increase the packet size from 16 to 256 bytes.

B

Here's the result of our TCP throughput study. The x-axis is actually the different packet size from 1650 256. As you can see here, only half of the CNF plugins code, skill as the packet size increase in particular final we've, copia Romina, could scale that CIPA throughput well pretty close to the baseline as the packet size increase. Here.

B

However, for this four we could see here they suffer considerable loss, especially when we increase the packet size from 125 128 to 2050 to 6. The throughput level is quite similar here for UDP proof put for obvious reason in general, including the baseline. They provide less UDP throughput because of obvious reason. In TCP we could actually make the physical link more efficient right by design. It's like that there mechanism.

B

However, the difference here is all the C&I plugins could actually skill given a different packet sizes. Although the the throughput loss is considerable, but the point is, is an applicant could scale as the packet size increase being a SS enin Romana suffer the lease throughput loss for UDP, actually.

B

Our third experiment is on on the CP overhead that an applicant introduced to kubernetes. We were using an app a pretty famous tool for performance benchmark. We particularly use their unique feature called UDP RR, which is utilizing UDP to measure sip utilization of each of this plugin. By using this UDP RR, we could measure the time spent in user space kernel space and waiting for IO.

B

Here's the graph on the two graphs on the left side are actually CPU utilization and two graphs on the right are CPU time or service demand. The service demand means a time that is required to complete one transaction of UDP.

B

With this we could actually show the cost of Siena plug-in from the request, ID or client side, as you can see here, most of the plugins. In fact, all of the plugins introduce CPU overhead, but it actually very very from the least, is around 10% to 40%, because they need to obviously see and a plugin. They need to allocate some IP from IP pools. They need to do in cap the cap. They need to talk to a WSB PC API, for instance, for AWS CNI and romana etc.

B

However, an interesting result here is: is this Siena plugins, which is viven cilium they for for weave? The weave adds around double the CPU overhead compared to the baseline and salome at least triple the CPU utilization here for weave. We think, because, because of the obvious, probably and their software routing, because they do the they do routings, they do pairings and they could save network partition and cluster and things like that, most probably add additional computation and then for cilium. They leverage the BPF module in colonel every time.

B

They need to generate the bite right and then inject it to the to the packet ploy. So we thought that add, except the cpu more. If you have a better analysis, please come to me the first experiment. We were looking into. How much time does a CNF plug-in use took to provision and network whenever a new pod being created so network launch time is, is very important in case you guys have a short, live or latency sensitive workload in your kubernetes cluster right.

B

So we measure the time when cubelet, when cubelet issues container create command or pod get command. In this case and in the example cubelet is trying to spin up nginx pod on the left and then until the engine next pod is responding to pinging requests from the pod. On the note B we stop the stopwatch, so it is a time from container create command until it responds to the pinging request.

B

We understand there might be a an overhead here, because cubelet is appalling events to the cube api server. Here, however, however, we didn't figure out a better way to measure this, and I don't think this will cause issue, because the comparison between CA plugins are Apple to Apple. Here's the result.

B

For some reason, flannel took more than 10 seconds to to spin up the network on a pod, so in order to make graph prettier I'm gonna take this out. Most of the plugins add additional network lunch time for this for CNI plugins at around 0.5 to 1 second additional network lunch time whenever cubelet creates a pod.

B

Keep that in mind that network launch time doesn't include the image being download it, because the cubelet state comes after that the cubelet Connecticut it comes after the image download. Also for this three Salem coupe, prouder and caballo. They add additional one to one and a half. Second,.

B

Our last experiment is on whether there's an impact of network policies to the network performance in our reference in a typical performance benchmark, there's no application logic running in the host level, so this leaves the host CPU to be available to process, whatever networking required in the host level. So with this we make sure to run Network policy test that doesn't saturate the network as well as separate the CPU.

B

This has impact of isolating the the network policy process by the CPU at first level. So we are borrowing a pool from our friends and panel network. We run the pool the client pod in not a in server body, not B.

B

We then configured the client part to send a fixed rate of 2000 SP requests with rate that we ensure do not saturate the network as well as CPU, and then the server we configure it with various response eyes from one kilobyte, 100 kilobyte, and then we make sure each request is- is perform in a new to speak connection by disabling the persistent connection. In case you guys are using nginx. This could be achieved by disabling the keepalive, a configuration for example, and then we repeat this best for 0 to 200 Network policies.

B

What this means is for 0 network policy. We do not run any network policy, obviously, but for, for instance, I'm giving an example of 200 policies. In this case, we configure the first policy as default, deny the we'll disable all traffic coming into the server and then 198.

B

We generated a policy, a random policy, for instance, in this case, allowing us to port a allowing us to port b, a b c d, random port which do not match the the port that the server is listening to then the last the last policy we configured with port 80, which will allow packet to coming in and matches the the port. Basically, in this case, we could simulate that the that the the traffic, the all the policies are being hit before when we were doing this test.

B

Here's the result for calico network policy, there's no meaningful delay at all being introduced in the response time at max around 0.1 millisecond. We could not this very slightly for 100 kilobyte response. There, when we try to increase the number of policies from 50 to 100 and then for we've, we've also provides us with similar result, which is a non significant additional response. Time here for cilium also similar result, probably around 0.2 millisecond for the blue.

B

One, however, keep in mind that the network policy that we tested with cilium is actually the native network policy feature that offered by native kubernetes cilium, actually provides a much better way by providing us a custom resource that they have called a cilium network policy that we didn't pass here for that custom resource they actually offer extended.

B

What native network policy can offer? The native network policy can offer a policy in l3 and l4 for complete ingress and limited egress, but for cilium network policy they could offer l7 as well with complete much better egress feature for coop router again similar result a little bit higher, but still not meaningful. They add additional 0.4 millisecond additional response time for the the green one, which is 100 kilobyte response.

B

We do not. We did not pass the other Sienna plugins, because, obviously not all Sienna plugins offer this network policy feature forum Romana they actually provides, but because of an open issue in github, we could not play around with the network policy when we set up with hops, because I think the latest cops release somehow changed. The default. Ip tables configuration that cause some issue with with Romana and effort policy there Vijay is going to share what are the takeaways that we can learn from this.

A

Ok, so forth, echoey is our starting with the CNI itself in general, very tricky to pick appropriate CNI plugin engine, because it totally depends on the use case and where you are running that particular given it his workload and the Kuban it is cluster.

A

There are a lot of trade-offs to be taking to be taken into account, including performance, security and isolation.

A

So if you need network policy other than you are going to look into something like calico or Celia, and perhaps Romana and terms of what works better on throughput, the default config flannel beavin Romana achieved much better network performance than the others in terms of the larger throughput of obviously, DCP has much larger throughput, then the regular UDP workloads, which is usually not much for most of you, perhaps most CNI, plugins scaled, better and UDP with about 30 or takeaway here is you definitely have some considerable amount of loss because it's.

A

Mostly and I plugins introduced some CP overhead of the vivre and see Liam being the largest we've as Pickers. We think it's because in V it uses the previous module of which is probably the cost for that CP overhead in see Liam, it's I think the beep-beep underlying BPF what you are currently Maudie. That is the reason for that CP overhead, but that's just a assumption right so also in terms of network launch delay.

A

There is a beast: nipple network launched early to launch containers, except for flannel, which I think in this particular test case was a flag and wasn't totally expected for network policies. There is a non significant delay and in the no and the basically hilly that isn't used app on the number of network policies you had yeah, so that is mostly take away, is around the policies for the experiment itself.

A

We need a more comprehensive analysis which, which will basically prove or add more info to what can be done to improve particulars in AI and all and have a bit much better picture of differences between inner plugins. For example, we can configure of some PC and I plugins to work much better on bare metal and bare metal workloads. Then we can do on on cloud providers.

A

For example calico you can disable I can IP mode to just use BGP and which be think we'll offer much better performance, another experiments way of thinking of as doing concurrent requests and testing. But you know all this.

A

You know plugins against those and see what is the performance and throughput and latency responses from TNI plugins and another testers workload, interference wherein if you already have some workload running on your Cuban ADIZ cluster, you can run these tests to figure out how does CNI plugin get affected or due to existing workload, and all we had a lot of references for preparation of this talk, three of them.

A

Three of them include- and this is an empirical study of computer networks, comparison of our networking solutions of our communities from machine easel and high performance network policies in Cuba, Nettie's fluster from folks fire networks.

A

We had few help from people as well during a J gather, know early and summer.

A

Thank you. So this is our contact details or is reach out to us. If you have any details to add to what we can do to improve the.

B

If there's any questions or we can share after this.

B

So your question is: how did weaken that Network lunch time right? How did you mission so, as you can see here from this diagram, we actually have a stopwatch, I mean stopwatch and then, when you deploy an pod, typical kubernetes thing rakubian, this scheduler will actually an update that somewhere in the impurity state and then couplet will watch that then say that oh, I need to spawn the pod. That is not running yet, in my note.

B

So when cubelet has its own state state ABCD, one of them is including container schedule and then container should be download. Don't need it then. After this, when everything is ready, there's a state in cubelet that says container CLE when Kotak when cubelet reaches this container create state, and then we start the stopwatch.

B

We start the timer and then the pod, the docker images already access straight after that, because sorry before that, then pod will run and then, when the pod is running, CNI will provision, whatever network required right, assign IPS, for instance, routings talking to at CD, for instance, there's a bunch of mechanisms there, depending on CNN plugin and then at some point the pod will accept connect TV, whether the two things that Sienna provide in the beginning, reach ability and connectivity from the second node.

B

We just spawn a ping request, then, when this spot starts receiving the ping request, we stop the timer so both those time, but only not no service here pod on me does that make sense anything else. I.

B

Sorry, can you repeat.

B

Packet loss right so he's asking whether there's a packet loss when we were doing this test. What is the answer.

A

For packet loss bidding actually identify any packet loss for UDP experiment. We did actually get a lot of packet locks losses which is acceptable because, but we per se for experiments.

B

Because the packet size is also pretty low, which is 256, probably depending on the type size depending on the instance type. When we increase to 500 or 1000, we will see that, but if we didn't do it, unfortunately.

B

Okay, I think the time is up. Thank you so much for attending my fault.