Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 15 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sponsor Demo: HAProxy Technologies - Benchmarking 5 Popular Ingress Controllers

Description

Performance has never been more important than in a cloud-native world. Cloud-hosted resources cost money and a slow-loading application can cause a suboptimal ROI. Have you taken the time to tune your Kubernetes ingress controller and proxy? Many organizations don't until it's absolutely necessary, and most users will typically run a default, out-of-the-box configuration.

In this demo, we benchmark five popular ingress controllers and put them head-to-head against each other with their default configurations:

Envoy

HAProxy

NGINX

NGINX Inc.

Traefik

We measure their average requests per second, latency percentiles, and user-level CPU usage; We monitor for any errors that are produced in order to go beyond performance and test reliability. You can verify the results yourself by downloading the GitHub repository and reproducing our tests. Who will come out on top? Watch the demo to find out.

A

Hello, thanks for taking the time to watch the demo in the following, I'm going to run some out of the box benchmarks against several ingress controllers.

A

First, a little about me. My name is daniel corbett and I'm the director of product at ha proxy technologies, I'm a security nerd and a cheese enthusiast.

A

Benchmarking is a science in and of itself, but many users do not have the time required to perform head-to-head tests against the wide range of ingress controllers that are available.

A

Understanding how your angus controller and proxy performs is important, as poor performance can cause you to use excessive resources, which ultimately translates to lost money, especially in a cloud-native world.

A

It can also lead to a sub-optimal user experience for your end clients.

A

However, when it comes to tuning various pieces of the infrastructure, many users get lost in the abyss of information online or just do not even think it's something they need to do. This means that many users end up running out of the box configurations.

A

This demo is going to focus explicitly on those out of the box configurations to find out which ingress controller performs the best. In a new setup, I've included a link to the full architecture, setup guide and code to repre reproduce these benchmarks within the h, a proxy technologies github located on the url at the top of this slide.

A

For this demo, I've configured a six node kubernetes cluster running on an aws c5 x, large instance.

A

One of the nodes has been dedicated specifically to the ingress controllers. The other five are where we will run our traffic injectors.

A

I have five pods running a standard, echo application in which the ingress controllers will be monitoring for changes in routing traffic to in these tests. I'm testing envoy, with contour as the ingress controller h, a proxy two different nginx ingress controllers and traffic. For each of these, I will be running the latest versions. At the time of this benchmark, which are shown in the table, you may be wondering why I have two different nginx ingress controllers listed. This is because there are two completely different open source projects.

A

One is a kubernetes community driven project and the other is directly from nginx inc. The biggest difference is the kubernetes community. Driven project has a very complex configuration and harnesses lua for many of the dynamic capabilities, while the nginx inc, one is a bit more slimmed down and does not use lua from here on out I'll refer to the kubernetes driven project as nginx and the other as nginx inc.

A

For the actual tests, I will use the tool hey. I will run two different tests in the first test. I will maintain 250 concurrent workers from one traffic injecting pod.

A

The second test will run 50 concurrent workers from five traffic, injecting pods each test will run for a total of 360 seconds. During this time I will scale the pods up to 7 and back down to 5 with 30 second intervals in between scale outs.

A

I will do this three times in a row and then I will make several changes to the ingress controllers themselves, such as adding and removing course headers and adding and removing path rewrite rules. I will also sleep in between each change for 30 seconds.

A

At the end of the benchmarking, we will graph the average request per second, the 75th 95th and 99th latency percentiles and user level. Cpu usage we'll also graph. The number of http error codes that we receive.

A

Let's get the benchmark started.

A

I'll be running a shell script that will handle running the benchmark as well as collecting and graphing the data during the tests. I'll also keep a grafana dashboard open, showing the activity to each of the servers.

A

A

A

A

A

The first test has completed: let's examine the graph data.

A

If we look at the first chart, we're able to see that h, a proxy came out on top averaging approximately 26 000 requests per second with envoy in second place at just under sixteen thousand.

A

We can see that engine x came in last with around ten thousand connections per second.

A

If we look at the latency percentiles, we can see that h.a proxy had the lowest 75th 95th and 99th percentiles with envoy and traffic neck and neck for second place.

A

Nginx inc came in at a close third, and engine x came in last with a massive spike for the 99th percentile.

A

Metrics, looking over at user space, cpu usage, we're able to see that h, a proxy maxed out roughly around 40 percent cpu nginx inc came in close close second at 47 and engine x came in third with 63 percent.

A

Both envoy and traffic came in at around 70.

A

We also have a graph showing us how many http errors were produced by each proxy.

A

It appears that neither h a proxy nor nginx inc produced any http errors, but traffic produced a little over 300 502 errors, nginx produced 28, 502 errors and envoy produced 23 503 errors.

A

Now, let's go ahead and start our second.

A

Test this time, we'll run 50 concurrent workers from five different traffic, injecting pods and let it run for another 360 seconds.

A

A

A

Okay, so the second test has completed: let's take a look at the results.

A

So this time we can see the average request for h a proxy is about. Is that thirty seven thousand, which is about ten thousand higher than the last test um and uh envoy again, is? Is that second, with roughly around seventeen thousand requests per second nginx inc came in at third, with around fifteen thousand requests per second, looking at the latency percentile graphs, we see that h, a proxy is again the lowest across the board for the 75th, 95th and 99th percentile envoy comes in second and nginx.

A

Inc and traffic are neck and neck for third, however, you'll note there are some really drastic spikes in the 95th and 99th percentile for.

A

A

In the user space cpu usage graphs, we see that nginx, inc and ha proxy are neck and neck with just under 50 cpu envoy came in around 72 and traffic came in at 77.

A

What's important to note is the significant difference in the request rate of h a proxy compared to the others, and yet it was able to maintain a consistently low cpu usage. Ultimately, resources cost money. So if you have an ingress controller and proxy that are able to handle more than double of the closest alternative and do so with 30 to 40 percent, cpu you're definitely going to save money.

A

Looking down at the error rate.

A

Looking down at the error rate, we could see that neither h a proxy nor nginx per inc produced any http errors. Traffic produced approximately 630, 502 errors, engine x, produced 27, 502 errors and 504 errors and envoy produced 14 503 errors.

A

That concludes the demo. Aj proxy has a reputation for being the world's fastest and most widely used software load bouncer in the world for a reason, give it a try today and you'll be certain that it's out of the box performance will supercharge your kubernetes environment thanks for watching.