VMware Contour Demos and Deep Dives, 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Performance Testing Ingress for Internet-Scale Workloads - Alexander Brand, Heptio

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Performance Testing Ingress for Internet-Scale Workloads - Alexander Brand, Heptio

Have you ever wondered how much ingress traffic a Kubernetes cluster could handle? How many nodes would it take to handle the traffic of an Alexa top-40 website? Understanding these numbers and how your ingress infrastructure scales is critical when it comes to deploying internet-accessible applications in production.    At Heptio, we needed to prove that our Envoy-based ingress projects, Contour and Gimbal, would scale to support millions of concurrent connections, thousands of backend services, and thousands of virtual hosts.    In this talk, we will explore the strategies and tools we used, the challenges we faced and the lessons we learned while running these tests. We will dive into kernel tuning, HTTP benchmarking, Envoy metrics, and more. We hope that talking about our experience will help when it comes to performance testing your cloud-native applications and infrastructure.

To learn more: https://sched.co/HChv

A

Thank you for joining me here. I'm Alex brand I work at hefty Oh, which I guess now is VMware, so super excited about that. This session is about performance testing, ingress for internet scale workloads.

A

So when I say ingress, I'm talking about kubernetes ingress. So for those of you that are not necessarily familiar with kubernetes there's a resource in kubernetes called ingress, very similar to how you have a deployment or a pod or a service. There is a resource called ingress which allows you to expose services that live inside of the cluster to the outside of the cluster.

A

So organizations typically have a portfolio of apps that they want to expose outside of the cluster and, if you're, looking for a layer, 7 load, balancing solution, ingress and ingress controllers are typically what you're. Looking for adeptia, we have been working on a couple of projects. One of them is contour, so contour is our ingress controller for kubernetes. It is built on top of envoy the proxy that was built by lyft, so we were very happy with with with envoy and the Android community. So contour is responsible for ingress.

A

Inside of a single cluster, but what we, what we've been seeing is that the majority of our customers are actually running multiple clusters and that's exactly where gimble comes in. So gimble is a multi cluster, ingress controller, a load balancer that we have built.

A

So why did we create gimble? So, as I mentioned, the majority of our customers are running multiple clusters.

A

They want to load balanced across a variety of clusters in their data centers, but more specifically, we actually built gimble in conjunction with or in a partnership with Octavio, which is a subsidiary of Yahoo Japan here in the United States, and the reason for this is that they actually reached out to us- and they have this problem, where they're running hundreds of services that they expose over the internet, that they've historically been running on OpenStack, but been slowly migrating them over to kubernetes.

A

So they've had this problem where they want to merge this two worlds of VMs and containers through like a single layer of load, balancing and that's exactly where like gimble comes in, so you can not only route traffic to multiple kubernetes clusters, but you can also route traffic to OpenStack clusters, which is really cool. So if we look at gimble from like a picture perspective, you can see here where we have the Internet. The traffic comes in through gimble and then gimble sends the traffic to the right cluster.

A

If we zoom in a little bit, we can see the gimbal architecture so in the bottom you can see the clusters where we're sending traffic to, and then we see this box that is gimble. We actually use kubernetes as a building block to build gimble on top, so we use kubernetes as a platform to build a system which is really cool on the data plane which, as I mentioned before we are using envoy and then from the gimble perspective, we have two main components or subsystems.

A

We have a service discovery, subsystem which is responsible for discovering services in the underlying kubernetes and OpenStack clusters, and then we have a routing and rule engine which is really contoured. So contour is that ingress controller which gimbal builds on top of and then also we not really. We wanted to build something that was easy to operate and monitor. So we also have out-of-the-box, monitoring and alerting solutions built on top of graph on ax and Prometheus, so we are in the performance track slide.

A

You know we gave a bit of an overview of what gimble is, but let's talk about its performance and and and how gimbal behaves on their load.

A

So, while, while working this project, we had some very interesting requirements, so millions of concurrent connections, thousands of services per data center, tens of thousands of endpoints per data center, and all of this we wanted to achieve while maintaining the latency, the p99 latency under 30 milliseconds.

A

So when we started thinking about this and we when we started thinking about how to test this system, we started realizing that there is a lot of variables that you can play with. So you know there a number of requests per second number of concurrent connections. What is the response? Payload size, how many services, how many endpoints so there's a lot of variables- and we have to try to you- know- boil down the problem a little bit, there's not only multiple variables but there's also multiple subsystems.

A

So, as we saw in that diagram, you had the data plane, the control plane, the monitoring subsystem, the discovery subsystem. So there's a lot of things that we had to test and make sure that we're behaving properly.

A

So when we started thinking about this, we initially went down the path of what we called micro benchmarks. So we started testing the system through very specific, looking at very specific variables. So, for example, what happens when I increase the number of concurrent connections on the system? Is the latency going to go down? Is it going to go up? It's gonna stay the same, so we were trying to solve and understand how that was gonna impact the system.

A

Once we started going down the path of more micro benchmarks, we started realizing that we also wanted to understand how the system behaves in a more realistic scenario, in a more production like scenario and that's what we kind of called my macro benchmarks. So you want to understand how the system behaves when you're throwing realistic load at it. So in this example, we wanted to see you know what happens when I send or I open a hundred thousand concurrent connections, and we send thirty thirty thousand requests per second, and what does the performance look like?

A

So, if you look more closely at micro benchmarks, an example that we kind of talked about already, this I grabbed from we had a huge spreadsheet that, where we listed all the tests that we were running, so this is an example of one of one of them. What happens when I increase the number of concurrent connections on the latency? So what is the effect on the latency, so we defined all the tests. We define the test which variable was under test, what were test cases and what was the expectation that we were looking for?

A

So some of the pros for mock micro benchmarks, they very quickly helped us identify bottlenecks and bugs in contour and some of the components in the discovery subsystem. So some examples here we ran into one where contour would actually panic when we were trying to create a lot of ingress routes.

A

So ingress routes is a CRD that we've built and when we started running these tests, contour would actually panic and like break down, and the reason for this is we had a race condition that would only happen at high load, so that was I was cool to find another issue that we initially thought was just a logging issue. It actually turned out to be much more interesting, so you can see here I'm. Hopefully you can see the graphs but they're like going up and up and up and up which you don't typically want.

A

You don't typically like to see that. So in this case we are looking at Q here. So we use a Q internally in the discovery system, where we queue all the services that we've discovered to be then sent over to the gimbal cluster, and in this case, what had happened is that client go, which is the library that we use to talk with? Kubernetes, actually has a built in raid limiter.

A

That has a very small default, so I think it's five requests per second, so we have to actually increase that limit and actually make that configurable so that the users of gimble could actually configure that, and you know, use it as as they really want it and get more performance out of it.

A

So helped us identify bottlenecks and then once we did that it also gave us. You know confidence that contour itself could actually handle a large number of all these objects. So we actually inside of contour. We build a graph that keeps track of all the objects and we wanted to make sure that contour could actually handle a large number of these and micro benchmarks.

A

We're really helpful there and then finally, micro benchmarks are much more easier than macro benchmarks to run, and so, if your 10 constrains, you know, if you're under any time like it's just very much faster to uh to set up than the macro benchmarks.

A

But of course the big con is that you're evaluating the system through a very narrow lens, so you're, looking at a very specific variable that might not really reflect real-world usage so again doesn't really reflect the reality of how the system is going to behave at scale and that's exactly where something like a macro benchmark comes in so macro benchmarks.

A

You know they test the system on their realistic load, they measure and evaluate multiple metrics and then, while you're running these tests, you can start seeing where the bottlenecks are and it'll give you an idea of what you need to do to actually scale the system. So perhaps you want to increase memory or you want to increase CPU or perhaps you're, hitting network limitations. So running macro benchmarks really gave us the this sense of where the bottlenecks were in this system.

A

One thing with macro benchmarks, though, is that, given that you're doing a lot more load- or you know a lot, many more connections or requests per second, you might actually have to scale your tests down to something that's more realistic. So we were a little limited with hardware. So what we ended up doing is we actually ran tests at three different scales and we wanted to ensure that the resource utilization scaled as we expected either linearly or sub linearly according to our expectations.

A

So you can see here and instead of running, you know millions of concurrent connections. We have to scale that down to three different scales and run tests across those.

A

So from an infrastructure perspective, we used a kubernetes cluster where we split up the nodes in a way that each node had a very specific role. So at the top we have a set of load generating nodes that would actually generate load against the gimble gimble processes.

A

Specifically envoy envoy was running on the gimble nodes which were labeled and then finally, we had a couple of back-end clusters where we were actually sending traffic to from a hardware perspective. All of these we're bare metal machines, so there were 56 cores, 264 gigs of ram and they were all connected with a 10g network.

A

From a tooling perspective from the load generating side, we use a couple of tools, work too and tcp kali and hope. I'm pronouncing that right then, on the gimble node, we talked a little bit about this there's envoy, contour the discovery system and then on the back end. We actually used nginx. So the typical example up and then you'll notice that Prometheus and Griffin is actually all over the place. Here. We really needed Prometheus Ravana to understand how the system was behaving while we were testing so super key.

A

So if we died a bit if we go a bit deeper with some of these tools work, if you're unfamiliar woodwork, it's an HTTP level, benchmarking tool that can actually give you some accurate, latency measurements. So you can run you can run work against your service. I run it for a specified period of time and it'll. Give you some information about how that service is behaving. It can actually generate some significant load.

A

It takes advantage of all the cores in the machine, and that was one of the things that we really were looking for when it came to deploying work on the cluster, we used a kubernetes job, so the job resource was actually very helpful helpful for some of these tests, and actually we can. We can see what that looks like so, if you're familiar with kubernetes. This is your typical kubernetes resource definition.

A

You have your API version, your kind, so in this case we're looking at a job, and the first thing I want to highlight is a generate name feature which I found is not very known and is not known very widely, so generating name will actually allow you to specify a prefix and let the API server generate a random name for your resource. This was super helpful for it for us, because, instead of having to you, know, create the job delete it created again deleted.

A

It was very easy for us to just submit a bunch of jobs and then the API server would be able to would generate the name for us. The next thing I want to highlight is pod anti affinity, so, as I mentioned before, work to would actually consume all the CPUs on the machines. So if two work two pods would land on the same machine, they would actually contend for the CPU resources and that's something that we didn't want to happen. So we used anti affinity to actually tell the scheduler hey.

A

You know, spread these pods onto different nodes on the cluster.

A

We also use the nib containers, and the reason for this is that we need it to tweak the kernel, so some kernel settings for specific workloads in this case work too. We needed to tweak some of those settings to be able to, for example, open the number of connections that we wanted to open. So if you want to open a lot of connections, you actually need a lot of ephemeral ports on your machines and the default range is very small, so we increase that range.

A

So in this example, it's from 1024 to 65,000, so we used an init container here to actually increase that port range and then finally, this is what work to command looks like. So you give it the duration of your test, the number of connections that you want to open, the request rate that you want to send against your service and then you know, where is the service? So in this case we are using the internal envoy and service inside of the gimble contour namespace.

A

The next tool that we used, which came along a little bit later, is TCP Cali. The reason for TCP Cali is that we found that work. 2 was not very good at opening a large number of connections. So, given that we wanted to open, you know 300,000 connections, we had to bring another tool in so TCP Kali is a TCP load generator and we found that it was much better at opening a lot of connections when it came to deploying this. It was very similar to work too.

A

So if we go here, we can see that you know it's the same job definition we use the job as well. What's interesting here is that we actually used completions and parallelism to be able to run multiple jobs at the same time and that way increase the number of connections that we were opening.

A

So, in this case, we were running six of these workloads at the same time- and this is what the command line looks like for TCP Cali, you give it the number of connections that you want to open the duration of how long you want to keep those open. And then what is the connection rate? So how many connections do you want to make for a second and then, as I mentioned before, we used nginx as the target service.

A

We just deployed it as a kubernetes deployment and we actually ran tests against two variations of of nginx, the classic or the vanilla that you can get out of docker hub, which responds with about 600 bytes of data, and then we also wanted to use a custom nginx that would reply with 22 kilobytes of data, which is the average payload size that Yahoo Japan was was experiencing when it came to testing against nginx.

A

We found that the default config that it comes with wasn't really enough for the tests that we were doing so we actually had to customize and tweak some of the configuration of nginx.

A

So here is a config map of the nginx config that we used and there's a couple of things I want to highlight here. So in line 5, we are telling nginx to fork as many worker processes as there are CPUs on the machine. So the way nginx works is there's a master process and then it Forks a bunch of workers that actually accepts all the requests or the connections. So we wanted to make sure that nginx was utilizing all the CPUs available on the machines.

A

The next thing on line 6 is the number of open file descriptors. So when you're accepting a lot of connections, you actually are consuming file descriptors. So we had to increase that as well. Then a couple other things line 14 how many actual connections each worker could open or could accept. So we had to increase that to around 65,000. Another thing that it's not really visible here is that we actually disabled the access log, so we found that the access log would actually slow down the nginx process, so we disabled the access log in nginx.

A

So, let's talk a little bit about results. We are going to look at four charts or graphs that they all look at the impact of concurrency and requests per second on specific variables. So in this case we are looking at the impact of concurrency and RPS on latency, and we can see here that during our tests, all the latency measurements that we got were under 15 milliseconds, so very much under our target anywhere between 5 and 12. Milliseconds CPU is where it starts to get interesting.

A

The more requests per second that we were sending the more CPU that Envoy was consuming. So you can see here that you know when we started at the blue line, which is 10. You know red, which is 20,000 requests per second and then all the way up to 30,000 requests per second, the CPU you know envoy was consuming more CPU, which kind of makes sense the more requests you're sending the more churned. The more processing that envoy has to do.

A

Impact on memory was also interesting. The more connections we were opening the more memory envoy was consuming, and this also kind of makes sense. You are keeping track of more connections, the more memory that you're going to use. So we can see here that the memory went up all the way up to around eight hundred and twenty megabytes and then finally, the impact on the network.

A

So how did the network ingress or sorry egress behaved here, and we can see that it looks very similar to the CPU graph, where the more requests per second, we were making the more bandwidth we were consuming and again this makes sense the more requests the more data is flowing through the pipes. So this was good to see as well.

A

So this is a summary of all the the graphs we saw, and one of the conclusions that we were able to make here is that a single Envoy, pod or machine or machine running envoy could actually handle a lot of load and if they wanted to go up to you know three million concurrent connections, they could actually just run ten envoy pods, which was which was interesting. So that's what it was great to see that the system actually just scales.

A

You know in a linear fashion, so that was that was cool to get some of that data, as I mentioned before we used uber, fauna and Prometheus, and this is an example of a dashboard that we built for our tests. So if we zoom in a little bit- and hopefully you can see it on the left- we have requests per second than connections per second and the latency, and then the total connections downstream and upstream, if we scroll down a little by the way, these are all coming out of envoy. So thank you, envoy.

A

Community super helpful, metrics out-of-the-box, so it was it was. It was great to have these metrics to understand the system. If we scroll down a little bit, we will see that we added a couple more things, so mainly how is Envoy using the CPU using memory and using the network. So we wanted to get on insta to get insights into that utilization. There.

A

So my main idea for this talk wasn't really about to talk about the numbers. It was more to share some of the lessons that we learned, so we it was, it was hard work. We made mistakes along the way. So hopefully you know talking about this is going to help you when it comes to doing your own testing with your own services.

A

So the first thing is document.

A

Everything I can't stress this enough, create a plan that just outlines how you're going to test what you're going to test part of that plan or part of that document should probably should be a like a results table or maybe have a results document where you already set up and know exactly which number is you're interested in, because, while you're running the test, we're going to get a lot of data- and you want to be consistent across test- runs so make sure that you have a table where you are actually jotting down.

A

Some of these results once you have that outline over that plan before running any tests document the environment so document, you know the number of nodes that you're running against the number of CPUs, that each node has how much memory the version of the software that you're running.

A

So all of these things are very important so that once you're analyzing the results you actually know you can compare and contrast across software versions or memory and stuff like that and then finally, I found it useful to just keep a journal or a scratch pad where I could just take notes when something weird happened. So I recommend doing that as well.

A

Observability very, very important, Promethean graph on a-- super helpful. The we were lucky that you know envoy has great metrics, contour and gimbal. They already had metrics. We were missing some that we added, so that was also cool, so yeah, it was again. Metrics are super key for these tests. We also found the node level metrics to be useful, so not only not only how you know, each of the components are behaving, but how are the nodes themselves behaving? That was also useful.

A

We used know the exporter for that and we found that creating test dashboards instead of you know, jumping across dashboards was very helpful as well, but you know in summary, just use and use some of these tools. You know, don't don't fly blind understand how your systems are behaving short tests can be deceptive. When we started doing these tests, we were actually running like 10.

A

Second, tests for 15 second tests, I think we might have gone up to 30 seconds and while we were running multiple tests, we were seeing that there was a lot of variability between them between the results and the reason for this is that it's just not enough time, so we moved over to longer tests or five-minute 10-minute tests, and you know that allows first of all your components to warm up and to actually be ready for the load that you're sending and then also in some networks, you're actually going to see jitter, and you know, traffic or yeah jitter, so with a longer measurement period.

A

You're actually minimizing the impact of that.

A

Next check the network, so someone can say this is a 10 G network, but effectively it might be much slower, so I'm, not necessarily much small slower. But when we started running these tests we just assumed that we had a 10 G network and some of the measurements that we were doing and the calculations that we were doing. The math just wasn't adding up, and one of the reasons was that it wasn't effectively a 10 G network. It was more like an 8 dot something so you have to keep that capacity in mind.

A

You don't want to overfill your pipes, because otherwise you're gonna start seeing weird weird stuff, so yeah measure your network. We used I, /, 3 4 for doing this, there's actually an open source project around running iperf 3 on kubernetes, which is just orchestrating everything which is really nice. So yeah check your network tweak the kernel. So we saw an example of this and the work job definition. The kernel can actually get in your way, so make sure that you check the logs check the system logs check the kernel logs.

A

Some of it might end up there. Some of it might not I found a couple of things that did end up there, but sometimes you just have to know a bit more about what to look for and then, as you saw, we used an init container to tweak the kernel or tune the kernel and that's kind of how we did it in kubernetes pine box rabbit hole. So this is an interesting one.

A

When we were doing these tests, we would find weird things happen, so there would be an outlier or you know we would run into some Network issues and we weren't sure if it was like an environment thing or a an actual bug. So we would actually try to investigate and try to replicate. Some of these issues that we saw the issue here is that you can actually spend a lot of time trying to track these down.

A

So just make sure that when you're running to these and you're going to investigate, just try to put a time box around it, because you might find that you're actually chasing something that isn't really a thing that might have been an environmental thing. So yeah just one time box, rabbit holes understand first automate later so as engineers as developers. We typically like to automate everything, make our lives easier, but we found that the plan that we had built would actually change along the way.

A

So the more we learned about our system, the more we learned about the environment, the more we learned about how to test this system. The plan that we were building would actually change, and the automation was actually would actually have to change as well. So wait until you actually understand your strategy, how you're going to test your system before actually building any automation. So once that strategy is solid, documented have someone else run the tests make sure that someone else can actually do it and then go and build some automation.

A

We actually found that creating and when the automation wasn't really worth it, because it's actually really hard. You know you have to look at graphs. You have to look at some data. You have to there's a lot, so it's it's actually not very easy.

A

So with that. Hopefully that was helpful if you're doing anything around testing at scale in kubernetes or you know anywhere I would love to learn more. If you're interested in contour gimble, you can check them out there. We also have a booth down and sponsor area I, guess and if you want to reach out I'm gonna, be here so there's any questions. Let me know, but thank you and happy testing.