Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automated Kubernetes Scalability Testing - Sebastian Jug & Naga Ravi Chaitanya Elluri, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Automated Kubernetes Scalability Testing - Sebastian Jug & Naga Ravi Chaitanya Elluri, Red Hat

Kubernetes supports large clusters according to the docs, but how does it actually scale? Who came up with those limits? What are the actual numbers? To challenge this we built a CI/CD environment geared towards deploying, and testing, Kubernetes at Scale. Our stack consists of Kubernetes, OpenStack for IaaS, Jenkins Pipeline, Ansible for automation, pbench a performance benchmarking, and visualizing tool, Prometheus and other Open Source projects. The stack has pushed the performance and scale limits of Kubernetes with kubelet, control plane and cluster density focused tests. In this presentation we will explore the story and challenges on how we built and tested this stack. We will demo the test harness and share the latest performance and scale results. Attendees will learn what the real scalability limits of Kubernetes are, as well as how to scale-test their own infrastructure.

To learn more: https://sched.co/GrYn

A

Hello thanks for joining us today. My name is Ravi Laurie I work on open shift, automation and tooling I'm.

B

Sebastien drug also work on tooling, mostly and go and I also focus on container runtime performance. We.

A

Are going to talk about automated Cuban at a scalability testing so who are we? We are part of performance and scalability team Metroid had working primarily on open shift, the wider performance and scalability team who works on the entire Red Hat's products. Portfolio I'm sure everyone over here might have played around with communities are open shift, which is an enterprise version of Cuban IDs at some point of time, quick show of hands. How many of you are running, kubernetes our open shipped in production.

A

That's a lot of people when running or bishops in production. One might have these questions in their mind. Does OpenShift support running applications at scale? If, yes, then, what are the cluster limits? How can we tune across so to get maximum performance? What are the challenges? One might race when managing and running a large and dense cluster? We had similar questions in our mind over the next few slides we are going to talk about how we build and used our automation, pipeline and tooling to come up with the answers to these questions.

A

Infrastructure is one of the key things we need to test any product at scale with this mode. If we build scale lab it's an initiative to operate when hats products in their upper limits, the scale that consists of three fifty plus nodes about eight thousand plus course, which is a lot of resources, and we keep adding more and more physical nodes to this lab every year. The resources are all shared and are available on demand.

A

Let's talk a bit about our cluster availability. We install 2,000 node cluster once every openshift release and runway Nia's performance and scale. That's on top of it. We maintain a 250 node cluster, all the time to resolve customer issues, to design new tests for features added upstream and for R&D work. We refresh this 250 notes cluster with the latest and greatest OpenShift, which once every sprint our Sprint's are usually three weeks. Long.

A

By the way we recently completed a 2,000 or large-scale testing for openshift 3.11 running kubernetes mode, not 11, I'll use this run as a reference to talk about how our automation, tool or automation, pipeline and tooling works. This is how our automation pipeline looks like the pipeline runs in stages, with each stage being an individual jenkees job. So all the blocks over here are separate. Jenkees jobs and the pipeline controller is responsible for time them up together.

A

The pipeline is flexible enough to allow us to run a particular job instead of running all the stages by default. I'll talk about what each job doesn't detail, however. The coming slides the first job in the pipeline is image provisioner, this job watches for new OpenShift code bits and builds a.m. eyes and cue cows with various packages, configurations and container images baked into it. By doing this, it reduces the in the time taken by the Installer to build out an operative cluster as the Installer need, not fetch any packages or container images over the network.

A

During the install time, this job has suffered to build images based on both well and atomic. Once the image provision is done, building an image, it calls the image validator job. This job validates the image built by the image provisioner by installing an all-in-one node OpenShift cluster.

A

So at this point we can be sure that the image is good, the next components of pipeline controller and watcher, given a properties file, which is where we define the parameters needed to build all the jobs in the pipeline pipeline controller, reads it or HTTP and then calls the respective jobs the pipeline can run multiple jobs in parallel and supports running the jobs on both public and private clouds.

A

We store all the job templates in a public github depository, it's all open source and it's in Djamel by the way. So it's easy to edit and let's say we modify a particular job template or add a new job template to the github repo. This is where the watcher component comes into the picture: the watcher component watches for these changes and creates, or updates the jobs and Jenkins.

A

Now that we have the image and properties files ready, we kickoff the pipeline double OpenStack. So why did we choose? Openstack OpenStack provides the virtualization platform to scale out the OPF cluster to a higher node count. So, by using virtual machines, we can make efficient use of the lab resources and also can run large scale tests on a smaller hardware footprint we use Tripler to deploy. Openstack trip flow basically brings up. The OpenStack under cloud which manages the hardware and installs of the stack over cloud workload is where we host all our open ships.

A

Virtual machines, Chapleau also installs, say accept cluster, which manages the storage for all these OpenStack virtual machines. This slide has details about the hardware that we use for our mm, not scale test run. We got 84 machines allocated from Skylab and we converted that to a 2012, open shift cluster impressive right. We use the same flavors as as an AWS.

A

The next stage in the pipeline once the OpenStack cluster is up and running is openshift. Everyone uses their own architecture to install and set up openshift. This is how we do it. There's a gem nerve or Bastion host from where we control the install, manage the cluster and run wages, performance of skilled us there's a load balancer which balances the request. The masters there's a DNS which manages a cluster dns.

A

This job installs the bare minimum operative cluster with three masters: it CDs three infra nodes, three storage nodes, two app nodes of worker nodes, I'll, explain the reason for just installing 11 node cluster in the coming slides. Most of we call this 11 node cluster as core cluster. Most of the nodes in the Cole cluster uses nvme pass-through, and that's because at CDs running on the master nodes, elasticsearch running on the interest, Infinite's rahat, OpenShift, container storage running on the storage notes, needs high, throughput and low latency for better performance.

A

We also make sure none of the core notes land on the same hypervisor, for the cluster, to be true, hey, CheY and also for the performance and scale test results to be valid.

A

The next job in the pipeline is to set up tooling. We built a bunch of open-source tools and cluster loader is one of the most extensively used tool by us, given a yama definition cross, the loader read, set and loads of the cluster with various objects. This is a sample cursor loader, config I will explain what this config does this.

A

This config creates a cluster project, 0 name space, and if the name space already exists on the cluster and deletes it and recreates it again, or else we can set if exist, to reuse to reuse, the same name space and it loads up that name, space with thousand passports, the step size is set to 50 and the pause is set to 60 in the tuning sets, which means that the cluster loader will create 50 parts in the first iteration waits for the parts to be up and running and then creates the second part second set of 50 parts and goes on till like it reaches a thousand part count.

A

This is just a simple example of what clustering can do. The things that cluster loader can do are unlimited. We can use the cluster loader to load up the cluster with bunch of open shift templates to create thousands of namespaces over thousands of parts, secret routes, etc, etc.

B

We've also embedded cross the loader in the atomic help it should test package. So those of you that have open shipped already installed, you may or may not have this package installed, but yum install it any one step away from loading up your own cluster, with whatever images or objects you like it's.

A

Also available in upstream kubernetes in case anyone wants to check it out.

A

The next tool that we use extensively is P bench. It's a benchmarking and performance analysis. Framework pH has three subsystems: P bench Asian, which runs on all the OPF nodes on which we want to capture the performance data, P bench asian, provides way, stools likes our highest, add bit stat and flexible enough to allow us to run our own tool.

A

The next P bench agent also provides benchmark scripts like file new poof and user benchmark, so using AP bench user benchmark. We can literally run any script and expect the agent running on the open chef nodes to collect the tool data. The next sub component is the event server. Once the agent is run, dunning the bender once the it's done running the benchmark, it tears up the results, ships it out to the server where the results get unpacked, archived and indexed. The last subsystem is the B bench web server. It's responsible for visualizing.

A

The data collected from the OpenShift nodes I'll, give a simple example to show how pH works. Let's just enable bit stat tool on the open ship, compute nodes and use pH user benchmark to run the cursor loader config that we saw before.

A

At the end of the run, we can see the memory usage of cubelet, cryo darker or any other process running on the opposite. Nerd, when the cluster loaded is loading of the cluster powerful right.

A

The next stages in the pipeline are conformance tests and scale up. This job runs cubanía days, end-to-end test and once the conformance is green, which scales up the cluster to a desired node count. The reason for running the conformance before scaling up the cluster to 2000 is that let's say we find out that the cluster is bad after after scaling it up, then we'll have to tear it down and build it from scratch again, which takes a lot of time.

A

So, in order to avoid that, we build a core cluster run conformance on top of it, and once the cluster looks sane, we scale it up to 2000.

A

The openshift installer is ansible based, so we usually set the number of folks to 50 for the tasks to run in parallel and speed up the scale of time.

A

The last stage in the pipeline is performance and scale test. We design various tests to look at the performance of plain cubelet, networking, HF, proxy and storage. We also have bunch of tests in the pipeline to validate and push the cluster limits.

A

We repeat the same pipeline run with the latest and greatest open shift bits once every sprint to take a look at the performance regressions.

A

The motive behind building the automation pipeline tooling and the scale lab is not just for our team. We are actively unloading, other teams to enable them to test their workload set scale. This way they need not have huge infrastructure, they need not install and maintain Alaska coaster with the latest and greatest bits. They can reuse the tooling and automation that we build instead of maintaining their own.

A

Adding a new workload to the pipeline is as simple as creating a PR with the right job template and the watcher component will pick it up rhiannon Jenkins job and adds it to the pipeline.

A

Sebastian will take it from here thanks.

B

A

B

We have our cluster running and then we need to run the tests and, let's take a look at what we found, some challenges that we face. So this is just a quick list of some tests that we'll be covering.

B

We have a good sized team, so we have specialists in storage, networking that take their respective components and make sure that openshift is actually compliant and optimized in each of these different test fields, and since we need to cover all the bases, not just the control plane, because the decisions that are made on behalf of Red Hat for the choices in HTTP for the choices in storage, we had, we got to make a lot more tests than just regular control plane tests.

B

The first test- I guess you could say, is really just the scale-up. So while you might not think it's a test per se, we do need to measure how long it takes to scale up as well as each step to make sure that it's optimized cached and not duplicated. If, obviously, if you're scaling to a small node count this all times, not that relevant.

B

But when you have 2,000 nodes take if things go wrong several days, so the node vertical is a most basic cubelet density focused test and what it does is create the max pause across the compute nodes, and here you can see the cubelet as well as the container runtime system resources used by both of them across the versions, 3.9, 310 and 311- and this is one of the things that we optimize and strive to maintain- is no performance, regressions across versions.

B

So it's the same test, adding bunch of features, but you can see things don't get worse. At least Master vertical is the follow-up more advanced version of node vertical. It's a more advanced cluster. This density test that doesn't just create pause pods, it's not just a theoretical, best-case scenario, but actually includes builds secrets, routes and all sorts of openshift objects outside of pods. The goal of the massive vertical test is to have a more fully loaded environment. That is more representative of real clusters.

B

So here we have the control plane again, the API server, the controller, as well as at CD, with a exponentially increasing project account and the projects contain the same amount of objects. So, as you can see, as we move to double the project count quadruple the project count, the control plane scales, much better than linearly so once again, scalability is something that we pay close attention to, as we want to make sure that you can run as many projects as humanly possible with your objects in them.

B

The first decision that we make I guess for you guys in openshift, is the choice of an ingress, HTTP router. What we use in openshift 311 is H a proxy 1.8.

B

So the major feature that we have in a sea of proxy 1.8 is the router threading, and you can see the difference between openshift, 310 and 311 in the different versions of HD a proxy. As we add in more concurrent connections, the throughput of the a tree proxy router scales quite nicely. Of course. This is also brought to you by another open sure, open source tool called MB. Then we've created specifically to be able to saturate these links.

B

Networking is actually a relatively complicated topic with openness shift on OpenStack, because each of these products carries their own Sdn. So this thing introduces a problem that we would like to refer to a chaise otherwise known as double encapsulation. Needless to say, that is not great for a footprint performance we use among other tools. We use you perf to benchmark a node to node service, the service and pada pada, throughput and latency for both the TCP and ATP connections.

B

The way we are working to get around the double encapsulation is a new product in OpenStack called courier, which leverages the network abstraction of OpenStack Neutron to isolate the networks of the kubernetes cluster. In this example, we have the default iptables, hybrid Neutron, firewall driver compared to the newer I guess: open vSwitch, firewall driver, but small node sizes, open V switch, produces better TCP throughput performance, but quickly falls off when the cluster gets larger.

B

As for storage, our storage solution on openshift is cluster, so otherwise known as Red, Hat, open shift, container storage form lay container no dip native storage in case you're, not keeping able the names. Essentially, we run a pretty decent-sized, Postgres or MongoDB install on the cluster and then load it up with PG bench specifically, and the improvements at 311, the Gluster block performance, has increased substantially.

B

Keeping with the theme default, installation of open shift does come with Prometheus pre-installed. The previous stack is cube, state metrics, node exporter and at very least alerting. So two topics that we cover with Prometheus is the research and development of guidance as to capacity planning for Prometheus. We want to enable customers to collect large scale, metrics or, alternatively, maintain 15 days of retention.

B

So the recommendation here is to have at least one terabyte of general-purpose SSD storage. We also applied synthetics large-scale workloads that are based on graph on queries, but applied synthetically just to make sure that Prometheus can scale and the next version of openshift not 311 will have Prometheus two and a half which does have some substantial, both memory improvements as well as scalability improvements, so configuring there's not really another test, but it's another tool that we've written.

B

One of the challenges that we're always striving to improve is the reality of the load test right. We started like I, said back with node vertical and then master vertical, and now we have the config mirror, and what this enables us to do is essentially try to extract all the objects that we can from large-scale clusters like open shift online and take out the user data and then transfer them onto our scale, lab or other large piece of hardware.

B

This enables us to debug performance issues that customers see is well on the field, because we don't often have access to their environments and we certainly can't reproduce all of their configurations. So this is one of our neighbor tools that enable us to do so.

B

So, as with upstream we've been closely tracking the cluster limits and each of these fields pause per node pause for core namespaces, we have tests specifically designed most of them using cluster loader. Has this very trivially easily to generate whatever you want, namespaces or the pods, or any object and load it up to hit these, but essentially every release at very minimum? We track and verify in the cluster limits.

B

So higher limits recently open shift. We started diverging a bit from upstream as our pod per namespace. The third seeing some issues for kubernetes start seeing some issues with pods per namespace. The API server started slowing down bogging down and we didn't see those same issues so given everything we do eventually or we try to make everything land upstream, some things that make it but we're working with sig's scalability as I'm a member of them to try to figure out. Why they're? Why we have this difference right now,.

B

Is it possible to run to a 500 pods per node, so we currently support 250 pounds per node? Theoretically, it is now possible to run 500 pounds per node if your application does nothing. So yes, of course getting there, there were some challenges even with DNS mask it didn't like it, especially if you have a small cluster. You have like 1, 2 or 3 worker nodes, 500, pods per node, sure 2000, pods or 2 thousand nodes. Later that's not gonna work it out.

B

So it is possible to run 500 pods per node, but the issue becomes when you have real applications. Real containers running on a node, it gets a lot more complicated first start is the first thing that happens is the container runtime seems to be the weakness at this point.

B

Preventing higher node counts, as the plug latency will keep increasing, as you have more applicative real applications, not just the control plane. Applications. The plague latency, which is a good indicator of health for the container runtime, will keep increasing over time and eventually the nodes will become not ready and things just escalate from there. The scheduler will say we just lost a node, we can reschedule 500, pods or whatever it might be, and there goes your cluster tuning so we'd like to take a layered approach to tuning at address performance issues.

B

As best as we can. We try to optimize for performance at the lowest level, which is the code level. We work very closely with engineering teams at it right had obviously, and all of our work does eventually for the most part, github stream. Next, we try to take tuning on so, if you're running ro, you definitely out of the box, get the tune, D daemon. Essentially what this is is a daemon that runs on your system and automatically applies our best practices, our best tuning knobs to your nodes, based on whatever profile you select.

B

So if you have a storage node, you can select for throughput or I/o, not both like someone was asking for earlier at the booth.

B

If you have an open shift, master node, it's all tuned out of the box, so you just need to select it it's set at boot and you don't worry about it. Of course, you can take this further if you have special needs. For example, the API server has PS and burst rates out of the box. We set this in as same defaults for any size cluster, but if you have these monster iron rigs with 96 cores and four different sockets, then we can do a little better.

B

If you up the QPS and burst rates, the scalability does drastically increase, but at some cost so every major release. We update our scaling and performance guide, and this guide covers all the topics that we discussed. Everything from the installation to nibbles the API server overrides that I just discussed the cubelet configs to increase the density.

B

The networking and routing optimization, as well as newer features that we're working at working with and adding to kubernetes, to support, running your actual workload and tuning your workload to make sure that you get the best performance for your applications, such as the CPU manager, huge pages and now GP is so like everything we do. Our code is all open source and upstream good, bad or ugly. It's here. So if you want to take a look run, it kick the tires a bit or fork it and contribute it's up to you guys.

B

What's nice already, we.

A

Are planning to test operators at scale, which is the buzzword for this conference by the way, and we are working on onboarding, more and more teams into a pipeline to take advantage of the automation infrastructure, including that we build and we will continue to push towards higher cluster limits, and we are also working on adding support to the pipeline to run in various public clouds. It does have support children, these Kael tha's in any cloud since you solve containerized its install phase that you are working on.

A

Thank you next.

A

A

Before we jump on to the Q&A session, I just have one last thing to say: we are hiring the famous last words. Please feel free to reach out to us to know more about the interesting things we are working at Red Hat. Any questions.

C

A

No, it's on the lab results are all shared like it's in the lab is like huge, like we just get like some notes allocated once every open ship release, we take them, install up and start open stack on top of it and then run our pipeline, and then we just give away the machines once we are done.

A

We use the same VM sizes like same machines with like same memory, resources, CPUs and everything all the time, so that, like we can get a run to run comparison between the previous runs.

A

You mean on the OPF cluster yeah. We manually tune the Cubist and buster's for a cluster, so one one time we want them when we are running a test which created ten thousand namespaces with like lot of objects, secrets routes and all that, like the API server got overloaded, and then we had to like tune the QPS and bus rates like double double them up: quadrille them and to like get the cluster back to run and state.

A

So this didn't happen on the 250 node cluster, which we usually maintain every sprint, and that's because there are just 250 nodes hitting API server instead of two thousand nodes. So, depending on the node size like we do, we do some tunings on the cluster.

D

B

Yes, we have a number of different test rate, so in different stages of reality, we do try to make this as real as possible and we do have I guess, like whatever database specific tests and all sorts of different tests that would have state as well. Sorry, good idea.

B

Yeah, so it's that's more of an upstream test and it doesn't work with all the open shift objects. So, right now, we're moving in upstream as well. I believe we're moving away from cue mark we're rewriting the upstream cluster loader to add the additional features. Also I'm pretty sure cube mark is the hollow pod one. It doesn't run on real, like pods and nodes.

B

It just creates like virtual ones, so all of these tests are on real Hardware using real nodes and all that, so it we're really trying to strive for reality, and that's where we come with above the tunings that Robby mentioned is just actual issues that we encounter yeah.

B

Yeah so since that's not GA, we can't discuss that at the point, but we might definitely maybe yeah.

A

But we'll have any Muslim.

A

B

So I think that started with the we have a large performance of scalability group that doesn't just work on openshift. Specifically, it works across all relics, so we needed something that was pluggable and modular to work with all the basic tools, because I'm sure you guys are familiar with that chart of Linux performance measurement tools. So we needed something that could capture anything and everything and could be easily extensible. And so that's why we wrote P bench was just we didn't want to have some sort of like tailor-made stuff.

B

It's pretty easy to just add a new tool to P bench and capture it and record it and compare it so really just came down to extensibility and standard tooling the room yeah.

A

It's quite flexible, like we can literally like create a script right now which collects OC, get nodes and then it's it and we can add it to P bench and then it will collect OC, get nodes every three seconds or ten seconds, depending on the interval. While your test is running so we can accelerate and stop like, depending on the other tools.

B

Stram, it's up.

B

Yeah, of course, of course, yeah so sixgill ability just.

A

B

To sorry going.

B

I would say we don't have plans, but that's something that I do care more about, so we should talk about why that is maybe offline or whatever not on video yeah.

A

We do report thanks to six scalability, so recently we ran a test to see if we can increase the pods per namespace more than like it's three thousand right now so upstream lower two limits, so it without like. Why not create a test and then try to see like if we can increase the number of parts per namespace, so we found out that it can be increased. So we reported the results back to six scalability.

A

So our plan is basically to get the change upstream first and then basically get that into o P shift instead of just modifying it in the efficient level. Yeah.

A

Yeah, we haven't changed any offer limits, so we are trying to get it merged upstream and then get it down to a publisher.

B

All right, thank you very much.