Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Scale and Performance Testing of Kubernetes: Answers for Specific Applications

Description

Scale and Performance Testing of Kubernetes: Answers for Specific Applications - Georgy Okrokvertskhov & Aleksandr Shaposhnikov, Mirantis

"Managing thousands containers can be challenging, but if you want to know how Kubernetes will behave at scale we might be able to provide an answer. In this talk, we share the data we collected in our scale lab, which consists of 500 physical nodes. Using virtual machines, we can simulate up to 5000 Kubernetes minions running actual workloads, and our tests are designed to reveal how Kubernetes behaves while managing a complex application (in this case, OpenStack services) at large scale.

After the talk you will understand:
1. How Kubernetes performs rolling-updates, from a time and performance perspective
2. How fast one can roll-out containers on 500 nodes with specific constraints
3. How traffic flows between services, and what networking performance one should expect
4. How a single Service can facade 1000+ Pods with or without Autoscaler, and any limits involved
5. How many Services 1000-5000 Minions Kubernetes can support
6. How long it takes to deploy Pods for a single Service via Autoscaler to handle 1000 workloads
7. How long it takes to deploy Pods for a single Service via RC to handle 1000 workloads"

About Georgy Okrokvertskhov
Georgy has worked with Mirantis for more than 8 years, starting in 2008. He has experience managing Windows datacenters for large, distributed companies. Georgy also has a deep networking background, which he obtained while working for Cisco Systems. Today, Georgy actively works on a performance and scale testing of OpenStack and Kubernetes.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So how you guys is it so this is the last section presentation before the lunchtime so like I think we will be like quick enough to not be late. That's a lunch time, so my name is giorgia a cracker yourself, my short name or nickname is Gosha, which is much easier and we have Alex traversing cough as well, and so here's a gender for today. So we will just walk through what we are doing, who we are.

A

So what is our test setup and then we will explain what we have as a result and what we were able to achieve in our labs and they're. Like do a quick summary. So what we should expect from cubensis at scale, especially when we are running applications and not just running some artificial or synthetic tests so who we are?

A

Actually we are a members of performance teams which is a team of 12 engineers at Moran chest. So me, righteous is a company who is who was primarily focused on OpenStack. But year ago we started actually looking into author technologists, specifically in soap.

A

Containers and kubernetes was one of their like interesting applications which we started to use and the first work with kubernetes was actually related to running kubernetes on top of OpenStack, so that was our first experience with kubernetes and we were mostly focused on on OpenStack and we are doing a lot of stuff in OpenStack community. So, for example, we have performance working group who is focused on the testing of OpenStack scale and before, and we plan to extend this activity on so kubernetes and containers as well.

A

So we are working closely in partnership with Intel and Intel actually provided us a physical infrastructure for running those tests. So all this like scale testing requires a lot of hardware and it's pretty expensive exercise. So thanks Intel, we have like two different data centers. So one is rented a truck space where we have like dedicated physical 250 nodes and we have Intel data center where we have 500 node level.

A

So Alex can walk your you through the hardware. So what we have in this lab- and as you remember so, this is insole hardware and it intellect so.

B

Yeah our lab, based mostly on Adele air 630 servers, which are two sockets, have almost of much Xeon v CPUs, which are 12 cores physical ones, with 12 future ones, have one quarter of terabyte of RAM to 800 SSDs and which is quite more important to one of the one of the latest in tow.

B

It's 710 Nix, which kind of giving us a hard time to basically to figure out the best in the nation of the operation, systems of care now new drivers and they operate you kind of nothing better, but they're, still kind of not always disabled, and you know kind of optimization, which we kind of applauding and stuff ramen, which we kind of playing Tony on turning off and working on the performance.

B

So another lab consists of the none of us, which are also having kind of top-notch CPUs raised the same one set up the same Nicks and oh it's person the same one.

B

So here's how like our network, looks like so we have our lives, Brian technology used between the racks and using those three and BGP for this despite the external barrel, a the future for doing some other stuff for different networking solutions, vendors, but right now using calico. So we have such a five flags that is the biggest for one lap. So we're talking about 500 nodes, and so that's it.

B

We have each and every server have to do all had Nicks, so each and every cell have for 10g ports which are connected to different switches, switches between themselves, carbon blood and that aggregated on the spine.

B

Since routers, we have busy people who, later everywhere.

B

Kind of I, already matchsticks lands but fix nice right now know you've not used on the streets.

A

Right so another important part of performance and scale testing is to be able to reproduce all those tests and for that we had to create automation and we heavily rely on the automation in this lab and the most actually painful part is to do automatic provisioning because we have to start from bare metal.

A

So we have automation which automatically provision all those like vermin, bare metal nodes in different configurations, and we can dynamically split those 500 nose for, like difference, smaller environments, if we need to run multiple different tests or do some experimental work before we run this at the larger scale and for provisioning for very metal part we use having it Kobler. We recently started to play with Ubuntu mas, which came with Ubuntu 16, so that's version which we use and 4q burnishers provisioning.

A

So once we have operational system and it's usually Ubuntu 1604 long-term support darshan so for kubernetes provisioning, together with Callie, we use carga, which is one of the related projects in cuber nature's ecosystem.

A

So once we have our environment deployed, we have like Jenkins, which used for running specific tasks, and then we run all those tests which we we should have and all the results will be collected in several databases. So we have was great databases just like store some aggregated statistics, but for time series we use influx DBM and once we have all these data, we have special tools which will do some analysis. So we use our language to do some.

A

Statistical research and all those results are automatically published and generate like aresty documents, which we can then upload to the internet almost automatically. So this system also used in OpenStack, and that was a community standard at those time when we started all those activities and what we are looking for is to extending the this approach for queue, branches community as well, so just like high level understanding what we have in more like graphical formats or in Jenkins system.

A

We wished us or all like jobs and components which I mentioned like Kobler, was SM in Flex dB. So when we deploy cuber images, we actually rely on the reference architecture, which is implemented in cargo installer and.

A

Cargo deploys highly available kubernetes configuration, so we have three nodes which are running each CD cluster and on the on the same nose. We are running cuban writers masters, so we have like API services running there. We have like Cuba's, scheduler and cargo itself uses hypercube distribution of kubernetes. So as a recent version, specifically what we used in our lab I mentioned here on this slide, and then we have like minions nodes and based on the Ektron, the specific environments. It could be like front one sets or 500 physical nodes.

A

We also use virtual machines, so we can create virtual environments when we spawn like up to four or ten VMs their physical node, and then we use them as a minions. So that's another option so when we need to test a large number of cubed or inches minions- and we do not have enough like physical hardware for that, we switch to the virtual machines, so testing approach. So we have two tools which we heavily using, so we have for OpenStack rally project. So this is a tool which was specifically designed to test OpenStack API.

A

So OpenStack project was all about like api's and we have a tool which allows you to create pretty complex scenarios and like execute varios API commands and collect different API related statistics.

A

But it looks like that when you're heavily rely on API, it does not necessarily- and it represents the actual state of the running applications. So the next step, like beyond API testing, is to be able to deploy a special application which allow you to collect application, specific methods. So, for example, we have a tool which France so called like master or collector, which sits on top of my sequel database, and then we run a lot of ports and in each port were on container, which has a simple agent.

A

We just reports back to that collector service, I'm up and running, and here's my time stamp. When I send this message so based on this tool, we can actually figure out what actual was going on on top of kubernetes cluster, and actually it will show pretty interesting results further on. So just like briefly what we usually measure. So we can measure API response time and there are several blogs on humanity's dot I. You are related to Cuba nature's performance and they test almost usually queue branches, API response times, but also with our like collector tool.

A

We measure the actual time of response from specific containers and we can actually figure out. Is there a difference or there is no difference between those like apiary responses, about support, originates and actual running application rigid?

A

So, as our title says, so we are running this for specific application.

A

So during the last year we made a pretty big effort on moving OpenStack from bare metal to the micro service application, which can be hosted on top of some platform like system like kubernetes Overland container management system like kubernetes, so OpenStack II itself was created as a serious oriented architecture, application which actually allows us to move it from bare metal and standard deployment approach to micro services model.

A

So there are still some stateful components in OpenStack, it's mostly related to storage and data bases which OpenStack uses, but overall, so each API components of OpenStack is like stateless application and we almost have one-to-one mapping between service in kubernetes and OpenStack CRS. So so far we achieved this configuration, so we took like most frequently-used OpenStack services and continued eyes them and we have like 19 queue: branches services which represents OpenStack components and each service use replicas stairs and, as we use self orchestrating micro service approach, we rely on config Maps.

A

So that's each component, which of this application. When it starts and running it will be able to find its own configuration and it will be able to test our other components which it depends on. So we can achieve this with basic configuration. So in the details it looks like that. So this is a specific OpenStack service. We just called opens like Keystone, and this is like identity management system for the OpenStack and, as you see, we have like deployment. We, which will create a service service, relies on the replica set and each OpenStack instance. I'm.

A

Sorry, OpenStack keys and Keystone instance will be running as a port, and this allows us not only to easily deploy OpenStack battle. Also, it allows us to do like day two iterations. So if we need to upgrade OpenStack service, we just like produce a new image for the Keystone service, and then we ask queue branches to do rolling upgrade and, as this service is stateless, it's pretty straightforward to use queue, branches, rolling upgrades to perform service upgrade for the open, so we as OpenStack API should be like externally available.

A

We use ingress our external load balancers, so distributed like incoming external traffic ranks among like port. So you can see how it looks like from the queue branches perspective and when we go to the big picture. So that's application like OpenStack, which is a massive application, and this is how it looks like when it's deployed at large scale. So we have three nodes with kubernetes master and we never deploy any other like applications on top of master nodes. So cube managers have those physical nodes like dedicated to queue branches poignant.

A

We also say that okay, less all OpenStack services deploy on their own dedicated knows just because each OpenStack service actually generates pretty huge laws on node. So it uses a lot of CPU and memory, so it's better to physically figure out what will be the placement rules and we use like no selectors to specify which, like deployment topology, will use and on each queue branches minyan. We run exactly one instance of OpenStack compute, because this OpenStack compute component will manage Lib the earth and KVM hypervisor on top of all this physical machine.

A

So we need to have exactly one instance of OpenStack compute on on each physical node, because that's the process we, which will manage underlying hypervisor on the operational system. Well, very good. So let's go to the interesting part results.

A

So alex is a original author of this collector system. So that's the results which we collected by this tool so Alex. If you can walk through what we did yeah.

B

Okay, so here's the dependency of the replica replica set creation time versus replica set size. So, as you can probably see, it's almost invisible and there is a kind of replica set aside- it's only one replica set it to mention. So if a burkha set size is 100 volts, so it's kind of quite quite fast, 500 volts, also kind of rigid quite fast. So it's kind of the time is seconds for their 1,000 2,000 5,000 10,000, as you can probably see it's pretty linear.

B

So scheduling of the ports in in in terms of one replica set is pretty linear. It's pretty fun thing to take a look on their percentile, it's kind of a problem. You see that this person percentile, is kind of pretty interesting. It's kind of quite like like it should be so, which is kind of very, very unusual. So I'm gonna probably try to explain what's happening. Yeah.

A

So before this, let me.

A

So those results are pretty consistent with what you may see on the queue branches, dot, I our research so and that's pretty much one-to-one mapping between the test type. So on cue branches blog, they were running like 10 replica sets and they were like measuring time off for replica set deployments there so number of odd.

A

So we replicated this scenario and we pretty much confirms as the the linear dependency between replicas size and deployment. Time is like our concurrent e in our experiments on physical nodes. So in this test we were running Q branches on physical hardware. So this is like five hundred nodes, Q branches, cluster. It's not a virtual machines and the interesting difference, as Alex mentioned, we have in percentiles.

A

So this 50 percentile is not exactly the median of this big 99 and new percentile bar and that's pretty interesting effect when you start looking into details and start looking into dynamic, so how those ports were created, and so is our like collector tool. We have and ability to figure out which sport reported at what time. So you will see some nice plotters and by accident or not, but those plotters specifically related, so number of 500, which is our number of physical nodes or number of minions I, don't know.

A

Is there any relation in scheduling or but that's like actual natural experiment which we did and we saw those dependencies. So the the reason? What why like 58th percentile, is not exactly the median of the 99% whole bar just because of those plateaus. So as you see, we were trying to create 1,000 nodes, 1000 ports and at 500 ports.

A

Scheduling was stopped almost for 10 seconds and then it start working again and then it stopped again for specific amount of seconds and then it's like proceeding. It looks like this. There is something in scattering which actually creates those gaps and we will see those effects on other tests. But when we say linear, yes, it lie near when you have a bigger scale. So when you're at like 10,000 nodes, all those gaps will will stay like closer and closer, so a bigger scale.

A

You will see a nice like line with some small like deviations from the linear law, but a smaller scale. Those effects are pretty not small and when we go to the bigger scale. So that's our another experiment. When we're we were performing rolling upgrades.

A

So on intial phase we deployed an application set with 500 pods, and then we created new image uploaded to register, and then we asked cube images to perform rolling upgrades as a result of parameters like update periods and falling interval. We have this nice straight line, which actually says, as this rolling upgrades like works as like pioneer algorithm by one.

A

It was like racing those sports, and there is no any like anomalies seen in this process and as you see it, when we like increase the scale, all those problems with steps and are not noticeable so.

B

Like a few words about it, so we exactly mention it. What the parameters we did we do used, because the default ones is a little bit different, so a big period will be three seconds and the POE interval, as far as the remember is one second, so because of the pole, interval is kind of slightly increased by default, its upgrades will be happens, a little bit kind of slower, so that parameters will allow you to put all these updates. I mean probably somewhere around 20 minutes so for for 500 size of the replica set.

B

But if you go with default ones, it would be kind of either maybe 5, maybe 10 times. So it's just two parameters because the polling gets real as far as I see as container is up and running the new portals are open right. It decided to go and great another one. So that's parameters very, very important. If you plan to do that faster, it depends on use cases yeah.

A

Okay, so the next step for us was to figure out if they're, in your difference, if we use node selector because from like our intuition, we were like thinking that probably this will somehow affect the the process of scheduling and, as our application heavily relies on the north selector.

A

That was pretty important for us, so we're on those tests and actually within, like the error of measurements, there is no difference, so there are still some like plateaus during the scheduling process, but from like time perspective, there is no any effect of the using of nodes selector on the scheduling process.

A

But when we go to more complex scenarios, which you might see actually in applications, so here we have multiple replicas says with 100 ports in each one and those replicas sets were actually different in a single llamo file. So this is a single cube, CTL commands to create this configuration and we were expecting that those replicas sets will be created more or less in parallel mode. Well, that's actually not the case, so it's not a parallel process. So initially it looks like a sequential process, so the first replica set was created.

A

Then the next one started. So there is a little bit overlap. So it's not exactly one by another, but this overlap is pretty small and the time gap between replicas set creation time is actually significant. So another interesting effect which we can see on this picture, I'm sorry is- is grouping.

A

So it looks like that. The there is some scheduling, algorithm or somewhat processing algorithm in queue branches which actually process several a few replicas sets at once. So we see a grouping like three or four replicas sets, which are created almost simultaneously, or at least almost together, but between groups. There are still gaps in time and this actually may lead to potential effects. When you use a lot of replica sets in your application.

A

So in in our case, when we have nineteen replicas sets which we create simultaneously for open, we actually expect that it will be created simultaneously and then our components will figure out. They are like dependency and orchestrates himself through the EGCG, which we use to figure out, who is already configured so where is allocation so on and so forth, and it should take some specific amount of time.

A

But what we see in the result is quite interesting so because of this grouping effect, which is not actually deterministic, so we don't know which replicas yes will be grouping together. So we have an interesting problem and which actually creates an effect of prolonged deployment. So within first minutes, like specific, almost random sets were created, and so all those components have actually dependencies on the show offices like RabbitMQ, which is used as a message buzz between all those components.

A

So all those services which were created on first miners', they were waiting for the availability of the ribbet MQ service and components which were created only on 3rd minute, just like because of this grouping effect and because it's not actually created in parallel mode. So in ideal situation and that's what we expected initially that when we create all those replicas set in at least one port will be created, almost martini ously, and then we will have processes.

A

Running services will actually detect this replica set in house situation and- and all this like self ARCIC orchestration will happen, but actually, as a result of those delays in the replica set creation. All the all the orchestration started almost at the like 4th minute, when all those replica controls were created, so that actually might be interesting effect on the application deployment. And this was caused by this scheduling mechanism or like a replication set mechanism.

A

Right we almost there yep.

B

So for the mint working, that's for its expectable results, so we just did a few tests for the figure out how the whisky de we so just to mention that the using Calico right now with the kubernetes so for the networking assumes that kind of you almost get.

B

The wire speed v is kind of precious moment to use or standard MTU 1500 and it's kind of limited by kind of you see speed of two links because kind of fusing commitments in this fallings warning some using kind of two bonds from all of that, and it's kind of perfect try to use for. We have to go and maintain the different ports so create based, adjust multi-threaded log between the different parts. So we tried that we see the expectable that sorts will be about statistics.

B

Oh you give it's kind of close to our speed, so you just kind of 90 percents fine. So let's go to a summary. Yes,.

A

So summary is based on what we saw, but still you need to remember that statistics might actually show different results may have like different interpretations, but, first of all, when you look on to basic like performance test like we have published on cue branches, dot, io that says, okay, we can easily.

C

A

Like a thousand of ports in cuban it says that's all true, except so there are other like effects which are not covered by those tests, and you cannot actually rely solely on those tests who predict the time of deployment of your application. So, as you saw, there are some another effects based on the replica set numbers or any other like object, numbers which cuban adjust might have, and your projections might not be very accurate.

A

So from networking sides we didn't see much effects, but there are different. There are some specifically. We saw problems with service discovery, so when we were running large amounts of workloads which generated traffic, obviously cube dns subsystem was overloaded and we saw like delay in the cuban artists service discovery.

A

Scheduling is linear, but still as usual, there are other effects wishing to take into account when you plan your application deployment and because of like replica set behaviour when, when you can rate yourself for orchestrating application, you might need to take into account those effects.

A

And there were few issues which we mentions when we were running those tests and those tests were running on the bare metal. So first of all, docker running in production might be tricky, so we saw multiple instances of problems when docker process was hanging so and the typical symptoms.

A

You see, you're like cuba, in sports, in status, benzene or like creating, but nothing actually happens, and then you like login to the system to the docker PS, and it just like things and the only solution for that is to restart the cure process which might affect your running or all region, not running. Containers on this node and queue branches itself does not handle this situation very well.

A

So just like way and waits- and there is no like communication between docker and you know, queue branches, but a cube crisis will never reschedule on such force, and that might be a huge problem at large scale when you have like production, environment.

A

So when you use calico or any other networking solutions, troubleshooting is a nightmare, so they use a lot of like underlying technologies, bgp IP tables and other side and is without like proper tools. It's almost impossible to properly like trace your networking stuff, especially when guys went like developers created their contingency, is trying to reduce the image size.

A

So they remove all like networking tools like yesterday, we were trying to troubles with one of those problems and we had no like pink commands in the container, so we were unable to do anything so that might be a big problem. So don't forget about like troubleshooting tools, because it might save you a lot of time and cube. Analysis itself is not very good with diagnosis and collecting all those like problematic areas. So we saw several instances of problems where.

A

Tns we know we're not updated and cue branches were unable to do anything, or we saw an instance where, when cube richest, node was actually like. Cubensis me on was not reporting back to the queue branches. Master and cue branches was unable to remove ports from this minion and we were trying to remove this namespace, and this never happened because extermination process was waiting for those sports and kubernetes was unable to do anything and it's hard to diagnose those problem. So it will require a lot of efforts before you figure out.

A

What's the root cause, so the last part like as we were, engaged with insole stuff, so I have to pose like Lego, not just about like performance results in their dependency on the hardware. So it's right there.

A

So here's the links so here's a tool it's publicly available on the github. You can use it so and we have a lot of queue, branches and OpenStack related star on the OpenStack performance, documentation page. So you are welcome to go and see what we have there. So there are interesting, like campers and so two branches, OpenStack components like environmental provisioning system, so it's not specifically related to a mobile sector.

A

Okay, that is thank you. Questions.

A

We end up with creating Pigsy server like with cobbler, which would strap all those machines. We are eBay my interface with a small initial image and then inside this image we are running torrent based system which actually pulls actual aggression, all system image, and this actually allows us to distribute the networking traffic.

C

A

Because we were limited to specific, like hardware configuration which we hear there's.

B

A torrent distribution kind of allows us to basically distribute traffic between the nodes between as soon as the node downloads, the image is starting to distributed.

A

So when we run tests, we actually were on the initial like warming route, we round which downloads all the images to the minions and then we rerun those tests. So it's it's pure time of kubernetes and dijkers startup. So it's not specifically like bounce to the image pool yeah. So we.

C

A

I, don't think so. No.

B

No right now, no, it's not tuned in this way, but as I know, it's a roadmap. We.

A

Use load balancer in front of register, obviously yeah, but not in front of queue branches. So we have pretty like powerful hardware, so at 500 node we didn't see any problems with running Cuba in Cuba native master or on a node and so like redirecting all the traffic to this. No, it's powerful machines like 12 cores and a lot of memories, so Cuban it is pretty capable to handle all this.

B

Lot and the only thing- that's really kind of thinking about that we have a limited resource on and the scheduling. So there is no reason there is only development going on the motor scheduling can kubernetes. So we kind of waiting for you because it should change the situation dramatically. Boss, mm-hmm.

C

A

Rally might help, but it's not specifically a diagnostic tool right, so there should be a set of tools, probably as a part of clearances distribution which are specific to cue branches problems right as well as for networking. So it will. So if your run, like things in production for troubleshooting, you'll, probably like have to have a special image which you can like quickly start continuous, and this image will have all like necessary tools for networking promotion because, like in typical situation in standard container, there is nothing at you. You actually can do anything.

B

It's for the kubernetes itself, I mean like refusing service so using H, a kind of not easy, but all bones, including KP tables, for that is some kind of chance of sending there's some particular kind of session to some budgetary server and it's kind of like be playing this 1000 know some of them are populated with some sometimes just 1000 service. At the same time, so we have 1000 records of the IP tables for the service itself and there's a lot of probabilities.

B

They different ones and it's kind of funny I mean if one of these services are not working and if the IP tables decided to scatter our name to this particular kind of basic in translate. This particular request this particular session to the series you don't ever know about it, because next one will go to the next one and that is kind of well very, very hard to figure out.

B

But what's going on because the tokete which was written it was written without kind of this is not just kind of trying to figure out how it works, but it's written in a way that is real, frail and any unpredictable thing. If something is wrong. So in this case one of the DNS services just sometimes is lock blocks to region, I mean using consumer producer/consumer way, so it's kind of fit segregate the requests and put them to the MySQL to decrease the cost of the doing that and increase the responsiveness.

B

So in this case there is each and every consumer or try to create connection to my skills, also be on our case. It's 96 consumers because of the twice of the number of virtual CPUs and one of them will fail, which one tools is just going exist, so I mean in case of the application we all done with some in this particular rate, so just die, and if you never know what was going on.

A

A

No, so the test was actually between multiple between minions, so the traffic was flowing between means, so there is no any workload which is scattered on the queue branches master, and we specifically placed those containers on the nose which are in different tracks, because then it will flow through the spine, instead of like being bounded by the top four of the rocks wish.

A

We did so, first of all, obviously it's not like very extensive test, so it's just like basic one, so we didn't see a huge amount of progress, but when we were running multiple composable containers, so we saw an even distribution of the throughput between containers. So you might see the situation when, overall, you have 18 gigabits per second, but one container will have like only 30 megabits and another one will have like 4 gigabit and another one like one gigabit.

A

So that's more advanced scenarios and requires a lot of like specific testing for the networking, but.

B

Also solve those problems because yeah because you go into the container work, you kind of you usually assume that you almost out even kind of much more containers because a bullet overhead much modern VMs, but in this particular case that each of them create each and every application will create different footprint on the networking. So I probably have to come up with some cure stuff.

B

In networking things, I mean on an important provider for the kubernetes, so in calico there is no such thing and the rest have to I mean not much I, don't know extra kind of solution which kind of fully implements or requirements which I'm kind of very very will be used from continue based kind of deployments.

A

Ok, cool. Thank you guys. Thank you for coming. That's lunch time, my let's not miss our lunch. Thank.

B