Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Autoscaling Containers... with Math by Allan Espinosa, Engine Yard, Inc.

Description

Autoscaling Containers... with Math - Allan Espinosa, Engine Yard, Inc.

"Docker and Kubernetes provides delightful APIs to show various statuses of our applications. From CPU, Load average, HTTP response times, etc. we all have what we need to make sure our app is running healthily. When things are on fire, we Ops people twiddle some knobs like spin up more Pods to keep things going. We use mostly our experience and knowledge of the systems that we are running to know what to do.

However, if you look at everyday things like your airconditioners and thermostats, they don't have an Ops team that gets paged to set the correct level of the coolant to set your room to the right temperature. They use some math behind called *Control Theory* to keep your room's temperature stable. In this talk, I will show how we can use the same concepts to autoscale and manage the health of our applications on Kubernetes."

About Allan Espinosa
Allan works at Engineyard where he supports customer's Deis, Docker and Kubernetes infrastructures in production. Prior to that he worked on managing one of the world's largest CloudFoundry deployments using Chef.

Allan is also the author of "Docker High Performance" from Packt Publishing. The book contains some worked examples and high level concepts on how to get Docker up and running in production.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, yeah thanks doctor and everyone thanks for coming. My name is Alan I'm, a Support Engineer at engineered and base. We have a book day. We make helm and work flow today, I'll be talking about auto scaling and how you can make your own with a bit of math. This is a bit of a personal epidemic, ish project of mine that I mess around from time to time. So yeah a bit about me, I wrote a docker book last January, so it's been out for a year now, so it's mostly obsolete.

A

I also wrote the the chef cookbook to launch a kubernetes cluster, so I made custom chef resources. So if you want to learn about more about that, you can look me up in the hallway afterwards.

A

A

I would like to talk about how we typically run and operate distributed systems. So typically we have op staff and we manage a server farm that receives traffic and so, for example, we monitor the CPU usage and then it the the actual usage gets fed into our monitoring system like hipster and then pager Duty wakes it up at 2:00 a.m. and then asks us. An ops person wakes up at night then check if CPU utilization is too high or too low.

A

That's why we got alerted so we check if our goal is different from the actual reading, and then we log into our fleet of servers or our cloud, provisioning, API and spin up more instances.

A

So you can see we have a similar structure for an your air conditioner. So here in an air-conditioner, thermostat basically regulates the the coolant which releases cool air, which makes your room cold. So you have your room temperature, your the weather, and then it gets affected by the cooling, a coolant of the air-conditioner, and then we get an actual temperature reading from our sensors and then the thermostats read that and then it compares with the temperature we set in the thermostat and the actual temperature.

A

So if it's too hot, then it releases more coolant if it's too cold it'll be easy space. So the difference here is that your air conditioner doesn't have an ops team that gets paged to adjust the coolant right. So so we have this automation now right. So we have in kubernetes you have your your horizontal pod or the scalar, which basically allows you to set the number of replicas in your replica set or replication controller, depending on the reading of the CPU utilization from from hip-stir.

A

So you set a target on your HPA object and you scale it so so we have that already it's working properly. So so one of the things that you can do is you can try to customize it, but the current object only has so much also by knowing the first principles on how an auto scaler works. Then you can know if your auto scaling policy is effective or is it stable?

A

Is it working as expected, or is it working good enough, so the math behind the air conditioner called control theory can help us figure out that so control theory is discipline developed how systems are dynamically affecting each other and using that information you can use that to create feedback loops to correct the target behavior and then there's a couple of results in math that describes how effective your feedback loops are.

A

So so control theory was developed in the early part of the Industrial Revolution, so the core part in the Industrial Revolution is the steam engines became more reliable by having like consistent performance. So this century centrifugal governor was made in order to regulate a steam engine. So a centrifugal governor works by having a spinning disk connected to the engine. So if your engine is fast, this will rotate fast and that spinning wheel is connected to two fly balls that will rotate rotate. So the faster these two two weights rotate.

A

Then it will go up and then, when it goes up it it it. It manipulates some levers, that's connected to a valve that regulates the steam intake. So if your steam engine is too fast, so the weights will go up and then it will close the valve, so the amount of steam getting inside your steam engine is lower. So it will try to slow down. So this this produces a self-correcting behavior so that the centrifugal governor was invented around 17th or 18th century. But then people did not really know how it works.

A

So it's only in the 19th century, where people got the theory figured out. So you can say that during the Industrial Revolution people have been cargo coding technology for 100 years or more so we're doing pretty good US tech industry.

A

So so in control theory, there's architectural II, a few basic parts. So you have your target input the set value so in an auto scaler. That's! What's our target CPU utilization for our system and then in the air conditioner, that's the set temperature in our thermostat. Then you have the output, which is the actual one, so the current CPU utilization and then next you have the control input which is dictating the number of replicas your your autoscaler sets, and then the traffic in this case is just modeled as a disturbance.

A

So the the traffic going to your servers will change the CPU and then it will.

A

Your autoscaler will react according to that, so there you have the basic structure. So this is a simple code that I wrote to illustrate that. So you have your output value, that's read through a sensor, so in the autoscaler I wrote it reads off the value of hip-stir hip-stir and then from the output value. The CPU utilization I get the number of instances or the number of replicas or using my controller and then I have an app to it or that make basically makes an API call that sets the instance.

A

Then next I have a control loop to periodically send off the values to to my monitoring system and then I do that every five seconds. So that's the control monitoring interval.

A

So this is an example of how it works. So I have my own autoscaler, not the one built in kubernetes, so you have the blue line, which is the CPU utilization and then the number of replicas.

A

So at this point, I increase traffic, which correspondingly increase CPU, and my number of replicas increased corresponding same goes when I reduce traffic, then CPU goes down as well, and then the number of replicas follow and I'll show that later on in a small demo, if we have time so, why does like this simple, auto scaling, work out, scalar work so in systems you can make a simple model if it's linear and time-invariant, so as you scale up and down, then it's not affected when what it's scaled up or down the effects are the same.

A

So in control theory, you can model your autoscaler as a these equations. So these equations are the same they're. Just one is analog and one is a digital form. So basically Y is the output, which is the CPU utilization and then U is the control input, which is the number of incense. So this equation just says that the number of instances you set in your replication controller is dependent on the CPU utilization and a proportion of it and the the change in CPU utilization. That's why there's a derivative written there in the digital form?

A

Where so that you can code it? You can model it such as the CPU utilization at the next monitoring cycle is, is dependent on the current CPU utilization now and the number of instances you set now as well.

A

Okay, so once you make a model of your system that you want to auto scale, then there's there are like four desired properties that you want. So even though you're not making your autoscaler, you can use these criteria to evaluate the built in horizontal path. Autoscaler, you said in kubernetes, so first is stability.

A

You want your scaling to be stable over time, so here it's an example of an unstable autoscaler, so at first CPU usage increase. So you add more instances, but then you add it too much so you scale down, but then you scale down too much. So you try to add more and then you remove instances. Then you keep adding more.

A

So in the end, like you, keep adding more and more and then reducing more and more, which makes an unreliable performance because from at one point, you'll have a thousand instances and at some point you have one so it unless your traffic is really like that, then it doesn't make good of a system as it should be.

A

So this is also another example of an unstable one. So, no matter how many instances you add, like your utilization, still goes up through the roof, so there's probably other problems in your application, like the networking or the database. You depend on if such a thing happens, so this is sort of stable, but then it oscillates through.

A

So you have the dashed line, which is your target performance, but then it oscillates through like too much instances and not not enough instances, but it's within a certain range. So it's stable, but not yet quite there, then this is what you want. So basically, as time goes on, then it the error margins starts to decrease, which is stable system.

A

So here is another stable version, which is a smoother one, so it gradually goes there. So a problem with this: it's sometimes it's slow and.

A

Alright, before I going the slow like another properties, you want it to be accurate. So it goes through the target performance at some point. So I want to go to, let's say 70 percent utilization. Then after some point your as you scale up, then you reach you go back to the target value of 70 percent performance.

A

So when I talk about the system being slow, so when you, when you scale out instances you want to, you want to settle a value that too fast enough to your target performance. If it's too slow, then it means in auto scaling. If you didn't have enough instances, then you already lost traffic, so you want it to be fast in terms of settling time.

A

Then remember the oscillations I showed earlier. So you also want your system to not have overshoots me, which means you overcompensate, and then you go back again in some designs of auto scalars, and/or controllers. You may want that as a trade off, so that it responds fast with a faster settling time but depending on I guess your budget, because you added too much instances, then you will probably not want that in your system.

A

Ok, so let's go to the auto scalars and how you can make one yourself. So here we have our. This is basically the replica controller, and then this is our auto scalar C, so, basically the to figure out the number of instances to scale.

A

We need to get our goal, which is the reference value and then the current value, so the target CPU and the current CPU and get the difference of that, and that would be our error and then use that error to get the number of instances using some like some kind of function so later on, I will show what functions you can use in making controllers.

A

So the first one is a proportional controller. Basically, then, the number of instances you set is the difference in CP utilization times a constant value. So here's a graph of that. So, for example, this is your target CPU utilization, which is let's say 70% and you're currently at 60%. So it means you need to add less instances. So so you add a bit of that and then your number of instances will decrease. The graph is inverted, but so there's an overshoot and then at some point it will settle down.

A

So in this controller, has certain properties like the higher you set your proportional gain, then you'll have more overshoot, but then you can have you can settle down faster, but then proportional controllers have a limit of having a large error margin in the end. So you the value, will settle, but it's a bit of your target.

A

Next, okay I've talked about that already.

A

Next, we have integral controller. So basically, the number of instances is dependent on the number of instances you have previously plus the lesson error value. So basically you're averaging out the errors over time and then multiplying that. So, if you're averaging your observations, then it means you're, reducing the the error difference, but then, since you're averaging as well, then your system becomes less responsive.

A

So another controller you can use to make it more responsive. Is you check for the the change in error? So if your CPU utilization changes very fast, then you can respond to it very fast, but since it's fast then it's also very sensitive to noise. So in practice people combine their proportional, integral and derivative controllers to get the performance they want in their controllers or in this example our auto scalars.

A

Okay, so I discussed all the types of controllers you can use for your auto scalars, so you have based off the model, but then, in the end, computers are not like fully deterministic, so there's nonlinear behavior as well. So if this was a perfectly linear time-invariant systems, then as I increase my number of replicas, then my CP CP utilisation should decrease proportionally, but then in some observations in generating my model, it's not the same value here. So there's like outliers here so even though I model it as a straight line.

A

There are error margin, so there's a need to reevaluate the model to check if it's really effective. So the same goes if you're using kubernetes built-in autoscaler. So you want to check if that the values you said, the target values you said in the HPA object- is making sense for your application.

A

So you can use control theory, not only in auto scaling, so people have like Auto tuning and auto configuration before so, for example, people have configured their engine acts or Apache services wherein they adjust the keep a lifetime out in response to the response, time of the application or in a or for nginx, like the memory versus the number of worker connections, so your queue is adjusted and then you can combine these parameters together to make more sophisticated, multiple-input and multiple-output systems.

A

So in summary, control theory is a model you can use to iterate on the feedback being received from your monitoring system. Then control theory also gives you a math framework to show if your feedback loops are effective.

A

Even next. Also simple models can go a long way as well, but then, in the end, like it's also important to keep reevaluating your models if they are effective.

A

So if you want to learn more about control theory, these are two books. I recommend the O'reilly book on the left is a good introductory book. It has like simulations in Python that you can test out. The W book shows like very concrete examples, so that's where I got the the keep a lifetime out and response time auto-configuration, but this is like more like rigorous textbook ish academic stuff. But if you want something concrete to do in your experiments, then this will get your feet more wet.

A

Just a few acknowledgments. The demo I'll show later and the benchmarks I did before were run on gke through through developer relations, credits provided by Kelsey Hightower, okay, I'll show a bit of demo, so here I have my application, which is.

A

Which basically serves a web app request and then I have a load generator. So this looks like the thousand women 1 million requests per second demo from Cooper net from Cuba I think last year, I.

A

So first we need to generate our model, so so I make a model by so by setting the number of instances up and down using a simple sinusoid.

A

A

So basically, this is a graph I used to generate my model, so basically I scale up and down the number of instances against a constant traffic and then the CPU utilization responds inversely. So you can see that it's not a perfect counter sinusoid. So that's why there's a non-linearity in the behavior so to show that I'm, not using kubernetes built-in autoscaler I'll, just show that okay I'll just show off my sleeves are empty right.

A

So after generating the model after getting the data, we generate a model. So I just made an R script that basically.

A

Generates the model from the data.

A

So I get my model, so basically our model is that the CPU utilization is the next value, is 0.94 of the current utilization and minus 0.2 of the number of instances. So that's the so we'll use that and then we will use that to identify what makes a our controller stable. So I already made controller for that.

A

So here's the controller loop loop I, showed earlier, and this is the controller so basically I set my proportional gain to minus 0.1 and my integral gain to be zero. So it's a simple proportional model, and you can see my controller here, okay, so basically the number of instances is the proportional gain times the difference in CPU utilization, okay, okay, so I'll just show that these libraries here are simple API calls to through kubernetes. So our sensor that gets the CP utilization is a basic API call to hipster.

A

So it gets the list of number of pods in our object in our application and then gets the CPU and then the actuate or the NAM. The one who sets the number of instances is just a an API call to the replication controller.

A

So you can see our controller running here, so it's taking a reading of the CPU usage and then setting the number of instances. So you can see here.

A

So here I have 71 instances, 270 CPU cores or is that merely CP was whatever hipster is giving you so so that's that and then it's receiving a traffic at 20,000 queries per second, so I will scale so currently, I have 20,000 queries per second, so I will double that 240.

A

A

Double that so we'll just wait a bit for the number of requests per second to go up, and then we can check again on our monitoring system.

A

A

Okay, all right, so the requests per second is starting to pick up now that we generated more load generators. Next, we'll go on to our controller, so the CPU utilization will then start to increase because we're sending more requests.

A

So currently it's at 273, then later it will increase. As the number of the pads in our replication controller starts receive more traffic, then it will start to consume more CPUs or so there you have it we're now at 400% CPU usage. So we can check our.

A

Autoscaler here so the number of CPU increase and then our number of instances is increased correspondingly, so.

A

Yeah and then when I scale it down.

A

Okay, so once the traffic decreases, then the CPU utilization will go down, so we can just confirm first if our traffic goes down. So it's starting to go down.

A

To beat a bit more.

A

A

I can just not wait for it, so so that's part of like controller design yourself. When do you want to respond, so you can also model the delay in your measurement, so the reporting in hipster doesn't really register right away when you add more instances so other, so it won't register the CPUC. So sometimes there's delay in the control loop as well. So when designing controllers, you can model that and design around that as well. So if you want to check it out, it's in my github account.

A

So it's a Espinoza, slash control theory, so you can play with it. So that's it! Thank you for listening and I can take.

A

Anyone have some questions or something yes,.

A

Yeah so the default I looked at the autoscaler code while working on this, so that so so again, sorry the question was: what's the model being used in the default autoscaler, so it's sort of a proportional controller. If you look at the code, it's basically they check the CPU utilization and then the number of instances is I forgot what's in the code, but it's I think it doesn't. It doesn't do averaging of some sorts. It does averaging at the monitoring level.

A

But so, if you probably put the model together, then it can have an integral component. But if you look at just the control loop, not the sensor leading one, then it's mostly proportional.

A

A

So right right, so the question was: is it possible to generate yeah to make models using really world data like pokemon goes so so when I made the the sinusoid like this was well I, pre-recorded it but I scale up and down the the instances against a certain load and then I get the linear relationship. So when you want to get a model from from the from the real world, then you you can you you basically get the two matrix you want.

A

So, for example, you have your Pokemon go load, so you get it from your access logs right like the requests per second and then you get the other variable, which is the CPU load. And then you basically do a linear correlation between two. So you just basically all the data you have there. You just plot it. On a scatter point.

A

Right right so yeah.

A

So the the thing about control theory is that so, for example, a proportional model doesn't really take samples, so it's it's pretty much set and then you can yeah haven't really thought about.

A

But yeah, so if you have data, then it only works in a certain region, so you design against that operating region and then just get enough data around that operating region. So, like your typical operating case right and then, but then, if your traffic loads too much, then your model is already invalid.

A

So you need to regenerate yeah, yeah yeah, so I think some people are doing that so that you you they combine multiple control loops, so they have a the fast loop which does stuff a lot of scaling and then there's a higher control loop, which gives parameters to the controller based on other data, which is sampled at the larger amount of time. So they do that kind of model, combinations.

A

Anyone else yeah, so thank you, everyone, oh there soon.

A

A

Right right right so like so, the question was: is: can you make a more predictive control loop? So so the answer there is that so what people do is like for, for example, you're planning an event. Then they change the target value so norm in my normal operation, I'm targeting 70% utilization, but then on further. We can, for example, for Black Friday I change, my target to 40% and the system will respond correspondingly because you change your goal so another one.

A

So the the one I mentioned where you have multiple cascaded control loops, so they use that set up to change the target as well. So you can do a more sophisticated model that way.

A

A

Yeah yeah yeah yeah, it can be just like on so an unopened, an open control loop, which you basically set the levers manually. So in that case your outer control loop is actually a human.

A

Yeah, thank you.