Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Do The Math: Auto-Scaling Applications with Kubernetes - Antoine Hamon, Nephely

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Do The Math: Auto-Scaling Applications with Kubernetes - Antoine Hamon, Nephely

What better place is there to host applications but inside a Kubernetes cluster? Yet what feature is more under-estimated than pod auto-scaling? Yes pod auto-scaling can be really hard, especially when searching for an optimized configuration. Whatever the application, goals are always the same: making sure our pods are able to scale-up quickly enough so users don't ever face any latency nor HTTP/5xx, while keeping the cost as low as possible. As both goals directly leverage the other, the complexity is all about finding the correct balance. Instead of putting the regular 80% CPU utilization threshold (which is sometimes fine-tuned with load-testing and/or feedback monitoring), would it be possible to have a more mathematical approach to solve this problem? During this talk Antoine will present one he created and explain how to utilize it within Kubernetes.

https://sched.co/ZelV

A

Hi everybody, my name is antoine ammo and I'm very pleased to be here for the first vietnam cubicle to talk about how we could use mathematics to better configure auto scanning.

A

I'm a freelance cloud architect and devops, I'm mainly playing on aws and openstack, and I'm currently working for an american based company named pipestream as a developer leader.

A

This story, this talk, is actually out of an article I wrote over a year ago, so I definitely recommend you to check it out. If you need some more details, there's also the link for the web app created that I'm gonna introduce later.

A

It's not quite a surprise that the micro services architecture continues to invade the software design. It's easy to manage easy to develop your smaller team, it's better at distributing the load. It's easy to create hd deployments and easy to manage upgrades, but the game surely won't be the same without cloud orchestrators such as kubernetes as they bring to the table a lot of useful features to manage the lifecycle of those microservices and obviously one of them is auto scaling.

A

What a blessing it is to watch your deployment's number of replicas fluctuating all over the day, gently sized to the current load, but don't get me wrong. Auto scanning is hard. A lot of parameters are involved in the process such such as the minimum and maximum number of replicas, the upscale threshold, the sync period, cooldown delays, etc.

A

So I wondered instead of keeping those parameters to their default value and set the well-known 80 percent as the upscale threshold. Would it be possible to find a more mathematical way of saying those parameters.

A

First, we need to set up some goals. We want the scale up to be fast enough, so users do not fix any error, meaning with a given number of running replicas. If you have a rapid load increase, we want the running replicas to be able to handle the load until new replicas are created.

A

We want a minimum number of replicas to be running, so this is in order to save some bucks and we want the model to be working for every given load variation and that's all tied down to fine tuning the upscale and downscale threshold and, as you may know, communities has a single parameter.

A

Let's try to translate those goals into a more mathematical problem.

A

As we said, we want the model to be able to work for all type of load variation, so that should also include a worse case scenario, but what could be a worst case scenario to me, a good definition could be to have a great increase in the number of users doing a resource consuming operation within a short period of time.

A

Let's say you are a cncf administrator about to release kipcon tickets and obviously expecting thousands of users buying them within 30 minutes.

A

Let's set some variables, let's say we call any. The number of users t taught this short period of time: the function. L, u of t, is a load generated by a single user and the function at total of t solar generated by all users on the system.

A

We are also going to introduce some hypotheses.

A

We are going to suppose that the loss is evenly distributed across all replicas, that your application is stateless, meaning that all requests are round-robin across all back-ends that the average request timing is shorter than the kubernetes load check interval and that you are dealing with a large number of users, and here we dive in you can get your aspirin ready in the mathematical world whenever you have a large number of events occurring almost at the same time, this even distribution follows a gaussian or normal distribution, which has the following formula: where nu is the expected value?

A

Changing? U will actually translate the curve on the horizontal axis and where sigma is a standard deviation, meaning. How steep is the curve- and we are dealing here with our first issue, because in the mathematical world, if you want to cover all events, you need to cover an interval from minus infinity to plus infinity and that's obviously not computable.

A

But if you look at the curve, you will notice that 99.7 percent of all events will occur between minus three sigmas and three sigmas.

A

So we can call it close enough to only consider that interval, even though we are choosing the interval minus three sigmas to three sigmas for the simulation, we are not going to choose the total terminal to be equal to six sigmas because, as you can see on the graph over 95 of all events occurs within the four segments.

A

So choosing the total time to be equal to six sigmas will add half the time compared to four sigmas for only four percent of users, which is too few. So let's choose the total time to be equal to four segments.

A

Hence sigma is equal to the total time divided by four, and since we want the curve to start at the beginning of the simulation, nu is equal to three-quarter of the total time.

A

We now have redefined the problematic into this following sentence.

A

We are going to calculate the law generated by 99.7 percent of all users, each one of them performing a resource consuming operation and where 95.4 of them are doing it within a short period of time replacing sigma and u within the gaussian distribution gives us that formula, and we have a second issue here, because we're going to need the integral of the equation, and there are no mathematical formula for integrating e with the polynomial exponents.

A

To work around the second issue, we're gonna use the riemann sum.

A

Riemann is a german mathematician who has proven that the integral of function is equal to an infinite sum of infinitely small rectangles, and that is going to be very helpful for us, because now we can consider that all users within a small rectangles have started their operation. At the same time,.

A

I don't want to go into too much details about the calculation, but we can now define the total load function as following after replacing variables with our current values gives us that results.

A

So we had succeeding in having the total load function, defined as a formula involving the number of users and the user load function.

A

All that we need to do now is to use a dichotomy algorithm in order to find the highest upscale threshold, where no replica ever reached its resource limits all over the worst case scenario. Aka, the total load function.

A

There is one last thing that I need to cover with you before we do the demo, and that is how we could create the user load function.

A

We have to know it's impossible to have a proper mathematical representation of the load, but onto a system by a single user, and that is also also true if you are not running real-time operating system such as linux and others, but I came up with a procedure that is to me close enough to be valid and the idea is following: if you have a large number of users repeatedly doing the same task at slightly different time, you should see the application resource consumption to be pretty much flat.

A

So all it takes to calculate the impact of a single user is to divide the resource. Consumption of the application by the number of users calculate average. Request. Timing, repeat this procedure for all tasks that you want to include. In your user scenario, add some timing and voila all right demo time spoiler alert. Since the demo is pre-recorded, everything's gonna be fine.

A

We will do two simulations for the first one. Let's say the resource limit is at one cpu: it takes 30 seconds to start and we have a minimum number of instance of 2. for the user load function. We will consider a user scenario with only two actions.

A

The first one would be something similar to a user logging that would take around two seconds and it's quite cpu intensive because of the cryptography.

A

The users wait sometimes and then upload a file that takes a bit longer, but is less cpu.

A

Intensive in order to lower the compute time we specify to the application. How long is the user load function? Then we specify our t dot and nu.

A

In our case, ninety-five percent of a hundred thousand users are doing the action within 15 minutes.

A

Then we choose the orchestrator. Those are the default values and as soon as you made sure that you had a sufficient number of points for the computation, you can run the test.

A

It takes a bit of time. I actually cut the video, so it doesn't take too long here, but it will eventually get you. The results.

A

The first graph gives you a representation of the user load function. The second graph gives you a representation of the total load function, and the last graph gives you the result. The orange curve represents the load appearing stance.

A

The dark blue curve represents the number of replicas that are running and ready, and the light blue curve represents the number of instances that are not yet ready.

A

So the upscale threshold is at 54.4 percent, which is quite low. The maximum number of replicas that are needed to handle the load is 78. and we have a maximum load per instance. As 94.6 percent, we are going to change some of those values, so we can highlight the impact they have on the system.

A

We are going to double the application resource limit, decrease the application, start duration and increase the minimum number of replicas, since the application takes less time to start, we are going to lower the readiness delay.

A

So, let's see what impact those change had. First, we can notice that the upscale threshold has a much higher and acceptable value.

A

Second, we see something rather interesting, where doubling the resource limit on the application divided by more than two the maximum number of replicas needed for the simulation as it was previously 78, and it's now 27.

A

I will finish this presentation by giving you some advices on how you could improve the upscale threshold value.

A

First, I, by far the most important or the one that has the most impacts. Your application setup time should be very fast, the faster the better the web application do not as of yet separate resource requests from resource limits and, as you may know, kubernetes calculates the auto scaling against the resource request.

A

So increasing the resource limits will give you a little more time to handle the load.

A

You can also increase the minimum number of replicas, and this is because the highest load increase per replicas occurs at the beginning of the simulation, and you can also increase the resource limitation on your application.

A

That's it for me. If you have any question, I will be very pleased to answer you. Thank you guys.