Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Autoscaling Intro – Marcin Wielgus, Google, & Solly Ross, Red Hat (Any Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)

SIG Autoscaling Intro – Marcin Wielgus, Google, & Solly Ross, Red Hat (Any Skill Level)

Come join members of SIG Autoscaling for a look at a "big picture" view of autoscaling in Kubernetes. We'll briefly discuss the different types of autoscaling (horizontal, vertical, and cluster), how they fit together, and where to use them. 

About Solly
Solly Ross is the SIG lead for SIG Autoscaling, and works on Kubernetes and OpenShift at Red Hat, with a focus on autoscaling and metrics. When not dabbling in other parts of the Kubernetes code base, Solly enjoys a wide range of programming topics, and can be found most places online as @directxman12.

About Marcin
Marcin Wielgus is a Senior Software Engineer at Google. Marcin joined the internet search giant in 2010 and since then he has been working on various projects, ranging from Android applications to recommendation engines. Currently he is a Tech Lead of the Kubernetes Autoscaling and Zero-ops teams. He also acts as SIG-lead for SIG-Autoscaling.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So let's get started all right, so this is the cig auto-scaling introductory session I'm Sally Ross direct x-men 12 on github and.

B

Hello, I'm, marching I'm, a tech lead of the auto-scaling team at Google and also I act as a co-lead of sakata's can occur with.

A

B

Okay, so they're our first question that people ask in their life. Some of them are evil like what should we eat today for lunch, some of them are changing your life up upside down. Like will you marry me?

B

Some of them are asked for just pure speculation, like whose article ad today will focus on something much much more profound, something that can be compared to the famous question of life universe and everything asked by Douglas Adams in his Hitchhiker's Guide to galaxy the question that my team asked itself every single day, which is how much resources do you need in your cluster yeah?

B

If you ask this question to developers, you will get various answers, you can get I, don't know or it depends or we need to run some more tests or you can get a semi random answer like 42, just like in Hitchhiker's Guide. This 42 number might be way too large and probably the guy who said that overestimated the needs. Sometimes, if you are lucky, the developers run some tests. Sometimes this test were based on the correct version of the application and rafi impression of what the traffic would be.

B

If you are lucky, this estimates might be quite accurate, but quite often they don't include the very, very important factor. The change in the modern software engineering world version change all the time. 2News delivery pipeline pushes the new versions of your applications over and over the new version might either be faster than the previous one, or introduce some performance regression that not necessary might be caught in the tests.

B

The traffic might change some real-world events may cause a sudden inflow of requests during a new launch feature may make the application much more popular miscommunication between various engineering teams may lead to inappropriate use of the API and so on and so on, and these changes may cause you a lot of problems. Your application might become unresponsive crash and drop same grief and serious embarrassment on you, not to mention financial impacts on your company and on your personal career.

B

You may have a superhero on your team, someone with special powers, full route access, unlimited budget on your cloud provider, a friendly spider-man from the neighborhood who will fix all of your resources and applications. However, he may say if you once or twice, but superheroes, get tired and of saving your butt over and over. They also want to have vacations. They want to have personal life. They want to see, have sick leaves, so you should not strongly depend on them.

B

You need a robot, a robot that will take care of your cluster and your application day and night, someone that doesn't need vacation and doesn't get tired, a robot called auto-scaling. The mission of this robot will be to automatically adjust resources to match the current needs. But what does it exactly mean?

B

The first goal, auto-scaling, is to save you from this a total catastrophe, crushing application, outages one-hundredth, which requests increased, allowed or performance impacting changes. The second goal of outer scaling is to save your money based on the first goal of auto-scaling. You could just provide it an insane amount of computational resources, balls and stuff and call it a day. However, you don't want to pay an insane amount of money.

B

You probably you want to pay just enough for what you really need, so the ultimate goal of auto-scaling is to keep the right balance between what you need to not to have your application crashing and what you really need in terms of spur capacity and resources.

B

So, let's focus on this problem: how to maintain the decent load in your application. You can think about pods and application as pipes. There is some water traffic coming in for the pipe. You don't want this pipe to explode due to high pressure or to be underutilized and to be wasted, investment, and you can imagine that on every pipe, you have this small matter with a couch that shows some metrics.

B

There are some valves on the goals that may indicate problems like if the load is starting to be too high for the pipe or actually for the pot itself. For example, it may happen that the pot is using more CPU that it has originally requested.

C

B

The load goes above the safe threshold. It might be beneficial to start a new pot. Luckily, starting a new pot is much much easier than lying. A new pipe thanks to deployment stateful starting a new pot is more like cutting a cookie with a cookie cutter.

B

Let's take a look at the more concrete example. Let's imagine that this gouge shows the ratio between how much resources the Petrelli uses and what the pot has initially requested. So here we have four pots. Each of them is running really hard to use 1993 percent of the requested capacity. This is not a safe situation. If a sudden spike in the traffic comes, your application may not be able to handle this additional traffic and the whole application may become unresponsive.

B

However, if we add one more pod and the traffic doesn't change, then this is quite likely that, due to the construction of services in kubernetes, the traffic's will go to all nodes to all posts equally, and in that way, the load on the other pods will go down, so it will go to the level of 70%, which is much safer than 90.

B

Something in this if the situation is different and your application is not using that much of the requested capacity it might be, it might be actually beneficial to remove one pod and have a smaller deployment, less pod and more resources in your cluster in general. We want to specify a target, not too big and not too small, something that we will be happy with. Then the autoscaler will try to add or remove pods, to move the load more towards the desired value, and that's what horizontal pod autoscaler does horizontal put.

B

Autoscaler is built into the core kubernetes from quite a long time. It adjust the number of replicas, it would maintain the target load. But what is the load exactly for a quite long time we considered a load as the ratio between what the pods are using in terms of CPU and what they have originally requested, because the underlying monitoring infrastructure allowed us to do only this, so the architecture looked more or less like this.

B

There was a hipster which style called metric server that was scraping couplets for the container metrics and he created a horizontal path. Autoscaler. That was talking to hipster to get the CPU utilization data and based on this data to appropriately scale up and down deployments with bots okay, so we have CP o based scaling. However, CPU utilization is not the only way to express the load for some people. This mass metric is requests per second for some people that run, for example, some batch processing job.

B

The right metric is the queue size for the others who are more focused on s, allows the right matrix will be the response latency, and for these reasons we introduced custom metrics with custom metrics user alt allowed to point horizontal pot autoscaler to some metrics that you have in your monitoring solution and scale based on it on gke, the metrics might be installed driver on some other cloud provider or on pram in Prometheus or in something completely different okay.

B

So it works like this. The pot on some other entity pushes a metric to a monitoring solution.

B

Then we have horizontal pot autoscaler trying to get metrics through a dedicated customer tricks, adapter from your monitoring solution and based on that it corazon ttle, put autoscaler controller, adjusts the number of replicas in your deployment.

B

Ok, how to use it. I said that horizontal pot, autoscaler, is controlled by the API object best draw, so you can create an horizontal pot, alter scaler pointed to a deployment and configure a matrix for it. The metric can be CPU load or a custom matrix. He. You also need to specify a target value for the matrix, and you can also provide a cup of matrix.

B

If you want to base your auto scaling on multiple inputs for CPU loads, you can also use a dedicated cube, City Auto scale entry for for more convenience, but for custom metrics. You have to play with Yama files when to use it. A horizontal pod autoscaler is best with stateless serving workloads, so workers that can be easily started and killed things that don't have too much data that don't need high initialization time and workers that can add or delete posit.

B

Will you may base your scaling as a set on CPU load or some metrics, and we have a couple requirements for the matrix? The matrix should be of a gouge type, so it should change in time up and down and hopefully be quite responsive and react fast to the changes in in the upcoming traffic. Horizontal pot. Autoscaler doesn't work with metrics like counts, or it doesn't work with histograms. So in order to properly use it, you need something that goes up and down basically histogram like a speed on in your car.

B

Horizontal pot autoscaler is not best suited for stuff like stateful set, although with the appropriate car it might be used to scale it.

B

There are two talks about horizontal pot out of scaling with custom metrics, so has not give more details about it. I will give you pointers to these talks later in the presentation, so I showed you one way of dealing with the loads in your pots. Let's take a look at the other one. Okay, that's me couple days ago, I had a family event for which I needed to fit myself into a shirt tie and a suit and I realized that, unfortunately, my shirt is too small.

B

For me, it would be very unconvenient for me to be in a small circle, especially around the neck, and there would be a risk that the buttons will not hold a lot and pluck off and kill someone either way bad. So horizontal auto-scaling was not an option. I couldn't get another elliptical myself, I'm a single ofin. Basically, so in kubernetes we have similar situations.

B

Some containers are either too small or too big to appropriately called the content, and for this low workload we have introduced vertical path: autoscaler vertical pod, autoscaler adjust container sizes based on the real world usage. Let's take a look at this case. We have a similar pattern. Previously, it's consuming 95% of its resources in horizontal pot autoscaler. We will add yet another pot in order to split the load between two. But, as we said, it's not always possible. So in vertical pot autoscaler, we are doing another trick.

B

We are increasing the pots increasing its request so that suddenly this 95 percent load becomes a much much healthier. 70 percent load.

B

Vertical pot, autoscaler architecture is quite simple: we have a recommender that provides recommendations about the size of the pot. We have an update err which looks at the odds and checks whether the current pots have the right requests set and if not the it, the auditor kills. These pots and the kill pots are recreated by their controllers. They go for all the admission plugins in API server and in that admission chain we have our own admission plug-in that takes recommendations from the recommender and applies them to the newly created pots.

B

Okay, so vertical pot autoscaler can run in three different modes. It can give you recommendations on mean that it will give you hint about the proper size of your pods and you'll have to apply this hint manually at your convenience.

B

It can apply the recommendation at the anytime, which means that whenever you create a new pot, it will get the appropriate resources. However, the VPI will not touch the existing pots or you can trust it fully and go full speed with vertical pot autoscaler and turn on auto mode. In this mode, vertical pot autoscaler will check whether the currently running pots have the right request and, if not, it will update them, but, unfortunately killing them.

B

Okay, so how to use it in a similar fashion, as horizontal pot autoscaler vpa is controlled by API object right now, as we are in alpha, where the API is based on customer customer resource definition CRD. So you need to create a CRT with the appropriate data. This data include the label selector for pots, which you want to auto scale. You need also to specify what mode you would like. We pay to operate in, and possibly some extra data for for deciding which containers should should be updated by BPA.

B

You can also provide minimum and maximum amount of CPU and memory that VPI should give to to your current containers.

B

More data details about a vertical pole. Autoscaler are available on our github page under under this link when to use it VP is suitable whenever horizontal pot autoscaler is not suitable, so it can scale the workloads for which you need to have a fixed amount of pots. For example, your stateful set or wore clothes that have just a single pot and workers that you cannot basically eat, to which you cannot basically add more pots.

B

Can you mix a horizontal pot, autoscaler and vertical pot autoscaler? Well, not really in not yet, and not always, horizontal pot, autoscaler and vertical pot, autoscaler kind of try to solve the exact same problem so how to deal with high load on your pot? How to deal with situation when the load on your pot is not so big and the requested resources are simply too big and your money is wasted. So when you apply both of them, they may cause.

B

Your workers to scale up too much or scale down too much in both cases, it's a little bit dangerous. You can, however, try to combine a horizontal pot autoscaler with custom metrics with vertical pot autoscaler, it's a bit safer, but it needs extra consideration and should be investigated or in our case basis.

B

Hopefully we'll have a better story about this in the future. Ok, so you have either a couple of extra pots or some bigger pots, and now you suddenly realize that, actually you don't have that much space in your cluster to hold all of these extra pots. What now with it like a dead end, no you run on a cloud and cloud can be consider as a huge ship. What you need to do now is to get some extra capacity at the right time and release this capacity when you don't need it anymore to bring.

B

This brings us to the very, very important point in seek out the scaling agenda. We want to automatically ensure that you have enough capacity in your cluster, but what exactly is the right capacity? So simplifying a bit kubernetes scheduler schedules pause based on their declared usage, for example. On this picture we have four nodes: three of them are quite busy. One node has just a small pulse and it's quite likely that this green one will fit into the third node like here.

B

However, if there is no space left in the cluster and all nodes are busy, then the only option is to add yet another node and when did note arrives, scheduler will reconsider this green pot and possibly put it on the newly added node. If the situation is different and not all nodes in the cluster are well utilized. For example, here we have only a small pot running on node number three and a small pot running on node number five.

B

We could move the green pot to the third node, freeing fifth node and making it available for deletion, and that's exactly what cluster autoscaler does so. Cluster autoscaler has two points on its mission. Whenever cluster autoscaler cease pending pot, it checks whether it would help if it added yet another node to your cluster. If the evaluation is positive, it talks to your cloud provider and buys you a new node and when did node finally arrives scheduler reevaluate the pot and puts the spending pot on the newly added node.

B

In the same time, a cluster autoscaler constantly monitors your nose and whenever your nodes are not that much utilized, it looks deeper into this node. It considers what would happen if the note was removed, whether all the paws that are there would be able to schedule elsewhere. Would there be any impact to important system components like DNS or hipster?

B

Would it violate some of the pot disruption budget and if the answer is negative for all this question and it is safe to remove the note and the note is unneeded for an extensive period of time- I mean about 10 minutes, then the note is removed. Reducing your monthly bill on your cloud provider.

B

There is an important distinction between a cluster autoscaler and horizontal and vertical tot autoscaler. Cluster autoscaler is based on scheduler, so it's based on pot constraint, sins based on the kubernetes internals, like both disruption, budgets and system components, and it doesn't take into account, for example, how what is the exact load on your notes? So if your notes are burning hot, but your pots have wrong requests, it will not help you helping. You is the mission of horizontal as vertical pot autoscaler.

B

There are there for making your request right or making the load spread across more pots than you originally have.

B

Okay, so how to use cluster autoscaler on dku, we have a single common that will start everything for you on other cloud providers. You need to execute a couple of manual cube serial actions, but everything is documented here. So it should also be quite easy for you to set up the fink. We have a support for a Google cloud, Azure and AWS when to use it.

D

B

Always unless you don't want your cluster to be scaled or you don't want to have like a changing bills, but most of big clusters are running cluster autoscaler right now, and they say that it's a good choice for them or you cannot run cussed out the scaler. If, obviously, your cloud provider is not supported, okay, so now is the time for a little bit of status report. Where are we with all of these efforts?

B

Cluster autoscaler is in GIS in September 2017 in scales, more or less to 1,000 nodes and supports 30,000 pots. It can go further, however, with slower response rate horizontal pot autoscaler is in GA, I mean the course of a potato scaler based on a CPU load. Custom metrics are currently in beta vertical path. Autoscaler is in alpha. We launched it in late March and will hopefully be pushing it more towards, but sometime this year. So what is missing there in vertical path? Autoscaler?

B

We hope that all problems with your request will be automatically solved, and one of these problems can be out of memory errors that can happen on your notes. Unfortunately, the current support in kernel and in cubelet is not sufficient. We don't have the right reporting. We don't have the appropriate logic in the kernel, oome killer to do, reporting and handling a properly. This needs to be fixed.

B

Moreover, currently it is not possible to update pot requests without actually killing it. So, even if you have enough capacity on your note to increase the pot, you need to kill it so that the kubernetes god is satisfied.

B

Eventually, we hope to have this in place upgrades in place, so you will have a much smoother ride with vertical path of the scalar horizontal photo. Scalar could possibly movie more configurable. We have a lots of hard-coded constraints that we are hoping to get rid of and currently custom metrics are has support on yon for to a monitoring solution, namely study driver permit use. We would okay.

A

Sorry about that there's actually an adaptor for cystic now, I believe it's closed source at the moment, although someone from sis can probably clarify, but they posted a blog post about it and if you google, for it, you should be able to find it. I didn't.

B

Know it that's great, but if there are any monitoring solution employees around here, please please, please get your monitoring solution integrated with horizontal portal to scaler. You.

A

Can come up and ask me for details about how to get started about that later and there's some helper repositories around. That should make your life easier.

B

What does there are dedicated autoscaler for system components? Cubed DNS has one hipster has one, and ideally we would like these outer scalars to be unified under horizontal port autoscaler and vertical patel to scaler umbrella.

B

Okay, this stock is slowly getting to an end. So I would like to give you a little bit more details about our seek so.

B

We have meetings every Monday at 4:00 p.m. copenhagen time on Zoom. We have a quite active slack Channel and most of our stuffs is in kubernetes / autoscaler repository on github, and there are other auto-scaling related talks on this cube con. There is autoscaler kubernetes workload with tomatillos at 11:00 10:00 tomorrow, horizontal put Auto scale. You know that scale on custom metrics at 4:35 also tomorrow and seek out of scaling deep dive on Friday at 4:25 p.m.

B

thank you and now is the right time for questions.

A

B

A

D

With the horizontal podcast autoscaler, do you notice, if, like you, are increasing the number of pots, but it doesn't help like that would mean that it continues increasing and increasing right. Do you notice the situation.

A

Yeah, so right now we expect that your metric will have to react to that. So if you use a metric that doesn't react to a change in in pod count, then it will keep scaling until you hit the limit. So in the horizontal pot autoscaler object, you can set a maximum and minimum number of pods and it'll keep its scale between those pots. But if your metric does not change right now, it'll keep attempting to scale as long as it thinks it has allowed to so.

A

This is this is a very important point of the horizontal pot auto scale or make sure you have a metric that reacts with your pots.

C

Thank you for the talk. First, yeah I have a question because I've been reading about the cluster of the Schuyler and also the audience on tests and I show that there is like different configurations running in the master or running in the in the workers. So what is the Pro and the cons to run in so non way on the other? So.

B

In you can you we are on the cluster autoscaler and master, because we take care of the Masters and we have the monitor it so, for from our point of view, it's better to run your cluster to scatter in such a way that is always available on gke. We make sure that master notes are available. If you think that your notes are more robust and more resilient, that your master note, then you can run it on the worker notes.

B

However, in most cases, master notes are really really work hard on it, and for that reasons, probably it's better to run a little master.

B

E

Hey thanks guys. Does the vertical part autoscaler play nicely with jobs, perhaps in recommender mode, so I'm sending jobs in can I get some numbers out that hey these jobs are feelings, are out of memory.

B

Yes, to some extent so vertical pod autoscaler doesn't have a response pointer. It uses a label selector, so you can select arbitrary set of pots. So if you want to scale vertically your jobs, you can do so by putting the appropriate label select order.

B

F

How do you go about actually kind of identifying appropriate limits to set on your containers of Danya pods to start with.

B

How do you identify yes.

F

I mean you with the vertical part or the scale in recommender mode cool. That's fine, but you're working with a set number of pods there. So the the recommendation there may not actually be that helpful. I guess.

D

F

You're trying to scale outwards, so how do you determine the optimum resource level to request for your pods to like at the outset? Okay,.

B

So how vertical photo scanner works? Is that a divert beginning whether it has no data? It doesn't do any actions in in time when it gathers more data, it has more confidence what the appropriate request should be, so in vertical photo scaler we have a field for status. The status contains three numbers. The one is the desired value for a particular resource, for example CPU and memory.

B

The other two values are minimum and maximum values that are still okay, so we start with very small minimum and very small maximum and in time we are reducing these values and updated when it sees that samples are not within these values kill the spots so only after we get enough confidence that they have wrong requests. They are killed at the very beginning point we have no data, it's quite likely that will not make any decisions.

B

We will make some decisions when we notice that, due to initial recommendation of, for example, we do to what you put initially in your pod, the post, our ooming, so they are using much more memory than you thought. So then we will react, really really fast and increase. It.

B

Any one of us know then I would like to thank you for coming and see you on the other artists killing session.