Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive into Autoscaling - Marcin Wielgus, Google & Vivek Bagade, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive into Autoscaling - Marcin Wielgus, Google & Vivek Bagade, Google

Come and see how to debug and optimize your autoscalers and decrease your monthly infrastructure costs even further. During this talk members of SIG-Autoscaling will discuss the internals of HPA, VPA and Cluster Autoscaler, their peculiar features, and ways to fine tune them across dimensions like cost and availability. After this talk you will know where to look for information about the autoscaler activity, what settings can be changed and which flags should probably be left alone.

https://sched.co/UagF

A

Everyone, it's great to see everybody interested about auto-scaling on the wake from Google to take a deep dive with everybody in the order scaling right. So there was this great presentation yesterday by my colleague Marcin about an introduction to order scaling and heaters upon the best practices. The types of auto scale is what they do and everything so when the slides and the recaps do come out of all the presentations do have a look at that, but for the uninitiated, I will touch upon some of those or introduction topics.

A

So taking resources for your applications is hard or they're, never standing still, there's always more traffic. Less traffic Black Friday's are the boundary Monday coming up. So it's difficult for you to set a decent or even you know, working limit for your for the scaling personally as a developer, it was really hard for me to be nice. Auto scaling does help you with this. It gives you or three types of water.

A

Skill is with three different purposes, or so we touch upon that a little bit here or to get the right amount of resource limits and request for your pods. You should probably go with the vertical orders order to scale it and to set the right amount of replicas that your application needs. You should go with HPA and to set the right amount of notes in your cluster. You should probably use less you're a scalar, so Oh VP a what does it do so?

A

How do you know the right amount of CPU or memory that your workloads need? It can be really difficult to figure that out, especially if you are starting out so VP, a actually transfer or historical performance of your application and the utilization or the utilization of resources.

A

Your application has, across time and I, do recommend to write resource limits or actually sells them for you, or this can actually alter your workloads to said our optimal resource limits for you, horizontal quadrotor scaling or so again figuring out the right amount of replicas that your part that your deployment needs is hard. Especially you know our production situation so or HP helps you with that.

A

It targets a certain target utilization that you that the user sets and if the utilization exceeds that, if your current utilization exceeds that or be scaled out by adding more replicas and effect or if your target, if your current or utilization is less than the specified target, it reduces the amount of replicas. So basically it pushes your current utilization towards the Stargate utilization. Oh.

A

B

A

The scalars, so how do you know how many nodes your lessor needs at this point of time? At any point of time, it can be extremely difficult because you might have something like HP, a turned on which increases them or reports, and do you really have room for your extra part now onto your of onto one of your notes? Maybe yes, but if you do or aren't you or provisioning somewhat, you can sort of save caused by only using nodes when you really need them so CA. Turning on CA can actually increase your cluster orders.

A

Sorry plus your utilization by quite some or level, so we call so scale down. So if your or if we scaled up and then it turned out that your workload is done and now you might not need the new it anymore, we actually scale down scale it down for you. So in this talk we'll take a deep dive into how you can use the cluster or a scalar to reduced your infra costs so because we add and remove nodes.

A

This affects your monthly in trouble. So if you do have a lot of nodes, you might have a bigger available.

A

Yeah all right, so, let's take to actually understand an optimized cluster or a scaler. We need to know what it does and how it does it. So, let's take the scenario where you have two node pools: a node pool is basically similar-looking nodes, a few similar degree nodes. So there are two node pools in this case. There's an NL standard, food and I know standard eight and one standard for has capacity for full CPUs, while TN one standard eight has capacity for 80.

A

So, given the scenario, we also see that these nodes are filled with blocks and these blocks represents you till it will be or the node being utilized by your boards. So if you take the first genuine standard food node, you can't see that it is sort of half filled and that basically signifies that half of it is being utilized by some of your ports already on it.

A

But above the yellow line you see, there are two boards that are of that request for CPUs and three CPUs each and they are not on any node the signifies that they are pending boards. They could have appeared from a new workload order, HP a or the scale workload could be or any of those or you know sources. So what cluster or descaler actually does is yeah.

A

So what plus you know to scale it actually does? Is it adds nodes for you so quickly? It doesn't seem to be working very well.

A

Yes, I'll stay closer to the laptop I. Guess right, so we actually add nodes for you. So in this case there are two new nodes that are added or the two NL standard for nodes that just appeared, and these should be able to house your pending boards.

A

So why did we scale out the n1 standard, for why did we add two nodes to n1 standard for nodes? While we could have added only one n1 standard eight, or can we also customize this behavior? Somehow the answer is yes, we can so cluster autoscaler. Has this concept of expanders or an expander is basically a a veil or an option to expand your clusters in this particular mechanism and each expanded?

A

Has this function called best option, which is which evaluates a bunch of options to ield the best option and each option in turn has a new group to scale out. It has the extra amount of nodes that particular node group has to be scaled by and the pending pods that will be housed by this scalar.

A

So, let's turn order. Scalar has a few expanders already implemented each with different parameters that are optimized, we'll go over them, so the first expander that we have is least wastage expanded. So um when you add new nodes, there is certain wastage in world, if you add them and underutilized them so V, basically, the sorry, the least least is expanded, actually pegs to nodes that are that have least wastage.

A

So we do have a formula here that basically helps with that. There's also the fact that it optimizes for options where you pay less no tax or because the mode notes you have the more tax you have to pay in terms of cubelet running on those extra nodes and also you know your demon sides and other ports running on those nodes. So we try to optimize for that, and this expired nice ways to expand. The next expire that I want to talk about is price expanded, so price expanded is slightly more complicated.

A

It optimizes food the cost incurred per resource requested by our pending ports. So if, in the earlier case, we had 7 or CPUs requested by our pending ports, and we want to optimize for the dollar cost of satisfying these sports, and we also want to optimize for the current cluster shape, so in general, it's better or to prefer larger cluster larger nodes for larger clusters or for reasons of management cheerin, and you also pay less tax.

A

If you have larger nodes, there are some formulas that I'll that you can do can do to understand exactly how it works, but I'll skip that for now. So the other expander that we have and I want to talk about is more sports expanded, so keep in mind that every option does not satisfy all your pending boards, especially if your pending pods ports are of different sizes. So most post expander takes the option that satisfies the most amount of ports.

A

Most amount of pending words or you can use this option when you need to scale out and schedule as soon as possible and satisfy as many nodes as possible or the last expander that I want to talk about is priority expander. So when we have options, we basically look at these options and a which will pick the option that has the highest priority specified in this particular expanded, and the priority here is specified by the user in a config map.

A

That would look like this, so in this case or the nvm for large machines have the higher priority. So we will try to scale that if that is possible.

A

So how do you pick the right call flow for your situation. How do you pick the right expander in most cases, just start out with your project price expander? It will work for the majority of the use cases and optimize what actually matters for you it in which might be priced right which might bid price it good.

A

So again, if, for some reason you feel the price expander is not optimal in your condition, then you can try changing it. Based on what you feel can be optimized all right, so moving on the scanner, also that scaled down for you, so take a look at this situation. This sort of looks like it like the earlier situation, but the cluster is pretty sparse.

A

Now it has a lot of emptiness on it and if you look closely, you can see and then standard four nodes which are completely empty and there are three and when standard eight nodes which could be, you know, optimized further they're, mostly empty. If you click take a closer look, you can in fact see that all these port, the three ports that are on the n1 standard nodes, can actually be fit into one and one started node. So that is what see it does for you so at equal to zero.

A

We analyzed your cluster and figure out which nodes might be unneeded. So in this case, your of two and one standard, four nodes and the three and one standard eight nodes are marked as I needed. These are your potential candidates were scaled down and if you look closely, there's also a simulation result: the CS c assimilate. If a particular, you know all the put all these pods in these unneeded nodes can be fit into our different destinations.

A

So we move on from here of two. So once a node is marked are needed, it has to stay, are needed for ten minutes or a particular window that can be configured before we actually attempt to scale it down or the stability purposes. We don't want to scale down too quickly, because if you do that, you take a hit on availability so and we don't scale down all the nodes at once, all the unneeded nodes as once, because that creates too much flux in the cluster.

A

So what we do instead is we try to scale down the empty nodes in bulk, so here the two and one standard, four nodes which are sort of empty. We try to scale down and at a equal to 10. We achieve that right.

A

So after that happens, we again don't remove all the in one standard, eight at once. We instead try to remove non empty nodes one by one. So we also wait so since a scale down happened at t equal to ten minutes. We wait a while before ria tempting another scale down, and this is for stability purposes again. We don't want to scale down all nodes, one all at once or in short delays that can severely affect your clusters.

A

Availability- and you will see this or- and this is the theme of the presentation, as basically cost versus availability, so at after the period of time, in this case that he folders ten minutes v re attempt to scale down and the second node the second in the standard. Node is scaled up and hence support on and go spending, but it can be scheduled on your n1 standard, eight, so everything's fine or we had a scale down again.

A

So we wait for a period of time before he attempting a scale down, and we do that at t equal to thirty minutes. The pod cuts spending on that node and then we have it. Oh, we have a very compact, looking Lester at T equal to thirty minutes right. So again, the theme of the presentation is cost versus availability.

A

You can queue in your cluster of and cluster order, scalar to sort of lean in one direction, and that is what we'll be talking about. But before that I wanted to point out, the CA is tuned for most of most of the serving workloads.

A

So majority of the workloads in general are serving workloads like a web server or some kind, and for that or the default behavior works fine more than fine, in fact, but if you feel like your workload is different somehow- and you have you think you could, you know tuned it better and make push it towards a certain direction or then you can tune it, so you can tune it in the direction of availability or the direction of cost saving.

A

So if you have a workload like, for example, a batch workload or so how are battling for different batch workloads show up all at once. The workload shows all up at once, or so you have to immediately and the just as fast as they came and they will get the workload completes and you would have to scale down quicker.

A

So, in those cases you might want to optimize see a further and we'll talk about how yes, so there are certain flags for optimizing utilization, nca and we'll go over them one by one, so the first one is scaled out unneeded time. So this is the time that a node has to be unneeded for before we attempt scaled on so having the higher value would basically protect you against flapping.

A

So if you remove a node quickly, it might so happen that a workload shows up, and then you have to scale up again, so there will be a lot of flapping in that case, so you would want to protect a keen side and the way to do it would be increasing this. On the other hand, if you reduce it, you will have faso scaled on potentially, and you would also have because of you know or higher see a reaction time.

A

The second thing I wanted to talk about is scaled oddly after delete. So after a particular scale down. We wait for a certain period of time for the situation to stabilize before we reattempt it via time scale down. So this flag affects this time.

A

Similarly, if you want to increase this amount or he will threaten to yourself against flapping and you would have higher availability in general, if you'd reduce this time, you would gain how fast is scale down. You would have fast. You see a reaction speed.

A

The third fact that I wanted to talk to you bird is the scale down utilization threshold, so um first know to be unneeded. How? How do we mark our node to be unneeded, so the utilization on that particular node has to be below a certain threshold for it to be deemed as unneeded, and this flag directly affects that value. The default is around 50%, so if you're a V market, we mark nodes as unneeded once they fall below the utilization threshold or 50%.

A

So if you have a higher threshold that allows for lucid packing of your notes because of scale Don's won't happen asked frequently, it also allows for the level of over-provisioning, so you can or privation the cluster will not look as compact, so you would have some level of over-provisioning which might be needed for availability reasons. On the other hand, if you decrease this value, you would potentially have tighter nodes. I have potentially also have higher utilization.

A

The fourth plans that I wanted to talk to you about is max empty bulky lead so as you've seen earlier see, it does delete empty nodes all at once right, but there's a limit to that. So the limit by default is 10, so we do scale down 10 empty nodes at once, but this can be increased or you would want to increase that if you want faster, clean up after let's say a batch job or if you have larger clusters, where are 10 nodes might not be enough to scale down at once or so.

A

In which case you would want to increase the sample or decreasing. This number protects you against making big changes in your cluster so again for availability reasons. You would want that there are also other things that matter as a flagship matter or I'll go over them briefly. The first, for example, is unready scalar under any time. So if a node is unready, we wait for a certain amount of time before attempting scale down or the flag directly effect side.

A

Scan interval is a time that CA is basically how often see area reevaluate your cluster or scale down delay after failure is so after, if, after failure and scale down, we wait for a certain time before he attempting it or reactant a scale down and this flag effects that um scale down delay after ID. So if we scale up, we wait for a certain amount of time for stabilization reasons before we attempt another scale done and this flag effects. Ak value, unremovable note recheck time out.

A

So if we're nude is deemed as unremovable, for whatever reasons we have a timeout for which we recheck, if the, if we can still delete this an immovable node and this flux value would affect the timeout.

A

So if you look at it broadly, the values on the left, the flag, values on the left, make your be scale, downs, quicker and if you have multiple scale, downs lined up and multiple clean. So if you, what a clean your clustered or the values under setting the values on the right would help you have the serial scale, dance, faster.

A

So there are game there are some non cluster or the scale of flags that also mattered. So for your controller manager and should you live queue, API QP s and Q PPI first matter so, for example, cube API a QP s is the QP s that your controller manager or scheduler or uses to talk to the API server. So if it's not set to a high enough value or you will have put for a poor performance that effects that can affect your or scale down since Caleb's API server flags, so there are.

A

There is the math mutating requests in flight and max requests in flight that signify? How many requests can your API server handle in flight? So how many concurrent requests can it handle at once and that number, if not set to high enough or to a high enough value, would affect the performance right? So how do you use these or flags in real life if you're, using or self hosted kubernetes, or it's simple and I'll just change or the value in your C? It should work.

A

We do recommend that you gradually change them and maybe even one flag at a time so that you don't affect it a lot, and you understand what are the consequence of changing a particularly value decreasing or increasing it, and over time you will reach what you feel is your optimal situation for hosted solutions, full solutions like gke, we have auto scaling profiles, the charge or beta soon, where you have a curated profile that you can just pick from and say: hey I want to optimize my utilization, and that would basically mean a certain list of flags are set to certain values to help you do that for other cloud providers or you would have to figure out or exactly how to change the flags and how to set it to optimal numbers, all right, um I guess, that's it for the presentation, but we can move on to the questions.

A

All right, they did not seem to be any unless I'm missing. Oh this one here this one here, do we have a mic.

A

C

A

So I'm going to repeat the question of for the benefit of everyone who couldn't hear that so the question was: how can we actually use these methods to handle cases where we have workloads that burst with more CPU usage, and then don't need that CPU with that amount of resources anymore?

A

Could we you you start these techniques set, so if they are the same workload, so, for example, if the work load already exists in a cluster, but the limits are being changed by something like VP, a and and thirds, you would need more resources in a short burst in that in those cases. Yes, you could use some of these flags, but then these floods, that is particularly discussed. Some of them- are useful for scale-up. Most of them are used for scale down.

A

So if you want to scale down quickly and use less resources, that is how you could use these methods to do that or another question.

A

Right, so the gentleman here was interested in figuring out if we could use less euro descaler in an on-prem setup.

A

So for this or I do also recommend margins or talk yesterday, but or you can sort of do it. There are methods to do it so cluster order scale. It supports a few cloud providers at this point. If we have a exhaustive list of floods providers that we support on Prem is not really a cloud cloud provider at this point particular point of time, because that is not implemented in the one Prem setup could become widely different.

A

But if you, if you could in fact implement an interface called cloud provider which exists in cluster order scale it in your own Prem set up. That means that you can use order scalar, so any any environment that can potentially implement this cloud provider interface can use cluster or a scalar.

A

um Any other questions sure, gentlemen, a friend.

A

C

So I want to be able to use the autoscaler while balancing the high availability. That I might ask my scheduler for so I might tell my scheduler I want you to balance my pause across availability zones, but this kind of the autoscaler kind of fights a little bit with this preference, because the scheduler doesn't know about the autoscaler. The autoscaler might only leave room on one node. So all the pods get scheduled there and it's kind of like happens over and over.

C

We notice in production is there is, should I just tune it to have more room like you suggested, should. Is there a better story for happen?.

A

So with that, yes, that's a good question there. Yes, at this point, we are scheduled a team and the cluster or the scaler team are working on something like a balancing out a balanced way of doing this like if you want to spread out your nodes across multiple new ones and have more balanced set up. You should be able to do this with solution that we are working on, and this is definitely.

C

On the radar, okay, thank you.

D

So all of the utilization that you had in your talk is all about requests right. Is there any work being done to use actual utilization at the machine to work on vertical pod, auto, auto scaling, and/or cluster, auto scaling great.

A

So this is a good question: why don't we? You know you just use utilization to figure out, and you know trigger scale of when required, etcetera and scale and well required. So there is a strict boundary between the auto scalars and we want it to be that way. So there is a separation of concerns with what we want the autoscaler to do. When it comes to the autoscaler level. We make certain assumptions. We make the assumption that everything in terms of resource requests is already perfect, because something like BPA would have done it.

A

So in this case we basically try to do the best we can based on this assumption, so he would probably want a junior VP a or to get the right amount of resource request and resource limit set and based on that, the cluster orders key load will try to do the best it can, and that provides us a good layer of separation of concerns. The.

D

Following question is: are you aware of the kernel bursting patches currently in flight on l, kml I? Don't.

A

Think I'm aware of that, so we can talk more about that after day, yeah.

D

We could talk about that afterwards, but I'd love to talk to you about that.

E

So we run pools of specialized worker worker nodes. Does the is the cluster autoscaler able to take into account things like taints that would be applied to a worker pool? Yes, so.

A

The plus you notice Kelly also works with chains and labels if they are appeared and the food scale, if your pod would only fit in those particular node pools, will scale those dedicated node, poles. Oh yeah, I did not notice this lines. Let me skip erosion.

F

Okay, I was wondering: does the autoscaler take into account like draining nodes before scheduling, to make sure that your workloads stay available? Yes,.

A

So we do drain a node before our video move it and that, and we also look at before even tree. We look at things like for disruption, budgets to make sure that the board on that can actually be recreated and can be fit and anywhere else so or we do do that.

E

G

To ask if you had any opinions on observability best practices, particularly about like why didn't this scale down when I thought it should is always the classic one? That's.

A

A great question, so one thing that we are working towards is visibility, so why did the cluster or the scaler scale up or scale down at this point of time, cm it's kubernetes events and these events have the reason why it scaled up or scaled down. Oh now, actually we civilized thing the event into an timeline or something like that is left out to the consumer of these events or something like gke has a product in beta value.

A

We try to visualize this over that time, like over Peter timeline, so we do or from the cluster, on what our scalar point of view. We do Emmit these events.

H

Are there any considerations that you should keep in mind if you're gonna be using the cluster autoscaler, along with like preemptable, x' or spot instances.

A

So it does work with p.m. tables um 100% sure about its part instance, isn't because not a weight of how exactly it works at a to base, but um in terms of preemptable it doesn't work with it in in case. It also is a weight of the fact that the M tables are cheaper, so you should be able to work with it perfectly.

I

Question about the cluster autoscaler well, because the looking at the pod resources versus cluster resources, sometimes when the noes are actually scheduling up, they are scheduling up by one by one, so like weights word, is there a possibility for to have like a more fastest gala? That goes more looking at the metrics of the cluster versus the metrics of the pods right.

A

So at this point of time we only look at we only scale up once we notice an unshakable point on there are so there are proposals about sort of doing this predictively making predictions about you know, can we already scale up without having a pending pod, but those are what they are, which are just proposals at this point or but we will be looking at that more in the future and any proposals at that all also welcome- or one thing I forgot to put it on the side- is a fact that we have or the scale C go to scaling meetings, Monday 7:00 p.m.

A

est DT, so any proposals are welcome. Any requests are also welcome. Awesome.

I

Thank you thanks. I.

B

Think you were saying earlier that a lot of the flags that you had up could be used for scale down, especially when you're talking about, for example, batch jobs specifically to batch jobs. Do you have any guidance or on the like either scale up or just in general, great.

A

um Scale up it's for a lot of providers: Caleb is already pretty fast, should be fine for the lot of cases or scale down is where we get the most amount of requests, and this should help with that. So it depends on you. If you're using a hosted solution or a self hosted solution, we're using a self hosted solution, you can use eggs all the contents in this presentation to figure out the right values for your batch workload and a few optimizes gradually, you should be able to find a sweet spot for you pretty soon.

J

Two questions one: if are there any claw cloud provider specifics that we have to keep in mind, and the second is, if you use custom metrics for horizontal part of scaling outside of CPU and memory, is there any bearing on cluster or scaling I'll.

A

Answer the first question or first so, are there any cloud provider or things that you have to keep in mind I think one of the biggest thing that you have to keep in mind is that the scale up or scale down time once cluster or the scaler decides hey. Let's remove this node the response time and the time after which it actually happens if we're completely on the cloud provider.

A

So, for example, cloud provider a might be able to scale down quickly but remove the node much quicker than cloud provider P or- and that is something that you'll have to experiment and figure out. Okay and the second question, could you.

J

Please repeat that yeah, if we have any non star, I mean not not CP or a custom metrics right, not CP your memory, but custom metrics for horizontal pod, auto scaling. Is there anything that we have to bear in mind when we do cluster auto scaling right.

A

um So nothing of that sort Annette when it comes to cluster to scale it because we do again have that separation of concerns there, but there are some best practices to set your you know choose the right metric of so, for example, if we generally don't use, advise you to use metrics such as memory and latency, so we can talk more about that. Okay, this is this I.

J

Guess so there are some gotchas which we can do. Yes. Thank you.

K

Hi um so I think you mentioned that autoscaler works with no tent, so I'm asking the question under the amount of tenancy contacts. So let's say I have some workload only can be rung in a specific node groups. So can autoscaler kind of like create a new node for me that I have bigger machines if I have to create a new notebook and autoscaler can like scale one the bigger load groups down to zero and move out the power to some smaller node groups? That can only run it so.

A

If the question is, can the autoscaler create a new load pool for you, you're.

K

Like with big machines automatically currently.

A

No no, but we have sort of a feature that disease in gke or because we understand how a Blood Factory, because we just work on gogledd. So we do have that feature for Google Cloud or we can talk more about that after the presentation answering the questions. The second thing is that does it? Can you selectively scale a particular node group? Yes, you can, if you have the right, teens and Toleration zon that know the right teens on the node group and the right holiday shins for your parts, I see.

K

So can autoscaler scale the node pou to zero.

A

So we are out of time it's just been notified. It should sorry for the questions that I couldn't handle, or we can't do that right outside. Thank you.