Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 19 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Autoscaling SIG - Solly Ross, Google & Michael Hausenblas, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: Autoscaling SIG - Solly Ross, Google & Michael Hausenblas, Red Hat

Every had a question about the nitty-gritty details of the horizontal pod autoscaler? Want to weigh in on the future of the cluster autoscaler? Curious about the ongoing work on the vertical pod autoscaler? If so, come to the SIG Autoscaling Deep Dive. We'll be discussing future and ongoing developments in all three autoscaling components, as well as answer questions and concerns about their functionality. While the session will be centered around areas that attendees wish to discuss, potential topics include algorithmic improvements to the HPA, the path to stabilization of HPA v2, and cluster autoscaler integration with the cluster API.

To learn more: https://sched.co/Grew

A

Welcome to the Sikh auto-scaling breakout session. If you are part of Sikh auto-scaling, if you contribute, then please come up to the stage and we have a few more. Thank you few more ads here and the rest is essentially Q&A right. So if you have a question, probably I'm gonna run around with that microphone for people to ask, and if we can use your microphone for answering things so Q&A, you can ask anything related to auto-scaling.

A

Who has a question first question.

B

Hi I would like to Auto scale of CRD that has a scale sub resource, but it's not connected to any pod. It's like an external. So how would I you know? I would like to use all the logic about the HPA, horizontal autoscaler, using metrics custom metrics things like that and and then scale out and in this custom resource, but the the HPA is well like. The name suggests like very mean it's really about that's right, but so we'd be possible to make it more generic.

C

So yeah, so that's not possible right now, mainly because the HPA has a lot of logic around detecting pods that are in weird states that we don't want to skew your auto scaling for normal use cases, but it's definitely a use case that has been brought up before and it's something we're interested in.

C

This is kind of the part where I go. Prs. Welcome, like I I, would love to see a good design for how we can move some of that logic around detecting pods. That would throw things off to a more generic phat form or continue to have support that logic, while being a little bit more flexible in the case where our scale sub resource doesn't provide a pod selector, for instance, I.

D

Is there any plan on making thresholds for auto-scaling so instead of having one, we could have like minimum maximum I've been playing with the tolerance in the controller and sometimes 10% is not enough. Jerry yeah. Do you plan having flexibility there? Okay,.

C

So to clarify you want to you want to configure the the amount by which it scales up in steps or like so so for a summary just to make sure everyone's. On the same the same page, the way the horizontal pot autoscaler works right now in kind of the high-level overview, is that we get a current utilization. We compare that to the target utilization forming a ratio, and then we multiply that ratio by the current replicas.

C

So if your target is five hundred mill, cores and you're using a whole one core in 1500 is two, so we multiply the current replica count by two and then same in the opposite direction right if you're, actually only using 252 of it 250 over 500 is, is a half.

C

So we multiply the current by half so to clarify so right now we have a thing where, like we only will scale up at once by a certain amount so that we don't like you know quickly, add 500 replicas to your thing and then it turns out that it was some sort of weird spike and we really should only have that at 10. So, are you looking to configure that threshold? Are you looking for something just closer to I want to scale at particular triggers.

D

So, basically, if I trigger at 500 mil occurs and I have 500 won, you won't trigger, because it's in the 10% tolerance I'd like to know if it's possible to have instead like 600 and 400. So like, like you, know, scope under which I don't want to scale. Oh.

C

Okay, I see, that would be a good thing to have. We don't have it right now, but again, like some of this, some of this stuff is is: is we try to cover some of the common cases that people were encountering and as we learn more about what people need to do with the horizontal pot autoscaler, we are open to more designs to improve it. Thank.

D

C

Like it's worth it to raise an issue for that, and and also like, if you can put up a design proposal or something that would be even better one of the things we always try to balance, is we don't make things too crazy, complicated right like we want to make it so that you can understand? Most people can understand the horizontal plot autoscaler, but it's flexible enough for people to do what they need to with it. As.

E

Someone who uses the cluster autoscaler a lot is there anything on the roadmap there that you can talk about maybe more notes so.

C

The big thing with the cluster autoscaler coming up that I know of so that's not quite my area of focus but I try to keep an eye on everything. That's going on. Segata scaling.

C

The big thing coming up is integration with the cluster API, so we're we're looking at integrating with the cluster API to avoid our current dependency on like manually, knowing how to interact with your cloud provider because maintaining cloud providers for everyone is a pain but and we would love to be able to just say: if you implement the cluster API, then we work with you and even if you have your own particular implementation, that's proprietary or whatever is.

A

There anything in particular you would like to see, or was it just general interest in the roadman.

E

Currently, it taps out of the thousand nodes for various performance reasons. You would love to see that go up.

C

All right, oh yeah, I'll, definitely make sure we pass that on, but okay cool so do.

F

You actually see that it stops working for you after Towson notes, because so the Towson knows threshold is something that has been tested, and this is that what we know works, but if you've seen things that stopped working after terms of notes, we'd, like we'd love to know like we haven't, observed yet anything concrete that starts breaking after this threshold.

F

G

Couple questions: is there a simulator where we can without having to spin up a lot of notes, test the different ultra scanners.

F

So I can say for vertical bottom to scaler. You can try it out using an off mode which just generates the recommendations, so you can take a look at what it would do to your pods for cluster to scaler I'm, not aware of anything.

C

For yeah, well, hopefully, when we move to the cluster API you'll be able to do a fake cluster, API implementation and then for the horizontal Potter scalar. You could probably run with a fake node like a kind of a fake node thing that accepted pots and marked them as if they were running kind of. We.

G

Have a similar testing framework.

C

Inside kubernetes for doing certain types of performance testing, and then you can provide a fake implementation of the resource, metrics API, which I think we have floating around somewhere. If it's not an official repository I'll make sure it gets out somewhere. Okay,.

G

And, secondly, it's anybody using a project, mentalizing called escalator for auto-scaling.

C

Can you repeat that escalator.

A

C

Am I am not aware.

A

Just a quick question who would be interested in such a simulator who would want to use such a similar just some gosh, the interest of it? Okay, cool? Maybe that's something not anymore.

G

Hi is there a timeline for when autoscaler will start using the cluster API.

C

I, don't know of the timeline. Do you have one so.

F

I, don't have the timeline, but there is one big problem currently with the cluster API that is being worked out between the auto scaling team and D team working on the cluster API, namely what the autoscaler needs. It needs to be able to delete a concrete machine like this machine. This is the one that I want to delete, and this is currently not incorporated in the cluster API and without that the scale down doesn't work.

F

So in the current implementation in the current design of cluster API, we can only do scale up and scale down would be impossible, as kaito is pretty important. So so, basically we're we're trying to figure out how how that will have that how it would be possible to integrate both clusters of scalar needs and not over complete complicate the cluster API. So, unfortunately, I don't have I, don't have any concrete plans for you, but this is the status of word where it's currently at my.

A

Guess would be not 2018.

H

Is there a way with the cluster autoscaler to tell if that it would that a node group can be scaled down, but should never be scaled up like for upgrade? For example,.

F

Can you repeat zyk to.

H

Tell that cluster autoscaler that a certain node group may be scaled down if nodes are idle, but that group should never be scaled up, so you can.

F

Said I haven't seen so I, don't believe it's possible at the moment. I guess you can try filing an issue or try to select channel critical to scaling and they'll. They'll definitely know if it's possible and if it's easily added, if it's not there, okay.

I

Hey Mike can I ask a question, so maybe a question for the for the group as well, so any thoughts around creating higher level constructs to manage the performance of you know a group of parts all together based in one matric. That would just make it easier rather than creating multiple auto-scaling rules.

I

You know each per pod or names paper, I would say per controller. There would be one construct for all of these, so, for example, I have a HTTP endpoint and that HTTP endpoint is basically served by multiple parts. Right and I would like all of these to be scaled up or down all together, based on that custom metric. So, rather than going and creating a rule for each one of them, I can just group them all together as one app one logical app and they all get scaled up and down together.

C

So you could probably do something similar, so it basically comes down. You can point the horizontal pod autoscaler at anything that eventually resolves to pods and has a label selector, okay, so anything that implements the scale sub resource and can provide that label selectors to select its pots.

C

So you could hypothetically implement some sort of higher-level grouping and your your metric in the horizontal pot autoscaler also as of the past few releases, does not have to be a per pod metric too so like if you, if you are measuring, if you were measuring your requests per second at your load, balancer level and you're, not for some reason splitting that up per pod you can.

C

You can specify what we call an object, metric or an external metric, which is basically saying I, have a single metric that corresponds to some object or I have a single metric that corresponds to some set of labels inside my metric storage solution.

J

Sure I have two questions around the cluster autoscaler I'll keep them short. The first one is that currently the cluster autoscaler can be configured in auto-scaling groups by using tags on them, but in GCE it uses a prefix on the name of the managed instance group. Is there any context on that decision or will in GCE we will be able to filter on labels anytime soon.

F

I'm not aware of any changes around the GC cloud provider for cluster to scalar. It might be on the roadmap. I might not be aware of everything, so I encourage you to file an issue I'm going to have to find out.

C

When, in doubt, always file an issue or even better, a poll, request, searches, I'm proposal, the.

K

Scale-Up delay has disappeared. Are there any plans for the scaled-down delay to make it more granular so that I can select / HP a definition of.

I

K

Minute, where this we.

C

Don't actually have a scale down delay either so right anymore right now. What we do is we kind of look at a window of values, and we pick the largest one during that window.

C

So in some senses it continues. In some cases it continues to behave a bit like a delay, but in other cases it will smooth out bumps right now. I, don't know of any plans to configure that window, but it's certainly something we could consider.

A

Alright, with that, we reached the top of the island, unfortunately already done, but I'd really like to know if that format in terms of Q&A and so on, if that's something useful or if you prefer more traditional like deep dive coverage and yeah definitely file issues and be be around on the selection. Some meetings thanks a lot for being here, geez.