Kubernetes Event-Driven Autoscaling Standup, 16 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-04-16 KEDA Standup

Description

Link to notes: https://hackmd.io/cEi_FerdQvyTvB1i-5U0kg?view

A

Perfect, okay, good morning, good evening, good whatever everyone thanks so much for joining I, see a few faces or names that I don't recognize, which is great. So we've got a few items on the agenda. We've got an upcoming Cato release. Coming on, there's been a lot of progress on some of the v2 work that I know, zip UNIX been working on.

A

I, don't see him on the call yet I see Ahmed here and we're also I think I mentioned doing the release, so maybe to start with, we can just go around very quickly and just do a quick intro. This is your first time joining. Welcome just say your name. You can say where you work.

A

If you want and then just let me know as we go around if there's any specific items that you're interested in discussing today as a group now make sure to keep adding them to the agenda and if you're just here to listen and hang out, then that's cool too. So I will start first, so I'm Jeff I work at Microsoft, one of the kata maintained errs and yeah I. Don't I don't have anything to cover today.

A

I can give an update if we have time on the maintainer PR that I open this week, but we can go from there. Tom I see you just joined with your beautiful sunny outside I'll. Let you in next.

B

Good evening, good morning, my name is Tom, cool I would hold its imagine a few things correct. Thank you. Let me then.

A

The list here- I, don't think I know how to spell attendees. So if someone can type correct that we have Jeff Tom Israel I saw you ping me: I actually haven't even opened the ping. Yet so I'll, let you if there was something allowed to chat today, you can just introduce it here.

C

No problem, thank you Jeff, so hi everyone. My name is Israel and I'm from Microsoft great.

A

And did you have anything you want to chat about today or just just joining.

C

Back in, it's been a while since I've participated, so I think about about about a month or two months, so I wanted to find out what has changed because it looks like it was a proposal to modify schedule and also the time at the meetings take place. Yeah.

A

C

A

A bit confused about that. Yes, we can cover that. We did cover that last week, the we wondered if it would make sense for us to shift the time of this meeting. We talked about a few options. We thought about, maybe bumping it up an hour. We are already moving meeting every other week now, I think there was. There wasn't strong consensus for either I think I'll have downsides, especially once life normalizes.

A

Whenever that is then a few folks, if we moved it up an hour won't be able to join because I live like school drop off. So we can. We can sync on that again: I meant to send the survey in the stock channel, but I forgot so honor it I'll. Let you go next.

D

Hi I work for Microsoft in the agile functions, serverless space and I've been a part of KDF from the beginning and one.

A

Of the main tenants, any items from you, honor it or just I- guess the only stuff that you instead, she should be working on yeah.

D

So we are going to Satish we're just going to start the release process today and I. Think that's I, don't think I have anything more I'm interested to know more about how the veto work is going. Okay, so.

A

We'll go on or sorry Ahmed next and then we can go to Sevilla Neku. Who can he who has the most on the BT so I'm at anything from you.

E

Not much for me I'm Ahmed, I work for Microsoft as well working um I, know, I, don't have anything else. huh Oh yes,.

A

And I did I know. This is a little bit on the spot. I I meant to ping you earlier I know you've been chewing on a potential proposal. I was actually mentioned this to Sibiu net before about a new scale, or do you want to chat about that today or should we do that in like an upcoming sync?

A

Let's do to the upcoming sync.

E

I want to write a document first, that I would share in an issue, and then people can read it and then in the upcoming sink, can talk about it more in detail. We'll.

A

Do that yeah, my bad, so so I think that works great, especially we've got a few folks on the line. See beinac. You won't go.

F

Yes, hello, my name is Mindy I'm from Red Hat I'm, working on OK de on on the v2 branch, specifically, so just a short update. We make some progress over there. I've made some refactoring on that, which is great. So from my point of view, like the main functionality for for the scale object is, is pretty much done to make the next main main.

F

Each item is the scaled objects, so we should probably discuss whether we want to do some changes in the in the scale of the specification, so it couldn't be implemented and, according the specification, I'm not.

E

Sure there are these little jobs. What.

F

What do you mean sorry do.

E

You mean skilled job or yeah.

F

It's called job here right.

A

Okay, great yeah, we'll chat about that thanks every night. Okay, let me grab I'm looking I listed, it jumps around a bit. Rogers I see you on the line. If you want do a quick, intro, yeah.

G

Sure, hi, I'm, Roger or RG I'm, also working with Microsoft I'm from a team that does code with customers and so I've got some essentially some use cases that I can share and that's true sweet.

A

Awesome, oh cool I see the agenda is oh. This is a different one. So I'm going to pop in up here and we'll say: Rogers use cases: okay, Abba Scheck. Do you want to a quick, intro sure.

H

Thing hi, everyone.

H

I am actually I just want to gauge some check if there is any interest in data streams proposal a new scaler, potentially yeah issue. Seven four six right I just want to check on that great.

A

Awesome yeah we'll cover that for sure we see Amit.

I

Hello, everyone amid knish I just wanted to contribute more for Kara and just like just tiny or clearly just know, what's happening. Basically, even I wanted to discuss about the scaled objects like I got so more about that. That's it from me great.

A

Awesome thanks for joining, let's see here, Rahul.

J

I work in the Microsoft cell scheme I'm in Technical Sales, and wanted to check whether it is whether Kara is the right solution. If I want to scale out my kubernetes cluster based on incoming FTP requests, also I see on the on the agenda here or from March 19 there's a line item around HTTP scaling so wanted to know whether it is possible to scale out for scallion based on incoming HTTP requests. Thank.

A

You yeah perfect thanks for joining rahul. Well, we should be able to jump to that at the end and if not, we could we could grab it up on slide. Two great questions and I think we've we've had some discussion about it in the past. Okay, going through I think we have everyone, except for Travis.

K

Travis with shutterstock we're interested in possibly using Keita, currently working on a POC and I'm really here just to listen and I guess get an idea about version 2 and what's coming up, 11.4 release great.

A

Awesome thanks for joining Travis yeah I feel free to jump in. If you have any questions and I've got an eye on chat to or or folks want to raise their hands as we go to, because we do have a few folks on on the call today. So thanks Aaron for joining so with that I think we've got a good agenda set, let's jump right into it. The first one is the kata 1.4 release I'ma.

A

Do you want to give a quick update on that and I'll open up with, like the commit log looks like for for since our last release.

A

Hundred I can, oh sorry,.

D

I told you that I, probably.

A

Did because whenever I'm, tired I keep mixing up your names.

D

Surprised if I did it's absolutely alright, it's fine I wish I was a half as smart as Emmett, but that's a different matter. So the only thing is that yeah, so we're going to start the release today and hopefully we will get it done by tomorrow. That's that's: ergo I have candidly not looked at the not looked at the commits, so I'm not sure what else is being included here. Yes,.

A

Our last cut was on March 12th, and so we've got a few updates and read means it looks like a typo, some improvements to the AWS authentication, the bug fix and, as your monitor leak with some of the scalars, a few more governance type stuff around our. How we now do DCO, which I ran into personally with my PR yeah. It looks like a few bug, fixes I, don't see anything big in terms of features or new scalars that are in this one.

A

So yeah, it looks like just a pretty pretty minor minor release, some improvements to RabbitMQ and some label stuff, so yeah pretty minor stuff. What will funnel it up in the changelog is Tom mentioned so we'll kind of roll up some of those commits into the change lock here and then also the releases on github. So, as we mentioned we're starting that today, so pretty soon, we should have that wound up for release and for those who have joined before where we're trying a release schedule of every four weeks.

A

So that would mean we'll release this week and then four weeks from now, we will do the one dot or the yeah the 1.5 release whatever gets in from here to there, which might include a new Redis stream scaler or some functionality there. So any questions or comments on 1.4 before we move on to v2 great alright, so v2 zipping, like you mentioned already, you've, made a bit of progress here.

A

I, don't know if there's any issues to update, we talked some last week about one of the goals we have for v2 was to make scaling multiple resource types more flexible, and so today we allow you to scale deployments and scale jobs, but we get a few requests to do things like hey.

A

Can I scale our NGO deployments and should objects and jobs really be the same thing, or should we split them apart and if I remember from our discussion last week, we are going to split up scaled objects and scale jobs and assumes to be like from your update, you're saying that most of the scaled object changes to make it more flexible so that you could deploy or scale like a stateful set or an agro deployment or a deployment. It sounds like that might be what you finished off last week.

A

Is that as that accurate- and you can just kind of pick it up from this? Yes,.

F

A

So the biggest one, then, is the scale job spec and did you say there were any kind of questions there? Should we look at the existing one or I? Don't know if there's a proposal floating around somewhere, probably.

F

We can look at existing existing like spec on the page, I know or I can recall. There were some, you know some requirements, but I'm not sure. Maybe film can come in I.

B

Can't remember yeah.

A

I know like, for the most part, I think this job target ref. These are just lifted straight from like the job spec of kubernetes. So these are all things that make sense to surface, because it's like when we, when we create the job. These are the the settings.

A

You should send a kubernetes I know the one weird thing to me and I and I, even am not even super clear where it landed, is when you go the approach of like a queue and you're like I'm gonna have a queue and I want to drop in a hundred messages and I want to spin up a job for each message, and you never want it to have more than like ten jobs at once.

A

I when I look at this spec would assume that I would set parallelism to ten and that that would stop it from from doing it, and that's actually not the case, and it's not I, don't even think it's max replica count either. If I remember right, there was some weird behavior.

A

Maybe this was a bug where, if I look at like the scale or 4q, it would actually be the queue length that it's looking at, and it's looking at like this number right here and that's how many jobs it spins up which to me it wasn't what I expected so I, don't know if that's by design today. That would be the only thing that, as we move into v2 I would love it. If it was a bit more clear on which one of these like, what does parallelism do? Maybe it's an advanced feature.

A

What is meant to replicas count do and what does like the queue length thing do and how did those work together in a way that it is least well documented, I, don't know if anyone has any thoughts on that or even could explain what the existing behavior is.

F

B

F

That we can probably create an issue for for this, and we can. We can like get there like the requirements or the improvements, because I recall never somewhat I. Don't remember so. Maybe I'll look it up later. So so we can start into the issue and maybe go from there. Yeah.

A

F

That's perfect, I think that's a great next Ally we had the other other thing for, for the week too was basically at the moment there is. There is a very limited way how to scale the deployment based all most people three years or we can. We can do it or multiple scalars. We can do do it, but the the behavior is not ideal. I would say so there were some discussions about about capping the maximum number of replicas and etc.

F

So we did some conversation on the issues, so I'm not sure, if no, no, the guy, who requested it isn't here, but probably this is the this is the other other topic that we can. We can rethink, we can discuss all we can improve you doing.

I

Yes, I'm here, yeah.

F

Yeah- and this was the second issue- that basically, if we use multiple scalars like most of the scalars, basically share share the metric nine, so the metric name is used for HP ice to to scale more employments. So if we, if we deploy scaled object, that consists of, for example, two two triggers and both triggers are using the same same metric name, the scaling won't work correctly. So we need to change change this, so we need to probably generate some unique, unique, not written any more. Something like that.

F

This is this is issue. I can tell you now, issue number.

I

Issues event, 33 yeah.

F

Exactly so, this is, this is good.

A

I can for for the v2 for for another improvement from my point of view, and this is independent of jobs or not it's just in general. If you have multiple.

F

Yes, yes, yes, yes, yes, yes, and then there was the other request for some scalars to introduce something like starting thresholds. So basically you can. You can define basically, for example, folder for the kafka you can. You can define like the electric short, so basically threshold based on what you want to scale and there was a request to basically introduce something like start threshold. So, for instance, you want to scale your phone for Africa when there is one message but scale, the other other like because on on different amount yeah. This is it yeah.

A

Interesting, it makes sense. So it's almost like the hope here is that we'd have some pattern for you to say: don't don't spin up the first replica until some threshold is hit and then, after that it just publishes the HP a target based on the leg threshold. So maybe maybe one is a bad example here.

A

I could see it being more like hey I want to scale for every 100 messages, but don't even start scaling from zero until I have a thousand things on the queue and then it will wait for it, the Kafka, the a stream to have a thousand unprocessed message before it even activates. Is that accurate?

A

F

I think yes, yes, so basically we can. We can implement this for some squares. This is not dependent on the on the skeletal check itself, so we can implement these features for for the square root zone. We all seen the need I see.

A

So it's it's, maybe not a a k2 core specific item. It would be something a pattern that we could introduce that scalars optional is going to introduce, as they do like. That is active thing. Yes, I see, yeah makes sense. That makes sense. Yeah I like the idea, so maybe that is something that I'll move it to the v2 milestone for now, so that when we kind of do these things, we keep an eye on it. I'll pop help want it on it and then I'll.

A

Keep this open I'll, try to add a little discussion, which is like I'm thinking like when we do the B to change that. If, even though this isn't necessarily coupled to b2, we could, in theory make this backwards compatible, but we were kind of like hey as part of you to were introducing this new pattern to called start threshold, and that way when folks start contributing scalars. Maybe it's something that we reinforce, or maybe we even say all scalars- have to support this property. I, don't know I, don't know if folks have thoughts on.

A

If we do this, how we should roll it out. Yeah, maybe there is one more.

F

Point to this, so what you describe is maybe something different than the guy sitting in the issue. I think that he wants to start the start, like the scaling when there is, for example, one one one message in the queue, but not like scale to the other Africa like until there is 100 messages. Oh so.

A

It's the behavior today that, if I, if I just did this today and I, had like threshold of a hundred, if there was one message, it wouldn't activate. I.

F

Guess he's boosting something like this.

F

Or maybe I'm like getting wrong so no.

A

I think you might be right, because that would sorry go ahead. Someone else I just.

G

Want to say, I haven't seen this activity, even if the lack threshold is let's say a hundred. If I have even one message, I see the pod spinning up brightly again. What.

B

Would you want to do that actually.

B

G

With Casca yeah the scenarios that I've worked with the top as well as went up, they are mostly real, real, really new, real-time streaming scenarios so where the customer is really looking for passing messages as soon as they show up. So in that, in such cases, they wouldn't want to wait till a batch of 400 messages is accumulated and they would just want real-time processing right out here.

B

I fully agree I'm more wondering why he wants to wait. I think.

A

I I think I understand it now. Actually that that helps so I think the scenario is sometimes he'll get 10,000 messages, and in that case he wants it to scale based on this number, the 100. Sometimes he might just get 10 messages and if it still scaled by this one, the HP would only give him one instance, but he's saying in that case, if it's greater than this number, but less than this one, he wants it to honor the lesser one.

A

So if you only had 10 messages- and you said it to one- it would burst really quickly for the small batch and process them very quickly, but it was a massive batch. It went to cone nuts, so I think yeah. So the scale would have to lie to the HP about the metric value when it was greater or equal to the start threshold, but less than the lag threshold. So the feature request is actually more around like having the scale or no.

A

Is there a ton of work or a little bit of work and I want to scale differently, based on those two options which is very different than what I, when I assumed.

A

And unless I'm less confident on that one I get what the the the intent is, but I don't know the right way to to expose it. So.

A

Maybe we park it for now. I can I can update this issue, but but I'm okay with it like.

A

If somebody builds this capability I'd, it probably is still cuz whoo it's up to each individual scaler to publish the target, so it would still be a scalar specific work item I just in trying to wrestle around like what's the right name for this property and and how would we expose it in a way that makes sense that you have like this bursty number, if it's a small thing, so anyone who's willing to chime in there, but that that makes a little bit more sense to me at least what's being requested?

A

Okay, any other comments on v2 stuff. We should slack here zip eunuch.

A

Great, oh, this is a great one. Tom I didn't even think about this as an option move stand up to the YouTube account so Tom I'll. Let you kind of cover what you were thinking here. Hopefully, there's.

B

A YouTube account who has all the stand-ups for all the projects I think it's that link. So maybe we should check if we should publish the recording as part of that amount. Yep.

A

Agree: yeah. We we have been and does it have it for projects too, because I know this is where they do like their cig stand-ups. But, like would I see a helm, a helm stand up on this on this page, so.

B

If you have a look, I service mesh interfaces also great.

D

B

They even have a playlist with all the standards, so yeah they're also they also just recently joined so great.

A

Yep I agree like right now: we've been publishing them to our YouTube channel, which is very one-off and so I love the idea of publishing to a channel. That's got a bit more power to help get get the word out more so yeah. That's awesome! Any thoughts on the next step here. Should we just ping our CNC, F friends and say like hey: can you give us knowledge on how to do this I'm happy to help with this one yep.

B

That should be a.

A

Surface, that's would be the.

B

A

Let me let me assign this to me then, and I'm gonna. Add a to-do item, because I have a few assigned to me. Do the YouTube thing for kata great best description of it, yeah I'll know what I mean.

I

Okay, talk about the way to stop so basically multiple scalars, so the idea is to like, whenever we had an external metric, it's hard coded within or the scalar something like Kafka scalar has lack threshold as the metric name. So ideally, I was thinking if we can have a way to in in the scaled object to mention what what should be the name of the external metric or if they don't give, then generate then generate a metric name on our own. So that was the plan.

I

So I was wondering: how do we do what shall I named it as metric name, or should it be more specific I want to discuss on that name. I said.

B

I

Way, what should be like the key in in the scaled object that should be there yep.

A

Yep III I personally think metric name sounds great I think your description is awesome. I think the feature sounds good. Anyone else have any thoughts on on what this thing is called that you would set override the metric name. We publish to the HPA I think.

F

It's correct, I think this. This option should be part of the of the triggers like much, but it should be like in the trigger section. Let.

A

Me make sure I'm thinking correctly. So if I look at the scaled object spec today, why is it not what I wanted it to show? You have triggers here and then there's metadata, which I'm not showing right now with this potentially be appear to metadata, so I would say like a metric name below the metadata property.

B

A

Think that's a great suggestion. Any other thoughts.

I

I can't test a bit of them, but I am not sure how to taste any other I, don't know test, monitor or I. Don't have access to go ahead with any of them. How do I do that? Any yeah.

A

I think the two easiest wants to test with, because we've got quick starts from both B Kafka and RabbitMQ.

A

If you have an image, if you like, publish an image of Keita, you can even just ping ping me on slack or something and say like hey and I've got a few tests when up I can just modify my cluster and pull in your custom. Image and I can run a few tests, but I'm pretty I'm, confident that if it works for RabbitMQ and Kafka that it will work for all the other scalars I assume based on where you'll be writing the code.

A

I don't think it's scalar specific necessarily so I'm happy to help here to do a smoke test for you to make sure it works forward for a few other ones. That I've got some demos around.

I

A

No problem great, thank you for being that any other questions on this one, and are you cool if I sign you to this as well I see you you've already been pretty involved in this issue. I, don't know if you you wanted to. It sounds like you're, actually coding it up right now, so I can just assign the issue to you officially.

I

A

You on it, that's awesome, really appreciate it. Yeah feel free I keep an eye on the slack channel too. So if you run into any issues and maybe I'm not being as responsive or someone isn't here, feel free to just flag us on slack and it will ping my phone and unfortunately, I get notifications hurt are too noisy now that I almost just can't rely on, but.

F

A

F

So, and if you could please like do this work based on the v2 branch, so like I, suppose you want to have this functionality, not in the in the wee one but in the v2. So if you could realize the thing.

I

A

Should we I'm fine with that? Is this a because it's a new feature? Should we just do this on both and just backwards it to be one or do you think it's easier at this point that we just make it v2 only I.

F

Don't know I figured we do all. This is good like if we like this will force the users to switch to being v2 so sure.

A

And I didn't do anything and I assumed based on the progress we're making we're. Probably thinking like v2, around May, June or more like June July I mean so I. Don't think that's crazy to have someone hold off who's who's done with this I'm at. Do you have any thoughts yourself.

I

E

I, don't have any cool.

A

Great all right, let's go with that, then, okay, so moving on to scalars in general I know there were specific interest from some folks on the call. I can't remember who wanted to talk about the Redis dreams, one Tom I, don't know! If that's when you want to start with her. There was anything else to you wanted to bring up on scalars yep.

B

H

Liz, Taylor and I have one for ready streams as well conceptually. This is a little different because in ready stream the object itself, it's like an append-only log. So essentially you have to use something called appending, increasing, increase lists using this specific command, I thought that can form the basis of criteria for scaling.

H

You know you determined based on if they were consuming off a data stream. So that's that's the idea I saw yeah. So it's to be honest, is very similar to Redis lists in in terms of implementation like the same authentication mechanism. It's just that you know the command we which we use in order to fetch thee lag.

H

It differs, and there are a few caveats which I mentioned in my second comment: if you can scroll scroll down please, but that is more with respect to, because this is like a squeezed, because structures is essentially unlimited, append only log, but this is moved from a consumer. You know development standpoint and how kada is not essentially or the scaling a or may not be just just empowerment's.

H

If anyone has feedback or inputs on this, but by and large from a conceptual standpoint, I think it's it's pretty work on the basis of pending items for a specific consumer group, ready stream. So it is sort of somewhere in between a Cochran's, humid group and Redis list. Very cool, okay, yeah.

A

I'll pause here: if there's any questions and I can I can chime in or or thoughts on this on this scalar.

D

Yeah Abby shaker so the the streaming work similar to Kafka is the idea, the same of checkpoints and and and.

H

D

H

And different, at the same time, it's similar I would say the difference is that you know ready stream, so sort of takes upon itself to manage the state of each consumer, so in Kafka. So there is a consumer groups right, you can have you know individual consumer instances come and go like you know, there is no restriction on that, but ready streams. What it tells you that you'll have to have a unique consumer instance within a group and a redis stream will store some state on behalf of that consumer instance.

H

So I have an application, which is mapped to open humor group, say, for example, application. A and I have two five instances to scale out the to share the processing load. Those instances have to be unique and Redis teams will store some state on the server side on the west side itself. So that has some implications in terms of say, reprocessing messages and you know dealing with failure scenario so on and so forth. One of these haven't mentioned in the comment as well. If you see the second item, what if instance fails? So?

H

Yes, it is conceptually similar but implemented differently within reading streams. If I were to summarize this okay.

D

H

A

It's good, though, to and and I think some of the even the college you have here where it's like the consumer has to a corrupts. There's the risk that it's going to keep increasing and kate is gonna, keep scaling, that's pretty consistent with with how qaeda works in general, with Kafka, for example. So I think this is awesome.

A

I didn't even realize there were hooks that we could dig into the Redis streaming protocol, so so I personally think the proposal and the caveat Elston really good and then were you interested is this one I should flag for help want it and we can see if someone in the community wants to pick this up or do you want to try to take a whack at this.

H

A

That's awesome.

A

Man, Redis can do so much more than I than I ever realized that things just chugging along.

H

Isn't the cue so there's some cool stuff coming in there do that's.

A

Cool great awesome, any other scalars tom. Is this the just make sure before I move on OS.

B

Mm, yes, so we have somebody who's working on this. My one awesome yeah.

A

I saw some questions on slack. They were, they were wondering about which STK to use, but that would be a very good one to have as well great Oh awesome perfect. Thank you, Tom, okay, so Israel dropped off already. He had to jump up for another meeting at the half-hour mark, I guess I kind of mentioned this. At the outset we were evaluating if we should move times.

A

I don't know if anyone cuz we've got a few folks who are on this call who weren't on two weeks ago, if anyone just x wants to throw in if, if we should definitely move I, think the the easiest one for most of our team would be moving an hour earlier than what we're doing and still doing it every other week, but I like I personally, can make most any time work so I'll just kind of pause here in case anyone wants to chime in on that ongoing discussion. This week.

B

The European folks would like and.

A

It's an hour and after it would earlier like what an hour be significantly different, or would it be like two hours or three hours would actually be the good one.

B

A

Great any other thoughts on that. Yes,.

F

It for me thank you.

A

Okay, yeah I. We might do this, then, because this is popped up so I wonder if, in terms of next steps.

A

I'm fine, if we go to nine personally I think I mentioned there. The serverless working group that meets at 9:00 to 10:00 but I have I, usually don't have to attend those anyway, because they're focused very much on cloud events which I'm less involved in changing.

I

A

Sure they might switch topics yeah so that so that could so that could make it a little bit tricky but I'm I'm open to it and we could always adjust again. So maybe I'll pop in the slack channel I will do it this week because I'm gonna add a to-do list. Item.

A

Do the time of Keita thing on slack I'll post a message on so I can see. If folks are ok with the 9 a.m. and that maybe we can move it. We won't plan on changing next meeting because I want to give at least a meetings notice, but next meeting might be our last one at 10 a.m. Pacific time, we'll see. Ok, great Roger saw you mentioned. You add a few use cases. I, don't know.

A

If there's anything, you just want to kind of share high-level with with the group here and then we can go to the Rahul's questions around cluster auto-scaling, an HTTP.

G

So on mine, actually a part of the use cases. There was also this small issue that I'm seen with HPA, which I wanted to share I, did not raise it on the Cana forum yet, but I probably should have with more details. So the use case that I want to share with share is about a customer. We are looking to create, create log streaming and scaling and doing some processing or parsing logic on some logs that they are getting from syslog.

G

We are so they are getting the log that in the protocol they are passing them on to event hubs and then using events of scalar on data to scale out as your functions, which triggered using event, hubs and functions are deployed on a TS. So that's the scenario and based on the processing. The auspice from the agile functions would either go to an output event tub or a dead-letter event, so that the kind of end-to-end sin are you finally going out back in back into as your sentinels for further processing of the process messages.

G

So that's the kind of end-to-end pipeline that they're. Looking at what I'm saying, though, right now is, my HPA is currently sinus stuff with the with the number of messages in the queue. So even though I do not see any actively processing messages, my HPA is stopped. Looking at a value of saying that there are more number of messages in the and that's, why does not automatically scale down my pond I'm, not sure? If anyone has seen this analysis, something I was hoping to kind of get an idea of what I should try to.

G

You know debug this issue further yeah.

A

And I, if I understand right, there are active containers, there's active replicas that are processing event hubs, so they're they're chugging along doing something but kada when it's doing its check on how much work is required. It's it's seeing that, like. Oh, it looks like there's no messages here which is potentially throwing off the scale is that is that right, right.

G

The other way around only thing is that, even if there are no messages in the event hubs currently that have yet to be processed, my HPA is still thinking that there are messages on the tube and that's. Why does not scale down so another thing that I tested is even with this is with the HBM, this kind of state, even if I add and more messages. In my event, my pods that are already created in those messages further ahead, so it's not that it's in a stock or a limbo state.

G

My spot dam go ahead, but my HPA doesn't seem to drain out or it's not getting the right value for the number of messages and I was wondering if this could happen, if I had any other local instances of the Latasha functions running and consuming from the same event hub, but maybe not recording back the messages in the check point in the same tack, point blobstore: it's.

A

A good question, I know behind the scenes, how it works for functions is, is there's a storage account and it's writing check points for each partition and when you run it in the Azure function service we named the storage account. Actually it has nothing to do with the name of the function. I think it's just using the customer group name yeah and so I am curious. It's like I was wondering at first. Maybe there was a world that it wasn't correctly.

A

Writing the check points back into storage and that's why the scaled scaler was thinking it was bad earlier, but then, when you said it resumes at the right spot. That was a bit strange. So definitely if you had some conflicts between, especially if it was pointed to the same consumer group I think things could get a little bit out of whack.

A

It is one, though, that, yes, if you end up creating an issue, there's nothing I can think of. Initially that would explain it directly. Teams.

D

I'm sorry for not responding before.

B

I know I, know I just but.

D

But one thing to check I would imagine that we have a log which tells when the HPA goes and checks. So what does that log say and what is HP actually see so.

G

I was I was trying to print what I see on my screen here for the HPA I'm not able to pace the screen shot for some reason. But let me just pretend out.

H

G

I can totally understand this is not the right forum to kind of discusses. We could move to like an email or something or the flack channel to kind of debug. This further I just wanted to kind of bring this up to find out. If someone has, you know, it's get them something: sunburst I.

D

Think that decides best to open an issues and you can in the Kayla repo and just all the logs and the details.

D

They are more eyes on it as well.

A

Sorry I was gonna I think if you turn on debug logs too, if you deploy Kate out with the bug logs enabled.

G

A

Thought there was a way that when you look at the law, it doesn't do it by default, but then these four cents goes when you turn on to bug logs I feel like I could see what the number K descaler was getting, which ideally would map one to one. What HPA is pulling from, but.

B

A

Could help too and in terms of just debugging, to get that issue up them, but yeah feel free to ping me as well. If you have any trouble with this, it sounds like something that I've spent a ton of time with event hubs so I'm happy to see. If there's anything, we do to help unblock awesome.

G

Thank you so much appreciate great.

A

Okay, so that brings us to the last one from Rahul around cluster auto-scaling and if I understand right. The a few there's kind of two things in here one is Keita today is focusing on horizontal pot, auto scaling, which means you have a cluster with like five nodes and a bunch of events come in and we make sure that we're scaling out replicas within those five nodes well and I. Think the first half of your question was: could we have Keita driving the scaling of the entire cluster?

A

So maybe you don't need to go from from two replicas to eight replicas alone, but you also need to go from five nodes to ten nodes and then I think the second one related to it was and what, if the event is actually HTTP and it's not something like event hubs or RabbitMQ or Kafka? Is that a fair statement? Rahul anything else, you'd want to add.

J

So I, you know if I want to scale out based on FTP request incoming request, but.

F

J

Or is it the right there right like technology? You know to do that. So the scenario here is they process like I, don't million FTP files? Let's say you know within a couple of hours and they were they wanted to know whether it is possible to first Moodle is structured to basic infrastructure to lineage, which looks like it is possible. But the next question is: how do you scale out when you get this burst, of incoming FTP requests, yeah.

A

Makes sense so I think I'll take a stab at this and I'll. Let others jump in nothing. An FTP right like File, Transfer Protocol is the one that they're looking yeah, so we don't cater today doesn't have any way to scale directly on anything that is push or like request, based that the TCP HTTP G RPC FTP directly. However, there's one pattern that may or may not help there's a blog post here, which I can paste into chat it's more around HTTP, but I clattering.

A

The pattern looks the same and how it works is if you can get metrics publishing from the FTP destination, so I imagine like in the cluster there's some FTP server, I, don't know if the FTP request Traverse through an ingress controller or even if the FTP server itself can emit metrics the way that I have done this in this blog post that I pasted.

A

Does it as well is that you actually create a scaled object based on Prometheus and you scale, based on the rate of events coming into Prometheus, so like I'm, showing on my screen now this is one that's HTTP trigger, so I have nginx that publishes metrics to Prometheus every second saying: here's, how many HTTP requests I'm processing and then kata asked from atheists. It's like hey. How many requests has this HTTP endpoint guy in the last five seconds and then based on that it can actually start to scale out.

A

This is, in general, a super common question. There's a few other options too. I, don't know how they work with FTP I'd have to rely on others on the call, but around these scenarios around HTTP in general, while kata doesn't do it today. It is something we're interested to see how we could make even the pattern. I showed a bit more first class, the SMI spec work that was mentioned my help, but that's more related to service meshes Kay made of serving, does do scaling, but zibeon ACK, I, don't know!

A

If you know this, would it do King? It would K native serving do scaling for FTP, or would it have to be HTTP in that world.

A

Yeah, so that's kind of the generic ones that we kind of tell people of kata directly. If you want it to do it, it's gonna need to pull some other metric other than monitoring the FTP request directly. If it's HTTP, Kay native serving, could do it today. You could also do a cheapy with this method. So I think this.

A

This kind of pattern that the blog post shows and you could ignore all the stuff about HTTP, but in general, using the Prometheus scale or if you can get the data and metrics into Prometheus, might be your best bet. I, don't know if that makes sense, or if you have any follow-up questions on mash yeah.

J

I think so figure out whether it is possible to some I'll publish the incoming request for FTP and looks like if I, if I can get this pattern works. Yes,.

A

Exactly right, yep, if you can get some metrics from the FTP, maybe the server itself or from I don't know if FTP goes through.

B

A

Also be possible if.

B

Your than actually you could also use that then.

A

B

You don't have to spin up a distance if.

A

You have the FTP metrics in at your monitor. You mean yeah, yep yeah, that's great! That's a good call out! Okay! Does that help Rahul yeah.

J

So it looks like you know: if I can get pulled off the matrix. That tells me the incoming requests for HCP. That would you what you are showing here on the screen that will be the pattern: okay, great.

A

Awesome yeah more on that too, and in fact, some of the scalar research Hammad's been doing is at ease for two weeks from now has been these types of patterns. How do we make it work more seamlessly with kata and ideally very simply because this is probably the number one question I get around. Kata is around the HTTP scaling?

A

Okay, so that's how we had in the set agenda we're at about 50 minutes in before we break? Is there any other topic someone wants to flag or discuss before we all sign off? Can.

B

You really really berate on the cluster auto-scaling what the scenario was, so it was it just based on the HTTP request, I.

A

Think it's just FTP Otto's going. There was nothing around the cluster right or did you also have a question around cluster at all.

B

Does anybody know it's the cluster autoscaler which metrics as well? No, it doesn't make sense.

A

Never mind when I was going to point to we do have this open issue which I would love to get around to some time is around? Is there a way we could integrate easier with the custom, auto scaler? It has four thumbs up so for people like it but yeah. This was a different one. There's some good discussion here from some folks around this could work. So this is something I think someone would need to do. Some investigation on I should do I should say investigation.

A

We don't have a label for that, but yeah there is there that separate one that I was talking about. There is an open issue on. If there's anything we could do and I think I discuss here. We can indirectly influence the cluster autoscaler today by forcing too many nodes or pause to be scheduled, but there has been asked like. Could we make that more direct, no.

B

I'm just wondering how we will be able to help with them.

B

A

You great okay, so thanks everyone, a ton for joining for the discussion as I mentioned. If you need something between now and two weeks from now, github issues is great and slack is also where we're chatting about some of this stuff as well on the kubernetes slack channel that you can find on the k2 landing page. So with that we'll go ahead and wrap it up for this week. So thanks again, everyone we'll chat again in two weeks. Thank.

B