Kubernetes Event-Driven Autoscaling Standup, 12 Aug 2019

Previous Meeting

⏯

youtube image

►

From YouTube: KEDA Aug 8 Standup

Description

Includes demo of the alpha KEDA dashboard

A

Sure, okay, great so we'll get started. We'll have a few updates. I went through this morning and I saw a few of the action items that had been followed up on and a few that hadn't. So we couldn't. We start, though choice. Do we have Li shake ours on the team and then Jasmine who's got a demo of this dashboard.

A

That's very cool that I'm excited for for her to show off in just a second I, don't see Tom on the call, the other one I was going to call out just as a follow-up from last time is the authentication PR which I was just looking at now, and it doesn't look like it got reviewed.

B

A

Didn't see Aarthi or Ahmed signed off on this I know Ahmed out this week. Aarthi is here so I'll just her on this issue as well.

A

To see that, as for the jobs, one Lee I see you're on the call, but Shawn I don't see him. Do you know if that one progressed at all, I was trying to assign the issue to to Shawn, but he hasn't accepted the invite in the org. Yet so I might just ping them on teams or something do you know if that one's progressed at all, leave from your knowledge.

A

Oh I can't hear, let me see I think my speaker might not be working hold on one second, because I can't hear anything but I see you on your muted yourself.

C

Is that does that make.

A

It better perfectly now my speakers are working. I can hear you yeah, alright,.

C

It was me, then I'm using a different audio thing and I don't know.

C

Yeah no I haven't heard anything since last week, so I can follow up or if you pay him on teams, I'm sure that I have about the same output. I know that they are there on a customer engagement, so they might have a bunch of different things to focus on, so they might have focused, but probably worth following up great.

A

Yeah I'll paint them on this and then I'll make oh, not the authentication, oh yeah. This is the job so and that's right.

A

Here, except to invite to the org okay. So then, before we go to SH asmin to demo the dashboard, which I'll record and probably pop up on YouTube and add to the notes for people who want to watch it. Lior shake are any updates from your side or any things that we need to take note of that I can at least even follow up offline with artheon ahmed on.

B

So I had submitter PR for the external scaler support quite a long time ago, and also no that's the so the two of them. So there's one for scaling, base and kubernetes events. Yes and there's the external scaler support and no no one's signed off from that. Yet now what I can do is for me next time you could I could demo that support, probably better and then also, then we could discuss it and close off the PR after that, yeah.

A

It looks great we talked about it last week and I know you weren't on the call. It's it this week's challenging, because it's a it seems to be holiday week for for a lot of.

B

Understandably,.

A

So so why don't we plan on that will at least cover it, I still I'm, going to see if I can get them to review it before the next Meetup?

A

But let me just add it here for proposed agenda that would be cool to do it because yeah we talked about it yesterday. I think everyone thinks what you built is probably great. It's just that we actually need them to formally review it so I'll, add the external scaler support demo, that'd be cool, or at least walk through yeah that'd be great, any other ones. Shay car that you've been tracking I know.

A

You also had the kubernetes events, one which hopefully they review as well, but I think this one's, probably the one that will be more interesting and it's also 22 days old.

B

Yeah so I could demo it so I've already got a sample and go, but I want to write a sample in different language, so I started with node and so I could show a scaler with a min node with that. So I'll do that. Next time, nice.

A

That's awesome, okay, great and then we anything from your side.

C

No very little I'm on other customer projects and other projects, so I haven't been focusing on chaos or much no.

A

Worries, that's great okay, then Shaz min I think the last one. Then, if you want to just show off your dashboard and then I'll post the demo as well on YouTube, so that people can get a feel for the dashboard, but I think it would be great to show that in kind of a public forum as well, yeah.

D

Sure um can you hear me I can.

A

Hear you great and.

D

A

D

On the call and like some reason, no one can hear me but okay great. Let.

A

Me make sure I've, given you the right permissions to projects. I'm gonna, make you a host which I think will let you share your screen. If you want to okay.

D

D

A

Yes, I see your screen: I see localhost 3000, yeah.

D

Yeah um yeah, so I guess this is the dashboard. um There are three main pages that you would see when you start to kind of or start up the the dash and the first one is like an overview of the kid, a scale controller, and so you have like this panel. That gives you a bunch of like details about the deployment of that scale, controller and below here. You have kind of an events log for all the scale, decisions that the scale controller is making um there's still some work. That needs to be done.

D

I guess like in the events in this like panel, um but I think like most of the information that needs to be there. Is there so and I also just want to point out that over here?

D

You see that it also tells you that, like the last scale time, so the dashboard will tell you but it'll, like always keep track of like when it's scaling out and then what I guess like give you that information, which is kind of nice um and then over here on this page, it's um very simple: it's just a table that just shows you all of your scaled objects. uh So right now, I have only like one scaled object deployed and that's actually my event up trigger um I, actually scaled.

D

It like, like 10 minutes ago, just make sure it's working so um again, like just basic information on there were no messages that are being sent to the event hub right now, and so it CL down to zero and there are no replicas and then also information about your HPA definition and reference, but I think the most interesting view that most people would find a lot of use out of is like the detailed view of this page, so kind of like the overview page, um you get information about the deployment, but instead it's specific to on the scale objects deployment are in that sense and over here you again, like you, have the loss, scale, time scale or active time for this deployment, so um I deployed well I.

D

Think in Pacific time like 953 am so it has that which is kind of nice and then there's still some work. That needs to be done still trying to figure out how to like get the x-axis to not be so crowded because before I had time stamps showing um when the scale time would be over, um but it kind of got crowded so for now, I, just like put it to the side, but you can still view it all over here.

D

If you hover over, um so you can see like it's hard to scale up at the same time and then um five minutes later it's scaled down, and that was just the cooldown period, so um yeah, and so you have more information this time about the scale target and then also, if you want to keep track of how many of replicas are currently uh there I guess for the specifics scaled object, you can vary uh yeah.

D

You can just like look right here to if you don't look at the graph and the information I guess about the trigger type. This is more like the metadata you would get from that sealed object. You go down further. This is where you have events specific to scaled object and Cl history. So you can see that I send it 120 messages at this time and then that's when it activated and then scaled from 0 to 1.

D

um Still don't have I guess like the one to end scaling that you that people might want to see that's something that should maybe be figure it out after I leave and then also here after five minutes, that it also deactivated so I think what I'm gonna do now is that I just have like a regular event. Tubb producer, that's going to send I just had 150 messages to this event hub and then it should be able to pick up all hundred 50 messages and then scale accordingly. um There.

D

It also takes a while, but I've also realized that sometimes it doesn't fully pick up all the messages, and that might be an issue with my event hub rather than the dashboard, primarily because all this information, the event logs and like the infant metrics to show you like what scale ditions correlated to what it's actually picking from the logs emitted from the scale controller and so I guess that is useful, because, if I'm sending my 50 messages, but it's only telling me like something less than that then I guess I helps with the debugging experience of.

D

Why did my scale Daughtery not scale as much as it should? So it takes a bit of time to kind of pick that up I could take any questions in the meantime.

A

B

You go for it, you.

A

Go first, I can go next, I already think I know the answer to my question. Okay,.

B

So this looks fantastic by the way: I literally really cool, so I just Iran. So what's a different way, what the overview page is showing and when you drill down here so the overview that does it show a summary from all skilled objects. Yeah.

D

Yeah and then, if you want to look at the specific skilled object, you have to go to it's on like detail. Word: okay,.

B

And then how are you getting all the event stats and the logs? Are you just looking at the logs from the yeah for the controller yeah.

D

um Oh, No, okay, so okay, something happened. It didn't pick up the info metric currently, but it did fill up I'm back to that question, yeah, so I'm right now getting all the information from the logs, which is kind of an issue right now, because I need a more scalable decision, but essentially I'm omitting the replica count every 30 seconds and kind of picking that up, as well as the input metric um that emits logs every 30 seconds after your scaled object is deployed.

D

So when that happens, I can start like also looking at the metric count, but it really didn't know if that up. Oh, that's something I might have to face yeah.

B

I mean there's some other was like yollie such as the metrics from Prometheus, and that's probably a more reliable way to get that, rather than the logs that could be I.

D

Think that's something our theme mentioned, um I think just for the scale of my internship I didn't have as much time to work on it. So we went for this for now, but I think that's an option. They're exploring yeah I know.

A

You brought up some concerns or kind of about the scalability of how you're fetching the logs and and I was gonna. Have this in my question, which is once we pop this into the kata org I, would imagine one of the things that we want to look is making it so like. If you choose, you want to install the dashboard that we encourage you to also maybe hook up something like Prometheus, and then we use Prometheus to pull in some of this stuff so that you don't have to just parse the logs yeah.

D

Also because um since I am parsing, all the logs from the scale controller, um an issue with that is that, like the more logs you collect over time, the longer it'll take to load and parse through all of the logs and so I've been taking the last 500 logs um and that's also just like I'm, also losing information. I, probably a wand of the kid of this kind of scale controller and my scaled object over time. So I I did on. Where is it right? Over I I wrote kind of, let it I don't know.

D

You meant right over here, just kind of like an overview of like the code base and then kind of like how to contribute, and then I didn't mention it in like that. The first document, like issues that need to be like resolved to so.

A

B

Another question I had is around the authentication for the dashboard I presume. I know this is a very basic question at the moment and so I presume there's no authentication to get to this dashboard right. It's just us proxy into it. Potential yeah, yeah.

A

B

Yeah, we probably need to add something to authenticate as well, so like the kubernetes dashboard gives you the option of uploading, a cube, config or using a service account, or it reads the authentication token from from your HTTP header, to make the corresponding cube. Cto calls for you, okay,.

A

That's awesome that I get feedback so I see you've got like you've been working on this in your own repo. Have you thought about when you want to push this into the public Keita org? Oh so.

D

Oh yeah I didn't have access to making the repo so I'm, just waiting for are they'd, create it and then all all appreciate great.

A

Awesome yeah that'll be great, and then we can start tracking some of these issues in feature requests as well as well as getting contributions, so this looks really awesome. Thank you.

D

Yeah, that's about it it'll scale down soon, just.

C

Just want to echo what everybody else is saying. This looks really sleek. It looks really well put together awesome job.

A

Great, so with that, that's all we had on the kind of proposed Israel I saw you jumped in. We have kind of an abbreviated meeting. Today, I did record the first bit and I'll post the recording, because I want people to be able to see this dashboard demo as well. But Israel is there anything on your side, any updates or questions or things you wanted to chat about before we sign up for the week.

E

No I missed the last meeting, so I'm just joining to get updates today, I'm still working on my tutorials, you know for the the various scaling examples. So when I have the videos ready, I will publish them and I share with the team awesome.

A

Great well I'll just finish, updating the notes, then and I think that's it. For this week next week, we'll talk about some of the external scaler support, I'm, still gonna ping Aarthi and on that about reviewing TRS to hopefully get a good sense for both the authentication, one I think is pressing the jobs one.

A

Hopefully Shaun will make some progress on, but we'll see that one's kind of rich on space and then the external scale or one and a few other ones, but I think those are the big three and then we'll chat about a more next week. So anything else, anyone wants to chat about. I have.

E

One question Jeff, then I will sign up the docker hub repo that we have to publish our images.

E

So the process for that is like to do a pull request for the docker file and then like is a dedicated CI job that will publish to that repo. Or how does it work? I? Think.

A

It's manual right now arm Ed's been doing a kind of ad hoc. We have a work item before we hit one, oh, that we need to automate it all through CI CD, but it's any time I think it's about once every week, then he updates the master one yeah.

B

A

Of them are updated. We had the same question last week. It's somewhat automated, it's kind of not to like I, think this master 1 this job right here for circle CI, it might have the final step. He published a docker hub, unloading artifacts. Maybe not those don't look like the right, artifacts, so I'm, not totally sure. Okay,.

E

Oh no problem yeah.

A

Iii know I guess we could even look if I go to like our docker hub of docker calm is that where we go and if I search for kada I'm curious to see when the last tags were updated to have a sense so nine days ago, for master and 24 days ago for latest and if I look at the last merge on our repo, it was also nine days ago. So it looks like the master one. The master tag is updated automatically and then the latest one is just kind of ad hoc by Ahmed.

A

Whenever he does it, okay, which there is a work item to automate all that as well. Okay,.

A

Great well thanks everyone, a ton and chasman, especially thank you so much for that demo and for that awesome work and then I'll, post the recording and and put it in the links and and make some noise about it. So people can check out that dashboard as well and we'll chat again next week.

D

Thank you thanks.

A

Everyone thank.

D

A