Kubernetes SIG Scalability, 16 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-03-16 Kubernetes SIG Scalability Meeting

Description

Agenda and meeting notes - https://docs.google.com/document/d/1hEpf25qifVWztaeZPFmjNiJvPo-5JX1z0LSvvVY5G2g/edit?usp=sharing

A

Okay, so um this is six capability meeting a 16th of March, 2023, um yeah and I'm, wondering if you have any topics, I didn't see anything on our agenda today.

B

Yeah, so this is the first time I'm joining the meeting, just kind of uh just getting our feet wet into the uh into what you guys are looking into working on and uh Sean just kind of uh added us yesterday to just get an idea of uh you know what you guys are trying to work towards.

A

Okay, yeah, uh hey Jim, um do you have maybe some topics to discuss.

C

Hey uh yeah so able to hear me yeah all right, uh yeah, so I think last six scale we discussed about um so Google is reducing the frequency of some of the tests right with scale tests and the pre-submits and stuff so um yeah. So I had a discussion within the team here and see how we can help especially, is respect to. Can we kind of load balance, how many scale tests or any any particular areas that I think we can?

C

We can start with uh by contributing so from from the AWS team we have uh uh today Daniel, pratik and Ashish, um so essentially we should uh and avoid traffic isn't here today right. um We would have loved having him here too, but Marcel. If, uh if you have any inputs there, essentially um what kind of tests can you get started? I guess we'll need to um do some setup and figure out how to get these tests, especially with the large scale configurations running.

C

We start with a smaller scale, um run like a weekly or buy like a alternate day tests, or something like that. So you should. We should talk about that.

A

Yeah yeah I think I think it would make sense to you know: um I'm not actually sure if, if there is currently support like in the cube test for AWS to to deploy the uh cluster for AWS, there.

C

Is support for cops in Cube test and.

A

C

In in the past, actually like uh then, to you know than three years ago, I added a cop's job for AWS for 100 node uh periodic job, and that was running, and that is passing also for a while. But I think that wasn't maintained. I didn't have a chance to kind of keep up with the job and eventually I think a bunch of things changed and the job broke and it was removed.

A

C

On those things, change I think cops uh because I see some co-op selected jobs outside of six scalability, like in six release and stuff for AWS on the on the test grid, so I'm, assuming it should work you will have to figure out over time. If, like cops, is the best suited tool or if we want to kind of you know, um use something, that's more convenient for for skill testing on AWS, but I think to start with yeah.

C

It's like kind of the low friction option.

A

Yeah yeah I think it would make sense and then I'm guessing like you would start with, like 100 note tests and then once we prove that it's stable, we can probably do the alternating that we mentioned last time right, like the 5000.

C

Yeah, um asking Daniel or later.

A

uh I don't know, I was asking you I think right. Sorry.

C

I thought you said: Daniel yeah.

A

C

I think, yes, that makes sense, essentially um we'll start with the 100 or 500 nodes and kind of get these tests stable and running then try to increase it.

A

So I'm actually wondering if, uh if for like, even like 100 best notes, um do we need like some additional AWS projects or or some resources, probably like the whole infrastructure right.

C

uh You mean with respect to uh projects, as in you mean like uh where we'll actually bring up these resources, billing and stuff, like that yeah yeah, exactly yeah yeah yeah. So for that, actually um that we are still figuring out I guess what uh what the next step for this is. We will so there's a. uh There are some credits that AWS pledged for the cuminous project.

A

C

Year, writer and I: guess it's three million dollars or something like that, um and we want to use that to kind of help. The project with CI testing and those kind of efforts right and one place is scalability testing, and this is the definitely a place where you can spare some of those credit rates. But these credits are actually their own by cncf.

C

um So essentially, what we'll also have to do subsequently is we'll have to talk to the Sig kts infra, okay and I. Guess they have a week your bi-weekly meeting on Wednesday, so we'll have to talk to them and also make sure that um it's okay for us to use these credits to help with scale testing but I think um I. Think the footprint of this shouldn't be that big.

C

Just yet uh if you're gonna run like a one job with five thousand nodes and then we'll have to figure out some of that on the Fly Like. If we have to Spread spread these credits uh with other efforts.

A

Okay, yeah yeah I think it makes sense yeah.

D

C

D

Topic like when we are starting with 100 node test, like what's the trigger point like how do these gets triggered like when there is a PR or during release like? What's? Are we planning something? No.

A

So so the trigger you should so actually before there were figures like it was triggered based on PR. But since, like you know two weeks, we disabled these and right now um basically what's happening. There is a CA job that periodically like every I, think few hours um creates Pro job, and this Pro job basically runs the scalability test.

D

Okay- and this is what we also want to do for AWS as well like periodically, we will be running the scale test now in AWS platform as well.

A

Yeah I think so right cm.

C

Sorry I wasn't paying attention to the last one. Can you say it again, pratik.

D

So uh I think Marshall was explaining that there is a periodic brow job that runs the scalability test currently so for AWS testing. Also, are we planning to do something similar where we'll have yeah.

C

D

C

Yeah I I think yeah, so basically I'm higher options is periodics and pre-submits so period. We should definitely start with periodics because giving continuous signal and if we actually run it with enough frequency, we can actually, let's even if this regression and these tests start breaking. If you run it with enough frequency will have a small window of uh for changes to look at like what regression happened.

C

Pre-Submit I think pre-submit. Also, we should figure out. We used to have 100 note, P, submit on GC and gcp. Before recently Marcel was saying we, it was stopped because running it on every CRA was turning out to be costly. It was running as a like the credits. Gcp has. We are kind of exhausting you're burning it faster than what we have for this year.

C

um So what we can do there is that we can figure out something frequently there's a way to also add a pre-submit job without mandatory running it.

C

So you can kind of add an optional pre-submit job which, by default it won't run, but you can kind of kick off a bot command on the GitHub here and and we we actually started this way even for the gcp jobs we didn't make it PR blocking, but we made it optional in case someone wants to, you know, run a scale test, so we could do something like that as well. Yeah.

A

I think even like right now we do have optional scale tests for like 5000 nodes.

C

A

Think it is I think it was maybe not anymore. I I, don't know wow, okay, it.

C

Was in the past I think there was two thousand a thousand at some point. Take long time back this one yeah.

A

Yeah but yeah, but we didn't.

C

Because I think we didn't want to go higher because if any member of the community can trigger it, yeah.

A

We haven't used like much this 1000 or 5000 scale, because uh we didn't have anything to compare against it. So I think we switched to five, but maybe it's not anymore. There I mean it wasn't used often right um only like on some really like critical peers, I think.

C

And I think we also discussed an idea around. um We can restrict the set of people who can call that bot command so.

A

That we cannot, we cannot.

C

Call left and right by people like uh maybe six we can. We discussed this idea that small group may be like six kilometer approvers or something like.

E

C

Actually trigger these tests on peers, because it's 5000 note, then we don't want it to be triggered carelessly yeah.

C

um Anything else, I, guess still early days it's much of the way you should think about it as you're kind of fingering out and if there's anything that from your end, let's say because I know right now for these regressions any scale tests, you're kind of like on the point for some of these uh resist blocking issues right.

C

um So if there's any chance that we can take off some of the burden there I think we should. We should talk about.

E

uh I had one question uh sham, so are we also planning to introduce some new new scale test scenarios or like it will be uh using some existing skill testing methods that we have.

C

um But you mean, like the tests itself, that we run or the Frameworks that use with us yeah.

E

C

C

E

No I was just checking out like are we also planning to introduce, introduce some new scale test scenarios uh for kubernetes or, like we'll, be using some existing uh scale test Frameworks like a verb test? We have.

C

Yeah it's there. We have some infrastructure already, which we can reuse almost pretty much in the current form like.

A

We're visualizing.

C

Metrics per Dash and the tab we already use like things like cl2, for example, it cluster loaded to yeah.

E

C

Tests and there's this whole Suite of tests there um for different different things right. uh We, the sixth skill itself, runs two particular jobs. um I. Think at this point it's probably more muscle know it better, but there are different uh tests that we run and we can also extend it to more.

A

um Yeah yeah yeah there are, there are a bunch of different tests, but there is like the main one. I would say right and and probably when we are talking about uh testing on AWS. We are thinking about the the main test right, like the 100, node and 5000 node test scale test that we call basically load test.

D

Okay, yeah, and also on that note, a couple of questions like I know we have this buffed at test grid and Perfect Like These are dashboards that we monitor today, because these are periodic. So we want to monitor them over a period so going forward. Do we envision like one common dashboard that is showing tests running both in gcp AWS or the reflex separate dashboards, I.

A

I think I think probably separate dashboards, because I can imagine that you know um like the regressions, can come from many different places, starting with kubernetes, but also gcp infrastructure or AWS infrastructure. Right so and also the character characteristics might be a bit different, so um I would be in favor of actually splitting them um and having like two separate dashboards, one for gcp and one for AWS.

C

Are you referring to the perfect or the uh test grid.

D

um So yeah the test grade, one I see there is so I'm I'm opening this for the first time, I'm not familiar sorry but yeah. This is test grid. It says six scalability, six calibility GCE, 6 calibility node I can paste it in the chat.

C

Oh yeah yeah this one right yeah, so there that I think we will have to have a different if it's a different job. Each job gets its own page. There and jobs are grouped together in under buckets like six scale, cigarilles and stuff, like that yeah, um so yeah I think we'll have kind of maybe a six scalability AWS or something like that.

B

C

Question yeah there'll be some more questions. Also we'll need to answer good Daniel.

B

I just want: what's the current process like you, guys, are running the periodic test and you notice a significant enough regression. What's the current process to act on that.

A

So um basically, um current process is that we have uncle, and um you know, um they're basically Fox Folks at Google, who uh who like our uncle like for a week and whenever it fails. They take a look and see whether it was kind of like lake or if it was actually, you know um happening uh constantly or if there is any regression in latencies all this kind of stuff and then based on that, we either like look for the root cause or maybe like the pr or like any other possible change.

B

Infrastructure and stuff, like.

A

That yeah yeah exactly okay,.

C

All right, this link, you paste it.

C

I didn't know, there's another seems like this is another copy of the test grid.

D

C

D

Yeah I got it from actually six scalability read me.

C

Oh I see: okay, actually, both of them went to the same page, interesting.

C

All right, I guess, uh I see others on the call too, um unless someone has more questions on this. Is this well there's going to be an ongoing discussion for a while? Some are still expect that it will folks will join yeah.

A

That's the call and.

C

Kind of specially said, set artists of the starting days will be a little bit of a figuring things out.

C

um Any other topics for today's meeting.

C

Any questions or hot issues to discuss about okay, there's one I wanted to bring up real quick uh once in the annual Sig update, I. Think someone from the um someone from sick really is speaking a while ago, I believe around uh scalability anal update, I, believe white Tech said he will be drafting it. Do you know when that update is gonna happen?.

A

You will share you, you mean the the yearly update yeah yeah, that one um I have no idea to be honest.

D

C

Yeah I think once that update is there uh I guess we should share that, maybe with the group here or kind of make it accessible somewhere in in the six kilometer, either or page or.

A

Yeah I think I think it's like you know the the common uh there's like one place where all the reports go like every year. uh I don't have link right now, but oh I can find it I.

B

Can just put here like.

A

C

If you can link that from our uh Community page for six kilobity, then.

A

C

Yeah: okay, thanks.

A

Okay, then a guess: if we don't have any more topics, I get I guess we can just finish a bit early and give back seven minutes to everyone.

C

Yeah yeah. Thank you. Thank you.

A