Kubernetes SIG Scalability, 7 Sep 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2017-09-07 Kubernetes SIG Scaling - Weekly Meeting

Description

2017-09-07 Kubernetes SIG Scaling - Weekly Meeting

A

All right there we go so today's the today is September 7th. So this is the September 7th 2017 six-day meeting public, recording and.

A

Let's see here, I I was out. I was out last week, so I'm a little behind on what's going on for their specific topics. Folks wanted to work on today discuss today.

B

I / I am submitted like a ton of issues.

C

There were like there were a bunch of regressions that he spotted some of them were at least some of them were like configuration issues like the cube. The fact that that cube, mark large cube marks were failing was because, like it was misconfigured, it's still like. We still see some problems in real clusters, and this is still like, not debugged, so like she I'm looking into it now, I.

A

Mean should be here, so he may would I think I see I. Think I see him there with someone else. Is that.

D

Yeah, so so we fix this cube mark issue thanks to whole thing. You pointed out this problem with monster size, but we we're yet so for the correctness tests I filed like seven or eight issues already for the failing ones and like actually quite a lot of them actually got fixed, or at least people sent fixes for four of them, and people responded for at least two others, and we were having issues a couple of regressions.

D

I think there are at least two regressions with our real cluster performance tests on 5,000 nodes, and one of them is most likely something around recent couplet changes to their docker conflict or the container regime, and which is basically it is like rocketed.

D

The pod, startup latency is 216 seconds or something like that, and it was this I'm looking into this, and there is another issue with API called latencies, so the API call latency is starting from about a week back, increased like thrice and which also is probably some regression and yeah I'm looking into it. It should hopefully get fixed Friday by the end of this week, yeah.

D

And that's pretty much it and on a side note like I, recently added a presubmit for qumar 500. So if you guys ever come across any PR, you think can potentially break scalability or effect in any way or if you just want to experiment, you might just. You could just run the qumar 500 against the PR.

C

There to clarify this is not like an able type by default. It's like it has to be manually. It.

D

Has to be manually kicked because we can't effort, you might find it against I mean it would be costly to run it against each PR.

B

Did you have a? Can you explain that for a second, are you planning to do like a label to do like you know we did like a runt n label to detect flakes right, which people don't really use properly or haven't used? um Are you planning to use like a labeling system to ink you and then you know, evaluate whether or not it's worthy of 500 node kick.

D

Well, I think I mean for now. The the the plan was to just basically whoever is reviewing should likely identify that scalability. The the PR could have scalability, in fact, and just manually run it against the PR. There is no automatically kicking it based on labels, but that sounds like a good idea, probably yeah.

B

I think you probably want to do a batch right like you'd, want to batch them up at the end of the week or something like that where you'd have you'd have one label that says good, mark 500 or something or some some scale tests label or whatever you want to call it I don't care, but then it could batch them all up and then run them as part of your periodic runs right. So that way, it's it's it's exercised before it actually gets merged.

B

Because usually what happens is it gets urged, then, that window of when the coup mark 500 test run? And then you have all this other crap involved versus like having the results for just that. One thing right.

C

I see ya, oh thank you. Mark 500 is like takes one hour or something so like this window is pretty small. It's like way bigger for larger cube mark this 5000. No, it's like 15 horses, but.

A

You were, you were concerned about the cost and not running it over and over and over again Ferb. Where were you.

D

I mean more than cost I I think we definitely can afford it for PS. We really think need running it, but I mean the issue is if many peers are running it at once, we we might end up using a lot of quota, which is I.

B

Was thinking of just patching all the jobs that need it right and then just running those in a in a cycle, because then it would take your 500 or run to be whatever hours to be hours, plus X right.

D

Yeah, that's actually a good idea. We probably can add q mark 500 to be to the secondary tests before just merging like in during the batch merges, but not to the not actually run it for those like in the first cycle. But then it is part of the batch before submitting. Yes, I will discuss this with tests in front of folks. They probably should know more about.

A

Thanks, thanks for the update I have one I have one topic, just a very short update that I thought I would any other any other topics for today. I.

B

Had a question for Y Tech, you guys are deploying you guys are managing such large single node master clusters, which seems crazy to me. But uh has there been pushed on your side or is there still push to deploy a che configurations? We.

C

Will be deploying it? Yes like it's one we are or will be. We are not. We will be ok.

C

Like it won't be default, but it one it will be possible with someone wants to do.

A

The this was actually smaller and topic to my update.

A

Aaron I met with the CCS folks yesterday and they were they were telling us that they're gonna step up to some provide some account budget to run tests on AWS and we were kind of revisiting the provider config stuff. You know this is this: is the five node the five node hae master cluster and was really lobbying them hard for getting into testing around that configuration and it seemed to be saying we're gonna? Do it I'm not sure about the timing on that, but they're?

A

Definitely they're, definitely ramping up, so Tama I think I think we can I think we can look forward to also getting some help from the AWS folks on this front, going forward.

B

Doesn't even know if they're plenty to attend some SIG's still feedback signal at all or are they gonna? Oh I've, seen I've.

A

Seen them in and out of things they've shown up here, occasionally just not regularly so since our meeting was just yesterday, maybe they haven't wake up there. They haven't quite gotten organized around that. Yet we're I'm gonna keep I'm gonna keep lobbying them for it.

A

So anyway, that was my only that was my only update. If there's, uh if there's no other topics for today, we couldn't call it a, we can call it a wrap anyone. Anyone all right, I, think we're done for today see at least some of you at the community meeting Thanks.