Kubernetes SIG Scalability, 30 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-09-30 Kubernetes SIG Scalability Meeting

Description

Agenda and meeting notes - https://docs.google.com/document/d/1hEpf25qifVWztaeZPFmjNiJvPo-5JX1z0LSvvVY5G2g/edit?ts=5d1e2a5b

A

Okay, yes, so this.

B

Is catholic so uh yeah? Maybe I will start uh so uh today. Is uh this is sixth globality meeting and today it's uh 30th september 2021 and um today yeah. We actually don't have any topics on the agenda, but if you have any questions then um then you can proceed with them.

A

Okay yeah, so my name is kaifi kathy jung, I'm working at intel. um I have a question regarding the uh in place uh on the scalability that in place cloud vertical scaling features how. How is that going? um Because we we did some pro some testing and then we found out there are some. The um the latency is a little bit high, so we're thinking about how we can you know um contribute to that.

A

So, but first, how is that that pr going.

B

uh Could you repeat which feature because I'm not sure if I'm familiar with with this feature.

A

Okay, it's called it's called in place. Part part vertical scaling. Feature number is one zero. Two eight eight four.

B

uh Can you maybe paste it on the chart, but it sounds like it's vertical pod, auto scaling. Then I would say it's probably auto scaler.

A

Yeah, it's auto scanner. Oh it's! It's not part of this working group.

B

No, no, actually, it's not oh.

A

Yeah, I think, okay, I joined the wrong one. Okay, I see I got it okay, I see. Oh that's why? Okay.

B

Yeah, that's why I was quite surprised with with this feature.

A

Okay, I see I have actually I have another question so in your agenda: expansion performance benchmarks. So what is there any document on? What kind of benchmarks um are there or what kind of performance testing are you testing like? You are testing the qps, the latency yeah, something like that.

B

um Yeah much maybe do you want to answer a question.

C

Yeah, I can try. So are you asking like specifically about like the printf or like or in general, about like the six credibility.

C

Okay, so maybe I will answer both so like I said this c6 cabinet we are running tests in most of them is in either of one of two scales using like the whole 100 nodes or 5000 nodes, and we are basically running some workload watching for like first of all, if this workload, if all the operations are finished are finishing correctly and then, if all the slots are are met, and we are focusing mostly on the latency seo, I think this is like the main area of focus in our tests uh yeah.

C

We also have like some specific tests for like generating high qps and validating some- um some- I I can't remember exactly but yeah, but we are generating, like I think, like thousands of kps and and seeing how the system behaves something like that uh yeah. I I think that that's mostly it and then for the p and f, we don't have any benchmarks yet and what but this is. This is going to change. I hope, hopefully soon.

A

Okay, so you're saying you're running your cluster size is 100. Node is that right and then you are testing 1000 qps and then to see how the system.

C

No like so so, basically, we are creating cluster, let's say like with 100 nodes. Then we are trying to create. I think thousand posts poster like those spots are gathered in some deployments in some demon sets stuff like that like and we are creating some some services based on based on those and then we are running some uh rank updates and and then we are deleting everything.

C

So all this time we are watching for the latency of this of the system and and and we are checking if it's in our slo, just like one second for most of the cost and like for some. These lists cause it's is higher, so basically, this is what we are doing and uh just to clarify. We are also doing the same in like 5 000 notes case and in the 5000 we are creating. I think 100 50 000 pots in in such a such a such a cluster.

C

Additionally, we have a test which creates, I think, like some some like. I can't remember exactly what I think few thousands of qps, but I would say this is like the less important that's for for us. We are mostly focusing on. Doesn't that the first one I described.

A

Oh okay, thank you is that, like a document on this data, like you know, for example, one 5000 nodes and then how many workloads and then you are mirroring what kind of things like this? Is there any document on the test that set up like some yeah? What do you just described.

C

Yeah, I hope I hope there should be some document discovering that, but I'm not aware of anything anything like that, so uh I think the only source of truth like the source code. Unfortunately, so I guess maybe just action item for for the our our seek to to maybe uh make it more clear on yeah. That's basically some some documentation describing that at least at some high level.

A

Yeah yeah, that would be great yeah. I don't know what testing you are doing and then what result you are getting so yeah. I think it's good for people to have an idea. Oh yeah, this you know for this co-base. You know the medical base we can support like you know how large the cluster is right. What's the qps, what is the latency.

B

So with the kps um so with the qps, it's actually um like you shouldn't think of it as a qps, because um in terms of requests, there are different requests and basically they like, for example, I can imagine cluster that you know um is doing 1000 qps just getting configmaps secrets, but then you can have second cluster with the same size but, for example, listing all the pots and those requests are much more heavy and because they are much more heavy, then you will not get 1000 qps.

B

So qps is not usually the the value that we are interested in. uh Actually uh we merged uh this. This question about the qps to one of our uh documentations. I will post it soon when I find it maybe after the meeting I will find it and also we have second document where, uh where we are kind of um saying, okay, so the kubernete there are some kubernetes kubernetes limits and uh and for example, we are supporting, let's say up to 5 000 notes.

B

So there is also this kind of difference that we are maybe not only saying. Okay, we are testing this this this because we are changing. Basically, our test. All the time, but there's also official list of uh limits that we support.

A

Yeah yeah, that makes sense yeah, I know yeah different, you know, depends on what kind of request right, you're sending right so might the qps will be different yeah. I think that that makes sense um so yeah, so you are going to post a doc. Your test load test document somewhere right.

B

Yeah, so um this meeting has uh agenda file and uh I will add action item for me and after the meeting I will find those two documents and I will send them. I will just add them. Add them here.

A

Okay: okay, great thank you! Yeah.

B

And actually um yeah um do you want to talk more about the pnf benchmarking.

D

No, I haven't got around like I looked up briefly, but I haven't. I don't have any pr to review yet. But yes, I I plan to work on it. So, okay.

B

B

So maybe I would just add here that.

B

Okay, so um do we have any more questions.

A

I have one more question: does this group cover the cluster scaling cluster, auto scaling.

B

No, no, that's the other one as well.

A

That's the auto, auto scaling group: okay,.

E

A

How about the horizontal part hpa also, the other group.

B

Yes, so basically, you can think of it like scaling up and down it's like automatically scaling up and down its outer scaler, and our group is making sure that one once it's scaled, then it works.

A

Oh okay, so your group is responsible for to make sure everything works when it scales our outer skills up or skills done scales in right. Something like that.

B

Yeah, I think it's good approximation.

A

uh Okay, so where can I know exactly like you are, like you know, for example, you have any new features planned for your. What group um any new features that your group will work on, or is there any a list or document which.

B

Have that um so actually, as a as a sixth scalability, um we have multiple different goals, which is sometimes it's also some improvements in terms of scalability but, for example, first and most important is protecting from scalability regressions.

B

So we are constantly running those ci tests and basically analyzing all the regressions, but also sometimes we we add some features and um I would need to think, but I think recently we had three three major improvements kind of, but we often work with other sikhs to to improve some particular parts of kubernetes as an example. um As an example, for example, there is, um there is a priority and fairness and um we were quite heavily involved in priority and fairness from also design perspective, but but basically improving.

B

um The other thing could be um immutable secrets. So if you have regular secrets, then um then what happens is that when the secret is mounted to uh to pot, then then kublet gets this secret from cube api server and then opens the watch and with immutable secrets. What happens is that it just gets the secret, but it will never get an update, but this, for example, also reduces the number of watches on api server. So these are two two improvements that I can think right now that were made quite recently.

A

Okay, got it okay, so basically, this group is concentrating on doing the ci test or whatever performance test, to make sure you know everything works um um when it scales out or something like that is that right is my understanding, correct.

B

I would say I.

C

Wouldn't say like this is yeah, I'm not sure like. I don't really agree like with this this statement. I would say that this is a significant part of what we are doing like as when the regression happens in our tests, like the debugging understanding, which exact pr is causing.

C

That might be very time consuming, but we I I just want to say that we do contribute to new, like features um usually like developed by other sticks, like I think, like the pnf, is currently a verbal topic uh that we as a secret, are contributing to to it. So uh it's not that you're only like running tests. There are also like the the changes, the um kubernetes, that we are involved in yeah.

B

Basically, based based on those tests, we are either finding regressions or finding places in kubernetes uh where we can improve, but also, but also we are consulting other six in case of uh they came up with new feature, then as a six capability, then we are also consulting them to make sure that you know this new feature will be scalable.

B

F

Okay, got it yeah. Okay, thank you. Hey guys, um quick question on that. So I saw marcia found this interesting issue with that. One second, for example, class uh calls call latency going up for exam class. uh Well, I I had a question there. So much do you know how come we didn't catch that in 120 itself uh did it was it hidden or like our tests were not sensitive.

C

So, which regression are we referring to.

F

Oh sorry, the so there is this thing with.

F

Trying to find it.

F

Okay, so this requested, this is the issue um I checked reported this one.

F

Nice dive deep there, um it yeah, I think what I didn't understand is how we caught this in 123 and not 121. Was it regression in 123 or.

C

I think like so so.

C

Yeah, so I think like this, uh no not sure like.

C

Basically, like this, this lock contention was with us, like, I think, from the like the very beginning of the project furnace. As far as as I understand so, I guess from like 1 20 1 19, something like that.

C

F

F

Or or like data or 120 121 like back then did our tests also cover apn f for uh or we enabled it recently for our 5000 node tests.

C

I think they they were covering like pndf yeah yeah. That's true, but maybe maybe I misunderstand like: are you referring like because uh like are referring to like the the last comment or like one of the, because I think there are two changes in in the in the issue you linked. So the first one is that uh basically, we are seeing some lock attention. It is causing some, let's say regression that some calls are slowed down and then in the last.

C

So then there is this fix which like improves this this uh locks, and then we are seeing that the latency of like the exempt cause, is going down to like it's going down. But on the other hand, we are seeing that some, some uh latency of some calls is growing like, for example, like the note high uh priority level is, the latency there is is, is growing, I would say even significantly so, and this also can be considered to be to be regression. So are you frankly, like the depressed one like that?

C

The those exempt like that this lock contention are like to the effect of the fix of these lock contentions, which has caused some latency to some other calls.

F

So so the second one you said about existing calls like the non-exempt calls uh seeing a latency yeah. That is actually now the right behavior, but it got exposed because of fixing, because I I assume the exempt calls are. It is happening because the exam calls are being handled first.

D

F

Okay, okay, yeah.

E

So I was actually referring.

F

To the the one second one, where the exam calls.

C

So so so so yeah, so basically, I think that this is like with us for forever, like since, like the very beginning of the preparation furnace and frankly, it wasn't happening, it wasn't failing our test too often right, so it was just a sound flake. I I tried to debug and found.

A

C

Some some some issue there, so it it was, I would say it was happening pretty pretty rarely so maybe for that reason we haven't like to catch this. This issue in, like the past version of the quarantines okay.

F

Okay, okay, then that makes sense thanks.

A

I have a question regarding the latency: uh is it like into an agency? How do you mirror latency from the client side, when the client send the request and then, by the time you know the client uh get some notification or query?

A

You know um to make sure that workload has been deployed or you may have different latencies for different segments.

B

So this latency that uh magic and xiaomi we're talking about is latency like pure api calls latency.

B

But, except for that, we also do have some slos for different latencies, that maybe you you would be interested more, which is, for example, pod startup latency and put startup. Latency is basically measuring the time from when you create pot and till it's basically running on the node, taking into account that the node needs to be stateless.

A

Okay, so when you create, is it married from the user side crunch site so when life? So first, if I'm a user, I send a pub creation, request right and then you know until the time that pause is created and running in the node. Is it like that you know?

A

Is it like entering latency or is uh um because I know credit card. It goes through several stages, right, first, api server and then the scheduler right need to schedule.

F

Yeah exactly and then eventually.

A

Kubernetes, you know ticket and then you know, of course specify the note right. You know that the complete on that no ticket and then go schedule. It go through the wrong time and then you know create that part. Eventually.

B

um Yeah, so that that's exactly this latency plus at the end one once the pot is running also, we include the time that it takes for watch to observe that this port is running.

A

Oh so include the time the use- okay, the user eventually observed that the part is running right. Yes, yes, so how does it affect? How does the user know the part is running? Does the user have to pull? You know, keep like get the list of status of the pod.

B

um So actually, uh api server has um like two main kind of requests. One you can, you can pull basically like you said like basically to get requests, but instead of it, api server also has watch requests and basically they work in a way that um you open the watch and you you say that you are interested in this particular pod, for example, and then, whenever anything changes for this part, you will get an update.

A

Yeah right so then, that that means the user needs to register with a api server. This is a watch with the api server right. The user need to write some code to do. That is that right, so that the api server will notify. Oh, you have some script running.

A

That script will does the registration with api server to get notification of that pause status. Is that how you do it.

C

Yeah more or less yeah, so we are opening watch watching for like we are waiting for a given pot to become like the ready, and we are noting like the current time and then comparing this with the creation timestamp from the post specification.

A

Okay got it. Thank you.

B

Okay, so um do we have more questions.

B

If not, then then I guess we can finish a bit earlier today. So thank you all for attending and see you in two weeks.

A

Thank you. Thank you for answering.

C

All my questions.

A

Thank you very much.

A