Kubernetes SIG Scalability, 11 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-05-11 Kubernetes SIG Scalability Meeting

Description

Agenda and meeting notes - https://docs.google.com/document/d/1hEpf25qifVWztaeZPFmjNiJvPo-5JX1z0LSvvVY5G2g/edit?usp=sharing

A

um So this is six collability meeting um 11th of May, 2023 and I can see that we have um topics today. So maybe, let's start with the cube field.

B

Yeah I I, uh um I'm, Ryan Halsey, now I'll I'll, talk, I'll talk back here, so there's a few people here today also who joined the call from from this group. So just so you don't, if you don't know um the keeper project, we have a. We actually have a sixth scale group in Cuba. We do. um We do some work, testing, scale performance, and so we had a few people joined today.

B

We wanted to I met with them with bush Tech and uh at kubecon and I'm gonna share a few things of what we've been doing. So, if you don't mind, I'm gonna share my screen. Yeah.

A

I think I need to give you permission, give.

B

A

Second, okay,.

B

I can also share it with you, the slides but I can't.

A

um Okay, I give you like okay, so now you should be able to host to present I think let.

B

Me see: okay, let's that should do it. Okay, you should see it now great.

B

All right so I put together just a little bit of content. Just so we can show you a few things of what we've been doing and have a discussion around a few ideas. Okay, so the the keyword six scale. Let me I'm gonna move my panel here. Okay, there we go all right, the keyword scale so that our mission at cubic scales, what we've been doing is um uh so we covert maintains its own API server. It's got its own component. It's got a scheduler, we've got a bunch of things.

B

We've got our own workload um that runs as a pod, but we, you know, we have our own apis, like VMS, vmis and so forth, and, um and so our perspective was that we should have our own scale and performance standards, tools and best practices and so on, and so that's really what we've been focused on in six skills. We we're focused on on these things, building out scale, performance standards and many of the tools around it.

B

So I'm going to talk a little bit about how we do the measure so you're going to see a common theme throughout this is um we want to use all the common tools about how we measure and that's Prometheus or you know, having something on a local laptop. So what we do is um we leverage.

B

um We have two metrics that we've created Prometheus that we use heavily to do our uh our measurements for performance at scale. First, one is it's called phase transition times and um I'll I'll go through on more detail. In that another slide.

B

The second one is the client go HTTP calls to the kubernetes API server, so the phase transition times we primarily use to measure performance like P95, we'll look at like how long it takes for for us to go through a few different phases, our virtual machine to go through efficient, different phases to reach a running State, and then we have like. So. The client goes think of like how many hcp calls are we making to the kubernetes API server.

B

This is like how many patches, how many times we patching our virtual machine um during its life cycle, and this is important as obviously for a scale right. We don't want it. We want to be a good neighbor in in the cluster. Keeper is just a guest. We don't want to be sending thousands of patch requests or unnecessary patch requests. So we monitor these things to make sure we we're being a good citizen in the cluster okay. So those are the two um metrics we use heavily so closer. Look at what these look like.

B

um So this phase transition times what we did is so the process or I guess the I should say the inspiration for this. Is we took that idea of a creation timestamp that you see on like pods or other objects and deletion timestamp things like that, and we extended it to um also include other things that were important to us. So this is a picture of a virtual machine instance and um what this?

B

uh What this has is the specific phases that uh the virtual machine will go through before we're actually in a place that a person can use the guest, and so these phases, like pending scheduling, scheduled running or all like, have different steps like pending, will be like. Okay, we've we're look, we're waiting for the scheduler. The coupon is scheduled to um to uh find a place to uh for um to land this. uh To win this workload, scheduling like we're in we're in the process of being scheduled um somewhere and then scheduled means like we've.

B

We've found a node where this process, where this guest is going to land and then eventually running running, is, is basically just means that the domain has been defined like we've um like we've actually created it went first, um doesn't necessarily mean the guest is running yet, but it just means that we've actually defined the domain, and so basically the the idea is we have these this information? We have this granular look at what's happening, and so now we're exposing it. We've basically taken this, and we have we've sort of done two things with it.

B

We have um we post the the um the timestamps in the status on the object, and we do this when we actually go through the phase transition, so we don't like, for example, we don't like to two update calls to do this. We we take um in the same uh uptake or the patch call. We will um to change the phase.

B

We actually will post um this face or in just some timestamp at the same time, and the second thing is we also get in Prometheus, and so what this does for us is we end up with um stuff like this, like we build dashboards around this, where we, we can clearly see like okay, here's like what we go through for this BMI, like here's with those granular phases.

B

We can see um all this different, um all the different data about um uh the phases and how long they take roughly, and so this gives us a really good picture of like how we're performing um in our clusters. So we see like lots of patterns, we see sometimes different phases take longer, and it can it's really easy for us to tell okay exactly where we're getting stuck and also for. If you know so for running that in production or if we wanted to do measurements, um we can see like papers where our bottleneck is.

B

Okay, so I'm gonna go to the HTTP calls so like I mentioned earlier. This is like thinking patch calls like we're making to the kubernetes API server, and so what we're doing is we? We want to measure this all the time we want to run this. um We basically we do.

B

Is we created a bunch of CI um that will constantly measure this against different PRS and we catch when we change this uh when when these linear changes, so, for example, here two actually in the last I think um yeah I mean this is just in the last uh five months where we caught um an increase in patch requests, and these were the two pull requests that we got them and then- and it makes sense like these- are have to do with like cleaning up finalizers and dealing with um controller revisions, so they're they're important for actually doing uh management of of the virtual machines.

B

But but the point, the important point is that we can see this. We can see this, as you know, as the code base evolves, we can see how this changes, and so obviously, when we can see this, it means that that hopefully, we can eventually apply a number to this and get us a sense of like okay. You know this is this is how we could affect scale like we're doing a lot more patches.

B

Now with this, when we pull on this pull request, this could affect scale in some way, so I I wanted to open up for questions or discussion, and this is just um these are just some ideas of like things that we've looked, that we kind of look to to want to develop when you know with uh with you guys and for kind of the question, I guess that I get all the time. It's like it's like.

B

How long does it take to get a like a virtual machine up and running if it has a bunch of things with it?

B

Like storage- and you know, we use this kind of cni whatever and, and the answer to the question is that it's it's unknown and, and the reason is there's just too many variables for us to be able to know this, and so like I mentioned you showed earlier on the dashboards like we can see with all our granular phages like how long things take, but I have no idea like an even even if I were to zoom in even closer to some of the steps.

B

How long it takes for like a PVC to attach like I, don't I, don't know what it is and, and so that's kind of one of the things that we at least initially was looking to collaborate with you, guys and um and I'm sure. There's other way, other ways as well, but like find ways like that, we could look to measure some more and look for ways that we can uh try.

B

And you know, zoom in on some of the different characteristics that Argus the different phases that um different workloads go through with pods and or even things in the scheduler of PVCs.

B

um So these are just some examples like PVC attachment, maybe Network attachments um or any other pod conditions that we can think of Beyond, just a creation time step or some things that we we had in mind the things that could provide a um a more granular picture of like what an end-to-end flow would look like, and so anyway, I wanted to open up for questions or thoughts on the site. um I'm, definitely open hearing what you guys have to say and that's my last slide.

B

So any questions or thoughts that people have.

C

Yeah so I think, thanks that that was great, and that was like um um super useful um I. Think the in general, like the defining like what or figuring out like what takes a significant part of time. Is it's not a problem that we really solved well um anywhere in kubernetes? um I? Think that the idea or the path that we are we are trying to to go with um through their project is like integrating better with tracing, but we still didn't figure out how to do it.

C

Well, so we are focusing on, like smaller parts of the system, now like um I paste it to the chat like one of the two caps that we are trying to pursue, which is like integrating cubelet with tracing um I think this did like there's also another one that like that I can find, which is like integrating API server with tracing.

C

This doesn't like address the um neither of those addresses like the the like um they really cross, cutting operations that take number of controllers, and so on and so on, but uh hopefully based on the learnings that we have from from from those, we will be able to somehow better instrument. The system um well I, guess like the the the main point that I wanted to to say is that we didn't really figure it out well in the project anywhere. Yet.

B

Yeah I um I from uh like from my perspective, um I, don't know the system. Maybe you can tell me if it sounds crazy, but my understanding is that- um and this is just a like I, don't know I'm just guessing here like we have. We have PVC is a bunch of phases. Right like this. Like I, don't know you can I think there's an attached phase, there's a pending phase or something like that.

B

um Well, I was wondering like if, um let's say we like, we were to pursue something like this kind of similar to what I was talking about with like what we do now with virtual machines. Would it make sense like if we were to um you know the moment when we a PVC changes its face to some other something else attached or whatever we emit a metric?

B

Maybe it doesn't have to be on the object. Maybe we could attach it to the objects like the Christian timestamp? Maybe it doesn't make sense, I don't know, but even a metric would do um I mean I. Think that would help a lot. Does that make sense like as a as an idea like?

B

Do you think that sort of do you think that would sort of get us closer to what our goal is like um like answering that question of like how we can get through like the more granular phases of what goes into getting a pod up and running.

D

I I think the I think what I found a bit unique about. What you are doing here is you're, trying to kind of embed the trace into the status of objects.

D

um So the metrics uh aspect that you're talking about we I believe we have it at few places already today, for example, with the API server for different parts of the API path and cubelet. Also, for example, has some metrics around Mount volume, Mount, latencies and stuff, like that, the the thing with them is they they aggregate. So it's you kind of know more about the trends and like how, in general things are doing, uh it depends on what you really want.

D

It's felt like when you, when you showed the the crd for the virtual machine. You want to have the record per object, um so I! Guess, if you need that level of granularity you, um you will need to persist this kind of data somewhere now, whether you do it on the object itself or I was actually just looking at the the cap that ytek shared and there it seems like if I'm reading correctly, they are trying to use some sort of a different uh structure for actually holding the traces that are disjoint from the object.

D

C

Not even an object, it's like integration with open tracing in general, okay,.

D

Open Telemetry collector.

C

The names of that were changing over time, which is the current one here: okay yeah, but in general, I I agree with like what shiam just just said that, like it depends on the goal that you want to achieve. If you want this for debugging, why my single instance took longer or yeah took longer or whatever like what was happening, um then metrics aren't super useful for that.

C

If you want to um use it as a general information for what are the slowest part and what we potentially should focus on to optimize, then metrics sounds like a reasonable option.

B

Yeah, it's it's the latter. It's like right, finding like if I were to it's like answering that question. Like you asked me: how long does it take for this virtual machine to go from you know the moment. I created it to the moment. I can hand it off to someone to use it's it's getting it down to getting the granularity enough. That I could say. Okay, we we clearly have um a bottleneck with whatever we're doing in storage.

B

You know, I can see it in and I can see it in our metrics that it takes it's taking longer than right now than it should be. You know whatever that is and I can. I can clearly see that or whatever other phase that we could point to as as being problematic.

E

um One more thing I wanted to add like the metrics or things you are looking for are mainly coming from cubelet side and they are not from the API server side like Network attachment were um and the volume attachment, so one possibility could be like something which API server does. Is audit logs and start in the recent kubernetes versions in audit logs we have started adding annotations for each step so where the latency is coming from, you could see in those annotations.

E

We could do something similar in cubelet. Maybe it's audit log or something else with the annotations where, when a pod is getting created in cubelet a network ads Mount sorry volume gets Mount, it's an annotation and network gets created, that's an annotation, so when you are checking for a particular pod in cubelet like why why it was delayed, you could just see those logs in cubelet and tell that could be another way.

B

The tricky part with going with logs is so now we yeah that could work. No, but the problem is that, like how do we view it? How do we take this and analyze computer and then and then we would need to have and we'll probably be like Cabana or something that's which is fine like we could you could do it that way?

B

um I guess you could use Cabana to get you that information.

E

So they didn't get the last last part. What's the tricky part of.

B

So what else yeah? What I'm saying is like, so you would need something to so like like it's, it's aggregating the data. How do I take the data and analyze it I, guess that would be the problem by the way like I'm, not just going through the logs and picking out individual pieces and looking at the latency item, I'd want to have a tool to to deal with um how I'd need to analyze it so, like you know, maybe Cabana or something that's. What I was what I was saying.

B

Maybe you could use to to accomplish this I, don't know if that's the purpose of the audit Beyond a tool, I've never used it, but it sounds like maybe you use Cabana to analyze. It.

E

All right that makes sense. No. This is more getting granular data on on a particular pod, uh for example, but if you or as Sean, was mentioning if you're looking for Trends, then the metrics would support that.

B

D

Also, like the other thing, is, um if you're capturing every single phase transition on the object, and does it mean that over time your object is going to grow or you.

B

Can actually responded.

D

B

A fix, it's banded, there's a fixed number of them. This will make and we're only going to post. um So only these um four and then what we do is like. We use the creation time snip and we also have a deletion time step and that's so our starting phase creation time set completion time steps then phase, and then these four in the middle.

D

All right, okay, yeah so today, in our scale tests uh this. This is a bit interesting in the sense that in our scale test today, we also have to measure this the one that uh we run using cluster loader. We, we are kind of getting these bits of informations from things like events and uh like modifications to the Pod itself like create timestamp, shity events and stuff, like that, your uh yeah you're.

B

Making it so how? How do you? uh How do you get the the scheduling like when you send like scheduling? What is that that's like so the when the Pod is, is what.

D

It's maybe oh I, think of myself. Remember. Is it the event the bind event or is it something on the board.

D

I, don't know if the part itself marks.

D

But I, it's essentially I think one of one of those two uh where we have where we either check I. Think for the event that scheduler emits when it um when you schedule, support or write something on the Pod itself, I can't recollect, which field.

C

Yeah I think if I remember correctly, like for scheduling, we are relying on events and for which.

B

Isn't great by.

C

The way, um but it's where we are and for phases within the world Cuba, is doing I think we are relying on fields from from the Pod, but we also like extended the metrics on the cubelet and we would like to- or maybe we already did, I'm not following that closely to be honest, but we are trying to migrate a bunch of that to to rely on the metrics for um for aggregations and for like get for like the aggregated view of the of the like how system is behaving and like on those individual fields in the pods or events only for debugging, individual stuff purposes.

C

F

I um I had a follow-up question on that, so I think um the the way how we collect the metrics as in is it coming from the phase transition timestamp or is it coming from the cubelet that really uh does not matter? All we need is a breakdown of let's say if the creation to running time went really high um after a particular release, was it related to a cubelet or was it related to Something in the cube Word Stack?

F

So keeping that in mind is there a reason why um these scheduling times are collected from the events and not from the scheduler emitting those latencies um in in form of metrics.

C

um I think it's it's um relying on the metrics is something or migrating more to a matrix in certain places is something that we started more recently.

C

um I think we ideally, we would like to rely on metrics as much as possible. um It depends exactly what exactly do you want, because um it's a matter of granularity, also like um it's yeah in particular, it's it may be.

C

It may be hard to um so for scheduler you can you can you can edit the metric?

C

um Maybe that doesn't make sense? What I'm saying now I wanted to say that, like you can emit the metric when you are making a sketch like um scheduling decision, but it's probably what you want from scheduler and what, where, when even is happening, I I guess: I wanted to say that, like uh we, we are potentially ignoring the the part of like sending the requests to the API server and so on, but it's probably work around the boat anyway and it's probably not what we are interested in anyway. So.

D

So if scheduler emitted a metric, which is a difference of the pods creation, timestamp and the point where it succeeded to schedule I guess we could collect the p50 P90 values yeah. It.

A

D

Makes it more accurate than what we are doing today, which is I believe some? It's it's second level granularity, it's not even milliseconds or anything um so yeah, I, think really I. Guess we could move to that model as well like why Tech is saying it's just.

F

We have been following and regarding granularity, I think we are also trending data, so um the implementation is such that it comes on a per object basis, but at the high level we are looking at P95 of creation to running or um or an average right, so I I think um moving to that scheduler metric will will then help us align that okay keyword started running on 127 kubernetes and it is seeing more time being spent in uh scheduling phase. Let's go look at kubernetes, 127 uh P95 for scheduling and see.

F

If there is a a problem there or or something like that, so it helps correlate the trends in in both the projects.

C

D

Yes, that makes sense. It's also what a lot of a lot of what we are doing today with with the test. We are running correlating a bunch of metrics with with time.

A

uh Also, you mentioned that you, you are wondering whether the change comes from Cube fears or kubernetes itself. So I was wondering. Maybe it would make sense in your case to actually like run cube field with fixed version of of kubernetes, for example right and then you.

A

You are sure that all the changes come from Cube verd, but also there's also something that we use for testing various things where, for example, um with the aircon compiler and where we have like fixed kubernetes version, and we just test column compiler, and this can also like just reduce the noise that comes from both projects at the same time and help you debug further.

B

Yeah yeah we do, we do do that. We we test across I, think three or I think it's three releases, um so yeah I mean that is. That is a point yeah. We would want to see that stuff, but I mean it goes further. Even like you know like Elena is saying like we, we want to see deeper into you know exactly like you know. We can see that. Okay, maybe it's slower, you know. Maybe it's maybe it's because of the.

B

If you ever change and it's Cuba has changed, but um nowhere we're like we're curious to see like even deeper like what is it that is going wrong in the scheduling like you know, maybe we'll file a bug like I, like that's kind of where we want to go with it. It's like you know. We can see that, like you know, we we've got a lot, a lot of large clusters. We run the stuff we want to see exactly where it is we want to.

B

We want to drill a little deeper and and hopefully find some things.

A

Yeah so personally, I think for right now we are using metrics and logs, mostly unfortunately, and it's it's painful.

B

Yeah, so we I mean we I, guess like what um this has been a really good discussion by the way and I guess like the what I, um as a goal like for for us I mean this is something that we've been doing and driving within Hubert and, and it makes sense to us- and it's worked well for us and I mean we'd- be interested in helping to contribute to this. This has been something that we care a lot about, because it's it's um like keyword is like we're.

B

I guess the point we're making here pretty over and over again is like thank you. It's just a guest and and the cluster and right like we use pods, you know like we use PVCs and stuff, and it's got It's got all these components, but um you know we want to. We also want to make kubernetes and that we want to make it. You know improve its scalability and performance because we rely on heavily so um it's it's. It's definitely a shared goal on this, and so I I we'd love to help.

B

You know in this area if that makes sense as a direction, if you guys think sounds like it. It does then yeah.

D

So one thing I feel: maybe you can help, given your familiarity with the space on the the things that are happening on the Node side right today in the test. For us, uh let's say when we are creating pods. The granularity at which we operate is after the party schedule and the uh and cubelete starts running.

D

We, for example, don't know how much of that time is going into Parts like, for example, mounting a secret or mounting a config map into the into the Pod or, like uh things like fetching, an IP for the Pod IP assignment.

D

So if, if there is a way, you can actually take your knowledge here and kind of measure that maybe via metrics or maybe something else it, it would be interesting to see that added as a measurement, which is a concept in our scale in our load testing, tooling, to kind of measure some aspect of how the system is behaving, so you could add a measurement for that. You can play around with it.

D

We're happy to help you there Marcel voice record I've, been I've, been using that for a long time for these skill tests too.

B

Sure yeah, we can do that I. What would you suggest is so what we could do is we can write some things up and as what we think could be an approach, what would you suggest is the right way to purchase? Is this something we should like? We can come back to this meeting and share our thoughts on this. If you guys want or can write an issue or something or what would you.

D

Suggest yeah, yeah, yeah I think anything, that's convenient for you an issue or you can come back to this meeting, uh but also uh I, guess it's I'd say start by playing around with that tool and see if there's ways in which we can enrich some of the some of that information. With these tests and okay, of course, you can also kind of post any questions you have on on the six scale, slack Channel and maybe tag one of us.

B

Okay, yeah, we can do that and the tool is the the cluster loader is that is that the tool right.

D

B

D

um So it doesn't have node level measurements today and that's something you can add there.

A

I'm wondering about one more thing, because um we mentioned that there are multiple phases for, for example, post scheduling and I'm wondering. Maybe we should even have like some brief summary of metrics that we already have and we can use for debugging because I'm not sure, if I'm personally, aware of all of them like with PVCs, for example or um or other stuff. And maybe we could just even like start with metrics, to understand like what we have covered and what we don't already.

B

Yeah we can take that as part of our approach. We need, we need I, don't know the answer to that either. So yeah and I think we want to know that I think it's important, so we can do that as well. We can come up with some of what's the current um set of things that are available and uh we can talk about them.

D

Yep cool, thank you um and everyone else we joined I guess uh with over time should wrap it up.

B

Sure. Thank you very much. Thank.

D

You thank you, which is really interesting to see this, so thanks for bringing this up, of course,.

B

D

B

You guys online and in future meetings. Thank you very much. Thank you. Bye thank.

C

You thanks for the presentation, of course,.