KubeVirt SIG Performance and Scale, 20 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG - Performance and scale 2022-10-20

Description

Meeting Notes:
https://docs.google.com/document/d/1d_b2o05FfBG37VwlC2Z1ZArnT9-_AEJoQTe7iKaQZ6I/edit#heading=h.tybh

A

Okay, uh welcome to sixth scale everybody. It's October, 10th I shared a link to the document in the chat. uh Please feel free at yourself as.

B

A attendee please, okay,.

A

uh Let's get started with the first uh item: okay, so we're gonna. Do some review of the periodic job results because um we've got some changes. The way I did merge. We got a latest patch merged for deletion times. Let's just take a quick look at them.

A

um Let's see I think it's only going to affect this test or these two tests, because he's the only ones who actually tracked Alicia.

A

So, let's have a look see if the data shows us so 100- and this is the 600.

B

All right, there's the 100.

A

Cool looks pretty good away foreign.

B

And then, let's take a look at the 600.

B

I think this is 600 right, I'll click, it.

A

No, this is also under.

B

Character: okay: this is 200. Okay,.

B

A

It looks like we don't get them okay, but maybe this was run to.

B

Maybe this was run before.

A

And answers that's odd foreign.

A

S on this one, that's weird that was from this morning.

B

Yeah I wonder why.

B

We'll pick a different one.

A

All right here is also 200. see if we have it. No, we don't. um Okay,.

A

Okay, that's strange I, don't know why these don't get it, so these are different they're different jobs they have or I should say they have different scripts that trigger these things, but it's all using the perf tool which should have the same code. Oh you know what it must be using it. Must uh it I think that no, that can't be it should be using the new, the latest Qbert. That's the only thing I could think of why it wouldn't be why I wouldn't be getting this. Why won't be getting your changes so.

C

Which are which are these jobs? um Can you this.

A

Is the um can you see my screen? Yes, okay, this is the um this. Is the periodic uh question density test yeah. So this is the one that runs in the performance cluster, it's isolated in the performance cluster that runs between 200 and 600. VMS, and then delete some.

C

And which one was it that you saw the successful it was.

A

Nice, okay- that was this one so as you know that so that doesn't make sense. So this is this performance cluster. Also, this one also runs on the same Hardware. It's just a hundred.

A

So, but this one had it I, don't know why yeah there it is so this one's got it okay, but I. Don't know why this one doesn't.

C

Yeah I'll try to dig into it.

A

It's a 10 20.. Oh, this is set from 10 19.. No, that's still I think it's still merged, but.

B

A

This didn't march yesterday this was like a week ago or like maybe this less than a week ago, but it's just still yeah. It.

C

Was this week actually but yeah? The CI was having oh.

A

Here we go so so that one has it it's just I, don't know. Maybe this okay, maybe we just have to have to wait, have a way to wait long enough and it just hasn't made its way through yeah, okay, all right. So then I guess so I guess the data we do have. So this is a hundred and then what did I say this one was. This was um okay. This is 200 okay. So this is a 200 that hasn't all right.

A

So we've got I think we have the data we want and we don't have the 600. But that's okay, we'll we'll get it eventually. So this is our numbers. Okay, so worst case 13 seconds. Oh boy, okay, that's interesting, P95! So, okay, six seconds and then oh, that's! Okay! That's pretty consistent! That's pretty good! Oh! Actually! That's that's really good! Okay, so we have so. We've doubled the number of VMS and our P95 is almost almost identical.

A

Okay, that's pretty good uh p50 is pretty close, I mean that's that's 0.3 seconds, that's not very noticeable, so that's actually really good.

C

Okay, yeah I wondered that in second in do we have like a page deletion time or or something like that.

A

Do we have a what.

C

um Graceful deletion time.

A

Yeah there is a default: okay, I, don't remember how long it is, though it's a good question I, as yeah as to whether they're affected here I I, am so I. Guess what what so there is a there's, a I guess the way to think about this. There is a default graceful deletion time, but if the guest exits, um it basically just gives the guests time to exit. So if the guest exits really quickly, then it will make it seem like um so I guess yeah it could affect.

A

Our numbers is I, guess the point yeah, because if, if we're going to give the guest time a deletion, we're also factoring that that is a variable that will affect this. So if we do I think we have probably a better experiment what to take away the graceful they are maybe lower it and then we'll see some failed where we may see some fails, but it should provide us a more consistent measurement on the control plane.

D

A

Okay and make a note.

A

Yeah I think we just kind of lower it or something.

A

Okay, all right that looks pretty good I I. This is these are good results. This is so. This is actually totally the opposite of what we're seeing on create. So that's really good to see that it seems, like you know, we're not getting any crazy numbers. Let's just see what happens on the 600 or whenever it comes through, we'll see how this holds the p50 or P95, but that's really encouraging to see that this is holding.

A

Okay, cool all right, I think we're good with the periodic results. I think the rest everything's staying green, so it looks good that we're staying within our thresholds. Okay, all right! Let's go to the um let's go to our second point, so we have um Caleb and and Tempe here, hey guys, how's it going.

E

Good, thank you, hello, hello. um We've been uh pairing with cats on uh a few different things related to uh the using Cooper, and especially the cluster API provider and uh cat had mentioned. That here would be the place to discuss um some things around the cluster API provider, so that um we can get some discussion around some of the behaviors of it. uh Oh and I think that has just joined us.

E

um I actually dropped some links because.

B

I have a few okay.

B

I think the overall pattern that we're missing is the ability to specify the style of initialization. um There is proud in it and no cloud and it's hard coded in such a good provider. October there. It is we just hard closed.

B

I think it's that Cooper B1 chronic configuration.

B

A

I'm having a little trouble hearing, um I I, don't know if it's on my end or but I I'm, getting like um I'm getting like every other syllable.

C

Yeah same with me,.

B

Sorry I have to speak a bit louder. I tend to have a soft voice, even person.

B

I'm working in this uh this provider here and we're trying to um uh to drop some more link s.

B

All right we were chatting about this then and wow I can't see it. Okay,.

E

So is the cluster API provided there's a secret, that is the user data for the machine with no Cloud coming up, and so there's a kubernetes secret that gets uh mounted into the virtual machine. Somehow using um no Cloud. Here it's got a hard-coded name, you see online 147, and so what we're trying to do is make sure that that user data gets mounted through I. Think is what we're trying to do.

E

uh And we're unsure if it's coming through.

A

Okay, so let me let me try to uh try to repeat back just so I understand. Is it so you want so you've got you're using Cloud net you're using the you've got a credit disk and you have um you've got the secret here. That's hard-coded! You just want to make sure you're trying to make sure that the the cloud in it config drive that you have and the secret is getting to the guest is that is that right.

B

E

So we're looking to be able to specify which strategy to use for the the provider in the no cloud or what was the other one that we're using load cloud or config drive. So those are the two strategies that we found and we want to have it so we can specify which strategy to use because I think it it's automatic and whatever it's picking is not the one that works for us kind of thing.

B

Find 145 needs to be able to be blocked out, pretty not just the cooperly one clutter knit configured but like cloud in it. No cloud.

B

So we have an option of the two different trials.

B

That sounds about right.

A

A

You want to get um uh so you just you want to get some of the data you want to get the data to the guest, um but is it your is you're just trying to find a way to do it, whether it's going to be using config drive or using the node Cloud format like this, that you're just trying to get it to the guest? That's what you're looking for.

B

So guidance correct uh when we're so in cluster API, we're stuck is using just cloud in it: config drive and we're trying to Target Palos and Talos, no Cloud, that's dial when we're using ubertips, and so you need to be able to utilize no Cloud as an option and right now it's hard coded to um Cloud emits.

B

A

um Well, let's see I think, um maybe maybe it just needs to change the uh to support either option I mean I. Let me see where um I don't know at the top of my head, but I remember where this is in.

B

Cuba, it means.

A

You can do a little comparisonic focuses in.

B

A

Blade do you remember on the top of your head? I was really like you I'm looking at us yesterday, because you patched this so I. Don't remember: I thought it was a bird Handler.

C

A

C

Then I patched.

A

This recently to actually.

C

Yeah in the cloud in it has its own package.

A

B

B

No I just saw it.

C

So guys well, Ryan Ryan brings that up. um So what I'm understanding is that this cluster API provider is creating a virtual machine instance uh from from that function, and you want to specify a config option where no cloud is also configurable, whereas it is right now hard-coded to a spec value, which is cloud init, config, okay, awesome uh by any chance to create an issue on on cluster API provider.

C

Cubeboard repository um I see on that Charter that they meet Tuesdays at 8 am 8 PM, 8, AM, Pacific, so um creating an issue on that Repository might get get you some traction.

B

um Which meeting is that we were trying to track down a little bit.

B

B

Okay, thanks all right, we'll create an issue. Should we create that and uh the cluster API provided for Cooper.

C

Hey I would create it against this repository uh cluster API provider, keyboard.

B

Absolutely we will do that.

A

Yeah that sounds good, guys. Here's the reference that I was looking for. So here's like the types um like you can see how um the way that this is done in tree with Cloud net.

A

You can see how it looks at the volume types and and that's how it gets resolved so I think, let's see where was the um okay still the volume Source right here, so it basically I it seems like it could just be done very similar to what's done here and and then just looking at the volume Source or the volume type and then um either going into no cloud or going into the config Drive type um would be how you can make it so that supports both.

E

Yeah that looks quite good.

A

Here, I'll give you a link, so you got this as a reference, something something you can reference on the issue.

B

Sure no problem guys thanks for narrowing, that down for corporate spice report, that's going to be great.

A

Cool, so do you guys and I have another another question for you guys, so you guys. Are you guys working on cloud? uh The cluster API provider for keeper is that uh sounds like something you're.

B

Interested is this.

A

B

What the effort is is to try to create for all cncf projects, the ability to uh click create a cloud native development environment. Right now, we're we've been working with cluster API and directory targeting.

B

Equinox and others we're trying to find something that'll scale better for cost, so we're using again cluster API to Target two birds. So we can have a you know, plus, if there's a bunch of machines to bring up um coder, which inside of coder here using terraforming to create workspaces and instead of a workspace as being individual VM. They are cluster API, created covert clusters, uh but we're going to have the fully working Ingress and everything with the uh the environment of the Dead container in the Pod.

B

That's exposed to connect through the wall specific and web.

B

To try to go hey here's how the cncf can help you onboard contributors fast to make some contributions to your problem and so I by the way hit the hacker and say. Look where we're uh we're part of the cncs Strategic initiative, uh Senate, Consulting and so part of our job is try to find things that can help out in the next two to five years to make things better ecosystem, um so not not specifically working on Uber cluster API. But my input was very aligned with the health of the ecosystem.

A

Oh cool, okay! Well, that's awesome, yeah that that's a cool use case. That's awesome!.

B

A

B

To see what's going to come of it, those things can be really wonderful and cool people gather around the idea and um I think we're gonna see some pretty lovely things happen within the next month.

A

Cool yeah I'll, be there next week cool, that's exciting: okay, cool cool; okay, thanks for sharing guys, okay, all right, I think.

B

Directing us here, whether we'd be a bit lost about helping people direct it to the right place.

A

Yeah good yeah we're glad we can help and uh it's good. Okay, all right, um I'll lay you've got one more issue here, so this is.

B

Another question.

A

Yeah, okay and formers in the word controller, okay, fun, stuff, I, remember having this conversation for many times over I guess. Maybe this is.

C

The the pr97935 removed what um informers for this uh cluster type flavors? Well, sorry, crd, flavors, Umi preference instance type. They removed uh the informers in the controller, and then there is another PR tracking.

C

um The changes that bringing it back I just wanted to make sure uh we have like we follow this in six scales, because it's going to be like it'll, probably impact uh the performance so.

A

Okay, first machine flavor and purchasing preference that.

C

It this um this was only effect affecting the virtual machine and like virtue, machine flavor and virtual machine preference uh controllers, so I'm not sure like it will I'm not sure how much impact we will have in the uh the density test numbers, uh but just wanted to share as something that that I found, while taking into other issues.

A

Yeah, let's see um so this is the only two. So it's only these two apis that regret I I'm, just trying to what's the reason he's being removed. I, don't so both players have proven problematic.

C

A

I'm actually surprised they were added, I mean I I, don't know what he means, but I I haven't dug into this. So I wonder what I don't know what he means by informers, because cuber doesn't really have them. They have. They have basically their own implementation of of an Informer. It's not really I, don't know if that's what he means or if you mean sexual control, runtime no.

C

The flavors, so um flavors is an API that replaced VMI preset in um version 48 through 55 and while working on that removal and after 45 flavors took VMI preference uh type. So it's the same API that has been changing names.

C

So while this rename was happening um on the uh on this API, the the word, what a client has a package that gives um interface to the actual underlying uh client go Informer and that was removed uh and for a while, like for for four or five releases, Upstream main doesn't use um the Informer for reconciling, and now it is going to be added back with the pr eight five. Three seven.

A

Okay, I, wonder um we don't we don't do anything with flavor um I, wonder if so that would be interesting. If you know to your other point of saying, if changing it back will affect our current jobs, that would be interesting. I also wonder: um I. Also wonder the yeah see we don't we just don't use flavors like it would be interesting to see what would how we can measure flavors.

A

um Then we maybe have a measurement on this, but I I, don't know I I, don't know if it's something that we could do but like um or maybe we could ask.

C

Fyi like this is something going on, we might be, might not be interested, um but the only thing I want to make sure is because this is something performance related. If this is this gets back ported, we should have it all the way back when we were it was introduced, so um that that is the one thing and then, regarding the measurements, uh I'm, not sure if we have it in our plans, um yeah.

A

Yeah, well, that's what I mean it's like. If we, if we don't have the measurements I, we have no way to say like I I like at least I, don't have a way to say like what the performance difference is like that's I, guess, that's sort of the thing I'm struggling with, because um it's good. We should keep an eye on it just but but in terms of like, if we see our jobs change but um I wish, we had something with flavors so that we could actually measure and see to leak and see.

A

If you know if what the performance changed for this is I mean I, don't think it would be that hard um if they asked Lee to write this I, don't think it'd be too hard to write and point them to what he needs to do. It's really just um I mean I. Think it's just uh it's just a. We just need to add a few lines of code just to wrap around the existing creations of the vmis.

A

We just need to change it to um instead of doing PMI, so you just create VMS or maybe even already do that, but we just need to do with create a flavor and then create them off of flavor. So it might be like a tiny change that we could do to support this.

A

So let's yeah I, don't know. Let me I.

A

I don't know if it's on this radar- okay, let's start a threaded slack I mean that might be the best approach, at least because.

D

A

Is just a good opportunity for us to get you know for him and for us to at least get an idea of what the the scalability changes and performance change? That is that him introducing these these changes.

C

Okay, sure um and just another side like the flavors API is not as critical I would say, is the VMI right like because we can like theoretically, there would only be a certain number of flavors that we create and it's not going to be used as much as um the the VMI instance.

A

Yeah you're right, so what I? What so? Here's what I'm thinking is like? We could talk to we, and maybe this this is a um uh you know. Maybe this is something we could um like. Have us an additional periodic or something we could have like we we already right now we create you know we have that density test. We have like we create 100, 200, 300 or whatever we can have one of those in there be um based off flavors or something you know just something like because we have a few.

A

We have a bunch of data points and maybe we can. We can um have one of them be because I think there's value in it, because we're like this is an API that people are going to use and I and I. Think um I I, don't think we have visibility to it, the overhead at all, so there's some value in at least half of that, and you know the hope is like because to your point we are still launching game on us right so like we can. Actually this would be good to see.

A

Okay, hey like we can compare to our other jobs and say like okay, here's 350 lines, here's you know what it is with flavors and VMS. You know, and it shouldn't really be any different. It should be the same. So it's it. Actually, it's a good. It's a good way for us to get um it's actually a good comparison for us to see.

C

Yeah yeah and yeah I I mean we can. We can start that thread and even.

D

C

It on the VR, if this needs anything, we can help out with.

A

B

A

All right, so that was I can start a thread on that. Let's just see what he thinks and- um and uh we could put him in the direction to do that- maybe he could be a separate point spr or a part of the yeah okay cool thanks for bringing that up. I think that's that's good! To have a discussion: okay, cool folks, okay, um any other um any other topics. I think! That's all we have on the list any other last minute. uh Topics before we finish.

A

Nope, okay, all right! Everyone, thanks for your time and next week, um next week's gonna be kubecon, so I I, we're not gonna, have this meeting, so the next meeting will be in two weeks all right. Everyone. Thank you. Thanks.

B

B

Thank you, bye.