KubeVirt SIG Performance and Scale, 12 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG - Performance and scale 2023-01-12

Description

Meeting Notes:
https://docs.google.com/document/d/1d_b2o05FfBG37VwlC2Z1ZArnT9-_AEJoQTe7iKaQZ6I/edit#heading=h.tybh

A

Okay, uh this is sixth scale. It's January, 12th, so um lubo I think the we can talk. I probably will just talk about um this PR. Since you brought up that there was some performance changes and I did a quick look at uh the the um end-to-end running times. I did notice a difference.

A

Do you have any thoughts on this uh kind of I mean you can see like some of the changes that came up were like the the number of requests for patch and an update or did change I, don't know if that's what what's the cause here, but that is different and then the um the creation times for the p52 and 95, or also noticeably different uh I, don't know. Do you have any thoughts? Is that what you saw when you did this test locally or what do.

B

You think no it's no, so let me explain so. Basically what the change is doing is uh replacing the Levert as uh some kind of metadata store for for internal memory store right and therefore we can have a quicker update of the metadata, and there were some worries that this might cause a little bit more events to be admitted to weird Handler and maybe degrade the performance.

B

But what I saw in local testing I was just looking into how many more events we get from the launcher to the Android and how many more sync Loops that will cause on the Handler side right and I just test it I think simple: create of simple grow, VM and.

A

B

Of the VM and I saw actually decrease of the of the sync Loops required for the VM to go to running State and to migrate to the another node. So I I was thinking. This might have some. You know savings on the time uh spend in the looping in the sink, and it might improve actually something.

B

But then you actually point it out that we, we are nine seconds slower on 95, right and even on on the other person deals so I'm, not quite sure, but then I was looking into other jobs or history of the job. Performance, jobs and I actually saw similar numbers uh from, for example, day before the the pr got merged. So I'm, not sure where your data points are coming from.

A

Yeah I looked at the I. Looked at this job, I looked at the the results from this job and then I went back about um I, looked at two different results from a week back and and that's where I pulled the um that circled this data from, and this is roughly around what we've seen it's like close to right right around 30., so um I mean it could be that maybe this is something to do with another job. I mean I can look at a few others.

A

We can go a little farther back and let's do a comparison everywhere. Maybe with the two I pulled a little bit of an outlier. Let me see: um let's do a quick look at this one.

A

So here's the third here's 32 yeah, okay! So here's close to what you were seeing on yours right, 32, 39,.

B

A posturing screen or.

A

A

There we go, you see that so this is about um I, think it was about a week. I pulled it from here a few weeks back so toward the end of the year. So this is um 32 39 39.

A

That's what it comes.

B

A

So that was and then what we say on your job was 34 39 47., so I will ignore the P99 for a second we'll say: 34 39, so it's kind of within the 32 39. still with him, so yeah I mean still within the a few seconds of it. So that's not so bad. Okay, okay,.

B

Yeah, maybe it would be interesting to see where we got the delay of nine seconds.

A

Yeah well, I guess what we could do is like, since this has merged. What we could do is um I can continue to monitor this and see if like well roughly, where this goes like and see if it kind of we stay with, because it could be that what happens is like we start to get it. We end up with a few of these, maybe every once in a while, or we even hit some jobs that are kind of stay within the 30s.

A

With your spine, like that's, that's acceptable, so I think that's that was kind of the point it's like to me like it just seemed like um the number of requests changed, but I don't know. Maybe there isn't much fun effect here.

B

I mean definitely it's interesting to see that we got only one patch uh I mean the in the previous results, which is which seems to be a little bit long.

A

Yeah- and you know it's sort of like I'm looking at um like here's another one that I took, here's 100 and here's 59 and maybe yeah I mean so I I thought that was Associated. Looking at a few of these I thought this was associated with your jobs, but maybe it's not oops wrong window.

A

So what, when you went through your change, um you saw fewer sync loops. um What what um what? What what or the number of which requests would you expect west of.

A

Like uh I, don't know what you expect to see that so it.

B

Could be that we have less patches and updates.

A

B

Because yeah, if you have a fewer sync loops and before we had on all of the Sim clubs, we had some updates, and now we have only fewer uh runs. Then maybe we can have a less patches and updates.

A

I think so I'm kind of thinking a lot here. What I think we're going to do is because kind of, like looking at some of the jobs and basically doing this by hand like looking through I, think I, think there's some variation and I think we know I think we're gonna have to do.

A

Is we're gonna have to take some of these numbers I'm going to have to post averages or something just something that it's a little bit more smoothed out so that we can examine it over time because I guess like this is a little. It's I think it's a little too raw, because if we're saying like a hundred patches and then I'm going to another job- and we're saying like I saw one here, which is a little strange.

A

Yeah I mean I, think I. Think probably what I have to do is we're gonna have to change our data a little bit kind of what I'm thinking is. Maybe we can maybe we can take like averages over time and post them somewhere, because otherwise I think we're gonna have a hard time, basically interpreting your results.

B

Yeah some kind of trekking tool that would.

A

B

Us to see how, for example, the PA 50p 95 LOLs.

A

In a time, do you know, do you know if, like there's something we can do in prowl like that, would we could we could parse this data and then take like an average of it and then post it to I? Don't know a web page like? Maybe we can post some kid? Maybe we can post it in I'm.

B

A

Sure, maybe there's a place, do you have any ideas, yeah.

B

I have an idea I'm not sure how we implemented it. So I would need to record my memory.

B

But let me pause the link so for the CIA Health we have some metrics, for example, what's the merge Q length or was the time to merge and so on and we we create these graphs and we could have something similar for the performance. Jokes you'll see my link, I posted into the meeting chat.

B

Yeah scroll down, you have these kind of graphs, which would be good.

A

B

Tracking the pin 50p 95.

A

Yeah that this this is exactly what okay. So how does this work exactly? This is um this is uh generated from prow.

B

I think we expose some metrics and then we periodically collect them and do a graphs for them. But I will need to refresh my memory.

A

Okay, this is helpful. I can look into this uh as well and see maybe I think this is Daniel. The maintainer and and Federico, and okay and I can ask questions around and um on slack, okay, that's that would be really good.

A

I think that would help us because yeah I'm kind of like looking at some of the comparisons like we have some historical data in the document like in our in our meeting notes as to what like we've seen over time, but we don't really have any smooth averages, so yeah I think it might be hard to say exactly what the you know: precise changes.

B

Okay, yeah; okay, if you have some opportunity to to play with it like on some cluster, maybe um where you have some better historic data that would be nice to to see what happened there. Yeah yeah.

A

I think yeah. We should be able to look back and once they have yeah once once they have a chance to play around with this. Let me we'll do a I can let you know what we end up. Seeing in the in the graph here sounds good cool, okay, yeah, thanks, Lugo, I, think I think that's probably all we have I I, don't we'll look at the performance periodic results on the time with that. I think this is a good action for now.

B

I will also try to look at a little bit.

A

Cool cool all right. Well, thanks for thanks for joining lubo uh I'll, let you know what we find and uh we can, when I have something we can. We can talk again in the in the future uh community.

A

Thank you very much cool all right. Thank you. Thanks, lubo, all right glad to hear that.

B

Bye-Bye bye-bye.