KubeVirt SIG Performance and Scale, 3 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG - Performance and scale 2022-02-03

Description

Meeting Notes: https://docs.google.com/document/d/1d_b2o05FfBG37VwlC2Z1ZArnT9-_AEJoQTe7iKaQZ6I/edit#heading=h.yg3v8z8nkdcg

A

All right welcome to sixth scale everybody, it's february 3rd, I'll link. The document notes in the chat. um Please add yourself an attendee, please um all right.

A

Let's get started with the agenda, so um a few items for today um want to review um the performance periodic job results because we're starting to see a lot more information. Now that.

A

Some more information that should be able to enable us to add thresholds, so that's really exciting. So let me see, let me open a few of these.

B

And we're going to do a little.

B

Quick look over these.

A

All right, let's start with the first one here so um okay, so this is, after the changes with the primer added the primer vm. So this looks really good we're so we're starting to get the and much more accurate, create pods count that looks really good.

A

102 also really good 108 yeah. Those all look really good. So um this is awesome and then like this too, like we're we're so we're getting this info now um all right. So that's that's really good. So another a lot of really important things about this is that I think this gives me a lot more confidence in these verbs.

A

So I mean we should talk about them like what we see. That's sort of strange here in this list like.

C

You see the delete.

A

What's that the delete? No so because marcelo, we run because now we're running the job in the um we're doing it before this week, we're running the job and then we're waiting, and then the delete gets run after the test completes. So it's way after the metrics have been scraped.

C

A

Could do we, we could do a delete, but we would need to do it as part of the test. Now yeah, okay,.

C

A

Yeah, I think, what's interesting in here like so what's what I think is really good it's this. This is really good. So we're like we're close to 100 on the patching vm vmis. That's awesome like fairly close to the creates.

A

That's pretty good, this one's. I don't know what this is. 650 645. This might just be picking up some other stuff. I don't know, I'm not sure what this is grabbing, but.

D

Yeah, is this really a get on a note.

A

Which you see here, yeah we're grabbing the the http request, counts during a period of time when we're doing the creates, like 100 vms, so yeah there's. No, there shouldn't be any other vms running yeah. I mean, as we can see here so yeah I mean this is get.

A

I don't know what endpoints are. No, I'm sorry we're looking at this. One get notes, count yeah, so this is get on.

D

A

D

This is this word handler, or is it controller.

A

I think we do it like this. We use by.

D

Controller handler and operator for for handlers it can make sense. Vendors uh do heartbeating on the node, so they get.

C

A

D

A

I see okay, so during this time, this is about roughly how many we saw. Okay.

C

Right, it's not it's not per second! It would be nice to see the rate, because you know if it's is, if it's doing too much get nodes per second yeah.

D

It will be you're, absolutely right, yeah, and I think it's not doing it that often, actually they get I'll just check it. While you continue.

A

Yeah, I know that's, um I I don't think I think it's okay, like I, I I'm just trying to pick out some high numbers in here like um update endpoints, I don't what about this one? um What up? What endpoint would we be updating here.

D

uh That's I think, gonna be that that's no lead election would be config map endpoints, interesting.

A

And there's this one too, so here we patch, so we do a lot of pat. So we do pretty much a one-to-one patch to vmis, that's cool. We do a lot of updates, so we do roughly eight.

C

To one yeah, this is interesting. I also seen my performance test like you know, eight updates. Well, I just I didn't see this. You know there's these numbers, but eight updates per vmi. It seems to be too much.

D

I'm not sure we have a lot we have to. We have until it reaches running. You mean right.

C

D

Yeah, I think this is this can be pretty accurate. We have a lot of sub stages, which we are reflecting in the vmi status like when the pod, when the pod gets created.

C

D

The schedule when it is running right, I.

C

Was confusing with patch yeah updates makes sense, yeah, sorry, yeah.

A

Yeah, so this one.

C

No yeah updates makes a lot of sense, yeah.

A

Yeah, so that would be like um yeah, so like every time we changed its phase. Right would be so we should expect at least uh what it would be like. Four, four five. So four with like pending scheduling.

C

Oh, this is one for phase, but it also has conditions yeah.

D

C

D

It's more than this.

C

D

It sounds reasonable. I wouldn't do it yeah.

C

It's really the.

D

Best, if you would have said 12, I would not have been surprised. Okay,.

E

It's tough to correlate it directly to all the different fields, because a lot of fields are updated at the same time, so you might hit running and have some conditions at the same time, for example, and then you might have some conditions to get set independently of any of the phases and it's really tough. Okay,.

A

Yeah, I know I was just trying to quantify and see if, like anything stuck out so okay, I mean, I guess so yeah. It sounds reasonable to people, but is anything in this list? Sound unreasonable.

D

I guess any points.

A

Updates con well the end.

D

A

D

I'm not sure that's for uh leader election is it. I thought it's based on conflict maps, but when it ends, I think we're using endpoints, but okay, great then yeah that would have been missing sounds good.

A

Okay, so about a two to one for those.

A

E

A

Okay and then uh just to get end points about roughly yeah, this one's not.

D

A

D

It makes absolute sense, and it's independent of of the this is just a coincidence that it's that it's kind of half of the notes count. That's just a coincidence. Yeah, okay,.

A

Yeah I mean, I think it looks pretty pretty good. I think, all those all those nerves, yeah.

D

Okay, doesn't look like any real outlier, really cool.

E

Yeah, how about um thresholds has.

A

That been discussed yet all right, I haven't oh, so that's where I was going, so I was gonna. So this part. So we can look at a few of these. Let's see here so I just pulled up three randoms, so we've got 37 on the 50 35 on a 50 30 out of 50., seven, thirty, five, thirty, okay, that's roughly like a ten percent range or so variation.

B

Let's see what about this one 58 56 48 yeah,.

E

P95 is something we could probably use. I would not use p99, that's like one percent worst case, but p95. It seems like if we take so the highest one. We got was 58 if we added like 50 to that. So that's that's 60 seconds, maybe um maybe 90 seconds as a threshold. If we ever give about 90 seconds, we did something terribly wrong.

A

Yeah yeah we're kind of hovering in that right around the under under a minute here for 100., yeah, okay and then right around under 40 for the p50.

E

It seems like we.

B

E

Be able to get that consistently lower than a minute.

A

E

One of the things about ephemeral, virtual machines here is that we should have already synced the image right, so the um I guess it depends on what image we're using I'm trying to think of. uh If um there's any case where we are pulling the image to nodes. First, uh the actual boot image, because we're doing that then.

C

I can see why, like.

E

So the first one on every node would be the outlier there. It would take an extra amount of.

A

B

E

Pull the image.

A

C

A

No, no, this is from the um this is from the periodic tab.

D

But isn't it running on martial's cluster now or oh, I don't know, is it? Is this running.

C

No, this is this is running in the in the workload no cluster. Oh.

D

Just keepers ti because yeah, okay, okay yeah.

C

D

The images should be pre.

E

Synced as long as we're using one of the pre-synced ones, let's double check just to make sure we're not doing anything. I can look real quick.

C

Yeah, I I think it's using the image that it's upload uh when you do cluster up.

D

Okay, so probably just a serious one right.

C

Yes, it's not using from from quayle. It's not not downloading.

D

If it's using that one, then it's present to all nodes, yeah.

A

How many, uh how many nodes is this just one node two nodes, the.

B

A

C

Yeah, so I have the other job running the performance cluster, but um it's actually uh not running right. Now um I actually create a pr because the image was using. um You know what was missing goal. Just something very simple, you know was just missing go um I have a pr fixing that, but it wasn't merged yet. So, when we have this merge, we can see the the the job right in the performance closer.

A

Okay, so with the performance cluster, this is like it's totally. It's totally different than this, where this is running right, it's its own, dedicated one.

C

A

Actually, okay, cool.

D

Okay, yeah and so on. It sounds great and in addition we can probably add some. What would also work on cubicle ti. Good is probably also some expectations regarding to the counts, the rest calls and so on, but on the other hand, it changes pretty often when you do pull requests.

A

D

A

To see that, maybe it's exactly.

D

What we want to see.

A

It is, I agree with you um like, for I completely agree because this is like. I think this is so I would say this is going to help us a lot with performance, and then this is going to impact I'd, say scale, because when we, when the number of verbs that we do right, is going to affect kubernetes, so we should be conscious of this. I mean I think um I mean I don't know. What should we pick out of here I mean I would expect like this one looks pretty good like patching virtual machines.

A

This should always be one to one. I think looks like it's.

E

Pretty close because it's patching the um what's it patching the ready status, dirty conditioner thing right from the from the pod right is that what it was it's sinking something from not guaranteed.

A

E

D

That's not, but it seems to be in this case pretty much.

E

D

A

So maybe not like, so maybe what should we? So? What would be like a case where it's not one to one I mean that's just like well so, if like, for instance like this, is an estimate, if it's not one to one like, let's say it was 200. That means that we had to. That means they all failed. Whatever the condition was.

D

A

Right, I think.

D

A

Seems like a problem right.

D

Yeah transaction matrix: we could just double it and say yeah for this one here and say: if we, if it doubles, suddenly, then something which maybe of interesting interest changed.

E

Yeah we're specifically looking for loops here, so uh something where, if a, if two controllers collide, then they're going to get in this competition, both trying to update an object at the same time, and maybe it eventually resolves itself. But that would result in several like like double or maybe even triple the number of uh patches and updates.

E

So I think patch and update to virtual machines and the ones that we are most interested in from a scale perspective, because that that multiplies with the number of virtual machines that we have our virtual machine instances that we have.

D

Okay, I think we can even be pretty close here and say for the patch it's two to one and for the update. We could even say it should it must be below nine to one or ten to one, because for as long as we can just change this number, with the pull request, when we see that it's legitimate that it goes up right.

A

Yeah we can just change it.

C

I think maybe also the number of get and least you know- maybe not threshold, but something to analyze here, because it will change. You know. Number of getting lists will change if the vm is running with pvcs or multiple needs things like that, um but you know we don't want to make. For example, get we have getting point get nodes only that.

E

That's for our uh heartbeat, so the get nodes is for the um bert handler heartbeat wow, 600 eddie.

D

The timeframe would be interesting on this one.

A

This is uh and interpolated over five minutes. That's insane! So a hundred five hundred five minutes. Yes, really weird, the last one.

C

Can you see the other uh runs.

D

C

Indicate that we.

D

For instance, have a get somewhere get node somewhere in our vm, create flow that then could be to speak. For instance, it.

A

Looks like we're consistently at like. Let's see this kind of a weird, it's not really correlated right. It's like six to one, almost seven to one, it's kind of strange! Actually, so I guess like it's so we're saying it's like maybe correlated with time, and maybe we.

D

A

From the word handler,.

D

Or something I'm just double checking our default heartbeat interval. So that's just that we know.

E

Must be doing something.

C

Well, it's five minutes, it's 100 percent of that. That's.

D

C

E

Doesn't look right? No, no.

C

I think this is.

E

Accurate uh because everything else is accurate,.

C

Yeah definitely something something wrong. Maybe it's failing. You know it's doesn't receiving the it's failing and then it's trying to get like like crazy. You know the the result, but if, if it doesn't get.

D

One over right now, it's one minute so from the two word handlers, we should see only five times to ten requests due to heartbeat well,.

C

Yeah dude, it's weird yeah.

A

C

So uh I think again things that I I think I mentioned before. We need to give here also the code you know to understand. You know which, which requests were 200. You know http 200 uh code for response and which ones has the 400 and 500 ones, because we could see here in the get notes if those ones are 400 and 500 uh answers uh called.

C

We don't know which.

A

C

And which yeah.

A

I mean yeah, I mean I guess marcelo is like I. Maybe we can. I I don't disagree with you. I guess this tells me you.

C

Have permeators there isn't it? Can you just check this.

A

Yeah I have to.

C

A

It's not running yeah, it's not it's. Also, not it's not yeah. I would have to be like my local cluster yeah, I mean I could try it if you want to, but um I mean all I gotta do what 10 vms.

C

But I think we we can extend it. You know for requests that failed and request that went through you know in here in the report.

A

Yeah, like I see here so we we patched the nodes, roughly uh kind of at the rate that you just said, roman, I mean it's fairly close, so it's 12.

D

A

But something is really getting.

D

The note too often this is not right. The patch seems.

E

D

But yeah yeah, the patch, is exactly what I.

E

Would expect yeah, I think it's our node labeler. uh We have a controller that.

D

It's on everything.

E

Yeah and I see that the.

D

Device plugins also have nested functions where they do get so there may be something to it.

A

B

So this one we will need to investigate.

B

A

Okay, I so back to what you were saying marcelo about the list. I agree with you about these. These are good safeguards like um because.

C

Yeah, we wouldn't.

A

We would be right- this is like this is good like we like the these can be expensive. That sounds that's really good. We only do one I mean this seems. I wonder I wonder if this would be nice to find a way to correlate this, like. I wonder if this is like.

A

I don't know what this would be list. um Maybe it's the only one I could think of here. It's like we ran the job once like what we created a role or something or like we checked something on a service account or what what would these be?

A

We checked the name space once I don't see the tie into this.

A

Let's see some of the others.

A

Oh, I don't even see them here actually.

C

A

Use one of them.

C

You know things that when we have like many questions like that, um we don't know also, who is actually you know actually uh requesting that you know. Is it very.

E

C

With controller you know, I think, once we have this, we need to have like another test and get you know this per component and also, as I mentioned, per it, failed or not. You know or went through the request, and then we can just please.

A

C

A

It in the performance.

C

Cluster anyway, because we have the dashboard, so I will try to make this pr working next week. So then we can. We can like book that.

A

Okay, I don't even see- um I don't even see these in the other ones. I only see that one only the list service monitors, which I don't know what I don't know what that is, and then there's no lists here, um I'm I'm guessing it's there. It's just I'm missing. I mean, like that's, probably a small window to like to capture this, because I bet it's literally happening once probably right at the start of the test, and we have to get the the data at the right point in time to see it with the interpolation.

A

It's possibly just missed it, but it I think at least it's it's pretty insignificant, though you would see. I guess maybe what this isn't, and maybe it's not something we care about. If, if we see like more of this- and we see like more of these show up, then I may be a little more concerned but yeah. I don't think we need to worry about that.

C

All right, I'm surprising that it doesn't lease anything. So it's yeah.

A

We mean you're surprised, like you, don't see, I don't understand like.

C

Because I was expecting you know some least, you know, um for example, somewhere at least in vmi, and then this wouldn't mean not good. You know you know to find those kind of things. You know well.

A

C

A

Mean it's good that there aren't any you know like this. Is I wonder what's doing this, but I it's it's pretty insignificant. Whatever is doing it.

C

Yeah, it's pretty good, actually yeah.

A

Yeah, it's really good.

A

A

This looks like from the word operator something yeah. Okay, I think that's pretty good and then uh so I think these seem pretty good. So we have. We can do thresholds around these these right here and I'll. Get you all these and then investigate this one, which is a little a little unclear. Okay, cool all right! That's good good! We're getting some data from that! Okay! Let's go to the second bullet point um from and I'm glad you're here today.

A

I heard you mention this in the community call um because I think it's gonna actually be important for our jobs and using the serial tag, because we don't want any of them running in parallel or we're going to get money.

D

Results, you definitely have tests which just not to describe or something just put a serial there. You don't have to put it to each it or so just on the global described set of serial.

D

Okay and everything is good, but.

C

D

Think you're, not are you executing the tests through heck funk test um we're? No, it's it's hack, perf, test, okay, yeah, then it doesn't then heck funk test is only real. Only attack, funk tests make sense out of the serial tag. So if you're just running it out of that you're not affected, but it may still be good just to indicate that they're supposed to be run in serial efficiency, but you should not be affected.

D

A

All right: cool, okay! Well, let's go to the third bullet point we'll go to pr, so uh this is one. This is a follow up to the um the pr from last time where I hardcoded the five minute range vector. So this is um what I wanted to do here is. uh Let me um go to the pr from last time, because I have a this: is the one.

A

Here we go, I think I had at the bottom, so the follow-up for this I wanted to do is that, like um I wanted to establish a relationship between the range vector and the previous grape interval, because they're they're absolutely related, like I did a bunch of testing on this and it's this is how the interpolation is, is determined and how the values get set.

A

um The other one is that, like we need to monitor the range vector and make sure that it's that it's the right length based on the test duration, um you know if it's too short, um we should extend it. We're gonna miss data. If it's too long, um we need to be cautious of that as well.

A

So there's there's a few things that that I wanted to capture and that's what I went through with this, and so the relationship that I established with the range vector is a 10x relationship between the range vector and the prometheus scrape interval. I set premium scraper integral to be 30 seconds as a as a global variable there. um I couldn't, I thought about trying to look up the prometheus scrape interval, but it's actually.

A

I didn't think that was a good idea like, but I'll give the user the ability to set this if they want to, but it's hardcoded to 30 seconds. That's what our tests are so 10x to that comes out to five minutes, which is what we're using uh taxing reasonable because of you know what we were seeing with the results from the five minute range vector time gave us reasonable, interp interpolation metrics for increase.

A

So um that's why I went with that and then, um like I said, the other one was that the setting the range vector to be close um to the duration, uh and so basically what I want. uh I wish I had my some actual data here, but basically what I want to do um there's a graph here.

A

What I want to do is like I want to get. Let's see if I find a good one here, I want to get the that. I want us to run the audit kind of like right near the end of this test. Actually you can see right here so that this value right here right, it's 22 and you can actually see it right on the end. Here.

A

It's um it's actually coming down kind of the end of this test, because the the primer tests, whatever is, is running in front. So what I did was is I actually increased this buffer, so I added a buffer between tests now of two prometheus grape intervals. So let's come down to a minute by default, to give us some time between tests, and I want us to like scrape right- one prometheus interval back where it's on the offset so kind of somewhere in this area.

A

So we land somewhere in that in that one minute buffer and we don't come too close to the end of the test because there are like. If, if we oh there, we go that's better. If we, if we come too close to the end of the test, I mean the results. Can uh let me kind of conceive like here?

A

We, they kind of move a little bit, so I kind of bring it in to where, like the results are fairly stable kind of like you can see it here on this, like when they're fairly flat at these points, where, wherever one to where I want to grab them. So kind of right, near probably like a five minute interpolation, it's like be like four and a half minutes is where I want to grab it for the most accurate results.

A

So really, the only changes is that it's like it's just to make it. So that's when this like, when we this is it's going to change it based on the prometheus scrape interval and the length of the test? That's really the gist of it.

B

A

Okay, um other pr's that are open. I I saw david you had this one. um This is your fix to the vm pools.

A

But I don't know if you had any comment on that, but I I did do a review and then it looks okay to me. I think I just left one comment: yeah. It's.

E

Tedious, it's the best way to describe it. I I can talk about it or all. I did was change the hash function that was originally trying to determine if the pool's template changes versus the vm and vmi that we've actually deployed.

E

So I'm comparing the hash of the pools template for a virtual machine compared to the hash that we've stored in the vm, the vmi, and that was wrong because my hash algorithm, I don't think I have one that would ever work accurately if we ever add or modify a field on the vm or vmi spec.

E

It would essentially just after an updated key vert caused all this to be wrong. So I changed it to use this controller revision and I can compare the controller revisions against each other. I'm using the exact same api version every time and it works. uh It's just really tedious. So that's all I changed cool.

A

Okay, all right and the third one. This is the the slos document that I've talked about. Previously, I don't I don't know what I haven't seen many comments on this, except from marcelo I mean do. People like are people okay, like mainly what I wanted to do with this, like I said, is that to describe what we want to do with our testing and kind of get to where we want to go with like um when kubernetes has this slos document or they they try to.

A

You know they have tested or confirmed uh things like that they slows they have for the platform.

A

We could go that way as well, and I think kind of advertising through our testing is the way we could do it. um So I mean that's kind of what I'm doing here with this uh just kind of laying the groundwork for that, and then we need to implement the testing for it.

D

I think it's a good idea. I just didn't get to look into it. I mean in principle expressing what we think from the test perspective that we're capable of doing is great when we describe it.

A

Okay, all right have a look at that: one: okay and then marcel you had this one.

C

Yeah it's merged, so uber now has the excitation extension to uh create via convert object, pm, vmis and replica set and wait for the you know the ready state um and and collect some detail: uh metrics latency, the metrics so from the vmis it's in the end. It's just create a map and get the timestamps. For you know all the states from the when the vm and vmi is changing the states and also the pod. That is changing so it it gets like. uh You know the time that the vm, for example, vm, creates.

C

Then the vmi is created because I want if the vm is creating the vmi okay um and then, when the pods created the pods in each eye. The container is in each eye, the pods are running and the vm vmware is running. The vm is ready. So it's it gets all of this. So yeah.

D

Very nice, this.

C

One's a fair example.

D

C

Yes, I have um actually, I run several tests doing that in the, um but it was um in other nodes. I I can send a picture about that later, because also it it's some test that I did um okay, I did some tests that in the in the ci also so I will.

C

I will share that later.

A

C

A

The is this the this is the burst test right, that's what that's what's important.

C

It's the first yeah, it's a burst test: okay,.

A

A

All right, okay I'll, have to try it at some point um on our internal questions.

C

A

That in the convert.

C

A

Oh cool, okay,.

C

Okay, how many what what do you mean, how many views.

A

How many vms have you have you tested with this.

C

A

18 000 wow.

C

It was a big test, but I don't know.

A

The results from that.

C

Yeah, I don't know if I can share these results, because it wasn't for appreciation.

A

Okay, cool, okay, all right, I think that's all we have so for um did we did anyone get the answer to this? Are we using precinct images for this uh for this job? I think someone said they.

D

Would say it was sent to us.

A

D

From there from there where we saw their results right now, yes, but um I'm not sure if it gets presents, could but could be.

A

Okay, um so for follow up on.

C

No, it's it's not I'm not using yeah, I'm in the in the performance cluster, I'm not using this uh image because it's you know it's a kubernetes that it's already running there and I'm not pushing the image so the first, the first image comes from the quay and yeah. You see like some downloading time so in the first vm that has been created.

A

C

We we can maybe improve that but yeah, let's put something to think about.

A

All right all right so for follow up on these, um so I can. I can take the action here um to put together thresholds, I'm going to do it for all of these. Basically, what we have here, oh this one's duplicate, so I'll do based on these ratios, we can start uh I'll start, adding like a a fail pass fail based on what we see on these thresholds.

A

Okay and then um I'll create an issue for this one um I'll, just open it and attach it here and then we can we'll see. We can do follow-ups on this one and see what people find about about this.

A

Okay, all right, everybody, I think. That's all! We have unless there's any more topics going once twice: okay, all right! Everyone thanks have a good day.

C

Thank you, bye.