Kubernetes SIG Node, 6 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Node 20220906

Description

SIG Node weekly meeting. Agenda and notes: https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#heading=h.adoto8roitwq

GMT20220906-170341_Recording_2560x1440

A

A

Meeting on September 6 2022, uh first off, we have driven with the Retro for 125.. uh Even are you able to do.

B

uh Sorry, yeah I cannot pretend that uh uh doc. If someone can make me the host.

A

I give you co-host.

B

B

All right just a.

C

One second.

B

Here we go, uh can you all see it? Yeah, okay, cool, uh so yeah I created a dog following the uh previous ratio sessions, so here are well. First of all, this is a link to our cab tracking dock, uh and uh here is a table for all of the Caps. uh We have worked on in the 125 release cycle and they are and their status. So in total we have 16 caps that we have worked now in the past cycle and uh among them uh we have completed 10 of them.

B

Although one of them got reverted, uh this um storage quotas and uh six of the cat, we had to uh remove them from uh the Milestone, which means we didn't complete them in the 125 cycle uh and uh uh well in place uh pod vertical scaling we merged to CRI changes. So we have made some progress uh on this cap uh so yeah. This is um well on what we have completed in the 125 cycle, and here are some um uh history data in the past Cycles.

B

So we can well see here in 125 we merged, like a majority of the Caps, that we tracked so yeah. We were making good progress there and yeah. So, let's uh we can talk about the uh in the past, recycle uh that saying sad went well and uh the thing said uh could have gone better, so we can um start the conversation here.

B

uh One thing I want to cover here is that uh for the changes that require a container runtime change, uh we merged the CRI changes first and that uh on blocks the container runtime implementation. We did this for the in place called vertical scaling. We did this for uh evented plaque, so yeah I think this is a very good practice. uh We probably should follow this. uh We probably should do the same for future. Changes that require container runtime change.

B

D

Thank you, yeah Ruben I, agree with that. That makes it a lot easier to get the full end-to-end support. Otherwise it ends up in a chicken and egg problem.

B

Yeah so I guess we well kind of um agree that we should do this for all future similar changes right. We want to merge the CRI change first.

B

Maybe we can make that like happen before the alpha uh change before the main Alpha feature is merged, which should aim to merge with your. It hinges. First.

D

Do you think it's a good practice also to have a separate cap for that, which is what I had initially for uh this particular one? There were two caps, one tracking just a CRI change uh that way. Any issues that are that need to be discussed. uh You can have a focus point for design discussions on that yeah or is it Overkill.

B

um I, don't know if it's just a too much work for the cap owner or the future owner. That.

A

B

Have to create two different caps, one for a CRI only uh well personally, I, don't think we really need that, but uh it's just a code. Wise uh it'll be ideal. If we can separate the CRI changes out from the main Feature work, yeah.

A

I yeah I, agree, I, think it's it's good to have a single place where we can look at all the relevant details, including what the CRA changes look like and like having multiple kits will be problematic. You know to get them reviewed and synced.

D

Yeah, okay, makes sense.

D

B

Anything else we want to mention things that went well.

E

um I can add something I think like overall from the test stability stuff um it we there's a lot of improvements compared to in the past, tests were pretty stable um up to the release and there was no kind of last minute fire or anything like that or regression that we needed to go really dig into so I think good job on all the testing efforts being pretty stable.

B

B

um Anything else well, I, can add another one. We have completed a majority of our caps compared to the previous psychos.

B

Okay, um any other uh things we want to talk about. Maybe uh things that uh could have, we could have done better, uh something that can be improved. uh We have talked about a CRI change. I think that uh well falls into this category a little bit uh I'm sure we can.

A

Like we mentioned.

B

uh The CRI changes could go first, uh yeah.

B

A

Log stream, so even though the cap itself was straightforward, the implementation is quite involved and so I think for cats like those it's it's better to like seek reviews earlier in the cycle, foreign.

A

Yeah yeah, okay, yeah.

B

A

Least, the implementation was uh way bigger than the than what it look from the kit.

B

B

Nice on anything else, we want to mention here, hello,.

D

I guess reviewer bandwidth, but that I believe is a problem for all caps. So it's not specific to in place.

B

Yeah I was capturing.

A

Yeah you should. We should capture it. Yeah.

A

D

Tests coming in late was I would have probably focused on that a bit earlier, uh if possible in the cycle, because I went from in my within one week, I went from being, you know a positive. This should go into. You know no, let's hold, because I found a few code changes that needed to be need to be made to the scheduler, even though the scheduling Sig was very on, uh they would go on that the they felt the changes were low risk, but something this big of a feature. We want to be cautious.

B

Just like uh um detected some uh change close to the.

C

B

Yeah all right to the code phrase yeah.

B

B

Anything else, any other um caps.

B

Or in general anything we want to um improve.

B

um Okay, if not, then uh well, we can um uh wrap up with our retro session. Then I will copy this. um What we have talked about to the main doc.

A

Thanks Reuben uh for leading the discussion here, and uh maybe we can uh start 126 planning next week and head back to the agenda yep all right. uh Next up on the agenda, we have Daniel ye on CRA stats performance, update.

A

Daniel I can give you co-hosts just a sec. I know you are telling to present all right go for it.

C

Yes, I think um yeah. The idea of just adding additional stats, the CRI um or efforts to eventually deprecate sea advisor have everything including Primitives exporting and such uh supported um and I. I know you're wondering like what's what the overhead was going to be with these, you know, fields uh how this container stats or lists pod uh stats was eventually changed.

C

So I'm, just kind of here to give a quick overview of uh you talked about a couple weeks earlier and responsibly sure it doesn't look like the new Syria fields are going to be too much additional overhead, um so I've uh the Stockman here that just kind of illustrates my findings.

C

um So my proposed change, I'm added added fields that are presidency advisor, is not currently in CRI includes, like Network stats process thoughts, um yeah some CPU stats, a wide variety of like Linux secrets. That's um so I think some of the main grpc calls we were concerned with were list container stats here, endless pod, sandbox snots, because uh all these digital new stats most directly affects uh these two RPC calls, um and so I was part of my change.

C

I, just I just added new CRI fields and their stats to V1 and I had I have the containerdy side here as well um and I'm just setting these to some some value. That's um some dummy value essentially um and I. I ran I. Think four test runs, total um I ran one: uh oh I'm, sorry I also use like P Pros profile. That's a uh go. Lane profiler gives like a nice overview of how much CPU time you're spending and which function. So that's super helpful, um but in total I ran I.

C

Think four runs one with just the main activity. Branch um one was my custom version 1080 um and then both those but with feature Gates enabled down below and on these I deployed 100 replicas of nginx and I used kind to uh run my custom container, D versions. So here is you can pick file I used the quality config uh pretty standard, just a lot of replica sets, because we want to see a lot of PODS how much uh these additional stats are going to cost us.

C

um So I'm, just gonna highlight here yeah key Prof uh grasp for this. um This is the main it's energy Branch, with no changes um with none of my custom changes. I think this is from August 31st I had the uh Visual terminates and my document, um but, as you can see, you know uh just filtering by these projects here it takes about the 0.11 cumulative time, uh which is the total function time um and flat flat time.

C

Here is the amount of time it actually spends um it's side of function and not calling other functions versus cumulative is the time it spends total. That includes call you other functions, so I had submit stats here like on Marshall, even which is uh related CR.

C

If you see like photobuffs um that here's the general list container stats um and again, this is the main branch here that we have our container D version, which overall is, as you can see this flat time here- is a little higher right for for marshalling um and I think that makes that makes sense just because there are more security eye fields and where stuff you have to deal with for the Buffs, all that.

C

But yes, but overall um aside from like on marshalling um the stats, are you know extremely similar um for uh those containers, thoughts which we actually calls uh runtime service? Client?

C

um Eventually, you know it's not Marshall, so Adam marching's a little higher, but the rest of the uh functions related to list container stats are I, think identical, um which you know indicates that this isn't going to be a major uh regression of with, in regards to CV usage by adding these new steri fields.

C

Yeah, and- and here is the uh flame graph for these, so flame grass are pretty cool. um They essentially show you. You know where, where functioned so, here's a root which is I'll get out of here, actually so so I'm just filtering by this expression here, uh let's pause stats, which is why it's saying root is only 13.31 of uh CPU time, um and that just means that we spent 13.31 per time in this uh function that we filtered here, in which case this is uh the spot stats.

C

um But, as you can see here, if we, if you do uh sorry yeah, if you do list container stats here, um I'll do this.

C

This container stats here, um as you can see this is like point- you know one one percent of time um and he calls all of our grbc methods here uh relating to the photobuff and then and such as the odd Marshall.

C

um All that and that's one of the things we're concerned about which is just the additional. uh You know: photograph overhead and grbc overhead there um and again this is our main containerdy version uh with none of my custom changes and so still look at the containerd version. With my custom changes here- and this is all for list container stats list- pod sandbox- that's it is a little.

C

It takes up some more CPU to have this um yeah, but, as you can see, my custom Infinity uh changes, um everything's super similar uh 0.11 or 0.09 seconds of CPU time. So no.

D

C

Major regressions there it's a little more like a Marshall in here, uh which makes sense just because you're, adding new Fields um around 20 or so to the CRM.

C

Oops, okay, all right.

C

um The next thing, I looked at um I was just listening to intercepts again. uh This is going to be listed. Hot stats. um Excuse me, you know container and pod level. So that's as a sea advisor currently supports. We need that for many different container, slash pod levels for uh each metric that we're looking at so here is the main you could look at this oops, okay! Yes, uh this is going to be the main version of uh of containerdy um yeah I'll, just better highlight it here. Actually so.

C

We do this, let's see list spots, that's that's the function. We're looking at here.

C

Yes, so let's pause that uh it takes up some more time and uh those container stats here, but if we're looking at yeah overall, so this takes uh 13.15 of CPU time here um and with the uh with the new CDI Fields. This takes up 13.31, um as you can see, that's 10.73 seconds versus upload for it um so again, not not really a major regression with the new CRI Fields, um but yeah actually what's interesting to note about Los pots, that's um if we followed the flame graph here.

C

um Those pod stats here we'll call list spots, that's partially from CRI. um So what this function does is like, depending on, if the feature gate is enabled or not, um it'll try to fetch some stats, the CRI um and but eventually it will fall back to fetching stats from City advisor.

C

And if you look at the flame graph here too, as you can see, sea advisor like get C advisory. Theater info takes up quite a bit of time. um If we can look at get pods that list pod stats here, that's 13.15 of CPU 10., um but really just getting the stats from C advisor takes up like 11.60, um which is honestly like most of uh most of the time. That's been here, um so um you know.

C

If that's eventually, we can try to reduce like the sea advisory implementation and get stats or something um we could save a lot of time um for, for instance, here right.

C

um There's this method that's called year, two um I mean if I was here so yeah, so this is It's called get spec, as you can see here, uh but yeah we'll just dive into where, uh where the functions are called from list plots that's.

C

um This will be a container info here. Get info, as you can see here, like updates back from actually calls gets back, which calls another gets back, but here common dog gets back is actually being called spice.

C

um So you know in theory, if you could, like figure out some way to cash it, even uh even just going with the basis that uh this the shorter time will be cached um if we gotta average. This among our four runs um whoops yeah. If sorry I think guys seem to have lost my tab here.

C

um Thank you. Sorry. Okay, yeah, here we go yeah. um If we like average this. Among our four runs, uh we could actually save like around three point, two seven percent of CPU time um just by caching common.get spec. Here um but again we are seeking to deprecate the advisors, so I'm, not sure if that would be the best use resources, but uh that's just something to note there um yeah and yeah.

C

We can improve it uh if we, if we want to probably um and just his another thing, I tested, I tested uh Prof with, was using again, like my main, like, like uh the man continuity release um and not using my custom maternity version, um but with feature Gates enabled and there's this cool feature yet called uh pod to container stats from CRI.

C

um If you said this to true, this enables the cubelet to gather container and pods. That's from the container runside, rather than straight from seat advisor and so Theory like supervisor is pretty slow right, as we saw here uh with the uh see the device container info, calling everything um and if you get more stats from just the Sierra itself. Instead of Galaxy advisor you could you know, the hypothesis is that you could save a decent amount of CPU type, um and so here is my side.

C

Here is the main container D release um strictly on.

C

Decides to make Affinity release with feature Gates enabled, um which will, in the future gate again is uh you'll, get pod stats strictly from Sierra um and, as you can see here, uh most pod stats takes up a little bit nine percents. This is the mint energy release versus here. It takes up like 13.15 um and now for a custom maternity versions with the new steered eye.

C

Changes um takes the 13.31 um without the feature you get and then with the feature get it takes up 12.19, um which is honestly a pretty solid treatment, uh but maybe not as much as we would like um and there's maybe some reasons why um they aren't as.

C

So either they aren't as um aren't as uh good as we might have hopes with the new feature. Yet um so so, essentially, what happens is like list pod stats. uh So here's the feature gate here. It says if P dot, Auto container stats from CRI, um then a little call big list strictly.

C

um But regardless, if you have a fallback here for, uh was pod stats partially um so yeah. If we could maybe eliminate this fallback or add some more stats this to your eyes, so we don't have to uh fetch it from C advisor.

C

You know, I, don't see. Advisory does have more metrics in Syria right now um we could potentially make things run a lot faster because you don't have to call that main C advisor method that takes up like majority of uh of your functions have.

C

um And I I think it was also strange to me was just you know: the Syria should have a decent amount of fields written and maybe see advisor to collect less Fields, since uh the fields are already going to be cached from our CRI collection.

C

um So if you looked at this uh flight graph here of, let's see if we look at this plan graph here, but eventually all the logic boils down to foreign.

C

C

Memory process etc for for whatever stats we are looking for, um but you know what's interesting so now here is that on.

C

And again this this is a run with the uh feature Gates enabled.

C

So, even with the feature Gates enabled only like pod CPU memory, stats are added right and look at this pod. Cpu memory stats are added, and so some stuff like out of process stats.

C

uh You know ad networks, that's um I actually accused that number, since that's, but like ad process stats, for example, isn't being called um and- and so this is, this is a thousand second run, and then this is a 1500 Second P Prof, as you can see, the Thousand second view Prof Ile um only like CPU pod seeking memory stats are added um versus.

C

Here you have some Network stats as well, uh for example, process stats aren't being added um and and which, in which case you know, I think there might be uh I think some investigation might need to occur as to why you know some of these odd pod stats at this itself are not being called.

C

You know, in theory, if these could be called, um maybe some additional CPU time to be saved as a sea advisor to have to do less work and uh touching these stats here, um yeah so I. Think, overall, though, um I'm gonna recap, this um there's not a major regression at all in adding these new CRI fields.

C

um I I said with the Prof and I'm, comparing like that: August 31st release of of uh containerdy versus uh my custom changes, um but yeah I think it is just interesting to note that touching metrics from C visor takes up a lot of CP time currently um and potentially that scenario you have improvements, I could be done just uh speeding things up.

C

uh Generally speaking,.

C

Yeah, that's uh should be it from my. uh If anyone has any questions or anything.

E

Awesome, thank you, Daniel for all the investigation and looking into that, um yes, I think like what this shows basically is. You know we had the earlier uh kind of conversation around if, with the CRI stats cap, um it's kind of was a performance impact around adding the rest of the fields to the CRI would be and I think this. This work and Daniel's kind of profiling and so forth shows that the overhead is.

E

It's should not be three highest quite minimal overall and uh in fact we have a issue with containerdy or with the key advisor today kind of not catching the the spec right. That Daniel showed so um by moving to to getting the stats from the CRI will eliminate that code path. So you could see some some benefits there um yeah. So it's kind of the findings from that yep.

A

Thanks I think it'll be interesting to also compare like right now. We are just returning hard coded values right like when we actually parse and fetch the values. If there's a difference or do you think there'll be enough- caching that there wouldn't be any additional impact.

C

Yeah I think it might be hard to say: I was read up, I know you know. Obviously it's obviously fetching. This actual stats does take more time too. As for my politics c groups and everything so I I think I would have to do some reputation. Maybe and I could run other people off and have some better results.

E

Yeah I think also one thing to point out like today. We already do that parsing NC advisor right, so if we, since the long-term goal of the cap rate, is to get rid of the parking and see advisor we'll just move that parsing to The Container runtime right, so that's like a fixed cost that will kind of have to pay somewhere right.

E

That's one way, I would think.

A

About it, yeah, okay, what's the total across Cube and the.

F

A

Right and we are moving some things- we are optimizing something so hopefully we.

E

A

Right in the end, yeah.

E

Cool yeah, so I think next steps. Maybe we can. We can discuss it in the follow-up, but I think next we want to see for for 125 planning actually or sorry for for next uh next kept planning. um We want to decide if we uh how we want to proceed with the CRI cap. If we're okay, it's going to add the rest of the stats to the CRI and and do that as the next step and Implement them. So we can start migrating those fields to the CRI and uh continue the effort on the cap.

E

So that's probably the next concrete steps you want to make here.

E

A

Sounds good thanks uh thanks David and Daniel uh you can move on to the next topic on the agenda. So next up we have uh kutong with an issue about unexpected initial delay of probes.

A

G

Yes, I'm gonna call um yeah, I wanna briefly talk about the issue regarding the initial delay seconds.

G

um I recently invested some issues uh on the probes uh invocation uh time, sequence and I find something weird in the in the 121 plus and I. Then I find a bigger issue which was already tracked in the in this describe industry. I pasted in the dock, um I think the the overall the bigger problem is, the initial delay seconds doesn't work, as the API says from the API.

G

It looks that it should be just uh the first time a probe got invoked is is from the start time of the container plus this initial delay seconds, but it doesn't, but our implementation is not like that um and I, and there's also concern about uh Google to restart, and at that time um um we it's not clear if this initial delay uh second still uh make a difference for different container like if to uh sorry.

G

If, uh if two probes have a different initial delay, uh when couplet restarts, um should we also respect the difference between them or we ignore it? That's not very clear uh and a second point is uh we already had some Jitter added in this uh before the before the probes uh brought before the props invocation uh I I from the from reading the code and the people's discussion. I understand.

G

That's that the purpose is to avoid standard herd problem uh where, uh during the couplet restart we it does um all the probes for all the parts. I understand, that's a problem, so we add the Jitter, but about Jitter. There is also problem. uh One is we are. We are giving different Jitter time to different probes. Even they are in the same uh same container and there's an ongoing PR to uh to solve this problem uh altered by math sales.

G

uh Another problem is what I just found in 121, because the Jitter is only added um uh conditionally.

G

That means, if say, um say if the say we have a container which have multiple probes and each probe have they all have the same period seconds and eventually they will run at exactly the same time, regardless of the initial delay seconds.

G

um So this is some problem. I find to see more details. uh People can look at the debug I pasted here, um so I want to bring this to people's attention. This may not have a huge impact to to most of customers, but some customers want the initial delay seconds more predictable, um I I'm, going to propose some fix to this. To make this, as close as to the spec says, which is the continuous start time, plus initial delays, probably doing some simple sleep instead of the current implementation.

G

um So I want to see if people have concern about that, if that's true I may suggest to I may suggest that we update our API DOC for this initial delay to avoid future confusion, yeah, that's what I want to talk about.

A

Thanks for raising this guitar, uh so I think maybe uh folks then take a look at the issue and maybe kitten you can come back next week uh with your proposed changes. Yes,.

G

A

A

Anything else folks have any thoughts on this.

E

Yeah I had a quick question if I can um I was just kind of curious, so with the um with the implementation change that we had, um how does it do we not add Jitter anymore I'm, just trying to understand the case where there's like a lot of um if that's over, because it's you know the overhead for the runtime and so forth of doing all those exact calls. You know at the same time, for example, for an exec probe.

G

Right uh I think Jitter is still needed, but uh um I wanna. The current time. uh Implementation of the initial delay is it's uh it's kind of a mask uh during which the the probe is automatically considered, as uh as uh as good I I I I, don't I, don't like that way. I may thinking we still need Jitter, but we also sleep a reasonable amount of time based on the initial daily seconds.

E

Gosh Roger, thanks for clarification.

A

All right, uh you can move on to the next topic for now. uh Adrian checkpoint, restore next steps.

F

Yeah hi, so um one of the discussion points during um getting the initial checkpoint support into um into the cubelet was always the the part um that all the memory pages are now written to disk, and um and what do we do to make sure? Not not anybody on the system can can access it so um for for the alpha release, we we said we just um store it in a way that only root can access it, and so um there's no immediate uh problem.

F

We are uh generating with a checkpoint, but for the future is how, if, if we want to extend checkpoint uh to to higher levels um besides the cubelet- and there were a couple of uh things we discussed how to how to do it and I I just try to think about it.

F

And one one idea was to add additional authorization to the cubelet API endpoint, but as far as I know, there are no. um There are no cubelet endpoints which have additional authorization. Currently, another idea was um to have the service run on a different port and provide um cubelet API endpoints, which require higher authorization like checkpointing, and the last idea I am currently thinking about is what would it would it work if we encrypt the image I see that cryo and containerdy are both using oci Crypt to decrypt images.

F

So um my basically my question is: is there anything anybody can think of like? Is it a totally wrong idea to to think about encrypting checkpoint images? This way we would not have the memory pages on disk um for everyone accessible. If we are talking and pushing it to a registry, then it would also be, and then encryption makes even more sense, so I'm just looking for some initial feedback. If, if people think that encrypting checkpoint images would be the right thing to do, or if it's not a really good idea.

A

Right, I I think it makes sense, but I think we'll have to talk about uh where we'll store the keys to encrypt right, because these images are only accessible to someone that has root on the host. And uh where are we gonna store the the keys used to like encrypt this yeah.

F

This is this is also not it's not not clear to me yet um because, um as far as I see, um uh cryo and I think also continually, they only support um decrypting images, but not encrypting. So it seems on the container engine level the there has been no uh at least no no visible um thoughts went into where to store um keys to encrypt images, not just to decrypt images, so um yeah, it's it's just an idea.

F

I had and I wanted to see if there's uh some some feedback, but yes, the keys need to be handled correctly. That's right! Yes,.

A

Really I think that's what I would worry about. It's it's a good idea, but we need to figure out uh how it would make sense.

F

Okay, so then at least that's uh some, some positive feedback, um so that's that's good and, and maybe I can try to see if I um can figure out how to implement something and then once we have an implementation, a proof of concept. Maybe it's it's it's something we can look into or continue from there.

E

So I had a quick question more general just around what you mentioned around the Google, API, endpoint and so forth. So I'm just trying to understand in terms of next steps for checkpoint research, it's kind of in general is that kind of the the next step is how we integrate it with kind of the Kubler API and um make it more than just kind of API call on Google it or kind of just trying to understand. You know how the work with the encryption fit into the checkpoint restore status in general.

F

Yeah, okay, so, as from from my understanding the the the the idea was, we only expose it on the cubelet API level for now, because we still don't understand what it means to have a checkpointing in in in kubernetes, really and and all of a sudden, because if I think the the concern was, if we go the whole way and expose it on the cube CTL level, then somebody could create a checkpoint and all of a sudden, you have all memory pages on disk, and so the idea was to to make to take small steps and only expose it.

F

The cubelet level for the first iteration so that we don't open up for anyone to all of a sudden be able to see all memory Pages written on disk. So that's I, think that was the idea behind doing it. Only on the qubit level.

A

Yeah I think also David, like there's a there's, a very like if we expose it at the API level and someone is able to uh to get like privileged access to the API, they can basically checkpoint and be able to read the memory of any container, so I think we're just taking baby steps here.

E

Yeah yeah that doesn't make sense I'm, just my kind of comment might be like. Maybe we should investigate.

F

And uh yeah, then one of the discussion points was: um if we want to take it further um to Beta or to to the to the cube CTL, then we have to find a way to make sure that it cannot be abused to um get the memory pages of a container just because you can do a checkpoint. No.

E

Exactly yeah, that's kind of what I was going to comment is: maybe we need to investigate kind of from the API side right if we can maybe add some special permission or role or something super elevated that you need to be to get these checkpoints right.

G

Maybe that's one.

E

Other solution we can look into like, in addition, maybe the encryption so today.

F

Because that's cool yeah, awesome, okay! Okay, thanks! That's I! Guess that's from my side on the topic. Thanks.

A

Adrian uh so last on the agenda. We have uh when I, uh with an update on in place vertical scaling.

D

Yeah hi uh thanks everyone, so, um with the In-Place update, I got to try out the latest continuity. The master Branch head of the master Branch I, built that and replaced the container de binary that when you deploy in gke, the default is 168 or something and replace that with the latest.

D

One and I tried out essentially I tried a true events code with the updated E28 test that does the full checks, which is uh validates that the update the resource status is updated correctly and it does update the iPhone one issue where the time it takes to run the end end-to-end test all 34 test cases was uh it jumped from 869 seconds to 2988, which is significantly higher, and it looks like uh the issue is not in the container of these side of things, but it's more on the once.

D

The update is done and you query and you get back the updated resources reflecting that into the status is taking a significantly longer more time and that needs investigation. I, don't think it's Alpha blocker, but it could be uh I, don't have strong opinions about it, but the issue is there I created it and uh we should track it and then fix it at some point of course, uh sooner than later now, that's the only thing that I found. uh As for a c group, we do support goes uh Ronald.

D

How are things going with uh I know you started looking into looking at it and I, don't know if you got a chance to take a full view of it and any more feedback on how things.

A

Are done, we and I have continued the review, so I I was out last week.

D

Just now no worries.

A

Vacation so I'll continue that and before the next week, hopefully we'll have a properly.

D

A

D

And the last thing I wanted to check was whether initially, we discussed or I think with the direct to split the pr into multiple, smaller peers. To that end, I created another PR for the API changes. uh Do we want to bring that in I? Don't know if that's something we can decide on if we, if that seems like a good idea, I'd like okay to test on that one so that it it's at least ready.

A

I don't see harm in it, he's testing it and okay.

D

A

D

Can you can you please yeah yeah I.

A

Can add that and then we can figure out like if.

D

What are those things yeah I wish? I could add it one of these days, I'll, send you a PR requesting to add with the contributor list. I should do that. Now it's been a while uh I'll do that this week, hopefully yeah.

A

D

A

All right, folks, uh that brings us to the end of the agenda. uh Do folks have any other topics they wanted to bring up.

A

Right, uh thanks for joining then see you all next week, bye now.

A