Kubernetes SIG Node, 14 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Node 20210714

Description

Meeting Agenda:

https://docs.google.com/document/d/1j3vrG6BgE0hUDs2e-1ZUegKN4W4Adb1B6oJ6j-4kyPU

A

Hello, it's july, 14th uh 2021, it's a ci subgroup of signaled and we are talking about node conformance tests, um so conformance tag was created for kubernetes certification program. It's supposed to be uh agnostic test that works supposed to work everywhere, uh and if you pass them, you certify it to be called kubernetes or something- and I think initial idea was that note conformance will be in the same vein.

A

So it's a subset of tests that doesn't require cluster. That can be run and if you can like, if you're can components pass this test and your node is certified as a like compatible or something. uh However, there will no agreement that it this. It will be part of certification program in any sense.

A

So, um as time passed, we like, I don't think there were any attempts to make it uh uh part of the certification program, nor any desire so far, um and um so the word conformance is initially intended to be uh same as conformance and cluster test, but uh it doesn't actually represent what it currently is.

A

um So I think um um we can like. Today we have.

A

Yeah uh ordering is a little bit weird. So if you look at this conformance tag description, there is a document saying how to do cri testing policy. I have this red sign on the link, so this policy explaining how to contribute runtime test results into um into common dashboard, but I'm not sure whether anybody using that um so there is a faq and.

A

Also, all sorts of documentation, but uh I don't know with anybody using this program and to certify the runtimes. I think we are running like container d tests right now and the cryotests it's part of our test grid and there is nobody contributing back cri test results, but nevertheless um I think this is the reason, by the way, uh elena, that initially you've been pointed to cry as a not conformance kind of uh is something that started not conformance because of this document.

A

I think- um and but I mean description of this document saying that uh that should be agnostic to cloud providers less distributions and run times.

A

So it's basically a set of tests that doesn't require any special handling uh so, like everything's supposed to be working without uh any special configuration- um and I think it's a fine definition- we can use it and I suggest that we keep like we rename conformance tests to something more appropriate and then um for everything else, for like um everything else being alpha beta features, plus uh some features that have special requirements.

A

We can use feature tag plus some extra text like alpha or special, to indicate why white makes it so special and then, when feature is graduated, we can rename it into node conformance or whatever new name will come off is, and this will indicate that this feature is supported by every distribution and uh it's part of like big, uh I mean part of required tests for node.

A

So this is a proposal, um and I think I just combined uh this document out of different sources uh and I think that maybe how we can structure tests and uh if you publish it as a policy, we can clean up all the tests like right now. Many tests have conformance plus feature, and I think this is not what everything like what we intended to do.

A

What do you think.

B

Like amen to all that, that was great uh thanks for doing all this research, sergey uh and brazil. I didn't even know about this cri testing policy. This is like totally news to me, so uh this is awesome. Thanks for doing all this digging, uh and hopefully the digging that I started at sig arch like a couple weeks ago, was also helpful uh as a starting point.

A

Yeah absolutely.

B

So I I think that we should probably like I would say next steps are probably to send something out to like sig node mailing list like share this publicly, maybe uh and get feedback. I'm sure that uh dims will have uh opinions. I'm sure that, like uh if we could get like aaron or ben elder to weigh in they should have opinions. I'm sure the conformance folks will have opinions so uh yeah. I think just getting eyes on this, and uh we might want to come up with before we we set this publicly.

B

We might want to figure out a new name for node conformance, but other than that. I think this is great.

A

Yeah, um I was thinking like not, I don't know like all. My names are stupid, so and names are hard.

A

Main suit or like main tests, something like that.

B

Well, so I guess, uh I think that we should totally move away. Well, I guess it depends like there's kind of two options right so, like historically, we've had this thing and, like the node end, to end test to this day like there's this idea of being able to run a subset of all of the tests just with a node without like a fully functioning cluster, and so I think that we might need some way to indicate that- and I don't think node conformance is a good way to do that.

B

And then I think, there's maybe a separate issue of like how do we make sure that all the cris are like functioning the way that the cris are supposed to function? And maybe for that we could replace like right. Previously, we had this, like conformance tag with the hope that this will be kind of like a you know, cri conformance or that kind of thing. But the conformance game has asked us.

B

Maybe please don't use conformance, because um that's confusing people- uh and I agree- I was confused- uh so maybe like what about like for the cri specific ones like. Maybe we could try the name uh like I'm thinking, cri validation or something like that.

A

Okay, but this is not shirai specific- it's basically like it validates all the components that uh present in the node, so.

B

Yeah, so I I I know that uh I'm saying we shouldn't we shouldn't like wholesale, just rename all of the node conformance tests, I'm saying for whatever meets this, like cri validation criteria.

B

We should re-label those as cri validation, uh and I think we should figure out a different story for what we do for the like test that just you can run locally kind of thing. You don't need a full cluster.

B

I, basically you know, node conformance is trying to do too much. We should split it. That is my proposal.

A

Okay, so you suggested that uh there will be no two categories, but, uh like third category will be well, I guess conforms will be south of sub-categories, like cri validation, configuration.

B

So I mean there's like the actual anything: that's a real conformance test should be marked with conformance and anything. That's not a real conformance test should not be marked as conformance, and so my suggestion is like. I think that you know all of the like the standard feature things that we used to call node feature whatever that we're trying to migrate to just use the standard feature tags. Those are fine.

B

The reason I think that we were tagging them specifically like node, as opposed to like uh you know, just a normal feature was because we could run them just on like a single node in a more minimal node, end-to-end test setup, and so that's the thing that we're kind of losing uh and that's part of like why. There's like a bunch of confusion here, is that, like historically node conformance was trying to be two things thing number one was, it was trying to be a reduced set of the end to end test.

B

You could just run on one node and second, it was also trying to be a set of conformance tests for nodes, uh and I think, if we like try to split that into two things. I think there's also overlap right, like some of the things that are conformance tests could just be run on, like one node, without like a fully functioning cluster, uh depending on what components are required, but maybe not all like.

B

For example, I think that there's no scheduler in the node e to e setup, so there would be some like um there's like some things that wouldn't make any sense to put there, because you have to have a scheduler, for you know like full cluster conformance.

B

Does that make sense like I think that we should treat the two things like? They are two separate things and possibly split them into two separate things.

A

Yeah um I'm trying to.

A

A

Yeah, I think that note conformance would be pretty much everything that doesn't require any special treatment.

A

On a single note, so you're saying there is a separate dimension for that is uh uh which part of cubelet we're testing.

A

Oh, like so see, right.

B

Now, okay, uh so that way, uh it's like clear.

B

B

Does that make sense.

B

Also for people who are not staring at my screen or the test text is too small or whatever I said so for conformance uh historically node conforms tried to be two things: one. It tried to be a set of end to end tests. You just needed one node for two: it tried to be conformance like tests for nodes uh and I'm suggesting we get rid of the name, node conformance, because it's confusing everyone and then for the conformance like tests that we want to like have the cri run against.

B

We should call them something else, probably cri validation and then for the tests that you just need one node to run. We should come up with a new name for them that is not node conformance and it's possible. There might be overlap between the two. So I'm sure that, like some of the cri validation tests will not need a full clustered run, so they would also probably be in whatever we want to call this like uh that. This needs a name.

A

Yeah, but isn't it all the tests that in this folder doesn't need any cluster.

B

This way, this is like anything and I think it's like tests, slash e to e underscore node, as opposed to e slash node.

A

So basically, all the tests needs to be marked with a tag, but I mean it's already marked with sig naught yeah.

B

But that yes yeah, but I think that's a.

A

Signal, but without cluster okay.

B

And then we could get rid of the special weird e underscore node folder, perhaps like, because that's been confusing everybody as well. uh So I don't know I mean that's, that's one way we could go forward. I think we need to get feedback on this, because maybe that's gonna just be too much refactoring effort or something like that.

C

A single note test or something would work easily enough or like kubrick, local or something.

D

Cubelet local looks nice to me because it the reason is, I believe, it's, it's not really a single node cluster, so maybe I misurd because it's a very, very specific environment, so cubelet local seems very nice to me.

C

Yeah, like it technically spins up like an api server and an scd, but like mostly for like dynamic, config and reporting by statuses and.

B

There's definitely no scheduler, because you have to schedule the and admit the bots.

D

A

Yeah yeah, this stack can be added automatically by seek describe right because it's applied to.

B

The international concern was so you know how this will. This definitely applies to everything in the test. Slash e2e, underscore node. We also have a bunch of tests, labeled sig node in test, slash e, slash and then I think, there's a bunch of common tests as well. That are us, so we do have some that require a scheduler. I think currently.

A

Wait, we don't have any tests in these folders that require scheduler right. We only have a.

B

Separate you know we understand, but then we also have.

A

Yeah yeah different one tests and not not right.

B

In case people are like what are you talking about? Are you losing your mind? No, there is in fact a test, ue underscore node folder and then a different eoe, slash, note folder. I.

C

I learned this the hard way today.

B

uh Yeah just now just.

C

B

A

B

Thinking you know.

A

Welcome to the club, our.

B

Tests definitely make sense uh like.

C

It's great I'm.

B

Trying to fix a fiction test.

C

I learned a lot of horrible things about our test suite in the process. There's.

B

Also, a big warning, so that's cool.

B

uh I'm just learning so many new things: misplaced and legacy tests. Okay, so apparently we're supposed to clean these up, sergei.

A

Okay, but yeah we're on the same page that uh this couplet local can be applied uh universally in sick describe uh function right because, like the same way, we apply sig note uh to all the tests. You can apply: kubelet local there.

B

Well, maybe what we should do is we should get rid of this, and we should focus on like building up this weird e to e node suite.

A

Okay, but then, uh returning to your first saturation about ci validation, I think I was suggesting that we will basically mark everything that doesn't require any special requirements and graduated with some conformance tag. um So you suggestion to split it into features rather than keep it in.

B

A

B

Conformance is very specific requirements uh like right now, I think they are only uh so conformance does have to be reviewed by a conformance reviewer, uh which is why I suggested we maybe poke aaron, because I know that he's one of the conformance reviewers like not in new zealand, so maybe a little bit better time zone overlap um anyways. So as far as the conformance tests go, I think they only really test stuff.

B

That's in a user-facing api and node has a lot of stuff so, for example, anything that's just testing, for example the cubelet talking to the container runtime uh anything, that's just like exercising that and doesn't involve like a user, submitting a specification like that's, usually not considered for a conformance test. So the conformance test is just checking like the surface area of the external api yeah.

A

That's right right: we need to rename it for sure. That's why, like, if you switched as a document, I suggested that uh we like in this category will be all the tests that uh feeds uh like that are agnostic and don't require special environment.

A

So all the cri tests will fit like most of the tests will fit into this description.

B

Yeah so there's there's a bunch of like conformance test requirements.

A

Yeah, I understand that I I'm talking about not conformance tech. That was never this yeah.

B

Yeah, it was never that so that's what I'm saying I'm saying I I agree with the plan for like for all of the stuff. That's like you know, features and whatnot uh like those should go there, but like for the stuff, that's ga that is not eligible for like becoming a conformance test, because I think there's gonna be a bunch of stuff that isn't I'm saying what do we do about those?

B

Well, maybe we make like a cri specific suite, that's a subset of what this is in order to make sure, because when I said like, can we just rename node conformance to like you know, cri conformance or something like that? I got no. No, that's not what node conformance is it's also these other things so.

A

So do you think we need a so in the document I suggest.

E

A

Everything that is vendor agnostic will have a common tag. So you're saying we don't need this common tag. We just need the like subsets like sub areas that we need to uh like highlight.

A

So I was thinking like we have. We need a common tag so something that can be run on single note, independently from any component that you're running you just run all the stats and you go from node perspective. So I wanted this tag for not yeah conformance.

B

It's intersection of.

A

This plus not conformance ish, like yes,.

B

So no no conformance goes away for like this sort of thing, the stuff that runs on a single node.

B

We call them one of these things, uh but then, if like, we want to continue doing whatever this, like cri validation thing is which right now we are using to select tests for the different cris like we're running node conformances, I think presubmits and uh a bunch of other stuff like for those things we might want to have a special tag for that uh and that maybe that should be cri validation uh for tests that conform with this policy.

B

Because, right now we don't.

A

So sirai validation is one of the features.

A

So let me try to explain how I see it like the validation is one of the features, we're testing, so it's area of kubler that we may want to mark, but we also need a common tag saying all this test iv and diagnostic can run on any runtime, so we we want to mark them all together and this, and by runtime I mean environment as as a whole, so it includes some configuration tests, some like conformance tests, some like my research allocation tests.

A

So all these tests can run independently from environment and we mark them as single tag. uh So we can run it as part of like pr validation or something like that. That's why I wanted like common tag for everything uh that is not the same as kublet uh local, because kubel's local is a superset of that.

E

Local may have.

A

A swap, for instance, and swap, cannot be conformance.

B

Well, swap can't be conformance, uh but I think a bunch of the cri stuff also can't be conformance. So this is why I'm saying.

A

B

Should stay away from.

A

Yeah, not the conformance, we need a new name, but not conformance, not yeah. We need a good name, not sanity. That uh mike suggests uh archon suggested.

F

Science is like base seat of tests that we will just say the cubelet is working fine without any problems. Okay,.

B

And then, we've also separately got this like cri validation testing, but that doesn't show that it like works in concert with cube. So maybe cri validation is also a bad name, because we also, we already have that in cry tools which is different.

A

Maybe not basic, instead of not conformance or not basic tests.

B

Well, you you'd emphasize like uh and like we wanted to be agnostic of environment, so maybe like node agnostic.

F

It's a complicated name: okay,.

B

All of these names are complicated.

B

I think I think, honestly, you know uh the brainstorming is good, uh but we've spent a while discussing this topic, so isis would probably like the best thing to do is to like think about this. Asynchronously send out a doc for feedback. So sergey can I action you to send out a doc, the node conformance doc, soliciting feedback.

A

Yeah, I think in pr may be better, so maybe I will send it as a pr.

B

And so I guess uh maybe also like.

B

I think that this stuff isn't reflected currently in that doc. So, maybe just like, add a plan to cover splitting the use cases for current tests, and I will leave that in your very capable hands. uh Sergey.

G

Do you so plan to get rid of this e2e legacy.

E

A

E

C

Like we'll get rid of it eventually, after we've all forgotten about it, and someone else reads the readme again in like five years.

B

I mean that's where we're at so you know.

B

uh Okay, let's see I put a thing on here for 122 burned down. We were in really good shape. uh The last time I looked uh so like not to give us too much of a pad on the back, but um let's take a quick look.

E

um Oh, I shouldn't have done that.

B

That's gonna take six years to load. Oh it did it.

B

Oh, this is not gonna, be useful for searching anyways.

B

So last I checked we were doing really good for 120 to burn down. um So there's a bunch of stuff here and not all of it is us, uh but like we only have nine things and we're like not even at test freeze, uh so that that's better than last release, um so great work. Everybody uh stuff that are specific to us is, I think, this bug plus uh the four tests that are being tracked, and I think that.

B

Is this one this one this one? Some of them are like they're, not regressions, they're, not new, uh so I think probably none of them have to land. This release, um like I think all of these are old uh flakes, so none of them are going to be release blockers, but we can continue working on them.

B

This is the only thing that maybe should be a release. Blocker.

B

So- and it looks like there's a thing going on with ronzi here so.

B

I will maybe take a for this one. I will try to follow up with that one online. uh Let's see, let's go through these rtm. What's going on with this one.

F

uh Probably we are waiting for some input from someone who is more familiar with the sea advisor.

H

ah Okay, so I guess you're, you already been pink someone uh bobby yeah.

B

Sergey, do you know what's going on with this.

A

Yeah, I will ask again: what's going on.

B

Arjuna, do you want me to unassign you so that I don't wake up? I don't end up pinging.

B

You, maybe david, didn't see it.

B

uh Cool okay, so no updates on that one. uh Oh our favorite test, uh matthias! I owe you a review or something I think.

A

I think I reviewed the hackmd document that you sent.

B

I haven't had a chance to look at it. Is it it's not linked here.

G

No, it's on the other one. The hack document.

B

Where, where was your pr I don't know, I can't find it. Oh, maybe it's this one. This was the one that I think I put a hold on yeah. We were digging into this and uh I okay, here's the hack empty. I haven't had a chance to look at this since last week because I got really bogged up with code freeze.

A

I think timing looks good. I only have one comment on proposed change. I don't know whether this change is actually needed. I mean it's, not a bad idea to do it basically idea here is that uh we have a radio state and we assign it in the very end, like we calculate ready, state and then assign very end, but there is like a go routine in the middle that may execute faster.

A

So I think uh I gave a comment that there is one more place that may benefit from uh being properly assigned.

G

I I didn't see your comments.

A

Because there is like stars or with some like sign in hakandi, if you scroll all the way down, yeah yeah, oh three, I think you're singing.

B

You so one of my concerns with this is what basically, when I was looking into the code, uh I did a bit of a deep dive for this and I sort of concluded.

B

I thought that this test was written kind of weird, but then I also went and looked at the underlying code that it was testing and that code was also potentially buggy uh so like there was. uh I think this yeah this case where I was like, I think, there's a race condition, because we go and we trigger this thing asynchronously.

B

But then we immediately go here and we set the status to ready which might overrun whatever ends up getting run. uh So there's a there's, a race.

G

B

So I thought this was this chunk here was kind of weird uh and I feel like probably fixing this. I feel like it's too late in the release to do this now, but this is something that we might want to do next release.

B

I think it's probably tough, like fine to tweak the thresholds and see if that at least reduces the flakiness of the test, and then we can take it as sort of a backlog thing to try to fix this, because I think this is a real bug and I think the test is catching a real bug, uh but uh having it, flake constantly is like unpleasant for everyone.

G

Yes, I agree, okay, so I will.

G

I will create one pr and one issue for that and assigned to the 23.

B

Okay- and I think uh this this isn't enough compared to like- what's we need to also increase the frequency right? Yes, okay,.

G

Yes, yes, yes, yes, yes, yes, okay, but I didn't want to change my apr uh for the test.

B

That's: okay! That's okay! uh I just wanna. I wanna make sure that I write down accurately because I'm going to put a comment on here: uh disgust in sig, node, ci meeting. I guess today is july 14th.

B

G

National day in prince.

B

Oh yeah, happy bastille day we will adopt the proposal outlined here to update the frequencies in the b2e test to.

E

Decrease flakiness and take a backlog item to fix the possible race.

E

Identified here and.

B

You will file and update sound good.

G

Yes, thank you.

B

Awesome: okay, that was in great shape, I'm so glad we're making progress on that because I know that's been haunting all of us. uh Okay pod should run through the life cycle of pods and pod status. What's going on with this one.

B

Something oh, this is the. This is the weird it's just failing for cubelet orphans uh and it looks like there was.

G

um There was like fix, and and but it's still failing in the node orphans, and now I have like a debug output which doesn't really help, but since I'm on vacation, currently I didn't take extra time to look at it, so I will but soon enough. I promise.

B

That's okay, it looks like all of these are memory manager, tests.

B

So perhaps does uh is anyone from uh the the memory manager side of the house interesting? Yes,.

F

uh I'm interested, but in general, like we have a separate lane for the memory manager, and it should be just in excluded from orphan tests. It's funny that we have them.

B

Well, maybe that'll fix this, then if we exclude that so do you want me to assign you to do that? Yeah.

F

Yeah yeah, I will just check the test like under the testing for job config.

F

E

uh And I will own a sign.

B

Making such good progress uh amazing, okay last one- oh this one is pause- should support pod readiness gates. uh What's going on with this one francesco.

D

um Is yeah I'm still waiting but still struggling to reproduce locally because in ci I will it's hard for me and uh true to be told that was super busy with the code freeze, so yeah this one and another one I'll need to prioritize. It's on me. Sorry, understandable.

B

uh I mean that's not.

D

Understandable still bad, I promise I do. I can do better.

B

No, no, I mean it's. It's fine, like uh I guess, since whatever happened over here like this is flaking less uh I mean. Maybe it's just because code freeze happens since the eye isn't necessary. Don't know, I don't know if, like uh frequency of pr.

D

The real blocker is that works like a charm on my environment, so I nearly need to invest some time on see why it breaks in ci. I guess.

B

One thing I'm a little bit curious about uh is like who owns this feature: pod readiness gate- I don't want an olive github.

E

I just want this repo.

B

Github you're trying to be too smart.

B

I can't find a cap for it.

B

Which makes me think it, maybe it's been around for a while.

B

Well, it looks like it went ga and 114. So that's probably why there's no cap, uh so it's been around for a while.

B

It probably means whoever wrote this is disappeared.

D

We have also pr to just increase the timeout. Let me fetch the link, but I'm not sure why uh why it is not producing.

D

All right, yeah ready, skate timeout this one so.

D

Let's share this in the chat.

E

Oh there's so much going on the trap.

E

Could this help.

D

You know it's the first try, let's improve the logs and give it more time, because I really believe I really suspect something in the ci. So yeah not claiming is the fix, but it should help.

B

Oh yeah, I guess this waited forever. It's.

D

Right until the test timed out, no it's less than minutes 30 seconds. There is a nice comment which explained why forever is 30 seconds.

H

B

uh So this just looks like it's kind of at least from this refactor. This looks like it's just kind of a poorly written test.

B

This at least just a little bit.

A

And you'll have better login.

B

E

I think this is fine, looks good to me.

B

We can probably skip the serial test since they're broken.

B

uh Great well, that's progress, awesome work uh and then I guess uh francesco you had a couple of things. We want to talk about cereal.

D

Yeah I want to to syrup, because it's quick and arthur may pointed to me that uh the cpu manager test is failing lately, so I believe, but I need to check and again, code freeze frenzy was why I didn't check yet that this bug I filed some time ago. It's actually broken now. The reason is the cpu manager entrance. They assume a very specific cpu layout. They are just fragile, so they can break. For example, the cloud provider just changes, cpu generation they can break so.

C

D

It could happen, and this is of course outside our control. So while I can say hey they break because of this fragility, this is surely something we need to address and it's a good chance. So once I get the chance, I will start looking into that and possibly fix this issue, so I was just point out that it could be the test fragile. So that's something we should take into account.

B

Let me pull up the serial tab.

D

And I, by the way I have fixes for those two, some somewhere, the.

B

Critical part on the yeah do we have bugs for those.

D

B

Because we should probably fix those.

B

There's a bunch of things that look like they're, maybe flaking, but this these two look.

D

I can fetch.

B

I

D

If you, if you want to look.

F

I have some questions regarding the dynamic configuration complete configuration. I saw it deprecated now. Yes,.

E

F

Curious when it will be removed because we need to adapt all our tests, cpu manager, memory manager and the opposition engine, it.

B

Will be at least one release.

I

B

Probably can speak to that since that's uh his thing.

A

Yeah it's at least one release. I think uh the only user right now we found is maybe alibaba and I'm not sure how vocal they will be about removing it uh but yeah. I am aware of that. That's relying on that. I thought that duplication will go faster and I will have time to start migrating. Some tests.

F

Yeah, but we still have some more serious duplication regarding docker, because our tests, all e3 node steel, is using docker, so we probably should move it to use continuity.

C

Yeah a lot of you know it relies on dynamic, kubelet config too, like the eviction test and stuff.

I

B

I'm going to file one bug for each one of these failing tests.

D

Yes, please thanks for some reason, I I didn't think about filing bugs. I.

B

Love how there's this magical- I don't know if anybody's seen this, but you could do this magical file, a bug thing in test grid, and it will just let you do the thing.

D

B

E

So I'll say, failing test sake, node uh cc, francesco.

E

B

Let me, at least maybe include the tab. That would be nice of me, wouldn't it I didn't do that on the last one.

B

So feel free to.

E

My first comment.

B

Cool, uh I guess that's kind of all I had unless anybody has anything else for the serial things.

D

Not to the moment.

B

I need to take a look at the ci board and see if there's anything there. That's waiting on me uh the as far as like features go we're in great shape for code freeze, and I know everybody's been working really hard. So thanks, everybody for all of the effort that you've been putting in things are looking really good.

D

Okay, cool: I will be better prepared.

B

It'll be uh we'll be like less exhausted from all of the code freeze and bug squashing and who knows what else.

B

Sergey got anything else.

A

No, I look at ci board and all the issues are assigned. So if you have any issues, just take a look um well like not issues prs assigned for reviewers and uh yeah now, maybe to approve like we have 13. That means.

B

Approval- I don't know if I can put me but oh there's a couple that are assigned to me that I haven't looked at so I'm guilty. Sorry.

A

All in the same boat.

B

Yeah, I don't know if there's anything else to throw in here.

A

And we almost had time so I think we can yeah get eight minutes back. I guess.

B

We've got these two new, wonderful uh things. Oh there's a few.

E

E

I think that is thing.

B

I'll get to that one later.

B

Cool- uh maybe let's just quickly look at this one, because I didn't see this wow. This was this looks like a scalability thing.

B

So I guess there's this one new yeah. This is gce master scale correctness that doesn't sound like us.

D

Elena, sorry, just a quick question: the the flake pod readiness fix or well improvement. We just reviewed should it be in the milestone because it lacks the milestone label.

B

um I think we'll see like, depending on what the fix looks like and then I can. I am a milestone maintainer, so I can add things to.

D

A milestone, okay, cool great, so.

B

D

B

And I will do that.

D

B

I feel like did we merge something recently that changed ssh things.

C

It changed some subset of them, but not everything. Yet I think.

B

Okay, so maybe I'll cc, I think that was sasha's pr, so I'll cc him and then see what happens, because I I guess this was just filed three hours ago so otherwise I think we're done.

A

Thank you for driving the screen.

B

Thanks everyone yeah no problem sergey, hopefully next week, you'll have your uh setting yeah fixed.

A

I have much better picture now right at least video yeah, okay,.

B

A

B

Thanks everyone cheers thank you.