Kubernetes API Machinery Special Interest Group, 18 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20230518

Description

- The Implicit Kubernetes-ETCD Contract
- Kubernetes control-plane upgrades
- Proposal KEP-4008: CRD Validation Ratcheting

A

B

A

um So hello, good morning, good afternoon, good evening, depending where you are I, hope, everybody's doing fine and healthy today is May 17th of 2023.. This is cjpa Machinery by weekly meeting in open source kubernetes, and um we have a number of items in the agenda today and we are going to go in order and the first one belongs to Han and Merrick. So why don't we start with that?.

C

Sure so we've actually discussed this before, but there is an implicit contract between kubernetes and ETD. America and I have drafted a doc on the explicit expectations that kubernetes has from etcd, um and we want to codify this. uh We we don't want to stop here at um just a written documentation of of what the contracts are between um the two binaries. We want actual tests in a shared interface so that, basically, we can concretely be confident that we haven't broken any underlying assumptions that we might have of the underlying storage layer.

C

um Merrick has like I I, think I had I had some code that I had shared with Steve at one point um and I think he was supposed to take that and do stuff with it. But um I, don't know what happened to that and so I think Merrick has been sort of following up on on those threads.

C

I think Merrick might even have a demo of what this could look like.

D

Eric yeah um so following handset can I share. My screen is that okay, so.

D

Is that okay? Okay thanks, so let me show or yeah just first to introduce myself. Can you see my screen and hear me? Yes, I'm Mark I work on hcd. You may know me from emails like this I. Don't know if you if this is readable but yeah I work on that CD and I spend a lot of time on finding uh or announcing and finding all the latest data inconsistencies on that CD.

D

um So yeah we had a lot of things to learn as a community and one of those things we're learning from our mistakes and uh like making inconsistency disappear. So one thing that I wrote is a new framework for testing HD, which required us to re-abulate all the API guarantees and watch apis, and we test it by having model tests, which basically means we implemented an in-memory, a fake of or something like fake of LCD.

D

That behaves like hcd has the similar not full but similar API like hcd it's in process, and we use it to compare behavior of HCG versus the intention. So if you can see you can we, we have couple of old history cases that we reproduced in reproducing the traffic and we compare hcd versus the model and we can get a nice craft slide.

D

This one which is I, don't know if it's visible but yeah. That shows inconsistency. So this brings me back to what handset is. If we have a contract, if we have updated API guarantees, if we, if hcd Community, focuses or I am proposing for at City Community Focus on or prioritize implementing an hcd, mod or kubernetes traffic, as defined here as a milestone, we can get a full HD contract.

D

So I took a word that handed and that's if that's not the super great but Han, basically Define an interface and has a publicly available start on on his spark search. Interface, uh thingy and I took it and if GitHub shows me and I took the the model that we have on HD State copied into kubernetes code implemented the same interface and have it running so it's required to disable a lot of things to do on the testing.

D

But it's small. It's some most of the tests work to to give you even more example of something more concrete.

D

I have a full set of press under API server package storage. That's, of course there is couple like couple of them that are removed. I didn't have time to fix them, because this is a POC, but most of them run and pass, and that's all from like the demon that I wanted to show. If there are any comments or questions, I'm happy to to share and discuss.

A

E

Yeah I have a I, have a question um so so that demo was neat. um It appeared to show this is what kubernetes uses, and this makes sure kubernetes doesn't grow anything else, but I don't think I saw where it says. This is where we make sure at CD doesn't break what Cube uses, which is the half I, was expecting to see.

D

So we I'm doing it on the the I mean on the issue. I, don't know if it's I think feather opened it on the issue. I'm disguising that there is list of Milestones that I slowly NPR that slowly implement all the features. That kubernetes depends on or add the traffic types and contract elements one by one, because it's pretty complicated contract that grow our coverage of those tests. So we currently cover some at large Parts.

D

If you can yeah look at the issue, you should get an idea and treat the peers.

F

E

F

E

Guess it's not clear from from the stock. It is the goal to actually be sure that SCD doesn't break the contract or is the goal to to say this? Is the contract from Cube and we're pinning it or um or any addition needs to be paired with something I like like I can see a lot of purpose behind. We want to make sure NCD doesn't break what Cube uses I see less purpose behind something that says we want to make sure Cube doesn't use anything else.

F

C

Are definitely not saying that yeah we're, definitely not saying that we are definitely not saying that we are just. We just want to ensure that uh ETD doesn't break Kate's expected behaviors.

D

And using this approach, we found cases where it broke.

B

D

Yeah, so we want to have so benefit like Main benefits. Is there was a request, like uh I think can't mentioned our previous discussion when we, when there was asked about having an interface for for testing of kubernetes testing at CD like or testing against generic storage? This is like uh one example of implementation for of this interface that we are working on and its implementation. That should be. uh That should be easy easily to yeah to to prove to cross validate both both projects.

D

G

I yeah just I'm gonna Echo it back to see if I understand. So what you do is you take SCD, you create a model of it and you want to you want to validate that.

G

The model is accurate, so you take it over to kubernetes, um you swap it in and if everything works, then the models probably accurate, at least as much as kubernetes cares about and then that would then allow you to go back to edcd and and use that model as a way of ensuring that SED doesn't be coming like, doesn't violate Kate's guarantees and then also, if you expand that model in some other way, you can go back to kubernetes and make sure you didn't break the model by checking it against kubernetes.

G

So that kind of the that kind of the um the thought.

D

Yes, yeah uh plus we do. This is a robustness test, so the model is just small part, because the main benefit is test, like volatility, that HD behaves like the model in and under any circumstances. So any injection uh at any I know, storage, failure, any network failure and we are like throwing the list of those failures.

G

I mean this seems like a really good way to really good way to test NCD, um especially that I like the model based approach, because these the model, at least for those that haven't seen Merrick, had a really good talk at kubecon. About this, um it can can do a bunch of more subtle things around um checking various like interleavings of operations that that might not be expected, but could happen in the real world.

G

um So that all makes a lot of sense to me.

E

G

Is there anything that you expect to do on the kubernetes side, or is it mostly just you know, tested against kubernetes every once in a while make sure it's working? Do you need? Do you need a harness in the kubernetes code base? So you can do that more easily.

D

um I I think you have more concrete ideas or.

B

G

Is there other things that you need on the kubernetes side, for this I mean? Obviously you swap it in every once in a while to make sure that the model still works.

C

You want to do. I would like to Upstream this thing so that it's just there.

E

So, for for the uninitiated, what what does does that mean like this exists in a package, or does this mean, like I, replace the usage of an NCD client with a wrapper around an NCD client.

C

Yeah yeah there would be a rubber yes, so yes, there yeah, there would be a wrapper and we we don't have. We haven't completely worked out where we would store this interface, but ideally it would be some place that both at CD and kubernetes could pull. From that way, we could Ensure that we are testing the correct intersection.

G

So it sounds like you want to enable like a certain kind of like test. Workflow. Is that.

E

C

E

Basically, yes, there's basically some tests. You want to run regularly. We have the side effect of Cube no longer owns its own storage interface right like so so as a for instance. If this moved to uh a.

A

E

2K, repo uh and everyone depended on that, and then he was like Hey I want to use this new method. uh It would put whoever owns the repo in a position to say no and like there is some concern over that. Where, where would you imagine this living.

C

Maybe case details.

C

Something like that: okay, yeah yeah it kubernetes but yeah for sure, like, like you, tell us something that can be imported into both without dependencies.

C

And that basically, uh should mitigate your concerns about not owning the interface right, because it is actually a.

E

To some degree, um yeah I'd.

C

Like to think about it, yeah yeah for sure for sure.

E

The idea of having the interface defined makes a lot of sense uh being able to use that to test the behavior of SCD to be sure that it doesn't break the things that Q relies on. Also very good I I have seen I. Think I saw in three five the test framework here talking about America. It was awesome. It was the one that let me inject failure at this point right, yeah, that.

C

D

C

E

D

B

D

That, like there was cases that we found that, like there's at one Edge case, that we found that Etsy or HD broke the contract, and it was a very subtle bug on client side I think it's already fixed and upstreamed on comparative side too yeah.

A

Okay, so there is a couple of hands: I think Mike has been waiting um first.

H

Okay sure yeah- this is really interesting and great um I was wondering how you test uh interleavings. How do you control time to get the specified interleaving.

D

So this is more going into my talk from kubecon. We don't depend on time we validate like with validate linearizability, which means we use uh linear, like we execute some. We have some execution history from tests during that test. There was some I know injections, so, but we have a history and we can use this history to validate.

D

If, uh if the history is consistent with the model- and we don't depend on I mean we don't to the we depend on linearizability Checker to find an order to uh of uh operations, and we, if we have an order, I mean it's. If we have an order, we can or linearizability tracker will say this works if it finds at least one path of model. That is correct and it will reject if there is no possible path. That model could take throughout all the possible execution order that uh yeah, if.

B

D

Is no path, the model or the history is not linearizable, and it means at CD is wrong. I can send you to like the talk or I can send you to the documentation of HD robustness, test that show exactly the a demo for this like it's.

H

uh Yeah, could you please drop those pointers into the notes, I'll. Try to ask just one brief question in summary: uh are you saying that uh what this does is it takes whatever happened and doesn't really take care to control? What happened and says is: was this a legitimate thing to do.

D

Yes, that's a good summary. Okay.

H

Right, my question was a little different, which is: uh do you have the ability to make things happen in a challenging way?

H

um You know, I I gather, know right so I'm going to suggest or ask uh you know one of the things that you probably want to do is you know one of the problems here. Right is there's those concurrent stuff and, depending on the order in which things happen that are beyond your control, the the code will react differently and it would be good to be able to make tests that actually, you know, reflect different impositions from the underlying reality. So.

D

You're asking more about injection, not validation, so for injection we have the decade dedicated library that allows us to remotely inject file points or arbitrary. Let's say semi-arbitrary code into HD. So basically say this is a fail point and we want it to you to panic when it hits it.

H

No I'm talking about like injecting timing right if you've got concurrent.

D

Activities we can either sleep or we can okay, that's some coordination.

H

D

Yes, you're talking about higher level or or how to use this framework, we are trying to use the framework, but at city has we already had this framework. We are just trying to grow, it I, don't know if it can cover all the edge cases. I assume like you described, but it should should like I'm, open and like this is open. I can take any example. I mean we can create an issue and grow a list of ideas. I, don't yeah it's it's. It should be.

H

D

Thank you. That's.

H

Good enough for now.

A

Very good book, then.

I

uh Yeah, hello I also wanted to um I'm, also working on NCD I'm, a little bit familiar with this work on the contract, but more familiar with the testing Just Wanna Give A another where to summarize it so I, think the golden standard in this type of testing is jabson test and if you're familiar with Jefferson, IO and I think this is actually taking this uh like a step further for hcd specifically, and the reasons are is first- is there's the Go fail injection library that look what Mark said that allows us to inject failures in more granular way than let's say uh Japan, because Japan is more like a black box.

I

Here we actually can do more and then this framework it allows you basically um mimic traffic, uh which is also great uh and to write tests around that and then on top of it. uh We do Loren's ability, validation uh through this uh through the library uh that is with open source and pretty good and then another thing uh Mark added was the watch validation. So we can. We can look because we're already injection all the failures right we can. We can do more in terms of validation. So it's not just learner's ability.

I

We can check other conditions, make sure that etcd specific Primitives work as they expected and so I just want to point out that you know the contract. I think is just for me. uh How I see it just like a summary of uh what happens in kubernetes, so people can write tests and understand all the scenarios better.

I

um So just uh hopefully this was helpful. Thank you.

A

Very good, if you can lower the hand, um so um anybody else so what's next here um I know David. You need some time to think about this. Maybe you know we bring it up again on the next meeting as a topic sure.

E

The the implications of changing our storage interface are significant, uh and we had previously made a pretty explicit choice to choose to base it on NCD and going back to the old doc about why we made that choice. Thinking about it, deciding whether it's right to change that now and the likely repercussions of that are all worth doing.

A

Do you um could you help me find the link to that old, look and Link it here in the notes for reference.

E

Sure I think it was an old position paper. We wrote.

A

G

Yeah and to that point, I think it would be really important that we, whatever testing happens, isn't like kubernetes development blocking right. So if, if somehow kubernetes changes in a way that works with the current LCD implementation, what the model isn't equipped for it, like I, wouldn't expect the kubernetes developer to have like their tests fail and be told.

G

No, you can't do this because um the model the model doesn't match what you do, even though de facto TD does like we would probably I would I would expect that to show up it's like a downstream like I'd say you can look at this, because maybe it's a problem with the model or um something like that.

E

So it gets complicated because once you give up the local control of your storage.

B

E

You no longer have that as your storage interface, so you couldn't add a new feature, yeah think about something like uh pagination support. You wouldn't be able to have it once you did this and it would be a different.

E

It would be a significant change, so yeah I will find that old uh position paper feta and we should re-read it and figure out whether we think it has aged well or whether it requires updates before we made any change here. We should make changes there.

C

That's great, it sounds fair to me.

A

Very good should we move to the next one. Thank you, American fun for it presenting here. um Okay and you are on the spotlight again.

C

Yeah, this is my other favorite topic, uh so basically, I have been looking at um various implementations of Open Source sort of prescriptions on how to do control, plane, upgrades and uh I've been kind of running. Around I I talked to Sig cluster lifecycle yesterday and um brought this up and they suggested I come here um so here I am uh basically the way that we do. Upgrades with Cube admin and Cappy um it occurs to me is not ideal.

C

So the way that uh the control plane upgrades from these Like official entry well entry for cube admin, um sort of mechanism for upgrading clusters basically involves upgrading a node at a time, and so what ends up happening? Is um you end up with a skew of you know? Api servers, controller manager, scheduler, and you end up with a non-deterministic upgrade sequence which to me doesn't seem ideal and in our official documentation it's actually quite fascinating.

C

um We prescribe that people use either Cube admin or that they do a manual deployment and the thing about the manual deployment is. It seems more correct to me than the way uh Cube admin and Cappy actually do upgrades, which is component by component uh I. Don't necessarily agree with the order of the components like SCD should be basically order in different and um yeah.

C

It's not clear to me that there is in ordering preference between controller manager and scheduler, but um ideally you would want to update web Hooks and then the API server and then either the controller manager or the scheduler um does. Does that make sense.

C

E

Don't know if I plus people is I, think I think what you're describing is that the way cubeadm does this is it goes to one node. It upgrades all the components on that node and now you have node one that has a VN or latest version qvps over the latest version. Controller manager, latest version scheduler.

E

But the position that you.

B

E

Is that the other nodes are at n minus one, meaning that lease acquisition can go back and forth between components uh and that the.

E

That you haven't done a clean layering. You can! Actually you could, if you, if you weren't using a localhost connection, have an old panu Cube controller manager, hit an old API server uh out of.

C

E

For cubeadm, is it wired for localhost, so they always say the same one.

C

I I believe so I'd have to double check, I think so.

B

I don't know if this is a good time to bring it up, but that point they're around to talking to localhost is that it's very much so you had your hand up. Is that a like an actual recommendation that we make that things should talk to local hosts too, because we've definitely seen issues like you described where you end up with new ones from two older ones and whatnot.

E

It's definitely not a recommendation uh now, whether it should be or not is also somewhat contentious. uh I, don't think it should be. A recommendation uh hitting and not ready, qvpi server is potentially problematic. Disallowing on ready connections is also problematic for different reasons, um and you know that every cubelet is not going to use localhost. So there is an advantage to having a single entry point for all track um that allows for consistency.

C

So my question is like what is the prescription from API Machinery on the sequence of upgrades.

E

Let's see, let's see what Moe's hand is first to make sure it wasn't day. I don't understand what this is.

J

uh All I was gonna, say is so I am unless they changed it. I would be surprised if Cube ADM is doing localhost stuff because there have been bugs in the past, like I. Remember when we rolled out token request, it would like the cube. Controller manager would be like. Oh this API server doesn't support token requests, so I'll do the Legacy Behavior and then like it would, because it would restart with, like you know, would get upgraded later.

J

So, like all the orderings were like kind of screwed up but yeah, it was like explicitly upgrading components in a way. That was not what we tell you to do um and having side effects from it.

E

As I remember our recommendation, it is to update the qapi server first before you update any nodes before you update the controller manager before you update the scheduler I agree that SED should be order. Agnostic and I also agree that the key control manager and keep schedule or ordering does not matter uh I, probably probably.

C

Could there's some like some of the controls could feed into scheduler? So that's that's. Why I'm kind of on the fence about it.

E

So I can say that the way openshift does this is we have one component that goes first, that updates basic configuration aspects, gets your crd schemas right. If you need to handle migrations that can migrate, data control paths, it can update web hooks if needed, then SCD and the qvpi server go in order and complete, and then we update the cube control manager and Cube scheduler in parallel and they complete, and then we update everything else.

C

So you guys are basically doing a component component, which makes sense. I think that that is like.

C

I mean it's, it's a it's a rational way to do it. You end up with Better Properties I mean like you, you kind of want to you want to update all of the controller managers at the same time, kind of.

E

um The choice between bringing them all down or rolling them out and the advantage to bringing them all down is that.

A

E

Don't have a flap chance, you don't end.

C

Up, yes, yes, you have, you have a flip-flop chance.

E

um What what we found previously, that our biggest problem was was that for a while, we use localhost and localhost didn't guarantee safety when hit by say a garbage collection controller uh for a not ready, API server and the properties of that were really really bad when it failed. We also found that having a single entry point gave us sort of one thing to watch where, if the controller manager is going to get there, we knew our cable it's going to get.

F

E

So it was just easier to watch for that, so.

C

Interestingly, uvip probably solves that problem about localhost and the garbage controller.

E

uh I, don't I feel a little silly. What's use it? That's.

G

That's that cap we mentioned I was about to bring this up too.

H

That's what happens.

G

E

C

E

Probably does improve that property quite a bit yeah.

C

Yeah so um so that becomes less of a concern, but uh so then, given the fact that it seems like API and Machinery stance on this is to go component by component as opposed to node by node uh shouldn't. We reflect that in the upgrade tooling, that we provide people.

J

Those API Machinery on cubeadm.

C

No, no I think it would be cluster lifecycle, but they agree that component by component makes sense, but they felt like the call belonged to API machinery.

C

Of the correct sequence, and so once once I finished here, I will go back and I will talk to them about and what to do with the Cub idiom.

E

I think writing down. The properties is, is valuable, um deciding about whether to bring all the key control of Interest down and then bring only new ones back up. That would be an interesting change. I, don't think we make a recommendation on that one one way or another, uh don't know whether we should don't know whether I have an opinion on that I.

F

B

E

Them to make a change, I think it's worth writing down both possibilities and explaining why we think one's better than the other, but.

C

I mean there's, there's a lot of there's a lot of different ways. We could go about doing stuff like this I I mean like I'm, not prescribing a specific upgrade mechanism. uh I just want to be a concrete.

C

The prescription.

G

Yeah I I think there's two parts to this: there's. What do we have to support um for the ecosystem, given that people are doing this different ways um and then like? What are we actually recommending and and like what is the fine print about that like what are because like if you're talking to local posts, there are definitely Corner cases that come up um if a controller manager flaps that causes potentially certain things to happen?

G

um But I like the idea of like going through and updating this, especially after uvip plans like I, don't think SCD should be in this list that we have in the manual deployments. Yes, it doesn't I think with uvip in place. You can pretty much say like um you should start updating your Kube API servers and then like everything else, should go in parallel, and that should be fine, because they'll talk to they'll talk to the newest versions of things. They'll be able to see everything.

G

The garbage controllers will be able to see the old stuff and clear it out.

G

But yeah we should do an update on I, don't know, I, don't know how how strongly we should recommend to other people what they need to do. I think I actually think it's maybe more of a benefit to fit to people to make it clear um what our constraints are, what we support and like what we think is a pretty good like um sane way to do it in a.

C

Nature mode should we even be prescribing Cube EDM in its current form, because I'm not sure if that actually makes sense to me.

G

I would have to look at it really carefully to see if we could reckon if there's something, that's really problematic, I I think we either have to reckon like realize that that's happening and make sure that we support it or we just have to like recommend a major change but I'm kind of I'm kind of in the situation where I think like there's a lot of stuff going on in the ecosystem, just kind of have a contract to support n minus one skew of all our components. Oh.

E

I think I have one slightly stronger opinion. Joe I think that that, if the configuration of a deployment or of a particular cluster installation is such that a new queue, controller manager or a new scheduler could hit or a new cubelet could hit an old Cube, API server.

E

I think that cluster topology is broken and and I think that's a pretty clear statement. I, don't know where the qadm is doing it not confident one way or another uh I think that would be one where we would say. That's a pretty serious bug. It needs to get addressed. One way to address.

E

It is to be sure that all qapi servers get updated first, and that is what I think given that would be my preference for what we would direct people to do if they make sure that if a particular cluster makes sure that a cube controller manager keeps schedule of our cubelet never hit an old qapi server, but.

B

Don't think it's an urgent.

F

E

To make a change pending pending some other failure, but I'm not anticipating on it.

C

That is a fascinating, very strong statement, so wait so you're, saying you're, saying it is actually incorrect for KCM to talk to an API server n minus one.

E

I think that it is I'm not confident that we guarantee that functions as desired.

C

So I I like that I actually like.

B

That a newer controller manager is not allowed to talk to an older API server, as per our versioning things, I've coached this with Jordan um language I. Like that run into problems, we run into problems, um doing exactly that on some of our internal classes and it's yeah definitely not supported. You need to upgrade the API server for any client.

E

I'm glad it's not me falling short on reviews when I review I I review it for um for compatibility with its current Cube API server, not for compatibility with the previous one.

J

Yeah and- and we have checks and like controller manager, that make that assumption like implicitly like hey, is this API and Discovery? Thus, I can use it well that that wouldn't make sense, because if you made that check once on Startup and then the API server changes beneath you, you're kind of screwed.

E

We've also made use of new CSR fields, for instance in CSR controllers, in the same release that were introduced based on the Assumption qvpi server always goes first, I think the only thing ahead of qpi server is hooks. If you need a hook that needs to get updated for compatibility with a new type, then that has to go first.

C

If you like, that statement should actually be in our upgrade documentation like that.

E

Statement if Daniel and I failed to put it there, then he.

C

B

E

Is the only one remaining uh sorry I mean Joe's here too, but he and I I thought many years ago wrote that uh if we mess that up, let me know Han.

C

uh It's that one I, don't I, don't believe. That's that's written in our upgrade documentation. Awesome, okay, yeah.

G

Yeah hon I also love just the other cleanup ideas you had like make sure it's clear that FCD is independent like for external things like web hooks just say, say if they're, independent or say what the flow is um we should just so. People know like what to expect I think is really handy.

E

I agree I like those as well, but uh but in terms of like that boundary I think it is wrong if you, if you can have a new Cube controller manager, talk to an old server needs to be updated.

C

I mean the the way that stuff is actually written in our upgrade. Documentation would actually kind of suggests that we should be able to tolerate n minus one, which is interesting.

E

We can tolerate a old Cube controller manager talking to a new API server, so, but only in that direction, not in the other direction as similar to we can tolerate old cubelets against new API servers, but we cannot uh tolerate new tubelets against old apis, a person.

B

On that no I don't know if we document specifically, how meant the version SKU supported between Cube controller manager and the API itself. I know we do with cubler I've assumed it's n minus one for control manager.

J

um I don't know if we make that clear.

B

E

You are corrected as n minus one uh I'm, pretty sure we wrote it down. Okay, uh here we go, it is in the SKU policy.

F

Yeah I think this SKU policy covers this already.

C

Interesting cool. Well then, our implementation entry implementation is at odds with our skew policy.

F

Yeah yeah all the components in this SKU policy say they must not be newer than the cube API server.

C

We should we should actually fix our upgrade documentation and.

F

C

Which is basically violating this property and Cappy I think which also violates this property. No.

E

It is possible to go node by node and ensure that they do not talk to an API server. That is not local to that node.

E

If that's what they do,.

C

Well, if, if you're not communicating on localhost- and you are going through a proxy, then then.

E

You're at risk, yes, yeah exactly.

C

C

And- and someone just mentioned that the localhost thing was likely removed from Cuba, so that means the implication of that is that that risk does exist in qbdm, upgrade sequence.

E

It's certainly worth checking yeah.

C

All right well, thank you.

A

Okay, so we wrap this topic up for now. Thank you. um I know. Alex has been waiting eagerly, so Alex you're, trying to shine awesome.

K

Thanks very uh I would like to introduce a cap today. um If you can go to files changed and then like just view the readme, so we can see the rendered markdown.

K

Can you hit the three dots in the top right and hit view file.

F

A

K

Cool so yeah I'm hoping to get the Sig to agree that this is a worthwhile thing to do, and some feedback on the proposed solution and hopefully have an alpha for 128.. um So yeah this cup solves uh two main issues. The first would be that it's difficult for users to add or change value valuations in their crd schemas.

K

um If they do so without incrementing their storage version, their controllers will break while trying to update crds that are are CRS that are now invalid from the new validations or if they bump their storage version. I just think that's just a huge cost to crd authors, so they probably choose not to do that. So what this would mean is that crds are kind of in stasis. If someone wants to change a validation, they don't.

K

If you are making a new crd and you think of you're thinking about adding a validations, you should also be thinking about how you can't change them in the future. So you should not add the validations in the first place, so this situation kind of just deters people from using this feature.

K

um The second reason is that, on the kubernetes API server side, we can't change how existing validations are handled or fixed bugs, for example, with string formats, because it would break user workflows.

K

um So can we scroll down to the um a small example here? So this is an example of like just to illustrate what happens to the user when they change a value, validation and their crd, so assume they have that first schema with. You know two fields that are strings, my field and my other field, and they create a CR that includes an empty string for my field.

K

Now they change my field to give it a minimum length of two and if they try to update the CR to add my other field without changing anything else, to get an error. This also. This also would happen to controllers, while updating the status field and even more sadly, users of server side apply, who submit a patch with just the single field that doesn't relate to the one that is failing, but we'll still get that same error.

K

um So this cap proposes a feature to try to avoid this by Pro doing a post-processing step on the schema validation errors. Can you scroll down to design details um we'll just do a post processing step on the schema validation errors find all the fields that have errors associated with them and see if they changed from old to new? If they did not change from old to new, then we will convert the error into a warning and if they did, if the value has changed, that has an error Associated. We continue to throw the error.

K

um There are some implementation complexities with making sure that this is a reliable list of field paths and that we can associate the old and new values, but in general, I think it's a feasible plan and I would like to hear the six thoughts on this.

E

Would this be supported with cell Expressions as well.

K

um As they currently are, yes, cell Expressions, the fields that you can refer to are limited to the field that you attach the expression to so the field path. That's reported by the error is going to be a strict superset of all the fields referred to by the expression. So we can do that comparison. There is a proposal to allow the user to set a custom field path to report the error on for the use just for the user user experience take.

K

We will have to consider that, while implementing this to make sure that we get like the original like root field path that contains all the Express Fields.

E

Very glad to hear that.

G

Yeah that was an interesting Corner case of this one, but I think I think we can work out the details on that I've been tracking this, so I'm I'm generally in favor of it I mean the main. The main user visible change will be that um if you do break your crd schema, um you will still be able to change things that don't affect the fields that you broke, but um I guess.

G

The only downside is that you might not notice what you broke quite as quick, but I still think you would notice it pretty quick, and if you were doing any testing you should be able to detect it and if you are using any tools to detect that you're breaking your schema, you wouldn't hit that in the first place.

G

So I I think it's I think it's pretty helpful.

E

Would would there be a mechanism that someone could use to get the old Behavior if they wanted to.

K

um Right now that would be limited to turning the feature flag off, but if that's something people want inside of the API server options, I'm sure we can add that in.

G

E

Worth debating um the feature, gate is going to go away and it is not common for us to.

E

Change Behavior without a mechanism to use the old Behavior we had so.

H

I think it's worth.

E

Considering being debated on the cap, but overall I'm in favor of this glad to see ratcheting validation go in.

A

Okay, so what's the next steps, so of course you want people to review the Escape. Are you targeting the 128.

K

Yeah I would like to have a basic Alpha in for 128., so if people can review it and for this version, that'd be good great.

G

That's good thanks, Alex.

A

Thank you uh and I think we have one more from beans that is requesting attention to this PR.

A

uh Okay, so probably this is for Joe David me. We can not do it in the meeting. um We can take it offline unless there is something I am missing.

A

F

A

G

Yeah just look at.

A

It pay attention to this. Okay, very good, we'll do these things when you see this recording. um Okay, that's all anything else that I missed. um It was a good meeting. I will upload the recording uh before the end of day today, send it to the channel put in the document um very good. Thank you, everybody for coming and we'll see you next time.

A

C

F

K