Kubernetes API Machinery Special Interest Group, 22 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20230322

Description

- deads2k: Deprecate and slate StorageVersionHash for removal? https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/2342-exposing-hashed-storage-versions-via-the-discovery-API
The StorageVersion API looks like a better footing for the feature and has traction: https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/2339-storageversion-api-for-ha-api-servers
- liggitt marking it deprecated is a great first step.
In general there was agreement that deprecating this one (in favor of the 2339) is the right way to move forward.
AI: deads@ will send PR to mark it deprecated, targeting 1.28.
- lavalamp follow-up from feature gate discussion 2 weeks ago: draft KEP for initial review
Daniel asks for feedback, especially around the User Stories to make sure nothing was missed.
-lavalamp follow-up from mixed apiserver version discussion last week: extremely draft KEP
logicalhan Two open questions:
where to store the public key for the apiserver
What approach to take to make discovery consistent
AI Daniel will include an "unresolved" section with details to bring up in future meetings and see if we can get an agreement on the way forward.
deads points out to KEP dependency chain: API Server Identity — Storage Migrator — this one
-jpbetz: Want to raise awareness of De-share InitContainer type from Container to SIG

A

A

For kubernetes Open Source by weekly meeting today is March 22nd 2023.. We have a nice agenda and we are going to get to it. um Let's begin with David.

B

Thanks so I wanted to talk about the storage version, hash, uh something we haven't talked about in a very long time. This is different than the storage version. Api. The storage version hash was a first attempt that we added to our standard resource discovery to uh to try to have something to key an automated storage version migration off of uh over time.

B

We found it just didn't work well because we had to understand the state of all the API servers and there's no guarantee that a client is going to be able to make those connections and do that I. Think the storage version is a much better place to continue from and got some traction from.

B

uh Wow names names are escaping me was it was Mo and was it Han.

C

Andrew cycle: that's.

B

Who it was Andrew uh so I'm proposing that we deprecate the uh storage version, hash field in the old Discovery document and plan for its removal in another three releases.

C

That sounds good to me. I, don't think anybody can be making use of it uh at the moment since you can't reliably connect to a particular API server or even know how many there are.

B

Yeah, um that's my thought. I want to bring it up here, also because I know Jordan never wanted to remove another GA API and technically this one was beta, but there was no way to really know that it was beta because we've been returning it by default to users.

B

um I agree that I, don't think, there's a reliable way to use it, but I figured I would be sure it wasn't going to give Jordan heartburn.

B

B

That looks like an on your head, be it uh look to me um I.

D

I think I'm gonna go ahead. I'm.

B

Definitely gonna deprecate. It then, or we'll definitely deprecate.

D

It at least at least marking it documenting it. That seems like a really good first step to my knowledge, there's only one consumer and it's our storage migrator thing that has never like progressed to support the state um yeah, okay, at least documenting, announcing trying to doing a sweep to find usage and saying hey. This probably isn't working the way you think it's working. Are you sure you want to be doing this? It seems like a good first step if that sweep turns up as little use as we expect then yeah letting it expire.

D

It's probably fine. Okay, great.

A

Okay, anything else, anybody else has an opinion on these.

A

No okay! uh So, like there's an action item for you David, uh you are going to Market deprecate um right away. Are you going to wait until somebody has a different opinion or.

B

I will go ahead and Mark it deprecated I. Think that's fairly non-controversial uh I will probably just snooze a reminder to come back. I, don't know somewhere during the summer and.

C

Is that for uh overall is that for 128, or are you going to try and pretend that it happened in 127, yeah.

B

No 128., okay, I, see no reason to rush it unless yeah yeah I just I happen to come across it. I was looking at it going oh I wish I didn't, have that to deal with yeah and want to remove it. Okay, okay sounds.

A

Good, very good should we.

B

C

To the next one then sure um the next two items are uh mostly just informing everybody that I wrote up caps for things that we talked about uh for the last two meetings, um I'm trying and failing to get over a cold, so I, don't I'm, not firing on all cylinders. So I don't know how much discussion uh we can really have, especially since I.

C

Don't know how many people uh read these, but um uh for this cap in particular, I would appreciate if people would comment on the stories uh and um I tried to call out a few different feature flags and how they would work with the system. So if you could point out ones that I missed or like the classes of feature flags that I missed that'd be that'd, be uh that'd, be great, especially David brought up an auth one. That I could not find that feature. Flags. Is that offsetting like now, GA and always on.

B

One that was I'm also worried about the cubelet settings. Where you make a change to enable say, swap on a cubelet, then you have a flag to enable swap on a cubelet, and then you try to disable the feature date and what should the cubelet do so.

C

Yeah yeah! Well, if you, if you read the kept now um I think that concern is addressed basically I say like some Flags aren't a good match for Central enablement and for those flags we won't use that system.

C

Are you interested okay, I'm interested in cheating to get to uh get some useful stuff on here.

B

I: okay, okay: there will be follow-up questions, I, think about like trying.

A

To separate trying to.

B

Separate the two in some reasonable way: um yeah.

D

Even having a coherent way to describe that, like things that we expect to be Global and things that we actually want to be able to roll out gradually to a cluster, so like swap support on nodes, maybe seems like a thing we would want to do per nude pool or gradually, where.

C

D

C

I left canarying for, like a follow-up at the moment like like uh I've got a I can't boil the ocean so um uh seems like canarying is a follow-on activity. Once you have something that could be theoretically centralized, um so I haven't described how that would work yet.

B

Okay, you have several caps open, a lot of them are really cool. uh You have this one I see there's another one on the list for uh a known version: interoperability, proxy. That looks interesting. I know you have one for and.

C

I have the fine grain Aussies yeah, it seems like I, like writing cups, but not uh finishing them.

B

Well, my question for you is: if I were gonna focus on I mean let's say two for review for potential implementation of 128, which two would they be.

C

um Probably the the two here, the the finishing the feature flags and the um uh the proxy one um I think it's my turn on the fine grain Aussie one uh Tim has left feedback. um I will yeah I, definitely overpass on that one. um So I don't know when I, if I can make a promise or not, but uh okay.

A

C

And uh I'll just go on to the to the second one, um which is the thing that we talked about last week. I wrote something up, there's a few open questions and it is not detailed at all yet so, um if you go there and be disappointed, I warned you, um uh but in in particular uh we're looking for an uh an idea for like uh do.

C

We need like a public key or assert or a CA cert, or something so that API servers can identify each other, um and if so, where do we put that?

C

um So if anyone has ideas on that, if you could leave a comment that would be great uh and the other thing Han and I were debating is uh given that resources can be added and removed.

C

Do we want to merge all possible resources into the discovery information, or do we want to show only the discovery from like one version of API server uh say that either and if so, do we want to use the new one or the destination version?

C

If that happens, to be a downgrade destination or or uh the alternative is to hide resources that not all API servers can serve, which I think Tim left in a comment there, um I I currently think that anything that isn't showing all possible resources will result in namespace deletion bugs um which means that we have to merge things which isn't actually as bad as it sounds.

C

I think we can just read the storage version API and uh construct the discovery information from that, instead of having to do something complicated like dial each API server or something like that, and today, we're currently in beta of the aggregated discovery stuff is in beta, and that is reading. uh The look I think that sends requests to the local API server. So um if we're reading the storage version, API versus sending Discovery requests to the local API server, I, don't think that's like more work for the machine. uh It's just different work.

B

Does this explore the possibility of having Discovery show all possible resources, but have individual API servers refuse to allow create unless all API servers allow creating or is? Is the proxy just demanded a way to avoid that.

C

um I think that is.

E

Proxy would allow you to avoid that situation. I think.

E

C

E

Just run up the traffic.

C

That could be something that we could consider uh in addition, though, uh I haven't I, didn't think about that at all um yeah, so I don't know give this a it's real short right now, so give it a read and see if there's a alternative.

C

There's a lot of to-do's.

E

So the thing that I was suggesting that Daniel does not like is um in order to make Discovery consistent I was uh suggesting that we just route Discovery to the Discover, the the destination version, um and uh while we would still orphan objects, um as Daniel pointed out, uh this is already an issue.

C

Yeah the the orphaning case, by the way it happens if you're deleting a namespace and doing a version transition at the same time and the namespace lifecycle controller observes a set of resources that doesn't include a set of resources that one API server can serve, but there's actually objects of that type in that namespace it could allow the namespace to be deleted and then, when you undo your version transformation and recreate a namespace of the same name, you have these zombie objects come back. So that's the and I believe this can happen today.

C

So it's not like a new failure mode, but it is either closely related to the problem. The steps seeks to solve- or it is the problem that's like the cap- is trying to solve. So that's why it seems good to me to uh not let that happen. That.

E

Was actually the thing that was described in the slack Channel as being the problem.

C

uh The no the slack Channel.

E

Discussion was was none about orphaned, orphaned objects.

C

No, the slack Channel discussion that triggered the conversation last week was that the namespace cycle controller like went crazy and did a bunch of list requests in the situation right right.

E

um Which we will avoid and we will solve that problem, we'll.

C

We'll solve that problem, but making it not do lists will also uh leaving zombie objects behind that occupy space and NCD and could come back to life in the future. uh That doesn't seem.

D

Right here, uh I think it might make it worse like there are cases today that, like fail safe by the name, space controller, saying like I'm, getting errors and so I know, I can't do things, and if we routed Discovery to the destination server and routed requests for apis, we didn't know about the destination server. Then that would allow the namespace controller to succeed and think it had cleaned up new cases where today it would hit an error and I guess, go crazy or back off, but in any case not delete the namespace.

E

Well, eventually, the upgrade will succeed and you will be in that state anyway.

B

D

Appreciate it, there are cases today that are broken. It would be nice if, instead of exchanging one set of breaking cases for another set of breaking cases, we could eliminate the.

E

Breaking cases, but no is that exchanging breaking cases, this breaking case already exists, and so we're not there's no trade. This is just I'm, arguing that it's personally out of scope uh that.

C

uh It would be a it would be a requested. It would be a highly desired uh immediate follow-up, so.

D

um Yeah, like we recognized that we can get inconsistent Discovery discovery that is inconsistent with what some servers are serving and true, rather than trading one inconsistent Discovery for another inconsistent discovery.

C

So another I should mention is the in the the previous cap. uh If we offered a hook for people over version transitions in regards to a specific feature, um you could let people like if a resource is going away, clear, the entire collection uh before letting the uh upgrade finish. That would also solve this problem um and more thoroughly I think uh upgrade.

E

Hooks yeah I mean, but that that's almost an that's basically a orthogonal feature.

C

uh I mean I: we've got to solve the problem somehow uh if we solve it with the other cap that I guess that's fine with me. um If we solve it in this cap, that's also fine. If we solve them both places, that's Overkill, but I'm. Okay, with that I think.

D

Yeah I I've definitely wanted a place to put like I, want to turn off this API before I do or upgrade or stop serving it or whatever. I need to clean stuff up like yeah.

E

D

The destination version no longer serving things. The older version had would become a non-issue which seems like a nicer way to handle that yeah.

E

I I agree, I, agree, I I think maybe the feature flag, Dynamic feature selecting is the place to solve it.

C

All right so I think what should happen here is all um leave an unresolved section in the cap and like like write at least a sketch of how it could go either way and uh when this is closer to being completed. Like maybe next time we meet with the Sig can help make a decision.

B

um I, haven't I, haven't read yet. Is this considered an optional thing? I remember there were concerns before about whether API servers even have visibility to each other. Yes,.

E

It would be optional.

C

Yeah, the idea is, um if you have network uh connectivity between API servers, then you get the proxying. If you don't, then you get 503s or whatever, whatever error is safest. Okay,.

B

A 503 and that can be based on something like storage version. Coupling to storage version means this would not be able to go ga until storage version when GA correct I.

C

B

Okay, have you so that chain also goes back to APS server identity? Yes, which means that your dependency chain on this cap is API server, identity, storage version and.

F

B

I'm not against it, I just want to make sure we agree on yeah. That's what that means. Okay, yep.

C

Yep yeah. This is why uh this kind of thing is why I really wanted the API server identity. In fact, both of both of the both of the Caps here depend on API server, identity.

G

As a reminder, all the code for that is still semi-broken for anything but aggregated API servers or it only works for cube, API server. It's completely wrong for aggregated API servers.

C

D

Been in master I thought we fixed it in master, no.

G

Because you told me not to merge it and then David and everybody else asked me to do a bunch more work, so I'd just die having it.

D

Wait what I only looked at the backboard I was like. Why are we backboarding this, but I thought it was 60.

G

No, no, no, you came over and like Mo. Wouldn't this make this technically slightly worse, so I'm like yes, you were right, Jordan, but yeah. It.

D

Doesn't sound like me.

G

Yes, it does but yeah the the gist is no not none of my. My PRS are still open, they're hanging out because there's like a 10 item, long to-do list now that I have to go address um to fix them.

C

Okay, maybe I can help her find somebody to help.

C

um Yeah so I guess.

C

It'd be good to get if possible. Like my in my dream world, we could agree on at least one of these caps for uh 128. um yeah. So take a read and I guess we'll talk about them again next week, okay,.

C

All right and that's all for me- uh should we there was at least one item on the backlog uh um is Joe here. Yeah.

A

Joey's here yeah.

F

A

Do you want to talk yeah.

F

um This was really short. Actually so, um and so it's it happened a while, because people might already be aware of it, but I I thought it was interesting to know, and so I was just going to mention it to the sick. So they're during um the development of um the the cap for sidecars by Sig node. There was a topic of uh how we should structure some types.

F

um So um there's three kinds of container types: there are um there's the normal containers, the main containers there are the ephemeral containers and then there's being introduced, and then there's the init containers and um the init containers are gonna, be used for sidecars. That's not actually really important here, but what was important is that ephemeral containers have their own go type and the main containers have their own go type, but init containers share the same, go type as the main containers.

F

So when, when the Sig wanted to go to add a new field to just to knit containers to support sidecars, they also were changing the same ghost struct. That is used for the main containers.

F

um And so there was some discussion. Should we introduce a new go type and split it in half, so that now a knit containers have their own containers type in its own field? And but there was a lot of Downstream implications to that, because the go types aren't just used for the server implementation, they're also used for the clients, and so that would technically be a breaking change, because you're switching a type and that can't be completely hidden from your clients.

F

So the ultimate decision was to continue to use the same type and not break Downstream, but I thought it was interesting because it kind of showed that, like the way that we don't insulate clients from the way that we use our server types is having implications on. You know our ability to make changes to how we how we build um how we build Types on the server side. That's it I just wanted to I just wanted to tell people that, so they were aware of this case. I thought it was interesting.

D

And one other place I can think of where we had initially similar types and then ended up. Splitting them was in uh web hooks I think webhook configurations we used to have validating and mutating web hook configurations share the same structure type for their like lists of web hooks, but then there was something that we wanted to add to mutating I forget what.

B

D

B

Was the call me again, that's.

D

Right, that's right, uh reinvoke, if necessary or whatever um that didn't make any sense for validation, and so at that point we did split the types yeah.

C

D

Building, forbidding it and validating one yeah.

C

The thing is, there's there's compared to pod container, there's.

D

Very few uh users of if I mean even even the.

C

Interesting web.

D

Hook, configurations there's basically one admission: plugin that looks at that field versus containers, which is like hundreds of references or lots of users of a utility method which iterates over all the various types and then calls a visitor so um yeah, it's not exactly the same, but I think. If we had been thinking ahead, we would have realized. Maybe validating mutating workbooks won't always be identical. Maybe there are things that are in mutating that won't apply to validating, or vice versa, and we would have tweaked that ahead of time.

C

No I think you had a hand up. Did you want to say something.

G

uh I was thinking of an idea, I don't know if it makes any sense. Could we can we try to catch ourselves if we do anything crazy here by having like just on the server side, some like fancy conversion logic, which, like makes sure that, when you convert from the outside to the internal representation, it doesn't set fields on container types that we don't expect. So that way, if you add fields to like init containers, even though it's using the same go type when it translates internally, there would be checks.

G

Saying well can't set this field for this yeah.

C

Yeah, so it would be completely possible to make the change in the internal type and not make the change in the external type. But the point of that the thing that people wanted to do was make the change in the external type right.

G

Right, I, yeah I was just trying to come up with something that made it so that at least you would catch bugs or something yeah.

F

G

Some Oddity of changing the external type by adding a new field that didn't make sense through a knit container, but did make sense for container or vice versa. Yeah.

C

Yeah, unfortunately, it's like the same. The external types that we publish are the same external types that client go uses so like you can't uh expose that change only to people who want it.

F

Yeah, ironically, clients non-go clients that have to generate themselves from like open API. Typically, don't have this problem they're fully insulated from our internal types, and you could you could split a type and they would just see the open, API change for one part of this API and not the other, and they would be totally fine.

C

Right um thanks, Joe, the last item in the agenda is a PR and I: don't see Eric on the call list, uh so why don't we just leave it on the backlog? I have I and others have left comments on this PR, so um uh I think. If we're going to talk about it here, we should wait until uh Eric shows up um if we didn't answer his questions already.

C

So let's go yeah very.

A

Short meaning yeah amazing. Well, thank you so much everybody for joining and I will upload the recording later and I hope you have a good Wednesday a good week and we'll see you again in two weeks.

F

Take care thanks, bye.

F