Kubernetes API Machinery Special Interest Group, 5 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20221005

Description

* [mo, anish] Discuss about KEP status and blockers for moving these to beta in v1.27? In regards to sig-auth/3299-kms-v2-improvements
**2339-storageversion-api-for-ha-api-servers
**1965-kube-apiserver-identity
* [lavalamp] quick followup from last time: status of logical clock vs RV
* [lavalamp] quick followup from last time: status of subresources vs fine grained permissions (follow-up doc)
* [fedebongio] KEP List for 1.26? Tracking Board for 1.26 & All Open KEPs with SIG API Machinery label
**Allow informers for getting a stream of data instead of chunking #3157
**CEL for Admission Control #3488
**Aggregated Discovery #3352

A

To three recording very good hello good morning good evening, depending where you are, this- is ckpa Machinery by weekly meeting for kubernetes open source. My name is Federico and Giovanni I'm co-chair of ckba Machinery. uh Luckily, in the call we have David eats my culture and technique of the Sig and Daniel also another technique opposite, and we are going to cover a number of interesting topics today.

A

um So why don't we get started uh practically more I read the first topic, so I will give them. You know privilege team to start covering that one. So all we do.

B

uh Yeah, do you mind opening some of the links, um so I can give some context so for the canvas it would be to cap. uh We want to have um the capability of being able to have automatic storage migration of of data as the underlying Keys change.

B

um That problem set is very similar or, in the sense, exactly the same as what um the storage version API tries to solve for um encoder version, so like schemas of objects uh and I suspect is probably the only way to solve it in kubernetes, based on what I've started to think about and gone through all the bits, but so those the API server identity, cap and the storage version kept. Are you know pretty tightly linked and have not been updated in orders or two to three years, depending on how you gonna count?

B

So what I wanted to kind of get a sense? There's like two high-level question height one is on these caps. um Were you guys happy with sort of the apis that you came up with in the code and sort of the approach, or was there like something like more dramatic? That needs to be actually changed about? The actual implementation that exists within kubernetes API today or is it just like hey, add some tests and you know like make sure it actually does what it does kind of deal.

B

That's like my first uh thought around that and the other thought is um um there's a lore in the canvas V2 cab. There's a small diff around uh proposed change to the storage version. Api to add some extra fields, um I wanted to kind of get a sense where folks thought about that, and if that would be kind of okay.

C

um So one thing I can definitely say is the the API server identity and storage version caps have kind of stalled out due to the contributors going off and doing other things. So uh I think it's not that we don't want them. We just don't. Have somebody drive them at the moment? I think the API server identity cap is actually pretty important and is blocking a lot of things uh and there's a bunch of people working around it. So I I think that's important.

D

I'm trying to remember where we left the storage version- I, don't remember it hitting a blocker implementation was I.

D

I, don't remember what happened to PRS, for it looks like I think I found APR from a while back that didn't merge. That I had reviewed comments on I'm trying to see it past me said this can't be fixed yeah.

C

Is that is that from Highway it is yeah you switch teams. uh I was been working on some other things, uh so yeah I, don't think, there's anything wrong with it yeah. It looks like.

D

My comment says that there was a simplification that I wanted before it merged, but that there wasn't any reason it couldn't work. So, uh let's see past me said we simplified with a flow where we yeah create storage, allow read-only requests mutating after we figure out the storage version, so.

B

um So, looking at this title of this PR, it looks like uh maybe the storage version stuff isn't wired up for open resources, yeah or it doesn't work, maybe for custom resources.

D

um Yeah, if I'm, remembering correctly, the problem, was a race on Startup to be sure that you knew what version you had and it was accurate before you started storing data right. So so part of this. If someone is relying on this for something like storage version migration, they have to know that the value that is present is actually being used and you haven't used something else to store instead uh and that's what I believe this PR was doing.

B

Okay, that makes sense um I. Is it safe to assume that, like the aggregator and aggregated API, servers are already correct, because the cube API server code is correct.

D

I, don't remember.

C

The advantage right do people who write aggregated API servers ever update the code that they are importing yeah, usually.

D

Every release- well, sorry, I guess it depends on who owns that particular one yeah.

B

So if it's David or me and yes, we do okay, I suspect.

C

That not everybody does uh is the quite that on top of it, it can't save everyone.

B

uh So I agree with David on this one.

D

um So it's easier for those, because their storage versions are pinned in the code. I, don't remember what we I. Don't remember the state, but I do know that it's an easier problem, because they're pinned in the code as opposed to mutable in an API definition.

B

Right, yeah, that makes sense um we're on the API server identity stuff. Were you guys happy with just like reusing lease objects and just giving them a label to identify them as belonging to API servers, so that sort of kind.

C

Of uh I thought that we had at least that much emerged in the code already no.

B

No, what I was asking was is that approach that exists in the code, uh as you just said, Daniel, is that sufficient, like you, don't want some other completely distinctly different API to represent API server identities.

C

C

As this was, this was what the Sig topic was about. I, don't recall, exactly um I think that whatever we emerged was on the way to the thing that we wanted, but I can't uh promise that without doing some homework.

D

It has been a while, but my memory, and you you probably looked at it more recently than I have uh is that is that we actually did say, reuse the existing leases. We we put them in a I, don't think we put them in default. We put them in some namespace where people was supposed to be our namespace Cube.

B

D

Sounds like a thing I would have suggested um and and then probably tried to copy the cubelet heartbeat code.

E

B

Yeah, so it it reuses the same lease controller. It just has a extra closure passed in that mutates the lease to have the label. Okay, it was like a custom garbage collecting controller, that's running in the API server for cleaning up those leases.

D

I, don't I don't remember being unhappy with that yeah.

C

That that sounds that sounds about right.

E

C

Don't know if that's like so like that sounds like the way the API server would figure out uh who all the API servers are it's not necessarily the way we would want to present that to users of API server. uh I can't promise that, without looking at how easy that is to consume, and if there's like other like like likely API servers need to communicate to each other what version they are, what what their feature flags are. Etc and I. Don't think that that is going to be captured in the lease thing.

C

I think the least thing is just to get the uh like identities of who is in the system right.

D

For use elsewhere right.

B

And so storage, so storage version does use that same identity um and I think that by all seems conceptually right and seems fine, um uh Tim IC, you had your hand up. Did you want to go.

E

B

C

B

E

The same thing that like having that um it, this has come up and um the context of a storage version migrator um that having the uh hi server versions would be important to that. um So I don't know. If that's you know a separate object or how that would end up looking, but.

B

Yeah, so storage version is its own API and so like that, I think sort of leads me into my uh other question um Theta. Could you bring up the KMS, the two cup and kind of scroll down until you see a get diff which one the first one you're that one right there and just kind of scroll down? um Yes,.

A

B

Right there, um so this is sort of like ignore the exact specifics, but this is the rough. A proposal I had of like hey I'm, just going to add fields to the storage version. Api that tell you like each API server would say Hey. You know this is my encryption State and then this is. We have a common coding version we would you could have a common um competition is.

B

Is that sort of like agreeable to y'all is that? Is that fair or like would y'all want me to go, build like a completely separate API? That's reusing the same information, our group resource.

D

It looks this looks related to me.

C

I mean I, think it's I think it's relevant uh foreign I guess my my complaint with uh this stuff was always like the implementation and how uh limiting it was from scalability perspective, um not with the API type things.

B

E

C

Is there I'm not so I so uh I'm? It seems reasonable to do something I'm not sure uh like if we're adding, if we're adding something to this, uh that we need to be kind of clear about what it is and how, uh like the literal, key ID, is that really something that we should share is a hash of the or hmac of the keys, something better to share uh I I.

C

Think I would rather have it be more opaque than less opaque uh if possible um and like obviously, it's optional and clients need to be able to survive without it being there.

C

uh I'm I think I'm more worried about who's setting it and how then then modifying the uh API version.

B

Yeah, so that reminds me of another question: I had so like right now, this object is managed um inside the API servers with the storage version manager obstruct, which has some really specific logic about like The Ordering of this stuff and like deduping, like cohabitating resources and just upwards are really gnarly stuff about resource semantics.

B

um Would it be okay to just enhance that to like take some I, don't know something that helps it understand the encryption related fields hashes or whatever? Or would you want me to like copy a bunch of code and make a different control.

D

I think it would depend on where we know these keys so.

B

I I, like I, think that's a.

D

Code level decision in my mind, but um I, see this API as representing. These are the things that the API server knows about the data that it is storing on a per resource basis and yeah.

C

uh So like, if, if it's a external process that knows what the keys are, I would expect that to send its own, like uh patch or server-side, apply uh request to modify these uh and the API server change would be like permitting an external owner of this field, uh and if it is known in process.

B

C

It's completely known in process to the API server, even for crds, yes, okay, then wiring it, along with the other, like I'd, expect it to get wired along with the other things that we uh wire uh from like what? What part of the code knows? It is the etcd storage layer, or is it the uh like uh command line, resource enablement, stuff.

B

Yeah, so it's the LCD storage layer um and it isn't funneled out yet because there's no user of it outside of that layer, um but it's it is. It's meant to be dynamic because they're, a plug-in the KMS plugin is allowed to change it at any time. So it's already sort of fully set up to be dynamic, um so it would probably end up looking like some interface um function or something that tells you the current information so.

C

If it changes, how does API server know like what part of AP, because I think this stuff doesn't basically doesn't change, except for the crd things and I?

C

Don't think API server does anything other than like make sure the object is there, but if you've got some part of it, that can change on some basis, then we have to add a mechanism that can like propagate that change and uh if we're adding a mechanism that propagates that change, it's not obvious to me if we should like make the current code also propagate changes or if it maybe it's just easier to have a loop inside like the NCD storage layer, which then sends a patch request or whatever. When something changes um like.

C

Were you expecting API server to like pull this thing or is it watchable or like how, however,.

B

Yeah yeah, so the API server um learns. This information from.

B

Basically, every time there's a right it there, it has a chance of updating its cash during that right process.

B

um So it's kind of like that. So it is pulled because there's no watch on it or because it's I mean I guess we could have built a watch for the status API for PMS, but it's just a pulled API, um so it would be um like today the storage version stuff just tries to update every 10 minutes because it's like yeah, it's kind of slow moving. You shouldn't be messing with this, but if you kind of messed it up, we'd fix it up every 10 minutes for you kind of deal.

B

uh This would kind of make it a little bit more dynamic.

C

Yeah uh I mean if you've got stuff.

C

Yeah, if the, if the storage layer knows it I, think the storage layer doesn't have the uh I, don't know it's a little weird to have the storage layer uh calling back through the front door of API server to modify something in another storage layer. uh So.

D

So we did have to address that with uh with a filter. Handler but like weird, is relative. The alternative which uh some of us have lived through uh is worse, alternative, being direct modification NCD of another resource that doesn't go through. Oh.

C

Yeah that that's unacceptable yeah.

E

I wasn't discussing that: okay, okay, yeah.

D

um I I don't find that to be overly strange, uh I think finding a consistent spot to write. It is important being able to describe when you dock the field, how much lag can be expected uh and everything in this type.

A

D

A reflection of what the server actually does. It doesn't actually drive the server Behavior, uh so great I.

B

Think that's a.

D

Tv with the field you're proposing right.

B

Yes, it, it is a reflection, and if someone messed with it, then it you made a wrong reflection and it'll get fixed uh but like it doesn't like we're, not not trying to we're not trying to consume this information in the API server. We're trying to tell other things that hey! You know, barring someone just arbitrarily messing with like core system resources. You can trust the data in here to be something useful for your for your purposes um uh for a Next Step.

B

uh So I can certainly try to resurrect that crd, uh P or crpr and make it storage version stuff work there. um One of the next steps that I thought might be appropriate is like the cap document seems to be not in line with, what's actually in the API server code, uh because you know those things diverse I was going to try to update it and like, and that would hopefully give you all a way of like being like yeah.

B

That seems like a good idea versus not um but they're kind of the high level I'm getting is there's not necessarily anything like invalid about any of these cups and like if I was to like uh make sure they had good test coverage and stuff, then otherwise, actually were feature complete for what they said. They would be that this could go to Beta in, like 127.

D

um I think we had beta criteria for the storage version. One did we have it for the identity, one.

E

C

Don't recall but I, so the the only problems I can make is that if you, if you uh chat PR's at me, that adjust the Caps uh I will uh review them. Okay,.

B

Yeah I I I, don't need like a promise on hey yeah. This will make it at this time, it's more of uh if there's obvious gotchas and blockers and things that you know are heart going to prevent it from graduating. Further yeah.

C

I, don't think I.

B

Don't think there was any.

C

Like uh blockers or like technical, uh uh like surprises, that we knew about and that's what stopped us is just like the people doing, it uh ran out of time to do it and did something else.

B

Yeah that you know that sounds fine, I I can find bodies. That's that that part is not hard um yeah it's at a high level. My concern is I'm, taking a Sig off cap and then deeply coupling it into two separates. They gave API machine Recaps and hoping that the other two graduate so I can graduate my thing. I.

D

Think you should just look at it as Daniel and I were pressing. It knew what you needed ah eight releases ago and uh and we will accept the thank you, foreign.

D

126. I'm I need to double check to make sure that it actually is. uh We did indicate this is what we wanted for beta. But if we've done that, then I think I'm happy doing that.

B

Yeah, if, if you think that there isn't that much left for it to go to Beta I can I can certainly push my efforts towards getting this move forward, because I I'm not moving KMS forward. This release on purpose uh so like I, have I have time it's but yeah, I I know what I need to do for KMS, I, don't necessarily know exactly.

C

Yeah I'm, just the average case for a cap, is that something surprises us so I think uh the the sooner we can uh start on it. The better um okay.

D

um I'll try to find the features, I don't even know if they have issues there's a bunch of labels. We have to add and I think we have to do that before tomorrow. So yeah.

B

Yeah, um okay, I have taken up like 10, more minutes than I planned on and I know. You guys know yeah, so I will stop talking. Thank.

C

You guys cool uh thanks, Moe um I. Think of the next two items are the gender for me and I'm gonna try to keep each of them super short. uh These are just follow-ups from last time. The current state, uh the current state of The Logical clock uh versus uh expanding the definition of resource version is that I made this issue and uh a number of people expressed opinions.

C

uh It seems to me that uh there were even some uh some additional Pros were pointed out for the second option of the adding a logical clock, but it still seems to me that overall, expanding resource version is winning.

C

um If you haven't read it, I would read uh at least the first post I've tried to keep it up to date, with all of the uh all of the arguments that have been brought to light in the comments, um but if, uh if the discussion doesn't evolve soon, I'll probably just call it for uh option one expanding research version and send a documentation PR, and we can be done with this- okay um and the second one, um and if you don't agree with that, then also go say something there.

C

uh The second one is the uh sub resources.

C

Fine grain permissions, uh in addition to the five the straw designs you all saw uh I, took that to Zig off last week and um uh I I think mostly the objection from Jordan that it, the first set of designs, only solve the like Grant additional provision to somebody who doesn't have it problem and not the like, restrict permission from somebody who does have it uh problem, and uh so, if we're up to uh I, think nine designs, uh the design six is basically something David was saying uh design.

C

Seven is uh my expansion of that and Tim didn't like that, so he wrote a design, eight and uh I thought that was insufficiently clear, so I've written to design nine uh and I still think it's not right, um I! Think design. Nine at least illustrates the problem with uh ex like parameterizing permissions and uh making them uh like, like, like it's in some sort of nested manner, making them restrictable. uh So uh but that's that's the current State and I.

C

Think Tim is like leaving comments on it as we uh as we speak here, uh so I'll try not to watch the screen, uh but uh yeah I I don't have a design that we could possibly make in time. For the kept deadline, so this is going to have to wait until next release to uh get a design yeah. So.

D

I have not been able to carve out time to review the designs. I've been buried in kept land since yeah.

D

The next two weeks to do it before we come back.

E

D

C

You aren't trying to write a cap.

D

C

I'm not going to write a cabinet, uh I I might be so I I, think uh design. Nine might be close enough to passing everybody's uh criteria that maybe a kept format to finish off the design is uh The Next Step, but I'm not going to get that out in the next couple couple uh weeks. Probably okay, maybe in the next couple weeks, but not.

D

In the next couple.

C

Hours, yeah, no, no, okay, thanks yeah uh yeah! Okay! That's all I wanted to I just wanted everybody to know, since we took like the entire time last time. Talking about those I thought, people wanted I, don't know uh where they ended up.

A

Thank you Daniel for the updates. This is great, um I mean. Do you have photo apps on things that you know we discussed in our journey and then we lost track? Okay, um so David Daniel. Do we want to do these? So, as you know, the process in 126 has changed. We have two opt-in, um the Caps uh that we think are going to make it or that we want to make it into the next three days. So this is the general board and then there is uh okay. It's one of my screen.

A

um You can filter here through our work values here, so we have three caps that I already Linked In the document uh so far. Maybe we will get uh most one uh into this two.

A

um So uh where are we here? Okay, um so how do we do? Is let's go one by one? I, don't know who wants to talk about them? um First, one is to allow informers for getting a stream of data instead of chunking polynomial.

A

um David I believe it was you who finally accepted into the uh yes.

D

A

D

Okay, uh this is one we worked with through with wojtek um and got that point where we even had an implementation last time, but you know we weren't Implement them, so we didn't merge.

A

Good any questions on this one: anybody who has comments concerns.

A

C

Probably want to go look at it because I don't remember where we got to on that one, uh but uh I I won't if David already approved it, I won't block it. Okay,.

A

The next one is sell for admission control.

C

Yeah uh I know David made a pass. Jordan made a pass uh I'm in a pass. uh I think we've got all of the unresolved items out of this um and uh there's one last PR that has the uh the review and flips it to implementable. So this is about really cool.

D

Yeah, as I recall, John had the prr on this I looked through the unresolved sections at like 155 um and I thought. That was pretty good. It was, they reflected the yes, we are going to do this. No, it is not in phase one so I was I was happy with that for secondary authorization and for namespace metadata.

E

Yeah, those are both definitely both Finn.

A

What needs to happen in practice? This needs to be merged before tomorrow, night.

C

Yeah I was gonna since I got uh I got like three or four of these I was gonna. Let David uh flip the final bit on this or approve this this last one. That's on the screen right now.

D

Sure uh I will line that up for this afternoon before I leave.

E

Awesome, thank you. All I will try and never write a cup that long again. That is a personal goal. So thank you all that for reading that one I appreciate it.

A

um I think the official deadline is tomorrow at midnight uh BST, something like that. I can double check, but I'm better to be safe than you know. I.

C

Think the changes since you last Saw David are probably very small and it won't take you much time. Yep.

D

A

Okay and the last one we have on the list is a great discovery.

A

E

C

D

I'll see it Jeffrey they're, free.

E

Oh yeah, uh sorry, can you guys hear me? Yes, yeah um I think we just have one last sort of question on that uh on the pr sorry, can you go to the pr.

A

Where's the beer, this one.

E

um Actually, I I can link it in the chat.

D

If this is the pr about like beta versus Alpha and feature date versus serialization, there's, actually a long slack Thread about that now and I. Think where it landed, is the idea of a more stable serialization makes sense, because we have a lot of experience with Discovery many years of experience with discovery, but we have controllers and ha concerns for the implementation of of the backing data that have not been tested. So a beta serialization would allow us to have clients honor it when it was available, because we know the serialization is stable.

D

It's not going to dramatically change and as opposed to an alpha which could you know significantly change over time, basically complete, discard and redo, and so we can gate the beta serialization with an alpha feature date without changing the the timelines proposed here. So the client would be able to use it if the server chose to enable it and uh and then the decision about whether to enable that will be left up to how confident individual individual distros are. And potentially you can even change your mind at a later point.

D

So if you're not sure enough turn it on when 126 comes out, when 127 comes out and it's Beta And On by default- and there were basically no changes, then you can come in and retroactively turn it on on older releases.

D

Managed to get Jordan Clayton myself and Antoine all on board with that.

C

Yeah I kind of back out of this one, because there's just too many cooks in the kitchen.

D

Yeah there were, there are definitely lots of opinions on this.

A

So what needs to happen to you know before tomorrow, night.

D

If I'm, remembering the PO well Jeffrey needs to read and make sure he agrees with the big long slack thread uh and if he does then I think it's just swapping the feature gate to say: Alpha leaving the serialization as beta. uh And then we are good to merge.

E

Yeah I've updated the cat based on our discussion, um so probably just needs a review now cool.

A

E

Go remember just uh if I could just say a couple words uh on behalf of the six CLI. It would be really useful for Coupe control to to be able to have a significantly more efficient Discovery. We strongly support this and we're hoping that this doesn't have to. We don't have to wait another four months for the next release.

E

D

Yeah, uh that's that's the goal of that more stable, beta serialization. So with a beta serialization, um we would be willing to merge into client go usage of that serialization, uh and that means that if the server turns it on for you in 126, you can use it uh in 127 it'll be on by default and there's somebody retroactively turns it on for you in 126, in their distro by enabling it on Alpha the client go automatically use it. For me,.

E

Cheers thanks great thank.

A

You David, um okay, and so we went through these three and then there was one more thing: I wanted to show that uh there are two more caps that are not in the world, but they are um open and tagging CDP Machinery with the naval 126, so I think it's this one and this one um so I understand they are not primarily API machinery and they may be. You know, on the side um involving us.

C

The yeah I could say something about this one on the screen right now. uh This feature is not really related to us. um They've asked for uh some things about uh like sending stuff in a delete request and having that propagated to conditions on the object and uh Jordan and I have not been in favor of that.

C

um uh Currently, uh the the issues where they interact with us is that they're sending um patches to update conditions before a pod is deleted.

C

When we talked about Alpha I think we got a item that requested them to switch to server-side, apply for updating those, uh and there was more discussion on this and uh they wanted to remove that for beta.

C

But in the discussion it turns out that there, the code making the patches is not really tolerant of other actors racing to make similar changes and is buggy like potentially adding a second condition or deleting or replacing the wrong condition, so uh I think they're keeping the request to switch to server side, apply or or or whatever else, we'll fix that yeah. So I, don't think we really need to pay attention to this.

A

And to correct what I said and Etc in the tracking board, it's just that he's under uh SYNC apps, primarily so um yeah I didn't see it and maybe in the same situation, happens with the other one.

A

This one, uh so this is trees. Three two waiting.

A

Okay, yeah, it's also in standard signal, so I think um very good. um If I, okay, I, you know, I, don't want to extend this meeting unnecessarily. If I remove the Milestone 1.6, there is a bunch of six that I will be imaginary too, but I think this is an exercise for another day.

A

Okay, so I think we are good unless somebody thinks otherwise. um Our caps are opt-in.

E

A

um People owning those caps are going to make sure that um peers get merged and they get sleep before tomorrow night um and that's what I have for this.

A

Anything else, foreign.

A

And I hope we can eventually, you know, get some help to keep making progress on this year.

A

um We'll keep you posted. Thank you so much for attending today. I hope you have a good Wednesday or the rest of it and have a good week.

E

C