Kubernetes API Machinery Special Interest Group, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery - 20220921

Description

*[lavalamp, pohly] Add whole-object logical clock field, or relax client RV constraints, to support “assumption caches”? https://github.com/kubernetes/kubernetes/pull/112202

*[lavalamp] subresources vs fine grained permissions. See doc (shared with api machinery mailing list).
thoughts on using CEL for this (design 5)
because CEL is non-default, optional, example of https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#ownerreferencespermissionenforcement which is off by default causing confusion for developers and cluster operators
thoughts on using secondary authz checks (~design 4)

*[deads2k] - choose which KEPs we want in 1.26 in the next week or so.
https://github.com/orgs/kubernetes/projects/98/views/1

*Shameless Plug: Virtual Session at Kubecon NA - API Machinery deep dive: https://kccncna2022.sched.com/event/182Mo

A

And we are on good morning good afternoon good evening, depending where you are, this is kubernetes sea api machinery by weekly meeting today is the beginning of a new season in all the hemispheres, the fall for the northern hemisphere, the spring for the southern hemisphere. So times of changes, thank you, everybody for coming. We have a couple of topics from lava lamp into the agenda today, so I would say: let's get to them.

A

I will answer so daniel.

B

uh Sure, um let's see, I think this first one. Well, I don't know we'll see we'll see if it's controversial or not. uh Basically, if you look at this pr, the the goal here is to add a and the author has called it a assume, cache, uh basically to add it to clanko. What this thing seems to do is when you make a write to api server.

B

The goal is to take that value, that you wrote and store it in the cache, so that if somebody else gets something out of the cache, then they they observe the written object instead of like the thing that was there before the written object.

B

uh So the tricky thing, of course, is like what, if other watch events are coming in in the meantime, how do you know if the thing that you wrote is more recent than the thing that you have in the cache or not, and also, conversely, if you're, if you get a watch event, how do you know whether the watch event was before or after the thing that you wrote and have stored in the cache?

B

um Is that clear description.

C

Clear enough to me three quarters of the problem.

B

Okay, what did I miss.

C

There is currently per resource and ordering for our listeners that says uh right right. Another object was updated before this object was updated. You will always see a consistent order.

C

The state that you see in the listener will have actually existed on the cluster, but if you record your update early, you can end up in a state where say a route has a route object has already claimed a hostname, um but the another object was updated in the middle that that when you do a list across all these objects, you don't see a current value of.

B

Yeah, to put it another way, what I just described is fine, if you're only looking at individual objects, but if you're looking at combinations of objects. Once you do this, you have put things in your cache in an order in which they haven't appeared in the server and if you compare two objects, one of which you've written to and one of which you hadn't haven't. uh We can probably come up with some sort of guarantee, but it may.

D

Specifically, you mean you mean objects of the same type or kind. uh That's.

C

The only guarantee you have today, yes, objects of different kinds. You do not have a guarantee of ordering, but at a single kind. You do.

D

Yeah: okay, very.

B

D

B

So uh I mean that doesn't make the approach that doesn't make it an impossible problem to solve.

B

uh Is there is that your prop jordan, that doesn't make it impossible to solve, but it does mean that you would you would like like, for example, you could uh just lock everything until you observe the right, the the watch event corresponding to your right um that solves the problem and maintains the property that david is talking about at the cost of like locking the entire uh cash for all consumers for a long time. Well,.

D

Slow down the client is never guaranteed. As far as I understand to see exactly that right. There.

B

Yes, yes, now you get to the problem for why I have put this on the agenda because there's actually no way to do this at all on our in our system, because you can't compare you're not supposed to compare resource versions right and you're, not guaranteed, to see an exact change. So how do you know if a watch event is older or newer than the thing that is in your cache right now, uh so I I can like.

B

I think it's actually not possible today and I think I can think of two ways of making it possible. um The first way is to relax our constraint on not letting clients parse the resource version. If we let clients parse the resource version, um which in fact, clients do anyway, even though we tell them not to, then you could compare resource versions to see if something is older or newer and solve the problem. That way.

B

The other solution I can think of is we could add a logical clock like the generation logical clock, but make it count the entire object, not just the spec uh so well. Actually, now.

D

That you mention it right in a lot of cases. It's the spec is what we're concerned about.

B

uh In a lot of cases, actually, this is controllers are actually more concerned with seeing their own changes and the controllers often make changes in status.

D

Right um but at least in in the controllers, I've written um the cases where I need to uh kind of do this kind of thing of keep concerned about seeing my own change. Well, I guess it's true for status too. I need to know.

B

Yeah, at least for this author- uh I don't know if he is- has joined the call. uh Probably not I don't see him on the list.

A

At least for this.

B

Author, that wasn't gonna work.

D

Yeah, I'm not sure I understand that um I mean I'll admit I've struggled with this too and I've. You know not solved it by um comparing resource by parsing the resource version. I've not solved it by putting in this kind of a cache, um uh but for status right for a status that a controller is providing.

D

I think it's valid, that the controller can just say hey. If I did a successful write, I know it went there. I don't need to. You know worry about it. um I mean.

B

The controller.

C

B

On your use case like if you're looking, if you're, if you're looking at the current state of the object to decide, if you need to do a thing uh and like and somehow it comes to your attention again in your cache, you could like double actuate uh or not actually, but.

D

I wouldn't be looking at the status of the object I'm implemented in order to decide if I have to do a thing.

B

That really depends on how your controller's written and what it does right, like you're, not stating an invariant that works for all controllers. I totally believe that.

D

Your controllers.

B

uh You can do this for, but but not everybody writes things that way. Maybe they should, but we can't force them to.

D

Well, maybe this is another reason why they should.

C

um I I can imagine I mean I've dealt with many cases where you have a lot of different writers and they care about seeing the changes.

C

I think I think I'd probably be willing to say at this point. It's been eight years. It's an integer resource version.

D

Well, you know we do still have work percolating around the edges, um that maybe it doesn't need. Maybe we don't want it to be a 64-bit integer. um You know I'd be up for something a little more abstract. You know we provide an abstraction that can compare for you know ordering um without revealing exactly that. It's an integer.

B

Yeah when I was uh pre-discussing this with david uh one of the things I thought about was because you I maybe if we let people uh compare for single resources, we should actually make it difficult to compare across resources.

B

The easiest way I thought of doing that is just to put a few like set a few bits at the most significant side of the n64, um so that it's different for different collections, which would make it very difficult to compare without a lot more processing. um So so that like. If we want to relax our constraint and say okay, you can do some comparison, maybe at the same time we can be like, but we're going to make it actually difficult to compare across resource versions where uh we are sorry across uh collections.

B

D

B

D

Kind of a hard.

B

Boundary in the system.

D

If you make an arbitrary length integer, um then it's really just a string of a different kind. Almost um if you keep it a 64-bit integer, now you're cutting into the 64 bits.

D

Let me remind you, I I don't know if, if you know steve presented his work here, but you know steve kuzner's work on the uh storing direct to cockroach, uh he pretty much needs all 64 of those bits, um they're they're they're kind of precious and he would rather have more bits, uh but he can kind of get away with squeezing it into 64 bits. But I wouldn't ask it to be squeezed into any fewer.

C

We're currently in integers, so we have infinite bits or sorry a string, so we have infinite bits.

D

No because clients actually parse it as a 60 sign, 64-bit int. So in fact steve has squeezed it into 63 bits um and we would break stuff if it were anything other than a. You know: a 60 science 64-bit positive integer.

B

Right, that's the issue is that clients actually, in fact parse this today, uh that's why we should have made that much more difficult in the past and I attempted it, but we were already too big at the time.

D

D

Yeah I mean, could I mean you kind of touched? The next question I was going to ask, is you know, could we do a transition where we introduce an abstraction to use instead of parsing and then at some point we change the strings that appear, so the abstraction keeps working because, of course, we've shipped an implementation that will keep working, but the strings will will break clients that are not using the abstraction.

D

The difficulty is.

B

Like how do you get people from all all different languages to use the same like if you, if you are going to solve this with code, like you got to implement this in every client library, if you're going to solve this with a with an additional field um that could maybe be okay, where, like resource version just holds the lower 63 bits, and this other field has uh all the higher bits?

B

That's maybe something that could be done.

C

Yeah, I don't think it's practical to create an abstraction, that's going to work on a java, client, a javascript client unless we actually went and created it. Yes,.

D

Yes, everyone is already convinced.

C

That people parse the string field as an int, then it seems non-contentious to say all right. It's an.

A

C

Now compare away.

D

B

I mean we do support. If we need more bits, then the practical thing to do is to add a additional field.

C

D

B

uh Database, I don't think you're ever going to use up to 63 bits. But yes, if you, if you're doing something more complicated, um sorry mike you were saying.

D

I was just saying we do supply a lot of client libraries.

B

We do supply client or libraries. However, most of the most of the client libraries have most of their code automatically generated from our uh api schemas.

E

Also, the client libraries we provide like then get used by programming languages which will pull out those string fields and parse them. Like the client libraries we provide, aren't doing parsing today, so the part thing is being done by other people's code built on top of those libraries and experience. Updating the client go like token refresh stuff in 114 time frame has shown like you can wait years and you won't get adoption of new versions of client libraries in the ecosystem.

D

Yeah I can see how that could be a practical problem right. So now, there's stuff going on in the chat right. The problem with string ordering is it's not consistent with the integer ordering we have now right if we could somehow switch to string ordering that would be much more powerful than 63-bit integers. uh That would be a great solution, but I don't we've just been talking about why we can't.

C

I mean yeah, just extending the radix. Doesn't it seems painful and 64-bit integers are big enough, for it should be big enough for anyone, uh but.

D

Well, that depends on what you're doing right, so I've contemplated um going to you know: sharding the storage so yeah and, and that would involve putting a version vector here uh and that's not going to fit nicely in this 63-bit integer.

C

B

It also would cause.

C

Problems when you try to get these clients that are built around it to use it so.

D

Well, that's the problem, we're talking about.

B

So we've discussed the you know what, if people use resource version as an ant aspect, but what do people think of the other alternative, which is uh add a thing that is actually an int? That is a logical clock for changes to the particular object.

D

So basically, you're saying take the generation idea, but make it apply, make another one that applies to the whole object. Correct.

D

I I think that would be a nice solution for a lot of things. Yes,.

C

I don't feel strongly between that and just saying resource versions in it.

C

B

Article, do you have a preference between those two.

E

um It defends me philosophically to give up on saying, don't parse, but I don't practically see how we would ever change it from what it is today. So if we can't ever change resource version from what it is today, we might as well get the benefit of being able to compare um yeah.

D

Yeah, I kind of right. I agree with jordan right. I would like to see this to be an uh opaque string, uh ideally with a wrapped in the distraction we ship. But you know the conversation has just reiterated. It's a 63-bit end. We can't change it from anything, but I mean 63 billion right and if we're going to insist that it's going to stay a 63 bit end, let's just admit that it's a 63 bit end.

B

What, if we do, both things.

C

uh Would you hold which two the cash, the assume cash pr on? Having that you think or would you say, we've agreed that resource versions are integers. You can do this eventually, there might be a better thing for you to choose, but well.

B

uh I I I think we need to hold the assume cash pr on answering the thing that you you brought up, which is like uh your your list. Order is now undefined. um You won't be able.

C

To solve the list order problem with a single with a with a per instance generation, you.

B

Can you can, you may have to wait extra.

C

What why how I think you still need the resource version.

B

I mean you're gonna see uh well, okay, so so are it depends on what.

D

Oh actually, oh right, yes, okay, you're using so first so to be clear right. The suggestion was to use both. So, let's just understand correctly the suggestion. One version of this is keep the resource version for bid. Parsing, add a whole object, generation number all right and then in lists there is no whole object. Generation number that makes sense, there's only a resource version, a but a controller that is caring about this stuff. It knows no, it's still not good enough um yeah. So daniel, I'm not sure. I understand what you're saying.

B

I would say I was suggesting like what, if we both added a whole logic generation and we let people parse resource version, if they wanted to.

D

Yeah, if we let people parse it, what's the advantage of adding a whole object generation.

B

The advantage of having the whole object generation is that uh nobody's ever gonna run out of bits.

D

Well, if we're concerned about running out of bits, that's the problem, even if we have a whole object generation.

E

I think whole object generation.

E

uh If, if I'm understanding correctly like we would need to bump it, we would basically be persisting a per object resource version and we explicitly don't persist resource version in ncd today, because it would make no.

B

It wouldn't it wouldn't increment with every change to every object in the collection. Only every change to that specific object.

E

Right but but that would mean like the no op rate, if I, if I round trip and I'm not going to modify the object in any way like it, doesn't actually persist. An updated.

B

Lcd we'd have to make an exception for this field. If the only thing changing is this field, uh then it's not a real right and we don't actually increment it.

A

It's congruent with what we do right generation. I think.

E

Right, I'm just thinking about practically like how we would implement like today. It's a byte comparison to say, like the bytes, we were going to put an ncd match the bytes that are already there. So.

B

I think it's a little more complicated, don't haven't we fixed the server side, apply uh no operate bug. I think it's much more complicated than a by comparison. At this point.

A

So, even if you did have it, what could you do with it? If it hasn't changed, you would know, even though the resource version has gone up, that your object hasn't changed just by doing a comparison, that's one property that would have a resource illustration doesn't have.

D

Yeah, I thought the whole point of introducing the whole object generation was to be able to continue to forbid parsing the resource version.

B

uh I don't know that I thought that far ahead, I'm just listening alternatives here.

D

Okay yeah, it seems to me if, if we allow parsing the resource version, then adding the whole object generation doesn't add anything.

B

uh It does add the ability to tell that you made a no upright.

D

Okay, anything anything really significant. I mean that that seems like a really corner case.

B

I guess maybe you could. You could actually already tell that from resource version not changing today.

A

Well, the resource version will change when anything changes in the system, so yeah.

E

So if you do an update and the resource version doesn't change, you can.

B

Already tell yeah: uh we we send back the global index, or do we send back the time at which the object was last written? I believe it's that.

E

The resource version of the persistent object.

B

Look, if you do, if you do a list and look at the resource versions of the items in contained in the list.

E

B

Not all the same as the list resource version right.

D

But that wasn't the question.

B

E

B

D

That you could detect a new upright by comparing the rv in the returned object with the rv. You know going into the right.

B

Oh because yeah, if you, if you, if you saw the latest version of the object and your right made no effective change, then the resource version won't change correct.

B

If you saw an old version and you're right made no effective change from the latest version, then you would see the resource version of the latest version of that object.

E

I mean you'd probably get a conflict because you were writing with a steel resource version, but.

B

Unless you wanted exactly the thing that was already there.

D

No, that would still be a conflict, though, wouldn't it I mean, if you I, I think, we're getting into really corner cases. I don't see the value in uh you know we're. We already live without this whole object generation. Today we're hypothesizing, really corner cases that could benefit from it. I'm just not convinced.

F

I mean this corner case is: like storage migration. It's important.

B

D

Storage migrator is the.

B

One thing that needs to touch an object and make a a write without actually changing anything about it. Well,.

D

Can you elaborate, though,.

B

The storage miter migrator triggers when there's been a storage version, change on api server or people should change. It should trigger the storage migrator and it is supposed to read and then write the objects.

D

B

The storage to become the new yes version and not the old version right.

D

And why would it care whether it was actually no upright.

B

D

E

C

A

The other direction.

B

Like our no op right, detection has broken the storage migrator in in the past because it's like, oh you're, not making a change. uh So I forget how we solved that, uh but I agree that we have strayed from the topic very far, as does anyone strongly object to uh telling people that they can parse the resource version.

E

Not strong, I think my.

F

I was gonna say my concern there would be is if you wanna, be fun and use cockroach, and you want to play with that field.

F

Yeah you're not compatible, maybe with the rest of the ecosystem, but we start telling people that parsley.

D

Is like really awesome. Steve has got a way to uh right now right. His code is already uh squeezing into those 63 bits he's.

F

Okay but, let's just say, say: cockroach changes in some way that needs more bits and.

D

I agree right. um The problem is that I you know the current state is, it is a 64-bit. It is a 63-bit end, but you can't parse it right. That's what was strongly emphasized in this meeting earlier right. So the the question is: do we one way of viewing what's happening? Is the question is: do we change from it's a 63 bit, but you can't parse it too. It's a 63 bit and you can parse it.

B

I mean effectively: it's not a change, because today people actually in fact do parse it uh yeah.

F

If we gave him a sorry.

E

If we wanted to frame it, you should use this as a big hint. Then, like yeah people will still not read the docs just like they don't read the docs today, but in the normal case, where you're working in a tv or you're working in a system that can fit within 63 bits like the parsing works. Fine and if, if any examples we give of parsing or comparing candlelit is a big end that at least starts to maybe shift the ecosystem towards like if you're gonna parse do it. This way, I I don't know.

F

But if you have a dedicated field that only increments in the same.

D

F

As generation but for a whole object- and it is actually an 63 or whatever, then you just tell people to use that field and there's no wrong way to use it right either. The field exists, because you have a new right server.

D

But, as someone said earlier, it will take years to get any kind of adoption.

D

I mean I'm sorry it'll take years to get reasonably completed. I don't even know where's the right. Here's the question right. There's this unbounded universe of code out there right. We can make recommendations we can expect most of it will change relatively soon, but that doesn't answer the question.

F

But isn't the goal to provide like a nice safe api abstraction, whatever you want to call it? Instead of trying to say we oops, and we should have hidden that field better, so just abuse it anyway.

B

Yeah I mean the issue. The issue for me is that there is literally no safe way to do this today, without either breaking our rule or just like not being able to do it at all. um Well,.

D

The issue is breaking if somebody, if somebody.

B

Has the uh if somebody has the need and we've made it possible, then they'll be.

B

Be able to take advantage of that, so I think.

D

B

D

Basic project governance kind of question: here we have a change that we'd like to make and we have this unbounded universe of clients right. We can provide a path, a migration path for these clients, but we have no way of um you know, making sure that everybody actually does migrate. But as daniel says, maybe it's enough to provide a reasonable path, a reasonable amount of time.

D

Anybody that doesn't with that by that time,.

B

I mean nobody needs to migrate. Anything right. People can't do this today, uh but.

D

We're talking about you know being nervous about saying that it's only 63 bits. Okay, we've been talking about alternatives to 63 bits. Maybe it's a big end. Maybe it's! We have some other field, the strings lexicographically compared we could uh take our existing ants and pad them out so that they lexicographically compare right. So it's it's actually both um big and in lexicographic string.

D

At the same time, uh you know, if there's any number of things we could do, but any one of them you know, is not going to work for all clients that are out there today.

B

Okay, so so, let's let's wrap this up, so we can so we can go on to the next topic. uh I actually uh mike. I, I think you convinced me the opposite that you were intending. I I think, actually it's best to make a um to make a new field and make it a logical clock, and if people have this problem, they can look at that field. If they're, currently pressing resource version, um we're not changing anything about resource version, so we're not going to break them at all.

B

But if people want, if, if in the future, we do want to expand our use of resource version into something else and worry about breaking clients at that point, uh our hands aren't tied. um So uh I guess I'll write that up and send an email to the sig list and we can for.

D

B

Sake of time, I'll.

D

I'll not respond yet: okay, okay,.

C

uh I'll say, I think that adding a field like that is likely to need a cap. That's pretty fundamental, that's every object out there sure adding a field, and I would like to see how it resolves the list problem.

B

So sure, uh um well, at any rate, I think the next step is uh probably a write-up of the pros and cons of each approach and we can have a more concrete discussion. Okay. The next question is on sub-resources versus fine-grained permissions.

B

If you'll recall uh one or two releases ago, uh we had a couple caps, one about leans and and the one about lean leans has been in the system for a while, and um we have not given the author, a very expedient response, and the last time it went through uh david was very worried about. I mean not that he shouldn't be, but was worried about like uh permissions, and how do you keep people from like wasting other people's resources etc by holding objects open forever and wanted some way to cap the permissions?

B

uh So then we latched on to the idea of. Maybe it needs a sub, a sub resource to expose this, and then the thinking was well. uh Finalizers doesn't have a sub resource, or at least not a universal one.

B

So maybe we should see if we can get a sub-resource for finalizers before we go making another thing and then, when I was reviewing that cap, uh I realized that, like, if we're going to make a sub-resource for finalizers and a sub resource for liens and like we already know that people also would like, like I said, resource for labels and annotations, and maybe owner references and probably a couple of document forgetting.

B

But if we make all these sub resources we're going to put a client that is like you know, you know how we have the apply status and update status and put status like we're going to. If we have like six sub-resources we're going to duplicate all those methods for every for everything and then for every sub-resource and then controller authors have to know which sub-resource to go to and if they want to do multiple things.

B

They've got to make multiple calls because they they have to use the sub resource or people won't be able to to configure their permissions correctly and it just becomes a large mess. And I don't like it at all. So I wrote this document with a number of other approaches that I was hoping we could debate here, so I think, fed if you could go if you could display the document. That might be a better, a better thing for us.

B

So I don't know if everybody had a chance to read this, I sent it to the list. I think. Last week I can go through the designs or if people already know what they think we can just open it up and people can talk. uh Would you like a introduction to each design.

C

Maybe jump the ones that are the most plausible we've rejected a couple. I think already.

B

Yeah, okay, so uh let's go to uh design five, let's go backwards because I think the I think five and four are the most plausible ones. um Design five uh basically makes us a part of the cel admission control effort um by making sure that that that, uh if you write a policy, you can perform a secondary, auth z check um to see if, like uh there's an example on the screen right now like either the labels match- or you have this permission, which would involve api server, doing a secondary auth check right.

B

So the uh has some advantages. I think david isn't going to like this one, because the cell admission is optional and somebody could turn it off. That's do you want to say why that's important to you, david.

C

Sure I think it's important to have behavior around what authorization checks are made. Not what the value of the authorization check is right. If cluster admin wants to be always allowed fine but which authorization checks are made, I would like to see be consistent across every cube api server out there, and so I don't want this check this make this authorization check when these fields are changing. I don't want that to vary between clusters, cause I go to cluster one.

C

Can you give us.

B

An example of like what goes wrong if, if those do vary.

C

uh Sure so so there is a gc admission plugin today that checks to see whether a user has permission to delete an object before setting an owner reference, and it is disabled today, and so some clusters you run on are protected. uh You cannot issue an update that results in a phantom delete, uh an escalation of the garbage collector to actually do a delete and other clusters are not protected.

C

So you got, you got two classes of clusters, some some where you're protected and somewhere you aren't, uh and it makes it difficult to know what state you're in, and I want to avoid that for a metadata field that exists everywhere.

B

Difficult for uh for who, for controller authors or for users or cluster administrators.

C

F

I was just going to give a quick little thing like we. The src recently had a vulnerability submission where someone was like hey, I can delete stuff and I shouldn't be able to it's like turn on the admission plugin and then you won't be able to, but it's in our docs and everything. But like people don't know it exists, um so.

A

C

A

F

With what david is saying, which is, if you have a nazi check, it should always run.

E

Or be on by default, I think the the issue we ran into with the garbage collection admission. One um was that it's off by default because it was added after the fact, and we couldn't turn it on by default compatibly.

E

So if, if we are, if we are strict by default, then when people are developing against a kind cluster or whatever they're going to realize they need this permission, they're going to take that into account they're going to write their controller write their permissions in a way that accommodates it. If a particular cluster wants to like turn off protections, it's like okay, but.

B

E

Is that the right to see is that the root of this.

B

Is that the root of this concern, like I develop on a cluster that doesn't have this thing enabled and then I go to a cluster that does have it enabled and my thing doesn't work and my my design is, is just completely wrong to make it work.

E

My concern is tuple one that most people don't realize it's the thing they need to consider and accommodate when they're planning and designing and developing, and then the other is what mo mentioned, where it's like people don't have a protection and are vulnerable to this sort of cross actor interference, and the answer is: oh: did you not adjust from the defaults to protect yourself.

C

I guess I would also say that I would like to say that someone who opts out of this does it by making the authorization check always return aloud, rather than disable. The fact that someone checks.

B

um Okay, so let's see joe you're on the call, what's the chance of making this cell thing mandatory.

B

A

um I mean right now our main focus has been on the extensibility feature, um so the idea of this being like a way to effectively add a first party admission controller, isn't on our list. We haven't put a lot of thought into it.

A

The first question I would have there is: if you're going to do something, that's standard. Why not just write an admission controller for it.

C

B

Would be option four right uh there yeah there's, so there is, I think, that's actually option uh three: okay! Well, why don't you scroll up fetty and let's look at let's look at another option here.

B

Option four uh yeah: we can talk about option three, first, okay, so just to just outline them option. Two is like how you would build this thing like to limit people to particular fields. If your tools were web hooks and crds and like obviously, you can turn those off or fail to install them option. Three is what, if we modified our back, which uh surprisingly you don't have to run rvec, you can run other authorizables.

B

So that also has this problem um yeah uh mo. I see your comment. Aren't all admission plugins disabled. uh Yes, they are, but that doesn't like. We have the code. We can still run an admission plugin, even if it's disabled right like we could, we could make a separate path for turning it on and off uh and make sure that it's on by default, so that doesn't seem like a huge blocker uh to me. I think david disagrees with me, but um I I.

C

F

C

F

B

F

That yeah, I was going to say maybe maybe the framing, is just wrong like like, for example, we do certain checks at the registry layer for our back and stuff, and you can't turn those off. I mean I guess you've turned the entire resource off, but you can't have the resource without the check.

F

This would be a little weird because this would be like a registry check that every resource has, but maybe that's fine.

C

I possibly uh I see the difference between modifying our back and creating a secondary authorization check, as as the rbac change is more of a how you would get it done, but the secondary authorization check is what you would actually do. I've I've grabbed these two objects. I have them before. I have the after. I know what's changing and now I need to check to see if that's allowed and the only spot I can do that is either in admission or in storage and then to check whether it's allowed.

C

I can make an authorization check and whether that is handled via some kind of our back rule or some kind of new authorizer. I could see creating a new type for it um or whether it's handled by a web hook. It would allow all that to function.

B

Yeah, I think, uh if you can scroll to design four, I think a new type is what I proposed here. uh So basically uh just to outline this, since it seems like this is going to be the the winner. uh We do two checks, the first one like okay, so first like do you have edit at all. If you just have a regular edit, then it does what it does today. If you don't have regular edit, then we check this some sort of perfield permission like like.

B

Should we go to the per field permission checking path? If you have that, um then.

B

There's something complicated to figure out which which fields you've changed, which implies which permissions need to be checked, uh and uh you have to have some permission that covers each field that you're modifying.

B

uh Hopefully, that's uh not super confusing uh jordan.

E

uh I I was just gonna say like what, whatever we end up with, I think a good property is that the expression of the permissions can be done in a single place or in a coherent way, like one of the scary things that I see, people having to do is sort of over grant permissions at the authorization level. To say why, yes, you can update nodes and then separately put some sort of admission protection, web hook or cell or whatever it is.

C

E

But actually not really only this one field, so I.

B

Don't yeah, I don't like that either and that's why this one doesn't doesn't layer restrictions on people that already have an edit permission. You have to have a special permission, so it probably would be configured in in two places.

B

I think that's unavoidable, but at least the the like permission that you have on the resource tells you like. You need to go. Look at the per field permissions to see what this person can actually do.

E

Yeah the implementation has to be in two places, and we can like be diligent to make sure that I'm talking about more how you express that, like how you program that in that program, but how you uh yeah, how you declare that permission to have this and.

B

Then we don't really have a choice: we're just going to do regular, auth, z checks. So all we can like we've got the resource information and the verb and the the the subject. Is that the right word uh and that's like that's like all you can that's all you can get out of a and it gives you a yes or no or a deny, allow. I don't know uh triplet right, but that's all you can do for an aussie check.

B

So we've gotta we gotta like um shoehorn everything into that template, um which is kind of interesting.

E

Right, but we can use things like that. We won't register that authorizer, unless you also have the corresponding admission hook on like like we can, we can make it hard to like so.

B

As as written as written, this is not something that you would put in the authorizer chain. This is extra calls of the authorizer chain, uh so that would always.

C

Happen, I think you'd have to add an authorizer to the chain that allowed this to happen. I would rather not see the second check you make as an admission plug-in I'd rather see it in the same spot, but exist on rest storage like be unturn, offable.

B

um It's been a while, since I looked at that code this so maybe I'm using the wrong uh the wrong words. I thought our authorizer chain was a set of authorizers which could fall through to the next one, but eventually you get a a low or deny answer right.

C

Yeah but the secondary check right. It has to happen at the point in time that you have both the old object and the new object, and that is the like: validate update method that gets passed to storage. As I recall,.

B

um Yeah, but uh why can't we call the? Why can't we call the ordinary authorization uh chain for that.

C

It could call the ordinary authorization chain, but I'm saying instead of saying here's an emission plug-in you can turn on or off like yeah.

B

That's what I'm saying this is not a new. This is not a new emission plug-in. This is this is uh like the configuration doesn't go in an emission plug-in and the calls don't go into an existing admission, uh plug-in or a new one. They go into various places in the code.

B

Okay, at least that's how I'm thinking of it like I could be. I could make stuff up, but.

F

So I I think uh one of the things jordan said earlier isn't necessarily addressed and then well. It could be addressed in any of these, but I think jordan, what you were asking for was a way to express the permission to do this conditional update without also granting regular update.

F

And I believe you could do that if the authorizer chain was set up in a way to like, I guess, can like tell storage that hey you're not make sure when you're in validate update. uh I allowed this to get to you, but only under these specific conditions being true. So.

B

F

B

So you're actually saying you want some sort of guarantee that if I give this person a you know per field verb that they may do, they don't also have the regular update permission on the regular update, correct. I mean that's a problem. Our our system has today right, like you.

E

B

Give somebody namespace uh update and then uh like block them from a particular object and like right, like you.

E

B

They disagree with different scopes.

F

E

Find green permissions like, on the one hand you might be, you might want to say, like you, can you can update deployments, but you can't set liens on deployments like because lien setting is like something I want to limit. So that's one one way to frame it. The other way is: oh, you are a lien setter. I want you to be able to remove your lean from deployments, but I don't want you to be able to update like anything in deployments. That's crazy.

B

So so uh you we can actually uh kind of turn it around. uh That might be an option which is like okay, forget about the initial thing and use negative permissions, um which is uh option b there on the screen, so like you're only allowed to edit spec. If you don't have the no spec permission.

C

I'm definitely a positive permission. Guy uh I've gone through this path once before, and I could not write the code to express uh our back in this very first iteration. Our back actually had allow and deny, uh and after a day and a half, I couldn't make it work so um yeah, I'm positive permission. Only.

B

uh So I don't see, I don't see how to address jordan's.

C

B

A

Positive commission.

C

I think the positive permission would be uh would be. You have the ability to set liens and if you just had general update, spec capability or update capability you'd be able to set. I see, okay, I finally got there. You.

B

C

B

I want to know that I.

C

Care a lot more about the second story. As a lien setter, I only want someone to set liens.

C

I don't feel as strongly.

E

Abdullah commented on the thread that you started: daniel uh scheduler has a similar uh yeah.

B

I looked at that yeah I looked at that. uh Is it a cap that he linked and yeah? Indeed they have a field that they want. uh Basically, the same thing for.

E

Yeah, they want to say like until these things are true, this pod isn't really ready to be scheduled and then they want to have like distributed actors saying like yep. I made this thing happen to make this time happen and only be able to set like their yep, I'm schedulable now bit and I'm trying to figure out how you would express that without doing that over granting like yep, you can edit anything in the pod and then somewhere else separately later say, but actually only this.

F

B

The problem isn't the over granting. The problem is knowing that that you haven't like given them edit, somehow through some chain of our back uh or other configuration.

F

Like if you have a net new field, can you not make it so that it always requires the net new permission check for it to work so like? If you have update it means you can update, but not that one field right.

E

Now the issue is for them to send an update request to change that one field you have to just give them like pod status, update or pod update.

F

No, what I'm saying is now so.

E

They can change.

F

No, I'm not saying we allowed that. I'm saying that if you have both update and update lien, you can do anything you want. You can set liens and change the whole thing. If you have just update lean, you can only do update lien. If you just have update you can update everything but lean or whatever the other fancy. Things are right, like they're bus, but there's just there's positive permissions. You can have them both together to do all the things you want, but.

E

I don't think how granting update lean like mechanically works, to allow an update request, but scopes it to just updating that one field.

F

What I'm saying is we expand the code for authorization to inform storage that hey I let this through, but you need to make sure that bad things just happen.

B

What is written here for option four, is you you get, is almost exactly what mo is saying right? You do this check uh the regular edit you don't get, then you do a you know.

B

Okay, do I get granular edit, some sort of granular edit that comes back true, so you've done two off z checks uh and you set something like in the context or something about something that goes with this request: an api server that like hey, you need to do the granular check later on, uh then that goes through and it comes down to the storage layer and it sees the old object.

B

The new object and this bit that, like hey, you gotta, make sure that this changes is authorized and then it goes through and computes the different fields and computes which set of uh authorization checks. It needs to do to let this in, but that still doesn't solve jordan's uh question of like how do I know if I've got this per field uh permission? How do I know that this person doesn't already also have the edit thing, but I think that's just a general fact of how our back or how how our authorization system works.

B

I don't know if I can fix that with this.

E

Yeah I so mo like to dig into that example, just a tiny bit. So the first check we could. You know you come in with an update request and we say like do you have update permission?

E

If you do like awesome, you can update whatever like go through and if you don't, then maybe we've got like eight like labels, annotations, liens, finalizers owner references, maybe like conditions like I don't know what else um we've got like six fine-grained things, and so we have to check all six of those and if you have any combination of those we then let the request through and say, but only if it's one of these, like three things that you have permissions for.

B

That's not what's written in that's not what's written as indesign four uh and design four is like. There is a single permission for update everything, there's a single permission for do granular checks and then in the storage layer. When we know the old and the new object, we compute, which fields have changed and we can search for uh uh authorizers which, like like we can we can. We can have a name for each field like like a programmatic name, and we can query authorizers to see if they have permission for that thing.

B

So if you've changed three fields, uh it's three off z, checks, yeah. I think there's.

F

But they only happen when you're all the way past admission and everything and you're just at storage right, you initially might do one or maybe two checks depending on the situation, but actually, um okay.

F

I I, while I'm not like ecstatic about the code complexity of any of this, I I find that it will probably be much easier to explain to someone that doesn't work on this code base with us, which is that this thing logically maps to this field. This singular check- and maybe you need this other marker permission- to say that those singular checks have to go through or something.

F

Maybe you can come up with a design that doesn't require that, but I still feel much better than saying: hey give everybody update and try to cut it back with admission.

B

Yeah all right, um I think, we're out of time. uh The plan is to hash this out again in next week's sig off meeting. So I.

C

Like uh for the record, I like option four, uh and I like four uh as a as a developer. I like 4a much better than I like 4b and as a user. If somebody else will take care of it for me forever.

C

I kind of want 4b as well.

B

Okay, so these are straw designs, so uh maybe we can write this like. Obviously this has to be a cap and, like you know, it's going to be a whole thing, so um yeah, okay, thank you.

A

um Thank you, daniel hi, um so david. I think you're next.

C

Yeah, I just want to make a uh we have to add the caps that we're gonna do for 126 before too long. uh So, if you want your cap included, you know bring it up on the mailing list. Bring it up here, actually probably bring it up on the mailing list uh and we can see about getting it added make sure it gets review bandwidth and such.

A

Because the process has changed right for 126 now it's uh different from there's.

C

An opt-in, yeah and the opt-in is coming up in three weeks. I think.

D

Just to be clear here are we talking about prs that add and change caps, or we talk about implementing caps.

C

We're choosing the set of caps, we're trying to target at a release and which ones will graduate or not.

A

Okay, thank you.

F

Yeah enhancements freeze is on the sixth of october, so you don't have much time if you want to change something yeah.

C

So far, I know of self for admission for alpha, I think, uh and if daniel wants to do something for sub resources or fine grain permission.

B

uh I don't have anything planned yet um uh yeah. I was just trying to get an answer for this poor fellow.

D

All right, so we are for a api paradigm, fairness trying to land this kept change to add limits on the borrowing and we are trying to uh get the implementation going. uh So can I again beg for some review on the kept pr for adding the limits on the borrowing um and yes,.

B

uh Yeah chad uh chatted at me. I I wasn't. I wasn't uh aware of that one uh I did. I did comment on the api one uh yesterday. um Thank you.

B

Let's see all right last uh last topic.

A

um Yeah nothing. It's just me saying that um coupon north america is coming, I'm not sure who is going, uh but we have a deep, deep dive session for api machinery. It has two topics. Thank you, david for taking care of it. um Jeffrey uh we'll also cover the second one. um So we have aggregated cube api servers and then open api v3, and here is the link for the session. I don't know if this is open.

A

Maybe if you're a virtual attender, you can also look at it, but then it will be public, so just wanted to advertise. It.

D

Do we want to add the api priority and fairness, borrowing into that it's kind of a big change we might want to socialize.

A

uh You mean into the talk seems to be a little bit late right.

D

C

The target is already set and we had to prep the presentations. So all.

D

Right that answers that, thank you.

B

All right: well, thanks for joining everybody, looks like we're going to end on time for once.

A

Thank you thank.

D

A