Kubernetes API Machinery Special Interest Group, 19 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20221019

Description

[andrewsy] Discuss kube-apiserver identifier format (KEP-1965)
[lavalamp] fine-grained-authz KEP pre-review (follow up from Sep 21 discussion)
[fedebongio] Virtual Session at Kubecon NA - API Machinery deep dive: https://kccncna2022.sched.com/event/182Mo

A

Okay, it's recording and good morning, good evening good afternoon, depending where you are. This is um kubernetes open source Sig API Machine by weekly meeting today is October 19th 2022. We are heading right into Q4 of this year. It flew by very quickly. uh We have two great topics today and I want to thank the people that helps us every week. You know have in every two weeks interesting things to Wisconsin, so Andrew um you're, the first one I will give it up to you yeah thanks.

A

um So a couple weeks ago uh we updated the cube, API server identity cap um in hopes to try to land that for beta and 126., and one of the criterias we added was to reach consensus on the uh the lease identity format.

A

So I tried to articulate all the options we had discussed on stock in the the pros and cons of each approach and the agenda notes um so I think after getting more familiar with the code I'm personally in favor of um the hostname one, the cube API server, Dash hostname uh for three reasons, uh the first one being that I don't buy the argument around wanting to run multiple API servers on the same host, and even if you did, you can work around that by assigning different pod names for the cubelet managed, um static, pods.

A

So I feel like there's like a reasonable prompt for that. Second being, if we're really concerned about uh hostname exposure, there are already places where the lease identifier contains host names, such as um like with leader election with controller manager and Cube scheduler, uh and that's the um after digging through a bunch of the garbage collection logic. It seems like creating um and requiring a garbage collection of leases for every Cube.

A

Api restart does seem expensive, because every time you garbage collect the lease it requires a realist of every storage version, object and updating its status. To remove the least that no longer exists and I know mo and David had some additional thoughts um regarding the KMS meeting thesis.

B

One second Mike, uh the the thing with two API servers on one system: I'm, pretty sure that we had at least one person who really wanted to do that uh to the extent that they added a feature to let API servers share the port. uh The reason they wanted to do that was uh that they wanted to reboot uh or, like start one and then stop the other on us on the same machine without having a uh outage.

B

um But it has been quite a while I, don't know if they still do that, but I assume somebody does next.

A

C

I just wanted clarification on the solution, for that I'm. Not quite sure, I followed is the idea here that the lease name is actually the Pod name.

B

Oh uh kubernetes uses the Pod name as the container's host name by default, unless you set the hostname field in the container, in which case kubernetes uses that.

C

ah So we're talking about the host name.

B

But if you run API server in uh it's like systemd or something, then you don't get that feature.

C

So specifically, when we say hostname here we mean the hostname, oh gosh,.

B

So API server uses the the syscall to acquire your one's own hostname, um whatever that is.

A

All right so I understand the solution. Now.

C

D

uh Daniel on the person that wanted to like reuse, the same port here in that case, is that user expecting the host name to be the same between those two things, because it's like.

B

They're, like I, don't think they I, don't think they care about the hostname. For that use case I, don't think they cared about the hostname.

D

I I guess I'm trying to logically understand what would that user expect to happen? Would they expect a new lease to be created for the new API server, or do they consider those.

B

Like oh yeah, that's a good question I. If I were doing that, uh I would consider uh the second API server to be an extension of the first API server and give them both the same name and hope that they didn't fight too much during the brief amount of time when they were both alive. That's.

C

Awesome uh so so hang on thinking that, through with KMS or or even with storage, uh we think it through with storage and it's gonna be careful I'm I'm, not confident that does good things. I also I also think I'm, probably okay, telling a person. If you wish to Run Two at once. We would expect you to give them unique names, um and even if we didn't support it as an argument, I I would find the setting spec hostname work. Is it pod, spec, hostname.

B

C

A

B

I think it's not that hard to uh to set that, even if you're not running inside uh uh cubelet.

C

So one of the things that I was interested in related to this is: are we going to go for graceful, lease release and I? Think we we do not want to delete our release when we shut down. We want to leave that up to GC. We we did settle on that. Then I know I confused the issue by bringing it up, but it didn't exist. The first time we did this cap uh and- and it does exist now and so I just want to have an explicit like we will not gracefully release I I.

D

I think if it would be a poor choice if we were going with stable identifiers to put the lease down actually.

C

Can we can we release and not delete it I think we can.

B

So um uh I I think we should start without graceful release and, if so, the case in which that would be useful. uh Like imagine, you're scaling up or down your control plane by adding API server replicas, uh and you know that you're removing one and it's not coming back. It could be handy to just remove your lease completely uh and instead of letting it be garbage collected. But it's not that big of a deal and I think we shouldn't Implement that unless, uh unless a lot of people ask for it,.

C

Okay, then I think I also like Cube API server, with a hostname, giving us a fixed name.

D

So are we leaning towards basically no config knobs and when the API server comes up, you know, however, you set up your environment, whatever hostname that process sees it was going to try to claim if you've done horrible things and there's multiple that are going to fight for it. I don't know.

B

D

Have it here that tells.

B

You that we're exciting or if you've put pii in your in your host name, um that's going to broadcast that to everybody. Yeah.

C

I think that the that's.

B

Already going to happen as.

A

B

Oh, it does, does it already happen in some way.

D

For the scheduler and the controller manager, maybe not the API, oh.

B

Do they put that in there I see I see uh okay, they might be in different containers and not have the same host name as API server.

C

uh In terms of detecting this went bad I, remember one of the prr answers was. You will be able to look and see that the Post-Star hook did not finish, and that seems valid to detect someone who had a problem right. They they had a problem, their API server was, oh, you wouldn't be able to help fight. They would both think they had it.

D

Oh also, what's the expected Behavior, if, like an API server crashes, and it's immediately restarted by whatever Watchdog is trying to keep it up, should like immediately try to hold like? Is it like, just not cool that that lease was technically lost for a little bit, because the process does withholding it got, killed somehow and well? I think the point of.

A

Lease is they don't do.

B

That yeah, we just decided we don't want to like try and gracefully shut that, like the the point, is to keep the lease operable to reduce churn in the rest of the system. uh So but no, the point was not about graceful right. He said crash, so there's no question of graceful, but.

A

The point of a lease is that it's a timed, you know time thing you take it for a while. So presumably a brief crash in recovery will happen, while it's already held anyway, and you.

B

Don't need to do anything right.

C

That's what I would expect as well, and that would be another reason to have this fixed name right.

B

And the principle is, we want to reduce churn in the rest of the system.

D

Yeah I mean I I. Think the human ergonomics of a fixed name are are nice. There seems to be at least you know some quantifiable benefit to just, let's not just recycle, a bunch of objects with basically no op updates that do actually change stuff and cause people to care and maybe run storage migrations or something and just you know otherwise, waste io on the system for no good reason. uh So all that seems like good reasons.

D

Daniel. Do you remember why you went the kept said it was going to be fixed? Then the implementation went to uuid and now we're saying we're gonna actually undo that and go back because I'm just so.

B

I I vaguely I vaguely recall having a conversation with Highway um I think we were worried about uh exposing the hostname that we weren't sure that it wouldn't be pii.

D

If we want, if we're still concerned about that- and we don't.

B

I'm, not I I, don't I, don't think I'm concerned about that. Like we're, gonna we're gonna document this.

D

I I guess I would ask if, if we want to avoid maybe having to care about documenting it, we could still use the hostname just hash. It first.

B

But then you lose a human error about ergonomics, yeah I.

D

Mean you, you get still the stability you just you just don't get as easy understanding of the map. Yeah.

B

I mean and and uh right it doesn't have to be the system host name right. It can be the container the host name reported by the container API server's end, which people can set without too much trouble, even if you're, not using uh uh qubit so like I, think that's as reasonable a way as any to set this bit of information.

C

It seems reasonable to me: I am flexible for for either one I agree that they would handle. My stability concerns.

B

I mean if you, if you just imagine like a random process running somewhere and it wants to identify itself to other other processes in the network or whatever a hostname is a pretty natural way to identify yourself.

D

And and we're just okay, uh we talked about it just a little while, but we're okay. If you accidentally give the same hostname to separate, distinct processes that you can't actually tell like, which one has it actually, they both think they have it. I guess.

A

B

uh Because they both think they have the same name.

D

They both think they have the same name and it says they're they're holding it so they'll be like cool I'm holding it and that's fine.

B

Okay, yeah. Maybe we should talk about that scenario a little bit more because imagine somebody's misconfigured it like they put they've hard-coded, a a host name in their pod configuration and they blasted the same pod configuration to all all of their uh control, plane, nodes um and so they've got like three API servers and they all think they're called the same thing. uh That's configuration error, but we probably it would defeat the purpose yeah. So so we probably need to make sure those things. Don't think that they're called the same.

B

um That's a good point.

D

I so I I think somewhere in all of our designs, or at least designs related to running storage migration. There's gonna have to be like they're told, like the expected number of API servers. That's at the current point in time. I think you need to at least know that to know that everybody so.

B

There's a present uh I forget exactly the format of leases, but uh suppose you could add, like I, don't know a label or something with a uuid uh and and it would be fine for that thing, to change every time. It's just if you. If API server, looks at the lease and it sees that it's got some other UI uuid, it overwrites it with its currently generated one, and uh if it keeps having to do that, then we know that we're fighting over the lease with some other system how's that that at least.

A

D

On on a restart, there would be churn like right.

A

So turn on the.

D

Lease, but not on any other object right.

B

Okay, so the idea to put it in something: that's not the name, so the name stays the same, and if people fight over this label um I mean you've, got a you've got a problem. If, if multiple API servers think that they're the same thing.

C

Yup I'm just trying to think of myself explaining this in a way that doesn't say that the label that you just described was actually another lease.

C

Right, like I, have a label I'm going to look at this label I'm going to make sure I don't fight over it with somebody else. If I am fighting over it with somebody else, I'm not going to use I'm not going to write my data out.

D

What if it was like much simpler logic, which is that, if you're doing an update to fix that you just increment a metric counter, and that's it like you, you keep doing it, you don't actually change your behavior at all. You just say hey. This update I did at least part of the reason I'm issuing this update to this lease is because this random uuid is different and I increment. My metric, and if you see this metric, going really high up, uh you know it's not working as you expected.

C

Okay, I think I think that might could do.

A

Yeah we're not saying we need to change the actual like mechanic of the lease holding we're just saying. We just need to know if two API servers are fighting.

B

Yeah yeah observability is the first is the first uh step. uh I guess the question is: if we need to do more than that or if making it observable um is enough, I mean.

D

C

D

C

For beta and then we'll figure out when we actually use it, if we need to do more.

A

B

C

Enough Mo, or is that not going to be good enough for your purposes.

D

I know I I think that's I I'm not worried about this, because I will not make this configuration mistake. I am because I'm aware of it so I'm not gonna. Do it I I'm more trying to help others understand that they've gotten here and so like metrics, you know if you're looking you'll notice, audit events, if you're looking you'll notice granted leases might be Exempted from audit events, because they're noisy.

B

um Yeah I think they are, um we can I think API server uh if it notices this number increasing, rapidly needs to uh put a warning or an error in its own logs. People do sometimes look at those.

C

Yeah, the metric is an easier thing for people to find.

B

Yeah the metric is is easier. Okay,.

D

No I I think I'm comfortable, saying that hey this is just a misconfiguration. It's just. If you give someone a reasonable chance of finding it I.

A

Hope you didn't notice.

B

Foreign you've got to give your API servers different identities or, uh unless.

A

B

Like deliberately trying to do the like takeover thing that I described but I think that's going to be a pretty rare case.

D

Well, for like the purpose of like storage migration, regardless of if it's for key rotation or storage schema changes does uh does any of this need to have like the number of expected API servers so that way on the outside, if you're, looking at storage version and there's only one API server, because the threes are configured have the same post, name I'm,.

B

Fairly against trying to program that, in like the original coordination mechanism, the API server used to put its endpoints in the endpoints table was uh based on everybody added their own IP. If it wasn't in the list and truncated the list to uh the number that they expected uh and basically everybody hated that system, uh uh and so we went to this like uh yeah I mean this. Is this is kind of a evolution of the desire to not use that system and have API servers figure out how many of them there are.

B

um If we think it is going to be hard for people to name their API servers, distinctly then I'm, not sure yeah I, don't know.

D

Yeah I think the only concern I have is, if you were in this misconfigured State like the encryption arrest stuff might like. You know you might trigger rotations earlier than you should, because you think the servers have converged, because you think there's only one right like one of the things I had in. At least you know, some of the alpha reasoning was like, like the the controller, that's gonna like kick off.

D

This thing has to know the number, because until it sees that fixed number and they're holding for some amount of time, it just doesn't do anything because it doesn't know if they're ready for the change. uh I don't know if that's okay.

C

A

C

Something you would expect to build into the system that you're writing to manage the keys. Or would you expect to build it in here and if the cleanup, how long would the cleanup have to be given stable names for you to have confidence that the server you're waiting for was never coming back.

D

Right um so I I think when I originally wrote some of that there was some specific numbers. I had written based on um forget exactly what, but.

D

uh Andrew, do you remember how long these nieces are held for, like I guess like if we I think it's an hour, because if we hard code that that number, instead of making it configurable and make it something you can rely on?

D

um And we are okay with something probing the leases based on that label. And thus they can like with something that's not built into the system. Necessarily uh doing that probe.

D

um I think it would be fine right, you'd be able to find the leases, um and then you know if they've all been held for some amount of time. That's greater than small number of minutes, but maybe less than an hour. I feel.

B

Like that's, okay I mean, depending on how worried we are about this. If API server detects the condition uh we could make it not like- maybe not right to the the storage uh storage versions, uh object.

A

C

The risk of making this more complicated are we okay splitting the lease name from the holder, identity and saying that for lease names we use qvpn server for holder identity. We have units.

C

Foreign, that way, you would have something that was persistent, but it would handle the the restart case fairly gracefully. It would effectively be that second label, you were talking about Daniel yeah um and it would cause the post start hope to like delay indefinitely, which would hit the thing that Andrew had already said. You can look at this and say if you don't go ready. This is how you know it why it has happened.

B

Yeah yeah, basically we'd be running a instead of one. uh Basically, we've been running a leader election per API server, name.

D

That seems, okay, now, I think you get better ergonomics right because, like it doesn't it's unlikely to come up and become healthy if the fighting is happening, um so it kind of dies loudly and then the person goes.

B

And yeah yeah that would be kind of.

D

B

uh If, if a failure mode for API server, it's like I can't I can't I can't lock my name so um I'm, just not going to start. That makes it not not work in the. uh You deliberately want two things to have the same name, but maybe we shouldn't let people do that anyway. S.

C

I, don't want them to do it anyway, just because I am I am concerned that I'm a little concerned about the storage version, because that would reduce that would create, could create churn right.

C

You do storage migration, but you don't need to I'm more concerned about the KMS usage where I decide, I'm okay, to remove something and now I can no longer decoup my data yeah right so so, and I'm willing to bet that the person who wanted to do that is Stefan he's not here, um but he would have wanted to do it to reduce API server downtime during graceful release. Timing.

D

So, let's define when I do that for openshift David like like. Do you need this.

C

uh You know if I'm, if I'm, choosing, which problem I am going to have uh I, am likely to say, I would rather have a reliable way to express the identity on the server than a convenient way to.

C

To run on the same port.

D

Yeah I mean like I, guess, uh basically what we just discussed I think I'd be comfortable. If that was like updated in the captain. We just said yeah: let's go. Let's do that.

B

Yeah, okay, okay, so uh on a restart I, come back. I see my old uh lease with my old holder identity. How do I know that that old holder identity is a former iteration of me and not some other API server that I'm fighting with.

A

C

A

On the lease object, right, yeah.

C

So the host name on the release object gives you the right lease that you're going to acquire I. Think if we did this, we would actually use a graceful release that does not delete the lease right. So the lease object would still be present, but we would remove our holder, identity and.

C

B

When you okay, so what? If, what if we crash and restart, then.

C

You have to wait for that. uh The the.

B

That's real that fire. That's real bad, though right, because it's not good aren't. We aren't these hanging around for an hour.

C

uh No there's two different processes. One is one is saying if this name, if this lease name disappears, I have cleanup to do on the storage version, and that takes an hour for me to decide to delete the entries from the storage version. Okay, um then we have a separate piece that says: uh I hold this lease as the holder for X many seconds and.

B

C

And so you know, X could be.

B

Okay: okay, no, that works good. That works good, so we're gonna hold it for like 10 seconds or 20 seconds or something uh we take that long to start anyway. So.

A

I still think that on every restart there is going to be some system turn because we use the holder identity in the the status of the storage version API, but uh but like there's, not that many objects. So it's probably not a big deal well,.

C

Sure you can change that older Identity, or should we use the lease name right yeah we.

B

Should use at least name? We should change that. Well,.

C

I guess I guess.

A

Okay, I added um some action items I'll take at the bottom of the agenda, so I'll send the cap here. Yeah.

B

Yeah this is so we've. We've we've effectively changed from a model where we have like three names uh in the cluster to a model where we have three hats, and uh each hat can only be one by one name at a time.

C

B

I think that's I think that's an improvement.

B

Okay, great, uh should we go to the next? Yes,.

A

um Thank you Andrew for capturing uh some of the discussion um and then uh Daniel next one is yours. Do you want me to open this one.

B

Sure um all right I will open my open it on my screen too. So I can, let's see uh so we talked about this two meetings ago um and we talked about it in the Sig auth, the meeting after that, and uh basically the designs. The last time this came around uh didn't didn't fulfill a second obligation, which is uh the ability to add a new field without including that field in the existing broad permission for rights, um and uh so we've come up with a design that addresses that and is slightly simpler.

B

B

The other concern that uh Tim Hawkins had uh was that if we don't want a system where we changed version- and that implies that you need different permissions because that's too much burden for system administrators, so the basic design now is to add to schema uh per field, a tag describing the permission for that field and whether that permission is uh like, uh like you, can grant that permission to anybody, and then they can edit that field or if it's, even if you have the regular permission, uh you still can't edit that field, unless you also have the special permission.

B

So this is the two modes um yeah, so it becomes a part of the API design to describe what permission, uh if any, you don't have to have one. What permission goes with a particular field, and this means some implications uh for like how you describe the permission. So instead of uh let's see where's that section instead of A system, that I was describing before, where the permission is generated from the field path.

B

um Now, basically, there's a style guide for how you set up a permission. uh I, the one I made up in the cap, which is completely open to change at the moment, is the permission. Name has to be a noun should be in a lower camel case. uh It should be referred to like a conceptual attribute, not a field name um that helps you reuse the permission in other contexts.

B

uh Right, like you, can imagine some permission, making sense for uh I, don't know like both pod and and like some sort of networking thing or some sort of storage thing right. So you want to grant that permission uh General generally over both of those types uh instead of generating an additional permission right that the more permissions you have the more work it is for system administrators to figure out what is going on and administer this thing um and uh uh the permission shouldn't reference, all the ancestor Fields right.

B

So uh instead of saying you know like pod.spec Dot, uh node name, you should just say like node assignment and you get a hierarchical system by assigning uh permissions to the like, so like spec spec has a permission associated with it and that if you have that permission that covers everything inside it right.

B

um So that's how you get a hierarchical system, um yeah, uh so I I, don't know um I, don't want to go through the whole design, because it's kind of long. But if anyone has questions now's a good time, David.

C

Can you go through an example of how you would have and exclude of a label pattern from the generic privilege so so I want labels with start with a pattern uh security.openshift.io to be excluded from the generic update.

B

uh In the design is written, you can't do that. People were only asking about new fields.

B

uh I didn't even think about uh labels with specific prefixes as being equivalent to new fields um and since the uh since the permission system is now going to be part of the spec and since the built-in the built-ins, including metadata, are shipped with uh kubernetes. uh There wouldn't be a way to configure that, if that's important to you, then we can see. If maybe we can.

C

I figure out a way you know, I was trying to figure out how to do it. I thought I'd be able to do it. uh What what if I said, I want to have an exclude of certain patterns on liens that.

B

Would be possible because it's a new field, uh oh.

C

Certain patterns I only want certain patterns.

B

uh uh Certain patterns.

B

I'd have to think about that uh that wasn't the request that I had been anticipating um the problem with exclude yeah it's difficult to support both include and exclude on the same field.

C

Okay, I'll, let Mo go and I'll think about whether the include and exclude on the same field is actually the problem that I'm running into in my head.

B

D

Well so, let's see I had a few things, um and you know some of this is like really small, like on some of the implementation details anywhere that you had about how like some of the wiring would work. I I think I would want the wiring to be written in a way that, if it's using something like context to pass state that it always passes some State, and if that state is missing, they just hard fails as an internal error. Saying I, don't know what checks I need to make instead of like. Oh.

B

D

It must mean it's.

B

Like the older, no yeah yeah, that's great, that's great! The the you know I. This was this is like the minimum viable uh design to get the point across. That wasn't intended to be complete in that detail. Yeah, but that's good feedback. I'll put it in when I do a revision. Whenever.

A

B

D

The the on the part about where you're like, actually it's like, really it's actually uh almost on screen right now, where it says, uh marking existing Fields. This way with break existing clients, uh because you know oxy is fully configurable I. One of the ideas that David had shared last time was basically retroactively, adding that stuff to like the bootstrap rvac policy to then allow and admission.

B

Yeah I think I actually mentioned that uh we can. We can. We can totally solve the problem for our back right, like we can take it away from the general permission and add it in our back. The problem is that doesn't help people who aren't using our back to configure their systems and uh yeah so.

D

My thought on that was there's two things: one. We could just hard to require you to start using our back. So that's that should just be a thing, but fine. If you don't, if you don't don't want to make that strong assertion, we could we could make it so that when our back is not configured, there's just an a new authorizer at the end of the chain that takes those Legacy permissions and just allows them to happen.

D

So that way you haven't lost anything the permissions like you were already allowed to do it in the old system.

B

B

um When it sees a request for a new permission, it translates that into a request for an old permission and asks the like goes through the chain again or something yeah.

D

It could I guess it's not something like that. I haven't, but something too retroactively like sort of do it the other way.

C

D

C

Mo I'm not sure that that handles a case where someone was using a web of authorizer and the web of authorizer would return no opinion, but nothing else would Grant permission. So we'll get a default deny.

C

I'm not adverse to trying to think of a solution there, but I.

D

um Yeah, it might be a little bit complex to get it to work. I, I I do think it's tractable and the idea of allowing environments to tighten this down sort of retroactively is appealing to me. So.

B

Along those same lines, uh one thing I've considered but haven't written into this is like if a system administrator does want to go through the trouble of locking something down extra and that's no problem for them, because they know how their auth system is configured and they can just add in the permissions to the people who should have it while taking it away from everybody. That part is easy. The problem, the thing that this design doesn't address is that, like the uh the schemas for the built-in types, are the same for everybody and not changeable.

B

uh So if we want administrator to be able to lock down labels in a special way in their system, um we would need like a way for them to override the uh the tags that we ship on the built-in schemas or something like that um which I don't have written down, because I don't want to do that unless it's absolutely necessary.

D

I'm not 100 sure if I just followed what you just said.

C

um There are three cases of type smoke, so there's there's one where you are defining your own schema. You have a field on your schema and you're going to decide. You need a permission to set this so.

D

C

Is the case where you are reusing a schema somebody else to find so you're either reusing a pod spec uh or you are reusing, maybe meta V1, which is very common, and you want for your type, your embedded version of meta V1 to have different rules and then there's the third case where you're a cluster administrator. The person who has defined the schema did not Define it as you wished.

C

It would happen, and now you as a cluster administrator, want to lock down fine-grained fields that were identified in the schema, but not identified in the manner that you would like.

D

Thank you, yeah. Okay, I follow now.

B

Yeah, okay, uh should we let Tim uh say something: yeah.

B

Do uh do this bread first Tim.

A

D

I'm concerned a little bit about what the user experience on this is going to be. If you have like multiple different permissions needed for different parts of the request- and you know maybe some are deny- and some are.

A

You know like the fail, opener fail, closed.

D

Kind of approach- and it's maybe.

B

So, okay, yeah, let me clarify a couple things. uh One is part of the point of putting this in the schema is that we will be able to generate a list of all the permissions that there are in the system uh and you know link back to the types that they apply to uh which maybe one permission May apply in multiple places. uh uh Okay, so so that's that's one, the other. The other other answer is like. Let me be a little bit clearer about how the the deny system work works.

B

It's not a deny system. There is no denying the the the the two options are um either okay, so uh yeah. Actually, let's just walk through the. Where did I write this the overall flow, which is yeah so the first step, a request comes into the API server. The first step is we check the do you have put patch create you know? Do we do you have a mutating permission?

B

uh If you do, that's, that's the regular permission, we'll mark it as such in the context or whatever uh like Mo wants, and um if you, if you have that, we don't need to check the specific field permissions, except for the ones which are marked as exceptional uh uh yeah. That's uh that's uh step. Three I think yeah. If you don't have, on the other hand, if you don't have the regular permission, then we're going to check the granular permissions for everything that you are attempting to change right.

B

So uh it's like the regular permission gives you you know 99 of the fields and for a couple Fields. There may be yet another permission that you need, but if you don't even have the regular permission, then you might still be able to make the change. If you have, the special permissions I feel like I, still didn't, explain that very well. But hopefully the highlighted bit on the screen makes it clearer right. But at no point are we looking at what you have and then denying uh that.

B

So it maintains the current uh it's like uh positive framing of our back.

B

B

Yeah, if you can think of a better way to describe this, then uh uh yeah feel free to. uh Let me know our Sunday Sunday uh poll to my Branch here.

A

D

D

If you like, let's say, there's a built-in type like pod, V1 and pod V2 are we are we saying that we, the authors of this, would be very careful to make sure that semantically, equivalent Fields require the same permission, and if we don't do that, terrible things will happen. Yes,.

B

And uh in order to help people do that uh the documentation is going to have to make a list of the fields, um so you change the AP API type, so I. If you make a conflicting field, you would probably see in this list. uh If it is reasonably organized, you would probably see uh that you've made a conflicting permission if you fail to add a permission that makes sense in the context of the field. I, don't know that, there's anything that I can do to help you with that.

B

um uh So there there is actually a harder problem with this, which is uh crd, authors.

D

B

Okay, if everybody goes off and makes their own uh their own set of permissions, uh it's not likely to work together and somewhere near the bottom of this I have a set of ideas on how we can do that. I think um I think it's below this Friday.

B

If you keep going down a little bit um yes by the style guide, um yeah so I suggested either uh put a registry file in the in the repository for third-party authors to you know just make a make an entry there sorted alphabetically, so you don't get too many contact conflicts or something, uh but go record that you're using some permission.

B

What it's about to make it possible for other people to use it, um and the other option is third party authors prefix their permissions with something uh unique to them like namespace, and if anybody has better ideas than those I would be happy to change them because I, don't think those are very good ideas, uh but yeah I think there's some there's some sort of problem.

B

There I think the build-in types are fine, because we can generate them from the docs and put them in one place in a list and I think humans are fairly good at like looking at, like we just don't Mark that file as like generated. Don't review, We mark it as uh please review the changes to this file, even though it is generated and I think people are going to be relatively good at noticing that you've, you know, made a duplicate permission or failed to use uh some already existing permission Etc or have produced a permission.

B

That is a bad one, because it's only ever going to be applicable in this particular case it will be impossible to reuse and so you're, just cluttering up the space. um I think that's something that human reviewers can do. I, don't know how to do that for random series.

D

D

The thought I have in my head I have absolutely no idea how to apply to crds, but for a built-in API. Since we do go through the like the Hub type Could, you actually Define the permission on the Hub type and then have it actually funneled back out to the associated Fields, oh um or Associated concept.

B

I think it's not that easy. uh You certainly can't get the same permission applied to different uh Kinds by that mechanism. You might be able to take care of different versions, but I think I I think it is less magical to just have humans specify it in all the places where it goes.

D

uh So is the expectation that, like uh before this thing goes beta, we're going to manually annotate every one of them so yeah.

B

That uh the expectation is that we will add these to uh at least the fields mentioned in this cap, uh which is mostly uh metadata and uh conditions, and uh there's like a couple of fields in pod, um I think mechanically. We could also uh do spec if people think it is going to be generically useful to have a permission. That means spec and nothing else.

B

That that one's easy to do mechanically and after that, like I I, think that's enough to like get this off the ground, and if people want to add more, they can just add them over time and I. Think it's not gonna like if you. If you come up with a use case, it will no longer be like a big deal to do this. You just add the stuff to this to the schema and check your changes in and you're good to go right.

B

You don't have a lot of manual as opposed to sub resources, where it's currently quite a lot of boilerplate and some not so boilerplate code to make a sub resource.

D

If so, if you add this after the fact, we're implicitly saying that those new things cannot be special in the sense that you can retroactively say right.

B

You wouldn't be able to retroactively, make Fields uh excluded from the regular permissions um unless we come up with a system for doing that. Yeah David.

C

uh I've been trying to decide if I thought that there were cases where we would want to exclude labels or annotations from a default update permission or convince myself in either direction uh and.

B

I've been yeah.

C

I've been unsuccessful at convincing myself that it's definitely not needed, which is not the same as convincing myself that it definitely is needed. It just means that I'm I'm uh there's three cases that I described um earlier.

B

C

Exploring what those three cases mean uh and comparing it against say, um I can think of a couple either labels or annotations. We have on name spaces related to scheduling where you can say this is the default node selector for my namespace and you must fit inside of it and an update to namespace permission. Maybe shouldn't include that uh and that's one example. There are other ones that are openshift specific.

B

Yeah so like just translating that into the terms in this design to make that work, you'd have to make.

B

uh Because it's still a a like uh positively framed system, like our backs, you would have to like first separate somehow separate the permission for that specific label from the permission from everything else and then Grant it only to the person that you wanted and exclude it from the regular permission.

B

C

B

That doesn't work too well with this, like hierarchical system, if.

C

You leave a sign, the mechanics of of expressing the authorization choice and say somebody built an authorizer that can do this. What I'm, trying to figure out is, is whether the minimum amount of design that we have is simply a a. What, if we asked, does the user have permission to send any of these labels and we just list them all.

C

um It will be extremely expensive with the with a Web book API today, because you would, you know, list every every label on every request, but conceptually would it give us what we wanted and then, if we built a a mass authorization check where you you have like a slice and you just issue the slice, would it do?

C

What would that function? Would that solve that functional case and then could we decide? Is it worth the additional effort to do this or not and I'm not making a declaration here, I really don't know, but I think that's I think I managed to boil it down to. That is the question that I have.

B

um I think that would work. It would only be as expensive as the number of labels with values that are changing.

B

um That would definitely be like extra effort, because that's not a uh natural uh thing at the moment, I'm.

A

C

I would like to explore the idea first and then side. The trade-off either is or isn't worth that additional effort. I, don't know sitting right here.

B

So, uh honestly, if I had that problem, I think a more natural way of solving it is with a.

B

uh It would be much easier to describe that situation as a forbid thing than as a permit thing.

B

C

Would be more natural in terms of if you were expressing policy for it versus in terms of of implementing how at what layer? We would ask for the request where the policy would be written. For instance, the thing I just described that policy would not be written into the schema of the label right, because the idea is right. The king of the label is immutable for every custom resource using it right uh and that that may not be appropriate for what we have actually created. Yeah.

B

Yeah- and you also, you also need the the uh the parameterized permission to not be to not uh like be uh recursively, inheriting from the uh non-parametized permission right in your in your condition, you don't want a general permission to write labels to imply that you have a permission to write a specific label.

C

Yeah, um that is true, and there was there- was one permissions model that we we did explore. That would have allowed that.

C

But I will admit that that was that was very difficult to understand. uh The first cut of our back hat.

B

Yeah, uh let's see while phrasing it in yeah, I, uh I I think I have to think about how to actually do.

C

That can we can we describe the problem first.

B

uh I mean the problem is: is basically uh there's an unbounded number of possible labels right?

B

So the the problem is to describe uh like the problem is to fail to say yes for specific labels, while saying yes for an unbounded unspecified set of labels and.

C

It's made worse by the fact that the schema author does not necessarily know what labels or annotations are going to get applied. Yeah I, just haven't even been paying attention and.

B

You have to, uh and you have to like, also not give all the enclosing permissions. Okay,.

C

Sometimes you want it that way. Sometimes you don't right.

D

So I try to parse the problem that we're talking about to me. I think labels and annotations. If you step back and think of them as like, distinct kubernetes objects. What we're basically saying is hey I want the ability to at any time retroactively, say something inside of this object is now special I. Think, that's basically what you're asking for right, which is that hey it doesn't matter that the top level field says you can have anything in labels.

D

This thing here retroactively says you can't do that.

D

C

Don't even know if I would agree that it's retroactive I think the fact that the object meta is is immutable. For crd authors means that even your first creation of this can end up in a place where you have no way to describe it.

D

Right, I think that the I think the retroactive part of that is the fact that you can't there's no way for you to actually go far back enough because you don't control the schema. Okay, like you, can't actually go far enough to be at the beginning of time, because we, the kubernetes Developers, start the time already before you see the code, basically yeah, um okay,.

C

Then yeah I can I can agree with that phrase.

D

um I, don't I, don't know if there's a way to build this without some order deny logic.

B

uh Mo, you've actually made me think of an alternative system for describing these things like like. If we figured that okay, basically, uh a label is an object. uh What if we made the request where what, if we made the permission, request substituting the labels key for the object's name uh and like the object's kind becomes label, uh uh uh then it's easier to see how you would configure such a thing and your ah system um I. Don't think that it would be easier to understand, though foreign.

B

David, how would you what can I write down to help you decide if you really need to see if you really desperately need this or not.

C

Do you mind if I try to open a PR to your branch where I try to describe.

B

C

It yeah just just thinking like I I'm willing to do that to try to describe what's in my head and then then, once we can agree that that's like a that description makes sense than trying to then trade off like. Is it worth it to try to fix this yeah.

A

And I'll try to.

C

Come up with some examples, the the node, the node selector one definitely occurs to me because it's it's a spot where effectively, you know created something highlighted. uh There's uh there is it's a spot where, for instance, the namespace API different sigs have not been able to add fields to it, but they have added meanings for labels that are then used in admission and the you know the off Sig has done this for pod security admission, for instance. It's the basis of how that works.

A

C

Everyone have should access to to updating a namespace, give you the ability to override uh safety. Permissions on every node I mean probably not.

B

Yeah, so I I think you can actually do the thing you described with the with this, as as written you just have to like not give anybody the the enclosing permissions and uh like list out the permissions for every label separately.

C

B

uh Your permission system has to understand like a wild card right, so some people get like every label um yeah yeah.

B

If your mission system can do it, you can do it with this uh I think if we're gonna promise something like that, we need to make a a checker or a like a verification system to like uh for a given actor like do the check and tell you whether or not they actually have the ability to set a label or not, because it's too hard to configure and be sure that you've configured what you think you've configured uh mo one more thing and then I think we're out of time is.

D

There is there some way to make like labels and annotations, or maybe even hold objects like in like have them be in the default state of the outer permission, doesn't Grant inner permission.

B

You mean like, like per object,.

D

The like on, like David's, like you, cannot set the security.openshift.io thing like is: is there a way we could express that that basically made it that you would have to somehow um I guess it doesn't work, because I think you'd have to enumerate everything somehow up front and it is intractable to start that way.

A

B

All right uh seems like we'll be having a follow-up. Maybe maybe we'll continue this in sagoth. uh Is it next week or three weeks from now, I'm, not sure November.

D

9Th is the next signal.

B

Yeah so I think that's three weeks from now. So we've got lots of time to uh get ready uh all right.

A

Very good yeah, we are at time um there is keep going next week and we have a deep that session and then it's going to be public. You can watch it I think some people hearing the call are going to be present. I know, Joe is going to be there and some others so um have a good coupon. uh If you're going and we'll see you next time. Thank you. Everybody bye.