Kubernetes SIG Auth, 30 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: SIG-Auth Bi-Weekly Meeting for 20230830

Description

SIG-Auth Bi-Weekly Meeting for 20230830

A

Three everyone: this is the cigarth meeting for August 30th 2023. We have a relatively light agenda today and not that many folks around, but we'll see what we can do today.

A

um Anish, do you want to talk about the secret uh feature, split first or the offline stuff first, which one do we want to talk about? First.

B

We can talk about the offline first.

A

Okay, um so this dock is linked in the agenda and I realize basically, no one's had time to read it, because we were still like in the process of putting it here. um The there's two things I want to get out of this uh conversation for now. uh um One is a high level agreement that the problem is sort of like an okay one to solve, and then some of the specifics so like at the high level.

A

This design is um focused around adding the capability in secret CSI to support a mode where the cloud environment that backs the provider that it's using is unavailable for some amount of time right. So the our use cases are around like Edge deployments. Where might you might not have the best network connectivity, but you still want to use secret CSI.

A

um So you know in systems that are directly syncing into kubernetes Secrets. This isn't really an issue because they already wrote their data into the kubernetes API, so it's just kind of there and they can just like if they're offline.

B

A

Doesn't really matter because the the Secret's already there, but for secret CSI every time it does the amount right it goes to the provider which does effectively a live, read against the back end, and thus everything has to sort of be online.

A

So this proposal uh uh basically says that we we will try to solve that problem somewhat generically in the driver with coordination with the provider. So that way, if you have arbitrary providers that want to leverage this feature, they can with relatively minor changes, it is explicitly off by default and requires opt-in by the Admin, as well as opt-in by the provider implementation.

A

Just all that seem reasonable.

C

I'm trying to remember how much coasting there is already built into CSI like if so, if, if a pod wants to mount a CSI volume, the CSI plugin gets consulted. Obviously, before the Pod starts the first time and it has to fetch the data and set up the volume out and populate it, and then it gets injected into the pod.

C

um While the Pod is running, the driver I think gets consulted periodically to see if it wants to update that data. If that fails, doesn't the data and the volume Coast on like what was previously set up?

C

Okay, so that that already works so for pods that are started already, we're not going to break it in an already running pod if an existing, if a container in an existing pod dies and has to be restarted that also coasts on the existing volume right like we don't reset up volumes for container level restarts okay, so this is only for like I need to start a new pod instance uh uh and I am actually setting up volumes fresh for a new pod and the backing provider isn't available. Okay, that.

D

Would be helpful.

C

To describe just to avoid freaking people out who are like every my back-end provider is on the critical path for continued availability of running pods like making it clear that pods and volumes do Coast. This is for new pods that.

A

Would help this is basically new pods on like scale up scale down um cubelet drains upgrades all that kind of stuff, um and you know for our use cases. We don't need indefinite offline support. We just need it for some amount of time, but I think you end up needing this. You end up needing a durable cache, basically to sort of survive like a cluster restart.

C

And this is proposing an API level thing rather than a node local thing, so that it could survive or help with node drains and pod reschedules that happen to hit a different node.

E

Right, okay, it.

A

Is not trying to like do something Fancy on the file system of the cubelet it's running on, um since that would inherently be problematic. Based on how scheduling happens, um okay, um I I, like a high level, does anyone have a concern about just in general, doing this at all like um again, like I? Don't think such a feature should be enabled by default?

A

Nor do I think uh it should ever happen without um the provider stating that they want this to happen um right, like the reason being there is um so like we support like workload, identity, style flows through secret CSI, where the service account token is being passed through, so that is being used as like an authentication of the Pod and then authorization against Cloud systems, so, like a provider, needs to be aware that, like those checks are going to be skipped, if they let this happen, do they have to make it do.

C

They have to be skipped.

A

um Or or more specifically, they could be skipped depending on what like, if, if we built it in a way where the driver would decide on its own, that it was offline and would just use as cash, then those would be skipped right. The the way this design has it now is that the calls are always made to the provider and only if the provider returns that a specific error that basically says hey I, can't talk to my Upstream.

A

If you happen to have a cache, go ahead and use it only then and on will the driver see if it has cached information.

C

Okay, but the cache is Key by service account.

A

Yeah service account and SPC.

B

So yeah also this we call not published secret ref right, so CSI supports providing credentials, is kubernetes secrets. So it's basically a combination of these.

C

Okay, so if one workload asked for a secret and got approved and it got cached and then the provider went offline and another pod with another service account, it's like hey, please give me the secret. It wouldn't get a cash hit, because it's keyed by service account as well. Okay, that's right, yeah and.

A

All this stuff is keyed on the uid, so even if you delete and recreate, then you still don't get to Cache it, because not the same service account.

C

A

C

This would be opt-in at the provider level.

A

And at the SPC level, so you, the admin when configuring your custom resource, say I want offline for, however long and the uh the provider has to go into a mode where it detects itself as being offline and then tells the driver that I can't can't do the mount.

A

But it's not because of an authorization issue or some other assist random failure. It's I believe I'm offline. Please go ahead and use the cash. If you happen to have it, but basically the driver and the provider are always in the path and no one's getting skipped so unlikely that something will happen in a way that is unexpected.

C

Yeah double double opt-in seems correct.

A

um With that one in mind, okay, so I'm not hearing any immediate pushback or don't do the thing. The hard part about all of this is having durable storage for an offline thing, because the point of secret csis did not use kubernetes secrets.

A

um So basically, we have three proposals for like where the durable thing is. One is kubernetes secrets in the namespace of the secret CSI driver, so at least they're like local to it.

A

The other is using a custom resource definition as the cache, also in the namespace of the secret CSI driver and using it there um and then the third is mostly just a combination of the two with the idea that you have a custom resource that holds all the metadata for the cache, but then pointers to kubernetes secrets to hold the actual um data which gets more complicated. Obviously, um the the gist of the overall concern I have is.

A

In a lot of kubernetes deployments, you end up having a lot of components like Ingress controllers that have very broad, like read: access to kubernetes, Secrets and I'm really hesitant to put your fancy secrets that you got from your key vault in that space.

A

But if we put it in a custom resource, no one will know to encrypt the thing at rest for their KMS policies, because it's a new resource and it's not a kubernetes secret.

E

And they may audit log the bodies yeah.

A

They might Auto log the bodies or do something terrible um yeah, so I I, don't I'm stuck right. I want basically neither for various reasons.

A

um So the one, the the one of the considerations with the custom resource approach, was to put a encryption key in a kubernetes secret and just use it to encrypt our cash before storing it in the custom resource and thus effectively.

A

Fixing all the problems by inheriting a bunch of complexity. Where do we put that key? That would be in a kubernetes secret right, so it's okay! If an Ingress controller can read it because it doesn't have read access to the cache right, so it doesn't have the complete access to actually get all the data.

A

B

A

Perfect, not great the at the end of the day, what we're talking about is like re-implementing encryption arrests inside inside of secret CSI, which is horrible.

C

Isn't this like well.

C

The people who want this could already achieve this by copying to kubernetes Secrets right.

A

uh As in the secret SYNC feature in secret, yeah, well I mean we plan on removing that feature. Okay,.

A

That that's the nicest thing where he's going to split it out into its own project and then it won't be in secret CSI. So it can't be the solution to this problem because it's.

C

um Is there a thing we could do with the key that would make it less accessible and possibly more ephemeral? And if we lose the key worst case like we lose our cash, like normally with encryption keys. If you really really really really have to never ever lose the key, because if you lose it like you're dead.

B

C

Here, if the, if we can make the keys less accessible, but the scenarios where you would lose, the key would not overlap with the scenarios where the back end provider like becomes unavailable like those are two different dimensions.

C

So the backend provider would have to become unavailable and you would have to lose the ephemeral key for you to lose your cash.

E

um You could have your own, like Oracle service in the cluster that uses TPS and if it like, you know, gets all good resources at once. You just.

C

Is is there any Central controller or coordination point that the CSI driver.

C

Implementations have or any communication mechanism that they have um where they could hold this key, so it wouldn't be stuck in a kubernetes I'm. Just I, don't know, I'm wondering.

A

I see you okay, Maybe like it would certainly be hard to generically build because I think you need, like some host level, help.

A

A

Trying to think maybe we could do something like that.

C

I, don't know it might be like a question to put in there like is: is there a place that would be maybe less durable, uh but under reasonable operation? We think it would. We would be able to keep the key um I.

F

Don't know so I I have a question like why can't we just catch the keys on the Node, why do they have to be in the API at all.

A

Because we need to survive, node restart. Oh that's! That's!.

F

Awesome so this is mainly about disconnect from the provider yeah.

A

You're disconnected from the provider, but you still want the you're.

C

On the edge through your providers on the edge- and you need to be able to do, node operations and scale up and stuff does.

E

This break node isolation. If it's individual CSI drivers that are.

E

Writing and reading to the cache and one becomes compromised.

F

uh Probably like, given that there's no Jordan and I were talking the other week about see if we could finally generalize node restriction to more things uh but yeah. Typically, what I see in these cases is a service account that a demon runs, as is just given the permissions cluster-wide, and nothing would stop notes nodes well,.

E

I mean even even if the permissions issue can be fixed right. The point of this is that the cache is shared across nodes, yeah.

C

And unless that too yeah, if the cash issued across notice, then yes, it comprises yeah.

A

It to be fair, though, like when you, if you enable the secret SYNC feature that exists. It also does the same thing, because it basically requires.

E

Nobody should ever enable that I'm.

A

Aware of that problem, yet it's like possibly the most used feature.

A

B

A

So that that is fair, it does weaken the boundary sort of inherently with what it's asking the system to do right. It's asking these gaming sets to basically coordinate in a central place and they inherently trust the central place. So if you can poison or otherwise mess with the central place, you cause all of the Damon says to misbehave.

F

A

I don't know there.

F

Are alternative ways that you could be extra careful about doing something like this? Like you make your provider an in-cluster service, you create dedicated like a dedicated node, that is there for the provider or something and then I, don't know somehow manage a a PV.

F

You know persistent volume claim for an attached disk that holds your cache that can move around between nodes. If that needs to survive. Notary starts and just be like super careful about isolating the whole thing, but having any cluster cache, which is way more work, but you maybe get better security boundaries out of it and avoid secrets.

E

Yeah there was like a service running on the cluster that was CSI. Driver X goes and sends an RPC in in that service can say: hey you have a reason to be decrypting the secret or you don't have a reason to be different in the secret based on the pods. I can see running on your.

E

A

Just to build this, it would be like a like a node authorizer yeah. But first is the secret CSI I, don't know cash.

F

Outside the API, so that I don't know but I don't know if you get um if that's better or not.

E

I, don't know how much to worry about it right because, it's like say, I'm using GCB Secrets manager, whatever it's called right and I keep my you know my banking access token in there by the time, I've enabled the feature that, like lets that token leave gcp Secrets manager and be stored anywhere except memory. I've kind of like given up on.

E

At least a microscope yeah.

A

These are, these are secrets that are exported out of The Secret store they're, not um like keys that are like asymmetric and the private key never leaves the wall. You.

F

Don't have it right right, I think we should try our best to preserve node isolation. For this feature.

A

Yeah I mean I, don't disagree there I'm just trying to think of how to do it.

F

uh I mean if we maybe a question for others like if we did use secrets to store these the cash like. Is there a path to extend node authorizer.

F

For that, uh which would at least give us the option to run drivers is cubelet and get some protection.

A

When you say extend like the authorizer like what would you expect it to do more than what it does today,.

F

uh There would there would need to be a way of tying that secret to pause on the Node yeah.

C

I think you get into a chicken if you want the driver to do the creation of the secret and establish the relationship between a secret and the mod somehow yeah.

D

C

That action is exactly what you would not want a compromised node to be able to do right, um even if we did find some way to like link a random secret and.

F

It wouldn't even it wouldn't even just linking the secret to the Pod, wouldn't be enough right, because if you're trying to survive node restarts, you need to link it to a deployment or a demon set or like something higher level, because the pod's not going to survive a node restart either you're just going to get a new one. I.

E

Think it would help right, like the existing node authorization.

C

E

F

E

F

Net authorizer grants read permission yeah selectively.

E

Based on you know, the graph of what pods are running on a run. Node.

D

E

So it can additionally check like I mean that seems like the check we have today right. You can read the secret if and we'd have to come up with I guess some way of encoding the service account name into the secret name or adding a field to the secret or something like that.

A

I can see Jordan's face, he's trying to figure out how what the awful mess that would turn the node authorizer into a.

F

Track yeah I I'm, not a fan of it, I'm, just like chucking things in node, authorizer.

E

It doesn't have to be in the existing, not otherwise right you couldn't. This could be done via like aggregated API server or a totally different RPC.

F

Service, well, that's interesting.

A

It's just like a random service running on cluster that you talk to, and it elevates you based on a certain principle.

E

Yeah, like we've done that in in gke, we have other components that kind of re-implement all the logic that authorizer, because we couldn't find a good way to reuse. It.

A

Now I'm curious if Jordan reviewed such a thing and if he complained at you for re-implementing the note.

C

Doesn't re-implemental just the relevant bit? um Yeah they being able to cross nodes is a goal because it helps with no drain and node operations, and it's also an anti-goal because it breaks node isolation like I, don't see, maybe.

F

Like is it sort of feels like a sign that the driver is doing too much like it's going up a layer and trying to coordinate things like it's on.

B

F

Node right, you know shouldn't just care about the node it's on and and now we're trying to make it do things that span nodes.

F

Like maybe this should be a controller or something in parallel.

E

Just to play volt inside your cluster.

F

But I mean something that caches.

A

Yeah because the license no longer allowed us to.

F

Like we need something that caches the secrets in a way that spans knows, does it have to be the driver that does that? Can something figure out that pods are scheduled on nodes and populate that cache concurrently.

A

Remember, though, the only thing that ever has access to the secret is the driver, because there's no record of it anywhere else right.

F

Yeah uh yeah I, don't know how is that gated I'm, like the.

A

The driver running on the cubelet has no special permissions when it when a pod runs and asks for the CSI mount. It is issued a service account token for the Pod under a specific audience. The driver then passes that, through to the provider, the provider then passes it to the cloud provider to do whatever checks using so so.

F

You would you'd need this theoretical controller to be in a very safe place and have token create for all pods, for it to replicate that path. Right right. That.

A

F

You'd, probably put it next to your API server. If you were going to do that.

A

Yeah sure you could, you could certainly run it co-located on your control plane because it's it's like the controller manager. At that point, it's very very powerful.

F

um But it would that is one possible solution to avoid breaking node isolation.

C

Well, that avoids breaking it on the way into the cache. It's still not clear to me how the actual drivers doing the actual work get access to this very secure cash, yeah.

F

You still need some way to um indicate that relationship, but if it's, if it's the combination of this thing, popularity in parallel and some well additional node authorizer specific to this thing, that also runs on the control plane.

F

That might be paths that does not breakout isolation right.

C

It feels like this. The current state of The Proposal is similar-ish to the secret sync like it's a little better because it's encrypted, but not really, because we're putting the encryption keys somewhere that you can sort of kind of get at.

C

um So it's like it's a little better, but it's not that much better than the secret sync.

A

Capability I mean the difference between this, and the secret sync is that all the stuff is local in the secret CSI driver name space and no workload can directly access it, where you still have to go through SQL CSI to be given it so I do think. That's pretty significant right, yeah.

C

But but in terms of like exposing data from the provider into API, visible objects that all the current ideas about the encryption key, you can still kind of get at.

C

Are like do we know who would use this feature and how much they care about node, isolation or like are the people who would use this feature the same people who would use secret sync today.

C

A

C

But sometimes they really don't care I'm, just wondering like if secret sync would work for them today,.

B

Yeah, they might not be the same set of users who use secret sync because I think like they explicitly want to use the mount capability like the volume, let's see if it's available in the volume- and they just want that volume to be also available in Northland scenario.

E

Kubernetes secrets are available in a volume you can mount them, you can project them. No.

A

They don't want them in the kubernetes API like where anyone can Flex with them.

C

Like the the combination of not wanting their secrets in the kubernetes API, but being okay with cached versions of their secrets in the kubernetes, API is a little weird.

A

I I think as much as we tried right.

A

We do have special namespaces well, we do have special places where we put special things and protect the special things right being able to create like tokens for service accounts and Cube system is effectively cluster admin, like you know, you've completely hosed everything if you have that level of access, but not providing it at every namespace to every pod sort of indefinitely, with no policy around how long the pods hold on to it in in restarts and stuff like I, do think there is a distinction there, yeah.

C

But create token access is different than broad read access and we know things have broad read access.

A

Right yeah well, I mean the problem with secret sync is it requires driver or whatever component? It is to be able to create sequence all right, so once you can create secrets, you can just be like I would like a secret and I would, like the token controller, to fill it in for me, with the nice data from um this service account and now in cluster admin, because I could create a secret in this namespace yeah.

C

So so secret sync current version of secret sync, where it syncs literally sings to a secret, and then we Mount the secret into a pod. Yes, but well, I guess: yeah yeah.

A

It's very hard to get out of this circle.

C

And we have something else in the agenda to talk about.

A

Well then, the other thing on the agenda is splitting up secret CSI into um basically effectively two projects right, one that does kubernetes secret syncing and one that does the CSI mount, which I can initially talk about more oh.

E

One final question: I guess knowing who would use it kind of helps us like if we could assume things like TPM availability that may change what we can do.

C

Is that what you were referring to like host support.

A

Yeah I mean: is there any easy-ish generic-ish way of using TPMS on kubernetes.

E

F

Right is your tire. Is your suggestion that, like, if you were say at Edge, running on bare metal, you'd have the same TPM that could still decrypt your secrets, like once it restarts or.

E

D

E

Like I know like in gcp, for example, every VM has a TPM but they're also kind of like you could look up the details of other nodes, DPMS and encrypt things for them.

E

Obviously, that doesn't exist if you're just bought some Dells and slapped them in a cabinet somewhere, but.

F

A

And my understanding of the use case is rather like normal, vanilla Hardware on the edge like in a McDonald's.

E

Well, I mean normal, vanilla, Hardware probably has TPMS these days. Yeah.

A

I mean has some TPM, but not necessarily like the fancy kinds that can be like sinking across DPMS. You know sort of the I.

E

Think yeah, if it.

A

Was just a single node, it would be a non-issue right. We would just be like yeah just use whatever the TPM has and you don't have like I I know various projects with Microsoft that can do the single machine case really well across. Basically any Linux distribution with some relative compatibility, but I just don't know if there's anything exists for anything beyond more than one Linux VM or machine.

A

C

A

E

C

The biggest open question.

B

C

This thought this talks about protecting the content in the API, with encryption and there's some weirdness around where to put the encryption key, but it doesn't really talk about no isolation at all and today the provider. Can you account for that because they're given the token and they can see like what node it came from? Well, they can see the Pod.

C

They can figure out what no it came from if they cared uh hopefully soon, they'll get the node information as well, uh but if we flatten the cache in the current version of the design, we lose that information.

A

Right and today the cash is only filled as I mean not today, really at the end of the only way to fill the cash is for something to try to use the cash on a particular node. And so, if you had node local caches, they would only have data available for.

C

Which, which would help in some cases um but so like if you had a Daemon set or something um like it, would help that right? You could do. You could still do rollouts and things um because you would have a copy and you could still key it by the node. So it would help things like Damon sets, um but it wouldn't help no drains and cross. Node scheduling, moves and stuff.

A

The very ugly alternative that I never seriously considered was because I just thought it was the wrong places that you could just tell the provider to do it themselves if they want to.

C

A

Cash, however, you want to cash right. Nothing says that the provider has to call a cloud API right that the ugliness of that is that you're at like, if, if more than one provider cares about such a feature, you've just asked everyone to do the same work over and over when the driver should have just done before it. Yeah.

C

I mean you have all the same issues about encryption and protection of the encryption key and breaking notice like.

A

You have all this I was more I was more saying. The one problem that's caused is closer to. Who cares to use it right? We don't have it in the central project, then you don't encourage people to do it again.

F

I'm not saying that's right. If the, if the provider caches itself, we don't necessarily break node isolation as long as it does the right checks for who should be able to read this. The secret right, yeah.

A

I mean like the provider is handed the service account token, so it could basically be a note.

F

Oh, the provider is also in the node right.

A

F

Never mind yeah I'm, taking yeah.

A

F

Cloud providing.

A

But provider as in secret CSI provider.

D

um Just quick time check.

A

You've gone in circles on this.

A

F

Two split I think I think I'm I don't have a solution that meets all the goals.

A

When would it be better like if we let's say we didn't try to do anything fancy about node isolation? Would it be better to try to pursue like a persistent volume approach where you store stuff there and have some way of accessing that persistent volume over uh custom resource, Plus, kubernet, Secret.

F

I, my instinct is that that goes in the direction of just deploying a service that is backed with that volume. uh Sure.

A

I mean you can do that. That's right.

F

Often volumes are bound to a specific nodes, so um you wouldn't feel I mean you could do.

F

I, don't think you'd be able to have one volume shared by all nodes unless you have like an RPC interface in between.

A

Right, yeah I'm, not not really asking about that aspect, I'm more asking if you built the capability of that approach, is that approach better than using the kubernetes API with custom resource plus kubernetes secret and encryption, and all this stuff to try to kind of hold it together, like the beauty of using a custom resource and kubernetes secret? Is that you don't have to install anything on the cluster? It's all just there, you're just I.

F

I'm I'm not sure that it's really better.

E

I think having an RPC service somewhere that you just ask I, am nodex or I'm CSI driver on node X I want secret y unseal it for me, that's quite clearly better right. That's.

F

Awesome, yes, I think there are trade-offs. You're um service has to handle encryption for you, um whereas you know Cube, API server has Secrets encryption already.

F

You have maybe more work to Implement that service to deploy it to keep it operational, to upgrade it, whereas if this was done with like an admission, your hook or an authorizer that is either another one or an extension of node authorizer, it's a little more built in maybe easier to manage, um feels.

C

F

You have the the downside with using Kate's is that you get the union of the permissions? You know you want for this specific use case, plus whatever else happens to be in our back, which you might not want. If you want to be more strict.

C

I'm gonna Echo Rita's time check, yeah, there's more things. If we want to talk about those, we should move on and we say we're going to have this conversation again once more people are around.

F

Yeah great yeah, okay sounds good. Now.

A

I really really need, like chat DVD, to transcribe this meeting. For me, because.

A

F

If, uh once it gets posted to YouTube, you'll get a transcript, usually off of YouTube.

D

Taking notes man, oh.

A

D

A

Cannot talk and take notes at the same time, I am incapable of doing this thing.

A

um Okay- and this tell us about the splitter.

B

Yeah, so this is a controversial topic, uh so I think I did bring this up with the cigars I think, maybe last year uh the current situation is in CSI driver, even though it is a CSI driver. We support sync as kubernetes secret, which we realize we shouldn't, uh but it was mostly because a lot of users who you CSI driver to like get secrets from external Secret store.

B

They also wanted sync, this kubernetes secret one for usage, is environment variable so that they can reference the kubernetes secret in their parts back and then the second most common scenario is: if they were getting, search and keys, they wanted it to be synced as a TLS kubernetes secret.

B

So they can use that with uh the Ingress resources um and uh we've had conversations in our team in our Upstream meeting, basically like the CSI community meeting, um trying to come up with a way and also trying to decide if we want to split the project and then make it its own thing, um and one of the other considerations was: there are actually a couple of other projects out there in the wild, which do the exact same thing right.

B

Like there's like external Secrets operator, and then there are like other controllers, which can get secrets from external Secret store and sync it as kubernetes secret today.

B

um So the conclusion that we came up with in based on our community calls was we want to split it, and we want to have this as a separate project, because we have a lot of users today in CSI driver using that feature who can reuse the existing custom resources and be able to easily move to the new project so like. That is our biggest motivation, so that, like all our current users, can still continue using it rather than having to ramp up and go learn a different project and then like migrate.

B

All their applications to use that um I did a POC and demoed that in the community called um I'm trying to consolidate that in, like a Google doc. That I had shared before so I'll present that, and also like redo the POC like demo in the next one, but I think the general questions are. If we split the project, um is it okay that we have those two projects blessed by Cigar?

B

Like that's the first thing and then one other question that came up in the discussion last time was: what would the auth look like today.

D

B

Drivers we basically get the service account tokens from cubelet, um so the driver gets it and passes it on to the provider and all of that right. But if we do this synced controller as a separate thing. uh Basically, this controller is going to have god-level permissions to be able to generate tokens on behalf of the work and the custom resources. That's requesting for these secrets.

B

um Also, we are planning to tie the custom resource to a service account so that we can still support workload, identity, Within, uh but yeah. This single controller is going to have God level permissions and then the third thing is today. The CSI driver is generic enough where the driver exists, as you see God sub project, but providers are implemented by each provider, so they exist in different reports like Azure and one exist in Azure Google, one existing, the Google repo and all of it.

B

um We are able to reuse. All of that right, like we don't need providers to make any change. Only thing is we implement this controller like car, the maintenance and the way we would have to package.

B

It is all those providers have to be packaged as sidecars in a single pod so that we can still continue using RPC, so we can still use Unix domain socket to communicate with different providers so like these are the three lingering questions like I will add this in the dock and present it uh with the demo next time, but uh looking for thoughts concerns that we should be aware.

E

Of you need to place anonymous calls to these customers Auditors. Anyone who files an issue in the repo complaining about Secrets thing.

C

But the controller wouldn't have to have Global permissions. It would only need permissions for the service accounts. It was asked to sync for correct.

A

But how how does it get that access.

A

We don't have like a if this object exists. Referencing this service account then, and only then made this controller service account, create a token request for the other service account authorizer.

A

um The one thing I was curious about maybe Jordan. You know better than me, because I haven't looked in a while. Can any of the validating admission policy would sell that has access to the authorizer? Can it do any checks?

A

Like certainly I know, we could probably write checks that just disallowed this Con this um controller from interacting with Cube system. Sure we can do that, even if it has Global like cluster wide ability to create service accounts, we could just restrict it from doing anything in Cube system.

E

You don't need to sell right, you can just use webhook, authorizer or whatever. No.

A

E

B

E

A

We ship with secret CSI yeah.

E

I mean a validating admission web hook. Would let you block creation of secrets that are not authorized, even if our back lets it.

A

F

Yeah I guess so I mean we've done that before so yeah. So.

A

Certainly I guess that is that's possible. I was hoping for something that could be done in cell, so we don't have to deploy an entire webhoe.

D

I mean soon with the new authorizer creature.

A

Well, I mean that still requires deploying an authorizer.

E

D

E

What is the policy decision here right? Sorry, the.

A

Policy decision would be that, if a like namespace level, admin made a secret provider class that wanted to use the controller to do the work to sync to a kubernetes secret, and then they created another resource that indicated that intent and they indicated which service account was necessary to use.

E

A

The authorization.

E

Decision requires reading these arbitrary other resources. In that question, right: okay, right.

A

So it's exactly like the node authorizer right, it's just a graph! If the, if you can make a path in the graph, then you're cool. If you can't make a graph in a path, then no don't do the thing. So.

C

Cell balloting admission doesn't have the related to look up other objects exactly.

D

Only what's in the request.

A

B

A

Mean we could we, we could ship like a default policy that doesn't allow like Cube system and other namespaces to be messed with, and then maybe also explicitly, or we could invert it all in, like only allow like well known, like only allowed, explicitly opted in namespaces to be used it. It is problematic, though right. You basically have to Grant the authorization cluster-wide and then constrain it admission somehow, which is like my least favorite part about admission and authorization when used together.

E

I, don't know how much effort should be put into making this feature secure right. It's like.

E

Security is out the window once you're like, if you're deciding to take things out of your whatever Vault or wherever and just like spraying them across your cluster. It's like.

A

uh I think the concern I have more is foreign I'm, okay, with this controller, having like complete access to your Vault because you've, given it that and it can do whatever bad things it wants with the secrets, but that should not give it cluster admin to the environment also like it shouldn't need that level of access.

C

Is the assumption that the vault is still carrying about the particular pods that requested these secrets, or would it be sufficient to just let this controller run with a single service account and sync Secrets out as that service account.

A

Yeah, so the Vault does care, because they're built with like workload, identity, integration right provider, cares very much so that you pass in the specific service account with a like I mean one one thing I know we could do is we could disallow token requests being made by this controller that don't have an explicit audience set because it has no business asking for the Clusters audience as a token, so we can certainly constrain it immediately in those ways um and we could constrain like I think the types of Secrets it could make.

A

So it can't make like I want a service account Secret.

B

E

Well- and it only needs to pass the workloads token. If we want this to work unmodified right, we could tell people okay, the way this works has changed. You need to go authorize the this controller's identity to retrieve these things from the vault.

A

Isn't that worse, though, like you you're giving up more.

E

What this is like.

F

C

Is all banana you could you could limit it to making token requests for the back end audience.

C

All right, you can do that.

D

Look we we can think about all these like ways. We can enhance it, um but I think the larger question was more if we split it out uh this controller. Is this something that say: goth is interested in sponsoring and there's pros and cons with this right.

C

I mean it it's already in a project we sponsor, like all the functionality that we find problematic, is already in a second project.

D

I would also argue that, would this centralized controller the the goal is to work with external Secrets operators so that they also get the extra benefits that we're enhancing all the extra security around it. Hopefully, because that's a project that a lot of people use today and they have the same issues.

B

C

If we refactor it split out a thing that we already have and don't make things worse in terms of the blast radius, that seems neutral-ish and.

A

I think this makes it better right, because today then there, if once this, was no longer in secret, CSI right, you no longer have to worry about that thing. Doing a thing you don't want it to do, at least from the secrecy perspective.

C

Finding ways to constrain the permissions of the controller seemed like the big piece here.

A

And this, where were the other two things that was one but you've had two other things: none of.

B

Them yeah the one the other one was when we moved CSI driver, like one of us, was to make providers out of three rights so that, like we don't package all of that in a single repo. So with this one, the code still exists in different repos like it's, not in kubernetes six, but we would still need to package the Pod manifest with all the different providers, because the communication is still through Unix domain socket. So.

C

It still needs to be through Unix domain socket, but you would only need the provider that you were using right. We wouldn't have one part with all the providers.

B

All right so either in the kubernetes six repo. Wherever the controller is we package it like in a hand, chat where users can pick which providers they want to deploy, or they would have to go to each individual, provide a repo to figure out deploying the controller and that provider. But.

C

Yeah I mean putting them side by side in a pod, so they can still speak over. Rpc seems fine and I like that. We don't require the provider to change anything. We can still use the same provider interface and make the same request to it so that one provider implementation can be used as a CSI driver or for this that seems good.

A

What was the third thing, so that's the one is the packaging constraints of like one of the goals of secret CSI was to not be king maker right and I. Think it succeeded in that by splitting out providers and I think this still continues. That Trend, then, is the permissions thing. What's the third thing.

D

There's a sub project right.

B

Yeah, basically just splitting it up and having both of it as a cigarettes up on you, these.

D

A

Think the summary I got from Jordan is already a sub project. I.

C

Mean it to me, this looks like taking functionality. That is not great out of a project that we would like to actually be something we wholeheartedly recommend and kind of.

A

B

C

Over here, saying like if you're using this, it's still here use it this way, um but and.

D

It's an opt-in thing, yeah.

C

Would we what would we do for people who are currently using it in like? Would there be a migration or uh like if you were doing this now, do this type of thing, yeah.

B

So in as part of the boc like, we are able to keep the same custom resource and, like the whole provider model, everything is the same right. So only thing is you deploy this extra additional controller and then there is like a wrapper custom resource which references, the old secret provider class and then has details about service account and all of that that the controller can use for workload entity. So that's the sample that I shared on the zoom chat.

B

um So in terms of uh migration, like once we have this separate controller, we will Define a plan for deprecating this in the CSI driver project, um which will probably be like a 2.0 or something um because it's a breaking change and we will also document how users can move to that like they can keep most of the resources that they have today. It's just like additional things to deploy and all of that.

C

B

C

The scope, the permissions- you would probably have to do some sort of admission thing, um which is kind of lousy. There are match conditions on admission web hooks in 128 in beta.

A

Wait, what is the main condition do in this case? Lets.

C

You skip your web hook, or so you could, but that means you're putting a admission web hook in front of like token request, which and then saying only intercept things from this. One service account like I, don't know if, on any cluster older than 128, it means all token requests are going to go to your web hook and you've. Just that can toast your cluster. If it's unavailable.

A

No, it will definitely toaster cluster. You can't get pods up.

A

um Okay, but but I thought is validating cell admission, not beta, yet I thought it was beta. Oh.

B

C

Is better than 128.

A

C

Right but it but beta apis aren't enabled by default right.

A

No I'm, okay with that I'm, okay, with telling people to go, enable the beta API to get the feature, and all that like that, that part seems much better than deploy an entire web hook. That might break your cluster.

D

A lot of cloud providers don't have it enabled them.

A

I mean depends.

A

I won't I, won't name any Cloud providers that have it enabled.

B

I gotta drop all.

A

Right cool, uh thank you discussion. Everybody um I think we'll probably have it again in a couple of weeks. I know you're so excited, but okay.