Kubernetes Data Protection Working Group, 26 Jul 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG - Bi-Weekly Meeting 2023-07-26

Description

Meeting of Kubernetes Data Protection WG - 26 July 2023

Meeting Notes/Agenda: -

Find out more about the DP WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMWare)

A

This meeting is being recorded, hello. Everyone today is July 26 2023. This is the kubernetes data protection wedding group meeting, uh so I think today um we will have an update from Carl Ivan um the seek off right. You guys went to see goth.

B

A

A

See God's meeting okay, so you guys want to share anything. So I can stop sharing or you just want to talk about it. Oh.

C

Yeah I can share some slides.

C

I mean I'll, see my desktop yeah. Okay, so I pasted the link in as a comment: okay agenda, yeah all right. So last week we presented that current design proposal right to see God and uh we got amazingly good feedback.

B

C

Yeah, essentially, you know they told us that Mutual authentication can be achieve between two arbitrary kubernetes services, using the token request and token review apis. Oh interesting, okay, yes, so it's possible, you know and then and the token we get back has enough information right, the user information field there. So we can do a subject- access review yeah, so uh you know essentially to.

D

C

Right they had no problems with all these parts of of our proposal. Right that there's a side couch hides the store. You know the SP and then and the sidecar talks to the SP bar Unix socket, and you know the application uses direct grpc and the fact that they are TLS and not mtls, because when we came down to this part right, which was you know, there's a set. We had this whole session construct to solve the M part of the TLs issue. Right, they said.

C

ah Don't do this use token request and token review apis.

E

That wasn't the only reason for that part right, like the other reason to have a session, was to obtain the end point that you should be talking to for any given pair of snapshots right.

C

Yes and and that part Still Remains that's over here, the advertising of the of the endpoint.

E

I can't see your mouse okay.

C

Oh sorry, yeah.

E

It's all right, I, see.

C

It now okay, yeah I'll, get to it in a few and I'll get to it in a few slides, right, yeah, but you're right yeah. So we don't need this token review token. You know we use the token review token uh request apis to solve this problem right. Essentially, what has to be done is the client obtains an audience scope token. Now this is something new for me, too. It's been around for a while after I dug into it, but I it just did not surface for me.

C

There's a thing called an audience scope token, where the audience is not the kubernetes API server, but the server or some other entity right which you're talking to not the kubernetes API server.

C

Then the client would pass the security token. And of course, in our case also, the snapshot. Namespace Etc right is part of the grpc parameters and then uh the whoops, where did I go, the sidecar right will authorize. This cycle is authorized to actually do an authentication check and to authorize the grpc, just like we had before in the web hook Etc. So you know it really solves a lot of things. So, essentially, the synopsis of what we will be doing now would be you know, as before.

C

The backup application will directly connect to the CSI grpc uh snapshot, metadata service and bypass the kubernetes API server, but backup application must first use get the token from a token request, API to get the Right audience scope token for the CSI driver concerned, then the Sidecar or shouldn't say session anymore, because not a session. Okay, the sidecar is available to front the SP metadata service right and its job now is to authenticate the client with the token review API to do the subject.

C

Access review for the you know to to make sure that the snapshots the client has access to the snapshots, concern and then do the usual translate of the IDS and proxy. The grpcc call so much less moving. Parts the entire manager piece is gone right or it's only the sidecut to be deployed uh with the SP service, but but something's.

E

Still missing uh how, where is the decision of whether or not to Grant the access like like? Under what circumstances do you say? No, you can't have a token.

C

You mean this one, sorry in the beginning, though, that is that is up to kubernetes right, but.

E

I mean so so so when, when an administrator installs one of these things, there needs to be an affirmative action where they say: I want this application that I'm installing to be able to call these apis so that in case some attacker comes along and just tries to call the same apis like they get denied. And this thing doesn't. Let me.

C

Jump Ahead rapidly what you talk about over here right, so um the CSI driver definitely needs to get permission set up to do the the authentication pieces I was looking but did not find doesn't mean. My search was correct. There is something um maybe the app every kubernetes service account automatically. Has the ability to get a token right, I haven't figured that part out yet, but.

E

I worry is that everyone can get a token, and we have to have some sort of role binding, that binds an identity to say, like this. Guy is allowed to.

C

Get this doesn't work that way anyone can get a token, but it does, but where you use the token and where the token is honored is what controls it. The gatekeeper the gatekeeping is done in the token review.

E

C

E

Is it what is what is checked there? Is it a.

C

Role: button yeah, let me go through, you know, just a few slides and then and then I'll be. uh Let's then we'll dive in okay. Okay, let's share you know we'll put that in the question in the parking lot, so we can trash it out. Okay, so just to go over there's the three phases. This thing there's a client setup phase where it gets where it gets the token there's the authentication and authorization phase where the sidecar validates that token on the grpc called and then as before, right the proxy grpc stuff happens.

C

Okay, so a little bit clearer, now uh zoom in a bit, uh so essentially what the client will have to do is it has to look up the volume snapshot. The volume snapshot, content objects to get the CSI driver name. You know at this point. The backup line by actually doing this is you know, has basically has to have permission on its own to look at these objects, which is cool uh then, before this used to be done by the manager.

C

But now the backup application has to search for this registry, where the uh CSS driver has advertised its sidecar right and this this object is same as before, but it has an additional strain, which is the audience string. This is typically the DNS name right of the service, but we could choose anything we wanted to right it's, so we will have an audience string out there.

C

um Then the the backup app will then make a token request, call uh and it'll provide in that uh body right the audience string, and uh you know it could be, and whatever other optional parameters like expiration or if it's, if it's uh done indirectly by a projected, you know service volume in a pod, it will be bound automatically to that pod. Things like that, so the validity of this token can be also set at that point. So the token which comes back right is an audience token.

C

It's I believe it's a JWT right and it contains in it the user information which would be needed later on for a sar.

C

Okay, then, the client, of course, will trust the ca certificate as before, uh which now enables it to make the RPC call the grpc call and the grpc call uh we we set up that session document we used to have before we provide this audience scope token. We did not have the snapshot namespace before, but we have to provide it now because there's no context involved.

C

Oh so this call comes into the side. Car and the sidecar will do a token review. The token review takes that uh audience, scope token from the RPC call and the sidecast knows its own audience string. How does it know it? It was configured that way. You know whatever. Whatever registry information was there, it knows that same thing, so it knows its audience string.

C

So it makes a token review call and the uh it sends it to the kubernetes API server which, whatever, however, it it works under the covers right will, will decrypt that JWT and make sure that the audience string right matches uh over here and it'll come back with. Authenticator is true and it will return the user info of that client.

C

That's crucial. Now the the psycha itself needs permission to be able to make this token review call and that's that's something which you have scoped in a later slide, which I just flashed early on um so now the sidecar now has user info. So now I can turn around and do a subject. X whoops subject access review right with you know, which is the call we were doing before on the web hook. So now we can do a subject: access review to check that the the caller has accessed the volume snapshots in that namespace.

C

uh If that comes back with true right, then we can proceed as before. You know and do all the other stuff which was load the volume snapshots, the contents get the IDS validate existence, Etc and then proxy the grpc call. So it's actually a much simpler solution. Less moving paths than before and definitely easier to deploy can.

E

You go back I just want to understand that last part. What what is the subject action view access review actually looking at to to decide? Yes or no. Thank you.

C

uh The subject there's the section called resource attributes where, uh where you specify uh the uh resource types um uh involved, is uh Prasad on the call, you can probably give us details. Yeah foreign.

F

Request so if the client has access to uh certain resources um for certain verbs right, so you have to mention the resource types as well as verbs like. If they have, they have access to get create list. uh The.

B

F

B

F

Will be checking if the client has access to volume, snapshots and uh um yeah I'm going to start short content. You know to uh to able to get the better data yeah.

C

Another specified namespace too I think you can control namespace access right. You.

F

B

C

B

Sorry Ben, you can also control like access to like a URL can essentially say. Like this token, uh it only works for this URL. So I think like um to your earlier questions right. So essentially, there are two or three things that helps to safeguard the um the I guess, the authentication and authorization of the token.

B

So let's call pointed out, anyone can request any token right from the kubernetes API server and it's bound to you like the service account and then the interesting thing there is like um the audience it's like it has to match on both like um the backup client side, as well as like a CSI driver's side, but.

E

That's secret: that's just a string.

B

Yeah, it's just a string, so there's one one layer of it and then the second layer of it is obviously the art back here that we're talking about right and then they are back like I think we um in one of our earlier prototypes from last year like and we discovered that not only can It authorize like access um to objects or resources. Like one snapshot, you can also like um Safeguard like a URL. That's like I think something that they call it non-resource. You know non-sumpthing resource on URL resource or something like that.

C

Yeah non-resource URLs yeah and one.

B

Last thing that we haven't fully Research into is a token request, actually allow you to say, like this token is bound to, and this certain object yeah next, so it could be like um uh you know like where was it? There was a I think Carl pointed it out somewhere there was like an object uh reference bound reference somewhere inside the token request, um yeah uh API, so right there yeah like um so that one there you see like the yeah.

B

So that is something that we will have to research into like can we say that? Okay, this token only works for this particular uh volume snapshot or whatever, like.

C

No, no, no you! It would be fun, it's an existential uh thing: it's really it! It helps control the duration of that token. So, essentially, you know uh if I have my backup, app and I'm launching a pod to do this stuff. I would bind that token to that pod and kubernetes can do it automatically with a projected volume right. So essentially, what happens is when, when you do the review request and you have a bound object- the cubelet make sure that the bound object is still in existence.

C

If the bound object has gone out of existence, then it denies the token, regardless of the expiry time.

E

Okay, but but what I'm? What I'm hearing is any kubernetes user? That's able to read volume, snapshots and volume snapshot contents and can guess the audience name can go through this process of creating it constructing a token and obtaining the the URL and authenticating to it and getting granted access to this API. Because all it's going to check at the end of the day is you have a token that anyone could get?

E

You have an audience name that anyone can guess, and you have access to read a few objects that probably lots of things have access to reads right.

F

Just to add one more thing, um so obviously, as god mentioned, there are a couple of two main ways to generate the token One is using the token request, API and other ways to you know. While creating a forward we can project, we can inject the token into the uh the board State using projected field for the first method. If you want to create the token using token request, API, uh we have to give the permission to this resource so service account token sub resource through the hardback rules. Obviously, oh yeah yeah.

F

um There's a certain reprojective token I think it's up to the who is creating the Pod might need to access, but I need to verify that yeah.

C

And Ben points out one more thing you know so so far I haven't, we haven't looked at whether we want to control access to the snapshot. Metadata service, independent of access to the volume snapshot object. So yes, when you're right, so we could have another permission defined which this Backup Service app has to get right. Okay, yeah, so we.

E

Can check this is a fantastic Improvement. I I'm really happy with this, as opposed to the old design, where we were basically rolling our own security. This seems way better. This.

C

E

I, just I really think there has to be an affirmative step so that someone takes as part of the installation that says this service account should be able to call this API and.

F

E

That access and then and if you don't do that it just doesn't work.

D

Do we do we need that so, what's what's the equivalent operation would be creating a volume from a volume snapshot right so can we check and see if they have permission, I mean what what permissions do? We need to do to do that? The.

E

The worry is that something that is only supposed to be able to like look at some snapshots now has access to call this API and like launch a denial of service attack on the on the snapshot. Differencing endpoint right. So someone who is.

A

Only supposed to be.

E

Able to do some simple things with snapshots is now granted access to this API that they shouldn't and they could go bananas and attack it. You.

D

Know I mean that's, that's a you know, but it could also be a bug in the software that goes bananas as.

E

Well so yeah yeah I mean you.

D

Know denial of service is usually something we have to deal with on the server side, so.

E

So I just I think that there has to be some role binding, where you specifically say: hey I'm, installing a backup application I'm granting the back of application the access to this backup API so that other things don't by default, get access to the back of API. That's.

D

All I'm saying but I mean in but not but in terms of, like you know, going bananas we can start creating PVCs like man right so what's what's our equivalent permission for creating a volume from a volume snapshot, I, don't remember so.

E

So you're protected by quotas and billing in that case right, but.

D

Also, there's a there's, a limitation right, I mean so so our so some security point of view. We're actually really concerned with. Can you get data out that gives you? You know information about? What's stored, I, don't think the CBT is especially sensitive, but you never know. There's always side Channel attacks from junk like that. But if you could read the actually read the data by creating a volume from the volume snapshot, then I don't have a big concern about you being able to get the CBT, so it can.

E

The basic rule is, you have to be in the same name space as the snapshot, and you have to be able to access the namespace yeah.

B

You can access.

E

This this name page for the snapshot is you can create a volume from it and the namespace is your security model for for that kind of data, so.

F

The concern here.

E

Is you have an infrastructure component that can see all of the snapshots.

F

E

Yeah they can. It can reach around behind and and attack this component, but.

D

It can also create new new volumes from the snapshots right.

D

So so that would be the question then. So what would we do that limits you? So it seems like if we limit people to the same level of access as you get for creating volumes from snapshots.

D

That's equivalent security. Does that make sense.

E

E

No I mean I I'm, not thinking about this in terms of end users using the system I'm thinking about this in terms of infrastructure, components that are providing infrastructure for data management.

D

I mean denial of yeah I know, but though you know so, denial of service is one thing right, but we've got to defend against that anyway, because bugs I mean you know bugs are just as likely or more likely than a denial of service attack, because it's like yeah, you denied people the ability to do backups so.

E

D

Need to defend against it anyway,.

E

But but I'm saying like I'm, not asking for something expensive like a just adding a role binding somewhere that you have to explicitly Grant before this thing will open up the API to you should be very, very lightweight yeah.

C

E

It costs us nothing and it gives you an extra layer of security, but.

D

We don't do that role binding for other things. If you just want to be able to you know.

E

No, no! No! No, when you install a new CSI driver as part of that, like you create a bunch of service accounts and you give the sidecars access to all kinds of kubernetes resources and people scrutinize the list of resources that the CSI driver has access to very closely and says you know. Obviously, CSI drivers need to be able to read PVCs and they need to be able to write PVS.

E

But, like the moment a CSI driver is like reading the pods people say: that's weird: why would a CSI driver ever need to look at a pod? You know, and so people look at what infrastructure components have access to and and basically what the proposal has stands says: we're basically piggybacking on the ability to read volume, snapshots and volume snapshot contents, and if you can do those, you also get this other ability for free and I'm saying: let's not make it. You know implicit.

E

Let's make it explicit that you grant access to the the CBT API. You know when you create that service account. It's just another one of your many role, bindings yeah.

A

This Arrow here right, we have this uh Snapchat data service. We have this.

C

This is this: is the public object, non-names based I? Think anybody can look it up, I mean yeah. You could control it through that Shin, but I. Think that's uh since the non. This is a global object. um Yeah you could control it so that are.

A

You Ben asking for another one, that is for the namespace the object.

E

No I I'm, suggesting that we use one of those uh with non-resource URL uh things in the role binding so that when you or sorry in the in the role this or the club, this would actually be a cluster role. You'd have a cluster role that says, you know, may call the CBT API and then, when you create the service, account that the backup application runs, as you just bind it to that cluster role and boom.

E

Now it has access and when that, when that subject, access review occurs, the thing it looks at is: do you? Are you bound to a role that has you know CVT API in it and if no, it says you're not getting in right, just just to have an explicit handle that says these are the servers accounts that may call this API, and these are the service accounts that may not call this API so that you know, and you can have it- you have a point of control. Okay,.

C

Oh all right, so we look into that. uh So what Shin just mentioned, uh access to the snapshot metadata service so far, we just thought about it as a globally accessible object, but if that was also if that was control, wouldn't that serve the same purpose.

E

C

No that's up to us right.

E

Well, it's up to the cluster administrator like.

C

That I mean it's always up the cluster administrator, so right.

E

So we we get to make a recommendation around by default, should should people be able to access this or not, um but but I I still think, no matter how many of these we have, there needs to be one that says: you're allowed to call the CBT API as a as.

C

An individual all right, so the concept of using a non-resource URL is kind of interesting right. Prasad uh any take on that you're very familiar with these apis.

F

uh They are about using non-resource URL uh in in the rules and rules. Sorry in the roles and role bindings, uh the only issue I see is um if.

B

F

Fixed service URL uh that would work um changes.

C

F

It validated any.

C

Arbitrary string, which is what's checked for later on.

F

B

E

If it has to be the actual URL, then it won't work. Yes, yeah.

B

Well, like is like sorry just to confirm, though it's not I, don't think you need the entire, like um the DNS name, plus the prefix and all those things right. It's just the the the the prefix or the subpath that points to the API. Isn't it.

F

C

Yeah, um which one is it it's in the authentication.

B

It's under our back, uh it won't be under um right, yeah and then.

B

So search for non-resource, URL or something but just.

E

Not really sure, let's see.

B

It there is like rules, dot, um non-resource, URL.

E

It's a partial URL, the user should have access to stars are allowed billions. The full final step of the path since non-resource Euros around namespace, but.

B

C

B

Has to be a cluster row binding can be a row binding.

E

That's fine because we want this to be cluster-wide, yeah.

C

Then again I mean it could be. It could be that we require that it has access to the I mean the URL doesn't have to exist right, it's just a name if it gets through. If it's got, if we're doing the token access the token review.

C

uh Sorry, the SAR, we could be looking for anything over here which matches something right. So it's up to us how we control it.

E

C

E

Again, my high level point is like administrators will want a way to control this. Yes,.

C

You need to give them something: yes, that that I get yeah and actually, in the previous proposal, we had access to the creation of that session object that was equivalent to what you're just saying so.

E

Yes, yes, exactly if there's some special object which the granting of access to it lets you do this and nothing else then like then it would serve. The purpose. I'm just worried that if we piggyback on existing objects, then things that happen to have access to those objects will implicitly pick up this new permission, yeah and that's bad yeah.

C

So do these objects actually have to exist, I mean what if we have I mean we have an API Group for ourselves right and that and that's the whatever we defined over here. um Would we actually I mean what what was that is. We have the CBT storage right, cbd.storage.gatesio right, V1, Alpha Etc.

C

So what if we had that API Group and we had a resource, called snapshot, metadata sessions, reception, better data, API or session or whatever, and it doesn't exist, but it's, but it is a a resource name and we're looking for a create verb, which is pretty much similar to what you know. What how you set up permissions for token request and token review right. You just use these names, but these don't actually exist in hcd or anything.

C

Would that work? It's something worth trying. What do you think, or we can ask, say goth about that.

B

So, what's the purpose of creating a R back that the resource don't exist or essentially like I, guess, a virtual risk.

C

That is it's well scoped, because for one thing we use our I API Group right. It's not an arbitrary thing. We use the iepi group, we, you know we'll specify the resource, a pseudo resource which represents what we're trying, which is you know, access to this snapshot metadata service and the verb is create, that's it which is not, unlike you know, token, uh the getting permission for token review or token token.

E

Requests I think the verb should be get, but but yeah it doesn't matter as long as as long as the system would support that. That would be a good solution.

C

E

In my opinion, yeah.

C

Yeah, the the in these cases, right they use create for all these things. So.

A

What yeah, whatever it's some verb.

C

We just we'll figure it out yeah, but if we can try, we can get if that works right and we can get it uh cross check with see goth right. That would be I. Think that would be a good point. What do you say? Pressure.

F

Yeah, uh it also makes sense to use non-resourced URLs.

C

F

Right so that administrator would know which apis uh the backup, client would call and whether they want to give access to that. If you are not.

C

So how would we name our grpc API? That's my problem.

F

um It yeah it could be, we can even use a star or any wild card that is allowed.

C

Yeah, well, it.

F

Is common critics? Okay! Let me let me.

C

Put this ad, like I'd, like I'd, like the rule to be able to identify, if that it's CBT, my Snapshot metadata, related okay, so I was just hanging on to that uh particular noun. We saw over here.

E

Yeah it, it should be clear to an administrator reviewing his cluster roles like what it means when he, when he sees this in a cluster role, and aside from that I, don't think it really matters. You know, as long as there's an explicit line in your cluster role, that granted you access and that's the thing that the subject access review is looking at yeah. Then.

B

The administrators.

E

Will know what to revoke if they want to revoke access yeah.

C

I think we should also look into the the fact that shin and mentioned why wouldn't access to this object be equivalent I think it's equivalent right, but uh you know and that's actually a little bit clearer than a non-url and sorry a non-resource URL.

E

Yeah that I I would be okay with that, it would say, get access on the snapshot. Metadata service implicitly grants access to this, because why would you be talking to this? Why would you be getting snapshot metadata Services unless you needed access, I mean I'm, sure I'll be proven Wrong by some Edge case, but that that feels closer to a good solution. Yes,.

C

Yeah we looked into that differently. Yes, thank you.

B

Sorry, there's still essentially two different things right. The mediteria service is a resource. The URL that you're trying to protect is a different I guess resource no.

C

Is it I mean this.

B

Is representative right this.

C

Is representative of any snapshot metadata service.

B

Is that a valid assumption to say oh I, once I get access to this resource? I'll get access to another URL, because this resource Snapchat matters there in the in the it translates to a URL of its own right like slash API.

C

Hello, this is, this is a permission.

B

One snapshot service isn't.

C

This a permission.

B

It's a different thing. This.

C

Is the permission that gives us the ability to look up to make this call to list right.

B

Right to list the object right, so behind the scene, these objects snapshot matter their service, it Maps its own URL, right, like in the in the kubernetes, like um you.

C

Know that's right, but it's a classic binding, but it's a class I mean really. The application doesn't know where the volumes are going to be right. The volumes are app is granted access to volumes, uh volume, snapshots and volume snapshot. Contents well, volume snapshots of the namespace, not volume snapshots of this particular CSI provisioner in that namespace. It's volume, snapshots of that namespace. So likewise,.

E

The point I've been making is finer. Grained is better right. This we're doing something coarse grain by proposing that a finer grain one would be would be an improvement if it's possible.

B

Yeah, like um I guess, I, just don't see like the need to force the assumptions that hey. If you get, you have permissions to snapshot matter data service resource and this URL. It also means you have access to like um the CBT uh URL um I do something like that assumption.

C

This object represents the service. I mean how much, how much more precise can you get? No I agree. It's not on a personal per object basis. It's a class of objects right, it's the type, but.

B

What's the concern with using the non-resource URL R back okay,.

C

No concern other than can we can we strain make it very clear by stringing in all these nouns right of some sort? What we're talking about yeah.

E

I think we should experiment here and come back with the proposal. um I, don't think this is going to be a hard problem to solve. No, but uh we just we don't have all the information in front of us so I think we're we're speculating and arguing about stuff. We don't know so.

B

Yeah, so sorry, uh so one other question um Carl so as far as I can tell like, if the uh going back to that bound object, reference um thing that we talked about so um so sorry go back to your slide. Deck I think that one thing that we are missing when we move to this token review or token request API, is that we don't have to wait compared to like the previous um session.

F

B

Like one thing that we we lose out is that we are not able to say hey this token only works for this session. With these two pair of this pair of snapshots right no.

C

B

C

That we've accomplished that I.

B

Mean I mean there's nothing on the back end. That's.

C

Saying oh I see I see that yeah, but you know that was an artifact that was an artifact of um how we did the session because we had to enumerate the objects so that the session could find out. Otherwise, you know the Creator could reference some objects in the request and then do some other set of objects in in the actual grpc. So we had to qualify that that is an artifact of our solution, which we have to do to make sure consistency was accomplished now right.

C

There is no session set up so the first time we ever see the first time. We ever see the request right. It's got all it's got this entire payload and that's it.

B

Right so essentially on the CSI driver's side. All we can do is like we got a token and then we say: okay, this token is authorized to access all these resources and all this um right, URL or whatever, but.

F

B

Nothing, you know to say that this session, on this token, only works for this and I guess session, because you know in the past I can we have a yes because.

C

We don't reach a way to represent it exactly we don't.

B

Yeah, which is not the end of the world, I guess it's.

C

Wrong- and we will still be doing this so the at least the existential check will be done right, we'll be looking up the object. You can't reference some arbitrary thing which doesn't exist stuff, like that, we'll still be doing that.

B

Right right, the most we can do is that we can say this token can get and list like volume, snapshots and Norms natural contents. There's nothing. That says like only this pair of volume section, and only this pair of volumes are content, or at least on the CSI driver side. Yeah.

C

B

I don't know if it is a big deal at this point at least like there's, some form of um uh authorization right at the back end I. Think one of the things that, like um the risk of um sounding like um a bit like oversimplifying things, I think this security itself, as we all pointed I, mean many times such a big thing like I. Wonder at what point can we just say: okay for B1 alpha one of the first phase of this cap.

B

We just said we have all the essential kubernetes security things, employees and then, if needed, we can come back with a second cap. Let's say this is how we Harden it further with more specific edge cases in mind, because right now we're just exploring potential edge cases. You.

C

Know I think that's what the phase is about, so people will try it out and come back with feedback right.

E

It can you describe the the the realistic use case for a backup application that that can only run CBT on some subset of the possible snapshots, like I I feel like a backup application needs access to everything to do its job. The the thing I'm concerned about is I. Think I don't want things that aren't backup applications to be able to call this API.

E

D

Yeah and I think that's fair, yeah well, I mean people have you know, people do run in some environments where they don't get full system Privileges and they want to have like backup of just their namespace, for example. So we had people asking for this. You know that they would want to just install Valero for themselves.

D

And you could see other things like maybe a replication app yeah, that's replicating the volumes in a particular namespace. So.

C

Couldn't that be accomplished, Dave by simply uh giving the backup app a role binding to permissions, the volume snapshot as opposed to a cluster role, binding.

D

Probably I'm just I think uh you know. Ben was kind of like what what's the use case for.

E

B

D

App, that's not plus the Y, so.

B

E

Think that's a real use case. Then then the ability to to do vetoes based on the token accessing a namespace is not supposed to see, starts to sound useful to me.

B

Right essentially, it's like we don't want, like 10 any one new space to request or change block.

B

But again right like can we can that just be like a phase two off the cap, because that seems to be a harder problem to come up with it.

E

If we solve the the cluster-wide backup problem securely, like that'll, be a massive win and and this desire to have per namespace backup application seems a little bizarre to me, but but maybe we'll entertain it later. Yeah.

D

Yeah I would say that um for the security, if we try to tighten it in a in a later version that tends to cause problems, if we're like adding additional capability, like you know, hey here's the ability to let users have finer grain, you know not, they don't need cluster uh roles or you know cluster permissions to use it. So we're kind of loosening it up, but also you know, make it finer grain, that's probably something we can pull off, but you know something where it's like.

D

Oh you just used to work, but now it doesn't that's, that's, usually a harder problem.

F

E

Don't want to over engineer this part of the solution. I mean I. Think we've made a huge step forward with this new design. um No.

D

I I agree. All I'm saying is that you know, let's not plan, to tighten things. Let's make it as tight. We can make it tight at the beginning and then loosen it. So, like your proposal that initially this only supports you.

E

Know all the way through like what does it mean to have a access to only do this in one namespace I'm.

D

Not arguing that we do that, that's that's something where we can that's loosening. Actually, so initially we're saying you gotta have! So if so your proposal was initially, we say you got to have basically cluster admin. You know very high level of privilege to use this so basically cluster-wide um permission and that's something that you don't hand out to every application right and then we'll give you finer grain controls in the future which won't break like. If you already had cluster cluster-wide permissions, adding the namespace scoping isn't going to break running applications right.

D

They already have permission to do everything anyway, that's kind of what I'm trying to say does that make any sense.

C

Yeah and I think Dave. It's probably already grandfather then, because essentially, if you don't have access to load volume, snapshots, in-app namespace, you can't do this.

D

We were right well, unless you guessed something well.

E

The the problem is the problem. Is this hypothetical future namespace backup application still needs to read several non-names-based resources to do its job, and it's not clear to me.

D

E

You're going to put a fence around it other than at the subject, access review, choke point that we just talked about right.

D

That's so in an existing application, I can do a volume snapshot in a namespace. I can create a volume from the snapshot in in a namespace and I can read the data from that snapshot in the namespace right.

E

D

You can't find the.

E

Volume snapshot, content, object, it's invisible to you.

D

E

A

My hypothetical.

D

Backup application right right now can do its work by taking a snapshot, cloning, the snapshot, attaching the snapshot as a volume and then reading the data out. Okay,.

C

That's an interesting point: there Ben, uh you said the volume snapshot. Content could be invisible, see the only reason we're saying load the volume software content is to find the CSI driver name, but.

E

Right right, that's why I'm saying like this backup application is fundamentally an infrastructure component. The way we've designed it to try to install it as a user application in a username space would never work uh and, and you would have to start granting it cluster-wide access to certain resources.

C

E

Moment you do that it can see the snapshots in the other namespaces and it can start doing all kinds of things you wanted to do. I.

C

Hear you but consider this right, I mean right now: it's talking about the volume snapshot existing most likely. The backup application was the one that created the volume snapshot right, in which case it had access to the PV and from the PVE. It could get the provision a name that CSI driver name. So in other words, it's already got the driver name. So it could just jump, skip this step right, just create the snapshot, skip this step about looking up the content and uh you know, go for the metadata service directly. So.

F

E

C

Really not about looking up the volume snapshot.

E

As much as you still need, you still need the volume snapshot content to to extract the uh the snapshot handle.

C

uh The backup applique client doesn't need to do that. The backup client has access to volume snapshots it's it's uh this guy right, the sidecar right. It's actually this step over here, which will authorized, can needs to do that. So really really in this bubble. Here.

C

The only thing we really need is to get the CSI driver name, so you could get it from the PVE prior to making the oil before after making the volume snapshot or you could look it up from the volume snapshot content, but essentially that that that box should really say return on the CSI driver name, that's what it really is about.

E

All right: well, it I think we need to. We need to make a call whether we think that the per namespace backup application is a real use case, because I do think we could back ourselves into a corner where we say we're not going to support it today and then, if, tomorrow we change our minds yeah, we realize we made a decision that that really really causes problems. I.

D

Mean given that volume snapshot is already available right, this is an optimization for volume, snapshots, so you're, basically saying yeah. We have this optimization of it available unless you know you're not you're running as a non-privileged application and what's the logic to that.

E

D

B

E

I didn't follow all that I'm trying to.

D

So so, right now, I can write an app I mean backup is not the only use case for volume. Snapshotting right, you could be. You know, there's a there's any number of things and it could even be something like uh say. For example, you have an operator that does its own backup. This is something we've talked about right um right now. It could do that by say, taking a volume snapshot and then doing a block dump or whatever it does it.

D

Maybe it knows how to read the format or something whatever it does right, but it can do currently. It can currently use volume snapshot, only it's resources, and that's just fine. So why would we add an API that gives you an optimization on volume snapshots but not make it available to anybody who can use volume snapshots.

E

The the the word use is doing a lot of work in that sentence, because what what an end user can do with the volume snapshot and what an infrastructure component can do with the volume snapshot is a overlapping two. Two overlapping sets right.

D

What's different right now,.

E

So the the ability, it's the ability to read the volume snapshot, content and get its handled.

D

That that's not a that's, not a usage! That's a detail.

E

Right but but in order to interact with the actual storage device and like do things you need you need that handle so like ordinary users. Don't need that information to do what they need to do, because the sidecars do the work for you the moment. You're writing a new infrastructure component that might have to interact with the action.

D

So we're we're adding a a utility to kubernetes, basically a function. You know functionality here so.

E

D

Functionality is.

E

I can create a snapshot, hold on, go to side, 16 or the the previous go back a slide or two in the. uh Maybe it was the one before that. Maybe someone after I the the new sidecar we're adding.

D

That's an implementation right where, if you look at from the top from the requirements point of view of what you're building, what are we building.

E

Well, who's going to ship the sidecar, the sidecar is going to be shipped by.

D

That that's an implementation, Ben.

E

The storage vendor right.

D

That's an implementation, so what capabilities are we providing and why would we not provide these capabilities along with the existing volume snapshot capability.

E

I guess I need to think through which pieces are likely to be present on any given cluster under different circumstances and whether there's any novel way to put them together that we haven't thought of, because what we're envisioning here is a cluster with a a storage device. That is explicitly supports. The CBT API and a backup application that explicitly takes advantage of the CVT API and the the glue between them is.

E

That is, is that snapshot, metadata object and the the rest I'm. Sorry, the the grp, the HTTP grpc, that that goes between the backup, application and sidecar right that that's the glue.

D

Right, but from a functionality, I mean I really want to attack this from a functionality. Point of view, because this.

E

Is there I get what you're saying I'm just I'm trying to think like so so? If, if we Design This with the whole cluster backup use case in mind, and then someone says, hey I've got an idea I'm going to write one, that's namespace specific like can they do that? That's what I'm trying to figure out right.

D

Or why wouldn't we want them to do that because we already give them the access to volume snapshot. You know create volume from snapshots.

E

Yeah but maybe maybe it would just work, maybe it'd be fine, but but you'd have to change the subject. Access review to not not check access to the volume snapshot content right because that's not actually required to do any of this yeah.

C

I think we will do that because it's not required like I said this really means we need the CSI driver name and that's it.

D

Which is already available, it's not a secret right, I mean.

C

It's available for two sources: right: the PV, all the volume snapshot content. So what we're saying is that the backup app has to have access to either of those two. You know you get the PV by the PVC whatever right and who do you get the PV.

E

Well, no, no, you could get that string from the uh from the storage class or the snapshot class. Well,.

F

C

If you as long as you get it, then you know that's true, then I'm going to definitely remove this VSC from here, because it's not necessary. We really only want the CSI driver name, so we can make this search. Okay,.

E

Why I I feel like if, if it's hypothetically possible, to just create a single namespace backup application, then there needs to be a handle somewhere in that subject, access review that says this? Guy only can access snapshots in this namespace.

C

Yeah yeah. That is exactly what.

E

Is doing CR.

C

Yeah, in which case I as I said, go there. We can do this with the current thing. It works good, okay,.

E

I'm happy so sorry to go down a deep breath, hole.

C

On that I don't know it's selfish, some flaws. Let me know we over I think I over specify this thing by mentioning the SC in the front shouldn't be there right. So it's good.

C

So, anyway, the next steps right, uh okay, we'll update it- obviously update this stuff, which right and uh we're thinking uh you know, prasad's already done some prototyping, actually, almost out of time Prasad, you want to say what you, what you found out.

F

Yeah I just uh validate validated the token request and token review part um I loved it. The existing entry prototype to get it.

C

So that that's really cool so yes and we'll we'll update the design with this uh with this stuff, so I think we're good it. You know I'm actually feeling very positive about it, because it simplifies everything- and it's definitely you know a nicer more elegant and easier to deploy model.

C

All right, that's all I had that's.

A

Great, thank you. So we don't have to go back to see goth right. We got other because.

C

You don't have to go back to.

B

Them no I think the next time some of them will be involved would be um at the cap level.

A

Yeah yeah Jordan.

B

David, it's uh they've seen the older PR, the older cat before um so.

D

B

Like um going down the path, they will continue to show some interest in the capital.

A

Okay, that's good.

A

All right um are there any other questions regarding this uh cup.

E

Ben I appreciate you being exhaustive on this: it's how we all kind of get to Crockett my pleasure.

A

All right, then, thank you so much all right, bye, okay,.

D

Thanks everybody good discussion.