Kubernetes Data Protection Working Group, 22 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG Bi-Weekly Meeting 2023-02-22

Description

Kubernetes Data Protection WG Bi-Weekly Meeting - 22 February 2023

Meeting Notes/Agenda: -

Find out more about the DP WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

Hello: everyone: it is February 22nd 2023. This is the kubernetes data protection working group meeting I. Think today the main topic is we're going to continue discussion on CBT. Then I can give a quick update on the volume group snapshot cap all right, so uh Yvonne or Prasad, who want to start I, can make you both co-host.

B

Yeah um so try to yeah so today would mainly the Prasad, um so I wonder, hang on a sec, hey call I, wonder.

C

B

What do you think shall we go over it today or shall we um because um I'm hoping like people like um would be around discussion.

B

D

This is a very small audience.

C

Will discuss with Evan and Carl? It's.

A

The same thing that you discussed last time did you uh last time? Did you share anything new? Was this uh see this slide? That is the new. Is this still old.

B

This is the uh is a wish. It was just like uh follow a follow-up from like the last conversation with the band like bandian um I,.

A

Also love you yeah. That normally was back here last time. Oh.

B

No have fun yeah, so I think, like Prasad, has a different diagram that he wants.

A

B

Yeah we went over this last meeting, but we didn't get a chance to record it because.

A

Do you have any uh uh slides or some you say you have a different diagram? Where is that diagram? Is this song? We can just take a quick look and see how that looked like.

B

Yeah yeah go ahead and.

A

Sure, let me stop sharing so this way you could share hi fun. How are you doing glad to have you back to the school.

C

Yeah I joined back as a as an individual.

B

I'm paid right- yes, oh this is uh so yeah um yeah Prasad. We can walk the group through, uh but just knowing that you probably have to redo this again in the next meeting. But at least like we can look up Jenkins to get her feedback and we put.

C

Yeah um all right so to give the context. So this is the kind of very high level theme uh where we are trying to achieve uh with CPT data path, uh so backup software would basically call uh CPT service endpoint and uh we will get the change block data and for authentication, uh so mpls is obviously is the is one way, but that adds dependency and extra overhead of managing certificates.

C

It is also add, depends your you know, third party tools, if you want to achieve the um mpls using service mesh or start manager, for example, uh so we uh discussed and thought of uh alternate to this. uh How we can you know, use kubernetes apis to um to have the authentication and uh that way we don't have to go into managing all these certificate uh certificates and uh we don't have to rely on uh the tools like service meshes or cert managers as well, so I would divide the whole workflow into two parts.

C

uh One is authentication and then then the actual, uh how the data flow happens so for authentication um yeah. So first talk about I'll talk about the components we will be discussing uh so CBT client is basically uh the client would be requesting the Change clock data so backup software could be one uh symmetic client you can imagine uh currently CPR server and the CBD Service Board. The security service board will basically have uh will still solve two endpoints. uh One is the AG API service endpoint and the possibility service.

A

Hold on for a minute, uh so are we going back to aggregate API server now I thought we are not either.

D

No, no, no, the server is one thing, not AG. Api objects.

A

But what is this aggregated.

D

Let them go through is the point is this: this first step is only to to initiate the transaction I set up a session, and then you are free of the cube API server for the rest of the data transfer. Okay,.

C

Right so adjective may not.

B

C

Term here we can say resource uh Handler or something like that um yeah, so, basically, um so once client will will so for authentication. You can kind of relate this workload with a GPS cover, um so client would basically uh using go client or any client would.

B

C

The volume snapshot Delta uh endpoint, then so this request goes through API server. um We can. We will be sure that this happens uh with TLS authentication encryption in place right, so all these calls will be secured.

C

um So as this CBD service or starts it will uh do this server side, authentication authorization using the service token it has, uh it will make sure it has. uh You know, permission to access the following snapshots and other required resources.

C

uh So once the client makes a request, the request will go through APS server and um through API API service resource will will. So. This is similar to how tag API service igps service, uh request flow happens right, so the request will go through the security part. The CBD part will authenticate client and it also do authorization using the service account token.

C

It provides okay, uh so it will check uh if it has permissions again to uh value snapshot, resources and if it is, it has permission to call uh non-resource URLs like the CBT endpoint uh it it serves.

C

So next step is the service creates a token and in the response it it returns URL um to get the change group data along with the token okay. So this object will not persist anywhere. It's similar to how we had designed the list change block API right, so there will be unique token per session and once the client gets the token and URL it will make the grpc call uh with with that client on the on the endpoint to get the change of data.

C

So this authentication flow will be secured as it's going through the qvps server and using the kubernetes ca. So so it will be secured and TLS communication uh The Next Step once it gets into token, um then when it when it calls the grpc endpoint um we so so, with the token, we can assume that the trust is already there, so we don't have to do again. The server-side validation, server, validation at client side, uh so that removes dependency on you know.

C

uh Having the certificates a client client doesn't have to, you know, provide its certificates, so we don't have we don't. We won't fall into that so we'll just using the token between call the grpc endpoint and get the Change clock data from the CBT service.

C

So this way uh we are um kind of splitting this flow into two parts. First part is to um to um do server and clients, so it validate authentication and then once this trust is there, we will directly call the grpc endpoint possibility service endpoint to get back to your data. This uh that makes sense.

A

uh Can you talk about the deployment model? How do you deploy? Let's see the you, have a your CPT power, so what will be deployed together with your with the CSI driver? So this uh CPT the side? It's like a side card, you put it like a side. Car yeah.

C

Yeah, it will be a sidecar to um the CSI plug-in container.

A

Deploy so if a CSF driver that want to support this, then they basically need to deploy together with.

D

This so I missed it. I pasted the link to this current CSI deployment picture right.

A

D

It would look pretty similar to this stateful set deployment. Part right, you would have. You would have a sidecar with this service right. uh You.

A

Were actually uh actually yeah, let's take a look on this one yeah, okay, so.

C

Yeah, you can assume uh CBD service are deployed on the same spot as a sidecar to CSR plugin, so that it can call the um CSR plugins.

A

Will also deployed with all the other sidecars right. The extra pressure snapshot is all here in this big pod right example.

D

It doesn't have to be the same part, but it you know it's up to the implementer right.

A

But then you will have to deploy several instances.

D

There's no have to it's a choice of the implementer but yeah so.

A

The thing normally I think choose to do that right. Otherwise, you true true I.

D

Mean otherwise.

C

You'll get requests.

D

A

Because otherwise you get the cost, and how do you know you don't take.

D

It I mean yeah, but but what I mean by that chain is that you know if the CSI implementer wants to do any other, weird form of ha or other things up to them right. We don't really care.

C

D

It would make a call through the Unix socket to the CSI implementer of this thing, which will actually generate the URL and token because that's obviously not in the not in the CSI in front of the general spec right, and that will be returned back through that Unix to that Unix socket back to the side car which will return it back to the client and then at that point onwards the client will make a call to that URL with the two and use the grpc API to pass the token through or whatever the protocol might be.

D

At that point- and you know that's talking straight to another Green Box in this uh in the cluster.

A

uh uh Okay, so can you.

A

Sequence: diagram: I'm. Not seeing, can you add a seat driver in this diagram as well, so this one does not have to see the driver. That's.

D

Streamlined that's streamlined, CBD.

A

D

A

D

A

There is a call right if you called.

C

A

Basically, you will add.

C

A

It'll be good if you can add that one.

D

We should start with that uh with the deployment diagram, it would be easier to picture.

A

Right, you could probably have both or you.

D

Could have a deployment.

A

Diagram and also have a sequence diagram since the this other one is it's like a secret diagram, but you probably I, don't know if you were trying to combine them, but yeah maybe have one like this and then the.

B

A

Will be the sequence diagram that shows that the driver there as well yeah.

C

D

A

C

D

On that precept, yeah.

A

B

So how did you say, like um token is and session is generated by the CSI Plugin or is it generated by the CPT Service.

D

Probably, by the by the CBT, sir, by the CPT service, I mean the the sidecar exists to listen and communicate with the CSI driver in a standard fashion correct. So uh all the implementation details are in the CSI driver, so it makes sense that the CSI driver returns both the URL and the token, because essentially, when that token is conveyed to to the through the URL into in the session, calls it has to be decoded by the by that receiving entity right.

D

So the sidecar has got nothing to do with the format of the token it's opaque, but it has to convey the token has to convey everything which is happening over here right.

B

So I I guess like I, just um okay, um yeah, yeah I, think I get that part um I. Just don't click to confirm. I, don't think like um so the CSI plug-in is like the plug-in. The container that's maintained by the provider right correct, so I just want to make sure that not every CSI plugin have to implement their own token. In general,.

D

No, no, no, in fact, the thing is the the way the CSI deployment model is that picture right that that little green container right it talks via Unix sockets to all these sidecars right, so each side car is talking to it. So there's a protocol there and uh all the logic. The business logic of the domain. Knowledge of that environment is in that green picture, right that the CSI driver can data. So there is nothing we can do in aspect which defines how the snapshot data is is constructed. It is up to the implementer.

B

Right so I'm talking about the authentication part, not the CBO.

D

The authentication part has to be defined by the standard.

B

So so, yeah again just.

D

Talking sidecar is doing that work, so if you bring up the picture of the deployment you can see where these two, where the, where the, where the two pieces uh you know, are handled.

B

So in this case there are two yeah right, so there's the green box that do the authentication, authorization, essay or the yellow box.

D

So but precise, can you bring up the figure Johnson? No, no, the the the CSI one, the the component diagram.

C

D

Okay, yeah so over here there's another side: car in that box call with the external attachment external provisioner. There is another side, car called say: CBT, whatever CBT.

B

Service all right so yeah that part we yeah okay, so.

D

I think that part is going to do the authentication and authorization of the call.

B

D

Succeeds, if that succeeds, that one is then going to use the uni the Unix domain socket to talk to the green box. To say this. The here here are the parameters return me a URL and a Tok and a token correct.

B

D

B

Me a URL and a token, so the URL and the token come from the green box.

D

Comes from the green box and.

B

Then, hang on a sec.

D

um Flip back to the other diagram and see that sorry.

B

Hang on a second diagram, um so if I'm, like a storage provider, I have to implement my own token generation mechanism inside the green box, correct correct another provider, I have to implement their own token generation mechanism. Things like that, their Green Box.

D

What is the token convey right so, and that depends on the on the protocol. We're going to Define for the grpc part see so uh Prasad. Maybe you can expound on that little bit more next time, because essentially, when we look at the API right, we have to convey through the API what will the context in which it's operating correct so so that token has to represent that context. Now how it's done is totally up to the storage vendor.

A

Yeah I think, if you can actually that sequence diagram you can, if you draw out other uh all the components, so the what is missing is the CSR. Plugin is not there right. If you have that, then.

C

A

Who is creating those? Because.

D

Right now, I'm.

A

Getting confused yes,.

D

I agree, one more one: more swim link one more swim, Lane to who's. Constructing this thing, yeah.

B

So essentially,.

D

B

So it's actually essentially for the um provider to provide a CBT service. They also have to implement an additional token management.

B

D

That's right, so so the piece whatever whatever is actually going to do the business logic right to actually generate the the diff or to generate the changed blocks. It doesn't come totally for free right. They have to do work so that work includes getting a URL generating a jrpc endpoint.

D

Maybe they spawn another process, they we don't care, they can do whatever they want right and they have, and they have to give us some sort of token, which is going to incorporate and represent the fact that you've gone through this authentication or authorization phase, because once you go into the yellow box below right in this figure, you are now talking with uh you're now talking encrypted, yes, but not with mutual authentication. Necessarily it's that token.

D

That token is delegating the authentication or authorization right through a secure, Channel and now can be used in this yellow box, with with lower with lower level of without Mutual authentication. Put it that way. Yeah.

B

So um so, essentially inside your CBT um RPC spec, there is an RPC for authentication specifically to get some sort of CBT token there's. Also a separate set of RPC calls to get the actual CBT metadata right right and.

D

They're they're.

B

D

I mean there would be. The first call is through the kubernetes API server. The rest of the calls now move into the data API right, a grpc based data API.

B

So every time you'd make a grpc call. You have to like pass the token into the provider framework or whatever and say: hey can I do translate token.

D

Yeah, essentially, you have a session right. You, you said I want to get the change blocks for this volume between this wall. With this volume snapshot and this volume snapshot, so it returns, so it authenticates your request, validates it and then returns your session token and a URL to contact. They use that session data to talk make make all your calls and you're done.

B

So as the CSI driver maintainer, which um I guess for the most part, I'm not- and it doesn't really affect me as much to be honest um so essentially, if I were to pick up this I want to expose this I want to implement a CBT. It's a bit scary.

B

So the kubernetes token there is the CBE token, which I may I may or may not have in my back end to support, and then there is like um the CBT RPC call um um yeah like I said for the most not like um it doesn't affect me like um you'll, be interested to see like.

D

Right we should. We should Implement something which, which has this uh a CSI driver component, which does this right and.

C

D

Libraries client Library example, client Library, built against the grpc spec right, which can demonstrate how to use it.

A

B

That'll be easier.

A

um It'll be good to also get feedback from the security team side. Just you see.

B

D

B

Addition to the CSI driver also has his own mechanism to authenticate with his back end right so, which is like you know,.

A

um Right now, or with you anyway, right so yeah.

D

B

D

That's in the green walk, that's in the deployment area, not so not this green box, but in the in the CSI driver deployment area, correct.

B

Right again, that is nothing to do with CBT right. It's just like a CSI driver, so I'm, just like I'm thinking from a CSI driver, maintain a perspective. It's.

C

B

You know it has to authenticate with the back end, which it already does. Yes,.

C

B

Then it has to have a special way to authenticate with CBT endpoint a token of some sort, um and then you have to deal with kubernetes token with you um I guess all this I can. If I press person say all this is here to avoid like a TSL Supply inside um authentication, yeah.

D

We absolutely do not want to redo what kubernetes does for for us today correct. We don't want to take on any management of network security, because that's that's not our 40 R40 is the domain over here. Kubernetes does all this for us, so we have to piggyback as much as we can on the functionality that kubernetes gives us.

B

Right and and I agree with, um you know that that principle so, but this that is this, like an extra like um session token, oh.

D

Okay, so this this is, this is actually we talked about this with Ben I think two meetings ago or three meetings ago. Essentially, you know this is a data API correct and uh we are we've all agreed right now, based on feedback that we don't want this to go through the kubernetes API service.

D

So that means direct communication with something.

A

A car hold on a minute I, so I think there are two paths right. There is a part of. It is still like a control path. You're talking about what are the blocks, then. The second part is the real data right, yeah.

C

A

Just the control part right, not really the yes.

D

So this is the control part. What Prasad started showing you in the in the Sim link, figure right and the swim. Lane figure, it's all about. It's all about the control path, the weight, the cube, API server, you're, making an up uh uh you're posting a request right through Cube API server, which is because of an AG API service type right register there. It goes all the way without creating an FCD object. It goes all the way to the service.

D

The service does all the authentication and authorization required, but doesn't create an object in that CD. Instead, it returns back here. Here are the end points to connect to now use the grpc protocol to get the change blocks.

D

That's all control.

B

Can we uh um I don't precise and I chatted with about this earlier too? Why can't we just use the um the service account token? What do we need? Two tokens.

D

Okay, so let's, let's not mistake the word token token, as a general, it just means something opaque. You can call this a session token if you want to make it clear, but the point, the point is the session token is something which was constructed as part of authenticating and authorizing, and it it basically carries your right to make a call to the other API.

B

Is the network token review and subjects review API? It's.

A

Basically, like credential like it could be like basic credential or something yeah.

B

Because the first in the first box on the right, you already asked kubernetes API server.

D

It's more than just look it's more than understand the domain of the credential. Okay, the the domain of the kubernetes credential coming in was all about security and identifying who the client was it's that type of a token right. It's a security ready token. This token right is something constructed by the CSI driver to understand that it's this volume, it's these snapshots, it's this user, it's you know all the underlying stuff, and it goes with the URL right, because that's also where the CSI driver decides what endpoint to expose.

D

So this this is data session data which is not in the in the kubernetes domain. It's session data in the CSI driver domain.

A

So what format will be this session token, so it could be a secret or it could be something else.

B

A

Is the uh is this is up to the CSR driver to decide the format of this token? Yes,.

D

That's exactly why we need another swim Lane to show where it comes from. They.

A

Maybe could you quoted something else, otherwise it could.

B

Be confusing true because.

D

The one thing so two things one is: we need another swim Lane, so that we can see where this token is created right and the other thing is, let's name the, let's name the fields so that they're very suggestive right, yeah.

A

Let me just record um uh what is a CP uh CPT, credential, CBT, URL question or add a add some explanation right. Even if you want.

D

To yeah, for example, I would suggest CS the CBT session URL and CBT session token right. That would be so much easier to understand. It'll be very clear.

C

B

Not sure, like you, um I mean yeah again right um just like ask them questions not disagreeing.

C

B

C

B

Know what additional information, what additional session information that will attach to you or provide additional identity uniqueness so.

A

I think this is a uh this is like the information. Maybe you need to access the the storage system or or maybe not right. This is the for the for the URL for the CBT URL.

D

B

D

It is information for that yellow box below right uh for the client to communicate to find the end point. That's what the you know the URL is about and for the service at the other end of the endpoint to understand the context in which the client is operating. Okay, so now.

B

That what is this context, though,.

D

The context is this request right. The request here is volume snapshot Delta, so.

B

What make this request unique.

D

It's this request, so the client made a post right and got a response. Then it takes that response and makes a grpc call so essentially the the it's a session. For that thing, it doesn't have to be unique. In any sense, there can be five sessions on the same two pairs of snapshots. Who cares right? But the point is it's: this client it's this request and something is going to handle this request. So that's what session.

A

Now actually needs to provide some some service, which can you know, give you this CBT data right, CPT.

D

To connect to that service.

A

You need to have some credentials, which is the session token. That's.

D

A

Then that can be well, maybe it's a kubernetes secret or it could be some other format. It.

D

Could be anything whatever.

A

D

Want to do exactly, we leave it to the implementer to do.

B

A

B

Again really I feel like it's not providing anything additional to the Token. The service account token. So.

C

A

B

Okay, so the request doesn't have to be unique. That's what we just said. So, just like some sort of thing that says this is a request from client a, and this is another request from client, B and client. A is asking for the CBT between snapshot, one and snapshot. Two yeah and client B is asking for the same thing: Snapchat one and Snapchat. Two. The only difference is playing in client B, which is really just in service account tokens which again.

D

B

Token review is doing right.

D

But all that is taken care of at the first phase, right, authentication and authorization is happening in the first. The first kubernetes call. So you know, let me make an analogy: Ivan, it's like open, like uh like open in the file descriptor right in posix. At that point, at the open call right, it returns an FD, and then you do I O with the FD right. Well, that open call is validate authenticating, you and you know, based on for your credential, your user, ID, Etc and and the and the path you are opening.

D

But once that is validated, it goes down to a lower level of the kernel where you get a where you get a file. Whatever you file, structure, Etc, sign, stuffed it to your user area and your process, and you got to file the script back from that correct. And then you work on that and multiple processes can open the same file exactly.

B

You do the ACL once right.

D

So it's a session, so the file.

B

D

Session so likewise over here.

B

You have one token from kubernetes and another token from the provider, so essentially I'm saying they are more or less they're likely. The same thing.

D

But the sessions are used in okay, maybe we're talking cross purposes in the green box. Above This negotiation is done securely with kubernetes API in the yellow box below this is done securely, but it's valid. It's done without Mutual authentication and it's valid, and the The Authority is conveyed right via the talk session token, which was securely established.

D

Think of it that way.

B

So the CSI driver already knows how to talk securely with their backend.

D

Of course, I mean the CSI driver. Implementation is not part of this picture. Right, I mean how it does it. It's up to the CSI driver.

B

Right so I'm saying like that, I feel like that diamond box is not necessary. Oh.

D

Because so so, what the diamond really he needs to add another swim, Lane, okay, there's another create session. Token, really comes from the CSI driver, okay, so there's and that's done through the Unix socket so yeah. This is the CSI driver component.

D

So there is a call, there's a call made here.

B

Regardless right, the CSI driver just passed along the to the back end and say: hey, give me a session: okay, okay, right, I,.

A

Guess maybe even even your question is: why can't synthetic directly just return those things.

B

Right, yeah, I guess regardless, regardless of how this I mean like with with the CSI drivers, try to do the session management or would it tap into its um like back-end session thing like you use AWS as an.

D

Example, the CSI driver has to do it.

B

Right, but would it like um um okay so forget about how the CSI driver does it, regardless with the CSI driver, generate the token or a CSI driver called AWS, SDS and say hey give me a temporary token.

D

I mean I shouldn't, say: CSI driver group create a secret and just in in its own, in its own namespace, which is secure right and give the name of the secret to as the token and then when the Call Comes in right to the grpc service. The grpc service looks up the secret and gets all the context so.

B

So yeah, so so, whatever.

D

B

Simple, it's a CSI driver, so let's assume the CSI driver managed the entire life cycle. The csis driver have a state of the session because if you crash a restart, you still remember where the token is good or not somehow the goal magically taken care of right um I would like to know. Okay, maybe like, if you want to extend this diagram a little bit. I want to know like um what country. What makes a session like?

B

What makes the session a session so client is say: I want CBT for snapshot and snapshot B and then.

C

B

B say: I want the same thing: okay,.

D

Don't bring in let's, let's stick with one request, so the session is.

B

Tied to this well, no, they have to be multiple requests right. If it is one.

D

Request but let's, let's handle, let's handle the problem this way. So this is a post call. You said: what is the session correct? The session is returning the result of this post call. The Post call returns an immediate result, but you haven't got the data. Yet right, you haven't, got the changed block metadata. So the session is all about retrieving that change the change block metadata and then it's when you finish with it you're done it could have a time to live whatever you want. As an implementer. You put that in right.

B

D

B

That's that's what it is. I mean it's so so why can't we just stop it after the kubernetes, API server said: okay, I can trust this request.

B

D

Can't we I missed the question so.

B

So why can't we just stop after we call the program review and the subject, access review apis? What.

D

Does that that doesn't return you the data which you're asking for right you're asking for the snapshot, Delta change blocks.

B

A

B

That comes in the yellow box, right that.

D

B

So I'm talking about only the green box, I'm saying like but I'm just saying like um why? I still don't know why, like the diamond box, is necessary when you're already.

D

Kill the diamond box: okay: how okay, really we have to the diamond box, is very misleading. We need a swim, Lane cut and paste that vertical line and put.

A

It under that I think uh your question seems to be like you, you think well, I can't even get that in just one call. The first call but I think the the problem is. If we do that, then it's going to go through the API server again right. That's.

B

A

We're trying to avoid.

B

Well, we are going through the API server.

A

Yeah but the aps server is not getting the data back right, so we're only saying: hey uh we're good. We get back the token.

B

Right and I'm not talking about the CBT data I'm talking about the authentication part.

D

You're talking about the authentication part.

B

Yeah I'm talking about only the green box, yeah.

D

Yeah! That's so that that that box should be on the swim, Lane again.

C

D

Don't know how Prasad draws swim Lanes, but again, whatever.

C

The thing is exactly.

A

D

Action taken on on that call right that authentication not declined that that actually is the target of the left of the left swim Lane. Then that makes a call to another as Prasad we can work on the figure later on, but now we have we have. This is the correct part: a set of actors involved right. The CSI driver actor was missing from this chart before so now you can see that the CSI driver actor creates the session token and returns it back, and this is done.

D

If you look at the on the look at the other figure on the deployment side, this is done through the Unix. Socket Unix domain socket right this. This sidecar calls this driver. The driver returns. This data.

B

So it makes sense so that skill has an answer. My question: why, like um okay, so go back to your uh your chart.

B

So if you go to your yellow box, yes, so scroll down a little bit so right here, right, um so okay, so like um so right here um so.

D

This is wrong. The figure is wrong here. These calls are made now not to the not to the side car but actually to this service. So you have to extend the yellow box to the right and talk to the second actor. It's the calls are made not to the side cup. The calls I need to the CSI driver piece which is listening so.

B

So regardless, um who we call so there's some endpoint out there um same process or not, regardless of um logo routines or whatever right, um I guess I'm still trying to understand like um so you say it's your grpc call with session token list change blocks. So why cannot we estimate part of the grpc call?

B

We submit the service account token, and then we asked kubernetes API server business. Or can this be authenticated and authorized if kubernetes API server say yes and then we just call like uh we just search the CBT.

D

I I hear you: let's say that let's say there were a thousand calls made right, or every call would have to be authenticated in that remember. This is now a data transfer API. This is not a a control API, so the whole. The whole point expressed in the previous meetings was about how we overload the API service. Even authentication requests overload services and the other part is the other part which Prasad I think had researched again. I'm, not a I'm, not an expert in security, so we really have to get this through security.

D

Is that it's very hard to authenticate to actually expose an API, which is trust signed by the? What did you say Prasad by this by the certification Authority.

B

So forget about um certificate. Signing so forget, okay, so the server will always have a certificate. Yes right, the client side. So let's say the client side doesn't have a certificate, so we don't want to do mtrs, but at least the communication would be over secure, grp, at least it's encrypted correct, so like regardless of thousands of requested.

B

The the green box will still need to happen right. So in the first part of the green box, is you still have to ask kubernetes API server? You should scrub a little bit. um What inscribe like that, that first gray box, token review and subject access review, whether it is 50 requests, is a thousand or million requested that first grade box will always happen. You still have to ask kubernetes APS. Yes, you will still ask to win at least appear, so it doesn't, it will still put load and pressure on the kubernetes apis.

D

Authentication.

B

And authorization yeah so, but what I'm saying is if you scroll down to the yellow box um um like when you make a thousand grpc calls each one of them will still go through the same mechanism and it will say, as the kubernetes API server can I trust. This service account token.

B

Without going to that diamond box right, like the the Diamond Box means, like um you know, we have to create 1000, like those volumes snapshot. Deltas um custom resources there in the green box, but this is just one.

D

Quality again, okay, so yeah I I, don't I, don't get the I think Prasad has to expound. What this thing is you understand what this volume snapshot, Delta object looks like and I'm sure Prasad has a different figure for it different definition of it. Maybe we should call it a session, talk response or something I.

B

Don't know what yeah token yes,.

D

Yes, that is your confusion. Right.

B

It is no, no, that's not my confusion, my my eyes Okay. So I still don't know why, like um the session, token, is needed.

C

This communication will be open right, I mean so, if you just send a service support token, along with the requests and rely on service to validate the validate the client based on the service account token, the communication will be still open. It won't be um a TLS communication right so.

D

The client has to trust the server and the server has to trust the client agreed.

D

So it can be done directly as in the case of the green box right, it can be done with mutual authentication and authorization done through CSI through the kubernetes mechanisms, that's in the green box, and that that's because the client is coming in with say the whatever Q kernel Authority give granted to the client right. It has all that type of stuff and we, the cube API server, is the one which is trusted and then there it then forwards the request to the AG API server, which responds validates all the other stuff.

D

Now we want to bypass the cube API server, but we still want to have mutual authentication accomplished at the end of that, for all the data transform so.

B

What we use is the delegation.

C

D

Since we use encryption over the Y in this yellow box, you it's a secure call coming in the client. Can trust the fact that the endpoint came as a response from The Trusted request and the service receiving the request can trust that the token right was issued by it, because the service cannot. At this point, the service of the yellow box cannot identify the client and doesn't have to actually you just trust the fact that the token conveys all the all the authentication granted in the green section.

D

You get you get the delegation part.

B

I I get it I. Just don't agree with that. Yeah.

D

Oh, no, no sure that that's okay.

B

D

A very fair, that's a very fair disagreement, and we need to put it through people who are better in security. Okay. This is a proposal so.

B

We can do this one, let's get the proposal again, yeah.

D

Let's get the components correct: okay in the figure, so we don't so we don't get sidetracked and then we can totally rip the security model apart. I have no problem with that.

B

Yeah sorry I'm just thinking um yeah.

B

Okay, um yeah I. um Do you have any other thoughts.

A

Yeah this I'm not quite sure if uh this model uh is Good from the security point of view, I guess this is definitely something that we need to ask them this. Is it.

D

I agree: I agree with.

A

That this is not something that we have used before right. uh So it's basically now you get the token now uh you just trust this right, the the and then the backup software will be using this token uh right. So that's the part I'm, not sure. If how secure is this this way? So like you, you, you do the secret, basically, the first one.

A

First time you call that's the normal path, normal kubernetes, you got the authentication everything and then you just trust that whatever you got right then, uh after that you don't have to go through that again, yeah.

D

In some sense, it's not unlike any web token right, I mean.

B

So I think the uh you see like we, we we scare like a phone away.

B

He was like this is why I don't show up to this kind of meetings. I'm just kidding uh yeah.

D

It's very nice that only so few of us in this meeting, so we can shake this out. Yeah.

B

Yeah I think, like um anyways I, finally tell me that he wants to continue to uh work with us on this, but just subjected to his own other full-time job and stuff, like that, but anyways regardless I think fundamentally right. I think uh the the main uh disagreement um is that, like um so so, I think Carl you, you put it nicely by saying that okay, so the the the client trust the server via the the TLs certificate. I think that we cannot get away with that. One.

D

Right, we just cannot yeah.

B

But now how does the the the the the other part of the equation is? How does the server trust the client so in this in this proposal, like we're, saying that the client, the server or trust the client, because the client has a token that uh the server issue- that's right? Okay, so which I think like from that?

B

If you talk, if we use like that flow, I, think that easy, even that that that that that language, that that makes sense, yes, I guess like yeah the main difference that this agreement is a couch I, guess like what I'm saying is like the server will trust the client, because the client present a service account token that the kubernetes API servers it can be trusted.

C

B

We don't have to agree wait. um We can agree to disagree with that, but I think do we agree that that's the fundamental like disagreement there yeah.

D

B

Folks wanted a stronger authentication and authorization I'm only going to trust you if you give me a token that I issued you versus like um I guess, like my my way of my language, is.

A

The things that is kind of out of our control is like how does driver implement this right? It's like completely up.

D

To driver yeah and by the way I I, really think we can make it really strong. If say, every grpc call can mutate the token and return. The next token you know I mean that's, that's how, for example, web authentication works right, the JWT is, is, is mutated and returned back I. Think every call something like that right. So essentially, this is a session, and if the session breaks, there's no retrying, you go start a new session, but.

A

Then yeah so do we need to you know uh design some some pattern for this. One like how you're you're saying it's a completely up to the driver. I guess: that's the part I'm they could. You know it could have something that is actually not really secure. So.

D

That's the part, I'm not sure how we can.

A

Guarantee that yeah sure.

D

Let me spin out an example: how I would do it just just a very simple example, because you know I prefer to implement simple B. So if I'm, the CSI driver, I, would say, construct create a token I would spin off a pod potentially would run this service. It gives you isolation, it gives you, you know rate, limiting everything on that thing and I give it all these secret information and spin it off and forget. So my my CSI driver now goes back to doing its own stuff right.

D

The client contacts that URL, which now points to that new service running in that pod and it does its session and when it's done it closes off and the Pod dies. If the if the client doesn't call and say some time to live, the thought just says bye-bye: it's dead right, something like that. I mean that's, that's how I would implement it as a naive first pass approach.

B

Right I think this definitely this design definitely definitely makes it very simple on the client side. So the part where you talk about when the CSI driver comes up somehow have a secret or something that it passed along the secret and it went back to do his own thing. So that is the part that I'm not sure.

D

So let me take a concrete example: I've worked a lot with VMware, as you guys know, right so, and right now, I the way we talk to VMware, we spin off apart. You know because it has C GoGo bindings Etc. We spin off a bar with all that stuff and it we give it all the arguments which include say the the volume either FCD ID and the snapshot. Ids Etc and we tell it go, do its thing correct, so I would consider doing it in a similar, Manner and spin off the spot right.

D

This will give it all the identity of whatever came in this box here like here, you know: here's the here's, the snapshot name, here's the here's, the you know, volume sorry, FCD, ID, the what's whatever it has to be in CSI in CSA object term, so volume name uh uh and snapshot, content, etc, etc.

D

All the data is then packaged and construct concerned turned into the domain specific FCD ID and snapshot ID Etc and given to the spot, somehow through a secret or through command line, arguments or whatever you know, that's totally vendor specific and the URL is given to it and the token is given to it right and then that guy then listens for until sometime. You know until it you know until the time expires Etc, but it listens for the client. When the client call contacts it, it knows what it's doing and it starts.

D

You know running the whatever the VMware calls to you know: open the FCD and start getting allocated blocks and change blocks and all the other stuff.

A

Okay, so how does this par affects other parts running in the cluster I? Guess that could be a question? You know I know I know, but you.

C

A

D

A

Like that right so.

D

um I don't know, any piece of work involves some CPU research.

A

I know I would just put a part around it, but potentially what other people would ask those questions? That's why I'm asking yeah.

D

I mean the backup the CPU cycle is the CPU work is, is targeted straight to my backup service right, but if I'm using a kubernetes service is like okay system acting on my behalf, that's all, but the work has to be done so.

A

So like and now you have been doing this right so like even today, is that so that's what you're doing you have like a pause just for this.

D

B

D

And it's actually very fasting. Well, you know again we're talking backup right, so we're talking the the milliseconds taken to spin off a pod is nothing compared to the seconds required.

A

What about your? So so all your like the data, the real, not just control Parts, but even data is also going through. It.

D

Yeah yeah, and at least all the data is done in there here when.

A

You have like a dedicated Network for this.

D

We use the vddk how the net, how the data comes to the to the vddk.

A

Is what do we suggestions? Is it dedicated backup, Network, yeah.

D

I mean no, so we it's it's not there's no dedicated Data Network in this environment, but the kubernetes spec. Doesn't the CSS spec? Doesn't talk about dedicated networks right.

A

No I know I'm just saying that people you're asking those questions I.

D

Mean I wish I really wish, because that would help us in the tkgs case. But you know, but anyway whatever, but.

C

B

I saw that the end to level workflow works right. um You know so um again be calculation of time here.

A

You, probably yeah, you probably could you know you also other than you're adding this assist driver in here in the diagram, maybe also good. If you talk about like how are you going to uh Implement how they are going to.

B

D

A

I need some details, so people can actually absolutely.

D

I think they're at least two implementation Styles we can propose. This is one style. I just said.

B

D

That you know the CSI thing just also handles all the volume requests together and one thing it doesn't care. You know so the URL I mean look at the variance variations. We have this return, grpc endpoint it could be dynamically it's created per session. It could be one common thing for all sessions. Right I mean there's so much variation here. I don't have enough Network background to actually propose what's right or wrong, but the point is it doesn't matter the spec says: here's an address.

B

Yeah again, from my perspective, I think the uh the the the grpc call, the grpc um definition I think those are all good. um My only thing is like the the authentication and authorization part.

C

B

Whether we need an extra session token from the provider or can we just trust the um community service account token I. Think that's that's my only main thing. Otherwise, like you know, this looks perfect. This looks great, um that's the main part for me, so yeah I guess. My concern is from like a security and like, um like the token.

A

Management any other time I need to join another meeting now yeah.

B

A

So yeah, so please update this, so we can come back and talk about this next time. Yeah.

D

Okay, so I think we should meet during the week and just figure this out. Okay,.

B

Yeah yeah thank.

C

B

If you guys want me to be around just ping me on slack or whatever, okay sure, okay, thanks guys bye.