Kubernetes SIG Auth, 19 Jul 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Sig-Auth Bi-Weekly Meeting for 20230719

Description

Sig-Auth Bi-Weekly Meeting for 20230719

A

A

Everyone, this is the Sig auth meeting for July 19, 2023. I, think we have a light agenda today um for folks that weren't aware yesterday was code freeze for uh 128.. uh If you forgot, then it might be a little late. uh Otherwise, if you have some exceptions or anything in uh you know, hope those get approved and get your stuff done.

A

um Otherwise anyone have any announcements or anything they want to discuss before we move on to the main discussion topic, which is a sick storage cap.

A

Open uh radio um so I scanned this cap, but I didn't necessarily fully read it or understand it. Yet, um let's see so Carl and others are you all on the call.

B

C

I I have a slide deck for the camera, which is in the chat.

A

uh Would you like me to make you host so you can present it that'll be great. Yes, yeah.

B

All right one moment, sorry, so what color is that I'm getting set up like um I just want to do a quick like um background, slash contacts intro there I know like yeah. The the cap yourself is huge, um so we're hoping to like um break it down, and you know pick apart things that are only relevant to um the focus of this um meeting.

B

So yeah. In a nutshell like this is a sixth solution. Data protection, working group, cap and uh I know, like some of you, may have seen this cap floating around like um since late last year.

B

The idea is to introduce, like um this new CSI API into the CSI spec, that allows user to pull and request, like um change, block tracking data and what that really means is like, if you can imagine, like a user, has like a stateful workload attached to a CSI PVC that could be, like you know, anywhere from one to ten terabytes of data, say on a typical somewhat typical, like the relational databases out there, and when we want to do backups of the CSI volume right, like I mean set up on every time, um have the backup software or the CSI driver, try to um walk or back up the entire 10 terabytes of data change.

B

Block tracking is the kind of service that allows like um the backup software to say. Okay, you know between you know, TN and TN minus one. What are the Deltas and changes that have happened at the Block Level in this volume and give me back those Delta, Delta and I'll?

B

Only backup like those does so, and the reason why we are here is because, like we last year like um as some of you who have seen this cap floating around might remember like uh we have tried like multiple approach to try to implement and present this um CSI API uh challenge. There is like the amount of data is, you can imagine that comes back from the provided storage provider can be huge right, say in the worst case scenario, every block um on the um this have changed.

B

You know, I think you have 10 terabytes of all these, like hundreds of gigabytes of metadata, coming back um going through the SCD going through kubernetes API server, finally back to like the the consumer, so we have to like um kinda in a way like I'm thinking out of the box and so that the conventional, like crd, slash controller approach, is not going to work. um We tried the aggregate, the API server approach that got shut down because of this unbounded amount of data. They can flow through, like um the kubernetes control plane.

B

So you know we try to put together a device, like um you know, a grp, PC, streaming, endpoint and then part of the requirements um is that, like this has to be secured um from, like um you know, a a workload as far as like um cluster um security perspective. So hence again, we want to take the time to talk, walk everyone through the focus of the cap and zoom in especially on the security aspect of it and give some advice from you folks to see you know.

B

Is this like the right way to do it? What are the gotchas and how to proceed from here so Carl um wanna take us through it.

C

B

David yeah.

C

So, let's Okay.

D

C

This is very situations, so this this particular slide deck matches. What's in the published cap- and you know- hopefully you know if, if y'all uh look at this server, it will help understand what we've written out there.

C

uh So, as I will mention the goals of this design, iteration- and there were two previous iterations right- was to provide a secure and idiomatic CSI API to it's all. Buzzwords, okay, efficiently, identify Okay blah blah blah, but you know the point is we're going to get change blocks from a volume snapshot uh between two volume snapshots and all all the allocated blocks of a snapshot in order to do efficient, backup at the Block Port level? Now uh so the the technical details of how to do that are pretty straightforward and well defined by industry.

C

The issue really is integrating this into kubernetes and uh requirements for us. You know are like we cannot be any less secure than the kubernetes API itself and we cannot. um We. We have to use the existing uh container orchestrator uh storage provider, interaction pattern used by the CSI API implementations. Today right. We can't. We can't uh impose additional constraints of the service on the storage provider. Right um uh to you know to do anything extra fancy in terms of how we implement this uh by the the co hyphen SP.

C

These are like terminology from the CSI spec, so I'm just reusing it. You know these abbreviations, uh then we have to relay a large amount of snapshot metadata without overloading the kubernetes API server and, as I have mentioned right, the through previous couple of iterations. We figured out that the only way to get this is to just bypass the kubernetes API server, because no matter what solution we do, it's going to go through the kubernetes API server, uh so we have to make sure that whatever we're doing uh you know adheres to these.

C

You know these are these: are the goals for this iteration I'm? Very briefly, going to flash this that you want to move on, because it's not relevant to the presentation we want to make. We want you guys to examine the security aspects of what we're proposing. This is the use of an API. That's what we want to check that our use of this API is secure, not the details of the API, but you know just for background information. It's a grpc service right, which has a couple of RPC calls.

C

uh You know once called get allocated that returns, all the the metadata on the blocks of a single snapshot and the other is called get Delta, which will return the metadata on the changes between two snapshots, and you know the the rest of these couple of slides just gives us the detail, and the format of you know how we represent stuff right. So it's all there in the K in the written cap. But you know it's just interest of time: I'm not going to go through this data right now.

C

Oh so I want to talk about. uh You know what we are proposing, so the CSI spec is, you know, is supposedly independent of any container orchestration. But you know reality is you know uh we're doing this from the point of view of kubernetes. So this is our kubernetes implementation to use that spec. So uh so our solution requires that a kubernetes backup application will directly connect to this grpc I just showed you and fetch the change block their metadata through this API, bypassing the kubernetes API service.

C

um So the back the way we are solution, roles, the backup application- has to create a snapshot session, request, uh CR uh to set up a snapshot session and which authorizes the use of this direct connection, and, as you can imagine, right we're talking to you mainly about all this authorization related issues, um we'll provide a snapshot session manager to be the controller of this CR, and this manager will support all the participating CSI drivers in any cluster right, though, an individual session cannot span multiple drivers, it's it's per it's per um uh driver and there'll also be a community provided sideka.

C

We call it the external snaps snapshot session psych up and that's that's provided so that the storage provider can um can uh uh be isolated and insulated from the kubernetes side of managing the session, and so I'll go on to this the details as we go back, but you know just keep in mind what the overall solution is. So I'll start with this very intimidating picture right, which has big picture of everything uh and we'll zoom in some of y'all may not even be able to read it.

C

It's got so much detail, but we'll zoom in through this and go into details as we go along.

C

So, uh let's start by, you know, try dissecting what we're looking at so, given that we have a you know, a CSI cycle right which the CSI driver is going to deploy, and this sidecar is going to be the path that interacts with the kubernetes clients right- and why is this this?

C

The actual vendor specific logic used to actually to get the change block or snapshot blocks is going to be handled by the SP Snappers uh snapshot session service uh and existing paradigms today says that the sidecar pattern is used very commonly with CSI, and the existing Paradigm today is that the sidecar talks to the to the uh csips through a Unix domain socket. So there's no security ramifications, particularly over here.

C

For this thing, it's deployed by the SP vendor, uh but the most important takeaway from this is that this, the storage provider doesn't see kubernetes clients. The client of the storage provider is kubernetes, so in the container orchestration, the entire system right and it's securely set up to work with the system. The clients of the kubernetes clients are not clients of the storage provider.

C

That is that's a really important takeaway from this part. So over here we see that you know. So. This is one security domain, the gray one, the gray area. It's it's, the Unix domain socket stuff. It's already an established pattern, we're just using this pattern and we provide the sidecar to handle the kubernetes side of it.

C

Now our design requires that a kubernetes backup application. Well, it's going to directly use this uh grpc service, so the the sidecar we provide offers such a service right uh and uh kubernetes backup applications run in the cluster, so we're really going to make TCP IP based grpc calls uh to the Sidecar. uh Now the tricky thing is these are TLS calls not. uh This is like no there's, no Mutual authentication involved. uh Why? Well the sidecar itself we sidecar! We require to be deployed with well.

C

You know to provide a CA certificate and you know things like that, um so that we can advertise the the server side uh credentials to the to the client um and the client can trust the certificate they call. This EQ calls no problem, but we don't have- or we don't know right that security is not our background, but we don't know of any standard mechanism for arbitrary kubernetes client to authenticate or validate itself to a kubernetes application right. So we didn't know about that. Yeah we looked at.

C

We looked at various forms that maybe we could emulate. You know we ideally we'd love to just use the whatever uh ID kubernetes uses when we use say cube cutter right, we log in and we do stuff. We love to just use that we love to use a service account. For example, we don't know how to so, um at least without sucking in and maintaining and and basically re emulating or actually modeling what the kubernetes API server does today. We we do not want to put that logic into our Sidecar.

C

So all the rest of the solution is all around. How do we? How do we uh validate this backup applications connection to the side car uh you know so because the because the sidecars address and certificate is well known, anyone can connect to it, but we can't let unauthorized entities connect to it. So the rest of the solution is all about this. How do we authenticate this stuff?

C

uh So what we're really talking to you about is to you know, validate what we're proposing over here in terms of this green box and how the soil session setup is done, and uh so that's that's the background of what why this is such a big, complex solution, at least visually, oh, by the way, interrupt and ask questions anytime, you want uh so let's go over the high level flow, so it's kind of recap of what I just said.

C

uh So there's an independent manager which provides the snapshot session, request, CR controller I'll get into some of the details later uh there is this sidecar and the psycha is advertised to the manager by the by the presence of a simple CR which just gives its uh um CA certificate address and there's a label on it, so the manager can find it easy efficiently. Etc, that's all standard stuff, um then the back a backup client will create a snapshot session request uh in the snapshot. Namespace the manager handles this request.

C

Right gets invoked and it will look for that. Sidecar and uh sorry. Look for the service configuration object which advertises the sidecar it finds it. Then it you know, authorizes the the caller and creates a a session data object uh with a large, unique name uh and that's this object is created in the namespace of the CSI driver, and the manager then gives the the uh the name of this object. uh It puts that name and the the ca certificate and URL of the site car into the result of this session. Request.

C

Cr, the backup client by itself does, you know, does not have access to view Or List these objects in the uh CSI namespace.

C

Then the backup client then uses this session token and makes the jrpc calls uh it's talking to the side. Car the sidecar looks at that token then goes and searches for that snapshot session data CR the existence says. Okay, you know this is the token is valid and then it looks inside and finds the rest of the data required. What is authorized to be used and does the rest of the rest of the the validation of the RPC at runtime?

C

And uh it then you know, proxies this call to the storage provider session service over the Unix domain socket and proxies the results back. That's that that's the high level view of the of the solution.

A

um As a as a quick aside, I I know you, you were mentioning that you did not want to like get in the business of like copying, authentication logic or mimicking API server logic. You have built an authenticator here, just not a very good one, not to be mean about it. No.

C

It's not a good one, and yes, we have actually what we're doing is we're kind of doing it by we rely on our back. So it's more like a delegation type of model right.

A

Irregardless of the schema right like, for example, you store a secret inside of a non-secret custom resource, and then you copy it to a different one right. When, at the bare minimum you didn't need to store the secret, you could have stored a hash of the secret like there's, there's, like all sorts of like better actual security ways. Jordan go first, yeah.

E

I'll, let you finish talking through it, but I um I have some suggestions when, when you're done walking doing the walkthrough.

C

Yeah, please that's why we're here right: okay, yeah.

E

Go ahead, yeah. Do you want to stop now or.

C

Let me finish: there's only a few more slides I mean the deck is big, but I'm only showing a few images. So it's not that much left. Okay, all right so just go to the details. So the session set up, the storage format is not involved at all it's, so it's only on the community side right. What we're providing over here uh so starting off from where the snapshot session request is created right. The application then creates that request and pulls to the result, and so it's that's what the application does.

C

uh The manager has a validating web hook to check the contents of the of the payload and validate the payload of the object and uh then uh a little bit later right to the listeners on the controller right yeah, the managers controller gets access at this point. It actually goes and finds the CSI driver name from the snapshot or objects mentioned, um and then uh it searches for the snapshot configuration CR by using a label search uh once it finds that then it creates this session data CR and updates the result.

C

With the token um this part is so you know how does this actually work from the point of view of the application? So you know, starting from where we left off the previous slide right. We had the snapshot session data uh CR sitting in the CSI namespace and we have the session request results.

C

So the client starts by trusting that CA certificate and then makes TLS grpc calls and it provides the IDS of the kubernetes objects. It's dealing with.

C

Sorry yeah the sidecar, then you know, looks up that CR and uses it to validate the invokers right to use it, and it then checks the RPC parameters make sure they are correct. As per the session and just you know, other types of stuff which are related to the actual request, and then it proxies the calls with all the translations uh required for the storage provider and proxies the results back to the client.

C

uh You know so there was the okay. Given our limited security background, we identified a couple of the couple of issues involved which we had to address. One was the unintentional exposure to volume, snapshot objects by an entity, who's authorized to create a see the this request CR. So essentially, we put a validated web hook in place to make sure that when you create this uh snap uh snapshot session request CR, you also have permissions to access. Volume snapshot objects in that same namespace. It's a namespace object, so that says from a backup applications perspective.

C

Just because you have you have permissions to create this request. Cr doesn't mean you have permissions to peep into you know, to use it to grab a snapshot data from any arbitrary application. You also need. You also need access to the volume snapshots in that namespace. In order to do this, that's what it was addressing uh and then you know we're worried about obviously about the spooking up the session token, because the sidecar's existence is well known, so the mitigating steps we take for this.

C

We use a very large, very long string of randomly chosen, kubernetes name characters for the name uh yeah we could put the we could use the name or we could put the name in something else and use a token search. We can. We can address all that right uh and that's what we're really talking about. uh You know some of the feedback we want from you guys. uh More importantly, the visibility of this object is restricted by kubernetes are back to the CSI driver and to the manager.

C

uh Essentially, we don't want the backup application to have access to this. uh Then we also impose a Lifetime on this object. So there is this inspiring field in the object and um the expiry timeout is enforced by the sidecar and is enforced by the manager. The manager will also delete all these objects as they expire.

C

uh So in a nutshell, I mean these are the permissions we were. We were uh defining or we are requiring sorry exist. Every CSI driver right in every infrastructure. They have their own way own ways of setting up uh policies to um you know do this, but uh essentially uh so we have three principles involved here. There's the backup application service account right that one's going to need access to the to create the snapshot, session, requests or objects, and is also going to need access to get volume snapshots in that same name space.

C

um Then there's the snapshot session manager, which pretty much needs permissions on everything, uh though, on the configuration object. It's only going to get unless so I'm going to delete those objects and there's the CSA driver service account. The CSI driver service account is going to need permissions to uh get list, delete the snapshot session data objects it. uh It definitely already has permissions to read uh volume snapshots, because that's the it's the entity that has been creating these volume snapshots to start with um the.

C

We assume that in our prototype right, we actually use the role binding to make this happen right, because uh we- essentially, if you see if there are multiple CSI drivers and they're installed in different name spaces, then they are effectively isolated from one another with respect to the snapshot session data because they have no business peeping into each other's data too.

C

um So that's the gist of what we have over here. So this is just a link, so yeah at this point, yeah go ahead and.

E

It sounds like you have. This is a good screen if we could just sit on this one for a minute, so you have the backup application, uh which has a service account. So this is a kubernetes service account already. So it has. An identity am I understanding that correctly, yes, okay um and it's making kubernetes API calls so it it's getting service account credentials, somehow, probably running in a pod being fed uh service, account credentials um and then the driver, so the the token that you're envisioning this thing getting.

E

It then passes into grpc request to which, which of these actors, the okay.

C

So the token is going to be placed in the in this request as a result, so this backup application then makes a call to the to the uh to the psycha I do have to script, go back one. It makes a call to the sidecar here. Passing this token on the grpc. Okay,.

E

And when jumping back to the page, that has the actors which actor is this is the site. Top This is the sidecar okay. So then it's going to take the token and validate it somehow that this is a good token and that it represents authorization to do this request. So it seems, like the are those the bits you're wanting feedback for like how how to get this backup application a token and then how the driver can use that token or verify that token is authorized. Is that you're?

E

Looking for okay, um so a couple bits of feedback on the the ideas presented here so um having a secret be the name of an object? An NCD is actually really problematic.

E

um For a few reasons, one is that anyone who has the ability to list those objects can see all of those Secrets. Yes, um so that's that's not great and then even worse, the name of objects is used in their etcd storage paths, and so, even if you like encrypted your storage information, the content of the storage and CD, um if the secret is the name of the object, that means the secret is uh unencrypted in the uh in NTD.

E

So having it also shows up in other places, it shows up in the URL of the object when you request it, which can easily get logged like um having the name of the object could be. The secret is not great, so um just right off the bat. That's that's not a great thing. Fortunately, we do have a way to create tokens um that can be used to verify identity to arbitrary audiences. So what I would recommend is using audience. Scoped service account tokens um for this.

E

So if the backup application is going to be making backup requests to some CSI driver, uh it can ask for a token for its service account yeah scoped to the audience of that CSI driver or CSI service, so that can be either a well-known name for that CSI driver or it could be. The service name of the CSI driver, I'm, not sure how how the backup application identifies the grpc service that's going to make a call to, but it would it would say.

E

Please give me a token for my service account that is limited to the audience of this CSI driver.

E

So it can either do this by making a an API call to the kubernetes API or it could mount in a token into the Pod like when you're setting up the Pod definition uh for the backup application, you could say inject a service account token for this audience.

C

Yeah, can you give us some more details on this, so essentially you're saying? Basically, then it comes down to uh oh sorry, going backwards is slow, uh it's this. So this whole green box goes away. I mean that's what we're trying to do to start with.

E

um Well, I I, don't know what other information you're yeah.

C

Yeah there's other information. You have to check.

E

Yeah but it's important it's.

C

Tremendously, what you just said, where.

E

The Authentication and authorization Parts- you don't need that anymore. Maybe you need it for other books. We.

C

Do need one thing, though: we do need one thing, and that is um so that again I don't know the apis. You just say you, you know who can use this API for the audience. Scope service account token, but assuming that it's that it's only doable by the backup client right, how when we pass it through uh um I mean yes, we this so This manager can validate the snapshot contents and all that part has to be done because the service account doesn't actually. This SP doesn't actually do that.

C

uh So when, when the when it comes here, the fact that we have a management, so so what checks that the that the service account the man, the backup plan service account, can create this audience scope.

E

Token yeah, so there's there's two ways to get a token. One is to make the API request directly. Okay, uh and so uh that's just a normal API call uh I'll I'll link to the API. So that's a normal API call that's authorized in in the normal way. So if you could do this in Cube control like as a admin, you could say, keep control create token for this service account and if you are authorized to you can create a token.

E

So you could do it directly with an API call or you can put it into the um pod spec.

E

And at that point you are, you are running you're, defining a pod, that's going to run as the backup application, and so you implicitly have the ability to get credentials for the backup application service account like you're you're running an application as that service account, and so.

C

Anybody right, the Pod split, would have that so so, by virtue of having made the call to get the token, the service account just carries those tokens with it.

E

uh You can request those tokens, so it automatically gets a token that identifies it to the API server. But what you're wanting to do is identify that service account to some other service like the CSI service. So that doesn't happen automatically, but you can add things to the backup pod uh to say: I want to mount in a token to identify myself to the CSI.

B

Backup Service um so Jordan, one of the uh we we did um look into like um the token review and token request, some apis and um one of the constraints and I don't know if it is still like um a hot constraints on our end. Right now is um when it comes to CSI spec. um We try to be like um platform agnostic as much as possible.

B

It was brought up to us that, like great, if we can um couple ourselves to the kubernetes and token review and collection of apis, but you know as much as possible. Csi try to stay like um you know, platform, agnostic, yeah,.

E

The CSI portion of it is agnostic, it's just an opaque token, so we're trying to figure out how to make use on either end like the backup application running in kubernetes context and a CSI snapshot service running in the kubernetes context. Both of those are kubernetes specific, so you're using the CSI Plumbing.

E

So the plumbing, through the CSI grpc request, is just agnostic, it's just here's a token session token sure, but the backup application is running in kubernetes, and so it's going to get its token from kubernetes and the backup service is running in kubernetes, and so it's going to verify the token against kubernetes mechanisms right. So the The Proposal you have is verifying the token by making a kubernetes request to see if there's an object named a certain way.

E

So it's all it's assuming kubernetes, um it's just as reasonable for it to take a token and verify it using a token review to get the identity out and then do an authorization check on that identity. Okay,.

B

So the token verification part happens inside the CSI driver right. It will still have to reach out to kubernetes and say: hey I got this token.

E

About what happens in the sidecar and what happens in the driver like if, if the point of the sidecar is to do the kubernetes specific things um and then hand it off to this is CSI driver and the CSI driver can assume that it was all authorized and verified already um that's a possibility, like the sidecar, can do that check. The sidecar can say you know I'm running on behalf of CSI driver Foo, so I'm going to take the session token.

E

Do a kubernetes API call to do a token review to get out the identity then turn around and do an authorization check on that identity and once everything is happy, then I'll pass it through to the CSI driver and the CSI driver just says: I assume. This is okay. Now I I'm I'm fuzzy on the line on what the role of the sidecar is in terms of enforcing kubernetes, specific stuff.

C

Yeah, oh so the sidecar really insulates the storage providers service from kubernetes clients, so it has to it's. Basically, it's a proxy, so you know it invalidates that the clients are correct and then it just passes. Does the business logic stuff to the service provider using whatever you know, as a service provider, the storage provider's got been set up to do support the entire cluster right. It's authorization support the cluster, but it doesn't know about the individual clients within the cluster. That's the sidecar's job to do all that. Yeah.

A

In this case, though, to George's point that that is your kubernetes interface, yes, abstract or hiding kubernetes from your storage provider.

A

So for that piece to be making internally a bunch of very kubernetes specific calls over a generic uh CSI interface as like the data transfer mechanism, seems totally valid right because even in this design, that thing is going to make a kubernetes calls just to figure out, like it's odd seat decision based on these custom resources, so I think you're sort of well within that the the the the thing I want to highlight, though in The Proposal, that Jordan just described, which was exactly what I was going to suggest, is you're, not storing tokens anymore right, they're generated ephemerally, ideally by the cubelet and mounted into uh a memory-backed volumes on the cubelet, but they're you're.

A

Not writing, adding Secrets out to the kubernetes API.

C

That we really want to know how to do that, but but I want to point out, there are two pieces of information we require in the sidecar. You know, and you know it's always- we always face this dilemma of to get one we give up the other, so uh so right now, so yes assuming we can use uh an audience scope.

C

Token right. Okay service account token, uh but we also have to convey to the sidecar the context under which we're operating, because we have done an authorization check right um at the in the in the controller. Now that authorization check could be done in the side cut too. If that token can be used in the same way, see right now we we are using a validating web hook to do that.

C

Authorization check for the snapshots and kubernetes API is validating the actual construction of the CR, which is I mean incidental, because that's you know that's only about because we had to make a session, but the check to do to authorize access to snapshots is a um SAR check right, Prasad chamin over here. You've done you right. You know this, um and can this account service account token be used for the same check exactly.

E

Yeah it it that's exactly what it can do. So there's two steps, one, you take the token and you do a token review to make sure that the token is valid for your audience and what you get back is the user information which you would then drop into a subject. Access review to do an authorization check. So it's two steps. One is verify the token and get the identity and then the second step is to take that identity and check the authorization.

E

And it's exactly the same as the subject: access review, checks, you're doing in the web hook.

D

C

That's actually, that's actually really cool.

A

um Also, we already have code that does all of this. We even have roles and robot I mean cluster roles and cholesterol. Binding is already defined for this. We have. We have written this code a thousand times, so it's already in libraries.

C

Yeah yeah I mean cluster Road cluster role. Bindings I mean we have to leave that to the CSS driver, but you know we can you know because they already have. It deploys CSI things with all the stuff, so we just have to advise people how to set what you know like I had in that last slide. This is what we need effectively set it up and whatever policies you already have right. So.

D

A

D

A

If you can accurately model what you want uh in the way that sort of kubernetes expects you ask these questions, I think you could get the oxy checks that you expect uh I. Think the certainly the can this identity, uh or does this identity have list access on these resources within this namespace. Yet we can obviously help you answer that question very easily. Yeah um yeah.

C

Can you comment on the maintainability of whatever you're not proposing, because you know we are not in the business of security and we just want to use apis, not not redo and repeat the same double logic used elsewhere and has to be maintained over time.

A

uh So uh I mean Jordan David. You can correct me, but I think this is pretty relatively well factored out in our in our code base. So I think they could just import it and reuse it and then just bump the library whenever they want to yeah.

E

The token review and subject access review apis are V1 and stable and those are the foundation of everything uh kubernetes does with uh delegated authentication and authorization, so they're not going anywhere. uh The API calls you make against those would continue to work, even if we add things in the future and there are helpers and foreign straightforward clients in a client go that will do these things.

F

A

Was actually more thinking about the API server like the aggregate API server code that does token review and subject access review using delegated authen and delegated? Obviously I feel like they could just reuse that in their grpc handling.

E

um I'm not sure they need all of that, like they they're getting a single token in a single field in a grpc request, and they just need to.

A

E

Don't really need the filtering wrapping mapping of HTTP request attributes to subject access review attributes like I. Don't really think they need that.

A

Or no no I didn't mean that layer I meant like the delegated authenticator gives you an authenticator, which you would then use against the token, and then the delegate authorizer gives you an authorizer I.

E

Would probably just use a client go directly that'd.

F

Be happening suggesting either to them, uh but if you actually look at the API, it's really well factored into I know about these things. I know what a user is. I know what a group is uh I know what verb resource I want to ask the API server about. Let me make that request and whether you use the the API server delegator or whether you use just client go and roll your own I think you'll be able to understand it.

F

The delegated authent also does things like uh retries, so that authorizer in the API server has some benefit Beyond, just uh just wiring, and it does retries. It has caching wires up arguments and standard ways. I could see that being useful.

F

D

F

As you do a remote request, everyone says well: okay, now I want a cachable now I want to cash over a certain lifespan and.

F

I would be consistent with those arguments, even if you did roll your own.

C

But Prasad Ivan, you guys are you all familiar with these apis I am not familiar with the API, so I cannot ask intelligent questions at this point.

D

Yeah I had a few, uh please correct my understanding, so once we generate the audience token, what are the ways to inject it to um or pass it to the backup client we talked about injecting it but cannot be changed by using API and APA uh yeah.

E

Yeah there's a field in the grpc API called session token I assume you would use that to plumb the token to the driver.

D

But so the backup client has to pass the um okay you're saying directly at the site. Car we Face the token and validate that or the backup client should have the token, and it should pass it to.

E

The side correct, the backup client should have the token and it.

C

Should add grpc requests we still.

E

C

Field, I guess the the key is 10 weeks so given can we do? Is there a clear, serialization mechanism for this token, so we can pass it through and deserialize it the other side. It's just.

E

A a jaw, it's an opaque token. You should be able to just set it directly and there's it can be treated as opaque. Actually, all the way through to the CSI driver, the CSI driver is going to take that token and put it into a token review request and pass it to the kubernetes API server and say: please tell me if this is valid and who this represents. Okay, so nobody needs to introspect the token anywhere in your chain.

D

A

And I don't know if it's relevant but I know secret CSI store, which is a CSI driver, uses um bound service, account tokens in a special way for uh sort of passing through application identities. I, don't know. If you have that same use case. The diagram is complex enough. That I can't actually follow. What actors are doing what, um but that might be useful to you guys too I don't know if it's relevant.

B

I think at this point, like um the token review and subjects is review, apis would be most relevant. We did um take a brief look at the secret CSI driver um yeah, maybe not for Alpha version. I guess um call to your earlier question.

B

um No I think this is uh I, think we that this token review stuff it come up during our previous discussion I think there was also a uh prototype being done on it, so we can always pull that back up to see where we left off the yeah. The reason why we started to roll our own like session code, looking thing I think is because um someone brought up that it has to be. um You know kubernetes on or platform agnostic.

B

If we can, you know again just be firm with the contract that says the site. Car is the place where all the kubernetes call happens.

B

um Yeah I think we can simplify that quite a lot. Yeah.

E

Yeah, the TSI bits are agnostic, but the implementations on either end are kubernetes specific, no matter way, no matter which way you go, whether you're using service account tokens or using like kubernetes object names, it's still kubernetes specific, but that's fine. The CSI bits are yes,.

B

A

And I think you already said that the the sidecar like uses UDS, to talk to the actual backup thing right so like.

D

There's no concern.

A

About authenticating the sidecar to the backup thing, it is effectively part of the backup, then application over a well-defined interface that abstracts kubernetes from it. Yes, it is correct, so I mean that that seems totally fine, uh I think you're, basically just gonna end up building a very kubernetes specific shim, um yes, I think that's yeah I think that's totally valid and expected.

C

I've been precise, if I understood this correctly, then essentially we will fold the the side: cow will fall in all the management manager. uh Part.

D

C

Organization right.

D

D

Accessory or authentication and validation just to confirm uh so the backup client would use the token from service account right.

A

The I don't know which actors which so like I would expect the the Shem between the the the backup thing and the rest of the world to use its own kubernetes service account to do token, review and subject access review on the incoming service account token, which belongs to whoever.

C

Will run with this lowest one CSA driver service account and the backup application will run with that service about.

A

uh Yeah I think so.

D

Yeah, as I will say, I think we had uh sometimes like we had discussed about this uh yeah we'll have to I guess follow up to that.

C

Yeah we I think at that point. We did not understand how we could extract the service account I think we uh Prasad I think we did not know about this. Api right token request getting this. Where is this where's the API? Actually so.

B

This one is different: this is uh the CSI driver, one yeah, so I think you had your second tab or something was the one that.

C

B

And review I think this is the one that.

C

These are the data structures. Where is the which is the API itself? Actually sorry,.

E

um I linked to the client go um I'm, assuming you're, going to be using clango or something built on top of client go.

F

um Right now, yeah.

E

I'd, like to uh the the point in the client go client set where you would make this create call.

A

So I mean but based on what they described, didn't sound like they needed Dynamic audiences per per app. So wouldn't it just be easier to ask the cubelet to do all the work.

E

uh This is token review.

A

I meant for the provisioning of the actual service account token. Yes,.

E

To get the token I also linked to the docs that show what it looks like inside of pod spec um I. Put that in the notes.

E

Right yeah there, if you want to pull it up, um that's an example of uh asking for a token for the vault audience and then putting it at a specific point in the uh in the Pod. So you could ask for a token, for you know the vsphere CSI driver or the whatever the audience that you come up with for the particular CSI driver.

E

And, and if you do it this way, uh you don't have to worry about setting up any other permissions for making the token API requests to get the token and the cubelet handles rotation of it for you. So it's a it's a Time limited token, but uh the cubelet will say at 80 of its lifetime. It'll fetch a new token and update it inside the Pod, so inside the side inside the backup service pod.

E

When you're about to make a backup request, you can grab the value from the token and make your request and be confident that it's always up to date.

C

Yeah, typically, the backup application Falls are kind of ephemeral they launched it's.

E

Like a job like it starts.

C

It runs, it finishes, yeah and finishes yeah. So if this, if this mechanism generates a new token each time, we wouldn't even know I guess so, yeah.

E

So if it's long-lived, then this is really nice, because the cube will rotate it for you. If it's only going to read it at startup, then that doesn't matter as much, but it also is nice because the cubelet will get it for you. You don't have to write the clients to request the token and make sure it has permissions and stuff it just gets handed the token so.

C

Prasad makes sense.

D

E

So that the key here is the the use of the audience. So if you don't include an audience there, then what you're actually getting is a token that is usable against the kubernetes API server, and so, if you hand that token to the CSI driver which you've effectively done is given the CSI driver a token that it can use to do anything, you can do against the API. So it's really important to include the audience here, so you can say here's a token that identifies me, but only for this particular audience.

E

You can't use it to like make kubernetes API calls.

C

Yeah so suggest you reiterate so essentially once we get that token in the in the. Where is this dude in the backup application yeah so talking from the backup application we send it to the site car, then the side cow can do. Subject access review on that token to make sure that the caller has access to the right things correct. First,.

E

The token review get the identity, then take that identity and make subject access reviews to check whatever authorization you want to check. Yep.

D

C

Okay, that's that's what we that's what we wanted and we couldn't figure out how to do it.

C

Clueless on the apis, I'm gonna have to read them up. Prasad, you you're more familiar with them, but we may have to come back and ask you guys questions yeah,.

E

No problem uh to do it: async slack, is great. um People are happy to answer questions in there. You don't have to wait for a meeting like this, but yeah I think this will work well for you.

B

Yeah, how is the um just curious, how does the um audience or the scope being is it being is being enforced right? Is it like how do when.

A

You just got a token review. It would then use that you tell the API server that I want this token to match The Vault audience, and so the apis are cool I'm, going to validate all the crypto and everything, but at the end of it, I'm going to validate also that uh it's not necessarily valid. For me, it's valid for some opaque string called Vault, which is the audience.

A

um So if you don't specify that string, what you're asking the API server is like hey? Is this valid against you, which is not the question you want to ask it? You want to ask: is it valid against me the back the backup service thing right.

B

So in this case, both the client and the CSI driver side. They have to agree on what that audience or this scope is yeah.

E

So a couple there are a couple ways to normally do it. One is like a well-known string like when I'm talking to Vault, I, say Vault um or when I'm talking to a vsphere CSI driver I'm using the name BC or CSI or whatever that's one way to do it. Another way is to use the um address or the name of the service you're actually calling to uh as the audience. So.

E

Presumably the backup client knows something about the coordinates for the backup service that it's going to make a call to, and so you could use that name as the audience. So if it's going to be talking to food.bar.baz to make the backup request, then it could ask for a token for food.bar.baz and.

B

So it was like the DNS name of the service DNS.

E

Name yeah exactly I've seen both of those they can both work.

B

D

C

Yes, well, it could be the provision name or the or the service name. I mean yeah.

D

E

Service name has the nice property that, uh like as soon as you hard code, something it can accidentally get left. uh If you.

B

Use the service name.

E

Then it has a nice property that if you talk to a different Service uh you're going to be asking for a different token, automatically yeah.

C

Yeah, which, which is probably good, yeah yeah,.

C

So this is what is this about? This is.

B

This is a different token request. This is within the CSI driver. This is between the um yes.

A

I, don't know if you need this here. This is about uh how um um for a CSI driver to get the identity information about a pod, that's going through the CSI interface, which is relevant in the case of something like secret CSI store, because it uses that token to then talk to like Azure, key Vault or Cloud KMS on Google.

D

A

D

A

In the use case, where you need uh the CSI driver to talk to some external identity- uh sorry, some external system, but it can't use its own identity because using its own identity, would require it to have all the privileges in the world. Instead, it needs to use an app local identity. So that way, applications can have access to like individual secrets and the CSI driver itself is permissionless.

D

C

This kind of like output, facing to a CSA drive into the infrastructure, correct.

A

Yeah, so the the token request API for service accounts was meant to be like a sort of like a platform agnostic way of about it in kubernetes identity from to the outside world.

A

Obviously it's used internally too, but that's that's what allows to work, because the payload and schema are well defined and providers support validating those tokens.

C

So can you go one more time over what permission so the CSI drivers service account right, which is basically the one which is going to handle the delegated SAR right. So does this need any additional permissions to use that token to look up stuff, or is it just implicit.

E

It it's not going to be using the are you are you talking about for the backup stuff? We've been talking about the whole time or this news.

C

For this specific problem we are facing here so.

D

C

Going to get the token the the audience token here right the audience scope token from the backup app coming here. Yep uh this guy is going to use the essay. After does a token review. You said: uh was that a token review API to validate that token yeah and then it's going to do an SAR. With that token correct so.

D

C

Bad idea, right.

E

Those two pieces of information for the for validating the token. If it needs to access other kubernetes objects in order to do its job, then it would need permissions to read those things I'm we hit. We didn't really talk about what other information was in these that it would need to slurp up to pass to the CSR driver.

A

um I think to you Carl your question, though you do need to grant that um that sidecar or service account the ability to make token review and subject access review, because.

B

C

C

So so this one needs a permission to make. Subject: access review checks and.

A

Token review.

C

uh Prasad got that yeah yeah, okay,.

D

C

Guy who understands this not me so sorry about the dumb questions.

B

Thanks Carl yeah I want to be I'm. Sorry, we are three minutes past just want to be aware of everyone's time. um I think uh this is where available, valuable and helpful. Thank you. Everyone for taking the time to uh talk us through this. um If you have any questions we'll reach out on slack. Thank you very much.

C

D