Kubernetes SIG Auth, 8 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sig-Auth Bi-Weekly Meeting for 20211208

Description

Sig-Auth Bi-Weekly Meeting for 20211208

A

Hello: everyone welcome to the december 8th 2021 meeting of sig off um looks like we got a couple items on the agenda. uh Let's start off with this announcement mo is this you.

B

Yeah, you want to open up the listing tiny text um so yeah myself, rita anderson natalie. We sent this out, I guess almost 10 days ago now and we got some good response and uh if you want to open up that neats kept link, I was going to do a quick highlight reel of what's on there um just to give this some more awareness. You know I don't expect folks to be working on this year, but hopefully folks that are interested uh will be able.

B

uh So anish and rita have been sort of leading the kms work. um There's a few folks that have said that they're interested in collaborating.

B

I think we were going to try to send out some kind of doodle to see if we could have some high bandwidth meetings on what we're thinking needs to be changed and updated in that api, so if you're interested, probably ping, guido or niche. So that way we have a sense for who wants to be involved.

B

I suspect this to be a pretty large effort, but I think there's enough folks interested to move it forward.

B

The next two caps on that dock- that we right now are similar in the sense that they both talk about uh configuration options on the api server that are owned by sig off that are flags today, and we want to upgrade all both of those uh implementations to structure config and various folks have indicated interest for both of those items. um So that's good too.

B

um I think this one is less, um maybe flushed out than the camera stuff. I don't think we've discussed this as much so I would expect these to start out more as higher level designs and because I don't think we've talked about either one of these in a cigarette painting. Before directly.

B

We just had like issues and asynchronous conversations about them, so both of those are up and then the bottom two ones are sort of placeholders for things that we have um uh that are within cigars curvy. But we haven't flushed out enough details for someone to maybe write a cap. All of these are targeted at you know. Someone has to basically sign up to write the cap and also maybe ideally, the same person, but maybe someone else also signs up to write the implementation.

B

We are asking that um the release that they kept is marked implementable, that same release should um that same release of the next release should involve the work for that kept to be started, so things do not get stale and we want to keep this moving and also just at the high level for anyone interested.

B

This is a pretty high bar. This is a long-term investment. It takes a lot of work to get these things through.

B

So just know that if you sign up for any of this work, I would firmly put this as not the getting like good first issue type of category. This is this is like the exact opposite of a good first issue.

B

I think that's all I wanted to say I don't know if rita, nish or natalie, you guys haven't, thought certain comments on this right now. I I plan on continuing this discussion. You know next year, since this was our last meeting, but I just wanted to try to raise some awareness here.

C

um Just for a concrete timeline, um I would guess that the enhancements freeze for 124 would probably be late january. um I think the last time I saw the release team was treating beginning of january as week. One of the release and the enhancements freeze is typically like week, three or four of the release.

C

So um if there's going to be a cap for these targeting 124, uh I would expect to have it reviewable like at the beginning of january, so that there's time to iterate on it and close on questions to get it implementable in time.

B

Yeah, it's pretty pretty tight with holidays, so I would be very surprised if any of any of this could be that done that quickly, unless someone was just doing only that as their full-time job for the first week of january.

B

um So, but I my hope, is that, even if you know we don't get any of this stuff in as implementable by enhancement space that we could get it in as a kept that can be worked on in one like merged and worked on on 125 time frame, because I think we've talked about this stuff ad hoc for a long time. But we haven't made progress. So it'd be nice to push this forward.

D

Sorry, just to confirm you said to make sure that it's reviewable by mid-january.

C

um Early, but uh it depends on how good of a chance we want of the reviews and iteration wrapping up in time to get it influential like.

D

C

Yeah like um if a kms like rework comes in like mid january, I really doubt like it would get attention and feedback and be able to iterate and actually close and be implementable by like a week later. So that's.

D

It okay, so I think the tldr here is. If you wanted to be uh me, the 1.124 deadline, then it really should be reviewable by early january yeah. I think that's fair.

C

And like it's better to aim for early and if it doesn't make it that's okay, at least we made progress like what tends to happen. Is we think? Oh we're not going to make the one dot whatever enhancements? Freeze, that's cool, we'll just do it for the next one and then our whole release passes and then we're in the same boat. So it's better to like get it up there and get it reviewed and um if it happens to miss 124, that's okay.

C

We've made progress and we're like in a much better place to get it for 125 percent.

B

That did remind me of something jordan, uh so you know I of the the top three on here I'm interested in willing to review, for I think we need we once we get to the stage of having cats, I think we would need some folks, or at least volunteer to review and be involved other than just myself.

B

Honestly, um I think I, I think, david's on the call I know david. You had expressed interest in the authorization one. I don't know if you'd be interested.

E

Yeah, I am um adding a second authorization book early.

C

Yeah, if we want to go ahead and add names to people who are interested reviewers, that would maybe be helpful. And if you're constrained to like reviewing one or two, maybe note that so that we can try to load balance and just make sure we have bandwidth.

E

That's a good point, and yes, I am.

F

um Natalie here should we just should we also be noting in in this doc? This is more a question for the niche reader and mo. Maybe we should note like just expectations for folks working on this. If we want this in one two, four, then, by end of the first week of january or early january, we need the kept kind of in a draft state for review.

D

I'll add that to the stock at the top.

F

B

Cool, thank you uh anything else I want to discuss about this. uh We have, I think, a few other items on the agenda.

A

Awesome thanks for uh putting this together mo and rita, and whoever else worked on this.

A

All right uh moving on client go tls config for ciphers and min version um mo. Do you want to.

B

Yeah you want to bring yourself here, uh so I brought this up. I don't know how. Many weeks ago now in the uk machinery- and I spoke with um david and someone else, uh great bad machinery in regards to basically incline go, I wanted the ability to configure the tls stuff more specifically. So today it hard codes to tls min version, 1.2 and no control over cypher suite.

B

I wanted more flexibility there, so I I don't know why I started the academy. She knew first, but I did so that's there and the pldr there I got was relatively positive response with a desire to be able to configure it like override the default globally and then explicitly override it at a rest, config level, so that, ah yes, they you are friendly.

B

But I I think you know such a configuration is effectively co-owned by this thing in api machinery.

B

So I wanted to ask here what folks thought like, for example, I don't think jordan, you were there when I had this conversation with others, so I'd be interested in your opinion.

A

What are you talking to? Is it always cube api server.

B

Not necessarily uh so you know, you know that same code is used for web hooks and other things. um So I you know I I would like to be able to enforce. You know things and also there's a lot of code within our like api server logic and aggregate logic where all the client stuff is like 18 layers in um so like having like the ability to globally override it to do something else would be helpful because most of it's not exposed in any reasonable way,.

B

But at the bare minimum I would like to be able to at least on a rest config, that I create be able to set a configuration.

B

So the at a very high level that I thought of for configuration would be a min version which would probably only allow 1.2 or 1.3 to be set, and then, when it is 1.2 it will let you control the ciphers and obviously not letting you control them. When you have 1.3 because code doesn't let you do that. So there isn't any point in that.

B

Does anyone have thoughts or concerns? We have roughly similar configuration for the api server itself as a server, basically asking for a similar configuration as a client interfacing using machinery.

A

Do you want to set them, or do you want to turn specific cipher suites off?

A

B

There's like a subtle.

A

Difference between those two.

B

uh So far, the way I've built most of this stuff is, I just set the ones I want and explicitly disallow any ones that I haven't explicitly picked myself. That way, there's no surprises um granted, I don't think the tls 1.2 cipher suite is actually growing. I think that's that's done so. Maybe.

C

The suite aspect seems sort of like a dead end, because it's tls12 only and then one three and up it doesn't apply like I, so I yeah, I don't see that growing. I don't know from it from an auth perspective. I don't object to setting these like it seems like machinery owns most of the plumbing there.

C

I would probably not want to expose these all the way out to like cube config files. That seems super fiddly for that level to be managing, but if you're doing it programmatically like you, can already override the client like. Why not?

C

Why? Let you plumb this parameter on them.

B

Yeah, you know the client configuration like it's for in my case, because I was using the code in a bunch of different ways. I I was basically trying to not reinvent half of flying goat and all the nice wrappers and all those things because they're useful and nice, um uh the I I don't, have an opinion on how far up it goes for, like you config my my gut says that uh I I talked with 6 eli a little while back about having a qrc file.

B

You know some place to have configuration that is outside of a cube. Config, that's for a user. I could see maybe some value there for something like this for a user to explicitly say I refuse to talk to servers and crap ciphers right, but even but that's just like a vague thought. I wouldn't I wouldn't tell anyone to do that, but my my for for I you know david. That asked me to write a kepler flesh out the details. My my thought process on that was um I I would stop at the rest.

B

In the big level, I wouldn't go further out.

E

Unless anyone wants me to do really badly, I think it would be possible for cubecontrol to control this setting without adding it to a cubeconfig. If 6dli felt strongly about it like flags, um it could be done that way.

E

E

Oh, like uh you could could even be like n bars just like yeah. I know that would uh drive some people a little bit batty, but it would be a cube control, presumably only thing, um but I you know it wouldn't have to be a cube config, but I would probably try to argue about how far up the stack it goes after the core library made it happen uh and a comment about fips compliance when I've done fips stuff.

E

It has been the case where people want to just completely remove ciphers from the os uh and from the libraries like there's a special open, ssl or whatever is important. There's a special ssl package that you end up using. uh So you can't have this be an accident in fixed components.

A

C

On the client side to do fips, uh you can actually build with the fips go lane tool chain, and if you want, you can import a fips only package which, like puts the tls stack in a mode where it will refuse to talk to anything. That's not that's compatible, and it does really weird and unpleasant things like reject talking to servers with keys that are too big because they were bigger than what was permitted. The last time that stack was validated. It's really unpleasant.

B

Yeah yeah, so I'm that that certainly is part of my motivator. I also just um in general, wanted to.

B

This is why I had more control on just to assert the set of behaviors that I'm expecting of sort of the stack. uh Certainly fips is a motivating factor. I've also uh because people are bad. I have. I have fips requirements that are like. I want fips, but like not all of the fips, so I can't like import the fifth only thing I have to like control how much chips only it is, and it's just yeah yeah. My life is very unpleasant, um so I yeah um ted.

B

I can't hear you if you're saying so I think you're angrily commenting.

A

Yeah, I don't think this seems too objectionable. I would say that if we have two allow lists, one from the go standard, library and one from command line, flags that exist in some configuration file or in cube, config or scattered all over the place, then I worry that those windows will move over time and then there won't be any intersection and stuff will just break versus like a mode where we, you selectively, turn off cipher suites. You don't have to worry about that like if we configured deny and you used it.

A

If there was a problem with some tls mode that you wanted to turn off before you know, go gets an update.

A

Maybe that's not a concern, since we just got tls 1.3 and everything is going to be static for a little while fingers crossed, but um yeah.

B

Of course, if you're using fips that make you turn off to less than that three, so there's that okay, so I I think that just I'm getting is like I could. I could actually go right account and get it would be interested parties and, I think, we'd be okay.

B

um uh Is anyone interested in being a reviewer on such a cat? Any chance.

E

B

Okay, mike, do you mind writing that down still understand, so I remember that they volunteered.

B

Now does anyone have anything else on this is adam adam? Are you on the call for your topic.

B

See adam, maybe he forgot that.

A

Okay, um does anybody else have context on this.

E

I do uh this is about the um trying to find a way to enforce some limits on ephemeral volumes. So when we created pod security, we said we were going to make it allow ephemeral volumes for csi drivers for all levels. That's restricted, baseline and privileged and doing it in that way allows csi driver authors or anyone to create an admission plugin that provides further restriction.

E

If we had chosen to go in the other direction and restrict inside of pod security admission, there would be no way to have a special set of rules for each one.

E

So one of the things that has come up is that some kinds of ephemeral volumes are safe for use by restricted users, and some types are not.

E

While it is possible for every csi driver that is not safe for use in ephemeral volumes to add their own admission web hook and manage that web hook to prevent usage by restricted users.

E

There is also an api resource that gets created for csi drivers. That is a convenient spot to hang a description that says either I'm safe for restricted users uh or I'm safe for level x, and if we were to do that, sig storage. As as I understand the last time I spoke with them, they were amenable to the idea of actually owning that admission plugin.

E

So this is not an extension or should not be an extension to pod security admission, but it would be a new admission plugin that can say I can recognize this csi driver as unsafe for restricted users and therefore I will not allow its use as an ephemeral volume in illustrated namespace.

E

I think the desire from the storage folks is to have blessing to interpret the namespace label provided by sigoth right. That label is pretty clearly, let's say hours, but they would be consuming something similar or they would be consuming the same thing and and having it apply in their admission building.

B

um Quick question david is so I I understood sort of why we went with the label based approach for on on name spaces for the new pot security stuff. You know how to just lot of nice ux things, especially around dry, run and other concepts. That kind of allow that to be really useful.

E

You're saying for a csi driver, why have a label instead of a first class field yeah? That was just just a question.

B

I I don't actually have an opinion. I was just curious. What was the reasoning in this particular.

E

uh The reasoning for this particular one is that the prototype was built with a label, because labels don't require schema change. I don't believe there would be a principle objection if the approach was generally blessed, adding an actual spec field.

C

Stepping back just a second, I I had thought when we were talking about whether to allow ephemeral, csi drivers or not when we talked with six stories. At that point, I thought the uh direction was to recommend that unsafe csi drivers not be modeled as like ephemeral, inline pod drivers.

C

I it isn't control over those parameters like control over unsafe parameters, fed directly to a csi driver like there's a way to protect those with a like a storage class, so that you capture the parameters and then things inside namespace is consumed in a safe way. I thought the guidance was to not create unsafe, ephemeral, inline csi drivers.

E

To my understanding, some already exist.

G

Can we get any like help me anchor my thoughts about this? What is an example of an unsafe ephemeral, csi driver like water, so.

C

That's a bad thing that can happen. It depends on what the csi driver does right. So imagine like imagine, a volume plug-in that is going to like issue kernel commands.

C

C

It takes as input from the csi parameters like things that translate directly into those kernel commands. So it's not hard to imagine like uh providing inputs that would crash the machine or like do really terrible things for performance or like who knows, and so when the values for those parameters are coming from a storage class.

C

uh It's a more reasonable assumption that, like the cluster administrator set up that storage class and said, like I'm, driving these unsafe parameters, but I'm selecting the values to set there. But if it's being driven from inline pod specs, then any random pod author can say: hey, please give me a volume and I can set any parameter. I want and it feeds directly into that drive.

C

That that consumes parameters that are unsafe or not checked or not bounded.

F

C

And that's why that's that's where I thought we ended up when we talked with six storage about the bot security standard stuff, which is why, like I, I'm kind of questioning like if you were going to write unsafe, csi inline ephemeral drivers.

C

Maybe this would be a reasonable way to protect them, but I thought that they had swept and identified. They were only like one or two and both of those hadn't really considered their use, and so I'm just wondering if that changed.

E

Well, to my knowledge, it didn't and they still exist and are still used.

E

But it's possible that I'm out of date there I I didn't see.

E

The I don't recall seeing changes in that regard.

C

All right, let me see if I can dig up the the zig storage meeting. I think they were referenced and maybe michelle went and swept the list. Let me see if I can dig that up and um that would maybe be good to settle like. Would it be less work and better a better long-term direction to encourage the couple ones that were identified as being unsafe, that this isn't a great use case and and see if it would be reasonable for them to re-posture those.

E

And come up with some plan to prevent their use in upgraded clusters.

E

I'm open to that, if it's practical to do.

C

Okay, I'll take that ai to dig up those references and see if we can pull in michelle or.

C

I think it's michelle he did this week, but uh pull in the storage leads and double check that this is necessary.

E

And so conditions on necessary would be if they, if, if those had not changed, were known to be risky.

E

We will need something to deal with them now we could say: go off and build admission. Web looks for these two. We could say: we've identified these two we're just going to block them. We could say: we've identified two, we don't know what other ones have been written here is a way to explicitly indicate. I am not safe.

G

Are these ones that we've identified third-party csi drivers? Are they something shipped by the kubernetes product, they're.

C

Third-Party csi drivers- I, if it's a handful of third-party ones and the recommendation- is that unsafe, csi drivers not be made. I don't really want to build in a mechanism for making it easy to create new, unsafe csi drivers and then like integrate with policy stuff.

E

When you consider whether those csi drivers are likely to provide a safe way in general, it seems like the answer to that would be a resounding no, and so you would be condemning the people using them to have to build their own validating admission weblog, and you could say that, but it doesn't seem awesome.

G

I mean, I guess to me, it sounds like this is a bug report or cve on those third-party drivers. You know you have built in one that can be used inline and accepts untrusted commands from an untrusted user.

C

And this isn't new like to be clear. This is this is something that it already existed. If they're unhappy with users having access to these unsafe options, they already had access to the unsafe options.

C

I I just don't think it's our gap to close honestly.

E

If someone were to close the gap, you would probably expect them to build it in nearly this exact way, correct.

C

um I would probably expect them to close it by not publishing like not consuming unsafe parameters from pods, not consuming it, but then trying defensively.

E

I'm thinking about a cluster admin who has this in his cluster already and is trying to figure out how to do how to move towards a best practice, and he can look at it and say I need to keep this problem from getting worse, but I can't change it today. How can I do this.

E

And we can say you have to build this from scratch and they can.

G

Yeah I if we found a bug in some other third-party component, that's deployed. We wouldn't build a pod security policy extension to blacklist that right, we would say, go upgrade.

E

Your third-party component and then I would not leave openshift clusters vulnerable in the time frame. It takes to do that. I would give them a mechanism to stop the bleeding now.

C

E

The naive way to stop.

C

That is to say all right: this problematic csi driver has a quota, a default quota of zero on it, so no namespace can use it by default and I will look at the ones that are currently using it and then like grant them exceptions and open open it back up to those namespaces and the new namespaces won't start using it and I'll burn down the problem I already have I mean a quote, is not a great like I wouldn't go to that as a design for this, but as a stop gap like lock it down and then make exceptions for existing users, it works.

E

So it sounds like it doesn't matter to you how many of these there are.

E

I just want to make sure we're grounded in that fact, like it doesn't matter to you how many there are we're just looking for completeness to say.

C

To me, it seems like an anti-pattern to intentionally expose consumption of unsafe parameters into pods.

F

C

Are a lot of things in pods that we wouldn't do the same way if we were going to do them today and ephemeral, csi drivers are brand new, like they just shipped last release or like a couple releases ago, so they're just in the like picking up use phase. This seems like the right point at which to say: don't expose uh any functionality to pod authors that you consider unsafe, like this seems the right point at which to do that.

C

E

Trying to decide how and why it would be a different than be perfect, but I will probably just continue.

E

I suppose the list is useful to know in general.

E

And then building it out so that the option is available, it would probably be enough for me. I mean it is possible to do it.

G

E

I mean as a stop.

G

Gap couldn't couldn't we ship, like um just you, know, throw up on a repo somewhere an admission, a a emission web hook that detects unsafe usage of of the known, unsafe, inline csi.

E

Volumes yeah, I mean that's, that's exactly what this does. So it's just a a question of.

G

This well, the proposal that was in the dock was like building some field or additional capability into pod security policy. Right.

E

It was supposed to be channeled through a new admission plug-in built on top right, like a it's, it's very easy to create a pr that does that. um I would not support adding this to pod security. I would support building something in to actually make this configurable entry, but lacking that you know someone can manage it through validating admission web hooks.

E

um I just would look at that and say the cases where you're likely to need this are not the places where the person has already built that validating admission level.

G

I guess what I mean is as sig author as signode. We would say something like we're aware of these two cases or three cases where people have written csi drivers that can be used as inline volumes and they accept unsafe parameters in an untrusted fashion from the pod we have filed bugs or cves or whatever against these. In the meantime, here's a stopgap admission web hook that will block usage of these.

E

Right, it's it's! You can actually make that block usage, uh something that is first class present always on and you add a label and you have fixed your cluster in the field uh and- and you know I look at that and say- am I likely to build that yeah, I'm 100 likely to build it. The question is whether I would build it upstream or downstream, and this is effectively the offer to put it in upstream.

C

Yeah, I I think I I don't want to encourage people to write, unsaved csi drivers and then distribute a manifest that has a label and think, oh because we put this label on it's totally cool that we wrote an unsafe csi driver like that. That seems like that's what this would motivate going forward.

E

Yeah, I guess that's just a matter of perspective.

A

Yeah until we have examples, it feels like this is a pretty hypothetical discussion.

A

But I think examples would be very helpful.

A

A

uh We have a couple more items on the agenda. um Should we reconvene uh maybe discuss on the dock a little bit.

E

Sure uh I I will have a look at the list that jordan put together um and.

A

Reconvene in january sure.

A

A

uh Mo next two items look like they are yours.

B

Cool, let's see you got two minutes yet so I want both of these to be kind of quick, but I know we've talked about this on and off for a while, and I wanted to see if we could sort of try to write down the mvp requirements, around uh client certificates for kubernetes service accounts and like serving certificates for kubernetes services.

B

That's like the bare minimum. We could do here with no spiffy, no fanciness, just the bare minimum x, 509 semantics that we all understand or api server, already understands.

G

um I've been meaning to write a cap for the client certificate piece of that I've just not had the time, but.

F

G

Someone is interested. I can actually do that. Basically, what I was going to propose was keyword. Volume sends the standard, cn format that we already have today to a designated signer and that signer can be the existing cluster certificate. Signer.

G

And then you hook those two up and you get a client certificate.

B

How do you plan on enforcing the boundness of bound tokens to bound certificates.

A

uh What which binding the pod one.

B

Or the secret one or just anything we come up with in the future right, but today the token authenticator for service accounts is what enforces well first, it's what expresses the extra fields that contain that information and also enforces that they're valid um client certificates today, with the schema that we have for them, lack the ability to express the uid or in extra fields of service accounts, and they also yeah just the way they're expressed, do not have um they would not like if you, if you created such a certificate, like pretend you just could today, with all the field set that the token authenticator for service accounts would be unaware of that and does not validate that the pod reference, for example, maps to existing pod, with that servicing et cetera, et cetera.

B

um So I know I think these are all tractable problems that are not necessarily hard uh they just they just need to be written down as like. This is how we plan on solving it, um uh and, generally speaking, that's that's what I would want. I don't I mean I guess I don't really have an opinion on if it's a new signer or the existing one as long as it functions, um because certainly the existing signer can, if you give it the right cn, we'll express the service account name and kubernetes would not really care.

B

um It's just that the uid and extra fields aren't expressed or enforced correctly.

G

Yeah, I think what I was thinking of was just in the cs, treat the csr as not really a contract for the shape of the certificate. You have to issue from your signer, but just a cryptographic proof of the identity of the pod, requesting the certificate, um and then you know the default signer that we already have will respond with one certificate.

G

If other people like other formats, their signers, can respond with other formats.

G

So yeah I can. I can do that surgery to extract the cubelet piece from the existing.

G

Google doc and bring it back at our next meeting.

B

I'm not sure if I fully followed exactly what you said like I. I guess I don't necessarily care as long as the enforcement is still there as long as the protections that we provide for bound service. Defense right, like in particular, uh bound service account tokens, even because, even though their jobs are still revokable right, because they reference objects within the api that we can look up and revoke them using, and I would want that same property for these certificates, even though we don't have like crs or any mechanism certificates in general.

A

So, are we only talking about clients of the cube api server? How do you want these client search to be used.

B

Yeah plans for the api server, I don't I don't I mean today. We don't enforce any particular trust chain requirements, so that's where the new signer could be beneficial, because you could explicitly say that this should that new signer should not overlap with any uh should not, but it's not enforced uh to overlap with any other existing signers trust chain. So that way you can keep them separate.

B

If you wanted to. This is what I would want to do. um I guess but, but I mean I guess like without that. Also I just.

G

G

I was just saying my thinking was to focus on the cubelet piece first, because the feedback I got before was um we need to focus on one piece of it. First and the keyboard is like, like the actual requesting of the certificate, is the most concrete piece that can be landed. First.

G

And then you know we can work on okay, cube. Api server should enforce some more requirements like object, binding, for example, on the certificates that it expects.

B

Which I agree that makes me nervous, because that sounds like we have an intermediate state. That's unsafe.

G

It's not unsafe, it's the same safety we have today. Today, you can go off in uh with some difficulty request a certificate for your pod and use it, and it doesn't have object. Mining.

B

What yeah I mean, that's what I mean like that, you don't give it.

C

One automatically.

B

Yeah, that's that's the part, that's what I'm saying. Okay yeah! You can do bad things if you have enough, like time invested in the bad things I'm saying like like I I would want people. I don't know what the mechanism would eventually look like, but I would want people to be willing to have this basically on for all pods all the time with no qualms.

G

Yeah- and I was thinking about automatic issuance- don't don't think about that for now, um just first, let's talk about how we actually get the certificate into a pod. If someone is willing to, you know, write a projected volume or whatever.

G

G

Do the other steps in that process where the we eventually end up with everybody, getting a nice secure certificate by default? And you don't have to use prayer tokens.

B

I guess I could ask: could we could we go the other way? Could we we pull out the object, binding validation that we have out of the the what's the space the service account token authenticator into a generic layer above an authentication and then come up with a schema for full expression of all the user info fields for certificates?

B

You have that, then your and your api server is now ready to, regardless of how those certificates get created and made enforce the same validations.

B

A

It's not particularly complicated.

B

It just does no one's tried to write it down, well enough to get it through into a cap and anywhere right. That's what I mean like we. I think we talk about it, but we haven't made progress. So I'm trying to like go from tiny pieces that I don't think cause contention of any sort really.

A

Yeah, so I guess I I think that client certificates for um the cube api server are as far as like keps go is gonna. It feels lower value to me than support for mtls.

A

Workload to workload within the cluster.

A

Yeah, so I guess, like you know, if the tokens still exist like what problem are we solving by pursuing this, um and I guess, if it's just to get our foot in the door, then are we actually really getting our foot in the door towards you know, like maybe higher value improvements down the line.

A

Agree or disagree.

G

I think object binding on a certificate is like it's pretty easy like we could. Just I don't like, we could extend the existing mtls authenticator pretty easily to support optional um object binding on the certificate like you know, we'll have to do some dance to get an oh, an appropriate oid to store this information under in the certificate, but then we can just stash the pod uid in there right and re-you know slice and dice our existing object, binding code. For the token.

G

um So if that is valuable, uh we can just, I think, that's kind of like unconnected or unordered with the rest of all the certificates. Work.

B

Does that make sense a few times.

E

Yeah I mean I, I did think we were trying to separate this into pieces so that it could land in smaller chunks, and I thought one of those was you needed to be able to have a serving certificate for all services and we agreed that one was the easier problem.

E

I'm I'm trying to understand whether mike your suggestion is to actually go back to trying to do it all at once, because I feel like that got bogged down in.

G

Like a client certificate for all pods or serving certificate for all services, because I don't think serving certificate for all services is a very easy problem. That's a very difficult problem.

A

So so my comment was only on this discussion. Field like it was focused primarily on uh mutual tls2, cube api server that seemed like between mutual tls, between pods or serving certs for pods, for other things, to talk to them that field that felt lower value than you know, the other. You know potential motivate motivations.

A

We have like a reasonable story for authentication to the cube api server already, um so I would say maybe we should prioritize the others unless this is like a clear avenue towards you know achieving the other goals.

A

In short order, uh do you buy it david.

E

As long as there's still some iterative path where it's right like a an on-ramp and not an on cliff, uh because I don't I, I want to make sure, there's some way that we can look at it and say yep. We can fit this much in this time. We can fit that much in the next time and um rather than just keep saying now, you can't come in because it's it's too big a change all at once so yeah as long as as long as that's still there then yeah, I'm okay with that.

B

So mike the piece you thought was most valuable was empty less between kubernetes services on like a single cluster yeah, so meaning we would need the serving certificate bit for the services and client certificates for the service accounts. Those two things together.

A

Yeah well, let's start making progress towards one of those it seems like our discussion was kind of around.

A

Yeah authentication the cube api server only or what we would need to bring uh mtls to keep ava server up to, like parody with um branch services, account tokens.

B

Oh okay, so like as as a very smallish starting point, then could we attempt- maybe just I I guess- I'm not necessarily falling to here where you said that serving certificates for all cube services is hard. I feel like this is the thing that david built years ago in openshift. I just just want the same thing. Basically.

G

The problem is service, lifetime is totally decoupled from pod lifetime, and the membership of pods and service accounts backing a service can change at any time in a very fluid way.

G

So you need to be able to invalidate those certificates.

G

As membership in the service changes, a client certificate for a pod service account identity is like doesn't have that problem.

E

So so you, I guess that seems a little bit odd to me that you would worry about about. You know when a pod leaves a service, you would want to remove it, but you.

E

Or when a pot is deleted, you want to revoke the certificate or are you but you wouldn't for a client certificate, or I would.

G

Say different when a pod leaves a service, other people shouldn't or other you know, pawns should no longer trust it to offer that service.

G

G

We actually don't solve this problem at google. um We we instead build. We only issue client certificates and we build a mechanism that lets you check whether the client certificate you're talking to behind the service you're talking to is um is valid.

A

That's kind of confusing I I would say that it is more like there is a mapping of services to service accounts and certs carry a service account. Identifier, yeah, okay,.

A

So I I think the thing with services is, um you know, they're loosely coupled to pods. um You know over time.

A

uh um Pods may be added added and removed to services. I think that has implications on potentially both sides, uh like you have to update a pods serving certificate to include new names. You have to revoke old, serving certificates for pods that have old names. If you want to do that sort of.

A

Like degree of isolation, another way to think another thing that we could do is just punch within namespaces, because pods can only back services in the same name space. It might be like an okay path forward.

G

My thinking for the concrete first step was focus on mechanism for getting a certificate into a pod and then letting then we can experiment with like okay. I want to think about service account, certificates or sorry. I want to think about uh serving certificates for a service. We can write a new signer that has some strategy for authenticating.

G

You know uh for issuing those certificates. um I don't know what that would be yet, but that's why I was focusing on the cubelet piece. First.

B

Yeah, I think I think that's fair. I guess I I don't know how others feel. I I to me if, as long as you keep the serving certificates short lived, you know some reasonable lifetime.

B

I I'm not stressed by revoking them if you can, if you're part of the service you're part of the service, um if you were recently part of the service yeah, whatever.

A

What about having to reissue.

B

A

Reissue certificates when services are created, do you think that is problematic.

B

A

If somebody had like a service, v1 service, v2 or I've, seen people like version their services by you know some like deployment, so it's like the uh creating new services and deleting old services happens frequently now each one of those events require. um You know, uh refreshing all certificates across all pods within the bacnet service. Like is that churn like something that we want to um that we're? Okay with do those updates happen? All at the same time um seems like a.

A

Potentially scary impactful operation versus, like you, know what we have before.

B

A

That make sense.

B

Yeah so so I think um david correct me if I'm wrong like in openshift, we kind of sidestepped this a bit just because we stored the serving certificate in a secret. So it was just mounted.

E

Right, uh you ended up having to say I have this pod. I know this is the service that I want to have. I want to serve for you, get to choose one and it will create it in a secret, and then your pod just relies on that secret. There's, no automatic injection, but it did make it very easy to go from. I have this thing that serves.

E

I need a certificate that other services are gonna, be able to trust, and I'm gonna at least say that inside that this contained two pods inside this name space, and that got us a very long way. I I do have to drop, but I can. I can appreciate that if that is not the direction you wish to go, uh that might not be a useful step for you to move in.

A

E

B

A

Hearing more uh thanks, everyone yeah we're a little over time. um So talk to you all in the new years.

B

Well, thank you.