Kubernetes SIG Auth, 30 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sig-Auth Bi-Weekly Meeting for 20200330

Description

Sig-Auth Bi-Weekly Meeting for 20200330

A

uh Hey everyone: this is the march 30th 22nd 2022 um cigars meeting uh yesterday was code freeze for 3, so I think everybody's good and tired right now. uh Thankfully we have a pretty light agenda uh granted. I don't know to hear how much discussion he wanted to have about trust makers today, but we'll find out and get to it, um but max. I think you have the first two items.

B

Yeah, I think we can drop the second one, because you already answered this so uh about the first one. The idea is simple: I proposed it on slack, so we have a bunch of apis to debug authentication flow. For example, token review, subject, access review self subject and this review and so on, but there is no api which helps you to get.

B

You know who you are your attributes, your user id username, extra attributes and so on and as we discussed on slack, it looks like a missing part of this, so I propose to add some kind of an api end point to check your attributes and also provide a corresponding obstacle command to like to to make the back easier to make debugging easier. Yes, that's the idea.

C

I remember this being rejected in the past due to concerns about anyone with a token being able to play it and see who it belonged to. I want to say the concerns were from mike denise, one of the googlers. Do you also remember that mode.

A

I do not this might have. This might have been before me. Okay,.

B

I think you can still. You can still use token review to get user attributes.

C

Most users cannot, which is why you're asking for this? If most users, could you wouldn't be asking for this.

D

Can't you just use uh can't you just make a request that gets rejected by rbac and it returns your username yeah, not seeing that as a like solution to this feature.

C

Request, but that seems.

D

I don't, I don't remember.

C

It uh so I don't. I don't remember whether that was there or not. I bring this up because I would like to be sure that same party gets asked again, um so they get a chance to explain it.

A

Yeah, so uh like so with my personal experience with this in gcp, is that gcp does or tk really does encode some encrypted blobs in the extra field.

A

Which, I guess they're happy with all the other web books and everything else being able to see because they're semi privileged um yeah, I mean I I mean. I would personally like to see this. uh I guess I'd, maybe even be okay. If it's our back is optional or something, but the our back to call the api is optional.

A

I guess that would be that would at least alleviate the concerns. um I know jordan who's also not on the call I mentioned concerns about just the extra fields in general. We don't know what people are putting in them and that that could be bad, and I feel, like you, shouldn't rely on hiding secrets inside of extra people can do anything.

E

Yeah I mean the so the problem maybe so like we could build something that works for our back, but when you're running on a cloud platform, usually you're integrated with the cloud platforms, I am abstraction, so um kind of who you are and what you can do is kind of a very open-ended question when you're using webhook token, authenticators and web token authorizers.

B

Or proxy or something yeah.

A

Yeah, I'm not saying this wouldn't work with all those I I should have used the word authorization, the idea being that if, if you built an api but then did not provide bootstrap our back policy for it, then it's up to the cluster admin to assign that. But that's what I was saying is that we could. If the concern is who can see this and you want to control who can see it then go for it? uh Don't don't give don't give anyone access or give whoever you trust to have access?

A

um Okay, like you know, I and kanzu and openshift clusters, users can.

B

A

Are and their groups, but not extra? Yes, maybe am I remembering I don't think you can see your own extra in. I don't think you can see your own extra.

C

I couldn't say for sure, but.

A

Yeah, I remember you can see uid username and groups, but you can't you can't.

C

See the last bit yeah and it exists in openshift, because openshift also has a user concept uh which is managed on the platform and that isn't the case for general cube. So.

B

As I see it most of the time, that's enough just to see groups and username- that's that's would be enough.

A

Would this new endpoint be authorized or available to all uh micah this question? Yeah, I mean so I think that's open-ended. We could. We could do something similar to what we did with the bound service. Token account discovery endpoint, where we we only bind that one to service accounts uh and something like this. We may.

A

I guess I don't know if that care, if a service account can see its own metadata, um because it's already in the in the token, so maybe we could also just the service accounts, um but the choice for binding it to like system called authenticated or system called unauthenticated is deferred to the platform for the discovery, and we could technically do the same here. The same concerns present.

A

It does reduce the usefulness of the api, because then it will maybe work or maybe not work depending on which platform.

A

But um we want max, we can try to follow up with mike and jordan. Make sure I I guess david you're here. Do you have concerns? What's your what's your stance.

C

um I don't have an issue with it. I just want to make sure that people who had previously objected to it uh get a chance to object to it again uh and provide those reasons and- and I presented them as best- I remember, but it was many years ago. This came up.

B

So, okay, I'll ping them and I think, we'll discuss the rest uh on slack. That's okay,.

A

Yeah um yeah, if you feel feel free to ping them directly in slack and try to get feedback uh on my side. I'd be happy to look at it kept, but maybe don't write one until you understand their concerns and if, if, if you want we can, we can also put this on the agenda for two weeks from now and try to try to discuss again if they're on on the call.

B

So I will do it on my own if it would so to be necessary.

A

uh I'm I'm more than happy to discuss the second item too max. You don't have to just take what I said as the final answer um did you want to discuss it.

B

Yeah, maybe someone has the same problem as I am they have just because you know we have this repo authorization that you know you can custom customize your like excel slides. So uh we use this to provide some kind of a multi-tenancy to forbid access to some namespaces or some resources during days or nights for some engineers.

B

So such policies and we want to enforce them, so we want them to be strict, but if like, if webhook doesn't work because of something, so um it counts like like okay situation, so you do not policies doesn't work. If webhook doesn't work, that's not. I don't think this is.

B

So in in our case, this is not useful. It became uh like we cannot use this.

B

This way focal.

C

Authorization, so so, just so, I'm understanding you want you're using a web of authorization as a deny authorizer, and when there is an error, what happens is it gets treated as no opinion, as opposed to deny.

B

But it allows so uh uh if it allowed buyer buck, uh for example, uh and uh we use, we use google just to deny requests so and if this is, if these requests are allowed by bugs, so they are allowed um if a book isn't unavailable.

B

Something like so.

C

It's not about whether the web hook is available or not is the fact. You need the web hook to happen before the rbac authorizer.

B

And, and for beat every request here, something like this.

C

Okay, this is similar to what daniel asked about a while back.

C

That sounds interesting enough to be kept worthy to me. What did you say.

A

Oh yeah, no, what I what I said is this is totally fine. We there's already folks trying to write a kept for a proper, structured config for multiple authorization web books, one one of which would be this capability of configuring. Your failure policy- and you know there. You know it could be fail close by default or it could be just there is no default, and you have to tell us, but I just I what I didn't want is more api server flags just trying to tack on features.

C

um As it happens right now, I believe we have an ordered authorizer but yeah. um I I don't object to a structure config, if that's necessary to do it. That sounds okay, uh the idea of being able to build something that allows a deny authorizer sounds very reasonable to me. uh It's a thing that that we've made use of before.

A

Yeah, I think the feature is great and I mean I feel like it should be.

A

It should be very easy and accessible, whereas I think today it's a little bit weird and and the deny authorizers were added after the fact. So I think you know the the failure, the failed, close behavior. Sorry, the the behavior of failing open, is backwards compatible, but not great.

A

Yeah yeah, I do agree. This whole thing would need a kept and, if we're going to write a cat, well, there's already an existing effort for a cat. Please just work on the clip together.

A

Anything else max.

B

No everything's, okay, so I I I am going to wait for the implementation of this structure conflict. I think.

A

uh So I wanted to bring uh uh sure I went to pr so so there was some recent effort by some folks to clean up or try to fix some bugs in our client-side oidc logic. So this is the the not not the server side, but but uh with cube, ctl and client going both, um and I wanted to get folks's feelings around one around sort of the the health of this thing that we have that we basically don't fix bugs in. Even though there's like known issues, uh they've been known issues open for years.

A

For this thing they do cause people pain uh and we basically haven't meaningfully changed it and multiple.

A

If you ignore my one fix like five years four years, so that's one bit I wanted to talk about, but then the other thing was the folks have thoughts on just refactoring the whole thing to do something drastically different just to not try to use the cube config as its storage layer and instead actually use the file system.

A

It's like one lock instead of this weird. So today it has a tier of locks where there's like the in-process lock and the config based lock.

A

Ish, um it's really messy and really hard to reason about what's gonna happen and uh well, there's bugs in it and folks were trying to fix the bugs and like tim and myself reviewed it and they were like so bad and then, like you, didn't make coach breeze, because I just didn't feel confident enough in what the change was doing to feel good about it.

C

Didn't even have thoughts on this.

C

I don't have experience in that area of the code david. You wrote it son of a gun, uh so I don't remember that.

F

C

Yeah, it's it's more accurate.

A

Yeah, I think literally, you wrote it. Eric did some changes to it or you and eric maybe wrote it together or whatever, and then it was just. It hasn't been touched and meaningfully touched in at least four years.

C

um Can you can you and sorry for for those of us who don't remember, can you enumerate some of the bugs in it.

A

uh So there I think, there's like two open right now, uh but they they all come from weird behaviors caused by this global client cache primarily around when refresh tokens are rotated and basically it gets confused and will like sometimes return to stale refresh token or sometimes not, and so you know on certain providers.

A

This is not problematic, because a lot of providers should just keep returning the same refresh token or no refresh token at all and just assume you're using the old one, whereas like modern providers, modern in the sense of more security, conscious ones make the refresh tokens one use, so you get a new one, every single time and the old ones invalidated. In those cases, this thing goes haywire and just falls off a push.

A

uh Yeah so I mean yeah. I have a hard time reasoning about this code, because there's there's locks inside this client cache there's like global locks, then there's this thing that hides a lock, because it's the cube, config abstraction um and all of this is really really muddled. If you ever invoke more than one tube cto command at the same time, because there's no synchronization against the cube config itself, so then like we're gonna, basically trash your food config, which was also not cool.

A

There's no flock on updates to the coop config unless something dramatically changed. No, that's terrifying! Yes! Yes, I know.

E

Like but but it's like, yeah go ahead. Three years I was gonna say um like for the out of three provider plugins that is like for the g cloud, one that's what we have done is we have a separate cache file where we use a simple locking. Actually we don't even lock it. We um just rewrite the whole file um every time like atomically, um if you have a complicated one.

E

One thing we looked at was um using something like both db, so just like a tiny key value store so that we didn't have to worry about, like all the complexity of locking the file and um ensuring that rights are consistent and all that good stuff, in our case, we're only caching a single value. So we just went with um atomically replacing the file yeah.

A

Oh sorry about that.

B

uh I wanted to say that I think this is a good pattern and we ended up using our custom, like exec plugin, doing something as the same thing like storing credentials in a separate file, and it works perfectly.

F

Atomic updates of the file- don't don't get around the read, update right cycle problem, though, like you could still have two processes that read the file and then race, to which one writes.

E

I think, in our case, it doesn't matter because um when you use g cloud authentication, no matter which cluster you're connecting to you're always the same person, it's a little weird.

F

uh Yeah, no, that there are specific circumstances where it doesn't where it doesn't matter, but for the kube config file, specifically for configuration files like there are. There are certainly circumstances where just doing atomic replaces is not enough. Okay,.

E

Yeah, because if you're doing refresh tokens, you got to read the reverse joke and send it and then write a new one. I.

F

E

Both db, I think, has transactions which would.

A

E

A

Yeah, so I mean, I guess: that's where this comes down to right is: how far are we willing to go to actually make this work right like in something correct? I shouldn't use the word right, because uh this happens to be important to people, because it's built into cube, ctl and very soon. This will be the only thing built into cube ctl, because the next release is when the built-in aks and gk stuff goes away.

A

So I'm just trying to sort of figure out how we roll out changes to this thing and how dramatic we can be with them um as a for example, we if we build in some mechanism that doesn't store credentials in the cube config anymore.

A

Well now you can't copy paste your config across computers, which, as much as I don't want people to do they do that.

B

But if like, if the refresh click invitation enabled for your open id connect provider, you, you cannot copy config around your servers.

A

Yeah you're right because as soon as you use it in one place, it'll invalidate the other one which, which is what you want.

A

But it is frustrating.

B

So this is a real problem for dex, because index uh rotation of refresh tokens was enabled by default, and uh we have a lot of bug. Reports uh about this- that coop city lost refresh tokens.

A

Yeah, it's it's! It's easy for it to do that, because it's just not it's just not sound! um I mean at a high level. What I was proposing is that we piggyback off the cache directory that is currently used for discovery and some other things which I don't remember what they are.

A

And start having a distinct, separate cache in there and using that file or whatever, as the actual locking mechanism, not not anything in process or anything like that and not having multiple layers. If you make a million clients, it's okay, you get a million copies, but they'll still be synchronized by the file system. It doesn't matter.

A

But I don't yeah, I don't know if anyone has strong opinions on this or if there's anyone willing to like work on this stuff, but.

A

Yeah, I I just don't think trying to keep it as is, is going to work out. I think we're just trying to monkey patch around structural issues.

A

Anyone else have anything on this: we can, you can go to the phd.

E

Can we use fuzz testing now in kubernetes or not the 118 stuff kubernetes go 118.

A

I have not checked, I suspect we haven't bumped to 118 yet because it hasn't been out that long it usually takes a while before.

A

I guess we're large enough that we do enough terrible things that version bumps of go calls us.

A

I don't know. Why did you bring up the fuzzing? I mean this seems like a situation that cries.

E

A

Fuzzing well try to just bash at it and watch it and float itself.

D

I'm I'm still confused about each client in this current implementation having their own mutex for protecting cube, config rates because they're all using the same cube, config right. So you have multiple clients with separate mutexes to protect rates to the same cubeconfig.

A

That just sounds broken, but no, it is broken at many layers tim, it's just like it's! It's written slowly enough that if you just scan it, you won't pick up on the fact that it's doing things that don't structurally make sense.

A

It's easy to miss because it there's not a lot of code, but it's all very subtle and it's very deep within client go and its interactions with its surroundings are not obvious.

C

And apparently it's my fault, um you are eric's or both. I I don't remember enough to have a meaningful opinion, though I'm sorry.

A

No, it's fine! I just I I I was hoping to try to get a little bit of discussion on this, because I I felt bad that I we couldn't come up with something within code for his time, but I was like I wasn't comfortable with the changes.

E

Okay, uh here it doesn't sound, like anyone, has a strong objection to beautify again.

A

Yeah it's basically, this is abandoned, go do whatever you want, just don't break it worse.

A

All right trusted anchor sets. uh I have not read this cap.

E

That would be because I just wrote it at 3am last night, so this is so yeah. This is some of the feedback we got on the original.

E

Certificate provisioner kept draft google doc thingy was that trust anchor sets were a useful piece of functionality on their own, so they should be broken out into account and landed, and then so I've done that uh it's largely the same, except I did a little digging into how certificate signing requests interact with our back and I copied that mechanism for trust anchor sets um so trust. The trust maker sets now carry a sign, a spec dot, signer name field.

E

um That is the name of their signer, that controls them, and then you can use rbac to say a user xyz can update all trust. Snickers sets associated with signer x.

E

The rest is pretty much a straightforward.

E

E

um So if anyone feels like giving it a look, please do and we.

A

Can have an in-depth discussion next time or something uh so I just did a quick scan of the api. The one thing that sticks out to me is um as a for example, on these things. um It is, is the concept of a crl discussed within this cap?

A

No can can we have a concept of a sphero.

E

I don't know very much about how crl's.

F

Work a few different ways, but the most obvious one for this is just a crl blacklist file deny list. Okay, yeah, deny list sure.

A

um Yeah I mean so it is possible for this, the crl endpoint to be embedded within the ca itself, uh and that's not uncommon for like the big cas to have it, because you know they have a ton of pki infrastructure, but um it would be nice to have the capability of you know. I you know I have this. uh uh The ca bundle whatever it's, maybe just a file on disk, and I want to be able to control revocation for it within this.

A

Okay, like the perspective, I'm coming from this side, is customers keep asking us to use their corporate cas for kubernetes and don't care when we tell them not to, and it is really hard and practical to do that without a crm support.

E

Do you think a better? It would be better to have a mechanism that is just within this trust. Anchor set object, have a list of revoked certificates or revoked certificate hashes, or I know, there's also like ocsp right, where you can designate a url that will serve the.

A

Yeah, I guess I wasn't imagining the url-based one, but um I guess you could have both. I guess it wouldn't necessarily hurt to have both.

A

I I think the url one is traditionally what people kind of expect, because that's what other tools tend to consume.

E

Aren't client don't client libraries, wouldn't they.

E

Wouldn't they expect that information to be embedded in the certificate, and not I mean I I'm only moderately familiar with go's tls stack um yeah. So black would imagine you can't just mix and match like oh hey, here's my I don't know I'll have to look how how does go. Take the ocsb url.

A

So last I checked, which has been a while, and maybe david you have more experience with this. Is it's not in the standard library but like cloudflare, provides some hooks that you can add to your go code.

C

There's some parsing methods in the standard library, but the standard verifier doesn't have a spot to drop it.

C

So there's not like a verify option, at least as of uh one level of go go is that 116 is the last time I remember personally looking.

A

Just so there would be, I guess um some implication for like uh the cubelet or whichever actor we're, is doing the verification to be enhanced to also consume the information in some writing shape or form.

E

In this case, it's the work every workload would have to be like cubelet is only responsible for dropping these files on disk right.

A

Right, um which, which is fair, uh um I I think what what I'm hoping for is, the mechanism to be available and less of the enforcement that workloads do the right thing.

A

You can't enforce that they do anything right anyway,.

E

Oh yeah, I mean yeah, I definitely could have a csp url or something like that.

A

uh David or like tim or anyone have you all seen requests for anything like serials and stuff, or am I just.

C

In my own world, I've gotten a lot of requests for it. uh The assumption is that it's built into the library, and we just aren't using it. I I don't know that I have an opinion about coupling it together here. My memory is that you can actually just say on a ca bundle. Here's where my crl is and if the client honors it it works, and so, if the trust anchor gives us a spot to place that which I haven't read this cap, probably because it was made at 3am today then.

C

Then it may just work, but I I was gonna withhold judgment until I read.

A

Yeah, I guess in my head, I was imagining um like if you have an existing kubernetes cluster with just its file-based ca bundle, and you want to you have some way of doing, revocation in the sense of knowing when to evoke some certificates that you have issued.

A

Maybe it would be helpful if it wasn't part of this one um okay, so I realized. Probably no one has read it too much, but it folks have thoughts.

D

um Just one thought from looking at the api here. um I definitely think this will be useful for organizations that maintain their own uh private cas for internal services. um I'm not sure what I don't think. Signer name is necessarily relevant in that case.

D

Something to think about.

E

So the use case I was thinking of is, I have you know some private ca and I want to use it for like mutual tls between workloads in my cluster.

E

So I create my own example.com, slash, super cool ca, signer and that's how I surface the ability to use that private ca hierarchy inside my cluster um there's some cases where the signer name is kind of redundant. Where like what mike was wanting where we could throw some certs into one of these trust maker sets and then use it in a webhook, backend config.

E

But I figured you could just make up a dummy, signer name in that case.

D

Yeah, that makes sense I mean you can always just make up a dummy, signer name. um I guess I'm just encouraging you to think about the case where you want to do like mutual tls or something between a service. That's outside of the cluster as well.

E

A

So jim, are you saying that maybe signer name would be like optional have to use it.

D

Yeah, that might be all I'm saying, which is pretty nitpicky at this stage, um but I guess just kind of thinking about that use case.

D

More generally, having looked at the cap.

E

In the model where, where I'm thinking about this so and and my my perspective is colored very much by working on that particular implementation, so I may be wrong. um The.

E

If you're using youtube to us- and you want to federate yourself with um you- know some other company- you want to talk to their workloads- that's still occurring under the auspices of your dot. Example.Com super cool case signer, but it will have a different named. Trusting set you'll have the default one for your workloads and then you'll have you know: billy bob's, emporium trust set.

E

And you have to surface both of those trust anchor sets into your workloads, because you definitely don't want to intermingle them.

D

Yeah that makes sense.

E

But I'll need to think about it. Some more.

A

B

A

A

I guess we can make a quick look at.

A

It might be more red than expected.

A

Nish? Why is this scary symbol here.

B

If you click on it, which one is failing.

B

I can take a look at that. One.

A

Okay, no, it's fine, it looks like uh otherwise the regular stuff is happy and then, let's see.

A

A

That's right: the service is kind of open. Stuff is like the only thing that ever shows up.

A

A

Bunch of tests right- maybe maybe this good one.

A

Okay, not very confident in this.

A

David, do you happen to remember what these tests actually do?.

C

uh My eyes are so bad. I can't read what you're showing me: okay hold on sorry, my bad.

C

There we go blind person, um so the opting out was to be sure that you didn't get it assigned by admission. I am surprised that it's flaking, I don't have insight as to why.

A

I mean all of these failures seem to be related when a bunch of other things fail, and I'm just wondering if it just timed out, because the whole info is kind of it.

C

Could, depending on like how the pod was trying to get created or something like that, yeah.

A

Kind of willing to cheat here and be like we just went through properties, and we don't need to worry enough. We can give everyone 15 minutes. So anyone just.

C

Well, I wouldn't say: take it easy, but I've got a bunch of other stuff to look at.

A

All right, let's take that, uh does anyone else have anything they want to talk about.

A

It's not, you can see all them.

A

All right, that is a yes well.