Kubernetes SIG Auth, 12 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG-Auth Bi-Weekly Meeting for 20230412

Description

SIG-Auth Bi-Weekly Meeting for 20230412

A

All right, hello, everyone uh welcome to the April 12th uh 2023 meeting of saigoth. uh Let's kick off, um we have quite a few items on the agenda. uh First off uh I just want to make a quick announcement in case you're at kubecon or, if you're able to join remotely. uh We do have to take off Deep dive. uh The link is provided We'll, Be, Love, uh we'll love to catch up with folks, all right, um I, don't think, there's anything other than the discussion topic.

A

So first up we have the external signing service account token cap, PR.

B

Ide yeah so um yeah so last time the feedback was. If our motivation is just automatic rotation, we can do that through files um like there's uh like an existing path to just you know, have it pick up changes from disk? um We don't want to do that. We don't want to have these keys on disk. The um idea is to decouple this logic.

B

You know fully out of API server so that then you can have this separate key server component. Do whatever you want like, for example, pull from KMS, um so I listed a few other use cases in there, but.

C

D

Right um so I, is it just two use cases rotation which we already said wasn't like sort of valid for this one and then out of process yeah.

B

Yeah, it's it's mostly around out of process so that um you know that logic can be decoupled out of API server um and then um people can, you know, extend it in the way that suits their needs.

B

D

How does this relate to the fact that, for example, the cube controller manager still has to find this stuff for the.

E

Same basically,.

B

Yeah um yeah, that's a good question. What is I I was thinking to have that out of scope of this um because I guess, what's our long-term plan with those Legacy tokens.

D

They're going to stay forever right now, that's the long-term current plan.

F

We don't gen, we don't auto-generate them anymore, um but people can request them and there is not a plan in place today to uh get those people onto a better system without breaking them. Basically, you can. You have been able to request.

E

F

Token, in a secret since 1.0- and there are definitely things built on top of that and yeah like.

G

We had, we had.

F

A reasonable path for migrating to Auto generated ones, and that's basically done. We have some cleanup work to do there, but uh there's not a path for migrating explicit dependencies on the requested ones. We could maybe look at uh making the one if you request a token in a secret like moving the generation of that from the controller manager into the API server. That might be a possibility, um but the nature of those tokens I, don't think we can change reasonably because.

H

They're not expiring,.

F

They are not expiring, yeah I.

H

Mean I think for the purposes of discount, like you, you can have separate Keys like for the for the Legacy tokens that don't expire. You can do it do different key pair that maybe could be on disk.

H

um It's up to. You know the operator of the API server if they want to trust that email or not, um but that would that would still allow you for the temporary ones to basically not have any key that you have to hold on.

D

What does that give you.

H

D

H

A way to run kubernetes without a exportable.

D

Key but you still have the exportable keys just on the controller manager still, and it's the one that gives you tokens that last forever. If.

H

The API sort of respects it yes,.

D

Oh so you're going to break the core API on purpose and basically, like fall out of conformance to do this. I.

F

Don't actually know if conformance okay.

D

If it doesn't we'll go, add that and make it.

H

Or give people the option to do yeah.

D

I mean yeah, like so I I, don't think it's out of scope, I I! Think if you're gonna try to fix it, you should actually try to fix it right. um So stepping back for a bit is the I. Also think. Let me actually pause for a second uh David. You wrote a comment there. You want to talk about that. First, since I think you wrote.

E

It before I was like.

D

I'll read David's comment, which is: would we recommend that require service account? Tokens are not usable between different clusters, certainly can't enforce it can't enforce it today.

H

What's, can you repeat the question I'm, not sure I understood.

D

uh So David would comment on the agenda, which is basically um Rita. Do you want to put the agenda back on screen real quick, um which is basically, would we recommend or require that service account? Tokens are not reusable between different clusters.

E

Right, so if you have an external signer.

D

Yeah, it's just some global Service, and maybe it just has one key or something right now all your tokens are like completely reusable with glass clusters right. That's not exactly the intent of this feature right, no.

F

E

F

They check existence of the service, account object and that the you it matches so like you couldn't use those tokens against the cube, API server or in a token review like it, would complain and.

D

Say I mean okay, sure like yes, the if, uh if your uid generation never conflicts, which it really shouldn't, um you should be. Okay, that's fair.

F

Yeah I'm trying to understand like what keys are being used that are more powerful than access to the cube. Api server would be like. If you have access to the keep API server, you can already do things. You could restart the qbi server with another key pair uh so that it would like spit out tokens and accept tokens with that key pair like if you have that level of access.

F

What risk are we mitigating? Is it a key that is trusted by other things? For other reasons, um yeah.

H

It's mostly about just the like the well the export ability right like if you have a key that or a set of keys, even if you're rotating piece that are exportable, um they can't there's no audit process on those keys right and external. uh The external.

H

The API, the verifying of those of tokens signed by the API server that are used by other systems, say cloud off somewhere is, is going to have to trust the API server. You know we're not necessarily the API server, but wherever you have the issue or endpoint for The Trusted set of keys. So even if you know, if a if a bad actor does who control the API server does uh add a new, a new, you know row key.

H

They also have to have access to the oidc issuer or the the public oiec issuer or like uh verifying URL right like the.

E

H

And so like in the case of eks, those are two different systems um and.

D

You don't need to do that right, like you, could just use like the request, herder stuff right, the the ability to just tell the API server who you are so you can just impersonate any identity and just ask you to Mint tokens, so they would get audited you're right that there was totally because it would be following the regular path. It would be a regular audit Trail for this, but it doesn't prevent you from basically infinite abuse of the system in an online attack.

F

It if I'm understanding correctly this is for taking tokens you get from an API server and using them against another system. That is not using token review right or verification. So another system just has a key set that it's verifying against and presumably.

E

F

That you set from a different, more trusted or more audited system right.

H

Okay, because then you then you have like yes, an operator could, at you know, potentially at one point, create a road key update, an API server, a key or music credential have access to the API server, but they don't necessarily have access to the external system that those tokens are being used for, okay, right and then so take the taste of AWS. If, if even even a, if, even if I'm, not running EPS, I'm running cops and I. Put my oh, my keys in an S3 bucket for my oidc provider.

H

That's not my my Q API server!

H

um Now I have now as a you know, as a if I want to get Cloud credential like AWS Cloud credentials by by assuming some role with with a token signed by the API server. I have to have the key. I have to have be able to publish that key to the public endpoint and, and it is now it's orderable or it's. It's yeah.

E

It's it's not exclusive. Now, if you have.

H

A non-exportable key.

C

A

Just a quick time check um just FYI: do we want to like time box this.

F

uh We had said we did uh I. My like my take is that this is pretty niche.

F

um I see why it's important in a like that narrow combination of use case, like tokens granted by QB, has ever used somewhere else. um I think the questions about Cube controller manager do need to be resolved.

F

uh Otherwise we were sort of like half fixing the problem in kubernetes and it doesn't really make sense and, like someone mentioned something about types of EX like extending this, and that raises big questions for me, like I, don't really want the token surface to be extendable, I'm, not sure exactly what was meant by that um I think those have to be resolved and given the list of what we have in Flight, I would rather see us like finish some in-progress things before we take on a new extension point. But that's my.

E

My perspective.

D

So two two things to add to that: I generally agree with Jordan Sarah one is: could you do this completely out of tree with just like a proxy in front of the API server to just sign stuff, with whatever you want to like intercepts the resource?

D

You already have to do uh their seven proxy if you wanna, if you wanna, intercept other things.

F

Like intercept the token request, API.

D

You can already you can already intercept everything, Jordan request, header lets. You do right, like that's kind of the point, uh I'm not saying it's a good recommendation, but if, at the end of the day we're talking about a significantly complex feature that has like an incredibly Niche use case, I'm kind of okay, saying that you get an incredibly Niche implementation right, maybe I'm kind of hand waving there. A little bit. I haven't thought through that too much.

D

But the the other thing I wanted to suggest here to see if it was tractable is so I'm generally like really not in favor of any more of these grpc things being shoved into like core critical parts of the API server. Like I've learned my lesson from KMS: it's just a just a crappy thing: don't do it don't add more!

D

um Is there a way that we could try to leverage that existing extension point to help and disregard somehow so like having some kind of way to like chain the KMS? To give you like a derived key or something, but something that's not as something that's not on disk, but maybe in memory, maybe with some kind of TTL associated with it.

D

So that way you can have similar properties, but the operation that's doing the signing doesn't require a remote RPC right like that's like the core thing that cam sv2 does for us, is that it no longer has the remote thing in line to all the calls, so it's way less dangerous and way less likely to cause an outage.

D

um I also think this thing will turn into a security check box that I'll be forced to check and I really don't want to have outages, because I'm asked to check the checkbox.

D

Anybody have any thoughts on that.

D

I'm just reading the comments.

H

Makes an interesting idea, but I have to think more about it like does that solve your problems? If you only have a provide like a KMS encrypted key on disk that can be decrypted, because it's still.

D

I mean it would stay in process right. The API server is not going to write it out. It's got. It's got to stay in process right, so I I think at some point you can start saying that someone's not necessarily going to be able to get into the memory of the API server. But maybe, if that's still in your threat model, then you get really painful.

H

um Yeah I mean if they have access to the same identity as the or whatever you know. Whatever permissions that the process that's running, the API server has to decrypt the token that's true. They could do that. So.

E

It's still exportable.

D

Yeah, so in that sense, it's there um sort of the way I've seen some of this stuff handled in like Azure ID and like GitHub, um is they they use some cert based signing and they issue intermediates that are short-lived.

D

So you, you can't really meaningfully export those because they die very quickly but they're, not in line to the request right. You you ask for it. uh I guess you I think it goes through some kind of CSR flow, but um you you generate your local key, you ask for it to be signed and you have it intermediate and you get to use it for a certain amount of time.

D

So obviously the root private key and all that stuff is sitting in the HSM and that's long-lived, and maybe you use cross signing to rotate it out whenever you feel like it, but inside the API server or inside whatever component you have it's not present.

D

The intermediate is present for, however long like it could be, as short as like 10 minutes right, because the point is to make it so that it's not in line like that's the core thing. I want to try to avoid here um and yes, I agree with Jordan's time box, because now we're at 20 minutes.

F

My guess is even a short, reduce period doesn't satisfy that unexportable requirement. That's my guess. Yeah I'm happy to talk about this face-to-face and yeah I am looking forward to seeing the folks who are there um yeah.

G

E

E

F

Terms of next steps, specifically for the the cap, maybe noodling, on the cube controller manager aspect like what what could we do there to maybe unify or yeah that that's a specific thing that I think we need to answer.

D

Cool uh we wanna Jordan. Were you the one who wrote All these caps out yeah.

F

I did um I uh like I, said I'm, not really expecting us to like leave today, knowing like this is exactly what we're working on for 128, but um I wanted to I, first linked to the open enhancements issues tagged with Sig auth, just to remind us that um there's a lot of stuff that isn't done yet that we kind of have in flight, and uh so as we want to look at spinning up new stuff. Also, let's try to um make progress and complete some of the in-flight stuff.

F

uh So that was just the first thing. Take a look there. If there are things you've been involved in you kind of forgot about or sitting here in beta, uh maybe look at what it takes to get to GA, and if we can make some of those things happen. um The second bullet was uh I. Think it'd be helpful to know where things are in each stage. So some things you need design some things already designed and we just need to like Implement them and make progress. Some things have implementations that need review.

F

So whatever the work is, that needs to be done like the next step. What we're wanting to accomplish for 128, let's be really clear about like what the goal is for 128 land of design or have an implementation and a review, and then we need to make sure we're load, balancing folks. So like this, this isn't really like a wish list like here's. What I want to be in 128, but I'm not planning on working on. This is more like here's, what I'm planning to do in 128.

F

So, if uh I put a lot of question marks and kind of put names of people, but um hearing from you all would be super helpful, like here's, what you are planning to do, whether it's work on a design or work on an implementation, um so I'm trying to take question marks off of things as people commit to them, or we get confirmation that people are planning to work on.

F

uh So that's going to be overview and.

A

Ideally, the ones that we want to focus on for 120 should have be added to a milestone right.

F

Yeah I took a first pass on things that I am planning to work on uh either as a reviewer or helping Drive implementation, or that I knew I'd seen other people like actively updating stuff on so I took a first stab at Milestones, but that's just a first step.

F

And then I roughly sorted this list uh in order of like things that were furthest along uh in the pipeline. So um there are a couple things that the who am I, API, very small surface area, pretty well understood pretty much just mechanically stepping through the progression. So I think that one um is a reasonable thing to Target GA I'm happy to review it uh and I need to get confirmation. Oh I see a uh someone in the chat. uh Hopefully this is a pretty mechanical graduation to GA um that we can get in.

I

um There are some tests required like to improve coverage and also, maybe writing a blog post about just by this, no obstacles. Okay,.

F

um The next one was the cleanup. This is actually around the Legacy token use, so uh tracking of Legacy tokens um is a pretty mechanical graduation.

F

um That concludes sort of the easy steps of the cleanup. uh The hard step is um actually doing the cleanup using the tracking information to do the cleanup so there's a implementation in Flight I'm on the hook to review it uh I think there's a good chance of getting that to Alpha in 128.

D

So we're going to have a controller that goes around deletes people's stuff.

E

Yeah, it's gonna go so well.

D

F

um I, don't know if we want to step through this Sergey is not here. Did you want to talk about KMS V2, no,.

E

D

Thought Sergey was here, that's where I saw him on. Maybe he had to drop down here? Okay, um sure, uh so we can uh near the later. Part of the agenda is like what I'm highlighting here like I. Have some stuff I want to make changes to for chemistry to to the crypto stuff, um so I, so in my mind, um KMS V2 cannot go to GA in 128, like I need two releases like I planned on making like some read level changes and then some right level changes.

D

So I needed two releases to do that um unless I wanted to add flags and I didn't really want to add Flags.

D

um uh Let's see, I see David's comment about API Machinery features, um so what's still sort of up in the air about KMS V2 is how much we're gonna try to make uh be automatic so like in terms of rotation.

D

um So if we, if we're saying KMS V2, has to have like near completed automated rotation, then we gotta, like finish all the storage version, stuff If. Instead, we would pull that out, maybe into its own thing on track it separately. Then it would not necessarily block kmsv2 itself from going GA right, so there could be a separate cap. That's basically like here's, the storage version API that does this right now and here's the API server identity, stuff. That does this right now and together and add some enhancements. And here you go.

D

Here's a new cap that says: how are you going to do automated rotation? um I, don't know, I, don't know where I've landed on that yet, um but that I think those are like the two big things.

E

D

We can come back to like crypto stuff, so, let's see do you want to keep going to Cluster trust bundle, see that thing.

F

uh Just to be clear, hey must be too staying at beta 128 with ongoing work in beta right.

D

Yeah, that's that's my plan. Unless someone has some seriously different opinions.

F

uh Yeah: let's talk about cluster custom doll.

J

um So we have one more piece of Alpha scope, work to land I, think in 128, which is the actual projected volume support, um and then you talked about beta, but I think it makes a little more sense to land cubelet certificate, support like cubelet pod certificate support and have them both be at Alpha first, so we can see how they play together if that makes sense, um so that was my kind of optimistic forward-looking plan was I. Have that draft kept about workload?

J

Certificates I want to turn that into a real cap, I'm working on a draft implementation as well. So hopefully it can go a little bit faster than this. Last time, I've learned some stuff about uh lending things in kubernetes.

J

The so that was kind of my thinking is try to get first finish up the projected volumes on closer trust, bundles and then simultaneously try to get an alpha of workload. Certificate projected volumes.

E

F

I'm not opposed to getting the projected volume in Alpha. First I do think there's value in the API, even if we don't have the projected volumes like just as a as a way to say here's how you get the signing certificates for this. Even if things were talking to the API directly um I I think that's a useful API, even by itself um so they're delaying.

D

It maybe the a small hesitation I would have with that. Jordan is what, if we want to make changes to that API, based on what we learned from implementing the cube.

J

Is I don't know like I'm, not aware of anyone like actively planning to use it like? We have plans to use it from gke, but that's all pretty much based on projected volumes. I think.

F

I mean having an answer for people who are like say: you've got these kubernetes signers for csrs. That's awesome where do I get the cas for those like having a thing to point them at would be nice. That's something we've sort of just said. No opinion talked to your cluster provider right.

D

Right does this kept cover actually filling out the trust bundles for the.

F

No but the apis are representative.

D

Yeah, okay, so what what you're saying is if the API existed and the platform wanted to use them to expose it as they could in a reasonable way, then.

F

You could say like give me all the cluster trust bundles for this signer name like.

E

Screaming sorry but I I don't feel.

F

That strongly, if we want to focus on the projected volume and like really make sure we love that that's fine.

E

J

I'm a little more comfortable with trying.

E

J

The complete set first all.

A

Right who's, gonna be the reviewer for it.

F

um I can review the API bits, but we need node and storage reviewers um for the cubelet and volume bits.

J

E

D

The dates here.

E

D

Have dates yet.

F

I mean typically the second release of the Year ships in mid-august, so code freeze is mid. July and design freeze is like mid-may -ish, okay.

J

I'll, try to keep that in mind. I have a unfortunate habit of scheduling vacations right around the freeze. I, don't know why that works out, but uh keep us clear this year.

F

That's not a problem, just we'll get everything done like a month early.

I

D

I'm trying to like look at, let's see it would be 124 right that we should look at for guidance on when code freeze happens normally.

E

I guess really: it's enhancement, freeze that we want I know this is wrong. This is in February.

E

D

Freeze for 125 was August like beginning of August and enhancements. Freeze was near end of June.

D

E

That sound right.

D

No that doesn't sound right because kubecon EU was until May that time and now it's in like April, so maybe there's like a whole shift. I don't know, you've got some time, but not necessarily as much time as you think.

F

That's a good General, Life principle. You have some time, but not as much time as you.

E

D

Okay, cool uh multiple authorization, workbooks.

F

um I'm very interested in this for two reasons: I would like to be able to run multiple authorization, web Hooks and.

B

F

Would like to be able to control the failure policy on a web hook like right now uh web hooks that there's an error calling them their timeout or something authorization's like well, no opinion go on to the next authorizer and I would like to be able to say if this authorizer is unavailable that counts as a or times out. That's a deal closed, um so I am interested. I can help with review um at the very least, maybe implementation or design, but uh I.

E

F

Up for some of this.

D

Jordan, do you hate me if I say I want to be able to configure them through the API.

F

D

But David last time I talked to David. He seemed to want it too, and I have folks that want it.

F

um Let me let me so I don't want to couple the ability to run multiple web hooks to an API if we also want to expose it via API I wouldn't want to run that API. But if someone wanted to work on exposing that optionally in a non-conformance way.

C

F

I, probably wouldn't hard block it, but I would raise big eyebrows about like. Why are we doing a thing that is optional and non-conformance and can't be relied on and at least I wouldn't want to run in clusters? I was responsible for.

D

um I mean if your customers ask you to you, might have to right. It's kind of it's kind of the gist of how these things go.

D

um Okay, uh yeah I, don't know if I have had any asks for the just the multiple bit like the multiple having more than one and then having the failure policy. I. Don't think I've necessarily heard too much about that. I have heard about I want to be able to configure my own web hook on your cluster.

D

E

Dude, um okay, uh I see you're typing right now,.

F

That's a good question, Tim uh so um being able to have some indication in the config like only route, this type of request to this web hook, um where what that condition can consider is like open to debate. um That's an interesting thing to settle in the design.

D

For it, I kind of blocked it on it, yeah.

E

A

Who who's the Implement implementer here? Is it there.

D

F

Don't know yet we're still in the design phase so pinning down the design uh is.

A

Looking for one.

D

Yeah but I did talk to like I think last coupon I talked to like uh Joe about like could I reasonably use cell for filters here and he'd seemed pretty positive about that and to me I think if you're gonna have failed closed web Hooks and more than one of them, you really probably want to filter to make sure that it only invokes it where it makes sense, um because this isn't like admission, it's going to break your cluster in a really special sort of way.

F

um So I need to Circle back to the current author of the Holocaust I linked and see if they can actually work on this this release and if not, then we need to uh have an alternate person to keep driving the design board.

D

um We don't have to think about it too much here. Jordan is the whole discussion about being able to restrict to some golden Masters? Is that in scope or out of scope for this restrict to what restrict system colon Masters? Can a web hook reject that user or that group.

F

um I think where we left this four months ago, I wanted a structure that could accommodate like ordering of the built-in super user authorizer in the future.

F

But I wasn't gonna walk on it to begin um yeah like today, it's an implicit, it's implicitly at the front of the authorizers and whatever else you specify like node web hook, rbac whatever um you can specify the order uh and then the super user one shows up at the beginning um a structure in the config that says here's the order I want everything to run in, and here's where in where in that order, I want the super user one to go.

F

I I think it would make sense to have a structure that can accommodate that there's. A lot of assumptions built on super user always is first and can never be blocked. So we'd have to resolve those before we actually allowed a config that reordered that.

D

Yeah, like aggregated API, servers that just hard code. That group is allowed to do everything or.

F

Loopback API servers AP cyber API servers talking to themselves and assuming that they won't ever fail an authorization check stuff like that. Yeah.

E

D

uh See next one if we're happy to go on is the structured, not structured, authentication, Web book, it's it's.

E

A job or yvc right, not Authentication,.

A

Is this a prerequisite for this one.

D

Which one which one is what.

A

Multiple authorization is.

D

A

D

Those are unrelated.

I

D

um So I plan on working on this one, this coming release and probably I'll, probably drag the niche and elect, and someone else with me then yeah and I think some of the uh I'm forgetting his name. But a member of the community helped write up a bunch of tests um uh that test the existing oidc functionality so that I want to get that merged sort of early in the 1 28 cycle. So we can have the CI signal.

D

So if we start messing with this code, if we actually get the writing code, this time around um we're not going to break stuff, um so I think Jordan, like probably you and I, need to sort of like agree on the API surface sooner rather than later,.

F

Okay, how about if we queue up some time next week.

E

Yeah yeah sure we'll.

F

Like arm wrestle over what we make a prereq, um the uh yeah for for 128, honestly, if we settle on a design for this and got test coverage of the existing code in place like I'd, be pretty happy with that progress. Even if the implementation didn't quite start yet. But.

D

Max I see you're on the call. Were you were you going to be able to devote time to working on it or are you busy.

I

uh According to implementation, I think so I feel like I can say that I'm confident enough that I will be able to implement this I have time, but according to design, there are still many unaddressed questions, so that should be discussed.

D

Yeah I mean I I, think whatever we do, we have to just make sure that we don't risk the existing functionality.

D

So if we, if we, if we, if we write code that messes with the existing stuff, it needs to be really strongly gated and ideally we don't mess with the existing stuff, if at all possible, um but we can figure that out, but yeah I think that's the core thing right is like. If you have an alpha implementation, it doesn't have to be perfect or done. It is Alpha. That is what that means. Yeah.

I

I, remember that the uh with concept or something that worked already so- and this was a completely different authenticator, so not messing with other authenticators at all. So as yes,.

D

I think that was my initial plan, but then Jordan threw a wrench in there because he was like Mo make it so that you can like copy paste. Your existing CLI Flags into this new config and it'll just work exactly the same. You know like that's just.

E

D

Like it's hard to do in the sense of like purely do it and make sure it never diverges from the other thing unless the other thing and this thing in the same code path.

D

I

D

um But yeah so that that's the hard part right is like I had basically thought that we would just copy paste the existing authenticator and just start ripping it apart and making it do what we wanted. But that's really hard to do if we have to maintain perfect feature parity with the other one or behavioral parody, at least um so, but I understand where you're coming from on that Jordan I I. Just that wasn't sort of what was on my mind but I.

D

It sounds like very much traditional Jordan move, slow, don't ever break anything and assume.

F

Everything I just assume we're going to be maintaining both of these forever and so having two that diverge in subtle ways, with not a real migration path from one to the other sounds like a nightmare where users actually and for us yeah.

D

That's fair, it's very fair.

E

E

D

uh So fine-grained odd, Z I see David's comment that says: Daniel's not gonna work on this, the 120.

A

I'm, sorry for the previous one, the review is, is it Jordan.

F

And Moe: uh okay: let's.

E

Put Mo first, okay, it's fine! It's fine!.

D

um Yeah you can do that.

D

uh Yeah I, think that one is fine, I I, think we'll. If we Max are you gonna, be a kubecon by any chance.

I

D

Oh, you are okay, that's excellent! So that way we uh all right, I will I'll find you then, and.

F

Okay, again I'm, uh you were commenting on the multiple authorization of my books. Would you be interested in helping review some of that as well.

G

Sorry was that to me, yeah.

F

Sorry um for the mult, the authorization config, you had asked a question about the condition stuff I.

G

F

Want to help reviews.

G

um I'll be around for a couple weeks and can help review um until August, so uh if this land, if this doesn't land in the 128 cycle, I'd love to be involved, but I'm, probably not going to be around for much of 128., okay, I I'm, sorry I totally forgot. First.

F

All right, uh the fine green off Z1 I'm happy to uh drop that out.

D

I mean I want it. It's super cool.

F

uh It is, we have a lot of work to do. I'm, I'm, okay, with reordering some things.

D

Yeah, um let's see I know Rob is going to be at kubecon, so we could talk to him about reference Grant and see how we want to move forward with that I think we had quite a lot of open questions about how.

E

D

F

um One of the one of the reasons I actually want to like try to wrap up some of the in-flight stuff is to free up bandwidth to think about sort of bigger system.

F

Questions like the reference Grant one um I think the reference Grant proposal is like half of a solution, maybe less than half, maybe a third of a solution.

F

It's like expressing a third of the intent, but then it relies on sort of controller implementations to do things and then actual authorization grants to those controllers to do things and I'm I wish we had time or attention to actually develop the whole solution for that space, where you can express intent and it actually gives the authorization to the controller and the controller has to make requests indicating I am doing this for this resource, and then the system says you: the controller can act for this resource and that resource was granted.

F

You know with this access by this object, like those are kind of the three pieces and the reference Grant proposal is like a third of that and I think, what's actually going to happen, for the other two thirds is, like controllers, are going to get super broad permissions and maybe use our library to check those permissions, or maybe just do things and execute confused Deputy attacks.

F

um So I think this is solving a real problem, but I I would at least like to think about whether we can actually solve the whole problem, which will require more work but Rob's. Not here um I think we should probably talk with Rob and Nick I think was the one who opened the housing.

D

Yeah, maybe we can, we should I, know Rob's gonna be a cubecon, so we should at least try to talk to him some in person and see if we can come up with something.

D

I generally agree with what you said: I'm just a little worried if we try to build the other two-thirds of it, it'll be like one of the largest kept cigars ever done.

D

Mostly because it was it's still like, gonna have some kind of out of tree, something like something's gonna, the things that are going to use. This are not in core, so we can't like enforce it. They behave in same ways.

F

uh um Did you put the two cell ones on or maybe you put the magic conditions.

G

Yeah I took both of those there I wasn't sure if they should be on this list, since they're kind of co-owned with API Machinery, um but they have some crossover to sagoth. So I just raise awareness that these are both planned to go to Beta in 128.

G

um For cell admission, I'd probably put Joe Beth's as the lead um for implementation, but there's a whole team of folks collaborating around that.

G

Not sure who's on point for review.

F

Okay, um I'm, mostly counting on the API Machinery folks, find reviewers, uh but yeah thanks for throwing this on on our radar.

D

Yeah I think a lot of people are really excited for the sales engine stuff. uh It makes like an entire class of problems to disappear.

G

um For match conditions, um I talked to uh Igor's who stole McCall about.

J

G

uh Driving this one um I'm gonna, try and get uh PR out later this week for um just to update the cap for beta.

G

I think the other one other thing to note about this is uh it's dependent on Cell admission control going to Beta? So if it looks like that's not going to happen in 128, then um that'll block this.

A

I'm, sorry, what did you say was the implementer for this.

G

uh Igor Village.

A

um Can you add the name later.

B

Yeah I can I can put myself in there of a tough one.

A

J

A

What's next Mo, did you want to do the crypto stuff.

D

Yeah we can, um if we're done with kept stuff, I'll try to make it not too long. You want to open up the dock. um Okay, so if folks haven't read this thing yet this is going to be kind of hard to follow along, but I'll do my best.

D

um So these diagrams describe sort of um how KMS v2's crypto stuff works today in 127..

D

um So obviously the part where the remote KMS encrypts a data, encryption key is sort of unchanged, um but currently on Startup and on key rotation. We do some fancy, non's calculation and uh sorry I'm, saying that wrong. When we use our data encryption key that has been encrypted by the KMS plugin every time we do an encryption with it. We generate a new nonce. The nonce is both partially random and it has a counter. We are limited to 12 bytes, because AES GCM is annoying, um so that's kind of how it works today.

D

um So there was concerns around basically anything that could cause the counter to be reset in a way that it's unexpected or reused in the way, that's unexpected, primarily around any kind of technology that would restore the process to an earlier State and then have it run from that earlier point in time, um then there's uh I've been reading up on sort of various guidelines around how long you should use a key for basically how like how much data you should encrypt with it and how many encryption operations you should do and all that's based on like the size of the messages and all sorts of other stuff.

D

But if you look at the most conservative estimates, um they end up being on the order of like a couple of hundred gigabytes right. So like you, you can actually very easily hit this limit on a high right load, cluster right, um so I wanted to try to basically address.

D

Let me step back for a second canvas. V1 had none of these issues because it always made a new deck. Every time it was going to do an encryption, so it was always a new deck, um so it had this horrible, like um failure mode where it always needed the remote plugin to be 100 available for every single call, but it didn't have any like cryptographic shortfalls it just just and it had a purely random knots.

D

So I I wanted to come up with an approach that retains the existing minimal Reliance on the external KMS, um but restores more of the um capabilities that V1 had from the crypto standpoint, um while not sort of gaining the massive cash size problem that V1 also has so like. In order to function efficiently, you need like an effectively an unbounded debt cache uh because you just have one-to-one mapping.

D

um So this this. Basically this this describes a um scheme. That's vaguely called nonsense extension. um So it's a it's a mechanism for adding extra bytes that are used to derive sub Keys, which is it, for example. It's done by the I think it's called Cha-Cha forget the name. Now the Cha Cha say: first um Mike I see you on muted. Did you have thoughts.

C

Yeah I'm trying to go back. Your primary concern is reusing the nonce when we're using a counter for the nonce.

D

So they're the.

G

Non's concern is.

D

If you snapshot a VM and then restore it, but you don't do it in a way that guarantees that the original VM never stopped immediately at the snapshot right.

D

um Does that make sense like basically, this feature makes it so that you could write State into edcd?

D

That was at a particular counter state at like time, 10, but at time five is when you took the snapshot, and then you restored the VM at that snapshot. So it's going to go from five to ten again, so it's going to count back up right. So all of those things that encrypts further will have collisions.

D

Sorry they have the potential of collision because there's only the four bytes of randomness.

C

Yeah I mean like there are some. There are plenty of protocols that use counters for nonsense, including TLs. This isn't, like a you, know, new innovation here, um I.

C

Yeah I wonder how um realistic this concern is.

D

Yeah, so that one I'm not super sure about um the I, think the gist, though, is, is that, when it happens, it's completely catastrophic right, like is.

C

It two uh two things that right to disk.

D

Yes, two things that write the desk. So if you have a single nonce Collision, you immediately leak the like the authentication, key um and I. Think I think you only need like a handful of collisions to leak, the full key you have to do like more complex math, but yeah you end up losing the actual encryption key and because we're using the same deck for all of them. You basically lose the entire backup.

D

Right, like that's, the real issue is that even if this happens point zero, one percent of the time or zero zero one percent. In those cases, the whole system falls apart right. So right now the canvas V2 docs basically say do not use VMware stay restores with this feature, because we can't guarantee that it'll work exactly like you expect.

C

Yeah I guess I'm trying to think through a scenario: uh the full scenario: um okay, I understand the concern now I have to mull on it. But thank you.

D

So that that's just one of the concerns, the other concern is basically depending on who you ask. They give you very different numbers on how often you should keep using the deck.

D

um So I wanted to come up with something that didn't involve more remote calls to the KMS plugin, while giving us like more unique decks.

D

um So the design basically says we'll just use an hkdf with a salt that we store beside like a public salt that we store with the existing nonce um and off when I get around to actually making an actual PR for this I'll have all the links to the stuff where I looked at this stuff. But it the scheme is generally known as like a non-sustansian scheme. Right is that you're you're?

D

We can go back to having purely random nonsense because we no longer have to worry about collisions, um but your derived key is mathematically unrelated to the original key because of how like the hash works, but it's deterministic uh based on the random salt plus the original key.

E

D

It's kind of complicated and I, don't necessarily expect folks who, like rock it without spending some time thinking about it.

D

Also on Mike on the whole nonce counter is pretty common, I I, don't think most of those things wrecked stuff out in disk in a way that you can observe the Collision outside of like the protocol.

E

D

Also I think TLS does use a prf like in this way in a similar way. It doesn't use a like a full hkdf, but it does use a suit. A random function to derive keys.

C

To derive the keys, but then the sequence numbers for like asgcm are the nonsense for the TLs record. Numbers are the nonsense for the uh in the asgcm Cyber Suite right, and they don't even have that Randomness that we add, um because we are, you know adding that prefix to the knots, the only in your PR, the there's, some prefix that is still random. So.

D

But what I was trying to get to is the the pseudorandom function used in TLS gives you like derived keys right. So that's kind of I think where TLS gets his safety. Is it doesn't like it's not even if it has a non-collision, it's not likely to happen on the same derived. Key foreign.

D

That might be wrong. I haven't looked at how all the guts of TLS and the prf work there.

C

Well, I mean it's just the same as the deck right. The session key is like the deck here and it's how about we TLS.

J

C

Is open is how long we reuse the deck, which is three.

D

C

Right now, in the current implementation or whatever.

D

um Now we reuse the deck forever. It's only on the remote canvas plug-in being offline and not responding to us with its current key ID that we stopped using it.

C

Oh, when did that happen? That was not in the reviews.

G

C

Yeah, okay, we.

F

Changed the deck on when the key ID changes or when the API server process restarts.

C

All right, I I thought we still had the 10 minutes and the last revision that I reviewed there.

D

Is no TTL at all it's forever right.

F

So good that we're good that we're staying in beta and it sounds like we have conversations to have yeah. We are at time actually.

D

um Yeah I can put this on like maybe not next to God's agenda, because I, don't think I'll be here for that meeting, but like the one after that- and we can chat again or whatever.

D

um But I do want to come to like a decision on this in 128 like I, don't want uh um I, don't want to have that restriction and the scary messages on the kmsv thing forever like, um but that does not screen production ready.

A

Good thing: we're not going in GA, 110, ., okay, I, moved the rest of the topics to the following week. um So sorry, folks, we if we didn't have a chance to get to um all right, that's I think! That's it all right thanks! Everyone. Thank you.

I