Kubernetes WG KMS, 18 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: KMS BI-Weekly Meeting for 20220118

Description

KMS BI-Weekly Meeting for 20220118

A

All right, hey everyone. This is the kms meeting for january 18th 2022. This was the fourth kms meeting in the series, uh so we were just continuing the discussion on how we want to frame versioning and ga requirements and such for the kms cap, so uh trying to remember my train of thought initially exactly right now.

A

B

We were talking about the audit id whether or not we want to use that as a ga gate.

A

Yeah right so right now, the the the initial cap document does not have um graduation criteria, because it's not a kubernetes rest api and um and what we're proposing is basically just one minor, optional additional field that doesn't have to be persisted anywhere. You know it's just up to the kms to consume it if it wants to so.

B

That one's a little bit harder.

A

To frame sorry, go ahead.

B

So I had commented on damian's point about that. um I I think, there's having the audit id there is useful, um but given that there are requests that don't have the audit id generated, I failed like it's. In some cases this wouldn't be quite useful for what we need in terms of correlating.

B

So if that's the case, meaning it doesn't cover 100 of all the cases, then I I really want to make sure that is not part of the ga gate, because I mean I I guess like I don't even know if it makes sense to include it if it doesn't cover like all the cases where we need correlation.

B

I think it's a nice to have if you want to use audit id for this, but it doesn't cover all cases, so I I don't think we should include it. If.

C

I can ask something about that like ah by saying that it doesn't cover all the cases like. Do you mean that some of the operations since they are writing the cash? uh We don't have an audit id on the uh kms side or is it something different.

B

No, it's something different. um Let me pull up the. It was something that mike brought up right, um yeah, so.

A

Yeah so he's saying that you know as we um uh the api server when it starts up to fill up the watch cache we'll do a full list on all secrets and all all api resources right, and at that point it will cause it to cache data encryption keys in its storage layer right so later on. When users are actually getting those secrets, you won't see it.

A

Those events in kms at all right, so those are those are unknown and then that that list from the api server also itself doesn't necessarily have an audit id, because it's just the api server talking to itself.

A

So the the more I've thought about it, the less I feel like we should do anything with iid, because it will always be a maybe api.

B

A

And and to me, it might make more sense to pursue the audit annotations approach, which is to actually include the kms uid as an audit annotation within audit logs. So if an audit log is generated- and it happens to also have a kms event occur, then the audio event will tell you about that, so you can search for it if you want to, but but the the.

B

A

B

Not true yeah, the inverse yeah.

A

Because the inverse is not true, let's just don't worry.

B

It's not useful right, I mean we could try to add it, but it's not going to cover all our bases um and just to clarify like what mike said was initial load. Is not audit logged but does trigger kms calls right? Yes, so, in which case we will get.

C

The key the case because if the kms is involved like how come the audit log is not produced.

D

So not every request from api server to kms is initiated by a user request right. So when api server restarts, it basically has to warm.

B

D

Cache right, so it's just going to make calls just to update its cache and then that is not correlated with any request from the user.

B

So from that perspective, this new uid is way more useful because we can actually correlate them.

C

Right yeah that may be kind of a topic for and, as I said, but like how can requests that are made by the api server to let's say in our case it's a kms, but in some other cases it might be an aggregated api. um How come these requests? Are logged like what? What is the the actual? um Like reasoning for that, because to me whatever the operation is as long as it goes from one point to another, even if it's made by the api server, we would want to track it.

A

Okay, but but these are low-level storage problems right. These are not part of the kubernetes.

B

A

That's that's why? Right? This is the storage layer being the storage layer right. It doesn't go through the kubernetes api because it implements the kubernetes api yeah right right. um So that's kind of the just. We do try very hard in the kubernetes code base to go through the front door, which is to go through the kubernetes rest, but you can't do that everywhere, because you can't implement the rest api without talking to storage directly at some points right.

A

um So I think that that's the general gist of it, but, for example, we would never talk to an aggregated api through some back door, because we wouldn't know where it was.

B

A

Know of the service to talk to um do you make sense.

C

Yeah he was level, but that's not the case.

A

Right um so I I don't know if we want to like if we want to have the audit log, have an audit annotation at some point with the kms uid.

A

That would need to probably be in this kept somewhere right as a as a thing that happens at some point in the future. I guess it doesn't have to be like initial implementation, whatever right, but it's probably worth calling out to see if people.

A

If people are okay with that as a future step right.

D

Yeah, I think that that's where I was a little confused. Should we add it? If we still don't know if we're gonna use it or can we add it at a later stage like so this part focuses on what we are starting with and then can we keep updating the kept with newer things before we say it's completely done so like at some point after investigation. We think the audit id annotation is something we can do.

D

Then at that point, do you think we can add it to the kep as a pi and then just implement it.

A

A

I would want that to happen, but my experience with the enhancements repo has told me that that's not how the enhancements needs and others want this to work, because, for example, with the addition to the csr api, I just tried to update the existing certificate signing request with the details of it right because it was sort of all there and I wanted to keep that kept as a complete reference of like well. This is what the csr api is and they're like nothing. This this cap is implemented. This kept is done.

A

You have to make a different kept to talk about your effectively your five line change today right. It has to be completely isolated and tracked on its own, and I was like okay right um and you know I got like, like. I remember mike being like really that you have to do that much work to like do this tiny thing I was like.

A

Yes, the cap is significantly more work than the code, um so I I I would be very hesitant to say that it would be okay to say, here's, here's a cap and like we did the uid stuff and that work is done now. We can just go updated to add more stuff about what we now want to do with this uid and because I think it's very hard for them to track right.

A

Like the idea of a cap that doesn't go through the phases, I think immediately will raise a bunch of red flags and be like what do you mean.

D

uh Yeah the way I saw the cap was it's like a design implementation noted down and like basically discussed before we do it and that's why, like it, doesn't fall into any of those categories of graduation and then, if we add something and don't do it, uh are we allowed to take it out later, like let's say whatever we're investigating turns out that it's not possible to implement then at that point, even if it's implement, we can't change it to implement it, but are we allowed to remove it from the cap.

A

I mean I so I certainly like, if, for like, the client go credentials. Plugin like uh when I took that kept over, I you know I rewrote significant chunks of it, because what had been implemented significantly diverse from what was in that cave, um and so you know I was like, like there's no point in this kept existing in the state. It's just confusing because it says one thing, but the code does another thing, um so I think that part is okay.

A

I didn't really get pushback on that, it's more of taking something that was considered done and then saying I'm going to add stuff to it. Now it's not done anymore and thus needs to be tracked, is a little more like like like.

A

If we were going, if we're, if we're gonna go from a document that doesn't have any phrases to a document that does, I suspect, they're gonna ask us to make two separate documents like one with the phases that is going to go through it with its own criteria and one that was just a point in time like kms uid kept. That just said, we would like to add this.

A

Uid: here's why here's, how we're going to guarantee all the things but like we're just going to do it, and it's going to be done and that's it like that. That document is just sort of done. Then.

D

Okay, quick questions right, so if we don't move it to implement it, then are we allowed to update the existing cap with newer things that we're adding? Because even now, if we add that in the future we're looking at audit id, that means we cannot move the kept implemented until we do that right because we said in the cab that we're going to do it.

D

So instead can we keep the cap in the provisioning state or implementable state and then like keep updating it and keep adding new stuff and then implement it, and then once we feel observability as a story is done, can we change it to implement it.

A

I I think the answer is yes, but I I think the weird part there is we need to have like, like what we're basically saying is we want to keep working on the cap and defining what it means for observability be done, but the way the enhancements repo is framed, as you define everything you plan on doing at once, and then assign alpha beta and ga requirements of all of that right and that's where I think the disconnect is, I think the re, the process for the enhancement people expects you to know everything you want to do and you're allowed to make mistakes and fix things and that's fine, but but you can't, I don't think you can change it as much as I think we're trying to change it right like we're saying that this is a piece like like.

A

Basically, what I think I'm saying is instead of having an observability cap, we might just have like two or more tiny caps related to kms observability, and I think that fits within the mold or what the repo wants or the process wants us to do.

D

Yeah yeah yeah. That makes sense, because if in the future, if you wanted to audit id annotation, if you want to add, implement details and motivation and all that it just turns out to be a whole new camp.

A

Okay, because it's basically unrelated right, it's like hey, there's this cool uid, but I want to show up somewhere else. Here's why.

D

A

So I mean, I think my my thought process would be. Is we we certainly want like, for example, clayton's, like the only approver for encryption at rest left that still works on kubernetes, uh so we would certainly want like- and you know, clayton's an api reviewer too. So we can. We can ask for his feedback on like how we should frame any of this and same with, like jordan or mike. If anyone has opinions, um I think I would be okay with just saying that this cap will go from like when it emerges.

A

It will be implementable and then, once the work is done like we're going to enumerate all the work related to this uid and we're just going to say it's implemented and that's like this kept will then just be done and the next time we want to make a change or next time we want to make an enhancement related to kms observability, we'll just make a new kept. That defines all the things that we want to do there.

A

I think this one should be relatively smaller than right, like really or I think it is small. When I was scanning it seemed pretty small. um The other thing I wanted to ask more related to implementation um is: uh did anish? Did you go into details about how we're going to guarantee that this uid is available in, like the kubernetes blogs.

D

uh I mentioned that we will log it in the api server as part of the changes that we are making. So I said for every request that we are logging for kms and also additional logs, that we add, we will add uid as a key to the log.

A

um I don't, I don't think that answer is kind of what I want, which is: how do we guarantee that this, uh like like? Basically, how do we guarantee that this uh persists over time right like over changes like as we as we go work on other other things right, like related to kms or whatever? How do we guarantee that those things have this.

A

Like the closest I can think of is so from the perspective of the stories api um or the internals of the storage api. This is represented as a transformer um and I think the transformer takes in a context like not not a go context but like a encryption context, and I think what we might want to do is put the uid in there and then write a wrapper implementation of the transformer whose entire purpose is to always make sure that that uid gets locked right along with anything else.

A

That we think is valuable as context but like I think, if we did it that way, we have a pretty strong guarantee because we're just doing it from the outside of the transformer layer. Just saying like this is going to happen. It doesn't matter if the rest of the layers um uh remember to do that as long as they log the error itself or whatever it's going to be in there.

D

Yeah, I can add that detail.

D

Yeah, I didn't go too specific into the implementation. I suggested, where we're making the optional fields and all of those, but I didn't go into detail on how we are implementing it, but I can add the transformer stuff there.

A

Yeah, I think we, um I think we need at least a little bit more right, um yeah. So right now, I'm I'm looking at it. The transformer interface has transformed transformed from storage and transformed to storage, and uh you know to take the data in and then the second argument is a context which today only has authenticated data, which I believe is the path to that cd.

A

So we could expand that to include whatever metadata that we want and then have like some a wrapper implementation that guarantees that it's logged appropriately on failures.

D

Yeah and then probably also success, right, like just at a higher level, must be.

A

Yeah yeah probably significantly higher, but yes yeah. I guess errors could always be logged in. I don't know whatever v makes sense, a six or eight, probably something like that.

D

But I can add that detail in the cap.

A

I think once we have that, uh I know I'm assuming you know, we've enumerated our motivations and stuff, which I think are pretty straightforward. You know, I see a thing in kms. I don't want to sit there and look at timestamps inside of api server logs and guess uh you know.

A

A

I think I, when I scanned it, I think I had seen something about uh like the name of the object or something like that somewhere in the cap. um So I for the grpc api, all we're saying is we're going to pass in the uid right, so I think that's totally cool. I think that fits within the the intent of that api. Just basically give the plugin the least amount of data possible for it to do stuff.

A

um I think we're fine if we want to log more stuff on the api server like the name of a secret and stuff, because the api server logs already contain all sorts of stuff like that, I don't think that's a big deal. Yeah.

D

Yeah, I think that's what I added I said like api, so we can log with some additional metadata that it has.

A

Yeah, so it would be good to enumerate what we want right. So I think what we want is uh group version, resource and name right right. Group version resource allows you to get a full location, and then the name pinpoints that location to a particular item right, um yeah and- and these are all on individual items right. So that would be enough right. You could say this secret in this. uh Oh sorry, you need the name space as well name.

D

A

So, do you need five things or six things right, the gvr namespace name and the uid.

A

So but yeah all that would be good to enumerate because then it would be clear to like clayton and jordan, and whoever else wants to review this that like and also, I think we have cigars tomorrow, right yeah um so, and so I think we have enough like we can talk about it tomorrow, the specific details and maybe even bring up the the oddities around fitting this within the enhancements reaper or or we can just say that we we want to just basically have this this bit of the kept just go.

A

Ga as it's implemented, because all it is is just adding this thing uh and see what people think and if people think no that's weird or unexpected, then we basically have to add a feature gate that says like. If you opt into this behavior gonna be very strange but um yeah. It is what it is.

D

Well I'll add these uh today, so it'll be up for review again. um So hopefully we can all do one more round of review before special call. Tomorrow.

A

I've been talking a lot. There's other folks have comments.

E

Do you want to talk uh also about other topics like the performance topics, and would you just like to focus on okay? I see it out of nothing. Okay,.

A

uh I I I'm totally cool with that. Are we done talking about observability just want to make sure I I don't have anything else to say this morning.

B

If we just focus on uid yeah,.

D

Yeah, I think we have long-term action items and also like the immediate ones for the cap, so I think that's a good place for us to transition to performance. Now.

C

Just one thing I'm wondering maybe, um since the main focus will be the the id with the log, would it make sense to move the addition of matrix as a library to beta so that at least like we have one focus for this release and then for the next one for like yeah for the graduation, then we have another focus.

A

I don't think I fully understood what you meant by beta.

C

um So like there will be different graduation phases, as we've seen, um and you mentioned that we like at first, we didn't really know where to put the audit id if it ever was like supposed to lived in this particular cap um and since, like we wanted to add matrix of the library like, should we still consider to do that for this release, or should we push that to another one.

A

uh I mean what I'm understanding from our framing is that it would be a different camp. Okay,.

B

Or or damian, are you more asking like once the uid is there? How is it actually going to be used? No.

C

No, no I'm more saying that as part of the observability improvements. One thing we wanted to do is introduce a library for like to provide generic metrics in both parameters in open, telemetry format that an implementation could use, and I'm wondering like if it's part of the scope of this particular cap.

C

If we should even consider it like to be done uh at some point later or if it should be done in another.

B

Right so like the reference implementation.

C

B

Yeah, that's what I meant by how it's actually going to be used, because we we're adding it right. But how is actually going to be surface is that where the metric stuff, the library comes in, that's a good question.

B

Do we want to commit ourselves to say the reference and implementation is part of the cap? Graduation.

C

I don't think so. I I like, I don't even think we need a cap for that because, like at the end of the day, we just introduce uh kind of an implementation which, like we think, is a reference but like nobody has to fully implement it or even like they can like. They don't need to abide to our standards.

C

So maybe we don't even need a cap for that.

A

uh You I think we might still need at least some part of a kept, because it would become like a sega, subproject um sort of be within our example repos and stuff. So um yeah. I I think it would probably be part of a larger cap related to.

A

Some aspect of kms, whatever we want to make it part of- or it might be like, like a like a mini cap on the side related to other caps, basically, which is like you know. If we have like a performance one and an observability one, both of them could have beta requirements. Let's say that, like you need that implementation to be done right, like like. Basically, it's like a dependent of that.

C

Yeah yeah this would make sense. The the only problem I have with like how we are proceeding is that it's something I mentioned on the cab, but we are basically trying to solve how we can correlate the data once like we've noticed a problem, um but we have currently like no ways to actually detect the an operation failed or I something happens like we. Don't have the metrics that surface this information today, um I'm not sure how users are currently able to notice issues, um but as far as I can tell like, it lacks information.

D

Yeah, I think, right now it's very provider specific like, for instance, for azure, like we have metrics around it right then kms, operation, space and I'm assuming, like other providers, also do the same where they generate metrics for failures and encrypted cooperation.

D

But maybe we can have metrics on the api servers to like aggregate these and say like hey, look, encrypt calls are failing, or do you call the failing.

C

Yeah, that could be a like a good thing to tackle for this particular because at least we'll have like something shared by all the implementation that would surface issues.

C

um Let's say like within with not that much information, because you can't really know like what exactly happened, but you at least aware that a decrypt or an encrypt operation failed. So that can be surfaced as a metric um and then have maybe another effort that would provide a reference library to like provide matrix, but on the kms provider side, not on the api server itself.

A

I don't have a strong opinion. I.

A

I I don't think it's a bad idea to say that this uh captured, like within the production, readiness uh review questions there is stuff about like well. How does how does the cluster admin or the administrator observe the system is working, and so we're improving that by saying that we're gonna have um better logging right but, like it explicitly says in the prr questions, that logging is like the absolute last thing. Can you like come up with something slightly better if right say um so yeah?

A

So if we want to have metrics, we could we could define those within the cap. Metrics are weird because they only exist at two stages: alpha and ga. There's no like middle yeah yeah. That.

C

That that will change, hopefully in the future, but yeah.

A

Yeah, so don't play with it, because because people think they can remove the alpha ones that have been sitting around for like five years, I was like no, you can't it's it's effective.

C

Trying to duplicate them in the case yeah. Okay, I understand.

C

Let's see I can I can help with that. If you I can provide you some metrics that could be useful tomorrow and if you want.

A

A niche any of y'all have any thoughts about adding stuff to the cap. Related metrics.

D

I think if we can define uh with what metrics we want to add, uh maybe we can do that. That's fine, like damien! If you're interested, you can actually just open a pr.

B

D

Kep branch and then we can just merge that and that will show up, but if we don't have it, I think that is also fine. I'm just saying like we don't have to block on it. If we don't have it, we can always add this part of the reference implementation. We can say like hey, we are introducing these metrics in kms and then we are also providing a ways, so you can consume it in your kms plug-in with the reference implementation.

C

Sounds good, I don't have anything else on the observability side.

A

uh So I think we still got 15 minutes or so. If you want to talk about performance stuff um doesn't want to leave that and have strong opinions and dumps.

A

uh If anyone has the doc, I didn't want to share this screen. I don't have it right now and my computer is in a weird state because I've been out of office, but I can make folks co-host if anyone wants to share.

B

uh I could share if no one else wants all right.

D

Let me I can try to share.

E

I'm on linux, I always try to avoid sharing anything as it usually causes a lot of trouble.

A

D

Your team is green.

D

Okay, yeah, I think in performance like there were a couple of smaller topics that we had written up right so like the first one was basically cache warm-up today. If the api server restarts, the thing is because it makes a list api call.

D

It makes a single call encrypt call for decrypt call for every single secret that it has and it wants to update the cache it's um calling sequentially and then this means there's a burst of calls right and then, at this point your kms will hit a rate limit and then because api server keeps retrying like uh your.

D

Your kms goes into like some kind of a jail where it cannot recover or basically every call is failing and then when api server cannot warm up its cash, because these operations are failing, it will fail all the health check and all that and then it will just keep restarting so, like. I think one. The first one was that use case where how do we optimize this scenario? Because if you set a really large size for the cache size, the api server when the api server is running?

D

It's really helpful because not all calls are made to kms, but if, by chance your api server restarts then you're basically like hit like now, you cannot recover at all and then I think it also becomes worse. When you have more, I mean if you have multi-master scenarios so like if you have aggregated api. So if you have three replicas of the api server, then each one is trying to warm up its caption.

A

Yeah, so I I think today the recommended approach for like larger environments with a lot of encrypted data through a kms, is to explicitly configure your api server health check with the parameters that tell it not to care about the kms part, so, basically check check. All the other health checks skip that one, because it's just like just this is not worth it, um which obviously is silly right like the point of the health check is, is the system healthy and you're like no? No, I just want, like mostly healthy. I think I'll just take.

A

Take it semi-functional, that's fine! um So I so I think some open questions around this are so today. You know we have that one-to-one mapping between data, encryption, key and uh object and std.

A

One, the I think, the main concern I had around some of the changes that have been proposed in the past, with like using the key for some amount of time or other other variations of basically less.

A

The less data encryption keys is what it comes down to is that um today, if you're on the stories, migration right, which is uh you know, part of the you know, we have a sig api machinery subproject, which is the storage migrator um and just the general concept of storage migration right, which is basically read all the data from the kubernetes api and post it back to the kubernetes api, completely unmodified for all objects right and then look at the resource version of all the objects that are returned for the objects that have the same resource version.

A

You know that they were already up to date with the latest and greatest and with the ones that do have a resource version changed. You know that they needed to be updated and they were, but the gist of it is that is the process that sort of has to work correctly. For you to rotate your keys right, and it does today right, if you do that, if you follow those steps today and you do the rotation dance, everything works, fine.

A

The the approaches I've seen where, like a deck, is reused for some time um had failure modes where they would be like yeah. No, it's everything is up to date, but it hadn't actually done the rotation because it had cached the deck for more some amount of time. I'm trying to remember exactly the flow it's been so long since I looked at that code that uh it's a little flaky in my head right now,.

D

Yeah, I think, like some of the different options proposed as one was deck reuse right like use it for a certain period of time and then like if for five minutes and then after that regenerate a new deck and then start using that for encryption. And then I think there was also a option where someone suggested you can use a deck per namespace.

D

uh So that was the second one and then the third one was rather than duration, use a certain deck for a certain number of operations. So, like you said, say, one deck is used for thousand encryptions and then once it hits that, then you basically use another deck for the next thousand and so on. Right so, like I think most of it was around reusing it, but how to reuse. The context was just like there were different proposals, name space duration, counter.

A

uh And this could you bring up the kubernetes repo, so if we could try to find some of the old pr's it might uh that you might actually already have it then um yeah, so that if we scroll down, I think we had some good discussions here somewhere.

D

Actually, I think that might be in the issue. I don't think there was an issue with link that.

D

This is also pr yeah. I think this was the one issue where a lot of things were discussed around deck, reuse,.

A

I, by the way, I've actually never understood what mike was trying to say with this diagram.

A

D

Not saying there's anything.

A

Wrong with what he is I just, I could never actually figure out what he was trying to get me to understand.

D

Yeah, the only thing I understood was you: there is a deck that is generates a local kick and uses that for a period of time. So this is almost like bringing the encryption key from kms to keeping it in memory or something like that, so that, like it, never has to worry about external calls.

E

So this is something I mentioned in the document too, so um I was a little bit.

B

E

Cautious with reusing the keys and then I kind of jokingly, said well if we would use the deck to encrypt other keys, so it would kind of be a local keck. uh This would be most probably fine, so I think this was just a diagram to show that you have a layout. This is remote kms. Then you have a local kms and then you, you have to delete that encryption. Keys.

A

Yeah and I think what it gets down to is like if, if I do a storage migration, I I have to be able to tell the system somehow it cannot use the local kms. It has to go all the way down to the real kms, get it to do work associated with that to make sure everything is up to date, with the latest foundation, sort of the foundational keys that are being used to encrypt everything, and- and that's where you get but like like.

A

Basically we're hit we're we're hitting one of the canonical problems of computer science right, which is cash and validation. Right like how do you know when the right time to validate your cashless, so everyone is up to date, and um uh I don't know what I I think that's my head at the bottom of this screen right now, just wrote down this okay. That was me.

A

I think talking about using citadel to play around with stuff too, uh because one of the things that I think still was never um defined is what is the actual cost of running a kms like what what is like the actual like latency and performance impact of just a kms right and citadel was like a perfect way to test that, because all it did was only ever have a local key encryption key that he held in memory, so it never made any network calls.

A

So the only thing that you were measuring is if you have an api server and you have citadel co-located with it running a unix domain socket. What is the overhead of talking over that unix domain socket?

A

I don't think we ever actually.

A

Found those numbers.

A

Because I think alex left google, he said thus no one really picked up that work.

A

I think that would still be valuable for us to understand what the cost of using a kms is, because today we don't actually tell anyone right, like we say, kms is the best approach, but one we don't define all the failure modes, but we also don't even tell you if everything is working perfect.

A

What is what is the actual cost of using one.

D

Yeah, I think two aspects of it right. One is the round trip time so basically like how long the operation takes and then also the number of calls it generates, like 100k cash size, I think with cloud providers, it's just if you actually have 100 000 secrets. I think it's it's really hard to actually restart your api server and like not have things crashing or hitting rate limits unless the kms plugin is keeping the encryption key in memory like if it's making network calls yeah, I think it's just.

D

It doesn't scale at all.

D

And I that is where, if we reuse decks in like some acceptable way, then at least like we can reduce that by a certain factor like if we re use the same deck for like, let's say thousand secrets, then for 100 000 secrets. We still would only make 100 calls.

A

So here's a question right like since we've been talking about our reference implementation and I realize we only have a few minutes left.

A

If we had a kms implementation that internally generated a key encryption key like locally, you know, went ahead and told the real the grpc api behind it, the like azure key vault or whatever hey. I need you to store this for me and like. Let me let me get it back after some point right and then it was then going to locally use. It right like like basically is this a problem that can be solved with some amount of key hierarchy within the kms itself right.

A

So it's even if we did, if even if you left the data encryption key purely one to one as it is today, if the kms itself, when it did the encryption stored, some metadata by like hey, I used this key encryption key.

A

um Let me you know, let me fetch that back. Let me decrypt like let me make one decryption call to decrypt it back into its full state. Now I can decrypt all data encryption keys that are being sent over the grpc api.

A

With this one locally, I don't have to make any network calls right and then it sort of becomes a kms policy decision. Then right like how? How much are you willing to use that key encryption key right? So you can then basically say well, I don't know I can use it for some amount of time. I can use it for a certain amount of operations, but I think that does significantly reduce the problem, though right like if the, if that capabilities in the kms or am I am, I misunderstanding, I'm missing some scenario.

C

I would say that the only thing that we wouldn't have to tackle if we have a local kkk would be to like.

C

Today the users have control of the kkk if they want to create a new one or to do a rotation every like seven days, they can do that on their cloud provider, um but if we have a local kk, we'll need to define like how like how do we change it? um So we will have to communicate like have a way to notice that the kkk on the server side change for us to change the local key.

C

I think that's the bare minimum that we should do um to leave the users kind of under the control of the key.

B

um I also want to raise that the reason why people want to use a remote kms is because it comes with all the industry standards and all the policies and all the uh compliance stuff right. So one now that we're starting to have a local one. I don't know what that means, for compliance and and security concerns go ahead.

A

um So is that any different than the fact that we already have a hierarchy with the data encryption keys right?

A

We, the api server, generate keys in a way that the kms cannot influence and we use an algorithm that the kms cannot influence and a storage format that the the cameras cannot influence, and then we just hand you the encryption key and say you have to store this. Give it back to me in a format that I can hold on to that's opaque right.

A

Is it any different than that? Because, yes, you have your industry standard kms that might be using like quantum crypt or whatever, but it's still limited, I mean kristoff is laughing, but I'm sure like. Maybe it is maybe he's doing something cool or something really fancy.

B

Well, the hsn stuff right, the hardware yeah right.

A

Yeah, it's using a secure enclave, hsn.

B

A

These are all very important and good things, but then what's actually left us, the weakest link is the data encryption keys that we are generating in memory using go code not using an hsn and then using them for the actual true encryption right so like. If you were an attacker.

B

A

B

Right now now, this local one is is, it is, is it uh on disk or in memory.

A

No, it would only be it would be generated in memory like so imagine the kms it starts up, it generates one in memory and it does one kms call telling the backing the real kms hey. I need you to encrypt this for me and um and give me the encrypted blob right. So that way it knows what the encrypted payload is right, and so, when, when data comes in from the kubernetes api server, it can figure out like all right like how did I encrypt this? Do I have that key now or I do it?

A

Can I go back and get it from the the real kms right? It's basically one level of indirection within itself, but it's always in memory right, so it should always be able to come up with no state.

A

Come back up, generate a new key and be ready to do encryption, but when it comes time to decryption, it has to basically warm up its cache using the real kms. But the idea there would be is that cache could be over many many data encryption keys from the kubernetes api server right, whatever the policy is whatever is. Okay for that plugin to do, but it's just one level of hierarchy, separated right, it's it's, but it's within the control of the kms plugin like basically today.

A

What we're saying is the kms plugin is just a pure conduit to the real kms, and I'm saying it doesn't have to be that way.

A

If we explore that as a potential choice, then like I, um I want to be careful with the decisions we make in the kubernetes api server because we make them for everyone and we and you don't really get a choice right one, because today you don't get a choice about how we use data encryption keys in the format right, that's not really great and that's actually, where some of the pinpoint comes in is like. I have a bazillion secrets and they're one to one and I'm calling as a niche mentioned.

A

If you make 100k incred decryption calls to your kms something's not going to work uh right, and maybe the kms plug-in can actually take a stronger role in making that right, and I would feel much more confident in saying such a statement if there was a reference implementation that did the thing that we were saying instead of like hand waving it.

B

I was gonna say this is not like. You should add this to the doc as in another option, I don't think I saw it.

F

This is new, so definitely.

B

Here, like over.

E

E

Definitely a local chems is a huge advantage um when we have maybe just one keck or we have very few calls. But this is quite a big implementation and I was curious. Why don't you have such some batch calls for? Okay, so, okay, I have a question for not k, keys, send them all at once to the kms and ask it to to decrypt or encrypt it.

A

That is interesting. um Maybe initially I can answer. Could such a is such a capability present in like azure key vault? Could I give it a bunch of a bunch of data at one time with maybe some schema for to figure out like which bits are like you know like the delimiter or, however, like the encrypted data is separated, and could it decrypt all of them and hand them all back to me. I don't actually have no idea.

D

Yeah, as far as I know, you cannot do that. Like I mean you give it something like you give it the plain text and it'll just encrypt the whole thing, but you cannot give it like a bunch of data and say: can you just do all of the encryption at once and give me back the encrypted data.

A

Well, I think sorry right.

E

Yeah, but could we just you know, um send them like okay? This is so like, so you as the kms plugin would set some id and behind it the thing that we want to to encrypt, and then they, then the cloud provider would just encrypt it and return. Also like kind of a json object where you have the id that we specified and then behind that we have the encrypted path. So we could equip several things at once and not have a single calls.

D

Yeah, I don't think it does that.

B

D

Also, it's highly.

B

Dependent on the cloud provider right like it, I would I wouldn't imagine, even if you had a call like that, an endpoint that supports that ultimately, you'll still probably get throttled, there's always going to be a limit.

A

Yeah, I mean, I think, a high level, though I think kristoff what you were describing is actually kind of similar to what we were just talking about right, we're basically trying to build a hierarchy. Somehow it's basically: where does the hierarchy live? Does the hierarchy live within the true kms implementation, or does it live within the cms plug-in, and I think the problem is kubernetes is trying to make a extension point.

A

That has to be basically the least common denominator of all the cloud providers and like pkcs11 and the other encryption standards right like everyone needs to be able to implement this grpc api, because that's that's sort of what we want to happen.

E

Yeah, but you know, we also specified the initial api right where we said: okay, we send you and we have two options and crypto decrypt, and could we just specify okay? This is the new api where you can encrypt or decrypt a bunch of stuff and then ask the clock process to implement it. Now.

A

So I mean so certainly the grpc api could be expanded to have like a bulk encryption, really really bulk encryption. I don't think matters. I think it's bulk decryption. We care about, like your rate, limited like the kubernetes api rate limits, how fast clients can talk to it. So I don't really think that's the case here. It's like when I start up and I have to read the whole world. I need that to work and I need to work every single time right.

A

In that case, if you had a bulk decryption capability, I think that solves the problem. I just don't think any kms that we currently have energy like that is in use today supports such a capability, where you send it not just a cipher text, but a series of cipher text at once and ask it to do all of the decryptions and then send all of the decrypted payloads at once. I just don't think that api exists.

E

Yes, so this would be so obviously your solution that you propose to settle and having a local chems is obviously better, but for I don't know if if it would be a huge implementation effort, I just want to just ask if it wouldn't be just easier to change the api and say okay now we would like to also batch encryption batch decryption things and before we close out one thing, that's super important.

E

We should always have boundary conditions what we allow the users to do. For example, um I looked up that you shouldn't use the gc when you're using asg sim use it more than two million times and for cbc I tried to find it, but it was more esoteric. So there was no no concrete hard number uh which you should take to orient it. So I would be worried that the customer says oh it.

E

I need to be phipps compliant and it's enough when it's once a year or something like that, and then um they would wonder that that that the database got hacked and that there's somebody catching it.

A

Right, I think, kristoff what you're saying is no matter what we have to make sure that, if we're using something like gcm or whatever encryption mode, we're using we stay, we don't let the user have an option where they can go out out of the safe bounds of that mode. No matter what we guarantee that, and I totally agree with that. Yeah.

A

Cool, I have to drop and thank you all for the discussion.

C

D