Kubernetes WG KMS, 15 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WG KMS Bi-Weekly Meeting for 20220315

Description

WG KMS Bi-Weekly Meeting for 20220315

A

Hey everyone: this is the cigot meeting for march 15 2022, uh specifically the kms meeting number 10.

A

um Do you folks have updates uh I've seen a question from you um kristoff in regards to some of the api semantics of kms?

A

Did that make sense.

B

Yes sure um so um so my current implementation was under an assumption, which is wrong, um which I can see from the source code that the kms plugin also does the encryption, um but it does not. So this is something that I need to change um but anyway, so my question was: um if we introduce a aes gcm just to the kms plugin, it's just half the way off of the solution right.

B

So it's pretty nice that we have stronger and authenticated um encryption for the decks, but we would also need to um add asgcm then to the envelope logic in the api server right, yeah.

A

So in the discussions we've had so far that that ladder piece, the envelope logic and the kubernetes api server is what I believe we were all discussing as this point as the thing that we're trying to fix, which is that, when you know, if we imagine a protobuf format that we are using to store encrypted data, the mode if such a field exists in there, that is in there and it says, asgcm it's referring to the cube, how the cube api server did the encryption of the data using whatever ephemeral key that it created.

A

It is unrelated to how the kms plugin would encrypt the actual data encryption key. I also agree that that should also use a sdcm, but the api server cannot do anything about that. The kms plugin could just literally just return the unencrypted key.

A

Does that make sense.

B

Let me repeat what you said: so the whole discussion wasn't about the encryption of the decks within the kms plugin, but not what the api server does. So this is what you said right.

A

No, no, the the discussion so far has been.

A

How does the api server use the dac that it generates in memory to encrypt the data, and it is today 100 hardcoded to ascbc and nothing a kms plugin does can influence that or otherwise improve it, and we want to as part of various changes to the storage and other things go ahead and tell the api server use asgcm, always, but instead of implicitly using it, let's put it somewhere in the storage format as the mechanism that was used.

A

So if, in the future, we want to do something else for any reason, we're ready to make such a change. But it's just it's it's. If such a configuration existed, it would be telling the api server.

A

How do you use the deck not telling the kms plugin what it should do with the deck when it says the grpc.

B

Okay and everything except the last part, so um we don't like the kms plugin what to do with the deck.

A

Yeah, so the plain text that's sent over the encryption call is the deck, but it is opaque to the kms plug-in what it is. It's just a series of bytes, exactly yes and the kms plug-in is passed very nicely. Please encrypt this and send me the ciphertext back.

A

It is up to the kms plug-in what it wants to do so in our reference implementation that would be asgcm using a key hierarchy, but the kms plug-in is too free to do whatever, including not encrypting, because you can't make it actually do encryption right. Just basically just send it back to us or something uh right like a as a like.

A

You know, one of the things that you know I had proposed a long time ago that I don't know if alex ever did, was to take my citadel thing and strip out all the encryption and just return the keys, as is as a way of measuring just the pure impact of having a kms plug-in at all right, because it's not making any network calls it's not even doing any encryption. But if it's present in the loop and it's causing grpc connections to be made and funneled through.

C

A

Server, if you you know, measure two api servers, one with and one without on the same hardware under the otherwise same configuration with the same load. You know you can kind of get a high level estimate of like what is the actual cost, pure, like 100 percent cost.

A

That we can never actually get around um once we came up with a completely different architecture, um but yeah, so I I believe in the key hierarchy within our kms reference implementation, we would use gcn and uh last time when we had talked about the the protoconfig that um the the the well the niche was looking at. You know that you know there would be this. uh This key.

C

A

Within the yes.

C

A

uh We had talked about you know. Well, maybe it would be nicer for it to be more strong, like there'd, be more fields available so that uh there could be more.

C

A

I could imagine metadata that a kms plugin emitted, which was well what is the mode I used to encrypt the plain text that the api server sent me, and you know it might be as gcm or whatever right. That way. You know I. If we go down the route of including metadata, I would, I would hope, by reference. Implementation would also include that that mode as a as part of its metadata, so that way we have both flexibility on the api server to change things in the future, but also within the reference implementation.

A

That way, we avoid as much impulsive things as much as possible.

B

Yeah, that makes sense yes, this was this was the second half of the discussion last week right.

A

um Dude, thank you for answering my question. I was going to ask an initiator. Did I say anything that you all felt was inconsistent.

C

That was totally right and but at least from the poc that I have now, I've used cbc. But this is good that you know we will be moving to gcm yeah.

D

I was actually going to ask like uh should that change happen like we could work on it in parallel right that particular change.

A

The the change.

D

Switch switching default to gcm.

A

Yeah, uh you know I had the pr like two years ago that basically did that and we just decided not to pursue it because we're like oh we'll, fix everything, and we just then did nothing so yeah. If we want to uh like. We still have two weeks in 124.

C

A

Wanted to revive that pr, which just basically gave the api server the capability of using gcm not making it use it. I think we could go ahead and add that as a thing.

D

A

D

A cap: do you think.

A

uh I don't think so. I think that's within the kind of the scope of like an implementation detail that.

A

It you know if it required a captain, I would say then we would just lump it in with the work that we're doing otherwise.

C

E

A

Really heavy weight thing uh for a cap.

D

B

But it might need that cap because um either we make it at an um property where we can, where you can configure how we want, for example, the envelope encryption to happen. If it should, the default would be the ncbc. So it's a behavior that you know or you could change the asgcm um or if we would just fully change it, for example to gcm and the encryption would work right. So we would need some kind of logic that, if gcm phase, then please try cbc and assume that it worked right.

A

uh So the the pr I had all it did was give the api server the capability of reading gcm data back, but it left it only. Let it use cdc and did not change actually any behavior. It just made it so that if, in the future it encountered gcm data, it would understand it and be able to do something about it, the idea being that then one release later or maybe even more, you just flip it to say I always use gcm, and then it doesn't matter because you can always downgrade back like you can.

A

You must be able to downgrade one release so that that flow still works and then, if you, if you cared as a consumer, to go towards gcm or go back towards cbc, whichever way you want, you just run: storage migration.

A

That was sort of the framing, because then it kind of lets. You basically uses the multiple release cycles as a way of avoiding the api, because it basically says this is the new default. It is obviously not compatible with the old one, but we can gracefully move you forward and let you gracefully move back and you're fine. So like we haven't, introduced any breaking change by changing the format. We've just pushed it across enough releases that you it's not a breaking change.

B

Okay, sorry, let's revive this product instead because it's useful because there was no wholesome solution yet yeah.

D

A

D

Up your pr again, maybe.

A

uh I don't know if I'll be able to pick it up in the next two weeks I mean I can certainly try if you know, if there's enough motivation and interest because, like I literally remember like mike's comments saying this is a great idea. By the way should we fix a bunch of other stuff while we're at it. I'm like. Okay, fine, maybe we'll fix a bunch of other stuff, and then we never did.

D

Yeah but as you said, this is like not it's related, but it's it. I think the isolating it to its own pr makes sense, uh and since it's not a breaking chain, you can introduce it in this release even right, I I so I I'm a proponent of you just reviving that pr and push it through.

A

Okay, so the I think.

A

This was like, oh it's a long ago,.

D

Oh, this is a pre-covet okay,.

A

Yeah yeah, no, I okay. I did work in vmware, so there's that didn't still literally work at a red hat at the time, but yeah but yeah alex still worked at google.

B

Yeah, so this is how we now count not not before and after christ, but before and after covert yeah yep.

D

uh Yeah, let's because I think this is one of those things that people keep asking for so mine as well, since we're making all these changes anyway,.

A

Yeah I mean if I I I'm.

A

And since you have the.

D

A

Sorry go ahead since.

D

You have the coders.

A

Yeah, I'm sympathetic to the need of making progress on this in incremental steps, especially since I I realize that customers are annoyed by this arbitrary voice. That's imposed on them yeah, especially when it makes no sense.

D

A

Okay, yeah, so we can do that totally. um So, if I recall anish, uh you were one, you were gonna, maybe present something today or show some stuff today you want me to stop sharing.

A

You're already a co-host.

C

Okay, uh so I have a working poc on what we discussed last time. One thing was: I ran into an issue with using protobuf for serialization, so just for the poc purposes, right now, I'm using json but again changing. That should be pretty simple right, uh but in terms of the things that I did basically, what I did was I created a new api new proto file, so v2 alpha one and then in v2 alpha one.

C

uh The things that we are adding is we're adding this new string uid field, so that we can observe requests going from api server to the kms plugin and then also like more mentioned right. This new key field right now, I am using this key only for the encrypted uh local kick, but I think we can also add additional metadata here and in terms of the azure kms plugin.

C

I was thinking we can send metadata on which keck was used to encrypt, uh which kms keck was used to encrypt this local kit, so that when we get it for decryption, we don't have to scramble through all the versions available. We can just pick the version. That's there in the metadata and just try and decrypt with it and then in the encrypt request. We are adding a new uid field and then in the encrypt response.

C

Basically, the kms plugin sends the encrypted local kit, which was used for the encryption um and then in terms of the changes that I had to make. Most of it was in transform to storage and transform from storage, uh basically reading through the serialized data and then sending the encrypted kick and then the cipher. In case of uh when you do an encrypt, you get the cipher and the encrypted check from the kms plugin. And then you add that into hdd and then similarly on the decrypt call we send the.

C

Where is decrypt.

C

Yeah and then in the decrypt call, we can actually send the uid and then the data, and then here it's nil, but basically I'm also sending the encrypted kit. So the kms plug-in has all the information when they're staying through.

C

So I think the majority of the changes were in the kms plug-in um and then I I looked at moe's citadel already right and I think that implemented a lot of things that we need. Although it was for getting the kms keck and using it, I was able to reuse most of it for actually locally generating a kit and then using that for encryption.

C

So um basically I'm creating a new keck service in my kms plugin and then I'm using some of the logic that's already used in our api server for generating the key and then also storing it in an lru cache. So I have a transformer lru cache where I'm storing the base64 encoded uh encrypted key and then the decrypted check so that whenever the calls come in, we can just automatically use that and then at any given point in time.

C

I have one keck, which is used as a nominated keck for encryption purposes, and then I added this encryption operation counter which basically can be configurable, but right now, I'm setting it to five just so that we can see in the demo. But what this does? Is um it reuses the same key for encryption certain number of times and then, when you hit the limit, it will rotate that kick so in this get logic, we are saying.

C

If this particular encryption check has already been used for five or more operations uh less than five operations, then we can still reuse this check for encryption, so just get it from the cache and then return it. So we can encrypt it. But if we have already exceeded five operations, then what you do is you go and rotate. This locally generated kick and then use this as the new kick going forward for all the operations and then here in rotate encryption check.

C

If we can see, basically, we are generating it exactly similar to how we're doing it in cube api server today and then the next thing we're doing is we're actually making a network call to keyword to say, like hey. I have this local kit.

C

Can you encrypt it with the kms keck for me and then give it back and this metric I added so that in as part of the demo, we can actually see for how many calls we are generating, but yeah after we do that we're basically adding the encrypted keck to the cache, and then we also have the encrypt key so that whenever the calls come in, we look at the cache not found in cache. Then we can go outside and.

A

Question and ish yeah, uh so maybe it was the last thing you just said uh where? Where is the logic for when you? I guess this is just encryption right? You haven't gone over decryption.

C

uh The decryption is basically in the get, so this is the part for decryption right, so when decryption calls comes in so basically we know that this encoder we get this encoded kit from entry, tech from the grpc call already as part of decrypt request. So the first thing we do is we look up the cache to see if it's already found in the cache. uh If it is not found in the cache, then we take that encrypted kick and then make a call to keyword to say: hey, look, I have this encrypted check.

C

Can you decrypt it for me using your kms key, all right, yeah and then, if once we get that again, we have a metric here just so for the demo purposes, how about also we can have it and then once we get it, we also add that check to the cache, and then we send it back so that that can be used for detail and then in case of the local encrypt and decrypt that we talked about. So we talked about aes cbc being used in the cube api server.

C

So um I followed along what civil is doing and used aacbc also for encrypting, the local kick uh in the kms plugin, uh so that is where cbc the go yeah. So this is very similar to what citadel is doing right so we're basically creating a new aes cbc service, and then we are actually doing decrypt and encrypt, and then this decrypt operation is basically decrypting. The local keck using the local kick to decrypt the deck and then encrypt is using the local keck to encrypt the deck. So nothing else changes it's just.

C

We have another kick which is there in memory and we're using that. So we have that and then what else and just here yeah I think majority of that was that and then obviously in uh in some of the logics where we were directly calling keyword, I just had to swap it to use the local encrypt decrypt. uh But I have a working cluster with this.

A

Can I ask you a quick question: yeah sure um could you could you show me where that get function is called? I'm just kind of. I was just kind of trying to mentally map the code in my head, so where, where do we call get in like relation to that, like a cbc service thing.

C

Yeah so basically, when we are actually doing decrypt, uh so this decrypt is called when the grpc request comes in right. So here when we call decrypt, we give it this encoded kit, which we actually got as part of the grpc encrypt our decrypt request, and then it uses that to say, hey, go, get blocked and then get blocked is what is just doing get. So it's doing go, get the keck.

C

So this key here is basically the unencrypted local ket that you can use for entry particular publish.

A

Okay, cool all right that makes that makes sense. So, okay,.

D

D

I thought we were storing that in storage right.

C

Yeah we do so this encoded. This encrypted kick is actually coming from storage right. So in a decrypt call, it's uh api server is reading it from hcd and it's giving us the cipher text and the encrypted check as part of the decrypt request.

D

So this is going to storage right. This get call.

C

I don't know this is in the comments from storage. This.

D

Is kms? Okay, so where are we getting that encoded k? This is before this call right. Where is that.

C

uh Hold on, let me show you here.

C

Okay, so if I look in hcd right like just because I'm using json like this would be a lot easier for us to read, but in hcd what we have is we have the encrypted data which is encrypted using the deck, and then we have the encrypted get so this encrypted keck is sent from the kms plugin to cube api server.

C

Every time a new encryption is done, and this encrypted kick is a local kit that was encrypted using the kmsj right and then, when we do a decrypt call from the api server. The cube api server reads this data from ncd. It sends the encrypted data.

C

And then it also sends the entrepreneur keck to the kms plugin.

C

So we have that encrypted check here now and what we're trying to do is look up in our cache in kms plugin to see if we already have the unencrypted value of it.

C

If we do, then we can just use that to decrypt the check cipher text, but if we don't have it, we make an external call to keyword to say hey. This key was used to sign up to encrypt this local kit. Can you decrypt it and give me the data.

D

Gotcha, so this keck.get is actually calling the kms.

C

D

It looks up the cache.

C

And if there is a cache miss, it will call kms. Okay.

D

E

You go into the get.

D

Call like yeah sure, sorry.

C

Yeah, so we're actually mimicking what we're doing today with debts in api, so yeah. So this is the get call. So if encore encrypted keck is nil. That means it's an encrypt operation right, because there is no data available. This currently I'm assuming it's an encrypt operation, but this can also mean this is all data. But, like that's the case we have to handle, uh but let's go down the route where this is a decrypt operation. So we are not in that if loop right so.

A

C

Not in that, if loop, basically, we are saying, go, look up the cache first thing with this encrypted kit. If it is available in cache, you return the unencrypted check, so it can be used for other further operations, but if it is not found in the cache, we already have the encrypted kit, which is this so we go and make a call to keyword to say, hey you.

D

C

Enter this from me, gotcha.

D

Yeah, so that's the call that I was looking for. Okay,.

C

Right so this is a call to keyboard.

C

Okay, yeah and then so I have a working cluster right, so this is basically like how we're storing it right now in hdd like, but we can move the proto after this, uh but.

A

I could make a quick comment. I think we already talked about this last meeting, but I I think, uh both in the code and in our apis we're going to be want to be very careful with how we name all the variables, because basically all of them are like kv. Key vault key this key that and it just like it gets like.

A

The questions that rita just asked, you know are basically like conceptualizing all right, like which part of this hierarchy are you in right and and it's hard to talk about it because you're just saying the word same words over here.

C

A

Different contexts right, yeah,.

C

Yeah, I think there was a point in time yesterday where I was just going around and found, because all my variables were like wait encrypted encoded, which one is what but definitely like when we clean up this implementation. Yes, the variable names and how we name things is very important to understand that actually.

D

I mean now that you explained it. I think the encrypt that can encrypt and encrypt data make sense to me. Like I mean this makes a lot of sense like the way it's stored, um so I'm actually, okay with the naming of these things in scd. I think it was just in the code yeah. It was confusing like get and decrypt, because everything is decrypt but who's calling decrypt right like it's a local d grip or it's a kmsd crib.

C

Yeah yeah yeah, okay, so yeah, so I have the kms plugin running. um So if we look at the logs like right now, I'm just logging some of the things so there's also the health check request and all that right. So if you see here basically like for encrypt operations that happen, it says, hey look.

C

I found the kick in cash, so I'm just going to use that for encryption, so you can see count two count, three count four and then because we set a limit of five, it says: oh I've already hit the limit, so I'm going to read and rotate the keck that I have locally so that for the next set of encrypt operations, I'm going to be using a new local kit for encryption and then in terms of metrics.

C

So if you look at this metric uh basically this is the total number of encrypt calls and decrypt calls that the kms plugin is doing right. So this includes calls from api server um health check from api server. Then health check from my kms plugin as well.

C

So if you see here basically there's like 4200 our decrypt operations that were called within like since last night and then similarly there's like about 14 for 4200 encrypt calls to it right and then the metric that I added the kms encrypt kms decrypt was to see how many calls we're actually making to key world. This number is divided by a factor of five.

C

So if you look at it, it's only 780 instead of the actual 4200, because we are only encrypting it and storing it in cash and then, even in terms of cache warm-up, we actually don't have to do a lot of warm-up. So we just when the plug-in restarts it generates a new local kit that it will use for new encrypt operations. But if api server sends an old deck, an old check, saying hey, can you decrypt this? For me, it'll say: oh, this is not in my cache.

C

I'm just going to go, call keyword, decrypt it and then put it in my clash.

C

So we can do that and then, uh in terms of uh all the configurations, I think we can have similar things right like today we can configure cache size and api server. I think we would be able to configure cache size even in the kms plugin, um for how many local kicks it can store in memory.

C

um I think that can be what kms plugin providers want to do and then also in terms of how many times they can reuse, a single local kit for encryption uh can be a factor that they can decide based on how many calls they want to cut down like based on rate limits available with azure keyword or something like that they can say like. I want to reuse this particular local kick for like 1000 operations or 10 000 operations. Basically, they will reduce the number of calls by that structure.

D

Can I answer a question, uh so this cache is for the local cac is in uh is maintained by the plug-in right, the kms plug-in right. um What happens if so, what if there are multiple.

A

Right so you're using azure key vault as your orchestration layer right so as they generate new keys. If a different instance of the kms plugin sees a key, it doesn't understand, it just asks key vault, hey. I don't know what this says. Can you decrypt it for me and then I'll cache it locally? Well,.

D

Specifically, I was thinking like multi control playing scenarios if there.

C

Are multiple plugins.

D

They all have their own cache right.

C

Yeah that also works really well, so, like none of them have to maintain the same cache. Data like they can just be totally have they can maintain their own set of cash.

D

Because check for the cash and if it's not there, it will try to get it from kms and then it will store its own, its own cash. Exactly.

A

Right but they're effectively stateless right, they have a cache, but it it when any at any time they can be restarted and basically rebuild that cache using the azure key vault in this case to fix it.

C

Yeah, that's why I was saying there is no cash warm-up required for the kms plug-in. It only has to get data and storing cash on a need need basis like if someone says hey decrypt this and if it doesn't have it it'll go, get it from keyword and then cache it.

A

Yeah so thinking through this, let's say the default uh key reuse inside of the kms plugin is like a thousand or less than or maybe ten thousand he's kind of use it a little bit uh in like like a medium-sized kubernetes cluster.

A

You probably only would need to make three calls to kms right, because you would basically have three api servers, so they're likely to have three kms plugins with three keys that were actively realistically being used at any given time. And if you know, if you're doing your regular storage migration, you basically pull less onto three right.

A

So even if you pretend it's coalescing towards 10., it's kind of irrelevant, like like the numbers, are so small now that it doesn't really matter because you're you're talking about you, know upwards of 10, 000 secrets, 100 thousand secrets, but we need, like 10 network requests, to actually get yourself into a state where your local cache has been warmed inside the kms plugin, to the point that it can decrypt all of that cd if it needs to so so.

A

Your cost is basically down to that thing that I thought experiment had earlier, which is what is the actual cost of running canvas at all, like those grpc overheads, which I don't see us getting around it like I mean the the local cache within the api server helps with that one, but you know that it's kind of to some degree it's a little bit irrelevant once you have just the less need of actually making remote calls.

A

um This is cool like this is kind of. What I expected is that it would, just, as you said, just divide the number of network operations with how often you were willing to reuse the key, and I know christoph you did some actual math to say yeah. You can basically use it like a stupid number of times and it doesn't actually matter.

B

Yeah two million.

A

Yeah, like it's just an absurd amount right like so, if you use it 10 000 times, you're like many orders of magnitude like below any safety margin, without like really any concern like honestly like I, I don't care if that, like inside the code. All of these are like inputs to some function and can be used.

A

I don't know if I would want to expose this as config that I tell a user to care about because, like I don't know like 10 000 is perfectly safe and basically reduces the problem that this is solving to exactly kind of zero or a hundred thousand whatever like. It basically makes the network problem disappear in my opinion, so I don't know if it like it's worth asking the user to understand the number, because it's kind of hard to even explain what it does.

C

Yeah, I think the knob is solely for like people who know what they're doing like if it's a cloud provider deploying it and.

A

C

Can calculate the cost then they can configure it. But yes, as for users, I think they would just be okay. Like I mean there, they have a problem with what is there today and, if you're, reducing that by a certain factor. I think, like that, still helps.

D

I mean, as a cloud provider, maybe you're like wait. Why am I storing more data in my std? I don't know that's one factor, but they might ask, but they don't care properly. uh Quick question.

E

D

When we do rotation, how does that work? We talk about that.

A

So I I think that's the biggest sort of gap in the current sort of like not the current like so the v2 alpha one api is given right is the bit that's missing is how does the api server know sort of like how far the kms plugin is kind of like kind of in that state right?

A

Because, right now, if you run storage migration on this cluster, depending on the state of the kms plugin, basically where's the counter at right now it will either do a storage, migration or maybe won't okay, I think sort of maybe actually I'm not even sure what so yeah like. I think. First, we have to define what is the expected behavior.

A

Do you expect the system to do if a storage migration like in this this cluster right now, if we ran a storage migration? What is the expected thing.

D

Do you mean I mean just to clarify? Do you mean if like say the remote kms, I got rotate key got rotated? Is that what you mean.

A

uh So I'm stepping back a little bit and thinking through the kubernetes api, so storage, migration from the kubernetes api is basically coupe. Ctl, get secrets, dash all name spaces shove into a file and then cube ctl.

C

Replace all of them here.

A

Yeah replace dash f that file right and that that that's the interface that kubernetes provides so I'm trying to stay within that for a second. So the expectation traditionally has been that the api server can use the positions in the encryption. Configuration object to figure out if keys are stale.

A

In this example, there's only one kms plugin and only one index into the list, which is that one kms plugin so from the api server's perspective.

A

Basically, there's never a need to do storage migration because there's only one right, it's only one in the list. It can never be stale, but, as we've seen that every five increments, it changes right, the key being used technically changes from the perspective of like the api server. Maybe we don't care about that five. But if you, if you changed the I I guess I didn't ask a niche: how? How is the real key vault configuration done on this plugin? Is it like just?

A

Are you just pinned to a particular version of a key right now.

C

Yeah today we pin it to a particular version, yeah here yeah. So basically we say the keyword name. Is this and then the key name is this and then we also pin it to a version.

A

Right so in terms of orchestrating the rotation right now, what exactly the assumptions in the api server are currently correct, because the meaning of the kms plug-in never really changes and the meaning uh in the encryption configure can't can't really change right now, because there's only one right. So in that sense, this is correct, even when this new key hierarchy is correct, because the final key vault key is still the same thing.

A

So it's kind of fine, uh I I, but it's not neces, it's not sufficient. If you want the key itself that like like, basically, you know, we talked about like labels or whatever like so. If there was like a way of saying, key label equals current fraud or whatever right. So, basically, instead of pinning to a particular exact version, you were pinning to a mean instead of a version, um then there has to be some way for that to be communicated through the whole layer.

A

Right like there has to be a way of the api silver. Knowing that you, you believe this to be stale.

D

Yeah and he has to know the old one and the new one in order to do the migration.

C

Right so I was thinking if there is a second layer of rotation, so if kms check gets rotated right, this.

A

C

Metadata that more talked about can be helpful so today, when, as part of the byte key, all I'm sending is the actual encrypted local kick right. But let's say we have an additional metadata field or we reuse that field in that metadata field. We would have this encrypted local key, but also like for azure kms plugin. Like we can say this was the version of the key that I used for encryption.

C

So let's say the key got rotated in key vault, so there is v2 and then there is v1 and everything. That's there in xcd. Right now is encrypted with p1 right. Even if the user does a storage migration immediately, then everything would get decrypted without any issues, because in the metadata we are saying v1, so we will go to keyword and say hey.

C

I don't have this in cache decrypt this with v1 and give it to me, and we still would only make one beat call because the first time it's not in cache, we make the decrypt call then for all the other secrets that come as part of storage migration. We don't have to it's already in the cache. Decrypt would work fine and then, when encrypt call comes in for all of them, we would actually encrypt it with whatever is configured as latest or whatever is pinned to the kms plugin right now. Okay,.

D

And just to confirm after you decrypt with the old version, you would up encrypt encrypt it again with the new one and then update the scd record with that property or the metadata. Okay.

C

Right yeah, the new encrypt, all of them will have v2, and then I think the question was: how do we expose it to users? But this is following the exact same procedure as today like today, when they do cuba city get secrets dash a pipe into cubes. It will replace when the whole thing succeeds.

C

The user assumes that the storage migration is done, that they can get rid of the v1 version. I think the same assumption can be applied here as well like if the call succeeds also in terms of the number of calls we make to kms we'll go down here.

C

A

So I I think the part that's.

A

That that needs uh some tweaking in the api server is the fact that, let's say you're in your in your kms uh configuration struct, you set the cache size to a million. Basically cache all data encryption keys, really care right. The api server will never make call to kms, then once it caches them. So when you do storage migration, it it won't do anything with kms it'll just be able to decrypt the data and it'll be fine right, but that's the part. That's missing is.

C

A

Has to be a way for the kms plugin to tell the api server. I realize you might be caching stuff, and I realize it's actually functionally valid. I need you to pretend that you don't have that cache and throw it away, because I want you to explicitly call me because I might want to make a change that.

D

Makes sense right.

A

um And then, then, what that does is it allows the kms plugin to be like okay cool? I am currently at version one. I see that you're also at version one. It does not matter that my particular local keck is different from the one that you used because we're still at v1. It doesn't matter right like it's conceptually.

A

Like you know, all of those local texts are just basically hierarchied under one true root key, and so we don't care they're, semantically, the same right, but if I instead see that I'm at version two now and and there's a little bit of ambiguity in the sense of like I'm at version two. How do I know all the other plugins are at v2? um I think there is some dot right there.

A

That has to be replied, but let's just pretend that that has been solved if it sees it set v2 and it sees that the metadata says v1 and it's like cool now. I actually do definitely want to cause a change, because I know that the real root key has been changed so now the hierarchy matters right like because it's actually a different hierarchy.

A

So then again kind of that that has to flow through in some way, because then you can basically just make easy tweaks within azure key vault and, like the rest of the system, will basically catch up. I do think there needs to be like uh I make a change in key vault and, like I wait a little bit to for something to happen. Basically, like all the kms plugins catch up to me, and somehow I observe that this has been caught up to and then I do, storage migration.

A

But it's still like way easier than the current thing. uh Rita.

D

uh Here: here's a crazy thought: what if we keep the uh the uh the remote sorry what if we keep the metadata in the kubernetes api server cache as well, that way it would trigger it will know like hey. This is out of date.

A

uh But what would it compare it to right? That's the difference. The camera.

D

um Whatever the the latest is.

A

D

Your point only plug-in knows what the latest says: that's not uh yeah.

A

Yeah, that's that's why right so, I think a a proper implementation of the kms plug-in would make sure not wait.

D

But could we have the api server cache when it calls the plugin? The plugin will tell like here's the version I want you to use and the cache will see that it's out of date right.

A

Well, so so, right now we don't really have the concept of a version right like we have like. So you know in initial proto there's a key. I think key is the wrong name. I think, should be a key id or something to not make one thing they're supposed to send a literal, private key or something or an encryption key right. It's supposed to be a reference, and it's okay, if it's an encrypted reference, but it's not literally plain text.

A

I I think what I would expect to happen is most of this stays opaque for the api server and the api server is basically it might make more calls to the kms than absolutely necessary, but then the kms itself, the kms plugin, to be very specific needs to be smart enough to be like, I see you're making this call to me api server.

A

I don't. I want to explicitly tell you that you don't need to change anything right like it needs to be smart enough to be like. I see that you have a v1 key, I'm also at v1. It doesn't matter that the local keck is different, they're conceptually the same key and it basically hands you back the exact data that you sent me without changing anything on purpose, to make sure that you stay aligned with me right. It's just a way. We have to basically have a way of a push notification effect.

A

Right like there has to be a way for the kms plugin to say: hey the next time you try to encrypt or deep or actually you know next time you try to decrypt data like next time you try to fetch data out of xcd.

A

um Actually, no, I think it's both next time you encrypt or decrypt data. You need to call me. I don't care if your cache is up to date and you can do it without me. I need you to call me so that way I can help you figure out what like, basically, what to do here.

C

Is there any way we can uh reuse some of the version? Grpc requests that we have today so like when api server starts up? It makes a version api version request to the gamers plugin to see if it's running on the same proto version.

A

C

Like v1 alpha v2 alpha one stuff as part of that request, can the kms plugin provide some kind of data which the which api server can cache and then periodically make that request?

C

And if that data changes, then it says: oh there's a diff, I'm just going to go invalidate my cash.

A

I I think that works, it's just that you have to somehow guarantee that before you run storage migration, that the periodic thing happened after you changed the real key in key vault right, so that might be okay right. Basically, instead of basically instead of the communication coming from the kms plug-in, I think initially saying well, maybe we can try to reuse the channel we have in the other direction and just pull instead of push. I think that might be fine right, like at the end of the day. Rotation is relatively rare right.

A

It's it's like it's a really important edge case that we want to have strong support around, because it's basically the hardest part of the whole thing, but it's also still a rare enough event that you could say that, like it happens like at worse once a week, basically like once, maybe once a day at the really crazy edge cases. So it's like okay, fine, if you pull like every minute asking the plug-in. Hey are you? Are you good now.

C

Yeah I mean, if you look at it, it's already doing a health check of every 20 seconds today so like this does not add a lot of calls like doing every minute or something like that and also like it's. It makes it a lot easier in terms of how we rotate stuff, like all it really has to do, is invalidate cache. So any new request that comes in now says: oh cache miss I'm just going to go. Call the games plugin.

A

Yeah, I think I don't think I would want to reuse the existing version api. I think I would want like a more dedicated one to just make it more clear, because I think the version api is just make sure we're compatible with each other and the health check api is like. Are you not dead right.

C

So I think, I think, in the proposal we had this health check, request health check response like I think that might be a good one, that that basically is the kms plugin saying yes, I'm up and running, and then we can also add certain fields there that we think are valuable like saying key rotated, true or false, or something like that, and we can just do that health check api periodically like we don't have to do multiple requests.

A

Yeah, I I guess I guess we might want to then sort of rename that to like status right like overall status, it's not just. Are you healthy? It's also like what is your current state. um I.

D

A

And yeah within the scope of status, it makes sense right to say I am healthy. I can talk to like I I you know like I I'm healthy, because I recently had a successful interaction with key vault. That might be the definition of healthy, for the plug-in is that within the last 10 minutes, I made a successful network, I'm healthy in that way and by the way in that last.

A

I I guess uh there has to be some definition of like, uh like how does a kms plug-in know that it's now at v2 versus v1 and like how recently did it check? There's some there's some ambiguity there. I guess of like there's a delay right like we're not not going to sit there pull every second trying to be like hey key vault. Did you get a new key? That would be silly, um so I I guess my question to y'all would be all of these sort of layers introduce some delay.

A

How how do we know when all of the delays have effectively caught up is because once all of them have caught up, then you can run storage, migration, okay, but before then running story, migration is safe. It's perfectly sound, there's nothing wrong with it, but it doesn't necessarily do what you want.

A

It might do partially what you want, depending on which kms plug-in you hit, which sub-state you hit, but today it's really obvious when you rotate versus not because you have to orchestrate the whole thing yourself, so it was really either you get it right or you break your cluster in some funny way.

D

That's why I was like, when you say, invalidate the cache, uh if, if we don't store the new version of it in the cache in the new updated cache, how will we know now if I need to invalidate it or versus not about invalidating well.

A

So so think about it like this right, let's say: there's apis around this and at time 2 you press a button in key vault that makes a new prod key available right.

A

If you have a way of asking all of the instances of the kms plugin and all the instances of the api server at what time and again you need synchronized clocks for this, but at what time did you make your last check if all of those are greater than time two?

A

You know that they know that there is a new key and now, if you run storage migration, that everything will coalesce onto v2 instead of v1 right assuming your clocks are all the same right.

A

So like you know, I could imagine that being sort of the interface which is.

D

I don't know like it feels weird to rely on time stamp for this, because things could go wrong. Yeah.

A

You're right right, so I think that's like the minimal implementation right. What I'm saying is, can we do something better? That's a little bit more concrete and specific right like what. What do we need to flow through the system before.

D

Yeah, that's why I was saying if, if, if plugin says hey, I have a new version now and aps service sees oh okay, I'm gonna invalidate the old cache and store this one in the new cache. Can we do that because because then next time it would just use the cache one and compare if it's the same. It won't invalidate the cache again.

A

So there would be so there would be some concept of a version somewhere like a key version or stats and something like that. There's some fake version string.

D

That way, we also don't need to worry about keeping the api server caption sync either, because each one will have its own cache and if it's out of date it would invalidate and get the new key stored in the new cache.

A

So when you do this periodic status fetch within it, it would tell you the current version of the thing right, so I think, what's still missing, there is how how does the actor that's gonna run? The storage migration know the current observed version right. Basically, if I, if I told azure key vault, I want to go to version three now.

C

A

At version two all of the systems are aware of this in the sense that they're gonna, as they do their periodic polling and stuff, they would catch up and learn about version three. But how do you know that they have done that? Basically, how do you know that all three api servers and all three kms plugins have observed the new v3?

A

Thus, when you run storage migration, the right thing happens that that's the I think the piece that's missing. I I think you're correct that explicitly trying to funnel a version through is better than the timestamp. I agree with that, but I think there's still one step. That's missing, which is how do you know? Storage migration is, can now be run.

D

Yeah I mean the api server is the one that's like that knows: hey now, there's a new one. I now have to go and validate the cache right like can when the api server knows that, can that be the trigger.

A

Well, the interface itself does not run storage, migrations right.

D

Can it uh can I create something that triggers the storage migration.

A

Yeah, so I I think I think the storage version hash that concept that exists within discovery that is used by the storage version, migrator.

E

A

I think that that's where we would want to somehow include this version information, so the hash changes, meaning that within the kubernetes api there is an artifact that changes when this is observed and thus, when you still have an issue right, which is that that artifact is per api server. I think maybe I'm actually not exactly sure on that.

A

So there's still a little bit of a coordination issue, but yeah like there has to be some way for you to say: hey all three api servers, I'm late for my next meeting uh that you have observed the state and dust storage migration will do the right thing. I think there's still a little bit, I think, of disconnect in my head how we, how we get.

D

Right yeah, I mean I think, last time you mentioned, there was like some a code for when storage migration uh happened like maybe we should all look at that.

A

D

Did you share that link somewhere? Sorry, if you did, I.

A

Have not, I have not gotten back to it. uh I realized we're over time, so we can probably stop uh a niche that was really cool. Thank you.

D

Thank you that was really great.

A

Very exciting to actually see cool stuff happening and I'm glad that our intuition was correct right. It's basically like you dramatically reduce the net request you make with just one.

E

D

All right, I guess, chat on over slack.

C

Yeah for the rotation stuff, yes,.

D

C