Kubernetes WG KMS, 25 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WG KMS Bi-Weekly Meeting for 20220125

Description

WG KMS Bi-Weekly Meeting for 20220125

A

All right so hey everyone. This is kms meeting number five, yes uh for january 25th, 2022., uh cheltenham.

B

Yeah, bodyguard christian! No! No! No! No! No! No! Okay! Thank you! So um so, as we want to talk, do you want to talk about the cap first or do we want to start with the performance topics.

C

um We can talk about the camp first too. I mean we talked about before the recording. So uh but yeah I mean right now, but the cap, I think, mo and rita, have reviewed it and then damian added a few comments. So we've made changes based on that. um I think right now mo has tagged uh clayton and david to take a look uh for the final review and approval.

C

uh What we've done is based on the previously got call. We have made changes to add a feature flag which is enabled by default, but if the user wishes to disable it, they can do that in the 124 release, and we have also made the uid field pointer so that if the user disables the flag, then uh the data that's been sent on the wire will still remain the same.

C

In 124 and like the previous kubernetes version releases and then in 125 we plan to make that flag, ga, which means it can no longer be disabled and it will be enabled by default, based on feedback.

C

Now any other questions on that yet.

D

No, I just wanted to say that I saw almost comments yesterday about the matrix and I need to check that out again because I really miss these metrics and I haven't checked at the time to really look into that again.

D

But if the metrics are already present, maybe what we should do instead of like adding matrix to this cap, is rather just update the production, readiness doc to add that we have metrics to know if there is something wrong with this feature or not,.

C

Right, but the production readiness for this particular clip is very specific to the uid feature being added right. Do we need to mention the metrics that already exist for the prr? Here I mean don't. We need to correlate how uid is helping here.

A

A

One one thing I would say there is that, um aside from the this, these metrics aren't documented as far as I know so because I had forgotten they existed because it's been so long like I I I felt bad because I actually reviewed them like I don't know how many years ago, like at least part of them, so I should know that they existed but I'd forgotten god. So there's that but they're not documented, I don't think they go into prr for the current cap, but they they should be discoverable without reading the source code.

A

um Right and yes, there are alpha, but all metrics are like vaguely alpha, so that's kind of unhelpful.

A

I I don't, I don't think the cardinality requirements of the metric stuff allow us to include the uid in any meaningful way in them, though,.

D

A

D

Which is okay, that's fine they're for different.

A

Purposes right: that's, okay,.

D

A

um Yeah, my gut feeling would be is that there's these are just separate things now yeah. You know we might review the metrics and decide that they're insufficient and then, as part of whatever further observability work. We do, we might say well, we need more or we might need to take existing ones to be better in some particular way.

D

Yeah my reasoning process was more like um in terms of prediction, my business.

D

I would kind of expect that we um go through the whole process that the user would have to go through, like in prediction, if they had to kind of debug kms, because, like this cap is about observability as a whole, um and the first thing that they would eat should hopefully be metrics, because that's how they would detect that something is going wrong and then by knowing that, and they would be able to use this cap with the uid that has been id to really um kind of have more awareness about, like which uh operation failed.

D

Why uh and yeah get more insight based on that to meet part of the overall experience, but I'm not sure if that's what they are trying to measure with this question or not.

C

Yeah, I think that makes sense like the way I viewed it. When I was writing, it was like vr seem to be very specific for the new feature that you're adding like how ready what you're adding is uh in terms of readiness.

C

But what you are saying makes sense right in the overall, like we can say: kms observability. What are these things, how it can be done, but I I don't know if it can, it needs to be in the prr, or maybe it's just some separate thing where we can say what is the general kms observability experience like we are adding uid, but we can also, like maybe document like as part of this metrics, that we document, maybe we can have like all these different things that can be used for observability as an end user.

D

Yeah, the only thing that I might be aware of is that if the production readiness team comes in and say hey, why are you adding uh logging stuff where you don't even have metrics? Just by reading like the actual question that they ask like we have metrics, we it just that it's not part of the cap, like we don't have added this.

D

We haven't added this information yet today, um and maybe that's what I want to see like they want to see overall experience like if we are missing anything before starting like, for example, if you start by tracing- and you don't even have monitoring in the first place- um that's something that they might want to catch with this production readiness stuff.

D

So I'm trying to like I'm not sure exactly what they are trying to achieve, but from their perspective I would kind of expect uh to have like the overall experience that would be described to me once something is. I did.

C

Yeah I mean: is it okay? If you wait for reviews and see if that's what they want? Yeah, um okay,.

A

One thing that this conversation came to mind for me is.

A

Yeah, I think all of the work we're going to do with this will require this, but maybe you know we can start off with just the uid stuff, because I think we would. We want to document this new behavior right so within right now the kms documentation within kubernetes just says how to configure it, but and that's it so yeah within that page. You know we should.

A

We should document the metrics, but we should also document you know it might just be like hey if you're running one two four, I guess you don't have to say one two four, because the docks are version, so the the you'll implicitly be at the right place.

A

If you're, um you know new enough docs, but you know they would tell you you know if you're observing kms failures, um you know in your kms, then you know you can try to correlate by looking at this uid in the chemist and looking at these logs and here's maybe an example log that tries to help you understand. What's going on.

D

Yeah, it definitely makes sense.

A

Yeah, I don't think docs are explicitly covered in the chemical. I think they're, like implicitly implied because every kep anything deserving of a cap is obviously needing docs. Basically, so just.

D

Yeah, it's something that we definitely might want to keep in mind while iterating through that.

A

Yeah and I suspect in the kubernetes docs, we don't actually document the grpc api at all. I suspect, which kind of shows how little faith we have in anyone being able to implement.

C

Okay, yeah I'm taking down some notes on the cigars agenda, so I've added this like we should document the enhancement and also the metrics.

C

Cool, so you want to talk about uh performance or do we have any other questions on the existing cap.

C

Okay yeah, so we can talk about performance. I think we touched on it uh briefly in the last sync up on what the issues were and then just to summarize.

C

uh Basically, there's a single deck per secret today and the main issue arises when you have a burst of secrets that is created, because now api server is going to call the kms plug-in to encrypt every single day, and the other issue happens when the api server restarts.

C

And then, if you have configured a big cache size at the api server, then it basically needs to warm up its cache, and then it makes uh a one, a single call for each deck, which means if there are a large number of text, then it's going to make a significant amount of calls and then the kms plugins on one side can get rate limited and once they get rate limited, it takes time to recover and the calls will just keep failing and because this the api server could end up restarting because it's not able to warm up this cache so like.

C

That is one big performance issue uh that is there currently. um So for that, we've yeah and listed different scenarios that are happening today and then I think the focus at least the first one is: how do we optimize the deck generation?

C

um And then I think, a couple of proposals that have been there like. So this was basically summarized from different github issues and like conversations and also like different suggestions from users was uh using a deck for a period of time, uh sorry using the deck for a period of time uh for encrypting secrets. So that means there's. No one is one mapping, but rather have a time period defined for which a single deck can be reused.

C

Then the second one that was proposed was maybe use a single deck per name space. So any secrets that you have in a name space all of them are encrypted using a single deck which can considerably reduce the number of decks that are generated.

C

uh Then the third one was maybe have a certain limit set on how many secrets can be encrypted using a single deck and then perform encryption using that deck for till that limit, and then once the limit is hit, um regenerate a new deck and then use that for the next set of secrets.

C

um Yeah so like. I think we have three of these uh suggested right now and then I think.

D

I think there was a fourth one as well, if I may had, which was like to add some kind of hierarchy uh to the to the kex, and that's like the first kk that is owned by uh the kms. Then have a local uh kk inside of the kms provider, and that would be used to encrypt like the dks.

D

Which was like what we discussed in the previous meeting, mostly.

C

Right yeah, I think mo, is also updating the doc for that one yeah.

C

Yes, I think sorry.

D

Yeah I was just going to comment on the on the the proposition of having like um dex per namespace. um I think the the problem that we will encounter like.

D

I really like the idea in the first place because, like we would still have like some kind of tenancy model in place, um but we have to uh remind that remember that, at the end of the day, like this feature doesn't only target secrets but can target any kind of resource and as part of all the resources that exist, there are some of them that are named space, um so it like. We wouldn't be able to cover this kind of scenario.

D

If we were to have like one name, space per uh one, deck per name space, we would have to have like one day dedicated to everything that is in depth, space, which isn't ideal.

D

Just to kind of remove that from the list just because it doesn't really seem feasible to me.

C

Yeah, I think that that makes sense, uh but also, I think, yeah mostly today, it's been secret. I think that's why the proposal was very focused. On the I mean, the suggestion on github was very focused on secret and I think once we do solve the performance issues, I think it opens up an option to use this for all the different components so that without hitting the radium but yeah that does make sense. If, because, if you don't have namespace resources, then it basically just violates the whole thing.

D

I don't remember, on top of my id for a downstream use use case. We have any resources that are not namespace that we want to encrypt, but I.

B

Know that we have.

D

Like at least four or five resources that uh need to be encrypted, not only like the secret, there are like plenty of other things that needs to be encrypted.

A

Yeah to your question, damien or oh, it's tokens are claustroscoped and openshift right, so their cluster scope, routes, config maps, secrets or name space.

D

Yeah, I wasn't really sure about the access tokens if they were namespaced or not, but yeah you're, right, yeah.

A

They're, their.

D

A

D

That that wouldn't fit uh this particular use case.

A

Right um also, just as a whole, like I, I find the idea of asking for certain resources to be encrypted and others not just to be like weird. I just want to encrypt everything like just move on like like. Why do I need to figure out what is a secret, because I don't know.

D

Which should it be encrypted.

A

In the first place, right right I mean just in general, though, like uh you know, you can add any custom resource to the api server right and you can, you can have it have any semantic that you want right. So, since you can't possibly know what custom resource will be added to your api server, you have to assume at least all custom resources are secret because you don't know right. So, if you're going to do that, you might as well just encrypt everything.

B

One question to number four: wasn't the idea to have kind of a local kms connected to a unix circuit? Was this not one into this direction, or was it really having a local kick? Maybe I misunderstood it.

D

I I think the idea was to have like the the cake uh hosted inside of the kms provider that is stopped into the api server via unix.

B

Okay, so let's answer it.

D

I think that was the idea.

D

And I honestly, like my the feedback that I wanted to give last week about this particular idea that, but we hadn't really much time, but that is what I really like about. This particular suggestion is that we still have the same level of security with the the decks so like the decks are rotated whenever the object changes, uh which is like the best that we can do in terms of envelope encryption um and the key principle that we have like on this feature.

D

That is the kms encryption is still here because, like um the owner at the end of the day of the whole uh key archi is the kms. So if the users interact with their key in the kms as long as we reflect the changes that are made in the kms to the kk that we have locally. So if the kkk is changing the kms and we should change it in the kms provider as long as we do that and the user are still in control of the key.

D

I think the key principle behind this feature are still like, like remains, even though, like we are moving.

B

um For the first suggestions, um so yes, I also agree that that the one with the local cac is the best solution. The problem that I have with the first solution is, for example, to have it for um for uh sorry. My kids are right now and that's a little bit um so the problem. The first thing with the first suggestion, is to use it for a certain amount of time is that it's really hard to pinpoint how many so we would have two different um kind of boundaries right for asgcm.

B

We can do it that many times, whereas cpc- you can have it that many times, and so I see that we have deployment on lots of different boundary conditions for depending on the encryption we use- and this might be confusing problematic to the user and by namespace reduces the problem problem um but having it pair amount seems like the best solution, because it's actually the problem that we have so this kind of pinpoints it, but at the end, having a local keg kind of another layer solves it.

B

So maybe it's like how soft engineering works right, so you can fix everything by adding yet another abstraction layer. So I think also, like they mean said that this works quite nicely, because then you have just one key: that you need to get decrypted and then you can decrypt all the other decks. So it's really nice idea.

C

Yeah, I think also like one thing is uh basically this would be something that each kms plug-in would have to do right, rather than uh the something in kubernetes uh api server. Rather than that, it would be something that each kms plug-in has to introduce and then basically build from there on their own in each kms plugin. So it's not something that would be common across all the kms plugins.

A

Right, but if we look at the implementations that exist today, it's basically just the three cloud providers right like vault, deprecated there and archived it like. So there isn't really many implementations to worry about right.

A

That's where the reference implementation idea comes in is that if we just do this for you in the one place right, because what we're trying to say is the only reason this is like this problem actually only exists because you're using a cloud canvas if you're using a local hsm like a hardware thing, it doesn't care how many encryption operations you need to do. It can do it just as fast as like literally possible right.

A

So you know it's just as fast as software will be right and what we're saying is: that's not our fault right, like we can't make your network faster. We cannot make your rate limits go away, but we also can't work around them in a good way, like every. Every way that we come up with to work around them involves basically a compromise of some some aspect of either our security or probably more important.

A

Just like the semantics of the kubernetes api that we want to make sure it's persist all right so, but if we provide a library implementation of this, I don't think that's that big of a deal, because all we're saying is well, you don't have to use it. If, for example, you don't need that functionality, just don't use it right and we could totally make it so that the key hierarchy bit is um and maybe like enabled by default, but disabled within the reference implementation. That way, if you're like no, I don't.

A

I don't actually need that indirection, because my kms is like right here, it's hardware, so that's right, so we can do both it's more of like I. I like this idea because we can build it once we can ask people to use it if they want to. I suspect people will, because why? Why would you write your own if someone already did it for you, like? I don't really see like there's.

A

No like competitive advantage of writing your own you're not going to get anything out of it, um and then it would prove out that we did a good job right like because people would adopt.

B

Yeah, so so I agree also on that that it's it's not really an encryption problem, but it's more like, like a kms provider, so an api uh network problem that there are too many requests.

D

Yeah, I think the aspect that was pointed out is that the problem itself isn't um in the kms feature, but rather in the fact that we are like some of our users or kms austin kms that need to go through the network, whereas, like some others just have our local. So all of our like of our the consumer of this api, basically um are not impacted by this problem, so it like. There is no actual reason to put the fix inside of the api server itself, but rather like.

D

If we provide the tools in the canvas provider itself or in the reference library implementation, then it would be up to them to really implement it or not if they really need it so yeah. I really. I really really like this id personally of like delegating this responsibility to each cloud provider to like make the performance optimization that are needed.

C

Yeah I like having that in the reference implementation rather than just documentation, because then we're basically.

B

C

Them to do it on their own, but as long as we're doing the reference implementation, I think, like that'll be really great like then it can be one way to have a common approach across different providers and then yeah. I really like the way you can just opt out of it. Then that'll be really great.

C

So shouldn't we sorry, I.

A

Was going to say one one, other sort of use case that I kind of want to handle whether it's not really a use case, but one other sort of proof of the reference implementation would be. Sorry. Words are hard, so we would have a reference library implementation.

A

I would like at least one reference implementation that, for example, uses like pkcs11, like you know, do like a yubikey or something right so like we would have the library, but we would also have we would pick some standard like pkcs11 and implement a full implementation of that using the reference library.

A

So that way, it's not just there's a tool box that you can use like it'll, be like a complete like end-to-end like if, if you're running, bare metal, kubernetes and you just plugged in a yuba key to the server or, however you get your kms there right, you or probably tpm is probably already on the server. You don't need to plug anything in then you're ready right. You just you just literally need to deploy this one process.

A

It'll, you know: have a unix domain socket it'll hook it up to your kms um through the tpm interface and like you're, ready to go like that's. You know it's going to work.

A

Obviously the harder thing with something like um tpm is really. You know it's not our problem, but you you would have to make sure that the tpm across your aha environment share the same encryption keys under the hood right, but that, but that's a that's a deployment issue. That's not something we can do for you because you know cloud kms does that for you right they, when you provision a key in cloud kms it's available across the globe.

A

It doesn't it's not just randomly like one place, uh but if you're doing bare metal hardware right you, you have to sort of take that into your own consideration. How you plan on distributing that key across your hardware.

A

Also, I don't have anything.

A

Sorry go ahead crystal.

B

I said that sounds exciting, so the previous company we hosted everything on bare metal and having having a set of cubic keys that we would just plug it in would have been just amazing.

A

Yeah I mean I recently had someone from vmware that was running kubernetes on vsphere and stuff and they were like yeah I wanna like so we you know they did the local key base encryption. They were like, and you know, is this good enough. I was like no how's that good enough, your key is sitting right there in plain text inside the api server. What part of that makes it good like you need to use kms and they're like okay, how it's like?

A

Well, you you go, write your kms plug in, I guess they're like, but we don't know how to do that. Also we like have other work to do. We don't have time to implement an entire like.

B

A

Just to like get something done, I was like that's fair. It's fair.

A

So yeah I'd like to have a better answer for individuals, especially you know if you're sticking within uh the well-known standards for tpm's and can messes and stuff right like you shouldn't, need to write, there should be an implementation.

B

Yeah also that I think in has this sgx right, so maybe um it would work also with this so yeah. This is a cool, quite wide white domain. That would be really nice.

A

Yeah with sgx, you know it would be the sort of the best implementation would be to get the get everything inside the uh sgx, but that's a call order. um But you know you could certainly um keep the encryption stuff inside of there and you know have some way of mediating between the kms.

B

Plug-In and such, but basically those are two different concerns right, having the additional layer for the kegs or maybe even something more dynamic like like a keychain or something that depends on just one um coming from the kms and the other. One is the reference implementation that adds a lot of benefits or sugar. On top like the pixie 11 implementation, right, yeah right.

A

Yeah yeah it's. I was mostly just saying that if you wanted sgx, you probably have to do at least a little bit of extra work, but not yet start over.

B

Yeah, because we seriously moved to the reference plantation wanted to be sure that they followed properly.

B

Okay cool, so we then sit down and what write a cap, how to do reference implementation and how we would see the additional um tech or.

A

So what just as a at a high level?

A

um Is there any anything we're like so today we're saying like if we just pretend, like you know, you're on gke and you're using their you know, so their kms plugin is deployed using like a 123 api server and everything is encryption is enabled right.

A

Api server is generating per write data, encryption keys, sending those across the wire getting them encrypted and getting them back right.

A

If we, if, instead gke, had this key hierarchy thing within their kms plugin.

A

A

Is there anything that's lost or changed in some subtle way that we're missing right? The you know. I want to make sure, for example,.

D

um The the only thing that I I want us to keep in mind is that to me, like the the key aspect behind this hierarchy is to have like the kkk that is inside, of the the kms being the main key. So whenever this one is changed by the user or even the kms, because like it can be rotated every seven days, for example, um I want to make sure that this always remained the main key and we never rely on the kkk that depend on the previous uh kkk from the like from the kms.

D

So the only thing that I would want to make sure that whenever a kkk changes in the kms, we are notified uh in the kms provider and we can rotate the kk at that moment.

A

Okay, yeah, so so what shouldn't? I think, what you're saying.

D

You probably need some better.

A

Terms for this, so we have the we have the like the master or not that that's a bad word not using that word. That was not allowed anymore, um uh the the main kek or the true kms.

A

Okay, that's the one that you're saying. If rotated, you want the ability to inform the kms plug-in of this event, so it can throw away its in-memory key.

D

A

Generate a new one that is now detached from the old hierarchy and is now part of the new hierarchy.

D

A

D

So um because I I think, like the the reasoning, is that to me what the users want uh from this feature is to actually like have the the main key always being on the canvas and everything encryption related should be controlled by this. Chemist, like everything like the chemist, should always be the main actor in the encryption like we should never derive like from another branch in the iraqi. I would say.

A

That seems fine, I I so I think the questions I have around this are then: uh what is the mechanism for the the true kms informing the kms plug-in of such a rotation? So we need that. We don't have anything today, but then also do we need a mechanism from the api server side for it to say that it.

A

That it wants to basically do like it wants to do a rotation in the sense of it has previously encrypted data from some some key encryption key or with some data encryption key that was encrypted with some key encryption key, and what it's saying is that it would like to redo that with the latest key encryption key is whatever whatever the definition of latest is because the api server doesn't know anything about your hierarchy. It's just asking for. I want you to be at the latest level right yeah, so I.

D

Think I think yeah I was just going to say. I think the the signal that okay cash change should first be propagated from the kms to the kms provider and then once the kms provider does the exchange. I should also like uh tell the api server that the k, the local kkk change and that if they want to do a full rotation of the data, um so that, like everything, is based on the new uh hierarchy, um then they can do it.

D

I think the event itself should be propagated uh from the kms to down to gps server.

D

And with that in place, we could even like. um So there was a discussion uh last time about like how we do a full rotation of hcd. That's something! That's not yet uh supported upstream, like when the kkk changes currently in the stream. We are not rotating all the data, which means that if a user has their kkk um being stolen or whatever like, there is a lick in their cake and they want to rotate them completely and rotate all the data that is currently in the cd.

D

They cannot do that currently because like they would have to basically generate a right request for each resources um that are encrypted, uh and if we have this mechanism in place, since we won't have like the huge load on the kms provider anymore, we can definitely upstream a code that will do the complete rotation for us.

A

I think I might need like a diagram or something to follow exactly how this would work out, um and maybe I I suspect, to get the most value out of uh this stuff is. I think we would need to go ahead and propose some structure change to what we store on disk right. So I suspect that the the real key encryption key, the the one used by the kms.

A

You know it probably needs to have some kind of unique identifier. That's you know immutable and only changes when you get a new kms encryption key right, so it's you know strongly attached to that and then presumably the local key encryption key like maybe like you know it would have maybe like it's. Maybe it's hash, like its unique identifier, might be its hash, plus the id of the real encryption key right.

A

So that way, that information is there and then that needs to somehow be funneled up to the api server and in and also needs to be present in an easy way within the actual stored data on disk. That way, you know all of this data is sort of flowing through the stack and then you can.

A

You can you can start building mechanisms to stay like well, okay, what was the real? Like you know, what was the real key encryption key is first everything encrypted by that, but then also um uh if there is a key hierarchy, right, like you can kind of try to trace down. Is that also up to date? If that matters, maybe it doesn't.

D

Yeah one thing I'm wondering is like because you you mentioned that we need to change how we store the data to kind of say with which kkk like which local kkk it was encrypted with. um But one thing I'm wondering is like: do we actually want to support um having multiple local kk uh encrypt uh like having encrypted data, and then cd are they? Do we want to go forward with the idea of like whenever the k can change?

D

We will do a full uh rotation of the data that's stored in the city, and so that we would only have like one source of truth that is used to like encrypt all our data, and we wouldn't have to keep like this information of like knowing with which key the date. The data is actually encrypted with.

A

um I'm not sure, because rewriting all the data in ncd no matter what would be expensive in terms of I o um and- and you know I want to get to a world where it's like a good idea to just encrypt everything, not like a really dangerous operational decision, yeah right. So for that to be reasonably the case, you know you should be able to do rotation of your. um You know main key encryption key.

A

You know every week every two weeks, whatever your policy is whatever you're comfortable with yeah, but not also, then be terrified that every two weeks, like your your cluster, is basically going to be dysfunctional for like an hour, while it's sitting there re-encrypting all this data.

A

Now you know it there. You know there is still some.

A

Like I'm unsure about like what this ends up being right, like this, like so there's, probably some limit to how much the local key encryption key should be used right and each each aha instance of the kms plugin should have its own set of local key encryption keys right right. Obviously any of them can decrypt data because, no matter how they are presented with the data, they can always ask the real kms for help in decrypting a local key encryption key that was originally created by some other instance of itself right.

A

So that's totally fine, it's just there, but there needs to be some finite limit to how much you use the local key encryption key right, um and it might just be that we, it might just be an arbitrary, like small number like we might just say that it's only used for like uh a thousand encryption events, because we know that's perfectly safe within any bound of any modern cryptography.

A

But also it really eliminates most of the cloud kms issues right because it will make it so that you know there's a you know, a three order of magnitude, difference and load against your your your upstream kms.

A

It might it might not even it might be like a hundred thousand, or I don't know like some like it, it's probably a number between ten thousand and less than like a hundred thousand. That is perfectly reasonable.

D

um One thing I'm I'm just wondering like: has anyone seen um like a scenario where a user would want to do a full rotation of the like the key that isn't kns, because, like some of the previous key got leaked and they want to do a full rotation of the data in the cd based on the latest?.

A

I mean, I think, that's a totally reasonable thing for someone to do right and it might not be based on any like belief of compromise. That might just be just good security hygiene right like yeah or you know, in in openshift, when I wrote the encryption controller right, it's basically, I think hard coded to just rotate every thing every two weeks or something like that right, not because I expected the key to be lost every two weeks or something he just does it like.

A

Why not um I mean it was to partially prove that I wrote all the encryption rotation logic right right, just by actually running it over and over, and if you still have a functioning cluster at the end, you know it's working.

D

So I'm wondering like how we could maybe introduce that to upstream um I mean this mechanism, full rotation.

A

Yeah I mean I, I think what we need to first start off with is um like.

A

Basically, what are the changes to the grpc api to funnel extra information through and then what are the changes to the storage format to store, said extra information and then then, once we have that, and we don't have to worry about the format exactly, we can just say that somehow we're going to store this new id field within this, the stored disk, because we can come up with any it could just be json for all that matters right I mean that's a bad idea because it's verbose, but it could be so once we have that, then we can start saying like okay.

A

What kind of component would you need to orchestrate a rotation right like can? Does the component need like read level access to raw fcd data? Does it or can do it through the kubernetes api right? Is it part of the kms plugin, or is it a kubernetes controller? What is it or is it a coordination among all of these actors, but I don't think we've.

A

I don't think we have the foundation yet to really ask the question of how would this work, because within openshift right, the reason the encryption rotation worked is because the api servers are self-hosted. You know like I would probe them for their state, so I could tell like how far along they were.

A

I could look at their configuration with their static config to be like all right. This is the key that you have by definition observed because it's on disk- and I can see that you read it- those types of things right so like each of those assumptions, has to be sort of pried out and turned into like a generic mechanism or like well. How do you know how far a particular pi server is.

D

Yeah right, um it makes sense like to wait until, like we have the actual mechanism of like funneling the information that the kkk changed and store it into hcd. Before actually thinking about this kind of problematics.

B

I think that the rotation is really important. It's not just good hygiene, because I mean today we most tls is with all the encrypts right. So we use new keys every time and we don't reuse them anymore, because back then it was numbered to re-use them a couple of times. So once the key got leaked, you could kind of decrypt the whole conversation and um this whole rotation can be a huge mess if you think of it too late.

B

So in my previous company, we had this whole mbap, encryption and figuring out how to do the rotation properly without any hassle uh was a huge mess. So so, basically, uh it caused more cves than benefits, because the user sensors never really use it. That much, and I also agree that it would. It generates a lot of float on lcd if you would also want to re-encrypt everything, um even the the dex um for us. For example, we sought medical data and the dex was always stored in next alongside to the medical data.

B

So if some suffix of some of the decks were leaked, the assumption was almost also that that did that it was already encrypted. So so we never reported all too much with decrypting the re-encrypting. The data itself again.

B

But again, this discussion is super hard to do without kind of a board or a common language that we agree on. So we had also specified kind of a kind of a notation how we, how we could write down what we mean by saying: okay, this keg encrypts this keg, and this kick and grips this deck, and this is and because can we become very confusing quite quickly.

D

Yeah, just to go back to your point of that, um this kind of mechanism like having a rotation mechanism in place, will put a lot of load on its cd. At some point, um I think, like the problem comes down to whenever, like we want to improve something in terms of security, uh there are always some drawbacks in terms of performance or whatever, and I think uh if you want to do a full rotation, if it's like something that you want to do every other week or like just once in a while.

D

I think just having 30 minutes of load on your lcd is something reasonable. If you really need to have this important part of security inside of your cluster.

B

I would think think so too, but um the the camera topic arrives. You know like in october last year and the future extreme from stefan was like, oh my god, how much more performance does it squeeze out of that cdb? I haven't already the beast on max slope. We don't have much to spare so so I've got always a thing that lcd is already reaching its limits of capacity.

B

So this is why I kind of uh assumed it's the same.

D

In openshift we're already doing this for rotation, so it's fine, we it! It won't be worse than what it actually is. So for us, it's fine, uh it's more like, but the other users like what would they expect uh if we want to have a mechanism to do for rotation like they have to be aware that it will put load? It's completely normal, but it's a drawback here that you have to embrace. If you really want your encryption to be secure from an end-to-end perspective,.

B

And also the future um outliner to maybe encrypt at some point. Everything address is also nice, which would be nice to know. Maybe it's a future task to keep this in mind.

C

Good, I think we need to spend some time on this one like basically like writing out a design spec on how this works.

C

I think, if we actually write it down, then we can also consider what different scenarios we need to kind of have right, like I think, rotations, one really great use case, but then I think it'll also help us visualize what we are missing missing if we move to this new model like how do we make it backward compatible like when game is plug and start using this, how the data that's been encrypted from the keck in the external store? How would that still work?

C

And all the scenarios like, I think all of us are in agreement- that this is probably something that we want to do like.

C

This is the model that we want to do the hierarchy, so maybe the next step would be for us to actually just dive deep into it and then maybe just write out a design spec on what the data would look like like what are the things in addition to what we have today, that we want to store in net cd and then, like just start building from that, like use that as a building block and then just keep going from there.

D

Yeah, I think that definitely sounds like a good plan, uh just focusing on like how we want to kind of implement this hierarchy.

D

uh What are the design changes that we want to make to the api and then really focus on the like the potential aspect that might be impacted so such as rotation, as you mentioned, such as, um like backup, opportunity and stuff like that say, focusing on that at first will help us kind of figure out like how it will impact the other aspects of the future.

C

Does that sound good now.

A

Yeah um do does anyone want to take an axe? Man do like oh, do do some explicit fleshing out, maybe in this dock of just uh like explicit mechanisms we might build, or you know new data. We might want to store this stuff like that.

A

So that way we we have something and it might it might even we might even want to first start with, like use user stories like what we expect, the final state of the system to look like a little bit more strongly, um and then we can try to address like which of those can be handled by what we're discussing.

B

I would love to contribute to that topic, but I don't know if I would be able to do it on my own, because I'm still kind of on boarding myself on on the details in communities.

B

So I would love to help someone.

D

Yeah I mean we can always all have kind of a look uh at how that could look like and then seeing during the next meeting about.

C

D

um What are the approach that we suggest? uh What is the possible like look of the data in the cd, how the data would look like that? I mean it's always fine to cross different ideas and implementation in that regard.

B

Okay, so the current action item would be that either we we individually create pairs or we do it on our own trying to sketch something out, and then we see kind of in this. This brainstorm mode like like decoupled or asynchronous brainstorming, that we come together in the next video and try to show what each other has and trying to figure out. They were forward right.

C

Yeah, I think that sounds like a good, short-term plan right like because we haven't synced up next week. So if we all have something, then we can come back with that and then like.

C

I think we had a good conversation today based on what mo wrote right and then, if we actually have something more then it'll be easier for us to just go through the conversation and see like if there are any downsides and if we can actually make this model work, and I think that we could do that in a couple of sync ups and then once we think like this is what it is.

C

uh Then I mean we can start writing an actual doc, which will have the exact design, and then we can just make that again.

D

Yeah yeah, that's what makes sense to me.

B

B

Okay, so ask my former colleague if, if the the encryption annotation that he created back then um is open source, so so maybe I could show it how beneficial it will. It is because, in in design um discussions with office issues, which do you mean which deck do you mean this is the one from the users is the one from this.

B

So um maybe I could also pitch that annotation, so it sucks able to get into it, but once you entered it super clear: what's meant.

C

Yeah and also in this dark, I think, like we've added these extra extra metadata here and there right like we had the kick in the observability too, like where we said. Maybe the game is plugin should written the kick so there's some data around what key was used to encrypt it and all those. So I think it's just basically moving it around and then just having one common thing where we say like what are the things we need and uh let's go from there, but I think yeah.

C

That sounds like a great plan like by the next sync up like we can all just see like I mean, try and come up with something, and then we can have show that and discuss and then once we're in agreement. We can actually convert that to a full fetch proposal and then just make that a kept.

B

And the end goal is to to create to create a cap, but the next cap will be for next release right. So there's not so much of a hurry right, I mean to be fair.

A

My tape wants to write like a cap that is sufficient for this. So, like don't, don't think that we have that much runway.

B

Yeah definitely but, but I think this is the most crucial phase uh where we might have had a bit more leeway, but once it's it's we agreed on something. Then we can um well if you would be in one building, one room it.

B

I would say: okay, let's take a day and bring some of that topic, find out all the solutions scheduled out, maybe discuss it on the full update and then we would be done but remotely it's kind of hard right or do we have babies, or someone of you has um uh heart with which we could have such a kind of white boarding session.

C

I can do friday if you have time uh if you want to pair on, like I think I don't have a lot of meetings on friday. So if you're interested I can pair and then we can brainstorm together,.

B

I would love to, but you friday is my friday evening, which is family night, so.

D

We can do that together.

B

Yeah definitely because because nobody see you based for me, it's always like um through discussions. I learn a lot and realize how few things I know- and this is why why I kind of shy- which I way off the super, to doing something completely on my own, because I I would be worried that I'm going down a rabbit hole and, at the end someone says yeah, but your first action, uh your axiom, um your assumption was wrong and then I wasted a lot of time.

A

I gotta drop y'all can keep talking. I gotta go.