Kubernetes WG KMS, 18 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WG-KMS Bi-Weekly Meeting for 20221018

Description

WG-KMS Bi-Weekly Meeting for 20221018

A

Okay, so hey everyone, this is the KMS plug-in meeting uh for cigar. This is the 19th meeting in this series for October 18 2022.. uh Before the recording started, we were discussing uh initious PR in relation to uh changing from a lru cache, with a maximum size to a expiring cash.

A

That's based on the TTL of the keys and in the initial review. One of the concerns I had raised is: is there a way to bound the maximum size of the expiry expiring cash to doesn't have to be small? It could be as like something like 100 megabytes or something it just just doesn't shouldn't be infinite.

A

um Is this is kind of the feeling I had, um and so we were discussing if we could rework the cache key logic to instead of using the actual?

A

um uh What is it? Does it use the encrypted deck as the key yeah yeah, so it uses the encrypted deck as the key um and we were like or I had made the suggestion. Maybe we could try to use the FCD path instead, though I'm.

A

Thank you. That would mean that if you had multiple like rights to the same place occurring that you could get, some cash misses I think during right like unnecessary cash missions. Technically during that, because.

B

Multiple rights should be fine right, like every time. There's a read: it's going to get only one Transformer, which is from the latest right.

A

Yes, uh well, I guess it depends right, so the transform the the part where we do the call to KMS that occurs before we actually write the bits to SCD right.

A

uh And I'll I guess that's not actually it's not guaranteed to be successful. You could get a conflict right before you finish doing it right.

B

So only after the KMS call is done. We do it right. So if the gamers call actually errors out, then we are not writing anything to hcd. We are not writing anything to the chat.

A

Right right so say say that you have multiple rights and you try to uh at the same time, so one of them is going to get a conflict and the other one's not going to get a conflict. So the one that got a conflict also called KMS and also did encryption and stored. Something in the cache.

A

Right and with an expiring cache, that means I will just stay around for 24 hours, even though it's completely useless.

B

A

um Yeah so I think that's where the previous lru cache's Max size was really useful, that yeah it was wasteful, but only if you like, like only if it really like uh mattered right like you, it was, it was bounded and it would it would roll off the buffer effectively.

A

At some point there wasn't a big deal that that could happen, um whereas if we change the logic to be purely path based, then you could actually end up overwriting a key like a uh an encrypted deck uh and it might not work um which, obviously, all the code could handle this right. It could be like hey. Let me get it from the cache and if the key, that's at that path doesn't match it. Well, fine I'll, just I'll, just go, call out to KMS and get the get it to decrypted uh thing anyway.

B

um Yeah like what you said, it's just like some additional calls. Maybe.

A

Yeah I guess I, don't I, don't I'm, not 100 sure. If the trade-off is correct there, but I do think it would effectively bound the size of the cash.

B

Using the length right so basically, you define the length and say like these many entries at the max and then, if it reaches that length, then.

B

But even in that case still we are not garbage collecting. If we hit the length we're just saying like Okay, the cash is full. We cannot add to it.

A

No, no so when I was thinking so what I was thinking is, if you were using the hcd path as the key, then that means that the maximum size of the cash is the is. This is the size of how many SED Keys. You have pretend that you encrypted everything just for simplicity's sake, Nobody Does that today, but if you, if you encrypted everything the maximum, is like the number of objects in SCD right right, all intents and purposes means the maximum size for most users today would be the number of secrets in NCD right.

A

So that would mean that you're effectively limiting the cash size to probably like 100k or something.

B

Okay, yeah, okay,.

A

And so, if you're remaining it to 100K and what is the are these like 32 bits, each all right, so you know so 3200k I forget that that's some amount of megabytes, probably it's either 3.2 or 32- I'm, not doing math in my head right now. uh uh But that seems okay right if we can make it so that the I I guess having a cash Miss is not just it's not a big deal.

A

We we would want metrics on that to see. How often are you like, uh like I, guess, there's two types of cash missions. One is literally just not in the cache and the other is it's in the cash but you and but when you tried to use it, it didn't work right.

B

Yeah, so do we want to distinguish between those two to see how often we have it in the cache, but we're missing.

A

I think so, because if we, if we didn't distinguish it, we we might not be able to make a good assertion about if this was like a good trade-off that we made right because we're what we're basically saying is hey. We want to use this different um uh caching mythology. So that way the API is easier to deal with, because the user doesn't really have to come up with some arbitrary fixed size that honestly I've never seen.

A

Anyone configure anything but like a really tiny number like what's the biggest number I can type into this field before it like overflows the end or something just because it's like I, don't like it's basically saying I, don't care right like it's like I, just want to cache and I. Don't want to call my KMS plugin right, um so that was the API I was hoping that we could kind of build out like hey, you know under like normal operating circumstances. You know you're very rarely, gonna ever need to worry about this config.

A

So let's just do it for you um and because we're gonna make it so that rotation stuff works correctly.

A

You're not you're not giving up any security properties of needing to talk to the KMS really because if the camera says hey, my key has changed. You will notice and ask it to help in that. In that scenario also, so that I we still get rotation correctly, um so I guess just I'll poke at that see if that makes any sense, yeah I'm gonna say since Mike was the one that suggested. The change I would also try to rope him in, like maybe pay him on, so I can be like hey.

A

You want to take a look at this and see if you have any thoughts that way, um we can get some feedback from him early too, but he because he might have had like a completely different sort of design for how to use that cache than what we're thinking about.

B

Chad, let me think of him. Yeah.

A

So I think so that's that's the I guess North Korea. Do you have anything about this particular thing.

C

I'm still curious about the texting part, but.

A

The testing part for the cash yeah.

C

I mean I'm I'm curious like how how we can implement it.

A

Yeah I mean there's I, guess what we could. We could certainly try to expose some some helpers, that we could call it integration tests that, like let us assert, like maybe the size of the cash and things like that. So that way we can be like you know like we could write effectively a stress test that, like just shoves, an enormous amount of data uh across a uh like like if we did like a bunch of rights to the same key.

A

We could assert that at the end, the cash value is still basically won, uh and then we could do a bunch of Rights across a bunch of different keys and then assert that the size is well. It doesn't matter like you know, it's still just the length of the different set of keys. It's never like out of that.

A

Well, we could at least get those kind of semantics out um and I guess that, combined with some of the existing tests that try to say that hey just turn the plugin off and then move stuff does it still like do reads still work. um You know those still give us I think some confidence yeah.

C

um I mean, as they say, like invalidating caches like harder problem to solve yeah.

A

Yeah I took a CPU architecture, scores that they discussed some of the like fancier cash, invalidation, algorithms and, like it just made my head hurt like it was just absurd like and like my teacher was like there's not actually like a formal proof that proves this is correct. It's just like we think it's correct. I was like that is not good enough.

A

Y'all like this is the thing that does all the Computing I need the thing that does the Computing to be correct, um true, so the the other thing that I just remembered uh and like this is one of the like the many com variant. One of the many comments I put on your hot reload PR is I believe what's going to happen today. If you hot reload, is you'll, throw away the edac cache.

C

A

That's that's the cash inside the Transformer that we're just talking about the one. That's not lru! That will. Oh.

C

Correct but that's fine right.

A

I mean it's not good,.

C

Yeah I mean I mean, but why do we want to even maintain the older one right.

A

Because that would mean what would that would mean that every time you hot reload, every read after that's gonna call KMS to refill the cash.

C

A

Every read that doesn't hit like the watch cap acity, invest.

B

I mean in terms of the feature like it is. It is same as today right like today, when you change the encryption config you restart API server and then API server has to again warm up its cache.

A

Yeah so I could buy the argument. That's no worse than what happens today, but I I. Think. The subtle difference is that um when you were manually starting the API server I think you had. You had more control on when that happened.

A

Right, like you, could you could make the argument that you would uh the way you would do your API server restart? Is you would bring up a new API server within the new config up? First, let it become healthy before shutting an old one down, basically to make it so that the the warm-up of that cache was hidden from users now, because it's going to happen on a running cluster.

A

uh That means, if, if you ever had, if you were ever changing the files on disk it'll notice and immediately reload and immediately throw away its cash.

B

So is there a way for us to retain the cash like I haven't, looked at the heart, reload PR yet, but if you can retain it like that will actually be like an added bonus of doing heart, reload right.

A

Right so that that's one of the things I was thinking about is like, should we change like the Constructor of the load encryption config to like pass in the cache uh like whatever cash is right and it'll be like?

A

Maybe it would just be shared for all, like all KMS plugins, like they'll, just be just one big cache, because it doesn't matter right, they're, just there's no like because the API server controls well, it might be weird, but there might be some weirdness there, but you know you could you could imagine that uh you could pass in like I, don't know a function that gives you a cash right and so that that way the caller could track. If you called it and like how many copies are whatever are out there or whatever else.

A

I'm: okay, if we don't do that right now, like I, don't think that has to like block your work. No, like you can just say that what we have today is good enough and we're still making we're moving the ball forward, but it might be a good thing for us to basically write an issue for and have it in our in our KMS backlog.

A

If you're like yeah, we want a hot reload, but like not throw away all that hard-earned memory just because like right just because it was like hidden under all the all the all the other layers right, like the reason, I I only noticed this as I was like going through. Your PR is I, was in the state of mind about thinking about initials, PR and then Europe, PR together and then I. Remember that inside the Transformer interface, oh yeah, there's this cache I care about a lot. Because- and this is like.

C

And that's actually good like we are uncovering a lot of edge cases yeah, but yeah.

B

C

Effectively so for the hot reload uh we did a API rewire so that actually nullified our work earlier or my work earlier. So uh I did the rewired stuff again yesterday and get it got it to the working State uh and more also added some initial PR comments. Thanks for doing that, so couple other things we want to get in into that pair.

C

So, for example, you want to make sure that when we are doing the hot reload, the Transformer swap is atomic like when we are swapping the older ones are not getting used, and things like that so and then like draining the old connection, so I'll try to get those things done in this week, so that we have some time.

C

Or we can get the feature in basically.

A

Okay, um yeah, so yeah I think you still have to figure out like how to do all the tracking stuff right. That's like I think.

C

Yeah I'm gonna work that on today.

A

um Also I'm trying to remember uh oh one of the comments I've made on your PR, which I think you can basically ignore right now, because that's kind of the state I ended up at is you you did the really well, you copy the really fancy file watcher based stuff.

C

A

I was like: do we really want this level of fancy like no.

C

A

Expected, like a simple pull, Loop right.

C

I I did I did have a simple Polo, but then uh uh it was keeping the file Watcher on infinitely. With this approach you we are, we are also even terminating the uh the file watcher.

A

Because you have a stop channel, so you would always know when to stop polling.

C

ah That might be true, but then also like work user I mean this exact same thing. I've done in that I've done many times for writing controllers.

A

Yeah, so I I think I I think it's fine, it's more of like I was just like like this is not a fast moving. Config I! Guess it's not like you.

C

A

Config, a bunch and so like, like when I, was looking through, like how the fs notify stuff is wired up which I realized. You know it's just copy pasted from a dynamic certificates.

A

I was like how does one actually assert that this thing is actually correct, because it's basically built on a bunch of os specific implementation details and then there's like four or five cases where it's like huh just go ahead and just add to the work you just in case. Something fails and I was like.

A

You have tests for like when this might fail like what? What what like? What is this like, you're, like you're, like guarding against so many theoretical failures, I, don't even know which one is a real possibility versus like something else right.

C

And I see what you're doing the.

A

Way: um okay, so that that part is like weird, but maybe okay, um so yeah I think it's fine to leave it. It's just uh I, just wasn't sure if, like the complexity, really was worth the trade-off like because like if you change your encryption config and it takes five minutes for it to reload versus it, takes like um like a few seconds to realize.

A

I, don't think you really care because, like you're gonna, do it like once a couple of weeks or something so I, don't think it really matters and at the end of the day the whole point of this is that the state would get funneled into.

C

Hold on one second.

A

uh What was I saying uh so would this state would get wired into like the storage version stuff so like there would just be controllers looking at it anyway, and they don't care if it takes a few seconds for somebody to change versus five minutes, because they don't actually know it just happens, so it just happens as a watcher and they're just gonna see it and do it.

A

um But you know you got it. So it's fine, uh no big deal there, um let's see so that that uh Rita's, not on the call anymore so I can I can get a little bit of update on well. I guess you know, like did you have other stuff.

C

Oh, no, no! That's it so I'm, just sort of focused on this thing so hoping to complete this and get ready.

A

Okay, um so on Rita stuff I have I, have reviewed her PR I think it's like 100 ready to go uh and the tests you know prove that it's working, because if you run the test against the head of Master, it shows that custom resources are not like. It fails, saying: hey this custom resource was supposed to be encrypted, but it's not uh so that part's really nice. So you know we got. We got good test coverage. Everything is good there, um so I think she's good, so I'm gonna I was I.

A

Just opened up the board real quick, so I was just gonna scan it real fast, so I I think the Wild Card stuff that you know like you have I think we're just gonna end up, probably waiting until 1, 27 I, just I would be really surprised if we get that far just because of how little time is left, uh but.

C

You know don't.

A

Don't wait to do the work because it'll it'll show up sooner than you expect it's just you know. Whenever you get to it, it'll be great Anish. uh You still have the issue for uh multiple um uh multiple KMS. Private providers can be configured with the same name. So we want to. We want to fix that. Yeah.

B

I'm working on that so I think I can get a PR out by tomorrow.

A

Okay, cool that'd be great um and then I will probably not probably not work on the staleness stuff soon, because I think uh the next thing I'm probably going to focus on is the the storage version support for custom resources.

A

It's probably the pr I'm going to look at to try to revive um so that way that that stuff is moving forward.

A

um So, let's see like you're working on hobby load- and we just talked about that- uh let's see in this- you still have the uid logging stuff, but I'm I I think that's to me less priority than like the validation and the expiring cash stuff. So you know whenever you get to it. Yes, it's fine.

B

A

uh And then the big thing that an issue and I must do is we must review Christophe or friends of mutation? Yes,.

B

But I will definitely look at it today, like I, think I have time in the afternoon, so I'll vlog that time and do a first review.

A

Okay, good that'll be great, so we can get something in. So let me think so. We we definitely want Kristoff's stuff in 126.. We want. We want High, reloading 126. We want read us stuff in 126. We want the seat, I, I, guess if we didn't get the expiring cash change in 126, it's not the end of the world. I do think we would want the validation stuff, though, because that's the one that basically has like a you know a release.

A

Note action required bit attached to it, so we want to get that out sooner okay. That way, if people get mad at us, they have. We know sooner rather than later.

B

Okay, okay, make sense, I'll get that appear out.

A

Yeah so yeah I, I, think yeah, so my I I guess I have two primary focuses before code for each one is um the reference implementation and the other is the stuff needed for um those other two caps.

A

um Some we've made some progress like we. We got some of the Caps updated um and there's some ongoing discussion about like having a stable identifier for API servers versus not um so.

C

A

Think that that is ongoing, but I think fine, I I think the biggest change, though, is making it so custom resources support the stuff, the storage version stuff at all, because today they just don't um so we need to get that sort of sorted uh because I don't want another bug which is basically hey: custom resources, don't support X related to encryption because yeah like yeah, you, you added uh support for custom resources, but now you can't you can't do rotation because yeah correctly supported okay, um I really can't win tonight. Oh.

C

A

See and anything else that we have going on.

B

What else is required for the Apes, our identity, apart from the State, the ID description for API server? Is there any other implementation changes or test changes? That's required.

A

So I think the ID thing is the main thing, so that main thing will either so I I can see like three possible options. We could. We could make it so that the ID is random. We can make it so that the IDE is configured by the admin per API server. So it's something that has to be different per binary uh or we could make it so that it's optionally configured by the Admin. But if you don't configure it, it's a uid, it just gets to it.

A

I I kind of lean towards the middle in the sense of like either make it random or make it always configured, but not necessarily both but I. I can I'm, not strong.

A

Being honest, I just think it's I I think if you want to give the admin the ability to control this, and you should like just make, do it like just just make it like a thing they do so they give it like a nice name or something instead of like you know like what will end up happening is like then like unkind clusters, you'll see like uuid spaghetti but like on, like AKs you'll, see like the number one or something right.

A

um Well so I just but I, don't think that's a big deal. um I I think uh Andrew Andrew Kim had opened some um tests that that he added for some of the controllers, so I think I. Think the biggest gap for both of the things is that there's just not enough test coverage both unit and integration, I think on all the different pieces, um but I think the main feature for the ID thing is just actually deciding.

A

Oh there's there's some CLI flags that are currently configurable for, like how the leases, work and I think we were discussing just removing the config and just hard coding it. So that way, it's like I, like I, just configuring, lease durations and like how often the API servers pull, is just asking for someone to screw it up and instead of just saying like this is how it always works on all API servers and like this is just a property of how API servers refresh their identity right.

A

um So I think that's the main thing there and then the main thing on the storage version thing is making it so that it works correctly for custom resources, driving uh and I I I have the pr open. Well, okay, actually uh take that back.

A

The other thing I think that's sort that might be semi missing on both of these, because I think they might need some more metrics, um but uh I have the pr open that was immersed, that added the custom resource or was trying to add the support for custom resources, and that PR has like hundreds of comments, oh wow, or is this in a certain amount of comments over like a long period of time like from Jordan and David and Daniel, and like basically like everybody Under, the Sun I was like God I, don't know?

A

What's actually I'm gonna have to like read the whole thing from top to bottom, like slowly as like, you know, go through like well, let's see what like they discussed on this really long thread and then I have to actually go read the pr and something and what the code does uh so yeah. That might be. Am I that's going to take some time.

A

I've been I've been semi-putting it off it's only it's going to be so painful and I don't want to do it, uh but I have to do it because we have to get it all in uh in like three weeks um for this week and yeah the week after keep calmness, I I don't understand why they would put code freeze right after kubecon, which is too late, is causing unnecessary stress, yeah, um so uh yeah I I mean I.

A

Guess you guys still have next week since it's just like Rita and myself they'll, be there they're working on this stuff. So, like you, you guys will still have that time to work on this stuff. Yeah.

B

Yeah and then, if there's anything else, that you need help with the other two tips in the next week, I mean post on the slack chat or the event. We can help out too.

C

Yeah, that makes sense.

C

Okay, any anything else.

C

I'm good I think that's it.

A

All right cool, so I guess we can have you know 13 or so minutes back and get back to all the work that we just talked about. Doing. Okay,.

C

Cool. Thank you. Thank you.