VMware TGI Kubernetes, 18 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: TGI Kubernetes 132: Sealed Secrets

Description

- 00:00:00 - Welcome to TGIK!
- 00:04:31 - Week in Review
- 00:23:15 - Sealed Secret Overview & Install
- 00:37:21 - Sealing a Secret
- 00:52:49 - Retrieving the Public Key
- 00:55:28 - Exploring SealedSecret Scope
- 01:00:00 - Rotation: Secrets and Keys
- 01:15:37 - Multi-Cluster Sealed Secrets
- 01:30:28 - Wrap-up

Show notes up at https://github.com/vmware-tanzu/tgik/blob/master/episodes/132/README.md.

Come hang out with Joe as he does a bit of hands on hacking of Kubernetes and related topics. Some of this will be Joe talking about the things he knows. Some of this will be Joe exploring something new with the audience. Come join the fun, ask questions, comment, and participate in the live chat!

This week we are going to do a deep(er) dive on Sealed Secrets (https://github.com/bitnami-labs/sealed-secrets). This is a way to encrypt secrets in a way where you can check them in and manage them with the rest of your gitops-ish workflow. In this deeper dive where we plan to peek under the cover and look at some of the day 2 things like key rotation.

A

Hello, everybody, hello, hello, hello and welcome to tgi kubernetes. I am your host today, joe bida, uh for those who are not aware, tgi kubernetes is a uh weekly ish actually been doing really well trying to get it done almost every week, uh weekliest youtube live stream that we do here uh as part of vmware tonzu, where me, or some of my colleagues like josh or you know, hopefully we'll get some more folks on board like we did last week, uh show up and really help to talk about.

A

What's new in the world of kubernetes, and then we dive deep into uh you, know some projects, some technology, uh sometimes we're experts on it and we're really sort of putting it through the paces for folks other times we're not really as much of an expert on it and we're.

A

um You know discovering it with folks making mistakes showing how it all works together, and uh this time we're a little bit in between so we're gonna, be uh the plan for uh this episode is to dig into this project called sealed secrets, which I think is really really cool. um So that's what tgi kubernetes is. We are now on episode, 132, which is absolutely crazy.

A

um I've been doing it for I guess years now, and here we go and somehow you know a lot of you all are still with me as we're actually playing around with this. So uh you know, first of all, hello to everybody who's joining us. um I am coming from you from seattle.

A

uh This is my office, overlooking usually I can see, uh get a nice view, but for those who haven't been paying attention here in seattle, we have a lot of smoke from all the wildfires on the on the west coast, and so the air quality is is actually pretty crappy these days I'll here, let me grab this I'll. Show you what I'm dealing with I I now have.

A

I now have an air quality meter that I bought on amazon. That shows me sort of what the air quality problems are. um The green screen is confusing it a little bit, um and this is telling me that currently in my office, the 2.5 nanometer particles are 37 micrograms per square meter or cubic meter, which is not good, and so now I know how bad the air is in my office, but I actually don't have a filter, and so there's like nothing. I can do about it.

A

So I think you know around here we're just waiting for rain so that we can actually start clearing the air a little bit. It's been pretty rough. Well, let me start off by saying hello to everybody from around the world. I you know, I really love the fact that I know this is not always the best time uh for folks around the world to be able to join us, but you know kick back grab a beer hang out uh and we'll explore some kubernetes stuff together.

A

So I want to say hello to to to steve who was here before everybody else saying: hi, uh martin from the netherlands uh sevi saying hi, also waleed good, to see you uh josh, who is a a regular host now uh saying, hi he's been doing quite a few of these things: uh rory bianca hello, ahmet mona, calling in from germany joy from richmond tim, which has a cool tim. How did you do the little like cat emoji?

A

Is that, like one of the standard emojis, I don't know that's pretty cool jim first time from manchester thanks for joining us anjuka from helsinki, uh amim paul from new york and st louis uh sevy, victor from new york uh rada, good to see you, carlos from miami uh very cool heim from israel, israel, tomas, philip from germany yeah. This is just crazy, morteza from tyrann good to see you uh gokhan from turkey.

A

Awesome welcome, welcome, welcome um and and mas emiliano. I just want to say it I mean my I'm sure my italian accent is horrible call you know joining us from italy and then deepti from vancouver. I'm sure your error is pretty bad up. There also uh good to see y'all yeah, 132. wow, um so one of the things that we like to start with, as we get going is, uh is we have some notes and we have a sort of a crowdsourced document that you can all join us at in the the url here.

A

If you want to join us, is tgik dot, io, slash notes. Let me put that in the chat here for folks, okay, dot, io, slash notes. um I was gonna, try and put it on the on my standard sort of breakdown on the screen here, but uh I started doing it and and obs defeated me in the 10 minutes I allotted before the uh before the the live stream here so anyways you can join us here if you want to put some extra notes in there as we explore stuff.

A

If you want to do some time code that helps me out later, um as I as I go through and uh uh uh get this prepared so that we can get the nice indexing in youtube after the fact that'd be helpful, but let's go through sort of news of the week. What's been going on, and thank you huge shout out to george, who is always super helpful here uh uh to help curate these links make sure that everything moves smoothly. Thank you.

A

So much george you're uh you're a huge help, don't know what I do without you, so um so core kubernetes uh the 1.20 release cycle. It's been a pretty light week, but the 1.20 release cycle is actually started in the and the release manager. This time around is jeremy rickard. So you know thank you so much jeremy for taking one for the team. This is a big job and I think a big part of it is executing the release processes for kubernetes itself, helping to make sure that things converge and then, along the way.

A

You know really sort of leaving the campsite better for the folks that come next right really really want to make sure that every release goes smoother and we automate things and we make it smoother one release over the other, and so you know we're on you know: 20 plus releases. Now how many releases has it been? Because I mean there were a bunch of releases pre 1.0 too, and it's always a big job, but hopefully it's getting better.

A

uh You know over time you can um so you can go to the kubernetes.dev and actually see all the information about the release. What the timeline looks like who's leading it. This is hugely great information to understand, what's happening with the release- um and you know big shout out to george and a lot of the folks in the contributor experience sig, who put together the kubernetes contributor website, where we have some place like this to bring all this stuff together, which is super super cool um and uh yeah.

A

So all of the uh you know all of the the information around the release. That's going on here so great way to track. What's going on there very cool all right. So the next thing here is that we have a cve that folks should be worried about.

A

This is a cv in the linux kernel, um but it's something that will impact a lot of folks that are dealing with containerized issues, and so the idea here is that memory corruption can be exploited to gain root, privileges from unprivileged processes, and so the idea is that you know this could be used. I think the fear is as a container breakout.

A

I haven't taken a close look at it, but um sysdig has a has a blog post here. I will accept cookies um around what's going on, um and so one of the things here is that the uh it looks like the mechanism here is through cap net raw, and so this is uh the uh the capability in the kernel that lets you essentially send raw packets would really spoof things on the wire. This is generally not a good idea that some this is generally not something that you want to give to containers.

A

uh uh There's there's really only very few places where you're really going to want to use this, and so, if you're, using pod security policies, uh you should really be using those to drop drop. The cat, capnet raw or you know other mechanisms to make sure that uh containers as they're launching are not actually looking for this, um and so it looks like it's a incorrect offset calculation uh using packet sockets, and so it's a relatively you know, advanced kernel, feature that most workloads won't actually need to need to use.

A

So something to be aware of, if you, if you're, not using pod security policies, if you're giving capnet raw to a lot of stuff, then you're going to want to make sure that you upgrade your kernel with uh with the patched version and uh and work through this.

A

um Let's see, we have a 1.19.2 point release, um let's see, uh looks like uh conversions for custom, metrics uh panic in cube control debug when the pod has multiple init containers or ephemeral containers, and so it looks like nothing super groundbreaking here I mean clearly, if you're, hitting that it's bad, um if you're using custom, metrics, uh there's probably something important there and then and then update the cni plug-in. So this is just a release thing, so nothing too outrageous going on there uh so good to know. That's uh it's good! It's always good!

A

When you look at the changes and you're like oh no big surprises there, um and then this is something that I think is really really cool. That I'm super excited about um is uh there's now a capability in kubernetes and here's a blog post on it for returning warnings, and so this is one of those things where we'll do things like deprecate some apis or find that somebody's using something that like we're like hey.

A

You know what we're going to accept this, but we really think that you know this is going to change or something's going to be happening over time, and we really didn't have a way to. um We really didn't have a way to actually return those signals to users and things like cuba, as they call things for cube control, and so this is a great example here of like now.

A

We can actually go ahead and do this and, if you're using say the wrong version of ingress, we're going to say, hey, you know it's deprecated and it'll become unavailable in this future version. So this is really a great way such that these deprecations and scheduled removals, which we do multiple versions in advance right, so we're looking at 1.22, which is probably not another, six or nine months from now.

A

um You know we want to make sure that uh we give folks fair warning as this stuff happens and it doesn't sneak up on them and we really haven't had great mechanisms to return it. So I'm really excited to see that we now have these mechanisms for returning warnings, and so, um let's see so uh and then there's ways that you can actually there's metrics for showing you that hey you're making a request to a deprecated api.

A

uh You know you can get the list of deprecated apis and understand. What's going on, there super cool going on there, and then uh it becomes an audit annotation.

A

And then this also applies to crds, which is super cool also, and so you can say, here's the deprecated warning, like you know, hey this this uh you know, example blah blah blah um very, very exciting. I'm you know this is really exciting stuff, and then you can do something warnings of as errors where it's like hey. If you get a warning, you want to make sure that you, you know, cube control exit exits out, instead of actually continuing to execute that so very, very cool. That's uh that's!

A

Exciting usability thing that we're actually seeing going on there. Okay, so we had some uh something got messed up in my uh in uh the uh hey george. Can you look at fixing this? Something got got kind of screwed up in the uh in the uh in the notes here um looks like things got jumbled any idea if oppa can take advantage of the deprecation features, I don't know. I think.

A

Let's see where did the uh warnings are now a thing? Okay, so we'll go through and look at this. It says that you know, let's see admission web hooks our primary way to integrate custom policies to allow a request to, uh but warn about configuration yeah. It looks like those warnings are actually supported now um and so yeah. I would imagine that this is something that could work with opa that'd be actually a fun thing to try.

A

uh uh I wouldn't be surprised if that were the case, so that would be really cool. uh Don't worry about it! I I assume that in good faith, the thing got got a little bit uh uh got a little bit uh messed up here, um so okay cool and then um there was a link in here, um yeah, the so okay.

A

So there's a link to the cap that will actually get in there and then um I know jordan liggett, who did a lot of the work behind this gave a demo at the uh at the the kubernetes community meeting, and so that's something that you can dig into also uh super cool. So let's check the cap real, quick yeah. So this is actually I mean like this is a great way to look at like okay. What do we want to do with a cap? Because this is the type of thing where it's a good thing?

A

Everybody can agree to it, but there's a ton of decisions to be made about. How do we do this in the api? Where do we do it like? How does this impact cube control? So this is the type of thing that's going to really sort of impact, a different parts of it and using the kep process as a way to really make sure that everybody's on the same page before folks put a ton of work in here, and so you know, I love seeing this stuff.

A

You know and then there's the implementation history of like okay, this stuff was started in april and uh you know- and now it's uh now it's beta and it's actually making progress. So great example of the kep process kept process in action. Oh steve's asking what about the t-shirt today? This is a t-shirt. Oh, I love this um I'll do a little plug here, because I love these folks.

A

So so much this is a teacher's t-shirt from theory11, which um these guys make cool playing cards and uh and they actually sell some t-shirts too, and I think this is where I got this one yeah there's my shirt so, but they have some of the coolest playing cards and I buy them a lot and I'll give them away as gifts and stuff.

A

But my favorite and I bought a lot of these things is: uh is they make some star wars playing cards? And so there's? This is a dark side playing card and this is the light side and if you go- and if you you know, if you go to their page online you'll, see that actually all the face cards are characters from the star wars. Series super super cool um and uh yeah, the but the, but all their playing cards are beautiful, and so it's the type of thing where it's like they're a luxury.

A

But it's like a 10 luxury. So it's like it's something that you can like splurge on, but not feel too bad about splurging on which is really fun, all sorts of really cool stuff. There, all right cool, all right so now across the the rest of the uh cloud native ecosystem gitlab had a great blog post that um yeah.

A

So you know I I usually carry some of these around, and so I you know I wanna the biggest star wars fan, I know, is tim hawkin who, who leads up a lot of the uh uh a lot of the uh uh cig networking stuff, but he's been in kubernetes forever huge star wars fan. I had these in my bag for next time. I saw him and I was going to give it to him.

A

uh You know at the next cube con, but then you know covet hit and all the cube cons got cancelled or went virtual, and so I'm really sad that I haven't had a chance to give them one all right, all right, cool, um so yeah. So the get lab folks actually had this great blog post talking about what's been working for them, what hasn't been working them with kubernetes?

A

And I really love seeing these things, because you know they get into the details of what is the tool set that they're using what are some of the surprises, because they did a bunch of stuff where they were running clusters cross a z. They saw a ton of traffic cross hazy right, not surprising and uh and that would that ended up being for them a cost on aws that they didn't really appreciate, and so these are some of those practical learnings. That, I think, is. You know super valuable to folks that you know hey.

A

There are pros and cons for running a cluster that crosses azs, and this is definitely one of those things to be aware of um things like how they're doing metrics and logging things like reserve, pod capacity and utilization.

A

So all sorts of really practical lessons here that, I think is you know you know really really great to actually see folks share, not just their failure stories with things like kates.af, but also some of their success stories and the lessons that they've seen.

A

And, uh let's see so the cruster I, this is really cool. um uh This is a so george was showing me this. I wanna.

A

Let's see um I was digging into this. This is like a raspberry.

A

So tell me more about this george, because I I didn't have a chance to look at this. What makes this different? This is like a raspberry, pi uh uh uh kubernetes cluster thing: what's the deal, what did you? What did you say this? Is clusters on raspberry pi? There's a ton of these, but uh the author, which is, which is george, has a a soft spot for cube admin based stuff, okay, so running a raspberry pi uh kubernetes cluster.

A

Okay, this is a cube admin based experience around this. To really show you how it works. Okay, very cool- um you know- that's been on. My to-do list forever- is to actually build a raspberry pi cluster, um but I've never had the time to get around it like I keep buying arm based microcontrollers, and I was just showing george, you know as we were.

A

We were talking about this, but before, like I've, been playing a lot with um like the latest one I bought is like synthesizers and stuff, and I, like I, I've been looking at, like even smaller ones like uh this is what I have here. Where did it go? Is this one called the teensy, which is really small, and you can do a lot of audio stuff with it and stuff so like? I want to do more of this stuff and actually start creating connections to kubernetes clusters and I'd love to play with raspberry pi's.

A

I just haven't had time to do it, so this looks like maybe this will actually get me going with it. It'd be super cool, uh let's see new version of helm released um and this one is a security patch. um So a lot of details there.

A

Okay, helm, repo ad will no longer actually overwrite stuff, so I wonder for notable security okay, so they have a security advisory and it's really great, to see uh helm really start taking advantage of security advisory seriously. um This is really cool.

A

Okay, so yeah, so look at this go before blah blah blah that there was a crypto package that they were using, that was parsing x, 509 certificates that could cause a denial of service, so yeah so good stuff like that, so super cool there and if I'm not mistaken, um if I'm not mistaken, there was also an announcement of like helm2 deprecation.

A

Is that yeah, so there's a deprecation timeline, which I think folks should be aware of? Also so helm 3 was launched in 2019 um helm to so security patches until november, okay, so yeah, so we're we're we're staring down helm2 deprecation, something to be aware of y'all should be, should be aware of that: okay cool, um let's go back here and then okay, so somebody put in a link here to the the cni benchmark.

A

Now this is really interesting here, because I know that uh the uh I believe the folks running this so benchmarks like this are always you're like okay, who's actually running it, and I think one of the things to to be aware of here is that you know sometimes this stuff ends up being run by folks who have sort of a horse in the race right. So you always want to be careful and understand, but it's really interesting to see folks start to deal with this.

A

um You know impacts of mtu weavenet, um so some really interesting results going on here. um So there's memory, usage versus cpu usage, um our horse in the race. What we've been investing here is entreo, which looks to be really interesting. So a lot of like looks like they're, really examining this from all sorts of different directions, which is really cool, very cool.

A

All right so yeah, so if you're, if you're interested in pushing kubernetes networking speeds, uh you know and uh performance, this looks like a great resource uh to at least understand sort of some of the ins and outs and the things that you really want to take a take advantage of very cool uh yeah, and I I like to see that that entree is actually coming uh turning up well here. So that's really good to see all right cool. um Let me leave the uh uh look at the chat a little bit.

A

So lots of these pie based projects are customer. They use specific distros this just so okay. So the nice thing is that just like pure cube admin for this stuff, which is cool la maddie's, asking long term goal for synthesizers, I want to do stuff.

A

That's electronics that doesn't involve computers, and I know that there's computers like built into each one of the modules that I'm playing with, but it feels like you're old school, doing analog stuff, which is a lot of fun um cool and then like yeah and then a pointer to the theory 11 cards. You know they should send me a commission all right awesome.

A

So let's go um and uh start digging into uh sealed secrets, and- um and here is the web page here, so I don't think that there's anything uh uh I don't think we have an official fancy web page beyond and so, oh first of all, I want to say thank you. I I think he was able to join us. Marco has been doing a lot of the work to maintain uh sealed secrets here. This was originally started by a bitnami alum, who I believe last I heard was- was at aws these days.

A

uh This is angus, lee's uh and so marcos uh has been doing a lot of the work here and uh just a big shout out to him. um What time zone are you in marco? This is like t. This is you're you're in a time zone where tgik is not the most convenient. So thanks for really making the time to join us. I appreciate it um so the idea here with sealed secrets, which I think is really interesting.

A

um Oh he's in utc plus two so yeah yeah, I apologize um is you know you can manage all of your kubernetes and I love the problem here. I can manage all of my kubernetes config and get except secrets, because it's generally a really bad idea to check secrets into your git repos. um It's a good way to get owned, and in fact you know there are folks out there that actually, you know, monitor all check-ins to uh github and uh look for something that looks like hey. This is an amazon.

A

uh This is like this looks like an amazon. You know access key and it's gotten so bad that now, amazon and github themselves will monitor for those things and warn you and I believe amazon might even auto revoke the key if it sees one of these things checked in, and so you know. But but you know, amazon access, keys and github access keys.

A

Aren't the only secrets that get checked in, and so you know, and especially through the the lens of things like git ops, and this idea of managing everything through a system of record like git uh secrets, end up being a real thorn in your side, um and so so steve is suggesting get leaks as a way to protect this. So what is get leaks.

A

This looks cool scan, git repos for hard-coded secrets. Like passwords api keys, oh very cool. You know I would be cool to actually scan get repos for anything for any secrets that are checked in right, because if you see a secret yaml that looks like a secret, you probably shouldn't check it in that'd, be actually pretty cool.

A

uh So paul's saying, if you put your aws key in github, you'll get several emails within a few minutes and aws does revoke it within minutes or so a friend told me it's too easy to happen, and when it's once it's out there, you can't put that genie back in the bottle, so yeah so get leaks, looks really useful here. So yeah there's you know defense in depth right, it's not just about encrypting stuff. You also want to make sure that you know humans make mistakes accidentally check stuff in get leaks.

A

It looks like a really cool resource there, um cool uh all right. So the idea here is how do we actually start managing secrets in the same way that we're managing the rest of our configs, whether you're using git or get ops or whether you're using any other mechanism to be able to manage this stuff? You really want to have your secret sitting next to everything else.

A

um One thing that I do want to actually so like we're, so we'll go through and install it and get going, but this is different from encryption of secrets using some of the advanced features of kubernetes, for keeping secrets encrypted and over time, there's been sort of a continuing march to make secrets in kubernetes more and more secret over time, uh and so uh there's still a lot of gotchas there and josh has done a couple of tga tgik episodes where he's delved into uh a lot of the sort of like what's happening.

A

I believe it was josh did one of them at least and touched on on sealed secrets at one point, uh but he was really looking at sort of like how do secrets work. What are the best practices around secrets at the cluster itself? Sealed secrets is something for managing config before it hits the cluster and there's a little bit of stuff that happens in the cluster, so we'll go through that flow and really get into into some of the details there.

A

But at the end of the day, what it is is, is it's a uh it's, a crd plus a controller that has essentially encrypted uh data and then that controller then decrypts that data and creates the secret for you, and so it's really a way of like you know, once it hits the cluster. That controller will decrypt it for you, so it'll actually be sitting in the cluster decrypted.

A

But it's a way for you to actually have everything through cube control be encrypted until it actually goes ahead and gets into the cluster. So that's that that's the general flow here and so uh uh so yeah. So let's go through and um and let's see here.

A

Oh, this is new. I hadn't seen this too sealed secrets, as templates for secrets all right, we're going to dig into that. Okay, uh I I reviewed this beforehand, but somehow I missed oh, I I skipped over that uh okay cool, so let's actually go through and get the thing installed and then we'll start exploring some of those more advanced features: um okay, installation!

A

You could there's like a gazillion ways to install this, um and so one of the things you got to realize is that there's a client there's a helper client that that we're dealing with combined with the controller and the crd, that's actually running on the server side, and so um let's go ahead and uh you can use customize and um or a helm chart or the operator for a framework.

A

But what we're going to do is we're just going to use the raw yaml, because that's how I roll- and uh the cluster side we have here, is on the releases page yeah. So that's what we're going to do. I'm way ahead of you, josh curl dash. Lo.

A

And so here we have controller.yaml.

A

Let me know if you all can't see this. uh I try to pre-flight this and make sure it's all readable. um Okay. So what do we got going on here? So we have a deployment, that's going to be put into cube system, okay um and then- uh and this is gonna- have a single replica um and I'm sure that you- you know- I don't know- I wonder marco- can you run multiple replicas? It's a sort of a high availability thing. Do you know, um and it's I looks pretty simple: we got a single container.

A

uh We got some readiness probes, uh some nice things here, so read. Only root file system run as non-root run as user. So a lot of sort of lock down stuff here. This is actually really good to see um standard in ttys and it has a a temp volume mount uh which is an empty dir. Okay, cool, um okay, it's noisy, so they'll fight with each other, so not really, okay, so yeah! So maybe uh uh is there a?

A

Is there a feature request in the sealed secret repo to do sort of leader election types of things? That might be a fun uh that might be a fun way to actually have somebody uh help uh contributing here to this that'd be a fun project to do all right. So let's go ahead and we have and then there's the custom resource definition here and uh and then we have there's a role. So this is one of those things where, um let's see so essentially get and create for the.

A

For the sealed secret, let's see this is a the role getting access to essentially uh pretty much everything, unfortunately right so role. Binding here is that we're seeing that essentially.

A

Okay, this is interesting. Okay, so by.

A

Default sealed circ, sealed secret service proxier.

A

So this is essentially saying anybody authenticated to the system gets access to the sealed secret service, proxy or role the sealed ser. So I'm not super familiar with this, so this is actually saying you get access to the sealed secrets: controller resource.

A

Okay, this is to be able to draw back the public key. Okay, uh moz is listening. Well, well, I'm driving, while they're driving, that's funny. um Okay. So that's really interesting! Okay, so this is the proxy resource. Okay um to be able to actually ask for the public key okay. I want to dig into that a little bit.

A

That's really interesting, because the I you know we'll go through essentially how the public key stuff works, so that you can you know uh and and what happens behind the scenes with the uh as we encrypt a secret, we'll dig into that. um There's the role binding around that so essentially anybody can get.

A

This is a fancy way of saying anybody can read the public key for uh encrypting new secrets and then uh and then there's a the controller itself essentially gets uh uh admin access, uh which is essentially the ability to list and create secrets.

A

um It's fascinating that it doesn't have the ability to read secrets just to list, uh but I guess it all will always list um so that, actually, when you do a list, you actually get the full contents of stuff. um Okay- and I think you know this- oh- and this is the unsealer. No, the unsealer does get the ability to essentially do everything you need with secrets. I mean you know.

A

Usually we we look at things that are dealing with secrets across the cluster with a little bit of side, eye but kind of that's this thing's job. So you got to trust it. So there we go uh so okay, cool, let's go ahead and we will apply that. I have two clusters running, because I think if we have time I want to look at sort of like can we create one secret and actually deploy it to multiple clusters, because I think that might be an interesting way to do things.

A

um Let's see so the devops live says. Would you agree that kubernetes secret definition should be renamed? It's just base64, not encrypt and relies heavily on admins to store it properly encrypted.

A

I mean it's secret in that the uh it's a separate thing that can be r backed and controlled, and we actually make it difficult to access it across namespaces.

A

You know anytime, when you have automated systems, it's not uncommon for there to be things that you'll call secrets, but they get passed around as values like everything else. I think you'll look at this with config systems, like I mean the one that that you know I'd, be you know like puppet salt chef, all those things have ideas of secrets and they may be passed around encrypted, but like the means to decrypt them is actually fairly widely distributed. Also, I think, over time, we're continuing to lock secrets down more and more.

A

The base64 encoding has nothing to do with encryption. That's just a way so that you can actually store binaries in there um versus uh uh you know. Having to you know manually have something know how to like. You know. You know uh base64, encode and decode. That's the only way to do it. um That's that's the only reason why there's base64 there there was no, never a plan to have base64 look like it was encrypted, uh but I don't know I mean like I'm like. I don't know if we should rename it.

A

I mean at this point: it's kind of moot it kind of is what it is. Okay, so cube control apply, dash, f, controller.yaml.

A

Okay, steve says: the key thing to remember is that the sealed secret can't be changed, think changing namespace. You have to reseal it okay, so, like the you know, when you encrypt it it's encrypted for a specific namespace, you can't actually reapply it to another namespace, I think, is what steve's saying here: um okay and so what we have here uh is uh so it went through and it went and we applied it. Cube control, get uh pods, namespace equals cube system, okay, everything's up and running sweet all right.

A

So now the the main so now. The other thing that we have here is that we have a thing called cubeseal, um which is the companion command line utility that actually goes with this stuff, uh and so I installed that let's go through and we'll look at, you know the instructions for installing that um I'm on the mac. So I installed that with homebrew, but that's also available on the releases. It's a go binary, and so it's sort of self-contained there. So you can go ahead and do that all right. Here's the main usage here.

A

um So one of the things that we do is that we can actually create a uh a secret.

A

So, let's look at this line here and and we'll decode I'll decode to you what's going on here.

A

So marco says, sealed secrets is very conservative. We try to avoid you shooting yourself in the foot by misconfiguring your r back et cetera. That's also why, by default, it lives in cube system. Okay, all right, so here's what we're doing so bar is essentially the secret that we're encrypting um and then we're creating a generic one where it's essentially, where from file where you know we're reading that from standard in and then uh and then echoing.

A

So you could cat a file in here or something like that right or you could have this be from file here and then we're writing that into mysecret.json.

A

Oh look at that. We got a deprecation warning here, so dry run. So what we're doing here is what dry run says here is: don't actually create the secret in the server instead just create it locally and then output it as json.

A

So here we have cat mysecret.json is essentially what we have you know is you know the the json version of the ammo that we would actually be upgrading to the uploading to the to the to the cluster, and so we're doing this just locally, and so we're essentially using cube control, create as a way to sort of template and do the base64 encoding here you can create the secret uh yaml any way that you want to. You don't have to use cubecontrol create uh you know, but this is just a convenient way to script.

A

It, uh let's see uh alex, says the key management to truly in cypher secrets is really tricky in such a dynamic environment like kubernetes, okay, yeah. uh So there we go and then maddie's asking about a key rotation, we'll go ahead and we'll get to that. um So this is that base64 encoded that we referred to earlier. We can do a base64, um uh is it?

A

Is it dash d yeah dash d, um that file? Oh.

A

um uh Let's see we can do uh echo.

A

Yeah, so there you can see bar that's just the basics before encoding, that's not encrypted right, um yeah, you're way ahead of me, marco, but then the next step of what we do here is we actually say: okay, we want to use cubeseal to take the plain text secret and turn that into the sealed secret, and so we're going to go ahead and do that now.

A

Magic happened behind the scenes there, uh but if we now look at oh we'll pull it up in vs code here, if we now look at the sealed secret, what we'll see is that this is json also um the api that this is now a sealed secret instead of a secret right. So we went from secret to sealed secret um and we can see that we now have a very long piece of text.

A

So this is a way of saying bar in a very verbose encrypted way uh and so uh yeah now what's going on behind the scenes there, all right so what's happening here- is that the controller when it started up in the cluster um created a public private key pair and the way that public private key pairs work? Is that essentially it's two two halves of a key and you can encrypt with the public key and decrypt with the private key? You can actually do the opposite.

A

Where you can, you know you can encrypt with the private key and decrypt with the public key too, but that's not being used here. That's used for signatures typically, and so you want the public key to be public, and so what cubeseal is doing is it's reaching out to the cluster, so it actually won't work without a cluster.

A

So if I actually go through and if I do uh export cube, control, cube, config equals dev null and I I don't know I haven't tried this before and I try and do cube seal like it'll say like: oh, you know no valid configuration because it doesn't actually know which cluster it's actually talking to.

A

So let's go through and let me fix that.

A

There we go um so now. I have that cluster up and running, and so we're off to the races here um so dennis is asking any reason to use yaml uh jason instead of yaml. uh I think jason's just safer to parse. Generally there's you know, I I'm gonna, give a talk at some point about all the evils of yaml and sort of like some of the uh some of the problems that and opportunities there.

A

Oh okay and then uh marco's saying that now you can actually do this with dash f and dash w, so it'll write to disk. So you don't have to do all the redirection tricks here, which is useful too, all right so cool. So we have this up and running. We have this secret and uh and if we go through and if we do a cube control, get secrets, uh namespace equals uh cubeconfig our cube system.

A

One of the things that we'll see here is we have there's a whole bunch of service account tokens for all the various service accounts in the and but we actually have this particular uh thing and it's of type tls, because I don't know why it's tls I mean you can use it for tls also, but um but this is the key, um and so now we can do uh get secrets.

A

Oh, let me do namespace equals cube, config or cubesystem.

A

Getsecrets, if we do that.

A

Okay, so there's a lot of craft here, but what we see here is that we have a.

A

We have a couple of files here, and this is probably the public in the private key. So this is the key, and then this is a public certificate. It doesn't have to be a certificate, but it is um that uh that actually shows uh what's going on here, and so this is the private public key uh key pair that we actually have going on, um and so uh what I believe and correct me if I'm wrong marco, is that you know we're using some advanced mechanisms in uh the api server so that cubeseal can actually.

A

It doesn't actually read this resource because that'd be pulling the private key, which would be bad. You may not have permission for it. Instead, what it does is it actually opens a a tcp connection to the sealed secrets: controller that then actually publishes the preferred uh public key to be using back out.

A

So, let's see so uh discussion in the chat talking about the encryption configuration right, so this is about how do you secure things from git into the cluster? There's a separate sort of thing, which is the encryption configuration of like how do you make sure that you actually store things in an encrypted way when you're writing to etcd inside of the cluster? And so really it's like. There's the.

A

How do you get your config into the cluster and then how do you make sure that, once in the cluster, the cluster is storing it and provisioning it in a more secure way and it's really sort of they're they're complementary to each other and they layer on top of each other?

A

um Okay? So that's what we got going on there. So now I'm wondering marco! Is there a if I look at cube seal uh cube seal help? Is there let's do a let's? Do a super verbose cube seal here?

A

Oh okay, we have the um it'll probably go to v10, maybe.

A

All right so yeah, so so this is what happens. Is that it's it's saying that it loaded my um my cube config from here, and then it actually made this call to this particular now, I'm running a kind cluster here, and so it's actually talking to my api server. So that's what 127.0.0.1.5288!

A

That's my api server and actually saying. Okay, I want to talk to the http sealed secrets, controller proxy that- and this is making essentially the the api server- has some capabilities for some situations like this, where it'll, actually proxy requests to a service.

A

That's running, and this is the way some of the things like logs and uh and the you know, access to the dashboard under certain circumstances are allowed, and so this is actually making an api server, a an http call to your api server and then the api server is then proxying that, through to the actual service, that's running on a on the cluster and it's asking for the certificate that it should be using to encrypt things.

A

It's getting back the certificate pem and uh it gives you 200 okay and it's actually returning the contents there, and so this is how the cube seal is actually getting that using that public key to encrypt things and then writing that as a sealed secret into that file.

A

Okay, so there we go so now now. What I can do is we're going to go through the next step here and I can actually go through and do a cube control apply, dash, f, sealed.

A

What do we have my sealed secret dot json? So this goes ahead and creates this uh in the cluster. And now we can do a cube control, get secret, uh get a sealed secrets.

A

We can see it's there, but one of the things that we can do now is. If we get secrets we can actually see. Oh look, here's that secret. So what happens is the controller watches for any sealed secrets and then decrypts those things and writes out regular old secrets? So now I can go through and I can get secrets my secret.

A

Oh yaml, there we go. We have uh uh there's the data, the base64, encoded, foo, okay, so here's the deal anybody can go ahead and that has access to the git repo can read that sealed secret.

A

Only people, with with permission to create a sealed secret in the in the right name space, can upload that and then only with people that have access to read secrets from that namespace or start pods that have access to that secret are allowed to actually then go ahead and read the plain text to the secret. So if I were a different user that weren't admin on this cluster, I wouldn't be able to read this secret and I wouldn't necessarily be able to.

A

I could maybe upload it to the wrong name space, and I believe, if I did that, then the uh the the uh the controller for sealed secrets would refuse to decrypt it, and so this this was a secure way to get the secret into the cluster. But once it's in the cluster, it's really just like any other secret that you've uploaded to the cluster. It doesn't actually save you or improve security once things are in the cluster.

A

It's really about getting it into the cluster is what sealed secrets does so marco's saying that using the proxy is currently a pain point, because some cluster configs, like kubernetes with private control, planes, uh yeah, so there's a lot of um smoke and mirrors for the way that that proxy mechanism works generally in kubernetes.

A

All things call into the api server for things like logs for things like these, um uh uh uh these htv things it's actually one of the few cases where the api server calls out to the cubelet, um it's kind of an exception there, um but it's used for streaming logs. It's used for uh for things like this, so it's a little bit of a corner case that I think if we had to do kubernetes over again, I'm not sure how we'd fit how we'd?

A

uh How do we do this, but I think we'd probably break the proxy out into a separate type of controller separate from the api server.

A

um So steve says the controller is globally scoped in terms of access to sealed secrets by default from memory yeah. I think the controller will look at everything, but I believe and marco correct me if I'm wrong- and I think this is one of the advanced features that we can get to- that when you actually seal a secret, you can seal it for a specific name space or you can actually seal it across the cluster and I think that's something that's worth getting into so there we go.

A

So that's that's the main flow that we're talking about here, um and so that's actually pretty great, because now I can create a secret check it into git and I can cube control, apply it like everything else and it'll just magically work, so that's actually really really cool.

A

um So some of the sort of the next things that we want to actually start talking about here. um So I want to understand this template thing, but like things that I want to investigate with the rest of tgik, uh what is certificate rotation? Your secret rotation looks like what are the different types of rotation and how do those work? How would we actually make this work across multiple clusters? Can I actually have one secret that I can actually deploy to multiple clusters?

A

What does that look like um and then the third thing that I think is worth looking into is I want to understand the scopes and the template stuff. That's here, because I think that's really interesting and I had missed that before.

A

uh So let me go ahead and read through that real quick.

A

um The previous example only focus on encrypted secret items themselves, but the relationship between sealed secret custom uh secret and says similar in many ways to the deployment and pod in particular, the annotation labels of a sealed secret resource are not the same as the annotations of the secrets. Okay, I get it so this is like what are the things that we want to actually put on the metadata, what is sort of the the metadata template of the secret that we're going ahead and creating?

A

So this is a way of actually specifying labels and annotations and things like owner references of the uh of the secrets that end up being created. Okay, that's super useful. There I get how that works, um and then.

A

Okay, so here's one thing that's actually interesting here is that we can go through and right now, cube seal by default will actually talk to your cluster and pull that certificate back out, but we can actually go through and we can grab this thing explicitly, and so, um let's see what's the best way to extract that marco, do you know cube control.

A

So we can go through and there's a whole bunch of things here. You can get it via cube seal. So how do I actually get? How do I dump the public.

A

A

A

Okay, cube seal fetch search, so we do cube seal fetch, cert, my cert.crt okay. So now we have the certificate. So you know how I did before where I set. You know: uh export uh cubeconfig to devnull and now cubeseal.

A

This thing's going to fail because it's like hey, I couldn't load the file. Well, one of the things that we can do here is looking at this is that I can actually specify the cert here.

A

uh I called it my cert.crt.

A

Oh and it says no ins, no namespace and cannot infer it automatically. Okay! So now let's go through and we'll change our uh sealed secret. We'll change this so that we'll go uh namespace default.

A

Boom, so now what we did is we sealed the secret without actually having access to the cluster, and so that's that's. An important thing is that you may actually have situations where you want to see all the the the um oh I could. I could pass dash and default to the cubeseal also, um but you may have situations who weren't a seal: the secret, where you know you don't have immediate access to the cluster. So that's a useful thing to know there.

A

um So that's uh that's cool okay, so the next thing is, which is interesting here, is let's go through and let's, let's experiment again, we'll do cube control, create namespace uh test. Okay, so we've created a oh. um Let me fix my cubeconfig.

A

So now we've created a test namespace and now what I want to do is I want to do cubecontrol.

A

So now, I'm going to look at my sealed secret and I'm going to be like I'm a hacker I want to go through, and I want to change the name space here to test instead of default and I'm like well, I got access to this. I have access to the test namespace on this cluster, but I don't have access to the default namespace. Now I can do a cubecontrol dash n test apply dash f, my field secret.json that went ahead and created it.

A

And now, if I do, if I do a get secret, you can see that the uh the the sealed secret controller did not decrypt at this time, because I took something that was meant for the default namespace and I applied that to the test namespace, and so this is a security precaution to make sure that hey, if somebody seals a secret for one namespace, they can't use it in another namespace.

A

But that's, I believe, where the sk, the scope feature comes in so now we can go through uh and we can say strict and this is. It must be sealed with exactly the same name and namespace, but we can actually change this so that it can actually be cluster-wide, and so now let's go through and actually let's seal the secret again.

A

Oh, it's also because you manipulated the sealed secret contents. Okay, does it actually does it actually seal the entire, including the the metadata so.

A

Maybe marco says no, so the metadata is not actually encrypt, okay, but yeah. So regardless I tried to use a cross cross namespace and it's like hey. I don't like you doing that, but now we can do we can call cubeseal, and what we're going to do here is we're going to do. Scope is cluster wide.

A

So now we're going to go ahead and do the same thing and uh and then let's look at the um the uh the sealed secret here. You'll notice that now there's an annotation here saying that this thing is cluster wide.

A

Okay and now I'm going to change this thing to test and let's see if this works.

A

So now it got deployed here now. If I do a get ah now, you can look and see that oh, it went ahead and actually decrypted it for me and so yeah. So you get the choice of whether you want to actually seal the secret just for a very specific name and namespace, whether you want to do it across the namespace, where you can rename it like.

A

Maybe you're actually versioning your secrets in some way that, like you, want to have the sealed secret, but then change the name as you actually apply it or whether you want this to actually work across the entire cluster, where anybody that has access to that cluster can go ahead and actually uh decode that secret.

A

Oh, it's raining outside. That's really good news for the air quality, I'm very excited! um So that's actually! So that's a that's a really cool feature there and I think it shows you that, like I'm sure over time the whole subtleties around namespace name, specific versus uh cluster wide a lot of folks have sort of hit their heads against that, I'm I'm sure, uh okay yeah mark. This is really interesting. The annotation gets uh put in the template so that it actually gets copied into the decrypted secret so that you can actually round trip this.

A

So if you actually go ahead and pull it, you can actually change it and then re-encrypt it without actually losing some of that metadata yeah. So that's actually really cool. So that's a that's a sort of reality of this that round-tripping and keeping making sure that these parameters are round-tripped with it that I I hadn't really understood before. So that's really cool to to see how that works all right. So that is the way that this stuff works um uh very, very cool now.

A

uh The next thing I want to do is secret rotation and there's two types of rotation that we should really talk about here.

A

Okay, marco says: the main problem is that the error messages are currently crap like there is no way to uh get feedback that hey. You know, I wasn't. I didn't decrypt this because it was only to you know you were using it on the wrong name space, something like that um yeah. The way I would do that is, I would have like a condition in the status of the sealed secret. uh That would be the probably the right way to deal with that, all right, so key rotation.

A

So there's two secrets that we're talking about here there's the secret for the sealed secret controller. That's that certificate and private key that are created when you first install this thing into the cluster. So that's the first secret that we want to worry about rotating.

A

The second thing that we want to worry about rotating is the secrets yourself right so, like you know any secrets that you're putting in there, you probably want to go ahead and rotate those things uh sealed secret cannot rotate your secrets. What sealed secrets can do is actually rotate the key that it uses and by default, that happens every 30 days, and what happens is that when it actually happens every 30 days, the sealed secret controller will go ahead and create a new secret.

A

uh A new uh key value, uh public key private key pair it'll keep the old private keys, but it'll only then serve up the latest version and correct me. If I'm on marco it'll only serve up the latest public key, and so that means that at that point, when the the the controller does this a key rotation, all secrets encoded after that point, will go ahead and be encoded with the new key, but all secrets that you were sealed beforehand that are now checked into git. That may be six months old.

A

Those things will actually still continue to use the old key and the controller will know how to actually decode those things and so, um and so that's something that to be aware of is that it'll actually but like, but for you to really sort of take advantage of that sort of that that new key that the controller is using.

A

You have to go back and reseal the secrets that you've had so far and so we're going to actually go ahead and do that so like what I'm going to do is there's a way to force early key renewal on the.

A

On the uh so maddie's asking, how does it maintain mappings? My guess is that there may be a key id, uh that's encoded in the uh in the sealed secret or it just tries them all and sees which one works right. Let's think at some point, like you know there, there's uh that might be something that works. Also, I'd I'd be interested to know whether there's actually a key id that ends up being getting used as part of this um okay, so we can do uh date, dash r, that's the current date.

A

uh Hopefully it's time zone aware and now what I'm going to go through is I'm going to change the controller yaml and we're going to go to the deployment here.

A

uh Command controller args, oh where's, args.

A

A

Okay, yeah tries them all and the key id is planned, but it has to do it in a backward compatible way.

A

uh Okay, so now what we can do is we have this flag to the controller called key cutoff time, and I'm going to set that to that this thing here.

A

Okay, so now what we're saying is that um this will force a new key, the controller, to actually create a new key. When I actually push this so cube, control apply controller, we went ahead and did that cube, controlled, cube system get pods and we can see that sealed secrets, controller.

A

The new one is running, the old one is being brought down.

A

At some point, okay and then, if we look at uh get secrets.

A

Now what we see is- and I don't know but like the certificate here- I think- is probably a new certificate and but this key is probably includes both keys, I believe- or maybe let's actually, let me see here- secrets yeah, so it now has a secret, but my guess is that the private key is actually now contains multiple private keys, uh all the old private keys too, and so I mean we can go through, and let's do this I'll grab this. It will do pb paste uh base 64-d.

A

And so this is rsa private key. Is this a single private key? Where are the old private keys? Oh they're, okay, marco's saying there should be multiple okay and I'm not? Oh, no, I'm not! I'm not seeing the multiple private keys here.

A

What did I do wrong.

A

So it's been running for two minutes: cube control.

A

A

uh Cube system.

A

A

hmm Okay, so marco said he missed the part where I triggered the renew. What maybe I did it wrong? So tell me if I did it wrong here. I said I said key cut off time and then I specified especially now in using the the date dash r, is that time zone aware.

A

um I would expect that this would actually create a new, a new one, so like what I was doing is the early key renewal. The key can be generated early by passing the current timestamp to the controller into a flag called key cutoff time so should be seems, correct, okay, um but yet here we are, um let me go through and let's do.

A

Oh, you know what that I'm a I'm a dumb ass. Okay, uh let's see, I should have passed.

A

Yeah, let's do that: okay,.

A

A

Okay, so now we have multiple secrets: okay and- um and how do we know, which one is the latest? uh It probably loads them all up.

A

Not dash 0-0 and.

A

Oh it's time stamp base. Is it based on sort of the creation time stamp then.

A

uh Or is it based on a timestamp? That's that's embedded in this, the certificate. Okay! So it's the creation timestamp that it's actually using okay, so so the last secret that is created is the one that uh that uh the sealed secret will use. But now what we'll find here is that I have a secret.

A

That's existing already, this mild, my sealed secret, and that was encrypted with this certificate here right, but now we can go through and I can I can seal it again, and so what you want to do is you want to do a cube seal and we'll do.

A

We'll take off the let's see we'll do.

A

And we'll do my sealed secret 2.

A

And now, if we do diff my sealed secret.

A

We can find that these things are now different because oh well, one of them was cluster wide and one of them wasn't but like, um but the. But it's actually now using that new certificate that it actually grabbed down from the cluster, and so we can go through and um yeah. There would have been different. Oh does it actually put a time stamp in there, so they would have been different. Anyways, okay, uh my cert.

A

So I will do my cert two here and so you know we can see that my cert.

A

We can see that these certificates are now different because we're using one certificate versus the other certificate.

A

So that's the way that you, you rotate it but again sort of like we can still have. You can have a whole bunch of secrets that are now encoded with the old secret, and so that's something that you actually want to be able to to to be aware of, um and so a couple of things that you want to do for good hygiene as you use these systems over time every once in a while.

A

You want to actually go through and and and there's some explanation about sort of, like the uh the the two different types of sort of secrets that we're dealing here.

A

um So you want to actually go through and actually call this sort of re-encrypt here and and make sure that we do this and and does the re-encrypt actually work in place? Is there a way to do that, but you can go through and.

A

Steve's saying that they auto they automated in metal um encrypted the same clear that creates two different type: it's not it uses a non. Oh there's a nonce in it, okay, cool um and so there's a re-encrypt which is essentially says, um use the re-encrypt, the given sealed secret to use the latest cluster. So how does that work? I.

A

Wonder so maddie's asking about the private keys for the life of the clusters yeah. So the first thing you want to do is you want to automate? You want to actually re-encrypt this stuff here. So let's go ahead and do this so we're going to go through and we're going to do re-encrypt. My sealed! Well, that's wrong! Okay! So we're gonna do re-encrypt.

A

Myfieldsecret.Json into temp.json, uh oh actually, I want to give this a v10.

A

ah So this is really cool.

A

Oh look at that. It's duffy, hey, how's, it going duffy, so this is really cool. So what it does is that when you do this rotation, you want to do the rotation on a sealed secret, but you don't want to necessarily um see ever see the private key and you may actually only have the crypto text and so the way the rotation does. Is it actually uses that same mechanism to talk to the controller and calls a different endpoint on that controller? Saying rotate it?

A

And so, when you rotate it, you send it one sealed secret and it sends you back a new re-encrypted version of that, and so this actually uses this so that you actually never see the private key. And so this is great because you can do a whole bunch of workflow here with never actually seen the private key.

A

So that's pretty cool yeah. So right only that's super cool okay, so you- and so so good hygiene is like once in a while once a month once every couple of months. You want to go through and do this re-encrypt or better. Yet you want to rotate your secrets right because you shouldn't be using any passwords uh over the long term anyways. But you know oftentimes, those secrets are manual and you don't want you know and rotating them as a pain.

A

So you should at least make sure that you're using the latest certificate from sealed secrets and then the question that um that lamati had which is like hey. Does it keep these things around forever? And the answer is yes, it does and so like when we went through and we actually looked at uh see, get secrets.

A

When we look at it- and now we have two of these secrets here, every 30 days, it'll create a new secret and it'll actually leave the old secret behind currently in.

A

In sealed secrets, there is no mechanism to automatically prune the old secrets, um and- uh and so uh if you do want to prune- and you have that re-encryption uh process in play, then this is the way that, like hey, you know what I want to keep six months worth of old keys in case something's left around for a while, and then I want to go ahead and re-encrypt.

A

It would be useful, I think, to to be able to do an audit to actually say. Are there any sealed secrets in my cluster that are encrypted with a secret, that's more than like, say six months old? So I think I could see uh some mechanisms for being able to say: hey. Does my cluster have a clean bill of health? Is it safe for me to actually delete and garbage collect this stuff? uh That would actually be uh be something that would be useful here.

A

um So yeah and marco says it can be useful to upgrade your sealed secrets to a new file format without knowing your secrets and so yeah interesting stuff. um So that's how rotation works here, um let's see and.

A

And so that is so the only thing that I want to play with, yet that I'm not quite sure how to do, which is how do we actually deal with multiple clusters? Can I actually deploy the same sealed secret into multiple clusters, so let's go ahead and see if we can do that.

A

What I think I need to do is to install sealed secrets into a new cluster and then go through and take the ceiling key, the late, the the one that I care about out of cluster a and install that into cluster b, and then automatically things will start working, and so I just need to go ahead and install the new one in there and labeling a ceiling key secret with anything other than active, effectively deletes the keys from the sealed secret controller, but it's still available in kubernetes for manual encryption.

A

Oh, this is weird okay ceiling, key with anything other than active all right. Well, let's get into that we'll go ahead and look at that. Okay, so what we have here is um we have. This is the new secret that I just created: okay and so, and let's make sure that cube seal uh oh and then move temp dot jason to my sealed seagra.json. So now this is actually signed with the new key.

A

And oh this one's cluster wide whatevs, okay and then we can go through and we'll do uh apply.

A

Okay and then we can do cube control, get secrets all right. So we got this up and running. It's been there for a while. Nothing has changed, uh and then marco says: okay, you you need to manually kick the sealed secret pod when you copy around private keys. Okay, yeah! That's going to be another issue.

A

If you do an h, a sort of multi because you may have like when it's time to rotate, you only want the the leader in the leader election to do the rotation, and then you want all the followers to to actually be able to pick up the new key, so that'd be another piece of the puzzle.

A

If you're going to do an ha version of sealed secrets, okay, so now what we can do here is we're gonna go through and we're gonna do uh get secrets out of this namespace we're gonna grab the newer one here that dash o json and then we're going to write that to uh what do we call this yeah pr is welcome.

A

I should get on that um ceiling. Key ceiling, key dot, json, okay. So now we have the ceiling key dot json. This is literally the keys to the kingdom right here, so you do not want to check this one in do not check it. In um so steve, says: ha sealed secrets is painful unless you actually do that yeah. uh Okay! So now I have another cluster here.

A

So this is a kind cluster that we haven't done anything with already I'm gonna do a q, a cube control, apply, dash f, I'm going to actually deploy the controller here and- uh and this thing is up and running, uh cube control, cube system get pods and you can see that it's been running for 12 seconds again. This is cluster two, I don't know if you can see up in the corner here we have kind two versus kind one.

A

um So this is up and running now this thing because it just started it's it. It has gone through and actually created its own key off the gate off the uh off the uh off the bat. So now, if I go through it, if I try and do a cube control apply, dash, f seal. uh No, it's the my.

A

Sealed secret.json.

A

Oh, what did I screw up here?

A

Oh test? Oh we'll do default.

A

It's cluster wide, so it should be fine, okay, all right, so that created it's fine, yeah, hey, no problems, cube control, get secrets, but it hasn't decrypted it because literally this cluster can't decrypt it. It does not have the private key to decrypt that that particular secret. So let's give it that private key so that it can go ahead and actually uh decrypt that secret. So if we go through, we have our ceiling key here, and- um and here you can see this- is the the label name active that says hey.

A

This is a key that that should be active. We can go through and we can do cube control apply, dash, f ceiling key.

A

Why does it have yeah okay ceiling key, so we can go ahead and do this and apply this into the new cluster, and so now when we go through and if we do get secrets in cube system now you can see that we have both of these things. Okay. Now, what marco said is that this isn't quite good enough.

A

One of the other things that we need to do is we need to actually go through and actually kick the um uh the controller to actually have it go through and update and the way I'm gonna do. That is we don't need it anymore, so I'm gonna go through and uh I'm going to delete that cut off time thing that I added earlier so now when I apply the controller.

A

um What did I do wrong? Do I need a oh. This is an array.

A

So now we've gone through and we've actually.

A

Deployment configured okay, so this will roll it.

A

And now we have the new one up and running and.

A

Let's see boom, it did it so now this controller had access to that new key. I kicked it. I added so now I actually have the same sealed secret actually being deployed to multiple clusters.

A

So I think that there's you know another place where I think it would be cool to have uh is to have some workflow really well documented for how you actually manage ceiling secrets across multiple clusters and there you know and there's there's um another interesting feature here would be like being able to actually say well. This particular ceiling secret can only be used with these name spaces. I don't know, I'm wondering what the scenario would be there, but I can imagine that there might be some scenarios that would actually uh work for that.

A

So that is um that is really cool, because I think that's one of the things that's like hey. This works great in one cluster. How do you actually start dealing with it anger with multiple clusters, and so one answer there could be that you actually have multiple sealed secrets based on which cluster you're targeting, but the other answer would be to start using uh uh the same ceiling.

A

Key across multiple clusters, and probably as you did, this you'd also want to start managing the certificate that you're using to sign it, also in a more explicit way. So you'd probably want to turn off the rotation that happens automatically and actually manage that in a in a more sort of explicit way across cluster.

A

In terms of when you update, when you change how you install your ceiling, keys, steve says that we have that set up for pi specific name spaces, that only certain people can create secrets for okay, very cool yeah, and so I think you know one of the things that I think would be interesting in cube seal. I wonder if, um if it supports this is, can you create a ceiling secret from cubeseal without actually running the controller.

A

So that way, if you're managing this stuff, you don't have to sort of like you know fuss with the controller to create a secret so that you can actually go through and then have this stuff use. It and I don't know, there's the recovery, unsealed decrypt, a sealed secrets file obtained from standard in using the private key passed in with recovery, private key, okay,.

A

Re-Encrypt, so you can unscale the secret manually with recovery, but I'm wondering like is there so I? What I have here is I have the the ceiling key right. This thing here that what I did is I manipulated a controller to automatically create one of these things, and then I grabbed it. I'm wondering like if I wanted to have a multi-cluster flow, where the, where I wanted to say, hey controller, never create a ceiling. Key, that's going to be managed from the outside.

A

Can I run the uh the controller in a in a no key management at all and then use cube seal to create a ceiling key out of band and actually manage that as sort of like an admin task? I think that might be that might be for the multi-cluster flow.

A

You know there's this idea like you'd, have one primary cluster and a bunch of secondary clusters. But could you actually say? Oh, I don't have a primary cluster. I actually want to have my uh you know: command line tools, a manual process for actually managing creating that that might be something that would be interesting for that flow, but very cool. So hopefully I think you know I'm running down the clock here and I think we've covered a lot of the the highlights of what we want to do here.

A

So a couple of takeaways here is you know. First of all, um sealed secrets is not a replacement for encrypted secrets in a cluster. um You know it's a uh uh that's one thing to actually be aware of: it's really about sealed secrets. Is how do you get you know? How do you manage secrets through things like get ops uh separately from? How do you manage the secrets in the cluster right? It's about getting things into.

A

The cluster second thing is that rotating the uh the the the ceiling key in that cluster is a little bit of an advanced procedure. It'll happen automatically for you, but if you want to make sure that you, uh uh you can rotate that you have to go back and essentially reseal or re-encrypt all of your secrets. Now you should be rotating all your secrets regularly anyways and they would get the benefit of resealing. But it's something to be aware of.

A

Is that you know there's there's your secrets and then there's the ceiling secret and those things have to be rotated separately, something to to really be aware of, and then the third thing is that it is possible to manage these things across multiple clusters, but you have to start managing those ceiling secrets and making sure that you can use those across the clusters that becomes a manual thing that you have to start dealing with, and then he says joe.

A

What are your thoughts on placing sealed secrets in git and allowing flux, argo cd to pull those into a cluster yeah? That's exactly! The type of thing that you want to do is that you want to be able to put these things and get deploy them like you do like you, deploy everything else, but know that uh uh you know they're not going to accidentally be published to github in a way that anybody and their brother can decrypt them.

A

Another thing to be aware of is um you know there are limits to cryptography and what looks solid now from a cryptographic point of view may not be solid forever and so theoretically, from a sort of pki point of view, taking that cipher text taking that sealed secret and publishing it to the world checking into github posting it on twitter that should be okay, but depending on the environment that you're in whether it's a high security environment, there may be regulations.

A

That say you want to be careful about even controlling your ciphertext and so they're sort of like what is reasonable and what is sort of the compliance and the regulations and the the different. You know the different requirements that you have to hue to so sealed secrets is secure as far as we know.

A

As far as the public key cryptography is secure, it's still worthwhile to think about protecting the ciphertext, and you know, and and there are going to be environments where sealed secrets may not be good enough, and that's just something that you have to know your environment. You have to know what's up to snuff, and you have to be careful about that yeah. um So marco says, uh with respect to multiple clusters. One big feature we're working on is to support kms service provided by major cloud providers. Oh that's awesome!

A

So now the ceiling secret is not actually stored in the cluster. It's something that's managed by some external service. That's actually really cool. um One mo's asking was storing secrets in vault covered for application level secrets. No, this is actually something that would be separate from from vault.

A

um I think that there is, you know I would be interested in exploring sort of the interplay between how things work with vault, where you're actually having the secret store, someplace else and you're sort of bringing them into your workload as late as possible versus sealed secrets, and are there ways that we can actually sort of interplay the management around these things, so very cool um and steve says this is what eric does with flux and sops. He links his blog post in the hack md.

A

Oh, so let's go ahead and look at that um managing what is sops.

A

Mozilla sops has support for multiple sources of key material.

A

So how does this work all right? So this looks like an interesting thing to look at so you're, using something like create key with a kms you're using flux where and how do they get actually decrypted? Is there a controller that does this or you leave it to the workload to actually go through and actually decrypt? It is that the idea.

A

So yeah, I'm guessing that I'm guessing that you leave it to the workload to do the encrypting. So there's no sort of automatic going on there.

A

A

So flux, maybe does flux, actually know how to actually deal with sops directly that'd be interesting, okay, cool! Well, that's it folks! I'm going to sign off. Let me uh switch these things here. Okay and so moz says that soft has nice integration with customized decrypt secrets, but then you would decrypt them on the way to the cluster. Okay, I get it so you'd be doing that decryption client side versus server side, but you would need access to uh to the kms provider to be able to do that cool.

A

So all right folks. So thank you for joining me. um This was a a little bit of a deep dive into uh into sealed secrets, something that I've been meaning to do for a while, because I've never had a chance to actually dig into this level of detail, so really cool to see that happening, and uh hopefully this is useful for folks- and I think you know, understanding the different pros and cons, pluses and minuses the different layers of security. Here, it's complicated stuff, but um but hopefully you know we have the right tooling.

A

The documentation here looks great. uh So hopefully this is uh something that's useful and you all can start applying it in a smart way, and so um thank you for joining and uh we will I'm not sure who's gonna be on next week. Maybe josh, maybe we'll have uh some of the newer folks like um like paul and uh uh who was who did it last week I wasn't able to tune in it was paul and- and I haven't met her- was that carla who was helping out I'm trying to remember um this?

A

Is the cora cora, uh paul and cora. um I heard they did a great job, um uh so yeah, so uh maybe it'll be paul and cora next week. I think we we tend to play by play by air and see who's gonna be doing this week to week but tune in next week. I'm sure we're gonna have something exciting uh from one of us and thank you all for joining us and have a great.

A