VMware TGI Kubernetes, 10 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: TGI Kubernetes 113: Kubernetes Secrets Take 3

Description

Come hang out with Josh Rosso as he does a bit of hands on hacking of Kubernetes and related topics. Some of this will be Josh talking about the things he knows. Some of this will be Josh exploring something new with the audience. Come join the fun, ask questions, comment, and participate in the live chat!

This week we'll be digging back into Secrets and that ecosystem! We'll There is quiite a bit of change in the way that secrets can integrate with Kubernetes and Josh is going to explore some of them.

Episode Directory (GitHub): https://github.com/vmware-tanzu/tgik/tree/master/episodes/113

Table of Contents:

- 00:00:00 - Welcome to TGIK!
- 00:04:41 - Week in Review
- 00:23:01 - Episode Overview
- 00:23:33 - Default Secret Behavior
- 00:41:06 - Encryption at Rest
- 00:58:41 - External Provider Trade-Offs
- 01:05:31 - Integration with Secrets: App Level vs Platform Level
- 01:09:02 - Vault Secret Injection via Agent
- 01:39:31 - Sealed Secrets
- 01:46:09 - CSI Secret Driver
- 01:51:46 - Wrap-Up

A

Hey, what's going on everyone, hello from Colorado, happy happy, Friday,.

A

All right, let's see who we got coming in if you're joining us feel free to say, hey and chat if you feel so inclined, I'm, Josh I, think many of you have seen me before, but I'll be hosting today, I'm signing in from good old Colorado.

A

Sebastian welcome Smitty glad to have you while Leeds same wishes to you. Thanks for joining us again, always a pleasure to see you all heed: Steve, hey how's it going howdy from across the pond over here.

A

Mozz hello, Martin, hello from the Netherlands awesome, welcome Maddie welcome back as always Jess how's it going Felipe from Paris, hey Philippe, thanks for joining us in joining us today, hyeme, hopefully I'm saying your name right, hi you, or at least clothes from Israel, welcome, welcome it's a slot.

A

I might have butchered that when I apologize from Finland welcome from Finland really great glad to have you with us today, all right we've got Tim Downey in welcome Tim Robert Dublin glad to have you Robert another person from across the pond Phil from New York City, hey Phil, how you doing thanks for joining us, hoping you're, safe out there Pedro happy Friday to you too. Thanks for joining us Thorsten from Germany, hey thurston! Welcome! Welcome, karthick glad to have you. We got Vivienne from Munich sweet Grigor from Lourdes Moldova awesome. Welcome! Welcome, Phil, Tim!

A

Hey glad to see you Timmy! As always new day. Hey how's it going Yash from India welcome, welcome, Meester from agile defense, welcome, Yuka Jukka from Helsinki. Welcome. Welcome glad to have you, hey, Michael, all right, all right! We've got Sahad welcome so hard, all right, let's get into some of this stuff and then I'll check back in a chat and see who else is joining us if you're just joining us now and you feel so inclined feel free to say, hey and chat super glad to have you all here.

A

I'm really excited about this episode. It's a topic! That's been I've, had lots of conversations recently about for random reasons and, as you might know, Duffy did some really awesome. Episodes on this topic- I, don't know if you're in the chat Duffy. But if you remember the date it was maybe a year ago, I don't know if it's been that long, but I think was it was like episode. 65 66 will will add the episode numbers inside of the the chat, the show notes in the hack MD.

A

But you know a lot has some things have changed right and I think there's a lot of really cool things going on in the secret space, so we figured hey, let's kind of cover it again, we'll talk about. What's new, we'll review what's already been covered and all that good stuff, so I'm super psyched to have you all with me today and let's just take one more look and see who else we have joining us all right, Wow!

A

There's a lot of people today great welcome everyone trying to remember where I left off Sahad welcome from Kashmir welcome! Welcome roberts we've got yet from turkey. Brandon annarbor, hey Brandon, thanks for joining us today, glad to have you vincit from Scottsdale, welcome, Vincent, Caleb, Charleston, North, Carolina or South Carolina. Sorry, not not! North, Carolina, hey Caleb, Brad, hey how you doin Brad from Virginia we've got Rudolph awesome, we're I'm, ready to learn Rudolph, so we're in the same place, awesome Dan from Toronto hope, you're doing well. Dan we've got Tom's from ton, Zia Tanisha.

A

Sorry, hello! Glad to have you right on sue from Paris welcome got another person from Paris. We've got Olaf from Denmark as well awesome Wow. We have a lot of people from all around the world today. Hope you all are super safe out there with all the craziness that's going on, but maybe we can take our minds off that a little bit and talk about some kubernetes how's that sound. So we can review what do we got going on in kubernetes land? Anyone in chat played with 1:18, yet I'm curious.

A

Let me let me know in chat if you've, if you've tried out 118 around 118, 118 . one is out and George as always went through and gave us some really cool links to some cool blog entries. You should check out I'm, just gonna high-level go over them. Real quick one is around API priority and fairness alpha.

A

This is super super interesting it you know might take my take on it is it kind of lets you put sorta like quality of service type things into the control plane, so you can basically say you know what gets priority inside of their communication with the API server. There's a couple ways that you can match match the requests based on what you're requesting for what the verb is I think there's even ways to do it from like the account that's requesting it. Maybe I haven't looked too deep in this.

A

It seems like one of those really cool features where, if you're running at like a pretty large scale and your API server is getting hit pretty hard from you know things all over the place. This could be kind of a cool thing to check out, especially over time is like you know, we have more and more services in the cluster that are talking to the API server. So this is a really cool set of knobs that you should read this blog post on when you get a chance, server-side apply just a small mention.

A

If you haven't checked out server-side apply. This does a lot of really cool stuff, but it kind of does the some of the diffence overside I.

A

Guess you could say um one of the cool things that's intriguing to me and I haven't gotten the chance to check it out myself is that it opens up the notion of being able to represent what the fields are managed by, which is always super interesting, because you have all these fields in the manifest that might be mute, be mutating inside of the cluster from certain things or might be mutating from your you're applying and changing of manifests.

A

So this is a really cool one to check out I'm looking forward to it other ones that we've got. There's a topology manager, kind of cool I, don't know how much you all are familiar with like Numa it, but it stands for where is it at non-uniform memory access?

A

So if you've ever looked at like a server before with like a multi CPU architecture, there's all these different RAM slots and there's this notion of, like ones that are local to that group, and then there's the other ones too, that it can basically go through this inner socket connection, which is pretty low level.

A

The the cool notion about this is this: adds this idea of topology, where the the simple way that I think about it, at least for now, until I dig in is like you can use topology to kind of dictate, where you're going to schedule, something to make sure that it kind of lands in a place where it can access that localized memory, so basically ensuring that you can have. You know the least amount of latency possible, the most efficient workload possible.

A

Gpu GPUs fall under this Numa category next, due as well again making sure that you're not adding extra latency by not using the hardware, that's available localized to the socket, so another really cool one, probably really in the weeds. For those who are trying to squeeze that memory out. I'm guessing topology manager, especially for like GPU GPU use cases, could be especially especially interesting and cool, so really sweet, ingress, API improvements.

A

This one is interesting, I think the two big takeaways yeah here they are path type, which is a field that will let you specify how ingress paths should be matched in an ingress class which is going to specify how ingress is are implemented by controllers. So this is one that's kind of near and dear to my heart. A lot of customers. I work with they'll run. Multiple ingress controllers, like some will use a model where they'll tell development teams, hey just deploy your own ingress controller and run it.

A

Some platform teams offer like a kind of layer of ingress as an inbound access, and sometimes even platform teams will offer multiple ones right. They'll, say: yeah, I'm, gonna, run contour and engine acts and traffic side by side for a variety of reasons. Now what that ends up? What ends up happening? Is you have this complexity of? How does the controller know what what ingress to satisfy right? What ingress belongs to it? And you get a lot of variance, because some controllers would expect a specific annotation.

A

Some controllers would expect some different qualifier, so I think what I'm most excited about is this notion of ingress class, where you can kind of specify the controller responsible for it and that inside of here you can see that we have the ability to reference the ingress class name as well, so yeah.

B

A

Pretty neat overall kind of a kind of a cool feature all right. What do we got going on and chat as well? Some conversations around debug, pod, yeah cool cool.

B

A

Good sounds good, tabbed, ID, right, cool, awesome, good conversations, I, don't think there's anything I need to add to it, looks like Duff's, got it under control and chat, and then there's also a post on Windows CSI support. I, don't know a whole lot about this. I'm not gonna lie um I. It's it's super interesting. It's really cool that the the cubelet that runs on windows nodes will have the ability to leverage that same storage interface, to provision things I'm, trying to find the graphic that kind of showed it for me.

A

Here we go the cubelet exe and then the underlying went windows hosted provision storage. So if you are running workloads again, these are workloads on windows and nodes. This could be a really cool one to check out, and it's just great to see the health of the the Windows community and kubernetes and that they're kind of really moving along with a fast follow around CSI compatibility and all that good stuff. So really really neat really really neat piece.

A

Yeah and while leads question in chat, I'd also be super curious. Who in here does use Windows nodes? Is anyone happened? Use them in production, I haven't personally run into a use case where a customer was running Windows nodes in production, but I have a lot of customers. Ask me about it, a lot right. So it's I think it's on the horizon. I just personally haven't haven't used it much yeah, so cool cool.

A

All right, good, stuff, okay, what else we got going on here, so the working group for resource management has come to an end. It's met its goals. It seems like the the notion, as well and I haven't been close to this working group by any means, but it seems like a big thing is a lot of the work that is going to kind of continue on has been subdivided into other, more specific working groups. So just an FYI.

A

If that's a working group, you were at all involved with or working with and there's also until at least April 14th, so we're getting close an application form if you'd like to shadow the 119 release, so do check that form out I think it's probably just a Google Form. Let's check yeah, so release team shadow application. If this interests you you know, get get more involved with the kubernetes project kind of get an idea for how these release things work.

A

Someday I aspire to do this myself when I get just a tiny bit more bandwidth with customer work, but it would be a really cool, really cool experience. To kind of see you know how do all the cogs work together in this, this massive machine of of kubernetes release, so awesome awesome all right and then also if you want to join kubernetes office hours, they are April 15th. This case it'll be next Wednesday.

A

We have links in the hack MD for the EU session and the west coast us session so be sure to check that out. So lots and lots of cool stuff in the news this week for the Cates Court. Now the ecosystem bits as well. Let me just kill off my slack before I get notified endlessly. Here there we go okay. So let's talk a little bit about the kubernetes ecosystem, so one interesting article that came out of hub spot, what is it hub hub spot was migrating zookeeper into kubernetes.

A

So their use case is pretty interested interesting in this article because they did it without downtime. So as I understood it greeting through it quickly, zookeeper runs on VMs right. They wanted to run them inside, run the zookeeper nodes inside of kubernetes, but basically kind of bring them into kubernetes without introducing a switchover or downtime. So they did some kind of cool stuff in here where they talked about how they took the kubernetes service and kubernetes endpoints.

A

Well, really the kubernetes service right and they had that going through to an endpoint and that was kind of level above the zookeeper instances. So basically zookeeper. The white box here is still running on, like a VM as I understand it right, but there's this kind of abstraction, where clients could take this path in going through these kubernetes services. So then they take the zookeeper clients and wild zookeepers still back on the virtual machines.

A

They have the clients, go through the services and endpoints, which is again pretty cool, just kind of routing through business as usual, and then what this enables them to do is over time. They can then have zookeeper itself communicating through these services and endpoints so kind of neat right, like they really leaned heavily into that service abstraction in kubernetes and made it so that it was all working in this kubernetes networking model, and you probably can guess, what's next kind of similar to Etsy D.

A

You know I'm, not an expert on zookeeper by any means, but the general notion is you want to make sure when you add a node, it reaches consensus right. So basically, you can swap out one of these zookeeper nodes with a kubernetes pod running zookeeper, and then the consensus can kind of build up in this flow and as far as the clients are concerned, right, it's business as usual they're talking to zookeeper. They don't know whether zookeepers running in a pod or running in a VM.

A

So then they basically had zookeeper running inside of kubernetes, which is kind of neat, so yeah I thought. This was really interesting and again just shows like the genius ideas you can come up with. If you just really think through like how can I use these kubernetes primitives to really benefit myself right, so we also have some hacker news comments in here to check those out had some interesting insight a lot of times when you talk about things like zookeeper and kubernetes. My experience is that folks will kind of get into this political debate about.

A

Should you be running zookeeper at all like and the thing that I always tell people and I work with my customers is there's nothing wrong with zookeeper, it's great like. Maybe you have a bunch of apps that already integrate with it right? Maybe you don't want to change the world and boil the ocean? Maybe your kafka system integrates with it really well and you're happy with it right. You know.

A

We love that CD, because it's the the data store the key value store, if you will for kubernetes, but it doesn't mean that applications have to use it too right. So you know it's. It's just really kind of cool to see how a lot of these workloads things like zookeeper. You can Rotom super well on kubernetes and keep a flow that your developers are somewhat used to with their applications. So pretty pretty cool stuff. I'm gonna check chat one more time here. What do we got going on to do?

A

Alex? Hey, welcome glad to have you Scottie nice to see you Scotty welcome, welcome cool cool right, yes, Steve Kafka and zookeeper for kubernetes for two years now, I just finished an engagement with a customer actually, where that was one of their primary use. Cases was Kafka in the cluster. Now Steve I'd be curious. Just for my own knowledge are you able to disclose how you run Kafka? Are you using that?

A

Is it pronounced rimsey the Kafka operator, or are you using like a helm chart or something like that to run Kafka in the cluster I'd be curious. Alright, so let's see what else we got going on here, so service mesh interface, intro to service mesh interface in the CNC F, so pretty cool I got to do my homework on the the SMI or service mesh interface, but love the notion with the emerging service meshes that we're starting to standardize behind an interface all the people that are working on this.

A

Thank you so much for for helping us build. You know kind of a consensus around, so it seems like we have this progression quite constantly with kubernetes right.

A

It's like, if you remember back in the day like we started, building these networking tools and then we were like ok now, let's do CNI and then storage started happening and we all started doing storage and then all the sprawl was happening with how you achieve storage and then CSI finally came out, and the convergence around these specs has been hugely hugely beneficial in making sure we can understand how a lot of these different plugins work so check that out. If you get a chance, what else do we got here so docker compose?

A

Has their specification published so could be an interesting read for you all in github here really really deep in the weeds, always cool to kind of see, you know docks and how people are thinking about these plans and especially for something like compose I've got a lot of adoption. You know what the spec looks like and what they're implementing against there's a another kubernetes visualization tool. That's kind of interesting cube, vyas like when I read it I want to keep saying cube envyus for some reason, but this this one is kind of interesting.

A

It looks super super interactive, so good job on the UI folks who are working on this project. It's kind of cool like let me see if I can find the picture. I was looking at before it's got like a bunch of columns and almost yeah yeah see so you've got these different columns. You've got like in this case. They represent namespaces, you can click on stuff, as you click on it. It shows up in the side here.

A

I think one of the really cool things that this is doing, which I'd love to see in more and more of these kubernetes visualization tools. Is it's proactively warning you about things right, so you can see like the little radioactive sign, I think that might represent over privileged workloads. They have time machine functionality to show you how things have changed over time, which is a pretty neat idea. I'd love to know how that's implemented.

A

How do you kind of keep all that history, but it's a pretty cool tool now I will say this is a tool that you'll deploy to your cluster and as a client, you would go use right. So it's not like if you've ever use something like octant, where you run it locally. This seems to be more of like a centralized, kubernetes dashboard. If you will so definitely definitely a neat one worth checking out. Ok, a couple more here, so Argos CD has been accepted in the CNC F.

A

If you use Argos CD a little clap for them great job. Argo is really awesome. How a lot of this get ups and kind of New Age deployment models are starting to make their way into the CNC F and then last couple ones we have here to sidekick. So this is kind of an interesting take. It comes from the Mineo group on running a little sidecar load, balancer that can load balance and do health checks and things like that between pods.

A

So the way that I think about sidekick is it's not like you would deploy a full on service smash right. It mostly just solves that kind of a little sidecar load-balancing concern where it can understand where it can send traffic.

A

It can make sure that can avoid sending traffic if the the the health has failed of that thing downstream that it might be sending too so pretty neat project I'll I'll be kind of curious to see how some of these more lightweight- let's call them like hmm I, don't know if service traveling the right way to put it but kind of like lightweight proxy service, routing projects, kind of pick up as a lot of the like larger kind of heavier weight. In some ways, service mesh projects are evolving as well, so pretty interesting stuff.

A

For many, oh alright- and this was a really cool one from George to, if you're, really interested in just taking your virtual backgrounds to the absolute next level, actually, like seven more next levels, been the elder put up this really cool post about how you can go in and modify how these backgrounds work. I'll, give you the too long didn't read. You can take a background like this right and you can basically have it. Do crazy stuff, like in his very last post here. Have you as a human, come in all like kind of I.

A

Don't know like scratchy and like as if you're you're kind of I think he puts it dialing in from the Millennium Falcon or dialing into the Millennium Falcon right, so pretty cool stuff. If you really really want to impress your coworkers, this would probably be the way to do it. Forget all this kubernetes stuff. Let's, let's work on the virtual background, it seems to be the new hotness right, so very cool thanks for putting that together, Ben and then Steve who's in our chat.

A

We have an article from him on how we do get ops at metal so check this article out from Steve it's on medium thanks. So much for putting this together.

A

Steve I love, reading more and more about how get ops is being achieved, like I I'm, usually like dude a lot of my customers, kind of like on the edge where they're they're like I, want to use, get ops I like the model but they're just a little worried about the tooling maturity they're worried about the DevOps flow change, this change that their developers are used to in all that jazz. So thanks for putting that together, it's always great to see you know kind.

A

These real-world worst use cases so shout out Steve who's, also in the chat, great job. All right, let's see who else we got going on here now all right, Wow lots of chat, very cool cool cool yeah so be sure to check out TGI, K, dot, io / notes, if you haven't already all these good links are, are in here as well: okay, cool Tim, your coworker became a potato I'm, guessing you're, referring to slack backgrounds, hopefully so at least alright.

A

Let's get into some goodness about secrets all right. So here's what I'm thinking for tonight, whoever put the old episodes in the show notes. Thank you so much be sure to check those out. 65 and 66. I am gonna start with a bit of a review. I threw some diet, well, actually reuse. Some diagrams. We made before to kind of talk about how secrets, work and I think that'll just be a good primer for all of us.

A

So if you haven't watched episode, 65 or 66, don't worry you're, not gonna, be completely lost here by any means, but I'll talk about some of this stuff in review that if you want to deeper dive into, you should check out Duffy's episodes, as always. Duffy goes into great depth and it's super super educational. So let's go ahead and start with the the default secret model. Okay, so default secret model. What does that mean inside of kubernetes?

A

Well, I've got a quick diagram, then I'm just gonna bring up here and sorry the the iPads out of maintenance. So you won't get to see any of my kindergarten drawings today. But that being said, we have got hoops sorry I, just trying to get my keyboard shortcuts here to work, I totally random, but today, right when I started my day, I have one of those. Actually let me let me grab it. I'll show you I have one of these like really nice control, keyboards I, think a lot of you have seen them.

A

They're they're neat because and I know a lot of keyboard people do it now, but like they don't have you can kind of clean it easily, because there's no inset down there so I woke up this morning, got on my computer to hop on a call instantly smashed my coffee into the keyboard and just drenched this keyboard. So now it is pressing Keys randomly throughout the day, which is a huge bummer right and now I am using this keyboard instead, which is a very different experience.

A

So if my typing is very bad today, you're gonna know why so just just bear with me on that. Hopefully, the keyboard will dry out I unplugged it right away. I, probably plugged it back in a little too soon to test it, but I. Let it dry out a bit more so yeah. It.

B

A

Yeah, it is a downgrade Duffy. You are. You are not wrong. My friend anywho all right, so secret behavior. Let's just talk about it at a high level right so kubernetes has this API object that we're fairly familiar with that's called a secret right and whether you're explicitly using secrets today I'd be willing to bet you that you are implicitly at minimum using Secrets today right so I've got a cube cluster here.

A

So if we just get nodes, it's a to new to node cluster running in my home lab right, and if we do a cube, cuddle, get secrets, here's a secret! Does anyone know what the secrets coming from right? So it's coming from I? Guess the names right there right, so it's coming from the service account.

A

So when you establish a service account in this case, there's a default service account in the default namespace, there's a token associated with that right and then that secret excuse-me becomes available to pods that use that service account and those pods can use that secret to communicate with the API server. So that's kind of how kubernetes implicitly runs secrets, but we know from experience that like when we run in kubernetes.

A

We want to be able to take secrets we care about in our application space and then move those over into a store somewhere that can hold database credentials. You know access tokens all that good stuff right. So that's that's something I'm sure we've all played with to some degree. Now. How do you do that right?

A

Well, there's a secret object in kubernetes, so what we can do is we can take an example of a secret object right and we can apply that secret object to the cube API server and effectively, depending on how you set it either you'll do the base64 encoding up or you could use this thing called string data which will then base64 encode it when it goes into at CD, right, you're, gonna end up with base64 encoded string data in at CD right now, kubernetes gets a lot of slack for this notion right a lot a lot of times and I'm.

A

Here you could google and find the blogs that are like. Oh, my gosh, why does kubernetes think that base64 encoding is a security mechanism like that is atrocious right and it's not really that way like I, don't think anyone who worked on kubernetes is thinking like hey, let's just pay, 64 encode the data and no one will ever figure it out. In fact, if you saw the tweet for this episode, I tweeted grab a drink right in base64 and all of you that I'm aware of that responded figured out what I was saying right.

A

So there's no hiding base64 encoding its encoding right. It's not encryption, okay. So what do we do here now? The key thing to again really grok is that kubernetes offers a separate API for secrets, and that's really the crux of this because away from config Maps away from pods away from all this stuff, we want separate our back in separate kind of settings that wrap these secret objects right. We want to be able to put specific permissions in place.

A

We maybe want how the verbs interact with Secrets to be a little bit different than how the verbs might interact with a config map. So this is kind of like a base layer saying like here's, this API here's, these secret data, things that you can inject figure out the safety by either extra configuration or integration with something else, and that's what we're gonna be talking about today.

A

Okay, so when we get into this a little bit again default encryption, we're gonna kind of lean into the notion of encryption at rest, but you know it might be helpful. Real, quick, let's just create a super simple object here, and then we can kind of build on this to make sure we're all we're all grokking at and again I'm just gonna glaze over it fairly quickly. Duff goes into this stuff in detail in his episode so be sure to check that out. So all I'm gonna do here.

A

Excuse me: I'm gonna, grab a secret from the good old interwebs. Give me just a second okay. Actually, I'll grab one of those string, data ones, so, okay, so we'll do them seek well. Actually, let's get into the TGI K folder. Just so I don't lose this stuff, so I'm in TGI, K episode, 1, 1, 3. Okay, that's way, I can commit all this in case you all want to play around with it yourself right, so we'll do them secret example, alright and then we'll do dot. um Oh okay, great! So here's the deal.

A

This is a secret object, secret API, no big deal just like our diagram. Where is that diagram? Just like our diagram? We're gonna save this and it will base64 encode it for us right now. I use this thing called string data, and this is basically a way for me to put in the raw text and then just basically when it gets stored in that CD it will be base64 encoded.

A

The string data will still show up in like the what is it the annotation that tells you what you applied so again, not that basic C for encoding is a security mechanism, but just be aware that that will show up in like true unencoded plaintext. So let's say this up: let's do a quick apply right, so we apply secret example, yellow now here's the deal I got secret, my secret in the cluster right and, as you could probably guess, if I cube cuddle get secrets, let's make sure I did it in default.

A

Right, yep, okay, cool, get secrets. My secret okay I can print out the secret now. This does not necessarily mean that the secrets not encrypted it is not encrypted because I haven't set that up yet, but it doesn't mean it's not encrypted. What this means is my user has the our back to go and get the secret right, so I can keep this experience of injecting secrets into pods, getting secrets in the command line and so on as long as I have our back.

A

So that's not so much a problem, that's more of an hour back concern. What is a problem for us, though, is of course, how the secret is stored inside of our our buddy at CD right. So how is that secret living inside of that CD? Well, let's go ahead and check it out.

A

So I know in this case: if I do a get nodes Oh wide, we will have master 0 and here's the IP address so I'm just going to SSH into master 0 real, quick, all right, I'm going to interact with with @cd on the host now Duff again glad you're in the chat. Duff Duffy actually has a really cool like little I care.

A

If it's a script or just a yellow file for a pod, where you can hop on to a node again, this would only be for testing purposes and interact with Etsy directly from a container. So maybe, if you've got that link around Duff, you could send it in chat. It's really cool, but I'm. Just gonna kind of do it manually here with a shell script.

A

Okay, so here we go, I will go ahead and call this at CD, C, I, guess, shell, okay and then we'll go ahead and just make sure it's executable once we're done so sure mod + X for the at CDC at CD, C, shell, okay, cool great! Alright, let's paste this in real quick here: oops caps, lock, oh no I'm on a different keyboard. It's killing me okay! So this this is a little script that I'm running basically I'm assuming that I'm on the master node!

A

Oh wow, do you all see those little glitches in my screen? That's weird, hang on one second I think it's uh I think it's my compositor crap. Hopefully this doesn't blow anything up if I.

B

A

Will be back but I, don't think I will peek. Oh sorry, I did peek he'll there yay okay, you all can still hear me right. Hopefully so I should have killed the weird artifacts that were coming in okay, so at CD CTL, which will run on the host and then will call at CD now it needs to know where at what certs to use for F CD. So since I'm using cube admin, I know where these certs are so again.

A

This is just a little helper script so that I can execute commands with at CD and and again figure out where there are secrets and things like that. Okay, so now, the only thing that I do need to go ahead and get to test this out is at CDC TL. So again, many of you have probably used this before. If you've played around with kubernetes at CDC. Tl is a command-line tool that will let you that will let you interact with at CD itself.

A

Okay, so now we've just got a go out and get at CD CTL. Let's see if there's a easy download link anywhere in here or if I got to get it from the hole hole at CD releases page, let's see that's CD, CD, okay, just probably a direct download somewhere and I'm, just not seeing it, but let's go ahead and grab the hole at CD. Tar I, don't think it should take too long. So we've got amd64, alright cool. So let's go ahead and bring that down.

A

All right, while we're downloading, let's check it out here what we got going on.

B

B

A

Right looks like the conversation about mechanical keyboards is alive and well yeah. You get so used to your keyboard right. You get really attached to it. I'm I'm, already pretty sad. What's funny too, is I just put brown switches in it too. So that's a bummer just spent like a couple hours putting brown switches in. Thank you. Okay, cool @cd, ok, cool Duffy put his at CD client in there too. So you should definitely check that out.

A

Ok, cool, alright everyone. So let me go ahead and unzip this real quick, so I will unzip the at CD tar ball. Oops got to use the right two commands here so.

A

Oops VF at CD there we go okay, so there's that CD CTL, ok, where, where oh, where are you at CD, CTL you're inside of G's CD at CD? This point I should may have been better off using Duffy's pod. Okay, there's that CD CTL, so at CD, okay, perfect at CD, CTL, lovely! Alright, let's get this thing in my path and then we'll move on so at CDC, TL.

A

All right user, local bin and pseudo alright. So again, just so, you have the context I'm on the host of my master node right now right. So if we just do a quick at CD, CTL make sure it works, looks good all right now, let's see if we can make our little our little wrapper script work as well, so we called it at CDC, shell, okay,.

B

A

Then what I will do here just to kind of bring this up is run a quick command with it to get a specific key. So what is the key that we used we'll do the default namespace real quick, so this will be at CD CTL paste this in here hit enter.

A

What did I do wrong permission denied client key? Oh no, my API server client key. It's not happy about that! Maybe it's a pseudo thing, because it probably needs to open it up and it's probably protected there. We go okay.

B

A

Did I what did I call that what did I call that secret again? Does anyone remember it was d.

A

It was TGI K episodes, 1, 1, 3, okay, let's see what we got here so I called that secret example yamo. We called this one, my secret. Ok, so, let's see if we can get to this path secrets my secret.

A

Ok, nothing came back secret. My secrets: it's been a little while, since I've used at CD CT I'm trying to remember what the what the command is again. So if I is it lists at CD here, let's get rid of this, so we've got at CD CTL help. Ok, I'll make this a little bit. Smaller. Sorry for everyone's eyes should be under.

A

Yeah, okay, so I've got list. Oh you know what I wonder. If I'm am I setting the right API version for at CD CTL, let's see here, that's CD see yeah I'm putting in at CD 3 cool, get from keys grep my secret, oh good, call Duff! Let me try that oh, it's, my secret is it that might be what it was. What did I put in oops get.

B

A

Let's see if your command there work stuff, get keys from secrets. Okay, so we'll do this! Okay at CD.

A

A

Okay, what did I screw up here? huh My secret.

B

A

Where is the it's so long since I've used at CD CTL? What is the list? The list command? Is there Nautilus command as it just get I think that might be the case. hmm So what do I have here so I've got? The house did I not have the name space before so it should be registry Secrets default.

A

Alright, my secret okay at CD, get nope huh I wonder what the heck I've got going on with this little script. I usually use this guy to troubleshoot.

A

Let's see no list in v3, okay, no list command, + v3, let's see if your recommendation works, there ya know huh super weird I, don't know what's going on with it. So, let's actually, let's just double check in the cluster real quick cube. Cuddle gets secret.

A

Okay, so I've got a secret in kubernetes called my secret, alright, which seems to be okay right, so I'm inside of the master, node and I am authenticating okay, so if I just get it should be under registry right.

B

A

See yeah there's no stuff; well, it wasn't a and one of them I tried. So I would think that this would work fine right at CDC TL there. It is okay, who knows what millions of things I might have been doing wrong there, but we got it good job. Everyone good job chat. So here it is. This is the point we're trying to make right is that it's not so much the fact that I can get the secret back in cube cuddle.

A

It's the fact that if I went on the node right and I just basically looked into at C, D and theory, I could find this type of data right. So this is all stored inside in here in plain text. Now what does this mean for us here? So the question really becomes. What should we do about storing the the secret data right now?

A

In our case, we're especially vulnerable I'd argue because not only is the obviously secret and plaintext but I'm in a co-hosted environment for my control plane, which means my API server and SCD are sitting kind of on top of each other, so the notion being since they're running on the same host as if someone breaks in to get access to the API server at CDs sitting there as well, there's not like a hypervisor isolating them or or anything like that, so I'm in I'm, in a bit of a rough spot.

A

Now here's where we're just gonna quickly kind of glaze over some of the other options and again this will be just a high-level review. Now we have two paths that we can go down and one of them is encryption at rest. Another one is also encryption at rest, but it uses this notion of envelope encryption right. So let's talk a little bit about these two and again I'm gonna, I'm gonna quickly show a diagram here which you all can can hop into those other episodes too.

A

If you want to check them out in greater depth, let me just quickly grab this one for my diagrams folder here. This one is secrets secrets: where are you secrets?

A

Okay, so I think actually I might have called this one encryption at rest. Yeah give me just a sec. I know you can't see my screen just yet I'm just trying to pull up a file super quick here. What is this one called this one is called secret encryption. Okay, let me pull that one in secret encryption at rest there. It is alright, great okay, so here's a model that we could theoretically go forward with. Okay. This is what we would just call encryption at rest inside of kubernetes.

A

Now, in the episode that duff did, you all might have seen him set this up and basically what he did. I am pretty sure. My memory serves correctly that he set it up is basically he went in and set up what we'd call an encryption key inside of the API server itself and I'm just gonna pull up a quick snippet from the docs. So you all can see this as well encryption at rest kubernetes. So if you haven't checked it out, you might find this page to be pretty darn helpful. Okay.

A

So basically what what we did in a previous TGT gik is we set up this encryption? Configure s configuration manifest now. This thing is a static file that you'd put on the host okay, so the idea is, you would provide some keys to encrypt and decrypt right. So in this case you might just have one key. You can see that there's a couple different key types as well all right identity is this blank identity. One is a default one that you're gonna want there for things that aren't encrypted but effectively.

A

When you put one of these keys in place and store it on the API servers file system, you update the API server manifest and tell it hey. I have an encryption configure configuration for you? Api server go ahead and load it right, so the API server can load this encryption, configuration and start encrypting and decrypting these secrets. So, basically, what you do is you go in and you make one of these files and you can choose which option that you want here, um there's a little bit of contention around what the right option is.

A

Some people feel like the the main recommendation, is a little bit more vulnerable than others. A lot of people really like this. This thing called secret box, but in the encryption community as I understand it at least it's a little bit less mature, although I think it's been around for many many years. So if you do go down this path, definitely like if you're, not a security expert like me, I would go to my security team and be like here's, my encryption mechanisms. You know what do I need to think about.

A

As far as choosing one of these different one of these different approaches to encrypting right. So basically you choose one of these and you make a key and then your data, as it goes into @cd, is gonna, be encrypted going forward right. So looking up inside of at CD here, I would just see a bunch of encrypted data right and again, I didn't want to just rehash everything that Duff did so I just took a snippet before we did. This did this session on what that encryption data would look like.

A

So if I were to encrypt this right now and add one of these keys in the exact same little piece of secret data that you see, there would actually look like this. If I did that xcd query right so, as you can tell with this data, it had gone into the API server, it went through the encryption key to encrypt it, and then it persisted into @cd inside of this mode.

A

Okay, so pretty cool I mean, if you think about it, that does take us forward a little bit right, we're a little better off, because now, if that person gets access to at CD and they look into at CD, they should only see garbled text like this right. But if you think about it, you know everyone in chat. If you kind of look at this diagram here, what problems might this I, don't wanna, say introduced, but what problems might still exist in this model?

A

Right cuz, you might think well, okay, just put an encryption key in, like everything will be fine, but think about, like my host right now think about my host that has the API server and at CD running on the same node. What are some of the key things that are problematic here right.

B

A

While lead, you said so, is the key and clear text in static manifest for the API server yeah pretty much well. You've hit it on the head there right. So this key that I would be putting in place right from the encryption example. Yeah it's going to be sitting in plain text on the API server right.

A

And to and I guess that kind of answers your question Maddie as well right like these keys- maybe you maybe you hold them in a repo somewhere, but at the end of the day you end up putting them in this encryption configuration on the host with the API server right so think about this I as an attacker go in and I get on to the node, with your API server well in a co-hosted deployment model, which many of us do run co-hosted control planes if we're not using like eks or something like that, the encryption keys sitting right there on the same host right the same host at at CDs running on right.

A

So you know we have a little bit of a concern there right. We have to kind of think about the fact that I could just grab that key go in and decrypt and all that kind of stuff I also have to worry about people getting access to that key cuz. It's kind of like I, put a lot of my eggs in one basket. They're my keys now live my private keys, live on the API server, something to think about right.

A

Another big thing too, that might not be super obvious, is now you have you own this kind of static process right that you're gonna have to do key rotation with you're. Also gonna have to make sure I mean those of you who run a control, plane I'm guessing you run more than one control plane, node right unless you're, just yoloing your your your kubernetes clusters or you're doing some weird, like edgy use case you're, probably running a multi master cluster.

A

So now, I have to take this config make sure it's on every single master, inconsistent and I need to rotate these keys over time. Now the rotation is kind of cool with this again, if you want to dig deeper look into our other episodes, but the idea with these is that you basically these are in order, is the way to put it. So basically, the idea is when you need to bring a new encryption key and to encrypt all new data.

A

You'd, add it to this list and that encryption key would be the new encryption key used to encrypt all data. Now, one thing to know like say you had an encryption key, be compromised. All your existing secrets aren't just automatically going to be Rhian cryptid right. The old encryption key would usually stay around in this model, at least for some amount of time, so that those secrets could still be decrypted as a now.

A

If you were to change the secrets over time persist new data there, you know all that good stuff they will get encrypted with the new encryption key right, but all those words right. Those are that's a lot of kind of overhead to think about rotating your secret keys. How do you kind of you know? Do all these different bits and make sure they're coordinated on all the API servers and all that good stuff? Okay, so you gotta think about those bits. Now you might be saying then well Josh.

A

This doesn't sound very good and that's not necessarily true. This is better right. The key thing with these security discussions at least the way I think about it. Is you start here which is base64 encoded data right and then you try to at least do as many things as you can to make it better. So if this is your only option, this is better than not having it obviously right, but there are some other options that are super super cool. Now, that's encryption at rest.

A

In a nutshell, you put the key statically on the filesystem, the API server picks it up. It encrypts everything going into the sed. What about KMS envelope encryption? Now this one I really really like the idea of it. I would be super interested those of you in chat.

A

If any of you are using envelope, encryption, the kms plug-in system and cube, please shout it out in chat I'm looking for data here, because, aside from people who run in like gke or run in as ear I, don't usually see this being used primarily because some of the plugins are a bit immature, but I love the notion of this. So let's, let's talk about that real, quick okay. So if we go back to this encrypting secret data, one more time, we've got all the providers right.

A

This is all the static stuff and then we've got somewhere in here. I, don't know if it's in this, maybe it's on the same page. Let's do kubernetes envelope encryption, kms plugin. Hopefully that's descriptive enough. That's what I'm looking for okay! So this is a cool model because it's going to use a key management service, a kms to go out and do some fancy stuff with encryption. So a kms plug-in provider could be vault, it could be Azure, it could be gke. A lot of the cloud providers.

A

I think really have the competitive advantage here because they already have like true kms is running in their clouds right.

A

But let's actually look look at a visual here, because kms is like for me in my mind, just kind of like melts when I think about envelope encryption without a picture: cuz I'm a very a very visual person. So let's let's look at this everyone- and let me know if this- if this helps make any sense. So let me just once again open up a new diagram here for you all, and this will be secret envelope. Encryption I believe secret. There it is okay, our diagram is about to get a lot more detailed.

A

So this is a diagram that I did, that replicates to the best of my knowledge or represents the best of my knowledge. The standard kms flow for envelope encryption, so pretty cool. So here here's the idea here- you don't have to be an encryption expert to grok this, and if anyone at chat wants to add some clarity to what I say here, please feel free cuz, sometimes I'm, not the clearest secret comes in right. Now, don't worry about too much about the word generate, but assume somehow a data, encryption, key or you'll see it.

A

If you look in Docs, it's called a deck. Dek will get created. Okay, now here's here's kind of the brilliance here right. You make this encryption key that encrypts this secret. So now the and we're still all by the way, we're kind of like in the API server right now that there's some technical Interac yura sees with this.

A

But, like you know for illustration, sake bear with me right so now we're inside of the encrypts data with key, so we go in and we encrypt the secret right and then what ends up happening is we use KMS to encrypt our data encryption key with a key encryption key or a kak? So what's neat about this is in the end we get to store the secret and its encryption key side by side. So you might wonder if your first kind of grokking this, like what do you mean?

A

We store the secret and it's encrypt keye side-by-side. How does that make any sense whatsoever right? You got to remember this encryption key started gray, and now it's encrypted this blue one here right. So the idea is, if you got into at CD and got the secret out and it's encryption key meet you. Your encryption key is encrypted right, so now you're in a place where you can't necessarily decrypt it. So you might wonder: okay! Well, then, how the heck do you actually decrypt it?

A

So when someone needs to come in and access the secret in the future, how do they do it? Well again, that's the beautiful thing about it. This key encryption key this CAC can actually decrypt the data encryption key.

A

So the notion here is: we can delegate that I like to think of them, like master keys in a way because they're, the key encryption keys that are decrypting the data encryption keys again, some people might give me some slack for calling it the master keys but like what will then happen is we can go out when we need stuff and we can decrypt and just send to the kms provider just the encryption key right, because, ideally we don't want to send our secret over the wire.

A

We just want to send the encryption key over the wire. The ke key can then go and at some layer the ke key can then unencrypted the de key, and now we can access our secret. The way that we're expecting- and the beauty of this is now you've got the centralized key management service. Where you can do your rotation.

A

You have all these fancy kind of our back and privilege settings that you can do you've kind of decoupled that full encryption concern from kubernetes and you've got this really neat model where you can just again store your secrets right next to their encryption key, and should you need them the kms server, the key management service provider is just there to decrypt data encryption keys. Should you need them right, so that is a pretty neat flow. Now the warning I'll give you about Dex and are sorry the morning I'll give you about.

A

Kms is I really like it, but it's something that's probably more relevant to you. If you're using a cloud provider, I love the model, I love the storing the data encryption key with the secret, it keeps things fairly simple: it moves the primary keys that you need to really really protect outside of kubernetes, which is a huge pro for me right, I, don't to put too many eggs in that kubernetes basket, and then I can use services that are well-established, like Azure Google's.

A

You know key service, stuff and Amazon's key service stuff, and so on now the thing I'll just warn you about before you get too excited about this because I remember when I first read it I was like oh my gosh. This is so freakin awesome. Let's do it again if, to my knowledge, Google's kms provider is maybe the most mature, which is really cool. So if you're running on gke, that could be neat for you. I believe Azir has one as well I'm, not as sure about Amazon.

A

There was an effort to make a vault one Balts kms provider, who's actually coming out of Oracle, but the project didn't gain wings for reasons that I don't really know, I, don't know if anyone in chat knows, but it seems like the project got archived, so it would be really cool to maybe use this kms model with something like vault, but as we're going to talk about just now, as we transition out of the review, there's a lot of other really cool ways to integrate with external systems as well.

A

So we'll talk a bit about that cool. Let's see what we got going on in chat, cool yeah, it looks like a lot of. You gave a lot of good reasons why that standard encryption at rest methodology probably had some some big risks associated with it right, yeah, cool cool yeah. As Duff said it. You know, the kms is rad, but you're probably only going to use it with a cloud provider unless someone has updated a vault version of it, which I have not seen but I'd love to see that eventually, cool cool.

B

A

Then, at the very end there IKS doesn't implement envelope encryption for Kate's native secrets, interesting they're, using the Flex wall, yeah driver or the CSI driver. So we're actually gonna talk about that. Eventually, for what it's worth, there is a new approach to doing or a newer approach to doing secrets via the container storage interface to container storage interface plugins.

A

So we'll talk a little bit about that as well, so pretty darn, cool stuff, yeah so and I know some of you are asking about the graphics I'll be sure to include some of these graphics as well in the in the in the maybe not the show notes, maybe in the github repo. Are these graphics helpful for you all to kind of understand the flow of all the stuff? That's happening here.

A

My brain like when someone talks to me about envelope encryption I, just can't rock it until I see a picture like this, so I figured I'd uh I'd, bring it up and and talk a little bit about it. Okay, so I think we're done with review now and I know we are. We are already about an hour deep, so we'll probably go for at least another 30 or so minutes. So if you've got time and a drink and want to want to keep hanging out, you know feel free to do that.

A

That would be really cool. So now we're gonna talk a little bit about some of the some of the options for today, some of the things that I. This is a very selfish thing, but I talked to customers about because they usually ask me about them. One quick mention that I want to make is just the notion of an external provider.

A

So here's here's one thing- and this is this- this might come across as an unpopular opinion, I'm not trying to be controversial genuinely, but one of the things that I'm not super crazy about with kubernetes is trying to make kubernetes be a true enterprise-grade secret store. Okay. So what I'm getting at here is like again, I know: I just talked up this envelope. Encryption methodology, it's great I, like it a lot at the same time. There's another part of me: that's just like wow.

A

You know what, when it comes to it at these large enterprises, you know they want, like HSM support behind the scenes for their secret stuff. They want all these things are. Maybe they've already got secret management systems and I really like the notion of aside from the things you have to store in kubernetes secrets, like maybe service account, tokens and stuff I love the notion of taking the secret concern, which is usually a pretty complex thing in a lot of enterprise environments and just keeping it outside of kubernetes altogether.

A

I really really love that model, an example being I have had customers who run vault and they choose to not run in kubernetes because they already had it running in vm's and all this, and that's that's totally fine like there's nothing that says the secrets need to the secret store needs to run in kubernetes. There's nothing that says the secret store needs to be persisted in kubernetes at CD.

A

I really love the model of taking that out so, like maybe you run vault in queue, but it's again not persisting at CD, which could be a really good model, because vault is so feature-rich right or maybe you run vault in a VM or some other secret store and you integrate kubernetes with it. I really love that model where you use an external provider rather than trying to retrofit at CD to use secrets. Okay, now again, not trying to be controversial. I have customers that do encryption at CD all the time.

A

It's okay, it's just about knowing your risks right, so I'd be curious to get some perspectives from what you all think in chat right like what do you think about running an external provider verse versus persisting a net CD I think there's a lot of trade-offs there to really talk about and that very smoothly transitions us into our last set of topics here, which is really like. What are the options you have today? Okay, so there's three that I picked because I talked to customers about them a lot.

A

One is I'm going to talk about vaults injection model, which I think at the time that you recorded Duffy I, know you covered vault, but I think you were using the cube vault project, which is another one that we put a link into. You should check it out. Apps code has done some really cool work to kind of create some neat secret management aspects built around vault to my understanding, but there's been a lot of work since that recording in having some more native integrations with fault.

A

So I just want to talk to you about those today, then we're gonna talk about sealed secrets. I was a little bit unsure of whether to talk about this. One sealed secrets is one that my customers asked me about a lot usually they're a little confused.

A

They think that sealed secrets is trying to solve the same problem that something like vault might solve, which is definitely not true, but sealed secrets is a really cool model and something interesting to think about, and then I want to end today with which someone was talking about in chat the CSI secret driver right. So we'll talk a little bit about where that's headed I am sorry. Spoiler alert really really excited about the CSI secret drivers. I think the model is really awesome.

A

It follows some of the design ideas that I talked about liking like having an external provider having kubernetes integrate with that. So will foreshadow that a bit I don't know if I'll get a provider set up but we'll get into it. So alright, let's start off with vault injection and enough talking, Josh, okay. So let's I'm going to open up another another diagram for you all here and also try to check chat as I go through yeah Duff I miss the crayons I love these diagrams, but I kind of like drawing like I'm in kindergarten.

A

It it uh it's really fun Bogdan, you ask: does the API servers still need credentials for the kms, though yeah, so there there are some details in the provider about how it authenticates with the kms. So this is why it's really applicable to a lot of cloud providers, because you can oftentimes use like some of their existing I am settings so that the so that the kms approach can actually go in and what did I do here?

A

Oh I have a window open, that's why so the the provider can go in and actually talk, maybe through, like I, am privileges that are available to it to the kms provider. So that's a good call at bugged and you would definitely need you would need access somehow to the kms provider in the cloud and then sorry, you totally just answered that.

A

So thanks for thanks for throwing that in chat and I should have looked a little bit lower, oh cool, so Caleb, you've using bonsai clouds, bank vault, mutating webhook, that's awesome, yeah, so mutating web hooks are another really interesting. One right doing some amount of mutation on the front end as secrets go and that's that's really!

A

Cool I got to check out the the bonsai stuff, okay and then the last comment in there I'm gonna call you pianist because you're making really good comments, but I I'm, not gonna, say another crappy pianist because it's too long yeah, that's a great point. You you can't! Oh wait. You can't ever like fully avoid kubernetes secrets, they're so baked in they can be okay. They do a lot of really cool stuff. I'm, not I.

A

By no means don't want to give off the I, don't to give off the impression that I would say: Hey never use kubernetes secrets, but what I do talk to a lot of customers about is like hey, you don't have to make kubernetes. Do everything? Do you have an existing secrets to our provider you're used to you? Are you interested in using one like vault and so on? So there's just a lot of options and I think a lot of trade-offs. Yeah ACP is a good one. Totally agree yeah. So it's it's! It's such a.

A

It's such an interesting, deep conversation right. You really gotta weigh all those different trade-offs. Okay, let's talk about sorry. I love this topic, I feel like I could go on for hours and hours. So, let's, let's pull up the diagram for what I'm gonna get into with vaults, and now some of you might know, vault is a project by Hoshi Corp. It is basically like a key value store that has a lot of really cool feature sets to ensure that you could store sensitive information inside of it.

A

Okay, now I'm, not a vault expert, don't claim to be a vault expert, but I have integrated vaults a couple times with kubernetes. Now, if you have vault and it exists in your environment, that's cool and in the past, there's been a couple ways that you could do integration with vaults. This is a pretty interesting topic actually, so let me just let me just talk about the two ways that you could do it right and when I talk to a lot of my customers, this is exactly the conversation.

A

I have with them right, so you have a secret provider. Okay, and the question is on on cheese, on what this keyboards killing me on. What level do you want to integrate? Now? What does that mean here? So, let's assume you have vault or some other system out there. Where do you want the integration to happen? Now, there's two really common answers here, there's at the platform level right and then there's at the app level. Okay, now I, don't know how familiar you are with projects like spring.

A

Let me just see if I can find the right. The right thing. Okay, so spring is a framework that you use in Java. It does a lot of interesting stuff. One of the super interesting things that spring does is. It has ways to in the kit. In the case of kubernetes, you can actually run things in spring, we'll talk to the kubernetes api server that will bring down things like configuration, maps and secrets right to your application.

A

So you just define it in your code and it integrates now it's interesting here, because you got to kind of weigh this. The one the one trade-off that you've got there is you've, got all right. Well, you're doing this at the app level. Now all the apps need to be aware. Now all the apps need to have the appropriate our back to read their config Maps and secrets right cuz. In theory, you could just mount volumes into the pod right.

A

We've always had volume mounting available to our pods, and then you could go in and just with spring say: hey look for my config in this directory done deal right, which I kind of like that approach a little bit more I. Don't super like the idea of making my app aware of the platform it's running in, but what's the benefit there right? Well, you have a lot of security because, like you can handle this, you can scope the security concerns in the app level.

A

You have a lot of customization, because now you can program your app to talk to the API server or to vault in its own special way. So this can be a viable model if you're willing to really take on that trade-off of making the app, in my opinion, a little too smart and again really binding the app to the system right, because the great thing about volumes is I, hope kubernetes has a really long lifespan, but let's say it goes away in ten years.

A

For some reason again, that's not gonna happen, not gonna happen, but let's assume maybe it did. Your app is looking in two volumes right. It doesn't know about the ecosystem. It's running inside of and I really like that notion when you can achieve it. So what I'm getting at here is you have to really weigh these two options.

A

Do you want to integrate with your secret provider at a platform level or an app level and I saw a Duffy mentioned in chat platform level can be one of the most convenient ways for the developers now you're taking away, in my opinion, a good amount of their flexibility, perhaps which could be a good thing, depending on how you look at this you're, also putting a lot of onus on your shoulders now you're responsible for how the secrets get in their app congratulations platform.

A

People you're responsible for that now, right, maybe not how it propagates into their classes and objects, but getting it into their app is now your responsibility and for a lot of big organizations. They have a delineation between platform and apps teams, or there's kind of this tug-of-war about who should own. What now I'm gonna focus on the platform narrative here, I'm not going to pull up a bunch of Java code with spring. Here is an interesting model that vault offers inside of kubernetes and I happen to happen to like this model. This.

A

This predates in at least to the best of my knowledge, a little bit of the CSI secret driver work. So here's what happens? You create a pod write that pod goes to the API server and you're gonna be able to run this little guy called the vault agent injector. So whoever mentioned that Banzai mutating webhook I, don't know how close this is to that. But this is effectively a mutating web hook. Okay, now what will happen? What will happen? Excuse me is when the pod gets created.

A

This vault injector is going to inject an agent as an init container. So those of you who are familiar with the NIC containers they come first when the pod starts right, the anit container is gonna fire off. It's gonna go over to vault and get your secret for you and store it in the file system. Okay, now there's also some other conventions. You can add too, like you can tell via annotations that you also want to run the sidecar vault agent.

A

Okay and this one is basically I- think it's actually literally the same image as the Anette container, but this one is gonna run over time, so that if there are changes involved or rotations of Secrets and all this kind of stuff, these vault agents sidecar can bring that back into the pod. So that's kind of a key, a key idea here right think about this for a moment we're not pulling our secrets from @cd at least maybe our service account, but at least not like database credentials and stuff we're going strictly to vault.

A

Okay, we're delegating all the all of the pieces to vault altogether and through this model, we're not writing code in our application that integrates directly with vault right. Instead, these little agents are giving your developers for those of you in the chat who are like more on the platform side, your customers, your developers, an experience where all the sudden a volume shows up in their container and it's got their secret inside of it right. That's better I should say really a directory shows up with a secret in it.

A

That's that's kind of a cool experience for them and it's it's again a little bit more complexity for you, because now you've got to manage the injector, maybe you're also responsible for vault, which in and of itself has a bunch of production concerns. You've got to think about and then you're worried about making sure that all these pieces can communicate right. So do you all and try this? What do you say we give it a shot, real, quick and see if we can set it up?

A

Let's do it there's too much delay in chat, so I'm, just gonna, assume you're all excited to try it real quick. So, let's see, if we can do this, um I've done this a Lyle back for a customer, so I think I can pull on some previous knowledge. So, let's hope I don't bring us into a rabbit hole of despair here. Okay, so again, I have cube. I have a two node cluster here, just so that you know we will do a cube.

A

Cuddle get pods, I have no pods running in the default namespace, so we're gonna start off by installing vault. Now, to save you, the time of me searching this I went ahead and grabbed the helm chart off of the Internet, so I have got the vault helm and it's the 0.4 dot Oh, which is for whatever reason what the docs told me to use all right. So here's what we're gonna do we're gonna run a command. Let me actually bring up the docs here too. So I can reference those for you all.

A

Once we get in the show notes, there's a really cool page here. It is yeah, but by the way, just to call this out, hashey Corp is crushing it with they're like learn. Website I, don't know if you all have checked it out, but the tutorials are really awesome. They do a great job here. So whoever works on the learn stuff at hashey corp great job. This is really really cool. Okay, so I think there's a helm, install command. How there's a couple things when you install how and actually let me just.

A

Let me just show you real quick to make sure we're all on the same page. There's a couple things that we care about. One of them, if you try this on your own, if you're watching this video later, maybe in trying it there's a dev flag that we want to set to true by default, vault we'll try to like create a persistent volume claim. I, don't have any of that stuff setup. So I'm gonna do a very small, naive implementation with the with the dev setup.

A

Okay and then the other thing I need to do is make sure that the agent is turned on. That's little. Injector I was telling you about so I need to make sure that this little this little guy right here is enabled so that that mutating webhook is is turned on right. So let's try this out. Okay, uh where is there I know they've got it somewhere in there, Doc's I saw helm command earlier there.

A

It is okay, server, dev enabled is true, and then I also think I don't know if the agents turn on by default. I might want to double-check that real quick. So let's go back to the values and I, don't know if everyone on this call is familiar on. This session is familiar with a helmet. These are just variables that dictate how the thing deploys right. So what does it call I? Think it's just called age. No, it's called injector. Oh there is injector okay, injector, okay. It is enabled by default great.

A

So let's roll with this, so I'm gonna just do their helm, install I'll. Do it to my local directory just to make sure there's no variants there. I guess I didn't really need to bring that locally. Okay, great, then, let's do a cube. Cuddle, get pods, real, quick. All right! I'll make this a bit smaller.

A

All right fingers crossed everyone. Hopefully this starts there. We go Hey, look at that alright killer, so we've got vault zero running and we got the vault agent injector running now again, keep in mind these two pieces are very important. What you are looking at here is vault. I have chosen to run it in the cluster for convenience sake. They even say in the helm. Chart that there's a ton of stuff. You need to think about for production, so don't just helm, deploy and expect it to be. Production, ready and I've also got excuse.

B

A

Need to drink more I've also got the heha the agent injector here. So those two pods are running we're good to go and if it put the injector in it probably put in a oh gosh, what's the resource name, everyone is it mutating webhook configuration? What's that API mutating webhook configuration Hey.

B

A

It is what do you I think someday? We should all have a pub quiz on the kubernetes api and see who can who can remember the most API resource stuff. Okay, anyways I won't get too deep into this, but this webhook configuration is basically saying: hey, API server, look for look for this, this type of operation on this type of resource, okay, and when that comes in that tells the API server go through the injector and put these little extra containers in its place all right all right, yeah.

B

A

So how many I'm gonna drink a beer for that, one for sure, yeah, Duffy you're on you're on my team for the for the API Pub Quiz, so I'm, sorry, but I already I already call that. Ok! So here we go yeah everyone's on Duffy's team, I know Duffy. This might become a one team thing. Maybe we could all go against Duffy. That could be a good idea. Ok, so anyways. What do we got going on here? So we've got the injector.

A

We've got the vault agent now I need to go into vault and I need, to like add a secret and stuff like that. Let me just pull this up in there documentation again. You know before right before I forget. Let me just put the link to this inside of the show notes, so this will be vault, injector how-to, so.

B

A

Case anyone wants to play around with this or, if you're, watching this video check out the show, notes and it'll have the link in there. Ok cool! So, let's, let's, let's instantiate this, so we need to create a vault secret. Ok now what I'm gonna do here is I'm gonna cheat by exacting into the vault pod. Ok and I am gonna. Do all the commands directly inside of the vault pod. So right now, I am in vault. You can see. Vault is the number one process there or actually no it's not.

A

The entry point is the dominant, so it looks like they've got that running is kind of Ian it and then vault is pit 6. Ok, so I am gonna, go ahead and create secret, so I'm, just gonna pull these out from their documentation and explain them again, mostly just concerned about the integration with cube here, not so much about the inner workings with vault. So we go ahead and we make a new path inside a vault kind of like our namespace ish construct in vault.

A

I might not be doing that justice, but this is like a category a folder called internal. Ok, so we've just made that exist now we're gonna roll with database credentials here so vault has another command that we can do called kV put again, you know think of vault as very much like this key-value store. That has all these amazing things to integrate, with underlying, not integrate but like to provide really secure key value store for cents of information, so we're gonna put something in internal and we're gonna call it.

A

Database config and it's gonna have username and password okay, now I'm, not a vault expert, but I. Think if we do vault Cavey did you do vault Cavey, it's either read or get I can never remember what their API is. Let's see if we can, if we can try this real, quick I'll drink. If I get this verb right, hey there, it is okay, so vault inside of cou brownies has a secret. Now right. It has no relation to Etsy D at this current moment great now.

A

What we now need to do is go in and set up vault with kubernetes authentication. Now this is a whole topic in and of itself, so I'm, sorry that I'm not gonna. Do it justice here, okay, oh good job chat! You knew that you knew the verb right away, good work, so here's what we're gonna do we're gonna turn on this thing. Called kubernetes, authentication, I, don't know Duff in your session. If you turned on, if you use cube off at all for that other project that you were using I'm guessing, you probably did.

A

But basically the idea here is we're. Gonna set this up so that vault can use the token reviewer I'm. Sorry. What is it token review request API? What's what's that API name, Duffey, here's, here's your chance! What what is it called again token access review? What is it where it can then go out to the API server validate the the service, account more or less the identity and then based on authorization rules we put in place. We can actually say who has access to what Oh Brad, Beach adavi.

A

Subject: token access review: oh good job, Brad, all right! So, let's, let's go ahead and set this up as well, so I am gonna run another command which basically writes the configuration to vault. This is all in their documentation. All it's basically doing is writing the auth kubernetes config, it's putting in the token reviewer JWT, which is available at this path. This I'm in the pod remember so this this environment variable should resolve this should be the service IP I.

A

Think of my API server, someone correct me if I'm wrong there, and then we've also got the CA certs that we can trust it right. Okay, now, let's go ahead and save run that so now we've got the config in place. Okay and then I'm also going to write policy, and this is basically going to be like the capabilities. So you know again speaking to the complexity. Now we have this another layer since we're running vault of potential policy, where we need to define the capabilities of certain things.

A

So the whole authorization authentication story here is now more complex because we have this other system that were kind of tying into kubernetes. Now I, probably shouldn't mention vault has a bunch of authentication mechanisms. You don't have to use kubernetes per se, but that's just what I'm, using here and I'm, really just making use of that access. Token review right. So let's go ahead and run that all right and then some questions coming in to chat, yep, I.

A

Think while lead you answered that question perfectly Oh Duffy see that's what happens when you go afk, you lose the you, lose the pub quiz so yeah to to your points chums. The the main thing about the operator is a lot more focused on lifecycle management of vault in kubernetes, right, I haven't looked at the operator recently it had some sharp edges back when I worked at core OS, so I don't know. Is anyone running the operator in production? Is that still a pattern that folks use?

A

It could make sense I, just I, never followed along with how production alized the vault operator ended up getting so just to be super clear everyone listening here, I'm, not using the vault operator I'm, just installing vault using helm and I, would theoretically if I were to continue with this, maybe use helm to do the lifecycle, updates and things like that all right anyway. So now, we've got the policy in place, we're all good. Let's go ahead and exit out of the pod.

A

All right, let's see if we can get this thing wired up so again, here's what we're going for we've got the API server. We've got the pods that it creates, and we've got two volt here as well. So my hope is, if I sign a container through the injector, we'll be like oh I care about you container, you want to see that's right. Here's some containers go ahead and talk to vaults.

A

Using your service account token, and then vault will do some magic to figure out whether you should be authorised and all that jazz and vault even gives you like another token, with a TTL associated with it. It's a whole thing, but nonetheless that's kind of the flow the flow we're going with here so inside of their documentation. They have a sample app, which I've just grabbed as well. This sample app is basically going to. Let us mount two secrets: that's what I'm hoping to test with you all that we can mount a secret effectively.

A

So let's do this thing! If we go into you Oh Josh, where did you put that actually I think it's one directory back, vault guides I think vault guides. Oh, they put all of their guides together in one repo. Don't they okay, give me just a second I'm gonna find the right directory. So I don't screw this up. Royally, that's good! That's good! I found it we're good we're good vault agent, sidecar cool, all.

B

A

Again, if you go into the notes, if you try this yourself this, this repo is in there there learn page, so you can check it out. Okay, let's do this thing, so we've got kubernetes. We've got vault setup to integrate with kubernetes for at least just authentication of that user. Let's go ahead and look at the service accounts. So one of the first things we're going to want to do is we're. Gonna want to look into example.

A

What do I have here? A service account internal example. Ok, so first things: first, we're going to create a service account that service account is gonna, be tied to the deployment because remember, vault and the way we've currently got it set up is going to be able to pair service account and the tokens to figure out whether you should have access right to that thing. So let's do this: let's go ahead and apply that file. So we'll do a quick cube. Cuddle apply, alright good deal. We've created the service account.

A

Now, let's go ahead and make an application. Now it looks like in there Docs. Let me bring this up. You're. Actually, sorry I've got it here. So if we do a deployment, one org chart. Okay, here's here's a deployment just so we all know, what's going on right, I have no idea what this deployment does by the way. We're not gonna figure out how to make the UI work and stuff like that. But it is a deployment which the goal here is again. Who cares what the deployment is?

A

We just want to mount the secrets that it needs and should care about. Okay, so the first thing we're gonna do is just deploy it, as is there's no real instructions for vault two to go in and actually to do it or two, it's for the injector more or less to actually do anything with this. So if we do deployment, a1 org chart yeah, Mille, okay, great, let's see if we can get this pod, so cube, cut, I'll, get pod right. All right, great, so org chart is running.

A

Org chart does not have any secrets in it. I'm gonna do a modified version of the instructions to show you all here, if I exact into org chart and then get into their bin SH. Okay, so now I'm in the org chart, pod right I do not have any active secrets. If you will, inside of it, looks like it's. Gonna eventually have us put this in vault slash secret, so this directory probably doesn't exist right now.

A

Can we inject things into here? That's the question, so remember: there's a webhook admission controller. Why didn't? If you look at this, why didn't the vault injector pick up and do something with the secret? Well, it didn't have any instructions on what secret do you want? What settings do you want for like refreshing, the secret? None of that stuff?

A

Was there right, so one of the models that they use and a lot of people who set up their own web hook- controllers oftentimes, evaluate annotations to determine behavior they're gonna have us put annotations into the deployment to effectively make that happen. So if we go into I'll exit out of this pod and we go into deployment 0, oh it's a patch file. hmm I, don't really love patches, we'll try it out anyways. umm So basically, this is going to add three annotations to our existing deployment.

A

First annotation, it's gonna, add agent inject true, so this is gonna, be like hey, go ahead and put the agent in grab the grab the secret and put it in. It's gonna tell us what the role is that it's using again, you might have remembered the role from when we set up vault and then it's going to tell us a effectively what particular config we want to grab right. So with this annotation in place, we should be able to go in and and put this in. So let's do it.

A

So we will, let's see, let's see if they have a patch come and I always screw up patches. They freak me out so I wonder if they have a example patch in here at all. Let me pull over the instructions for you all to see patch patch patch checked, that's good patient! There's a patch command! Now, that's funky! Weird!

A

Do you need a cat like that? If you a patch, is that a thing? Can you not just point to a file I guess: I haven't done a patch in a long time, anyways, okay, so patch I guess this would be patched kind of like yeah, yeah, okay, I see what they're doing sorry. That was a silly comment: okay, go ahead! I think you still could point to a file, though I don't know if the cat things required anyways.

A

So, let's new cube cuddle, get pods all right, we're in knitting again right here now, let's see what we've got going on here. If I do a get pod for this pod Oh wide ho jeez, not a wide butt. Oh yeah, mole! Okay! Let's look at let's look at this pod, real, quick container, not ready, I'm, looking for an an it container here, everyone! Okay! So you can see the volume mounts are here now right, so the intended volume out.

A

Yeah I know I need to embrace customize Steve. It's it's a long journey for me, I'm, getting there um where, where, where I was my container friend, there's my net container? Okay, great, so here's your net container right and it is doing some stuff to make it be able to communicate with volta. Now it's interesting that the admit container, I I remember this. Now that we're looking at it again, um they shipped the whole vault.

A

They use the whole like vault image for the unit container, I guess you'd, it just seems like you, don't need the whole image. I think the vault image is like hundreds of megabytes or something seems like you wouldn't need that just for the client aspect of vault, but maybe that's just a convenience thing anywho the key thing I'm moreso trying to get at here is we all of a sudden have this init container and this an it container was sourced the mutating webhook right. That's the key thing to keep in mind. Okay!

A

Now, please tell me you started no okay, what's wrong with you, so it's still a knitting. Let's see if we can troubleshoot this real, quick, so we'll get pod. Okay, we will paste that in okay, let's get the, what do you think? Should we get the logs I? Guess, let's get the logs for the for the admit container here, vault agent in it, let's see here, okay, see.

A

Interesting error making API requests; okay, invalid.

B

A

Name: internal app.

B

A

Quite interesting, it makes me think that I missed something in fault.

A

Yeah interesting now, when you say I'm, not authenticated Duffey I'm thinking that I'm failing to resolve the role name at the vault level. Are you thinking that I'm failing authentication to the cube API server from vault.

B

A

I not apply the service account. I did.

B

A

Think I, let's see, let's see here, okay, so so quick, quick sanity check, right, cube, cuddle get service account. Okay, we have okay sanity check here. Everyone we've got internal app in place. Right that looks good yeah, I, agree, def, I think it's that vault itself I got it I. Think I must have done something wrong. Configure I bet. We can figure this out. Okay service accounts they're in their deployment. Let's just make sure that there's no chance that their thing is wrong which would suck because I just talked about up. How awesome is is.

B

A

Good, so we got the right service account pairing here it seems like it seems, like I didn't, configure vault right. Let's, let's see if we can figure this out real, quick, so I'll exact back into vault.

A

Okay, all right I'm, back, wait double-check! That am, I involved exact, yeah, okay, I'm back in vault. What do you all think this is so I went through and okay? We know that we know the secret exists involved right because we did a vault Cavey get. Oh you mean the K. The Oh Steve I, see I. Think I'd be surprised if the service account in vault wasn't already set up to work with this, but you might be right.

A

I'll double check that in a second good point, let's go ahead and do a vault Cavey get alright. Okay, let's do a get against internal database. That's where the file was internal database config. Oh, wait check that out, create a path for secret web app config- and this says internal database. Config mmm are my eyes deceiving me here. Does that make sense internal database configure its okay? Let's make sure that one's there first internal database config? Okay secrets: there: okay, we've got kubernetes authentication, enabled I think that's a true a true statement. Right.

A

Okay path is already used at kubernetes, fair enough.

A

We wrote the authentication config stuff, which should be this.

A

Ok, data written I, think I wrote this out and actually, let's let's double check here- everyone cuz I just want to make sure I swear. This should resolve to the service IP right. Okay, that resolves to the service IP.

A

Let's just make sure the tokens there I gotta. Imagine it is token. Okay, I think we're good. Actually, you know what you can also do here: I'm remembering some vault stuff as we go, you can also.

A

What is it called you can? What does that come and I just did there? It is. You can also read this I'm, pretty sure: okay, good issuer, an a kubernetes CA cert.

A

That is good. That is good. Okay, so we've got that. What else do we got going on so I? Think I? Did this I probably should double check right.

B

A

This is the as I understand it. This is the mapping to give me the capability right, so this policy right for the path, internal data database, config.

A

Internal data database config. Let's make sure this is right- internal database config, huh if anyone's really good with vault, does it make sense that data is here. Is that just a standard convention that goes inside of this internal piece?

A

Because that does look different? Doesn't it hmm I wonder that looks very strange.

A

Let's I guess you can't hurt to open it up even more right. So let's just do that real, quick, the next step. I didn't do what next step that I'd not do here cube.

B

A

Where we got this, so we've got that okay got the secret. Sorry I think I scrolled up too much, so this oh did I not do this. It's funny how easy it is to skip a step. I'm, pretty sure I didn't do that right. Is that what you all are talking about a chat right now, I, don't remember doing that!

A

Is that what y'all are thinking in chat, this guy I, think that might be it I am. It is super interesting that that path was slightly different, though isn't it? Maybe data is just like a standard convention. I, don't know, okay, let's try it so exit out.

A

A

Boy, look: what we got that looks pretty promising. Doesn't it whoo good job? Everyone thank you chat for actually following the instructions, unlike somebody so we'll go inside of this one which I think is slightly different, and if we have a secret I, think I think we did the thing okay. So what was it again? Everyone? It was secret.

B

A

Boy, Noah's, vault, right, vault secrets, hey check it out, awesome, very cool all right, so we have the injection working, which is which is really neat good job, everyone that is worth cheering all.

B

A

There's a bunch of other stuff with vault that I won't get into, but I'll just high level. It real quick because I want to make sure I can do a little bit of justice, sealed secrets and a little bit of justice to CSI drivers before I say goodbye to you all. So that's really exciting I'm glad we got that working okay, now, there's a couple things that are probably key to mention. Let me start with my diagram. Actually I had this conversation with a customer recently. It's only the reason why it's so fresh.

A

There is another annotation that you can put in place. If someone in chat wants to paste it. If you know it, it's something like secret, update, true or enable or something, but basically the idea is I deployed today with the anit vault agent in my app container. If I add that extra annotation in it will also include a sidecar container, which is another vault agent, so the idea is the anit container happens to make sure the secrets there before the app starts.

A

But then, if you put that other annotation in place, it will actually sit there and it will kind of go and talk to vault and over time. If you change a secret or something it will actually go in and change that secret. So look for that annotation! It's it's pretty cool! Another thing! I just want to give a shout-out to vault. I! Won't!

A

Do it right now, but, as you probably noticed it kind of outputs vault data in like it almost looks like it's like pumped print effing map in going right for those of a yoo-hoo right, go legs, probably pretty familiar, so there's a whole templating system that you can use in here. A lot of these. These secret approaches support this. But just look for. Let me see if I can find a quick snippet template template templates there. It is so you can do stuff like this.

A

Where effectively, you can put in templates that kind of look like if any of you have done, go templating before kind of look like go templating so that when the app gets its secret, it's in a format that it's comfortable consuming or knows how to consume so again, like kind of like we were talking about zookeeper earlier right, like the idea would be.

A

Maybe you could move apps to kubernetes and like everything about how they consume secrets would not have to change at all, because you can do these really cool, clever things to make it work right.

A

So there's again an annotation somewhere somewhere for that sweet Oh data is a standard for v2 stores, okay good to know Caleb. Thank you, cool, okay, so um I know, we've used up a lot of time, but if you've got about five more minutes, let's just quickly talk about two more topics: real quick that I think are really interesting.

A

I won't implement these ones because we've been together for a while now I want to give you all your weekend, but I'm not that you're forced to stay here, of course, but I want to end this on a bit of a good notes. Let's just talk about sealed secrets, real quick and then let's talk about the CSI secret driver, and this will be a really good kind of like cliffhanger for those of you who are interested in it and if you won't ever want us to double click on it.

A

Maybe we could do a TGI Kay on one of these as well, so vault, sealed secrets, quick and dirty with second on vault, steal secrets, sealed secrets which comes from a company called bitNami formally now vmware, so actually part of our family I promise. This is not a biased look at all at at secret things, but I get asked about sealed secrets. All the time people say: hey do I need to use. Vault could I just use like sealed secrets, or something like that. So sealed secrets is a really cool model and I hope I.

A

Do it justice, because I haven't necessarily used it in production yet, but I have used it in development environments with my customers. The way that I think about sealed secrets is there's not it's not solving the same problem right the ideal. The idea of sealing a secret is that you could even store the secret in something like your git repository fully encrypted and while people all over the world can see the secret it's in an encrypted format in the notion, with sealed secrets. Is that something runs in the cluster that can decrypt the secrets.

A

Should they need to be decrypted right, but it isn't a replacement for actual encryption in @cd, and here is why here's here's a quick flow to kind of show you a little bit of what I'm what I'm getting at here? Okay. So, let's look at sealed secret diagram, so file open, I'll, move this on over to diagrams, let's see where's my sealed secret friends, a secret sealed secret.

A

Okay do I want to show you all the behavior one. Let me just check that real quick, yeah I think hold on I'm, just I'm, debating between two different images. I think I, like this one, better yeah, okay, I'll just pick one that I can always show the other one too. I have I, have two graphics for this one, all right. So here's the deal with sealed secrets.

A

When you deploy sealed secrets, it basically runs a controller inside of the cluster. Now, what's cool about this controller, is it has the private key that it can either generate or I think you can maybe provide it? A private key I'm, not sure, but it can minimum generate a private key and it can rotate that key over time.

A

What happens? Is your users use this little utility called cube seal fetch cert, and it actually goes out to the sealed secret controller and gives you a public key back: okay, not the private key, but the public key. So what is public and private keys and asymmetric encryption right? The notion most of the time sorta is your public key? Can encrypt your private key? Can decrypt right. So the idea here is that, with the public key users can encrypt all over the place. They can encrypt all their secrets up.

A

Make them these really nice encrypted secrets right and then you just can. Basically you know I'm not that I'm saying you have to do this or should do this, but you can just store literal secrets in public facing repos, because unless that private key gets compromised, it is encrypted and unusable or seeable right. It looks like Steve confirmed for us, so you can use a private key if you want to provide your own. So so then what ends up happening? Is you end up submitting this CRD, which is kind of like a wrapper?

A

If you will actually I guess I'll get to that in a second? So, what's basically happening with the sealed secret controller, is there's a secret key and that lives in at CD? Ok, so here's why this thing is not necessarily a replacement for encrypting and it's solving a completely different concern right. The Etsy cluster is storing. It holds in plain text format. If you don't encrypt your at CD right, it is storing the private key there.

A

So in theory, if Etsy GE gets compromised, the private key can get compromised, and thus you can now decrypt the secrets. Another key thing to keep in mind, and this is what I'm going to switch into the other diagram, just to kind of show you what this flow sort of looks like.

A

Where do I have this one sealed secret, sealed secret secret, sealed flow I think this? Is it yep.

B

A

Great so what's happening here is users are creating this sealed secret in green now sealed secret fully encrypted. If I got my hands on that sealed secret I would not be able to figure out what the secret contains without actually getting my hands on the private key and decrypting it. But the sealed secret controller is doing a bunch of magic. For example, it's taking the sealed secret and it's actually storing a version of it if you will as an unencrypted secret inside of that CD. Ok at least last time, I checked.

A

This was kind of the general flow. So again, the sealed secret is not inherently safe in the kubernetes cluster right, but the Seavers isn't necessarily trying to solve that problem. I would argue, and hopefully again I'm doing it. Justice feel free to correct me. Anyone in chat, if you have different perspectives here, but I would argue that sealed secrets is doing a much better job of keeping secrets safe outside of kubernetes right, because the general idea here is like especially those of you doing a git ops flow.

A

Maybe you want to declaratively store your secret in a manifest somewhere right. You want to commit it into get. Maybe it's not a public get repo but, like you just want to commit it there, and if someone sees that secret, you don't wanna have to worry about it, meaning anything because it's fully encrypted right, so you're, basically running all of the decryption services in side of your kubernetes cluster, which, depending on what problem you're trying to solve that could make total sense.

A

But the key thing is again: you're encrypting yourself at this level right or whatever store, really that you eventually store the sealed secret in, but inside of this level, for those of you who are platform operators again, the last I'll repeat it one more time, just keep in mind that you still need to solve for s CD encryption, because at the end of the day, not only is the sealed secret controllers private key stored in s EE, but there are also copies of the the secret itself right so again, hopefully, when I'm well, it's clear what I'm saying here, really cool project really different concern.

A

In my mind. Okay, so that's a little bit of a summary on sealed secrets: if you get a chance and want to try it out, it's a pretty. What's really great about sealed secrets is they're pretty darn lightweight, so you can run this controller in your cluster. You can make they're sealed secret CR DS, just like vault. They have like a templating thing. You can use too so feel free to check this project out by bit. Nami it's it's a pretty cool one, all right!

A

Last but not least, and then I'll give you all a break. So you don't have to continue to hear me blabber about all this stuff. Okay is CSI secret driver, okay, now again, actually I think I. Think a lot of the CSI drivers might still be a little bit like experimental, we'll check in a second. So it's probably good that we're not testing it just yet, but now that you have a primer on how that vault injection worked, I think you'll find this really really interesting.

A

Let's talk about this and then wrap up for today, so the secret store CS driver I, really like this idea. I think I, like this idea, I haven't personally gotten to use it just yet. The general notion is to mount secrets, keys, certs and a pod using a CSI volume. Now maybe it may be the backup a little bit I, don't know how familiar are with si. Si si si is container storage interface kind of like how we have CNI as kubernetes progressed. All these people started building storage solutions to integrate with pods into kubernetes.

A

Some people were putting them in their cloud providers. Some people were running them as processes or containers outside of the cube API server controller. There is a ton of variants right so like like we should we eventually standardize on this notion of CSI, which is an interface that defines things you should be able to do, and that interface can then be implemented by providers. So like taking secrets out of the equation, there's CSI providers for AWS EBS. There is CSI providers for VMware's V San right.

A

It just depends on what you want to plug in so taking that CSI driver model, a step forward. What's pretty darn neat here is taking that into the realm of Secrets.

A

So, as you need secrets, this underlying provider can go out to a secret store right and actually grab that secret and make it available in the persistent volume itself. So the cool idea here is more so that you don't have at CD in the data path of your secret per se. You're also loading it into the containers file system may be on, like Ram disk or something right and in my mind what I really like about this idea.

A

I, guess really the plugins and how hard into the plugins get is really going to determine the success of it. What's great about it is I love the idea of just throwing a driver in and not worrying about other complexities in kubernetes, like maybe I use.

A

Excuse me, vault and I just want to put the CSI driver in and then boom it just works. There's no complexity per se with like the mutating web, hooks not to say mutating web hooks are a bad design. They're great! It's just like it's a little bit simpler because there's this underlying driver. Looking for when it needs to satisfy mounting these secrets and it can communicate with the provider as such, so I think this high-level one is just kind of the design, I'm pretty sure, because I think there's an azure and a hashey Corp implementation.

A

Let's see here, implementations, implementations or providers, I should say that's, probably a better way to put it. Where are the the known providers? Let's see if I can find vault real quick here we go okay provider, so there's a zero key vault provider, hashey court provider as well right so- and these are the only two I'm aware of so. If anyone in chat knows of some of the other CSI secret drivers feel free to put those in, we should probably I'm guessing.

A

Actually, let's check out vault I think it it's still like an alpha or something yeah. Okay. This is an experimental project. So, if you all are open to it, I think a whole session. I, don't know how mature again the plugin is I, don't want to put the provider developers. You know in the fire here if they're still kind of working on it, but this could make a really cool, T, gik I think so the idea again is with that CSI secret driver setup.

A

You can now add the vault provider and then not too dissimilar to what we did with the knit containers right. We now have this driver underlying that could actually go in and instantiate the secrets inside of the pod, which is pretty freakin cool. The biggest difference here again is that whole front end flow, no mutating webhook. No, you know injecting containers in the pod.

A

This feels a lot more like a a true platform service right, because you have these drivers running there, they're just waiting for what, when they need to do work, and they just make it happen, just like the storage providers do as well so I hope this has piqued your interest a little bit again just take what we did with vault and imagine moving the concern of getting the volume or sorry getting the secret, storing the secret all that stuff into this little thing called a CSI driver which is again I.

A

For me really really exciting, so, let's see how all this goes over time. I'll see, Thea zero ones, more mature, I, haven't looked I, know, I. Think someone in chat mentioned the.

A

Missouri KS was maybe using this I, don't know. Maybe you can correct me if I'm wrong, but yeah. They have a secret store here as well print provider, so all right, pretty freakin cool right. So um all in all, we covered a ton of stuff. Today, everyone, this was really awesome. um We went through kind of end to end with diagrams and out again I'll put some of those diagrams in the repo.

A

If you want to look at them again, but we had, we went through kind of a visual of every kind of default option and then kind of looked at the ways you can go even further right with vault injection. We talked about sealed secrets: the CSI driver. All of this stuff is really really cool and again coming away from all this. The thing I want to encourage those listening is really that, like kubernetes, is this really amazing, extensible pluggable system and the benefit in downside to the Hat?

A

Is that I can't just come on here and say: here's how you do secrets right. This isn't some turnkey platform where, like they're just there and there's one way to do them and that's it, which is awesome because, all of a sudden, when you start working in a financial services contacts, the secret requirements are different than when you go work at your startup, where you're trying to just pull some database credentials in for your Postgres system.

A

Right like it, looks different in different places, and we have all those extension points in kubernetes, which is so cool and I love how the ecosystem around Secrets is finally starting to really harden and come to life right so again, I hope that kind of excites you and know that this is really a journey. It's all about just like how can I make this a little more secure. You know what problems should I solve for, try not to boil the ocean right.

A

Maybe just encryption at rest is fine or maybe you're at a point where, like you, want to go, implement your own kms plugin right who knows, but you just have to kind of gauge that and know that you have a lot of options and how your workflow is and how you want to provide secrets at a platform level or an app level is really gonna dictate which of these options you go with and how you implement it and that's about it for today. So I hope you all really had a great time on this session.

A

I had a blast chat. You were super interactive, so thank you so much for helping me fix things, and you know hopping in and doing a great job, giving some extra contacts. This was a really really awesome session, we'll think more about a future CSI secret driver session. I saw a couple comments in there as well. So as always, everyone super super awesome getting to do this with you all we'll catch you next week, Joe or Duff will probably be back online. We'll see you then okay cheers everyone. Take it easy.

A