Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hacking Helm - Paul Czarkowski, Pivotal & Scott Rigby, Codeacademy

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Hacking Helm - Paul Czarkowski, Pivotal & Scott Rigby, Codeacademy

Helm is the best way to build, package, and run Kubernetes manifests. However it has been considered by some as a fairly insecure way to deploy software, mostly due to its server component Tiller. Let’s put that to the test and hack (and then protect from those hacks) Helm. Paul will talk about the architecture behind Helm (v2) and how it is seen as particularly vulnerable to hacks designed to either gather information about deployed applications and even access the kubernetes cluster itself. Paul will then demonstrate several hacks in growing sophistication that do exactly that and then show how to protect yourself from those attacks. Paul will finish with a summary of how you can use Helm as securely as possible.

https://sched.co/MPXN

A

Hello: everyone welcome to hacking, helm, I'm very happy to welcome Paul Tchaikovsky from pivotal and Scott Rigby from code Academy to the stage they will give you a presentation about hacking, helm so enjoy, and thank you.

B

Hi everyone I, so will kick it right off, so we are indeed Paul and Scott I work, a pivotal where I'm a developer, advocate and Scott is a DevOps engineer, yep cool! That's me! So we're gonna talk about helm, obviously, and really talking about the security of home, so kind of a rough idea of what we're gonna go through up on the board there. So we're gonna kind of start off and Scott. Why don't you start off by just giving this a brief like intro to exactly what home is really briefly sure.

C

Yeah I mean helm is a it's described as the package manager for kubernetes. It is a package manager for kubernetes. It is useful because it helps with bunching up your configurations. You say a bunch of UML configurations for your cube, primitives, it's templated, so you don't have to repeat keys all over the place. There's you.

A

C

Templating, so there's a logic in case you set this value, you can deploy these kinds of workloads etc, and there are a lot of other benefits, including versioning roll backs, so it just really helps with deploying the other. Really nice thing is helm, charts, I realize this is not all about helmet, just a brief intro.

C

They are meant to be reusable generally, so you really can write them any way you like, if you want, but there are also a gazillion or actually more like 600, something shareable Reap multi-purpose and reusable charts that are on the chart hub, which will link to later so yeah. That's just a brief intro. Many of you probably know in fact, just out of curiosity how many people here use helm now. Ok, so I really didn't need to say that there might have been two people in the audience that don't.

B

All right cool, so there are where's my clicker there we go so there are four main components of home and we thought we'd quickly go through them, as that will help set the stage for like the security concerns about each one as we go through. So obviously we have the home client all.

C

B

And it's basically just how you interact with home right, so you have the home client installed, Inlet on your laptop and like you're, doing a home, create to create a new chart, you're installing charts up into your kubernetes cluster you're, interacting with a repository, etc. Right, so that's pretty straightforward and really there's not too many like security concerns. We have it kind of the home client side of things. So then we have helm charts so maybe Scott you wanna talk a little.

C

And so of those configurations, I was talking about our just. You know package up into tarball and ultimately those I mean I think that's pretty kind.

B

Of I kind of covered it earlier when.

C

I catch up, there is the chart yeah.

B

So everyone here has used home, so you know what a home shot: yeah, a bunch of templates, shot, yeah Mel files are metadata at a chart and then a set of default values that will get rendered in if you don't override them. But.

C

Potentially your dependent charts, etc. That's kind of what we're covering here and.

B

Then you have your helm repository, which is like a server up on the internet somewhere that is hosting those helm packages, so those tarballs as they host those tarballs plus some sort of like index file to say what is available on that server and you can host a helm, repo and anything cuz. It's just static files, so like up on s3 or on github pages, the home community actually has a tool called chart, releaser, which is designed to host those chart, a chart repo on github using github releases and github pages.

B

So that's quite useful and of course, if you're using that you sort of offsetting a lot of your security stuff to like github or to s3 or whatever just.

C

Out of curiosity, how I am very curious now for soon you're all here and you a lot of most of you know about hell, how many people are hosting their own helm, repos right now: okay, wow! That's that's surprisingly great and we'll we'll link to the hub. If you don't already have your repos linked to the helm hub now, that'll be yeah that'll be up here. So.

B

We also have the tiller server right and it runs inside your kubernetes cluster as you're all aware- and you know it's probably one of the most controversial parts of helm and usually when people are talking about helm security, they're talking about tiller and so we'll talk about a few things around tiller and securing it and also how to do naughty things. If it's not secure, but really it's doing like the rendering the templates, deploying resources, etc. You install it by using the home client, you do helm in it.

B

Hopefully you're actually doing a little bit more than just helm in it, but that's kind of the basic simple command to install it and it creates a deployment in the coop system namespace and it creates a service of type cluster IP right. So that's just kind of a command line, sort of description. What I just talked through I'm Scott. You want to quickly go through the like helm, chart workflow what exactly happens as you're, doing like a home install and then we'll start to move on to the security bit sure yeah.

C

Okay, right there we go yeah so right now we're gonna we're gonna be covering the general use case. We realized there are a number, a number of different workarounds and ways to handle this and we'll cover some of those briefly here as we're covering some of the hacks, but really we're just talking about a helm, install where are you or installing your chart, either from either from in this case, from aretha right. So the helm client makes a call to your helm, repo it.

C

It goes ahead and then downloads that chart package essentially like just you know, grabs that grabs that tar ball, that you find that if I'm some of the metadata the packages, it's unpacked and its default values or are going to be used, it rendered it goes ahead and renders out the templates and it passes that information to the cube API like install or upgrade the application, including maybe we already cover yeah, including any values that you set any custom value files or any values, except the command line, the right, those that that information is all passed, the tiller server.

C

So all that so far has happened locally right. All that information is all passed. The tiller server which goes ahead and renders those templates and works with a cube API to deploy yeah.

B

Right so that brings up like a number of security concerns, so if we go move on to the next rabbit I'm, so like it's the helm, repo secure, is it using SSL a chart signed? Where are you storing that values file? Sorry.

C

And if your charts are signed, are you? Are you validating? That's signature, yeah.

B

Like where are we storing the values like, especially for like the values that they like your passwords and secrets and keys and stuff, and is that secure? Are they being stored in plain text and get? And then, of course, it still a secure who's able to access it and are my workloads protected from somebody else's workloads is what I've installed through Atilla accessible by someone else. Who's also got helm and that same cluster.

B

So when we talk about securing tiller and the other components, first I think we really need to talk about like the tenancy model that you've chosen for your kubernetes cluster. Before you decide how exactly you want to secure helm, because your home tenancy is kind of roughly gonna reflect your kubernetes tenancy for sure.

C

Or share that list all.

B

Right so we kind of we came up with kind of these four models, we're like either.

B

Everyone is like a sysadmin and everyone is the cluster admin right, in which case like you effectively, don't really have a tendancy modal like everyone is good and then you have like every user has their own namespace, so you started using using our back and your locking people down to just a particular namespace, or maybe it's like each team or project has a namespace right or then there's also that, like, if you're doing multi class to you, but every team has their own class and that often a lot of folks go back to that, like everyone that has access to that cluster is is kind of the god of that cluster and you just email around that x.509 certificates and like nobody, cares right.

B

Well, hopefully you don't try to do that. So, like yeah, that's it like everyone's cluster admin. You don't really care in that case, like you, don't really just worry too much about securing helm from each other.

B

If you're doing the namespace model, if using namespaces as a tenancy model, then you do need to start thinking about how am I going to protect each namespace from not just the users but from tiller and from how helm works. And so that's when we're like. Okay, do we need to install helm Atilla into like its own namespace or something like that and then obviously again, if each team has their own cluster and everyone's kind of a super user in that cluster, then you kind of well.

B

We can have one big tiller, we're not really trying to protect things from each other, because everyone has already got on kubernetes, so it doesn't really matter if helm or also has some elevated permissions across the clusters, and then we have to think about like what like. Are we doing our back? Are we doing a back like what's going on? Well, first of all, you're not doing a map, okay, back, like that's kind of a relic from a long time ago.

B

You shouldn't be doing it, which means you're not just doing that helm in it you're actually doing a like, creating a role and doing some other stuff. Alright, so you gonna do our back. Let's go.

C

C

Which side are we actually okay, yeah, so basically yeah we're? What we're talking about now is just making sure that we are. We have art we have tiller set up for whatever model we've chosen right, the there to you know to real types of users. We're talking about one is you know the the normal service account user, the others actual users humans, but first of all, we'll talk about you know tiller is, in this case a rope pretty much a robot right for us. It's a service account so generally.

C

You're going actually, why don't you.

B

You want to lock down the resources to the bare minimum required for, like that robot user right, so Atilla itself gets a service account and then you'll do role, bindings and plaster role bindings to specify is it accessing the whole cluster or is it accessing specific namespaces exactly.

C

To match the tenancy model that we mentioned earlier, yeah yeah.

B

And again, you're not using a back because it's not three years ago, a user accounts. So this is your your users, so the people that are going to be running home, so the home, client and it'll also be doing like coop cuddled get nodes and Kuk I'll get pods and stuff like that, and so, usually you think of kind of having your Super User, which is kind of up cluster admin rights and then a regular user. That probably only has access to a set number of namespaces and maybe also has a reduced amount of resources.

B

They have access to yeah. And so, when you're looking down Atilla, you need to be thinking about that tenancy model and who currently has access to like cluster admin. Who has access to specific namespaces.

C

Yeah I'm not sure if we really like noted this yet, but the the point is that if, if tiller is installed cluster wide in the in the tenancy model, where you, where you you actually do have user access to specific namespaces, there's really the big problem right, because tiller has cluster admin access now and you can access tiller using ultimately coop proxy from your client.

C

So now you are somehow granted that cluster admin access and maybe the person that's setting up your cluster- has no clue or maybe that person is you right and are not thinking about the fact that this privilege is given to the users yeah, so we're talking about privilege, escalation right. Let's.

B

Have a look at what that looks like, so you can see, first of all, we're installing tiller and a somewhat better way now we're creating a cluster role, binding we're giving it a service account, etc. But this is giving them a cluster admin rights right. So a person who is not cluster admin but has access to helm is effectively inheriting Helms cluster admin rights, and so this this up here is, if you're doing.

B

If everyone is cluster admin, it's kind of okay to install helm this way, depending on how how secure you are, but if you're actually restricting people to namespaces.

B

You probably want to do something like this, where you're creating a service account in a specific namespace I'm, you see I'm doing the dash n to denote a namespace I'm, creating a specific role in that namespace and I'm, assigning that role via roll binding to Atilla in that namespace, and then, when I do the helm in it, I specify which namespace to use and then, when I run helm, install I also specify that and of question to environment via environment variables.

B

So you don't actually type in the dash dash till a namespace every time and so doing this you would install a tiller for kind of every user or every namespace, depending on exactly how you're doing it, and then that that helm, that tiller user will be restricted in a similar way to the what your kubernetes users are restricted to.

B

So you end up with a slightly more secure model, but you do have a you do end up with a bunch of tillers lying around I do a workshop, where I kind of do that model and I. Think I install like tiller like a hundred times in the cluster right which isn't too bad because it doesn't take up a lot of resources, but it is. It is something to think about so once you've got it installed.

B

There's some attack vectors of like running things from actually inside the cluster and being up to able to access tiller from inside the cluster. So it's got! Why don't you? Why? Don't you go through that? Okay,.

C

Yeah, it's a skip ahead, accidentally yeah right! So did we list out these attack vectors, let's see in any case yeah? So basically, the three are some similar to what we had just mentioned a second ago that a lower privilege user trying to gain higher access. The second is some bad actor in a pod trying to access tiller from inside your cluster. So like you're, you know you've installed something and you're not really sure where it came from right or it slipped through. They.

C

Someone in that pod can go ahead and you know try to contact tiller, we'll talk about in a second and you know get that get that cluster admin access we just mentioned and the other is to launch extra pop sorry launch extra, pods and, and then the last is is when you just have you know a really like creepy chart maintainer right, an already a naughty chart: author who's. You know you do helm, install some chart and what's in it, what's it doing?

C

If you didn't inspect it carefully, it could be doing some interesting things so yeah. So the first one we can talk about the sort of the service right. Yes,.

B

We do a tiller in it right it installs a deployment facilitate also deploys a service Pattillo and if you have- and it has a cluster IP in a cluster port right. So if you have access to that service on which, like pretty much anything on the cluster, would by default, you can start doing things. So you can see from this example. I'm just running a random pod that has helm in it and I'm starting to run home commands, and you can see.

B

I can look to a helm version and see what versions are running, etc right, so without a lot, without really having any real without any privileges, apart from exec into a one pod somewhere, whether that's because I'm running coop, cuttle, exec or because I've like hacked your your WordPress server or whatever I now have access to tiller on your cluster and then I'm, inheriting, whatever access tiller has in your cluster. So suddenly you know from being someone that should be unprivileged.

B

I now have like full cluster admin rights in your cluster, so I can do bad things. So obviously, one of the first things you can do is just go ahead and delete that service right. It is not really needed for humans to deal with tiller, so you just do, could cuddle delete and get rid of it, but a patient user would still be able to access if they can figure out the IP address of that pod.

B

That's kind of what I'm showing here right, the first attempt it's using what would have been the service, its failing. The second attempt, I kind of figured out the IP address of the tiller pod and then I can still actually run, commands and undo stuff right.

C

So it's still good to do it's just one one extra layer of security so.

B

How do we, how do we solve that.

C

Yeah, well generally, you want to you and note that this still doesn't solve it all for you, but an additional layer is yeah. You kill you kill that service. Then you make sure that tiller is only listening on localhost. That's that's super important and there's you know a really there's a really great there's. There are some links back here. There's a really great bit, Namie article that lee goes through these this little section of the talk and gives fantastic examples of how how to do that.

C

So you don't have to kind of like quickly sketch this out yeah.

B

You can see here we're actually just patching the existing deployment and just changing the the like the container command row to an args to just be listen to localhost. So now it's just listening to localhost, so anything outside of the pod can no longer access it right. So it doesn't really matter where you are in the cluster. um You can't get to localhost in that pod unless you're. Actually a member of that pod, which you know you're not right,.

C

And by the way, the reason that you want to restrict a localhost in case it's not clear- is that Windhelm client, your binary from your workstation remember, is talking to tiller it's using coop proxy and that's essentially talking over localhost yeah.

B

So when you do coop kernel proxy, which is what helm client does it gets, it basically proxies you through to localhost on that on on that pod, which means running, helm, install, etc, will still work from a human perspective.

B

And helm proxy that sorry, coupe caudal proxy is controlled by our backs. So you know, as long as you're got the right are back settings. Users should only be able to access the the tillers that they have access to by the kubernetes are back. If you need to install if you need to access home from inside of your cluster, which there are some legitimate reasons, if you're running CI inside of the cluster, you may need to access home to do helm, install or helm upgrade or whatever.

B

If you have the helm install installed, and you have the right coupe config file, it should continue to work as you're expecting as you expect. But sometimes you want to actually talk to the the tiller API SAR. And if you want to talk to the tiller API you, then you actually need that service back.

B

So you need to actually secure that a tiller service some other way, and you do that by using TLS verification so effectively you're creating a like a CA and a bunch of certain keys, and you tell tiller that it should trust anything. That's been signed with that same CA and then, when you're running helm, client commands you, you provide those certificates and etc, and as long as those certs match and assigned by the right CA, you can now run helm commands effectively.

B

Now it is really painful to set up so most people don't do it, which so, usually you just don't do like there's, not a lot of reasons to access the helm API from inside the cluster. So it's not something you would really need to do. I have seen folks actually force everyone to actually do this for users to access it, but I feel that is usually a little bit overkill like you've already got your coop cuddle. Yes, sorry, your kubernetes are back sort of giving you some protections anyway.

B

So I feel like this is a little bit, maybe over the top, but everyone has different ideas of what what is acceptable risk when.

C

B

To security and.

C

It's a tool, that's meant to be flexible, so if you want to run your CI in cluster, you know go for it. You just need to make sure I mean the goal of that is. If it's not clear, it probably is to most of you, but you just need to make sure that the actor calling those commands really is Hillary needs to know they are who they say. They are yeah.

B

And then I guess the other thing you could do is you can use network security policies? We're actually met that a little bit earlier to restrict who can access the pod, and you could even maybe do something with sidecars have like a proxy in a sidecar and have that proxy have like different off mechanisms to access it. So there's a few a few ways you can work around it. So why don't we move on and talk about naughty chart authors, our favorite right, so maybe Scott. You want to talk a little bit. This yeah.

C

Right so I kind of indicated this earlier that that, uh if you, let's even say you're installing from the helm like community charts, mono repo right now and you do a helm in stall, stable, whatever right, stable, WordPress or something like this. That.

B

C

Yeah yeah, so apps can definitely limit you a bit but uh but let's say someone for some weird reason: snuck in a PR that no one noticed and it's got you know some. You know well, you know whether they're Bitcoin miners or you know something more, even more malicious. That might you.

A

C

Not just sort of like eat up your resources, but actually mess up your resources that could be in there. So.

B

How do we protect against that? You.

C

Well, you've, really can't you can't entirely. Of course you can't entirely you you've have to inspect what you're installing, but there are a couple of different tools to help. You know limit that attack surface right.

B

So, first of all, you have plug security policies, but they're really hides and we're really lazy. So no one uses them. Sorry, you have admission control of web hooks. Oh wait! A sec wait a second just because.

C

I did this twice before it. You don't have to be honest, of course, but who is running pod security policies on every all of your workloads? That's pretty close to what I thought. Yeah I mean I I've people know I'm with you yeah I'm. Definitely with you Craig. Definitely.

B

Is he's good work, Craig and in fact, I think Craig will tell you how to do that in his book managing kubernetes clusters, which he'll be signing at 3:30 there you go, there's a there's, a plug for you, so pod security policies, admission controller web hooks are great. You tie them into a tool like open policy agent and then you can use this I think it's a DSL for that is Rieger and you can do very interesting policies around what is and isn't allowed to be done so like.

B

If it's a deployment, the image must come from my registry. Maybe a particular type of user can't create a service of type load balancer. Maybe you can't launch an image within a Bitcoin in the name. You know a lot of stuff like that, so it's really useful. It.

C

Looks great yeah, yeah and.

B

Then Gauss rush Grove actually gave a talk yesterday on actually shifting the open policy agent test. All the way left to doing it with unit tests as we as this tool called contest that will actually just basically look through your Kubb manifests before they're actually deployed and will run those open policy agents tests against it, and so you can put that in your CI system and get really fast feedback on whether or not you're doing something that is gonna violate the rules that you're putting in place inside of your clusters.

B

So if you're not doing admission controller webhooks, if you're not using opa or tool like OPA, I, highly recommend it, and then you also have the mutating controller hook. So you can actually like intercept and change things that the naughty chart authors are doing say you want them to always make sure they have certain annotations or stuff. You'd actually modify their deployments to have the right annotations, which can be super useful as well yeah. So that's kind of all you can really do for naughty-naughty chart.

B

Authors is have policies in place and enforce those policies around really obvious things, and you can also enforce best practices as well, so really really useful.

C

Yeah right, so the other thing is right, like you can't quit, isn't there so you could just not use tiller at all. You know that is. That is one way to and again we're talking about how the current helm version helm to here now. There's one way to do it, you can, you know you can use, you can use chart templates to go ahead and render help you can still use charts. You could, you know, render those templates I think that's what this command yeah.

C

You can render those templates using that that are that, come out of that chart and then just pass those directly to keep control. That is one. That's certainly one way what you lose from that is, you know you lose Helms versioning and rollback capabilities, basically you're using losing the the deployment niceties that and what some people consider. You know like requirements for them that that tiller gives you yeah.

B

And this is a pretty common pattern that I see and you even see other other open-source tools using it like sto, does all of their manifests in helm, but they run helm, template and share the actual rendered out results. They don't actually distribute it with home. So it's definitely even if you're not distributing with home using home to do. The templating is still super useful, and if you do this, you don't need tiller.

B

You'll also have the like the tiller list, helm plug-in, but instead of running tiller up in your kubernetes cluster, it actually runs it on your local system, because really till is just working with the criminate kubernetes api. So there's no reason it has to run inside the cluster, and so when you use the helm till a plugin, you tell it to start. You run your home commands and then you tell it to stop right.

B

So tiller is only running for the length of time that you are doing this, so it's called tiller list, but it's not really till elicits, like occasional, tiller.

C

It's like serverless, that's Taylor, less a Remus Remus one of the original chart, helm authors and who works at J. Proc wrote the tiller list plug-in I have never used it myself. So I can't give perfect my stamp of endorsement, but I I looked it over and it looks like a really great pattern. I know a lot of people use it. So definitely something to consider right.

B

So that that's kind of Atilla we've secured tiller is solved. Everyone is unhackable now right. So what other things can go wrong? You want to talk a little bit about this kind of like the way the values can leak secrets can leak. It says.

C

Yeah yeah, absolutely so, are we on the value leech slide yet, okay? That being so, oh the sad face. Yes yeah. So this was actually a pretty interesting thing. It was an interesting thing for me to learn because I won't go into too many horror stories, but I did not know immediately when I first started using helm. That I wasn't quite aware of the tenancy.

C

How that how Helms tendency model or say tillers tenancy model can can match that your kubernetes tenancy Tennessee model or be mismatched, so, for instance, when we were talking before about let's say, you're using a you're doing multi tendency with you've got either each user or each project or each team having their own namespace and you've got all of that lock down really well inside of a cluster right and then the this situation we talked earlier, where perhaps you've got helm installed globally. We're sorry tiller installed globally.

C

When you install a chart and pass values to that chart that override the default values, all of those values are stored somewhere. You know tiller has to enable in order to be able to do roll backs and have that function on they have to store it so where they store their stored in config maps wherever you install tiller. So if you are installing, if you install it globally in queue system, those config config maps for every single relief chart release the values that you pass. Your custom values are stored in that config map.

C

So if you see where I'm going here, if someone normally say you're still limited to a namespace, you can't really just go inside of coop system and get that configure yep right. So if it's protected that way but helm has this awesome command called helm, get values and you can still get the values from that otherwise protected config map for someone else, some other teams namespace, you can get all of the you know.

C

Clearly again, if you, if everyone is cluster admin access, this isn't really a problem for you, but if you do have that other tenancy model, this can be a really big problem. For you see.

B

You can see with this example right, we're running home LS and we're great getting the release names of all of the charts that are installed and then we're doing at home, get values and we're simply and then we're crapping for the word pass right. So we're literally gonna get anything weird like password written in it that's stored in a home value. So if you have you get access to home, you can run this and it's not great, and you don't have a ton of protection against this, so hopefully home 3 without Atilla will solve that.

B

And so that's why it's super important to lock down helm and lock down tiller to be just the users who should be able to access it for the particular namespaces to be able to do it. Otherwise, you do have ways to leak, values, leak secrets through helm itself, and you can see on this previous slide. You can do a get config map, the release name and the version, and you can get all of the values they are base64 encoded, but like so what that's just an extra command to decode them right.

B

You should also be storing your helm, charts and values in version control, I assume everyone is storing them and get right because we're doing get ops, we're doing you know CI CD, etc, infrastructures code. So it's all up there.

B

So now like we're, how are you storing your secrets, he's storing them in github with the other values, or are you storing them somewhere else, and you shouldn't be storing them in github, even if it's a private github enterprise or whatever, like it's still secrets in plaintext, on something that, in theory, a bad actor could get access to and I? Certainly don't control who has access to I. Do weird things to my github enterprise that some other teams, so I shouldn't be trusting that which meeting about that zero trust. So what what?

B

How do we want to like? Where do we store our value? Scott yeah.

C

I mean right so, like some of this isn't really specific to helm, but then it does really relate to what we were just talking about right. You know, obviously you don't put passwords in API keys and in your get train, of course, but but but but clearly you know there are ways to get around it. There are ways to work around it, so you know one way is to in store.

C

Excuse me store encrypted encrypted secrets, right like say, using vault or or using some other home made another project that does encryption right, one way or another, but then, of course you have to have some mechanism to decrypt that properly and you need to make sure that that's running with your with how you setup your releases with with tiller, you could do and to be quite honest, this is what I still do.

C

This is what I do, just because it's kind of the most foolproof, but it is also a little bit manual that you could, you know, do coop control create secret. You know create for every single secret that you need again I'm not really advocating for that. From a security point of view, it's a it's a good way to do it, but from a usability point of view and a workflow point of view, it might not be as nice as what you want. So there are some other options there. There's that lovely project bitNami.

B

Tools: yeah secret, yes and then, of course, it's you've probably got some form of like encryption storage available to you. So whatever you have is hash cops, vault or something like that. You can probably use that I've certainly used like a shared key pass file and then, when I actually create secrets, you're just creating secrets and putting that in there now, when you're, actually writing helm charts a lot of chart. Authors will actually just put the like secret values as values in like the the values file.

B

I prefer to provide an option to also give a already created secret and if that's set, then you don't actually render out those secret those secrets, and then it's on you to then create those secrets. Before you run the helm install. However, you would do that so like if it's like a key dot, JSON file, you can like do it. You know create secret from file or whatever. If it's up involved cobalt, there are ways to get values into secrets from how she got vault, etc. Yeah.

C

And there and a lot of the community charts do have that logic built into them, not every single one. So you know, if you see you know, if you see one that doesn't you could feel free to like add the the 812. You know pull request. That's are open right now and I anyway. That's almost too sad to be funny, but the I have seen people you do use what you were talking about, Paul and tie a service and with you know, let's say LastPass API and automatically creates. We have one minute.

C

Oh, let's move on yeah.

B

So we just wanted to kind of finish with like um three is coming, so like did everything we just talked about. Does it even matter and well? Connors, yes, and no like tiller, is going away, but it's probably going to be awhile before we had a lot of us adopt helm three, because we have so much invested into the current version of helm and so helm.

B

Three will move towards using crts and other more like kubernetes native things, and we won't have tiller but really we're just gonna be shifting around the attack vector and we're probably gonna, be opening up more security vulnerabilities than we're closing right when we get rid of till it's.

C

Always because.

B

We're terrible so like you still need to keep all of these concepts in mind and be thinking about the two can a tenancy model of your cluster and that sort of stuff. So we're really looking forward to with helm three or the new and surprising ways that we're gonna get hacked and people are gonna, get free, bitcoins, so they're gonna mine off our our clusters, so that that is kind of it. So we have kind of that's all. That's what we went through.

B

Here's some references and further reading a lot of the stuff we talked through, especially the the blocks of text. Demos like patching the deployment are in that bitNami blog. So that's really worth reading and there's some other links there and these will be shared somewhere at the end of the event. If you want to get access to these, and so that is it and we are right at time. So we probably don't have time for questions. Is that with the Q&A.

C

We did want Q&A yeah, well, we'll we'll be out there if you want to pull us aside. So thank you all very much. You.