Cloud Native Computing Foundation Kubernetes Community Days (KCD) Africa 2022, 24 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Optimizing Secrets Management by Ifeoluwa Oluwole

Description

This session aims to improve the lifecycle of secrets management from integration with secret management tool, to secret creation, rotation, and deletion.

---
KCD Africa 2022 is the 2nd iteration of the Kubernetes Community Days Africa, a CNCF-powered free community event. Visit https://kcdafrica.com for more information.

A

All right, so the next section is um on the topic: optimized secret management, by um if you're, oh, please, don't kill me after I pronounce your name. But yes, if your lua is going to be taking that I don't want to mother the second name. He is finally called ife devops.

A

So that's what I'm going to be calling and throughout the section this section um he's he has over 80 years experience and um across um various sectors, including banking telecommunication, media security and consulting ife, loves devops, just like the name effed devops, just because of its broad um coverage and dynamics and uh there's never a dull moment with ife. Of course, car racing games and other musical instruments are things that, if I find fun and exciting so we're going to be hearing from ife in a bit.

B

Pleasure to be here, thank you so much so um welcome everyone. um I'm sure we had a nice break. um Oh I mean I may just choose to dive right in um secrets. um It's a um simple concept. Sometimes it could also be complex.

C

Are you sharing any clients? We are only seeing a loop of.

B

Oh okay, let me nothing to share, but I could just um leave it on my yeah just okay. Maybe I could leave it here.

C

Okay, if you're not showing anything now you can just take the screen off then, when you want it on, you can put it back.

B

Okay sounds good: okay, okay, yeah, so um yeah just I mean the brief intro um secrets could be a simple concept could be a complex concept at times, but over time I I noticed that the approach to it affects how you know: devops teams perform you know in terms of um security breaches or even in terms of um how holistic or how optimal secrets are managed.

B

um So I thought to myself: okay, based on a few challenges and scenarios. I found myself in, of course, based on several use cases. um How should secrets be handled? um Can they be handled you know within the cluster? Should they be handled by a secret management tool? um How do we refresh our rotate secrets?

B

um Should they be part of gates? I mean, with the whole new githubs model. How should we approach um secrets so based on the challenges are faced and how to efficiently manage secrets? I I encountered some solutions. um You know came up with a stack in how to you know, make this seamless and easy to manage.

B

So this is not going to be the conventional um tekiteki breakdown, but I'll just try to make it as simple as possible enough to relate to it. um Okay, so I'm going to just demo something simple using um using um some tools that make up this tech stack just to ensure that we can optimally manage secrets. um I mean at the end of the day, the whole the takeaway from this is um whatever stack. We choose whatever back-end management tools.

B

We choose um it's something we can relate to it's something we can easily manage, and you know, most importantly, something that um meets the needs of. You know the team and the organization in terms of managing those. You know sensitive, um sensitive artifacts in our environment, okay, so I'm going to share my screen now.

B

So I I just have a shot um I mean I I just did some brainstorming, some typical challenges that I had faced and maybe what you know others might be facing in terms of um the overall concept of of secrets. You know how to manage the entire secrets life cycles from when they are created to how they are rotated, so how they are even injected into um the clusters. You know the environment to how they are deleted, also disaster recovery strategies for secrets.

B

So um you know we have several solutions that could back up the kubernetes clusters, but um how well are the secrets also backed up? How do we have? How do we ensure that if we lose our secrets, we can recover them things around githubs, you know: how should we commit secrets to git things around show the actual secrets you know sits in the cluster.

B

Why not I mean it should be managed by an external secret management to maybe properly encrypted and also guide against um accidental division.

B

So things like that, uh you know some of the concerns that made me come up with this with this same idea, so the stock that will be used here. um Obviously, now secrets operator, I mean formerly called um kubernetes um external secrets, but that's been deprecated, so it's now based the external secrets operator. It's you know a whole lot of improvements, um so that would basically just be an abstraction layer.

B

um So on a high level as well there's a stock, it will be using called reloader by um starkita.

B

I think it's a swedish company also, so they also share this solution with the community and in terms of the cloud provider I'll be using it or using eks on amazon and our backend circuit management will be secrets, manager, aws secrets manager.

B

So some of these words, some of these terms or tools, might be new, um but I mean some of us might, you know, resonate with some of them, but nothing to um get worried or get scared off I'll. Just deploy this in a very simple fashion, and then maybe we can run through questions. You know observations concerns um yeah, so I already have my eks cluster set up. I think one of the sessions in the morning went through. You know, setting up a mini cube cluster, so any of that works as well.

B

If you have, um you know, connection set up to your secret management tool, but before now I have my eks cluster set up already.

B

As um of the prerequisite, so the first thing I'd like to have will be to set up reloader.

B

um The charts are available in the github repository, um so I mean I could probably share this later um so I'll just be installing um adding the reloaders home chat to my local repo home repo. Just do some updates and install reloader, so I'll just grab this and you know steer so it's grabbing the latest charts and we'll install now downline will still come to that, but maybe I should just mention this now. um Setting up your letter is pretty easy. It involves just adding an annotation.

B

um No have most of the heavy lifting has been heavy lifting has been done in the setup.

B

So once you install the charts in your cluster once you add them, you know reloading your cluster. You are good to go. Okay, so reload is added um check to be sure. So um I can just do keep city I'll get all just to see that my reloader app is running.

B

Okay, so I see it's running, deployment object, looks good and the port is also running fine. um Now the next tool to install here will be auto. Deploy will be external secrets operator, um but one of the prerequisites or um the requirements would be to set up or define your aws credentials.

B

Why do we need your aws credentials, because you need external secrets operator to be able to communicate with your um secret management tool in this case I'm using um aws secrets manager. So that means I need aws credentials. You could use um azure key volts or maybe gcp's vault or even hashicorvault.

B

Any of those other systems can work behind the scenes with eso, but in this case I'm using um aws and, interestingly, I'm using irsc.

B

This is a concept I won't dive deep in it's just im. Rules for service accounts so, rather than me, defining access, keys and secret keys. I have a role defined to define the role.

B

This is my policy here. I just need four permissions. I need to you know, describe the secret list secrets and get secrets. I can just show you that in the console.

B

I have a ws single sign on setup, just in case you're wondering what um this is so this gives me you know a more easy and streamlined access to my environment. Okay, so let's see what we have set up for um external secrets operator.

B

Okay, so I have a rule called external secrets and the permissions right here. um Dutch resource get secrets. It describes cigarette and list secrets so and it can only access my secret manager resource in aws.

B

This is all you need just um the user in this case, I'm using a rule with a permission that just needs access to aws recruits manager. You can streamline your resource further, you know to for the secret, but we'll get to this part in a bit.

B

So I have my role set up.

B

I have my permission set up and I think I'm good to go at this point to deploy external secrets operator.

B

Straightforward as well, you can grab the helm chart for external secrets in the helm, charts and repository for external secrets. um I have a repo added locally already, so it's optional for me to run this I mean you should probably see it already exists.

B

um Yep already exists, so I will install external secrets now. Take note, I'm passing in um my values file here, and this is simply the same. The default from the charts and the only tweak here is, I am adding my row like I said, I'm using im rule for service accounts, so this enables me to um grant a service account in kubernetes um to grant it permission to assume or work with an im role in aws.

B

So with this, my service account can access aws. As long as I have this annotation here and that's all the rest of the values file is the same. Just use the default um yeah except you have some other um interesting levels of configuration.

B

Okay, so I will install this.

B

I'm doing this in the namespace by the way. Before now I created an external secrets namespace, but this can be in your default namespace um I mean I recommend it to be in a separate namespace, it's good to always use a proper namespace strategy.

B

So this will install the external secrets operator.

B

It's crds as well. I installed.

B

Okay hold on, I think um this is for.

B

Okay, considering this okay.

B

So I think that is pretty short already installed this, but let me just be double sure.

B

Okay, so I think yes, it's installed great, so um yep external secrets installed successfully.

B

Now the concept of external secrets is, after deploying um the operator, you need to set up a secret store. What is the store? The store is simply a backend object that establishes connection to your secret management tool, so it could be hashtag vault. um It could be a jockey vault like I said it could be in our case, whereas in aws secret manager- and this is the manifest here- all of these charts and manifests are in the home chart repository, but I will just explain what um the secret store does.

B

So this is the money. First, um I'm giving it in aws secrets manager, um because that'll be the name for aws um I mean just defined. It can be honest. You can call it anything.

B

The region of um my secret, my manager, will be eus, so that's the london region, so that's the region I'll be using, which is this region here.

B

Secret manager.

B

Right here I think I should be in the okay.

B

Okay, so let's go to the london region, eus too, that's where we should be.

B

Okay, great um next up the service is secret manager. So um in external secrets, when you say secrets manager, that means you're referring to a ws secret manager, I think, if you're working with azure key vault, it has to be maybe something like um azure maybe kv, and I can't recover. You know it's it's there so, depending on the backend tool that will determine the kind of name that should be here.

B

But of course this can be any name, because it's just a friendly for you to identify it as one of your kubernetes objects, so I'll deploy the um cluster secret store now and while that will be deploying, I will explain um one concepts.

B

Now, in external secrets, we have what we call the cluster secret store or secret store, and you have what you call cluster external secrets or external secrets.

B

If you notice it's just the word cluster that is different in both, and why is that? um It means you can choose your object to be accessible throughout the kubernetes cluster or you can choose to tie it to a particular namespace in this case, I'll just like to have one secret store to serve all secrets in my cluster, so I'll go with the cluster secret store, so this is defined. This is created. Okay, so, um let's see.

B

Let's be sure, let's I mean: let's describe this object, let's, let's see what's happening now we have a cluster stock created, of course, with back-end connectivity to um aws.

B

Although they keep ctl gets cluster secret stall, let's see what we have. Okay, we have this created.

B

I think I'd like to describe this as well.

B

I missed the word described.

B

Okay, so when I see an event like this, this is just a way to validate that your external secret operator successfully, you know, can successfully authenticate to aws using the role or the credential you provided maybe an access on a secret key stored somewhere on your role. In this case, I'm using irs like I said so. This is good. This is a good sign that we can interact with aws and we can also interact with the aws secrets manager.

B

Now we have um we load that deployed. We have external secrets operator deployed. We have this store, created and deployed as well, which connects to aws. I think it's. This is now. Let's get to the fun part where we start creating secrets.

B

um Now, like I explained the concepts of plus text on our secret or external secrets, I want to be able to create secrets from anywhere and map to any namespace. So that's why I'll opt for the plus text on our secrets.

B

Now, before I create my cluster xml secrets, there will just be one prerequisite um and I'll explain that now this is what the closed site in our secret looks like, and this is what the manifest looks like I'll just run through this, so that you can understand what each section means.

B

Of course, the name is the name of your closed sectional secret object.

B

Now the external secret name is the name it will use to create, or it will give you external secrets. What's the difference there, the external secret will be created in whichever namespace you define so remember. You are managing a kubernetes infrastructure. You are using. You know the namespace strategy to properly isolate workloads. Your environment I mean, maybe you have um a namespace for dev or a namespace for staging, or you have a namespace for critical apps and let's create collapse with several levels of isolation.

B

At the same time, what about whichever secrets are consumed in the name? Space has to be within that namespace. That's the whole idea of the virtual isolation. You know with regards to namespaces, so here I'm saying um creates an external secrets and provision into this namespace in this case I'm using a default image. So this can be any namespace. This can be maybe um abc app namespace.

B

What does refresh time mean here? Refresh time simply says.

B

External secrets operator make sure that these external secrets always exists in this name space and who is in charge of that? It's the close text now secret. So every one minutes, the cluster external secrets, checks to make sure the external secret is in the default namespace, but it gets even more interesting. I mean there's another layer just to ensure 100 or maybe near 100 protection or availability of your secrets.

B

The next layer there is external secrets and tone also creates a secret within the same name, space so you'll end up seeing three objects: you'll see an object of kind, external secrets, external secrets, you'll see an object of kind, close text and all secrets and you'll also see an object of kind secret.

B

Of course, the one we're all familiar with is secrets which that is what our application will be consuming, we'll get to that part in a bit.

B

So for the external secrets, it now creates a secret called test secrets. Of course, this secret story, f is just saying: hey. Remember I created a secret store name named aws secrets manager, that's the store! You should connect with back end to fetch my secrets, so that's this. Just like um a reference call for synchronization, hey check, my aws secret store, fetch all my secrets or fetch my secrets named this. You see a section called um data from or extract like. I said this might be new.

B

This might be strange, but um you know I'll advise you check um the docs as well um I'll. Just try to do justice to you know, explain a bit further, but um the more you relate to this you'll understand what each section you know is all about.

B

So three kinds of objects will be created when I apply um this manifest, um which I'll do now. So, let's create our close text and our secrets- oh before that, like I said, create secret and secret manager, and why am I doing that at a point when I'm telling it to fetch this secret? I need to make sure that I have a secret named this in aws or in my secret management tool, any tool I'm using.

B

um So I'm going to show you that so I have the secrets defined, and this is the name of the secret.

B

So what I'm telling it here is extract the content of this from aws and please create a secret for me in kubernetes with those contents and how do you connect? Please use this to connect this secret store now this year, I'm achieving a couple so many things here.

B

So I'm going to apply this and we'll see what happens. um So, let's just create this. I just have the syntax handy just to save time and effort. Okay, so I have the cluster actionable secret created so.

B

Let's see, we should have something called close to external secrets, yes, which in turn creates.

B

Content creates an external secrets in default, namespace right here, which in turn creates a normal. You know the regular, the usual secret object, which is what you know the application will consume. So now, let's see get secrets.

B

And here we have secrets, I mean we're going to see something fun down the line I could choose to delete the secrets um I mean this is one of the problems, the pros of um external secrets because of the synchronization um I can choose to delete the secret and it will create its cube, ctl, delete secrets, test secrets so to refresh every I think, one minute or even immediately, let's see um yep. So you see, let me just clear this okay secret here, 50 seconds old was deleted and here it automatically recreated it.

B

So, let's say um someone erroneously, you know deletes um maybe just a typo and deleted the secrets.

B

You can be rest assured that that secret will be recreated if you're, not using an external secret management tool and if you're, not an external secret operator, um there's no way to there's no way to recall the secrets, unlike deployment objects or maybe state two sets that have maybe um things like revision history limits. Where you can, you know roll back objects or depending on. If you do upgrades, there's no there's no way to roll back or recover.

B

You know objects like config maps or or even secrets once you're deleted, unless you have them saved somewhere, maybe locally or um in a private, in a repository to yourself.

B

Okay, so we have created our external secrets. We've also indirectly created our secret. So I think we are ready to test this. Let's, let's deploy a test application.

B

I just have a busy box here which I'll run which I'll deploy and I made two additions to the busy box. The first one is the annotation for reloader, and this will simply ensure if I update my secrets, I don't have to manually, do a ruling restart it automatically deploys um restarts all the pods.

B

Now that will come in handy when you have lots of it may not mean a lot if you just have one deployment, object or two ports, but when you start dealing with hundreds of deployment, objects or stateful sets- and you know you start managing large, huge, um a huge number of micro services and that's when this will come in very very handy.

B

So I have this annotation here that says: hey just look out for all my maps, secrets and config maps. If you notice any of them is updated automatically restart my ports, that's the first addition I have here. The final addition I have is, of course, I'm bringing in my secret. So I'm just saying: hey um fetch some environment variables from these secrets.

B

Remember this secret is the third secret that was created from this, which is right here. So this is the third object that was created by me applying the cluster external secrets.

B

So, let's, let's deploy our test app, let's see what happens.

B

I don't have a ready syntax for this, so I'll just run this yeah, let's apply okay. Let me just clear this to have a clear screen. um I'll gets all now, let's just see what's happening.

B

Okay, I have my pods running. um Let's okay, I think um let me execute one of them just to um let's see, what's going on.

B

We just want to validate our.

B

Environment variables, okay, so say: um trains env, okay, I think. Let me just um sort this properly. Let's see.

B

Okay, I just want to see what I need: okay, so this uh environment variables in the port, let's see and they match what we created here.

B

So what's the entire flow of this um I'll just run through it, um external secrets operator um connects to aws through the cluster secret store. That's the first part. The second part is the cluster acts on our secrets, creates an external secret in your defined name. Space and classical secrets always makes sure that the external secret exists in turn, the external secrets created the secret object, which you know I assigned to the port or to the deployment, and it always makes sure that that secret object always exists.

B

So if I delete the secret objects, external secret automatically recreates it if I delete external secrets, classics in our secret automatically recreates the external secrets, so you see there are you know more than one level of protection here to prevent an accidental deletion?

B

So let's say: um let's see what happens now. I think this is the final interesting part. um I'm going to modify um a security. Let's say.

B

I add maybe one of the locations here, so let me um update the secrets, let me add a, let me add, let me add something here. Let me see um kcb, um let me say, city.

B

So um let me just add, um let me start the main capital here in lagos. So I'll just say, um let me say credit card and I will save.

B

Oh okay, so this is saved. So let's monitor what's happening here.

B

If I, if I do it gets all, we might observe something, um let's see if that will happen. I expect um reloader to go into action here.

B

Okay, so I think the refresh time is one minute, so just wait for a few more seconds and you should expect any deployments to kick in.

B

B

That is active, so we see here that the last set of pods um are terminating while new ports are running. Remember this happens in a rolling fashion, so there is no downtime up. Time is guaranteed um and, of course, this also respects your um your update strategy. So, if you're using maybe a fresh redeployment of everything or a rolling restart or a ruling, upgrade that's what will happen reload, I respects your update strategy, but by default um the default kubernetes of this strategy, which most almost I mean everyone uses this the ruling upgrade.

B

So it ensures that um it iteratively shuts down each pod, one at a time and then spins up each new port. So it's example: the one brings up a new port takes down part two and brings up a new port.

B

Just to be sure that all our new pods are running and.

B

To be double sure, I will um just exactly you know connecting some of the new pods and be certain that we have.

B

um We have our new environment variable.

B

So I'm going to print the environment, variables, env and search for cd, and I have kcdc.

B

Voila, so this um I mean.

B

It looks like a lot of heavy lifting has been done here and this achieves the goal which you want to achieve. Now I mean the pros of this stack. I just did some brainstorming and I said, okay um secrets cannot be part of the github's model. I mean you don't want to become it's an actual secret manifest into git, but with external secrets operator and with this entire stack as a matter of fact, you can have your external secrets manifest in gates, in fact, um depending on your github strategy.

B

In some cases, if you're using tools like argo, cd or um jenkins, mgx 3, you can have some levels of integration where, when you make a change, aggro city automatically, does um you know recognizes that new artifacts?

B

At the same time, if you're using gx3, you can, you know, build a model that says um automatically connects to aws and create secrets. So that's the next layer, maybe um the next um I mean maybe the next kcd, who knows maybe that will be featured. But that's something also. You know that's possible. You can automatically create secrets from aws without manually, defining your secrets, so just have something in um just properly define your github's model and that is very much possible automatic secrets. Rotation.

B

I mean the option is here: um you can configure imagine this database credentials or you know any other form of credential. You can configure automatic rotation um dr strategy. Well, I didn't adjust to save cost since it's just a test, but you can also set up some replication. um If I attempt to create a new secret, so I think it's it's even here um you can choose to relocate secrets to other regions, so you're, not um scared of the fact that the region may go down or your cluster might be on.

B

You know unreachable or you know your cluster crashes and all of that that automatically handles your disaster recovery strategy. Also secret synchronization, like we saw um you delete a secret, it gets you delete a secret object, it automatically gets recreated um you um you update the secrets it automatically gets. You know the pods gets redeployed in a ruling fashion.

B

I mean a couple of that and the concept is stuck. Nothing is perfect, there's always room for improvement. So to me um so far, so good, I feel, is just one more layer of operational management which happens just once, which is at the point of setup now, because the rest of the operations are basically, you know, I'm automated.

B

So if I'm to deploy a new application, I just have to create my secret manifest, create my backend secrets and aws and redeploy whenever I update my secrets: reload automatically reloads.

B

So I feel with this um I think optimizing secrets you know, should be more user friendly shouldn't. Be that scared topic um handled by only the maybe experts or the gurus, and you know I mean the kubernetes community. I think everyone can relate with it. Everyone can just deploy a simple stack depending on maybe your cloud provider or your segment.

B

Tool of choice and you can achieve an optimal security management life cycle. You know without um being an expert, so I think that's the end of my demo. I'm going to stop sharing.

B

Yes, thank you guys.

B

So if you have any questions.

C

Awesome that was a great session. I've always looked forward to secrets and different other options around the uh secrets management. I think one of the most popular ones is vault, but I've never seen a demo of that of aws.

C

um I don't we don't have any questions yet from the audience. um I don't know if anyone has any questions uh if we can share for if, if you're lower, to answer.

C

Okay, someone is asking: swedia is kubernetes service that does this.

B

Kubernetes service- um I guess you're referring to the operator external secrets operator before now. It was kubernetes external secrets, but um that has been deprecated and improved. So we now have the external secrets operator. So for now we can say that is the kubernetes um controller or service that does that's. That is the backbone of this entire stack. Basically,.

B

Sorry, I think I think you're on mute.

C

Don't mind me, I muted from the microphone, but I was seeing the screen as as, if everything was fine.

C

Okay, so thank you very much for the awesome presentation and we look forward to having you on our subsequent event. It was a very great presentation.

B

Thanks the pleasure, thank you. Okay,.