Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Secrets Store CSI Driver: Keeping Secrets Secret - Anish Ramasekar, Microsoft & Tommy Murphy, Google

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Secrets Store CSI Driver: Keeping Secrets Secret - Anish Ramasekar, Microsoft & Tommy Murphy, Google

Applications running on Kubernetes require access to sensitive information (passwords, SSH keys and authentication tokens). But how do you configure your applications when the source of truth for these secrets is an external secret store? What if you need to store, retrieve and perform zero touch rotation of these secrets securely? Meet the Secrets Store CSI Driver, a sig-auth subproject providing a simple way to retrieve secrets from enterprise-grade external stores such as Azure Key Vault, Google Secret Manager and HashiCorp Vault. In this session, Anish and Tommy will demonstrate how to use the Secrets Store CSI Driver to mount and rotate sensitive information from external secrets stores in the Kubernetes application. They will also discuss trade-offs of the CSI driver versus other solutions to accessing external secret stores and how CRDs are used to make pod portability across Kubernetes environments possible.

A

Greetings and welcome to our talk on the secret store. Csi driver keeping secret secret.

B

I'm anish ramshakar a software engineer at microsoft. In seattle, I worked on the security projects in the container upstream team. I'm one of the maintainers of the secret store csi driver and the azure keyboard provider for secret store, csi driver.

A

And I'm tommy murphy, I'm a software engineer at google in new york city and I'm working on the secret manager products and I'm also one of the maintainers of the secret store, csi driver and the google specific provider.

A

So why are we talking about external secret storage when kubernetes has a perfectly good one built in first, the kubernetes secrets may not meet your data arrest encryption requirements, although there are kms providers to enable data encryption, but depending on your organization, you may have already standardized on a third-party secret solution and you're looking to use that your external secret solution may have some sort of secret rotation story or integrations that you're looking to leverage.

A

And finally, if your organization has already invested in auditing and alerting on a third-party secret system, you may not want to duplicate that effort for your kubernetes secrets. So what do you do?

A

There are a few options to consume external secrets. First, you might look into modifying your application to fetch the external secrets from the external api directly using the sdk sdk provided by your provider.

A

This may not be possible, though, depending on your deployment, you may not have the code to edit or it may be prohibitively expensive to implement these changes and, if you're targeting deployments against multiple secret providers, this effort would need to be duplicated for each secret system.

A

So you may look at just copying the secrets to kubernetes secrets, perhaps with the controller. This is portable but won't and won't require application changes. But you may lose out on some of the benefits of your external secret store. Like data rest, encryption and the identity that's used to access, the secrets may go from the workload to the controller. That is duplicating the secrets.

A

You could also use a sidecar to fetch and write secrets. The sidecar may be injected using a mutating web hook here. The pod identity would be used to fetch the secrets, but the sidecar and the web hook may add additional operational complexity that you're not uh prepared for.

A

Finally, there's the secrets store, csi driver which uses the container storage interface specification that we're going to talk about today and we'll cover some of the features that we think make it a good fit for consuming external secrets on kubernetes originally developed at deus labs. This storage driver is now being built and maintained as a sig auth sub project.

A

The driver allows kubernetes to mount multiple secrets, keys and certs stored in enterprise grade external secret stores into pods as a volume. Once the volume is attached, the data in it is mounted into the container's temporary file system.

A

As a storage driver, it provides a familiar file system, mount experience to your compute workloads. It's also pluggable and supports multiple external secret providers. Without modifying your application or changing the pod yaml, it can load new values of secrets throughout your pod's life cycle, and it can also even sync those secrets to kubernetes secrets for further compatibility with existing deployments.

A

Finally, it supports both linux and windows.

A

You can use this driver if your cluster is on 1.16 or greater or if you have windows, nodes you'll need to be on 1.18 or greater.

B

The current supported providers are azure keyword, google, secret manager and hashicorp quad and in the future we are looking forward to an aws secrets manager, provider.

A

So how does the secret store csi driver work? The driver is installed as a daemon set onto each node of the cluster. Additionally, there needs to be a provider specific daemon set deployed alongside the driver when a pod is created through the kubernetes api. It's scheduled onto a node.

A

The cubelet process on the node, looks at the pod spec and sees there is a volume mount request.

A

The cubelet issues, an rpc to the csi driver to mount the volume the csi driver creates and mounts the temp fs into the pod. The csi driver then issues a request to the provider.

A

The provider talks to the external secret store to fetch the secrets and write them to the pod volume as files. At this point, the volume is successfully mounted and the pod starts running.

A

So, let's dig into some yaml here's an example pod. This pod mounts a volume at far slash secrets. This volume is a csi volume. Instead of a secret's volume, the driver name in the csi volume tells the cubelet to use the secret store driver for this volume.

A

The volume also references the secret provider class app secrets. So what is the secret provider class?

A

The seeker provider class is a namespace kubernetes custom resource that is used to provide the driver, configurations and provider specific parameters to the secret cs4 csi driver we've looked at the pod yaml and the secret provider class cmo. But how do you know which versions of secrets are being used by your pod?

A

The secret provider class pod status is a namespace kubernetes custom resource that is created by the csi driver to track the binding between a pod and a secret provider class. This resource contains details about the current object versions that have been loaded into the pod mount.

B

Let's look at the demo now. First, I'm going to show you what is there on the cluster, so we have a kind cluster with a single node. That's running kubernetes version 1.20.2 to show how the secret store csi driver enables port portability across different external secret store providers, we're going to start with deploying two applications written against specific third party secrets, and here we're going to be using azure and gcp.

B

First we're going to see what the pod yaml looks like for these applications. Using the external secret store api to access the secret.

B

This is a sample yaml for a pod using the azure keyword api to get the secret from azure keyword. As you can see here, the keyword name and the keyword secret name are provided as arguments for the container and the credentials are obtained from secret store threads. The kubernetes secret that I've reconfigured.

B

This is a yaml file for a pod, using the google secret manager api to get the secrets from google secret manager. The secret name here is app secret and the version that's being fetched is latest.

B

Now, let's go ahead and deploy these pods in their respective namespaces.

B

So now we've deployed the parts, let's check if the pods are running.

B

Now that the pods are running, we can query the logs for the pods to see the secret that's being used by the pods.

B

So, as we can see here, the azure key vault pod is logging. The secret from azure keyword and the google secret manager pod is logging, the secret from the google secret manager.

B

Now, instead of having different application implementation for external secret store, we have this application that was returned to consume the secret from the file system. Instead, using the secret store, csi driver we're going to show how this application can get secrets from either secret bracket.

B

Before we jump in. If we look at the code here, we can see that this part is trying to get the secret from the file system and log the secret.

B

As a first step for the demo, I am going to deploy the secret store csi driver using help in the cube system, nail space.

B

It's recommended to use a separate name space for the csi driver, pods other than the ones that are used for the workload. Now, let's take a look at what the helmets I've deployed.

B

We have a csi driver daemon set here. The csi driver pods need to be run on all the nodes so that the cubelet process on the node can talk to the driver to mount the volume. Now that the driver has been installed, let's go ahead and deploy the providers in the cube system namespace.

B

So first we're going to be deploying the azure keyword provider and next we're going to be deploying the google gcp plugin.

B

Let's check to make sure that the provider and the driver pods are running now that the csi driver and the provider pods have been installed and running. We are going to take a look at the pod, yaml, that's being used for the the application.

B

So, as you can see here, the board specifies the volume mount. The name for the volume amount is secret store in line and the amount path within the container is going to be mount secret store and in the volumes the same name is being used, and here the volume type is csi and we can see the driver that's being used. Is the secret store, csi driver? And here, in the volume attributes we have the secret provider class csispc that we will take a look at next.

B

So when we took when we take a look at the secret provider class here, we can see the provider that's being used, is azure and in the parameters we can see, the keyword name is kubecon. Eu 2021 and the objects is an array where you can define multiple different objects and the object type here is secret secret and the name is app secret. uh The possible values for object type are secret, key and search.

B

Now, let's also take a look at the secret provider class gamma for the gcp plugin, so we can see here that the provider is gcp and for the parameters we are trying to get secrets, so the secret name is app secret and the version that's being fetched is latest.

B

So next, I'm going to apply the yamos to create the secret provider class in the azure and the gcp namespaces.

B

Okay, now that we've configured the secret provider class now, the next thing is to deploy the same pod yaml that we looked at in both the azure and the gcp name. Space.

B

So when these parts get scheduled to a node, the cubelet process on the node will see the volume definition in the pod spec and invoke the csi driver to mount the volume the csi driver will mount the temp address and make an rpc call to the provider to fetch and write the secrets to the file system.

B

Let's check to see if the pods are running now that the pods are running now, let's look at the logs, the pods are running. Let's first check the pod mount to see if the files exist.

B

So, as we can see here in the volume mount path defined, we can see the app secret file and, let's try to log the file content and, as we can see here, we see this secret is from azure keyword.

B

Now, let's check the logs for the pod to see what the content is. So this is for the csi pod in azure namespace, which prints out hello from azure keyboard and let's look at.

B

And let's look at the logs for the csi port in gcp name space, and we can see that it was able to fetch the secret from google secret manager. And now, as you can see, the same application is working against either sql store.

A

Now that you've seen the driver in action, let's dig into some more advanced features. The csi driver provides an optional feature to sync. The mounted contents from the pod as a kubernetes secret.

A

A common usage of this feature is to store the tls certificates in an external secret store and have the driver sync it to a kubernetes tls secret for use by an ingress controller. In addition, the synced kubernetes secret can also be referenced in pod spec to set the secret as an environment. Variable use the optional secret objects field in the secret provider class to define the desired state of the synced kubernetes secret object.

A

And it's generally accepted best practice to periodically rotate secrets.

A

If your external secret store has an automatic rotation feature, you may be interested in how your workload can get the new values of the secret whenever it changes the driver supports automatic rotation by periodically reissuing rpc calls to the provider to refresh the contents of the mount.

A

The driver will emit a kubernetes event and resync the new values of any synced kubernetes secrets.

B

Now, let's take a look uh now: let's do another demo for the sync secret feature. First, we're going to see how the secret store csi driver can sync the mounted content as kubernetes secret, we're going to enable application to work with nginx english controller with dls certificates stored in key vault.

B

So for the demo, I've already created a certificate for common name localhost using the step cli. So let's inspect the certificate using openssl, as we can see here, the issuer, the common name here is localhost and in terms of the validity the certificate is valid for until april 6th.

B

So so now I'm going to upload the certificate to azure.

B

B

Now that the certificate has been uploaded, let's go to the next step, which is to deploy the ingress controller on the cluster.

B

This installs, the ingress controller and all the other required manifests. Now, let's take a look at a sample secret provider class that is going to be used for syncing as kubernetes secret. So, as you can see here, the name is azure tls, so the provider that's being used is azure and the parameters field here matches what we saw previously in the previous demo.

B

So the object that's being fetched is the certain the secret that we just uploaded to azure keyword. The secret objects field in the secret provider class is what tells the csi driver that the mounted content also needs to be mirrored as kubernetes secret. Here. The secret name is what is the desired name? The type is what the seeker type needs to be and, as we can see, the tls key and tls cert are mandatory fields for kubernetes secret of type tls.

B

So now we're going to go ahead and deploy the sql provider class in the cluster. Now, let's take a look at the sample pod service and ingress definitions for the demo.

B

So here we have a static pod, called foo app and the volume mount in mount secret store and the volumes being referenced here is for the csi driver, the secret provider class. Here azure tls matches the name for the sql provider class. We saw in the previous step and also as part of this manifest we are creating service and also another bar back end.

B

And when we look at the ingress specification, we can see that we are using tls for localhost and the secret name here. English tls csi matches the secret name, that's defined in the secret provider class.

B

So let's go ahead and deploy all the required odds back now, let's check to ensure that the pods are running. They are so now. When we try to curl the endpoint, we can see the certificate that is being used is the one synced by the secret store csi driver as kubernetes secret with an expiry date of april 6th. So let's verify that.

B

So when we look at the output here, we can see that the subject matches the certificate that we uploaded and also the expiry date here is april 6, which matches the certificate that we uploaded.

B

As we talked about the secret provider class in the presentation before it is used to show the current version of the object, that's being used by the part. So now, let's check the secret provider class for this particular port to see what version is being used.

B

So, as we can see here, the status is mounted and the objects is matches. The field objects that were defined in the secret provider class and the version that's being used for the part currently is starts with d7f4, and there is also other additional metadata that can help for cluster operators like the pod name and also the mapping with the secret provider class name.

B

Now that we've seen how sync kubernetes secret works, let's also check out how rotating these secrets would work. So for the purpose of the demo, I have already generated another certificate with april 14th as the expiry date. So we can see the changes, so, let's inspect the certificate before we upload it to azure keyword as we can see here. The subject is localhost and the validity for this is april 14th.

B

So now I'm going to upload the certificate in key vault with this new cell.

B

The auto rotation feature in the secret store csi driver should enable us to fetch the new rotated certificate from azure keyboard and update the mount and the send kubernetes secret. So now, when we try to call the same end point, we expect to see the certificate return to have an expiry of april 14 instead of april 6..

B

As we can see here, the expiry date now is april 14th, which is indicated that the certificate that was rotated in the external secret store has successfully been updated in the kubernetes secret on the cluster. Now, let's also check the secret provider class part status again to see what version of the secret is currently being used.

B

Now, as we can see here, the version before started with d, and now we can see that the version here has been updated to the newly updated secret.

B

So, as part of the demo, we will also try to rotate the secrets that were used by the application as part of the portability demo to see if it can pick up the latest value before we rotate the secret in key vault or google secret manager. Let's start tailing the logs to see the changes happen.

B

So now I am tailing the logs from the csi pod in azure namespace, and also the csi port in gcp namespace. The csi pod implements a file watcher which detects any changes in the file system and automatically logs the new secret.

B

So now I'm going to go ahead and rotate the secret in key vault to the new value.

B

As you can see, the csi pod in azure news space now has the new value. So now, let's go ahead and rotate the value in the google secret manager as well.

A

Okay, just did that.

B

And there you go, we can see that the secret store, csi driver has updated the content in the mount for the pods and they were able to pick up the latest changes using the file watcher.

B

So what does the future look like? We are working towards the stable release for the driver. This includes increasing the test coverage and also finalizing the driver provider interface. The cncf is commissioning a third-party security review of the project and we're looking forward to more community involvement.

A

If you'd like to get involved, you can join the csi secret store channel in the kubernetes slack. We also have a mailing list for notifications of new releases and security announcements.

A

We use github issues to track bugs featured requests or to answer questions asynchronously and also. Finally, we hold a bi-weekly community meeting.

A

Here are some of the resource, some resources from this presentation that you may like there's the document link to documentation for this psi driver links to documentation for the individual providers and a link to the examples from this presentation.

A

Thank you for watching and remember to keep this secret.

B