Cloud Native Computing Foundation ServiceMeshCon EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Shh, It is A Secret: Manage Your Workload Certs in Service Mesh without Persisting any Se... Lin Sun

Description

Shh, It is A Secret: Manage Your Workload Certs in Service Mesh without Persisting any Secrets- Lin Sun, Solo.io

Most service mesh projects provide self signed CA but that is NON-STARTER for a production environment as most organizations already have their PKI system in place before they adopt any service mesh. While many service mesh projects have added the support for plugging in your intermediate CA or external PKI system, they however require persisting the intermediate or root CA’s private key as Kubernetes secrets which is a security concern for them. This talk discusses a few innovative approaches in the service mesh community to tackle this challenge and the tradeoffs among them.

A

Today, I am so excited to be here. Physically, you know the first uh service mesh conch after covid. I'm sure is that for any of you, the first time traveling to a real conference after covet, raise your hands. Yeah very excited. You are here, so we are going to talk about it's a secret, manage your workload, search in service mesh. Without persisting it.

A

Let me quickly introduce myself: I've been working at ibm for a very long time. I think some of you know me from ibm. I changed to work for solo about a year ago, so I'm working on open source at solo. I've been a very long time contributor to the istio project. One of the founding members currently sits on istio steering and technical oversight committee. I wrote a book about istio.

A

It still explained to help our users get started with istio, I'm also one of the new cncf ambassador and I run a hoot live stream, which I learned about service mesh, istio envoy, ebpf, graphql technologies with you together with the audience.

A

So um this is servicemashcon. I'm sure you've all seen some chats like this right, the service mesh architecture right, but today we're going to focus a little bit on the most useful benefit. I guess widely adopted. Reason for people adopt a service match is mutual tls. Would you agree with that? A lot of people looking at service mesh because mutual tias raise your hand. If that's the reason.

A

Yes, I know we've heard that repetitively from our customers right, so we're going to dive into how the certificate is distributed, we're going to dive into how to plug in your external certificate authority and we're going to also dive into a couple of approaches to not persist your private key on disk.

A

So since I come from istio, I'm going to start using istio as an example to talk through how things work, I believe other service mesh works. Similarly, last time when I look at them, so if you use the default profile that comes with istio, you will have self-signed certificate, which means istio control plan is serving as the certificate authority for you. That means the certificate for each of your workloads in the service mesh right. So this isn't.

A

This is a typical, but it's not really what you want to run in production, because typically you already have your own pki system. You probably don't want anything self-signed, that's not integrated with your existing pki system. So the way it works is. It still has an agent which runs in the same cycle in the same as envoy proxy.

A

In the same cycle and the istio proxy, is your agent sends the certificate signing requesters to istio control play and then israel control plan who access the certificate authority uh sends the certificate back to istio agent, then back to the envoy proxy. So that's how your envoy proxy gets the certificates and it also handles certificate rotation before it expires.

A

So if you take a look at south signed as root ca, essentially in the istio system, namespace you're going to see hdlca secrets. So that's the self-signed generated by istio control plane at the boot time of the control plane. You will also see a config map that contains the root certificate for the self-signed route.

A

So, let's take a look at the generated htoca root secret. Essentially it can. It contains four of the files. uh One is the ca search.pam. That's the generate root search, also the private key. So this may be a concern for some of your organization, because it essentially persists that private key on disk as part of the scd as a secret, and it also contains empty, searching and root cert in this case. So if you decode the c8 key file, you will see the private key right. So why is this an issue right?

A

This is like the key to your house or apartment right. You don't want to give that to anybody so, and the other thing I want to point out is by default. The certificate is good for 10 years. So what this means is you give somebody else, the house or the key of your house for 10 years right? If somebody take it, they can access your house for free for 10 years. So certainly not something you wanted by default.

A

Now, let's talk about how the trust is distributed in this model, so uh we issue the startup.

A

It has a namespace controller that propagates the htoca root, search as a config map to each of the name, space in your kubernetes system and when your proxy starts proxy as a site card to your full service, it's going to mount that certificate as a config map on the amount of the configmap search as a volume into your proxy container.

A

So, interestingly, if you use istio today, if you do a quick check by creating a namespace full, even though, without annotate or label it a cycle injector, you would immediately see the stod creates the htoca root search for the full namespace. So that's how the root cert is propagated to each of the namespace.

A

Now, let's talk about in this model, how the trust is distributed. So uh isil agent is going to send a certificate signing request to ecod right. We talk about that's acting also as a certificate authority, which means the certificate and approve the request and sen minster certificate and send the signed certificate back to istio agent in the meanwhile ecod also propagates the config map into each of the namespace, which allows uh the I'm sorry. I should talk about the in this sequence first, so the first sequence is actually uh copy.

A

The config map into each of the name, space which allow the proxy to be able to amount the config map at the boot time and then with the root certificate and also the service account token. The isro agent can send the certificate. Signing request, which is dod, looks checked. Make sure it's good approves that and sends the means certificate back to the istio agent, so the workload search by default expires in 24 hours, but that doesn't mean you know. The service manager waits until the last minute to refresh the workload search in istio.

A

There is a configuration as an environment variable which we don't expect you to config, but for some reason you might it's a 0.5, which means we will rotate the certificate every 12 hours for you so way before it expire. We will rotate for you so now. This is how the default works, and now, let's take a look at how to plug in your own ca. So if you look at uh link d, it's essentially during install time. You can specify your ca search. You can specify your issue key, whether using linkedin, cri or helm.

A

So this is not surprising with kuma. You can also specify your key and search as a secret and configured in the mesh configuration yamo. So these are the ways you can plug in your key, your external ca with istio. There are two ways. The first way is through a ca search secret. The second way is, through api, called match: config ca certificates.

A

So let's take a look at them, so the first thing we want to look at is we see a uh series search in this. In this example, essentially you as an admin who is plugging the external ca, you would populate the ca search secret in the istio system, namespace, where you installed the seo control plane, and then you would specify the search and key and also the search engine if it's an intermediate search and also the root search.

A

So you create a secret populate that into the namespace and when stod boots, it's going to check whether the ca search exists. If it exists, it would use it. If it doesn't exist, it would fall back to the self-signed flow, which we just talked about right. um So in this example, then the rest of flow is pretty much same as what we discussed earlier on with hdlca root cert as a config map propagated to each of the name space.

A

One issue, though uh this one still have private key persisted as a as a secret. So may not exactly be what you wanted um all right. So we smash config so why there are two ways to do this in istio. The reason is, we had uh user, sometimes once multiple root cert, which is a common scenario as you transition, maybe from one root cert to another root.

A

Third right during transition time- or maybe you want to do a federated mesh that has different root cert, so it still has a configurable api to allow you to do that through match configuration, so you can configure mesh route search your actual root, search, image config, which hdod will propagate to you. Is your agent through the xds api. So this is an example of how that looks like.

A

Essentially, you can configure ca certificates inside of match config, which is part of the istio operator api, and you can configure your certificate right there. So one thing is: this is compatible with spf such as spell so very attractive if you're using those environments.

A

So now the next question is: can you plug in multiple ca search for istio control plan? The answer is yes, you can actually do that for the scenario I was mentioned early to federate across different uh istio meshes or maybe in the root transition time frame. So in this case you would have match config istio actual root, cert config through mesh config, along with the ca search as a secret where you install into your kubernetes.

A

So one thing I want to point out is: I find out a little bit confusing is a registration authority which does approve the request. As I learned in these all these concepts, the other concept is certificate authority which signs these requests. It typically happens after the registration authority approves the request by default, all the flow we just went through hcld works as ra and also ca.

A

So that's a lot of work by istio. So now, let's take a look at another community project. It's your csr! This is a very cool project.

A

It's essentially seocsr acting as a registration authority, so the certificate signing request was sent to istio csr, which approves the request and forward to certificate manager which acts as a certificate uh certificate authority notice. In this case, uh one thing really interesting is stod is not acting as ca or I so you have to kind of disable these functions in sdod to tell istio you know, I don't want you to be my ca. I don't want you to be my eye, and I guess say one biggest advantage of this approach is certificate.

A

Manager is very well known right, so it supports a lot of issues, so all that issuers that certificate manager supports you know comes with uh this approach like vault and many of the other ones. But uh one thing I do want to point out is in this approach: private keys here persisted as secret.

A

The other thing the israel community is innovating is the kubernetes csr integration. So kubernetes has a csr api. I believe the api just becomes stable in the most recent kubernetes release, but in ico the integration with this is still actively under development. It's still experimental.

A

So in this mode, uh icod works as ra and kubernetes ca or any other customer ca who choose to implement the kubernetes csr api as a controller can act as ca. So on this approach, when community first introduced one of the huge benefit of this approach is, it doesn't need to persist the private key in kubernetes cluster, which is very cool.

A

So, let's walk through how this approach works. So essentially the service, a or full here would send the certificate signing request to stod and the sdod would act as a registration authority right. It would approve the request and it would actually generate the kubernetes csr as a requester, so sending notice the csr request here from uh from istio agent to sdod. It's just normal certification request and the icod to certificate authority in this case is actually kubernetes certificate.

A

uh Csr request: the format is a little bit different, which I will explain shortly so in this case, sod is going to approve the request and generate a kubernetes csr request and then forward to kubernetes ca or other ca who implements the kubernetes csr controller and and then eventually, issues the certificate back and through sdod is sending the certificate back to the istio agent. So that's the flow of this flow, how it works and when I take a detailed look of how the certificate works in this case, so you can see.

A

uh This is a x509 certificate request that I generated using the open, ssl key command, and this request essentially and on the right side, is a kubernetes certificate. Signing request right notice. It's a different. The the certificate, typically standard certificate uh request, and the request is essentially in the body of here with our base64 encoded format. So essentially, what's on the left side with basics before encoding is part of the request of the kubernetes certificate signing request. You also have the requester here in this case.

A

Is the groups- and you also have whoever is the signer for this kubernetes certificate request.

A

So one approach with this approach: it does require implementation of kubernetes csr controller, so by default, kubernetes has a d4 kubernetes csr controller, I believe, search manager. It also implement a kubernetes csr controller, even though I believe they said it's uh it's alpha, it's not production ready. Yet you could also implement your customer controller. So there are api for implement controller.

A

So it's all pluggable, which is very nice, but one question I would say is there are very limited controller out there, because the kubernetes csr api is very new and just become mature, so the controller has not been picked up like, for instance, certificate manager has a controller, but even with that controller, it's having very limited issue being supported for that controller.

A

The other thing to think through is do you want your service mesh, also using a kubernetes control, plane trust domain right because you may not want to use the same trust domain as your kubernetes for signing your workload certificates. If so, this may not be a good approach if you choose to use the kubernetes controller all right. um The other approach uh which we're doing at solo is not changing. Anything in issio is we actually run a cycle next to istio.

A

In this case, istio continued to control plane continue to act as ca and ra, and the psycha is taking the responsibility of getting getting the certificate signed without so, let's take a look how this approach works. So if you have multiple uh kubernetes cluster, multiple sdod right in this case, you would result less api to your pki system. Less api cards, because essentially you're still using seod to sign your workload certificate, but the intermediate key and search would be generated from the cycle to your pki system like vault.

A

So essentially, only this cycle costs your pki system, and then each of your service still gets their workload certificate from sdod, so this could potentially allow very large scale of service mesh with many clusters.

A

One last approach I also want to highlight is what we are working in upstream uh upcoming, upstream 1.14 of istio release with spell integration, so the hpe team shout out to their team, contribute this function in upstream. So uh for folks, who don't know spell it's really cool that it can attest workload and issues speedy identities by your test workload, you can attach the workload by like the image name by namespace by part, so it can and it's configurable.

A

So you can tell spell how you want to attach your workload and spell can based on the rules and configuration to a test for you. So in this case the spell server would act as ca and I for you and it it can support two key um management strategies. So you could config well to say you know I want to persist the private key in memory. Oh, I want to persist the private key, I'm sorry, I don't want to purchase the priority key or I purchase a priority key on disk.

A

So that's that's configurable with spell, and if you use this mode, don't be confused that you still have hdlca secrets, because that's not really being used because spell is working as the ca and I not sdod so just walking through real quickly how this model works. Essentially, uh it still opened up a hook point uh to allow a uds socket on istio agent. So whenever it's your agent boots, it's going to detect whether the socket exists.

A

uh If the socket exists it will detect. You know to fetch the secret attempt to fetch the secret from the circuit and then the spell agent is going to do a testation on the workload of the of the service, uh just making sure it has the right. It's the right workload with the namespace service account and image name, and then once that's all done it can. You know, issue the certificate workload certificate back to back to the psycha, so the normal flow xds configuration still goes through istio agent to istio control plan.

A

It's just the certification bootstrap flow is uh going through because of the socket existed. It would go through the spell agent in this case, so we actually did a hoot live stream just a week ago. So, if you are interested in this check out our youtube channel solo, I o check out our most recent hoots.

A

I only have one minute so in conclusion, we talk about many approaches right. We talk about the default, how the d4 works. We talk about how you plug in your own ca is possible. How you plug in your own ra, is also possible.

A

How there are some of the approaches out there to not to allow you not persisting your private key so such as kubernetes csr control, plane cycle and also on the external ca integration with envoy sds. With spell as an example.

A

Now I think I may only have one minute four questions. Do I have a minute for questions or no.

A

All right there's a question there.

A

Like there are certain limitations, can you describe those limitations yeah? So it's still csi, it's actually very well known in the community. One of the challenges is uh the persistent private key is a security concern. So if that's a concern for you, you might want to see if that's really the right approach for you, that's the main thing one. A couple of things I like about is your csi. That's super many issues. My main concern is the persisting yeah.

A

The private key, like your private key, is like the key to your house, and typically the rookie is good for 10 years. So, if you persist on disk, if somebody grab it, you know they can access the whole thing for a very long time.

A

Well, the the approach, at least based on the tutorial I've gone through you still kind of have to generate. Yes, you can provision, you can use certificate manager to create the keys and search, but the way is your csi implement today. It still requires you to create that in the installation directory as a secret if you ever tried that project yeah well, I hope this is helpful. I'm out of time. Thank you.