solo.io Hoot Livestream, 21 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rotating certificates in Istio

Description

In this live stream, Hoot #47, we'll look at certificates in Istio. We'll talk about how to plugin your own CA certificates, rotate them without downtime, and show how to use cert-manager and istio-csr to issue workload certificates.

A

Hello, everyone hope I'm coming through I hope. Everyone is doing well today. uh Welcome to hoot number 47.

A

and today I will talk about certificates in istio. If you're here make sure you post a comment, say hi that way. I know I'm not talking to myself only, um but all right, let's get uh let's get started uh so, as I said, I'll talk about certificates in istio today.

A

um What I'll do is I'll briefly explain what the default installation does in regards with the workload certificates and then I'll go through uh uh the scenario of rotating the ca certificates without any downtime and then at the end. I'll also show you how to use cert manager and istio CSR for your ca and workload: certificates, hey Marino, hey Arco, nice I'm, not I'm, not alone. So that's good. It's always good to know I'm not talking to an empty room. I! Guess um all right so before I go into uh into the that topic.

A

I need to briefly talk about just identity just so to to set the field. So everyone knows everyone has the basic understanding of everything um in terms of identity, and you see spiffy here and spiffy. Ids I I talked about spire and spiffy Moore in like a couple of weeks back in a previous hoot.

A

If you want to go and check that one out, uh but just for this talk today uh now the workloads that match for them to use uh Mutual TLS and in order for us to enforce any access control uh policies, we need some sort of an identity and in istio this identity comes in a form of a spiffy ID and that spiffy ID gets encoded into an x509 certificate that gets signed by a trusted Authority, so spiffy, ID, spiffy, cluster, local and as default essay hdb bin on the slide.

A

Here, it's just a string that um and uniquely identifies a workload and spiffy itself. It's just a spec that describe how and where to encode that um spiffy ID inside of an x509 certificates.

A

So when we're talking about so we have the identity, we have the workloads, we have the we have the spiffy IDs and we have the certificates. Rather now, when we talk about uh delivering those certificates to workloads, there are three components that are involved here: dsdod or the control plane. Specifically, the component inside of the control plane called Citadel, the istio agent that runs in uh it's a binary that runs in the istio proxy and the envoy proxy itself.

A

So Citadel uh acts as a certificate Authority that signs and um issues the certificates uh to workloads the istio agent, as I mentioned, it's a binary that runs in the sidecar proxy, and this binary is responsible for requesting a certificate for that workload and then making that certificate and key available to the envoy proxy and the way it does. That is uh via something that's called secret. Discovery service, that's an API in Envoy.

A

uh Now the SEO agent is also responsible for rotating these certificates before they expire and then finally, we have the um the envoy proxy uh that just connects to the istio agent and receives the certificate and the key, so it can use for uh for communication.

A

So we know that we know how the uh how the rotation, how the issuance of the certificates work, but what happens um if I install this here I have a default SEO installation and what happens specifically in terms of certificates CA when SEO D starts up so on Startup SEO D knows about some base, CA data and what it does it distributes that to each namespace uh in your cluster in a form of a config map, that's called a Coca route cert.

A

So if you have your cluster and I'll show you later in the demo as soon as you install istio, if you look at the config Maps across your namespaces, you will see that is Coca Roots art config config map in each one of those namespaces. So that way, once that's in all the namespaces, when the pods start up what they do, is they mount that config map and then, when istio tries to uh talk to sdod istiod will present its certificates.

A

The agent can then accept that connection now I mentioned the sum base, CA data uh uh that istio knows about Now by default, uh istio istio-de. When it starts up, it's going to self-sign the CA and then use that for signing the workload certificates.

A

uh However, that's like using anything self-signed is not good right. uh um So what we can do is we can modify that and instead of having istio using the self-signed ca, we can provide our own CA. We want sto to use so there's a couple of options here um now I mentioned that base CA data right that SEO uses to push into those config Maps That Base CA data can either come in a form of a CA certs secret, that's uh created in the SEO system namespace.

A

So when the crd starts up, it's going to look for that CA search secret and if the secret is there, it's going to take the information from there and push that CA data from that secret to those config Maps. The other option is using mesh config. So when you're installing istio there's a global mesh config settings where you can provide the ca search directly and then what istio does I'll take those and push those to uh to the config Maps, hey I, do welcome hope everyone's doing well.

A

uh Thank you for saying hi as well um now, as for the external Cas, uh the two options that I'm going to talk about, both of them will use cert manager. First one is using the kubernetes CSR API, so there's a certificate, signing request, resource and kubernetes that can be used when requesting the certificates.

A

Now, in this case, when using kubernetes, CSR sdod is going to act as an RA, so ra stands for registration Authority that approves their requests as opposed to the ca, which is the certification Authority that signs the certificates, so istio will create and also approve the CSR, and then the cert manager will sign it. So cert manager is uh configured using an issuer to sign that CSR. Now these issuers uh insert manager represent Cas and there's a bunch of options that you have that you can pick from.

A

You can pick Vault, you can pick Google Cast, aw, AWS, private, CA and so on. The benefit here with this one, and the next option is that the private key is not present in um in the secrets you're, not storing in a secret. You shouldn't be storing it into secret anyway. Instead, it can be stored outside Google, AWS Vault.

A

You pick your uh your option, 80 back hope, you're doing well, uh and the last one uh the last option here that I'm going to mention and talk about- and this is what I'll show as well in the demo later on- uh is called istio CSR.

A

So this is another project that you run inside your cluster and, in this case you're using a Certificate request, resource from the cert manager to request certificates, and in this scenario, istio CSR is going to act as a registration, Authority and then cert manager is going to act as a CA. So the flow is very simple. A very, very similar.

A

um The difference here is that with St or CSR, the workloads in the cluster gonna talk to istio CSR, directly they're not going to be talking to a Cod anymore, all right. So this is just like an overview. There's no more text on the slides from this point on and I have only a couple of slides to explain the concepts and then I'll go into the demo. I figured having diagrams like this is, um makes it much easier to understand versus just me being in the command line and typing bunch of commands.

A

So I'll look at a scenario where I have a default installation of istio, which means I'm using a self-signed CA. So what istio is going to do is it will create those config Maps, the istioca root cert and the workloads are going to get issued. Those uh issued certificates they're signed by that self-signed uh CA.

A

So let's say I have that default default installation. Let's say I start with this one and then I say okay now, I want to create my own Routier and I want to create an intermediate certificate and I want to use that intermediate certificate. As my CA right. So what do we do? We use those intermediate certificates, I'm, calling them a here or blue whatever, whatever is easier for you, uh we take those certs, uh we store them in a secret called CA certs. We we would restart.

A

Istio istio is going to pick uh pick that up it's going to create the sdocia root certificate in a namespace and then any new uh workloads that are going to come up will have that new certificate or certificate signed by this Intermediate A.

A

Now this is all good in theory, however, if any of the existing workloads, A or B, the ones that are using the old self sign, CA will try to talk to this new service see. So this won't work because the certificates are from completely different Cas right.

A

Having being in this position, it means you'll, have downtime. You'll have failure, at least until you don't go and restart all the workloads. So all the existing workloads or old workload workloads pick up this new uh get the new certificate signed by this Intermediate A.

A

So, let's assume we got to that place magically all the workloads have the certificates issued by the same CA. So there are no more issues. Everything works fine now, let's say we want to introduce, for whatever reason we create a second intermediate CA.

A

Both of these A and B are rooted in the same route. Now we replace the ca certs with intermediates from B. This gets populated. This gets pushed through. Let's look at the same scenario: service C comes up. It has this new certificate from intermediate B. However, in this case the old, the workloads using certificates signed by the old um old intermediate CA, this will still work because they stem from the same root. So this works fine.

A

If you have a one root: multiple intermediates, if you're, swapping or switching out the intermediates, this is going to work. So what you'll do in this case is you can either wait for I, think it's 12 hours, the default rotation interval and then, after like 24 hours, all the existing workloads that used to have previous intermediates uh or certificates signed by previous intermediates will have the certificate signed by the new intermediate.

A

Now, let's complicate this even more, why why not? Right um now? Let's say we want to introduce a new route right. So, whatever reason we don't care, we don't want that old route anymore, we're going to introduce a new route and a new intermediate uh uh from that route, uh and we push that. What happens we end up in that same scenario, that I explained first right, service, C, talking to service b or service B talking to service C, it's not going to work because it's completely different routes. So how can we solve this now?

A

The solution is fairly straightforward. What you need to do is you create a combined root certificate, so the one that contains roots from B and C. However, your actual CA, cert uh uh and key will still be from that new route right, so you deploy that so your CS certs have that combined root certificate that gets pushed into istiocia root cert, which has two of them now right service, a talking to Service, uh uh C or service B talking to C works all good, because the istioca roots has combined root certs in there.

A

So once you're at this point, what you can do is you can just uh once everything moves over right moves to that new route, all the um workloads get issued new certificates.

A

Once that happens, you can just remove that uh combined root cert to only basically just to include that uh new Roots certificate and then your basically in a scenario like this, where you have a single intermediate and all the workloads are using that one and there's only one CA root cert in the cluster.

A

Now that will be the first demo that I'll show and then the second demo will be showing the cert manager and sdo CSR I will be doing a self-signed scenario here as well. uh However, I'm gonna use cert manager issuer to issue that self-signed CA cert all right, so the diagram, uh the diagram might look complex, but it actually it isn't that complex right.

A

So, as mentioned we're using CA where is insert manager, so what I do is I create this self-signed issuer first, so number one right and then I create a certificate resource uh and that certificate is gonna drop.

A

The certificate files into a secret called seoca.

A

Now, from the previous scenarios, we know uh that we need to provide the root CA to stod the way that we've done, that is by just creating the ca search secret or just letting the stod create a self-signed CA. However, this time we'll um create a secret that the istio CSR will mount in the step three and then we'll also create another issuer called seoca in step four, so this issue will point to the istioca secret and it's going to be referenced by the cert manager.

A

So whenever the workload certificate is needed or requested, cert manager will use this istioca issuer to create the new certificates. Now, at this point, we can um we can start istio CSR and that one uh is going to mount the root CA, so the one that we extracted right and then. Finally, we will install scod and we'll see in the stod settings we're going to turn off the ca server functionality and we're going to point setting a field called CA address. We're going to point that to istio CSR.

A

So now, whenever uh SEO agent uh and you work with the proxy comes up, the sto agent will talk to istio CSR for um for any uh uh any certificate needs right now. This self-signed uh setup is okay for testing right, but for like more serious deployment production deployments. Of course you you want to use something like Vault or Google or AWS to store your keys right instead of storing them in uh in kubernetes.

A

Now here's the exact same scenario as the previous slide, however: I replace that CA issuer with uh an issuer for Vault, so Vault issuer. So now all the root search, all the intermediates. All those secrets are basically stored in Vault and they are not stored in kubernetes anymore right, they're, not storing kubernetes Secrets, which is what we want right.

A

You don't want to be storing these secrets in the cluster so just like before we have search manager, we have istio CSR installed and this time we're using this fault issuer and this vault issuer is configured and knows how to connect to uh Vault and issue the certificates based on the ca that's stored there, seod just like before sets the ca address back to usdocsr.

A

So, whenever the workloads are requesting, uh the certificates they'll go to the istio CSR directly and then istio CSR in turn uses the Vault issuer to issue and use certificate to uh workloads when needed all right. So this was uh enough. Diagrams uh yeah I like these diagrams as well. It just it's so much easier to explain. Concepts like these, with uh with diagrams than just showing it in in code or command line. So let me switch to uh foreign.

A

To figure out the sharing here, if there are any questions, feel free to ask at any time and I'll answer them as I see them.

A

So let me let me figure out a good.

A

Let's try this one.

A

All right, I'll make this.

A

I find a good okay, I think this is fine right. If, uh if it's not mention it in the comments, if you don't see it, but it should be, should be all good. Everyone should be seeing my uh my terminal now all right, so what I'll do is I've created the uh the same scenario that I mentioned in um in the slides I basically have Intermediate A and intermediate B that are rooted in root.

A

One I have the intermediate C there's going to be rooted in a different route and then in the combined folder I just have that combined uh roots in that one file and I'll I'll show that in more details as um as I get to that point, so let me just make sure I don't have anything I, don't have anything installed, I'm using uh using kind. It's a single single node cluster, nothing special. What I'll do is I'll install I'll, install HDO.

A

Hopefully it doesn't take too long.

A

And then, once that's installed, we'll just do the typical labeling the default namespace and then I'll deploy sleep and HTTP bin workloads see wait for those to come up and what I'll do next once these come up, is I'll also uh deploy a peer authentication policy. That's going to set the mtls mode to strict.

A

uh So that way, since we're we're going to be working with certificates. That way we'll know if something breaks right away. uh Instead of like looking at the details of the certificates, it's just much easier all right, so we have both of them running just double check, yep and then I'm, just deploying the pre-authentication that sets the mtls mode to strict, as opposed to uh permissive and then to check that everything works. I will send a request from the sleep pod to the HTTP bin, so shouldn't be any issues here.

A

All works uh all good, all right, so we're using the self sign search here. Let's also look at the config map, so let me show you the config Maps notice, there's sdo CA root, cert istioca root cert uh in all the namespaces, except for the cube ones. Right. um Let's look at the details of that root, search file, so I'm just getting the config map getting that file specifically and then piping it through openssl and looking for the issuer.

A

So issuer is just clustered on local, but what we'll do next is I will create the ca search secret that uses the intermediate a certificate. So let's do that I clear the screen again so create secret generic CA certs in a SEO system and then I'm just pointing to cert keyword, search and search chain from intermediate a folder.

A

So once that's created, let me reboot I'll restart a Cod I think there's even a setting in stod that uh was added recently I, think that would automatically reload the certs.

A

So this I just see figure three starter: it's fairly fast yeah. So this was restarted and then let's look at the logs from sdod just so we can see what's happening there, I'll just grip for uh this guy yeah. So we can see in the logs of stod that we're using this Intermediate A uh from the exam that was rooted in the example root uh ca1. So we're not using the uh cell sign CA anymore. The STL self-sign CA anymore, so what happens now?

A

These two workloads are still using those previous search, so if I would try to make a request, this would still work right. There's not going to be any issues here because uh they're still using the same certificates rooted from the same route. However, if I delete, let's delete the HTTP band so as I delete this when it comes back up, it's gonna get those new get that new certificate signed by um the intermediate a so this time. If I try to send a request, clear the screen so same thing sending a request.

A

This time we get an error, so certificate verify failed. uh We're just going to expect it right, because they're rooted in different uh uh different routes. It should not work at all. So, let's just delete I'm going to delete the sleep.

A

Sleep picks up the new certificate as well.

A

This both of them are running and then sending a request just to make sure that all works. Now all everything works and both are on intermediate hey.

A

So the next thing I'll do is I will rotate the intermediates so from a we're going to go to B uh and both of them are routed in the same route. uh Yeah, so I could say: Auto reload, plug-in certs. Yes, exactly yeah you're right, that's the that's the setting that I was uh referring to so yeah, so it is, it is there all right, so what I'll do is I'll delete the existing CA search and then I'll create another one, but this time we'll use intermediate B.

A

So let's do that and because I didn't set that setting I'll just delete a Cod.

A

Oh, oh now, let's double check the logs again oops copy paste there yeah the logs from sdod and we'll see we're using intermediate B. Now, however, it's still the same route right, uh let's restart HTTP bin, so HTTP bin is going to pick up the new cert from intermediate B and once this one's restarted. If, uh if I make a request, it should work fine right because both of those are from the same route right. So there's no issues here. So let's, let's do that final!

A

One now I'll delete sleep as well, so both of them have same intermediates. So what I'm going to do now is I'll switch to intermediate C, but that one was signed by a different route right. So I'll do the same thing as before. The only difference is going to be that I'll combine the two root certs in uh in one file right, but everything else is going to be the same.

A

So this is the combined root search that I'll refer to, and this one is coming from um root a and the second one is coming from root: B. That's it it's all the difference and then we'll repeat the same, delete create thing, um create the CSS note that I'm still referencing intermediate B here and the only thing that I'm changing is I'm, just adding that extra root certificate from that second route into the combined one right. So no changes here. Theoretically, only the root Cas have changed. We added one.

A

So, let's do let's do this and take that root, cert out and just store it in the combined route, and then we can parse it and to make sure that we have two Roots right. We have the first root and then we have the second one, which is very aptly named as new root.

A

So this is just proves that we have the istio took that those combined um that combined cert and then distributed that into the istioca root. Cert.

A

What's the lead HTTP bin.

A

If we try again, they should work fine, there shouldn't be any nothing, no, nothing! Nothing should be affected here and then to lead to sleep again, uh no issues there, then, as the last step, now that we have both of those root search, there add the last step where we would do is we would switch over to intermediate C for any new workloads that come up we're still using the combined one, though right.

A

So let's do this, and now, if you look at the issuer in stod into logs you'll, see that it's intermediate C and it's the new root right and let's restart HTTP bin again HTTP bin is going to receive a certificate from the intermediate C, while the sleep is still using a certificate from intermediate B from a completely different route right. However, this is going to work because we have both of those root Cas in that config map right.

A

So the last step here that you would do is you would update the ca search again, uh but this time you would remove uh the combined root certificate. So it's going to only include the intermediate seized route and, of course you would do that once all of your um existing workloads would switch over to the new uh new root or new intermediate I guess: hey scheike,.

A

All right, so, let's go to the second demo and I'll switch clusters. Istio CSR.

A

So empty empty cluster don't have anything installed yet and this time I'll start with installing uh the shirt manager. So this is this. This demo is going to show how to use SEO CSR and have the search manager and istio CSR deal with the intermediates and root Cas and issuing the certificates all right. So first one is just going to be installing uh search manager.

A

Amazing there we go so we have the search manager, let's check the search manager, manager, namespace, you have the C injector web Hook and the search manager up and running, uh and next what I'll do is I'll create that certificate issuer. The sell signed this year right um that I shown in that diagram right. uh This is just so much easier to uh demo than just setting up vault and everything, but you should for sure, go and set that all up.

A

uh So let's do the this is how the issuer the sell sign issuer is going to look like just call this. Your the kind is this year there's also the cluster issuer kind, which applies to the whole cluster, but I. Think the assert manager docs are very prescriptive and are saying start with the issuer. First uh issuers are I, think bounded to a specific namespace right, so um anything that the issuer can issue to. Is this namespace in this case right?

A

Whereas if you have a cluster issuer, it's uh it's cluster-wide anyway, so self sign right, no settings we're just saying: hey this issuer is a sell, signed issuer. So let's create this one. Once we have the issue, we can go and request a certificate, so let me show you that.

A

All right so certificate kind from the cert manager, we're saying, create a certificate called istioca. It is a CA. It's valid for 30 days drop the files into istioca secret in the same cluster and then there's the private key setting, subject right and issue a reference, so we're just referencing that self-signed issuer I created 30 seconds ago. So let's do this so now. The result of this one is this: CA secret is Coca secret that was created with the certificate key and the ca.

A

So if you remember the diagram, what I'll do here is I'm just going to pull out uh uh the TLs cert and then we're going to use this as a CA right. So let's do that.

A

I'm, getting the secret and I'm just pulling out the TL TLS, that's CRT file decoding it into ca.tem and then I'm, creating another secret that will be used by istio CSR overall reference at ndsc or CSR to act as a root CA right, so creating a generic Secret istio Routier from the file that we just extracted from that uh secret that was created, but by that self-signed. Yes, sure, let's do this and now, finally, what we need to do is we have to create another issuer.

A

So now, we've just used certain manager to create a self-signed CA a CA certificate, but the last thing that we'll do here is we'll create another issuer. That's called the Coca, uh and this is what the SEO CSR will use to issue that certificates to or close right, and this one is pointing to that istioca secret. uh That was created. Let's do this now that we have all these secrets in place, and next year is in place. We can install uh SEO CSR right. So this is how this looks like.

A

What we're doing here is we're mounting that istio root, CA secret right and a volume called root. Ca and then we're just mounting that uh uh Secret in as the root CA file that we want to use right. So this is just telling it hey. This is the root CA that you use when you're issuing certificates all right. So this, let's just make sure that this is installed.

A

Cert manager.

A

Still running it's going to come up there, you go it's one of one, it's up and running, and now we can install this here all right, so this is going to be a longer.

A

Longer yaml here a little bit but uh oops I can scroll, wait what okay? Let me I just change the fonts a little bit and then I'll go back to a bigger fund.

A

So this is just the istio operator default profile and notice here that CA address. So what we're doing here is we're basically telling istio and consequently the the uh SEO agents that are running in in the proxies right to use this as their CA right when they're requesting the certificate- and this is just uh uh um istio CSR service- name- that we just deployed right.

A

We're also disabling the ca server functionality in the pilot and then there's an overlay that just mounts the um uh the secrets from the files, not secrets from the cert manager right so they're mounting the ca root, cert uh uh and the SEO dtls. So let's do this, so this is basically just installing SEO at this point, um all right and I can and go back to bigger fonts.

A

I this and this one should be good right.

A

Okay, all right, so we have istio installed, let's just double check it's up and running, and then we'll do the same thing as before: we'll label the namespace we'll deploy HTTP bin, we'll deploy sleep.

A

And let's wait for these to come up now, this time as they're coming up, the SEO agent is gonna, use that CA address it's not going to talk to you. Sdod for uh for the workload certificate instead is going to go to uh istio, CSR and istio. Csr is going to use that issuer uh to get a certificate that was signed by that self-sign CA that we've created right.

A

However, if you're using wall Vault or or like an external see like a proper external CA, um you would have that single issuer right and then all your certificates, all your secrets would be stored outside of the cluster somewhere, but the flow would be exactly the same right.

A

So when the club, when the workloads would come up, they'd talk to istia, CSR, istio CSR is then using the uh the issuer to go and talk to and Grant their certificates for um for your workloads, all right, so we have those up and running and what I'll do is I'll just use the proxy config secret command to uh uh to take out the certificate to get the certificate, basically decode it into the workload search File.

A

So, let's do this uh and I should have the workload search File here, yes, stair and then you just use open, SSL uh to just decode it and then in the output. If I go out here, you'll see the organization here is the cert manager. The common name was the SEO uh CA that was set there and we have the spiffy here as well. Right, nothing has changed. Nothing has changed from the istio's perspective right. This is a valid certificate, valid workload certificate um that workloads can use right.

A

It's theoretically doesn't care as long as it has this spiffy spiffy ID here and it's a valid document all right. So that was that was my demo. I. Think we're like 40 minutes in perfect I was perfect on timing again. um If anyone has any questions in regards to this, I'll gladly try and answer them, uh if not I'm, just looking to post a link to um the previous hoot, that I did a couple of weeks back where I was talking about.

A

um I was talking about spire and that one has more of a um more details on like how spiffy Inspire work uh in general, and you should definitely I guess. I can I can show it here, but it's not going to help much No One's Gonna copy paste it from here. But if you go to our Channel, you should be able to find it there under live all right if they're, uh if there are no questions thanks everyone for joining uh hope, this was useful, um make sure to like And subscribe.

A

If you have any questions you can post them in uh under the video or if you have any ideas for or wishes for the topics that we should do next, you can do the same. Otherwise, thanks again and I'll see you all next time.