solo.io Istio 1.6, 13 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio 1.6 Deep Dive Certificate Rotation (Part 1)

Description

In this 4-part series, we look at how to rotate Istio CA certs without interruption.

In this part we look at using our own CA and what happens when you naively rotate them into Istiod

About us https://solo.io
Manage multi-cluster Istio with Service Mesh Hub https://solo.io/products/service-mesh-hub/
Istio 1.6 https://istio.io/latest/news/releases/1.6.x/announcing-1.6/
Questions? https://slack.solo.io

A

Again, thank you for stopping by our youtube channel and in this series of short videos, we'll walk through s, Tio's certificate authority, handling of certificates, rotation and minimizing downtime. For these types of events, my name is Christian Costa I'm. Writing the sto in action book I've been contributing to and involved with, the sto community for the last three years or so for three and a half and I now worked for solo dot IO.

A

So in this demo, what we're gonna look at is rotating or reinitializing. It's do with a user provided route, see a chain and we're going to do this, because by default, this C will create its own CA and we'll use that to sign the workloads. Typically, organizations already have PKI and their own routes, and they want to derive the workflow certificates from that, so that other parts of the organization can share in this in this trust model.

A

So to do that, what we're going to do is look at the book. Look at the cert change that I've already set up for the duration of these videos. We have two routes and that will become apparent in the third and fourth videos, but not yet we're going to focus on a root certificate called root. A and this root has been used to sign an intermediate certificate, intermediate roubaix, and it's this intermediate that we will use to sign further certificates, including s Tio's workload, leaf certificates that get used for mutual TLS and identity.

A

All right, we also have a second intermediate. Ca has also been signed by root a and then we have a totally different intermediate CA signed by root, B, and so we'll see this in the the progression of the videos here. So the first thing that we're going to do is we're going to demo creating our own, or you know, telling SEO to use our own certificates. Not its own self signed is to your generated once the first thing that we're going to do is we're going to take a look at this.

A

Is the self signed user or a steam generated root? Ca that comes with this. Do let's look at connecting to an HTTP pin service from our from a sleep pot, so from sleep? A log in call ET pin everything works. Fine. We can see through the headers that there are search, Ayn's, being propagated and indeed for being used for mutual TLS.

A

Let's see exactly what that certificate chain is and what root is being used. So we come back here and look at the chain, we'll see a leaf node leaf certificate. We also see the root. The root certificate here in that and in sty, should see that matches what we expected.

A

So everything is functionally fine here. Let's update is Gio to use a CA that could be used to sign these more phone certificates that derive from a root that we own. Not that was generated by is diem, so the first thing we're gonna do is create a secret called CA certs CA certs is the magic secret that sto will look for at startup time and startup time only at least at this point, and if it exists we'll use that to to generate its its certificates for the rest of the system.

A

We can see we are going to provide the intermediate that has been signed by route a so we created a secret. Let's restart s T of D: let's go to canines.

A

Let's test your system, we should see that had been restarted.

A

And it looks fine now what we're going to do is we're going to take a look at the logs in HD of D and make sure that indeed we are using our own CA certificate that we just created.

A

Alternatively, you would see the actual root CA spit that gets spit out here in the logs if it was automatically generated by HD, but we don't see that so that's all very good. Now, if we take a look at the proxies themselves, we start to see some trouble, because the workloads originally were provisioned with the sto generated route and then signed by that route. Now, we've introduced that new one and so a tooling that we would expect to show and connect to the different components in sto, not working.

A

Furthermore, what happens between the services? Let's say we delete the HTTP bin pod and we're gonna do that, because we want the HTTP bin pond to pick up the new certificates and signed by the new route that we just introduced.

A

Now, if we go into Oh, actually we can sit. We can take a look at the logs here. So if we go into sleep, let's go into the sleep pod ticket, the look at the logs now, if we call HTTP bin from sleep, will notice and it doesn't work, that's because the routes have changed, and this is a problem if you're expecting a mutual TLS in the system and you change the routes, you will see. That's connections, don't work anymore.

A

If we come back into our logs here and try to find that one specific connection that failed, we should see that.

A

It failed because the certificate could not be verified the roots on either these two workloads, or rather the certificates that have been used, are anchored in different roots and they don't match up and they're, not trusted so step number. One here is: let's: let's go ahead and get everything on: let's restart it so we're starting to sleep pod, and we should see that come up successfully and now, when we make the call between sleep and h2b bin, we should all be on the new root CA that we introduced and the call should succeed.

A

So this leads to the obvious point of all two points: if you change the roots, let's actually run this if you change the roots and they don't match up and the different trust stores don't match up for the different workloads you're going to have down time. A second thing: if you live in an environment where you have existing PK I use this deal with that PK I, don't use the self-signed certificates in the next video. We're gonna take a look at what it.

A

What happens when you start rotating these certs, like we just did between the workloads and then we're gonna, show you how to do this in any way that doesn't disrupt traffic, even if you have to change the root CA itself. So in this demo we showed you change the root. Ca you're gonna have interrupted traffic, but we'll look at which is for not not having that interrupted traffic stay tuned to the next video.