Istio IstioCon 2021, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: I want to sketch a mesh for you

Description

#IstioCon2021

Presented at IstioCon 2021 by Christian Posta.

Virtual conference presentations lack the dynamic and expressive feeling of a live talk in so many dimensions, and explaining complex concepts can be difficult. Even in person, one of the best ways to convey problems, solutions, and architecture discussions is through diagramming and white boarding. As one of the authors of Istio in Action for Manning Publishers, I’ve gone through many refinements of diagrams to help explain Istio. In this talk, we’ll use live diagramming, architecture sketches, demos and no slides, to illustrate how best to get started with Istio and iteratively adopt it into production. We will cover concepts like sidecar injection, the power of gateways, best practices for tying in PKI, and DNS proxying as we slowly add new workloads to a mesh.

A

I'm going to run a presentation today, that's a little bit different than uh than others. I've done in the past, where I'm going to make it. I don't want to say interactive, because I can't get feedback from you all right now, but a little bit more dynamic, where I'm going to talk about various uh concepts related to adopting istio in production and we're going to use a diagramming or our architectural, uh drawing type of approach to this along with live demos.

A

So, let's get started and see how it goes so, my name is: uh is christian posta, uh I'm the global field cto here at solo, dot, io and at uh at solo we're working on building uh technology on onboard proxy and istio and webassembly to connect your applications manage any of those apis that you might be running inside of those uh those deployments um and then run those on any any cluster number of clusters, uh clouds or or whatever, and simplify the whole experience.

A

I'm currently writing istio in action, which is a book from manning publishers uh and and what you'll see is the the content from this talk is slow. Some of it's, you know taken from the the book that I'm working on some of it's taken from various workshops that I've that I've done, including a new workshop that we're going to run for the first time on march 9th that is specifically oriented around uh istio day two operations and we'll go into a lot of hands on detail.

A

It's it's free, uh I'll, lead it and uh and you'll you'll get a little bit more experience. So the things that we have learned in the field working with customers around the world with uh with istio. So let's get uh let let's get going here. There will be a live demos and so on so.

B

Let's uh see, if I can I eject this there we go all right.

A

So I'm going to start kind of at the high level, but we're going to quickly drop into uh various uh various details of what we're going to cover today, which include things like deploying the control plane in a way that makes it operatable or operable.

A

You know so, following some of the default install guides on istio will get you a good poc environment. But what does it look like to actually install it? uh Deploy workloads create gateways, uh secure the mesh with um your you know.

A

You know it still provides this mutual tls capability, but there are some inherent things that you might want to explore or change about how it manages the the the ca signing, certificates and so on, and then, lastly, we'll take a look at expanding out to vms and using some of the capability that comes in istio 1.81.9 around dns. So let's, uh let's get to it and let's see the first thing that I want to point out is uh how a service mesh is.

A

You know, used to help solve some of the same challenges that we've seen in the past. So it's not the the problems that we're solving with the service mesh are not entirely new they're, just reimagined right, so we're deploying into cloud infrastructure. That's highly dynamic and workloads are coming and going and scaling, and the network is unreliable.

A

We know that, but even more unreliable and and where we've tried to solve this in the past was using things like api management or esps and so forth, but even in the api management case, what we saw was you know. Traffic would flow into the system as we might expect from the edge.

A

But what we saw is that workloads inside the organization wanted to take advantage of the same capabilities like routing and security and some of the telemetry collection and to talk to another service. It would actually call back into the gateway and continue to do that.

A

You know to to the extent that this api management piece, which wasn't really built for cloud and dynamic environments, uh became a big bottleneck right, and so the alternative is to instead of having these these large centralized, uh uh you know things through which all the traffic would be tried to force through.

A

What we want to try to do is decentralize those operations and those enforcement points and telemetry collection points and so forth, and run that across multiple different clusters, multiple different clouds, deployment deployment targets and so forth, where you might still have you know something acting as an api gateway or you might push api gateway functionality down to the clusters to get a completely decentralized view.

A

And now, when the traffic comes in, you know into one of the clusters, the workloads are talking directly to each other, and policy is being applied in line with the request and not forced out through the uh you know, back up through the gateway like you may have seen in in the past, all right. So the the service mesh provides of a piece of the puzzle to help decentralize policy enforcement.

A

Telemetry collection security around uh around these apis and uh and do that in a way that's fit for for cloud-based deployments like those you would see in containers and in kubernetes, so you say: okay! Well, that sounds good. Now I do want a service mesh.

A

Maybe I do have this problem where we have this, what large scale set of heterogeneous applications written in different programming languages, and we want to provide a platform service to the users of the of our system that solves for these types of problems uniformly, regardless of what language or framework that you're using? So what do you? Do you look at the various different uh istio implementation or service mesh implementations um you find you know. Istio is one of the most dominant service meshes. That's actually deployed right now, but there's others and you can go.

A

Look those look at those, uh but but what the service mesh does is it puts these little uh proxies these little agents with your individual services right? So you might have. Let's see if I can do this uh right, we we we put the envoy, proxies or or a proxy in this case. If we're talking about istio, it is envoy proxy.

A

Next to the workloads which there are the enforcement points, that's where we can do things like telemetry collection or um security enforcement and so on, but something needs to needs to manage these envoy proxies right, so istio actually is made up of two distinct sections, all right: the the control plane which you install and is what the operators and users of a mesh use. You write your your your configs.

A

To the control plane, the control plane then synthesizes, those configs into config, that that is native to the proxy right, and so what we're going to see is in in in this first demo is how to get started with the service mesh by installing the control plane. First, all right we're.

B

A

We're not gonna disturb any of our workloads, we're gonna, install the control plane and ideally install it from uh from a day two, let's say perspective, knowing that you're gonna have to operate and maintain this thing into the next versions and upgrade it and patch it and so forth. So, let's, uh let's really quickly take a look at what that looks like now. You know if you go to the istio website.

A

The the way you install istios is is pretty simple. It's still cto install and that'll install the default profile, and- and you know that that should be good enough to get you up and running. But what we're going to do here is uh take a slightly different approach: we're going to install the control plane. First, so istio's components are you know, mix and match and interchangeable. You don't have to adopt all of the the components at once.

A

You can start small and start to iteratively uh grow into the rest of the capabilities that you need and that's what we're going to do here. We're going to use this particular configuration file and we're going to create the seo system namespace and then we're going to install the istio control plane. Now.

A

One thing you might notice here is I'm using a a config to the install called revision, so a revision allows us to uh version our control, planes and istio components in general separately from each other, because you know, if you're an operator of a system, a service mesh is a critical piece to that. uh To that um you know how the system operates. You know you're gonna, be patching it and and uh versioning and upgrading and so forth. So let's start from the beginning.

A

With that in mind, now one of the problems with this, this particular functionality is still being ironed out upstream and I'll. Give you uh I'll give you an update on on something that will change in a second, but one thing that we'll see is: if we take a look at the logs for our uh istio control plane, which we see here so in our seo system namespace, we have the control plane.

A

But if we take a look at the logs, there's one little thing that we're just unfortunately going to have to work around the the revisions capabilities uh needs a little love like I said, and but that's okay, we're just gonna uh we'll work around it because we know we need this capability and uh we'll just create the istio system service, which will then solve this problem. So let's go ahead and and take a look at that problem should be gone now we should see there we go things. Are things.

B

Are ready to be.

A

Pushed we're not seeing the the validation controller error anymore, um and so this.

B

A

This is an important first step, so if we take a look at our at our namespaces here, uh we have istio in action right. We have our apps running here. If you're familiar with istio, you would see would typically see two out of two. If there's an istio cycle practice, there's not.

B

A

Istio side car proxy here um and and like I said, if we look into this co system, we have the control plane up and running now. Let's come back here see if I can drag this back.

A

So I mentioned the the control plane, part and the uh you know we installed using revisions and that will allow us to eventually do upgrades canary based upgrades where um you know some.

A

Some workloads will be managed by one control plane and some workloads would be managed by a different control plane and then you're in control of slowly rolling that traffic over from one control plane to the other, all right so there's this revisions feature and the idea of this running side by side is intended to give you the most control over how you do your upgrade now that same approach is also used at a different level to introduce workloads into our service mesh all right. So if I come over here, let's see.

A

What we're going to do is we see we have our workloads, we're already running in kubernetes.

A

What we want to do is slowly introduce some of these workloads into the mesh and then start take advantage of things like telemetry collection or uh security, or you know these types of things so, first of all, let's make sure that our our applications are working correctly. So if I go into one of the pods and start start calling them, we see we get a, we get responses that and then http 200. So that's good.

A

Now what we're going to do is we're going to label the namespace that has these workloads with the correct revision for which we want the the the right control plane to inject the right proxy next. What we're going to do is we're going to deploy a canary version of our app and what this is going to do is create a new version of our app that should have the sidecar proxy.

A

We can see it running here and from here. We want to go back and verify all right. We have some web api deployed with the data plane, some without the data plan. Let's go make sure everything is okay. If it's not, then let's roll that thing back, let's get that out of there right.

A

You can verify that the traffic continues to work and and so forth and then and then introduce the data plane into the original deployments or or do you know, a rolling restart of the original deployment. Now this is one approach right. That's not the only way you can introduce the data plane into your into your service.

A

You can use kubernetes canary rollouts.

A

You can use istio's manual, sidecar injection, so there's a few different approaches here, but the idea is slowly introduce the proxy into this service, and so now we restarted, but once you're confident that it's working then roll the rest of them out and then you can either pull the canary out or you know, get rid of the original deployment and scale out the canary.

A

um But so this is one way to iteratively, introduce the the data plane and do it do so in a way that's safe, you're able to recover for from it roll it back and then, if everything looks good, then let's just speed it up and uh restart all of the workloads. Here this deal action.

A

We can see that uh now they're all being rolled right and it's because we labeled the namespace with the correct revision that the istio sidecar injector knows, which version of the data plane to inject all right. So that's all good.

A

Let's, uh let's come back here because now now we got to think about how we're going to expose these uh these applications, how we're going to allow traffic from outside of the mesh or outside of the the system into the into the mesh.

A

So if we take a look at uh uh one of these examples, so we have approximately this deployment right now right we have the control plane, it's managing the the data planes services when they communicate with each other they're going through this decentralized envoy architecture, and so what we're going to do now is we're going to add we're going to add some way for traffic to come in from the outside and into into the mesh.

A

Somehow, what we're going to do is we're going to install a ingress gateway, istio ingress gateway that is and the way we do that and you'll notice is we're going to again decouple and separate that out from the rest of our istio installation. When we install istio, we just installed the control plane, part now for reasons of upgrade and not having things too tightly coupled we want to separate how we upgrade the gateway as well.

A

So let's take a look at doing.

A

That so we're going to use the same issue operator, configs or similar istio operator configs for installing the gateways and we're going to start with a empty base profile in the previous, when we saw the control plane, we said minimal so that we can add and configure- and do you know, control the the the control plane configuration as much as we can here we're going to start with empty and then we're going to add the istio ingress gateway.

A

Now why this now this will be installed separately, as you can tell from the control plane. So if I started doing control plane upgrades, I'm not going to try to take down I'm not going to take down the gateway, also which might be serving live traffic from from clients outside the mesh all right, so we want to treat those completely independent.

A

So if we look at the control plane namespace, we don't see. We just see the the single control plane. Instance there now we're going to install the ingress gateway, we're also going to give it a revision so that later on down the path, when we start doing upgrades, we create a revision of 1.9. Let's say we can slowly control how traffic gets shifted from one version to the other.

A

So let's go ahead and install that yep.

B

Sorry is it possible for you to make the the font a little a little bit bigger on the terminal, yeah.

A

Yeah. Thank you. Thank you.

A

Hopefully, that is a little bit better.

A

So now, if we take a look at the components running in the istio system, namespace, we see that we have our control plane running as well as our ingress gateway running all right. Let's, I can make it just a tad bigger.

B

A

Way, no, thank you. Okay, awesome yep. um Now we can see that the service has also been exposed on a uh on a cloud load balancer here and then what we're gonna do is we're gonna expose our application all right, so we're just going to look at uh simple virtual service uh and and gateway resource, so for those of you that are familiar with istio.

A

These are these are these are how we can expose ports, open ports and expose service routing for traffic to come into the mesh into the system here. So let's just go ahead and apply that.

A

And now we should be able to call our service by calling that ip address that went a little fast by uh by calling that ip address, and now the traffic hits the ingress gateway. The ingress gateway knows how to route it to then the next workloads now another piece that uh you can do so the gateway, the gateway is good at allowing traffic in there's going to be typically um some some requirements about what traffic is allowed to be let in and which traffic is not allowed to be.

A

Let in istio has has an approach for for doing some of that, at least, which is uh writing, authentication, I'm sorry, authorization, policies and and other policies about what traffic can should be rejected. If it doesn't contain uh specific, um let's say tokens or users, or you know these types of things. So what we want to do is if the request coming in contains a jot and is and is approved by the you know.

A

The issuer of that token is is approved in our configuration, then we'll let the traffic in otherwise we will block it if it doesn't have an approved jot or if it doesn't have a jaw at all all right. So that's one example.

A

We can do other things like apply cores policies so that, if we're using browser-based applications, we can, you know, specify the correct course headers and, um and so we can, uh we can do uh tls termination or pass through through through the gateways.

A

uh We can expose multiple domains and multiple certificates on the uh on the on the gateway. So.

B

The gateway is.

A

Pretty functional for a ingress gateway, but if you need something more more powerful, more capable, then that's, probably where you'll look at something like a glue edge, or you know some other api gateway, ideally based on envoy, in my opinion.

A

So let's take a look at what that might that might look like so in istio we can specify a request, authentication and a request. Authentication can basically configures seo to say, hey watch out for jot tokens and we will restrict access to particular services that get exposed on the gateway to those that have valid jot tokens.

A

So let's apply that in conjunction with this config. We also need to tell its deal how to how to allow certain users- and in this case we're going to allow jot tokens that have been issued by a certain provider and we'll apply it we'll apply this policy to the to the ingress gateway.

A

So I applaud apply this authorization policy. This will so in this case, if you're, if you don't have a jot, it'll fail the request, if you do have a jot and it's valid it'll, allow it through if it's an invalid chat and so on, it'll fail um just real quick.

A

We have uh a number of sessions and by we I mean uh some of those folks at solo. We have the one that I'm doing here uh indeed and yuval will be doing a really interesting one about debugging webassembly when you inject web assembly into your envoy proxy for in seo, a customer of ours is doing a really good presentation on their deployment of istio in a really large scale, and then wednesday, we'll be doing a deep dive.

A

Lawrence from my team will be doing a deep dive on istio um authorization policies so that this is one of those kind of more complicated areas of istio security, api and and we'll do we'll do a deep dive and if you're interested in multi-cluster type stuff, I think doing a debugging of uh of failures and issues across multi-cluster, sdo and so on. All of this, based on on working with our production deployed customers um as we support istio for them all right. So we'll give that a few seconds.

A

Let's try to call that service without a token, and we see that it is indeed forbidden and let's actually pass in a a valid token.

A

This might take a little while, because the token is kind of large, but we should we should see. Then the request go to the gateway. Okay, we'll evaluate the jot validate. It then run its uh role-based access control against it, and we should see once this thing's done typing. So this.

B

A

Demo- and I just have a script that types this stuff for me, otherwise, as you can see, it would take a little while- and we see the request went through this time, so the gateway is properly enforcing the authentication authorization pieces here. Okay, that's all good! Now, if we take a look at the uh let's go in action, if we take a look at this, we see our workloads, they have the uh sidecar deployed the seo sidecar and including from right here.

A

So as traffic comes in from the gateway, uh let's just use that last example, the jot token comes, trap comes in gateway, does its validation and authorizations and then passes the traffic off to a different service which may then in turn also call other services and and so on, all of this traffic.

A

All of this traffic here is or can be, configured to be encrypted and done so with with mutual tls.

A

Now istio is the one managing the certificates that go into these.

A

These different uh workloads and istio is acting as the ca and the workloads are basically sending csrs to istio. The ca is signing them and then mounting the mounting. The cert directly into the workloads and in a default deployment istio. What it's going to do is create its uh its own. Let's, let's call this a self-signed certificate and we'll use that as the ca, but there are other oops might not be able to see that there. um There are other options, however, so in istio 1.8 and continuing to evolve in 1.9.

A

What we, what we have is the is the option for um the the same sort of model right, the the workload will come up. The workload will send a csr at this point. Istio will validate that.

A

Yes, this is indeed a workload that that can be run in in the mesh and deserves a certificate, and then it will create a kubernetes csr and then it will flow through the mechanisms that kubernetes allows right now, where you can have various issuers and an approval process and so forth that that then allows you to to manage the certificate creation workload, certificate creation outside of istio itself.

A

So that's one one approach using the the kubernetes csr uh another one is: it follows a very similar model, but instead of calling out to kubernetes it'll call out to a different component that actually speaks the istio ca apis right. So, instead of going there it'll, it would go to a different component whose entire job is to do this. Do this type of thing now these are all approaches that are available in the in the community. Continuing to improve one issue that we've run into a major issue.

A

Security issue with uh with our customers is the the certificates for either in the default case, or even if you mount it in the the certs here or even the workload search that get that get created here.

A

They're stored as kubernetes secrets- and that includes the certificates and and the keys all right. So what we need is an approach to you know issuing safely issuing a signing certificate, but not store this in kubernetes secrets, where theoretically people can can access them, and so one of the one of the models that we, let's, let's use.

A

There we go: uh let's use a real example where um a customer of ours actually a handful. This is a common pattern.

A

Customer virus wanted to store its root, ca and and keys, and so forth into uh some sort of hsm uh where, where it would be stored, either offline or not available, but then vault would be able to issue intermediate signing certificates that a you know, a different another component could use to to create certs, and so the way this works is we have a little side car that gets deployed with the control plane, and what this does is on startup.

A

It will reach out to vault and say: hey, give me an intermediate signing certificate that I can store locally as well as keys and, and I will and then and then the control plane will continue to act as the ca uh as csrs come in. You know, it'll it'll, just work uh the same way that it does today, but the the key point here being that this little sidecar writes the the response. So the response is a key and assert, writes this into uh memory all right for this particular for this particular pod.

A

So now the intermediate signing certificates are available.

A

And the the normal workload certificate provisioning happens as as you would expect in istio, but everything is stored, safe either offline or uh in in memory, and you don't have any secrets leaked into into std or into the kubernetes cluster or anything like that. Why don't we take a quick look at that.

A

And I believe we're running out of out of time here. So the first thing that we're going to see is we have.

A

If we can, we should see come on come on, we see, we have vault running and vault has it's it's, it's storing it's uh the root, ca and and keys, and all that the first.

B

Thing we're going to do is look.

A

That istio, when we started it, created its own self-sign certificate here and if we take a look at the the the public ca the root ca. We see that this was issued by that self-signed cert.

A

But what we want to do is we want to install that little sidecar to handle the workflow of getting the keys and the and the certs for istio to use. So, first of all we're going to delete that secret. We don't want to use that secret anymore and then we're going to take a look at how to install that that sidecar. So the first thing we'll see is that we're going to store it locally in memory, we're not going to write anything to disk and we're not going to store anything in um in secrets.

A

And then the next thing that we will see is that we'll store the um that in memory disk at c ca certs, so that when we have the let's find the init container.

A

Init container right here: what will what we'll do is that will handle the orchestration of the vault? um uh You know procuring those intermediate certs and then and then storing them in memory uh or sorting to this location in the pod, but in in memory. Ultimately, so let's apply this.

A

Okay, now we should go, get the positive mystery system and see that indeed, we are up and running.

A

And then, lastly, we'll take a look at that config map again the root ca and we should see cross your fingers that it has been issued by fault by the pki that we installed outside of istio and now all of the workload certs and so forth will be provisioned with this or signed with this intermediate signing certificate.

A

So it appears that I am running short on time, uh but I guess I guess wrapping up so this. This has been a, and the next step I was going to show was the vm integration and and dns, but I don't want to rush it and leave time time.

A

Maybe for a couple questions, um but I guess the point is this: sort of stuff is pulled from a lot of pain and heartache working with folks working with our customers in production, running histo, istio at very large scale, and so this is something that we have a lot of expertise and we can help you with.

A

We offer supported istio, uh it's still with the long term, support for n minus three. uh We offer fips builds of istio typical enterprise, slas sub, 1, slas, and so on.

A

This particular content was, as I said, based on the learnings from the field which we're going to go into much deeper detail, hands on with our with our workshop.

A

You can see some of the labs from the workshop getting familiar with is with envoy not from a hello world perspective, but you know really understanding envoy envoy is the key to the service mesh technology and then connecting up to and integrating with your existing observability systems uh continuing to onboard services into your mesh, and then, of course, you know how do you figure out when things start to go wrong? So we'll take a look at this.

A

This is part one of this workshop series um and the other one will go into even more detail like doing things like ci, cd and so on, but.

B

A

Out uh check out what we're doing at solo- and uh you know continue to invest in in istio and using istio istio is the most dominant most powerful service mesh so uh definitely continue to continue on that journey. So uh with that I'll leave a minute or so.

A