solo.io Hoot Livestream, 12 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Episode 11: Advanced Istio Configuration with Envoy CRDs

Description

Join Scott Weiss as he dives into Istio service mesh and how to do advanced configuration with the Envoy Proxy CRD.

About us https://www.solo.io

Questions? https://slack.solo.io

Code Samples: https://github.com/solo-io/hoot
Suggest a topic to cover here: https://github.com/solo-io/hoot/issues/new?title=episode+suggestion:

A

uh Hello, everybody um welcome to this episode of the hoot. um Second, I'm gonna show you what my stream is set up. Okay, um thank you thanks for joining me uh so today. um So just to give a shout out. This is the hoot brought to you by solo. I o um uh we go over technical, deep dives um in the service mesh space we go into envoy.

A

We have a great series on that by yuval kohavi, I'm going to be doing a deep dive today into istio and some of the advanced functionality that you can get out of istio using their envoy filter. Crd.

A

So just remember to like subscribe hit the notification bell, so you can follow all of our great content. Okay, let's begin so. Istio provides a lot of service mesh functionality out of the box.

A

If you are already familiar with istio, you may know this. If not, some of the features that you can get are both on the ingress, as well as between services in kubernetes clusters.

A

It provides service, discovery and load balancing that allows you to automatically route between groups of services.

A

It's something that kubernetes also provides by itself, but istio provides more advanced load balancing features, for example, selecting the algorithm based on which the load balancing takes place.

A

It also provides secure, mtls service to service communication out of the box, so every service that's running inside of your mesh, is injected with a side car the side car forwards, all traffic that goes through it and that way the originating pod for request will have its traffic encrypted on the way out, and then the receiving sidecar on the destination pod will decrypt the traffic.

A

So the services can be written to talk in plain text, but the communication itself will happen in mtls when it's over the wire, which is uh both client and server, are using a tls to authenticate and encrypt traffic.

A

Istio provides a number of sophisticated traffic control features, so you can shift the way traffic flows through your mesh. You can use this to achieve a b testing canary deployments. uh You can forward traffic to a particular subset of a ver of a service. um I I don't want to get too deep into this. These are all native istio features and there are there's really a lot here.

A

You can also control using policy uh security uh who is allowed to talk to who and that's also enforced by the proxy, as well as getting a lot of metrics out of the box from istio. So you can get a lot of logs and metrics based on the traffic. That's happening inside of your service mesh.

A

So all these things can be accessed using or or are enabled are configurable using istio configuration crds. They have a service entry for adding service discovery endpoints into the mesh destination. Rules. Allow you to configure load balancing, they also allow you to configure options for mtls traffic control typically happens with the virtual service crd.

A

This policy stuff is happening inside of authorization policy and then um traffic metrics are are built into the proxy and they'll they'll be propagated to prometheus or stackdriver there. There are a number of options there, so that's great when you want to access when istio gives you a lot of functionality already, but what happens when we want to uh extend functionality?

A

We want to do additional things that istio doesn't necessarily leverage, but envoy does that if we were to build our own service mesh, we could leverage this functionality ourselves, but then we'd have to reinvent the wheel. So what what does istio give us to actually allow us to do it um so, for example, cross-cluster networking um adding external auth and rate limiting servers uh performing failover between services?

A

So if you you see of a list of services, you have a priority order so that, if one fails, you want traffic to fail over to another service, deploying wason filters, which is a a feature in envoy that allows you to write your own custom filters. Filters are things that can act on or observe requests as they pass in and out a request and response, as it goes through the proxy.

A

We can actually mutate, reject and record these um requests and response uh and and the wassum gives you a dynamic way to do it without having to build your own ins, your own image for envoy, um and then there are a lot of envoy functionality that are exposed in envoy configuration, but a lot of it's pretty low level and is not exposed directly by istio.

A

But these are things that are are of a lot of interest to various users. Here at solo, we build tooling, that's basically extending the istio mesh. We are essentially providing a an enterprise product version of istio mesh, and so we leveraged this, what's called the envoy filter crd, to provide a lot of this additional functionality and our our product automates, this stuff away, glue mesh.

A

So um as a user, you would not you would you would actually have a crd directly related to these domains, but if you're doing it yourself, which we're going to get into a little bit today how to do that, uh you would you would leverage this envoy filter crd, we'll take a look at how that works.

A

So, just to get a little bit into deeper. What actually is this envoy filter crd? So it's kind of a swiss army knife of customizing istio configuration that gets propagated through to envoy so where the istio crds are, do not provide enough of a service area to control the envoy config.

A

You can use the envoy filter to apply opaque patches. It's it's got a semantic similar to json patch, a little bit where you basically give a matcher to say. I want to match a specific part of the. The translated is envoy configuration and I want to merge or insert some additional raw envoy configuration kind of bypass the istio translation loop as a whole, so just to get a clear understanding of that, istio can be understood as a translation engine.

A

It takes these high-level crds that are user-facing or maybe automated by a controller like a virtual service or a destination rule. These things are crds hdl parses these in in its internal translation logic and then spits out things like uh envoy clusters, envoy virtual hosts, which are the configuration resources that go directly to envoy. That is served by the control plane um and again, I won't get too deep into the control plane versus data plane architecture um just suffice it to say that the way envoy works today.

A

It's a reverse proxy, uh similar to nginx he proxy, but has a very large number of features, and one of the things that sets it apart is the way it's designed to receive its configuration, which is it connects, rather than being given a configuration file directly on disk, which is possible to do with envoy. But the primary use case, for it is to run it and connect it to a live server that is dynamically and continuously serving configuration.

A

It's extremely helpful for things like service discovery, which can change very frequently across a large number of envoy instances.

A

Proxy instances need to be updated with endpoints and so having a a server that can stream those results down to the proxies ones, that being more efficient and much easier to manage than trying to use configuration files to do it so anyway, what istio is is actually that control plane and it also it's its configuration is being driven by these high-level crds, but those high-level crds don't necessarily give us uh enough control for some of these use cases like the ones I pointed out so the workflow to use the envoy filter is to give istio the normal set of configuration that you want to give it, and then, let's say, there's something that you want to patch.

A

You give it an envoy filter, crd that targets the output configuration and this it does require a bit of playing around with istio to know exactly how istio is going to translate its configuration and so that we can create patches that that successfully line up with what istio is producing.

A

That's. Why again, if you're not wanting to get very hands-on, if you're, not wanting to potentially break things, and that's one of one of the warnings about envoy filter crd- is that it's a break glass configuration so you can go and configure things directly that go to envoy and istio is unable to validate them or prevent them from potentially causing outages or issues. So when you modify these, you really want to know what you're doing or you want to use a product.

A

That's built on top of it, like blue mesh, that abstracts that paint away from you.

A

So just for an example, I have here a virtual service on the left.

A

This is some reviews, virtual service, and what it's doing is it's routing all traffic that matches this host reviews uh book info, blah blah blah, and it's going to intercept that traffic and make sure it's all routed to this v2 subset. um And so we can see that that gets translated here. uh So on the right we have the android configuration, and uh this is just a subset of the configuration it gets translated to, but we're showing the uh the interesting piece here, which is uh we get a route.

A

This is inside of um a virtual host, which is um the envoy equivalent more or less to a virtual service. It's a mapping of a host to a set of routes. So here we can see we have a route that matches prefix and routes to a cluster that contains our v2 subset, uh okay, cool. But let's say I want to add some some advanced functionality. For example, let's say I have a custom auth server and I want to use to authenticate all of my services in my mesh.

A

I want all my services to be authenticated through this um custom, auth plugin, and just so you, if anyone is not aware envoy, has a filter that can be enabled that says, send all requests that pass through this filter to an external server for authentication.

A

The server receives the request and it can reply whether, with a response, whether the request should be admitted or rejected, based on something that's known to the server and is opaque to envoy.

A

So here's an actual example of what an envoy filter would look like to add this functionality. So I have a this envoy filter crd and I can specify any number of config patches, so each config patch tells istio where to apply in in the entire scope of the envoy config. Where should this patch be merged into, if you think about there's like a kind of a vertical flow from crds into envoy config? But now we take the envoy filter from the side and we merge it in so here.

A

What we're saying is we're going to merge into a route named reviews, v2 routes, and we have that from the virtual service. This name will get propagated through the envoy and we're saying we want to merge in a type, a filter, config, a per filter config.

A

So this is telling in another envoy filter crd, which I haven't shown. We've added the xdot filter to the http listener. On this envoy, so we have a listener. That's sitting, that's filtering requests. It receives requests. It's now going to activate the x dot filter.

A

Now, when the excel filter gets called depending on the route that gets hit, we have per route information. We have a per route context so that the the external server can know uh which route was hit, and so we have an id here that tells the external server hey for this route. We want to use this custom x, dot, config and that's something. That's that's opaque. It's it's known to the extelt server.

A

So by applying this envoy filter alongside the virtual service, we now have a computed envoy configuration for the route that contains exactly this configuration. This configuration specified in the envoy filter has just been propagated through to um to envoy it's it's been merged into. The virtual service inside of envoy and envoy will now um perform the x-dot functionality, we'll it'll it'll augment the existing routing functionality with this um x-dot config.

A

So are there any questions on how this is working? So far, just taking a look at the questions here again, the idea is that istio itself does not expose an x, the x, dot filter, there's no way in istio natively um to configure this part of envoy so using the envoy filter crd, we're able to expose a more uh sophisticated functionality and extend the functionality of istio to do more than um it's.

A

It's vanilla features uh expose okay, I am not seeing any questions, so I am going to jump into a demo of how we use this envoy filter crd to establish um multi-cluster networking.

A

So I had started on an architecture diagram here and let me just fill this in to provide some.

A

A

Okay, see my battery is running frighteningly low, so we'll try to make this fast. Okay. So essentially, what we have here is we have a cluster one and cluster one and cluster two and a pod in cluster. One wants to make a request out to cl to a pod and cluster two. So how do we do this? With istio? We have istio running in both clusters.

A

There are actually a number of possible options for this, but I'll just tell you one way to do it, and uh this is the the way that we recommend in the way that our product works, blue mesh.

A

So what's going to happen, is we're going to create a few resources here to configure istio so that this pod can make requests out to this pod over here? So the first thing we're going to create is is called the service entry. We create a service entry in the source cluster here that is going to point a dns name that we choose over to this ingress so that the pod can make a request out to the service entry and the traffic gets routed out to the ingress.

A

All right, so that's the first step. The next step that we need is. We need a a gateway crd on the remote cluster that tells it to listen for traffic on this ingress. Okay, now the last piece, the tricky piece here is, if we're doing mtls, let's say, and we want the downstream client. We want this pod here to verify the tls credentials of this upstream pod. We need the ingress to pass through the tls information and basically, we want to pass through this whole connection here and make the the ingress a pass through.

A

So we can create a gateway crd see if I can show that here, so here's the service entry we'll be creating.

A

um Now we have a gateway crd, it's going to match some special suffix that we've created and we want to pass through the tls connection. But the thing is that when this ingress receives the request for the service entry, the host header is going to be some.

A

uh You know: custom dns, name that we've created, which is going to be something like reviews.bookinfo.uh. You know cluster2.global, okay, this is a service entry that this ingress has no idea about, but that's okay.

A

We're able to get around this limitation by using an envoy filter and what this envoy filter is going to tell the gateway to do it's going to modify it with a filter that exists inside of envoy to tell it to match this remote cluster.global or cluster2.global suffix and rewrite it to doc, cluster.local.

A

And so what that's going to do is that this request is going to come to the ingress and it's going to rewrite from cluster.global to svc.cluster.local, which maps to a really a real service inside of our cluster.

A

It maps to this pod and we're able to transmit requests all the way through from pod one to ingress to pod two.

A

So I'm going to demonstrate that now let me just tear it down, because it looks like I forgot to do that.

A

Deliver the service entry, all right cool, so what we're going to do just to test it? First, I'm going to send a request that goes from the product page here. Let me show let me show the pods in both clusters, so I have this management cluster here you see, I have uh some pieces of the book info app installed.

A

If you guys know this is the book info example app, that's packaged with istio and it's got a steel running and in my remote cluster I also have some pods running and what I want to do is I'm going to try to make a request out from this product page here to this review service in the other cluster, and you can see we have an ingress setup in in both clusters, so they can talk to each other.

A

So the last thing I'll show you is the request that we're going to test and you'll see. We we send the request to this reviews book info remote cluster.global and we see there's. We could not resolve host all right so that that makes sense because we don't have any service entry set up for it. So the first thing we'll do is we will apply the service entry, and one thing to note here is this: ip address and port is actually the ip port of the uh the ingress that's exposed.

A

So if I show you, um uh we have a the istio ingress is listening on node port on this uh tcp, which is what we want um 32 000, and so that's we're going to send our requests.

A

So let's go ahead and apply this service pb paste and we created a service entry. Now, if we try sending requests we'll get a request, but we're going to get service unavailable because the upstream ingress doesn't know about it. So let's go ahead and now we'll create our gateway here, the gateway tells it to pass through the request.

A

So now we'll apply this to the remote cluster, configure that ingress and try our request again. It's it's not going to work.

A

um It still doesn't know where to route the request, but the last piece here, the real juicy one is this envoy filter crd and the envoy filter says, um like I showed before it's going to put this tcp cluster rewrite filter inside of the filter chain on this gateway, and it's going to replace that dns name that it doesn't know with one that it does so, let's go ahead and apply that and now fingers crossed.

A

We see that we're able to get 200, okay and we're actually getting a response from the remote cluster.

A

So this is just one example of how the envoy filter crd can be used to orchestrate these more complex use cases and leverage functionality that is either not exposed by istio or not leverage functionality. That requires, let's say a more opinionated way of configuring istio, because istio has its own way of configuring things. It actually does have an option for multi-cluster networking today, but it doesn't necessarily support this use case where I want to make a request to a specific pod in a specific cluster.

A

It's opinionated around more about load, balancing requests between clusters. When you have multiple instances of the same pod that live in in different clusters, so it's a bit of a different use case, and this is really the the beauty of the envoy filter crd, because it allows us to do virtually anything we want to. We can add additional clusters into is into envoy to allow it to connect to other types of services.

A

We can add additional filters. We can use it for wasm, so we can tell envoy to load wasom filters using this, which is actually the original use case for the envoy filter crd. It was specifically for um use with wason filters, which is why the envoy filter crd name makes sense, but now it's its purpose has grown and it's it's really allows you to do a kind of swiss army knife of customization of configuration, uh okay, so that kind of wraps it up. As far as I'm concerned, I don't see any questions.

A

Are there any questions on this for the live stream here.

A

People can, of course, come join the solo. I o public slack, encourage you to try our product glue, mesh, so glue mesh.

A

Automates away all this concern so you're able to use higher level crds for these use cases, and you don't have to worry about managing these envoy filters, which are pretty fragile and they need to line up correctly with the output config. So we need to know that these matches here correctly match the things that uh istio translates to, which can be a cumbersome task. It can be pretty challenging and fragile.

A

So we encourage you to skip the pain and just try us out, but of course, the purpose of this podcast is really for education and to share what we know so, um please feel free to reach out with any questions like I said, I'm on our slack. We have a lot of engineers and feel people who are on our slack are happy to take questions and yeah.

A

Thank you so much and stay tuned for another exciting episode of the heat. Take.