Cloud Native Computing Foundation EnvoyCon 2019 (San Diego), 21 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dynamic Request Routing With Envoy - Ben Plotnick, Cruise

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Dynamic Request Routing With Envoy - Ben Plotnick, Cruise

This talk will discuss using Envoy to reroute requests in production to desired destinations. This includes substituting your own version of a production service in the request path to test your code in a real-world environment. It will cover extending Envoy using a custom filter as well as using distributed context propagation and custom chrome and mobile app extensions to enable developers to test their code on the fly.

A

B

Everybody sufficiently sleepy after lunch, good cuz, I'm gonna, put you to sleep.

B

Okay, I'm just gonna, get started.

A

Everybody I'm Ben, plotnick and I'm, going to talk today about dynamic requests, routing with envoy a little bit about me, I'm, currently on the path team at Cruz, we're a self-driving car company, but the past few years. I was in the service him for a space at Yelp, and that's mostly what I'm going to be talking about, or at least the work that I've done was on that team. So what does dynamic request routing?

A

Let's check the docs under dynamic request, routing section, it says per request routing we can, let's see override the D tab entries with the l5 d tab header. This doesn't sound like envoy. Let's read the previous paragraph dynamic request: routing is one of linker DS, more powerful and flexible features.

A

This is awkward, so anybody know where service mesh con is alright. So neither envoy nor linker d2 have d tabs, but a lot of this work was heavily influenced by finagle and liquor D. So actually, what is dynamic, request, routing I think of it as a way to route based on dynamic attributes, which is equally vague.

A

For the sake of this talk, let's make some arbitrary distinctions, so weight based routing, for example, is where you have like some requests get foo and you send it to different backends depend depending on some different probabilities, so you might send 5% to your canary. Let's call this arbitrarily traffic shifting rather than dynamic, requests routing.

A

Something else might be header based routing where you determine the back end based on a header. So in this example we have this X version header and we might use the header to metadata, filter and envoy to route based on that header, and this is kind of what I mean when I'm talking about dynamic requests or dynamic requests, routing and dynamic attributes, but again arbitrary distinction. For the sake of this talk, let's go with it.

A

So let's took take a look at something a little bit more complicated. So let's say we have some deeper call graph where we have a front envoy and then some back-end services, a B and C, and actually we have envoys sidecars on each of them. So I've drawn little pink, rectangles and I've logically split up the incoming and outgoing traffic, but it's the same process and, let's say I want to deploy a new version.

A

So it's B, Canary and I want to ensure that my request hits that B canary or maybe I spin up the version of B on my laptop and I want to hit that version.

A

If I try to just use the X version, header like we did in the previous example, this is not really gonna work because the like what is a gonna see and what version are we even talking about so to figure this out? Let's take a look at two questions, so one is: how do we shuffle data around to make routing decisions and two? How do we make routing decisions based on those data?

A

So the answer to the first one of how to push the data around kind of has an answer and distributed tracing and it's called distributed context propagation. So in tracing, if you've seen Zipkin, you might have seen xB three trace ID this header by contract. Every service needs to propagate it from wire to wire across the process, but I think we can generalize this. So we come to our first requirement, which is that we need a generalized way to propagate context.

A

So let's say, for example, that any header with that starts with X C T X must be propagated across the process and we mandate from the CTO that you have to do this. This is a lightening talk, so I'm not going to go into that in any more detail, but there are other advantages to doing this. Okay. So now we can propagate some data, but in order to know what we want to propagate, we have to figure out what we're gonna do with it.

A

We need a propagated header to somehow express that, like, instead of railing to X Ralph, why we don't have D tabs, but we can take some inspiration on that and, let's kind of arbitrarily say this is X CTX override.

A

We say we have key value pairs where we say if it's going to foo instead send it to foo v2, that's going to bar, send it to this IP.

A

You can imagine something more complex like the D tab scenario, but this is a start, so we have another high level requirement, which is we need to be able to parse this XCX header and redirect request to a desired end point, and let's see what this looks like this is going to be an envoy filter, so get ready for some pseudocode if you've written a filter or you've looked at any filters.

A

You've seen something like this, which is the decode headers fill function, and this is one of the places where you can write some code in in a filter. So the first thing we're going to do is we're going to parse that header somehow so we're gonna get this map, which maps from where the request was going to go to where we really want it to go. Now we need to decide where we're about to route so that we can override it, and then we have to say: ok.

A

If there is an override, then we need to somehow override it and I'm gonna get into what that looks like right now.

A

So, let's take a simple example: we have this XC TX override and we say a request going to foo instead, send it to like v2, and maybe that represents something like metadata on the end point. The override route function could look like this, so inject the new destination, where we want to send it to as metadata for subsetting. So this will become a little bit more clear, but basically we propagate some metadata from our filter to the router, where there's already functionality, a non-void that does metadata sub setting.

A

But maybe we have some arbitrary IP so, unlike before where we were using metadata to say where we want to go now, we want to actually have a real destination. So it's not a pre-registered endpoint, it's just an arbitrary new endpoint. Maybe this is my laptop or something envoy doesn't know about this endpoint. So we need some way to tell it.

A

There is functionality and envoy that allows you to do this. It's called the original desks cluster or the original desks functionality. I guess and we can use this.

A

We could just build right on top of this, so we can inject the ex envoy original desk host header, it's kind of long, but it basically says instead of going to this, one place go to another place and the way it does this is it kind of on the fly generates an endpoint and redirects the traffic there, and so we have this XCX override and now maybe we want to send it to something: that's not an IP.

A

It's just like my host dev or something or some like DNS, name and envoy, doesn't know about this, so we can't just use. This is actually kind of like the previous example right with the IP, but now we have this added difficulty of DNS in pseudocode. It doesn't look so hard because nothing does we okay. This is gonna get a little complicated here. We schedule an asynchronous DNS lookup on the thread, local event loop and then we add a callback there.

A

That says when the DNS lookup is done, add the original dest header with the resolved IP. So basically, once the callback is called, it looks exactly like the previous example.

A

When you do this, the request is kind of halted for a moment, but this is generally done by user by like developers. So it doesn't really matter side. Note, I wrote this filter before the dynamic forward proxy was was written so doing DNS in the request path was not trivial, but I did it?

A

Okay, so now we're able to arbitrarily reroute requests by propagating this override configuration with distributed context, propagation and then use our custom filter that like evaluates this and does something.

A

Unfortunately, the developer experience of sending headers is terrible. If you're working on mobile, it's you have to use like some proxy if you're working on web, then you have to use some like extension or whatever. So what we did is we wrapped to this functionality in some like sugar, so for Chrome we built an extension that that basically just wraps injecting a header for the app we added debug view.

A

So in the debug builds of this app, you could just open it up and and add these overrides and they'll get automatically injected on every request, and then we have CLI support, which is basically just a wrapper around curl. Unfortunately, I don't have screenshots, but trust me. It's worth it to befriend, a good front-end and mobile engineers. My front ending skills are terrible.

A

Okay, if you're going to do this yourself, there are some things to be careful about security, I think any security minded person here is probably like sweating right now, because this allows you to arbitrate arbitrarily route to like anywhere. So you want to make sure to lock that down like we didn't allow this in prod.

A

We allowed like a subset of this functionality and prod.

A

You want to make sure that you're stripping out these headers and pretty much any distributed context at your edge, and you want to make sure that you're not injecting arbitrary data that might like crash your filter, also don't write your filter to crash and actually when it fails, it's it's kind of tricky to debug, because, like it's, this weird behavior that you're causing envoy to like kind of deviate from what it's doing and the override logic is not trivial to like figuring out where your request is going to go and then overriding that, like is not that trivial or maybe it is, and I just didn't figure it out, but yeah.

A

That's that's! Basically dynamic request routing. If you're interested in discussing discussing more about this or about anything you can reach out to me afterwards or here.

B