Istio IstioCon 2021, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: What Envoy Hears When Istio Speaks

Description

#IstioCon2021

Presented at IstioCon 2021 by Rob Salmond.

Istio listens to Kubernetes and speaks to Envoy. We will explore these conversations and learn to understand what’s being said.

A

What you're going to see uh is some material. I has adapted from training material that I produced for my colleagues at the cart labs. It includes a lot of original diagrams and information, but I've also borrowed liberally from the istio community diagrams from youtube talks. I watch blog posts. Things like that. I have tried my best to attribute all of them.

A

um If I fail to say out loud what I've attributed uh the actual sources from this material, it will be in the speaker notes and I'm going to provide the link to the deck at the end. So you can follow up on any of that interesting stuff because lots of deep depth in those resources for any topics you might be interested in. I'm also going to try to unravel some issue and envoy terminology as we go along.

A

But if I do use a term you're not familiar with feel free to pm me in the slack or find me another way, and I can try to explain that. The only assumption I have is that you have a pretty good understanding of kubernetes already and kubernetes sort of vanilla, kubernetes networking and from there we'll uh get into how sdo works. So this is me um frequently seen wearing unusual hats on the internet. I'm not an istio maintainer, just a long time user.

A

The I joined this organization, the car labs a little over two years ago. We had production istio in place when I joined it was my first experience with it. um So uh so my all my everything comes today, just just from you know, uh hard-earned shin banging on on rough edges, and things of you know the wisdom of production as they call it. uh So if I say something today, that's not true.

A

That's totally on me uh and me failing to understand something. I really look forward to being corrected uh by people watching who who understand stuff and maybe better than I do, and I provide the twitter link, not because you should follow me because for me, twitter's just the pillow that I scream into it's mostly be swearing at the cloud, but I will tweet a link to this deck afterwards and you can find those links.

A

I mentioned okay today we're going to talk about these four subjects a little bit about what istiod is and how it interacts with kubernetes and envoy uh talk briefly about exactly what envoy is, although we've already heard some of that today. So hopefully we can buzz right through that we're going to go deep into how envoy works and uh we're gonna at the end, we'll take a brief sort of high-level tour of what an istio-generated envoy, config actually looks like, and it's pretty beefy okay.

A

uh So this is what the envoy community would refer to as an envoy management server.

A

um I think uh it was mentioned already in one of the talks today um that uh actually it was like during the opening statements I think, somebody's talking about originally getting with matt from lyft and talking about you know how to program this proxy that they released, and if you actually go to the envoy proxy github org, you can find a project called the go control plane, which is sort of the canonical way to control envoy proxies, uh which is what histod is sort of uh built on top of uh other names that you will hear for seod floating around the community because of the sort of winding path that it has taken to where it is today, you'll sometimes hear it referred to as pilot or discovery or the issue discovery service um kind of just leftover names that people use in github issues and things.

A

So if you see that that's what they're talking about this is a dramatic oversimplification of the architecture. I've got a link here in the notes. That'll. Take you to a more complete description of that, but we're going to talk about a couple of these components today um and specifically: well: okay, so there's you know that does more than just that, so there's lots of logic in this dod now we'll get to some of that in a second we're also going to skip citadel completely um there. uh You know this.

A

This is the component that handles um circuit signing if you're very interested in that topic. uh There's a talk tomorrow that looks like it's going to cover that stuff uh called know your peers by alex von boxel. I think that's where you'll find excuse me a deeper discussion of these things, but for now we're going to talk about gallium pilot and we'll start with galley.

A

So what I would consider the core function of sdod to be is to read from kubernetes and write to envoy it's. You know, sort of a one-way pipe in general um and if you deign to read the istio docs, which you should, what you'll find is that it says very clearly quote: istio does not provide service discovery. End quote: um I don't know who wrote that and if they're here today, maybe they can explain to me what that means, because one of the things that gali does is it discovers services in kubernetes.

A

uh It also contains a service registry, where it sort of keeps track of all the things that it's discovered um obviously has native support to discover services inside of the kubernetes ecosystem, but it actually uh comes or came originally with other service discovery mechanisms.

A

Now those mechanisms aren't as important in the kubernetes world that we live in today, but there are kind of being subsumed by a technology. The community refers to as the mesh configuration protocol you abbreviated as mcp all over the place.

A

This is a mechanism that will allow you to sort of generically reach into hdod and tell it about things that maybe doesn't know about yet.

A

So this would be something like google has a project called traffic director, but you can imagine a scenario where you know managed service can reach into your cluster and say like oh, that managed redis instance. You just launched it's now, reachable from inside the service mesh and it would be doable achievable through this mechanism and sort of taking over this service discovery sdod to see what it has discovered when it wasn't doing service discovery. um You can port forward to this port here and check out this endpoint and then uh inside of there's.

A

It provides all these debug endpoints, where you can get a list of all the services. That's discovered all the end points things like that in the notes. With the slide deck you'll find a link to a complete list of all the debug endpoints that are available.

A

Okay beyond service discovery, galley also serves uh web hook requests. Now, we've there's been a little conversation about um how istio leverages kubernetes web hooks to provide some of the services that it does so I'll. Just sort of skim through this briefly, but basically the two uh primary types of web hook that are utilized are the mutating and validating web hooks. So there's a mutating web hook which is utilized to deliver the pod injection.

A

So what happens? Is you or something operating on your behalf will say to the kubernetes api? Please create a pod uh when that happens, the web hook configuration causes kubernetes api to fire, a request over to galley, saying here's the pod I'm about to create it will then munch the yaml hand it back, and when it's done now, that pod will have a sidecar and a container and so forth attached to it and same thing for the validating web hook, that's used for, if you you know, cuddle apply some.

A

Some crd ammo gets passed over to sdod for validation. If something wrong with it, it will try to send back a helpful error message, and so those two web hooks work and this diagram is lifted from the galley design dock, which is also linked in here.

A

This is the specification which tells kubernetes how to execute the web hook. So in this case, you can see that anytime, these api objects are either created or updated. It's going to cause a payload to be posted to a service of this name in the namespace of this name at this uri and same thing for pod creation. This is how the web hook is configured so pretty straightforward, kubernetes crd setup, one interesting side. Note the uh istio cuddle, cli pronunciation.

A

These two catalyst cli uh actually duplicates not duplicates but reuses a lot of the logic for the pod munching. So if you're curious about what is exactly going to happen to your pod spec, when you, uh when this weapon is executed, you can test it using the command like this, uh and the pod injection is actually quite configurable. So you can use this to tweak to test your tweaks to the pod injection logic and run it all completely offline.

A

So you can just make sure it's going to work before you put it up in the cluster okay. So that's gali, talking to kubernetes reading from kubernetes understanding, kubernetes we'll talk a little bit about pilot um pilot takes the services discovered by uh galley, and so do that's where it maintains this state of the world and then it takes that information and uses it to program envoy.

A

Now. This is a slide that lifted from a talk given by niraj potter, who spoke earlier and john howard who's going to speak. Actually, I think he already did, but he's probably going to speak again because he's everywhere um talk called debugging your debugging tools and I've got a link for that as well. What I want to do, it's not important that you understand everything in this slide. I think all I really want to get the point across is that you can see there's this very linear relationship between like kubernetes object, envoy thing.

A

You know there is a clear path from kubernetes through istio to envoy, for for a lot of these objects.

A

Okay- and that's all we're going to say about pilot, because really what pilot does is less interesting, what's more interesting, is how envoy responds to what pilot sends down?

A

Okay, so you're right now, we'll move into the insides of envoy world. This is how you can tell that an envoy is not a commercial product, because when you go to the what is on void page, it tells you it's very simple.

A

This is lifted from uh matt klein's, an envoy internal, steep dive presentation, which is very good. You give it all over the place. You should definitely check that if you haven't seen it, a lot of stuff isn't important for our needs today. What you can tell is that it's thoroughly buzzword compliant we're going to get into a couple of these things in a bit.

A

uh One interesting thing, which I think probably I'll mention earlier, um is that istio uses a fork of envoy. Why is that? Well, that's because uh istio adds a considerable amount of functionality to envoy and, as you have probably already heard and we'll hear more of this week, that functionality is going to be provided through webassembly extensions, as opposed to compiled in static extensions, but for the time being, most of the stuff that istio adds to envoy is actually still written in sort of static, c, plus plus.

A

That goes through this webassembly null vm thing that you'll hear about later this week, but um it yeah it's. This is why we need a fork and um eventually the fork will go away. You'll be able to do this dynamic um module like I'd add all the functionality needs dynamically, but uh until then we're stuck with this fork.

A

uh Incidentally, john again, um if he says something he's probably right about istio anyway, okay terminology, um I dislike these terms, but we have to talk about them because they're used everywhere um in envoy, land downstream means where connections come from and upstream means where connections go to. So this is, like you know, clients and backend services. You can essentially think of these, as I I've had a really hard time getting these words to stick in my head. When I first started, um I find drawing your network diagram.

A

This way kind of helps orient the those words in my head anyway, um if you think about sort of the idea of north south traffic. So in this from this proxy's perspective, you know here's the front end app with a sidecar. The clients are uh downstream and the back end is upstream and then from this proxy's perspective, the front end is downstream and some other back-end services is uh upstream.

A

um So anyway, I'm gonna say that a bunch, hopefully that settles in your head. I've also found this metaphor helpful.

A

What you should do is abandon any metaphor that uses water, because the stream part doesn't make any sense there, but hopefully that helps and then the other important terminology that we need to talk about are clusters and endpoints and listeners and routes again. These are all first-class uh envoy constructs. You can think of clusters and endpoints as basically host names and ip addresses. um Cluster is an unfortunate name, perhaps because we spend a lot of time talking about kubernetes clusters in an envoy cluster is not a kubernetes cluster um yeah, it's basically a hostname.

A

It's like you know something some abstract thing to put in front of a one or more ip addresses. These are all examples of cluster names that istio would create for you that would program a sidecar with names such as these.

A

As you can see, they don't comply to sort of normal, looking dns host names that you might be familiar with, but yeah these are all totally valid and then listeners and routes basically describe sort of you know what to listen to and what to do with it and and where to send it right so where to send it. So so what listen to um that is usually going to be like an ip and a port for tcp and udp traffic?

A

It could also be a path for unix domain socket um and then the listener is also going to configure.

A

What would what an envoy pronounce is a consider, a set of filters and into a filter chain and that's attached to a listener, and those filters define the logic of of how that connection is processed before it gets routed somewhere.

A

And then a route is a special type of envoy object that only pertains to http traffic and that's used for matching the properties of an incoming request to some back end. So you can imagine you know. If the header is this, then it goes to that service.

A

If the uri is that it goes to this service things like that, it's your sort of basic, you know nginx type, stuff, okay, so with those terms in mind, now we're going to look into how they function, uh I lifted this diagram from directly from the envoy blog uh from a post called uh how to write envoy filters like a ninja um which I've got linked here as well, um but this is sort of to give you a sense of of what I'm talking about when I'm talking about the filter chain.

A

Now I'm going to use and that's sort of an abstract metaphor here and hopefully most of the folks remember familiar with some basic python. What we're looking at here is a fully functioned python web application very simple one um and we're just going to sort of use this method as a mental model to think about a filter chain. So in this case um you know, in order for this uh web app to function, you have to define a function that complies to the reference function. Signature.

A

The whiskey is the sort of the canonical python method for uh serving web traffic. So you have to match this function, signature. You have to call this start response function and then you have to return a response body and that's it now.

A

You've got a functioning web app and you can run this web app like we're doing down here and send a simple web request in and get a response back, and you can imagine that when you're running this python process somewhere between you know sending this extremely simple client request uh and executing the the python logic some extra processing had to happen.

A

You know somebody had to decode a tcp byte stream, recognize that it's http like decode, that into http and then turn it into python data types that are suitable for you know, calling a function, and so so, if you're going to do that in an envoy filter chain, it might look something like this. You know you start by reading a rob block of tcp bytes.

A

You know trying to interpret them as http, and then you know um converting them into into python data types and then maybe calling some link function now now you know there are no, uh I don't think anyone's doing android filters like that, but um you know this is a. This is what you're looking at here is the simplest possible onboard filter chain which is an empty one, and the default behavior of an empty filter chain is to close the connection, um and so a chain that goes nowhere.

A

An empty chain is, is how outbound traffic policy is enforced. You'll you'll wind up with an envoy listener that has an empty filter chain, and that's it. The connection just gets dropped. I think in practice. Actually it's not empty empty. I think the stats filter is always there to provide those nice metrics that you see in cali and prometheus and things but yeah as long as if the chain, if the end of the chain isn't a outbound connection somewhere, then it just gets closed.

A

um So this we've got just an example here to think about um of us of a source and destination app. You know, you've got some source, apps got a couple of pods in it got a destination kubernetes service you're going to send a request to, um and we know that you know this uh service is going to get discovered by galley right. We've already talked about galleys out there looking for services and when it finds a service like this one, it's going to produce a filter chain.

A

That looks something like this, so it's got a listener that matches the service vip and it's got a single, uh a link in the filter chain, which is just the tcp proxy filter and all it does it's just a dumb. You know carbon copy, the bite's over in this direction, kind of thing, and it's going to send it to a specific envoy cluster. If you know it's like a host name, that represents some back number of back end that are capable of servicing the request.

A

So this is what that same. Filter chain looks like in an actual envoy configuration and you can. You can get envoy any sidecar to to display this stuff for you by using the istio cuddle config command, you can ask for other current listeners routes, etc. That are configured, and so you can see. This is again the listener name. Tcp proxy is the only link in this chain and there's the outbound destination cluster.

A

um Now, in practice, like I said, uh istio always injects uh the stats filter, which is you know, part of the.

A

This is why you go and go out and spend big money on istio, because you get all this visibility um and so even for a simple tcp proxy you're always going to see this in here um and then, incidentally, if you're looking at your istiometrics and when you see cx as a suffix like underscore cx as a suffix on a metric name, that means connection, which means that the statistic was generated by the tcp stats filter.

A

And if you see rq, that means request, which means the statistic was generated by the http stats filter, um which gives you which understands http and gives you a little more fidelity there.

A

Okay now I've said the word extension a couple times I may have said plug-in, but you'll find in um envoy, and you know in slack and github and so forth, people will sort of use the words extension and plug-in interchangeably to describe basically any of the code that go hooks into envoy to expand its functionality.

A

um And while it's true that all all of the filters in the filter chain that we're talking about are extensions, not all extensions are filters or filters or they're extensions that do all kinds of other things. Now. This is just some of the things: the istio functionality, that's um I shouldn't say not just issue but also envoy functionality. That's provided by extensions that have nothing to do with being part of the filter chain.

A

So when I think of the data plane, I think of just that filter chain- that's that is the heart of the filter chain. So so these are the types of filters that that can be assembled to compose a chain, uh and there are more than these. These are just sort of the core ones that we talk about so listener filters always executed.

A

First, um they typically are going to inspect an incoming connection and try to determine some learn something about what the nature of that connection and then attach a little bit of metadata to inside of you know, istio memory or the sidecar memory.

A

I should say about what it's found so typically, this looks like checking to see if it's tls, if it's tls, then that you know this look, a boolean gets said: you're checking to see if it's http, that sort of thing and then usually what that means is that a particular network filter winds up um processing that connection based on what's discovered by the listener filter.

A

Now network filters tend to be layer, 7 aware so you're going to find network filters out there for redis kafka things like that. Typically, a network filter is going to implement some kind of layer. 7 features usually something like application, aware: connection, pooling or smart routing. Something like that, and if you think about I mean you could implement an entire network service uh right in the network. Filter actually- and in fact, envoy comes with a couple of them.

A

There's an echo filter and http direct response filter that you can program to say if request comes to slash, foo return, a 500 or something like that. I mean technically that's a complete service and then just because http is so important in the modern web. There's a whole subcategory of network filter. That's just http filters, so uh the h, the http network filter, is called the http connection. Manager you'll frequently see this shortened to just hcm and http connection manager then has many other filters that can hang off of that to provide http functionality.

A

So this is where you get your rate. Limiting gzip compression request, routing cores headers. All that stuff is all implemented as uh http filters.

A

Okay, so we're gonna step through this listener filters always executed first they're asking some sort of question they're providing metadata, and now we move on to the hcm. This is where we start decoding tcp bytes into real http constructs.

A

uh Then we start providing our http value. You know we're collecting http stats, we're you know, injecting headers and stuff like that. Now it's worth pointing out the uh the http router filter filter needs to be present. Otherwise the request doesn't go anywhere, so it's usually the last one in the filter chain and then once the request gets routed well, that little red arrow there is actually doing a lot of heavy lifting. So we're going to take a quick diversion to just talk about how requests actually get routed.

A

Now this diagram also lifted from matt klein's envoy internal steep dive, talk which again linked here. Please go watch it. uh You will learn things, there's a lot more going on here than we're going to talk about, but so far with the little filter diagram that I showed. We have traversed this much of the path.

A

So, keeping in mind that you know cluster means hostname and upstream means: where connections go once a routing decision has been made by some filter, either tcp or http. That filter is then going to ask the cluster manager for connection to an upstream service, and that's that's sort of this component here. So there's a manager for all these envoy constructs the cluster manager handles you know things like knowing which endpoints are healthy and doing load, balancing and connection pooling and those sorts of things. There's an alternate diagram lifted from the envoy docs.

A

Just to give you a different visual representation of how the managers relate to clusters and listeners that they're responsible for okay. So with that little red arrow uh covered, there's one more filter abstraction worth talking about before we move on, and that is the this thing called the filter chain match the filter chain match basically allows you to make choice about which filters are going to be applied. Based on what the listener filters discover and so remember we talked about listening filters.

A

You know doing some sort of test and reporting back and then once they've extracted some information about an inbound connection. You can uh use the filter chain match to test those properties that were discovered by the listener filters and then go down one of several arbitrary paths.

A

So, for example, if you determine that the incoming connection is http because the listener the http listener filter to determine that's true, then you can pass it off to the http connection manager and then do your http routing and so forth, and if you can't determine that it's http, it's probably safe to just treat it as tcp. So that's the default and then router to an stream cluster, and this is what that looks like in an actual envoy config. So here you've got this listener.

A

This one's listening on this 1510 and you can see the filter chain matches here and basically it says if this happens to be http one one point one or two, then we're gonna apply the filters found in this list, and this is the the empty match condition you know if the any other connection. Essentially, that is not these things. We're gonna apply the filters found in uh this filter chain here and uh to be clear.

A

You know, I'm really wanna just connect these sort of abstract ideas to the concrete envoy, configs you're gonna find because sooner or later, you're gonna find yourself uh dumping on voice and looking at them and scratching your head, and hopefully this will. This will help with that exercise.

A

Okay, so thinking about that example of connecting from some source app to some destination app that was discovered by galley, and you know, which then composed a complete filter chain for proxying tcp requests to that service when the source app actually makes an outbound request to this ip address and this port.

A

How does it actually end up in the envoy process in the same pod, instead of actually being routed out through the network interface to where it's supposed to be going? uh Well, if you've been around kubernetes for four seconds, you'll know that everything is done with ip tables, and this is how we do this here too. um So this is a dump of all of the iptables rules from inside of a pod with an istio sidecar.

A

These have all been added as part of the sidecar injection process by in the knit container um and that logic for that lives inside of a command called pilot agent. You can find in the repo. It also does a bunch of other useful stuff.

A

I'm going to sort of briefly summarize what you find here, but if you're interested more about this jimmy song who's speaking later today during the chinese track, has an astoundingly in-depth blog post about how all of this stuff works uh linked in the slide notes here and uh strongly recommended.

A

So basically, what we see here is that um we're telling any outbound connection coming out of this pod to pass through that particular ip tables chain. The whole chain is out here, but the most of it is not important. All we're really saying is that anything that goes basically anything not everything, but basically anything that comes out of this pod needs to pass through this one chain and that chain is going to redirect it to localhost 15001, and the same thing is actually done for traffic coming into this pod on 15006.

A

And what that means is that you can uh pop into uh a sidecar pod, uh yeah pop into a pod and look and dump the listing processes from that sidecar, and what you'll find is that there's actually only two ports listening for all of this traffic, I mean a couple: other sort of maintenance ports like debug, port and statsport, and things like that, but fifteen thousand and one fifteen thousand six.

A

This is the only place that data plane traffic is passing through, um and this is sort of the sort of secret of an envoy listener that you don't see um documented really well anywhere, is that most of them are logical. Most of them don't actually open ports.

A

So the way this will fit together is that you've got your source app. You know it shoots. A laser out. Tcp shaped laser bounces off an ip table's mirror goes into an actually open port on this virtual outbound makes its way into a filter chain, gets processed, and then the pass-through cluster which is sort of the default destination for a lot of traffic that hasn't been matched yet.

A

um Has this option called original dst, which uses iptable's kernel magic that I don't understand to recover the original destination port and ip address for this request after it has been rewritten by iptables iptables has hidden it somewhere and allows it to be recovered. So so he starts out with a destination ip import here gets rewritten by ip tables to this, but by the time it pops out this end, it's actually been restored, and so what that means is that now envoy can look through the entire collection of listeners.

A

You know this huge list of all these other services in the cluster and see any of these match, uh and if it finds one that matches the characteristics of the now restored port and ip, then it can use that as a match uh and execute the pass. The filter chain pass the connection to that filter chain.

A

Okay, okay, so with that bit of sort of how envoy functions taken care of we're going to move on to talking about um how the envoy configuration looks and, like I said I do this, because um you know eventually, sooner or later, you're gonna have to dump a sidecar config and have a look at it and it's a little bit daunting. So I saw this screenshot from a screencast from the solo.io folks, particularly this fella yuval who's going to be speaking here as well. This week he and adit who you saw speaking earlier.

A

Can we talk about wasn't filters? I guarantee you will learn something from them: they're, very smart people. um I've got a link to this as well in the notes and he's absolutely right in this talk. He even though youtube flubbed the transcription uh envoy config, is not meant for human consumption. It's not it's not fit for men or beast, um mostly machines. Now this is a full complete envoy config, I pulled it right out of the envoy. Docs seems like it could be reasonable.

A

uh However, this is an envoy config that istio generated in an otherwise completely empty cluster. um Now uh I managed to pack this mere 90 kilobytes of json into uh 2.5 megabytes of png. So I could like show you this high level view. We are absolutely not going to go drill down to what these lines mean, I'm just going to give you a high level overview of sort of the 10 000 foot view.

A

Okay, everything to the left of this line is bootstrap. Remember we're talking about the pilot agent, the init container that goes into the sidecar. This is all provided by the pilot agent. This is just get get envoy off the ground.

A

This is the, and this is where that comes from um up here. You've got things about the the node in this case. It's just a pod so like what is the pod name? What cluster version is it and so forth here? This is all uh extension and this initialization boilerplate, so all those extensions we talked about the ones that could be filters or not filters or extending istio in some way.

A

This is all initialized here down here: we're bootstrapping, prometheus metrics and the health check, endpoints, we're setting up uh local clusters and then we're talking about envoy clusters. So these are like places that you can send a request to, in this case, they're the local prometheus endpoint that provides the metrics how to find sdod and how to find zipkin for if you're tracing here we're bootstrapping, metrics and logging and the tracing configuration, and that's it- that's bootstrap out of the way so forget about that in here.

A

All it is, is a list of clusters and, like I said, I pulled this envoy config out of an otherwise empty gke cluster. So this is just like you know the default uh kubernetes services that gali discovered out of a mostly empty cluster. So it's like you, know, kubernetes master, like the api server. Whatever else is in there shows up in this big list. uh This is all we have been pushed down by seod, so forget about that list of clusters.

A

Okay, now we're into uh listeners. This is probably where you're going to want to look if you uh wind up with a istio config that doesn't doing what you think it's going to, we thought it would do. This is the part where you're actually going to want to scroll past all that other stuff and look here, because this is where um things like routing matches are going to happen. Like checking headers checking, uris things like that.

A

We also have routes down here as well as listeners. These are both the static and dynamic routes um and, like I said, those things that are places that you could send an hdb request if it matched some characteristics and then finally, you can have secrets, so these would be the mtls certificates for um mutual like uh in in mesh traffic, and if you happen to be looking at the ingress gateway, you'll find your uh client terminating secrets there as well.

A

Okay, so that's it! So so I mean there's, there's really a lot here, but most of it is boilerplate and and a list of clusters.

A

You know: that's it, the you can skip over almost all of it when you're, when you're debugging, even though it looks pretty intimidating so now, there's this little block that isn't colored in this is a magic piece of bootstrap that basically tells envoy. Now that you've started up. Where do you go for more information? And basically this is telling envoy go talk to sdod for all the important things that it's that gal is discovering.

A

So there is a cluster called xds grpc and it. This is how it looks in the static cluster definition. Basically, you know a service called sdod in a namespace called seo system on a port far far away uh and that that gets your sidecar started.

A

Okay, so that's envoy config, I'm going to park it right there. I know that was a lot. So here's some cute pictures to soothe the spirit, uh and I genuinely hope I didn't make too many grievous errors, and I genuinely hope people learned a new thing.

A