Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio - The Packet's-Eye View - Matt Turner, Tetrate

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Istio - The Packet's-Eye View - Matt Turner, Tetrate

The Istio project reached 1.0 this summer, and is mature enough to have LTS releases. It's getting a lot of attention, but in a lot of ways it's still a mystery. You've probably read about it, you might have tried it, but do you really understand it? It promises advances routing, security, and resiliency, all for free! In this session I'll present a practical introduction to the operation of Istio - what features it can bring to your environment. What's unique about this talk is that we'll be exploring the different parts of Istio by following one plucky little packet into the mesh, through it, and out again. As we meet each component we'll learn why it's there, what it does, and see a demo of how to configure it for common tasks. This will leave you not only with slides showing example configs, but a valuable mental model and a unique insight into the service mesh's operation.

To learn more: https://sched.co/GraU

A

Good morning, thank you all for coming this. This is quite a crowd, and it's not me, that's obviously a testament to how popular and yeah the history of project is how much interest everybody's gone in. So this is. This is awesome to see. My name is Matt Turner, I tweet and have a website, as you might imagine, I'm a software engineer at Tetris we're a startup doing SEO and service mesh related stuff.

A

We are, of course, hiring that's all I'll say on that and I'm here to talk to you about SEO, specifically the packets I view. So what that means is what we're going to try to learn today is how a packet traverses through an sto system and obviously that leans heavily on the Envoy proxy. As you probably know, and the exposition I'm going to give is sto in a kubernetes cluster, because that's by far the most common deployment scenario- and hopefully this is coupon- we should all know a little bit of our communities.

A

So already so that's that's the easiest way to explain it as well. This isn't like an introduction to SEO I'm, not going to give you the pitch of artists. Ears features there's a bunch of great material out there on that already.

A

What I'm gonna cover is how a packet goes kind of east to west through the system, we're gonna, see how envoi gets involved and we're gonna see all of the control plane components involved as well on what they do, and what this will hopefully do is build you, a usable mental model for sort of reasoning, about sto, for understanding the emergent behavior that you see from this whole service mesh for understanding how it does what it does and why it does it that way. There are a couple of interesting architectural features that we'll cover.

A

So, if you ever find yourself having to defend this, do to our to a boss or somebody who says I, don't like it I, don't think we should use it. Hopefully this will help and while this year is an extremely mature and stable piece of software, if, if ever, you should find a bug in the rare rare event that you find a bug, hopefully this mental model will help with with debugging as well just to get the most out of this talk. It would be useful if you knew a bit of basic networking knowledge right.

A

What's an IP address, what's a root table a bit of kubernetes knowledge? This is cute con, although I have talked to you know crowds when nobody's even heard of kubernetes. But if you know what a deployment is a service, you know the fact that there's no overlay network that will be that will be useful and, as I say, this is a deep dive. We're going to look at how its geo works.

A

So if it would be great, if you already knew kind of what it what it is and what it's trying to do other because I'm, not I, don't have the time to cover that thirty five minutes isn't very long. I've had to cut a lot of the introductory slides, but hopefully everybody can keep up so just to kick us off. This is your standard sto diagram right straight off sto. This is this is what we're aiming for. This is what would be explaining.

A

So what we're looking at here is is a service mesh, as you know, right and ultimately, what we've got is two services service, a trying to talk to service B and the way I'm sure you know that that happens in SEO is through a proxy. It's especially the employ proxy, which is just graduated into the CNC F. Congratulations to those guys.

A

So we we use what's called the sidecar model of taking one of these, employ HTTP proxies and strapping it alongside every service in that services pot, because we used to have monoliths write service, a and service be used to be classes in a Java program and service. They used to talk to service B by doing a method call. Then this on microservices thing happened and they're now separate UNIX processes on in separate containers on separate machines, so that method call became a network call and that can be slow and it can be unreliable.

A

So for a very brief history, you know the what we originally did. I guess was add a library like history or finical to our code right, which would do things that you now need now you're, making those Network calls, which is reach eyes and timeouts and all those kind of things there are a few problems with that. You know: we've added. We suddenly taken this, this micro service and we've built I'm going to call it add, mini, lift right.

A

We added a whole bunch of code back in like a really big fat library that did all of this stuff. That probably had way more code in it than our business logic. And so every time that's got an upgrade or a CVE. We have to roll the whole thing. We have to go to redeploy our service, so you better hope the tests pass and it builds on master and they just use a lot of memory and they're.

A

You know history, it's a finical, the JVM, only that weren't these libraries didn't exist for every language, so there are a few problems, so this has now come out of process. So you know the sto team looked around, and so the envoi proxy is a really great HTTP proxy. It has all of these features we need. So what is do does. Is it's strapped it alongside so I? Guess a survey mash if you're being pedantic is this mesh of services service alien service be with with their proxies? What eise do an sis do?

A

Is this control plane these three pilot, mixer and Citadel components that I'll go through and what they do is they configure? They do magic and they can figure the proxy to do magic. To give you the emergent behavior that you want now these three components have an API in front of them. This control plane API, which is how you, as a user, can figure them like the three separate components, are invisible to you. You just interact with this API and this API is declarative. It's like the kubernetes api.

A

It's it takes yeah, more files that declare the state of the world and the behavior as you'd like to see it. That's a really nice property of this system. So we can begin our packet, which looks like a nyan cat, because I think I've just dated myself there back to about 2005. This is a big room with some big PA I should have had this slide like play the tune, but there you go.

A

Nyan co comes in and the first place, and so let's say the the request we're looking at is coming from one of your actual end-users on a laptop on on a mobile phone somewhere on the Internet. The first thing it has to do is cause ingress, so there is an ingress. This is standard, ingress, controller, kubernetes, ingress controller. Yes, tio provides you with an English controller based on the my proxy right and what this would replace the the nginx or the H, a proxy ingress controller that you use. Now you can mix and match.

A

You can keep your engine X in ingress controller and just use the sort of service to service sort of mesh parts of its do, but if Co does provide its own ingress controller and it's based on all the way, which is a great proxy and it's configured by the same configuration files that configure the rest of the service mesh. So if you go for this, do and use their ingress controller as well, then some configuration objects will affect the behavior not only of the properties in the mesh, but also this fits back of ingress.

A

So it's a generally good idea. Tm those those own voice just do host and pathways routing. Like any other ingress system. They are controlled by the same control plane that can show off the rest of its do, which we're going to look at and, in the sake of time, I'm just going to skip this and I'm just going to kind of assume this works. So the first thing is you: in English controller, say your user wants API dot, foo, comm, /v, 1, slash users, write routing, happens that Lai seven.

A

We decide that that's that's place service. A QED I won't go into any of that anymore, so our packet is now on its way to service a and it gets there. And what is it find? Well, there's no big reveal here. We know that there's an envoy proxy alongside every service right. This is kind of a logical diagram, so that gray box is is it's not a hostess like a service I need to the blue boxes.

A

Is, is a kubernetes pod, so we know there's m'boy in this pod along slices every say, but what does that actually mean like what is a pod? And how does this interception work? Well, that's it's early in the morning! So, let's go back to some theory. What what even is a container? Well, there's! No such thing as a container right. There's no first class container construct in the Linux kernel FreeBSD had a jails, Solaris had zones plan nine. Has these fully formed lanes basis line?

A

Nine is the best operating system, but we're on the wrong, but we're on the wrong we're on the wrong timeline, so we're using a student's copy of 4.4 BSD. Instead, what it has is it has these namespaces, these six namespaces. You can see here in these colored boxes and they are software isolation mechanism, so they isolate various software resources in the system. The first one is this grey box is the the mountain namespace. It's a lot like chroot. If anybody remembers that it changes the processes, view of of the file system right.

A

So what we have here is is a sort of nginx container. It's just called nginx and it's got two processes in it nginx and then a like a supervisor D to keep that running. These are standard. Linux users based processes they're just in running in a slightly different context. They're a little bit isolated, so the man namespace changes the view of the file system. If you think about that, that's entirely necessary because containers run from images right.

A

So if, if nginx here looks at user Lib, it might see Mazal Lib C and an alpine user space right, whereas the hosts might be GFC and Ubuntu user space, because it's running from from an alpine based image. So you need the mountain named space to change what it sees when it looks at /o the root of the filesystem there's a few more. The UTS namespace isolates the hostname, so this container can have a separate hostname to the other containers and to the wider host. The PID namespace isolates processes. The process table process IDs.

A

No, you like exec into a container. You do it PS. You can only see the processes in that container. The user namespace isolates user IDs. You can't see other users and if your user 1000 in this container, that is not the same user 1000 in another container, so you can't read their files. You can't nmap their pages, the IPC namespaces. If it I came, it stops that barrier prevents system fly by PC in SH, M stuff, that's more just preventing a sort of attack vector these days.

A

Nobody really uses that and then there's the network namespace that isolates networking stuff and what come on to that, because I'll see that's very important for our service mesh. So what is a given? It is pod, as we know it's one or more.

A

In this case, two containers kind of kind of strapped together the kubernetes documentation, healthfully describes it as the atomic unit of deployment, and, if you unpack that it is it's that it's the unit of stuff that you have to deploy together, you can't split these containers up and the reason for that is they're. Not each one of them is in a full container. They share some of these namespaces.

A

So necessarily they have to share the same kernel right because namespaces suggest a kernel thing, so they have to be on the same host, so they both have their own mount namespace. If you think about it, they need that because they're both running from a separate image. So they don't think that separate view with the filesystem. They both have their own UTS namespace. So they can both have different host names, but they share the same pid' namespace. So these processes can see each other and signal each other.

A

They share the same username space, so user ID 1000 for one is the same as the other and they can share pages. They can system five IPC each other if they want to and also they're in the same network namespace the same network context. So you think of a very sort of beginner description of equipment. It is pod, it's one or more containers that that share an IP right share an IP address. So within this network namespace there are several isolated networking things there is.

A

There is, as the most obvious example, there is one interface: it's called a zero. You know one IP address, so these these processes all look for each zero. They see any Sara with this with this IP, this 42. That would be different in a different part. That's how you can tell it's a different interface in a different pot. They also have a loopback interface that they can use to to each other, and that is different to the loopback interface in every other pod.

A

So the other Pods can't listen in sockets right, UNIX domain, sockets, open TCP streams, they're isolated in here as well, so is the root table, and so is the set of IP tables rules that are in play.

A

So I could write an IP tables rule that says: drop off the traffic write anything trying to come in or out Levin jannettek supervisor D fluently gets dropped on the floor, but this doesn't take the whole host down because this doesn't leak out of this container out of this network namespace into the wider host and it doesn't affect any other any other containers or pods either.

A

So, for example, nginx wants to offer a service it binds to port 8080 TCP anything that hits address. 42 port 8080 TCP gets to nginx flu and B can now not bind to 8080 because it's in the same network namespace, it's just gonna, get porting use error, so that was an original example. I guess of the of the sidecar pattern.

A

Right of it's the UNIX philosophy, you take one tool that does one job and does it it does it well, which is nginx, and it knows nothing about, for example, log shipping right it just dumps to standard out. So you add a sidecar, you had a fluent D.

A

Alongside that who's, the logs are can knows where your blk stack is and says the logs off to there that's the cycle model, so we can replace that flowing D with something a little bit more interesting, say a non-void proxy, and what that can do is offer network services like the fluid D offered logging services and the way that works is because we have one set of IP tables rules within this network.

A

Namespace that affect everything in this pot so on void, could write out a bunch of my p tables rules to say all traffic that comes through me. So everything trying to get to that port 8080 across this little gap is sort of redirected into on boy and then envoy actually spits it back out to live back and then from loop back, there's more rules that get it there. That's how the interception works, that's how it's transparent to to the application. You need no application code changes.

A

This is one of the big advantages of a service mesh. You need no application service, didn't serve its code, changes to take advantage of this and that that's how that's plumb together so the how about the way, the the way that's actually built it's kind of interesting. So when, when kubernetes starts a pod, when the cubelet starts a pod, there's code, while your CRI plugins so contain a D or kayo or something we'll start we'll make, the system calls to start to build these namespaces. Let's write the conference chair who works.

A

Accra does actually does an amazing talk on this. You kind of live codes, one or two hundred lines of go that goes through all the system calls to set up these names bases and basically make a container I like strongly recommend you find a video for that, but you'll see how I plug-in starts to build these namespaces. You know it kind of gets these far at this far and then it's about to run the container you've asked for now, as they say, is, do is transparent.

A

So, as a user, you submit a pod spec to the cube API server. That's only got the one container and you just want to run nginx. So that's what you talk in minetti's sto uses a mutating web hook. Admission controller I couldn't say that when I was practicing the Hat but changes that plot spec a bit, so it adds a couple of init containers as we'll see, and they also adds that Envoy container to the list of containers. So the first thing that happens is an init container.

A

It's just the our point image and it just uses the shell to run sissie TL and turn pour Dobbs on and that persist, but for various reasons that you know that again is isolated by these namespaces and that persist for their lifetime with this pot. So that tells you something maybe about this stability, that the team is expecting. That's not actually true anymore, that that went away in one, the oh, but it's a fun slide to live in right. What does happen and does still happen, is the next in that container?

A

Is this image called sto slash proxy in it it one's a very long shell script, I've done like a three-hour version of this talk, if you want to, if you can't get to sleep, go watch that and I go through this in painful detail.

A

What it does very briefly in a non so perfect way is to is to set up these iptables rules to do this. Interception right there's a couple of reasons: it's an inner container one. Is that again it's a single thing: it does one job and it does it well on void. The Envoy container doesn't have to know how to do this, and, secondly, manipulating iptables rules is a privileged operation. You can do it as route. What you actually need is cat net admin that capability, so this init container runs with that capability.

A

It does its thing and then it goes away. It drops privileges and then we're left with the list of containers right, the one that we submitted in the pod, spec nginx and then the one that that mutating webhook admission, controller added, which is, is tier proxy and has envoy in it, and by that point the IP tables rules are set up to do that. Interception and all the way as it happens, binds to 15,000 won on long loopback and that's how it locks into the system.

A

So I think we've covered how the packet kind of gets into the pod. What a pot is and how it bounces through envoy to service a so it does that on the way in and on the way out. So next routing. How does it know where to go so service? A has asked for service B but like where, where is service B? Where should this packet be Stan? How does it get there? So you can't just let keep proxy to it right. You could.

A

You could throw this at the wire at the kubernetes network and then coup proxy would send it to one of the service B parts at random.

A

You just get this round-robin load, balancing as I, say: I, don't really have time to do the sto pitch, but one of the the massive advantages of this do is because it's on the wire, because it sees all traffic and actually picks it up and passes it at all points, because we have this envoy sidecar everywhere we can do much more intelligent routing, so, rather than picking a service be pod at random, like queue proxy does, we know which ones got is currently offering, like you know, empirically a posterior I offering the lowest latency, which one has the lowest load.

A

We can even know which one is closest to us like in our hosts in our zone. So sto wants to take that decision. It doesn't just want to let keep proxy do something. So how would it do that? Well, if I look in cuban, it is, if I ask the API server, you know what's going on well, what is this service until we actually go for HTTP we've been here, but what is this service? What I get is is one IP, and this is the cost or IP.

A

So this is no good I fling a packet at this and I end up on the queue proxy path and I just get one at random, so we can and we want to do better than that. Okay, so we know this service discovery. Information is also exposed to have a DNS. So let's look in DNS know again one a record cluster IP. This is cluster IP range. This is not a part IV. This isn't really good enough. So I could guess right.

A

I can look at the pods and I could kind of when I can be like well, it's HTTP bin, so the pods probably have that kind of name the I got lucky here, but obviously this as I'm sure you all know, isn't sufficient in the general case, so that everybody here knows what a kubernetes services it's defined by that label selector, which can be arbitrarily complicated. They can pick pods that are more than one deployment or daemon set or whatever. So we got lucky here, but this this isn't good enough either.

A

Luckily, kubernetes does give you a way to effectively get it to execute that label selectively. For you, so there's a resource type called endpoints, so you don't are usually program endpoints in, but you can read them out every time you change a service. Endpoint resources are made so if I curate the endpoints for HTTP bin, what I do get is is pod addresses. This is pod. Ip range that I get three pod addresses, so these are the pods. So now this is just an example to show that is in a kubernetes environment.

A

It's geo can go and get hold of the actual workload endpoint addresses, and then obviously it can look could be information coming out of all the envoys in those, and it can make a much more intelligent decision about where to send this packet.

A

All right, yes, and if I look at if I dig into the endpoints object in cube, I get some useful stuff, like you know where it's running, but the host and their zone in the region. So this is the first control plane component.

A

This is a thing called pilot that sits up on there up there on the control plane and what it does is it sends this config to to the Envoy proxies it effectively sets up the routing the path and host base routes to tell the Envoy where to send these packets and as I've kind of kind of alluded to one of the ways it does. That is, it goes and talks to all the different service discovery mechanisms that are in play.

A

It goes and finds out where these endpoints are so in the case of us just running in this kubernetes cluster pilot doesn't even polka vanities pilot establishes a watch on the endpoint objects in the kubernetes api server.

A

So every time a service or a pod comes and goes, it gets an update and it kind of thinks about it and crunches it and compiles it and then sends route information to employee if you've ever configured, nginx right, you've got sort of server and location, a host and path based invoice data model- it's not entirely dissimilar to that.

A

So it sends those routes down to the Envoy.

A

As a quick aside envoy implements this envoys is API driven and implements this API, but the lift team have kind of pushed out as an open standard and they're calling it the data plane API. So this is meant to be an open spec for a comprehensive API, for configuring proxies the idea being that this is a nice open source ecosystem where you could swap them void a proxy out. If you wanted with any other conforming proxy that talks, the data plane, API I believe today, nginx.

A

Does that but, like don't quote me and I think maybe another one does so as I say, pilot talks to the service discovery system. If we're only in kubernetes, it's only going to talk to the cuban eTI's api server, because every workload it's going to want to talk to right. Every service is going to be a port and kubernetes is going to be able to tell you where it is.

A

If you're running outside of kubernetes, then you would configure pilot instead to maybe go talk to zookeeper if you're running on vm's and they all register with zookeeper. If you're in some hybrid mode- and you can do this easily- you can extend the mesh it's called. So you can say right, though my service mesh covers everything in this kubernetes cluster and also these VMs outside it.

A

Where I've manually deployed an employ the you would discover the pods through the kubernetes api server and then again your VMs might register themselves in console or zookeeper or something so pi that can be configured to talk to more than one of these service discovery systems and it'll it'll pull all the data in it'll crunchy little deduplicated and it'll write.

A

You know one consistent set of rules to all the invoice, so what can pilot do because pilot can eventually is effectively configure invoice, its routing and in orderly, so the proxy like features I, don't have time for demo in this slot either. So what it can do is it can set up that ingress routing. It can tell the the ingress controller to do all the things we want and then it can use other envoy features. So we can set up traffic mirroring, which is an envoy feature right.

A

Let's at this apiece who configure a gift Envoy to say when you're routing to this path, you're actually sending it to service a and to like service a prime. At the same time, it can do traffic shifting right in classic every testing. Of course it can its proxy.

A

It can do canary deployments now, if you think about Canaries they're, really just a special case of traffic, shifting they're, just a much smaller you're just sending to do services, but you want to send a very more amount, 1% or one request, a second or something to one of those. One of those end points, but sto can do this because envoy is on the wire. You couldn't do this with kubernetes right.

A

If you've got three pods in a deployment, the best kind of traffic split you can do is one for there, two thirds with the way that cube proxy works because we've got it employed on the wire pilot can tell it to literally. Just do you know Rand times nor point, not one, then just do a 1% thing, and then everybody has other features, because it's layer, 7 right. It's an HTTP proxy. It knows what a bad HTTP response looks like you know you get bytes back, but they're not just white to employ.

A

It knows that that's an HTTP 500 as opposed to a 200, so we can do circus braking. It can do fault injection. It can do all that kind of stuff and that's what pilot configures it with so can the packet come across the void, yet kind of gets serviced be well. Maybe there might be in noise where service for years? It knows how to talk to it. You know whether to retry or whatever, but there might be some policy in place.

A

It might be that there's an ACL and access control, this denying service a from talking to service B. It might be that service B has has a rate limit. We don't want to talk to service, be too often now in isolation in the dark. You know, pilot could configure envoi to say, there's a rate limit on service B. If you're ever talking to service B, you know a thousand QPS max, please in this case we've got three pods so or the service bees would get 3,000 QPS right.

A

But let's say we ship a new service, a that's just the codes a bit inefficient and the autoscaler makes suddenly means there's 10 pods. Well, there you've got 10,000 QPS coming across the service B as a side effect of that. So that kind of you can't make these decisions in the dark. That kind of mechanism wouldn't be good enough and you need global coordination. All right say all the service bees. We know every time they get a request. They talk to a database that database has a certain size.

A

So, however, many service beats there are between them. They can only take a thousand QPS that kind of globally corded global rate limit needs global coordination and that's where the second component comes in and that's a thing called mixer a mixer is kind of on on the path. If you want it's kind of on the wire every time, anybody gets a request, it calls to mixer, and it says what should I? What should I do with this is okay to send it I hope we pop the rate limit for this minutes.

A

This second does: does the policy allow it so yeah things like ACLs things like are back service to service are back. You know. If that changes, you don't want to wait for pilot to recompute it and push it out. You want to make that decision like right now, so those requests going to mix her as well and also it doesn't have to be necessarily a thing that's in with in Sto, so your policy decision of can service a doctor service B might actually be farmed out to an external policy agent right.

A

So, in which case all of these M boys need, rather than them knowing where that policy agent is and knowing how to talk to it and have credentials for it, they just talk to mixer thinking yeah. This next diagram shows I. Do it better? Mixer will just talk to those external systems so, as I say, every request that comes into an employ get sent. It gets past the headers, the metadata get sent off to mixer and their mixer can do various online things. Okay, so pilot does configuration that kind of changes on human time.

A

You know new pods come and go maybe every few seconds, but for things that are like per request, that's mixer, so the metadata gets to end up to mixer and it does two broad categories of things. The first is to report, so it'll it'll, write logs about all the packets say into elasticsearch. It'll create metrics about all of these requests, but because you've got an employ on both sides. You know what the real latency is.

A

You know what the actual bandwidth you know, what the actual set of byte rates are, so all of that can become made into metrics and that can be reported up to, for example, a Prometheus system and again, rather than all the envoy's having to know where each of those systems is and have credentials for them. They all just said it's a mixer, mixer, aggregated and mixer sends it off.

A

When you install this to your out-of-the-box it'll it'll ship, your Prometheus and then elasticsearch, just as an example and it'll configure itself to just put your metrics into those systems, but you can configure mixer to send metrics to stackdriver or to cloud watch shorter and like sis diggle to any other system that you're using, and you only have to do it once right. It's only mixer that has to know how to do that. The other side of what it does is the check side of things, which is where it says.

A

While we're trying to make this request, is it allowed and that's where it would I'd say, a rate limit which it'll just count up in memory, for you throw some service to service our back rules for ACLs and again these are internal things. You ride, sto config files that describe those our back walls, but you can configure mixer to say every time you get a packet I want you to go talk to my LDAP system and see if see, if the owner of service a is allowed to talk to the owner of service B.

A

So this all comes through mixer. This has this makes a lot of people squint right, because this looks like a single performance bottleneck and it looks like a single point of failure. So why is the architecture like this? Well I mean it is a single point of failure in that it's a single point that can't fail, but the whole system is really designed to be to be resilient to that and if anything makes a make to reliability better so and the key to that is caching.

A

So, for the reporting side of things you know we we passed the headers, we send them up to mixer mix them sends them on that's asynchronous. So if mixer is down or can't be reached that doesn't block the main m'boy thread and voice continues to forward packets, that's a synchronous, that's batch dot. Some aggregation can also happen client-side, so you said less stuff across the wire.

A

So that's absolutely kept out of the main thread out of the out of the way, and if it has to fail, then it you know if that buffer gets, that it will fail and envoi will continue to honor. The routes that it's been given by pilot verbs for the checks, the key there is caching so envoy, has of what's called a filter. It's got a plug-in, a plug-in mechanism, so there's a there's, an envoy plug-in that talks to mixer and gets these answers.

A

These yes-or-no check answers and that that client for mixer is what's called fat right. It's it's a big lump of code. It's got a lot of. It's got a lot of logic in it. Most of that is around caching. So envoy, pauses, pauses to say an HTTP packet sends all of the headers up to mix her and says right. A user was to send service a wants to talk to service B and it's accepting the encoding Jason and it wants gzip compression and the content length is like 72.

A

Mixer goes and looks at all of its rules and says: that's not allowed. You can't do that. That request can't go ahead. It's a no from me when makes the sense that reply. It sends the cash key with it. So it says right so: I looked at all of the headers you sent me, but the reason it's a no is just your source and your destination right service. A is not allowed to talk to service B at all, so I want you to cache this.

A

No and I want you to cache it under a key of source service, a destination service B. So if another request comes in from A to B, it's automatically in, though the fact that it's you know it's a different language, it wants be zip encoding right and it's you know XML rather than Jason, doesn't matter that the cache key for this is just it's just the source and destination.

A

So there's minimal cache keys to send backed by mixer with its replies so that this cache can be, as you know, as wide-ranging and as optimal as possible. Mixer also gives to TTLs for that answer. So it gives a cache expiry in terms of wall clock time, I think it's a cache expiry in terms of the number of requests, so we can say it's a no and it's valid for a second or three hundred packets.

A

Whichever you know tons first, so that that's all about making sure that if mixer goes down or is slow to respond, that you know you, you hit your cache absolutely as much as possible, but still you have this safety.

A

So if something comes into m'boy and it's not in the cache, it's a new kind of request pattern that we haven't seen before you know: mixer will go in the ass and you've got you've got an online incident policy check and you've got sort of global rate limit enforcement, and that's kind of the reason that in this picture, I took it out of this little hole up here and I moved it down it's technically control plane, but I would say it's more more. It's closer to the data plane.

A

It's more like this, the sort of slow part of the data plane, because it is on that request flow, but a lot of steps as I say, have been taken to mitigate any any adverse effects of that.

A

So what can mix to do well, as I say, to board things guessing arm reporting check. The reporting is your three pillars of logs tracing a metric, but there are lots of pluggable backends. It can talk to and you're checking is checking access, control lists based on our back policy and rate limits and a few other things.

A

So can we finally send the packet across, maybe not because this lady here is called Eve and she wants to drop in on our conversation right, so we said that services, the business logic becomes super simple. These services they don't bother with retries or timeouts, they don't throw circuit breakers and they also only talk HTTP. So no longer do I have to make like a TLS certificate bundle it with my application, worried about rotation, which nobody ever does right.

A

These things always just really long lived on weakened order that, for me it can and it should- and it does so what the envoys will do is they will put that communication in a TLS tunnel on the wire so that the communication is encrypted and you can't listen in that needs certificates, and that brings us on to the third and final control plan component, which is Citadel, which is basically a certificate management system that generates and rotates those certificates and configures them into the air voice.

A

This is mutual TLS. So, if you think about that the way the web works, you know this server would offer up a certificate. The client would check it against its CA bundles that are installed, but the server has no real way of validating the client like the client. Doesn't your web browser doesn't present a certificate, which is why websites have to ask you to login right. They have to move that out to an application layer, because you have to prove who you are in this case.

A

Citadel gives certificates to LS certificates to both sides, so both their voice have assert and they can do mutual TLS, mutual verification of each other and then based on you know, based on that, I've got a certificate with a CN. In you know an identity. This is I'm service, a that's! What the sort of services service are back.

A

Authorization rules are based on the whole thing, about identity and or authentication and working out who these services are and if they really are who they say they are or whether they should get a certificate with a surname is a whole nother talk, it's the spec or the spiffy spec, that's quite interesting, so I would say. Finally we are done. The packet the packet has reached the end.

A

We've seen it to come in, it's been routed by by ingress provided by an Android proxy configured by these to a control plane, it's bounced in and out of, employ with transparent injection of the proxy transparent interception of the traffic. It's come across an N TLS tunnel, it's obeyed or the policy. And again, it's done the same dance, it's bouncer than out of employ. On the other side, a couple more things to say: I think I'm actually doing just about right on time.

A

Another couple of things to say is that it, of course, the control plan has an API which are used to could configure these components, but they need somewhere to store that config and they need to be a you know, watch the configure updates all that kind of stuff. So what you really need is a sort of database and an API server to front it when we're in kubernetes. We just hijack the kubernetes one.

A

So when you install this geo, it actually writes a bunch of CR DS into the kubernetes api server to say: hey you're, expecting some more configuration documents with the following schemata and then what is do CTL does, which is the equivalent of cube CTL. Your user sends the STR config just off to the cuban Etta's api server. There sto format, resources but cube is expecting them because the CR d's have reconfigured right and it uses the cube EDTV to store it.

A

If you are running outside equipment at ease, then you have to run this infrastructure yourself. The one final feature that it has is is egress control. Kubernetes, obviously doesn't do this for you there's no real concept of an egress controller in kubernetes. Normally, there's no egress object. Sto does do it. So again we can apply the same kind of network services to egress. Let's say you know: service B ultimately wants it. Gonna get a picture of a kitty off the internet. We can have this this Bank of egress proxies.

A

You know, retry connections to Kitty comm in case it's a little bit unreliable, apply a rate limit, so everything in the cluster is only making 10 requests a second to this to this website on the internet. So they don't think we'd do essing them. You can do path, rewrites, host, rewrites or you know all the features offered by the egress and you or you could say what my cluster is an internal service.

A

You know, I have my data and my logic are inside this cluster I'm serving my users, who are using my light on my iPhone app. Nothing really should be reaching out to the Internet if anything tries to connect out, I've probably been owned, so I'll actually want to deny it's quite a common configuration is I want to deny all connections to the Internet and, if anything tries it. Of course, you know this this.

A

These n voice talk to mixer to so they will report that up into mix so it'll appear in your logs and you can just set up at me, Prometheus alert that says something's trying to talk to the Internet like. Is it just doing a shitty self update, or has it been owned?

A

So we saw how that packet in traverses the SEO system, leaning heavily on the my proxy- and this was in the context of kubernetes, which is hopefully what you're using. We saw how the control plane does that and how the various components do, what they do, and the split between pilots kind of ahead of time, human time static, configuration our mixers online policy enforcement and hopefully that's a useful mental model for for what's going on, and what a virtual service does you can?

A

You can imagine, hopefully, which one that is and what a destination rule does and yeah they go in. Take your take the lid off from that and have a look around sto. So thank you very much for your time.