Cloud Native Computing Foundation EnvoyCon 2019 (San Diego), 21 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building Low Latency Topologies with Envoy - John Howard | Snow Petterson | Liam White

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Building Low Latency Topologies with Envoy - John Howard, Google | Snow Petterson, Square | Liam White, Tetrate

This talk will go over how Envoy's load balancing algorithms work, with a focus on how the different components of load balancing (priorities, localities, etc.) work together and how they interact with mechanisms such as retries and outlier detection. We will then make things more concrete by detailing how these components were used to implement Istio and Square's locality load balancing features.

A

So we're here to talk to you about multi-region load, balancing with envoy, basically we're going to look into how sto and square does this and kind of use this as a way of exploring how to build low, latency topologies with onward so I'm snow and one of the senior maintainer Zafar Moya and I work at square. It's been. What's my option, work working on this load, balancing stuff.

B

I'm John I work at Google on East you and.

C

I'm Liam my work at saturate a now.

B

C

Of maintainer, okay, so let's go over the agenda quickly.

A

C

By laying out the acceptance criteria, I'm cover some of the built in envoy features for locality, load, balancing and then discuss why they didn't really meet any of our requirements. Then we'll deep dive into some envoy internals and how we use them to build low, latency, topologies and then we'll use two case studies from mr. square to show how you would go about implementing that and then finally, we'll cover some of the gotchas that you might see in production, and you need to think about when designing and operating these type of features.

C

So, let's start with what problem we're trying to solve we'll look at. Let's look at how you tend to build applications to increase their availability. So, for example, a multi easy application running with sto might look something like this right. This is a single mesh. We've got multiple clusters, a single control, plane per cluster and any service a in here can transparently call any service be, but just because any service a can call service B doesn't mean that it should so. The first problem is latency.

C

If you are calling something local, that's gonna be pretty quick, but that's gonna get progressively worse. Obviously, as you get further away because we're bound by the speed of light and for those running in public clouds cost egress traffic is costly right as something you don't want to do.

C

So, if you're calling something local, you know get charged for that, if you're calling something external, that's gonna be expensive and in fact, some of the existing locality functionality and Envoy was actually motivated by this, as this Twitter thread shows right, although latency is a concern, egress traffic costs were actually the primary motivator for this functionality in Amelie.

C

So when we're building out these features, basically we want to maximize availability but minimize the latency of latency and costs of our application, and to do that, we need to keep traffic in the same AZ where possible, but we want to be able to fail over to other regions, but, and we want to prefer closer regions. So I mentioned a minute ago: the existing functionality in Envoy the kind of zone aware and locality weighted low balancing stuff neither SEO r-square actually use this functionality. So let's kind of look at why they didn't meet our requirements.

C

So the most obvious one. If you go and read through the Envoy documentation, is a zone aware load, balancing now zone, aware load, balancing actually does kind of meet the keep traffic local where possible. But the problem we had and when I say we I mean is do hats- is that the Envoy cluster name must be static? You have to know it at bootstrap and in history. Oh, this isn't true.

C

We could have done some hacky rebooting envoy and then setting it every time it changed, but we decided not to do that just for pain reasons, basically, so the reason it's static as far as I understand it is because the zone aware stuff was introduced around the time of the v1 api's.

C

We're typically configured in general was much more static and it was often pre computed and served from a web server or s3, or something like that and the reason the zone aware load, balancing stuff didn't meet square requirements is basically it wasn't expressive enough to describe their topology.

C

So the other option that kind of looks like this is locality, weighted, load, balancing, but really locality, weighted load. Balancing is designed to solve a different problem right. It's just weighted based on locality. We actually want a hundred percent to stay in region, everything's healthy or in a date sensor when everything's healthy. Now you could probably hack locality weights at low balancing to do most of this, but that's relatively horrible, mainly because the routing envoy is the only thing that knows whether or not it's able to talk to another envoy network partitions aren't symmetrical right.

C

So, just because the control plane can't see the envoy doesn't mean that the other envoy can't see the envoy, so snow mentioned snow did a talk last year on pecan and they mention he mentioned that Square were looking into the locality load-balancing, but they hadn't yet come up with a solution on how they wanted to do this. So I grabbed him afterwards and we kind of came.

C

We had a little discussion and came to conclusion that we needed to build this out of some lower level concepts in envoy so I'll hand over to snow, so he can go through and discuss these.

A

Right so to get this kind of locality below bounds, this kind of region behavior that we want I wanted to kind of explain it in terms of the load, balancing components that AMA provides so liam talked about localities. We call it a load balancing which is not quite sufficient, so I'll talk about that and how I didn't how it worked is priority load balancing, which is really the feature that you want to use for this kind of topology.

A

So basically the way that only those load balancing is that it will first select a priority to send traffic to a and M points are placed into a priority and all the end points in the lowest priority will be selected first before any of the end points and higher priorities. As long as that, priority has enough available host once hosts are becoming becoming unavailable, healthcheck failures and other mechanisms I can flag a host unavailable kicks in, and then traffic start to shift over to all their priorities.

A

Now, once the priority husband is like that this is when locality selection comes into play, it's an optional feature that you can enable, and but if you do you get weighted round robin between your localities, which are basically grouping of endpoints scoped to a priority.

A

Once a locality has been selected, the specified load, balancing algorithm that you specify on the cluster if you like, random, run Robin, whichever comes into play so regardless, where you select you're still going through this Priority Selection optional, ocala selection, they're separate from the algorithm specified so to see what that looks like consider this setup, where we have eight different endpoints kind of split into two priorities: each with two localities. In this case, assuming everything is healthy. All traffic is routed to p0.

A

Now from p0 one of the localities got selected and again we select an endpoint here. You saying which I have a load. Balancing algorithm was configured now. This is the happy path where everything is healthy and you've successfully contained all your traffic to hit the same region that you put into p0. But let's look at what happens? One thing becomes unhealthy, so this is where the concept of spillover comments in we're only assumes that each of you priorities and localities are over provisioned in the sense that it assumes that you put more endpoints in there.

A

Then it actually needs to handle the current, the the expected load. So there's a forty percent default over Pershing's factor that provides a buffer where, if a few endpoints goes down, nothing happens and once a certain threshold is here, you will see that your spillover percentage, the amount of traffic that goes elsewhere starts to increase linear with the availability ratio.

A

So going back to our example, we can see what this looks like with weights added in so here we've just added in the the weights of each priority and off the localities. So we see here, p0 is still receiving hundreds in traffic everything's healthy and our localities in p0 gets six percent of traffic. As one endpoint goes down, you see that l1 has been scaled down, and that is because l1 is at the 50% capacity, which is enough to trigger spillover, because the over-provisioning fact there is not sufficient to kind of cancel it out.

A

P zero, however, is still at 100%, because p zero is 25% unavailable, which is not enough to trigger spillover. You take another endpoint away, and now you see, P Zero is dropping down to 70% and 30% of traffic spills over to p1, and this is essentially what happens when you didn't get region failover.

A

With these setups, where you're the closest region is no longer sufficiently available to handle all the traffic, so traffic has been now being sent elsewhere, then the last bit I want to mention is that there's also support for the grading endpoints, which is a basically a third health status between healthy, unhealthy and then this degree in the middle, which is a way to say that endpoints are still up. They should still receive traffic, but they are, they should be only used after there's, no healthy, endpoints and yeah.

A

So with that, we have a bunch of components around priorities and localities. So let's see how this is used in practice in sto and with square.

B

So with the easter EO implementation, we are essentially just using the priority field and we determine the priority based on how close of a match a given. Endpoint is based on the region zone and sub zone. So, depending on where you're running, you may only have a region zone, so the matching order is basically just a exact match.

B

So the highest priority is one where you're in the exact same region, zone sub-zone, followed by one that just matches the region of zone, just the region and then finally, will start spilling over to anywhere in a normal typical deployment on kubernetes, we are able to extract this information automatically from the kubernetes node. For other cases we're running on VMs or if you're, calling say external services, you can also manually configure the localities per endpoint.

B

So what that may look like is if we have an endpoint in the US West one. Our first priority is going to be ones that are also in US West one and then the P one would be ones that are just in u.s. West, but in a different sub zone. Then finally, just ones in the US and finally, we'll start spilling over to Europe and other regions.

B

So for the configuration we want to keep this pretty simple rather than having a lot of fine-tuning available. So in general, a user just needs to enable this and then to find some health checks on their service and we'll start doing the locality load balancing for them. We'll talk more about why the health trackers requirement, we do have some fine-tuning available as well. So one case that users were running into was that at the final level of spillover we start spilling over to any endpoint in the world which may not make sense.

B

For example, you may want to spill over from us West US East first before we start going to Europe, which may be a lot more expensive in terms of latency. So this example configuration would configure us was to spill over to us, east and vice-versa before we spill over to the rest of the endpoints, since it still a fairly new feature in East yo. We started implementing this earlier this year, so we're still adding some things in the next version of East Joe, we're hoping to add a per service configuration.

B

Some users have applications that are already doing load, balancing that this sort of conflicts with or they don't have an even distribution of their services, so they want to enable or disable this per service. Another thing we're hoping to add is cross region retries, so, rather than waiting for endpoints to be marked, unhealthy will start actually spilling over on the retry level and snow. We'll talk a bit more about that as well.

A

So the way that squared does this is the priority assignment is very similar to how is the air dosage just some heuristic, based on the strewing equality of the zone and region of various endpoints, which works pretty well, we use localities as the weight way users might do, green-blue deploys and various kind of traffic shifting within the priorities we use both features for us most of the Fela romantics are on by default everyone's running with them, so we don't give people much control complex static priority ordering between regions.

A

Most of that is all just very opinionated. Li stated by our control plane. What we do allow is various runtime overrides, so we use the subset load balancer to allow users to restrict the set of endpoints that should be considered for load balancing. So in the context of region failover, this is an effective way of locking down which region should be considered for a given request. So, instead of this being static configured in some llamó, you pass a header that specifies which regions you want to consider.

A

So, by default we have this big cloth sort of has everything in there with all the priorities assigned, but users might lock it down to a smaller subset. This also has the benefit of saving us on health checks, since we, since we use a single cluster instead of one cluster per possible subset- and this is basically generally useful for people who are using a very low latency requirements.

A

So they know that the thing they're calling exists in multiple regions, but some of them are too far away and there's no way you will ever get a request fast enough. So there's no point in ever failing over and making this a prayer request thing, as opposed to a per service, makes it a lot more flexible and also, as John mentioned, there's the whole idea of region cross region beatriz, which is instead of relying on health checks and available of the changes to do failover.

A

You can have envoy send subsequent retry attempts to different priorities, and this allows you to do spillover without there being any health changes. So for us, this is especially useful for active, active systems that are running and multiple regions, and they want to be able to utilize all of the resources to serve a request.

A

If you're waiting for health checks to change, clients will see requests to the close of region and potentially fail without ever trying the other region until health checks, catches up and sometimes you're, not even going in healthy, because most of your service might still be up, but a subset of its functionality is no longer working, and so you don't want to go completely unhealthy you're able to have retries go somewhere else. You don't have to you.

A

Don't have to go on healthy to your trigger spillover, and this also works just the way you margin with subset matching where it's lets you clamp down on which regions should be considered for retries as well, and which is just how we can enable this by default and tell people. If you don't want this, you can disable it and that's greatly simplifies our control play and logic, because we just enabled everywhere and make this dynamic config for clients.

A

We also have a since we're medium-sized company. We just know all the regions were in, so it all easier for us to give an ordering from to indicate which region should be tempted in which order. So for us we could just put all of the orderings in the control plane and that becomes kind of the the way. It's done.

A

There's nobody to configure this, because it's not neat because we know which one are closer and then the last thing I wanted to talk about from square is that we use the concept of the greater than points to do.

A

We entry, we use active health checking and we have active degraded health checking in the sense that we can have up streams, respond at specific header so that they get marked as degraded and again for these active active systems. There are cases where people want to. They want to perform some maintenance on one region and they're not going to take it down.

A

It's gonna be up there and functional, but like maybe high latency, maybe maybe they want to make sure that if something goes wrong, it won't blow up they're able to market as degraded which, instead of disabling traffic to it, redirect all the traffic to the other regions. Now, if these regions get overloaded and go unhealthy, you're still you'll still be able to spill back into the degraded cluster because it's still there and usable, but you shouldn't try to like shift traffic away.

A

This is also a good way of shifting traffic away from individual hosts within a larger, larger cluster, so certain systems again that perform many different tasks, might marked himself as degraded when certain functionality is down, but it will still be, but it still able to serve other traffic. So that's a nice way of habit.

A

You know so that when individual self chicks determine that you're unable to serve a request, you don't need to know if this luck is a global thing or not, because if everything mark suppose degraded, it's completely safe because you're still preserving the original priority load. So it doesn't matter if it's a host level thing or system old thing, it's safe to go to mark you surpassed degraded.

A

So now we wanted to go over some of the gotchas that you run into about this. So it's where the biggest thing that we ran into was the fact that connections set up when you do failover is way higher than normal, and this is a kind of this funny thing where people get used to the very low connection setup of local routing when your local region, so all of their monitoring, is used to seeing very low latency. Now there, whenever spillover happens, they suddenly get flagged as an outlier in their APM solution. But oh no.

A

This is this request. I took four times as long as it's supposed to, which is because it's just like 30 millisecond request that suddenly has to do like a 200, millisecond TLS set up across across the country or across the world. So it's not actually like a bad thing, but it's a thing that people notice and react to.

A

So we had bunch of people come in to our channels and complain about Alma being slow when what was happening is that Hamid does not preemptively create connections to upstreams, which means that once once you start failover, you have to do and setup which can be much more expensive than what used to.

B

So another issue you may run into is: if you have an uneven workload distribution so generally, if you're doing say simple round-robin your traffic distribution will be even and if you were to turn on this feature and and your workloads were evenly distributed, you would see that same even distribution of traffic, but if you had, for example, only a few endpoints in like the local region like, for example, here we have just a single end point in the priority: zero now we'll see all of our traffic going towards a single end point, and all these other end points in the second priority- are no longer receiving any traffic whatsoever.

B

So this may not be so horrible if there's very low load that the first end point can handle it. But it's very easy to over run a service like this, so our initial goal was to reduce latency and reduce cost and by turning this on, if we're not careful, we may do the opposite. Our latency may go up because we're over overloading service, and now we may have all these backends, which are underutilized, so we're wasting money on them. So it's important to take note of this when you're using this feature.

B

If you do need this uneven workload, distribution and then you should definitely be cautious, so another issue you may run into is with health checks.

B

So this feature is really only as good as your health checks are because the health checks define how and when spillover is occurring so for in the worst case scenario, if you don't have any health checks defined, we'll never spill over and so we're just sub setting which endpoints will ever connect to, and so you may have only unhealthy endpoints and by unhealthy I mean logically they're not responding properly from on voice perspective.

B

There's nothing wrong with them, because we don't have any checks defined and now we're sending all of a request to these bad endpoints, while there's these other healthy endpoints in a different priority that we're ignoring. So it may be sort of obvious that you shouldn't use this. If you don't have health checks and that's actually how Easter works, we won't enable this if health checks aren't defined, but the same problem occur.

B

If you have say unresponsive or poorly defined health checks, so if you had a health check that took five minutes to fail, for example, then you may run into this scenario four or five minutes. So it's important that you tune your health checks very well.

B

If you start using this, another issue you may run into is with conflicting health checks, so this is sort of a subtle problem that we often saw on kubernetes and so people running East Rio on kubernetes will often have health checks defined in envoy, but they'll also have health checks defined in kubernetes, and these may or may not be the same things.

B

So what will happen is that kubernetes health checks will start failing, and so the control plane is now removing those endpoints from the list of endpoints it sends to Hanoi, and so now, rather than the list there are, that percentage of healthy endpoints going down and eventually triggering spillover. That percentage is staying the same because as these endpoints are unhealthy, they're just removed from the endpoint list entirely, and so generally, if you're doing round-robin it doesn't matter.

B

If an endpoint is there or if it's unhealthy, you will get roughly the same behavior, but now with locality load, balancing that actually does make a difference, and so what may happen is if you're always just removing these endpoints right when they're marked and healthy you'll never get spillover until every single end point in the region fails which may be completely unexpected and you may be wondering like.

B

Why is nothing spilling over and it can be very hard to reason about so, for example, here envoys starting to spill over 30% to this distant region, because two endpoints are marked unhealthy.

B

The control plane now removes these endpoints from the list an envoy stop spilling over, and here we would actually expect that we'd be sending some traffic to the distant region and so yeah.

B

So you definitely want to be careful if you have a situation like this, this could happen outside of kubernetes, but that's a very common one that we've ran into.

B

So, in summary, in using envoys building blocks east geo and score we're both able to satisfy our requirements for keeping traffic local when possible, we've had a lot of success when doing so. There are some rough edges that you can run into, but I think they're, mostly avoidable, with some caution and like I said, we've been very successful square is using this in production as well as many used to East, Rio users and I. Think we've got some good feedback from this feature.

B

B