Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Envoy, Take the Wheel: Real-time Adaptive Circuit Breaking - Tony Allen, Lyft

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Envoy, Take the Wheel: Real-time Adaptive Circuit Breaking - Tony Allen, Lyft

Modern service mesh deployments are equipped with hundreds of tunables, such as timeouts and circuit breakers. Finding ideal initial values requires deep technical expertise. Workloads change over time, requiring regular effort to re-tune stale parameters. As a consequence, configuration errors have become a source of operational toil and one of the major causes of system failures across the industry. The service mesh should aim to expose a minimal configuration surface by dynamically adjusting parameters based on observations. Tony Allen will provide a deep-dive into how Envoy’s Adaptive Concurrency Control feature dynamically tunes circuit breaker thresholds using real-time sampling of request latencies, removing the need for periodic adjustment. He will also discuss lessons learned deploying the feature to Lyft’s production service mesh.

https://sched.co/Zes8

A

Hello, everybody, my name is tony allen, I'm a software engineer at lyft. I work in the resilience group on envoy related things and I'm going to talk a little bit about what we've been working on for the past year, which is the adaptive concurrency, control, filter and envoy.

A

So the adaptive concurrency filter is an http filter that was implemented in envoy. It measures request, latency, baselines and samples requests that are coming in after that and compares it to the baseline. So if sampled latencies are increasing from what the baseline would be, then we'll allow less requests through the filter.

A

So this was inspired by a 2018 blog post from netflix engineering.

A

So we took those ideas and implemented them inside of envoy, with a few changes that I'll talk about later on the way we're going to do things here is we're going to talk a little bit about simulating traffic. So that's how I was able to test various workloads and develop the filter, so I'm going to show various concepts using simulations and I'll describe what the graphs look like first off and then we'll talk about concurrency and circuit breaking.

A

So why would one want to limit concurrency requests where latency is coming from that sort of stuff kind of detour go off into a little bit about envoy and envoy filters, which leads into adaptive concurrency control? How that was implemented and then tails from lift infrastructure?

A

So the simulation framework that I'm using is something called buffer bloater. I wrote it to test the adaptive concurrency filter while developing it the way it works. Is it spins up a client and a server locally on the machine and I'd spin up a envoy process?

A

The client would send requests to the envoy process, which would route it to the server.

A

The server would take the request sleep for some configurable amount of time and then send a http 200 reply back through the envoy to the client, so you'll have control over the rps coming from the client and how long so you can do this in stages. You can say I want 100, rps and then bump up to 200 rps for some amount of time. The server also will have a configurable latency distribution.

A

So you can specify you know what percentile you want to have. What latency it'll gather, snaps and plot them which we'll see later on, and it allows for uh cool simulations like serger server, degradations, uh ramp, ups and rps, and that sort of stuff the adaptive currency filter really relies on or to understand it. You need to look at temporal behaviors, so this really helps out with that.

A

The source code can be found at the link in the bottom left there.

A

So this is a configuration file for this thing, the only things to note are that the client has an rps and a duration that you can specify, and this can be staged, so you can simulate burst in traffic or ramp down that sort of stuff. The server itself will have a latency profile, and that also is staged.

A

The output from the simulations looks a little bit like this, so at the very top you're going to have request latencies, so each blue point represents a request and the y-axis would be request latency and some unit of time. The unit of time is really important for the stuff I'm trying to show the rps is the second line there. The rps is static. It doesn't change for the duration of that simulation and then timeouts below that there were no timeouts in this one.

A

So you can see that there's a p99, that's very obvious. In the request. Latencies, you can see the p95 and the bulk of the requests after that.

A

That profile didn't change through the simulation and then, furthermore, there's 503s that would show up below the request timeouts the number of requests that are in flight from the perspective of the client and then the request success rate. So this is all the information I'm going to try and point out what the relevant pieces of info are for this as we move on so in the netflix blog post in 2018.

A

There was this interesting diagram so on the y-axis you have rps and on the x-axis you have time and what it's trying to show is that there's some capacity that let's say a server would have- and that's you know, unknowable for the purposes of you know any configurations, so the rps is stable in the beginning and it's below what the capacity the server is and then about halfway through it bumps up beyond what this capacity is and you'll notice that this latency in the blue line begins to increase at that halfway point and as it increases it's going to go into timeout town and then at some point it's just gonna.

A

Have this comic book explosion, so I wanted to kind of understand. What's going on there? Can the simulation framework reproduce this uh long story short it can so the latency at the top is steadily increasing from that midway point that the rps increases beyond what the capacity that server would be about three quarters, the way through we're seeing timeouts, which will cause a success rate to plummet to zero, and that's what we're seeing when there's that comic book explosion nice.

A

So let's talk about cues and concurrency, uh now that the simulation has been explained so there's some fixed number of requests that a system can handle at any time and that's dictated by various resources available to whatever computer is running on right, so cpu storage network that sort of stuff anything beyond that, as in like, if requests come in and need more resources that are available to that machine, a queue is going to start to form if the incoming rate of request doesn't change.

A

So that's what we're seeing here about the halfway point: we're seeing the outstanding request count just steadily increase and as it steadily increases, we see the latencies steadily increase until some point. All the requests just start timing out.

A

So not all this cueing is bad. uh Temporary increases in queuing delays are totally okay. This happens in real life all the time right, uh there's unexpected bursts. We recover from them.

A

So what you see here is a simulation where I bumped up the rps to the same amount that we saw on the first simulation, but then quickly dropped it back down to normal levels and you'll see that at the bottom there the request queue begins to form, but then just as the rps goes back down to normal levels, the queues burn down and then latencies return back to normal. The success rate was unaffected. There.

A

Now trouble kind of starts to form when latencies get too high, the latencies increase due to cueing delays, and if we have a burst, as we saw earlier, that just doesn't go away.

A

It'll begin to negatively affect the callers which lead to cascading failures, and what we mean by this is your server is going to be inundated with requests. A queue is going to start to form. Latencies are going to increase.

A

All the callers to that service are also going to start to form queues because they're dependent on that service, so that they can service their own requests and then all the callers depend on that and then next thing you know everyone has cues. All the latencies have increased timeout town comic book explosion.

A

So how do we fix this circuit? Breaking is the answer in envoy today and the way those work there are five different circuit breakers, but the one we really care about is this max request circuit breaker, which will limit the number of outstanding requests allowed to a particular cluster.

A

So if I'm an envoy, I receive a request and I need to route it to a cluster. If that cluster has an upper bound on the number of outstanding requests- and I have that many outstanding requests to that cluster- I'm just going to return a 503 to whoever sent the request- that's routed there.

A

So let's revisit that traffic overload scenario and see how circuit breaking would have affected this simulation.

A

You'll notice that with a well-configured circuit breaker, the latencies are under control. So if you pay attention to scales there at the top, if you can see it, uh the p99 latency doesn't really increase half of what the request latency in the non-circuit breaker simulation shows.

A

Now you can look at the timeout frequencies, there's no timeouts with a well-configured circuit, breaker scenario.

A

There are 503 responses and that's due to the circuit breaker, rejecting requests that would have otherwise contributed to the queue that would have formed and you'll see that the active requests count for the well configured circuit, breaker doesn't exceed whatever the max request. Setting was for this particular simulation, so the success rate doesn't doesn't uh plummet down to zero at about three quarters of the way.

A

Now not all circuit breakers are configured well in real life, they're notoriously hard to configure. So if we have a poorly configured circuit, breaker, that's still better than having no circuit breakers at all.

A

You'll see that in the poorly configured case, the request latencies are much higher, we're still getting some timeouts, or rather we're getting timeouts that we weren't, seeing in the well-configured case and the queue size is going to be much larger right because we overshot what the circuit breaker setting should have been, and the success rate suffers because of the timeouts in conjunction with the 503s, but it's still better than what we saw originally when there was no circuit breaking.

A

So how do we configure these things? You need to understand the service limitations, and this is done by performance testing, profiling of a particular application or service. You can ramp up the traffic. You can pay attention to the latencies see when it tips over see what the q was at that particular time and then set the limit to that. You'll also want to allow headroom for births. As we saw earlier.

A

We have bursts they're, okay. If they go away quickly and we want to keep the currency values up to date now this is pretty hard uh system. Topologies change people push code that affects the performance characteristics of an application.

A

This is very hard, so service owners really shouldn't have to profile a concurrency their application. Ideally, they would just focus on the application and not have to think about network-related things. The whole point in envoy is to abstract the network, so it's hard also to account for bursts. It's unclear what is acceptable like what is the time length for a burst? uh How big can these cues get?

A

How do we measure when it burns down the queue if things return back to normal and manually, keeping the concurrency value or manually keeping the circuit breaker values up to date is very hard.

A

So, in an ideal world we wouldn't have to do this manual configuration uh the system would figure out its own limits. There's no need to know about system, topology or hardware, and then a lot of cases. We can't know this if you're in a cloud vendor who knows what these things are running on and what abstractions they put in place.

A

We would also want it to adapt to changes. So if I push a commit that makes the performance characteristics of an application suffer or uh dramatically benefits the performance characteristics such that I can handle more requests simultaneously. We want this thing to adapt to it and we want it to be cheap to compute, because the purpose is to run an application, not just to run envoy.

A

We can do this in an envoy filter, but first we must talk about envoy filters and how they work. So an analogy I like to use here is uh pasta. Sauce is the pasta noodle what envoy filters are to envoy, so the pasta noodle is just a vessel for the pasta sauce.

A

An envoy is just a vessel for envoy filters, so here's a basic envoy, config, okay, we don't have any filters configured here and I'm just going to spin up an envoy with this configuration and run our quest through it and see what it does and what ends up happening is that envoy closes the connection because it has no filters, so it has nothing to do.

A

So, let's zoom in on an envoy process and see, what's going on in there, so you'll see the greatest hits of envoy they're the listener. The listener filters network filters.

A

The listener is a construct inside of envoy that you just tell it I'd like to listen on some port and all the requests that come in that are destined to that port hit that listener, and then they kick off the chain of events that send things through the filters.

A

The listener filters are something that allows code to be executed upon, accepting a connection so upon the first, except on that socket. We can run some code in these listener filters, so something that would change connection metadata.

A

If I wanted to change the remote ip or determine if something is a tls connection or not, this is where that kind of stuff would happen. You can also rate limit connection, so the local rate limit filter in envoy is implemented as a listener. Filter.

A

Now the network filters are envoy filters that operate on bytes that are coming in over a connection. So there's read and write filters and the read filters execute code when data is received over a connection and the right filter just does something when data is written.

A

Now a really important network filter is the http connection manager filter, and this is what has the http filters that everyone knows and loves so it'll parse raw bytes over the connection, like any network filter, but those bytes are converted into http message objects, and this is going to allow envoy to operate at a higher level of abstraction. So we can do things on a per request basis.

A

Instead of a per series of bytes basis inside the http connection manager, we have an http filter chain and that's going to be agnostic to whatever the underlying request protocol is, whether it be h1h2 grpc, quick whatever, and these http filters are going to have a decode step, which is what which is uh what's going to operate on inbound requests. So imagine I'm an envoy.

A

I receive an http request, then I'm going to decode it and then send it along it'll go somewhere, I'm going to get a reply back and then I'm going to perform the encode step and then return it where it needs to go. Not all http filters need to do things on the d code or the end code right. They can do one or the other or both.

A

So now that we're all experts in envoy filters, let's talk a little bit about adaptive. Concurrency, I'm going to keep this as simple as I can. It's it's a complicated concept, but I'm gonna talk a little bit of a high level and gloss over some of the nitty gritty details. Perhaps at a future time I can talk about envoy for evil, wizards or something, but not today.

A

So the adaptive currency filter is implemented as an http filter since we're operating on a per request basis right. We don't really care about bytes over the connection we care about what those bytes represent.

A

So it protects a upstream server by enforcing an upper bound on outstanding requests, similar to what a circuit breaker is going to do so we can turn away excessive traffic by returning 503s and not forwarding it along to the application we're trying to protect and this filter can sit in front of the circuit breaker and be used in conjunction with it. So one reason you'd want to do this is if you want to toggle the adaptive concurrency filter on or off, then you still have circuit breakers.

A

Protecting you in that case, or you can compare the two.

A

So one important concept with the adaptive concurrency filter is this gradient. So this was discussed in that 2018 netflix blog post. It's inspired from latency based tcp congestion control algorithms. So you know everyone's first introduction to tcp. Is that we'll you know adjust our uh our tcp window if we drop packets or something like that, but there's a drop base. That means some nick buffer somewhere has filled up full of packets right and then a request was dropped, so we're adjusting it.

A

Then the latency based algorithms are going to prevent that from happening or try to by observing whatever latency was measured during the handshake and adjusting the tcp window. Based on how long it takes requests to go over the network.

A

So we're kind of doing the same thing here, we're going to measure some kind of ideal latency and use that as our baseline and that's what we'll call the rtt ideal or the ideal round trip time, and then we're also going to measure just sampled requests right. So once we have this ideal, as we move forward sampling requests, we can slice up various time windows and then we can summarize them and call this an rtt sample.

A

Now this gradient value, which is related to these round trip times, is nice because it informs what direction we want to take our concurrency limit in this filter. So if the ideal round trip time is less than the sampled round trip time, we know a q is formed right. The sampled latencies are increasing from what is ideal, so we want to lower the concurrency limit if the ideal is roughly equal to the sampled or the ideal is greater than the sampled latencies right we're doing pretty good.

A

So we can increase the concurrency limit and see what happens so we can represent the new limit like this. We'll have the current limit. You just multiply it by the gradient and you get the behavior I mentioned a moment ago.

A

uh You also need to think what about headroom for bursts and also what, if the rtt ideal is roughly equal to the rtt sampled, your gradient's going to be about one we're, not really going to be changing the concurrency limit. So we need a mechanism to push the concurrency limit wider and we can do that by just adding some headroom value right, which I think by default, is going to be the square root of what the concurrency limit is or what the new concurrency limit is.

A

So this behavior of having a currency value and then probing into or basically widening the concurrency limit and allowing more requests through until the latency begins to deviate, in which case we close it, as shown in this netflix blog post figure and a simulation I performed in python. I just scripted this up, so you'll see an actual limit in blue on the left side and then the same thing in orange on the right side.

A

uh The way I simulated this was if uh the concurrency limit was larger than what the actual limits should be right then inject latency into this and then see what happens. So, what you end up, having is the scenario where we widen the concurrency limit. Wait for some kind of latency deviation to occur, close the limit and then wait for things to return back to normal. So you'll see this bobbing motion and it tends to hover around what the actual limit should be.

A

This also adjusts itself uh to service degradations, so you can imagine a scenario where the ideal concurrency will be lower or higher in the case of a patch that changes the performance characteristics for the better. This methodology still ends up finding what the ideal concurrency is in these simulations.

A

It's a little more complicated, as I said earlier, right, there's this buffer value and there's all kinds of stuff in there, but I'm not going to talk about that. Maybe folks have questions after I can answer them.

A

So, let's run through a life of a packet or a life of a request for the adaptive concurrency filter. You remember our fellow from earlier very smart. He has an adapt currency filter, a circuit breaker which we'll ignore and we'll call this the lime surface.

A

So this lime service can handle one request. At a time you can see the concurrency limits one. We have no outstanding requests. So when a request is coming in through the filter, it's going to hit the decode step in the filter and in there we're going to mark a time stamp. We're going to say what time is it right now and then we're going to check against how many outstanding requests we have right and if the outstanding requests are less than the concurrency limit, we can go ahead and just forward that request along.

A

So, while the lime service is inspecting this line or whatever it does, another request is going to come in now our concurrency limits, one our outstanding requests are won, so we can't let this through. We've already hit our concurrency limit, so we just reject it with a 503.

A

Now the line service is done, doing whatever it does, and then it's going to send a reply back and that's going to go through the encode step and we can derive the request latency at this point. So we know when the request went through the filter and was forwarded along to the line service, and then we know when the reply is coming back. So we can just subtract the two and then get what the request latency is sample the value.

A

Now our rtt ideal uh we're going to want to periodically recalculate this and the way that this works is we're going to fix the concurrency limit to some low value that we can either specify, because we know what our application should be able to handle at uh in the worst case, like there's, no reason that our application cannot handle whatever this low value is and then we'll aggregate all the samples and summarize it or you can just leave it at the default, which I believe is three.

A

And then it's going to periodically recalculate the concurrency limit at another interval and that's also sampled or that's also summarized via percentiles.

A

So here's a basic config we'll have our concurrency limit update interval, which in this case would be 500 milliseconds and then our min rtt recalculation, which is that rtt ideal and that's at about 300 seconds there.

A

So, given these two things, we've only specified two pieces of data. How often am I updating my concurrency limit? How often am I recalculating the ideal around trip time, just in case service characteristic change, or I have a noisy neighbor or something like that.

A

So here's the adaptive currency filter in action compared to a well configured circuit, breaker you'll, see that in the middle there. In our scenario the rps is going to bump up and then a q is going to begin to form and then the filter is going to react to this increased latency. Due that q lower the concurrency limit- and you can see there that there's increased 503s there's a higher rate than there would be for the rest of the simulation.

A

The cue size burns down because we're not letting as many requests through and then latencies return back to normal and then we're periodically just uh returning 503s in scenarios where the latency gets higher than we would like.

A

Now compare this with the well-configured circuit breaker, uh the the latencies are tighter in the adaptive concurrent currency filter. The success rate's mostly the same, and the cue sizes are roughly equivalent as well, but what's interesting is that the adaptive currency filter only needed two pieces of information.

A

It didn't need to know how many requests the service can handle. We didn't have to do any kind of measurements of that kind.

A

We just had to know how often should it calculate what the ideal latency is, and how often should it update the concurrency limit?

A

I think that's awesome forget about circuit breakers, I'm just kidding. We should probably use both if you want to toggle the haptic concurrency on and off uh the configuration for adaptive currency can get more complicated if you want it to right, there's all kinds of values like a jitter, a buffer value, all that kind of stuff that you can specify. But one thing I want to zoom in on is this min concurrency setting and that's just going to specify the minimum allowed concurrency limit?

A

Okay. So when you're recalculating the min rtt or the ideal round trip time, the filter will pin the concurrency limit to this value and then make its measurements.

A

If that value is too low, what you can end up with the scenario you can end up with is that the success rate is going to drop because we're rejecting lots of requests during the measurement window. So you'll see this periodic drop there now this can be mitigated with retries okay. So if I have a fleet of servers that are all going through, this min rtt calculation I'll send a request to one it's going to return a 503, and then you can just retry somewhere else and hopefully they're not in a measurement window.

A

We can help that along by introducing this jitter value, which is going to randomly delay the calculations to prevent an entire fleet from sinking. You can imagine some scenario where you scale up and you have lots of servers that came up at the exact same time and started their min rtt measurements at the exact same time.

A

So what this jitter value is going to do is introduce a random timer and then make sure that things don't line up. So you can see here over multiple simulations, the bursts and uh 503s during the min rtt measurement window. They don't line up right, so if you hit one and you're rejected and you retry on another, the chances are very high that you're not going to hit a mid-rtt window.

A

So I'll talk about our experiences at lift and default settings. So we track the p95 latencies and we find that. That's uh that's what we want, but that's for the rtt ideal and the sampled latencies- um and I didn't put this here- but we by default, uh take a measurement of 500 requests to calculate that p95 for the min rtt.

A

We have a 500 millisecond sampling window. So that's every 500 milliseconds we're updating the concurrency limit, there's a 50 jitter and a three minute min rtt window. So that means every three to four and a half minutes. There's going to be a in rtt measurement window we have 100 buffer, which means we'll allow a doubling of what the ideal latency is in our samples.

A

Anything beyond that. Then we'll start to clamp down on the concurrency limit and we have a minimum currency of 25 by default.

A

Some services change many of these values, but the thing we find that people tend to change the most is the minimum concurrency. This is actually pretty easy to change. Folks will just look in their dashboards and look at their service at steady state when it's not on fire, see what the number of active requests is over.

A

You know, seven days or so, and then just pick something that's about that value, because you know that things are fine there and we want to keep the latencies about where they would be there before adopting the feature for a new service. We always verify that the downstream callers to that service are retrying on 503s it'd be unfortunate if they sent to requests were rejected by the filter because of some mnrt calculation window and then never retried somewhere else where it most likely would have succeeded. So we have an increased success rate with that.

A

Because of this we use retry budgets in lieu of retry circuit breakers.

A

You can imagine a scenario where you have very high rps service and you enter a min rtt measurement window and lots of requests get rejected. So you have lots of retries and by default envoys retry circuit breaker uh setting is, is three active retries that are allowed.

A

The retry budgets do this as a percentage, so you can say 25 of my outstanding requests are allowed to be retries and you still get the protection from retry storms this way, but it will scale with the number of outstanding requests you have for a service.

A

um We then set the min concurrency to roughly what the steady state number of active requests was. As I said earlier, some general observations almost all adapted concurrency events that I've witnessed are due to a service degradation, not spikes in rps, so a bad deploy or an upstream dependency or like a third party, that's having a latency or an outage. That's where it's coming from it's never too much. Traffic is coming into an instance causing the latencies to increase.

A

So, let's take a look at that bad deploy scenario. This was a situation where a service at lyft had done a deployment, and it had caused cpu utilization to increase on the majority of the nodes in that cluster, and it was independent of the number of requests that were coming in. So, if you just sent it no requests, the cpu would still be redlining.

A

So adaptive currency, in this scenario, notices that, oh okay, so it doesn't know that the cpu utilization is increasing, but it is seeing that the latencies are beginning to increase, so it starts shedding tons of load and what it's doing is it it's? Keeping the sampled round trip times within 2x of the min rtt, because we have 100 buffer value, so the average min rtt is in yellow there and in green. Those are our sampled round trip times and it stays roughly within uh well within 100 of what the min rtt is there.

A

So all that load shedding was just for the purposes of keeping those two lines close together. But, as you can see, you know, 100 cpu utilization isn't doing any favors to the request latencies there.

A

So thanks I'll take questions.