Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Reducing Mean-Time-to-Detection of Incidents with an Envoy Service Mesh - Constance Caramanolis

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Reducing Mean-Time-to-Detection of Incidents with an Envoy Service Mesh - Constance Caramanolis, Lyft

Incident management is inherently stressful and is made worse when the diagnostics and observability data is lacking and heterogenous. Lyft runs Envoy at every hop of the network providing best in class observability across the entirety of Lyft’s network topology. Homogenous data reduces the time it takes to identify production issues. This talk will simulate a production incident at Lyft and guide the attendees through a page from the dreaded PagerDuty notification to resolution, by showing how engineers use Envoy’s extensive observability to identify and root cause the incident and remedy the situation, thus reducing mean time to resolution.

To learn more: https://sched.co/GraX

A

My name is Constance. Careless I was an envoy maintainer up till a few weeks ago and I'm here to talk about reducing, mean time to detection of incidents with an envoy service mesh. Yes, I do come from lift, and so that will be the context of how we're seeing everything today. So first I would like everyone in the room to close our eyes and imagine you're back home in your bed, it's middle of the night you're getting the best sleep.

A

You could ever get wall away at a conference in a hotel room, and then you hear the ding ding ding ding, Pedro Duty is waking it up at 3:30 9:00 in the morning pater duties, letting you know that front envoy at lift success rate has dipped below 98%.

A

This is where the adrenaline kicks in you're no longer asleep, and your start to panic, who deployed at 3 o'clock in the morning. What do I do? How do I fix this lift is hard down and I have no idea what to do, and so this is how I look like a 3 o'clock in the morning when I get a page.

A

Good thing is it's actually afternoon, and none of us are on call for lift or any of our. Hopefully none of our services that our own company, so as a group, let's all breathe in and breathe out,.

A

Today, what I'm going to do is walk us through this incident and how an engineer at lift would take steps to remediate the situation and obviously we're going to be using envoy to achieve that before we actually start breaking down the problem and the page. Here's just a simple reminder of what lifts architecture was I. Wasn't joking when you say you'd see this form to format, avert apology.

A

Many many times we have an edge envoy that routes that proxies all their requests that are coming from various clients on the internet and then enforcing requests to internal services on each of our services and edge envoy. There's an envoy running the handles ingress and egress traffic.

A

So at lyft, whenever we have a page, we have to send an instant response report email, the first one goes out: john-boy is paging due to success. Success rate, dip investigations ongoing operations, a little bit understanding of impact update whenever we figure something else out. So this is what we know so far. We know that calls coming into edge envoy are failing.

A

This is a piece or a page at lift. This is where everyone wakes up and they snap to attention. This also usually means that there's some form of customer impact and so we're on the west coast. 3:49 is almost 7:00 a.m. on the East Coast New York's, a big market Boston's big market East Coast, it's something to take very seriously so just because we are sleeping, doesn't mean that other people across the country are so.

A

We need to figure out what is causing the success rate dip on the yes on the left is a little bit of diagram is showing what edge configuration look like. We do usually group routes by domains and then the routes and I'll determine which service needs to go to. For this one. We're gonna be caring about the prefix show map and that Ford's request to map render and everything else under this domain.

A

If it hasn't matched a previous route, it will go to the render service, the page that triggered the alarm and our unfortunate wake-up call. It's just a ratio. It's looking at the total amount of 5-xx errors over the total matter, request sent, and so one of the easiest way to actually identify what serves is causing it. It's just look at the 5x x per service.

A

From the point of view bed envoy and as we can see here, it is the map render service, for whatever reason it is returning an abnormal amount of 5x x so to keep on figure out what so, what to see what we know so far is we know that a request from some clients a to map rendering or returning an error and with a little bit of side investigation. We actually know that clients can actually render a map, and if you can't render a map, no one's gonna request a lift. So this is pretty bad.

A

So how do we keep on figuring out? What's going on next, this is a snapshot of what our service service dashboard is that lift, and this shows pretty much all permutations between one service and everything else it talks to. You can also limit it between survey and service B. This will show you the amount of requests ups in response time. Success rate.

A

What's really interesting about this is it applies to every single service that lifts since a lot of ways use it every hop and the one thing that we want to look at since we know that map render is my returning errors.

A

Is it the maps or render service itself, or is it forwarding errors that I've seen from other callers, so we're gonna, look at the service errors from upstream services and we're to see here that it is a driver service, so, instead of looking into how we can make map within focusing our intentional map render we're gonna keep on following the errors until we see some root cause, if it's either a service itself or an external service, or maybe something else in one line now, one thing I want to clarify upstream versus downstream.

A

We at Envoy, we use those terminology as well. As you view it in the context of the response. Map render is dance no sir, to driver because it receives the response. Drivers upstream to map render, because it sends a response. One really awesome thing about envoy that I will not show you today cuz. It could actually take about two three hours to go through the entire.

A

Seeing all the different statistics is that you're able to see all the stats of from map render sending upstream requests to driver and, from the driver point of view, you get to see all the calls going into it, and so now we have a little bit clearer picture of what's causing the page right now. We know that driver is returning errors, but why is driver returning errors? A really nice thing about this is that's reiterating.

A

The point that we have on we're running everywhere is that let's apply the same methodology that we use for map render, let's look of any of the upstream services or causing errors, and once again, in this case it's actually an external service called photo filters and as we're figuring this out not too long ago, there is a new feature called photo filters that drivers would actually apply kind of, like the snapshot filters to their little photos very popular service in this hypothetical situation, and unfortunately, is that this wouldn't when this is returning errors, no maps are being rendered.

A

So something is going on. Very something wrong is going on wonder. The thing we know too is actually the CPU percentage of the driver. Instances is really high, that's not so good.

A

Now this is the root cause we stepped from the initial page, where we had a success rate ratio, and then we just followed the errors from the initial hop on edge. All the way down to photo filters root. Closing is only part of it. The actual interesting part is remediation, but we'll get to that in a minute. Once again, we've send an update email to everyone, so the instant report email says that we know what causing issue, but we're still talking about how we want to mediate.

A

We also know that now the maps are being rendered in clients, so first we're going to do. If we know our external provider is serving serving us errors, let's contact support, downside is is that there is no ETA for fix. They know that there's an issue, but they can't figure it out. Their machines are hos. They are not be able to take more requests. We are pretty much stumped once again panic and chaos.

A

If you we, no one can control the whole internet, and so since we can't fix our errors, you could be a little bit. Stop there. Let's talk about remediation, so first thing we can do so. We have to pursue options full one of two options: one we can wake up a driver, a developer and ask them to start coding, error, better hair error handling at four o'clock in the morning.

A

Unfortunately, unfortunately, all know that this is not the best way to handle issues in the middle of the night, so the second thing we can do is potentially reduce the load on a photo filter. Api. We were talking to the support, since we know that there instances are super. All the their machines are pretty saturated right now, reducing the load might actually help them, get back to a stable spot and return fewer errors. Now this is where Envoy is really really fascinating.

A

It that we have this thing called main ins mode and what maintenance mode will do is a when it's in your configuration. You can use, specify all the closer's upstream clusters, so you talk to. You can eventually say hey in this example. The external photo filters of stream service is not doing so well. I would like to shed five or ten percent traffic away from it, and what that will do is it will actually automatically return a 503 to the caller and not even forward request to the service the upstream service.

A

So this could give a chance for photo filter API to recover and then, after give the driver service an opportunity to go through its error handling path, and so this is an example of throughout the root cause path of where we're applying 1q surround boy finance mode.

A

So now we apply the mainus mode and we're waiting. We give it a few minutes. We sorry we apply range mode through runtime team and they'll usually take about a minute or two to get everywhere, and so after waiting a few minutes, you see or things better well.

A

Unfortunately, not success rate is still the amount of errors that drivers returning is pretty high and the actual cpu percentage hasn't changed. So we have two options. Clearly, our driver service can't handle this matter my request, so we can scale up one unfortunate thing of our current infrastructure. Eyelift we're migrating kubernetes, it's not done yet, but scaling up for us is very slow. It can take about 20 minutes for new instance. Come up and 20 minutes in our industry is very it's kind of detrimental right. It's easy!

A

It's easy to lose trust really hard to regain that trust. So, whenever a rider camera costs ride for a few minutes, they're gonna switch an alternative, so scaling up is an option we can add to this, but it's still actually not going to do anything to help us get riders back in the cards.

A

One thing we actually were realizing is that, since this new code path to apply in the photo filters is that the circuit breaker settings weren't properly configured and when the circuit breaker settings do is tell all down, streams are communicating to driver or any other service is I know how many max requests in this case is h2. This can actually handle, and if you must configure it, you can be flooding your entire OSIS, your entire service.

A

So one thing we do here is that we're gonna set a gel. The circuit breaker settings driver service to give it a better chance to reduce CPU saturation, and that way we'll still be returning some errors, but that might actually give us a better chance at keeping those whose requests are going through be successful, and so just showing us here. On top of the maintenance mode, the photo filters, we are applying circuit, breaker settings, and now we wait. We get a few minutes. Can't we're testing it manually. Can we still request rides?

A

Unfortunately not- and this is where, at this point, we've all held it's middle of the night and we've been trying to think about different ways to work around the driver code, mapper under code different photo filters and just not working, and so when we're releasing this feature, we release it under a runtime key in edge envoy and what this runtime key does it actually says once the runtime key said 100% when something matches to it send all traffic there?

A

This is really useful when you're rolling out a new feature if you're testing something out- and you want to spend only 5 or 10 percent, set this one time, cue to 5 or 10%, and only 5 or 10% of the requests will go there and everything else will do you fault to the previous matching route, and so what we do is we decide to shut it down this code path as well intentionally was had some bugs in terms of hailing error cases, things weren't fully fleshed out, and this is not something that we need to solve a 4 o'clock in the morning we're going to detour all traffic from show map the shows to map render and go back to the legacy path.

A

Now that we know that we've decided to switch all traffic to the legacy path, we're gonna wait. We expect everything to go back to normal and we're gonna wait while we're waiting for everything to go back to normal. Let's do recap so. First, we started off with identifying that the photo filters, an external service that us at lyft or if for your own company someone you have no control over and they're returning set of error. You can stop sending requests to them without having to make any code changes. This is called maintenance mode.

A

After realizing that this doesn't work, we can take another step back and notice that we had poorly misconfigured driver and there were sending too many requests that can't be handled by the service. So, let's reduce the amount of requests that we send to it and hope that things get better.

A

Unfortunately, they do not, and as the last result or as a last option in this case, we're just going to stop sending all requests on this path and we're gonna find a better solution in the day time when we're all well arrested and we're gonna go back to the legacy path that had been working for X amount of months days years after this amount of time, we actually confirmed that the drivers that the map is correctly rendering and all on all clients, application success rate is back at 100% and that we can start serving rides again.

A

Now this is element. This is a contrived example, but it is meant to show just a few of the ways that envoy can be used to handle it remediate. An incident I talked about circuit breaker settings very quickly, but we can also handle max pending requests. Max connections, so doesn't only apply to h2 outlier detection.

A

If it were the case of the driver service, instead of it being the entire fleet that was being hosed, you could actually imagine that maybe two or three of however many instances you have are just poorly performing outlier detection, we'll look at all of the five xx per instance and identify the ones that are poorly behaving and stops any requests and for a certain amount of time and we'll see after a certain amount of period, is it acts healthy again can respond just request properly if it can it'll bring it back into the pool and keep on sending requests to it?

A

But if it starts failing again, kick it back out access logging, one thing that we could do once we identified that the map render service was an issue. Is you could actually look at the access log of which prefix was being hit? You can group those in and find it out of the pattern tracing if you were earlier at Leto's talk leeta talk to us about how tracing shows the entire flow of a request. Sometimes it's not in this.

A

Sometimes it is an occasional night, but there's a weird one-off instance of something being wrong. You can use tracing to identify, in particular cases what is going wrong there and to see everything a deep dive. Another example with the edge envoy configuration what, if someone misconfigured an edge- and you start returning 404s if you're using a static config, it could take you a while to actually remediates at if you use the dynamic configurations already SCDS call them XDS. All's I, don't discovery a service api's, you could actually roll out a bad configuration.

A

Give it a 30 seconds a minute realize is bad and then send another update to remove that bad configuration in a matter of minutes. Hp header options, if it's a bad request, request timeout retries. These are things that actually we configured within the header and not something that's actually in the code traffic shifting is what I showed you in terms of the runtime switch and maintance mode was the first thing we talked about photo filters.

A

So how is this help? Look.

A

You don't have to be on the networking team or on the driver team to actually apply this knowledge. This knowledge applies to everywhere that ongoing is used. So the benefit of this is once you teach your engineers these sets. They have a set of tools, they can apply it anywhere. I have two beliefs when an infant happens that the first thing you need to do is keep the lights on most of us work in a server in a world where, if you're, off or even 10 seconds 30 seconds, our customers get really angry.

A

So keeping the lights on is a big priority and the second priority is in allowing those who have maybe a deeper understanding of the application logic or anything that's required to like anything associated there cause they can focus on fixing that, while other people are keeping the lights on, why this is even cool is that you can actually have a junior engineer who just joined the driver team. They can be apply in the maintenance mode. They can be traffic, shifting turn on and off the route.

A

Why senior engineer is debugging with the photo filters contact support? You can scale yourself and not have it be stuck to only one sort of team.

A

And with that, it was pretty short, but leaves a lot of time for questions so I want to say thank you, everyone and questions. Please.

A

A

Like automatic remediation, so his question was in our configuration. We had a backup and there's our concept in envoy. They can do automatic remediation between one and the other. We don't have that, but it is something you can actually program around. You can either do like consuming the metrics around like certain endpoint being wrong and then, after do a little bit of a loop in terms of updating the configuration, but there is no automatic way right now it something would be really cool. Oh.

A

I I turned off that one wasn't really at three am because I always forgot to take snapshots of 3m, but I did actually just tweak the phone Oh Matt I'm on my phone, but I have definitely been woken up at all times of the day.

A

Sorry say that again.

A

I'm sorry I actually can't hear you autoblog odd log, um that's apparently based on just how you actually implementing yourself weird like so that's where we'd have our logs with like we for our configurations. We actually put everything committed within a git repo, and we use that sha as our version history. And so then we can see what we deployed when we don't have it deployed. That's how we keep track of all that questions, anyone anyone outside no great thank you.