Cloud Native Computing Foundation ServiceMeshCon EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tap Tap, Debugging an App with your Service Mesh - Jason Morgan, Buoyant

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Tap Tap, Debugging an App with your Service Mesh - Jason Morgan, Buoyant

Microservices are great for all kinds of reasons but troubleshooting issues in a distributed system is rarely easy. Add to that that getting multiple independent teams to agree to use a standard set of metrics or debugging strategies is really hard. Issues often turn into a blame game where DNS, Kubernetes, the network, or the mesh all take a turn. With Linkerd’s built in ability to tap into, and analyze traffic we can quickly identify and isolate problems. We can do that with zero code changes, without needing app teams to expose their own metrics or become experts in Kubernetes or the mesh. When a problem pops up Linkerd users can rely on the mesh as a single source of truth to help quickly identify issues and drive down MTTR.

A

Hey folks, hello and welcome to servicemeshcon eu today we're going to be talking about debugging, an application with your service mesh tap tap. uh My name is jason morgan, this person here also that person there. I am a technical evangelist over at point. That means it is my job to talk to folks about the lingerie project, encourage them to use it and evaluate it and help folks as they move from development through to production. With linker d, you can find me on twitter at r jason morgan.

A

You can find me on github at jason morgan and you can find me on the linker d slack at jmo, so this is about the end of our slides for today, we're going to do everything as a live demo or as close to a live demo as we can get considering the circumstances.

A

So, let's talk about what it is, we're gonna talk about. um I've got two applications that are running in my kubernetes cluster, that are having some issues and we want to diagnose and try and do what we can to remedy those problems with our mesh. So I've got one application, which is a web front, end backed by two grpc services and another.

A

That is a series of web applications talking to each other over rest, emoji vote is, is the grpc application, so what it gives us is a web front end that displays a number of emojis allows us to vote on them view the leaderboard and see what the what the current state is of uh of the voting and it you know it seems to be working as designed, but we're having a problem with it.

A

I'm getting reports from my users and I want to fix it, so let's go ahead and take a look at the service map for emoji vote. So what I have here is a vote bot which generates some traffic. It talks to our web service and then the web service makes a grpc call over to voting or emoji and to be clear, I'm not getting that grpc thing from looking at this graph.

A

I just knew that already, because I work with the emoji vote app a lot, so I want to debug and I want to get things going so the first thing I'm going to do is just talk to my kubernetes cluster, ensure that I can communicate and that things are are running as as expected.

A

So I can ask about the nodes. I can see that I've got my three nodes in my k3s cluster, this k3s plus they're, provided by the good folks over at sivo, who are actually going to be announcing something here at cubecon. So please stay tuned for that.

A

uh Now that we've checked on our nodes, we're going to just check on the state of our mesh, so the linker dcli bundles in this check command, which will check on the health of the control, plane and all of its components, and it will also look at any installed linker d extensions and it will run their health checks as well. So we see that the results are green check, mark mark for both, which sure seems good to me, so we're going to move on and actually troubleshoot our application.

A

um So the first thing I want to do to troubleshoot my app is just take a look at the namespaces in the cluster and see what does linkerd see in terms of the golden metrics for those namespaces golden metrics being you know, success rate uh volume of requests and then um and then latency right. So we can see. We've got a bunch of namespaces here. Pod info is seeing a ton of requests, but is responding great and has a 100 success rate, linker d, viz and liquor d dashboard also both have 100 success rate.

A

One nice thing with linker d, the control plane components are also part of the data plane. So we can use the same debugging techniques we're going to use today on our various applications to check on the health of our mesh or debug issues. Should we run into anything now we have emoji vote and books app that are both seeing a sub 100 success rate. So, let's dive into emoji vote and see if we can't can't get this problem fixed.

A

So now that we know we have two namespaces with issues right, um books, app and emoji vote. Let's look at the deployments in emoji vote and see what their their relative health is. So I can see right away. Votebot and emoji seem to be succeeding. All the time have low latency, that's great, but web and voting are both reporting sound problems. So, let's dive in a little bit further.

A

So now that we've got the statistics on the deployment objects overall, we can actually dive into each one of those deployments and see what live calls are going in and what can they tell us about the health of the application? Now to be clear, I haven't instrumented anything inside these apps. These are normal grpc apps, I don't have tracing enabled or any anything in particular. This is what the linker d proxy is able to tell me about my application traffic natively with no configuration.

A

So I can see what pod a given request is coming from what pod it's going to the method as well as the path, that's being that's being called right and and the number of requests coming in. So I can see right away that web here. My web pod is talking to emoji and those should all be successful. We can see the call as list all and find by short code if we scroll over to the right. We see that those are both 100 successful, great in line with our expectations.

A

Based on the statistics we've seen, votebot talk in a web has two calls api list and api vote. So, let's see how they're doing api list is at 100, but api vote is at around 85 to 90, so it's not succeeding all the way and then webb is talking to voting a lot right and it's talking to voting and it's hitting these individual vote, these individual vote urls and let's look at their success rate. So it looks like it's good for most things, but vote donut right so donut.

A

What should be our most popular emoji is actually seeing a zero percent success rate great is actually seeing a zero percent success rate. uh So, let's go on um so we probably have. We probably have our our problem solved already or diagnosed, and what we'd like to do now is just just triangulate a little bit. Let's go check out the voting service, see what it reports.

A

So I do a top on that on that voting deployment right now, I can see that all the calls to voting like we saw in our traffic map are actually coming from the web service. Over to voting. We see the the api calls that they're or the paths they're taking inside the api, and we can see we can see how many uh and we can look at the success rate so so far everything seems to be going great right. We don't have.

A

We don't have any failures, but let's give it a minute and see what else pops.

A

Up all right, uh thank you. uh We have vote donut coming in starting to receive some some traffic and we see that it's actually hitting a a zero percent success rate.

A

So right here, let's go ahead and let's go ahead and check on that right. I can go over to my emoji vote app. I can click on donut. I see that I've got a 404, so we have a real problem, so I have more than enough to package over to my development team. So they know where to begin looking for the problem.

A

Great, so we've got that, but let's, let's continue to dive a little deeper. Let's see if we can't, if we can't grab some of that live traffic over to this this voting service and see what we get now. We've got the linker d vis tap command here and tap is going to snoop in on the calls between the two proxies and get a bunch of metadata around them, so that we can.

A

We can see the current state of our environment, uh so we we run our command and we can see that you know we have calls like this vote for running man. It's indicates that it's an mtls call, which is the default for linker d. When you install you get mtls between all uh all services, uh we have a status code of 200, which is great and a drpc status. Okay. So let's look for donut donut see if I type that right scroll around here there we go.

A

We've got a donut call here so now I've got a path of emoji v1 uh voting service vote donut, and we can see that while we still have that tls is true and the status code 200, we also get our grpc status of unknown, which is actually a grpc error.

A

Okay, well, the nice thing that we got out of that out of that last call is the particular path to look at for voting donut errors. So we we check out, vote donut and, let's just let's again, let's update the path that we're calling to. Instead of just the base url to the full voting service vote, donut see what what all calls come in now.

A

This is going to populate as soon as as soon as our traffic generator votes for donut or I can actually pop over to my web app and try voting for the donut again.

A

Great, so we see some of that traffic coming in.

A

We see some of that traffic coming in. We have still the tls true sasco 200 and that grpc unknown. So I could save this off and pass it to my developers, which would be handy for them, but we can actually take. We can actually get way more detailed information so that same tap command right, I'm going to run it again, but I'm going to change the output format from just that default terminal to a full json output.

A

So we'll see we'll see the same basic thing, but with a ton more detail and we could save this off bundle it with. You know with the message we send to our developers on the voting service and let them let them start solving the problem.

A

So here right we see the we see the output of one of these calls right now is jason, so I've got a bunch of metadata about the source, the destination. uh You know the the request, information and all the headers involved in this call all right great. I can also, if I look down a little bit further, I can look at the headers for another request and see that you know we've got our grpc status and we've got our grpc error message right there cool so save that send it to my devs.

A

I feel like I've discharged my duty to the team in terms of in terms of debugging emoji photo and now it's on the developers to actually get it fixed. uh So that's been great right, but this this troubleshooting process actually took longer than it needed to take right because I was looking around. I was waiting for things to come up. You all saw when I went to voting service. I didn't see anything about the donut until a fair bit of time in right, so I had to do that kind of aggregation myself.

A

um So one of the things that's really nice about linker d. Is you install it and you don't have to you: don't have to build or configure a bunch of custom resource definitions in order to make the mesh work. You get all the value with a very simple, install and and inject on your applications, but they're one of the two custom resource definitions that linker d does use. Is this service profile? So let me just create this and we'll talk about what we did.

A

So I use the linkery cli to look at the proto file for my emoji service. So emoji is a grpc service. It uses these protobufs and we can actually look at them and see what are the actual valid calls for this service and we get.

A

We get this service profile object, which will allow linkerd to do some more intelligent stuff with the traffic for this application, and it will be able to do things like collect and maintain the data about the given routes right there inside the service, so that we can more quickly debug issues like this when they come up, so I'm going to go ahead and create and apply a service profile for the emoji service.

A

I'm going to do the same thing for the voting service again, just using that just using that protobuf file and once those are done, I'm about ready. But there's a third service that I care about here, which is the web front end now. The web front end is not it's just the rest service, so I don't have a grpc file to work with or a profile to work with, and I also don't have a swagger file.

A

So I need to actually just take a look, so I can either write write my own service profile based on what I know of the application, or I can also use linkerd's tap functionality to in to to watch the traffic. That's coming to this web service and and decide how to build a service profile based on what we see so for the next 10 seconds. We're going to watch the web service see what comes in and auto generate a a service profile.

A

So we have. We have our new web service uh service profile created great we're gonna. I just give myself a reminder to go hop over to the dashboard and we're gonna take a look at how we can debug this even faster if we were using service profiles. So again, looking at the emoji photo name space, I can click on web. I can get my route metrics.

A

You know. I can see that you know, as we're coming in api list is staying at 100 api vote. Now again, it's only been a couple seconds, but over time we're going to see some of those failed transactions come in and this this api vote service is going to start to degrade right and if I'd left this running for hours, it would be a lot more obvious at what the failure rate actually is um so again gives us an indication that we've got a problem with with voting. So we look at our voting service.

A

And again right, instead of waiting for a vote, donut call to come in right which it did right away, but you know you saw it doesn't necessarily do that. We can look at our route, metrics filter on the success rate. So now imagine the situation.

A

I've been paged because there's an issue I come in, it's it's some amount of time later and I can just immediately look for the pass inside my api and see what's what is either responding slowly or is starting to see errors and then and then immediately go direct direct, the the right, uh the right ticket to the right team so that we can get get this thing fixed.

A

So that is emoji vote right. So that's that's as far as we're gonna go because we have some fundamental problem with our application, but now, let's change scenarios a little bit. Let's talk about not a grpc application, but instead an all rest-based application. So we've got our buoyant books, app and and books. Let's take a look at that inside that terminal. We're inside the dashboard real quick books is, is a bit more of a complicated service. Let's give this a little refresh. um Books is a bit more of a complicated service.

A

We have a traffic generator just like we did with emoji boat and three services, but web app is talking to both authors and books and books and authors are talking to each other. So there's some additional dependency there and my failures with with books are more intermittent, so it's a little bit harder to to get a sense of really where my problem is now. Luckily, in this case, I've already got I've.

A

I've gotten I've had the profile set up this entire time, so we're going to just use these service profiles, because these are all rest apis with swagger files, we're going to use those those profiles to get a sense of where our problem is and we're actually going to do a little bit to resolve it before our devs have to get involved.

A

So we start, we just look at the routes for the web app service right because that's kind of our entry point to this app. So, let's look at what's going on here, we can see a bunch of the calls are seeing 100 success rate, but post the books and post the books, editing and individual id uh are problematic. Okay, so, let's, uh let's see if we can't get this a bit more triangulated, I'm gonna see how is web app doing in its conversations to both authors and books.

A

So when I look at you know from this web app over to authors, let's see, let's see how how the routes look looks like every single call from from the web. App over to authors is succeeding 100 of time, so we're pretty good on that route great, but we still have problems with that app. So clearly it's a problem with web app talking to books.

A

So we look from web app to books yep there. It is uh postbooks.json and post book or sorry put on a particular book id dot. Jason is failing somewhere around 50 of the time, so we're already getting a lot closer to to our root cause right, and our goal here is to drive down that that meantime, the detection on the problem, so that we can get this resolved as quickly as possible.

A

uh Now I'm gonna look uh now that we've seen seen this. We also have that dependency between books, app and authors.

A

So, let's just let's just go in and check how what is books talking to authors about and how are those requests doing so now we look all books app is doing, is a head request on authors by a particular id and we're seeing that right about 50 successful, so should we clearly have we clearly have an issue and if we look right- and that was that was quick to diagnose with uh that- was very quick to diagnose with um with the routes in place, but you'll see it's a little bit harder.

A

When we look at the author service directly.

A

Right so we look at the live calls for the author service and we can see that that sometimes we're seeing we're seeing failures. But it's not it's not in a way. That's clearly aggregated right, I've got a particular author id that's seeing some some failures and and some that are succeeding right, but when I, when I view it from a route perspective where linker d's been been aggregating, the traffic based on the the api calls defined in the swagger dock. I see that you know the particular head.

A

The particular head request to any author id is failing about 50 of the time, so I've got like I've got the I've got the problem identified right and and that's good and and so we're we're there right. I can go ahead and alert my my authors, folks, that this head request is failing half the time they need to look at at the code, that's responsible for that response, but I can also because it's failing about half the time and succeeding about half the time.

A

I can actually use my mesh to solve some of this problem, so we're going to look at that service profile for the author service and we're going to we're going to change it so that we we fix the problem for tonight and they're able to take time and deal with it in the morning.

A

So our service profile has the various routes for this service and we're going to go into this head request that we saw before so head authors, id dot. Json is failing we're just going to go in here, and we know that it's safe to retry. This call so we're going to add a field that says is retriable is retriable. True, I'm going to hope that I type that correctly and and so we're telling the mesh hey. When you see these calls proxy go ahead and just retry it. So no app logic has changed.

A

No, you know no, no fundamental issues to the code or no. No changes of the code have been created right, but instead I'm letting the mesh take responsibility for trying to solve some of this problem.

A

And now we look at the routes from from books to authors right and we can see that the success rate, while the actual success rate is staying right around 50 percent, the effective success rate is going to steadily climb and it's going to climb all the way up to 100 because it through those retries it's going to it's going to succeed. Eventually we're also going to see our latency go up, but as long as it stays within reason. You know it's overnight, we'll we'll page them first thing in the morning.

A

They can respond to it in a timely fashion, so we'll break out of this.

A

And again now I can look at the routes from from the web app to books where we originally saw the issue and see what this looks like and now our success rate has gone up to gone up to 100 and we're feeling good about about that result.

A

So, let's pop back into the slides, let's pop back into our slides and that's the end of my talk, thank you so much for staying to listen uh like I said earlier, you can find me on twitter at rj's morgan. If, for some reason you want to see my github contributions, you can find me here and I'd love to hear from you over on the linkerity slack at jmo thanks so much and have a great day goodbye.