Cloud Native Computing Foundation KCD Washington DC 2021, 8 Nov 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Debugging an app with a service mesh, Jason Morgan

Description

Microservices are great, no doubt about that. But, when it comes to troubleshooting issues in a distributed system, it’s rarely easy. Getting multiple independent teams to agree to use a standard set of metrics or debugging strategies, makes it even harder. Issues turn into a blame game where DNS, Kubernetes, the network, or the mesh all take turns. Linkerd, the CNCF's recently graduated service mesh, has a built-in ability to tap into and analyze traffic to quickly identify and isolate problems. To do that, no code changes are required, nor do app teams need to expose their own metrics or become experts in Kubernetes or the mesh. When a problem occurs, Linkerd users can rely on the mesh as a single source of truth to help quickly identify issues and drive down MTTR.

A

So let me let me get started and share some screens and let me make sure I pick the right one, so you don't see all the other mess I have going on. uh Everyone see a terminal, hopefully I'm just gonna go ahead and assume that you do um before I go too far. I want to give you all a couple options, so I'm going to do a talk today where I debug some applications using linkery in a kubernetes cluster.

A

So what I'm going to pass out here is a couple publicly accessible couple publicly accessible websites that you can. You can take a look at and you can see if you can debug emoji photo or books app before. I do so feel free to ignore me and do that or follow along and let's get started um so just to kick off. We have an application.

A

It's our very important voting service. Where we decide what emoji is best so go ahead. Pick one um view the current leaderboard we see, there's been a lot of votes, so it's uh obviously a pretty big deal but fairly well distributed between our icons, um but I'm getting reports from someone that it's not working quite right, and I don't know why, because everything kind of looks up.

A

So let's do some debugging folks in the backstage or general. If you could put in the chat uh how you feel about the size of the text, that would go a long way and thank you so much so we're going to start right, I'm just looking at my kubernetes environment and let's get uh let's get some notes.

A

So right now I'm running an in-memory cluster. On my on my environment, uh I've got I've got the one component and it seems to be healthy. So that's good. So kubernetes is all right. uh Let's just check the state of my service mesh with the linker dcli, uh so I'm gonna run through a bunch of tests to validate that this is in a in a good state. Looks like hold on. I've got my handy dandy, laser pointer uh one more. There we go looks like all our status checks are.

A

Are good, so linker d is healthy.

A

My app seems to be healthy, so, let's, let's use linker d to narrow this down a little bit, so you can do this via the dashboard or via the cli, I'm going to do it via the cli and then we're going to pop into the dashboard and see some of what what we're doing here. So I'm going to start I'm going to use the linker d visualization extension, to give me a sense of by namespace. How are we doing in our in our key golden metrics about this application right now I want to.

A

I want to be clear here. This emoji photo application has not been specially instrumented to work with my service mesh. I have not built in any metrics or monitoring into my app all the all. The data that we're going to see about this environment is data that the service mesh generates for you automatically without any changes to your application code.

A

So we kick off. I can see that I've got some namespaces. I've got books, app emoji photo linker, dvis or linkery and linked evs. The key things I'm seeing here are of that namespace. How many of the pods in that namespace are part of the mesh, that is, they have the linker d. Two proxy are very small proxy written in rust, running beside them. um So all the pods and all my name spaces are meshed. So that's good!

A

I can see a success rate that is problematic right, so that is of the calls that are going on in my environment. What percentage of those are returning a successful status, whether it's a grpc status code or an hcp success response right now, I'm seeing in emoji photo 95 are successful, whereas in all name spaces it should be 100. I shouldn't be seeing these failures.

A

I can also see the the saturation at the time, as well as the latency by latency bucket right, so we're going to go ahead and debug and figure out what's wrong with emoji photo. So let's join me on this journey or again go to that link that I sent at the beginning. Let me just make sure that no one's saying anything weird in the chat um yep go ahead and link that dashboard.sivo.59s.io to go and solve this in the dashboard.

A

So I'm going to go now, look at the deployments in my emoji photo namespace and just see the the statistics there.

A

Once again, I'm seeing the same the same basic information, but now narrowed down inside the namespace to a certain set of deployments.

A

So of my of my components here, I've got a vote bot, uh a web service, a voting service and an emoji service. We're actually just gonna pop another dashboard and see what that what that looks like. So, if I go here and I go into my namespace, I can see a little graph at the top any second. Now there we go a little graph at the top. That shows me how these components are talking to one another right.

A

So it's something that you can't see as easily with the cli obviously, but once again looks like web and voting are having problems. Webb's, the web's, the front service or the first service that's getting hit as part of the traffic coming in here.

A

So, let's go and take a look where what I want to do now is: I want to see inside web what paths in my application are being are being explored and what what do I know about it right, because I want to see what's wrong with web right, because it's starting to see problems. So how do we? How do we get this fixed?

A

um So what I'm looking at here is there's a lot of text on the screen, so I'm going to try and make this a little bit a little bit simpler. The source is what pod is a given call coming from a given. um You know call from one application to another on a path inside their api. So what pod is it coming from? What pod is it going to?

A

What was the method if I scroll over to the right, what actual call is being exercised within the api and then how many have I seen you know? What's the response time and what's the success rate, so I'm looking right, I should expect that calls calls from web to emoji are successful, because again I saw that the emoji voter wasn't having any issue or the emoji component of my app wasn't having any issues but um but web to voting is having some problems.

A

So if we scroll over here web to voting, we see that you know exploring this. This general api vote uh is only hitting an 85 success rate and hitting the voting service and going to vote. Donut is zero percent successful. So if you go to that, emoji photo app that I put in the chat at the beginning. Go vote for donut you'll, see that that something is wrong so already here, I've got a sense.

A

uh I've got a sense of of what's going wrong, but we want to want to dig a little deeper. So it's it's one thing to know: okay, there's something wrong with voting for donuts, uh even though donuts are the best they're, not getting their proper representation in our application.

A

So, let's, let's go look at voting right and make sure that we can confirm our finding here.

A

So now, once again right, I see the source now for for our voting service right. If we remember from our little graph, our voting service is, is kind of a dead end in the path right. So we go from vote bot to web and either emoji or voting right, so voting's not calling to anyone else, and we look at our terminal and we see everything's coming from the web pod to the voting pod.

A

Okay, we can see we can see what api calls, what api paths are being hit and what the success rate is and then eventually, if we wait here long enough, we're going to see that some sometimes donut is being voted on and it's not succeeding there we go.

A

uh So if we go back, we've got vote donut and a zero percent success rate. And now, even though you know, we've tried to vote for donuts three times five times. Oh six, thanks anybody who's voting on it appreciate it um that that voting is, is not working correctly. uh Great.

A

So that's cool right, so I'm like now totally sure totally sure that um totally sure that I know my problem, but let's, let's actually again gather some information that we can use to send over to our application developers right. So I'm not I'm not the application team, I'm the person, I'm the platform, engineer I'm responsible for this environment, so I want to give them the information. So they can go solve this thing as quickly as possible.

A

So I'm going to use linkers functionality to do a tap right in the top top pick of the or the the title of the talk is tap tap right. um How do we? How do we diagnose our apps with linker d and tap- is a a vital part of that right? So tap will allow me to look at the calls from the web deployment uh to the voting deployment right and right now, I'm just going to look at the at the base path here, I'm sending it to less so we can see some stuff.

A

So there's there's a lot going on right in terms of in terms of calls in my environment right and I want to look for donuts.

A

Next come on buddy you got to be in here somewhere.

A

You know I just never seem to get this working right. uh I want to look for calls to donut really thank you. um So I want to look for donut um and I'm having a ton of trouble getting less the work here. So we're just going to quit out and we're going to refine my query and look for what's happening with the donut, so I'm going to do that same tap command.

A

Where I get, I get the live traffic between my components right, but now I'm going to narrow down my my search string from the base path inside the api, which is everything over to the actual voting for donut and we'll see what calls are coming in. um What calls are coming in and what's happening with them. So here right, I've got a request: information. I've got source information, destination information.

A

Yes, it's tls, because if you're using link or d every connection is mtls every single time, I can hear the I can see the method um there we go. I can see the method and I can see where it's going to right and I also get the http status code is 200., so this looks successful from an http perspective, but when I dig in a little bit deeper, the grpc status code is unknown, which sounds like it just doesn't know what's going on, but that's unknown error in grpc. So this is our application.

A

Throwing a grpc error, which linker d is detecting telling us is a problem, and now you know I've got some details. I can send my developers and they're ready to go: go fix emoji, boto yeah, but wait there's more doing my uh who's that dude the slap chop guy um doing my best snapchat chop uh we're going to take the output instead of just seeing that little text string, let's get some json output so that we can. We can get a lot more details now. I can save this off to a log.

A

Tells me everything I could want to know and a lot that I don't particularly care about about this interaction between my applications. Save this off. I give you the api path. I give you some examples of the metadata around the calls between your components and you've got you as a developer. Now have every all the ammunition you need to go figure out. The problem inside your application code and there's no arguing hey is this: the service mesh is this the platform? Is this my downstream app right?

A

We can see where, where the problem is so that's, that's solving emoji photos problem right, so here I'm done with my my original goal and where am I at on time, uh 50 all right sweet, so I still have a little bit. This is good, but we could solve it faster next time right. So, if you go look, if you go look at the environment right and you go into voting right, so it's cool to see it's cool to see the live, calls right, but what you're?

A

What you're at the mercy of here is what gets what comes through as you're watching right or we could tell link or d hey, listen, linker d! This is how our api works. I want you to store, metrics and store data about these individual routes, and I want you to display them in a way that makes sense to me. So this is the default generic storing of data, but we're going to improve that using using service profiles, so service profile is a way. This is one of linker d's, very few custom resource definitions.

A

Now to be clear, you can use the linker d service mesh in your environment without having to use any custom resources, so your application that works in kubernetes continues to work in linkedin natively, with some enhancements that it didn't have before, um but we we now want to teach. We want to teach our mesh something about our in this case our grpc application. So I'm going to take I'm going to I'm going to use liquidy and I'm going to run the liquidy profile command and I'm going to look at the protobuf file for mojiboto.

A

So that's this. That's this definite api definition file tells grpc applications how to talk to each other, um and I'm going to build out I'm going to build out a service profile for the emoji service.

A

What we create is a new custom object for linker d. uh It's a service profile. uh We give we give basically a definition of all the routes and what what things are safe to do with them right. So here I can. I can begin applying this.

A

Right so I can create this file. I can send it over to cube ctl, apply and I'll I'll, create my I'll, create my service profile.

A

Do it again for the voting service.

A

All right, so that's good for for our protobuf, uh our protobuf files and the apps that we have it, but in our case web service it doesn't have a swagger file. It doesn't have a protobuf file, it's just kind of a very simple web api that we wrote specifically for the demo application. So what I'm going to do is I'm going to create a profile, but I'm going to do it just by watching telling linkery to watch the traffic that's coming in and use that information to create a profile of it.

A

So let's do that so we've got. We've got 10 seconds of of me tap dancing here. While it goes and views what calls are coming in the web over the last 10 seconds and then uses those to build a service profile which just finished, creating let's go check out our dashboard. So excuse me this is going to be small. Sorry about that, but I'll I'll fix that or well it doesn't really matter you're, just seeing you're just seeing a launch of the dashboard.

A

I don't need you sorry a lot going on here um here. I've got my other in-memory lingerie instance. Now I can look at emoji photo and what you're going to see on this instance that you won't see on yours is: if I go into web right now, I can go to route matrix and I can see what does it know now about the particular api calls right if I want to build in retries or I want to build in timeouts or anything like that. This is where I set it up.

A

Right is inside these service profiles that make sense.

A

um Cool uh now, let's go to the second half, so that was that was debugging emoji photo. So that's the core of the demo, and this is how we debug an application with um with linker d. Beyond that, what I want to show you is: how do we go beyond debugging using service profiles? How do we debug fast there and then beyond that? How do we actually solve our problem?

A

So I'm going to start so I ran this command and I'm actually going to need to make it a little bit smaller. So I ran this command, which is basically instead of that. That thing I saw about give me the stats by deployment or namespace or whatever it may be right, I'm going to actually ask for the routes right just like we saw from that back tab. What are the routes in our environment for this books, application, which is another one? You can look at in that terminal that I sent so here's books app.

A

We can see that you know posting to slash. Books seems to be failing about half the time that sucks posting for a particular book id in an edit also seems to fail about half time, but our latency looks good. So you know, what's going on, we've got. um We've got some some issues that we want to solve.

A

So now uh let's go and actually let me let me step back and stop rushing for a minute. um It's worth it to look at the structure of the book's application right so with with emoji photo. We had a traffic generator a web front end and then two back ends that the traffic ended at the individual back end here right. What we have is traffic a web front end and two back ends that talk to each other right. So there's there's a little bit more complex stuff in here about the interactions between our applications.

A

Let's pull this over try and get to look a little nicer, so there's a little bit more complex interaction. So what we asked for was, I believe, web app to books so tell me about the routes between web app and books. That's what we told linkery um and it gave us some information right and now we're looking at or right, we're asked from the routes for the web app uh now we're gonna ask for the routes from web app to our author service, and I wish I'd muted my emails as well.

A

um So here I can see you know to the author service. What do we have going on looks like everything from the web. App to authors is happening without any issue, so that that's a good sign that our problem is is with books or with web apps somewhere or the interaction between authors and books.

A

So do the same thing that we asked for before, but now we want to look at when you're talking to the book service what's happening when we call the book service right, most things seem to work but posting to books.json and putting to books. You know whatever the id is dot. Jason is failing about half the time right so that sucks and um yeah that sucks and we want to find a way to little thing popped in the wrong spot. We want to find a way to resolve it.

A

Right and just one more we're going to ask from books to authors, what's going on right, so we got web app to books. Now, let's look book books to authors. Well, books talking to authors is seeing this this request. The particular author id fail about half the time.

A

So our problem is right: the author service, let me go back and show this in the in the ui, make a little bit clearer. The author service.

A

We go to our routes.

A

Oh right, sorry, this is the wrong window. This one's got it uh our author service.

A

So again, if you're, if you're on that that um that link that I sent you there um you're not going to see these profiles because the profiles are being built in a different linkerity instance as part of a cluster, but those ones that that I put in the chat those are open and available for use anytime, but my environment, I go look at books app. I go check out the author service and I click down here on my route table right. I can see.

A

I can see specifically what what call is having the issue right um and now I can. I can begin to solve it so long story. Short authors fails about 50 of the time, but I know that this this request here is safe to retry. So what I'm gonna do right? Maybe I got paged it's like two. In the morning I have intermittent failures with my author archiving service: okay.

A

Well, I'm just going to fix it at a platform level, a platform engineer perspective, we're going to fix it and then we're going to let the then we're gonna, let the um we're gonna, let the application team come in and deal with it in the morning.

A

So what I'm gonna do is I'm gonna edit the service profile object and I'm going to add in retries and those retries are going to. Let me are going to. Let me fix the issue and I'll show you how how we'll fix it and what that will look like using linkedin reps.

A

So here I've got my service profile object. I've got on each one. I've got the individual wraps or reps that are routes that make up that application, and my problem is here with this head author's id json right. This request fails about uh about half the time. So, okay, let's let's solve it right now!

A

Well, let's solve it from a platform perspective or band-aid. It that's a better one. Let's put on a band-aid until the morning when the actual application application developers come in, I'm going to say, is retriable true, I'm going to save this change.

A

Service profile is going to get updated and suddenly what we're going to see is these requests are going to are going to go from failing to succeeding, so I'm just going to watch the routes for books app. We need to make this smaller.

A

Can you all still see this? Okay?

A

Can I get a thumbs up from folks watching chat?

A

um Okay, what we're seeing here is the success rate for this path go from about 50 and it's slowly going to climb up to a hundred right. But, as that happens, I'm also seeing my latency increase right, so it's minor, but it's going to get it's going to get a bit higher.

A

I can also add some things like hey, let's put in a timeout, to keep the latency down, no matter what happens uh with these failures, I'm already up to 100, so I should move down fast there, but we're going to now see if I look at my other routes, we're going to see the effective success rate for these components all go to 100 right, which is good right and all we're watching now is. Does the latency?

A

Does the latency get too high right? Because that's my that's my concern now that I've addressed the failure issue. So if you're a user using my app, it might sometimes go a bit slower, but you're not going to see the thing just give you a random failure, and that is the demo um yeah, that's the demo. I appreciate you all taking the time to watch.

A

If you want. If you have questions you want to poke around, I've got the environment up and running. So we can. We can look at things, live and and poke through here and, if you're interested go check out, uh go check out this site to see a live, linker d dashboard that you can mess with.

B

All right, thank you. So much jason. That's.

A

uh Yeah sorry, one more plug before I go uh check out. If you need it link, are you 2.11 not getting started? This is interesting. You want some of this in your environment, check our getting started, guide and see how you can install a linker d service mesh in your environment in 10 minutes or less.

B

All right, awesome, okay, so um I do have a question. I saw multiple numbers here on the board. The p99s and latency.

B

Is there a way I can tie this into some kind of an sla um breach or something so that you know I can. I can use this not just to monitor, but also proactively trigger an alert or page of duty. Saying hey, you know we're breaking this a layer. This is happening and you know we're almost there at 90 percent.

A

Yeah great great question right, um so, oh yeah, let me post the getting started link in the chat. Sorry about that uh so great question. So linker d, the open source project, linker d, uh comes with a built-in prometheus inside the linguity dashboard component. You can configure that prometheus to do a bunch of stuff, so prometheus is a powerful tool. Linkerity is a powerful tool.

A

You can get the data, so linker d's, individual proxy endpoints expose data in a format that you can set up to be scraped by whatever your preferred monitoring solution is right and pull it in right and I know datadog's got an existing integration. So, if you're looking to go that route, you've got it right. On top of this right, the company that makes linkery buoyant has a commercial extension you can put on top of linker d.

A

That will give you the ability to set up slos on your various services and set up alerting, that's already, obviously well integrated, but there's there's a ton of good options, including just using prometheus and setting up alerting through that.

A

Does that make sense.

B

It's awesome. I did see that grafana. I was just wondering if there is something I can set up alerts there. Yeah.

A

Okay- and it's really easy, if you already have prometheus and grafana, it's really easy within linker d to use your externalized one, there's a ton of really good documentation on it and if you go to the linkerity slack community. So let me just add that slack.linkerd.io.

A

Where are you? If you go here, you can join the linker d community. If you have questions about anything open source that you want help with pop on in tons of folks there to help you, you can find me at jason on there and you can find a bunch of other folks as well and looks like we've got. You know close to 7 000 users registered at the moment.

B

All right, awesome, yeah. I see the link there as well, and one final question um was regarding. So if there is a service, that's not running in the cluster, let's say that I have a database, that's like you know, running in rds or something and um running link id in you know. um Eks does that kind of show any information any any debugging information that, oh, my queries, are failing or or this is the latency with the database as well.

A

Yeah, so that's a great question so whether actually, whether or not your database is in or out of the service mesh linker d is going to show you http and grpc metrics for your application. We'll give you a lot of details on that for a connection to a database. The best you're going to get in linkerity is just a raw tcp connection data right. So if we look down here at some tcp metrics you'll see again, you know how many connections are there. What's the data read and written, and that's it right um yeah?

A

So that's the like that. That's long story, if you're off cluster, so linker d is a kubernetes focused service mesh. It is not does not work on your rds instance or anywhere beyond that right. So you're not you're, not going to get tcp metrics for a connection to an external database.

B

Thank you so much all right.

A

Yeah and and brandon, if you, if you want that that links available, if you go into, if you go into one of these um one of these namespaces here, specifically the emojivota1, if you click on something like the web application, sorry brandon asked if we could dig into the individual file inquiries.

A

So if you click on web or voting you're going to see down here right as you scroll down you'll see the live, calls that are coming in to this application, then you can do something like sort by success rate and see exactly what what path in the api is being exercised and what the failure is as well as how fast it went. How many times it's been hit. All sorts of details like that.