Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 4 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Observability and Debugging Cloud-Native Apps using Linkerd & Conduit - Franziska von der Goltz

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Observability and the Depths of Debugging Cloud-Native Applications using Linkerd and Conduit - Franziska von der Goltz, Buoyant, Inc (Intermediate Skill Level)

Observability and monitoring are different, but complementary, needs for production applications. While monitoring focuses on measuring the overall health of your systems, observability aims to provide granular metrics that describe the behavior of your systems along with their context. Due to both its architecture and focus on providing observability, the service mesh can be a powerful debugging solution when trying to figure out what went wrong with your cloud native applications. In this talk, Franziska shows you how to debug your Kubernetes applications using features of a service mesh in tandem with use of other CNCF projects like OpenTracing. She compares and contrast two different debugging techniques using both Linkerd and Conduit. After attending this talk, you’ll be able to use either approach to debug your Kubernetes-based applications.

About Franziska
Franziska von der Goltz is a software development engineer working on open-source service mesh technology at Buoyant. Prior to Buoyant, she was a part of the education team at the Hackbright Academy in San Francisco, an engineering school for women with a mission to change the ratio in the tech industry. When she’s not writing code, she loves traveling, enjoys tea, and worries about speaking at conferences like KubeCon.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

All right hi, thanks all for coming to my talk: observability and debugging micro services with linker D and conduit I'm, Francisca I'm, a software engineer at buoyant and I work on service miche's, most particularly linker d, &, conduit I, think we can all agree on that. Debugging micro services is really hard, and that is because also services are owned by different teams and the application logic is split across service boundaries.

A

Services are comprised of many running instances and those instances are potentially being rescheduled at any moment without any advance notice, and as a result of that, the tools that used to work for debugging monolithic applications don't really apply in this new micro services architecture, and we see a new set of tools popping up at every conference. We go to where you have to find something that helps those debug our applications.

A

So we've got fluent e because we used to get away with just adding logging to one of our instances and we kind of get a good idea of what what was going on and with this new architecture. This. This doesn't really work anymore because the logs are distributed. Among many services, we've got Prometheus and we have open tracing because, with you used to be able to see the life of requests through one instance- and this also changed with micro services, because each service calls another service and so fluid, as so open tracing kind of lets.

A

Us trace the services from start to bottom.

A

One of the words that keeps popping up and trying to help with this new. This new architecture is the service mesh. We still get a lot of questions on. What is the service mesh? How does it help me? Do I need it. What can I even do with it like it's becoming this word, but there's it's not quite clear yet, just because it's really new, like what does it all entail.

A

So I want to take a couple minutes to go over that and talk to you about what a service missha's, how it can help you to debug your applications, so service mesh is a dedicated infrastructure layer and is, and it's adding reliability, security and visibility to a cloud native application, and it does this by managing the service to service communication at runtime.

A

It also provides many other features, such as circuit, breaking and request, routing security policies, etc. That are the focus on today's talk, though, is how to use a service mesh to debug a micro service architecture.

A

With that, I want to talk a little bit about what what consists a serviceman, what consists of service mesh of, and we have a control plane in a data plane and the data plane is usually implemented in a light lightweight array of proxies. We can see here if we're talking about a community specific application, we have all the applications in a pod and then we have as a citecar injected the blue squares which resemble the conduit proxy or the service mesh proxy.

A

Sorry, I'm gonna talk a lot about cotton hood later on yeah, so they're all connected, and then we also have the control plane and the control plane is kind of the users gateway to the hot path. We're where the proxy sets in it's. What you, as a user, interact with to get. You know to get a UI to get to see the metrics to see to set up policies that you're going to deploy.

A

Now I want to talk about the life of a request like one one request from service a to service B when it's being routed through a service mesh. So here we have to pause and we have service bar and service. Foo and bar is calling service food. But what happens is that bar calls the proxy that is, that is deployed alongside its pod and then that proxy calls pod to the proxy that sits within pot two and then the proxy impart two forwards it to the food service and because of where the proxy is positioned.

A

In this request, it knows a lot of things. It knows the latency of things as a Nick. How long did the request take to get from bar to foo? It also knows whether it's exceeded or whether it failed. It also knows how many of these requests happens over a recent time.

A

It knows the origin, like the identity, what Ric, who sent the request and who is the receiver of the quest of the request? What instance was the receiver of the request?

A

It'll? It knows a lot more. It knows whether or not it was retried. What instances went to as I just mentioned, and many other many other facets thanks.

A

So now to the question of how we can we use a service mesh to help us debug our application, and that is because we're the proxy sits. It gets a lot of. It gets all this information and aggregating that and using. It allows us to paint a really really specific picture of the application, and that allows us to really easily display top line.

A

Metrics such as success, rated, the latency distribution and the request volume of each of the services that is running on a service mesh and that is tied to identity, including the caller and callee relationships. Knowing who who sends a request to whom.

A

Servicemaster bugging or micro-service microservices debugging is fundamentally different. They're, like a different class of Diagnostics information. We can know- and we want to know of what is the success rate of the food service which service calls foo, which services plural. They might be multiple.

A

What is the success rate of foo when it's called by a specific service bar in this case, because that might be different than a third service? That's also calling through and, of course, what's the request, volume, latency distribution. All these things are very important to us as services and with a service measure. We can do all of that without having to modify our application at all, doesn't require any changes into application code.

A

So I want to talk a little bit about conduit, which is the demo that I'll be running in this. In this talk as well and conduit is a completely open source service mesh license under Apache 2. We built it from the ground up for kubernetes and for that reason, because we wanted to be able to deploy it as a sidecar, which is what makes kubernetes so powerful and so great. We made it ultralight and ultra-fast, and what does this mean?

A

It means that we built the data plane is tiny, tiny, rust proxies that, with a lot of requests, only take about 2 megabytes of memory per pod, all that, while only adding about 1 millisecond latency in a 99th percentile.

A

Well, if we link back to the diagram earlier, we have a control plan and we all that with go-to, really seamlessly integrate with the kubernetes api, because that's where we're extracting a lot of the information from it also supports h2h, 1g, RPC and tcp protocols, and we just released a version over 1 alpha last week.

A

The goals we had with conduit are like service measures are a little complicated to kind of get on like what are all the things that I need to set up, and we wanted to have a zero config service mesh that we could offer. That will add reliability, security and visibility for kubernetes, specific applications.

A

Alright, it's demo time, if you want to follow along here or later, I'd, encourage you to already install the conduit CLI, which is easily like you need to have access to kubernetes clusters, so you can just run it in mini cube. That's what I'll be doing and you just have to curl conduit, don't I'll, install and pipe it into your shell and then copy it into your path, but that's kind of all there.

A

All the instructions are are there, and so what we're gonna do is we're gonna install an app into a kubernetes cluster, just just plain old, plain old app and it's gonna be filling some of the time, not all the time, just some of the time and then we're gonna try to use the kubernetes dashboard to see if we know what's going on what like, if we can get any indication of why our services are failing, but we're not gonna have a lot of luck with that.

A

Unfortunately, and then we're going to install the Condor control plane following by installing the conduit data plane into the application without having to take the application down, just be a rolling update and then we're gonna use conduit to trace the source of the error across the application. So you can see what is going like to kind of identify the failing services and the failing edges.

A

Now am I gonna be so happy cuz. We know who we need to talk to.

A

So this is an overview of the app that we're gonna, deploy. It's an emoji voting app, it's very it's very, very complicated topology I'm kidding. So it's a forced service, app. It's a web service that talks to an emoji service and a voting service and the voting service in turn then talks to the emoji service to actually do the vote. We also have a vote bot that just is there to generate traffic, so we have to click ourselves. The entire time.

A

So if you decided to follow along I hope, you got to the point to installing the conduits yay and then we're gonna we're going to go through the next steps of installing the control plane and actually injecting them together, like I'm. Also gonna. Do this life here.

A

Let's hope this works.

A

So I already installed the emoji photo app wrong button and it allows us to vote for emojis, sometimes at work, and sometimes it doesn't even the same emoji clicking multiple times just starts working again this it's not very clear. What's going on right.

A

So we're trying to get our pods.

A

We can see that we have the emoji photo app deployed here and then the regular thing. So it's it's, it's it's working it's running and if we.

A

Open the mini cube dashboard in the emoji photo namespace we don't get any indication of things are wrong: everything's I, don't hundred percent, we think is great when we click into one like it's all good. This is just this is because all the pods are healthy, like everything is good, but we still see errors right and and kubernetes at that level doesn't know why, like the calls are being made, the instances is up and running, but something is going wrong somewhere that we're not just not quite sure how to figure out.

A

A

So now we're going to install the conduit control plane and, as mentioned earlier, we just have to run conduit, install and then we're just gonna pipe this into a puddle apply.

A

We can see that there's a bunch of there's a namespace conduit that was created. So it's not it's not doing anything in the default namespace or anything else, and we can see that we have a bunch of deployments and services that are being created so conduit also ships with a with a UI, and if we open the UI, we can see that it's telling us the service mesh was successfully installed and it detects the deployments here. But it's also telling us that we haven't actually connected any of the deployments to the service mesh.

A

So when we go into deployments, the links are grayed out and we don't see any success rates or request rates, because obviously we haven't actually injected Roxy.

A

So when we do that, this is the next step, we're installing the conduit and what we're doing is we're getting the emoji vote. Aoyama file- that is anyone following along that is actually doing this and wants me to pull up the bitly URL cool.

A

So we're piping the output off just the Yama file into conduit inject and then the output of that just into coop cuddle apply.

A

And we can see that a bunch of forest services remains unchanged, but the deployments were configured. So what is going on here behind the scenes is that conduit is taking the Yama file and is transcribing it and rewriting it and actually modifying it to add the conduit proxy in the Yama file, so the user doesn't have to worry about. Like did. I put this in the right spot is like the poor. The correctly this is all being attracted away from the user, so.

A

For looking at all the namespaces again, we can see that we've got some terminating and running again and the conduit the conduit namespace, is still up and running the emoji voto namespaces. There is a bunch of changes being made at the moment.

A

Now all the pods are up and running again and we haven't actually need it to take down anything. So if we run conduit dashboard again, we can see what changed so now we can see the banner that we have, that told us. You still have not connected your deployments, it's gone. We can see all the deployments here and we can see that the proxy status is green and if we go into the deployments, we can see a high-level overview of the success rates, the request rates and the latency distribution.

A

And then, if we decide that we want to take a little though we want to take a look and check out, you know the emoji service. It actually opens a pre-configured graph on a dashboard that shows us the success rate, the request rate, as well as the inbound deployments and outbound deployments. But since it's a it's a leaf service, it doesn't actually call any others just there, and here we can see the voting and web service as the involved deployments.

A

If we just want to stay with the dashboard, we can actually just we don't have to go back and click on it all of them ourselves. We can just check out the voting service and check out the web service right there.

A

So if we want to use the conduits a conduit, I, don't know if I mentioned this earlier ships with a CLI that also allows us to just dive into all the all the things that we saw and more, and so, if we just run conduit, stat.

A

For the deploys, we see the same thing, we see the name of our services and we see the success rates and the response rates and Layton sees across the three percentiles. You might wonder why the vote bot is just showing dashes. That's because they're incoming requests and the vote bot is just generating traffic.

A

So there's not really anything that they're successful, but you know they're not being called okay, and we also see that really none of our services are performing that well, like the highest success rate, is less than 82%, which, as we saw earlier, using the app like it's it's, not a good experience. You don't want your users to experience failures constantly as well as he also don't want. As a user you're not gonna go to a site where you're just clicking you're like ah does it work this time?

A

Does it work this time and sometimes and like it just seems completely random? You don't know, what's going on so we're gonna start so now we can actually just look into specific, deploys and see what services are there talking to we're going to start with web, just because it's the user facing one? That's where we clicked it's like. We know we know web is running, but we just kind of want to know more.

A

So we're just gonna stat and we can just add a from flag and pass it the deploy name- and here we can see that yes, web does call emoji the emoji service and the voting service, as we saw in the topology graph earlier, and we can see that the emoji service has a hundred percent success rate, but the voting service there's only about 55%, so that would give us so we'd be like okay. Well, what about the voting service? It kind of seems like the logical next thing, to look at.

A

And the voting service also calls de mogi service, as we saw in the topology graph earlier and again, it's only about 50%, 52 percent success rate.

A

Well, what we will want to do here is you probably all have the graph in your mind and it's a it's a very small application, but we want to look at what are all the services calling the Emoji service because there might be there might be others, then you know just we just want to know how are all the services performing that are calling our emoji service and the way we can do that is, you might have guessed it. We have a two flag, mm-hmm, no, so creative, so yeah.

A

Here we can see the emoji service is called by the weapon. The voting service and the web service has a hundred percent success rate. Yes, it's displaying all the emojis, but the voting service only has a fifty to fifty three percent success rate again, so we we identified the edge a little bit of like what something is going wrong when the voting call service calls the emoji service we're not quite sure what yet.

A

But if you think about having an application with hundreds of hundreds of services, now you at least know like what edge like what kind of two services do. You need to look at which already narrows it down a lot more than when we just looked at the dashboard, and everything was green, but we were clicking on emojis and it just wasn't working.

A

So what if we could just look at all the requests happening right now from one service to the other? We can, you know, surprise and the wonderful command that lets us do. This is to have tap, allows us to just tap the request as it is happening and we can tap from deploy, which just means all of the deploys talking it to a specific service. But we already identified that it's that the error is happening between the voting service and the Emoji service.

A

So we're just gonna set the voting service here and we're just gonna tap from voting to emoji.

A

And here we can see all the requests that are happening, that the volt bot is generating. At this moment.

A

That's pretty cool right, I think so.

A

Okay, so let's investigate this a little bit further here we can see that the destination is the emoji service and it's h1. It's a get method. We can see the path that it's that it's calling its node face, emoji we can see the latency and it's returning a status 200. So this seems to be all like. It was a successful request, and here we can see that we have a 500 error and we can see that the it's also calling the emoji service, which all be like.

A

We set it to the flag to only call the emoji service from the voting service, and we can see that something is weird in this URL I. Don't know why there is a fail, endpoint I don't know, but there is and we're getting a 500 back. So we know that there's this there's this one endpoint that is being called for rather reason and it's returning a 500. We actually see quite a few of them.

A

But ghost emoji works, so that's cool, so now we see all of them and we might just want to I just want to know what about all the failures like is there? Is there some sort of? Is there some sort of pattern that we could match? Because you know this is this? Is a small demo application, but in real life application you have more, you might have more than one endpoint in that service that is being hit, so we can just grab for the 500th since it's a request.

A

I also need to get the line up before and the line after.

A

So if we tap again, what this will do is actually just log us or not log, just print us out the the failing, emoji requests.

A

We can see here they all have a status 500 and they all call this fail endpoint.

A

So now we know now we know what what team we have to go to right. We know that we have this team, the voting team that is calling a fail endpoint in the emoji team. That's returning a 500, so there's something there's something: the team communication that that that we'll need to work on internally, but we can yeah. We we know where to look. We know we have a lot more visibility than we had just before.

A

When you get a phone call, because someone was trying out the application or customer support was like well, we had someone calling and like it didn't work and I tried it and it didn't work and I. Don't know why, like within a few minutes of just not having conduit installed, I, have to CLI installed but not having it in our cluster at all. Now we can see like we have narrowed it down pretty pretty rapidly.

A

Okay, so just to iterate again, because I know there was a lot of commands that you sweat li class, just want to kind of talk about what we just did. We had I already had an emoji photo app installed on kubernetes without any service mission ejected, we saw errors when we were using it, but we saw nothing on the kubernetes dashboard because it was just our services were healthy, but something was was wrong within them. Then we installed the conduit control plane and our winds.

A

We installed conduit the control plane and then injected the the data plane and we used. Then the combination of the stat and the stat commands to identify a falling call deep within deep within our applications, topology, and then we were able to go to the correct team we liked, and we did that all why well not touching any application code. We didn't need to add anything in any of in any of the applications. We could just leave. Our editor closed.

A

So in conclusion, debugging microservices is very, very fundamentally different from debugging modeling applications. We all know that I think that's why we're here and the service mesh is really uniquely positioned to provide that debugging information at the right layer of abstraction, because you don't want to be overwhelmed with information.

A

But if you think back to you the example request from the service bar to service foo, where it sits, it's just really uniquely positioned to extract all the information that we want and it's super easy I just got this and besides us a few type errors like it all went and we didn't have to modify our application.

A

I know I said this a couple times, but I think it's really really important, because having all of this in the application code means you have to also update it and in every application and then adding things to it like this allows us to just leave the application code untouched.

A

So try it out. We have. We have a booth that you can come visit if you want, if you have more questions or if you want to get started or run through a demo, we also have a great getting started guide on condo that you can just follow that it walks through the some of the same requests at some of the same commands that I went through today and also Condit is 100% open-source and it works with kubernetes version, eight one, eight and up so thank you very much for coming.