Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Understanding Microservices with Distributed Tracing - Lita Cho, Lyft

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Understanding Microservices with Distributed Tracing - Lita Cho, Lyft

With microservices, one request can go through hundreds of nodes. Not one engineer can know all the possible paths of the request, How can engineers infer how the system behaves? Metrics? Logging? These tools have their place, but neither of these inherently constructs a journey of the entire request. What if we want to optimize the overall request latency? Figure out how many additional hops the system will make by adding a new API call? I am here to talk about how distributed tracing tells a story about your system. I will go over how you can see the entire picture of what your system looks like, and with this data, make investigate and triage systematic issues, and make impactful, data-driven, performance optimizations to your system. I will go over what tracing does well and what does not. I will also go over how we leverage Envoy and OpenTracing to the instrument.

To learn more: https://sched.co/GrXj

A

Great, thank you so much for coming to my talk. I'm really excited to share with you my talk about distributors Tracy. My name is leta Joe I work on the networking team over at lyft and I'm here, to give you more of an intro. If this talk is really meant for people who has never they've heard of distributed tracing, but they don't quite know what it is or like what it's used for. So this talk is meant for you to explain like what distribution is.

A

What is it good for and like give concrete examples of how itself some of my problems and how we rolled it out at lyft.

A

So I'm using my mouse isn't working alright, so before I jump in I wanted to explain my background. A little bit so I've been um I've been working at live for four and a half years and I'm.

A

The first part of my career I worked on lip line, working on rolling out features and launching lines to all the different regions, and now it's like news nationwide, which is pretty awesome and then I switch over to infrastructure, and it's really cool to be able to bring that Park knowledge over and apply that to infrastructure and make engineers more productive and be able to have them ship code using the tools that you've built and previous to that. I worked on and during my time in infrastructure.

A

I've also worked on to your PC and standardizing our services. Serious communication with protocol buffers, then, before lyft I worked at DreamWorks Animation, making tools for artists and working on rendering algorithms and managing our render farm, and it's been pretty cool too. Throughout my career, I think I really enjoy making tools and making people other people more effective and that's it and distributed tracing is one of those tools that wouldn't Stu teach them I like what it is and how it's useful. They get really excited about it.

A

So this is the overall agenda, so I'm gonna go over an overview over what distribution is I split it up into two parts. So the first is the visual visualization aspect like what are you seeing? What kind of data it can you see and like what kind of questions can you answer with distributed tracing and then I'll go into the implementation portion like how do I get this into my system?

A

How do I instrument so that, like I, could get that data to view and then the second part of the talk is going to go over like how lyft integrated with it and because and how we rolled it out to all our micro services and I'll go over the challenges like integrating and then some learnings, so that, if you decide to integrate it in your company, like you, you could take these with you.

A

So before before, I can jump into what distributed tracing is and wanted to give context of what the lay of the land out there before we it we got distributed tracing. So what are the current of the really tools that lyft had first and foremost, is metrics, so there everyone I feel like that's their first way to monitor your system like they're, very cheap.

A

Let you fire them within your application, and you measure things like all rights through protes CPU, saturation and memory usage, and at the bottom there I like have a sample like we are most of our services and lyft is using Python and I have a sample there of how application engineer could fire off a stat? It's very easy.

A

Just one line of code and what's also cool about this- is that, as you horizontally scale, your metrics don't get more expensive, because it's just a number at the end of the day, so and they're very easy to reason about. Like you see, did you see the rate of change and you could and you could also use, create your alarms base it off based off of these graphs?

A

The second thing is logging, so there Jenner, they're, like string base, generate about an application and they're used to display like errors and stack traces and they're meant to use as a debugging tool. So let's say you see a spike in your dashboard or five hundreds. You can go into your logs and like look at a machine and see what is happening there.

A

So they're men they're not used to be they're, not really used for monitoring or like seeing the health of your system, but more for, like debugging, when you know that your system is something's wrong with your system, so metrics and logging are great, and this talk is not about how like distributed tracing is better than any of these or like they're supposed to replace any of these I think they're, all complimentary. You need all of them.

A

Metrics and naki tell a story from one perspective from one application, and this was pretty good for us for a while, because we were a monolith. So back in the day, lyft was a PHP monolith and all the transactions happen in one of the same computer, and so a statin law. Could you could set and log off of this monolith and it could show you the whole request flow and because it was a single process, one request took one process in the same box, so that would that worked really well for us.

A

But then we started moving over to micro services and now, like request, started going through multiple computers, doing multiple network hops and now it's kind of hard to reason about like why, like a write, request failed because there was a lot more dependencies to and more logs, we need to stitch together and we did it for a little while we added like a identifier, forever, a log per request, and we could search off of that identifier like a request, ID or write ID, but then and and then you could like search for that.

A

Id and I kind of figure out what's happening so yeah here, I'll go through a normal flow, but how like I would go debug.

A

Let's say: I see a spike of five hundreds and I found like a write, ID that that went through like that got that 500 and I'll show you, like my debug flow in the log metric world, so I go searching for a write, ID and I notice here that it went through like these services, like the regroup aSG's by services.

A

So- and here are the timestamps right- that's who I could tell that there was some ordering where, like the for the topmost log, is the most recent and the bottom is the the oldest and here I'm pre.

A

This mental model in my head, that, like oh, like this, require request, went through envoy and then we talk to public API, then rides API and then the API service, however, like that worked well for like when we were like 10 services, but now we're like way more than that, a single request can go through twenty and thirty services, and we have now like over five hundred services today and we have at each request, can like one write request can go through 200, plus RPC calls and yeah. So there's now.

A

Our system is way more complex, so I I'm, not gonna I, don't want to go through pages and pages of logs trying to figure out what the root cause was or what the or what even like answering the question of like what does this request depend on is now a lot more complicated, so yeah, it's like way too much to fit in my brain. Is there a better way to display this data?

A

That's digestible and like show me exactly what I care about versus, like sifting through logs I, mean sure we could make a bit better query language or, like basically come up with ways to make it work with logs, but is there? Is there a better way, so that is what distributing tracing is trying to solve?

A

It's trying to solve the fact that now requests go through many systems and many micro services and trying to show you that data in a way, that's like that's digestible to the human and figure out and figure like do root, cause analysis quicker and answer your questions about your system. A lot faster. So here's that same example in like a distributed tracing form so notice. Here. The top level is the root trace.

A

So that's the initial, that's where the request started off where and notice here that there's a like a span, so each row is known as a span and that signifies a unit of work and so and each span must have like I'd like a name so like an operation name, and it must have a time like I must have a start and end time, and that shows you like it's relating C and so, as you can see here, as you go down the rows and they're nested.

A

And that means that, like that unit of work, that envoy call made a network call to public API like ingress and egress so like it came in and that came out of that box and made another call to write API and again to the API service and it's at the rightmost corner. It shows you timing, information of how long a unit of work took.

A

So, let's go to more like a less complex example like a more contrived example. So here we have like a client in a web server and in the web server talks with three little micro services: oh sorry, micro services in one database, so the client makes it initiates the call and that's, and that span signifies the how long that request took and nested in that it made a call to the web server and the web server then call called a service to make sure you.

A

It authenticated that request and that concurrently called the billing and verify micro services, calls and then process some data and then stored all the results into a database. So let's say the database is on fire like let's say like it's, but let's so usually that gets shown as a dashboard.

A

What is it as a five hundred? And if you were to look zip through logs, it would be it's pretty deep. The error is pretty deep into the requests, right and and I would have to sift through many logs if I could find out. Oh, the database is not working, but here in the trace in one page and quickly see like oh that's the thing that's causing the error. That's the reason like I should investigate that and page the right person.

A

The second most useful thing that I found distributed chasing useful for is performance optimizations. So here this is a very common thing. I've seen out, lift but transformed into this example. Basically we have this again client web server call and we notice that the building system is calling the database get user three times. It's the same exact data.

A

It's getting, and here in this view, is very clear to see that it's duplicating work and cause a necessary load to the database, and maybe we could recommend like a caching solution to the payment systems because like why it should not need to get the called database three times for the exact same data.

A

Okay. So this is great like there's like this visualization is very useful. Like I, like you stores a lot of data in one place, how do I get this out to get get it into? My system like it must be very hard like do. I have to get all my product engineers to like add more code and how do I make sure it's like correct and yeah. There were a lot of challenges of getting adding trig just adding just there are challenges to adding distribute tracing into any system.

A

One is the fact that, like we have a polygon, Microsoft, micro-services architecture, so we have we support both go Python, Java and PHP, and we don't want to have to write clients for every single language for this and also trying to get get and, and so what is it, try to get libraries written for this? It's we more add more languages. We don't want to add more client, libraries or like libraries for distributed tracing. The second thing is: we do not want product and ship engineers to think about instrumentation and there's many reasons for this.

A

One is the fact that it's hard to get engineers to get change their code. They just want to focus on shipping features and they just want to like they just want to hit their ok ours right. They don't want to go back and like look through all their legacy code and add, like the these instrumentation code into every endpoint. That's a lot of work and and tracing it's not very useful. If, like one service instruments, it's like the network effect, so the more networks are micro services that instrument with distributed tracing the more useful.

A

It is so like the fact that to convince the first engineer to move over and they don't they're not going to see any benefits for quite some time.

A

Secondly, we don't want any like product engineers to make mistakes, but at the instrument right. So if the data like it's mislabeled or like they don't pass in the right tracing ID, then like um then, then, like they don't get hooked together and and now the distributors trace doesn't make any sense.

A

Lastly, we want to be able to standardize, so we want like the names of the traces to be consistent like we want, like. Let's say we want every single ingress trace to have like the endpoint path name or like we want the start and time to be exactly the same, like those really matter because, like if you're gonna make decisions on them like they need to be standardized and accurate, so that people can make good assumptions right, we don't want. We don't want every single tree like we.

A

Don't want people to read the code of where the instrumentation was taking place because it's different for every service, that's not as useful.

A

The other option people have thought about is like you could monkey patch all of this to all the services. So, like, let's say, like you use like a like a web framework, you can monkey patch it there one that doesn't really work for like languages like go, but secondly, it also it's not a great. It's not great.

A

If, like things were to go wrong and then like your that, your us infrastructure team is stuck with all the support burden to like fix it versus like versus, like the product engineer, to be able to fix it themselves, and it's also not a very like nice thing to just do it, do it behind the scenes and then they're just and they're, not aware, cuz, and also it's easier to make mistakes. This way a lot less and, of course it's we don't want vendor lock-in.

A

So let's say if we were to use like a vendor specific instrumentation library then like worse, and that we and we we as a lift. We want to be able to switch between vendors and like try different tools as the industry changes, and it will be really and we don't want to do a huge migration if we wanted to switch to a different vendor.

A

So this is what open tracing is trying to solve open tracing is, is a CN CF CN CF, a project that is trying to standardize all the instrumentation, and just you in tribute of tracing one great thing is that it's it's basically a interface where you all you have to do is like, and they have implementations of many different languages, and all you have to do is adhere to this interface and and then what you you you get all you get all the benefits and the and the vendors all have to implement the on the collector side that interface and then you could swap out vendors very easily again, it's supported by the CN CF Foundation and has libraries from many languages.

A

So it supports all the languages that we we use, such as Python go and Java, and PHP and yeah it's vendor-neutral. The idea here is to make it so that you how you're emitting the data is different from who's collecting it, and that way, like you instrument your code once and then you don't have to worry about like migrating over to a different vendor, you could just swap them out pretty easily.

A

So let's do a concrete example of like what this looks like in Python. So this is how you would initialize a tracer, so you do have to initialize specific, and this is good, a change depending what vendor you use, I'm, showing two examples here: one with light step and the other with Jager, which is the open source, tracing solution and light step, is a paid vendor. So it's pretty simple you you instantiate the tracer for Jager. You set up a configuration and then pass that configuration in to instantiate the tracer and light step.

A

You just pass in your authentication into the instructor, then this is what the instrumentation piece looks like. We have like a endpoint, and this three lines of code is essentially the open tracing the open tracing instrumentation code. It's not that much so with that with statement. When you exit out of that with statement, a that's when the timing start the beginning of it is where it starts and I'm going to exit out.

A

That's when it stops the timing and that's like the basics of it, you could add more code here to add tags or, like add, like baggage data as well. But for the most part, that's all you need to get to fire off a span, and here here's like a I know. This is a pretty that's a lot going on here in this picture, but like this, that's soulless, but this is what how the Geiger collectors work. So we have.

A

Basically, you have your application layer and then you're open tracing api's, and then that, though, that pieces of code is firing off to the Geiger client and there's an agent. They have Jaeger runs a sidecar agent that collects that data and that's mostly for configuration. So if you want to do dynamic, sampling or like you want to configure it dynamically, that's what that agent is there for and all that data is getting sent to the collector and the collector is digesting that data and then saving it to Cassandra and the UI part.

A

The Yaeger UI is reading directly from the Cassandra Asia, and this is the part I wanted to show is that the application is not even aware of the Jaeger client. It's not a where like they there is that abstraction, but do do the open tracing API, and so you could swap that blue stuff out for other options as well.

A

So this is great, like it's not that much code that your project engineers have to do it's and- and you have to do it once- and you never have to do it again, but you still have to still edit the code and is there a way that we can do this without having part of introduced to anything like? Is there a zero touch? There's a zero touch implementation possible, and yes, there is we. We did not do this.

A

We ended up using envoy and the service mesh to get tracing out to all the services and which is very, very powerful concept once because all our services run, and for those of you who don't know envoy, is a it's a proxy and as a sidecar proxy, that's run on every single micro service and all network calls go through envoy.

A

First then hits your application and because we already run on a server smash, we're able to turn on tracing for free for all the services and envoy is the one that's spinning off speeding up all the spans. So this is so at the end of the day. This is how it really works. So we have. Let's say we have application, we have a hosted container and we have it, and we have application running in our sidecar running a sidecar envoy running on a separate process. Envoy is the one that is sending this fans.

A

They are the ones that are and they're all standardized to the same way, because it's all one unified code base, so they are the ones I send the ingress fans and the egrets fans, and they send that directly to the collectors and the only thing that the application has to do to tie the two spans together is send this header. This ot spam contacts, header and all that is, is an identifier at the trace ID to every single hop, and we make sure we do that by implementing that in our envoy clients.

A

So we do have to refer to every single envoy, client we support and various languages. We do have to make sure that the header is set and sent over to the every hub, but that's way more easier to maintain than like trying to create when trying to either monkey patch per service or trying to trying to get every product engineer to instrument their code in the same unified way. So.

A

So that's essentially that's how we rolled it out. We were able to add a instrumentation data to envoy and we were able to get tracing for free instantaneously as we rolled out this coach agent envoy, but now that we have this data now that we have these fans and we have this tool, how do we get people to use it, because it's a very new tool like now, people are used to looking at their dashboards and they're looking used to looking at logs but like? How do we get them to use this other thing?

A

Now, that's like another thing in their belt like how do we show value the biggest thing that we did was integrating it into people's workflows as quickly as possible? So this is at lyft. We use Griffon a--, the goof on our dashboards and that's what people look at every day and we added these dots here and each dot you it's a link to a trace, and so, when you click on that, we you are able to see, get a trace view of like let's say you see a spike in five hundreds.

A

We would click on a dot within that graph and I'll show you a trace that resulted in a five. So we quickly try to get you into that context and show you the traces that are useful for you as quickly as possible and usually still interact with the tool that you're used. To second thing, is we also added traces into pager duty rotations? So, like you get paged and there is a trace available or we will link you to that trace.

A

So let's say you got paid for like yeah, hylene, C's or 500s, we'll show you an example trace for you to help debug and so interviewing, with the tools that people are familiar with. It was really in central for adoption, so yeah, the biggest takeaways is like we, we integrating with envoy, was a huge win so that we didn't have to get engineers at any code and we get that Network effect instantaneously achieved something similar with libraries or like.

A

But the thing is that you that zero touch experience the fact that, like no one had to edit any code was very powerful and then we that we wanted to tie everything together as much as possible so like we want to use the tool so that logging, stats and tracing should reference each other and like try to work seamlessly as much as possible so that people get used to the idea of like how wire trace is useful. Like we don't like, we don't want people to go.

A

Looking the tracing, UI and like trying to look for like truth, looking at your first trace, is pretty a pretty jarring experience so like trying to give you the traces that are the most useful to you to in your service. It was essential for adoption and, lastly, sample II. I didn't have time to go well.

A

I didn't make a sigh first sampling strategies, but sampling is quite hard because, like there's a lot of data that you, but once we started increasing the amount of number of microservices, simply we had to start a sampling because we were just emitting so much data and sample sampling figure out. The right sampling strategy is quite hard because it's very business-related like how I sample at live, will be very different from how you sample at your company, so like I just basically touched like the tip of the iceberg.

A

There's way more information out there, I have, if you're interested in learning more I, definitely recommend starting off reading the open tracing homepage and getting familiar with like the instrumentation libraries URI. Who is the creative Jaeger? He has really great open tracing tutorials, where you have like a crud service, and you start adding instrumentation and seeing those the the spans that you admit into a UI and gives you a concrete idea of like what is happening.

A

He's also here at this conference, so feel free to reach out to him he's happy to talk about tracing as well. If you have, if you want to go really deep, I recommend reading the dapper paper, which is Google's tracing tool and that's where all of this all started and it's a very easy, read and I.

A

You want to see like how they all started, where it originated from I highly recommend reading that and lastly, if you want to see a talk about how you can apply tracing to Kaos engineering and like how you could leverage that data into where to inject fault, next I recommend watching the talk about orchestrated chaos. Thank you. That's just that's the end of my talk. I really appreciate you coming to my talk.

A

I think we have some time for questions if people are interested or.

A

Me Oh what oh yeah.

A

Yeah, so the question was like: are we emitting any standard tags and that and yes, we are envoy- does emit a sender set of tags. One is usually with like, and that's also configurable in the Envoy configuration. But this tags we emit are usually the path name and the upstream who called you that you like the upstream.

A

What is it the upstream service and then yeah and there and then there's all their like business, specific tags and what's really cool, is that we, if you want lip so envoy, gives like, like the just the network latency but like. If you want to dig deeper into your application, people could add custom spans as well and they add custom tags there, all the time specific to their service.

A

A

I couldn't quite hear you are you using a service mesh to connect to envoy? Oh man, you have to come closer sorry. This room is too big.

A

A

So the question is: is that have you encountered a issue where the Envoy latency is not acceptable for a service?

A

Not no, not at lift, for the most part is not envoy, is written in C++ is very performant, and so a we have not noticed do even through looking at the distributed traces like we have not noticed any lanes, big latency effects due to envoy, because we send it like because envoy sentences to spans one span once once it gets the request and the other span of like when it sends it off to the application and that it's like there, it's so minimal like so it's not due to envoy that of the hylians.

A

If there's High, Line C's yeah.

A

Am I using? Are we using a service smash to inject envoys? I'm, not sure if I understand your question, because the like so envoy is being run as a sidecar and it creates that service mesh and so all network traffic is going through an envoy before hitting the application through localhost yeah. Oh.

A

Oh I see what you're saying yeah. So the question is like: how is the issue to commit as a control plane to configure our envoys? No, we you couldn't create. We have our own control plane and that's the most I could talk to you more about that or my colleague Jose here could talk to you at length about it, but we basically launched envoy before a CEO was around. So that's the reason why we don't use these do yeah.

A

Yes, yeah yeah, so right now, oh so the question is is like something has to instantiate the trace. Is it Envoy that insanity is the trace that it's correct, so when so, we run envoy at the front edge proxy as well. So that is the thing that instantiate the trace, we're trying to move that over to a client, the mobile apps. But right now we are like it's the front envoy proxy, that instantiate sit yeah.

A

Do we turn on tracing for every request? Yes, except except the only thing we don't turn it on for is GRP streaming because of the fact that you're tracing was designed like the due to the are traces are designed to only handle a finite like finite amount of time for the length of a trace and trimming can go on for like an hour or two and and at that point the traces don't make sense where they become so long that it's hard to digest. So we're trying to still figure that out.

A

The question is: do we inject TLS in our servers, smashed we're in the process of injecting TLS.

A

Have we had to the question is how we had to rewrite our wait. Yeah so repeat so,.

A

Yeah, yes, we use HTTP going.

A

No I don't believe we have to yeah. We do not inject at this moment in time. We don't inject the circuit, let's start into the outgoing requests, so we have not faced that challenge. Yet we are in the middle of our migration over. So we probably will face that challenge sooner rather than later. Yeah.

A

The question is: what is the memory footprint for running envoy? It depends. It depends on like, like how you configured it and what service like what services they could talk to. But it's not it's very minimal, but you should be in tracing wise, is very minimal, because envoy sends us fans quickly quickly as I can't off to the collectors. So.

A

It really yeah the look. He wanted more details on the memory usage. It really depends on like how like how its configured it could be in the like. It could be in the tens of Meg's to a hundreds of Meg's I've, seen it all, depending on how its configured.

A

Yeah, the question is like: how did we get the integration done in Griffin? Oh yeah? Oh, so our vendor light step created that integration. So basically we send the query of like what we're interested. What that graph represents and then light step has a plug-in to like generate those dots, yeah.

A

Say that can you repeat that question.

A

How do we, the question, is: how do we trace the base requests with envoy Deo database? Oh, so, how do we trace our database request with envoys hello I see so that so we do how it works. Is that all database requests goes through envoy outbound. We don't measure it when it's at the database, because we don't have a proxy on our database, so we can only measure like how long it took to to egress out to the database and come back.

A

Can you come closer yeah I'm, so sorry I can't hear I apologize. My hearing is very bad.

A

Yeah yeah, so the question is: are you able to tie traces two logs? We aren't. We want to do that. We currently do not do that right now, but it's there's many ways to do that. So one is like in the log send the trace ad or like in. If there is a span context available, you can add logs to your span, so you do it both ways. It's up to you.

A

It makes the spam more expensive if you send it with if you send the log with the span, but another thing I've heard people think about is like connecting your elasticsearch database with your Cassandra tracing data and how the UI merge that together, that'll be pretty cool, but no we don't the ties together with logs. It's not really great right. Now, oh.

A

Yeah, okay, oh yeah, I will upload this slide. Yes,.

A

Yes, that is correct. Yeah envoys are speaking up in tracing API, let's collect it depends like we have our vendor, giving us the collectors and that's collect. That's the thing that is collecting all the data and yeah so Jaeger, so you could use Jaeger as your collector as well. I. Think that's it for questions. Thank you so much. Everyone.