Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jaeger Intro - Yuri Shkuro, Uber

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Jaeger Intro - Yuri Shkuro, Uber

This session is an introduction to Jaeger and distributed tracing. We will do a demo of the current Jaeger features, talk about the roadmap, and finish with a Q&A. After this session the attendees should better understand how Jaeger fits in the observability space for cloud native applications. For more information on the project everybody is welcome to attend the Jaeger Deep Dive Session.

https://sched.co/Zewm

A

Hello and welcome to uh project introduction for jaeger. My name is yurish. Crowe um this month is the fifth anniversary of jaeger, starting october, uh from the first commute it's in august and so congrats to the project um as far as the agenda, so I will start with a brief introduction of uh why tracing is an important um part of the observability for your systems.

A

I will do a short, live demo of jaeger features, we'll talk about jaeger architecture, we'll talk about sampling, and I want to finish with discussion about the relationship between jaeger and open telemetry and how these two projects are um to say a few words about myself. I'm a software engineer at uber.

A

I started jaeger as an internal project, and then we donated it to cloud native foundation.

A

I'm also a co-founder of open tracing and open telemetry projects, which are the instrumentation uh projects for tracing, and I also published a book last year about tracing and my experiences at uber, as well as experiences with jaeger, and how to deploy it and configure, and things like that, you can find uh my contact informations on on my website, uh with a blog and another information.

A

So, to start with with the search question of observability, um why tracing is important in microservices, especially in microservices architectures, and why it becomes popular today, so modern systems have to deal with a very large scale, and the traditional wisdom is that we cannot scale up our servers because there's a limit to how much any given server can process, and so usually people say, okay, we need to scale out and build architecture in that way. But what does scale out actually mean the simplest uh approach to scaling out?

A

Is that if you have a monolith service, you can just replicate it multiple times in many hosts and put a load balance in front of it and if that's the whole of your architecture, then you have a pretty uh clear picture of uh how to debug these things and what type durability story is it because you could probably get away with just metrics for every single instance tagged with the instance name, and then you can investigate problems with that architecture, but in reality this is not how the systems are built today, and this is another type of scaling that we can talk about.

A

Is it's scaling the systems in in depth, uh meaning that we don't have a single monarchs? We actually break monoliths into many many different pieces and many different layers, and so any individual request that that hits the architecture has to go through all these many layers, so modern architectures are really deep systems. uh If you look at any of them and deep systems present unique challenges in terms of observability.

A

If we look at uh this picture, which is just a screenshot of uh microservices layout two birds few years ago, so imagine that there is a request coming from the mobile application. That request might look like this when it hits this architecture right it. It goes through a large number of services all participating together and inserting one single request, and we can look at it in a different way.

A

This is the actual representation of the real production request from from uber right might be an application start, and we can see here, there's like over 100 uh remote procedure, calls within that execution, uh 14 levels of depth. So why is that a problem for observability? Well, the problem is that now we have to answer questions uh about. uh What's wrong with the system, uh given so many different nodes in the system that where the things can go wrong right and how do we start with that? We we approach it usually with.

A

Oh, we have some business metrics, maybe the number of trips in a particular city and suddenly that metrics dipped, and so we have an alert and we try to investigate. But then then, what? Where? Do you go from that business?

A

Metric saying something is wrong to where in this massive architecture, a particular change happened that is causing this problem, and this is where tracing really comes in, because tracing allows us to look at individual requests in full depth end to end, and if there are errors happening, they will be captured in the trace, um and so the the conclusion here is that this is why tracing is important, because uh in in the modern deep systems, um sometimes it's much more difficult to find where the problem is given the how distributed the system is rather than what is specifically wrong, because once you narrow down the problem, there are a lot of variety of different tools that you can use to troubleshoot.

A

It within, let's say a single instance of a service, but even to get there is very challenging, especially if you're starting from an alert on a business metric.

A

um I just want to give a brief kind of crash course into tracing. If you don't know how that works, uh the the tracing is built on the concept of a context, propagation, meaning that when the request comes to our architecture, in this case let's say service a at the top is the kind of gateway service, maybe or api service, and as soon as the request comes in, we assign a unique id to that request.

A

Let's call it a trace id and we store it in a thing which we call a context or a metadata, and that metadata is attached to every single other rpc call or request that the execution of this top level request involves right and so as service or a call service b.

A

We will pass that metadata in and so on and so on, uh and what that does is basically that allows us to tag the a single execution across multiple services and then assemble this data on the back end into coherent representation of the of the execution, and this is kind of a typical representation of a trace in the gantt chart, where each service is represented by a certain span of time, uh where, like from start to end of the operation within that service and then there's a hierarchical relationship between uh between services um like a causality which is being tracked by this metadata being propagated.

A

Now, let's look at the demo of jaeger features, so I have here um a clone of uh main jaeger repository from github yaga tracing, slash jager, and there is a directory for examples with a single sample application called hot rod. So in that application we can see there is a docker compose file.

A

Which loads uh so-called jaeger all-in-one? This is a binary of single binary that combines all of the jaeger components in just one executable, so you can kind of easily run it, and then there is another uh docker image for the actual demo applications right. So.

A

Let's start it.

A

uh We can see in the logs. uh There are two things, so jaeger is the the main jaeger back end. So it starts on on this uh port 16686. This is the query series which also shows the front end the ui and then the the gem application started a whole bunch of services. uh The the one that's at 8080 is the one that we actually want to look at. So let's go there. So we can see.

A

This is a kind of a mock rides on demand application where you have customers and you can click a button and get a car right. So, let's order uh a few cars- and we can see here- there's like license plate eta of arrival- then some tracking information for for debugging purposes uh and and the latency, and you can see that the latency kind of the more requests we put in the the higher the latency becomes right so like.

A

If I, if I do another request, it will be short, but if I send many of them, then we'll see that the latency keeps climbing right. Let's look at how these requests look in jager, so we'll go to localhost. At this point that I mentioned previously- and we can see here in the services drop down that jaeger already detected all of our services in that mock application.

A

It also detected itself because jaeger query itself is instrumented, and so we can get traces from it um so, but we're interested in in in in the hot rods uh application, but before we jump into traces, would it be nice if we knew what the architecture of that application is, because all we've seen so far is the front-end and so jaeger has the system architecture tab for this there's two views.

A

One is not like useful for this case, but the dag view is is is good uh because it gives us very clear picture of what all the services that are included in this application. How they're connected uh this is a number of requests that were executed during my run of the demo. Let's go to see what what actual traces look like right. So there's a few ways that you can search for traces.

A

One is we can just go and do a blind search. Once we selected the service, we can just see and find all of them right the other option. We can do search by specific values if they are captured in an instrumentation and in this case the instrumentation records these license plates as the tag driver on the span of the of the service, and so we can put it in a search query.

A

We get just this one trace right and if we go in that trace, uh so let's, let's just go through that view, which is kind of the main view of most racing systems, uh at least for for a given trace right. So here on, we have a timeline.

A

So this is also known as a gantt chart view um and at the top we have uh what we call a mini map, it's kind of the small presentation of the trace, it's useful when you need to navigate very large traces, because you can zoom in here and uh jump into various places, but in this case uh almost all of the trays well, actually half of it still fits only on the screen, so uh the mini map might be useful as well then, on the left, we have um all these services uh laid out in the hierarchical manner, showing what uh how how which service called which are the service rights.

A

um So we can, if we collapse everything and then open only uh one level, we will see that uh actually one more level we can see here that the front-end service called the customer. Then it's called the root service multiple times uh and also called the driver service here right, so, which is what what we've seen in the in architecture, diagram um and uh and basically every every bar here represents a single span within the service. uh You can see that let's say if we pick a front end.

A

There is more than one span here, even though logically there's only one operation, because uh the way the tracing instrumentation is done. It creates a span not only for every incoming request, but also for every outgoing request. So this http get dispatched. That was the entry point into the front-end service, but then, when it made a call another http call to the to the customer service. There was another child spend so-called client spam created there, and if we click on any span, we can get uh additional details.

A

So let's look at the at the root span. So the one thing that we have in spans are things called tags. Tags are just some uh metadata that you can attach to the spans and it gets recorded and sent in in the background to the yeah tracing back, and so we can see there is useful things specifically like the the exact url. uh You don't want to put the url in the span name, because that will make it very high, cardinality and difficult to aggregate.

A

But in the tags you can put any kind of cardinality, and this is just, I think, nonsense for the ui cache investing and then another interesting thing here is a span kind which says this is a server span right. So if we look at the next one, as I said, this is an outgoing outgoing call uh or sorry. This one doesn't have this this one.

A

It will have a span kind of client so which says this is the uh the call I'm making out of the process to to the next server, and we can see that the url is it's hitting the customer service going back to the top span. um Another thing that we can see here is that the locks section right so uh in in the output.

A

Here we can see that there's like as I was executing this request. There is a bunch of logs written to the standard out right, and this is normal.

A

You can pipe them into some central uh look, aggregator aggregator, but they're kind of very difficult to use, because again, if, if there are multiple concurrent requests, you get a single jumble of of these log lines mixed together, whereas here these logs belong just to a single request right and this, the way they get into the trace is that if you look at the implementation of hot rod application, it has a special wrapper for the logger, so that logger not only logs to send it out but also writes to the current span, uh and that allows you to kind of get the exact same information uh but uh like filtered out from the from the rest of the noise that you can see in the logs.

A

So this is a very powerful feature of tracing that allows you to investigate problems um within like very specific uh execution of your of your total trace. Now, let's uh expand everything again and look at uh what what the trace can also tell us. uh So one thing: that's immediately obvious is that there is this mysql select which takes uh a majority like two thirds of the total time of this request right.

A

So if we were investigating the latency problem, then this is clearly the path where we would look for for why this this particular places is very slow. It also has some interesting logs, I'm not going to go into into this much too much, but it says waiting for log behind three transactions. So it's probably not the actual query that was uh taking so long, but the the operation was simply stuck in india service right, so there might be some resource pool contention here.

A

um The the other thing we can see in the trace is that uh these multiple calls from the driver to radius service. Some of them failed as as can be indicated by this red exclamation point, and the way that that is detected is that if you look at the tags, you will see error equals true right. So again, this is uh instrumentation. Setting that up, you can always do it for your own spans or your own operations. uh How you see fit uh jager simply just displays this.

A

These errors have the ability to a bubble up the tree. So if I start uh collapsing uh things- and we can see here that the driver is shown with an error, even though there was no specific error in the driver, uh but then if you expand it, then it's actually below right. So sometimes it's useful when you're looking at the very top of the trace to know if there are any errors and then you can drill down into where the exactly they happened.

A

And finally, uh another thing that we can easily tell from the trades in terms of like analyzing performance of the applications that these calls to radius seem to be happening all in uh in a sort of a staircase pattern right, one after another. So that would clearly be an indication of something that potentially can be improved, uh because if we know the the business logic of the application here, uh it looks like it's getting like driver informations for different drivers.

A

And so why couldn't the task radius for that in parallel or in a single bulk request right. So that would have saved uh this this much time.

A

Basically, by doing this, all in one and finally, the last segment where the front application calls into the root service asking what is the closest route for the driver to get to us so that we can compute the ata, then we can see here that uh this execution is not as bad, because there are some concurrent requests going on, but there is no more than three concurrent requests for some reason right. So this stop uh one two three and then once as soon as one star stops, then the other one starts.

A

So this is again an indication of some sort of resourceful contention. This is like an executor pool, uh that's uh limited by three and you cannot execute more than that, and so, if you run multiple concurrent requests, then they're all gonna be blocking on this thing. But in this case we don't see it because the contention, the main contention is really the mysql uh span.

A

If you, if I go back to search screen, the one thing I didn't talk about, it is uh is uh kind of what what you get on the search screen. So um we you can also search for traces uh by other attributes. Specifically by duration right, which could be very useful, because if you are capturing a lot of traces in the system, some of them are maybe very short and quick, and you are not really interested in investigating them.

A

You really want to look at what your p99 latency traces and presumably you have some metric which says. Oh your p90 lite entity is like two seconds right and you can put a query saying: can you show me traces which are longer than 1.5 seconds right and so then boom? We only have one trace here in this case. If I make it maybe like one. Second, I get a few more right, um so uh that allows you to narrow down the search and then investigate uh the.

A

What the differences are uh in in this traces, but sometimes um looking at one single trace doesn't necessarily reveal all the problems that might be happening in terms of performance and uh typically, when you use normal uh performance profiling tools like memory allocations, you would take a snapshot before and after and then you compare those snapshots right and so jaeger also has this ability to do the comparison.

A

uh You can select two traces uh and then click compare and what it does is that combines them into into the graph representation and uses green and red color coding to indicate where there are missing or extra nodes in this case, because there's only one uh extra node. So the the picture is not that interesting, but if we had a very large trace, then that picture got a lot more interesting and I can show you later in the slides example from real production.

A

Traces and finally, the last piece I want to show here is: we've looked at the system architecture before this one right. There is another view that jaeger has, which is called deep dependency graph. It's built from the search results. Let me remove this uh duration so that I can get more traces, so 12, traces uh and then deep dependency graph builds uh uh an execution um like a rigid representation of that of those traces. However, it looks very similar to the system architecture, but there is a significant difference in that this. This graph is transitive.

A

So when you see an error going through the like front-end service to driver and then to radius, we know that there is actually a path in some of the traces that goes through these two services, whereas in the system architecture that's not guaranteed. So um just because there is a kind of an error from front end to customer doesn't mean that there is any request from front end which will reach the mysql right, because this graph is based on just the pairwise connections between services.

A

And so why is that important is because uh sometimes sorry, you need to search again when you're investigating like dependencies of the services. You not only want to see what are my immediate dependencies under on the front-end service, but you want to like look for deeper dependencies, but only those which actually matter, because if there is a dependency of root service, let's say it's doing some background caching and hit some some storage here which never affects any of the front-end requests.

A

Then we don't show it here because it does not come up in the traces right and another feature of this graph. Is that uh it's? It is not going to show you the whole architecture, but only pass going through the focal servers, which is why it's in pink here. So we can refocus this view to another service, and then it will become even smaller, uh for example, in drive right, because there's no paths that go through driver and also through the customer service, so they're not going to be shown here.

A

And finally, this graph is also uh can show you uh operation level view not just a service level view right, and so we can see here that there are actually two different operations from the regis that are being called, uh and so they can. You can switch to this in the layout view you can.

A

You can change the graph to to show you different information, um so this graph, we we built it as a sort of a platform to to do more stuff with it, specifically overlay and real-time information, like what's your current latency or error rates on this graph right now, it's not hooked up in any way, but this is kind of the future direction that we're going with.

A

So I think this. uh This is uh all I wanted to show you in the demo so and then I will switch to my slide deck um and I'll start with this.

A

Sorry, with this uh view that I mentioned over production trace where you compare it to traces- and we can see here that, because um this this, these two traces are much larger uh there also they have a lot more interesting differences right and so at the red at the bottom shows that a whole number of calls uh did not happen in the in the round, right hand, side trace, uh and so, for example, if we look at the duration, the left one was 2.7 seconds.

A

The right one is 1.4, and so well, because this whole section is missing, that kind of explains why it was faster, but it also potentially there is an error there right and so yeah. I will skip this detail, there's another way to to look at this uh comparison of the traces.

A

So this comparison is currently structural, so we're just comparing do the nodes exist in one versus another graph right, but uh we also may look sometimes at the uh the latency differences within each individual span right um and so the latency difference is, is kind of uh gives you a very different picture, uh but also drives your um sight immediately to the problems where you want to investigate right.

A

So here we can see we're using like a heat map, color coding for for, like the darker red, the darker, the red, the the more latency differences in is within these notes, uh and then the quiet notes means that some of the notes were not even present in, in the right hand, side trace, whereas gray means it's like. There are no difference, and so we can see, for example, that kind of the the overall latency difference came into in this path right.

A

So it got lighter light lighter, but this is kind of one of the suspect spans that you might want to drill down and and look at what was happening there uh and if you um mouse over uh these uh these nodes that actually shows you how much extra time was spent in this specific span and how what the percentage of time over all of the trace was wasted there. um So I believe this this view is actually still not in the main branch it's like in the pull request.

A

Unfortunately, so you can probably try it right away, but um so, uh in conclusion, you can use tracing tool to monitor transactions in your distributed architecture and see where the requests are executing, which services has been hit and what happens in every step of the way right. uh We can also do root cause analysis by looking at individual details like what tags exist in each span, uh like maybe sql, query or a url, uh or some errors where they happen, and we can drill down into those.

A

uh We can also look at various patterns of how the timeline time layout of the trace look, and so we can detect immediately patterns like what's the longest critical path. What's the staircase pattern, why it's happening sequentially and things like that? These visualizations make it very easy to troubleshoot. um And finally, we can do um various service dependency analysis on the traces by using this system graph and the transitive dependency graph, and, as I mentioned, all of that is based on the distributed context, propagation, which is provided by jager sdks, and so now.

A

Let's talk about jaeger architecture and overall jaeger project, so uh jaeger as a platform uh consists of these four components. uh It has a a number of client libraries, also called sdks or tracers in different languages. You can see it on the left. They all implement open tracing api, um and so those are the things that you put inside your application or inside the framework that you're using within the application right and they collect data and they send it out uh to the jager backend and that's the the middle piece here.

A

Trace collection, back-end, uh which includes storage, uh some pre-processing, some potential aggregations and things like that uh and back-end also feeds into the data mining platform, where you can run big job analysis, big data analysis like flink jobs for real-time streaming or spark and create aggregates or views of your traces, for example, the system graph uh in the demo. It was like all in memory, but you can deploy it in such a way that it will actually compute uh from from a large amount of traces and give you the visualizations.

A

And finally, the the front end is embedded in the jaeger query, as I mentioned, and provides the different views of the traces.

A

um One thing that is worth mentioning is that jaeger project by itself does not provide instrumentation right.

A

So if you have no instrumentation in application, you're not going to get traces and we don't have auto instrumentation agents either, uh and this was a country's decision, uh because those aspects of of distributed tracing are taken care of by projects like open tracing and open, telemetry and I'll later speak about what that means, uh and so you need to get instrumentation somehow so that you can start exporting data and yeah deals with collecting that data and presenting and analyzing it.

A

So as far as history. So by the way, jager means hunter and don't spell it please with jagger: that's not the official name. It was inspired by dapper from google and and open zipkin. As I mentioned, we created the tuber and then uh donated to cloud native foundation, and now it's a top-level uh graduated project at cloud native foundation.

A

um So how jaeger fits in in your architecture is this? Is the slide tries to explain that? So let's say you have two services: a and b right. As I mentioned, uh you need to have some form of instrumentation in those services and there are various options you have you.

A

You can have open tracing instrumentation with lots of libraries are supported by open tracing that you can just plug in, and you don't have to do much in your code really just initialize some things and uh and then we also include a jaeger sdk which simply implements that open tracing binding so that when instrumentation captures the data, it just gives it into the jager tracer and then tracer is possible to send you into jaeger back end right. However, there are two data paths that you can see here on the screen.

A

The the the top one in in a solid line is the so-called in-band data, which is when service a makes, a request to service b. It includes a certain metadata about the trace in that request. That's a very small piece of data like usually trace id span id and sampling flag, and there are different formats that are supported. So jaeger has its own native format.

A

That was originally developed a tuber, but there is also now a w3c standard format called trace context that you can also configure jager sdks to use to communicate between services, and we also support zip code.

A

B3 format, which is another alternative of that, and so that's how the trace information gets into the service b, which it reads that metadata and then creates uh again tracing data to send it out of band, and so the trace data that goes to jager back end is really is sent in the background uh by background threads, um and it does not like happen on the critical path of the application where this part happens, the top part on the critical path right.

A

It's part of your request, execution flow and, as I mentioned, you don't actually have to have necessarily jaeger tracers in your service, because there is other ways you can instrument. You can instrument it with zipkin like a brave sur library, or you can instrument with various open telemetry sdks. uh They all kind of support jaeger, as as a data format, except that, if you zip can down jager itself supports zip code format, but jager backhand can combine all the data and present you in for a form of traces.

A

So this is the architecture so um of jager itself right. So on the left again.

A

Imagine that you have your application and you have jager client or jager sdk iranian inside the application and the typical deployment that we recommend is that iran, jaeger agent, which is a small process uh as a host agent, so that you don't have to like run many of them on one host, although if you do want, you can run it as a sidecar in the classic like pod uh kubernetes pod, so that every application will have its own eager agent is really it just is a proxy.

A

It knows how to find jaeger back end and and send the data there. um So you don't actually have to use it. You can configure jager clients to go directly to collector, but then you have to deal with some discovery, maybe like uns, uh sir dns uh name, uh so that uh like collectors can be located, whereas when the agent is local to the host, you just send it to a local port and also uh with the agent you can use udp so that it's sort of like it's a telemetry data.

A

It's not a big deal. If you lose it, whereas when you send to jager collector, you can't really use udp, so you have to use uh some http uh protocol to send the data and then jager collector receives all these traces from multiple applications and saves them to the database. You can also have spark of link jobs running off of that database uh and then the jager query visualizes. That thing and the last piece which is shown in the rats here is- is a control flow.

A

This is um something that we've built from the beginning into jager architecture, uh which I'll speak to. uh When we talk about sampling, it allows you to push configuration back into the jager sdks to affect how the sampling is done in the application.

A

However, this was the architecture that we initially uh run a tour and later on, we switched to to to slightly different architecture where, after jager collector, we introduced kafka before a component which writes spends in into storage right, so the jager collector got split into collector and in in gesture and indexer, and the reason we did.

A

That is because, um when you sometimes have a traffic spikes uh or some application uh is deployed with, like a sampling of hundred percent, it's very easy to send too much data that jager collector is simply not able to say fast enough into this database, uh because database have like a throughput limit um and whereas kafka is usually a more elastic uh storage, you can think of it uh that that can accommodate uh huge traffic spikes, and so this is one reason we introduced it.

A

So like we don't lose data when there is a traffic spike, we just write more to kafka, and then we experience a certain ingestion delay or lag as a result, because it takes a bit more time to save it to the database, and so the traces are not immediately available in to the query.

A

But another serious reason why we introduced kafka is because it allows us to to start building flink jobs uh to do aggregations like the dependency graphs um in real time rather than iranian spark job, which has to read the whole database and and like this is uh doing it in real time is much more uh interactive. When you get uh data faster into into a system and it reacts faster, so this is what we're currently running at uber and uh again you don't have to use kafka. You can go directly to the database.

A

It's really it depends on what you want to get from this, and the technology stack uh to mention for jaeger is it's. uh Backhand is all written in, go. uh We support pluggable storage. There are two ways: there's a several back-ends which are natively, supported directly by the binaries that we uh distribute specifically cassandra, elasticsearch badger, which is a sort of single node storage on disk and also users like a toy in memory, implementation that is used by all-in-one uh binary.

A

uh However, there is also another uh plugable solution, called jrpc plugins, where you can implement any kind of storage, backend uh communicating over jpc with the jager end with the jager collector, and that allows uh sort of us to to extend the capabilities to other storage engines uh without kind of bringing all of that maintenance overhead into jager main jaeger repository. uh The frontend is built in javascript and react is pretty standard.

A

uh Instrumentation libraries or like sdks, are all implemented in open tracing and we integrated with various like kafka and apache flink frameworks and, as I mentioned, zip code compatibility involves two things: we can uh jager clients understand, zipkin headers format on the wire, and it also zip can collect sorry. Jager collector can also receive data from zipkin in various formats that you can support. um It can even read from kafka like in in the zip code street format, for example.

A

um Now, as I promised, let's talk about quickly about sampling. So, first of all, why do we sample right? Why why it's even a topic- and the problem is that trace tracing information is very, very rich. uh You imagine like, as I explained in the demo, for every rpc call. You have two spans on the client and on the server, so each each span can have all kinds of tags and like attributes like with urls, it can have logs.

A

So this is pretty bulky objects that we have to ship and that happens for every single rpc request or actually hundreds of the rpc requests right. So the the volume of data accumulates pretty fast, um and uh so if your service is doing like, I don't know, ten, ten thousand rpcs uh imagine how much data you're gonna accumulate per second, so storing all of that can incur pretty large storage costs.

A

That's one reason for sampling, but another reason is that uh just collecting all this data from the application also has an impact on the performance of your application. So you may introduce latency because you waste in cpu cycles on processing all this data, and so sampling is usually the technique to deal with that overhead and with the with a large storage cost to avoid them.

A

So there are two types of sampling that typically used in the tracing system. One is head by sampling where the sampling decision is made at the very beginning when the trace is just starting.

A

So when you create a new random trace id, we flip a coin and say: okay, we're gonna sample it or not, and once we make the decision, that decision is fixed for the life of the trace and it's propagated as part of the trace context, so that every other service which participates in the trace it will use the same decision so that you don't get like uh partial traces somewhere here uh and uh because uh that is pretty cheap way of doing the sampling.

A

It has very minimal performance overhead, especially when the trace is not sampled. All your uh instrumentation is really effectively a no-op um you you get very little overhead, um and this is the default mode. That's supported by jager sdk. The downside of upfront sampling is that uh your 99 of your requests in the system are gonna, be normal and not particularly interesting right.

A

You really want to look at outliers in terms of latency, maybe errors uh and those happen much more rarely and as a result, if you also in like, let's say they happened, 100 000 times and plus you also have a sampling rate and one thousand then really your chance to get an outlier or anomaly is one in a million.

A

uh So that's kind of a bad thing about head based, sampling and, unfortunately, there's not much that can be improved about it because uh it simply doesn't know anything about what will happen to the trace when it makes a sampling decision. It's like done at the beginning, and so um the way the head-based sampling is done in jaeger is that each sdk can be configured with different samplers. You can say: use probabilistic sample like a coin flip with a certain rates, or you can use rate limit and say in like this many per second.

A

uh But uh the interesting part is that is that all sdk supports so-called remote sampling, where, as I mentioned in the architecture diagram, the configuration actually comes from the back end and that's very powerful, because when you have very many services in your architecture and many different teams running those services, those teams don't necessarily know what kind of sampling is good for for that system.

A

uh They also don't know when the traffic patterns changes and how do you need to reconfigure the sample or redeployed application, uh whereas when the configuration comes from the from the center location, you could do all of that in a much more intelligent way at minimum.

A

You you give control to the sampling to the team that runs the tracing infrastructure so that they have sort of like levers in terms of how much traffic they want to ingest, um and uh but you can also make something more intelligent, like adaptive sampling, which calculates things on a sort of a control loop and reacts to traffic spikes. This is what we use at uber, um and the one thing is is also interesting. Is the configuration can be done per service and tournament points?

A

So it's very often the case that a given service may have multiple endpoints, with very different uh rates of of queries to them or requests, and so you don't want to sample everything at- let's say one percent, if the difference, if you have multiple like orders of magnitude in the qps of the end points, so this configuration allows you to do at the individual level, and so you can read the documentation of how to configure it now.

A

Let's talk about tail based sounding so tail by sampling is, is different, uh completely different mode, where the sampling decision is made at the end of the trace, and because of that it can be much more intelligent. We can look at the latencies. We can look at the errors or some logs. Whatever anything that looks interesting in the trace, we can. We can affect how we sample those traces uh um in in an interesting ways right.

A

However, that uh requires that all these traces still need to be stored somewhere because, um like traces are distributed, you have all these spans cramming from all these different applications. You kind of need to assemble them all in one place. First and you don't want to store them on disk during that assembly time, because uh then we defeat the purpose of sampling.

A

We we want not to hit the disk, because it's very expensive, so it really means you have to allocate a lot of memory to store all these traces until they're done, and you can make a sampling decision, fortunately they're, all short, usually traces like last no more than a second, so most of them can be expired from memory very quickly, so it doesn't necessarily introduce a lot of memory overhead, um uh but there are some like architectural things. You need to do to support that and another another kind of downside of tail-based sampling.

A

Is that because we need to collect all the data from from all this every single request? That means that it has the maximum performance overhead on your application right, and so it's a trade-off. Whether you want to afford that and maybe in increase in latency, you can sometimes combine the tail-based and head-based sampling. So let's say instead of sampling, one in a thousand. You say well, let's sample one and ten, but then do like for every tenth.

A

We kind of do the tail-based sampling so that that allows you to control the costs and performance uh so how sampling tail based sampling works in the aeger. So there's nothing that needs to be done on the jaeger sdks because you simply configure them with either 100 or like a fixed percentage, uh where the the what the magic happens is really in on the back end, but jager components themselves, don't support tail by sampling, but we now release uh new components called open, telemetry collectors.

A

uh Those are jaeger, binaries, specifically built with from the open telemetry collector. We just support tail by sampling and you can configure those collectors with various sampling rules by like latency or certain tags like error tags. um Unfortunately, at this point, uh open telemeter collector only supports a single node mode, which means, if you can fit all your traces uh in one node memory, then you're, okay.

A

But if you need a kind of a scaled out solution, uh then that's not currently available, but there is work already happening and there's a blog post by grafana how they did it. So it will. It will be available in the near future uh and, finally, uh to close this, I mentioned that I want to talk about uh open telemetry, so open telemetry is a new project in cncf, which is a it. It's a descendant of open uh tracing and open sensors.

A

These two projects merge and it deals with again establishing a unified instrumentation framework so that you can reuse the instrumentation in multiple applications right. It does not deal with the back end collection of traces, except for the collector, which is kind of an intermediate piece so to to illustrate that let's say uh we're we're dealing normally with jaeger and open tracing. This is how jaeger historically evolved, and so you have application your application at the top there's like three types of applications that typically can be instrumented for tracing.

A

Some of them are explicitly instrumented directly in your code. You can start spans and write where is metadata into it or more often you're in the middle uh box.

A

Here, where you use some rpc framework, which comes with the middleware for tracing um or there's another option where you can sometimes there are in certain languages, there are auto instrumentation facilities where you can just attach a library to your binary, and then you magically get traces like in open tracing, that's available, for example, in python and in java, and then all these instrumentations they talk through open tracing api which, as I mentioned, calls back into the jager sdk and then jaeger sdk sends the data back to the jager back end right.

A

So it's very kind of all clear here. So now what happens with open telemetry, uh so open telemetry presents a different, slightly different api than open tracing. It's conceptually still very similar traces and spans and everything. uh But the method names are slightly different. So but you can just use the different types of instrumentations for open telemetry and then you don't have to have uh or run jaeger sdk.

A

You can just run standard open, telemetry sdk, which is included in the project in all languages right and then that sdk has the ability to export data directly in the jager format.

A

So you can still have jager agent or jaeger collector, except in jaeger data spans, uh but you can also, alternatively, run open telemetry collector directly, which and then export data in the open, telemetry format, which is kind of a standardized way now to represent traces and jager, is gradually migrating to that uh and then open telemetry collector will still be able to forward data to jager back and in the format.

A

And uh so, as I mentioned, jaeger uh components now exist that extend open telemetry because collector in open telemetry is written on go. We can just use it as a library, so we've built our own versions of those binaries which have the same capabilities as a open, telemetry drop stream collectors, but with additional jaeger extensions, which are kind of uh specific to jaeger.

A

For example, we can plug in directly our storage implementations into a collector so that you don't have to run like multiple services to uh to push the data through um and we're also converting uh jaeger uh implementations for storage now to work directly with the open telemeter data model, uh which is slightly richer than open tracing, because it will kind of provide us better compatibility and uh path forward, uh and so we're trying to uh kind of reuse.

A

All the good thing that open telemetry is building so that we can reduce the efforts we need to maintain, for example, all the jager sdks, which was pretty expensive um work in the past. um So if you want to learn more about uh uh jager uh you can you can attend a deep dive which will happen uh on the next day uh and so pavel will be talking about more about jaeger architecture and other things. And finally, these are uh the ways that you can get in touch with uh with the community of jaeger project.

A

There is a blog post, there's a twitter account that you can follow and there's online chat where you can go and ask questions things like that. So that's the end of my talk. Thank you very much.