Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jaeger Deep Dive - Pavol Loffay, Red Hat

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Jaeger Deep Dive - Pavol Loffay, Red Hat

This session is dedicated to an in-depth understanding of the Jaeger project. We will give a short demo of the recently added features, talk about various topics including the architecture, adaptive sampling, multi-tenancy, and configuration, and review the roadmap. After this session the attendees should better understand the Jaeger architecture, how to deploy it and get the best benefits, and to make contributions to the project.

https://sched.co/Zexk

A

Hello, everybody and welcome to jaeger deep dive project session. This is the second presentation uh at cloud native con virtual. The first one is jager intro and uh if you haven't seen jaeger in action, I encourage you to watch recording, uh I'm pretty sure there is very nice uh demo uh of jaeger in action.

A

So in this talk the agenda is we'll talk about the project in general, then I will spend some time talking about how tracing works under the hood. We'll have a look at two concepts, uh so I'll explain how the data collection works inside your application, but also how the tracing infrastructure works under the hood.

A

Then we will jump through a couple of different topics, namely sampling, jaeger integration and architecture and zip incompatibility and last but not least, uh we'll spend some time talking about new features since the last kubecon something about roadmap. And finally, we will do q a so just briefly about myself, I'm a pablo lopheim, I'm software engineer at red hat and I have been working on jaeger since basically, it has been open sourced in 2017 and apart for jaeger, I contribute to a bunch of other projects that are distributed.

A

Tracing related, namely open, tracing, microprofile, open tracing and also open telemetry project.

A

So jaeger is a distributed tracing platform- it's not just you know, a single server that you run on your laptop or or as a deployment, it's much more than that. It's the whole ecosystem of different components for different purposes.

A

So on one side there are instrumentation libraries or sdks that run inside your application and they collect data and they report it to the tracing collection backend.

A

On the other side, there is a visualization or the user interface that we use to to see our distributed transactions and in middle there's, also a data mining integrations and platform that it used to to aggregate the data and derive. I know another interesting insights from the data would we collect so jaeger is a project, it's inspired by google dapper and open zipkin, and it's mainly because it uses very similar data model.

A

As these two system use, uh it has been created uh at uber in august 2017 and two years later, it's uh it has been open sourced uh in 2017.

A

the same year. It also joined cncf foundation as an incubating project and last year. So, basically, again, two years later, it graduated to top level cncf project.

A

So from the technology stack, jaeger itself is written uh in golang and it's a very pluggable system for storing data so with upstream project, or we directly maintain implementations for cassandra, elasticsearch badger and in memory for demo uh and test purposes.

A

But the storage layer is very is pluggable, but it's also quite simple to implement. So if you have a expertise in running different storage, uh it's very simple to to implement storage right and uh and query interfaces.

A

The ui is um is rack.js. It's very nice ui. uh You will see that you know in a jager intro and the collection, libraries or the sdks, they implement open tracing api and they are available in many languages.

A

So, for example, in go java, python, node.js c, plus c sharp php and ruby jaeger also integrates with apache kafka and flink and spark for streaming and data processing. We will see that more in architecture.

A

So, as a community, uh jaeger is hosted on github uh under get racing organization. The main repository has roughly around 11 000 stars, and there is about 1 700 contributors across the all repositories.

A

Inside the main organization, there is roughly 15 maintainers again across all the components that are in the organization. uh We do have also twitter channel uh and guitar channel.

A

Well, where you can ask questions uh or race problems, uh and we always try to help our users to to resolve uh issues and uh get uh help them to get started with jaeger.

A

We do release roughly every one to three months. uh We always try to release once we have reasonable set of new features uh and jager itself is distributed as a set of docker images, uh but also said as a set of binaries that you can get from the github release. Page.

A

So how jig works, but more in general, how tracing works under the hood.

A

So on this diagram uh we see a bunch of stuff right uh and basically there are two important pieces here on the top. There are two micro services, so our standard applications and on the bottom there is tracing infrastructure and we see this micro services that they embed tracing api and tracing library.

A

So the tracing api is something that you know. The application uses to report tracing data to create events and the tracing library is the implementation of this api that reposits data to the tracing system.

A

Now, if we zoom into the application, we see that uh the application consists of different layers. There is some inbound layer. Then there is some. You know something in the middle so, for example, business logic and then there is a the outbound layer which can be, for example, an rpc layer to the next service and every layer talks to tracing api, and we call these points trace points. So uh so this transport trace points.

A

They report tracing data via the tracing api and uh and so with the strains points we call them also instrumentation. So instrumentation is just you know, a a piece of code that basically emits tracing events uh based on the behavior of the application.

A

This is a little bit different to metrics and logs, uh because tracing uh is stateful, so it carries some context with it. So when we see when a request arrives to the first micro service, what tracing api does uh it will create a special context with unique ids?

A

These ids are also generate generated by the tracing api. Now this context is propagated to all downstream. uh You know, instrumentation points and all downstream microservices. The context doesn't change, and this is important because the context doesn't change, because all these events will have the same ids.

A

So later the tracing system can correlate all these ends together and reconstruct the full trace or the execution flow.

A

And so the context propagation is basically the foundation of distributed tracing.

A

So let's have a look at the tracing infrastructure itself, uh so once the data leaves the tracing library, it goes usually to uh to collector or agent that stores data to a trace storage.

A

Apart from trace storage, there is also some data, mining and the presentation layer.

A

So let's have a look at how the tracing works uh with uh with open tracing and yeager, so open tracing a jaeger, many people, they use these technologies interchangeably. But actually there is a clear difference between these two.

A

So on this diagram we see again on the top, the application or the process and on the bottom, the tracing infrastructure, the application. We see that it's instrumented in two different ways: it uses the manually instrumented frameworks and automatic instrumentation so manually, instrumented frameworks. This is just something like a library uh that you can pull in into your class path and it will emit tracing events now automatic instrumentation uh is, for example, java agent that does bytecode manipulation to instrument your application.

A

So all these instrumentations they basically use just open tracing api and then the tracing library implementation. In this case, jaeger just implements this api and reports data to the agent or collector, so the tracing api is important because uh there's there are so many trains points in our application, and so, if we wanted to change the implementation- and we haven't used the api, we do have to change all of those uh calls to the tracing api.

A

So the tracing api basically abstracts the implementation, which makes it easier for us to switch to implementation, which is very important. uh However, with open tracing there is one slight problem is that if you want to change the implementation, we have to recompile and redeploy our applications.

A

So once we look at the bottom, there is agent and collector, so the the library that lives in our process reports, data to the agent or collector, uh and it can choose where the data will report it right difference between agent and collector is that agent lives very close to the application.

A

So it's something like sidecar container or it lives on the same host as the application process.

A

Then the data goes to collector and collector just stores the data into storage.

A

Now, let's have a look at the open, telemetry versus jaeger, so we see the open. Telemetry is also the instrumentation api, but it's also the tracing implementation, and this has couple of advantages over open tracing, because now, if you want to change the implementation, we don't have to recompile and redeploy our application, which is a big plus.

A

What is maybe confusing on this slide is that open telemetry is also the agent and the collector. So this is a new component in open telemetry, which is called collector and collector is a component that can receive the data, can process it and then can send it to the other systems.

A

So difference between jaeger, collector and open telemetry collector is that jager collector is also able to store the data in a storage.

A

So, let's have a look at sampling, so sampling is a technique, uh how we store less data that we actually receive, uh and there are basically two uh approaches how to do sampling the first one is coherent, head-based, sampling and other one is tail-based sampling.

A

The coherent, head-based sampling means that the sampling decision is made when a request.

A

First time reaches our infrastructure, and so we make the decision. And then the decision is propagated to all downstream services and these services they have to respect the decision that has been made at the beginning, and there are two implementations how we can sample at the hat. The first one is probabilistic and the second one is rate limiting probabilistic basically says, uh for example, sample only one percent of all the requests. What you receive brain limiting says, for example, sample only 10 requests per time. You need so, for example, 10 requests per second.

A

Sampling is usually configured per service, and so this is a problem because some services they receive a lot of traffic at some end point but other ends points might receive very low traffic and if we use a single sampling configuration across all those endpoints, the end points with low traffic will pretty much not be sampled at all. So jager allows you to configure tracing also per endpoint level to solve this problem.

A

The other feature what jager provides is centrally controlled sample, sampling and so in jaeger. There is a configuration file where you can configure all the sampling strategies and this configuration is then pushed to jager clients reporting the data, and so the other approach to sampling is tell-based. Sampling jaeger at the moment doesn't do tell-based sampling. It is something that we are looking at uh and I will talk about it uh uh in the roadmap, so integrations jaeger integrates very well with kubernetes.

A

There is jaeger operator and helm chart, so the operator is a component that can deploy but also manage and upgrade your applications. So it's much more robust way of deploying uh you know business application into kubernetes.

A

If you want to use plain kubernetes, manifest uh eager operator can be used as a cli tool to just generate this manifest. Based on the configuration you provide to the operator.

A

uh The next integration with is with prometheus, so jager server and all the client libraries they report metrics in primitives format, the open telemetry integration, all the open, telemetry clients or sdks they ship with jaeger exporter.

A

So you can use open, thermistor sdks with jager beckoned, an open thermometric, collector ships with jager receiver and exporter.

A

From the service mesh linker d and istio, they come with jaeger as a as a standard tracing system and ny uses native jaeger, ziploc splash.

A

Client, so let's have a look at architecture, so when jaeger was introduced, this was the the default architecture. So, on the left side there is the host and container. So our application are reporting data to the collector and then collector was storing data to storage, which then jager query was using to get the data for the user interface.

A

The data processing was done in batches, so spark job was reading the whole indices and tables from database. It calculated some results and stored them back to storage and, uh and so this architecture had a couple of problems, uh especially with scalability uh and uh traffic spikes.

A

So if there was too much data going to the collector, the storage couldn't keep up. So we just dropped the data and also we couldn't calculate uh sometimes the uh the aggregation. We couldn't run the aggregation jobs because there was just too much data uh in the database and it was very resource uh heavy to run the spark jobs because they had to pull a lot of data from the database into memory.

A

So to address this problem, we introduced kafka that sits between the collector and storage so collector. Instead of writing to database, it writes data to storage and then there's ingestor that just reads data from kafka and pushes them to database.

A

This allows us to run streaming jobs so instead of patch jobs that basically these streaming jobs, they connect to the same topic in kafka and push data or push results to to storage.

A

So zipking compatibility, so jaeger is fully compatible with zipkin and can ingest zipped in data in pretty much all the formats, and it can also read zipping data from kafka from the client side, jager clients. They understand, zipking context, a context, propagation protocol b3, so you can use zipkin instrumented applications with the acres in instrumented.

A

Applications, so what is new in jaeger since the last kubecon in north america?

A

So we have done a lot of progress. There has been a lot of new features, a lot of boxes, bug fixes and a lot of stability improvements.

A

So one of the major features is data loss, statistics for clients, so jaeger clients they can drop data. It's obviously a problem with clients drop data. They drop the data because they want to keep a reasonable quality of service. They don't want to overlap, overwhelm the back end, and so this is a problem, but it used to be bigger problem because we couldn't tell how much data the clients are dropping. So now there is a way how clients can notify the backend, uh how much data they dropped.

A

We have also done a lot of security improvements so, for example, kafka, basic authentication and tls for query service and also eager operator it's able to automatically provision certificates, but also renew them if they expire from the usability standpoint uh we compiled and we distribute jaeger for different architectures.

A

uh So apart from linux and windows, we added arm 64 and ppc 64..

A

The next topic is trace dsl, and this is something that I was personally really excited about, and I uh I worked on this, so we wanted to make it easier for data scientists and devops people to uh basically run programmatic uh queries against jaeger and to make this possible and simple, uh we developed a trace, dsl a trace domain, specific language uh that is able to you know, query data, but also extract features from the data, and so this dsl is used in jupyter notebooks that connect to jaeger query, but also to kafka for streaming processing, uh and so the main use case for this is that sometimes, if you want to verify your hypothesis, uh the verification requires aggregating a lot of data, and this is just not possible in the ui, because you cannot aggregate or you cannot manually, investigate thousands of traces in a user interface to really run such a such investigation.

A

You have to write a piece of code that would do it for you.

A

The next uh new feature is trace, quality matrix, so the same repository uh with the trace dsl. It also provides some support for uh for metrics, uh and this metrics basically says uh how well is your: are your application instrumented?

A

uh So this is very crucial, because if your environment is not well instrumented, you will not get the full power of distributed tracing and the last new feature is rebase on open telemetry.

A

So we have basically rebased all jager beckon components: agent, collector ingester, all in one on top of open, telemetry collector, so all jaeger, all these new components are basically open geometry collector and we just added storage functionality to this new component.

A

So on this slide, you can see the configuration of new jager, open geometry collector, uh and what is different to to the upstream open telemetry is that these components can store data to storage.

A

So we see the section with exporters and with the there we see jaeger elasticsearch, and we see that jager elasticsearch, which is just uh you know, it's the standard, elasticsearch configuration.

A

And to make this very simple, we have made couple of decisions to to support simple upgrade from the current jager components to this new set of binaries, uh and so we support uh the current jaeger configuration. So you can also configure this new component uh with the current jaeger flags and current dagger configuration file.

A

There's a lot of remaining work if you're interested to to participate or see the progress you can subscribe to to jager tracing, slash, jager repository and watch for issues tagged with arya, auto.

A

Okay, so what is on the road map? So we will continue with uh integration with open telemetry and once open, telemetry collector reaches ga. We will release this new set of components based on open, telemetry collector, and we are also looking at scalable tail-based sampling from open telemetry.

A

If you want to learn more about jaeger, there is egotracing.io website, but also block uh on the medium, where we publish interesting topics uh related to jaeger, but also to tracing in general.

A

If you want to get in touch, the best place is probably the the community channel, but also github and mailing list. We also do b weekly community meetings where you can connect and and talk to us in person.

A

Okay, this is everything what I have prepared for today and if you have any questions, feel free to ask them in chat.