Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tips And Tricks To Successfully Migrate From Jaeger To OpenTelemetry - Vineeth Pothulapati

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Tips And Tricks To Successfully Migrate From Jaeger To OpenTelemetry - Vineeth Pothulapati, Timescale Inc

Speakers: Vineeth Pothulapati
With Jaeger announcing the end of life for all Jaeger client libraries and recommending all users to migrate to OpenTelemetry SDKs, for many users, it is becoming important to plan an effective migration from Jaeger to OpenTelemetry. But this migration isn't simple. There are multiple ways to perform it, each one implying different tradeoffs; besides, Jaeger has some fantastic features (such as remote-controlled samplers or per- operation-based adaptive sampling) that could be lost during the migration, if best practices are not followed. In this talk, we’ll discuss some of the best approaches to the migration from Jaeger to OpenTelemetry, the different paths that organizations can take and its related implications, and what users can do to not lose any of the Jager functionality they really love when moving to OpenTelemetry. We will illustrate everything with a demo application, which will be using the Jaeger client at the start of the session and will have migrated to OpenTelemetry by the end.

A

Hello: everyone uh thanks for joining today's session and today's session is about tips and tricks to migrate from eager to open Telemetry. So, first things. First, how many of you are using eager already, okay, open Telemetry? Have you started using open, telemetry, nice?

A

Okay, then I think you will understand most of the eager parts and open Telemetry parts that I'll be discussing today. So the agenda for today is prerequisites first, setting the ground correct. First, like what are the expectations from the talk and other things and why we have to migrate, and we will also go through a brief architectural overview of both eager and open Telemetry, and we will be talking about levels of migration and I also have a demo on how we migrate an application, and the last one is eager and open Telemetry boundaries.

A

So today everyone is a bit. uh uh Why are you didn't confuse that? How how and where do I plug agar over open telemetry?

A

So about me my name is vinit potlapati I'm, a product manager at timescale, I work on the observability products at time, scale primarily focused on Prom scale and tops I'm. Also, a maintainer of open Telemetry operator, so, if you're already using an open, Telemetry operator, would love to hear out your experience and if you would like to check out the products that I work from skill and tops, they are open source. So the links are shared at the bottom. uh It's time scale slash prom scale and time skill.

A

Slash tops, okay, so the prerequisites so throughout the session I'll also be using Tom as Hotel, so open, Telemetry, synonym or a short form is Hotel. So the commonly used phrase so I'll be using that more often and the stock is focused on traces, so open Telemetry is also getting into metrics, logs and other things, but we will be confined and focusing on traces. The stock isn't intended to push the migration or I'm not selling.

A

Anything here, I'll just share what are the ways how you can mix and match the things and what are the upsides of migrating and, let's understand the components. First, before we get into the talk, so we know there are multiple components involved in tracing both in eager and open Telemetry, so the instrumentation layer usually has API and SDK. So in eager we use open tracing API and the SDK is eager.

A

Client, libraries and the agent and collector is uh all eager, so eager also has an agent and collector, and it also offers some native storage options within the collector, uh like Cassandra elasticsearch, and there is a grpc based uh mechanism to plug the external databases as well and eager also has a visualization layer which we'll be discussing later today and how it helps you to visualize the traces, even if you are using open, Telemetry and in open Telemetry.

A

All it has is the instrumentation layer and The Collector layer, so eager is all the way from instrumentation to storage and visualization, but open telemetries on the instrumentation and collection stage. Okay, why you have to migrate?

A

Now we have been hearing eager and been using agar for a while and igora served really well for most of us and people, love, Jaeger and open Telemetry has been announced in 2019 cubicon na. So exactly it's like four years uh in San Diego and since then the project has been matured and it's expanding itself into different observability signals and adding new capabilities. So, let's discuss on what are those capabilities and why does migration would make sense?

A

So the first thing is eager client libraries have been announced, end of life support earlier this year and eager community, and the maintainers also are supportive and advocating towards open, Telemetry SDK. So if you are using eager already, you have to move from eager client libraries to the open, Telemetry SDK and will be briefly talking on how to do the instrumentation layer, migration and the second one is open.

A

Telemetry is a new standard, as we all know for in instrumentation and collection of data, so it takes some industry best practices that were part of open tracing, open, sensors and all, and it also adds new capabilities that were not existing in eager instrumentation side so and the collection layer is really rich. It gives you options to configure different sources and destinations to ship. Your data and, and the last thing is open. Telemetry does support Auto instrumentation, which means you don't have to do any code changes.

A

You can just run a sidecar and it just works so levels of migration. So there are two levels of migration: one is the instrumentation layer, the other one is the collector layer before we get into the migrations and how it's done. Let's do a quick uh walk through the in eager and open Telemetry architecture. So the first slide is the eager architecture you can see. On the left side. There is an application, the eager client and the eager agent.

A

So the application is instrumented using eager, that's uh and the spans are being pushed to eager agent and from there you can see there is eager collector and the database and the UI. So this is the complete echo system and the components that are involved in eager architecture and the spark jobs are something very optional. If you need them, you can run uh coming to the open, Telemetry architecture. So here in the open, Telemetry architecture, we do not see the UI layer or the storage layer.

A

As I said, it's all about the instrumentation and data processing pipeline, which is open, Telemetry collector itself, so you can see here open, Telemetry uh collector being ran as an agent. So um so there is so much of confusion between this agent and collector. So we collect the agent or something if it is run within the host or as a site card to the application.

A

Then we say it as an agent, but it's the same, collector, that's being running and collector acts as a centralized processing pipeline, where directly your applications can send the spans to The Collector or your agents can send the data to The Collector.

A

Okay, let's start with the instrumentation layer, so you can see in the instrumentation layer. As I said there is this API and SDK so to understand what are the components involved in API? You can see this image. It has the Tracer API context API and the meter APS the meter apis for metrics and in the SDK you can see there is this propagator span, processor and aggregator. So these are the functionalities that the API and SDK offer you during the instrumentation.

A

So, as I said, the migration can be done in instrumentation using two ways which is open, Telemetry, shim and complete reinstmentation. So we will be doing a demo and discussing more about the re-instrumentation, but there are libraries already available uh in the internet uh on how to do the open, Telemetry migration using the shim. There is a blog post in medium, which uses the Java application as a demo to migrate from open uh the eager client libraries to the uh open Telemetry using the shim. So the shim is all about.

A

It consists: a set of classes that implement the open tracing while still using the open, open, Telemetry construct. So with a very minimal code. Change like the blog post, that's been authored by jurassi's uh published in the uh internet, so it's already available and it's just hardly takes five minutes to migrate by just swapping out the dependencies and the Imports and the second one is uh it's all. It's also helps you to do a to get on to the open, Telemetry SDK. In no time you don't have to do the complete reinstmentation.

A

So if you have less bandwidth and if you want to use uh some client base sampling in the open Telemetry or you want to use open, Telemetry SDK for some reason, you can definitely start with a shim, and this is the blog post. That's I've been referencing to that's, been mentioned here and the complete reinstmentation, so why you have to do the complete reinstrumentation, so the complete reinstrumentation offers a hotel as a package.

A

So you get all the capabilities right from the scratch from the code of open, Telemetry, the semantics and in future you can also expand your open Telemetry instrumentation into metrics and logs. So this is just a starting point for you if you get started with open Telemetry and it's also easily integrated with auto instrumentation application.

A

So if you, if you want to do an auto instrumentation for few applications and if you have other applications, if you which you need more granular details, so the auto instrumentation gives more often higher level traces and spans with manual instrumentation you'll have more flexibility over what you want to capture and what you want to measure. So it easily gets integrated with auto instrumentation and the normal apps.

A

So it's demo time, let's just get started so my demo is, uh is a clone from open tracing tutorial. So there is an open tracing tutorial in uh in GitHub authored by Yuri to understand how the instrumentation works so I'm, just taking the same application, I'll be showing you what's the eager instrumentation like and how's the opened Elementary instrumentation is like, and it's a very simple app. So we have a greeting client and we have a formatter and a publisher.

A

So the greeting client basically sends a request to the formatter and the formatter, basically formats the string and then greeting client publishes the string to the publisher just to do an STD out. So the demo is available in this GitHub repository, so I just pushed the demo to uh to the GitHub.

A

Okay I hope the demo works. Okay. So, let's jump into the code before we run it and see the differences. So I have the eager application here and the this is the eager client go. uh So no. The this is a client application which basically sends the request to the formatter and the publisher.

A

So here you can see it's all the normal code. It's nothing fancy it's a main uh here. You have the main and then the interesting part is here in the tracing. You have to see the init Tracer, so we all know that if we want to start with the tracing, there is an init function which basically gives you the Tracer. And then you start using the Tracer referencing it across your life cycle of the code.

A

So here we have started uh instrumenting and we are saying okay start span and then finish- and here we're also using the baggage as an example, how you can set the baggage to do the context, propagation and pulling it. So if you do not know the featured baggage, so it's a it's. Basically, it gives you an ability to attach metadata during the context propagation without a need to change the API.

A

So if you have to send some metadata from your application, a to all the way down after four requests, you can just put it in the context and it just flows through the last application that you want to send it so without any disturbance or changing the intermediate applications to capture it and forward it forward. So it just gets attached to the context. So here uh we are calling the format string function and it's making an HTTP call to 8081 and we are injecting a few data.

A

So the some data, which is context and HTTP headers- and this is HTTP headers carrier here and yeah. We are doing a request and yeah. Then we are also calling the print hello, which is a publisher service. Here, is the publisher service, so yeah and the publisher, the formulator service is nothing but a server just running uh just waiting for the request to format the string and send it out, and if you see the publisher service, the publisher service is all about just take the request and do an STD out.

A

Okay, so first we will run the eager all in one which is uh which basically runs the eager UI eager collector and in memory storage by default.

A

I hope the font is visible. Okay,.

A

So the eager collector is up eager all in one and now, let's.

A

Let's run the eager application so now, I'm, first running the publisher, go run publisher, dot, go it's up.

A

A

So I'm sending a string as a greeting called hello cubicon from my client to the formatter and then publisher. So we can see now the formatter is running, the publisher is running and the client is uh here so first, let's check the eager UI.

A

So as I just uh restarted, The Collector, so now you can see uh okay, there are no services or no data because we haven't yet pushed the data to Jaeger. It's just the eager, UI Egger, query spans itself and now I'm just doing hello cubicon and the demo just works, as this is authored by Jurassic like this is this is not uh not just Yuri, so it's from open tracing demo and now, let's check the traces here.

A

So the traces are here and you can see it's. It says hello and you can see the request has went from Hello World to formatter to publisher. Now, let's just check the open Telemetry code and check how it works. So I'll be a bit quick here: yeah uh the open. This is the open, Telemetry instrumentation. So now you can see you have used the hotel, Trace baggage and all the inputs here so I'm not using anything from eager or open Telemetry. And it's it's the same.

A

You just init the Tracer and then you can start the context and uh I feel a few areas. It's even simplifies, for example, doing the context baggage and headers. It felt easy for me at least in go, and here we are doing an HTTP request uh to uh formatter, and here you have publisher. So let's now run open, Telemetry and check the spans.

A

A

The formatter is up, and now let's do the same from the client, but it's instrumented using Hotel.

A

Okay, so I'm just doing the same now, hello, cubicon.

A

A

Did I change anything no.

A

Let's just check it out, that's format.

A

Okay, I'll just give it another 30 seconds. Let's just check I just.

A

Let's just check it is: did the spans come out.

A

Now the spans aren't out from open Telemetry, yet.

A

Okay, I think I got it yep. Now it just worked. It says the comma yep now you here, you can see the greeting client and uh it's just a few seconds ago. It's been, it has come from the open Telemetry, and here we can go to the formatter service and check the tags. So now here it says, uh go client and it's instrumented using the library open telemetry.io. So these are the spans from open, Telemetry instrumented using open, Telemetry and they're the same.

A

So this is what I wanted to show it's a very small demo But. It includes all context, propagation using the baggage, doing an init trace and all like the basic stuff that we use during the instrumentation and yeah. That's that's! The demo I had just to show how easy it is like to run and use agar alongside open, Telemetry instrumentation. So now the eager collector is running and uh the recent feature in eager has supported uh ingesting the otlp uh traces uh coming from your application.

A

So the data is coming in the otlp format as the exporter, and now we are pushing it to the eager and another interesting fact that I miss to explain is that. Okay, if you see the Tracer here, you can I have written a code in a way that you can.

A

Okay, you I have written a code in this way that now you can configure the eager, endpoint or open Telemetry endpoint. It just uses the uh any Tracer. If you want to use eager endpoint, you want to send to the eager it's it's this init provider for eager. If you want to use open Telemetry, you can use open Telemetry, it's just logic, to send traces that are being instrumented using open, Telemetry in both eager and open Telemetry formats.

A

So if you have some limitation on the backend that you have to send traces only in the eager format, you can use something like this and it just works.

A

Okay and the question comes as the context propagation like now, you have tens of applications that are already instrumented using agar and you you are starting a new application uh from scratch for the development. The always the question is: do I need to start with eager or do I need to start with open Telemetry. So the recommendation is, you should start with open, Telemetry and the context. Propagation just works because eager supports eager supports the eager, B3 and W3 see context.

A

Propagation and open Telemetry does support all three, but if you have different applications talking to a same application in different context, propagation, the open, Telemetry SDK also supports uh enabling or using multiple contacts at the same time. So if application is instrumented using open Telemetry, it can understand it can receive the request using the context, propagation eager, w3c and B3 at a time.

A

So you don't have to uh just stick with only one context propagation, so it just works, and there is a article in medium written on explaining the same, how you can do mix and match match of context propagation between the applications? Okay, the impact.

A

So now we have seen the migration and how you can do the migration now, what's the impact after you migrate, so obviously you have an improved Tracer implementation, as I said, the baggage and other things just work and the switch to the open, Telemetry SDK, while using uh the open, open tracing, is also a possibility, as using the shim just makes this easy for you, and there is also an improved performance. I keep reading in the readme and in other places, so the open, Telemetry sdks are more performant and access to open, Telemetry framework plugins.

A

So we are, there. Is uh each language has its own way of integrating into the SDK, so it it has a feature Rich framework, plugins and now let us understand migrating in The, Collector layer. So, as I said, we have two levels of migration. One is the instrumentation layer that we have seen that's the code, and now we will see the migration into open Telemetry without touching the code or without disturbing your applications.

A

So this is uh the usual architecture. When you are deploying eager collector and on the left and on the right, you can see how you can add open Telemetry collector into your existing architecture. So you don't have to do any open. Telemetry code changes in your applications. You can just use open, Telemetry character, because open Telemetry collector can receive the data from eager and different formats. So we'll be talking about how you can configure the collectors in the upcoming slides, but this slide is all about.

A

If you have an architecture as a left acre collector, your applications are sending traces from eager to The Collector and it's going to the stage storage backend. All you have to do is put an open, Telemetry, collector in between the applications and regular collector. So the question now comes is why you need an open, Telemetry collector when you already have an acre collector, so the open, Telemetry collector eases the uses, the data collection pipeline.

A

So if you have metrics traces logs all coming into from your applications, you can just run open, Telemetry collector and it can do it can process. All this, you can configure the pipelines. For example, you can also use span metrics processor, which is which basically generates the metrics out of your spans, and the processor is available in The Collector and here in the bottom I'm just putting a storage backend as prompt scale, so it just works with open, Telemetry and eager and let's understand the differences between the eager and open Telemetry collectors.

A

So the eager collector has more flexible sampling, whereas open Telemetry collector has sampling processors too, but the eager collector has more in terms of, for example, the Adaptive sampling has been released in eager one two six, so it's available in collector I'm, not sure it's there in the open Telemetry but yeah. You have wide uh options for you for the sampling and eager collector and an eager collector. You have entry uh storage options where you can use Cassandra elasticsearch or use the grpc based storage plugin for the backend in open, Telemetry collector.

A

As I said, you can use receivers. Exporters and processors, which helps you to build the data pipelines, com data coming from different sources to the different destinations.

A

So this is the hotel collector configuration you can see on the left. It has the receivers, exporters and services. So in receivers you basically configure what are your sources? Where will you receive the data from and in the exporters? You will basically configure where you want to forward the data to in Services. You will basically map this receivers and exporters to build a pipeline.

A

So, on the right hand, side you can see my open Telemetry collector is receiving data from applications which are instrumented using eager and in the middle you can see the applications being instrumented using open Telemetry, and you also have a Prometheus receiver, which can scrape the targets to get the metrics. So you can just configure the receivers and it just works, and even during the exporters you can parallely stream the data to your SAS backends or in in-house storage systems anywhere.

A

So you can also duplicate the data between systems and if you want to move away from one storage engine to another, there is zero lock. In all, you have to do is change some configuration in the open, Telemetry collector and it just works. So this is the anatomy of the open Telemetry collector. So you can see if you see The Collector on the left side. It's all receivers. It's otlp receiver, Zipkin eager receiver.

A

So if you have any Legacy applications or any other applications which are sending you some data in different formats, Hotel collector should definitely have the receivers for them. I think there are already like 20 plus receivers in the open, Telemetry collector country. We can just use them, and these are the exporters, and you also have processors where you can translate. The data, do some kind of sampling, filtering and everything and the exporters basically map the state or send the data to the backends and the impact.

A

So if you use the open, Telemetry collector, as I said, this makes the transition much easier and it's easy to start. It's all about just running the open, Telemetry collector and it's it gives you the ability to build the pipelines on how you want to send the data and where you want to send the data uh and all yep, and in now you have seen in the in the second migration which is okay, you can add open Telemetry collector uh in in the with your a good collector.

A

So now we will also see how why you have to use otil in Jaeger, so you can receive data from multiple sources and process the data using wide data processing tools, and you can also export it to the multiple backends and it also supports metrics, traces and other things. So here you can see uh it's the same, but here we are adding the open, uh uh open, Telemetry collector, just in front of eager collector, so that it just works.

A

It just sends the data to the eager collector, whereas in the past we have seen, uh we are just replacing the eager collector with open Telemetry collector, so it just works. So if you have any storage, backend, that's compatible with open Telemetry collector. You just put the exporter saying that hey I want to send the data to Cassandra or prom scale. You just configure the exporter and it just sends the data here.

A

If you are using eager in memory or Badger storage, you can still use the eager collector, but all you have to do is configure eager exporter and it forwards. The data coming from your applications to open Telemetry collector to the eager, collector yep and the impact is you can get best in both the world. So you can, if you have any sampling mechanisms that needs to be done, you can do a sampling in open, Telemetry collector and you can do a sampling in eager collectors too, but it gets complex.

A

You just need to take some conscious decisions on how you want to do, but you can Leverage The, in-memory and Badger storages and eager also offers a grpc based remote right plugin, so it just plugs into the databases that offer that integration and querying and visualizing the traces. So if you move to the open, Telemetry collector, there is no uh path to how you connect to your traces, how you visualize the traces and where do you store the Tracer?

A

So the open, Telemetry project is all about collecting the data instrumenting the data and collecting the data, whereas with eager you can use the the eager UI to visualize the data. So if you are moving from eager to open Telemetry, you should make sure how does your eager UI integrate to for querying the traces, and these are the hotel and eager boundaries. So, on the left side you can see, uh its hotel is all about the API SDK and The Collector. Whereas uh eager is the query and matured native storage backends.

A

So in future the eager will be evolving into a platform for uh traces, whereas open Telemetry will be more like a instrumentation and a collection pipeline for all the observability data.

A

Yep and, in conclusion, all I want to say is start with hotel. In some capacity it could be the instrumentation layer using the shim or it could be a complete reinstrumentation or you can just introduce the hotel, collector and start using the spanometric processor or the other features from the open, Telemetry collector.

A

That's all I have. Thank you. Thanks for joining my talk.

A

Before I end, my talk I just want to just show you. The eager UI has recently launched a monitoring tab which can give you, uh which can visualize use this red metrics coming from your spans. So now you can with open Telemetry collector, you can use the span metric processor, it generates the metrics and you can send it to a Prometheus backend and the eager UI directly queries the data from your Prometheus and visualizes this. So this is not built. It's all out of the box built by eager UI.

A

All you have to do is run an open, Telemetry, collector and configure this parametric processor. So it's really cool I just wanted to show it if you're using eager. You can leverage this feature yep and let's get back to questions.

A

I'll I'll be around yeah. If you have questions uh just you can join me.