Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Native Instrumentation for Open Source Software with OpenTelemetry - Ted Young & Ludmila Molkova

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Native Instrumentation for Open Source Software with OpenTelemetry - Ted Young, Lightstep & Ludmila Molkova, Microsoft

If you write OSS software, you may have noticed that it can be surprisingly difficult to provide logs and metrics to the users who run your libraries. OpenTelemetry is designed to solve this problem. OpenTelemetry allows OSS projects – web frameworks, databases, message queues – to move beyond testing and begin to participate in runtime observability, while still giving their end users control over where and how the data is processed. In this session, we will cover: The benefits of OpenTelemetry instrumentation. Guidelines for natively instrumenting your OSS library. Tricky parts, gotchas, and how to minimize them. We will also briefly cover the current state of OpenTelemetry project: Which APIs are currently stable, and what long term support means. Methods for combining tracing and metrics to simplify instrumentation. How to get involved with the native instrumentation effort in OpenTelemetry.

A

So let me introduce myself, I'm ludmila, I work for microsoft. I work on azure, sdks and observability uh later on. I will tell you about our experience with instrumentation, with open telemetry, uh the ups and downs of it, um and share some practical information and yeah yeah great.

B

Hi and I'm ted uh one of the co-founders of the open telemetry project and uh I work at lightstep.

B

Maybe we should do that? Yes, alrighty so yeah, let's get started uh so normally, when I give talks about how open telemetry provides value. Usually uh it's some slides, like this talking about tracing metrics logs, how they're all braided together and how that uh brings a lot of value, but that's not what we're going to talk about today today. Instead, I want to focus on this other problem, which is that instrumenting open source software sucks, like let's say your library wants to log into error, sounds like it would be so easy.

B

But it's this surprisingly difficult journey, you don't know what the application owner picked, but the only option you have to pick generally is standard out and that's probably not right, especially for their metrics. So what do you do right?

B

The real problem is that your library needs to support many applications, and so, when it comes to picking a solution, uh you have to defer to the application itself. The application owner is the person who picks where the data went and there's no one right answer for that.

B

So, since all of the data needs to go to the same place, each individual library can't pick a solution for instrumentation and observability, because the instrumentation is usually tied to a particular data system. So you instrument with a particular library- and that tends to pick a particular format, sends the data to a particular place and every library can't pick a different solution there.

B

So life is terrible, uh but that's not the only problem. The other problem is that open source libraries have to compose well with other libraries, which means they have to be very picky about their dependencies right.

B

Libraries don't get to synchronize with each other, so the dependencies that one library takes on might conflict with the dependencies that another library takes on and those dependencies might have dependencies with which conflict with each other.

B

So since instrumentation is always going to be out of date, somewhere uh version, sku can be a big poisonous effort here, plus any other dependencies that you might have, uh if you're hauling in say, like a big implementation. When you haul in that instrumentation, so once again, life is kind of terrible, but we want to solve this because instrumenting open source software is worth the effort.

B

A lot of the heavy lifting in most applications occurs within these open source. Libraries frameworks, database, clients, http clients. All of this stuff is doing a lot of work and we want to know what's going on in those libraries.

B

So how can we fix this with open telemetry? uh The first problem dependency management? By taking a separation of concerns?

B

um We can look at it this way right, like the library authors, they only need to write the instrumentation and the application owners. They need to choose what to do with the data. So we can decompose observability into these two pieces uh which in open telemetry, we call the api and the sdk.

B

So library authors only touch the api application owners only touch the sdk. The api only contains the tools for instrumenting libraries. It's a very thin layer of interfaces.

B

The sdk, on the other hand, contains all the nuts and bolts for actually processing that telemetry and it's a very extensible framework that lets you send the data in any format to any place.

B

You then take the sdk and you bind it to the api at runtime, so you actually aren't even forced to choose the sdk we provide. You could build your own sdk if you needed to so. Why does this help?

B

Well, it helps because separating these concerns means that the api no longer has any dependencies, because it's just this thin layer of interfaces, so it can't create a dependency conflict because it has no dependencies.

B

There we go uh the sdk. On the other hand, uh it's only loaded once and it's loaded by the application owner, and so the application owner now is managing the dependencies and you're not going to see version conflict happening within that sdk, because it's only getting loaded once so. At the end of the day, there's always a way to avoid dependency conflict here, because all the dependencies are coming in in one place and application owner can manage that, but that doesn't solve the other problem, which is backwards compatibility.

B

We have to ensure that instrumentation is always backwards compatible with prior instrumentation, because again, everyone is not going to go through and upgrade all their instrumentation all the time. At the same time, we want to keep adding more functionality to open telemetry, so we do this. By separating the api into stable packages and experimental packages, stable apis can only make additive changes that are fully backwards compatible.

B

Anything else we do has to go into a new separate package that is initially labeled as experimental and open source libraries, as long as they only depend upon the stable apis.

B

Well, they're never going to have a situation where they're, having version conflict between the versions of the apis that they loaded up so once again we're seeing separation of concerns, making it so that the api never breaks.

B

Anybody, because we're never going to break the api- and I should say just in general breaking apis- should never happen with any kind of widely shared library, because at the end of the day like what takes longer like carefully designing this api and ensuring that we don't create backwards, compatibility issues or fixing like millions of call sites, because we happen to break something.

B

So the point is actually none of this was specific to observability any library. Any widely shared open source library should be thinking about these kind of design concerns and just at open telemetry.

B

We just care about stability and long-term support quite a bit, so you might be saying like this is great, but you said stability is important, so when will open telemetry be stable? Well, today, the tracing portion is totally stable. uh Metrics, we're hoping to stabilize by end of year and logs will hopefully be stable in early 2022 for each language. You'll want to look at that particular implementation, but it lists which packages are stable and which are experimental also coming in the future.

B

There's going to be more kinds of signals, rum, ebpf profiling, so we're going to keep adding stuff and you'll be able to take advantage of that once it becomes stable. But you might be saying great. I've got my library today. I want to do instrumentation. So what should I be doing and to answer that question? We have ludmilla here with some real world feedback.

A

Yeah, thank you dad. So let me share the story that we've done in azure sdks. We instrumented our client libraries with open telemetry. We've done it in java in dot net in python. In javascript, uh you find you can find bits and pieces in go, uh we've done it for almost every generation, two of our sdks, that we have storage, keyboard, cosmos db in javon. Our messaging is the case event hub service bus event, grid control, plane, sdks and much more.

A

uh So when you work on something like this, um what would you think about? The first thing is like what you users want to know. What do you trace- and here I'm mostly talking about tracing- maybe a bit of logs and events on on spends, but I don't cover a metrics yet um so, let's talk about tracing, so what your users would want to know about the library um and like the first place to start is some complex api.

A

Calls that your library has the public api, which internally does some communication with underlying services and users would want to know about the outcome and duration and also some domain infor so like. If your library is a database, you probably would want to include the database name uh table name or collection, maybe a statement and other things.

A

So this is a good place to start uh and one of the benefits of having this public api spans is like users, don't know when they happen, it's magical uh and connecting them and understanding where, in this, in their code that they wrote this pen is collected, what it describes is beneficial, so the other layer of instrumentation is outgoing network. Calls in libraries. You, usually you don't want to do this, because it's supported by auto instrumentation.

A

uh The one thing you should be cautious about is that your your public api call, uh it should be correlated. It should be apparent. Maybe to this underlying uh network call, so users can see and like having those two basic layers uh helps users understand uh like the flow uh without like asking. Let's say for your support: they can see. Okay, this is how my library communicated with the service, and maybe they can see.

A

Okay, there were a five free, tries or long call to the service, so they come less to you as a library owner and they could be able to figure out more issues on their own or when they create an issue. They will give you more context, so this helps users with observability and you as well and at the same time you create the solution, mostly for users, even though you benefit from it. It's not the library, internal observability solution, okay, so the important part is semantic convention.

A

So uh first uh think you do you you can check, is your is open. Telemetry has a solution for you which spans to collect which structure they should have the relationships between them, which attributes to put on spence, um and we have a conventions for http, jrpc database messaging, some entry pieces, kubernetes fast and more so the semantic convention. They are absolutely critical for user experience, so users can let's say, query their traces for http status code and for all http clients, hopefully in the world, in all languages. It's the same key used everywhere.

A

All of the nice visualizations you might have seen are built on top of semantic conventions. Topology maps all aggregations, those beautiful gantt charts. They are so beautiful or cascade views of traces that you maybe saw they are all depend on libraries following the semantic conventions and going forward, they can bring even more value, so the core piece of distributed tracing is context propagation, so laborers participate in it by propagating context from the wire within the application and to the wire again on the way to the next service.

A

Let's talk about what does it mean? So if your library is a web framework or maybe a messaging consumer, then you should expect context to come from the wire. You either get an http request or maybe a bunch of messages, and you create a child's pen.

A

Well, if you've got a bunch of messages with different context, you kind of can link this context to the newspan. You create the important part that you take this pen and you pass it along to the user code and you do it through some open, telemetry apis and it's an implicit context. Propagation.

A

So you don't invoke something in user code to give them the context it just flows. It depends on the language like in go. You have an explicit context in other languages. This implicit thing works in some languages better or worse.

A

But this is not your concern. You don't invent a new way to implicitly pass context. You use in telemetry api and it does it best effort to propagate this context within the process to everything else.

A

If you can use some additional explicit way, it's nice to have this extra thing for your advanced users, who maybe do some background processing, creates threats or do something that generally doesn't work well with implicit context. Propagation, so implicit is supercritical.

A

Explicit is nice to have, and it depends on the language of course, and then the client libraries they take, this ambient context from open, telemetry or usually they don't even need to think about it. It just happens underneath the open, telemetry apis, they create a child spans and they pass this context along.

A

So, for example, if your library writes some logs for your own observability or maybe for users, then if you make this com context, we call it current or implicit, uh then your logs can be, can get the context stamped on them. So you can use the same thing for your own observability and improving your logs and then the network clients that happens underneath you don't necessarily instrument them, but they create child spans from your library spends because you made this context implicit and they pass context over the buyer uh to the next service.

A

uh There could be other layers of instrumentation. Somebody may want to trace some dns calls or tls so making even this low level context, current or implicit also makes total sense, and always if you can like this implicit part, is super important. If you can have an explicit pass and get context from users go for it, it's critical for some of your advanced users.

A

Okay, so let's talk dependency ted gave you a great introduction in why it's not uh such a big deal, but I know you're concerned I'm concerned too, so uh you take dependency on open, telemetry api. It's tiny package: it is stable, it follows semver, it has long-term support guarantees, so open telemetry takes api stability seriously, uh but semantic conventions are experimental. It's not the api. We expose it's like a soft contract on telemetry. This attributes names which attributes to put they may and likely will change in some cases.

A

Maybe even the span structure of the relationships may change so be mindful of that. At the same time, it doesn't break your runtime. It breaks some user expectations on the back end.

A

Okay, so to mitigate some of these concerns, you can make your instrumentation opt-in at least initially so expect that it will take a few iterations before you like figure it out. You may be, you want to get some user feedback. Maybe you over instrument it then they don't need it. It's too costly. Maybe certain operation is too noisy or maybe you are under-instrumented uh and then just give it some time to mature, get some feedback and eventually like we want uh instrumentation to just light up.

A

So you have your library, you have open telemetry, sdk and user application, and it's just all users need with opt-in motor, maybe plug-in mode that we usually have it's easier to make this dependency issues hidden, keep them hidden, but you um and you can start with that, but eventually you want to have it all in your library. So you don't need to document it.

A

You don't need to make this plugins, discoverable uh and usually plugins break just because people forget to install them more because, like maintainers, don't even test with these plugins um yeah, but this is the necessarily evil for the few of your first iterations okay, then you probably think about performance as well, so um the open telemetry api um is knob without configured sdk does nothing it's a tiny package. It does not affect performance uh when open telemetry sdk is configured.

A

Usually there is some sampling and it reduces performance impact significantly.

A

So otherwise, open telemetry is non-blocking. It consumes limited resources, but some overhead is expected. We work an open, telemetry heart to minimize this impact, but you should expect some impact and so for users uh who can like reason with it because of the value they get, but by all means benchmark your library and if you find anything, I'm sure all open telemetry languages as the case will be happy to help with optimizations.

A

One thing you can do on your site is don't trace for both operations. uh If something is verbose, use logs, if something for your own purpose use logs users can tweak the log level to get more or less of it. If you want to share something with users, but it's not a span, you don't have to create spam at an event on this pen.

A

Okay and related thing is be mindful of user costs. They pay for telemetry a lot. They pay for ingestion for storage, for processing, for everything if they host it themselves, if they uh use some um observability back-end regardless, they pay a lot.

A

If you are not sure, if that, then you think will um serve solve some problem for majority of your users, maybe don't instrument it or maybe put it behind feature flag or something before you learn more okay. Now you know everything that there is to know a quick recap.

A

uh Your core functionality should work, regardless of anything that happens in the tracing world. If it's not traced, at least your library works, uh then dependency. How is real, you can ship in plug-in or in some opt-in mode until it matures, uh then semantic conventions is absolutely critical for ux.

A

Please please, please follow them, and if you don't find one help us develop the context propagation, it's even more critical than semantic convention. Maybe I don't know how to say emphasize it more propagate your context, implicitly and explicitly, and when you test your instrumentation do test it. Try it out in some typical user application.

A

So pick a popular web framework enable all auto instrumentations the draw make sure your library telemetry correlates well with like incoming http requests underlying network layers, stuff like that yeah and come join us in instrumentation sake, help us polish. The last details about the instrumentation um help us develop new semantic conventions for popular technologies and just share your feedback ideas.

A

B