Cloud Native Computing Foundation OTel Community Day NA 2022, 29 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Debugging OpenTelemetry - Ted Young, Lightstep

Description

SAVE THE DATES!
ArgoCon – Register Now | September 19-21, 2022 | Mountain View, California - https://events.linuxfoundation.org/argocon/

KubeCon + CloudNativeCon North America | October 24-28 | Detroit, MI - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

KubeCon + CloudNativeCon Europe 2023 | April 17-21 | Amsterdam, The Netherlands - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. Go here to learn more: https://opentelemetry.io/

Lightning Talk: Debugging OpenTelemetry - Ted Young, Lightstep

How do you debug your debugger? How do you observe observability? Installing OpenTelemetry can be difficult – distributed tracing in particular is prone to installation trouble. In this talk, we'll briefly hit on the top issues users can face when installing OTel, and what tools we could provide to make this debugging process easier.

A

All right welcome uh to my short rant on debugging, open telemetry by me. I go by tedsuo on the internet, uh so, what's the deal new users get started by installing open telemetry in their app right? This is where everyone starts and we're always looking for ways to make it easier for people to install the open, telemetry clients and all of that stuff.

A

You know like improving the docs, various kinds of automation, yada yada and when all of this stuff works, when the installation works properly, it's great it's super smooth, but what happens when the installation doesn't work sadness? Sadness is what occurs suddenly brand new users who don't know much about this are thrust into a position where they have to debug a tool. They literally know nothing about.

A

So once metrics and logs are stable. I really suggest that debugging is an area that we put some of our focus on uh so in this lightning talk. I just want to briefly cover all the ways new users may have to debug their open, telemetry installation, starting with the exporter connecting to anything.

A

This is usually the first place, uh people look, uh it might be misconfigured. You can find connection errors, hotel log level equals debug. Is your friend, that's pretty simple.

A

Next up is buffering confusing the heck out of everybody right because the default configuration is meant for production, which means your data is going to buffer for some time before it gets flushed, and that means you might not immediately see data if you're just running like your app in like developer mode and you're, just kind of like clicking on things a couple of times and you're like is it working?

A

I can't see what's going on it's just because the data's been buffered, so this is something that confuses new users and last and definitely not least, is the data correct?

A

Is the data coming out of your application actually correct, and this is the part I actually want to focus on for the rest of the talk, because debugging these data issues is terrible, instrumentation might be missing, context, propagation might be broken uh and the thing is with when you have a connection failure.

A

You have an error right, so there's a breadcrumb for people to follow and a straightforward solution to fix it, but when instrumentation is missing or context is broken, you have nothing you just nothing, no errors, maybe some weird data, maybe not, and this is terrible right. This is really terrible, because new users have no idea what correct data is even supposed to look like you know if you like turn it on and get it connected, and then you see like one span you're like is that right?

A

Is this? What I'm supposed to get you don't know if you're, a new user, you don't know anything, especially uh if you're new to tracing you, don't really know what to expect.

A

So how can we make this situation better for people and my suggestion is: we have to find some way to set expectations right, because, when users test their instrumentation by running their app and clicking on things, open, telemetry doesn't know what they're trying to do. It doesn't know what kind of data it should be expecting.

A

So if there is a way for that user to describe the transaction that they're triggering, then we could create some expectations which would allow us to test the data actually coming into a collector.

A

So what would this kind of testing look like? I don't know, but I think we should research this and have a look at creating some kind of testing language which is simple enough and free enough from open telemetry technical jargon that inexperienced new users could actually successfully leverage it to debug their installation experience.

A

So just to give a quick off-the-cuff example. If you were writing this, you know like a test. You might write something like this. You should expect a transaction that transactions should be coming from this particular service and it should contain these following components, and I can write this down because I know my app. I know what my app is called and I know the pieces that should be involved in the transaction I'm trying to trigger.

A

If you were writing this in like yaml, let's say so that you could use it to configure a collector component. Maybe it looks like this.

A

I don't really know something like that, something like that would be really helpful for new users and if we had something like that, we might even be able to go another step and create say like a web ui for building these expectations, something that would walk new users through the experience of describing the transaction they're. Trying to trigger uh a potential source of this information might be uh their package manifest like in most languages. There's some file you can grab that will tell you which libraries are installed in that application.

A

You could then match those libraries with instrumentation that we have available and start creating some expectations with the user about what kind of data should be coming out of their app when they click on something.

A

And if we could do this, then new users, I think, would be able to get their heads wrapped around the kind of telemetry that there should be expecting out of their application. So this is just one idea. I think, there's a lot of different ways to attack this. I think there's a lot of different ways in general to combine testing and telemetry, and I think this is interesting. I think this is an interesting place uh for us to go once we've got kind of all the table. Stakes nailed down.

A

So if you also find this interesting uh hit me up on slack, thank you very much.