Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OpenTelemetry: The First R... Morgan McLean, Tristan Sloughter, Sergey Kanzhelev & Chris Kleinknecht

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

OpenTelemetry: The First Release, What’s Next, and How to Get Involved - Morgan McLean, Google; Tristan Sloughter, Postmates; Sergey Kanzhelev, Microsoft; & Chris Kleinknecht, Google

Earlier this year, the OpenCensus and OpenTracing communities merged to form OpenTelemetry, the first version of which will be released at Kubecon. OpenTelemetry provides libraries and agents that capture metrics and distributed traces from your applications and send them to backends like Prometheus, Zipkin, and Jaeger. The project is backed by a large community of end-user developers and the majority of cloud and APM vendors. We’re always interested in welcoming more people to the project! In this session we will cover: - What’s included in the v1 release, the project’s overall status and production readiness - Community structure, including governance, SIGs, and how to get involved - Recent integrations with various frameworks, clients, and Kubernetes itself! - Related projects like W3C TraceContext - What we’re working on next, including more languages, more integrations, and logs

https://sched.co/Uake

A

So you are here for open telemetry, maintain your track session. I will talk about open telemetry and how to get involved into it. What it consists of I'm, Sergei, Concetta I'm from Microsoft.

B

I'm Tristan slaughter I'm from post mates, hi.

C

I'm Chris Klein active.

D

Google+ from Google, so I assume that most people here, because this is the maintainer strack talk for opens limits. We already have a least cursory understanding. What the project is for those who don't open selamat tree provides the components that you need to extract critical information about your applications performance and about any outages or other problems you have specifically. That means that it you can extract, distributed traces and application metrics. We provide all the components you need to do that.

D

We have an API that defines these, so you can use the API to define a trace to find a span capture a metric or capture metadata. Next, we've integrations, as I mentioned in our talk yesterday. These are sometimes overlooked, but they're in some sense, the most important part of open telemetry and the projects have led to it open census and open tracing. These integrations are the reason that when you link open telemetry to your application and Stan, she ate it, you don't need to go and use the API and manually instrument your whole app.

D

That would be very painful. These integrations exist so that you don't have to do that and they exist for things like HTTP, client, libraries, RPC, client, libraries, storage, clients, etc. All the components that you really want to capture data from next we offer libraries. These are the actual actual implementations exporters, which is the way that you send to capture telemetry to your backends of choice and we support many of them. And finally, a collector chris is going to get it. Do a deep dive on each of these coming up.

B

Thank You Morgan, so, in addition to being a collaboration between open tracing and open census into a new single project, there's many contributors across the industry from APM vendors like data dog Google, was there dynaTrace, as well as end users like post mates, Zillow pager duty, and we also have a broad range of language library support from your usual Java Python Ruby go all the way to Erlang elixir and Russ coming.

B

The project was only announced in May, but since then, we've got close to 200, individual contributors taking part and it's growing every day. Hopefully some people in this room will add to that number.

A

Open telemetry defines three times of participation, it's kind of levels and the philosophy of open, telemetry, participation and membership is that your contribution defines your participation.

A

So it's there are rules to became an official members of open telemetry, it's very easy, but there are rules. You need to demonstrate code reviews, you need to do issue triage to become an approver and then the became a maintainer in the demonstrate. Some continuous commitment as a project and understand, understand the values of it and make sure that you go into the same directions and as everybody else.

A

A

So when so many people going together and building something, we need to make sure that we all go into the same direction and, as Tristan said, we already have so many members that we needed to think about garden structure from early on when we think of governance structure, we put members top and front so maintain years of individual components has a lot of autonomy, what they doing, how they doing it and when they doing it, members of open telemetry, elect governance that upholds and protect the vision of a project.

A

So we all doing the same thing, but then governance enforces regions to the technical committee which you all can participate in. If you demonstrate your knowledge and understanding of the project, technical committee defines specification that guides all the members, so the perfect trifecta of powers, but again members are front and center. As Morgan said, open telemetry is a complete solution for telemetry collection needs, as and as any complete solution. It has so many components and modes of operations.

A

Sometimes it may be hard for you to understand where to get started where to get involved and in this presentation, crease will walk us through open telemetry, how it works and which components are involved to make it happen, and then he will simplify it for you by saying what components in to pay attention to, depending on what your role you are with that I'm question to Chris.

B

C

So I'm going to talk through the components of the libraries and then hopefully by the end of the talk it'll, be clear: why we split them up this way and then, depending on what you're working on what part of the library you should be working on, that is, if you're working on it at all, so like a quick show of hands who is actually using Ben senses or has used open senses, nice there's a lot of people all right: how about open tracing open tracing users right more more users than open census?

C

So hopefully, if you've used either of the two, this will look pretty familiar.

C

C

C

Yeah Ted points out. We could use open telemetry to debug the latency of the clicker, so this should hopefully look pretty familiar. We're looking at a trace here and the spam in the middle has some attributes, and this is being displayed in iaeger, which is another seems you have project. So the actual thing that the library needs to do is produce these spans right.

C

So hopefully this will look pretty familiar too. Basically, we've got here is a set of services. Your upstream services, your downstream services, they're color, coded in the image and traced by itself describes the flow of a single request through a collection of services, and then the individual span is the thing that we care about producing from our library in the case of tracing.

C

But the spam is only useful in context, which is where the other libraries here, or rather the other services, come from, so we're basically trying to do three things right: we're trying to instrument we're trying to propagate the context between the services and then we're trying to send the spans that we create off to specific backends, and so this is kind of these. Are three different flavors of the same type of work which is instrumentation so looking at the concerns this way right? We have instrumentation context, propagation and exporting instrumenting code.

C

It actually looks like putting the open census or open tracing or now open, telemetry library in your library, if you're, a library owner or your application, if you're an application owner and using this to create spans for like some useful work- and here you can see, the useful work is called, give cube con talk and we're trying to instrument give cube con talk, so we're gonna create a little span to wrap this. All of these examples are gonna, be Python for two reasons.

C

First, python is hopefully close enough to pseudocode that everybody can read it and see what it's doing. Second, because I'm the maintainer for the open, telemetry Python client. So if you were in Python, you should come talk to me and we should work together. So the second thing we've to do is propagate context. This is handled by some libraries that are not open. Telemetry like if you know, GRP, see context in Java or, if you're familiar with go. They have this, like capital, see context that is usually in process context propagation.

C

What we're trying to do is between process context propagation. So we've got two things. We take the context from the wire put it in the application and we do the same thing in Reverse on the way back out, and you can see here we're doing this by way of a couple integrations right, so we're saying use B 3 as the the format for the trace metadata.

C

This is, if you've zipped in there using B 3 format, and then we've got a flask integration flask as a web framework for a Python that is dealing with the incoming requests and then we're using a request. Integration requests as a library again in Python for making HTTP requests and attaching request data requests metadata on the way out.

C

Last thing we got to do is ship it off to a back-end. Jäger is an example, the backend from the first slide, and the way this works is basically just in config. So this is not on the hop path of the application. This is something you would do separately, you're gonna in step. One instrument, your application and now in step, three you're gonna, configure it and so we'll see in a minute. These can actually be different people doing the instrumentation and configuration okay.

C

So now we're to talk about how the components are split up in the library and we're going to talk through the five components that Morgan mentioned at the beginning. I know this is kind of a slide full, but hopefully you can see. The important thing here is that the open Salama tree client libraries are split into separate API and SDK packages and that those have different audiences and different concerns. So who here is a application owner? Like you own, a application, you instrument it great a lot of hands how about a standalone library.

C

So this is not something that you maintain great awesome as a service, just something that other people will use last thing who works for an APM like the the data dogs in the world. The dynaTrace is dynaTrace in the front row over here, so application owners back to this diagram. You can see the thing that you care about, hopefully, is your application business logic. You want to instrumented you're gonna create spins in the business logic there you're only gonna depend on the API alright. So the API is this two things.

C

It is both a set of interfaces that we instrument in it like implement using the SDK, and it is a minimal implementation that those things like context propagation for you. So it's confusing that it's called API, but it includes some implementation and the SDK also includes some interfaces, but you bear with me: we call them API and SDK. So that's your application logic and then at configuration time, you're gonna want to pick a particular version of the implementation of the API and load.

C

That version and we'll see why you'll do that in a minute when we get to APM vendors. So you might do both of these things, if you're both using third-party libraries that have some other implementation, or rather that have some instrumentation that you don't control and you've got your own business logic that has instrumentation. You do control and you want to pick a particular implementation to use at configuration time now, if you're, a library owner you're, also going to depend on the API, but that's it a library owner will never depend on the SDK.

C

Now, if you're, an APM things get more complicated. Usually what you'll want to do is ship your own exporter, so this will take open, telemetry data it'll, convert it to the shape that you need for your back-end and it'll. Do the actual work of shipping it off to the backend. You can also, if you want to completely rewrite the SDK now. This is if you need to make like some invasive changes.

C

If you want to have actions that happen as spans start and end, or you might do both of these things and basically just use the minimal implementation of context, propagation and the set of interfaces in the API.

C

So those are the three people, the three personas that might use the library and why they might use them. These are the different components. Do have any questions about the personas real quick before we jump onto components? Okay, okay, so we talked about client. Libraries talked about the spec things we haven't really talked about. Our integrations and a lot of the work in open telemetry actually happens in integrations, which is making this work with other libraries exporters we talked about.

C

This is vendors that want to ship an exporter or rather want to ship instrumentation code from an application out to the backend, and then we talked about the collector, which is just a second, a big, separate thing from the library, so client, libraries. This is the thing doing the actual work you see.

C

We have tracing metrics and then logs, hopefully, if you're interested in logs come talk to us too, because we have work to do in the future on logs, and this will also do stuff like resource detection, where, if you have certain environment variables that will find them and put them as attributes on your traces.

C

The specification describes how the client libraries work. So, if you are writing your own client library in some new language, you're going to follow the spec and then a lot of the work in the client libraries, it's pushed up to the spec. It's not clear how something should be implemented or watch design.

C

This way, we have a conversation in the specs repo exporters, I've showed in the example, it's possible to configure multiple exporters if you're exporting to the collector, which we'll talk about in a second, you can actually swap out your exporters without restarting your application and then integrations handle context propagation they handle integrations with other libraries and services.

C

So one big use case here is that, if you're, using a library that say integrates with a database, you might want that library either to be instrumented using open telemetry or for there to be some integration that wraps that library, so that, when you make a call to the database, we generate a span right. We maybe do things like include the query plan as an attribute of that spin.

C

Finally, we have the collector or agent we've used, other APM's you're, probably used to seeing an agent or a sidecar that runs with your application. That does some work, choosing which spans to sample or choosing how to aggregate metrics. So we do a lot of this in a separate component called the collector and it's possible to run the collector both as an agent that runs along with your application and as a whole, separate standalone service that does the aggregation and another layer of export out to different backends.

C

B

Thank you Chris. So after seeing this talk and some other talks at this conference about open, telemetry, you're, probably thinking how do I get involved, I got to get involved now and there's a number of ways that you can get started getting involved and there's a number of personas that Chris talked about of where you would be coming from myself. Working to post mates and chairing the observability working group at the in the Erlang ecosystem foundation, come from a user and app developer. Who uses these libraries way of thinking.

B

So we have a unique perspective compared to the people. You know working for APN vendors and it would be great to get more end-users involved as early as possible working with this and trying it out and developing within the different components.

B

So some of you might already be involved which, if so thank you and keep it up, there's more ways than just code that you can be contributing. You can start telling your friends and co-workers write blog posts and, if you've only been contributing and then encode or use and filing bugs. You should also check out the special interest groups and look at attending some of the zoom meetings and getting more deeply involved with the project.

B

If you're thinking about starting to use open telemetry, you should head over to the website, which has some Doc's for a quick start and in a number of languages like Ruby and go. If you don't find a QuickStart for your language, you should check out the the github repos. A number of them have some readme information on how to get started or dig through tests open issues.

B

If you don't find anything and people will help, fill in and get on our git er channels and start asking questions if you can't find a way to get started if you are using open, tracing or open census.

B

Like many of you are it's a good time to start looking at the future of those projects which is open, telemetry, seeing how, if it's your needs or doesn't and bring those concerns up with the with the community and definitely check out the open, telemetry and open census shims, which will provide a way to keep your opens limit or your open census code intact and still talk the open, telemetry data format.

B

So if you're looking to start contributing today, first, you can check out our get involved page on the website and then find the repo that you're interested in taking part in for kit and start hacking away.

B

You can look for good issues to get started with by searching in the repos issues by the tag, good first issues or if you simply find something interesting or missing, dig in talking the getter channel and get some feedback from people who are working on it as well and started hacking away if your org wants to give support to open telemetry talk to us after this presentation come up to us and we'll talk about how that can work or join us in the Gator community channel, say hi, and someone will get back to you eventually very quickly with how you can do that.

A

Thank You system, so a stressin mentioned there are other initiatives that we all participate in like open. Telemetry is evolving project and we try to keep up with modern trends on how to monitor applications, how to observe applications. Some of the strengths are stable enough. Some of the concepts stable enough for us to put them in the standardization bodies and w3c trace context is one of the initiatives that many open telemetry participants also contribute to this double three treys context. Just today, it hit another milestone of maturity.

A

It's almost final specification, so one more months in until the final, so it happened today, which is great.

A

Okay, I'm here I need to describe what that means. So w3c can trace context. Is the format protocol how you propagate context for distributed tracing from one component component to another, and this specification describes semantics of this view of the context and the format it means to be serialized and digitalized too, and also talks about interoperability between multiple vendors.

A

That may oh, like my monitor one component, another component, this very powerful thing and we found it's stable enough to put it in a standard body and make sure that everybody liked not only people doing observability but also people opening platform service meshes and proxies, can also participate and help us visit.

A

E

A

You Morgan I.

D

Just wanted to clarify there are different phases of things becoming a w3c standard. By almost final, we mean there haven't been any changes to the spec in six months or a year. This is just a bit going from being a like editors, draft to being the actual like final approved, w3c spec. It is something you can depend on today. There's nothing we're just like closing some sort of loops on it.

A

Thank you and finally, I want to talk a little bit about non code participation. We want you to get involved and to get started with that, but sometimes you you don't want to cause. They just want to talk to us, and this is also consider participation. We absolutely looking for feedback so contribute to the picture here.

A

We have no way to say whether you using this application open telemetry at all or not whether you what environment you use in open sea limited for what features you like and what features you don't ever use, and this kind of feedback is also participation in a project. Give us come to us and tell us what you like and what you don't like and we'll help you so before we wrap up and switch the Q&A session.

A

We have an extended Q&A session today with many governance members I want to just describe what you just saw like we, we walk through structure of the project, how we operate, how we define the vision of for the project. We also walk through architectural project and what components we ship in first release of open telemetry. It's a lot of complexity, but we try to simplify this complexity for you and it's a lot of components.

A

The first release we've been major one, and then we walk through with the ways how you can participate and watching me to do to come and work with us, and now we will switch the Q&A and we'll invite MoMA or more people on the stage on a podium and I will happy to answer any of your questions.

D

I know there one or two others so we're going to be joining a bit late, I think Constance is on her way and of course, this being the last day of the conference there's some people already had to disappear.

F

D

First, we should introduce the people who weren't on stage.

G

My name is Bogdan I work for Google I was one of the first person that did open census and then also part of the steering committee of the merging project called open, telemetry and now I'm, one of the Technical Committee, and also part of the governance.

F

My name is Ted.

F

The open tracing branch of the family I'm on the Governance Committee these days, mostly I, work on the specification and I try to facilitate the project. So, if you're interested in getting involved, you can reach out to me and I'm happy to plug you in and answer your questions. I'm.

H

Sarah, Novotny and I came from the completely neutral branch of merger in the sense that I was the fifth person on the original committee that brought the two projects together. So I was the the tiebreaker of open census are open, tracing was if there was factional, is Asian, but there really wasn't.

H

It was kind of Awesome anyway, so I've been helping with governance and setting up and bringing together the two projects and we'll continue on the steering or the Governance Committee over the next couple of years, as it transitions to a healthy, happy open source project. So.

D

Think now we can take any questions from the audience. We can happy to answer both technical questions, tax questions of a project direction. Anything you want complaints about the weather as.

I

Someone that has been using telemetry but more explicitly is there any recommendation to to start and build a stack from scratch. I know you were referring to getting started, but that's one thing getting a library, it's a different to say: okay I would just want to get a full stack there and and start experimenting. Is there an easy path?

I

D

We talked a bit about this yesterday, no Sergey or Chris. Do you wanna, expand on it.

A

So when you're talkin full stack, are you talking about infrastructure, piece of it or application piece of it ever think yeah? So, in the first release we will consider it on application, monitoring, increasing application creation and application metrics, and this will be like topic of a first release and then later we will beef up all our offerings and, like collector, will start them saying more infrastructure kind of metrics and more like, like we will kind of extend it into like a wider range of telemetry sources.

A

So open telemetry is about collecting data. We can gather a lot of telemetry sources and bring it all together. One thing is, you need to add back end to this mix, so you need to choose which back end to your vision. You can use either open-source backends like Jaeger or parameters, but you need to manage them this component yourself, but then you can bring some managed solutions like like telemetry vendor solutions, and then it will give you more kind of insights and more data. Now this on top.

H

Yes, so so functionally open, telemetry is the instrumentation portion of this, and this is the piece that goes. You know through libraries, for SDKs into your into your application, but then there is a visualization layer, which is the idea is that this is the common language that many visualization layers can look at. So it's more difficult than building like here's, the full stack and I just drop my app on to it. I.

F

H

F

Just add one extra thing there, which is for languages that support something like an auto installer like Java, has like a Java agent pattern, for example, will be providing auto installation. So if you have an application, you can just auto, install open telemetry and it will add library, support and instrumentation support for you.

B

F

B

Of course, service meshes are all the rage these days and so I know some of them are open, tracing, enabled and hopefully will become open. Telemetry enabled so you'll just be able to turn that on and you'll start getting data without actually touching your your application code.

J

Hi my question concerns the API design and development you've gone through. Have you run into cases where you have to present a least common denominator? Interface? Are there escape hatches if there's a particular vendor that allows you to express things that you can't capture what the API is stated.

F

Yeah, so it's definitely a the case that we need to wear I, see a sort of least common denominator, and the API is actually more around handling edge cases than it is about vendor features. So much traditionally, though, you're able to at least an open tracing, you were able to kind of typecast to a vendor specific thing. If you had it, we really really don't want people to be doing that, though, so we consider that to be a bad smell if you're feeling like you're.

K

F

To go around the API to get something done, but one thing that does feel good is: we've had a lot of like in production testing at this point for both open census and open tracing. So when it comes to designing this API we're not really doing it from scratch, so we feel pretty confident that it is state-of-the-art. As far as the state of the art today and.

A

I mean philosophy, open telemetry. We really believe that all extensibility points that you ever need. We will provide the small callback small extensibility points. We call small hooks, but there is always a big hook. You can replace entire think and like put a completely new SDK, and we keep this door open for larger innovations.

L

Hey, so during the talk today, you guys teased a little bit about logging and how that may find its way into open telemetry in the future. So is there any more information you can give us at this time, yeah.

D

Not a whole lot new now so for anyone who's in the open census. Community. Prior to the merger, we were working on a logging specification we'd like a product requirements document after logging that number of vendors and large users who commented on given the merger between open census and open tracing and open telemetry that took a backseat, because the getting the communities aligned was a much bigger goal.

D

Logging is once again something that certainly come up. Maintainer zuv kubernetes have been asking us about it and I think some of the vendors also been asking about it just given, but some of them are expanding into logging space as well. To be blunt, I, don't think we have any fully formed thoughts about it. It's something that early next year, we're just gonna, pick up again as a brainstorming sort of brainstorming event and go from there. I. The priority at the moment is finishing.

D

The metrics and tracing implementations for the libraries and getting those out debate in the GA I.

M

Yeah kind of actually on the status of the items you have on priority for the roadmap for those of us who are using an exporter to say Jaeger for our different racing and stuff, like that, what's the what's the status on bridges or library interfaces so that we can continue to, you know export to those but adopt the open, telemetry standard. Yet so.

F

It's a it's a hard requirement for this project to have full backwards compatibility with both open tracing and open census. The idea being you shouldn't have to make a gigantic all at once: change in order to switch from open, telemetry, sorry from open tracing to open telemetry, for example, the switch would be first installed, the open, telemetry SDK, and it also will because open tracing is just an API that tracers combined to the open, telemetry tracer just binds to the open tracing API.

F

So all of your open tracing code will work with the new SDK and then you can progressively transition your code over to using the open, telemetry API.

F

We in basically every major language, we have a version of a bridge because we want to make sure we don't get to the end of the API and discover that we've done something. That's you know painted us into a corner real.

N

Quick when you all, if someone asks a follow-up and they weren't liked on the follow up, make sure you repeat the.

A

Question? Yes, no back here.

N

Can hear things all right? We have another question so.

O

How do you existing prometheus, like custom, prometheus metrics that may already exist and fit into this kind of ecosystem.

O

I guess application-specific ones.

F

So, like you just already have Prometheus metrics in your correct.

E

G

That's that's a very good question, so I think, for the time being, there is a big difference between metrics and traces, at least the way how Prometheus did metrics before they are context agnostic. So they you can easily have inside the same application.

G

Metrics recorded with two different libraries, and they will still work, does not true for tracing. So one of the major difference between these so so far, we would recommend, if you wanna, do the transition between the Prometheus matrix to open telemetry matrix to 4/4 time being you, you will run both you will. You will have both libraries and until you finish your transition, we do have in our collector a way to scrape Prometheus metrics that will transform everything into open telemetry. So that will we do the transformation outside the process.

G

That's because the Prometheus client currently does not allow us to to have a hook inside the Prometheus to to export our format. We are pretty compatible with Prometheus and it's one of our goal to be compatible with them and we will support directly export into parameters from our library and we also gonna support open metrics, which is hopefully coming soon.

P

N

Ask I have a comment question since Liz isn't here one other thing: we're trying to do that. I, don't know if got on a slide was to find a lot of educational resources for people, so we've been creating and we weren't running these open telemetry workshops at a few events expect to see more of those next year.

N

But all of those resources that we're creating for those workshops are Apache 2 license or the same terms as everything else, and the intention is that you know you could go back into your organization or to your local meetup or your local technical group and then run open, telemetry workshops and trainings yourself. So if that's something that interests anyone, you know those this room in the community for YouTube.

Q

So I saw the demo yesterday, some you know hello world examples. Are you going to do more advanced examples like async tracing? Yes,.

G

The mutex so in open census, we had a nice example. It's called hipster shop. It was built for for Google next as an example app for for a bunch of features that Google wants to show I think having something very similar with that in open telemetry with more like that's an app with seven languages, eight or nine different micro services, with I. Think with everything that you can imagine I, think that will be very useful.

G

I, don't think we'll be able to cover all the possible examples in all the languages, but having some examples for a scene. For example, yeah we'll have two or three languages with that scene, not all the of them but having an application. That is like that that people can understand and can see, I think it's it's very valuable and we will invest into that after G. But.

A

It's important for us to understand your particular scenario. So if you have some question and you can figure out how to do that and you need a Gemma just tell us on the guitar or whatever, and it will give us a strong signal that this is kind of scenarios that we need to talk more about.

R

Hi, so this is maybe more a question for the open censuses. That's where the exporters are coming from more and any thoughts on. Where would you put the exports like? How much would you merge, metrics and traces going out back up to the data center label and then export from there or is it more put the exporter on sort of every single service base.

G

So I'm, not a hundred percent sure I understand the complete question just.

A

Five seconds and so open telemetry provides a complete set of components that enables you to send telemetry to any back-end. You choose and you can start like easiest integration is out of process collector because it's easy to see vision. You can do it at any moment of your application lifecycle and then, if you cannot run any external process, we have in process expertise. And if we under implemented this in process expert, then you can use that expert on send data directly to the backend.

A

But then we have all sorts of combinations like some people, running parameters, metrics and taking them right locally. So they can do like very fast, a scale out and scaling and all like longer-term metrics you'll be sent to the back end. So you can do all sort of this scenarios as well.

A

We also saw people doing three-tier architecture when they have sidecar collector first and then collectors that aggregates across entire cluster, and this scenario allows us to do very, a primitive sample inside car, but then, like way, more elaborate sampling decision like tale based sampling when you sample, based on a failure in a very last service in a cluster level collector and only census, data to the back end. That saves you a lot of money on like zillions, your back-end.

F

S

Sorry I was doing track host for something else. My name is Constance I'm from spunk we used to be omniscient super excited about the community. We all need your input and I. Don't know what else I'm supposed to say. You're.

F

Also, a member of the Governance Committee and.

S

I wanna govern three that were too high.

E

Yeah, so my question is: if I'm currently having a application, that's using Prometheus client for metrics. What's my incentive to move to open telemetry for metrics like.

G

Okay, I'll take this because I was probably the guilty person that pushed for having metrics inside open telemetry, one of the major difference and I. Don't know if you try to use open sensors, we do have our metrics our contest context-aware, so which means there are certain things that you can do by passing context around.

G

For example, if you are recording a metric in service a which is called by service B, you can inject labels from from the other service, which is something that parameters does not support, and this is something that we we had in Google for a long time and we thought that it's super useful and we have clear examples inside the Google where what this was super useful for us and yeah. So that's the probably the major difference between us and Prometheus. Anything else is almost the same.

G

So if you record your metric- and you just configure to not be context aware like to just not extract anything from the context, then you have exactly parameters, and probably the that's not gonna, be a big reason for you to change. But if you do want to have the cool feature of being able to propagate labels between inside the process or between processes, that's something that parameters does not have. Yes,.

O

G

Matrix library so far, that's why we added yet one and I hope is not an n plus one I hope will become the one yeah.

B

This was one of my favorite features of Open census when I started working on clients for open census and being able to have that context of you know: HTTP status code.

B

What method was called in one place shared that gets used by my metrics as dimensions as well as show up in my traces, my trace fans when I look at them. It's really very useful.

A

But again it's not a competition, we working with them and try to resolve like how we can get a single guidance for for end-users and like what developers and guide them with the transition.

F

This is about when you have like an observability system that has both tracing and metrics in it. You end up with this context. Propagation and that's the real difference is being context where so.

Q

Have any of you thought about doing like a mini con at one of your offices, where you invite people to do like a whiteboard session for maybe three week three days to maybe a week, and if so, when can we expect that I.

S

Guess, like vendor, had on I'm, sorry, yes or like, but also like as I'm saying as a vendor, but also like. We definitely talked about how important it's about the community I think we just have some other bucket list things right now.

S

We need to do intermittent just ministration stuff like getting to incubating status, but it's definitely something that we care about, because we know that in terms of like, if we don't address the end, users needs and we actually don't get the feedback in terms of what's useful and like hey, there's certain things are really difficult to use. We're not gonna be able to be the successful project we want to be so we want to do it.

S

D

They'll definitely be something we just have to actually spend the time to plan it, which requires us to not be aq.

H

Which actually opens up another thing? If any of you want to help in that, please just reach out cuz. It sounds like Morgan and Constance at least want to see this happen.

G

We we already thought about having something like that in summer, but because of the merger, we keep postponing it until we have kind of a better version for our libraries, because we will be a bit premature right now to talk about a bunch of the things yeah.

F

And new, relic and life step are talking about hosting regular meetups up in Portland, so Portland Oregon, peeps, say hi.

D

You work at New, Relic.

T

I kind of question about the context, propagation and the w3c process and kind of my question is like: do you see like less cloud native software getting involved here like like Postgres as a database server? For example, you know there is no way in the wire protocol to say: hey, here's a trace or contact propagation like so. How do do you think about these kind of like non cloud native things? I mean posters is cool, but like my sequel and there's a lot of databases out there that they kind of don't use HTTP.

T

How should we think about that? So.

A

I cannot give any commitment here, but we're talking with TGS guys it's a sequel, server people and we aging double three ship product column TTS. So we I mean we. We clearly see interest and like from no sequel and like some storage, guys, like I, mean I'm talking and behalf Microsoft, because, like I've worked there and I work uses people like storage. People also want to give you information about how your transaction went and one wire it was slow.

A

So the aging the trace context there as well, like we using some custom format right now, but we switched into double tree. She eventually and it's kind of the same kind of ballpark of people, yeah.

F

Now a dad there's just a little bit of color when it comes to doing context propagation. Some protocols have metadata in the protocol, so there's a place to put this kind of information and some other protocols like memcache, for example, don't have a metadata concept, so it's much harder for the protocols that don't have any concept of metadata. It becomes a lot trickier to decide where you're actually going to put this information, but the ones that do have metadata it's just much more straightforward, yeah.

B

Like Kafka has headers, and so that's a good place just to standardize on how to how those should go in there and get read and linked around and I. Think I had a conversation with people at rabbit at one point so that might be coming from.

U

Uber and member of the governing board and used to work a lot on the pond tracing so now, telemetry.

F

K

I've just put I'm open tracing in my application that was received very well and now I have discovered open, telemetry and that we should be switching over to that. So that's interesting. We get a lot support for open, fed, metrics and stuff as well, but now I've heard the term open metrics being thrown around. So how does that relate to open telemetry I.

G

Will take it for the moment open metrics as a project just defines a data model for the metrics. It does not define a client or is not a client they want to. So there will be multiple clients, open, metrics, compatible and will be one of them. So in that regard, I don't think there is any overlap between the two projects. We are client libraries that we are helping. You instrument your application or your service. They are a data model which we will support and we'll expose that.

F

Another way of saying that is, you could get your data, your metrics data exported in open metrics format or Prometheus format, or stats D or open telemetry. We want to be agnostic on that front and.

H

I'll have one additional thing to that, which is there isn't have written shims to do the backward compatibility token telemetry from open telemetry to open tracing. So it's not a big rewrite for you immediately.

S

There you go, um there's also the collector, where the collector actually supports, like Zipkin Jaeger, all these other things. So if you just want to do forms like first doing data collection and using your existing instrumentation, that is like the first port, like first step forward in terms of adopting open telemetry.

S

So you have one collector: that's ports, metrics and tracing it'll, eventually support logs, and so that could be your first journey towards helping us grow the community and then after update your instrumentation, to whatever say, if you want to use the open trade, open, telemetry, metrics or a different protocols or anything else,.

Q

So are there any plans on adding specific information to spans at either individual span level or an overall level so that we can potentially lock down? Who can be which spans for security purposes?.

G

It's are you talking about accessing the spam data, or are you talking? Okay? So first, if I understand correctly the question you are talking about annotating who is accessing or who is viewing some spam information. Some spend data correct.

G

That's probably something at the backend level, where you should have support for some kind of ACLs or or some kind of logging, like security logging for for access access, logs or whatever is called I, think I. Think from our perspective we will produce the data and we we can probably make sure that we you. We do not send the data in two places and we just send to the trusted back-end that you configure, but that's probably everything that we can do from the client perspective. So.

A

I can tell, from my perspective, that we have a lot of experience in sending telemetry from windows and all all regulators from entire world. Looking at the city, meter, Trenton's and like trying to catch us by the hand and I've seen you doing something wrong, so we have a lot of experience, attributes and data. Now the question is: do you want to take this experience and move it to open telemetry project? Is it separate one?

A

So maybe we can just be better off by doing it in back-end and provide a custom solutions that you can install on top of open sea Dimitri, but if it will be a large demand, this is one of the area we as Microsoft really interested in. So we and we can bring this expertise to the project actually.

S

Their support within the open till I met. You collector that, because it's processing, all the spans right, actually there's a redaction filters too so like in terms of like PII. We, like we've partnered with some people who are using open telemetry that they react in terms of like they have secret key in their trace or user ID. There's also like one of the things we have an issue is to actually hash the actual values, and so that way, then you can like hash it.

S

So no one like we can't read it, but then, after once you have control people who actually have key. Then you can decode it yours and so that's part of the collector. So the collector has a lot of powerful things. You can at least to transform the data in terms of what you want to do. I.

C

Know we also talked about putting reduction in the SDK itself so like before it ever gets to the exporter already. The attributes that you want to remove have been removed from the span, and this is like because even if you control the exporter, you might still not want anything that you know. You won't redact it to hit the exporter layer in case you're, gonna swap the exporter out and then you know, lose access, control.

N

Don't see their questions I think there's more time. Did people just want to go to the line and talk about like one thing that they would either like to see moving forward in terms of where's people can specifically involved or just what they're most excited about you know tell since we have everyone here, I.

U

Could start one thing that I really like about open telemetry is the fact that we pull the context propagation framework below their durability layers. So observability is just a function that uses context propagation rather than providing it, and that opens up a lot of other functionality that can be built, and if you interested in that there was a talk at observability cymet this Monday about various things you can do with context propagation they can S&T was a durability. They can like debugging and production, testing and production.

U

Other type of things, and so open telemetry will contain that that layer for now in the future we may move without, but but that's like one feature that I'm really looking for it. An uber is big user of that feature already, even though it's an open tracing.

S

Okay, I'm gonna say to you just because we're gonna we're gonna try to have up the crew one thing: actually it's really exciting, but open, telemetry and I think that it like the problem that suffers a lot in terms of projects now is that we have one metric, like you know, there's a metrics, there's a log, there's a tracing there's all these different conventions and projects out there, but like open telemetry is kind of like slowly.

S

Turning to the superpower, where we get to actually combine all three right- and you can say you can combine even more and so like that's actually where, like observability, is going to it's like, not really doing any form of like one-dimensional monitoring, observability. It's how you combine all three of that, and so this project is that first step towards combining you know when you add metrics to your traces and after have nice integration with your logs like that is just really cool.

S

The second thing is, I would say: Auto instrumentation in terms of like you know, we're probably a lot of people here at KU con already working on the cloud and have a lot more of like the faster iteration turns out.

S

Adding a lot of frameworks and libraries on the fly and auto instrumentation is a really good spot to actually help people by past the initial painful part of instrumentation, because, as we all probably know, one getting by and in terms of instrumentation takes a really long time as an organization and also, if you don't have the experience in terms of like knowing what to observe auto cementation, can lead to fill that initial gap. In terms of like seeing how things are, and then it doesn't find.

S

Oh I know maybe I need to instrument this one piece of code, because that one's actually really slow or I need to do a little more things there, because there's so many issues, and so auto instrumentation and open telemetry has a superhero. Okay,.

C

I know this is Justin's beat, but I want to put in a plug for working on the language clients. Hopefully we have a couple different like fun. Communities that are easy to start working in I can speak for Python if you're interested in working on the client or especially, if you have another library that you work on, that, you would like to add an integration for that's thing that we're always looking for, and we need a lot of different people working on it, because we need domain expertise and the different languages.

C

So if you have it, please come talk to us or find us online.

B

Yeah, what I'm most excited about and why I got involved in open census and why the Erlang ecosystem foundation is a part of this is to bring the barrier of using our length or elixir in a company down a little bit, because often you go in there and you become. If you were able to already have an Erlang system there in the company, it's often its own corner. It doesn't talk to anything else. It's not able to use the same APM vendors you're.

B

So no, they eventually just want to get rid of it, because it's a pain to deal with so having these standards that vendor start supporting, allows us to be just another citizen with and the Cooper net kubernetes cluster and talking to the same vendors to visualize and get your logs metrics and traces, and we don't have to be the oddball out.

A

So I'm, mostly joking out of work like people asking what you're doing and I'm saying I'm actively eliminating my job, so I really hope that in some time we'll be talking, coop consent how to correctly Amity- and nobody shows up because silly me three alternates they're like. Why would you talk about how to collect a limited, its built-in everywhere and I hope it will happen soon, and this is what I'm working towards I'll.

D

Echo Chris's sentiments I'm excited for the language, client, libraries to reach beta and then GA early next year and then to see people starting to write. Integrations projects like this are only as powerful as the number of integrations that are written for them. So once we establish that quality level in the language libraries I want to see those integrations start flowing in.

H

And I'll reiterate the integrations from a slightly different perspective, which is this will only be as successful as we want it to be and fulfill the vision that we have for it. If we get really good user feedback, we understand what you need and we have you come help us and join us and engage with this project. It can be commits. It can be questions, it can be.

H

Comments on on discussions participate because that's really the power of doing this in open source and standardizing and I want to also say thank you to all of the project leads who have been participating in this so far because taking two competing, or at least slightly contentious groups that have overlapping and product overlapping projects and merging them together for the the benefit of user, is huge and so I'm asking you all to help us make this as successful as we think it can be.

F

That something that I'm excited about and been excited about for a long time is having the ability to instrument open source code directly, so someone who's had to maintain open source libraries in the past. The observability was always a sticking point. They I want to report an error. I want to tell the user how to operate it, just admitting a log and suddenly I'm like well what logging library do I use if I just pick one and everyone picks one. Do these libraries then compose together? They don't that doesn't really work very well.

F

So I was always frustrated having to just spew stuff to standard out, because there just wasn't a great way to plug a real system into my open source code and the way we factored out open, telemetry, that's now possible, so I'm really hoping to see library and framework developers starting to take ownership of their own observability, not just putting in the you know the metrics and the trace points, but thinking more about shipping playbooks for the users explaining them how to tune their libraries, what to do in the event of errors like how to observe their own stuff, I think that will work much better if the owners of those libraries are thinking about that and I think they'll be able to think about that or be more interested in thinking about it.

F

They've got tools that let them actually do it. So that's what I'm excited about I.

G

Think, having tonometry everywhere is something that I've found very compelling when I joined Google like eight years ago, I used to work for Adobe and we had zero telemetry, we didn't know what's happening. We didn't have alerts first time when I saw alerts was when I joined, Google yeah. So having the experience that I had when I joined Google and the revolution that I had, in my mind, I hope. That's gonna happen with all the products and he's not gonna be only in Google uber and the big tech companies should be everywhere.

N

All right, any other questions, I got one more question.

F

Q

We created a client called a tenable sampler for your Java, open census library and was wondering I know you have your exporter that you're planning on doing some sort of a tunable sampling, but do you have the ability to do it at the client level as well, so that if people aren't running kate's or if they don't want to install some sort of process running that the actual code, that's on deployed, could do it dynamically without having to restore it or you know, deploy java, app, etc.

Q

We did submit a 20-page paper to your spec as well. That's all that's up there.

S

Can we have it as homework and get back to you absolutely I.

U

Think one thing I can say to that is that some of that functionality exists in iaeger clients, the ability to sort of affect sampling from the back end and for configuration and bug. Dan gave me his word that that functionality was supported and open telemetry. So because our goal is as a Yaeger project maintainer.

U

We want to eventually stop supporting yoga class because today is not going to be any different from open telemetry, and then we will do that like as soon as there is a future party and that's one of the features that definitely need to exist in the client level. I think.

G

I think already in two languages are so people start working on what they call the morph sampler, which is just gonna, be an implementation of what we call a sampler in our ecosystem that will talk to I, P port whatever and we'll we'll just configure itself at the runtime. So yes, we will support that now. Now there is a question about what protocol that will talk and we may have to have remote sampler for every back-end, because every back-end may want to configure different things and may talk different protocols and I.

G

Don't know if we have time until GA to define yet another protocol for that. But definitely we will have one for yoga and if you already have one, please help us with the with the protocol that you support and we will be more than happy to help you have that functionality included. So from from the perspective of plugins.

G

We already are like in ready for having this and we'll make the demo that we'll have, or the the the example that we'll have will be for yoga, because we committed that that's going to be supported, fully supported in open tournament.

P

We have need for the remote sampling feature with the eager to adopt open to omitted click. The collector internally, which we want to do I, was actually just coming on to take on the issue and for the collector in Jaeger. Just a proxy is it gonna be more complicated and the collector is it? Are you gonna have to like abstract it to Europe the open, telemetry standard and define that, like basically can IPR a proxy, the Yaeger back-end? Now or should we? Are you guys, gonna define something more complete than that so I.

G

Think there are two things: remote sampling. If I understand correctly, you want to affect that at the client level, the.

P

Client, the client just has an endpoint that it checks. An endpoint is generally the Jaeger agent, which proxies backs correct. It's a it's a very like simple change. Unless, of course, there's a lot more going on with open telemetry the collector stuff, like you want to define your own abstraction so.

G

So, first we want to support our library to talk directly to the agent okay. So what we will do we already support an Yeager exporter, that exporter will be configured to push to the agar agent and then the remote sampler specific for Yaga will talk with the same endpoint and retrieve the same data.

G

Now, if you are want to put a collector as well in this infrastructure, I think we haven't made yet any design around that we, we are focusing right now to make our client libraries just be able to replace the Yeager client library, which means to collect the the agent that the agar has has to be there. For the moment.

V

I was just wondering um given what you Ted and ser gays talked about about this goal of having telemetry ubiquitous with libraries and frameworks for any library framework. Authors. Do you have any advice in terms of timing of when to jump on and start using the API or just best practices where's that just an evolving story? Well,.

F

I really really would like to see people testing it before we say it's 1.0 and not use 1.0 as the the reason to get involved in it. I would say a number of these languages all have tracing fully-baked in and metrics about to be so it's a good time to really start kicking the tires on it. Now we're hoping to be moving into beta rapidly we'd like to wrap up this specification work by end of year and March I think is a stability point that we're aiming for, though giant Asterix.

F

We are engineers our ability to estimate time is poor yeah, but that's what we're aiming that's what yeah, but two weeks, yeah two.

V

F

We're aiming for March so.

Q

Not to talk about the spec, cuz I know you want to finalize it in the next month or so I know you have this term called baggage, so do you have a way were within a corporate infrastructure systems can identify themselves so that way, downstream systems can then say: hey I'm gonna stop the trace, because sample is true. Yeah. Do you have that ability, no I'll submit a ticket yeah.

F

This is actually the the baggage mechanism is actually very basic. It's it's more like just basic propagation, and one issue is that there is no really standardized concept of security zones or network boundaries in a way that you could actually like grab so I. Think it's an open question: how best to do things like ceasing baggage propagation, so please get into it. I think one.

G

Thing that we can offer from from our API library, implementation is for the samplers, give access to the baggages. So the least again coming back to the same question for some implementation of remote sampler that wants to use the baggages, they should be able to do that without having to change our make cord library.

D

Any other question: okay,.

J

D

I think that's it so we'll stick around for a little bit to answer any questions you have, but thank you for attending this and our other talks on open flow. Each week you can have a great conference.