Open Telemetry Uncategorized, 5 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-10-05 meeting

Description

Open cncf-opentelemetry-meeting-3@cncf.io's Personal Meeting Room

A

Howdy, hello, Peter.

B

A

B

um During lunch, I apologize, I would say hello, but there will be gross too ignorance.

B

Hello there it's good to see you on.

B

A

C

A

You have a bigger office today.

C

I'm in the living room- yes,.

A

B

Clay, do you need anything from Miami with um you spoke with Gabrielle and uh those folks.

A

B

Clay with one of my colleagues I, wasn't sure, I think what they're good, but just let me know if you need anything from me, I wasn't sure if you they got back to you and so oh.

D

Yeah, hey thanks again Eric um synced with him on slack and they had some great uh input. So much appreciated.

B

Right, you know I'm good at how I'm introducing you to people who know more than me so cool yeah.

D

B

A

All right since that we don't have many items on the agenda. We have one right now: um Aaron I'm, going.

E

To get started sure I can kick this off um so I, don't know how everybody how familiar everybody is with this, but uh currently The Collector is using open census and they're trying to migrate to open Telemetry.

E

This has caused a big uh to do in the go Sig, which is where I'm coming from I'm one of the maintainers and the go Sig um about how we can unblock you, guys. The Open, The Collector, um there's a couple options that I laid out in the ticket I, don't know if that's the right ticket to be tracking this by the way, I'm sorry I, just kind of looked and found the one that seemed the most relevant to this.

E

But uh the big sense here is that we need a way for um the sense that we get is that we need a way to combine the open census data that uh you're currently collecting with the open Telemetry data that we want to generate from the open delay inventory so that you guys can gracefully migrate from one to the other um and I kind of laid out three options: um option. One is probably the longest running option of.

E

uh Let's uh resolve this at the spec level, so that we can actually write something that we can guarantee is stable um and I hope I link to the right spec issue. But this is the metric provider issue. That's been discussed at the set spec at least once um and I will go, find that spec issue. uh The timeline for that essentially would be spec agrees to it gets accepted, and we would uh probably the next release cycle and go release an API for you to use because there's a lot of attention there.

E

um The option two is: we could put some of that code in the Prometheus exporter directly. um We have a little bit of reservations on this and uh the the biggest reservation is. This code doesn't seem like it should live there.

E

It should live somewhere else in our API, um so this would be creating an API level for you guys to use, but we want some kind of deprecation plan, some some way to know that we can go and safely remove it front and not break you, uh your uses of it- and this is not something that would end up in the overall stable API.

E

um Sorry I should be sharing my screen. I guess.

E

E

Let me pull that up. Sorry.

A

E

So uh issue one is the metric producer API, uh we finalize it at the spec issue. Two is a temporary API um within the Prometheus exporter, but we want to develop a plan and if you want to go with this, that's fine. Let's work on a plan. The other option that I thought of this morning was Prometheus, allows you to have multiple multiple collectors registered with a particular register.

E

Would it be possible if you guys just register the open census and the open Telemetry and then maybe sometime in the future, when this API is settled, the metric the metric producer API, is settled um you can you can migrate to that.

C

I I think that's a good, a good idea so consolidating on the Prometheus side. That comes with the downside that we cannot support any other exporter for the moment which I'm fine, because I believe it's uh we, we were surviving with this for years, so I don't believe we we need to rush into changing that support. For the moment we understand the downside. We document it for the moment until the bridge or anything we may not even need the bridge.

C

If we have this, we may start changing everyone to open Telemetry and we may not even be blocked on the bridge. So we have to exit paths from this state either we change all the codes to Hotel go, so we no longer need a bridge and we can change the support to allow other exporters based on the hotel goal or you are ready with a bridge then. Hence we can support other exporter, so the Exit Plan being when are we allowing? When are we going to be able to allow multiple exporter or different exporters?

C

That execute plan is either one of these two choices, so I think I'm? Fine with that. If somebody comes up with that proposal now there is another PR coming from the uh from that: guy uh paivo Gustavo, something I, don't know how to pronounce the name uh in regard to this, which is uh which one to use, uh but that's a separate topic. Probably if I don't know, if that's what triggered this discussion or anything.

E

A

It uh the question that I have is um is the bridge, um so is the problem that both Hotel SDK and the open census SDK are trying to use the same properties register at the moment.

C

No, the problem is the problem is right. Now we don't have the same register right now. We either initialize open sensors exporter with Prometheus, or we initialize open Telemetry go with arrays. So now, if you are recording data with open sensors and we we initialize uh open Telemetry, you will not see your data and vice versa. If you record with open Telemetry, we will not your data.

C

What we try to become to to come into a state where it doesn't matter which API you are using for the moment, uh uh you will see the data, that's the state that we want to be the intermediate State. The exit path from this state is, we want to move everyone to go only use Hotel go when we are ready. There may be here because I see you, you have some concerns about histograms and stuff, we'll try to address one by one everything, and once we are confident and ready that everything is there.

C

We'll drop the open sensors, but we want to be in a state where you can record with both apis and the data are visible. We got to in this proposal that uh that Aaron has gives us that, but with the downside that only Prometheus exporter is supported for the moment, we understand the exit paths that I explained five minutes ago or whatever, whenever I explain them so I think personally, I I believe it's it's very good and Aaron.

C

If you can summarize the plan that we have right now, the when are we gonna be able to support multiple exporters, and when are we gonna be able to drop entirely Open Access as a summary of the discussion that would be great for everyone to understand the whole conversation.

A

My question was uh geared towards a different aspect of that and, and that is the bridge itself, so is the bridge um functional as it is right now, no.

C

No, that's not that's not the case. That's the case. What we will do urasi is. We will actually bridge at Prometheus level, so essentially.

A

The one, so my question is um because of something else, so we promised a couple of years ago, when open Telemetry was formed, that uh people that are using open sensors would still be able to migrate to open a Telemetry with no efforts, a bridge or a shim would be provided for for those users.

C

A

And that's that's why I'm asking if the bridge is functional, so are we satisfying that requirement or are we dropping that requirement? Are we saying that those users who trusted that statement are not gonna.

F

Do this there is a bridge, it is functional, it does not function for exporting to Prometheus. It doesn't work with pull-based exporters. That is a problem that needs to be solved. There's there's work on going in the specs trying to solve that. That will be presumably step two of this plan. That's where we get to the point where we would be able to support other exporters but are still using open census for instrumentation, because we're using that bridge.

F

um What we're trying to do now is identify. Is there a way to get us to be able to use the open, Telemetry SDK sooner, while still exporting to Prometheus as the only destination.

C

Correct and also also generosity, to be honest, that's a problem for for for SDK is not for us, so I think we should not discuss here, even though we may have people that know about this problem, and everything in this call I think it's more or less a problem for for the open, Telemetry goal is the key to to have that from our perspective of The Collector, we should support the receiving open census data and export opencso data, which we do sorry Anthony if I cut you, but I I want this discussion to be in, in other things like specifications or other things.

C

For my from our perspective, we should keep this on on our use case now. It would have been great duracy to your point if we were the first adapters of the bridge sure, but but I feel like these people are, are eager to start using open, Telemetry, uh uh API and SDK for metrics, so I'm willing to accept this or otherwise we can make a call and say: okay, we want to wait for the official bridge and stuff like that.

C

I, don't know if that's necessarily the case I would like I'd rather start and deprecating that and start using open, Telemetry and be earlier adapters of the API and identify issues possible or things for for these folks.

A

Yeah, so for my purely uh open, Telemetry collector perspective, I agree with you and agree with the plan. I mean, let's be practical and let's get it done, uh and um now, with a broader perspective, we discussed before that. We want the collector to be a one role model for the the community uh in general, so we should tell them how to use battery excuse, traces and so on and uh the way that we do things like or not like it or not, is going to be seen by external actors and replicated.

A

So if we do that by by doing you know the way that we're doing by.

C

The way that solution for Prometheus I, don't think, is the wrong if you are purely using Prometheus exporter, that solution is not necessarily wrong because you keep the open, Telemetry SDK, with their pros and cons there. You keep the open uh census SDK with their pros and cons there, and you just make them merge into the Prometheus. Where is the like the exporting path? So I?

C

Don't think it's necessarily the worst thing to do, even that in the in general, uh because even the bridge will most likely just put a slim layer between between these registry parameters registry and the open, Telemetry SDK, As I understood right now, because the the bridge will still use the open, Telemetry SDK to compute the Aggregates again. That was my understanding as couple of days ago. I don't know if that changed it.

C

So so that being said, it's the bridge will be for, for in open uh in Prometheus case will be just a small shim between this to to talk a similar language and not talk open, not talk, Prometheus language, top Talk hotel language between them anyway, too many things I feel like this is great Aaron. Thanks for for this recommendation, I think it solves our now. The thing is: does open sensor support that gather and stuff.

E

There is configuration options on open census to do that. um I believe Gustavo, who looks like he's in the call uh confirmed that that worked um like I'm I'm, not 100, sure I've I've not experimented too much with it. So.

C

Yeah, let's, let's see a PR, let's experiment a bit, we did and they and then maybe maybe one of the thing you should also maybe document as a temporary solution for for for even for open Telemetry go on the open, Telemetry go because I believe it's it's a nice solution for for this problem.

E

So, uh just so that I am accurately capturing this. The uh current plan that you want right now is to use Prometheus as the bridge um until the uh until we go. Sdk has uh the necessary API to have a working Bridge so that right.

C

So so, yes, yes, so I just said, there are two exit plans. One is these the other one? Is we switch everything to to open Telemetry already, so we no longer need a bridge because we are out of open sensors.

C

G

um Gustavo, when you did you notice any other differences between the exporters when you're, using open census and open Telemetry together other than I think you mentioned Target info being different. Was there any other major differences between the two.

A

The only difference is was supposed to attack it to info and also the prefix, but uh the prefix PR was smashed like yesterday.

G

Okay, I I think if, if we were able to do it without any breaking changes like keep the metrics exactly the same, then this might be feasible. Otherwise, I feel, like a feature. Gate would be necessary here. We.

C

Already have a feature gate there for enabling open sense, open, Telemetry API. So we will protect everything across that feature.

G

Gate I just mean that so in that case, then we would be maintaining uh side by side, hotel and open census.

G

Telemetry like, for example, in Ops report, would we have both open census and open Telemetry and you'd use one or the other, based on the feature gate.

C

For the for the foreseeable feature, I think I would do that anyway, because I, because open Telemetry go is a very new product. I want to have an escape path. If, if, if anything breaks but but let's say in in couple of weeks, I think we should look into not having that.

F

Well so I think David, maybe asked me a slightly different question here.

F

um Where are we going to support stopping using the open census, API and emitting the same metrics with the open, Telemetry API through a feature gate or is the feature gate only going to control using the open, Telemetry API for new metrics.

G

Basically, the value of uh merging open census and open Telemetry metrics together in a single Prometheus registry would be that if half of your metrics are open census and the other half are open telemetry, then you can like mix them right. You can combine them, but if we're planning to maintain distinct sets of telemetry anyways like Side by Side, open census and open Telemetry, then I don't see necessarily a reason to combine them. If that makes sense,.

C

I, don't think I I would prefer not to have that side by side or anything, but I would prefer so I would I would start with side by side for for at least or two David for at least for some of the I. So I will not jump and go and change everything. But for the moment I will change.

C

Let's say Ops report, as as uh as Gustavo did, keep that side by side for a release for or two or two and then figure out, okay, everything works and then I would like to move to the next stage, which is the one that you propose.

C

So user can choose between the two apis and uh like not necessarily choose users start to move to open term. Let's say: go to say that and uh and and start deprecating support for for that for the open sensors.

H

The pr you're talking about the one I just paste it in a.

C

Chat yeah the only problem with this PR by the way I have we shouldn't expose US Open hotel for metrics. That's just an internal thing that we we don't want to expose publicly, because the final state is. We want people to only use Hotel.

G

Do you think there will be some uh libraries where we'll want to have side by side then and others where we'll want to have um like open, Telemetry, replace open census, as in will different parts of the code base be in different stages of this progression?.

C

I, don't why why why would we somebody want to right now use open telemetry.

G

I'm sorry I'm just I'm just trying to understand.

C

um So my my so libraries should not be able to use open, Telemetry yet Library anyone else except our core thing. Where we do the transition for the moment, should you still open sentences until we declare that okay, we are confident into the setup that we have. You should no longer use open sensors. You should start using uh open tournament. They should never have side by side. We do side by side because we want to be cautious if we want to test these things, but not not externally, we should not spread this state.

C

We are in a in a state that we want to be for a for a cycle or two just for for our safety, but not something that we want to spread into the world. Okay.

G

Cool, so it's like for now. This is basically we'll have side by side and someone will be able to.

G

Someone will be able to try out just the hotel, Go instrumented stuff and when they do that, they'll lose all the other open census based metrics um elsewhere. Right.

F

But they should get the same metrics produced through the open Telemetry go as you go.

G

So they'll get just the ones that we've just the core Ops report- ones right that we are maintaining side by side.

C

Right now, that's the keys Adam's proposal changes to the next level, which is we will get all the metrics.

G

From both combined correct, but that means we can't have them side by side, because then they'll Collide right.

C

What does it mean side by side? We.

F

Can't be producing out of odds reports, both open census and open Telemetry metrics concurrent right. We.

C

Rewrite now produce one or the other, because we want to be safe, so we right now have code to produce both. But it's not running. The two codes is running one or the other, based on the feature flag, yeah, so ET side by side in some ways, but is not both cold. It's defined by the feature flag, which one is called it's an if statement if feature flag, do open sensors, otherwise do open term material the other one.

C

Because of because we are worried to swap that right now and we want to have the safetiness of reverting the change.

A

F

Some something where a data flow diagram could help to make sure that we're all on the same page as to what's coming from, where I'm going to. Where.

G

C

I'm not doing it since I'm not owning this. If somebody wants to do it, foreign.

C

D

um I'll I'll chime in uh jacobarnov from light step uh is gonna, be the tech lead on this. um Gustavo is also helping out. um Jacob is on vacation today on PTO, um so he wasn't able to join, uh but this is kind of a follow-up item from The Proposal. We shared uh two weeks ago from kind of going down this migration path, and um you know, sounds like we've got some things to share data flow diagram kind of what the migration path is. You can share that in a future second meeting as a follow-up.

F

Yeah I think that would be helpful. I think we're all on the same page here except it sounds like we may be talking past each other at some points. So having a bit of clarity around how that migration is going to work, it would be helpful.

D

Yeah sure things just just so um I've got the right list, and- and um this is kind of from my notes- it's you know- I think maybe a visual representation of how this all works.

D

Because it's you know it's probably easier to kind of understand that way, but also the steps of you know following Aaron's recommendation once that's done when when can we drop open census and and what that looks like and then what the path looks like to supporting multiple exporters uh after that and then I think it's part of that multiple exporter uh conversation. It's also, uh how does the bridge fit into that?

D

um There are probably some other things too, but um we can certainly figure that out.

G

One one thing I just would like to have in that is: can you describe whether we're featuring how the feature gating is going to work? Are we going to feature gate per library that produces instrumentation? Are we going to feature gate um when we're doing sort of the global setup for endpoints?

G

If that makes sense,.

D

I think so so this would be. um This would be when it goes outside of core and into the contrib uh as well. Is.

C

That I don't think it should ever go outside of core until we are ready in court like I, don't think we should spread this more than it's already toxic in the core that we have multiple libraries and everything.

D

H

Fair enough, so no go ahead yeah. um What are your thoughts on how this should then happen in the control repo? Are you thinking that, once we have confirmation that open Telemetry is good enough, um we'll just switch all the control packages to using it or.

C

Are we thinking we're not switch it, but but but what we will do is from now on, you use only open Telemetry. You cannot use open sensors from the new code and all code. We slowly move, but it's not like it's going to be feature gate or anything. We are already confident that open Telemetry works. So we're not going to ask third party or external things to protect feature gate or anything.

C

They would need a feature gate if they do while we are also in transition, but once we finalize and stabilize in core I, don't think they need to do they just record to the new API, so they don't need any feature view. That's what that's where my disconnect is coming from we're already confident into this. So you just work on the new, well API and that's it.

H

Yeah I guess I guess in my mind the disconnect here that I'm seeing order confusion that I'm seeing is that there's kind of three ways that we can go about doing this work and like there's, there's a pre-owned card that uses a feature gate to emit, both with open Telemetry and with open sensors. So that's one way right. We could just use the libraries directly there's the way that we had talked about, which is using the open census Bridge, which we are now saying, is not going to work.

H

So we're not going to wait for it and then there's a Prometheus Bridge right, which would then not be a no no change on the contribute side for any of the libraries that are producing metrics.

F

um Just to require I, don't think we're saying that the bridge won't work, so we won't use. It I think we're saying that there's a path forward sooner if we just use the Prometheus register as a temporary bridge, but that limits us to only Prometheus. If the bridge is ready before we're ready to completely transition.

F

We can start using the bridge and gain multiple exporter, support then and then completed by the transition eventually- or we may just go straight to that last stage, but that will depend on the the timing between the bridge being ready to support Prometheus and other receivers and the the go SDK being stable and well tested.

A

uh But do we have uh fishergate controlling the like the whole flow, including well Prometheus? Why do we need bridge at all right, I? That's that's consistent! For me. We.

F

We need the bridge if we want the existing open census instrumentation to be able to be used with non-prometheus exporters through the open, telemetrico SDK.

A

Yeah, but as they suggest that we, we can have side by side and have be sure to get to control like the whole flow of the metric reporting right. So we have like feature parameters for open sensors premises for uh open telemetry and we can switch between them.

A

C

B

C

For the diagram, let's do it for the diagram. We are just going in rounded Corners here. uh Let's wait for the diagram to understand better uh I. Think I think we all agree that this is the path forward. Let's uh clarify in in that diagram, if needed,.

D

Just one quick follow-up on the diagram, so Jacob by Gustavo air and we'll we'll put something together before the next Sig meeting. But this is specifically the the data flow diagram for uh feature. The feature gating correct.

C

And for how how things are working between the two libraries? What what can users use? What cannot use if, if they have metrics with this, are they visible or not and stuff, like that, okay, cool? Thank you! Okay! I have to leave guys. Sorry thanks.

A

All right, um I think: that's the only item that we had for today. uh Anyone wants to discuss anything else.

A

All right, so that's it see you next week, folks.