Open Telemetry Uncategorized, 9 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-03-09 meeting

Description

Instrumentation: Messaging

A

A

So there we are Advanced accounts.

B

C

B

C

This uh SFO Earl Gray I, see.

B

I'm on in San Francisco for a team on site, so yeah I get a full conference room for it.

A

Nice is it hot.

B

In uh San Francisco, it's cold and no.

A

The earl gray yeah.

C

A

Not hot Anthony.

C

I gotcha: they need to turn up the temperature, it doesn't come out. Hot is John, Luke Picard there I think, is what Anthony's asking.

B

That's obviously why I bought this group instead of the other ones, because they're all named some different drink I'm like oh girl, you're right yeah. Look at that one.

A

D

No I'm new here so I'm also happy to introduce myself at some point. I, don't know how long people trickle in oh.

C

Yeah, sorry Katie. um We normally wait a few more minutes, but yeah you're welcome to to introduce yourself if you'd, like yeah.

D

I just want to let you know who I am so I get to say hello, uh because I'm not normally in these calls, um so I work at datadog. Now uh previously I was uh at Google for a while um working on the go standard library for the last few years and then um came over to date.

D

Night last year, so um yeah I worked on the um I, rather I work now on um tracing complications um and Dave docks I work on the client Library, so um I'm the lead of the team, who will be uh using essentially the like otel API.

D

um You know um repository that is being maintained by by this group, so yeah I, mostly just want to come to have a chat about the um the ticket file just to talk about backwards, compatibility and see if I could help it all, just also from like a go perspective or from my team or like why it's important to me or any of that and just want to be able to chat about it and help.

D

However, I can and then also just like, if there's anything that you know, um data can help with or anything you just want to talk about. In general, with me, I'm also happy to just listen to so um yeah. That's why I'm here just want to say hello.

C

uh Cool well yeah, welcome uh and I would recommend. If you want to talk about that issue, uh if you could add it to the agenda, uh I don't know if you have access.

D

Absolutely yeah I posted on the slack, the other like a couple days ago, um but I will find you mind sending the notes in the chat, if that's not too much trouble to just send the link. Thank you I appreciate that yeah. Do you want me to add it to the end or I? Guess.

C

uh Yeah I I could probably go above that, but um where.

D

I'll just put it there, you can move it yeah.

C

Well, I think we actually have uh Quorum given uh David's out for a while.

A

C

So yeah we could probably jump in here. um Okay yeah, if you haven't yet already. uh Please add yourself to the attendees list and uh similar to what Katie was just doing. If you have something you want to talk about, please add to the agenda and uh a lot of money. We can get started.

C

Cool so first up, uh Robert has uh opened an issue kind of a meta issue that was addressing something that was brought up for this compatibility between the V1 11 1 and the v1-112.

C

um Although I do think that this was uh related to the metrics packages, so I don't know if that's the right version, but um anyways the the issue that we talked about last time, where we have this uh cross module boundary internal access to packages, and so I think this is a pretty comprehensive list. I think I talked with Robert a little bit offline, so I'm going to try to do my best uh to kind of give a synopsis of this, but um yeah.

C

There was this proposal that he has for this new utility called go cpy, which I don't know is not go CP, but um yeah I think this is one of those things where we decided we. It was a. It was an interesting idea if we could just have all of the shared code copied via like a generation.

D

C

uh To prevent the cross, module, um Imports and I think that was good. uh The only thing is, is that like I, don't know if it's still an open question here, um this uh internal Global package is an interesting one because it requires for the logging. uh All of our internal logging for the project is listed here. I thought that he listed it in this issue. um Oh yeah, here it is uh yeah so like this won't work for all of the issues.

C

This one might be a unique one, because we either need to export it publicly, uh which I think is there's like an upside and a downside to that and then or we need to state that this is like. If it's going to, you know, be used across uh module boundaries that it needs to be a stable, API.

C

um So I think if there's like an interesting idea there. The only issue also is then like what is the forward compatibility look like because if we wanted to do this- and you know, do some sort of copying, uh if we did the copying in all the internal package, Imports changed and then we deleted the old internal packages.

C

That used to be existing like we're going to incur all of the uh breakages that this is trying to avoid so like do these old internal packages just have to live on indefinitely, for compatibility reasons, um I think is another big question, so I think uh I think it was a good idea. Robert opened this up, um so I think that it's worth, if you have questions of your own or answers to any of these questions uh going in in taking a look here, I think it's worth commenting on it.

C

I'll pause here, see if there's any other questions, people have.

A

I think I generally support this approach. um The the internal Global one is kind of the the odd one out here. um I think they're. The the right approach is probably just to say that it's an internal package, but we will maintain a stable API for it, because we can't really copy that around everywhere. I, don't think.

C

Yeah I mean so yeah, so I I definitely like. We definitely can't copy it because you need like a global uh implementation, but one of the other proposals I thought there was like a yeah there's like I. Think it's in this is the pr um was.

A

You know you could also.

C

Make this a uh external package, but then you run into this problem where you have like this. Like you know, if you expose this as log internal, you get some benefit right, because then it's actually exposed as an exported package. It's not internal anymore, so it would need to conform to the API guarantees and like there's no reason any of us should have any mistake to say like it does. It also provides access to say, like instrumentation Library if they wanted to hook into the hotel logging, uh they could do that.

C

It does then also add this confusion that we haven't really resolved here, where, like there's, an error, logger and there's an error Handler so like that would probably be need to be better defined for especially for the instrumentation libraries which could use potentially both- and they may you know, have this on their own question about it.

C

um It also means that other you know third-party systems could use this in ways that they don't really expect, meaning that they make mistake this as like a log API, given the fact that we are eventually going to add a log API so like yeah.

A

That's my biggest concern here is: we we don't want to be producing an end. User-Facing logging API- and this could be mistaken for that, especially if we're saying that it would then be usable by instrumentation which would use it in a way that probably looks very much like just using a logging API, um so I wouldn't support that.

C

Yeah, exactly like I was on the offense for the same reason, um so yeah I mean, if that's the case, then I think that I think what you just described, Anthony is probably the best uh is also I. Think Robert was thinking was just like you know, uh throw up your hands and just say like okay. Well, this is just gonna have to happen like this internal package is going to have to be shared across module boundaries. We need to comments that this is done and make sure that it's not it's.

C

It's given the same stability guarantees that all of our other external facing apis contain.

C

B

This work here seems to be targeted at allowing us to have internal apis that we want to mutate, but not have those mutations break between version and version, conversion to version and I think that's a lofty gold, but if we do have something that doesn't need those same requirements, then we just set it as aside as something different and I think that's what we're hearing, what we're sitting at here right like the global we don't want to copy around, because then it's not global right. That's the whole of the whole reason we.

A

Have the global package so well and more specifically, because it has State I, think all of the rest of the packages we're talking about are helpers and things like that that don't carry a global state that needs to be synchronized across users exactly, but that that's what I'm saying is. Yes, it has State.

B

And it's the global package, because it's it's the one Global you know Singleton that we want to have of this. So if that's the case, then why don't we treat it as something different from this proposal? Right like it? Doesn't it's not covered by this and yes, we do need something else for it and we just probably just put a documentation in there, because we don't have any tooling right now.

B

That um does some double check for us that we haven't broken an external API other than ourselves being good at that right, like ourselves, double checking each other. So.

C

Yeah I think I think I understand the same thing that we're all saying as well like I, think the global is just going to be the odd one out and it needs to be noted as such I think that uh Robert wants to update the contributing with the general guideline here, and so maybe this just needs to be an asterisk. That's saying it like this is the one exception.

A

um I mean potentially uh I would just.

B

Go to go to Baton just say it can be documented within itself. Right like it's within the global, because if we I don't think we're going to come up with something else that has a global state or doesn't that we don't. We want not any answers to use, but we don't want to copy it around.

B

um But if we do, we have precedence for it and and can keep going like. We can keep doing this.

C

You know if we make a case for it and figure out that we do actually need that right, yeah, yeah, I think we're all saying the same thing. um Okay, so I think with that, if you have um other like suggestions, I think that we haven't touched on here or um other packages, I think that we haven't actually identified here. Can you please be sure to add them here and then also maybe just like uh Robert was asking.

C

If we wanted to go forward with this I think so I don't know how do we uh move to from proposal to accepted uh but I think it? It sounds like there's consensus here that we want to move forward with this.

C

Does it sound good.

A

Certainly not hearing any opposition.

B

C

Okay, cool uh awesome next up uh I've got this new addition to the metric data package.

C

This adds a new Exemplar type um and it only adds an exemplary type to the metric data package, which doesn't actually implement exemplars um I wanted to kind of bring this up because there's this open issue that I created looking at the stability of the SDK, which points out that, like histogram data, point right now, we need to contain exemplars in the future and it isn't defined generically over any sort of argument, and so um how this would evolve in the future will change depending on its stability guarantee.

C

So if we do this after video 1.0, it is going to have to do something um along these lines where you have some sort of well. This is this is I, think what it currently uh is proposed here, although this is going to be redefined generically. But you know, if you didn't do this, you'd have to do something where you have like uh similar to like the generated probe of some sort of interface, similar to what we do with aggregation somewhere else and I. Think that we could do that.

C

There's also this other thing where the Exemplar value could be redefined so that it itself is doing the interface. So how far you want to push down this interface is something we could do. Host 1.0, um but I think that the ideal solution is to just Define the histogram data point generically and I say that, because the histogram data point actually contains uh extrema and a sum that are currently defined as 64s I, think, specifically for compatibility with otlp but like they are all translated based on um a conversion when they're generated.

C

um So there's really a you know, not a reason they should be 64s. They could be based on the type of the aggregator or aggregation that is actually being produced. So I think that there's like a reason to correctly Define this with a generic type um and because of that I wanted to implement uh and add this in. So we have this ready for a forward-facing uh implementation of exemplars.

C

This isn't a full implementation of exemplars I do have like a working proof of concept. That is end to end uh for, like uh one aggregation, I think last value, and it's like really messy. uh Let's just say that it's really messy um in fact like yeah, it moved all the aggregators around.

A

C

It but I think it works. um Well, I, don't think I know it works. It produces data. So I can say that, like this approach does actually produce exemplars it it made, it may uh potentially be optimizable I'm, not like the one thing that I think that is um something I didn't take too hard of a look at is, if you want this slice to be over pointers to exemplars or just to this slice of exemplars.

C

The otlp um protobuf has pointers but like this is already an indirect reference type to slice so like it didn't make sense to me why that was in the building of these exemplars and the aggregator. It didn't make sense to actually build them as pointers so like I. Just left it as that, um but otherwise, like this looked pretty straightforward um from what I was seeing before him.

B

I saw I started looking through it. This morning um it looks like it's pretty straightforward. uh um The only thought I had is I. Think I know it's a different thing, but we are going to eventually have to support um exponential histograms and um are we going to have to go through this same.

C

Yes, that is also, on my mind, yeah, so I I definitely saw as like. I wanted to look at. This Josh had already proposed an exponential histogram uh proposal, so I was gonna like look back because I think the plan was to always add because we have the histogram data point I think the plan was always to add an exponential histogram data point to the metric data package, but I I completely agree.

C

I, think that we need to look at that before we go stable, just to make sure that, like there aren't hiccups like this, like um you know like we could make this change going forward in a backwards compatible way where we don't add this or we don't change the signature, but like I'd like to look at the exponential histograms, similarly just to see, if, like there's an optimal solution, that we should probably try to do before we stabilize it, um Aaron I know you're leaving uh on leave next week.

C

So I don't think you're gonna have time, but yeah I mean I may not get to it next week as well, but I would love somebody else to look at it if they have time as well. I think that's that's worthwhile, um but otherwise yeah. That was something I was going to look at next.

A

New histograms need to be generic across antigen floats because the boundaries are floats.

C

A

Make sense to store anything other than an editor or to record observations is, is anything other than? Is there anything other than a float.

C

It does because uh they have exemplars and exemplars are examples of the measurement itself, so they're going to have a type. The Min and Max currently are defined as floats, but that's necessarily that's not representing the underlying measurement right. The measurement itself could actually be an integer here, similar to the sum. The sum could also be an integer.

C

So I think that if you define these as uh generically, the next step is I kind of commented in this issue to Robert is like I think these should get redefined to be generic as well, because they would uh authentically like not authentically without translation, represent that measured value.

A

Remember does that fit into the data model like the data model defines them as floats and integers will exceed the range of loads. You.

C

Mean you mean the otlp right.

A

C

Yeah I mean yeah, I, think you're right so I mean, but the thing is like I, don't necessarily think all export formats are going to consider them. That way, like I, still think that you can like similar here to like um well exemplars exemplars are a good case, because the otlp defines them both as floats and as integers. So those those are something that we would want to preserve the um the type of the the measurement down the pipeline to the otlp.

C

So if you talk about the oclp data model, exemplars do contain that and if histogram data points also contain exemplars, then they would need to retain that information. But you're right. The min max and some are not some they're all represented, as floats Downstream in otlp. But I also think that, like, if we're going to include this uh change, to have a generic there's, nothing to say that, like um other export formats aren't going to, you know, be able to support integer uh values of these or yeah. Actually, that's a good point.

C

They may not um support integer or um floating Point values of these so like it might be useful to provide the option.

A

Yeah I guess going back and looking at the the data model, spec, not the the Proto definitions of the spec. It doesn't talk about the data type, just the presence of.

C

A

Max summon count, oh.

C

Right yeah in the specification, yeah you're right: it does not um no yeah.

C

um You know, but like yeah, like this data type I think is there's no reason to have this be a generic rate. Counts is just going to be that I.

A

C

A

C

Exactly yeah, and so like um yeah, but I do think that these these could be enriched by looking at it as a generic type and, if we're going to add it because exemplars do need to be defined like like I said like they don't they don't have to, but I think like the most um appropriate design for examplers would be to try to Define them over a generic argument.

C

Just for the compatibility reasons similar to how we do all the rest of this. You know the data points um you know like that is something else to keep in mind is, like you know, all these uh data points.

C

huh I thought this was added at data points. I, don't know why this is. This might not be correct. Oh no, there's a okay, sometimes GitHub, confusing.

A

Yes, okay, so exemplars are also in data points right. So that's also.

C

Something to keep in mind is that, like this is already defined generically, um and so this works really well. If we didn't Define exemplars generically and we try to push in some sort of um interface conversion here, like you, would have both a generic argument and you would have some sort of interface um uh type switch to try to decode these into otlp, which is not like the end of the world.

C

But it does kind of screen that there's some sort of like mix up here and I think that you know unifying on a single approach by using the generic argument that we have for data point makes sense so like if that, if that's the case, I think that our design here can also be copied to the histogram data point as well.

A

Again, this is this is a change. That's just to the SDK right. There's no API changer.

C

uh Correct yeah, uh SDK and uh exporters um because the exporters rely on the metric data. um But yes, it is not the API right.

A

Yeah I view exporters as part of the SDK right yeah. That's.

C

Yeah um I do remember now that I was looking at this. One thing that I that I did kind of want to call out is that, like I, have the spanity and the trace ideas just bite slices here. Eventually, the metric SDK will need to import the um Trace, API I. Think because that's where the spam context, uh tooling, is and uh all of the keys for context as well.

C

So you know I didn't, put uh a trace dot span, ID here and a trace dot, Trace ID here um just to avoid the import right now, but I'm not. This is like.

A

C

Only thing I'm, not 100 sure if this is the right data type representation in Exemplar um I guess is the question.

C

For what it's worth, uh the otlp as well I think is defined this way, um it's just a bite slice, which is where I kind of hold this from I'm, not mistaken.

B

Yeah, that step would mean that all of the metrics rely on traces.

C

No matter what they are going to, yes, they're going to rely on the trace, API yeah.

C

Yeah um so yeah similar to the Proto, it's just a bite slice uh but yeah. So that is something that, uh in in the POC I have, for example, is that that did happen. I, don't think it's the end of the world, um because it's the API, which is like pretty solid stability guarantees, and it's still the API is being imported by an SDK. uh You know if API was imported by the other API.

C

That seems like it'd, be a little off um and we might have to readjust, but I think if this seems appropriate, uh given that's that's kind of what it's intended to do, but just wanted to call that out um in in the review process.

C

I also I think uh I've been thinking this morning and like I guess, um I might be shooting myself in the foot with this, uh but one of the things that I'm also thinking is that, like in this PR, we add the Exemplar type. We could just update the histogram data point type to be defined, generically first and not add the Exemplar at this point, but I don't know like I had a whole working, like example of it.

C

So it seems vital to me at this point I think that the key here is just going to be like decoding the metric data test or metric data package to um the otlp or the Prometheus, or the standard out exporter. But I already have like I. Think I already have a working example for botlp I. Don't foresee it being too much more challenging based on this data type of Exemplar. So.

C

um Okay, SFO Earl Gray says he has to leave and I have 15 minutes. Okay, I, don't know when that was uh um okay, cool with that said, then, maybe we can move forward. I think I've probably talked this one into the ground a little bit.

D

um Zero pressure, if you can't, but just because I will have to make sure I leave it too. If there's any way to move mine up so make sure it happens today, if not no worries, I can always come next week. No.

A

Yes, if aren't a slave, can we pull that up? Yeah.

C

C

Cool so Katie I'll hand it over to you. We can talk a little bit about this. uh uh This issue here, yeah.

D

No thanks, and so basically yeah I mean I. Think probably most of you have already read it or know roughly what it is, but there's just several addendums in the um API definition where it says what methods may be added to this interface in minor releases and I.

D

Think, as we all know, like that's gonna break backwards, compatibility requirements, um it's likely to break builds um so I definitely have just concerns about because, like we're going to be implementing this API um inside of datadog, like we already have PRS out that are doing that and I'm worried about.

D

um You know a future where this is going to be extended in a way. That's not going to be safe, so the conversation's just been about like what can we do to make it safe um I have some proposals of things you can do, because we've also run into this before where we rely on interfaces and our code and then when we want to extend the interface, it wasn't straightforward to do that.

D

Obviously so we had sort of workarounds that worked for us, um but yeah I just wanted to talk about this a bit and just make sure just as a user, essentially of the API that I can feel safe using it, knowing that it's going to be safe going forward. Essentially, um you know and not be surprised, like six months down the line when suddenly tons of customers and like the whole ecosystem is broken because of something like that. So um yeah just wanted to talk about that. A bit.

C

Yeah, um so yes, this is. This is an interesting question and we've recognized it since, before the stability uh guarantees were implemented on the tracing API.

C

So the open Telemetry project has a pretty strong stance that the API uh can't be broken from the instrumentation side and that's pretty key, because if we have all these libraries using the API to instrument something and you upgrade the API and it breaks- that's that's a big problem or there's some sort of incompatibility between uh the API of one version and an instrumentation code and the API of another version and instrumentation code. So that's something that we're pretty um strongly in favor of uh preserving. That includes also major major version uh revisions as well.

C

So if you know you build against the V1 API of open Telemetry having to upgrade to the V2 is a big task, and it's not something that is going to provide the largest amount of instrumentation use in open source projects in adoption of open Telemetry. So the key there is that we wanted to try to provide uh compatibility, guarantees for the like, uh V1 or V2, or whatever major version apis I think it was for five years in open, Slumber tree.

C

So with that in mind, the downstream side of the SDK is uh not given the same stability guarantees and that's because the API needs to be stable for instrumentation. However, it also needs to evolve, and this was something that was discussed at the open, Telemetry specification level.

C

There is a pretty strong desire, and- let me just be clear, like go is not the only language that suffers from this um you know C, plus plus, is the other one that kind of rings out. In fact, it's actually worse because they need ABI compatibility.

C

um So, like the the issue, there is, though, that the API needs to still grow and what they consider a backwards compatible way, and that is not necessarily in line with what go provides. You know case in point. If you go, look at the recent Edition, that's being proposed to add a link method to the span, that's something that is in line with the open Telemetry project.

C

But if you want to do it in the go ecosystem traditionally the go uh you know, Community does not consider that a backwards compatible change, because anything that implemented the span interface is no longer implementing the span interface right.

C

So in that sense, we've also looked at it and said that if you're, a implementer of the API you're, an SDK you're, either a vendor or you are um us and so giving us uh that added burden that if you wanted to support a newer version of the API, you need to be stay current uh with what that API defines and allowing that breaking compatibility was something that we as an open.

C

Telemetry Community, as well as the go Community just assumed, was like that's something we're going to need to be able to do and that's why you see all those things documented the way that they are, because we want to be clear up front that that's a possibility of this project.

D

So I see the point. The issue is that even as an SDK writer I can't control the ecosystem as a whole, so like even if I require a higher version to build it. I s like it still could be broken for other people like they could be.

D

um You know relying on a different package like there could be customer. A who has, you know, is importing us and is importing another SDK for some reason, and you know we're requiring a higher version and we implement the higher version, but the other one isn't now their stuff like. They can't do anything essentially um because they can't use ours, but they also like because it would be.

D

um You know, using the new interface if it was the high version that it would break the build um for the other one so like we can't control them. What like other sdks do so that means like every single SDK would then all have to be all unlocks up at all times, which isn't going to be feasible. Necessarily.

D

C

Isn't that feasible, though I.

D

Mean I don't know like there could be an SDK that someone builds and then they just stop maintaining it, which is really normal, and you know there could be people who build.

D

um You know an SDK and then stop for six months and then come back to it.

C

And I agree and I think that I think the users of that SDK. um You know I think they need to be aware. That's the facts, because if a user of the SDK is not something that they can rely on, then maybe they shouldn't be relying on it. I think is the case like implementations of the open, Telemetry, API I think are uh you know, that's why that's why we have it documented as such?

C

Is that like, if you take on that burden, you need to understand that like this is something that you're gonna have to maintain in the long term.

C

Like yeah, I I agree like if you want to go, build your own SDK and then you want to abandon it like that's something you're completely. You know allowed to do, but I would also say like if somebody's gonna depend on it like I I, don't run their project and I don't run their SDK, but I would probably put you know big banner saying, like you know, buyer beware: this is not something that I would recommend using uh because it's not going to be maintained in the long term.

C

You know I do know that, like as a vendor, if you want to support the open slump, shoe project, like uh you know, I'll I, definitely know from like the two vendors I have worked for, like we've taken that commitment seriously like we are like committed to making sure that people are on staff to to follow the API, because open summary as a whole has decided that that's the approach they want to take.

D

Right, I guess it's just: it feels sort of anti-go in a way if that makes any sense like.

C

It is not like it is right yeah, it is, and that's that's why we wanted to make sure that it was documented. As such, um you know, yeah, and the thing that we didn't want to do is is to make it anti-instrumentation I. Think that's that's the key thing from the open Telemetry in um perspective. Is that, like the adoption of the open, slasher API, it needs to be solid um and I.

C

Think that there's there's a a balancing act and the go ecosystem is is very much in the in the case of the downstream, like it's not necessarily like they thought this through and like we're like we're gonna. You know, you know Implement something that is um not supported by open television, but it's it's designed so this uh interface stability guarantees are, are there open, Telemetry has a different purpose and, and it wants to actually evolve in a particular way, and it's not compatible with the go ecosystem uh standard right.

C

That's not to say that I think that, like.

C

I think that, more generally, the ecosystem of SDK developers is exceptionally small compared to the uh ecosystem of instrumentation users and I. Think that I think is where open Telemetry was correct in understanding that and Landing, where they landed, I think is, is appropriate and that's I think why this project has gone forward in adopting the versioning instability young bites. We have.

D

So I guess um I I, understand like we might be in an impasse, but if I might propose one alternative way of sort of meeting both needs is something that we have done uh when we needed to make backwards and compatible changes to interfaces in you know our datadog Tracer API is you can essentially just Define another interface like let's say you have a span interface that has you know five methods.

D

You can define an another span, interface like span with links or something like that, and it embeds the spam interface type and has another function on it and then essentially it can be, for you know whoever is going to be using. That can then do the necessary casting that they can do to see which one it supports, and then, if it supports that they can use that function. If it doesn't, they can just make another Choice essentially and then yeah that does start to bloat the API.

D

But then maybe let's say you have a couple of things that you've done this for maybe you've added two or three methods over six months and you're, like okay, now I think it's time for a major version bomb. Then you do a major version bomb and that pulls in all of the API. You can condense it back down and then you don't have to have all of that code anymore and it keeps it backwards compatible. In case someone wants to build.

D

You know an SDK, a small SDK for some reason that maybe doesn't have regular updates every week or, however often we're going to be changing. Things still allows you to have backwards compatibility without inflating it forever. So it's sort of a balance between like yeah I understand you can't make a. Maybe you can't make a major version bomb every time. I understand, that's a burden, some have done it like I've, seen like grpc I think has had like, or maybe it's redis has like V8 or something at this point. Some do it.

D

If you can't I understand so, then what you can do is, then you know, like I, said, make more interfaces that embed that type and then once you've gotten to a point where it makes sense to expand it. Then you, then you just move to a new version and you just clean it all up um and that's what we're personally doing to maintain it.

C

Yeah and um we've documented that in our uh contributing guidelines, as well as as an option, the the key thing here, though, is the uniformity that open Telemetry specification requires and whether that's something that is allowed I think is also another question.

C

And then it's also a question, like you kind of pointed out, there's the API blue and whether we want to allow that, especially given the fact that we've already defined our versioning stability guarantees to allow evolution in the way that we've uh would update an interface so I think if there's a balancing act here, um I just want to make it clear like we don't want to break that interface.

C

um You know, I, don't think anybody on the call is like we're just going to do this willy-nilly, um but I do think that we want to make sure that we understand that, like you know, if the rest of the open selection Community says like we need to add a ad link method and it needs to be on a span can't be some other interface like go. Go, can't block that and, like you can go. Look at that other conversation we are.

C

We are actively trying to make sure that, like that doesn't happen, but we also have to be willing to accept the fact that it could happen, and if that does uh impact the SDK developers.

C

I think that trade-off is something that we would rather um have happened than affect the instrumentation authors, um and you know I think that's where the balancing act comes in is, if you do add all these interfaces like you're saying the API blue becomes a little bit untenable, and so, if you're, an instrumentation author and you wanted to say start linking things- is the benefit of having the backwards compatibility for sjk authors worth an API author trying to figure out how to add a link to a span when the span is already defined, but there's some sort of other function or interface.

A

Like that's another thing is that you know writing.

C

A function is something that's also considered as well and the conclusion at this point is: we don't have to make that decision. It hasn't come down the pipeline yet, um but when it does, I definitely wanted to make sure that, like it's, it's communicated clearly to the outside environment that we.

A

C

Might update uh interfaces, and so when that happens, I think it's worth as an SDK author, if you're specifically now like you're saying like starting to pick up, implementations of the API understand that in doing so, if you want to provide forward compatibility support, that may be something you need to keep keep up to date with.

D

Okay, it yeah it doesn't seem like there's anything else. I can say at this point, um so I think I'll just have to let it be um I think it's probably just a.

A

Fundamental disagreement.

D

Honestly and I don't know how much more I can say at this point other than what I have um so I can just let it go I just wanted to talk about it. um You know like, like I, said we can make sure that we're up to date as much as possible, but we don't have control. You know.

D

I, don't have control as an SDK author over somebody else's SDK um and I personally think that yeah it could be a bit frustrating for a customer to have to maybe figure out how to make the code changes to support a new interface. But at least that's a workaround. If an SDK is broken, there is no workaround and they're just broken, and- and um so that's that's why I lean that way. But it's you know I, don't maintain this and make those decisions so um I I can't say anything like that.

D

I suppose um I just wanted to make my point.

C

Yeah and I I want to say that, like I hear you um and I I appreciate you raising the issue uh you're, not alone this. This conversation's happened, I think three times now um and all of the points you're raising uh I've I've said myself.

C

um You know, like I, think that there there was a really good reason to not do this, um and so, like I hear you like it's, it's definitely a concern from I think not just myself, but everybody on the call is aware of these things and it's a tough place to be in uh the other side of the conversation I'm trying my best to represent that uh and that's from the open, Telemetry community at the specification level.

C

So um I appreciate you, you just like kind of having that conversation at the end of the day, I do um yeah, I, think I think you're right. Maybe we could just leave it there. I I wish I could give you a better answer. um I guess as well. All I can say.

D

Yeah appreciate taking the time either way thanks yep.

C

A

Realize also like is.

C

There anything else, other people would like to add. I know, I've been kind of maybe uh monopolizing the response there I know. Anthony and uh uh Aaron have been talking a lot about this as well.

B

I will say that we have done this conversation multiple times with very similar or the exact same kind of proposals, each time and the the two key takeaways are. We don't have to deal with that right now, um but it may be something that we do need to deal with in the future, and we are also doing our best to preempt the specification which we are also part of, but also in some ways. We wouldn't do to not make these kind of breaking changes across apis across implementations in other languages.

A

Yeah I would remember to stack of that. I think we're we're all on the same page with what Tyler has said, um and we are trying very hard to make sure that the specification understands the pain that this causes to us and to implementers of the SDK and so that they don't put us in a situation where we have to do this. When there are other options.

C

uh Okay, I know it doesn't take this thing away, but uh yeah we hear you, uh we hear you okay, uh next up on the agenda, uh I wanted to touch base with uh uh Aaron since he's on the call hopefully and go through this next one. uh This deprecation notice for all the v20 I'd like to see if you had any update on this I, know um I posted some some comments here. What's the what's the status here, um it just needs to be merged and uh the tags pushed.

B

Well, I guess address the feedback and then but um the plan is this: Branch would not exist um after the tags are uploaded. So this this is a short-lived Branch. Just to get tags made.

C

uh Okay is the code accessible after you delete the branch, though, uh via the tag.

A

We're interesting tags are identical: they're, just pointers to commits yeah.

C

Okay, but if you, if you delete the branch it doesn't delete, the commits that can are contained in the branch.

A

Not as long as something's referencing it, oh okay,.

C

uh Then I'm excited to see this because we might be able to fix that in the other Branch. We did this on yeah I added a branch protection role for uh the point release meeting for David just a little while ago, Okay cool so yeah, that's the case. Okay, I.

B

Don't know when I will be able. Well, then I will actually have time to do it because I have a pack day today and then travel tomorrow and then off every time. But um I will try and do it before the end of the week. Hopefully.

C

uh Okay, so I guess there isn't a big rush. Yeah.

C

uh But that being said like would it be helpful if uh Anthony or I helped in this or is there's just so much domain knowledge in the path that you just want to keep going with it? No.

B

That's fine! If, um if somebody.

A

B

Can like actually get this merged and then my next step after merging this was literally create those tasks that I listed there and then push them up and then delete the branch. That's that is the three steps from this um I I can put a quick comment before I have to run off, but.

C

B

That sounds good because it looks like Robert has.

C

Some uh minor things as well, and um so we can I think maybe address that uh Anthony do you have Cycles to address that.

A

uh I have some yeah I have some time this afternoon that I could do this, and if we, if we could have it ready for Merch this afternoon, okay.

C

uh Yeah, let's, let's try to do that then, uh because I it you know, I'd like to get it out, but, like I said also it's not like the end of the world: okay, uh okay, cool! Then next up uh uh boy, yeah, maybe you're not going to.

A

C

uh Let's just talk about this really quick, um the centered out metrics, encoder I know that Aaron had provided some sort of work around for the time stamps ignoring the timestamps. Here uh we have a user uh Peter, uh who is, you know, essentially cleaned that up and turn it into a legitimate um uh PR. It just needs more reviews. It's just forever to be on this.

C

So far, uh it wasn't slated for something we needed for the ga, but I think it's ready, um yeah and so I think that other than that Disney's reviews just want to call that out.

C

uh I say that also because it's really good for examples, I think as we move into trying to document our SDK, this would be useful, um but not the end of the world. Okay, cool and then uh next up. I did want to talk about this, so I added this uh PR to use assertable in the attribute sense. This comes from the concept that currently in the metric SDK pipeline we um require attribute, sets to be used to for attributes, and this is specifically because uh in the metrics pipeline, attributes can't be duplicated.

C

uh I think it has a lot to do with the cardinality um and the representation at like the downstream side of things. That being said, so we use sets everywhere uh in the set creation. We actually don't use. Assortable and assortable is something that the set creation uses to define the set uniquely.

C

So it actually will take all the attributes, uh sort them and then do it unique across them, and it does an allocation every single time that you want to do that, and so what I did originally was in the metrics SDK provide a sortable and uh fool those sortables using the sync pool. The thing that I realized, though, is that, like the SDK is not the only place that would need this uh and it would get more.

A

C

If the attributes package itself pulled the sortables so anytime that it doesn't provide one from the user, it could just use one from the pool. This is very similar to like the format package, the fmt package. It uses a pool for a lot of its um formatting, so I think this is like a really positive change.

C

There's a definite benefit to the allocations, uh especially so I mean just kind of walk through I built a little bit of a benchmark here, so the one that I would probably pay attention to is this so for our instrument, implementation and our observable. This is the synchronous, and this is the asynchronous. What this does is. It creates I, think uh three different recordings of a measurement and each time it's going to allocate a sortable, uh and so this is essentially like in a collection cycle for every recording that you do.

C

You have an allocation uh and what this does is it removes that allocation. So this is a real hot path for the metrics SDK. uh So say you have, like you know, 100 uh thousands of uh recordings during a collection cycle. This would reduce that allocation to two. Instead of a thousand, uh so that's a really big reduction on the GC. uh Obviously it's using a pool, so there is going to be a um a slight overhead for this simple case where you have.

C

You know: One recording for a collection cycle which is going to probably be where you have like examples, and that kind of thing which again like is not like it's going to add a small overhead to I, think the memory that's allocated for the pool and the um computation time because it needs you know to synchronize locks but in if you're, using such a simple case, I, don't think that that's going to actually impact it.

C

So so I think here the downside is really negligible compared to the upside of trying to clean up our allocations across. You know, hopefully uh you know a lot of recordings, so I I ended up moving it to the attributes package, and then this uh uses uses that in the um metrics SDK, so yeah I think it's worth taking a look at uh and appreciate the reviews. I think it's some a really positive change. That is not it's only 87 lines of code.

C

So it's not too much to review before we actually get this uh released publicly. So I wanted to call that out.

C

uh Okay, I think with that we've gotten through the entire agenda, uh so.

C

I think everyone had run so I think we're here with us policy, for anybody else is something we want to talk about. That's not on the agenda.

C

Doesn't sound like it, anybody have any cool use cases of open, telemetry.

C

You know who I haven't seen at senior while Steve Steve Harris used to have a bunch of interesting things that I would never think of open telemetry.

A

Yeah rip crypto.

C

Yeah right, uh okay, I think we're in with that, we could probably end it here. We're nine minutes short of the hour. So I think this is probably a good time thanks. Everyone for joining um looking forward to collaborating I think just also a heads up that, like we're in one week of the API being out for the RC, so we've got another week and then probably next week, we'll evaluate our our next steps on that. So just a heads up uh come to the meeting.

C

If you want to know more better than that, we'll see you all asynchronously or uh next week.

A