Open Telemetry Uncategorized, 2 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-02-02 meeting

Description

Instrumentation: Messaging

A

A

A

Hey Tyler Anthony, it's going well.

B

David, are you going to um kubecon to you.

A

I will not be I'll be out on uh on leave for a few months, probably starting in March.

B

Oh okay, can I get you all my PR's before then.

B

um Well, yeah, okay, cool good to know.

B

How about you, uh Anthony or Mike.

C

No I'm not going.

D

C

This one, unfortunately yeah.

D

Yeah I've submitted a proposal to talk about the Target allocator and the operator for scaling promises collection. So if that's accepted I'll be going whether it was not clear, probably not given several concerns right now, yeah.

B

Yeah I had a proposal as well, but uh I haven't heard back either I'm kind of the same boat like if it gets accepted, I'll I'll probably go I'm, not really excited to fly all the way out there, but um yeah.

D

Yeah, especially from the West Coast, it's probably kind of rough yeah I'll.

B

Probably have to fly to your neck of the woods first and then go all the way over there. Yeah.

D

Yeah, it would likely do the same thing: I did last year, just drive up to Toronto and then I think I can get a direct flight from Toronto to Amsterdam, as opposed to flying out of Buffalo, where I'd have to two or three Hops at minimum.

B

Oh yeah, that's a great idea. He's got to do the border crossing.

D

Yeah, you know I'm going up to Pearson's First Flight, okay, go ahead, whatever yeah.

B

B

uh Cool well I Aaron had messaged uh earlier saying he's without power today, so he's probably not gonna make it. So, hopefully things are going well for him. uh It's not a great position to be in, but we could probably just jump in here. I know: there's not too much on the agenda right now. Foreign sharing. My screen, though.

B

uh Yeah so I mean, if you do have anything you want to talk about. Please add here as well also, don't forget to add yourself to the attendees list, uh so it started off. I guess maybe also just mention uh in case it wasn't seen uh I guess it never works. um We did a release uh four days ago for the new API and symmetric SDK uh stuff.

B

It was actually like just over a month, there's a lot of stuff in this, uh including the specification or the fancy confession stuff. So it's a pretty big I think release Milestone.

B

So um with that said, I think that I wanted to kind of touch base on like what the next steps were for the metric signal, because I think that's one of our big um pushes going forward uh in the time that we've released the you know within the past four days, uh we got two issues, I think related to feedback on the um uh metric API. One of the questions here was around the synchronous instruments. Accepting a context. It's um I think a fair question, but one of the things that, like you know, is like.

B

Why are we actually doing this and like currently we're just doing it to see if there's some sort of like deadline that we exceeded, which isn't really useful but um okay, I? Guess it's in this issue as well like uh one of the things that, like we had always kind of talked about, is correlating traces uh with synchronous instruments. So you need to have a context uh passed to be able to do that.

B

I, don't know if that's still a thing would plan to do in open, Telemetry I, don't know if any other languages are actually does anybody know photo languages are doing that, isn't.

D

It required for examplars, you have to be able to correlate a trace ID.

A

That was my thinking as well.

B

uh Okay, so wait so then for exemplars, you can't that's not going to work for a cigarettes instruments I'm not that familiar with the Exemplar spec. Yet.

D

That's correct if they synchronous instruments operate outside of a context within which you could obtain an example, an example spam okay,.

A

Does the observed method not have a context I.

B

Guess it's not anymore yeah.

D

um Even if it did, the context would be um relevant to the reader that is observing the Callback, not any of the observations or the any of the things that resulted in changes in the value.

A

Right right, of course, like if I were calling some external service to fetch these values asynchronously um it. You could have traces about that. I suppose um that you got exemplars for, but it wouldn't really be anything to do with the underlying data. Yeah.

B

Yeah, uh that's a good point, though, is yeah. Okay,.

D

Yeah, this is the same issue that we have with logs and why tigraine has been pushing very hard on the yeslog proposal. That context be.

B

Included everywhere, yeah, which isn't really gonna happen. um Unfortunately, uh okay, I think, if that's the case, I think that's gonna just answer that question like we're going to need those for the singers instruments, so this could probably get closed.

B

um One of the other feedbacks was bugging was looking at the registration API and one of the things he noticed is um one of some suggestions on like what the distinction between like a single instrument. Callback like this with in 64 callback and the registration, a registered callback on the meter itself should be called. uh I was kind of clarified. One of the things he pointed out is in Java for the single instrument.

B

Callbacks, there's a way to close uh the instrument, so that essentially means that the Callback becomes uh unregistered um yeah, remove the registered callback via uh it just essentially and registers it, which is um suggestion he added that we might want to add as well. So our is our observable instruments would have a closed method on them.

B

I I'm a little bit I've been thinking about it. I've been thinking about it for two days now: I'm I'm on the fence on this one I think I'd want to have clarification from the specification about this one, uh because I mean we specifically didn't allow the registration uh from single instruments to be removed. Given the API statement that uh API support creation of asynchronous instruments have passing zero, Mark callback functions to be permanently registered uh to the newly created instrument, and so the permanently is like I mean I.

B

I think that it is clear that permanently means that it shouldn't be like the instrument Still Remains valid, but the thing's unregistered. But then there's like this close operation that bognan has where it's like it seems like. That's all it's doing is just unregistering it, um because I also think that this note means that it's not like closing any other callbacks from like our multi uh instrument. So it's it's really.

B

Only the disclosure method is only on registering this one callback, so like I I, think that I, don't know like it seems I actually think it seems kind of useful like if you know this instrument is not can be used, then to remove it, but it also like it's it's a little I'm, hesitant to add it. If it would contradict the specification, I see an X from Anthony I, don't know. If you want to comment.

D

Yeah I mean the the spec seems pretty clear that it is permanently registered like if that instrument still exists and the Callback is no longer you could say yeah it's registered and it's been in a state where it's not going to be called, but that doesn't seem to be what the spec is saying there.

D

um I I would be very wary of adding this and I suspect. Java has made a mistake here.

A

I agree for what it's worth I think users are free to do nothing in their callbacks if they want, but I don't know. If we need to provide a mechanism to like explicitly unregister it.

D

If they want to make their callback set up in such a way that they can later change a value that it will look at, that makes it short-circuit and not do any work great go for it.

B

Yeah I was thinking that as well right um sure, there's going to be this function on the call stack, but I think that then they need to be judicious about that, and I also think that, like if they really care about that function on the call stack like uh just use the register callback method like because then you can unregister it. We built specific functionality there around.

B

It I think boggan's asking us, because it would uh be ergonomically easier to just do like a single instrument, but I'm also really hesitant about the um specification compliance here. Given how much we just went through trying to be compliant um trying to not take us out of compliance.

A

It's also can be done additive right, so we we could decide to punt on this. If we're not sure yeah.

B

And I kind of mentioned that and uh his point is that then you would still need to do some sort of um uh interface assertion type assertion there uh to you know say that it implements the closer which uh it does disrupt like this ergonomic reason, but I mean overall like I, don't know, I, don't know if it's a strong enough reason to say we need to do it now.

D

Do they need to check whether it implements the closer interface? If they've depended on the API at a particular version of the SDK at a particular version, then they will know whether that method had been added to those types or not? That's why I have minimum version resolution.

B

I see what you're saying so like yeah. If we added this to the API at a later point, then it wouldn't need to do that right. Right.

D

So we added in you, know 1.25. Then their dependency is on 1.25 and they can. They can know that go mod is not going to resolve a version lower than that, so they will have the feature.

B

Well, yeah, but then that also gets back to like we would be adding a method to an interface of the API, which is you know, our version and compliance uh issue that we've talked about many many times, but I I agree like I. Don't think that we need to add this at this point um else. I'm also hesitant, but I would love it if uh Anthony and David.

B

If you could comment on this issue and just kind of uh voice that opinion as well, because I wanted to hear more from the community on this one as well.

B

B

um But other than that I think, let's go back to here.

B

uh Looking at the metrics API GA project board that we have um 38 done issues this um these two we just talked about- but this uh because I think the only leftover holdout from the specification- and this is at our new OP instrumentation or no implementation- should be logging, uh which we've kind of all agreed that isn't. I opened this specification PR just before this meeting uh and it's essentially defining the no op implementation at the specification level saying don't log anything don't perform operations, so it should resolve this so that this issue is closed.

B

um You know it's kind of a bigger PR, so I I don't know how quick this is going to get uh implemented, but yeah. So I I think that just kind of heads up on that one, like there's still I, think one thing that we need to clarify at the specification level that we said once it's resolved this, and this should be closed. If I'm not mistaken, this was kind of a honker of it.

B

Yes, because it does address this issue saying that metric apis are not supposed to validate names, and then it will close this so yeah. When that's done, these two should be done, and then we just need tce review, um given we're going to address these other two for the metrics API as far as I can see, so it might be worth opening this issue for the um TC review or the API at this point. What do others think about that?.

B

Thumbs up from David I think we're good.

D

um Okay, as long as it's not bugged and being assigned and telling us that we need to stop registering callbacks in order to be compliant.

B

uh I yeah I mean I hope that it's not something like that, because I think that's got to put us in a little bit of a quiet wire but we'll see I'll open up the issue. I think.

B

Yeah, okay, uh cool, so that's the metrics ga uh for the Beta release of the SDK I wanted to kind of go over this. Similarly, these are kind of related, so they're here I think these uh sorry, this one we just talked about the duplicate register callback from uh suggestion from Bogdan. This would be related to the close. This could also be just implemented. The SDK, if we didn't want to pollute the API but again like I, think that's for discussion on itself.

B

There's a counter metric named duplicate, total I think this is a whole other bug from the previous release uh given somewhere yeah. Given this I think this has actually been given this. It's been addressed a while ago, so I just wanted to verify that it's looking for a response. This is the only other thing outstanding for the metrics uh SDK I air is not here, but I kind of get the impression that he's working on that. uh Given yes, this is opened.

B

This ad Benchmark of the histogram allocations, I think I'm just going to assign this to him and think about assignments. They can change really quickly. So I think that we have somebody actually work on I. Think the last issue. This actually looks uh ready to merge, so I'm just gonna merge it um unless there's outstanding, no yeah, that's excited emerge.

B

uh Cool all right so other than that the SDK development, there's I, think nothing really standing in our way for the beta.

B

So the outstanding question I think that is kind of uh being unasked. Here. Is this uh message sdkga task?

B

We, you know I, think if we're working towards a release candidate for the API, like the TC review, is going to be needed or maybe not like I I'm wondering what other people think like. Should we cut a release candidate for the metrics API and then have the TC review that or what what do we want to do uh in this process or what others see as the the process here.

D

I think we should probably do the review before making the release candidate. My ideal release candidate would be one where, if nobody comes up with any issues, we just change the version releases 1.x instead of RC.

B

Okay, yeah: that's it's fair!.

B

uh Okay, so if that's the case and I think that we have a path board for the API uh somewhere here, um you know: we've got issues, we've got um sign off, I think there's a line of sight, I think for then we need the SDK GA development.

B

So all of these uh draft topics in this backlog here this project essentially needs to get broken out. I'm happy to do this as well um looks like. We also have some things in here. We may need also a new project called the vegetastic post GA, just to get a placeholder for all the tasks that we kind of want to punt on until after a ga, uh so I'll add that as well.

B

B

Other than that, I think that.

B

I think that that's about it for getting a lot of sight on the uh the ga right.

D

Well, one of the issues of the jayboard was Prometheus. Exporter needs. uh Open. Metrics support is that needed for GA.

D

Second, one this one here.

A

uh Maybe I don't actually know if that's necessarily within our control.

A

A

I think that might end up being an option. That's passed through our exporter to the underlying library, to tell it how to serialize.

B

uh The Prometheus Library, you mean David right, okay, uh yeah with regards to it being in the GI I. Don't actually think it needs to be right like there's nothing. A specification says that, like an SDK needs to be able to export open, metrics right. Yes,.

A

That's correct, like I, don't think Prometheus exporters are required as part of a ga release. Okay,.

B

So yeah this could probably get go ahead. Anthony.

D

Let's just say it seems to be proposing adding an option um to the exporter, which we should be able to add at any point after ga yep.

B

uh Yeah, okay, so I will.

B

I'll do that as well um Escape integration, testing framework- this is another one, so I definitely know that this proposed structure for the placeholders for the docs like this is a good one to I think have ready for GA, given we're going to have a need for docs after the ga uh we've talked about this for a while. Is there any reason we would want to keep or move this after ga, I I think we need this 4ga unless there's conflicting thought here.

D

B

uh The integration testing framework, though that's the one where I'm a little bit more like I, don't.

B

So I've looked at this a few times, so the the metric test package, like it comprised of two different parts. One of them was like testing utilities to to essentially get readout when you wanted to like do like um some sort of SDK testing based uh uh testing, but we have the manual uh reader now and that, like works, really well in conjunction with the metric data test package to validate any sort of, you know, output that you would expect from instrumentation. So I, don't think that we need that component of the best success package.

B

But the thing that it also had before was like this end-to-end testing of the SDK. So, essentially, any of the functionality investigation have it tested it.

B

I, don't know if this is like really, this I think was a bit aspirational, um given the fact that we have like a lot of integration testing for like the meters and like the readers and the produced metrics that you come out with like what's pretty close, it's just not generic right, like you, can't take a a testing framework and then apply it to you know some other SDK uh because there's no, like you know, functionality there, but I, don't know if that's really needed, especially I, don't think it's needed for GA, so I'm kind of wondering if this issue should just be closed.

D

Didn't we move away from that sort of model with traces like we used to have a trace SDK test package that like tried to set up a harness around the SDK and put it through spaces, and we could have moved away from that? Didn't we.

B

uh I I'd say we moved away from it, but uh if you're into archeology it's still there, um we still have it in the in the repo. um So whatever it was existing is still like a part of the the framework I think it's I think it's internal though, um if I remember correctly so I, you know we could try to do that here, but I I, honestly, I, don't know if it's kind of like what you said Anthony like we moved away from it. I don't know if it's worth spending development effort on that.

B

uh Okay, I I opened this issue. I will try to I. Think I want to close it. um If there's any opposition, there's a lot of people that thumbed up it, uh but that's also really old, so I don't know. I will read back into this, so this might just get closed um other than that. I think that all the other stuff just needs to be uh broken out into actual issues that we did similar to the metrics API.

B

Okay, uh I think with that I think I'm through all the things we wanted to talk about for the metric signal. You could pause here so there's something else. People wanted to discuss it's not on the agenda.

B

Okay, um all right, then I guess we could also just talk about if anybody's used open Telemetry uh in the past week, some cool ideas, little projects and use cases.

C

That's something that uh it's been going on for a little while, but I, one of the maintainers for the descheduler project and kubernetes, and someone came in a couple weeks ago, asking to add Hotel tracing to it. So then reviewing some changes from them and trying to add a some basic traces and kind of update our metrics and stuff to get that. Well, so it's pretty cool to see in the wild.

B

Yeah, that is so it's for the D scheduler or is it the scheduler yeah.

C

The the D scheduler, it's a sub project, the schedule, maybe someday we'll get it in the scheduling too.

B

Yeah I mean the more we can find it in the ecosystem, I'm guessing you're, coordinating with David on that one.

C

I got some feedback from them. Okay, I had some questions on the PR but um yeah. He pointed me in the right direction on a couple things.

A

B

Cool, how is it so? Maybe this is a question for you and David like how do you actually set up like your uh export pipeline for the instrumentation, or is that just something that, like.

C

B

Guessing like with the cube API right like it's running on its own platform, how do you like tell it where to send those traces.

A

C

A

Control plane is.

C

Living somewhere.

A

D

A

C

A

I think it's not particularly straightforward. It's a lot more straightforward. If you're just doing YouTube cluster, we still have it without.

C

All the details of how we're gonna, actually.

A

Use them or provide it as part of like the managed service.

B

Okay, yeah, it's just more plugging the instrumentation right now and then manually getting somewhere set.

A

Up, but we both all the instrumentation I know of, is currently in Alpha, and so it's not it's actually kind of hard to even turn on and uh GK cluster, and also the because of the hotel HTTP stuff. From last release. It's uh it'll break any port forwarding you try and do that so yeah, it's okay.

C

So in my use case, it's a little more straightforward, because this is it's not a baked in control, plane component, so we're pretty much just wiring up the SDK as it exists.

B

um Oh okay, yeah, that's pretty straightforward!.

B

Yeah David that that comments about HTTP, recording breaking is I, think there's a lot of questions in the specification right now about like breaking more HTTP semantic conventions and it's like that's.

A

Frustrating it's not a convention that uh it was. This was the like. Http has a special constant called nobody.

B

Oh yeah, okay,.

A

Empty body this was like a one-line fix that I submitted a while ago, but it turns out that, like some weird library that we use or special code that we wrote for port forwarding relies on it somewhere. Yeah.

B

uh Yeah I think I know what you're talking about I think this has to do with like our HTTP Snoop stuff. Maybe in that HTTP Library, yeah, uh okay, all right, yeah I think it might actually be an open issue. That's active right now on that. One too.

A

So I may actually ask for a cherry pick of that into other releases, because Upstream Cube can't update to the latest otel go releases because of and I have myself or at least Google to blame. uh There's some like package refactor, that's going on with the Google apis.

A

um It's one of those where everyone has to update all of your dependencies have to update past a certain point before you can and of course, Cube Imports, the whole world, so they're just waiting, yeah wow.

B

Okay, uh I mean I'm not opposed to backboarding, like a patch to a previous yeah I.

A

B

It up but okay, but.

A

B

I think it'd be a one-off, like I, don't know how we we'd have to essentially create a branch to tag or something like that, but yeah.

A

Do we not have release branches or is there just the main branch with tags? Okay,.

B

Just main branch of tags, but I mean we've also talked about it. You.

D

Can create a bridge from a tag at any point, though yeah right, yeah.

C

B

Yeah and it's something we should probably work on I know Aaron's, also looking at deprecating a bunch of things that we have abandoned, so that would also be a similar pattern.

B

uh But yeah, if when it becomes an issue for it, please create PR or an issue for it too. We'll do thanks.

B

Okay, anything else. People want to talk about.

B

Awesome uh well, we could probably set it here. Then thanks everyone for joining. uh If you have more comments or questions, we'll see you all uh in slack or in the issues, asynchronously otherwise uh same place same time next week, very much.