Open Telemetry Uncategorized, 26 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-01-26 meeting

Description

Instrumentation: Messaging

A

A

A

A

A

A

A

B

Howdy uh wisdom, your mic is on and we're hearing some background noise. Just a heads up.

A

Well, whatever.

C

Hey, can you hear me? Yes, okay, man, I do not know what's up with zoom, but it is super frustrating.

C

um Oh I think I might have figured it out.

C

I don't know, maybe not.

C

Anyways um welcome everyone. I think we could probably get started. uh Just give me a second here, I'm gonna, just uh this given already. uh Please add yourself to the attendees list and if you have something you want to talk about, add it to the agenda.

C

uh Okay, cool uh so to start off, I wanted to talk about a issue that I'm having with the link. Checker I, don't know if anyone else noticed it yet, but uh pretty much like all the check links, jobs are failing and the reason behind that it seems like this uh URL here results. uh It should resolve um for people that were logged in, but her I don't even know if it's going to resolve um or other users, it was not resolving.

C

Although this doesn't look like it's resolving uh I think that just has something to do with my connection.

C

C

Double check that maybe I don't know, maybe other people can resolve that right now, I, don't know uh yeah. That is bizarre. It was resolving right before there's nothing. Computer issues, though so who knows um okay, but I, didn't know what we wanted to do to resolve it, because it's the project board so like this should actually be no problem here. I, don't know why this is being such a, maybe github's down any other people access this.

D

I get a 500.

D

C

D

Click or go through that.

C

Okay Aaron: do you get the same.

E

I can't hear you here: it works for me.

C

E

Working for you, it works.

A

C

B

You hear me now.

C

Yeah yeah, okay,.

B

um uh It's working for me, but I mean uh GitHub may be flaky right now, so.

C

Yeah, okay, there we go looks like uh yeah GitHub. It was taking a little while um yeah, so it resolves for me, but when I opened this in, like a a private browser or something like that, it'll fail.

C

um So I think that has to do with the fact that I'm not logged into GitHub uh when that happens, which is uh not ideal, uh especially given like this is um important to you- know, communicate to the broader audience like our progress. I, don't know if it has something to do with the fact that, like this is like the new projects board and like there is this classic project sport as well. um So I don't really know how to resolve this man. This is slow.

D

Yeah it fails for me in an incognito window, is what I found.

F

Okay, yeah you're able to access it when you're authenticated I wonder if it has to do with the private projects, because all of us who are able to access this have access to those projects. Oh.

C

Okay, yeah like these two down here, which are probably irrelevant at this point,.

B

I mean at this point: it's speculation, because projects aren't associated with a like a a project. A like open, Telemetry go uh It's associated with the uh organization.

C

Well, kind of this page is associated with open phone should go right because it's the projects associated with this project through this tab but I, mean I, know what you're saying.

F

um The visibility into the project I think is controlled by organization membership.

C

Yeah, um let me see if I can remove those Evan. Could you try refreshing, really quick.

C

Let's try the same actually.

D

Still getting save era incognito window for me, but yeah.

C

Okay, yeah me too: okay, uh yes, it wasn't the project being private, so yeah there's something wrong here uh this. This is also private, nope, okay, I.

D

C

uh But this is not a good look to have all of our uh CI systems uh checks failing it's not a required check right now for the links, but I'm just wondering how we want to resolve this.

C

I think this is something that's in um I, don't know where this is I think this is probably our contributing documents or something like that or maybe in a readme I, don't know if we just want to remove this URL there's also this original issue here that this is talking about about that wanted to authenticate um requests to the project using some sort of like uh user. In fact, we might be able to do this with the open, Telemetry bot, which is another issue.

C

um The only question that I have then is like I think that could resolve the issue, but then it also brings up the problem that like, if what is this checking, because is this: if this is checking what the you know, the community experience is going to be. This is for users that maybe aren't logged in or aren't members of the organization. Then maybe we don't want to do that. uh So yeah I was kind of wondering what people's opinions on this issue uh were.

F

I think this should be raised to the TC and have them interface with GitHub or cncf. So cncf can interface with GitHub. Whoever has our relationship there. We ought to have support of some sort.

C

uh What would the TC do, though? They would try to make sure that URL resolves, or they would do, what open.

F

A case with GitHub like if, if these are burning 500s, when unauthenticated users are hitting that's a GitHub problem, you have a lot to know about it. It may be much broader than just us.

C

Yeah, okay, I I thought so as well, um given that the it was like a dramatic change, it happened like kind of overnight um I kind of figured GitHub would have detected it and rolled things back, but okay I'll all right. That's a good point.

F

There have been a number of incidents with GitHub over the last few days, so it may be related to something that they're already aware of and working on, but without opening a sport case. I, don't know that we can know that they know.

C

Okay: uh okay, cool. That seems like a reasonable path forward and then for our CI system in the interim. Do you think that we should just remove that URL or.

B

Is there a way to just Mark that URL as we need to skip like that, could be the short-term solution.

C

That's a good point: um let's see, I think that there is, let me go, take a look at the job.

B

We probably don't need a solve it here, though, like I would suggest us, like. The uh actions are like see if we can skip it and if not then remove like that file from from evaluation until.

C

uh Yeah I mean I, think that's that's fair, um but I think it also just.

D

It works for me when I go to the old, the classic projects page and it works for me on other repos, where they haven't created a new startup project yeah. It's.

C

uh Oh, it looks like we have our Milestone thing in here as well. I wonder if that's a similar issue.

G

C

Been existing for a long time, um okay, I think. That's also a fair thing.

D

Milestone requires you to log in.

C

A

C

The same for the projects- okay, okay,.

A

C

We have a path forward on that I will um there's some action items uh in the I'll try to follow up on those afterwards, uh okay cool, then moving on in the uh agenda next thing I wanted to go over was the release preparation so for the next metrics release, uh the only outstanding PR is this extrema type for the histogram, uh which looks good. There were some questions from Josh that were also raised, but I.

C

They weren't I, responded to him and it's been a while so three weeks, I'm kind of just assuming that they were. Hopefully the responses were valid, looks like Anthony. You just proved this uh uh yeah. It looks like the coverage test is failing: yeah, gosh, okay, but that looks good. um It looks actually I think that should be the last thing in this Milestone that we need to actually accomplish.

C

So that should be done. I think after this meeting, um if I'm not mistaken, is there any disagreement with that.

B

C

Good uh okay, so then the only thing left for the stable release that I have uh is this update the server request to accept the server name so in the process of trying to update contrary by to realize that those server request, um method or function that we have here in all of the semantic conventions is uh useful in in a lot of respects, because it just takes the request, but there's a there's, an issue here where the um uh server name so like there's this idea of this primary server name that is supposed to be generated from the virtual host, like What's, called the primary virtual host and the submit to conventions, and then, if that's not available, uh then it's through the target, URL and then I think the host header, um so there's like three different sources for this, what's called like the net uh host name attribute thing is: is that the server request function never understands what the virtual server name is because that's not included in a standard go request, uh so it needs to be passed in some other way.

C

uh It's not necessarily like required. I, don't think uh to you know it's not an absolute requirement. Specifically. uh That's not what I wanted um and I say that, because you can also have it. uh Let me just kind of go through the working example that I have here. um Yeah I think this yeah. This takes a long time so.

C

Sorry, there's uh there's a lot of changes here which I want to go over as well, but I just wanted to pull one out so in the situation here like where we would actually use service requests right, like you can also specifically add this uh service name, and this is because we know what the service name is. We know that we take it's going to take priorities.

C

This kind of assumes that the attributes that are added to a span are going to use this last in semantics, which isn't actually a valid uh situation all the time. uh So it's really not ideal to try to just assume that, like you know, this is going to set this key here. This net host name key, but you also maybe want to overwrite it with something that you know is going to be the virtual host service name.

C

So for that reason uh wanted to add this server string here. So that's what this PR is doing.

C

uh Just kind of a heads up so I think this is needed before we want to release this, because this would change the API. This uh server request function has been released yet so, hence, there's no change log here as it's changing something that is uh still a work in progress. Yeah. Let's pause here, see what others think.

C

I did want to include this in this release, though. That's one thing that I wanted to make uh possible just because it would change something it would be a breaking change if we release this and then don't after we try to change it after the release. So I wanted to get this change in before that. So I think that this is another uh blocker for the release um that's on here, so that needs reviews.

F

So there are I think changes to the semantic conventions going back to 113., um but these are This. Server request function has been added since the last release to those that's correct, yeah.

C

Yeah, exactly so one 11.2 I think is the current stable release and that doesn't include the v113 cemented conventions, so um everything that's added in the v113. All the way to 117 is going to be in the next release. um So this changes all of it. This server request function was added in the v113. That's where we added a HTTP conf uh uh package as well, so yeah. This. All of these changes that are included in here, which are for all the semantic mentions back to 113, haven't been released yet correct.

C

uh uh Which is I think um a useful thing. We also had this request uh pointer conversion that we just merged in actually going through this process of upgrading semantic conventions in contrib, first to I think look at the ergonomics, as well as the missing pieces. So this is a useful exercise, but I think this is also something to call out that. So we have this update server request that needs reviews.

C

um The contribute updates, I think are also something we want to merge before we have um the go ahead for the release. I think that we could probably create an issue to track a release at this point, given we have a clear line of site with this PR and then these two PRS, but I just wanted to call out that this uh PR for upgrading the semantic conventions here it needs to uh also equip merge. There's some leftover to Do's, based on the existing PR that we just saw.

C

um But it's a you know it's a chunk that I didn't want to include in the release PR, because it does do a lot of upgrading, uh mainly around the semcom package. It represents everything to 117. You start using, it starts using the new HTTP conf in the netconf packages it links to the related PR and the specification for all the changes uh related to the scientific features of HTTP as well as for messaging.

C

This is a big one that also changed, uh namely around certain uh attributes being removed, and um one of the other ones for the HTTP is that the header values need to be in a particular format in the HTTP. The hotel, HTTP Trace package set them explicitly. There may be other issues with the instrumentation that are doing things that the semantic conventions say you should or shouldn't do. I tried my best to go through them all, but there's I don't know like what 30 packages there so I'll be honest.

C

I might have missed them, but I do know that it's not relying on any removed semantic conventions um and it's not, and it's anything that isn't well known like HTTP or messaging change. I was able to address those. So as far as I know, this should be. It should be accurate, but I'm, not 100, guaranteeing that this is perfectly symmetic uh convention compliant. These are also not packages that are stable, so we do find bugs I. Think changing them in the next release is something that's possible.

C

uh Okay and then the only other PR I think uh before our release is this update. Metrics uh used to the new API uh I haven't opened this up last week, I, don't know uh uh yeah I, don't think we should change this in this release. um There's some conflicts here, but other than that Aaron. What's the status of this, can we get an update here.

B

uh I had forgotten about it and it probably shouldn't be a draft anymore and I should just resolve those conflicts. Okay,.

C

uh Okay, cool I can yeah I'll, let you handle that then, but this also needs uh review, uh and we should merge this before we do the release as well, just to make sure that we similar to the semester convention, is catch any sort of ergonomic issues or um compatibility issues as well.

C

Okay, cool um I'll pause there I think that we could I'll see what like do. You think we're ready for this um release, issue that we normally create for our next releases in hotel.

C

I guess the other side of that question is in those PRS that I just showed you does anybody see something that they explicitly think is not going to be merged and that we are going to have conflicts and we need to hold off on uh planning the release.

F

Don't see anything I think I've approved those first, two PR's, um which are the ones that would block the core release, and then we can move on to the contribute. Okay, perfect.

C

uh If that's the case, I'll take the action item here.

C

uh Issue to start tracking the issues um and then once these um shares emerged, then we can start process.

C

Okay, uh that was the tough stuff. The easy ones are here, just a quick call out for open, Telemetry bot. This is more I think for the maintainers than anybody. uh These use a um specific bot for the open television project to uh allow when we create Dependable PRS that we don't have to close them and reopen them.

C

So they kick off the CI system, essentially they're users that have personal access token, which Anthony by the way is at the org level uh based on my reading of this stock here, so I, don't think we have to do anything specific in the repo for this it just uh based on looking at the Go build tools as well repo like they don't do anything specific there, and this worked for them. So this Theory should help yeah solve solve it.

C

Okay, um uh cool uh I, just I, think you're up next, uh for this is sampled to open tracing Bridge.

G

Yeah, so it's a very small feature request, but uh it's been silent for a while, so I figured okay, let's try to figure out whether we uh want to go with this or is you know? Is it going to be discarded so basically, the side of it? A little bit of context is that uh we essentially try to move well.

G

We currently use a open tracing or throughout the code base, and we want to move towards open Telemetry, and you have this open tracing Bridge thingy, which is like a very thin wrapper for the whole open telemetry, uh but in general it works all right. uh There was only one catch in our case, so in some performance critical uh areas I, we would normally do a little bit of hackers, I, think solution.

G

uh We would be casting uh from open tracing API to specific implementation and to figure out whether the span is being sampled so because the bridge hides away everything, except for the actual official API of open tracing.

G

It's basically impossible to do this kind of check, and so what this PR does is just adds is sampled method to that bridge so that we can do you know cash to our interface, which has a sample and continue it's exactly the same kind of API that uh Jagger Jagger, whatever it's pronounced into English, is implementing and also it's implemented in open uh or open Telemetry.

G

uh It's not implemented in open tracing, and you know we have an issue open for them, but I guess it's not ever gonna be a result, because you know it's kind of dead, sorry, uh so yeah. So my question is: basically: is it okay to make this kind of change and merge it or you if you want some kind of changes, I'm, absolutely okay, to make those changes and so on.

C

uh Yeah this looks at a high level, get to me I'd love to know how the people think otherwise, but um I I, apologize. I haven't had time to take a look at this.

C

uh The one thing uh there's there's a few nitpicks just in the quick review of the pr that I'm looking at um is the the big one is going to be documentation so right now the bridge fan context, which is an unexplored type, so I'm guessing it gets returned as an interface um as you've done in the testing like it can be cast this like sample level interface, which is totally valid.

C

I think that's a great way to approach this, but we need to document that I think uh in some way, so that it's not uh just you and me uh that know about like how to do this, um but other than that I mean I. Think this is a. This looks fine to me um outside of uh a PR number in the changelog. As far as I can see, but yeah.

G

uh If you could comment where exactly you want to.

C

Go yeah yeah that.

G

Would be great.

C

I can definitely do that. um Okay, I'll put it on my actual items after this. Anybody else have thoughts on this one.

F

So this is an interface that opened tracing, provides that we're implementing here and we're looking to add a new method to the concrete implementation. That's not on the interface and so everybody who wants to access. This is going to have to cast it to some interface that they create that.

G

Yeah, what other implementations of open tracing would normally do is not to hide away the whole struct. It would be a public struct, so you could catch to a specific implementation struct instead, but I feel good, because it's a bridge, maybe it's better, to keep it hidden or something.

F

F

Tracing expects it to be used, I'm less worried about this, but it feels a bit off to me.

G

Yeah, so, as I mentioned, uh our solution is a little bit heckish, because I think originally, it was not intended to be like exposed in open tracing, but open Telemetry and Jagger, and some other implementations also expose. This is sampled or similar method, uh so I kind of followed that button again, if you prefer to expose this information differently, I'm, also, okay, with it at the bottom line. What essentially we care about is being able to access this information for optimization purposes.

C

So Anthony one thing I heard I I might have miserred you, but I. Don't think that uh this is an interface that opened tracing uh exposes this.

G

Is something does not.

C

Yeah so I don't think, there's any way in open tracing to access the sampled state of the span and.

F

So that's my expectation, but this bridge span context is implementing some interface that open tracing provides, isn't it that doesn't have that is sampled method on it or whatever.

C

I see so your concern Anthony is like in a normal, open tracing world where this bridge is meant to. You know, take something that was working for open tracing and move it to open Telemetry. Why would you ever need to do that? Is that the question Anthony.

F

uh Well, I guess: my question is: do implementations of this other implementations of this open tracing interface normally also provide additional methods that you can access by casting to something else. Is that a common pattern there or.

G

Would this be yes, yeah.

A

Absolutely Norm.

G

Yeah, so the common pattern, at least from what I've seen, is to not have this struct hidden as private. It would be normally a public one. You know capitalized, so in that case you cast to a specific struct implementation rather than in this case it. You would have to write your own interface, which has this sample. Then you would have to get that one. So this part is a little bit, not that common I think uh and it's uh yeah whatever you prefer.

F

Yeah I know I I, think I'm fine, with the casting to an interface to to use this method, um even if every user has to Define their own interface. For this, that's perfectly fine too.

F

They know exactly what they need, um but I I guess, just if the, if the expectation is that open tracing users are going to get something that has an interface and they're going to know that there's an underlying implementation that has more information that they're going to want to get and that they typically go and get then yeah I'm fine with this um yeah I would like to to be pointed at someplace else, where this happens just to to kind of see it in in use.

F

But I think this is a good way of doing that. If that's the way, we have to go.

C

A

And I think you.

C

Think there are Anthony is that they just need to be informed, also where this interface is but yeah I agree. I haven't looked too deep into your issue yet, but hopefully you have a real world use case of how you're using this right. uh It was just.

G

uh Yeah uh I linked to Jagger's a sample method, uh which is what we use internally, uh I'm, not sure if that part of code is actually open sourced on our end, so like the usage of it, but the usage is very basic. It's just casting what token tracing exposes the interface of it to Jaggers implementation.

G

It's just casting just like here.

G

I I, don't think this is a very common use case. I think it's a very, very use case. uh There is an open issue in open tracing, but not a lot of people actually are interested in it.

B

I mean open tracing is being deprecated uh yeah.

G

B

My question with this is like: is this replicating a pattern of how open tracing did sampling or did open tracing, not even touch sampling, and if that's the case, then this information.

G

Was hidden and basically you were expected to use the span as it if it was sampled every time, but it would go into a knob implementation.

C

Yeah I think this is just like a a long-term feature that open tracing was looking to add eventually, but uh given the project, Direction turned into open Telemetry that, like it just I, think this is one of the things that a lot of web tracing folks included in the tracing, API or consumptually. Because of that, but it was never added to open tracing um yeah, I I mean, and that kind of makes sense like because, like in theory, you don't really need it.

C

uh You could just do all these really expensive operations and like it would just throw it away. But it's nice to know that, like hey, if I want to go, determine all these attributes or I want to go determine all these. Like events like, maybe just don't do them, because that span's not going to actually get recorded foreign like just based on what I'm seeing like it already holds this context right, because the bridge fan context has access to the uh is tracing and it doesn't increase the API surface area like directly.

C

It increases it from an you know uh if you know how to cast it kind of type. So, like I, don't know it's it's it's as long as it's documented and it's not just like completely hidden I, think that seems reasonably.

C

Okay, uh cool I think uh yeah I'll try to comment on this and everyone else uh who's on the call. Please um go ahead and comments.

C

Okay, it looks at that. We are done with the listed agenda. I'll pause here see if anybody has something they wanted to also bring up. That's not listed.

C

Okay, uh cool: can anybody have some cool uses of open Telemetry? Past week foreign and David I haven't seen Scott in a while later.

E

No, it's it's planning time in the kubernetes community, so nothing happening right now.

C

Is there any planned work in kubernetes for more Hotel integration.

E

Yeah, so the the cubelets integration right now is super basic and so they're hoping to move that to Beta next release and right now only the container runtime interface is instrumented, which is a grpc interface to talk to container runtimes, but there's obviously- or if you don't know, there's a bajillion little grpc or HTTP servers that are part of the cubelet and So.

E

The plan is to instrument most, if not all, of them and actually tie all of those together right now, the like it's not correctly plumbed with context, so you don't actually get them linked together, but there's plans to make that happen. So should.

C

E

C

Yeah I think it's really useful. Okay, like once you have that feature, there's uh some product opportunities to get some observability to coup uh kubernetes. There.

C

uh Cool yeah- that's really interesting.

C

uh Anybody else get some perspective news for open television.

C

Okay, well, if not as uh we could have seen in this meeting, there's a lot of work to do uh so, I think getting some time back is always useful.

C

Okay, I think with that, then we can probably edit here uh thanks everybody for joining we'll see you all uh next time saying play same time or async or slip bye.