Open Telemetry Uncategorized, 6 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-02-06 meeting

Description

Open cncf-opentelemetry-meeting-3@cncf.io's Personal Meeting Room

A

B

A

C

How's it going.

D

B

And Josh, both of those tips are yours. We were just doing um some triage uh and I was curious if, if we needed these still or what.

C

Yeah, this is about resource stability. um I have this.

C

uh Those are both failed out depths because uh of various reasons, um but I think the the main question I had yeah anyway we'll talk about that, because if you look um further down or no I, have it right here stabilizing resource semantic inventions is like the thing that we decided to focus on HTTP but I feel like we can stabilize HTTP, but if we don't stabilize resource at the same time, maybe the impact is limited, so I do think we need to, uh and specifically me, because you see my failed oteps pick that back up and start driving to consensus there, but that's something I'm planning to to do.

C

I just wanted to um talk about it today. I have specifically I think there's a few things we can look at, um but we can start walking through the agenda. I find I added attendees back in, if folks want to add their name um and we'll get started in like two minutes or so.

A

B

B

Apologies I.

A

B

Stay for 30 minutes, I have a conflict this week.

A

C

um Do we have anything that I think is process related? We should the only thing, maybe we'll skip to one of the process. Topics. First, then, if you only have 30 minutes, then we'll come back um sound, reasonable, that's good! Okay,.

C

uh All right we'll give another 10 seconds and we'll get started. If anyone has anything else to another agenda, feel free.

C

Okay, so um let's go through process topics first, since um uh Ted has a conflict so we'll time box us to 20 minutes um which puts us at about 25 after the hour, um all right. So first off this was added um to the meeting notes. Previously it was added about a week ago from Daniel. um This is a request to include request headers in Span attributes for http, with a proposal for how to do that in go um and I think this might be a very complicated request.

C

uh Trask, do you want to take it from there, because you added a comment.

D

Yeah I think so we already have um semcon for capturing requests. Headers I think this issue is about configuration of those like. How would you specify that you want to capture specific request? Headers and um Java has a environment variable system property that we use already I. Think python does, um but the request here was from go was to make sure that you know to spec that unify that configuration property. Somehow, so that's why I was proposing um sending it to the config file working group.

C

What what exactly do we say in our semconf about these particular attributes.

D

uh If you can pull up the um the HTTP tracing.

C

Working on it, uh oh I wasn't sharing the right tab for a while, sorry, all right so Trace, some kind of.

E

D

And here I got it: oh Armin beat me to it, he's dropped the link, but if you search request dot here on this page or go to armin's link.

F

C

uh Unfortunately, with my computer doing request, dots gotta be a little easier. You.

D

C

These are, these are the custom ones right.

C

And it says instrumentation should require explicit configuration which ones are captured right.

C

So I completely agree with you. We should punt this over to uh probably the configuration Sig at this point now I the the process question I had here myself was: can we just specify that things have to be done by other things like that uh I think so?.

D

B

D

Yeah go ahead. Oh.

B

uh No, you go first risk.

D

uh I, don't think we're saying that they have to do it. I think we're just saying that if that's, that is where this question belongs currently yeah or or I I, guess that is a question for their scope. Also, you know I they're, currently just scoping the I guess their scope is currently just format, not specific environment variables, but we currently have a moratorium.

A

D

Specific variables until that config format is done, yeah.

C

um And so I think it's two there's two compounding things here. One is this is a um there's, a security risk to not have this as a configuration thing um and some languages already had configuration that does it so asking go to do it. It's like. Well, you should because the other languages do it. So we need to be consistent, but we have that moratorium.

C

um Okay, it no matter what I think you're absolutely right. This should go to the config group. um I will make a note. Is.

A

Daniel in the call.

D

I, don't see him no.

C

Okay, um I'll do that off camera uh uh so to redirect to some kind of.

C

Correct to configuration.

A

C

C

Okay sounds good uh next one: how.

F

Can I ask one question in that regard: do we have tea decided to stabilize all of the HTTP semantic conventions at once, or would it be okay to say, request and response? Headers are more convoluted security risk and whatnot, and we would leave these out of scope for now and stabilize the rest first.

F

D

A

Think we need to.

D

Have that flexibility to um Mark to make that decision? For example, on the metric side, um we very likely may only want to stabilize a couple. You know the duration metrics initially.

B

C

So to answer your question: Armin I, don't think we're going to take everything we have in HTTP and stabilize it. I think, there's going to be like a set of things that stabilize and there might be things that we drop, um but this particular if you're asking specifically about this attribute thing, it's a should request and I I, don't think we're going to change like. Are there plans to change how that works for including attributes from their quests in the um Trace like I, don't see, that's changing at all that marketing experimental would be worthwhile.

D

C

The the name might change.

F

uh Because it follows the the pattern of including the in the name right or why are you? Why are you saying that trust.

D

Oh um only just from a prioritization um uh perspective, I I'd like to keep, uh we may uh I'm not saying we would, but just we may want to reduce scope of stabilization.

D

If we find that there's questions like that, like on the ECS side, They Don't Really, they don't really have that concept of the keys being um Dynamic, um so I actually had found in their repo the ECS repo that they had rejected sort of a similar semantic convention attribute because if they didn't have the ability to have Dynamic keys.

G

I have a broader question about this: the headers are optional. Attributes optional means that there is no contract, that they are there and nobody should expect them. uh Also they are Dynamic. Is it something that is even a subject of stabilization.

C

uh Yeah, the the stable thing would be um so there's two questions that I think are important in what you're saying one is: is open, Telemetry instrumentation allowed to produce that attribute in advance? That's.

A

The one I'm more worried.

C

About here is I would like, like I, think it's useful to to have these when users want them, and it's a thing that users are going to pinpoint and Target kind of a thing, so I'd want open, Telemetry instrumentation to be able to create them. So that's, that's part. One of the question part two is: um is it important for us to like? Can we actually declare something? That's Dynamic is stable.

C

The only bit we're declaring is, if you put one of these Dynamic things in here's, the prefix where it belongs, so it doesn't conflict with anything else. Right and I.

C

Think it's fair for us to for, for that second thing to declare this prefix is stable and it gives you a place to put all these custom things that are useful to you and it also answers the first question of can open Telemetry instrumentation produce this without violating the spec right, so that, to that extent, I think it's useful if you think open Telemetry instrumentation can produce these attributes without violating any spec or anything without us explicitly calling it out. We could go that route as well.

G

I think my question is: what is the benefit like? What um what do we guarantee that the namespace would not change, but even if it would like, you cannot build basically any experience. Well, maybe some esoteric experience for HTTP, specifically for headers.

C

It it gives us an area, we can't specify attributes. So we know that we won't conflict with the custom ones.

G

I see okay, yeah makes sense.

C

Yeah, it's not a it's, not huge um like it's it's it's not that big of a deal like if you wanted to cut scope for it. I think that's! Okay, but um if users want this- and this is the thing they've asked for- and we have no way to do it from open, Telemetry, instrumentation I think that's problematic.

G

Yeah yeah, it makes sense well I, don't believe that stabilizing same space would be, would increase the scope of this conversation, maybe more than it is.

C

Yeah and anyone else have any other thoughts there.

C

So I think, um basically related to the specific current semconf will defer to the HTTP group um what they want to include in scope with their initial, um with your initial stability things and like you can decide if it's worth it to Define this or not like that. You know um as long as you understand like what what the pros and cons are.

C

One thing I am concerned about is: there is a hesitancy to create instrumentation lessons in the semantic conventions, and so, if we, if something is not specified, it's almost like it won't get implemented right now. That's that's one concern that I have in general around hotel that we need to be careful of. So that's the only thing I'll caveat with, um but I think with that said, like um the HTTP 7 count can make a uh you know, decide whether that's going to be included instability. It's a good question.

C

um We'll kick this over to the configuration working group and let's move on to the next, uh the next topic, which is actually somewhat important, and this is how to allow experimental experimentation, um there's an Otep that um so Daniel, um daila and I were working on this. A little bit is he on the.

A

C

No, um we were trying to come up with a way that we can specify um kind of experimental semantic inventions so that instrumentation can Implement them ahead of them being formally specified uh so that we can have open Telemetry instrumentation that you know has all the things that open Telemetry once that that is considered greenlit and okay and that people can try stuff out.

C

um But when we looked into the details effectively, we're taking some references here from like HTTP um or w3c um conventions and uh using experimental qualifiers on attributes uh in HTTP was an abject failure uh and for various reasons.

C

So we don't want to continue like to basically make that same mistake. So, if we're going to do something that needs to be slightly different, this is a proposal of what's slightly different means.

C

um I just dumped a diatribe of my own onto the document of where my current thinking is, um but I wanted to see. If folks in this meeting have a chance to look at this before the meeting, otherwise look at it we'll discuss it next meeting.

G

I did I left my comments, my most her I think the major part is that that it's very specific to attributes and it won't scale Beyond attributes, because the semantic convention, the unit, is not attributed as like a scenario and the only way I see to evolve with this versioning somewhere for specific semantic convention.

A

C

I I saw your comments, I think that's a good call out um and and the the notion to limit the scope of this I guess the question here going forward is uh we need to think about how we Define the next set of semantic conventions and whether or not we want to do it only through expert committees like we're doing with HTTP or if we want to enable kind of organic long tail growth. That's that's kind of my thinking. Right now um and I see.

C

This proposal is enabling long tail growth, where we kind of open up the the uh the registry where people can just Reserve new names Define their instrumentation. Once it gets enough adoption it becomes stabilized.

C

um There is a bar to be included in the registry. The bar is um meant to not be subjective, but I think still is. If you read the the details, um my my current thinking, based on like seeing this in practice and seeing your comments of villa, is that we don't need this process right now.

C

um Specifically, I think we need like we're using expert committees to get a corpus of kind of core semantic conventions done um once that work is done going after long tail makes sense. I think we need to solve the problem of. How does the next committee go about building semantic conventions um with HTTP? We have a bunch of implementations already. What does that look like going forward for where we don't have instrumentation.

G

This doesn't make, but it makes sense for other specs. For example, we are working on messaging, uh we have messaging working group and there we have a complicated process. We create all tabs uh it's hard for us to keep track of what the target state of what we want to achieve right.

G

um But, more importantly, we have extensions for databases for messaging for what not, and somebody defines protocols and people who do this- they don't benefit from this things being in the spec repo. The aprs die right or it takes enormous amount of effort to emerge. So, regardless of the stabilization, um it's a separate effort, I think we probably would be nice to create a better place for people to evolve their sandbox semantic conventions.

C

Yeah, do you think this has to be the same kind of a registry where they evolve Okay so right now, the way this works is once you've claimed a name you own it forever for all time, because in practice that's kind of an inevitability once you've claimed a name and used it for anything it. It's like licking a cookie like no one else can take that cooking anymore, because we don't know if if you're gone, diet disappeared, whatever it's it's just there forever and that's kind of what the w3c group has done.

C

Right, um I think there's an interim process available, specifically for like messaging, where we make an area where you can Define all these things and have them stored somewhere. That's not official! Yet right where you haven't licked the cookie you haven't committed to any of them and then eventually uh you can propose all of them kind of wholesale and get them in right.

C

I. Think there's there's room for us to make a process where that happens. um That is and then I think after we've gotten kind of the core things done with that other process, then this kind of cookie licking process makes sense for long tail.

C

um That's my that's my thinking but like let me know like does that sound reasonable? Is that do you think we can make it work.

G

Yeah I think it sounds reasonable. It's not the burning need, um but when we introduce this procedure for experimental extensions or experimental semantic conventions, we need to like think how the whole convention will work and if we apply it to poorer things right so, for example, we create HTTP and RC like one point or c one right, uh then uh the fact that it means we're stabilizing the HTTP only and general part of General.

G

Probably uh if we introduce an experimental thing, uh it means that we need versioning for the whole semantic convention right so um and then the evolution looks like the evolution of any other software thing.

G

um So my only point that I think is burning now is that we cannot Define experimental attributes. We need version the whole semantic convention.

G

A

The word generator.

B

I, you know, um but the only thing I'd like to add to this is I think this is all actually like a really good idea to create. You know a different process for having a registry and adding things to it.

B

uh I actually, don't think they should be experimental, though I think going forwards like if you propose to add a convention, that's like totally cool, but that namespace once it goes in you know any future changes to it have to be made using um uh using some kind of schema versioning, and if you want something, that's just like totally not compatible with that, then maybe that should just be a new entry in the registry rather than mutating the one that's already in there does, that does that make sense.

H

uh Not quite for me at least uh mostly because so you know we do have schema versioning. uh How does that relate to the semantic convention? Are we just saying that, like maybe like uh the association between a given attribute and a schema, is the mapping kind of thing and that's how you version it uh and then, like it's just kind of like a static like you know, Universal set for things in cementic convention, or are you saying that the it would somehow be able to use schema versioning to actually version a particular?

H

You know basically, Keys you know names and the semantic dimensions.

B

uh Yeah, so we we have like a a schema versioning uh methodology in The Collector, where you know that, um like the spec gets released, it has a version number. So you know the version of that uh schema for that semantic convention in that spec um and you can uh in The Collector convert it to uh another version of that schema and you can't make changes to um semantic conventions that can't be converted in that manner.

B

Once they're stable, that's, like kind of our definition of stable is like if you want to change that convention after that, there has to be a way to to translate it um at least translate it backwards.

H

um And this will be true for some I I still don't see how that uh how we can use the existing mechanism to the schema to version um semantic conventions. We have schema versioning I, don't know how that directly translates to semantic conventions. So.

C

The version of the semantic convention is the version of the schema, and the version of the scheme is the version of the spec right now yeah, um which is to say we're planning to never break compatibility and we're planning to never.

A

H

What I was going to say? I mean, like you know, in contrib right now, there's a bunch of stuff that has zero relation to the stuff we put into semantic conventions with the specs, so I mean I.

B

H

Love for the state to be there, yeah I agree.

B

um Yeah that thing we that we have to do a lift as part of the semantic convention work to like actually go through the stuff we have in contrib and like lift it up onto a proper version of of the schema. So we need to do that as part of stabilizing the stuff that we have. That's you're 100 correct there. How.

H

Do uh how do we deal with this in both contrib and custom distributions? Like uh so? Is semantic convention intended to always be a global view that is shared everywhere across the world or will ever be able like? Let's say that, like you know, like uh Huli, the new company comes up here, wants to implement their own hotel collector stuff like? Can they have overrides or a domain specific semantic convention for their stuff? uh Or would it.

C

Only they can ad I mean like they could. They could uh have their own set of conventions that they apply with, like a schema, URL, okay for Telemetry that they make. But if they say this is open Telemetry, they have to abide by the send conf that we've provided. So if they say this is open, Telemetry compliant instrumentation. That means the labels and things have to abide by one of the semconf releases right.

H

So if they're, making their Custom, Distribution or they're just adding to contrib it's just still, they should still make changes if necessary to sorry. If they make any change to a semantic convention, then it would be necessary to change the uh the global Hotel specification. You know this that the things we're talking about here, the singular set of okay.

C

Yeah so so that doesn't mean they can't make attributes that are outside of semantic conventions right they could make their like. They could make their own set of attributes that they add, but they can't change, conventions, work and claim Hotel compatibility.

H

B

And that's kind of why I'm saying like once we get through all of our experimental stuff like like a lesson learned is it's like it did not help us at all to have a bunch of semantic conventions out there in the world in experimental mode for some long period of time like we.

F

Did ourselves no favors.

B

By by doing that, it just was the way it rolled and like going forwards. I'm totally down to like open up the registry and have people just contribute their conventions. But I would like those conventions to be at least the people submitting them to feel confident that they are fully baked enough. That they're fine, managing stability going that from that point onwards, and if they aren't then like don't submit it.

F

B

You do come back where somebody else comes up with some totally different way to like uh instrument Kafka. You just have a different name space for that that kind of instrumentation.

F

C

B

C

B

Like confusion going on.

C

I'm gonna have to call time here um so I think.

C

One thing I'm going to call out is just the T: the tldr is uh there's a high bar for contributing some kind of attributes. Just and- and that's I think should be the lesson to us from w3c um of if we aren't careful of what we send we're basically kind of locking in, and we probably need something else to help address the full issue here. That's not like this proposal should I think I personally think it should get adopted, but I don't think it solves the full problem.

C

I think it just opens up more and so there's probably something else. We need. Okay.

C

um Last thing, I want to call out we're at time box, but there's this issue here, um which is basically should metric descriptors, be a sentence, and it shows a bunch of examples um of metric, descriptor, noun phrases with periods, um so I actually like what the issue is going. After of, can we look like um open metrics, where everything's a noun phrase with a period uh and I think we should probably just like.

C

Add this to our documentation um explicitly like the description of a metric should be a noun phrase that ends with a period if anyone's against, like comment. Otherwise I'm thinking, uh I'll, just open a PR and we'll resolve that one.

C

Okay, I hear I hear no, no one making comments there um cool. So, let's move up to some of our project uh tracking stuff. um Most important thing to call out is I opened a bug about how we don't have any stable resource semantic conventions at all um and I think the impact of stabilizing uh HD semantic conventions. Sorry someone's backing me right outside my door. um The impact of stabilization conventions is limited without resource semantic inventions.

C

So what I'd like to do is kind of come up with a prioritized list of resources that we wish to stabilize, and then we can start working through them. I, don't think we have a resource semantic convention group like we do with HTTP, but I think we might have a lot more interested parties than we do in hcp as well. So what I'm proposing is we go to the specification meeting with we're going to stabilize this set of semantic conventions?

C

Here's what they look like um after we've done an evaluation of what they've like an ECS anyone. Anyone kind of concerned with that.

C

Okay, so then, specifically, the next thing I want to call out with resource stability. uh You can see my two oteps here. um There are some concerns around metrics and Metric stability and what it looks like for Prometheus we'll see you Ted, um so the first semantic convention I'd like to go after is actually the service dot star resource and Magic conventions, foreign. Why do I want to stabilize them?

C

Well, we actually have already specified that the Jaeger exporter is dependent on service.name, because that was a requirement for jaeger for us to actually just freaking work with their ecosystem at all. um So similarly, if you look at the Prometheus world for us to have Prometheus compatibility, it relies on the service dot. Star namespace on resource specifically open Telemetry requires service.name.

C

We set it to unknown if the users haven't told us, but that is a thing we inherited from Jaeger. It's a thing: that's common in some observability tools, not all, but it's uh it's pretty critical for the the whole ecosystem of hotel right now, given our Reliance on it, um I don't include a link, but if you want, we can pull up in the spec where we have Prometheus OTF compatibility, but there are certain key attributes in Prometheus that are converted to service.star namespace attributes, and these are required for us to actually work with Prometheus.

C

So my opinion is effectively we've already baked in the semantic conventions in a way it's in two experimental specs, but in a way where these are kind of stuck um and I, don't think we can actually change them anyway. If we wanted to I personally think they're uh generic enough and useful enough that uh I see no problem with us stabilizing them um wanted folks thoughts here, I was going to take this to the spec meeting and, as our first proposal I can make a PR that basically marks those particular semantic conventions is stable.

F

um Plus one on this and I think that we we have to mark them stable, SDR anyway, because they are kind of transitively stable, because the resource SDK, which is stable on its own, um already defines that they need to be set and how they need to be sent. So it's kind of stable anyway, already.

C

F

Least, for service.name not sure about the version.

H

And when we see changes we mean non-addited changes right like so.

F

In the outside, if.

H

We had the language around like a scheme that relies on changes to considered breaking change and require a major version bump. That's only for, if we're changing the definition of an existing attribute, not adding a new one, adding a new one could be minor right.

F

I believe so yeah, so if you would add like service dot, description or service, dot, owner or responsible team or whatnot, if you add something to it, that should be should be fine but the the meaning and, in the particular case, with service.name.

F

um The fact that it's always set I think that's something that that we should be able to rely on already today.

C

Yeah so effectively right now, this is marked the status experimental. So my proposal is I would mark this as status mixed and then underneath service there's actually a couple here that we could probably stabilize but underneath service. Why are you having trouble um this? This section will be marked as stable, so this would be specifically service.name uh service, dot, namespace service dot instance, ID and service.version service.name is a required attribute of otel.

C

It's actually required in the resource SDK special specification, which is called out here uh with the default value, um and it's required as part of us. Integrating with jaeger just in general, um service.instance ID is also part of the Prometheus specification. If you want I can pull that up and show you all um now. Did we move that one I don't remember if we moved it here, I think it's here now it might be in date of a home uh nope.

C

We move it's still in data model, yeah uh I kind of forget which, which things have been kind of merged in overlapped, extreme manipulations. Oh did we did move it out of here. Didn't we.

A

C

Does anyone remember where we moved? It is another semcom.

C

Supplementary guidelines- that's where it is.

C

No, does anyone remember where the Prometheus Samantha convention thing got murdered.

H

I just do the GitHub search every time. I need to find it to be honest,.

D

C

Specification compatibility, gotcha, okay, there, it is so inside of here. If we look up service dot, you can see service.name and service.instance. Id are the equivalent of Prometheus job and instance, labels, and they are required as part of that uh that particular specification. So if we come back to semantic conventions versus metrics resource semantic inventions- and it is the actual top level um and if we look at service, the only two that aren't otherwise vouched for are namespace and version um and I'm.

C

Okay, if we want to like remove those from our initial proposal for stability, but I also don't see any any problem, keeping them, because I think they've been around an Hotel long enough. That I don't really see a I. Don't really see many problems with that I am going to do a quick check through ECS, just make sure we're not stepping on anything ECS related, but unfortunately, serviced on instance, ID and service.name, as you heard, and saw we're pretty tied to them right now in terms of our just ability to function.

C

Any concerns, thoughts.

E

I have one question: please uh Siri, look now from Grafton Illinois do.

E

Eservice.Instances.Id actually used broadly because when I looked at, for example, the Java Auto instrumentation agent, it's not used on I have not seen it used much.

C

Yeah, so it um this is. This is an important, so it's only recommended it's not required, which is why you don't see it for Java Auto instrumentation.

C

However, if you use Java on an instrumentation with the Prometheus exporter, and you scrape that with Prometheus, you will get service instance ID of what Prometheus would see, which the instance ID would be the job Name colon Port that the Prometheus scrape occurred on so right now instance, ID is used to to basically fulfill the instance, uh part of Prometheus scraping and pull-based metrics, where you need to identify where you were pulled from um I think we could actually do better there uh by having a specification for what push-based metrics have with instance, ID I'd actually like to be a little more aggressive, with instance ID.

C

If you look at my oteps or some of the proposals, I put forward in the TC um I've been proposing. Actually, we leverage instance ID more across Hotel, because I think there's a lot of value in having a common understanding of instance, ID that you can link at a minimum where the rest of the resource attributes can be expansionary or declarative.

C

An instance ID is identified, um I'm, not arguing that yet all I'm saying is right now, when you're in a pool world, if you're in that Prometheus world within Hotel or in Prometheus incense city, has meaning and we're tied to that meeting.

C

um If you're in a push-based only world like open, Telemetry, just pushing um instance, ID doesn't tend to show up today, you can make it show up, though, because I believe there's SDK environment variables to fill it out. Something like that. That.

H

Was like a rule of thumb, scrapers or more instance, ID and other ones aren't or that's.

C

Just the way of the world's kind of today um yeah, the other thing is: if you look at this example, it's a uuid um in practice. uh Instance. Id is not a EQ ID. It's it's a like IP address port number, for example: oh okay,.

H

C

H

Was at a prior job before that would often conflate the term instance with host, where instance was for an example of VM customer on a host, and then you have like you know, the actual host itself is a different thing, so it would be really nice to have a semantically and evil thing. Sorry I see the the clerk do.

A

You have your Henry.

C

A

H

C

The way yeah cereal, sorry.

E

I at grafana we are reviewing how we are mapping um uh metrics in in our Prometheus compatible storage with remote right, where on which envelope on uh instant.id and I.

E

Remember that the hotel specs that says that it would, it could be a uid that is generated upon restart on I, would be questioned before uh putting this on the spec, because I am I, remember I think it is anurag from AWS who was questioning the benefit of having an idea that is not stable across restarts on uh as a vendor, um I didn't understand how to make sense of something that is not stable.

H

When you say stable, you're inconsistent, so you started up will be the same thing.

E

Yeah, you restart, you get the same idea on not something that is generated randomly uh typically. If you go into a Prometheus world, you map this to a label onto every time. You restart. You would have a new time series until you have a high cardinality issue.

A

C

In in Prometheus world, this is the same as um it I know in Prometheus is what is a job and instance right. This is the instance part of the of those default resource labels, so service instance ID is meant to be the same as Prometheus in a sense which is stable across reboots generally.

C

As long as when you reboot that process, you have the same port and you have the same IP address or the same host name right, so I think we I that's a good call out of if we stabilize and I I'd be in favor of this.

H

C

F

H

That name, if they're not making it consistent across reboots right like if you're using service instance ID it should. We should scope down the specification for such such that it is stable, because if it's not stable, then we shouldn't make it a recommendation. Is that is that, where the consensus is forming or.

C

So the the we have we have a few problematic things. I literally was having a discussion with Cloud run and I.

C

Think AWS Lambda might be in a similar boat around what is the same thing uh since you're like cloning, something possibly a thousand times um or more uh right, and so the when you're on kubernetes, knowing if something's about the same, is easy if you're on a VM knowing if something's about the same is easy when you're in some of these other weird environments, knowing if something is the same, is really complicated um or can get complicated, so I don't think we want to force it to be exactly the same in all scenarios.

C

uh I think we want to encourage it to be the same, uh but.

H

Like leave a little yeah.

C

H

Right, we should more clearly Define what we mean by the same and what it means we're going to restart. Yes, yeah.

C

So I think that's a good call out zero like we should probably, uh if we're gonna Mark these as stable, um take a look at the language on server instance ID, and try to make it a little bit better here, so that it's going to serve the same purpose from a push standpoint as it does in a pool standpoint.

C

um I've done a lot of thinking around that a lot of writing that people don't like, which is why you haven't seen it publicly, because I sent it through the TC and no one agreed, um but effectively. We want some kind of an instance. I would love to have an instance ID where, when you're that Prometheus world you get that stable, you know, instance label and if you're in a push world, we have some way of creating a stable label that we can put there.

C

That is relatively stable and I'd like to give sdk's guidance on how to produce it um and then give a set of conventions on what makes a good instance label, but not lock it down where you have to like, say: oh, this is an IP address, because again this applies to things that might not have IP addresses, potentially.

H

The IP address might be in our private Network and then Collide. So the.

C

Exactly which one do you pick yeah I I originally had a proposal of the uh the instance. Id should be the best guess at what IP address Telemetry will be exported from, and that is a terrible specification. Just FYI zero go ahead. I.

E

As a kind of vendor a Stone Point, what is it? What I'm looking for with the instance ID, is to explain to the analyst on how many instances there are Services running and then to be able to groups uh indicators per service on to say all service behaves the same or don't behaves as a or instantos behaves the same or behave differently on um on this use case is a bit different from what you were referring to as the Prometheus conversion, the Prometheus adapter requirement, but what we said on the instance ID.

E

So what we said internally is on I guess many vendors do. It is, if you run on a VM having an instance makes sense if you run on a kubernetes having an instant make sense. If you run on a functional service, maybe.

H

It doesn't make sense, I think it still would honestly. So, the back of my time at Lambda, you'll notice, like uh once the runtime environment, has been provisioned it'll, be consistent. So, yes, like a Lambda, might spend spin up a bunch, I mean or any function as a service. To my understanding, my expertise is more in Lambda, though, um but when you spin up, you know multiple concurrently, you can actually look at the cloud watch.

H

Log streams and they'll have a uuid suffixed after the function name and that uuid is like literally like they're reusing the same instance of Lambda. So to me that would make a lot of sense from like a like, if assuming other vendors with functions of the service.

E

It's a danger zone because it's a kind of implementation, detail of the vendor.

H

Yes, sorry uh yeah you're right, it's implementation detail, but if we think about it from like a runtime environment, instantiation I think that would be the instance then so like yeah, you call it a function or whatnot, but really we care about, but like to your point, they're about like concurrent instances like I, want to know like how many instances my collector is running at the same time right if so, there's only like a finite number of runtime environments spun up with an instance of The Collector, and so as long as like it's kind of we don't care about how a vendor would Define a runtime environment but, like you know, there's some unit that exists of a runtime environment, details of which I don't think matters that much as long as the instance.

H

Id is persistent and cross restarts on the same runtime, environment. I. Think it'd be okay, because if you like you know, if you have a Lambda function, you wait 15 minutes it like spins down you spin up another one. That is a different instance, but you look at the time stance and seeking. Currently, there was only ever one instance at a time, so I think I think that's more about it's in a kubernetes world anyway, sorry you're, saying uh zero yeah.

E

Yeah um I get I think as a vendor. We we typically at girlfriend apps and just when we have two high cardinalities on metrics, and so it's almost if we would do a processing to protect us against high cardinalities on this instant ID, and so maybe they take. This looks like a uid. I truncate on I protect my uh my storage.

H

So is that a implementation detail of your vendor, though, as far as the I mean I'm sure we have so I'm a slunk right now, and you know we also have problems with high cardinality in a prior job. We had assorting issues of uh cardallian, Cloud watch, so I think it's a common problem for vendors, but at the same time I don't know like it is a different instance and so like if the if the value of the like so like, let's call it. The key is the service instance ID.

H

The value like is changing, but I think that's also kind of just I mean that that's the world we're living in whether or not not even just functions as a service, but like like I'm taking all these ephemeral things we're spinning up in the cloud. That's that's the expected value and it's expected for the instance to change.

C

H

C

Gonna have to call attention, unfortunately, I think this is a super valuable discussion. It is I want to continue it. Let's continue it on. um So, yes,.

H

Sir I'd love to talk to you offline about this, sir yeah.

C

A

C

In in slack, um but more importantly, um I'd like to uh possibly next week, have a PR proposal of whatever changes we want to make to uh the definition of service um attributes. If we need- um and we can talk about what the proposal will look like in our next meeting.

C

um So let's, let's get that together, because I'd like to try to get these marks stable relatively soon I. Think trying to encourage instance, ID to remain stable across restarts, is important for a variety of reasons. We can talk about that next time, but, let's, uh let's talk in our slack chat, yeah, it's Hotel semantic conventions, WG and uh thank you, everybody cool, um since we only have 10 minutes left, uh let's step into Trask, you wanted to talk about blockers for HTTP uh semantic convention.

D

Yeah, um so there were two: uh if you can pull up the meeting notes.

C

Oh sorry, I was looking at them myself, but not sharing, because that's the kind of guy- and here we go.

D

um So I think uh these two, you answered um I'm I'm happy with that answer. We couldn't move on um the second two. Now that the um the HTTP metrics yaml PR has been merged. uh Do we believe that we can remove the blocking label from these to.

D

And I guess I should say: I I think we can, but I want to confirm that there's nothing I was missing about these. Yes,.

C

I think I think we we should be able to the the I mean. You saw my answer at the top one as long as we make sure that we don't have duplicate attributes where the same name means different things or the same meaning has different names, then I think we're. Okay, uh that's what all of those blockers were kind of about um to try to prevent that from happening. That's what actually all four of them in tandem, Were Meant, to prevent.

D

But as long as we've done, the last the third I didn't follow. I didn't follow that Association. On those last two I thought. The last two were just about yaml.

C

Those were about getting into yaml, but basically in putting an emo, we would discover all the conflicts was the thought I.

D

See so that's one thing we haven't done is there's no attribute sharing between traces and metrics, but I think lindmill I think you had done a PR, a build tool PR that allowed that now, yes,.

G

uh We can have, we can Define attributes in one place as an attributed group and we can refer to them from any other places. I can do this change if we're ready.

D

Yeah yeah that'd be awesome. Thank you. So.

C

It sounds like what we can do is Mark all of those as not blocking HTTP and just have what Lin Miller is going to do as the blocking task and then and then we're done cool right awesome. In fact, if you added that to the build tools thing with Miller I think you could probably close two of those of those requests as being done. If I'm correct, like we now have a shared place to put attributes right.

G

Mm-Hmm yeah yeah, so let me um start working on this I'll check. If those two can be closed, then.

D

C

You yeah I'm, pretty sure what you did solves. Those two I was actually going to mention. That too is I. Think I think that works solved both of those we just haven't done it yet so um cool. uh We have seven minutes left and I think what consider moving the meeting to weekly I'm the I'm, the long poll here I apologize. This is bi-weekly because my calendar is hell.

D

um So I my request is, um it doesn't have to be at the same time on the alternating week, um but uh the HTTP group would has uh we're are targeting um our final proposal to the community uh five weeks from now, and we have several blockers on this that require this group attention.

D

um So my request is at least for the next five weeks. If we could meet weekly, that would uh make me feel at least a little bit more optimistic about hitting that date.

C

Yeah, let's um can you send out a another? Are you I saw your Doodles before I appreciate it yeah, let's send out a poll for the second meeting time for the every other week and then uh that'll help.

D

Yeah cool great and we'll do it uh I'll limit it to the morning hours, um uh since we have at least Armin and I. Think surreal is in Europe also.

C

Awesome all right, I think that's what we had in the meeting today. um I think we're starting to Chip Away um in two weeks, I'd like to have a proposal around Reese, the the um server stabilization. So let's remember to continue that chat, but additionally, I'm going to be bringing the kubernetes simcov with a proposed like I, also want that to be the next thing.

C

We start because I think, once we have kubernetes once we have um service I think we're actually in decent shape for HTTP in general, and then we can keep expanding for VMS and other environments and things with resources, but because we don't have a resource them kind of working group. We might need to act as that for now.

C

D

Yeah traffic, um since we have a couple minutes, um one thing: I noticed when you were looking at that Prometheus um uh scraper and the dependency it has on service instance ID as I notice, there's also net peer name um and a cut something else: net, pure name.

A

D

Name: net host port and HTTP scheme.

D

And so I wanted to get your I can drop a link in here um so to the topic of the ECS. You know the ECS exploration um and aligning with ECS whether this becomes uh blocker. Essentially for that.

C

I think this causes huge friction. uh I, don't think I wouldn't consider it a blocker I think this is one where we need to go to the uh go to the Prometheus folks in The Collector.

C

um So there's there's two aspects of Prometheus, compat well, I should say: there's three aspects of prevent this compatibility. We have to worry about number one is: can we suck in Prometheus with with the open, Telemetry collector and then output Prometheus with the open, Telemetry collector right? So is it a pass-through? So if, if I use open Telemetry to scrape a Prometheus endpoint, do I get the exact same data out the other end?

C

These are the attributes that we use to do that funnel right to make sure that whatever comes in a goes out, B I think it's possible for us to change both of those without that breaking that use case at all. The second thing that we want to do is figure out. How should, if I push open Telemetry? What should the output Prometheus look like and what attributes? Let me impact that output Prometheus right, um and so that's where these come into play.

C

That use case might be broken, but that might also not have a significant amount of usage yet and that's something we can talk to those communities about um and then the third. The third use case is um basically when I scrape Prometheus and send it out otlp right. What should o2p expect of Prometheus data.

D

Okay, I will um just add it to the ECS uh analysis, issue spec issue as a something we need to follow up on. If we end up going down that route.

C

Yeah I think I think for 90 of usage today, changing it shouldn't be a terribly big issue.

C

As long as the the three pieces of data still exist and as long as that end-to-end use case is upgraded simultaneously of Prometheus into Prometheus out is upgraded. At the same time,.

D

C

D

Long as it's just renaming um as long as it ends up just being renaming, we have the schema translation that we could fall back on as far as justifying the the it not being a breaking change to something stable.

C

Yes, yeah. We could also in Prometheus, basically double write those attributes for a time right and that I think is also reasonable, cool all right. Thank you. Awesome thanks. Everybody we'll see you next week, depending on the doodle thanks.

F

D