Open Telemetry Uncategorized, 3 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-03-03 meeting

Description

cncf-opentelemetry@cncf.io's Personal Meeting Room

A

Good morning,.

B

Good morning, everyone morning.

C

Looks like we have pretty safe agenda for today.

A

B

Just pinged uh with Miller.

A

And I was hoping Josh, but let's get started.

B

um So yeah so I went through the remaining um issues in our projects, both of the projects, um the HTTP semcon project and the general semconf project.

B

Hey Josh heylon Miller.

D

D

B

Morning so we've got um these uh I'm just looking at blocker for stability and not the ECS ones um and same over here.

B

And so the plan for I think the plan we discussed on Wednesday um uh seems good to me, Josh just to catch you up with the we're planning to um still try to get all of the PRS in by the end of next week, um short of it completely ignoring ECS, um and if ECS comes in whenever it comes in, then we deal with that.

E

B

um So the remaining ones, um so a lot of them have uh open PRS already, um which is great. These are the ones that don't have a PR out. Yet um this one um I, don't think, is blocking httw stem comp anymore, but um I'm waiting to hear from Armin just to confirm that he agrees um this one is pending the TC discussion decision.

B

um This is marking the attribute requirement, level, stable and um so I have a proofreading PR in itially, and then my idea was to send a PR to mark that stable.

B

um But a couple of questions I had um that came out of discussions and a couple of other issues, um so Josh um in your stability yeah um do attribute requirement levels applied. It sounds like we're trying to say that it. You know the stability requirements are more on the consumer side or is it both so.

E

um Yeah this this that you're getting into really fun questions, there's what we intend to do with these and then what's actually enforceable.

E

um So the idea behind the semantic conventions is when users don't touch anything a tooling vendor can like use it as kind of an API to generate out-of-the-box, dashboards and manipulations. However, we're still allowing users to do whatever the hell they want in the middle.

E

So if a user goes in and like changes around attributes, we might need to like. We have two ways to deal with that in the spec right one is, we could just say this is open and users can do it, and people have to understand it happens. Number two is: we could actually start requiring, for example, The Collector to drop the schema URL on anything that has a manipulation of that fashion, meaning it no longer abides by the semantic convention.

E

I think those of those two options- I'm leaning towards the first one for now where, basically the idea is, we do our best job to give the best possible API for Telemetry, and we give users the most flexibility to do whatever the hell they want.

B

Yeah so making that sort of a back-end, a tooling like if you're the two some tooling, might be more or less resilient to those changes in the middle.

E

I mean we can we can basically recommend against users doing this, but I don't think we should prevent them um is my opinion.

E

So we could, you know, create some sort of a thing where, if users do this kind of manipulation, we drop a schema URL to help back end tooling, if this becomes very problematic in practice, but I also suspect that it won't be as problematic in practice.

E

um I think users are generally going to keep semantic convention, but attributes um generally.

E

Especially the required ones.

B

Yeah so I think that's really the only place. Those required ones that are potentially um problematic I mean that if somebody dropped one of the required ones, so I was kind of looking through and um like. If somebody wanted to on HTTP client spans.

B

um Drop the net peer name, um so I was wondering you know: does it make sense to mark this? As recommended I mean we could change that requirement level so.

E

um Can you real quick? Can you go back to the list of blockers because I think with Miller opened or raised my attention to one two days ago, that I think is actually a blocker if it's not already listed.

D

E

D

Yeah trust, do you have um backward compatibility for requirement levels uh in your list or, like actually I, think this? This can even come later. The thing the problematic thing that we currently state that adding can you attribute is always allowed and my impression, if we change this is the minimal thing we need to change and we can Define even more about the backup stability.

B

Is this what you're talking about.

D

D

E

D

E

So this is specifically for required attributes. So if you look at required attributes, we say that they will always be there versus recommended right. We say they should be there. So there's a question of whether or not required attributes are uh something that we can ever act ever after the initial specification. It's.

D

The comment above.

D

Sorry uh did Josh just reply to my comment as get up.

B

Got it? Thank you.

E

So there's one where we also need to enforce schema URL, but basically, if we add a new attribute that is required, then if I have an old version and I go to the new version, I actually can't synthesize that attribute yeah so effectively, you know going forward required becomes interesting. The first time you make the semantic convention you can add required, but from then on, you can only add, recommended or lower in terms of attribute levels. You.

D

Cannot conditionally require it with default value.

E

Conditional, oh yeah, sorry um I forget the order, but yeah the anything below required. You can add, but you can't add required anymore after the initial release. So then there's the question of is is required great. In this context, should we try to be preferring conditionally required and recommended?

E

D

Yeah, so when we Define this levels, uh there were a lot of concerns about required, attributes and moved down in some fair amount of work to investigate where we use them and they're. Actually, we don't use them excessively and reduce the amount of required attributes the acceptance, GDP and network we mostly use them to say, okay. This is the system right. The the thing that identifies semantic convention, plus uh maybe one or two other things, um but um it's a good question whether they're the ones in HTTP are, can be relaxed right.

E

Yeah yeah I'm I'm, almost wondering if the only required attribute should be those kind of some kind of identifying ones going forward like. Maybe we just explicitly do that only because you know that this does limit like some utility later on, but I think it might it'll be clearer and more consistent, like, as we add, attributes and expectations on what goes on.

D

Comment as well we're kind of trying to equate two different projections of semantic conventions, one for instrumentation one for consumers. Then maybe we just clearly state that okay, this is required for instrumentation, but for Consumer it's not it's difficult, but we can find a way to make it like to explain it more clearly.

E

Right and I think that that's exact yeah the way the way you're thinking about the problem is the right way to think about it. Where we're trying to use one thing where we might need to that said, we might also be able to collapse what we have into one thing like it could be that what we do with required and optionally required.

E

It's the same thing instrumentation is required to provide it when it's available right, but a consumers should not expect it to be there, and we just have one thing which is required right.

D

Yeah and that that consumers should expect is.

E

Yeah so basically required becomes conditionally required effectively to some extent where we say: instrumentation must provide it when they have access to it, and then consumers basically have this this unknown of they don't know if it will be there or not, because it depends on whether instrumentation had it. So when we do our conversions, our up, downgrade conversions right and I add a new required field.

E

I don't have access to it on the left-hand side, so I don't need to provide it because it's implicitly conditionally required right.

B

So I got lost briefly on conditionally required.

B

um Recommended so Josh, are you saying that recommended means instrumentation must provide if available or.

E

B

E

Saying basically required and conditionally required to kind of collapse a little bit.

B

But what doesn't require it already mean they must provide it.

E

But but it only means instrumentation must provide it.

D

It feels like we need an extra um sorry, the next to a requirement level uh like it must be there. It's like it's always there right. It's above required.

E

You must provide, but consumers may not get it. Yes, that's.

D

Exactly what we mean yeah, something that users Cannot drop so if they drop it, everything breaks.

E

That yeah well, that's true, I, don't know how we're going to enforce that just yet! Yes,.

D

Definitely this this is.

F

D

Kind of forms I'm.

F

Worried about the direction we're going, this seems to be very complex like why do we want to attach some semantic convention required filled with the schematic commission? Identifiers I feel identifier should be something totally established from this. Like you, don't have a you can have a single event with a database and HTE things both exist and how you identify that.

D

Thank you. We are talking about existing attributes, they are there, it's the DB dot system messaging.system and for HTTP. We don't have one, but it's basically http.net right, yeah.

F

Yeah, but we shouldn't use anything like http.m method or something as identifier identifier should be separate. I want us to have the separate of concern like one shouldn't use, HTTP without method for identification, because if you have a very noisy stream, there are one million different type of events. Do you expect something like a logic there to tell first describe if there's an HDMI, so there is not, then something else. This is super inefficient I mean shouldn't, encourage that, and people need identifier. It should be something else.

D

I think we had a quite long discussion in the community on this, and Community is not ready to introduce this thing. There is a lot of problem. You cannot understanding yeah.

F

Exactly so so, the message here is we're talking about semantic convention. We don't give a to identify. Identifier is a very good problem that we'll address later.

E

Yeah I, don't think the identifier bit changes the required discussion. We just had, though,.

A

Yeah, um like you're right entirely.

E

That we, we shouldn't care about identifiers, we're dynamically duct typed right now, it's all structural typing. It's there's no like this is definitively this type. It has exactly these things, exactly that's something we want to know for us, but the the question here is when we think of these required attributes who target audience of it and I think let's shift the document to be about the instrumentation, not about the uh the consumer.

F

Yeah exactly yeah the semantic convention is talking about. These are the things you want to have then go for it, then, with the fun like for instrumentation library to be a qualified, HTTP instrumentation, then there's something that you must provide and authentication should be removed from the entire discussion, not to confuse anyone.

E

Yeah I, don't I, don't think identification is in um in the current discussions here. What I'm? What I think that tell me if this straw man's correct from from what we've been talking about? What what I think we need to do is in the attribute requirement levels and the metric requirement levels. We should say that these are requirements for instrumentation providers, but consumers of the instrumentation cannot expect attributes based on these requirement levels because of version SKU and stuff like that right, the um from a schema, URL standpoint, there's still this issue of.

E

If something didn't exist previously, I cannot provide it.

D

C

Sorry sorry I will be possible like better to not call it required, uh like I. Just uh have something something different. So like a from my understanding, Greek wire, there's like a you know, stand name is required span. Id is required, Place ID is required, uh but looks like like if I'm following rights to the discussion, it looks like everything that comes from HTTP conventions or Messengers or whatever semantic conventions is actually optional. So if it's not there at all, then it's it looks like it is still fine right. Yeah.

E

Yeah I'm actually suggesting we only have conditionally required recommended um and uh option. What do we call it now, opt-in.

B

So you're saying Josh to move this into conditionally required. We.

E

B

A

F

E

And we only allow conditionally required going forward, because, basically, it sets the right term condition required means instrumentation must provide under a condition that condition could be always right, um but from a user standpoint, since it's conditionally required, they can't expect it to exist all the time.

A

E

The conjunction for instrumentation can be different for the condition from schema, URL transfer right so effectively we're getting rid of required and keeping conditionally required.

C

Right so basically like for consumers, let's say for a platform which comes like for younger. Let's say uh we just need to expect that like a Name ID is will be there, but everything else can be just absent right, it can be, uh it can be. There cannot be there. So basically, it's always if, if you want to like do some some analysis based on this.

A

D

I have two questions: the first one who I should Define the default condition right and always would not work if we get to the pr stage right, um conditionally required without condition means that instrumentation must provide, but users can drop like effectively.

E

Yes, well, one of the I think the default condition would be instrumentation from this specification version onward.

A

E

Would that would be the default condition? Great um should provide it and and from a schema, URL standpoint. It would be like you know. If I get data from a previous version, I don't have that condition so I don't have to provide that value. Oh.

D

D

Okay, I think there are, it will be difficult to explain, but when we can find a way. The other question is so what we're saying we are doing like this enforcement thing applies to instrumentations and we just don't tackle the consumer aspect of it at all, burned out.

E

For now, yeah I think so like the way I'm thinking about this is, if I were to, you know, pull in.net and pull in the HTTP, instrumentation library and I would have dump a TLP Json I should be able to take this end, conf look at the otopjson and verify for any particular spec version. Here's what came out and that's with no config right as soon as the user starts configuring things and touching views touching. You know Samplers that kind of junk all hell could break loose.

E

That's not we're that's out of scope for us, that's what I'm, basically suggesting and if they put a collector in the middle and start doing crazy stuff again out of scope, but that that Baseline, if they take that Baseline and shove it to a database, that database can expect this Baseline and we're optimizing for that use case of users. Take the you know, curated semcon, that we have and leverage it, um and if they want to do something crazy, they do something crazy and I. Think there are spec holes things in the spec.

E

We can do to make the manipulation case, not as terrible. You know, like maybe defaulting schema, URL away in those instances. Maybe um yeah I think that's actually the right thing to do if we want to account Downstream that this no longer advised by simconf is get rid of schema URL for anything that had like a manipulation that could break schema. Euro.

D

Yeah, yes, and we can tackle now, this is visible. Justice should have should, if vendors would not like it, because they would like some certainty and like in messaging discussions, we always get to the point where what what consumers expect to get out of it right. So it sounds like it's the eventually we will have to tackle it. It's just not the right time for, for it.

E

Yeah I, don't I mean I Pro I, don't feel prepared to tackle the entire problem just yet. So that's why I think you know, and we can actually explicitly State this somewhere in the spec. If we want um of you know, we are targeting that exact use case of I install a library, no extra config I dump Json of the otlp. It should look like this.

A

E

And then, if that's how you flow into your back end, this is what you should expect and users doing anything else are outside of our guarantees: I'm happy to actually update semconf stability conventions kind of around that, if that makes sense. This is why, when you look at when I was trying to write like what semcom applies to I was focusing on the instrumentation API at one point, because I feel like that's the only thing that we're targeting right now, so people can go write, instrumentation.

E

um Yeah anyway,.

D

Yeah, it will resolve a lot of questions right. It sounds good.

E

Okay, so uh I think I want to make sure that that particular topic that you raised it was there a bug associated with it.

D

The suggestion um that we have, if.

E

You could open a bug for that that'd be great and then I will um prioritize. Writing some spec for it. The the thing that I'm is that I think is important for this working group, though, is I. Think if you took the approach of every attribute is conditionally required or below or whatever is below required. Yeah um then everything's gravy right.

E

If we leave things as required, I think we have to answer the. What the hell does required mean uh part of the spec. But if everything is you know below that, I think we're reasonable. We don't have a problem, we're ready to go.

D

I think even that's been, can never create required attributes yeah. Maybe we will never have them. Then.

D

D

Cool I will create an issue and we'll get forward.

E

Yep, let's pop the stack to the original question, then that was tangentially related right, attribute level requirement needs marked as stable, um so I think.

F

E

That issue is opened. We can address the attribute level requirement stability document, but I I feel like that. Let's address that bug first and then we do the document.

B

Cool and I I can take that on the pr on for um this language after you open that issue, but Miller okay,.

D

Yeah I'll: do it right after this meeting.

B

Cool because I think that actually ties into the discussions in these two issues also.

B

B

Because over here the concern was, you know we wouldn't be able to add something later well. This is for metrics.

B

um Okay, so Josh so you're saying we could add things later to metrics now, as long as the schema URL gets bumped.

E

That nominally that's true, uh um that starts in April. um The the thing about metrics is, if you add something, it should not change. The number of Time series.

E

So this is, this is problematic here.

B

Right um so I think the per just a high level The Proposal here is offering an opt-in um kind of attributes, this code class. That then, would turn off status code. So you're kind of opting into that I.

E

Think that's totally fine again, the user is opting into it. The thing that would have to you'd have to preserve then, is effectively if I have the same inputs. I should have the same time series at the end. So, however, your bundling status code that that needs to remain the same in the semcon for metrics.

B

But sorry I don't really understand that what it, what does this say? Okay, I'm serious number of Time series remaining the same mean.

E

uh The number of Time series would be the number of different metric points that I would have in something that that instrument computes. So.

A

E

If I'm doing latency, right and I get a particular URL and that URL has a 501 and a 502 error right when I'm using status code, I should have two Time series, because the URL had two different statuses and that's my polynomial. If I'm using status code and they go 5xx, I should have one time series and always always with that same input. I should have two for the one and one for the other. If that ever changes where I end up with two instead of one, my metrics break my alerts break my magnitudes break.

E

um So that's that's. What can't change okay.

D

So, looking at this issue, this assuming the original status code is preserved. Adding can you say it didn't write say it is called attribute would not change. The number of things here is right.

E

D

E

Wouldn't yeah, so if you add status code without removing sorry, if you had status code class without removing status code, you wouldn't increase anything, and that's totally fine, but dropping the status code is where you switch up. The number of Time series right.

F

So we have to solve this for the initial stabilization or we should stop talking about it.

E

I actually think, given what Miller just said, I think this is a very important feature for metrics and we should totally make sure that it's addressed but I, don't think you actually need Ed, initially, first of all, I don't think status code. Is uh that much cardinality that it's a pro that it's super problematic in practice. Yeah.

A

E

Actually use it and we have tons of problems with cardinality, um so uh yeah I think folks that do that in Prometheus are kind of hyper optimizing for a little bit of storage. It's definitely something we want to support I. Think it's going to be critical for people who are hyper optimizing, but I, don't think we need it initially and.

F

And for my things, I I feel a better way. It might be to shoot this as another dimension for histogram buckets, so we're saying, histogram buckets can be multi-dimensional and you define what's the, how do you bucketize the status code.

F

But we should talk about this offline, Josh, yeah, I.

E

Don't I I kind of get what you're saying, but that's that's yeah the buckets are a different time series: I I, don't I, don't think this is a histogram thing yeah, but we can talk about it. Offline.

B

So the reason why I think this is uh important discussion isn't to add status code class, but to make sure that we understand as we're defining what stability means that we're not locking us into some place where we couldn't have a path forward in the future.

E

Yeah yeah, absolutely the um one thing I will say that I think we should probably make explicit if you have features like this, where there's an opt-in where for metrics I, would drop status code and use status code class right, we, we should not expect a hundred percent compatibility of metrics generated with that option and without that option.

B

Yeah so I think where I was getting I was getting confused. Here was when we were talking about stability applying on the consumer side, um and if we have a conditionally required status code which is conditionally required would mean to me on the um the consumer side. It would also be conditionally required if, if the status code was available, I see I like I, like this direction of relaxing a lot more stuff on the consumer side and only focusing on requirement levels being applied at the instrumentation side.

E

Yeah yeah I think we. um We should ensure that consumers are flexible, but this is still useful to them um and there's a there's, a dance that we're doing there.

B

Yeah- and maybe we can put better in for enforcement in the future, but I like the idea of relaxing that for now for initial stability, okay, cool, sorry, we were over time.

B

um This was really great and we will try to rework everything based on um relaxing consumer stability and focusing on instrumentation side stability.

E

And so the the definition of what stability means, um if you haven't updated your comments or whatever, please take a look, it has no approvals yet, which has me a little nervous, uh it's also large and aggressive. So um we need to get this wording right. Yeah this one I, so you're I want to get this through as soon as we can.

B

E

Have I seen that already.

B

Is this already updated because the earlier it was talking about consumers and tooling um yeah that was kind of the specialized tooling, our expected German staple for that to lean.

E

Yes, but if you look look at what we're enforcing um oh.

B

The things provided to the apis, okay got it yeah I see so you you're already ahead of you. You've already got this factored in over here of only at the only applying at the API side. It.

E

Might be worded poorly but yes, that was the that's the intention.

A

B

Yeah, okay, cool cool thanks. Everyone thanks.

D

Thank you bye. Thank you.