Open Telemetry Uncategorized, 14 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-09-14 meeting

Description

Open Telemetry Meeting 1's Personal Meeting Room

A

Okay, it's five after I had a couple things I wanted to discuss.

A

Let's see feel free to add yourself to the attendees list.

A

Is anyone here from the prometheus side of the fence, or is everyone primarily associated with open telemetry.

B

And on both sides.

A

Well I'll, uh I guess I'll ask my questions then, and then, if you have any feedback or comments, we can talk about it, um but we've been working in the collector and specifically in the prometheus receiver in the collector um and we've come up with a scenario: that's very difficult for us to handle, because sometimes we might be scraping, open, metrics and sometimes we might be scraping prometheus.

A

In prometheus, your metric name in your type comments, always matches or not always but generally matches the name.

A

Of the metric points for counters, so you can have a counter. That's called like foo, total and in prometheus. Your type comment will also have you know, type foo, total counter in open metrics.

A

The type comment doesn't include total anymore, but it still exists in the in the metric point itself, and one reason why that makes it tricky for us is because it's difficult to tell if a.

A

Metric stream named food total is a counter named foo or, if it's a oh sorry, yeah a counter named foo or a gauge named foo total um and right now it seems like we basically have to choose one or the other and it's difficult, because they could in theory also coexist, which means that you have two metric families with different names. But where the actual points.

A

Have the same or have the same name right now, we're sort of ignoring the case of a gauge.

A

We're ignoring the case of gauges named with something ending in total and trying to solve for the case that abides by naming conventions, but it's just something. I wondered if anyone had any thoughts on or ways that others have have solved. This.

B

So outside of primitives, um how does the type of a metric gets identified?.

B

If it's not by the does name make some well so.

A

There's you mean like in otlp, yeah um otlp is structured, and so it just has a name, and it has a separate type and the name and the type aren't necessarily related.

A

In that, like there's no type suffixes in in otlp, it's just a name and then a structured piece of information that has the value and if it's a histogram, we'll have all the buckets.

B

Okay- and you don't have the gauge versus contour issue, because.

A

A

You get a thing so the issue to be very specific. Is that given a metric point right so, like the data point belonging to a metric and prometheus, it doesn't seem possible to identify with certainty what the underlying name of the metric was, because you aren't sure if you should trim suffixes or not um so we do some heuristic guessing in otlp. You don't have that problem, because all the data comes bundled together in a structured way.

A

So you don't you don't have to try and remove suffixes or anything to get from the point to the metric name, because they're all part of a single data structure.

B

B

Sorry, I'm trying to grab to get an understanding of the issue and I might not actually be even like the best person to answer that okay, but we might want to actually go to prometus 100 meeting, but so like to summarize it's when you receive a promoter's metric at the data point level, you don't have the information of the type and right now it's this. It's heuristic based on the name.

A

It's that prometheus handles things one line at a time, or at least we're using the appender interface. So we get like a line that says: here's your metric point.

A

um It's you know. The name of the metric point is foo total and we can't tell if we should associate that with a metric named foo total or a metric named foo, because total sometimes gets trimmed and other times doesn't so that that's our fundamental issue.

B

So so at that point the type information is not available to you.

A

It is available to us. The issue is that there isn't.

A

Necessarily, there isn't necessarily a correct answer: you could have a gauge named foo total and a counter named foo in the same set of scrape results, and so it would be ambiguous whether any point named foo total belongs to the counter named foo or the gauge named foo total.

A

Does that make sense.

B

Yeah but the type will have the name as well right.

A

B

A

We can get metadata correctly, we can identify the set of metrics. We just can't associate the points with the correct metric like we have to make some guesses and sometimes wrong.

A

Yeah I can bring this to a prometheus group. I suspect the answer is you're. Doing the correct thing. um Bogdan asked me to check with prometheus folks to see if there are any other ideas or if they thought about this at all, when they were writing the openmetrics spec.

B

And could you remind us the underscore total comes from in the name.

A

Under so underscore, total in prometheus is a recommended suffix for counters, for counter yeah for counters and in histogram there's a series with underscore total um in open metrics. It isn't part of the name underscore total isn't.

A

um Instead, the name for counters never includes underscore total at the end, and but all the points in that series in open metrics for counters will contain underscore total at the end.

A

So open, metrics and prometheus differ in their handling of this.

B

So prometheus has both a naming convention and a type so yeah, whether you're like go gage or a counter.

A

It's a convention, so in theory in prometheus it's allowed to have a counter that does not have the underscore total suffix that's allowed in prometheus. It's just. It goes against their conventions, so they have a counter type and independently. They have a convention, openmetrics sort of formalizes it but also changes it so that the name no longer includes total.

A

Only the name of the points includes total.

B

I'm looking at the ticket now.

B

I my gut feeling is that we either need to only base the translation on the type or on the name, but if we try to do both we're gonna get into like some unexpected results.

B

I don't know if the best effort is actually better than just saying only one, either the type or either a name.

A

Meaning sorry, I don't know if I follow.

B

Meaning, that's I'm actually wondering if we should try to be that smart with the metric name.

A

Well, we have to be to support counters at all. Unfortunately, we're not necessarily trying to be smart. We're just trying to associate data points with metric metadata.

A

If we don't do any of this, then we don't get any metric metadata like type information for our metrics.

A

B

So, what's your proposal david.

A

My proposal is that we should do the right thing for metrics, which follow naming conventions and do the right thing, sometimes for metrics, that don't follow naming conventions, but for people that do.

A

If there's ambiguous scenarios like a gauge named foo total and a counter named foo, that we should.

A

Do the right thing for the metric type, which is following conventions.

A

B

B

Yeah I'm looking at prometheus, dark hips should unfortunately.

A

Right so that's the prometheus conventions.

B

Yeah I mean I'm just if it was must that would simplify.

A

No prometheus has very few musts.

B

And I guess the reason why it's assured it's because, like there is precedent where, like the naming convention in existing infrastructure, metric, doesn't match.

B

And so the question is like how we're gonna handle that with tlp.

B

I guess the the issue that your eyes was actually raised. Indeed, with such a case right, like it's envoy, so envoy, doesn't respect committee's convention.

B

Actually, let me find it there's like invite crystal stream and then.

A

Yeah, it's a bit of an.

A

uh Yeah, this is right, so this is a case where they're in prometheus and they're, not following conventions, but the the issue for us is. We also don't know whether we're scraping prometheus or open metrics.

A

So, looking at the issue, if this were open, metrics, the envoy cluster upstream rq, total metric points would correctly be associated with the envoy cluster upstream rq metrics metric type.

A

Knowing that this is prometheus would help us, because we could use different rules.

A

But I don't believe we know that at this point.

B

Well, we are in the prometus receiver, but you say that the parameters yeah. Okay, I see.

B

Yeah, um the only way to do this is to take from this add some metadata that will help us make the requisition if it's coming from prometheus.

B

If we can differentiate, data is coming from prometheus or not, then we can make the decision.

B

So if we can ask the prometheus folks if there is a way to do that today or if there is some metadata that that could be added, that will help us make this decision.

B

It could just be the scraper that can add this information.

B

The scraper package.

A

Yeah yeah right, like we do for metadata actually.

B

A

Be useful to stick in metadata potentially, but or maybe alongside it.

B

Yeah, I'm just sharing the prometheus calendar. It seems like there's a contributor meeting on monday and then the dev submits on thursday. In case, we need to discuss directly with the parameters community.

B

With regards to propagating the source.

B

What other type of source are we gonna propagate like besides parameters, like I mean, is it going to be like something? That's just for primitives kind of like.

B

A one-off, or do we think that propagating.

A

I mean I assume that something like I assume that a dashboard trying to use this information like grafana that still has metadata and metric names separate, also has to try and figure these things out, eventually we're doing it very close to the source.

A

So it's more visible because you lose the data if, if we can't figure it out correctly, but I imagine it might be useful for them.

A

There it is so there's.

A

There is a meeting on monday, it's a little early for me.

A

There's a dev summit.

B

So I know that chromatus has also a concept of metadata. Already. That's not very used, but indeed like the intent was to have the something like a dashboard or things like that. To use it so.

A

B

We introduce that we need to see if that actually gets passed all the way and like if there is like a mapping with the prometheus metadata itself.

A

Yeah, so that's what we're using today we're basically joining the metadata that comes with metrics, with the points to get otlp metrics.

B

B

B

The link is all I meant like the metric metadata.

A

Anyways yeah we can I'll, try and find you the calendar things and I'll go present there see if there's any ideas.

A

I I think we've spent uh a good amount of time on it, so we can move on if that's okay,.

A

um The other thing I wanted to ask about, I'm not sure if I'm mostly trying to.

A

Sort of trying to understand the current status of this. So at one point in the past, it was forbidden for metrics on a prometheus endpoint to have this same name but a different set of labels, so that wasn't allowed um it was. I remember, because c advisor used to do this and um which is a project I maintained and it blew up in our face at one point because prometheus started.

A

Not allowing it and dropping some of the metrics and it confused a lot of people, but apparently this is something that's since been relaxed, and the question basically is: if we have use cases for it like infometrics that have dynamic labels. We want to add.

A

Is that something we should take advantage of now or is it something we should uh try and still not do.

B

Looks like it was relaxed quite a few years ago, like now in 2018,.

A

I know I missed it.

A

I've been spreading lies since.

A

I'm mostly looking for the current guidance, because I don't think there really is much.

B

Look we should we should relax it to be more compliant with comedians.

B

Like any reason why we would not like, I guess it's more complex like you need to react to label change now, but I think you need like what uh michelin had said. If prometheus does it, we probably need to be compatible and handle it.

A

No, no! It's it's not sorry! This is about on the exporter side, so should open telemetry, ap or sdks produce metrics, which have.

A

In which different streams or different points have a different set of label keys.

A

It used to be disallowed, I'm trying to figure out if it's still highly discouraged or if it's basically just something allowable that we should do. If we know what we're doing.

B

So we sort of when it was disallowed, it was on like a prometus endpoint scraping. That's right.

A

uh It was both, so it used to be that your client would drop silently a bunch of your metrics.

A

So if you in c advisor's case, we had metrics with the same name and two different sets of labels and it would flip-flop between which one it ended up. Actually, returning on the endpoint.

A

And I believe it would also reject the scrape at the server side if you tried, if somehow you worked around, that restriction.

B

B

So isn't this the case? If we're in the collector case, aren't we gonna have any way that when you effectively have the collector potentially scrape from different original endpoints, but then expose it for potentially having probabilities to re-scrape it and then we'll have like label varying.

A

Are you saying like? Could this happen on the federated prometheus end point? Yes, yeah! It can definitely happen. I don't. I imagine prometheus made an exception for themselves back then, but I'm not sure.

B

Well- and I'm saying like similarly like, if you use the collector, the the open telemetry in in the way that it fits from multiple promoters endpoints at the export level, you're gonna get different labels for the same metric name. Potentially, yes, that's correct. I.

A

So maybe this isn't a big deal, though, and we should just go ahead and do it where it makes sense.

B

Yeah, they are just two different time sheets, pretty much.

B

A

Okay, well, that's that's all I have to discuss then.

B

Thanks david, okay,.

A

Thank you. Thank you.

A

uh If there's nothing else, then uh we can take off a little.

B

That was have a good rest of the day. You too bye.