Open Telemetry Uncategorized, 29 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-07-29 meeting

Description

Instrumentation: Messaging

A

A

B

Should we uh should, we get started, sure looks.

A

Like we have a few, a few new hello.

A

A

Let's start from the top.

A

New component, oh yeah, this person, just uh oh hold on a second zoom, is in the middle of my screen.

B

Cool um this is just a sponsor needed sounds like.

A

All right, azure, metadata, compute name.

A

Resource detector, hot name, I think that's a typo.

A

C

There might be a specific code for that particular section of the resource detection.

B

Oh, we don't have a label for it.

C

So and even if it did have a label well, I think if I had a label would.

B

It work, I don't know, would it break because it's yeah.

C

The resource detection, processor, internal azure is uh pablo mx psi.

C

I don't think the make target I added is going to handle that situation.

A

C

123 or something like that.

A

B

Yeah, I have a feeling our our script is not going to be able to manage this many levels of.

C

I've nested yeah. I think it's making some assumptions about being able to apply the receiver or exporter at the end of it as well right so right.

A

Yeah, this is not gonna work. Okay,.

C

I think we should probably just throw the resource detection processor label on there and then ping pablo specifically.

B

Is this gonna like alert a bunch of people.

C

uh I think it alerts dimitri.

B

C

B

All right, fine I'll just.

C

No, it's not dimitri. It's uh it's.

C

A

A

I'll comment and I'll add a priority needed.

B

A collector are not able to connect to elastic apm docker.

A

C

There's more to that first line, I think if we scroll over.

C

Looks like it just couldn't reach it.

A

Yeah I mean it.

C

Looks like a back end.

A

Collector mm-hmm.

C

I think we put question on it. You can take bug off a question on for now. I think it seems.

B

Actually, there's a waiting for author over here.

A

We'll do that I'll. Take the bug off. Okay, all right.

A

I guess I should probably not look for sponsors needed as well: anyways, okay,.

A

Hotel p receiver.

A

This client starts publishing, metrics and collector crashes.

A

Yeah, this definitely.

B

A

A

Seems pretty bad? Oh.

B

I see it's the remote right exporter right or the translator.

C

Translator prometheus remote right, yeah.

A

A

Oh, what I guess another.

C

Nested problem, we only go one deep, I don't know I would throw.

A

C

Put it on the exporter itself,.

B

Yeah, because that's what they're is that? What they're using.

C

I think I mean that's what they titled their issue.

B

I mean this looks like a p1, at least because it's.

C

Panicking yeah.

A

It's just gonna.

C

You can ping if you want to ping the people that own the translator prometheus there are that is called out in the package. That's.

B

A

Sorry in the code owners it's a dashboard.

C

But no, it's actually not anything. It's dashboard, someone named bertie sentry, who I I've not seen around.

B

I guess yeah, I guess that is the translator, that's it seems.

C

Like it's happening in there, probably.

C

C

You can also do its 30th century yeah. That person there yep.

B

Pinging everybody all right, yep.

C

I actually think that we've had more success recently getting issues and code owners connected. It's been nice yeah.

B

Yeah, I think it's been, it's been really good. I I like how much easier it is for us to also triage things like. I feel like not searching for people's.

A

B

A

C

Yeah, it's way better. uh I.

A

C

A

Any successful.

C

Runs of the adding labels yet, but I wasn't expecting that to happen very often.

B

Do we have a receiver prometheus remote rights.

C

Prometheus remote right receiver.

C

We have a prometheus exec receiver and a prometheus receiver and a simple prometheus receiver.

A

B

To remember, there was a.

B

I feel like this is not the first time this has been asked.

C

Exact receiver.

A

B

Yeah, I don't know I'll I'll dig into because I do remember this being talked about at some point. I don't remember what the outcome was, though,.

A

Aerospike receiver should support tls config, okay,.

C

Enhancement and uh we've got a tag for that priority. Three or two.

A

Yeah, I think I think all enhancements are probably yeah.

C

B

A

Be fine threes at the start, yeah yeah, all right.

A

Can multiple filter processors be used for pod level, metrics.

C

I don't see why you could use why you couldn't chain named filter processors. Together, you could chain any processors together.

A

B

I don't see a reason why.

C

It says it doesn't work and it starts, and then it starts working if you remove it.

A

My remove one starts scraping metrics nicely.

B

Yeah I mean it seems like if it doesn't work. It seems like a bug, but.

C

There might be something in the filter processor that says it has to be the last. No that's the routing processor that does that. Does the filter processor say it has to be the last processor in a uh in a pipeline, because the filter process or the routing processor, I think, gets all upset and I'll equip you about being the last one. It'll crash, the collector, otherwise.

C

I don't think the filter processor has that restriction.

C

It just drops data right it. Just all it does is remove data from the incoming payload.

A

D

You filter processors,.

A

With four logic, like can you say only.

D

Include things that match this or this with two processors, or is it going.

A

C

A

Everything that matching one.

C

Yeah it'll run the whole burst processor, and then the output of the payload would go to the next processor. Having that processor would have no knowledge of the one before and it would just do it again right. Okay, so I.

D

Think that's what he's running into yeah! So if these are both in the same pipeline, they don't have any overlap. Everything's dropped right.

C

Is everything getting dropped? Is that the outcome of the include statement.

D

I mean he says uh at the bottom when I remove one, it starts scraping nicely. I don't know if you know, if you interpret from that it starts outputting. Metrics like it may be, scraping with both of them, but he's just not seeing any output. So he assumes the scrape isn't happening.

C

Right yeah, it seems likely that it's dropping. Let me look at the how the include works there.

B

Right, like it's possible that it's just dropping all their metrics yeah.

C

D

Everything matches the first thing to tell, but if these are in the same pipeline then I would.

C

D

C

Yeah the way they include works is it says any names not matching. Filters are excluded from the remainder of the pipeline, so the first, the first one says: if you don't have that metric name, you're dropped and then, when you put the second one in there, the metric, the only remaining metric name has to be pod network rx bytes. But the second one says: if you don't have pod memory utilization over pod limit, then you're dropped. So the second one drops all the metrics that the first one kept right.

B

They're both includes.

C

Right, yeah, so that looking at the read me, the include, says and read me could be wrong: I've never actually used. The processor include any names, not matching. Filters are excluded from remainder of pipeline, so it's the include, is saying, only keep these metrics.

D

Yeah so assuming these are in the same pipeline, I think the behavior is what's expected. Yeah.

B

Yeah, exactly because you're you're, including like say, you're, including only the top, then you wouldn't have anything else.

C

Then you wouldn't have anything to include from the bottom one. The bottom would say only include these well, none of the ones that get spit out from the top are included in the second, so they all dropped.

D

C

Now it is looking at a different attribute, but uh that's irrelevant. I think right. The metric names is what.

D

Yeah metric names, I mean if it's not gonna,.

C

So for metrics in the include, you can use metric names and resource attributes, but I don't know how they're how they relate to each other.

D

Yeah, I don't either, but.

C

I think it seems likely that they're just dropping all their metrics.

C

And if we're wrong, then the filter processor owner can correct us.

A

B

So, what's the what's the work around, if you want to know multiple matches like if you.

C

A

C

Because I think I.

B

Guess I guess what they really want. Is they want to be able to match both of these right.

D

Yeah, a regular expression that matches both right, you'd use, parentheses and the pipe character to have like an or.

B

But this is matching on different values in the resource attributes right.

D

Oh and yeah yeah.

C

So that's why I think yeah. I think what they're trying to do is drop based on the first resource attribute and metric name, because it uses both.

D

I mean, depending which exporter they're, using uh export to cloudwatch. I mean it could have two parallel pipelines.

A

Yeah right with.

D

The same receiver, I mean I at that point: you're you're definitely off the beaten path, but I don't see any reason why it shouldn't work.

C

The the example in the readme also has two filter processors listed. um I don't know if the intent is that you would chain them together, so it seems likely that they're dropping all their data, but probably need pmm, sumo or or boost chicken to follow up.

B

Yeah yeah like basically what they want is they want to be able to put two to include statements.

C

Yeah, I should.

B

C

I should really add drop to the transform processor because the where clause it can be so complex, now yeah, but that's not. It wouldn't be hard to do what they're trying to do.

D

C

You could do this, though, right.

D

And then, if you added like drop, if not, then the filter processor almost is obsoleted in a lot of cases.

C

Yeah I need to use this. I I can't add: yeah exclude is probably what they want as well.

B

No, no, I mean they could use expressions.

C

Oh, probably, what's the expression? Oh, that's got it. Oh, oh cool yeah. They should.

D

Just use match type expressions, so you can can an expression use a um regular resource attribute no regular expression. Oh.

B

A regular expression.

D

C

It looks pretty static.

D

They also have attributes on both of them, but they're different attributes, so I don't know how they uh expect those to play together in the issue. Yeah yep.

D

It was a fairly complex if you tried to do it in a single filter. It'd be a fairly.

D

I think. Parallel pipelines is probably that the first thing that I would try here as long as the exporter, uh you know as long as the back end, that they're exporting to doesn't care that it's coming from two different exporters, then, who.

D

Cares because you should be able to use the same the same uh receiver into pipelines.

A

A

All right, uh so you can use um regular expressions as part of the expression, uh some text that that also could possibly work. But I guess, let's see what they're looking for.

B

Right, do you want to add a note in the in the issue about that.

A

B

All right processor is grouped by some metrics.

C

B

This one very well.

C

What's up, I said I don't follow, I'm not following this issue. I'm not comprehending yet.

A

I think he's looking for like a group by and like the transform processor.

C

Yeah, I can't do that yet and in fact the transforming processor grouping is a stateful thing right.

A

B

I have a level of granularity too low. That means too many metrics.

C

That's what the group by processor does.

D

It combines yeah, you take a bunch of of counters or sums and then like group them by. I assume a uh an attribute right.

C

Whatever consumer group is in his.

A

C

Yeah, can it already do that? I think you can there's a group by attributes processor.

B

There's the group by attributes processor.

C

D

Yeah, I would have to ask right.

A

A

A

Oh yeah, this one's mine, I love it yeah five minutes left.

B

um I don't suspect, we'll hear back from the code owner because we've painted them for weeks and weeks now so um well. Some.

C

Of those issues were years old, some of those from last year weren't they.

A

B

So I think I think this I'll just.

B

I don't know p that p1 or p2 p2.

C

Okay, we could do p1 for it on min 10. I think that could also make sense, but it's not a bug, so p2 is also yeah. It's not like it's actively breaking people's instances.

B

Except for the poor contributor, that's been sitting there waiting. His message was like hey, hey, coco. What do we do? What can I do here? I'm, like I, don't know man, nobody. Nobody said.

A

You can take over here, unfortunately,.

B

That's kind of what I said: I'm like hey is this something you're interested in because we're looking for contributors. Remember.

C

And take care of it yeah.

A

Yeah, uh so this one's already assigned to all right, this one's looks like it's already, it's already being fixed, so I'm not gonna. Let's plug that.

A

P2 p2 is good yeah, pretty standard.

A

Okay data point: you set one new attribute value. Why.

C

Does the prometheus, why do all the prometheus components show up so often in our issues list? Is it because they're the most used.

D

A

That's what it is, there's.

D

Also, a lot of them.

A

C

Nice, oh wait: this one's done. You can close this issue. This got merged. I think I'm aware.

A

C

A

B

C

Those are my favorite yep. Those are the best ones yep. That was actually a pretty good bug find because there was another pr open recently for the cumulative delta processor and the person was like. Oh wait, I think my change, but then it turns out it was that bug. So two people ran into it this week so happy that that's fixed.

A

Yeah and count sampler throws no pointer exception. Well, that seems pretty terrible. Yep.

C

That's a panic, they didn't say it, but that would crash the collector. Wouldn't it.

B

Is it they're using the account sampler spent count sampler, along with the tail sampling processor,.

A

It seems like a.

C

B

P1 is like things are crashing. That seems like a good definition for p1 p0 is like this is actively stopping.

A

C

From starting the collector, I think p0 would probably have to go on a core component. I.

A

C

Know if we could put a p0 like on a on a on a regular component yeah do we need to is do we need to add another label? There's a tail sampling processor on here. um The span account sampler, it's all. It's all the it's all just the tail sampling process, so the span count. Sampler is one of the options. I think it's the tail samsung processor.

C

So if you use that with an and policy now, I thought this got fixed, though hold on go to the pull request, because there was a pr recently in the tail sampling processor to fix some stuff with ams.

B

All right, because they're using a slightly older version.

A

A

C

Maybe it's maybe it's still open. Oh wait. Is it that one that's been open forever.

C

And we're waiting on anthony to respond. I think.

C

No, I mean it could be that one, but that's not what I was thinking of.

A

So there is a pr open for this uh 11 534. It looks like.

C

Oh wait: this is the same dude. I asked him to write a unit test and he was like yeah sure I'll. Do it.

B

Yep yesterday he said yes, I.

C

Can oh, but but isn't the person that opened the issue.

A

D

He said in the issue that he was creating a pr.

C

So they're just not linked.

C

It's hard to keep all this stuff straight.

A

Okay, this whole all right, so this one's already josh we're good and we're at the time.

A

All right thanks.

B

A

Thanks everyone thank.

B