Open Telemetry Uncategorized, 8 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-12-08 meeting

Description

Instrumentation: Messaging

A

Hey everybody, I I, wonder whether Ludmila and Johannes will make it or not. um I, don't see any comments on the stock Channel or the dog. Do you have any diameter.

B

I, don't know they didn't mention anything.

A

Okay, um let's well it's five past starting time: um I, don't know! Maybe yeah I guess they're not coming. You know it's Christmas season, almost so I guess they are probably busy um yeah.

A

Maybe we can discuss, uh you know like Tyler and me we are getting involved in these well Tyler he's mostly just to stay in the loop, um but I I am, as you know, trying to catch up and review stuff um and I see a few items in the gender. Do you want to discuss anything with us.

B

um Let me open it just a second perfect.

B

Understand Edwin.

B

Yeah so there's the attributes PR by Luke, Miller, okay,.

B

For somebody convention yeah.

A

The fear is I think it's Dawn I, think that it has found um agreement in this working group. I. Think that you are, you left some comments so um but other than your own comments. uh Amir.

A

On that front, my plan, my the plan for myself, is to finish reviewing that it's a BPR I'm trying to go over everything um but yeah, hopefully I I will I will approve that or just provide minor feedback, but the plan I guess based on the fact that there's already four approvals from this working group is that we will try to measure it very soon. So we are doing a specific release today. So I don't think this will go into that just to be on the safe side.

A

You know, but I can imagine this won't be merged. You know the the earliest tomorrow or yeah and uh yeah I. Don't know whether that works for you.

B

Yeah until it's not released, it can't go into instrumentations.

A

Yeah it can yeah I think it can um usually I think. Well, that's not the instrumentation maintainers, but usually what they have done um is that as long as a PR actually got merged in the specification they they start using. That there's always a risk that we know before the next release. We do some changes and you know this can still change, but as long as you don't like, let's say you update your instrumentation code and you don't release it till the next prescription. Release is actually a release. You should be fine.

B

A

C

Hi everybody I saw you.

A

B

C

Discussing so that's great I feel I will share I think uh we need not to wait for a draw because he's on vacation to speak, but otherwise, since we have Coral.

A

Yeah, okay, actually we'll meet as well, so yeah I will. Let you drive, then the the conversation I'm just here to stay in the loop. You know and hopefully merge the first PR later today. It's a big PR. So sorry for the delay. I am almost done. Reviewing that.

C

Awesome, we are glad to have you here Carlos, so.

D

C

D

Reviewing it back.

C

So and that's I mean that's already. uh The first point here on the agenda. I think you already talked about that. There is let me list PR. We have I think most people from this group have uh approved already from Amir, not true. If there's open commands from your site, I didn't look at that.

B

C

B

Didn't have the time to approve I will do it after the meeting. Okay.

C

Awesome so we are kind of we have like more or less Anonymous approval from our site and then you're just waiting for uh I think Carlos. You approval with lockout at least two of those that we need here. Yeah.

A

C

Awesome so so much for this PR, then uh somebody put this on the train the last week, I'm, not sure whether it was you look middle or somebody else. That's some kind of uh I I didn't have time to go of it in detail, but it seems to be uh like a timeline proposal of getting the semantic conventions table from from Ted and I didn't go through the details.

C

Maybe shall put it on the trender because he he seems he left some comment here, but uh yeah it seems, there's a goal to get all the semantic conventions stable until July, 1st 2024.

C

B

C

Sorry go ahead, I'm.

B

Here, second, in a year and a half right.

C

That is in a year and a half for all the Romantic conventions, but yeah I think there is several iterations here and I I think those semantic conventions are kind of uh that. The plan is to get those stable in the first in the in in the first group and I'm, not sure, there's no time it says current the true if they should be stable immediately but I guess there is some uh pressure here on the messaging uh side or the for Kafka and revenue that also pertains to messaging.

C

So uh I, don't know about this church Justice currently I, don't know about the exact timeline here and we can check back with that about details. I think we should do that, but I think we are anyway in a uh with the attributes, PR merge. We are in a pretty good position uh and we made like I'm, making a big step towards uh reaching a quality power that part that can be declared stable, so I I will look into this document. uh I will read through maybe leave some comments, if necessary, also on your site.

C

If you're interested have a look and uh leave some comments on this.

A

C

A

I can go ahead. Sorry, yes, Carlos, go ahead. Yeah yeah I think that well, first of all, I think that the Korean um stage that it's mentioned there is that that's in progress either been planned or discussed at the moment, um so messaging domain. I. Think I can imagine that this is what you know.

A

What a team is focusing right now, but I wonder how's the uh the interest of working right after on Kafka rabbit, mq I think they are, of course Very related, but they have I, don't know whether we have an expert, for example, on those on those Technologies like because this document takes for granted that we can find. You know, people who actually know uh you know very deeply what's happening and for Capital gravity, mq I, don't think we have people or correct me if I am grown.

C

No, we don't really have any uh any Kafka and private mq dedicated experts, I, think in this group, currently or or and and if so, I I, don't know about this expertise. Maybe they are I know we have like. We had some participating from the rabbit and queue side. I think I forgot his name, I, think he's also in the in the early line, sick and I think we could get some rabbitmq knowledge from there for Kafka yeah, I'm I'm, not uh I'm, not really sure so the area. That's a good question.

C

How people then procedure also from messaging side I mean they are? We are currently mostly working on the stuff for tracing I, mean there's also still a gap for generic messaging uh metric semantic conventions. So maybe for this group that would be a next step actually before going to the more specific cup, correcting Q Parts, not sure what kind of the TC stands on this is, but currently yeah there is for for messaging metrics, there's nothing there at all.

A

Yeah, that's a good point. Well, I mean first of all, this is a proposal and I think it's a great proposal, but it is still a proposal so yeah, actually that's a good thing.

A

um Yeah I think HTTP is the the one of the few ones that already has a lot of metrics coverage, but yeah you're, right, okay, yeah, okay, I, will bring this with uh up with that. We can either comment there, like hey, like um yeah, like two address metrics.

C

Because, at least from my point of view, I think, with the current uh composition of this group, I, think the general message Matrix. That is an area where we would have more uh definitely more expertise than into a very like deep calf, correlating queue specific stuff.

C

C

Site and also try to join one of those, uh so many convention seek meetings on Monday.

C

Okay, that was for this one then. The last point we have is the next step after the attributes, PR, which is yeah hopefully killstobing. Merge is talking more about the Sprint store, current links and the we started talking about the amir's document last week. I'm not sure how many of there have been any changes on your side, but I would definitely to Dave would love to come to some comments that were left on the span. Structural tab.

B

Yeah I walked a lot on like writing it in a formal way. uh I have a document. It turns out to be really big because I described all the decisions and why we made them. It's not.

E

B

Yet but I hope I will give you a like a draft to so you can take a look.

C

B

C

This document here that is linked here, it's a different one,.

B

It's something that I work on: it's still a walking Focus get it finished soon and Consulting.

C

So maybe let's wait until you have that you know state that they can review and look at some comments on the span structural tip today. If that sounds fine for you, yeah.

C

Because here that's this, uh that we had that we had our big old tab that was split up into like attributes that little winner was working on and this one here that just covers this band structure, and this is also a bit left aside until now. The span structure single after later, but I think now that we are coming closer to finishing the attribute stuff I think we can.

C

uh We can continue discussing on this one, because there have been some comments and I think the most important one or the most interesting one was those were comments from Miller uh and I think the most fundamental ones where uh I think this one here, because we in this current document that was kind of uh concerned was more or less that we came in previous discussions. Is that we settle uh what did we agree on having exclusively links for correlating producer and consumer operations or traces so in the current state of this Otep here?

C

What is recommended is that consumer and producer always separate faces and are linked together and uh basically, what loot wheeler brings up here is. There is some need to discuss it again, because there is obviously also pressure from customers to go more towards the direction where we actually have parent tribe relationships in scenarios where this is possible, because in some scenarios multiple traces can be confusing, not really wanna kind. Of summarize.

D

uh Yeah so like uh a short history, so we supported uh links for messaging scenarios to measures the case. What we found out that every once in a while there was a customer who comes and asks what do I see multiple traces for this operation.

D

um Yeah I'm, always sending one message and always receiving one message: I cannot query uh things, don't work or what happens? What are links and I found that I'm trying to explain my technical difficulties and staying like closer to the spec and but customer experience like this left side. So uh recently we updated it. We got some feedback from customers that when we have just one message, we use parental relationships and it makes customers happy and happy.

D

Customers are more important to me than following the specs, so I suggest we can at least make it a reasonable experience like make it allowed from the Specter perspective.

E

So I have I have one question there. I've been thinking about this same kind of problem and and I agree that uh there are certain circumstances where having it as a single trace is uh certainly beneficial.

E

um One question I have, though, is uh across the various uh uh tracing back-ends or vendors.

E

Is there a time limitation uh after which uh spans cannot be added to a Trace so like, for example, if, if there's a delay in processing a message such that you know it's not added to uh or it's not processed until, like you know, 15 minutes, 30 minutes, you know a day later, uh is that span gonna be able to be added to the trace after the fact.

E

And I guess uh my suggestion, then, is add a an additional level of uh criteria to your suggestions, such that, uh instead of just looking at. If there's only a single message also add a criteria around the time between when the original, when the message was created to being processed uh and if it's exceeded a certain certain amount of time, then I think it makes sense to try and create a separate Trace, instead of attaching it to the original Trace.

D

Well as like as a backend, they have no idea right. I'm, sorry as a client Library as an instrumentation I have no idea when the producers plans were sent, um I can try to figure out, but it won't be reliable but at the same time I don't believe this problem is specific to messaging scenarios, or you can have very long Trace, even with just the parent trial relationships without messaging, um so I think uh backhands that do this and they don't.

D

Maybe they have a better experience for the shorter periods of times like buffering or doing something. But I. Don't think this is a great behavior for the background to just drops fans that are that came after um so here, I would rather uh advocate for the back ends to gracefully accept long traces for all cases, not just messaging.

C

Yeah I I think the title's question I think this is a very backhand specific question. What I can do with those later writing spans? I mean I can't just say like give from an example from the one I'm working on and therefore like the usual 26, but the usual like experience, you can add spans at even days later that that works to trace I mean various limitations is when you have more like some specific analytic uh mechanisms that work on traces. For example, when you say, okay create metrics from traces.

C

There often is a Time window where your aggregate racism that, after this timing, you basically you then create like this metric data point based on the trace and every span that comes after that will not be considered. But uh it's again it's a very back-end specific question and there might be lots of variation there and also agree with that.

C

Miller I think it's it's uh doing like uh following his proposed and saying: okay, we not just take into account whether it's a single message or not, but you're, also taking into account the latency that we have between producer and consumers.

C

Think it firstly makes life for instrumenters much harder and in some cases you might not even be able to determine this latency and also, if you are able to determine this latency, you might still run into viewed edge cases because you have, you might have a single message, but the single message might be consumed by multiple consumers and one consumer might consume the message. One minute after it was produced.

C

Another consumer might consume it an hour after it was produced, and then you would have like a weird state where you have one parent child relationship and one link and the other consumers in many cases cannot know from each other.

B

Inside an endo correctly, we talked a lot about it in the past.

B

Yes, what our conclusion was if I remember correctly, is that in the case of a batch, because in the case of vegetables, no way to do it at balance child because because of the Fannin so batch requires the links and I think what we agreed for a single message both are allowed so I, don't know what what will be the default, but you can either represent it as a link the same as the batching scenario, or you can do a better child relationship and maybe some everyone can configure their instrumentation to to do it.

B

The way they like with the pros and cons.

C

Yeah I mean we did not 100 settle or anything. Yet that is right. It is still like order PR, it's not just a work in progress, but I I remember there was a this. This PR at least went the route of for now, just sketching it out with links exclusively and I. Remember, like two reasons: I put here uh by the event, is throughout the first consistency so that we are consistent over our messaging scenarios and for and for backend.

C

Basically, when you see uh when you see there's okay messaging space, you know okay, I I followed this special structure of links and I. Don't need to consider like to do anything else.

C

Also, it's kind of consisted in ways because there is, uh there are already scenarios even for a single message uh for the single measure message: scenarios there are scenarios like the pool based consumer scenarios where you have for receive operation that basically pulls spans from the intermediary, and also here parenting might be hard because the receive operation could already have an other parent context.

C

That is there, so here uh there might be the of course they can go with the route and say: okay, we use you can use parents wherever possible, but there is kind of even a single messaging scenario. We cannot say: okay, we use we use parent trial relations tip for our single measured scenarios, because even there are variations and then other other reason- and we talked about a lot that in the beginning, I remember is that we are keeping possibilities open for intermediary instrumentation.

C

Basically, when you have the producer and the consumer to have a private child relationship there, it's harder to fit intermediary instrumentation into into this picture, and if you have only let's say partly intermediary instrumentation only having the dependency environment type relationship might even break that in some instances and then, instead of linking to the producer context, you link to the intermediary or your parent to the intermediary context, then you might want to risk that the intermediary context is not kind of correctly parented all the way back to the producer context.

C

So that wasn't another reason that I remembered why we came up that we came up with this just linking scenarios, however, I think when we can come to some kind of a uh solution for those points. I agree with what Miller that for many, especially customer scenarios, just having the Bond compact Trace, where everything is in might be preferable.

C

I mean one thing: just I thought about this. One thing that I was also thinking about and I'd, be curious about people's opinions here is that one thing we could do is that here in the spec or in the semantic conventions, we basically say: okay, we recommend, or you should add a link between those between the producer span, the publish span and the consumer span to receive span or the delivers.

C

Then I think that's something that we basically could put there's a should, but in addition to that, we could give the freedom to allow to uh add a parent trial relationship. In addition to that, because then you would have the consistency of always having the link uh in certain cases where it's possible, you can have the current child relationship, you have it in the same Trace and if you have some intermediate instrumentation that you need to fit or that you want to fit in between then yeah.

C

The current child relationship will be broken up, but you will still have the link that you can follow. I mean it's a bit weird. You basically have then two spans and you have both. You would have like the link and the permanent file relationship between the same two spans, but guys it's just something that came to my mind and I I would wonder about people's thoughts on that.

B

Yeah I really support it. I want to add that when processing expense, uh if you have to build like the entire Trace, if you have to aggregate the spans and build the trace and be able giving an arbitrary spent to tell who his parent is and what attribute it has and derive things from a parent span, then it makes processing way more complicated.

B

If the link is always sitting on the spend describing what it links to, then you can just by examining a single span. You can know the information that you want to get so it does a lot of benefits to it for processing and visualization like if.

E

If the parent span.

B

Is for some reason lost, then you can't tell anything about the consumer span right, because you can't go back to the parent and examine it and understand things, but.

E

If you always use links.

B

Then everything is much more uniform. In my opinion,.

D

uh Can you elaborate what what's better with links.

B

D

For example, today.

C

B

I have a trace and I see a consumed spell right and I want to know who sent it. Then what I need to do is I need to um to to look at the parent, spend ID and examine it right. So if, if I'm holding the consume span, I see the parent I go to the parent I check, if it's a published span, I check if it makes sense that it's uh what I expect it to be and this process while it is possible, it's more complicated than just looking at the link.

D

Right what links give you gives you that parent doesn't.

B

So it gives me the links attributes which are always present on the consume span. So if I lose the parent spend like I, don't know anything about it. I still have the the coding of data in the consume side, for what is links represents and then also it makes it easy and because because I don't have to complete the entire Trace I can just process spends one by one I, don't need to Aggregate and be able to jump from one to the other. So.

D

B

D

This is the presence of attributes. It doesn't really matter what their relationship are is for the single case. We can put attributes on this plan itself.

F

B

The ways to to address this but I think it will be very um easy if the link pattern is always used like together with the parent child or or only link, but if we know that there's always.

F

B

Link, it's um makes it a lot easier.

D

Like link combines as parent child in good, but if, if well I, think our idea about attribute so far was to put attributes for single message case on spans anyway, and then would it be beneficial to have a blink? Was the context and parent-child relationships and all the attributes and spam.

B

Thank you so that we, if I, remember that what we ended up thinking was the best. Is that, because sometimes you have a mix of a batch and one badge in the same instrumentation Library, it might be desirable for a instrumentation library to follow the same structure for both single and better options.

B

So what we talked is that we we don't want to not allow one of these combinations, so instrumentation.

F

B

Use the link and store things there or we can do it on the span, attributes.

D

Yeah, let's say: let's take a simple case: um instrumenting Reddit and queue which I think does not support batching at all, uh and then what strategy would I pick the strategy? Okay, maybe if I could attribute some links? Okay, if you allow both right if I put attribute some links, then yeah create links, but if I decide not to what is the benefit for me to ever, create in colleague, yes,.

C

B

B

Yeah, so in this case uh there might not be a real big benefit, and but it.

E

Means that if I write the.

B

Code that the bosses dispense then I have to uh like ask myself: if it's this, if it's the link option or the parent child option and have a branch of different ways of processing them, and if I only go by the link, then it becomes a half to walk.

D

Yeah so I think we discussed that rigor for the back ends. They don't know which strategy instrumentation library takes right. So for the back ends there is a the algorithm is merge, Spann attributes so as link attributes. The combination that Union of this two represents all attributes for specific message.

D

uh In this case, it's the single Branch, your your merge and then you process it uniformly for any possible scenario.

B

Yeah but let's say you want to you, have the closest spend and you want to know who published it like a consume, spend and you want to know, publish it I. Then you have to go to to go to the parent and check the attributes to see if it's a published, spend and.

D

Then you know why.

B

Because you, you can't tell if the parent is the publish or not right.

D

So assuming I don't have a parent at all right, I'm receiving good from it's. My public API I'm receiving things I want to put all the information I need on attributes anyway on link or understand right so uh and then the question is: where do I put them? Do I always need the link or.

B

D

I, don't think the.

B

Specification should restrict it you can put in whatever you find it more suitable. But uh let's say the you get like a trace from a messaging.

E

B

Don't know you don't uh prepared and now you see um a consume spell and you don't.

E

B

Anything right, so how do you know who published this message? You have to build the trace right and go.

E

B

Parent and check, if it's a published pen, it might be a published pen. It might be something else right.

D

Yeah, but uh if we use this experience, if we need to get the parent or the link spend, there is no difference between spam and Link right.

B

D

Right, so, if you need to check who published the span, you either follow a parent or you follow a link.

B

Yeah, but if if there is a link, I know it links to the public spell right, because there is semantics for it. But if.

D

B

D

B

A parent I need to visit the parent I need now to distinguish between a case that a parent is to publish which is possible and another case that a parent is just something else. That's also allowed right, maybe I.

D

B

Have a link, I have a parent, but the parent is the ambient context or something like okay.

D

So what you argue for is that for backends, if there is a link, you can all it doesn't tell anything about parents, but having both parent and Link can be beneficial because then the best and why it should simplified to it.

B

Yeah, that's why we can tell if the parent is just something arbitrary or is it really the the publisher context which we are.

C

Just to jump in here I think we have two like two viewpoints here that are not I mean not necessarily conflicting, but I think there is the Viewpoint from basically the back end point of view that Army product. Of course, we already agreed that we say: okay, when we have this consistent way of correlating those producer and consumer spans, then it's easier for backends, because there's just one way to look at things.

C

You have the link, you follow the link and you know uh uh you can always basically follow the same pattern where when it can be links or it can be parents, then you are kind of already have this fallback that you need to consider and there's some kind of caveats that come with that and I think what Miller says it did it from customer point of view uh it, regardless of any backlink kind of complications.

C

There are cases when having everything in a single trace in the same in the same Trace with the parent trial relationship that this is uh more user friendly for for many customers and I. Think that's also a valid point: I see a charge. You have your hand up, go ahead.

F

Hi, first time caller here, I've been listening. I think I understand I just caught um this question that I'm hearing back and forth I wanted to try and clarify um what. So when you have a parent and and Amir has been using.

F

The word published, uh you don't know whether the spam is published, I, I, think of the word exported or or sampled being other other words for this probably the same thing, but when you have a link, what what's to prevent the link to being to something that wasn't sampled or wasn't published, I I, don't I have never heard that rule before um and it caught me by surprise.

C

Well, I think we might have a slight misunderstanding here, because when we are using the term publish we are using. We are referring to the messaging operation of publishing a message to the broker, that's kind of what we're using as published, and it's not it's not related to uh to whether the parent context for sampled that's a whole different kind of norms that we didn't fully open. Yet.

F

Right I wanted to invite everyone here to there's been a beginning of a discussion in the sampling stage, which is a week from today. uh In the same time, slot and I loved all if you're interested in sampling in links. We should talk about it in a week. I was here and listening to all the other complications that come up in messaging and I can see.

F

There are a bunch that are not related to sampling, but I, but I feel like, and this might be a controversial opinion and I don't want to offend, especially with Mila but I'm wondering if, like the it's the product that the vendor sells, that determines whether the user is happy with links or not so there's a classical Trace product out there. That has a lot of history, which is traces, are tree.

F

Data structures- there's always a parent trout and if you use the parent child, the product works really well and if you don't you're doing something that doesn't fit the product and the user is not going to be happy if the product doesn't fit messaging and and this effort to squeeze parent child back into the picture is to make the trace look like a tree again, so that the product works I'm, also representing a vendor that doesn't have first class link, support here and I'm, actually pushing back on that.

F

I think that we can do better and that we shouldn't be afraid of links. However, they do complicate sampling and, in the the thread that's been discussed in the sampling, Sig I believe we're starting to see a deficiency in the sampler API the sample, the sampler API being deficient, because we don't have a way to um and I think this might help with amir's concern in today's sampler world.

F

We don't have a way to say: I, don't want to trace my operation, but I need I'm part of a link that needs needs to be assembled to somebody else's trade um and uh Jesse. I. Think you're unmuted.

C

If you're listening Jesse, can you mute please.

F

um uh So so, if I say something informally here because formalizing, this is not actually not easy, but if I say something formally like when you link to another span longest man just so that the the back end can Assemble the trace that it had, which was which terminated in a message publish will say, but but there's this extra span.

F

That was part which was created as part of a new Trace, because it's a graph not a tree and that that new that new spans not could be sampled, now there's an opportunity to completely lose information at that point. I had a span links to something I'm interested in collecting and tracing, but I had decided not to sample it.

F

So my I've broken my link, and so the discussion I want to have next week in sampling stake is how we can avoid breaking links um and I think what that means is informally log the span, even though it's unsampled, which will give you a span object, you can staple on to another Trace.

F

That's the highest level idea that I I think we can save links here and then I, don't think it should matter whether you use a parent child or or a link, except, as you point out, there are attributes on links and there are not attributes on parents should children. In both cases you should be able to tell whether the target was sampled or not and I think we're just afraid of links that started my Outsider opinion and I'd love to have this discussion in the week at the sampling thing, I.

D

Would argue that the.

E

Other thing that you lose by by using links and not parent-child relationships is, uh is something like baggage uh baggage would also be lost over that uh that method.

D

Yeah and also the experience users pointed out, I agree, uh I'm, not afraid of things. I came up with using links by default everywhere, so uh but the consistent feedback is uh that it's not that the users. Yes, they don't understand links, but they don't have link related scenarios. In fact they have a tree, but we represent it as a graph right and what they want to have.

D

They want to query, and regardless of the visualization, some nice experience we create, there is I, think are important and if we um the moment with the links and down the parent trial relationships, it becomes like twice as complicated uh and still we are still in the world where no back-end supports links well and I would rather listen to my users and do make them happy with the current products and later on figure out how to work with games better.

C

Yeah I, really like your question that you asked, and you think maybe we can think about this one for this because I said: okay, we have I mean we have scenarios where we need links. When we have a batching scenario. We cannot go without links because we cannot have Market preparements, but there is basically those things, those those scenarios where we could have both we could have a Parent Trap relationship and we could. We can't have links those.

E

C

The single message: scenarios where there is no batching involved and I I, think a good question of your brought up is if we go with links or for those scenarios. What are the benefits that the customers would get from that because the benefits, as you mentioned before for parent child, are clear. They just have all the information in single trace, but other cases where even I think that's a good question, a single message scenario: customers would prefer a link relationship to the parent trial relationship.

C

Are there those cases and I I think at least one case uh one could bring up this year if you have the intermediary instrumentation present. If, for example, you have a rapid mq scenario and you only have single message scenario, but you want to also see kind of your broker trace this kind of integrated into your whole Trace picture and I. Think in that, in that case we don't yet have this at all for, like we don't have any messaging broker in the media instrumentation, but I.

C

Think in that case is having a link between producer and consumer is kind of beneficial, because then you can kind of turn your broker instrumentation on and off, and it kind of is supposed to integrate well into the existing kind of two link traces.

C

Whereas when you have a parent trial relationship between the producer and the consumer, Trace, it's gonna be harder to fit like broker instrumentation in between there I mean you might then actually Force the broker kind of either to break this parent tribe relationship between producer and consumer, or you might Force the broker kind of to link somehow into this parent child producer. Consumer conglomerate.

E

I would also suggest that there are. uh There are cases where the API would tend to uh imply that there's batching, but, logically, uh from a from a user's point of view, uh it should be treated as uh as not batching so like, for example, with Kafka system.

E

uh The API is designed such that it encourages batching mainly for performance, but most users still treat that as if they're processing each of those messages into independently and individually, and so just because the uh the API says that hey this is a batch that doesn't really mean that the the user thinks of it as such, uh and so uh I I think that we need to be careful not to uh go down the uh the approach that, oh just because it's a batch means that uh it should be treated as such um I mean I.

E

I. Think we really need to be looking at. Where is the affinity for a particular message? Is the Affinity uh uh for that message with the the current batch and treating it as a batch and uh logically thinking of it as a as a whole unit? Or is the affinity for that message with the the the remote uh the the producer and uh in that scenario, I think that it makes sense to have the the parent of that Trace uh belong with the the remote uh producer and not the current batch? Does that make sense.

C

When you talk about remote producer, do you mean the message producer yeah? The message.

E

Producer uh and and I understand that uh what I'm suggesting is is difficult from an instrumentation point of view. I I've written many uh of instrumentation over the years and uh you know being able to take a list of messages and split it up such that. You know that okay, this is the the current message that I'm working on and Associate time, processing it and and such that that's difficult uh when the the API and uh uh is not designed to really make that instrumentation easy.

C

But then you might have a badge that contains messages that come from like several Upstream remote message: producers I.

E

I yes, I, understand and so like. For example, when I wrote instrumentation for Kafka the way that I handled this is I I uh for specifically Java I instrumented, the uh the iterator and such that okay. For each of the messages in the uh in the list, I'm going to treat them as independent uh handling uh and and parent that, according to the message uh from the remote and not treat it like a batch just because most Kafka doesn't think of it. That way.

D

And there is a tricky part here where, uh if you write an instrumentation with something like Java byte code or monkey patching you, you have some freedom to uh rewrite things, but it was native instrumentations. It can be difficult and like if we want to produce some instrument. General purpose, instrumentation uh instruments and iterators can be well scary.

E

Right, especially because the apis that these uh libraries have designed don't really lend itself uh to to the kind of instrumentation that would be ideal. Yes, so I I sympathize like I'm just trying to make it clear that I understand the the the concerns and the the frustrations with the uh writing. Instrumentation like this.

D

I I guess our uh way of thinking and correct me. If I'm wrong was that as an instrumentation in a spec, we can uh have the most common denominator, it will receive the batch of messages. This is a batch. We receive the batch right and then users are free to create individual processing stats or maybe instrumentation libraries can do on their own if they want to right and then we can kind of have the Both Worlds, but in stock we are defining this most common denominator.

D

C

Talk about this place when you, when you receive a patch. Basically, you have just a single span uh kind of modeling that operation that receives the patch, but we would still basically currently we still put it. There is a short that we are one shot. This band should link to all the different producers for each basically, which requires more or less to look at each single message in the batch to determine who are or the different producers for those messages.

B

C

B

We all agree that it's sometimes users will want better and child relationship, and sometimes user will want the links, those um rationales with both of these suggestions and I. Think if we just don't limit it, we like the instrumentation, shows what to do or or be configurable, so it can do either this one or the other one I think it's a good way to go, um but I also think that also, if we capture it as parent child, we can also add a link like adding a link doesn't mean that we don't do better child.

B

E

Can say that so I I agree like in the case where you are uh doing the uh the parent-child relationship for the message. I think that it should still have a link to the batch Trace, like uh one option, does not necessarily preclude having a link to the other.

C

Yes, that's what I put out here I will I would be interested in your take on this. Look, we love when we say. Okay in this semantic conventions, we basically say: okay, you should create a link that you should always do when it's possible and you can, if that is beneficial for your windows in this particular scenario, also add the parent trial relationship. What that do you think that would work for the customers that you that you would have in mind? Do you think that would be a feasible path to follow us.

D

uh Yeah I think this would work and I I agree. We can do this that doesn't have any uh uh I'm not opposing this I just have one concern that we are duplicating things right and I'm I agree. We can do it not a big deal, but if what like I've still don't understand, why it's so important? Why can't you just stay with parent child? In this case?

D

Like? Can we do better than this.

B

Because how do I know if I see a parent? How do I know if it's the publish or not.

D

If it's a consumer's plan, its parent should be published.

B

Yeah, but maybe this parent is like an ambient context that we captured while uh calling this suit like there could be um I. Don't think we need to encode this restriction into the specification, because I.

D

See I see what you mean yeah, so basically, if for some probably invalid, but still possible reason uh receiving happens within some ambient contacts. You want to capture those the ambient and the link right, and the only way to do it is uh yes. I agree. Okay, this is a good one.

C

Just to this point, I think the ambient as I think we have. We have come across some good point here, because both of these kind of points here basically point to the fact fact that there might be another ambient context already active here in the receive operation where basically to receive operation is triggered uh to receive messages from from the broker by some ambient context more likely. So there might be an ambient context, and also here basically when we want to link to or when we want to include the intermediate instrumentation.

C

Also there might be an ambient context that kind of links or that kind of parent trials back to the intermediary. So I think that is uh Theory I think we have nailed down a case where also for single messaging scenarios. Links might be particular, and that's the case when there is an other ambient context present that we don't want to lose.

E

I I think there needs to be a distinction between a receive, for uh you know as trying to get a response from a prior message that was sent so so that's when an ambient uh span would definitely make sense and having a link to the the remote uh definitely makes sense.

E

But there's also the the case that I think uh having like a message listener, where uh the affinity for the the trace I think makes more sense to have the uh the remote uh um sorry, the uh the message uh context, uh the the parent, the the producer of the message, be the the parent of that Trace um yeah, just just a distinction between a receive uh versus a message. Listener.

E

C

F

E

Some other people joining the call here soon for the next meeting and just a couple minutes.

B

But I think if, if we get messages by pulling or by listening, so I graded, those like um maybe in one option, we don't need the link and then the other. We do need the link, but the back-end poster, so they just get spends and knows nothing about them. It can tell if this span is uh generated for medicine pattern and that's why the parent must be the publisher uh for something else.

B

It has to have like logic, to examine the parent and understand what it is and if it makes sense that it's the publish, yeah.

D

What what can we summarize, this discussion with I think two sentences? The first one blink is a shoot or maybe even a must and then instrumentations may make uh also uh use received context as a parent.

C

Yes, I think that is currently that's, not even that's not blocked by the current version, so the links are uh should and here's we'll say the okay. When you make this link, you must not parent with switch, but I can I can add this SMA that is located. You may also uh add uh this remote producer context. As a parent, if possible, decide. I can add this wording and then maybe we can continue discussing on top of this mental model.

D

Yeah yeah and also I think we we need to say that and then contacts takes precedence, so you cannot said parent if the reason I've been in context, it's quite it's kind of hard in too many conditions, the blood see if we can make it clear.

C

Yes, I ever think about that. I can either write that yeah uh that you may set this as a parent, if possible and just kind of leave all the details out but I think in the Otep we can definitely add more details, so I will I. Think I will elaborate a bit on that also the capture, what we what we discussed you now and then maybe in the next session we can.

C

uh We can continue discussing that so for for next week, uh as we are, we are almost out of time here actually was wondering as uh Josh said, that in the in the sampling sick, we are talking or they will be talking about links uh and sampling, uh in particular like what also pertains to our use cases here. I wonder if we should uh kind of move the the messaging this messaging work group meeting next week, so that we can participate in the in the links discussion for sampling. What do you folks think about it?.

B

Can we move it one hour ahead, so we have the sampling and then the messaging.

C

I think one hour ahead might be rough for West Coast Forks, because it will be 7, am I mean we could we already. We already had it on Tuesday some time ago. That would be an option that we say Okay a damn PST, uh uh no events that we moved into Wednesday did you say Okay, we move it just uh we make it uh on Wednesday next week this messaging work group meeting and on Thursday we try to make it to the sampling discussion so that.

F

Does that we could also just try to split the hour or something like that, like a sampling group is small and there's only one topic right now. um It could just be this group discussion with the sampling box present I can try and redirect them to your to your Zoom. Even there's only about five of us.

C

You can do the two so that we basically meet in this. In this.

F

I think this we can try to use this Zoom I'll I'll, try and get that to happen. um Okay, I'll bring the sampling crew next week, um since this is a weekly meeting. That's in every other week meeting um and it's great I really learned a lot from listening to this conversation. No.

C

Thanks for waiting.

F

Also 220 has got amazing diagrams. These are really helpful, um I think in the context of a Spam link sampling conversation I will be using those diagrams. Oh wow,.

C

So you're very welcome awesome.

B

So there we go, I will be on vacation on the like in two weeks and the three weeks from now. So I will join next week, but then I will not join for two weeks and I want to ask. uh You also have holidays right.

C

Yeah I think we will discuss that next week. Then what we do our holidays, I think at least for maybe we will make a break for two weeks or we will need to make a break for two weeks anyway. I will put that on the trend for next week to discuss.

B

And just a heads up I'm away next week as well.

B

C

Awesome so thanks everybody that was a great session thanks for all the people showing up and see you next week, when we talk about links and sampling.

D

C

B

C

D