Open Telemetry Uncategorized, 21 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-07-21 meeting

Description

Instrumentation: Messaging

A

B

Oh join me asking how to pronounce your name. I've never been sure.

C

It's a bit tricky. uh It's joao.

C

Yeah, there's the the a has this tilde on top okay: it has this sound. Okay, let's say like show: um okay, yeah! It's all right, don't worry about it! It's it's tricky.

B

If I say something close you'll know who I'm talking to yeah.

C

There we go all right, the folks here in in german speaking countries. They also don't know how to speak, so they they have a hard time as well. So.

B

Where are you in austria or where are you yeah in australia? Is that an austrian name or is it.

C

No, no I'm from brazil originally. So it's a portrait.

B

Oh okay, okay, okay, yeah yeah. I didn't think that uh the tilde over it would be a german thing or.

C

No yeah exactly no, no okay, if it wasn't maybe yeah yeah, they do love that you're. Almost here, oh yeah,.

C

Let's see if um the others join us on, I think.

B

I mean amir said he won't be here this week right right, yeah. I remember today, so maybe maybe.

C

B

Yeah, maybe lid mila might yeah. I don't know.

C

Summer, heavy vacation, I got, I guess, yeah.

C

Right I'll I'll start sharing and then if people join then.

B

C

Is it this too big to see or is it okay.

B

C

B

Flip it over to my bigger monitor here.

C

Now, oh I'll zoom a bit because.

C

A

Good morning or good evening, sorry.

C

Good morning morning, guys.

A

C

B

C

I think if you just.

B

Me if you were able to make this window not as wide like no, it's probably the full width of your monitor. Then it probably would probably end up looking a lot bigger on on some smaller screens.

C

Does this help? No, we.

B

Oh you're, sharing your whole screen yeah that doesn't help too much no.

C

Okay, let me just show that that window then.

C

Yeah, what about now.

B

Yeah, that's that's a lot easier to read. Yeah thanks cool.

C

Right so then, let's go to the agenda first, what.

C

So I I duplicated the agenda from last week here removed some stuff that was not necessary, anymore, so yeah. So there is this this task here to do. I I will do I'll, probably start this, uh maybe next week or so when I have more free cycles, but I'll do this I'll move the otep text that johan has created for the context propagation to the current spec?

C

uh I I don't think that's going to be too much work so or too too. Polemic so should be easy. I guess right there's this other thing here that he also mentioned uh but yeah. He said he said it's also fine. If we leave this uh for later so unless anybody else wants to do it, uh I don't not really sure what he had in mind. With this fan structure, I think he said he would was going to put something together, but I'm not sure if, if he had time to do it.

A

I I would like to do it, but I don't have some.

B

A

Yet so see by the next week for sure.

C

Yeah yeah same for me, I I was able to squeeze this in, uh but uh not, I guess not much. No much else do you know what he had in mind with this um propose. The instruction is that the link thing uh uh like what what all we have in mind.

A

Yeah, I think it's everything else accept attributes.

A

It's like the the producer's pen and receive spam, and um this things.

C

Okay, I guess that's also where the adding links after spain creation is going to be discussed, probably as well right, yeah, okay, okay, so I'll just write this down here.

C

Just don't forget about it, but yeah we'll see you next time and I think last time we were talking about the destination name right. The massive destination name attribute, and I saw that you left some comments. Let's just look at it. I was just going to type here, but I didn't have time. um So I looked at the links here, thanks for looking as well, but look at the rocket mcq and then it says that uh the message in the batch should have the same topic. I don't know if this is like our thing.

C

That is just a recommendation or if they have to be but see that we have to be the same badge same uh same topic. Yes, I.

A

See? Okay, so that may be you we can remove rocket and cube from this list. Yeah. I was looking into the code. Save your in the text. Sorry.

C

No, no problem yeah, but I I was looking at this and then I saw here that they had the same topic. So I think this one should be good for us right so for the kafka one. I didn't didn't, follow, really how this would mean, but.

A

Yeah, so they have a couple of um things below uh a couple of links and like from the api perspective uh yeah. In the first comment from the api perspective, uh it looks like uh yeah send data to a single topic.

A

uh Oh sorry, send data to multiple topics. You see that the next api I already.

C

Use is, if you send it to multiple topics, but I guess each message has the topics that they go through. That's what right all right: okay, okay, produce! The data object to encapsulate the topic key and message: data; okay,.

A

And but we create the span for a send, call right right, don't have a topic for this one, and I also have we in azure have apparently one of the services allows you to send related to multiple topics. At the same time,.

C

In azure you mean okay, let's so see man.

A

It sounds like in multi like in the majority of cases.

A

Destination name makes sense right and it will be uh available, but in sound as edge cases, it probably should go on the on links, attributes or if we have spam for messages and then spam per message, yeah and then the question is there is no uh the sorry. The net peer attributes are also not required because amazon things don't have them right and or gcp things, don't really have them, and then uh they basically have no guarantees.

C

Yeah, there's, like maybe 70 guarantees or something like that right for the most part they have, but the other cases yeah there was a list that johannes put was in one one of the comments or was it the document.

C

There was a list that johannes put together with all the messaging systems.

B

Yeah, it was in one of the comments. Oh.

C

B

It was resolved.

C

Yeah, I think it's this one here, maybe nope. Oh no,.

A

I think it's on the same uh line. I added my last comment.

A

But how do I think it's easier to find it? If you go to the files and then they're there, then they are not collapsed.

C

Right, I know they are claps.

A

Find the one that is not collapsed is not resolved mine and the same line will be there.

C

Maybe this one, I hate it. This does this jumping around when you collapse, yeah.

A

It was on messaging destination, name, I think, or.

C

Yeah this right here, rich.

C

Destination name was this year.

A

And not, there was a different one.

C

ah Here there you go perfect wow. This is.

A

C

I don't know what this is this line. It just firefox doesn't like this, but okay, let's let's go through this, so okay, so he had here kafka, but it's a topic name, but this is the problem now right.

C

Yeah, okay and then the rest of it so uh and on they wanted to decide about azure was there any of those or the other one that we need to look as well, but the.

A

There is a side there azure event degree. So.

B

It's kind of cost.

A

But it does not have topic or yes, it has a host, but might not have a topic per cent. Call.

A

Oh, it's not it never has a topic percent call might have it per message, but not percent call number percent call.

C

C

See if we can start a new dock, so just rewrite this to know somewhere else.

A

Oh, I can add it to the comments there.

C

A

I I cannot uh by the way I created the gist with it. uh I can just share the gist.

C

uh Sure of this you mean.

A

uh Of this list, yeah.

C

Oh okay, you want to share, or you just want to.

A

Get into the chat and then I.

C

A

uh Let me let me.

A

Just share second.

A

Okay, here we go.

A

You see my screen.

C

Yes, okay, perfect.

A

um Okay, so uh I also reworked it to be assuming we will put um the server the actual server context fit into not to name another port um so for kafka. We'll have these things always present.

C

So the host is to put on the net peer name.

A

Yeah so like this is currently the logical destination for in in the general conventions, so it makes sense to re, try to reuse the taste.

C

A

And then for kafka, it's maybe per message, probably not. It seems like it's not the common case to send to multiple topics. At the same time,.

C

Yeah it's hard to tell right. um If I send one message only then it's fine, but for batches uh I mean it could be that the batch also contains all the matches for the same topic. There is no way to know right.

A

Well, you like instrumentation, can no. The question is: do we force instrumentation.

C

Exactly yeah yeah, that's what I was thinking. It's possible right, of course, but um yeah.

A

Yeah and then uh also it depends on the spam structure. Do.

C

You want to do you want to edit this to put this uh api link that we just read, then we don't lose track of it, because that was very clear right.

A

C

I'll just send to the chat here.

A

C

A

Maybe I can put it later: okay, yeah.

C

Also also also good noise. Yes, that's fine.

A

Yeah, thank you yeah and uh one thing yeah, so I was talking that it also depends on the span structure. So if we uh we currently don't require spam creation per message right can it is possible. It's made right, it's not should uh at least I think so, and then.

C

I think so, yes.

A

Then what we can say about it that, okay, if you decide to instrument you create per message, then you put topic name there.

C

Yeah but will be hard to say this in the conventions right. I think the only way is uh like you said uh whether it was uh conditionally required something like this that if, if it's present then um and then maybe there- we can put this, uh but I don't know if the conditioning required if we can put like per system, it's gonna be a lot as well right to do like that as well. So maybe generic say like if it's available then put it otherwise.

A

Yeah, so if it's available and if it's the same per batch right, yes, then conditionally, it's conditionally required right right based on this condition and otherwise it's recommended to, and then there is a question either we uh create a links for each message right and then put it on the links attributes or we put this attribute on per message. Spam.

A

But in in both cases it actually implies that we have a unique context per message, so we are in a difficult spot here.

C

uh No but yeah, no, um I mean uh each method. Will it its message: will will have its own its own uh scope? Yes, uh its own context, yes, because what you're saying is put the topic on each message like this when you create, but that could be possible, but um actually that is, that is what we're doing uh so it it would be actually always there. I don't know if this information, the topic name in kafka, is available. When you create a message, for example,.

A

And I'm not this link that you have and I'm not sure if I still have oh here, it is.

C

All right, one of the comments, the producer apis.

A

So yeah so here is an example.

A

Sorry, but basically yeah, so this isn't a producer record in my cabin. So uh I think that the interesting part here is assuming we want to anyway, regardless of even the message destination. We want to put something about each message: maybe a message id or something specific to the message which is unique in all cases and then, if we don't have a context per message, uh what do we put on link? What context do we want to yeah.

C

No, no, we will have one context per message just because that's how we're going to do the linking on the consumer side, I.

A

Don't think we have it in a span structure. I don't think we require this context for message and we are saying some context should be on message, but it's not necessarily unique. It can be the same context for all messages.

C

That's a good question.

A

We talked with johannes about it uh about the current that this, the what tab you're going to yes, add, and I think it's it's there.

C

Yeah, I'm opening right now I don't I don't remember, but I I think I I remember both like discussing both things. um Having always a different one or having shared one. um Let's see.

C

C

C

Yeah we can skip the motivation. Probably it's probably yes,.

B

C

There requirements.

A

Requirements: okay, yeah message, creation context right and what johannes explained.

B

A

That it means that they they never specify which, what which message creation context. So it's either the context in which message was created.

C

C

Right, yes, yeah exactly so I remember now so either one context per message or if they, when the message is about to be sent, if it's, if it doesn't have a context already, then the segmentation can create one and all of them will have the same. I think that's what what it is then. Yes,.

A

Yeah and then the question is: if we allow this, how would we put unique per message attributes where we would put them.

C

Yeah, but I'm not sure if we do have uh per message uh specific attributes um today.

A

Let's see, I think there is message id which there's.

C

A message id yeah, yeah.

A

C

Only thing that I think that we can could be um destination destination kind.

A

C

Message, ideas.

A

If topic name is a well, I mean you cannot probably send to topic and a few at the same time. So I think this should be fine. Yeah.

C

Yeah but there's the destination id. Oh, that's the problem. Yes! um Well,.

A

C

Yeah, well, I guess, for example, if it's, if it's uh something that is um important, then there is has to be a way to create this. um There will be expand per message, then uh right. So if, if, if, if people want to have the message id, for example like this, then needs to think in a way that this is always created when you create a message.

A

Yeah, this is one possibility. The other possibility might be that. um Actually there are three so the first possibility. We create a span per message right and we do this. The second possibility. I think we briefly discussed it. It sounds hockey, but it's a possibility. We create this fan context our message: we don't record this pen, but we have a link to it.

A

um I think we discarded this idea because of that and the third possibility we add a link per message context, but it can be linked to the same thing all the time, which is also hockey.

C

Yeah, but isn't it the same, as is if we create just one context upon sending and then put the same in all of them?

C

um I think that the the difficulty comes, because uh maybe maybe the the the application owners will have to create this fence right, depending on the sdk or something um so, for example, if I do like, if I create a message, it's just like a new message or something, then how how people will create stands right for this, and I'm not sure if the libraries will do this as part of instantiate or creating a new object right, um but the other other way would be before sending the batch, the library that they're sending just place the span on their own, for example.

C

If there's no context in in in the method, then it creates a new stand, a new context for each and then you can have all the attributes like the id there and also the destination name, uh but it will be a. uh It will be a one context per message, then not just one. That's another way of solving it, because yeah.

A

That's like the integration.

C

A

Yeah and it's a good point that this fan can be created, the message stand can be created by the application and on the library, uh and we can just solve it with links right, because uh links on the send will no anyway, but all three approaches are hockey.

A

The span per message is happy because of zero spend duration.

C

A

No span is happy because there is no recorded span and uh um link to the same context as haki, because it's also hockey. So we don't have a nice solution problem.

C

I think the stand duration being zero. I think we can argue that for this case it I mean okay, it's always not great- to have a zero duration spin. But for this case I don't think there's another way to to connecting them like like like this is desired, for example, connecting a batch of messages to a consumer. I mean how, how would do you do otherwise, I didn't would yeah invent this other concept of a partial context or something I don't know.

B

Is it reasonable to add an event per message to the batch? Send spam and the event would have all the attributes pertaining to the message that you know like the guys that the library is sending the span would traverse all the messages and kind of say like oh, you know, added message added messenger added.

B

Isn't the right word because I don't know but anyways if you just record information for each message as individual events as you traverse them, you get a timestamp which isn't too interesting, but then you could have all the attributes for that message. In the event.

A

It is I I like it. The problem with it is that events don't have convicts. So what we would- and I think this this is solution that has the most uh future. There are multiple places in open telemetry when they want to create the zero duration spans, and maybe the right solution would be to say. Okay, some events can have context, and here is how you attach context to them. I mean not the parent context, but their their own unique context, and they are just the zero duration spans.

C

uh But isn't it the same? For example, if I, for example, if sending the message uh like when sending the badge, we create a context, and we put the let's say the attributes that are common for all messages into the context and then we add, add events on this context for for master specific things, and we have like a lot of events like do you want to say like then I will have an event containing uh because events can contain attributes. Oh no, I'm wrong.

A

C

Attributes right so then we have have events with some attributes. um For example, the message id will be an event uh attribute.

A

Yeah and then the guidance would look like if you decide to instrument per message and attach unique context per message. You do zero duration span. Otherwise you do an event.

C

Yeah but the the context will be the actual span right to the sense then, and that one will be the one to use for on the consumer to link to to um to the message um otherwise yeah otherwise create zero duration stance. I guess per message and then you put the things on the mess and then you link to all the messages yeah, it's a. I think it's a fundamental problem.

C

I don't remember if we bumped into this before about the the specific problem about having message specific uh attributes and then having only one context, then you lose there's no place to put this yeah.

A

Yeah- and there is a similar problem on in databases where they don't have a unique context for a bulk right, but they have unique attributes.

C

A

C

For uh for a bulk sequel operation, they have can, can you repeat it again for a.

A

Long operation, so for when you write multiple independent queries like multiple independent queries, so then uh the operation that performs them as a single operation right and each of this queries does not have a context per cell right. It doesn't need to it's a database call, but attributes are unique.

C

Yeah, like the the the parameters of the.

A

And there is no so for your an undefined area.

C

Let's see yeah.

A

So I guess what we miss is: maybe some analysis and some proposals, or maybe some just pros and cons, um I'll try to put thumbs up put up some document, and maybe we can have something by the next time.

C

Yeah, that would be cool.

C

Just want to, can we go, can, can you go to the um or tap again.

C

Okay, there is nothing actually about the individual context um for the consumers. Okay by the producer should be propagate to consumers.

A

Okay, if we decide to that, we need a new, unique context for a message. I think it will be an additive change to this.

C

Yeah, that's what I was looking for exactly um because it's only because it's very vague right now right, right.

A

C

Probably this issue that automatically context to each message: okay, shouldn't be attached in a way that yeah yeah, but I I I still think that we ended up with the idea that a new context will be great for each message and I don't think we discussed much about the zero duration span. I think we just said: okay, we'll do it like this and no no, no discussion anymore.

C

um I think we did discuss a bit and I remember all these other options about also the the stripped-down context you suggested as well, and things like that, but.

A

Structure, that's exactly the one.

C

A

Yeah and it's great that we they're thinking about attributes first, because apparently it's important.

C

Yeah the attributes really review some stuff. I guess yeah that we didn't think before. I guess.

C

But okay, but if we, if we summarize, if we have one context per message, uh then we are fine right because we can put the.

C

We are all we're pretty much good in that case, because the destination name we can have um for for the others we already had, but for kafka that is different, then we will be covered right and we can have message specific things like the id uh and then the consumers can link to this. We will be at the let's say, the real context of the creation of the message.

C

So we are fine right, um but if we don't have this, meaning that we don't want to create a lot of spam, for example, then or if, if the the application owners don't create it or they estimate them whatever, then the latest point is when the mess the batch is being sent uh packed to be sent then will be once then created. One context created and all of them will have the same.

C

Context inside of the message.

C

And then we don't have any option of putting the definition name, for example, or the message specific attributes.

C

And in this case then, does it even make sense that there is multiple links for a message, because it's going to reference the same context.

A

Right, that's a great point right, and maybe the way out of it is saying that if you're allowed to send batch to multiple destinations, then you are forced. You must create a spam per message.

C

Right, but even in the other case, where, like normal case, when there is a badge and it's for always for the same topic, if we put the same context, if it's only one context, the same context for all the messages, do we even need to put in all of them, because it's going to create links for the same thing right.

A

Right so in this case, I don't think we we should have links, but we still might have her message, attributes like message id and there will be no place to put them on.

C

Okay, so that's okay, but no what I didn't get that you mean creating a context per message in the case as well.

A

I mean if we say that, okay, there are some cases where you have to create a stamper message. What happens in other cases right? You don't have a spam for a message: yeah, no.

C

Exactly yeah, no, but what I was thinking is: um if we, for example, have this option that no span only one span is created and the same is used for other messages, um because it's what the otap says here right so produce should attach producer. That's a message message: creating creation context to each message. So if the message arrives, for example, at the producer the moment to be sent- and there is no contact inside the message- um then somebody should miscreate this right. It will be one context or it will be multiple for each method.

C

I think it's going to be multiple, because if we have only one uh then, what's the point of of the consumer, adding links to all the messages in the batch right.

A

Well, I think there was this idea that maybe it's too hard this this. This is the way of avoiding this zero duration, sprint right and all the hacks around it, and it was the way of saying. Okay, we just are not it's not possible, maybe to instrument each message or it's. uh It sounds too complicated, and maybe people don't need it.

C

A

It was like this okay- maybe it's too hard. Maybe it's not needed.

C

But if you, if you're gonna visualize this trace, then you'll see the create the whatever trade here and then multiple receives linking to the same here. To do all the same thing um right.

B

It's one scenario, but isn't it also possible for a consumer to receive a batch and every message in the batch it receives? Maybe came from a different producer batch and maybe some of them not from a batch at all. Right like the batch, doesn't follow through the entire messaging system as one unit. I don't believe right.

A

Yeah, it's reshuffle for sure.

C

Okay, so I can send what one message and it can be put into a batch later on by some some other thing- that that's what you mean or.

B

Yeah, I think, when you pre, when you produce a batch, it's a shorthand way of saying send all of these messages with one call right um and then they get into the message system and then how a consumer creates a batch and receives a batch is kind of separate from how the producer.

C

Sent a match: okay, so the it could get. For example, two batches in one batch in the consumer side.

A

C

C

A

But in this case it will be.

C

Okay right because if there was like two producers sending let's say two batches separately, but then the consumer got only one in one call got the two batches in one batch. Let's say, uh then, if we do this linking, even though, if like it's just two contexts, it will link to the correct thing and I think it will be fine.

A

In this case, yes, but if you send 10 messages, each of them can be received and processed in 10 other different batches right. So you send with one batch 10 messages and each of them is independently processed within other things, and there you lose like even knowing where this message went. Did you set the right topic name on it, because on the other side, everything is the same.

A

So like monitoring anything without her message. Context is very limited, but I guess what you said that okay, maybe it's not possible, and then this is the best we can do.

C

B

I feel like the best way to structure things so that you can see what happened. Visually is. If you do have a span per message, um it seems like modeling it. That way is more important than, and maybe I don't appreciate, the full uh impact of zero duration spans. I mean it just sounds like they're frowned upon, but um yeah.

C

I I I also don't uh uh in some situations it does make sense, because it's it kind of exposed like a smell or cold smell or something or something that doesn't look right right and most of the cases. It is true that is wrong, but I don't think in this case we can do it in a different way and I think it can be like a one exception, for example, because how would you do it other way right, like we just discussed 30 minutes, yeah.

B

I mean the other thing, that's possible, but it it's not as neatly tied together as if we go back the idea of an event. Let's say per message, and one of the attributes in the event is a message id and then, when you create a link back to the spam that had all the events, one of the attributes in the link is also the message id and now you can kind of tell which event inside that. But that's you know what I mean like it's it's hard to build something.

C

B

Yeah, it requires a human to follow all that to expect a gooey or back end. To do that. Sort of thing automatically is, is, I think, quite a stretch right, whereas the links, if you have a span per message and you link directly to the span- I mean that's. Obviously the guise can do that very nicely. Yes,.

C

Yeah, it seems like uh like uh uh don't want to say native, uh but it sounds like a native thing like like, like it sounds like you're doing the correct thing right with the following: the apis and so on. Like you create a link and get the context, you link to this at a link to this context, it's it sounds like crack to me. It sounds um intuitive.

C

Let's say like that and then easy to to easy, probably for for back ends to reason about things and so on, and also we solved the problem with the kafka, multiple topics and also the sub problem, with the massive specific attributes, um all all uh solves itself yeah. We should.

A

Recommend it right.

C

Yeah yeah, I think.

A

So it's not a must, but it's probably a shoe.

C

Yeah, I think I think it is statement that is highlighted now. I just need to say, like a product, should attach a new message, uh creating context to each message or something like this, but.

B

Attaching a context doesn't necessarily mean it's a span right, it's just a context. It's just something you can create a link to you. It's not necessarily a span. You can add access to right. It.

C

Is a spanish yeah? It is because a span in a context is kind of. I think it's kind of a interchanged use, but in this case to create a context context. It is they stand right, so we need to create a span and then expand is the context. And then you add attributes to this.

B

I I I I mean so if, if I'm uh I- I don't know if I agree with that, right uh context is something that allows you to link to a spam right. It comes out of a span, but you don't necessarily have access to the span.

A

C

Yeah, exactly I don't know, if there's an api actually to create what this has been, but not, it would be like a non-recording stand or something just to use as a transporting mechanism or linking mechanisms right. You know.

A

Expand the context you don't need uh to create a spanter creator.

C

Oh, you can okay, okay! I I never created a context alone. Okay, yeah! Then, then! That's it! Yes,.

B

But I think what makes more sense is if, when you go, when you have your sdk is doing its instrumentation and it's sending a batch. It creates a span to send the batch and then it creates a child span for each of the messages that it's sending. So these are like send spans, not uh message creation contexts.

B

um But if, if it's not the message creation context, then yeah, then we're not getting at the consumer side. We're not getting a link back to something specific to that message.

A

I remember now some feedback. We've got from users about the case where they own a producer, but they don't own a consumer, and this looked for them, weird that they have span per message and they don't. They feel like it's uh like they're wasting uh their budget on this fans.

A

B

I know that yeah, I know that came up where spans are expensive and you don't want to create too many of them. For no reason, as is one of the concerns, but.

C

Yeah, but this could be something that the library can do right to disable or something like this. I guess if they don't care right.

A

And I guess, like the question, is okay: if we say we should have a unique context, let's spark a span versus context, discussion, a unique context for message. Then, but users can disable it, for example, and then um I don't think we need to provide any further guidance. Actually we can leave it to the future ourselves to deal. I don't feel it's super critical.

C

um Get current contacts, I don't think there is an api to create a contact scene, opengl.

A

uh You can pick a language.

C

No, I mean in the specification there is no api in the specification, because I'm looking at the context specification is if there is a get great key, get value set value, but in get current context, but no way to create a content. I think the creator context is when you create a span that that's what I meant that they go like handy hand together. I know that after you create a span, you can get the context only and pass it around. For example, that's probably that's what they do in go.

C

For example, they use the context in go to uh it's not a standard. It's just this context thing, uh but.

B

I think have, if you have a context that doesn't necessarily allow you to change the span. The context came from right.

C

No, no, no, you can just attach things to it um to a context, but you cannot change the yeah.

C

Yeah, this is also something that probably uh needs to do some prototyping right, because the way this context I was thinking is that would create spam right for this. um So for each message you create a spam and then put this inside, because when you create this fan, you will get the trace the span id and so on. You can put the attributes and then you travel. You put this thing this object inside, but the span is already created yeah, so that there will be like a lot of spans uh showing which is.

A

Yeah and I guess what we can consider the options and I'm thinking about it, how we can present it. I don't know in this spec general spec medium right um and it seems there is some concept. That's missing can open telemetry for this, like the either the link without the context or the well. We need a context in this case or a context without a span or zero duration span.

C

Yeah, could we create a span that is not recorded or and because that will give us the context right, but this fan is not recorded.

A

And this is a hack right.

C

It's a hack, yes, yes, it is.

A

C

Not show the spanish.

A

C

Go ahead. Sorry.

A

And like when, we think about this options uh and also there is a event with context right, so something um the way to represent zero duration spent. So it seems whatever the current up until monetary providers will be a hack, and I think the context without a span is at least top hacks, and then there is a question: uh how would backhands treat it, and if we want to push this proposal through the um spec of progress, then we would probably need some back ends.

A

Who can implement it and see how bad is it and it's it's a long road.

C

Yeah, because if it's just a context, nothing, it's not a span, then um yeah you're going to link to it and link to nothing right, because it's it's nothing that is stored, that you can reference to afterwards. Yeah.

B

So does this help us, so maybe I'm missing something just help us figure out where we're going to put the destination name for all of the individual messages when the destinations are different. I'm not sure how I don't tie these. I'm not tying these together. Right now,.

A

C

Think discussion came because of this because because of this problem, yeah because the because each message has some specific attributes and if we don't have a context for it, then there's no place to put it right. And then we uh yeah.

B

Right but a context can't have attributes has to be a span that has the attributes right exactly, but.

A

Then we can have a link to this context on the sand. Call that will have this information and then, if users decide to attach their unique contacts per message right, which they can do if message is forwarded, for example,.

B

Because you're saying add, attributes to the link.

A

B

A

And maybe the way out of it could be if we say okay, we start with the spamless context.

A

And we say that we just have this context, and maybe we will find out that it's impossible in some languages, let's see, but then adding a span to this context would not be breaking and it can be done later. If we find out it doesn't work.

C

Yeah you would attach it, it would create a new span from a contact, something like this.

A

A

You can create a context right, at least there you I played with it, and there are maybe some some difficulties, but you can. um But then it's an internal detail. It's not a public api.

C

A

Later on, you decide to create a span instead of a context. It will not be breaking and you will add a span.

C

Yeah, I guess so highly theoretical, but I guess so yeah and goal is the same. uh You can also have the context in because it's just a language construct. I guess nothing, nothing, hotel, specific.

A

And in java the complication is how you generate a random thing and the random generator. That's coming with open. Telemetry comes with sdk another api, but.

C

I guess I do things or.

A

C

A

I guess this is a small problem.

C

Yeah yeah, but the problem with with the with the context without the span and then uh when in the consumer, when you, when you link to it what it's going to be right, I don't think no. No, no server today can can handle this right. It has to be expanded to be able to link them together and see, see it not race.

A

You would have a link to the context that doesn't exist and the link on the other side to the context that doesn't exist. Yeah. I don't think existing backhands work great.

C

Yeah but there'll be no way to visualize this right.

A

C

um Well, this index thing has to be stored in the back end, so so it's able to link to it right to display it. Otherwise, you you have a link to something that is uh where there's no reference to it right in the storage.

A

So like what I'm thinking is it probably existing backhands would not work well, I can play with that general effects, but you have um spam uh sand linked to this ephemeral thing and then there is the consumer spend link to this ephemeral thing and on this way you can link the producer and consumer because they are both linked to the thing that you have. So your index should be a bit smarter right, but it's like there is no computer science problem around it. There is enough information.

C

Yeah yeah no like if the context has a span id or something or something like that, that is able to be indexed and it's fine, but but it's it not being a span like. I don't think none of the current uh solutions will be able to handle this right. It's it's a new concept. I guess it's not gonna yeah, and if it's something that has a scan id, then it's under the spin. It isn't the same thing.

C

A

C

A

If you are backhand, you should be ready that you have links to things that don't exist on your back end, because people integrate over messaging systems right and you can you can somebody else. Can all this telemetry right you? You need to be able to say. Okay, I understand this is something else, um but it's the question: how do you search for the other side.

C

Yeah, exactly I mean you, you will receive something in your site on the consumer side, you have the attributes, because it's in the message uh you can display them like, for example, you can display that message id or something, uh but if you don't have on the other side where this came from, you cannot like navigate to this right, but you do this. You can be able to show to show underneath this information again this link and what information you can take right.

A

Yeah, so it's like this, this more generic way of putting message id everywhere and correlating by message id, but it has some future um as a as an open, telemetry.

C

Is any something that is something like baggage that could, uh because package is also an arbitrary number of things that you can just pass around right. uh I wonder if there's no structure ready to carry this around, that we could reuse, I'm not sure.

B

C

I didn't look into much baggage um stuff.

A

I guess the spam contest is.

C

A

Right three city and spanish shouldn't be enough.

C

Right: yeah, okay, I wonder how we can what we should do now, because it seems that without this uh we cannot um do much yeah.

C

A

Well, do you think.

C

A

The other thing we discussed that remember, we wanted to add links on the consumer side to messages that don't have context.

C

You wanted to have links to the add links on the consumer side to message. They don't have contacts okay, but link to what then.

A

Link to invalid context, and then we can attach like on the receiver side, you have no guarantee somebody sent you a message without the context right, easy.

C

A

And then we discussed the idea that we wanted still to attach message specific attributes right, and we also wanted to count messages and the using links to invalid context made sense there.

A

So maybe we can say that we should have context per message span or context, let's figure it out later and we will have attributes per this link.

A

But if somebody doesn't want to create this spam or context a name.

A

And if it's a badge called, they should have links to this messages being sent in batch.

C

Well, what would be the link then on the message you would just put something there or like the id that you have in the message or.

A

Yeah, so you would have message id, that's the the is the one thing, the other thing. If you have topic per message, then you should put top per message there in destination and.

C

Okay, but this solves the case where, uh when instrumentation didn't create a span per message per message,.

A

So you should create one, but we understand there are some conditions and if you don't, if users disable it, this is what you do.

C

But then isn't it um also wasting things because they, if they don't create a scan permission because they are concerned about wasting things right, but if they it doesn't the other side as well? Isn't it also wasteful again.

A

C

Because they will be essentially doing the same on the other side right, just creating a zero duration context, just in a way to attach some stuff to it. Right.

A

Well, I mean: what are the options? There is a unique information per message right. It's a.

C

A

Name or a message id and we can express them as array attribute arrays right, you can have an array string as an attribute. Well, this is a hard thing to do, or we can put them as links. I think links are a bit more expensive in terms of you know, ingestion right, telemetry injection then attribute to race, but at least you're structured. Somehow.

C

Yeah, I don't know, maybe we should really challenge the thing that I don't want. One spin per per thing. uh Maybe we should just say like. If you don't want this, then you're disabled. You don't have anything good luck, debugging! Your your applications.

B

I was, I was thinking it's maybe along the same lines that you know, should we reduce the scope of what we try to cover here to for batches. We say that this specification says what you should do if it's the same destination for all messages in the batch and much like we're saying for transport instrumentation, you know we're not going to handle that. Yet it's a future just kind of leave it open for how you want to handle it if you're doing different topics in a batch without saying exactly what should happen.

C

B

Just leave it for a future.

C

That's one way yeah, but but this doesn't solve the thing about uh having having to create one context or one span per message right.

A

The message maybe.

C

Because of the message id yes, I see what you're saying yeah so like I was thinking like. Should we restrain ourselves because uh there might be people or or users that think that this is too expensive and because we're creating a lot of problems? Because of this? um Because we don't want to have one context because like if we, if we think through having one context per message, it makes everything easier right for everybody.

B

C

It makes easy for, for the specification, makes it easy for people writing instrumentation, because yeah, but if, if the in this spec, you say like you should do this, but you still like are free to not to disable this feature. If you, if you think this is yeah too much, uh but then I mean you will not have any any any instrumentation or any any observability in your stuff.

B

I think this, but I think the span that's per message that can contain the per message: information like message id and essentially destination, name, that's sort of at the send time right. I don't think the creation context can help like saying you have a unique creation. Context per message: doesn't help find a place to put per message information during send.

C

Yeah, I suppose I mean a new context, can be if it's not created when a message is created, it can be created before sending right. It's just another step, another iteration and go through all the messaging and the instrumentation can create one context per message. Yes populate everything it can be. It can either be done, for example, in the library where the message is created or by the application owner. They do it manage because they want to control everything or or um like it's.

C

It's similar to the thing that I did to this uh in the past. With this cloud events thing uh when you create a cloud event, it's just like new object um and you can do you can create it yourself. But if you pass it without a context, then the the library creates one and puts the things inside.

B

And so the point being, if you do create a separate context for every message, then you'll have links to unique contexts um and for each message and that'll yeah. I can see that makes sense. Then, if you don't and it's one, can you create many links to the same context in the same span? Is that allowed.

C

A

Is but I like doing what you said about doing it to solve it, so imagine we if we define this, be if you disable it right, if you don't follow our sure dividends, uh if we specified now once we maybe get to solve this problem three years from now, we will be in a worse place than if we don't specify it. You see what I mean. Maybe we just should not say anything.

C

A

What to do in case, you.

C

You mean what to do in case. You disable this.

A

Right, yeah, okay, oh sorry, context per message, then we don't cover it. uh Maybe in v2 we cover it.

B

A

We don't need to do anything now, so we can do anything we want in v2 after we learn more.

B

Right yeah, but.

C

If you're not dealing.

B

With a if you're not dealing with a batch, though we would still say you wouldn't need a link for this information, you would just put it directly in the span for that. One message then right: yeah, right, yeah, exactly okay and there.

C

Will be only one uh span is the same as the sense then.

B

C

B

Seems reasonable.

C

Yeah, that's that's pretty reasonable. I think that one is easy. I think the batch is the problematic thing, um but the other thing that I just uh we we're way past now. um If we we can cut it short. But what I wanted to say is this like. If we have one context per message like this um and it's not created upon sand, uh you will have a lot of scattered uh in the trace. You had a lot of scattered, great, great, great right to like very short span, zero spans or something um not sure.

C

If we thought about that uh already or even if these fans are creating before scent, you will have like the sand. That's a parent, and then you have like a lot of small expanse. They need if we go with the idea of the context being a span um which I think it will be like that, because I not sure if, if it will be received well this idea of stainless context- maybe I don't know um not even sure where we would go to discuss about that stuff.

A

Yeah, so maybe by the next time, I'll try my best to prepare this uh list of options and pros and cons, um and if I will have time I I will chat to our folks who work on the visualizing. This messaging and links and.

B

A

They can give me some insights on how bad is it. I think amir had some opinion about it, and we can also ask him.

C

Yeah yeah, I I can ask also inside here to see if what people say. Yes, okay,.

A

C

Then thanks everybody. I will put try to put some of this thing that we discussed in the agenda, so we have it some for next week.

A

Yeah and also uh the link to my just.

C

ah Cool cheers. Thank you. Sorry. Next.

A