Open Telemetry Uncategorized, 2 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-08-02 meeting

Description

Instrumentation: Messaging

A

A

B

Just uh check the.

A

C

For the rum sig or maybe I grab the wrong zoom link.

A

Yeah uh hi yeah, it is for the ram sake, but the main one is tomorrow, okay, so this was added um for some additional discussions on the same topics of ram, but not uh many folks join. Not many folks are aware of you know this.

B

A

Okay, this one.

B

Specifically for the otep that you all are working on yeah.

A

We started for uh discussing a few tips, but then you know the rotep uh didn't move forward, so we continued this meeting for anything else in general, so usually um martin and nev, martin from neuralichnov from microsoft, myself and stefan and a few others. You know we hang out here but yeah. I think how about you, which company you're from.

C

B

Coming from grafana, um I recently.

C

B

Kind of app observability team coming over from tracing and so just kind of getting familiar with with rome, and what everyone's doing here, okay, okay, yeah.

C

B

Think I'll double check because I think um cedric, one of our new pms he's been going recently and so kind of filling in while he's out.

A

B

Thanks for clarifying as well about the meeting yeah.

A

uh I have one topic to discuss, but I wanted uh martin.

A

Nev, did you have any topics? I I wanted to talk about the uh you know the map data type for attribute values.

A

uh Looks like uh I, I I'm not following your pr closely, but if it's going the way of getting that support only for logs.

A

Is that the way it's uh going, uh nev.

B

Yeah, it's gonna be problematic because I ideally we want to have it for both logs and for at least span events, um but at least the pushback from the spec guys is they don't want any hints in the spec of that? It may be at some point in the future supported, so um hopefully we can get it in for logs once we do that, then we probably need to have our defined events first before we can even try to push span events, um I don't know it's, it may be a case of for span events.

B

We just say it's adjacent blob, which is it would work and and likewise for if we said event, data is just a string which is adjacent blob. That would work too and in some cases might actually be smaller than than a nested attribute.

B

um Okay, that would only be for the json exporter. If you're doing a an hotel exporter in binary, then it would probably be smaller to keep it as a nested attribute.

A

A

Yeah we could, we could go that way.

A

Okay, so martin, now that you're here I wanted to um you, know, clarify something um I I think uh I don't know if it was last week or the week before we talked about uh the performance timing parameters.

A

Today they are represented as individual span events under a certain span, and you wanted to move them out to be standalone events, because you know not all of those all of that data is available soon enough and that could you know, delay uh sending the base paid span. You had this one concern right and.

A

A

Option is: uh if we keep them under uh uh spans, uh you know we could generate a new span. uh You know which would anyway, you know these. The document fetch is a child span of the document load span and you could create one more document. Fetch.

A

Basically, I'm wondering if it makes sense to create one more span when the remaining data is available and then send it out.

C

One more span that represents the the fetch document patch.

C

Or the document load, uh like the the whole.

A

um I I don't know, but basically another span that would have the original document load as its parent original.

C

A

Load span as its parent, so so, given that uh you know the initial data spans can can go out the door and then any additional information, that's available after a few seconds, you know, could be sent out as additional spams and you could always link them to the to the initial span.

C

Yeah, I think it's not what's happening now.

C

A

It may not be happening today, because I I don't know whether uh all of the performance timing parameters, all the data is available in in one go or over a period of time.

B

um Why are we looking to break it up because, like we're we're looking at defining an event which contains them all yeah and my my understanding is the resource timings at least for a page load? Are a single entity, so they'll all be there. It won't be a case of it'll, be partially populated, at least that's my experience.

A

Oh yeah, we could, uh we could delay. um You know the entire in all the resource timing uh spans uh to a later time, when they're available, you know we could send out the base page span. uh Initially, uh whenever it's uh the page load is complete, I think martin's concern was the performance. Data may not be available until after uh a little later.

B

And it may not be like for fetch and xhr it takes about 10 milliseconds before the resource timing. Stuff is populated after the fetch and xhr response has returned. um So I don't know about the page load. I haven't actually done explicit timings on that, but it's possible, um but again we're talking about wrapping this and sending this as a span as opposed to sending as an event, which is what we're trying to define which may have a span linked to it like the the existing experimental ones.

B

Are there doing it the way they're doing it, because that was the only way to do it, we're now defining what we wanted to do moving forward. um If people still want to span, then yes, they can go and create a span and then fire the events.

B

But I I don't think we should be saying that we're going to credit, spain.

A

C

Now I was just going to say, like I yeah I like, I think that the span is is right in my mind, kind of optional, like it's the context that you can, that could that can be present and it's really up to the instrumentation like we can still have a span, that's generated right, but but I think it's important to capture the the navigation timing data always um and as much of as much of it as as possible.

C

So you know if the if the document load event doesn't fire, then then we can at least send some attributes from the navigation timing. Api.

B

If it's there yeah yeah, so for microsoft, we effectively send. We have two different events that we send. One is the the page load event which is affecting the document load, and then we have the page view performance event, which is the uh you know, a summary of the performance timings and they are sent explicitly separately.

A

So um there is uh one concern I have sending it separately, um which is.

A

um Like at least it could be like vendor specific, it could be just a limitation on our side, uh but but basically uh the uh the advantage sending along with the original span you know for for that hr is, is that you know the the performance data uh we know it belongs to. You know that xhr.

A

Otherwise, the identity of that original span needs to be repeated in the event.

B

Are you talking about ajax requests or the document load event.

A

B

A

B

A

B

So let me clarify so for application sites. We send two separate events for the page load, which is the page load and then the page view for xhr. We actually delay sending the event for the what we call remote dependency for the ajax request. Until the timing is available, all we timed out trying to get it because the timing for the xhr won't always be there, because if the resource timings is full, it'll never happen um so yeah. So for the xhr, I agree, including it in it.

B

The performance data in with it is needed because they are highly linked that you know they are directly linked to it. um But for the page view and the page view performance, um I keep keeping those separate, especially for client-side redirects, because you may not. You may not be able to wait for the performance time to be there.

A

A

I think the the reason um martin correct me if I'm wrong, but the reason you you didn't want to wait until the performance data is available was to handle the cases such as you know, somebody closing the browser or somebody navigating away from the page right, in which case um you know there is an urgency to send the data out um so.

A

C

A

uh Is that the reason.

C

I think that's one, that's like the main reason I think, but also also just uh not relying on tracing like, I think getting you know, sending that event is much much simpler and.

C

Wider, I guess than than like, creating a span and.

A

Now span already exists right, so you just need to attach additional data to the span. So should we wait or not was the question.

B

Well, the span may not exist depending on the order. Generally, it probably will exist, but it depends on your application.

A

um Can you clarify why, when the span wouldn't exist well,.

B

Who who's creating the span.

A

So so um we are talking about the performance of a url fetch right, so there will be a span for that.

B

Yeah, so for an ajax request, depending on how that's constructed there should be a span that gets created around the outside of that. So I think for the experimental ajax things I think they do create a span before they do the request, but they close currently close the span when the response is received, which is before the performance timing will be available.

B

So there are those instrumentations if we want to include the performance timing in with that, we'll have to wait, you'll have to wait, um and then you end up in a situation because you're waiting with the timeout, if the page navigates away, while you're waiting you'll never get called, and you won't actually send the request.

A

Okay, okay yeah. I think we'll have to experiment uh how it is behaving today.

B

Yeah page unknown is painful.

B

All right, if everything is a spa, we're cool we're shiny um yeah. We can wait as long as we want, uh but yeah yeah. That page unload is painful.

A

B

But yeah that would be the instrumentation's role for the span, in my view, and in the page load case, which is what I thought we were talking about, you won't it'll, be the application. That's created the span, not the instrumentation, because the instrumentation will fire on the load event and today it creates a dummy span. So it can send events which we could continue doing or we could have an instrumentation that could continue doing. But then it's going to potentially wait until the the timing. Is there or just assume the timing?

B

Is there and just go for it?.

B

But really I'm trying to abstract it out. So if we just define the event, we can have instrumentations that do have a span and it's rotations that don't have a span or we have the instrumentation. That says, I have a flag just to say whether to create a span.

B

I I don't see it as a there's, a limitation of defining the event that it must or must not be in a span, because in the case of uh your back end, you want the span in the case of microsoft's backend. You don't need this man.

C

Okay, I would leave it up to instrumentations yeah.

B

Yeah um so whatever whatever configuration you supply for your customers and say: okay, we're using telemetry here's our sku, you just have that configured with whatever instrumentation is going to be right for you.

C

Like having, but having like the events, just like the simplest way of gathering the data right, like you have you will you can just get like the events that represent the resources and the navigation yeah, um and that's like the simplest implementation and then like. If you want to have more context, on top of it and, like you know, uh link link different like show attribution between uh what happened on the page. Then you can generate spans like you, can have a separate instrumentation that generates plans. Yeah.

B

And like if you're in angular react or some other uh you know spa type framework, you may want to start the span while you've finished going and resolving all your router and setting up and then finish a span and then anything happens in between there should automatically get linked to whatever the the parent bank was.

C

A

Okay, so this is about the data, whether it should go as an independent event or are as part of a spam, but on the other topic of whether the data should be.

A

Set of individual span events are, should be uh a set of independent, just attributes. um You know they like the http request. Let me show you what I mean so the http request, headers um is, is a map, but they are flattened at this at the source itself. It sits flat. Let me show you.

B

And I think we're gonna end up. You know, based on the pushback that we're getting from my nested attribute stuff, we're gonna have to end up, I think, defining some scheme to flatten things. Even if log supports nested, uh I I think the way things are going that doesn't have to find my thing. There's gonna be there's gonna, we're gonna, define a strategy um and that things are gonna have to fall back to.

A

Okay, um so here the http request header is is is uh ideally a map is suited for it, but given that the map doesn't exist, they they're just recommending that you send them as a separate attributes and then the back end knows that the http request- header, uh you know- needs to be converted to a map. So so so there is a specific logic looking for this attributes with this prefix and then they will.

B

Collapse it into.

A

It so, whereas I was suggesting.

B

But it does mean you've got all that prefix that gets repeated every time, so it comes back to the size of the event.

A

Yeah I I was suggesting something like this. Where uh there is a there is a map you know in between. That tells you that tells the receivers.

A

When to convert to a map.

A

So so with either of these, you know we could uh drop the idea of having maps at the client side and then at the time of uh export uh or even at the time of generating, if we could flatten them.

A

And then, if we send it as an event, you know it would go as attributes on that event, individual attributes if it goes in a span or a span event, it's the same thing. It's where we we just don't need to send individual uh span events, which is uh too much.

B

Yeah, I I the the way the experimental one sends individual span events. Now, it's that's just wrong, but that was the limitation of the time.

A

Correct correct: okay! um So how do we arrive at a decision.

B

um Well, we we're I'm not quite sure the question so effectively. We have we're defining the shape of an event as event, name event, domain event, data which has all the data and all the the data for what we're looking at this. The screen here is the individual fields of that particular event.

A

B

um So the existing experimental one which we're looking at now, I think we just say it's experimental. It's not the recommended.

B

So in in this it wouldn't be events. It would be still be a single event which would have a domain a name and then the data, but because span event. If you're going to send it as a spam event that span event, data would need to be a string because right now today span event doesn't support nested attributes, which is why they sent today as an array of events.

B

That's partially why I was originally driving my spec clarification in one big: go to say: we're going to support it because protobuf supports it, but we're getting pushed back. So I'm going to divide it up in smaller chunks and based on the timing, wise. It's going to be a long pole before we get there.

A

So I think we should at least get it removed for logs if, uh like as in there has to be- I I don't know I think, for this one for this topic I mean, um I think we should time box the decision making and then just go with. You know.

B

If you go back to like what ted was originally proposing, um eventually, he wants span events to go away so span. Events would actually you you add spam events, but they actually still get sent as log events with a link to the spam.

B

Like that, that was one of one of ted's earlier um definitions, unfortunately he's not here um so yeah. The the whole thing of like having events as events like events should be events that should be the same. We're just dealing with this limitation of spans today.

A

Yeah, in that case, um you know for for app dynamics. You know we will.

A

Not put that in the events at all, we will put this data in in the original spans attribute, not the span, events attributes, but the spans attributes.

B

Okay, so it's not even an event, it's just the so in the span itself, you say that would limit in a span to only having one event, though, because we're saying event name of it domain no.

A

So so we would have to uh like, for example, uh have a annotation, a prefix. So let's say.

B

That was with the underscore map thing you were talking about, yeah.

A

Either that or even there could be different types of uh timing data right, um the performance, timing, data there are: there are different entry types, so the resource is one performance is one and there are a few other types, so we could say, timing data dot, that entry type dot fetch start.

A

You know equal to. You know this, this long value. This is long value.

B

Which is just the the plucking strategy, yeah.

C

B

Yeah you, you can make it work, but it's still going to be a lot of work, because if you just have a straight attribute, you yeah it's you'll, end up having like your http request.headers.map and then a number, um because if you've got the same type of event, then yeah, so you can make it work and that'll be that particular instrumentation. But.

A

B

A

Yeah, I think uh at this point, what we really need is a conclusion on the map type, whether attribute values can be maps um and and then the we could go ahead and make the changes um so.

B

uh Maps are nested, there are three different things.

A

Well, how are they uh different things? I I wanted to understand that too well,.

B

So you could have like they are defined as a map today, where you can have a map of values, but then that map only represents a single attribute, as opposed to a nested attribute where a an attribute can have a child which is an attribute which is not the map, so that would be a map of maps if you want to take it to that extreme.

B

So just declaring something as a map doesn't mean it's nested, that's just saying: well, I've got a map of things where this name can is only repeated once.

B

It's still a flap back like a dictionary well.

A

It depends on uh how you define a map right, a map. If we go by the current prototype of definition, uh a map will again be uh a key plus any value.

B

Yeah, it's it's not map that gives you that functionality, that's just the product, the any value that gives you that value.

A

B

Yeah, which is what I'm trying to clarify, I'm trying I tried to come from the approach of saying: protobuf supports it. So therefore everything supports it, pull stop even though the spec says span and metrics don't support it. The transport supports it. So therefore we have to define how it's handled, um but people are still pushing back and saying spans and metrics. No sorry but yeah. I agree, you know, protobuf supports it. If you, if you define protobuf to the protobuf spec, everything supports nested attributes.

B

But technically it's not the map aspect that gives you the nesting. It's the fact that the any value can contain in any value.

A

So where does that issue stand today? uh I know you're getting a pushback, but is there any uh likelihood um like, even if we uh make it accepted for uh logs alone,.

B

A

For logs it's it's only a matter of uh implementation right because the spec already allows it correct.

B

So, even if my my thing doesn't go in logs with support, so it's already there, um I said the original scope of my change was to try and effectively crack open the door a little bit. I'm now trying not to have the door slammed completely shut.

B

um I did publish an update to it this morning, so I have a table in terms of which I've exposed explicitly defined as this table. Doesn't it's not the be all in the end, all it's just a quick look up guide and I did that so that specs later can change and say span. Events can do this, but this table doesn't need to be updated for that to be supported.

B

um So the heading of that said optional, yes, no, um which made it appear that my change was enforcing, that um logs don't have to and spans do have to so I changed that wording to supported, um which is not quite slamming the door, but it is closed out further than I'd, like even.

A

The other way spans don't have to and logs have to.

B

Yeah well the the wording because I had optional and then yes, no based on almonds. It made it read around the other way. So that's why I changed the heading to say supported.

B

So it now says the heading says, supported it says: span and metrics no logs, yes, where before it was optional span and events, no, which read that they must be supported in logistics. So that was the change I made this morning and then I have a bunch of open comments which I've got no response from the last week and a half.

A

Okay, but uh how will the implementations uh handle it.

B

Yeah, I don't know, that's as part of my comments. That's part of the issue in that arman and I forget the name. The other person who's commenting are suggesting that sdks need to completely separate and have two different types of attributes: yeah log attributes and normal attributes um to try and counteract that. And I'm saying well, no, that's just overhead, especially from a browser side. That's an overhead! I do not want to do because it's yet another type.

B

We have to define what we could have is interfaces and have an extended interface, but it's just a complication. We don't need because we don't need to.

B

But yeah, who knows, I don't expect it to be checked in anytime soon.

A

B

But to say logs already supports it. We can continue moving forward by defining it and then once we define what an event looks like, we can then say: okay. Well, we want to also have this as an option to send as a span event, um and the fallback today which isn't already included in the attribute spec, is that if it's not supported it should be json encoded.

B

That's it's not mandatory, but that's the suggestion that's already in the attribute spec today and that if it's not a native one, it should be encoded and they say they're suggesting json coding.

B

Okay, so right now that would be how a span attribute would be handled or should be handled.

A

So what what is the uh concern against just using json encoding, json, encoded string.

B

A

Even in the logs case,.

B

I'm almost saying, let's just go for that, which is why I'm hesitating in responding, because that is the easier path uh when it comes to constructing it. Going jason's up stringify on a you know, resource timing is going to be a hell of a lot faster than constructing an attribute structure from the resource structure.

B

But if we end up going otlp, which is what um daniel was suggesting, is the norm today, except, I think it's the norm, because it's all node, if it's binary, encoded binary encoded, will actually be faster as of attribute than it will be as a string encoding, because the string encoding isn't going to get it's just a string, it's going to go, as is from a coming from the backend perspective.

B

If it is just json encoded, then it's nice, it's just a string. You just unencode it using whatever serial deserializer and away. We go so.

A

um So I think, uh even for the browser case, um you know we, uh you know, given that the json spec is not stable, yep. um We we will have to go with the protobuf binary protobuf.

B

A

With um so so, if, if that goes out the door first, then that will most likely be the default uh for most people.

B

Yeah, I think well, the hlp exporters, I think, are already non-exp they're already ga right so that that already exists.

A

But on the browser side um you know so the plan is that once we so basically we we also want to build a an agent package. um You know choosing some defaults, for you know uh several uh features.

B

A

Right so in that, uh if let's say you know, purvi is working on the prototype of support. So if let's say you know given that that is uh supported officially you know that would go in and if that gets released as the 1.0 version of the browser agent, then that will what uh be used by atmosphere.

B

Yep and that's fine as long as you've got somewhere to send it, um and in you know, if that, if that is the default, then having it as nested attributes will compress more than having it as a json code string.

B

We're talking bytes we're not talking like huge maps, but it will be smaller depending on the data, but generally will be smaller, like all you're going to do is look what you get on the screen here, like fetch, start two very long numbers. You know those long numbers will become four bytes or eight bytes, uh depending on the type of encoding rather than the 30 pod bytes. Just for the fetch that one because they'll get you know, they'll get encoded as integers and doubles so yeah.

B

A

A little lost uh frankly on um whether to what is the implication of this nestor attributes on if it goes through or if it doesn't go through, because at some point we have to drop the ball and accept.

A

You know one approach.

B

Yeah well, which is why I've been talking about just this data. We say the data is a nested structure. How that nested structure eventually gets sent, whether it's as a bunch of nested attributes or encoded?

B

From my perspective, that's the exporter's role, not an issue within the case of defining what the event looks like um so, at the exporter level. It's then you know, you've got the encoding and the decoding on both sides um in the specs that we're defining we're not actually defining directly. What that exporting is like yet.

B

um For logs, necessary attributes is supported, so we're saying it's a bunch of nested attributes, but having that as an option to convert that to a json structure, even if it's a jason structure of nested attributes, which would also work, um it's still valid, it doesn't change the shape of the event.

B

It does change how you go and implement the back end and get your data there. I agree with that which I think, where you're coming from santosh you want to, you want to be able to like export it and have it imported and start using it. Now.

B

um I I think from an experimental perspective that would be nice, but I still think we're a little bit away from.

A

uh But even in that approach, where you leave the decision to the exporter, it may still not be possible to generate a map for an attribute value to begin with, because uh you know not, and I'm not talking just about javascript, I'm even including, um let's say java for android and ios.

A

um And I and I don't know whether they have this requirement. Whether this is applicable, that if it's only let's say javascript, then we could add the support for.

A

An attribute value being a map, and and then you know, you generate a map at the source. But then the exporter will, depending on what's configured, it will choose the appropriate method for either flattening or moving it to a json or camera.

B

And that's what we value today is the map already.

A

It's already a map.

B

A

Even though these uh spec doesn't allow it.

B

um Not even even for a span, so it's an attribute. It is a map right. So no.

A

Attribute value.

B

uh Let me go have a look um because I know they define it as a single attribute value was uh where's. My.

B

Let me go to my specs. I've got links in that.

B

Any value, so any value today is a string bull into double array, keylist, which is a map for bytes.

B

And then keylist is a key value here. So keylist is is the is the map today. So it's value six.

A

So yeah, and and is that uh common with the node as well.

B

um I believe somebody tried to find the javascript implementation.

A

Because if it was um common with node, then how come the node folks don't have an objection.

B

uh Don't have the objection to which the the map.

A

To the map, as the value of an attribute.

B

Well, I think again, you you're confusing map with nested map is not nesting.

A

But even the map uh isn't allowed, as per the spec right.

B

Well, our map is allowed.

A

Not not per the protobuf, you know, uh as per the spec, because the other languages do not support it.

A

Like let me show you for java what I mean.

B

Let me go back to the attributes.

B

um Yeah in the spec, it actually defines it as a an attribute collection, which is a map.

B

They're saying resources, metrics data points spans, spain, events span, links and record logs may contain a collection of attributes, so your maps are already fully supported.

B

It's just the subtleness that a attribute value may contain an attribute value. That's the bit! That's the nesting, that's gotcha, which is only defined for logs.

B

And the key point of the collection is the actual map. Name can't be repeated by a dictionary.

A

So I'm looking at the java implementation of attributes and there, the attribute builder interface, uh for the put, I only see the the second parameter value being string along double pull and.

B

An attribute is defined to only have the native things that you're looking at today, but there is also the concept of an attribute collection, which is the map of attributes.

B

So if you go and look at a the span, attributes you'll find the span. Attributes is an attribute collection.

A

Yeah, but uh an attributes value cannot be an attribute collection. Can it be.

B

uh An attribute value- I don't want to go back at the top here- uh an accurate value, primitive string, building, double precision, int an array uh and then I've added the nested so yeah. So an attribute value itself cannot be a map.

A

Yeah- and that is what uh we're talking about- isn't it.

B

uh Indirectly, because a nested attribute isn't a map either a nested attribute is just another attribute.

A

So either ways my my, I think it's a little confusing.

B

I'm missing something here as part of the nesting at root. uh Is it a case of it's just actually yeah? I might be missing something here, a nested attribute, so in any value.

A

So this is uh this is what uh you know, um I'm I'm seeking right. So an attribute look at this and this is uh a collection of attributes. uh You know this structure and it has one attribute called timing. Data whose value is is a map.

A

So so this uh structure is it possible uh today.

B

Yeah, that's the I think, I'm missing. I may.

C

It's not possible like this, but it's possible in I mean.

A

C

Protocol it's possible, but.

B

C

Different than this, though,.

B

Yeah now it is possible to protocol because and any value can contain a key value list, which is a map. So yes,.

C

But the but it's but it's the key value list is uh key value is is, is is an array of of uh key values which is a string and then the any value, and anyway it looks a little bit different right. I mean it doesn't. Look like this is like a simplified map, whereas, like the in the protocol, the json protocol would look more, it would have more more structure.

C

A

For each value, yeah.

A

Yeah either in that form, or you know this form- I I don't think it's possible today, uh at least on the java side. I I don't know about the javascript.

B

uh Yeah, I think the the key value list is not supported today for spans. I think that was added for logs, so once people start implementing logs, that's where we're going to start getting hitting the issue, because they'll need to support this key value in this.

C

So so is it implemented anywhere anywhere like use for logs? You said.

B

Has anyone implemented logs yet.

C

I mean is, it is, is the nested uh attributes or the key value attributes implemented anywhere.

A

What logs or log.

B

That was my question. Is anyone implemented mugs? Yet I I don't know.

A

Yeah, I don't think so.

C

I mean logs, the dog sdk is implemented in java and.

A

C

A

Said in python too, so let me check in python.

C

But I I don't probably not I I doubt that it's that it's implemented the key value list.

B

Now you got a picture.

A

Yeah, I'm not next of valid, attribute value types is bull string by its interfloat yeah. Even in python, they don't support.

B

Straight attributes so have they defined? Did I didn't see logs there listed so because the only place that's going going to exist is for anyone who's implemented logs.

A

Yeah, I I don't see anything related to the locks here.

B

A

uh Actually, it should be in the sdk.

A

C

I mean the thing with this: you know that that's kind of my mind also is is whether there are any back ends that they could actually uh in you know, parse or interpret those those kinds of values.

C

My my guess is, is probably not right now.

B

um Yeah, like the collector, will do it, but in terms of the exporters, the exporter's probably done to it today,.

C

Yeah and like vendor backhands too.

C

um Yeah, I I did discuss this with uh with our ingest um and you're, like you already, could not do that right now, like we, don't we don't? uh We cannot parse uh complex types.

C

So I would be I'd be curious, like what uh what other vendors are thinking. As far as supporting this.

A

You know, I think none of them support like it. It has to. They all, have to follow this spec right. I think that's why it wasn't uh added yeah now and even even us, you know we need it um for the purpose of uh you know a smaller payload size and and also to avoid unnecessary conversion.

A

You know you, the data is generated as a map and the receiver also needs it as a map, so why unnecessarily convert in between, um but but that is where uh you know, I'm now thinking in the interest of both time and simplicity, that you know we from our side as well. You know we drop the idea of uh you know. Nested attributes.

B

Because I I you know, in terms of our sdk, we have a custom field which we haven't even talked about, yet that lets applications put in their own data and that is nested.

A

Yeah, so so in, even if, for all such scenarios we will have to, uh you know, provide a way to flatten that our you know the either flatten it or we. You have three options right. You know the nested protobuf as a json or as a flattened. uh You know uh as a flattened set of attributes. So the first option you know, given that it's not possible, then you know it's. The other two you'll have to go with either as a json. Encoded string are a separate set of flattened attributes.

B

In terms of where it's at today, because no one's even implemented logs for the logs export.

A

Now, even even even for the logs, I think uh you know we could we could, you know close the loophole and and then suggest a change to the spec to not support logs, because it looks like none of the sdks you know want to do it.

B

Today, yeah because they have they haven't implemented the logs.

B

Like from a microsoft perspective, we have to support it, we have to support customers being able to send nested attributes and it's unstructured data. It's they. They they define what the data looks like.

A

Actually, actually, if you are, um you know really strong about it, then I think this is a good time to explicitly point out in the logs api spec, because that is an end user callable api. So far, it has been only at the sdk level, which means you know it will be only used behind the scenes.

A

So if now, with the api, we are giving the ability for end users to create log records and in that spec you know, people might be missing it, but you know we can remind them by by talking about it explicitly, given that respect supports it, like.

B

I I think, if you look back at uh like tigran's original short code, he explicitly called it out there and in terms of like the the nesting he gave. Examples and people have provided examples of nested attributes. So I I don't think it's going missing. I think it's just the case of sdks haven't implemented it yet and haven't hit the complications of having to have different types of acrobats.

A

What I meant is that they better think about it now than later, because it might not be crossing their minds now, while reviewing the spec.

A

But but you're sure that a map support is required, like you can't do away with not having.

B

I I could not do away with not having map support yeah. That was a double negative. I need.

A

That what what about the map? Support for uh spans.

B

Ideally, map support for spans um would be beneficial because any event today that I send for application, insights or the internal version can support nested attributes which are supplied by the customer, not by me, not by the sdk.

A

So so maybe you know even for customer supplied uh maps, you know you, you know your sdk could flat could convert to a json encoded string, wouldn't.

B

A

B

Well, that's that's the exporter's role, not the sdk's role, so the exporter, yes, might just convert it to a flat nested json encode! That's that's the point at which the it is the exporter's role to convert the data into a form that can be consumed by the back end.

B

100 agree with that statement how they do that is entirely up to the exporter and I foresee, while there is a common otlp exporter that will send to a collector um and back ends, you know vendors may choose to only implement the collector as the only endpoint, which is one option that we're looking at once it hits the collector the collector protocol supports it.

B

So therefore they have to come up with the strategy to convert at that point anyway, because, while we're defining what the open telemetry provided code support that doesn't stop anyone else from doing it like um ikea, uh have already have their own definition and their own code to produce otlp they're using none of the open telemetry code.

A

So there is one statement you made, which is uh kind of uh you know odd. To me, uh which is uh the sdk is supporting uh the nested attributes, but the exporter you know, choosing to convert it to the format uh the dotel pinitz.

B

Not the format otlp needs the format that the the vendor needs.

A

I of the vendor needs.

B

Because the so like, for example, today, we have exporters for open, telemetry so effectively, everything in front that the application uses is all open telemetry, but then we have our own exporters for sending to azure monitor from node or in java or to our internal version. So in fact we have two different exporters that convert. We don't use a collector today for those two okay and those exporters do the conversion.

A

Okay, we are out of time um yeah I mean, um I think, either approaches have you know pros and cons. So we we just need to uh agree on our consensus. um Yeah.

B

So I I guess to summarize what I'm saying is an event: data is a bunch of nested attributes and we're defining them as nested attributes, so they will go to a collector as necessary attributes, but the if it's going to an exporter and not an otlp exporter, they can choose to convert that to a json form.

B

We could define it and say event: data is either nested attributes or a string, and therefore the string is already pre json encoded now from a browser that would probably be more efficient from a cpu perspective.

B

And a json exporter expected, but not necessarily an otlp exporter size, place.

B

But in terms of defining the shape of the event, I still want the shape of the event to be defined as a map of maps. The nested attribute attributes not just a straight map, which is technically just a straight dictionary, which is flat.

C

So I wonder like, if um are we gonna add like? Are we basically going in the direction of having of sending everything in a single in a single event, data attribute, and if that's the case like, should we open? Should there be like some kind of issue like to discuss this with the community or like or proposal?

C

uh That's that's. What, like, the sdks should do to implement, sending all attributes for all events.

B

I I thought: that's what we had in terms of the when we were defining the api, so we had the api, take a name, a domain and then the attributes which effectively is the data.

C

So I think this the proposal didn't have like the event. It had event name even domain, but it didn't have anything about how the data sent the data.

B

Because yeah, once we define an event that an event always has those three, then we just go and define the shape of the events.

C

Yeah, so we would, I think, am I correct, like sometimes they're, like that old tap on the specification that you have like, doesn't talk about that, how it's, how the the data is sent for the event?

C

Sorry said it again, so like the the old tab that you had and the spec pr uh for the event api, it only talks about name and domain, but does not talk about the data.

A

I added it today for the um but yeah. No, no, I'm sorry! I you're talking about uh yeah yeah. We! I don't speak about data yeah yeah. What I added today is the is the enumeration some possible values for the domain. uh But if you're talking about the data yeah, we haven't specified that.

C

So so what do we need to do like to to specify that like? Should we should we open an issue and like for for discussion with with other people, um or should we just add it to the pr.

A

No, I think uh there is a bigger problem. I think we have a. We have a much larger problem. I think you know until 10 days back or we have some other tc member. Who is uh you know, working with us closely? I think I think we are we. I think it's I. I don't think we are able to make a dent.

A

Because it needs um you know discussing in detail on a call, not just you know, like a 10 minutes in that spec meeting or or on on on the issues and beers.

B

Yeah on our three-month issue trail over pr's now.

A

No, even uh our uh you know this six started, you know I was just telling somebody the other day that we martin and I I think we started almost nine to ten months ago right and uh it's uh going, you know very.

C

A

It's yeah, it's going slow.

C

So so we are like it's so it seems like like we are very they're, basically blocked then like on this, like we can't, we can't start implementing events until this is resolved, so we need, or we make compromises.

A

Like we, we uh prioritize uh testing the prototype and compression and then give up.

A

You know some of the other requirements.

B

No, we we can still keep moving forward like, but for for javascript we need to have the log implementation, which I think, martin you've already started on. So we need to have that and then we need to have an implementation of the api, um and really all we're talking about here is how we're sending the data object right.

C

Yeah yeah, that's right! So.

B

That's trivial in comparison to the other work that we need to do so we're not blocked. We have.

A

But we will soon be blocked now. I think.

B

Well, now, logs logs already logs defines the key value this. So if we, if we move forward and say okay well, this is only going to work on logs. It will work on logs once logs is implemented. That first.

A

So so, let's uh then um take it up with uh the j6 tomorrow and see what they think uh where like there are. There are uh two approaches right to supporting uh nested attributes in logs.

A

You know you either have separate definitions for attribute basically two separate apis for attribute values, yep one for the purpose of spans and one for the purpose of logs or have a common and then you know have it support uh the yeah yeah. The.

B

Way the specs define today they have to be separate. You can have the logs attribute, extend the span attribute, um but there's already today, there's attributes defined in the api and then there's span attributes which was completely defined separately, but it's now defined as an alias to attributes. um So there's already the pre-concept of span attributes. So I think it's already going to end up being log attributes which I ideally wouldn't like, but right now as per spec, there is no other way to do it.

A

C

So I was wondering just like just to step back step. One take one step back um like should we like, should we do? Do you think, do you think it's we can move forward with the assumption that we'll be sending all everything in one one attribute the data.

A

In one attribute.

C

Or should we should we uh open an issue so that, like you know, some people from the tc could come in.

B

um Perhaps we should open a a discussion on the general shape of an event and see what sort of pushback we get yeah.

B

Okay. So it's saying that the general shape of a of an event will be name domain and data and then maybe custom data, I'm still in two minds where that custom data should be. But if we're going to have data as a top level, then maybe we say event dot application or event.custom or something, and that would solve that problem.

A

I think the the problem with uh one one um could be minor one, but one issue I see with that approach. Is your api spec to set the data? um You know I mean you'll be able to set the data only once even using one. You know call.

B

Which is what you can do now right with the api. You, you emit an event and you give it.

A

No, no, so so when you are constructing the event today, you can say event dot set attribute and you can call you can set multiple independent attributes uh that are part of the data. Let's say your data, has you know three different, uh you know things so you could set them separately, whereas with the new form the client we'll have to construct a you know a map and then set that against the data using set attribute.

B

Yep otherwise, because if you don't you're going to say, create me an event, I had all the attributes of the event and now met the event. I I thought the api was a straight emit the event. It wasn't a builder pattern, at least in the spec level yeah.

B

So if it is straight emit, then you've already got that problem, because once you say, omit the event for this name and hear the attributes you've already had to pre-construct the attributes anyway.

B

B

Anyway, we are at time a little.

A

uh Martin, do you want to at least uh take it up with the j6 tomorrow on on on the topic of uh supporting the nested type for logs and how? How should you know what would be an acceptable implementation.

C

um Yeah we can.

B

We can raise it, but I'm I'm pretty sure daniel's just going to say depending the other maintainers. Are there they're just going to say, it'll be a separate implementation, because that's the only way to deal with and making sure you stay spec compliant. Today, yeah.

A

Okay, then maybe you could, uh you know, update your pr with that then have a separate implementation.

B

I have a discussion in there talking about this problem um that once people start implementing logs they're going to hit this issue and the person called out that that they should implement it as a separate implementation, and I pushed back and sort of said that that's part of the issue, so that's still an open conversation. I didn't resolve that because I replied to that over a week ago. Oh speaking of replies, I'm really bad at slack.

B

So if there's something on slack that you want my comment on, if you could tag me because technically we're not allowed to use slack inside of microsoft, um we're allowed to use it for the open, telemetry stuff, but as a consequence, it sits in the corner and gets ignored most of the time because I have psych, I have teams, I have email.

B

I have too many ways to get interrupted.

A

Okay, um so yeah, we can talk about this uh in the loxic as well. Tomorrow, yeah yeah, okay,.

B

Yeah tigran may have already hit this issue where they came to define it for logs anyway. So you.

A

Know I think he he agrees. I think uh we we just need to figure out a way to you know, move it forward.

B

Yeah, how we get the rest of open telemetry, though agree yeah yeah, because I think ted fundamentally agrees with nesting tigran, fundamentally, groups of nesting.

B

We have all this pushback from the back end exporters. As far as I can tell so. The people who originally just implemented the exporters said: okay, my my you know, prometheus or jaeger only doesn't support nesting. So therefore you can't send me nesting. So therefore, the sdk can't support nesting this. That was just like the lowest common denominator.

B

Yeah, I think that's where it all came from, but there's now years, where of history, to try and move the snow against.

A

Yeah, okay, all right cool, good, okay, see you tomorrow,.