Open Telemetry Uncategorized, 18 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-10-18 meeting

Description

Instrumentation: Messaging

A

A

A

B

Nice Santosh.

C

uh Hi hi Martin how.

A

Are you good good, what's going on.

A

B

That a virtual background or a real.

C

B

It's real okay, wow. My room has to be really clean because people you're not the first one who asked.

B

B

Hello, all right.

A

D

The schema uh you know stick that started us today: yeah.

C

I did not Monday right yeah. Yesterday, yeah.

D

Yeah I missed it; there I realized it around 9, 15 or something. The thing is going on. Yeah I mean uh okay,.

A

Yeah I'll try next week, yeah.

A

So um yeah before.

C

We get started on the main topic. uh The schema I have one quick question, um uh so there is a um this is a question to uh nav to do and uh Martin.

C

um Basically, there is a discussion going on in the log Sig um about you know, making the events more clearer uh against logs there's a lot of uh confusion and some new um ideas are emerging to rename events to something else.

C

um So in that context there was a definition of what is an event that was put out in the spec, and it said that event is a log record where the there is a name attribute, and then there is a uh name, a trip event.name. As an attribute on the log record and event.domain attribute on the scope to me, it was uh complex definition of what an event is.

C

You know to me an event is just a log record with you know these two attributes and you know no matter I, don't want to uh go into the details of you know where these attributes actually end up on The Wire.

C

um So um in in that discussion um you know there was uh you know. One argument was that you know we ended up putting the domain attribute on the scope. You know you just duplicate it on every log record which I think for for our purpose for the browser instrumentation, you know we we don't prefer, because we wanted the pillar to be smaller, um whereas if we keep it on the scope, uh the you know, the the language, the the definition of an event becomes complex, but then uh I learned um this morning.

C

Let me share my screen that um that the definition of a log record today.

C

um Like if you go to specification logs, um there's a data model, so the log record data model. um If you look at the.

C

Fields, you know they have resource and instrumentation scope and according to tigran in a conceptually, uh these are part of a log record. Although you know in in a in a strictly speaking under the wire, you know they might come in in other places. So it's a little conflicting to me um so with this argument, I can keep the definition simple, but still keep the event.domain at the scope level. Yeah.

E

um So I I, just so in the spec seg, which was just prior to here, uh Tigger and opened up a new PR, which I've placed into the chat um to clarify the instrumentation scope, will be build time. Values only uh which means putting the event domain in the instrumentation scope will be not allowed. Oh.

E

And I think we have we've actually already defined some other events where we wanted to put the value into instrumentation scope, and this PR is saying that that should not be done. uh He specifically called that yeah I think the process ID thread ID and session IDs are currently being put in there today, uh which they should not be.

B

So, okay, that's surprising to me because he was I think he was like a while ago when we talked about this. That's what.

E

I think it was too yeah exactly all the all. The uh the immutable values would end up going into scope, but now he's raised this well and saying: no so.

B

Okay, okay, I guess: I have to read this. His reasoning, yeah.

C

Okay, so with this um with this, then okay, so the domain will move into the individual log record. Then yeah, okay, but which is not good for us.

E

um No well originally I was thinking that effectively, it would be like you know. The event name would be browser dot whatever um and splitting it out into a separate thing. So you had a bit dot domain, an event.name, it's like okay, so I was conceptually thinking that it was actually just part of every event anyway, um and it was only really that you effectively created a a log of antimatter which defined the domain and I went okay. Fine. That means we can.

E

We can have it as one um if there's still a place on the log. You know the the transport to be in one place. That would be nice, but yeah.

C

Well, I think the problem, um you know it could be minor, but it is still uh it's going to be repeating on every signal correct and in our case it's it's fixed browser right. It is not even like a session ID, it's not even varying anything. It's.

E

Going to depend on the events that we're emitting. um If we do come across some common events, then it won't always be browser.

D

uh Can you can you give us an example, a common event, you know what would.

E

Be a little like page view, so so, if we identify that page view is common enough between mobile and browser and something else, maybe we don't Define it as browser.pageview.

E

Probably a bad example, because I think that one will be browser, but.

D

Yeah I I right after that I think it would not be the same enough. Phd probably is not going to be the same. You know a lot of the attributes are going to be common, but um the page URL that we talk about browser. You know mobile devices, don't have that concept right.

E

Yeah exactly, which is why uh it was a bad example, but.

A

E

Yeah there may or may not be an event which we say this is common enough between the different domains um like for mobile, that we, we say: okay, let's just Define the event once, um but we could also say: let's define the event once, but it could be in multiple domains. That's which is another way we can do it so.

A

C

Okay, so um then looks like uh we don't have much choice, right, I, think if, if this PR.

E

Ior the first one goes in then yeah we've got to just put on the log record, okay yeah, which will happen as part of the log implementations. So, okay.

A

Okay, all right.

C

Then yeah then that's it.

D

Okay, so the last piece I, don't you know fully follow? uh What did we say if this PR goes in, you know we'll end up using record. Is that what we.

C

Said and no so um so, how.

A

Do I say it so.

C

I'll show you an example: I'll.

A

C

Here there is a scope um and the you know in the today's modeling you know we had uh you know attributes, uh and then there was a domain. Sorry there was a event: dot domain equals browser right. So this is how, as of today, uh all events will have the domain uh at the scope, level right and and then the event.name is inside an event.

C

Even dot name is inside an event so and an event is a log record that has to have event.name and event.domain.

A

E

So this will also I mean if we had custom ERS created custom domains, they would affect me that would have to be sent as a different payload with it sitting in the scope, foreign.

E

Events which are like browser dot whatever, but then they emit their own custom event yeah.

C

So for the customer event, um you're right I think they will have to obtain a different logger uh with uh with the different event.domain in that scope, yeah.

E

um But in terms of coalescing into a single packet, leaving the browser um so I'm not familiar, where does the scope believe in terms.

A

E

Know the um outbound.

C

So the scope is definitely enclosing all the spans, but if the scope is different, let's say the same: fetch uh Sim Library uh is is used to emit logs. But in one case you know the you know the the domain is different. uh Then they will have to come in. Let's say the scope spans is an array yeah right, so this yeah.

E

I think that's a modern question when I look at it: okay! Well, the instrumentation is the runtime, so it would actually be different instruments. The same question as when you have multiple instrumentations. uh There must be a way to have it there. So yeah yeah.

C

uh But whereas with this change now um this attribute will move wave up to nine yeah yeah, it will move here so, which means that you know this is going to repeat.

A

Yeah, which is not ideal but.

A

C

Okay yeah, then, uh if we can't do much, then we can't do much question.

A

If the event domain is related to the instrumentation being used it trying to Define that build time of when you build this rotation.

C

Only only cases, custom events, so so you're right that, uh maybe maybe we could go that way too- that for all the instrumentation generated events, the default uh events, you know we could hard code this at build time.

E

That we end up creating a browser event instrumentation that effectively packages limits of all the browser events yeah. That would be technically a way to work around that yeah. Then.

C

It could I.

A

Guess, you're not going to have a lot of cross domain instrumentations anyway, like maybe fetch, is in browser and track native. But like that's kind of.

C

Yeah I, actually uh can we uh find out one thing that, uh irrespective of where uh this domain is whether at the scope level or at the log record level, where do the back ends? What do the backends want? My understanding is uh only on the wire for the purpose of wire efficiency. You know we want it here, but eventually they they will end up going closer yeah, because.

C

um It it will be easier for the you know, the backend logic to look at you know both these attributes at one place um rather than two separate places.

C

All of these events end up going into an event store. uh Then, in that case too, you know they. You know there will be events coming from different domains. You know they will all go into the same storage, in which case you know these two. You know they will be together.

E

Yeah yeah, it would, depending on how the how the back end's plot the output over all the different Scopes resources, Etc um yeah,.

C

Yeah, and one thing I noticed, is uh uh a lot of distance here. I, don't know how many, how much people also represent the back-end teams?

C

um So maybe you know if you also could find out what your back end folks think like like what I'm trying to get to is um the event dot domain. It is okay for it to be at scope level in some cases, at the log record level. In some other cases, the back ends will will normalize it and move them all in in one form when they receive it.

D

Okay, this is for the packets that directly process otlp signals right.

E

Yeah that receive it yeah.

D

um Currently, you know from our side the backend teams are not really looking into it, they're just starting to describe to surface in terms of enabling otlp based ingestion, um it's starting as a POC this this semester, um so so far, everything that we have in mind is that will protect the back end by doing the snapping. You know our exporters will take care of that.

D

So anything the exporters will have to massage the data and try to get it to the same anyway. We, you know none of this matters to our current backends sure sure, uh but this will actually be a question for people that have uh otlp capable backends at the moment. Yeah um we need their participation, see if they're you know, participating there and influencing these decisions.

A

E

um Probably the other note on this based on this PR I, think and I could be wrong uh in our spreadsheet. We actually defined some attributes that we were going to put at the instrumentation scope level, so we probably need to go and just tag those so that if this PR goes in which, based on the feedback that came in the respect, meaning it probably is very likely to that.

A

E

Can come back and address those.

C

Oh, um which ones in our I.

E

Don't remember which.

D

One click scan I couldn't find any. You know it probably. You know it should be in the description Fields. If, if any uh I did a quick scan, while you guys were talking I couldn't find any, maybe you know.

C

But we haven't uh put anything at the scope level. It's mostly at the resource.

A

Card: okay, now.

E

Yeah, like we talked about it and then didn't do it yeah, okay,.

C

Okay, so um to to conclude here, um it's it's possible that we might still continue to put this attribute at the scope level, but at the same time for custom events, the same attribute could either go at the scope or at the individual log record level. So we we still want the spec to allow for both scenarios. What.

B

C

This is that what we want.

C

E

Okay, I I, guess terrific. Ideally we want to keep the payload as full as possible. uh You know your question about how the back end is going to deal with it. That I think sort of needs to be answered a little bit as well, um because by saying it could be an either we're effectively complicating.

A

E

Then we have to Define which one takes precedence, which.

C

E

Would be the event but um yeah so.

A

Even smoke becomes difficult, yeah a.

D

Question on the event that domain right, I I know we discussed about it early on in things. Does it have to be a separate thing, or why can't it be just the namespace if they even name itself.

E

D

Yeah, the only downside is, if there's a common event across browsers and uh mobile devices, say you know we have a problem, you know the easiest solution, for that would be crude solution is just duplicate the event you know, call it different and there are going to be subtle. Differences might as well. Do it.

C

The reason it was introduced to the domain attribute is uh the open. Telemetry agents will be collecting events from different. You know domains different business contexts.

A

C

To to avoid any potential conflicts, if you know there could be events with same name but could mean different things right to avoid to resolve that conflict. uh This domain was.

D

Understood so namespacing is probably the better way right. So that's how we have traditionally done it internally. It's all namespaced yeah and the only downside of doing the namespace is, if there's a common event, then you know the moment. You call something browser.ph view you cannot use it for anything other than browser which I think is okay. You know if it if it seems like exact event, we'll make a copy and then call it like that that at least gives it good money.

B

I was gonna say that there's also second downside, which is the repetition of the namespace.

D

That's correct.

C

Reputation, I think it is coming back. uh Martin.

B

I know but, like you know like where, if at least like, if you keep it separate, we may we might have the option of putting it on the scope.

A

Right yeah, I, agree, yeah.

E

Or if we rework a little bit, it's okay, the domain is hotel and then we say the the name is browser. whatever. So therefore we could say Hotel will never change. So, therefore, I could say on the scope. It is a build time.

C

But Domain Hotel is not very helpful right because there is other teams. Other instrumentations, that um like domain, is not serving any purpose.

E

Well, it's serving the purpose for dealing with Hotel defined ones versus someone else defining it, which I think was the um pigman's original uh thing in the log sick. So so, if we then say the event name is then unique by saying your browser.pageview?

E

Okay, it's still a combination of Hotel browser, page view that makes it unique so.

D

Another thing I would like to bring up. um You know when we talk about the payload, um optimization and stuff right, the repetition and things um I'll. You know I'll just ask this question right now. How often do we? You know what is the common batch size for browser events?

D

You know, how often do we flush um we're, probably not talking hundreds of thousands of events in a single patch, where this optimization will pay a lot of dividend right if we are generally doing it on average, the browsers, usually just move pages in under pretty small batches. That's my take on it. You know, I'll leave it up to you guys. You know you know you have experience, you can talk about it. Nev does a lot of batching in um are our staff.

D

So, in our name I'd like to hear your thoughts too, we might be giving too much weightage for optimization in the browser. Then it's actually required I I would say uh if all you're talking about is 10 events on an average in a batch yeah, you know putting it in the scope there and then paying the price for somebody to take it out and then put it back in. There might not justify it and I'd like us to discuss about that. A little bit.

E

It depends on the application, so we have some teams where yeah they they do. A trivial number of events, like all the auto attached stuff, there's a trivial number of events um if they start emitting their own events.

A

E

The 64k unload limit goes out the door pretty quick, um even with our you know, with the 1DS one and two second flushing. um There are teams that effectively we lose events, because we can't send them all because of the 64k limit. Today,.

D

Right I mean we expect the badges right, so the 64k limit. um I guess that's that's another thing also.

E

um Yeah because send Beacon effectively it's a 64k limit of all outstanding requests, so even splitting it which we do try and do we try and send a a single Cent Beacon and if it says no I can't do that. We then start sending each individual one at a separate event until we get to the Limit um because yeah, if you've got 128 uh K of events, you're we're only ever going to be Alison, that's the first 64k, even in multiple batches because of electron Firefox, because the way chromium handles it is effectively.

E

It's all outstanding events that the browser has backed. It's, not the fact that you've you've sent sent Beacon and okay now I can do another, a lot of people. Okay, that's not the way it works.

D

Are you talking about unload time frame right when you're unloading.

E

Only during Android correct.

D

Correct so in in normal case and I think uh yeah that that's definitely a legit use case the normal times.

D

um Let's say you have a bunch of events that you want to send I think we were talking about um optimization. If I have to send 100 events, I, don't want to repeat the browser 100 times exactly.

D

um That is what I was going after. It's like maybe.

C

D

C

Also getting to the same uh conclusion that maybe we should now that the compression API is uh coming, we could be more forward-looking um and, and then maybe in in scenarios in in for applications that are uh that emit a lot of events that emit a lot of data. You know, maybe they can enable compression um and and leave it to them. Yeah.

E

uh Which will work for probably some cases like I haven't looked in depth at the the person like does anyone know? Is it anything interface which I'm assuming it probably is.

A

E

So I miss that.

A

It's piping paste.

E

Okay, so it's synchronous- oh that's nice, um which means we could use Fetch with keep alive during unload, because we can't use send Beacon because we can't send any headers. Okay.

B

There's no question about performance right I mean it's going to take some there's going to be some overhead, so.

A

B

E

But it's gonna be native um compression, so it's not going to be as bad as you know: we're not paying the payload of getting Jesus and then the CPU that effectively do compression in JavaScript.

C

Yeah, it's not going to block the main thread. It will hopefully happen in the background. Yeah.

E

um And realistically, by the time this rolls out, most browsers should support it, because today it's only chromium, the current chromium ones, right I, don't remember what version it came in. It's like 100 and something.

D

Sir uh version support chromium, 80, I think.

E

80, okay I didn't realize that long ago,.

A

E

A

C

Yeah, it's only Firefox and Safari that don't support it, but pretty much.

E

Any chromium price browser should it should get a butterfly yeah.

C

Yeah and yeah I think very likely. The all browsers will support in a few years yeah. Maybe so, if we look uh forward, then maybe compression is not that bad.

C

Now it simplifies our our discussions right now. We could make faster progress. Yep.

B

I would I would want the only thing that I would add. Is that, like we do still have like I know that we still have a lot of users on Old browsers, so it's gonna take take a while before like you, you can. You can like apply that logic like for everyone, so there's going to be there's gonna, there's gonna have to be some kind of fallback mechanism for older, browsers, yeah. The.

C

Problem is, in that case it's going to be bigger size on The, Wire.

D

Yeah, the the payload uh price is going to be heavier uh I. Guess what we're? What sometimes was saying was that compression is an option that they can turn on. uh Not you know we're not going to turn. You know we're not going to just turn on by default or anything like that. Somebody says still so much payload price we're paying and things and then the option is you can turn on compression but understand your user base will have to come. You know it will have to be supporting this.

E

It's like the fun stuff of browsers effectively the the browsers that don't support compression are also the ones which are older and echo, or with uh pay more for the price, because they're yeah they're going to be using more bandwidth incentive.

D

For them to move security.

A

B

Saying sorry, sorry I was just going to say that it's like it's not the like the the customers, just like the owners of the website like the web applications don't have control over it. Yeah yeah.

C

And also um wire size wise, um if, if we we could enable this on, our customers could enable this on browsers running on mobile devices, which are which are more likely to work on mobile network, um whereas the desktops you know more likely are going to use Wi-Fi more more often than not. uh It's.

E

C

A standard but.

E

Yeah, if you look at this, like the fact that Safari doesn't have it, it's like there's all your Apple devices, yeah.

C

Okay, um so yeah I think um then I'll not say much uh in the log segment. On this point, I think it looks like even the domain will move into um the individual log records and uh as.

D

A namespace for the event name is that what we're going with or do we want a separate field.

C

I think that discussion is already over I I. Think if, if we were to I think what you are indicating is uh you you want to put browser Dot, Page view right, yeah I think we had discussed that at length a few months ago and uh we it.

E

Pretty much got rejected at the log Sig level. um I think that was the original movement, which is um just have the single event name.

C

Yeah I don't remember now, but um I think it was easier if we retain the original name event name um yeah I, but but I think it will be like going back on on I think there are already apis defined. uh Assuming this is going to be a separate attribute.

B

C

B

There's still experimental, so we could I mean if we had a strong, no.

C

I think it's going to a lot of churn, so I I prefer yeah. I know it's like progress, uh rather than be right. So much yeah.

E

So the only way we can get around, that would be to say we Define the domain as hotel or hotel ramp or something and then have the name as the browser.pg.

E

That's a hack, but in terms of that you know there will be two Fields. uh The value of those few Fields is what we're going to be defining as part of what we're doing here so.

D

Now we're going to keep even that domain and you know I I, don't think we need to do that. Hack, uh I'll just go and it just separate Fields. Definitely it's going to be a little bit more uh for the payload size, also, if you're gonna stamp it to every event. um That's what I was thinking, but if you, if we can't combine them yeah, that's fine.

C

Yeah I think from a back-end perspective. Having separate Fields is easier.

A

C

As they are looking at fields for routing purposes, they only look at event.domain.

C

um If it is even dot name, then they have to look at every event. Split to a string, split pick the first part and then do it slightly a complex, not a big deal, a.

D

Little bit more complex but yeah, I I, agree: yeah, okay,.

A

E

The other thing we came up in the meeting was this PR um yeah, so I think this is. This: is a leftover from um uh Tigger and pick this up. So this is effectively nested attributes.

E

um No did I paste the wrong way to yeah yeah, two Triple Eight, um so yeah, it's it's effectively, reactivating the original 376 and started restarting the discussion of uh this takes it further than mine, where it's effectively attempting to Define nested attributes for everything which would be great to have.

A

C

Is it even for resource.

E

This will be for everything anything that is an attribute that um supporting it, uh where it just explicitly stating that if you're you're, Jaeger so Jaeger or Zipkin that affected it should be Jason and Cody, okay, um but yeah. It's really that key value map aspect of it. So.

C

So I actually an uh an off topic, but it's been bothering me: why does open Telemetry um care so much about you know: non-otlp destinations, the Jager and Zipkin like aren't they supposed to evolve at the same time to keep up with open telemetry.

E

um It's all the way they store so I think uh yeah and Prometheus. Don't really have uh I'm, not an expert on it. So I could be saying the wrong thing here: uh the ability to stored uh data in a nested form, so it has to be planned so there's part of it when they Define the otlp spec. They defined it as planned. So the factory, the receivers just get it in store, and that happened to the conversion.

D

Okay, does it does that answer your question centers, um you asked, you know. Why is you know why are ignorance, looking special yeah.

C

Yeah I'm not fully convinced yeah.

D

I had this question also early on when uh you know when we started working on over delivered staff, my takeaway after talking to a few people, was that I think eager and Zipkin, you know ended up being the different de facto destinations or something when the project was so young yeah and they contributed their stuff into open, Telemetry and probably there's an implicit you know thinking or um was uh obligation or whatever it is to treat that as silver slash.

D

um That's basically what it is. If you look at the way the the exporters are set up uh every other exporter other than eager Zipkin and from I think for metrics. uh Everything else is in contribute or something like that, not the main mainstream repos. So that kind of tells you these are first class. You don't want to get into the discussion of first class, which is the second class things, but these are first class supported stuff. That was my takeaway I.

D

Don't think anybody would um you know validate that or whatever that was my takeaway from a while ago. Okay,.

A

C

Okay thanks, so we can switch to the other topic: um Gamers, okay, yeah, schemas and um I added an agenda item to to basically understand the next steps like I.

D

You had a review of the timeline before that. Do.

C

You want to oh yeah, yeah, yeah, yeah yeah, so so I added this diagram.

C

You can see my screen, um so this is the timeline where uh the the top part, the the up, the the text on the above this blue bar, indicates. You know the events that occur and the ones at the bottom indicate what we do. The audience does uh so here. I think uh I have put what we discussed, which is when the user initially visits the page. Then you start the page load spam. Let's call it yes,.

A

C

And then you emit a page view event um with the trace context of this span included in that event. So so one event is emitted right away. It will have to have a higher priority so that it doesn't wait for batching it. It gets out the door right away and then the page loads, the resources that the images CSS JavaScript files and then for each of them. We emit uh oh before that, when the initial HTML loads, you know we we emit the span for that page fetch for that HTML fetch, right and.

A

D

Sorry, uh you can explain the second arrow down uh emit Pages yeah.

C

D

The corresponding event uh occurrence that happens in the browser.

C

Yeah today, the the way uh things are modeled is there is uh there is a top level span uh for page load and it has two child spans? One is a page fetch which is the which indicates uh the span, for you know, fetching the HTML and then the second span, that is a child of the page load, is a uh you know. One span for each resource fetched.

C

And all of this together they are children of a parent span, which is for the page load where the page load starts. From the beginning till the point when the unload of the top window is fired,.

D

Did you say onload or unload on load on load? Okay, so Dom complete yeah? So what I don't understand is the difference between the page load span. You know the the time frame and the page fetch span. I, I! Guess: I, don't understand that page fetch. Can you.

C

D

C

Yeah there is a difference: yeah there is a difference, so the page load is. uh It includes fetching the base HTML and then you know fetching all the resources that it refers to and whatever initialization that the browser does all that uh is, you know together. We want to call it as a page load.

C

Today, I think it's called document load but yeah, um so that is the bigger span, there's the parent span and among the multiple child spans. The first span is the page fetch, which is just the network request. The API call browser makes to get the HTML.

E

Performance effectively stuff yeah resource performance, yeah.

A

D

Thank you when does that end,.

C

Yeah number three times, uh I I think as part of the page view performance uh timings. You know that would give us uh the fetch start and fetch, and uh so we would okay use those timings to to Mark the begin and end of that span. Okay,.

D

E

But in the proposal you've got here, you're.

C

E

It's an event, then that's just linked back to their parents.

C

uh No, it is uh it, this is a span, but I use the term even amid uh Loosely yeah.

E

Okay, to try and just generalize, okay, yeah.

B

Yeah, it is a span, but it's a span. That's that's synthesized from an event and it can be, it can be synthesized in the client or on the back end.

E

So we could just Define that as an event and then the user could choose whether they want to be a span or to be an event. Because yeah.

B

E

It doesn't really make sense to have a zero like this band for that yeah.

C

Now this is not a zero duration span right. This has it takes some time. This is a this is actually for sure for a span, because it is a network request. It is an API call to get the HTML, so it has well-defined. You know, beginner.

B

Yeah so so all the resources fly right, like the the page can continue emitting resource resource events, as as different resources are loaded throughout the lifetime of the page, yeah.

C

Yeah, so those uh so um yeah.

A

I also did a drawing that's related to this in the chat. Okay,.

C

B

Yeah so I'm not I'm, not I'm, not saying that, like you can definitely represent these as spans because they do have a duration. I'm. Just saying, like you know, they're not they're, not like in context. They don't happen in context, because if you get them from the browser as events, so you have to like generate the dispense after the fact.

A

C

Yeah after the fact I I agree, um but after the fact is only because there are limitations that we cannot uh capture while they're happening right. We rely on browser to tell us, and that is why it's after the fact, but uh but strictly speaking uh from a timeline wise, you know they. You know the document fetch fits to be a child span of document load.

C

And and in same as with the resource Fetch and the resource fetch only reason they are a child spam here is just because there is a context. You know there is a parent context, whereas here there is no parent context and therefore- or rather in this case, you know, there is a parent- is a click.

C

um You know if there is a context that is very clear, which can clearly tell you that yeah, it click is the parent. Then you know you use it as a parent, otherwise they will be an independent span, but they will start address.

E

Yeah I I still see these as a red stock yeah, whether they're sent as a span with the span of it, but that's implementation. But these are just events. I I agree that that's a span to say: okay, we started we're waiting until we're we're finished. That's a span to group all the other events under it.

A

E

But yeah in terms of the doc Fetch and the resource, that's just if I'm understanding correctly, that's just the resource timings you're, pulling out of the the um the browser resource metrics, which are only available after the fact, and therefore there's an event saying I'm going to admit the fact that this is how long the page took. These are all the resources I fetched since everything that went so there's.

C

One advantage, though, if you represent as a span so think of, uh if, if you represent the document, fetch as an event, um then it is Big, it will come harder to correlate with the back ends right if it, if it is represented a span uh where you you get the trace, ID back from the back end. Somehow there are, there are mechanisms.

E

Yeah I I think where you're going is it's more of a side effect of today, because all we had the spans if you've got events being emitted which are linked back to the the parents ban I.E the document load when it comes to represented as the UI you've got like. Okay, here's the document load and then you can have like little stars to say, and these are the individual events for for the detail um and then based on the name. You can then figure out um what you want to deal with it. So yeah.

A

On the other hand, also in the UI, you could also build a tree of.

B

A

Base that goes a lot of these resources and you could also link to that. Hey check out APM site yeah.

C

Yeah I think it gives you an end-to-end correlation right. The the tray started from the browser, but here is how it propagated on the back end.

B

Right, but can you do that I mean.

C

There were discussions where the server would send the trace.

A

B

C

A meta tag or one of those headers.

A

B

Yeah yeah so like from sorry sorry, just one more comment. So if you, if you take, if you take like the the back end us to be the parent span of the of the client's pants.

C

No, no! No! No! No, it is still um the the the the browser span is the parent. It's just that uh the back end will tell you hey I'm using this. As you know, my parents, so you use this as your stand. Id, it's a little confusing.

B

Yeah but I mean it's it's the browser that makes these resource requests that has like.

C

The resources separate resource uh they are loaded from a CDN right. Typically, uh you know there is no need to correlate with back-ends. It's only the initial HTML page fetch that needs correlation. So.

D

That's the the document load, the blue document load in the picture. Well,.

C

um Either that or it could be, the document fetch so so there are two ways you know you could even.

D

Document fetch already has a parent right. If you represent this, that is a span. It already has a pattern, which is a document load that the browser started.

C

So it depends on what we want I think in in one uh way of doing things. uh You know the document load could be uh the only spam and uh all these document fetch resource fetch. They could be events and the document loads uh span, ID and Trace ID could be. uh You know, I I, don't know whether it's only one of them, but it could be given to it by the backend.

D

Given to it by the back end, so that's when the starting of that phrase is really the back end right, correct.

C

D

But why would the backend then give it if, if it's not propagating it down so.

C

I think it needs more uh uh explanation than this, but the the tray starting in the back end. It's uh it's uh it's a wrong way of saying, uh because then it you would think as the APM span being the root, the root is still you the browser. It's just that the browser. Doesn't uh you know, give you a chance to start. You know, give it a trace ID for for that uh span, so you would create a span but not have a trade. It's it's confusing. No.

D

No, no I I think I know what you're what you're getting to is it's if somebody opens up the browser and starts typing an HTTP say google.com.

C

So your instrumentation hasn't loaded. So you you there's.

D

Nothing for you to okay, I. Remember.

C

Start the trace and therefore you make the request and the browser would not have sent the transparent header.

C

Starts it um practically, but.

A

D

What what exactly do we need to be gained by that um the.

C

Gain is the correlation you, you would be customers who would be able to uh see that this HTML request started from the browser. But then you know this is the API call on the APM side it it connected to and then further there's.

D

There's no API call per se right we're asking for a page.

C

Right in some cases you know uh there are static pages. In some cases they are Dynamic, it could be. The HTML could be generated by an API.

D

It's a dynamic page, then there's a process in the server side. That's generating this this whole page and part of that process. They could be calling databases or whatever even before they give you the first byte yeah.

A

D

Lot of interesting things that you can correlate there, that makes sense, um but that could not be the default is what I was saying. You know there's a lot of static, Pages cache Pages, um the home pages are usually cached um the first. You know when you hit google.com.

D

um Those things you know the the server would not even give you a reasonable ID that you can use, um so you will not have that. We cannot rely on it 100 of the time yeah. So.

A

Yeah I, don't think the you can say that clearly, even when you have a server serving from a CDN there are, there are Ray IDs, for example, for Amazon. So if your server is instrumented even for static files, if you have an engine instrumentation installed, you're gonna have some IDs.

D

E

Call this Advanced correlation, so uh I think I understand where you're coming from now Santosh. So my question then, is so you're saying the documents which are, which is all the timing, the DNS look up and all the rest. You want that to be rooted against the requests or the trace parent coming from the server yeah.

E

D

It comes from the server, it won't be always there's no point. If it comes to the server, then you.

E

D

The browser creates one so.

E

So in this particular case here we're saying the dot pitch is routed to the document load and the document load is routed to uh to that one. But technically, if you look at Advanced correlation the other doc fetch starts before the top load, because you've typed in the get yeah.

C

Yeah yeah I I see a conflict now yeah I think um the conflict is only because, um because of you know this, this number two yeah. We want to emit one event right away and that needs address ID. uh You know um the trace context uh be part of it and if we are waiting for it to come back from the back end,.

A

But that will be included into HTML anyway, which.

E

A

E

A

Yeah yeah, but that's maybe yeah.

E

So I think that's that that question is more okay. So what do we use? Who do we use as the parent um personally I still see them as events but uh yeah? You know it's it's. Where does the group Live, where when does the span start um and yeah? Who, who is the parent of the individual events, so um I I? Think if we start the page load expand as soon as the SDK initializes, then we just say: that's, that's it.

E

That's that's the parent, unless you want to try and emit the event for the page view based on if there happens to be a transparent header. Oh it's a meta tag, but that's I. Think more. The instrumentations uh level, not in terms of us defining what the events are. Yeah.

C

I think uh by default um you know most customers will not be enabling their backends to send the trace uh yeah wherein so that is an advanced case. So by default, uh the browser is the one that creates the trace ID. But you know we should do the implementation in such a way that when the server provides a trace ID, you would use that.

E

um There's probably more a consistency, level thing too: it's like um how do you want to represent it in the UI? Do you want to have that initial page load represented as an event, slash child span of the page load, or do you want it to be effectively just tweaking out just before the the page load event so effectively? It then becomes the parent of the page load span if it's happening first, so I I would think we just say.

A

E

Create an initial span, which is the paid Lodge span, and then we just root everything off that like you've got here where you're saying one gets created a span s two gets emitted with, so you can increase context. I'm playing two would get emitted with span s as the parent. You were going to get emitted with a span s as the parent. Four we get a what the series of fours would get admitted with pen uh span s as the parent um and then at some point. Spanish would finish.

E

um The interesting thing is what happens if span if we end span s before we finish fetching all the resources, um so that would be like a six, uh but that's like delay loading. So that's.

B

um I mean so so we have we have like in at New York like we had sort of like a span for like that is the same. Like page load span that ends when the onload event fires when it becomes interactive, effectively yeah. But then we also have another event or another span that can go beyond to unload because like if, like single page applications are like fetching resources like you know, Ajax data.

D

For example, so yeah the right side of the diagram there's.

B

D

Those things so.

B

I guess my my kind of comment would be like do we need to like? Is this more of a discussion like on the on the like instrumentation level, like you, when you have like multiple different plugins, and you use the plugin depending on what you care care about?

B

You know so like we could, we could discuss those details like when we actually write that instrumentation, like or or are we trying to like figure out if this is like a single source of Truth, or you know like this is like the spans, or this is the events you should be looking for.

C

um No I think if we discuss the possibilities, then we could do the implementation to to enable all those possibilities right, yeah.

E

Yeah because because really then you start talking about a you know, a page ready span how long the the grouping thing. How long did it take until the page was interactive yeah and then you could have the separate page load. But then again you could just say: well, the page interactive becomes the becomes. The parent span s for everything anyway, um and then we say: okay, the page is ready, which is like number five. Instead of saying we end the span. We just change that to an event: slash child Spain.

B

So I can see like how, like we have this existing instrumentation that generates these spans and if some people like it, then they can continue using it. You know, but yep.

E

Yeah we just updated to conform to whatever the content is of the event. Yeah, like you know, what we're defining is the events we're not precluding and saying. Okay, this other thing is like banned and you should never use it again. It's a case. Well, if you want to use spans, then you use that one. That's part of your distro.

B

E

Also came up in the spec meeting this morning about you know: uh at what level should the distro name be defined in the spec, um because yeah there is no straight Hotel destroy.

E

We can only come to do the web stuff we're going to be collecting a bunch of stuff which you might want to call the district a bit like Python and Java. Have it today, but they're, not necessarily well python calls the destroy. Java calls are just an agent um anyway interesting side, discussion.

C

C

C

Yeah, um maybe let's continue um this discussion tomorrow, uh so maybe we could come back with more refined thoughts on on on on this discussion.

E

Yeah I I. If we continue to look at them as events and then we say: okay, the event could be defined as a log event or it could be included in a span uh like if we just conceptually say, let's just Define the events um when we come to do the instrumentation, I I think that it's then just an implementation of.

E

Is it a bug event emit or is it plug event with a span or so it doesn't matter the fact that we've got the data being sent at those points in time uh we we, you know, we've already defined what they are, rather than keeping going backwards and forwards between. We need to only send it as a span or option.

E

We haven't even defined the data properly yet so that's my view anyway.

A

E

This is, this is a good discussion to have, but I I think there's probably still a little bit too early um or.

D

Or we could do it in parallel if we can find a way to do that, uh but you know I yeah I'm at the nail on that we need to First figure out. What do we want to capture? You know how we capture this. This is important. No.

C

But what part isn't it? Where aren't we reaching uh an end? I thought we we captured.

D

I, don't think so, there's there's more events that we need to go through um in the spreadsheet. That is more than we identified that we need to go through and uh flush out right.

C

The trouble that I'm facing is uh like uh like, where, where is this going to go next, like? Are we planning on putting this um in, let's say in the documentation here, you know this table uh this. You know the the these rum events that we talked about.

C

um Where will it end up eventually um we're.

E

Gonna have to create specification definitions for these, so yeah uh we've got to Define. What is the shape of an event like? Are we going to go event like we have event name event domain? Are we going to go event data? Are we going to go nested attributes?

E

Are we going to go uh Jason blob or are we going to just say, they're all defined as attributes at the top level which, from a space perspective, it's not good um like that discussion still needs to have in terms of you know what is the shape of the data once we, we know what the shape of the data is. We can then say: okay, well, whether it's a log event or a span event.

E

This is how you would map it um and then once we've got the the event, the shape of the data for both a log event and the span event.

E

We can then say: okay now we can say we we want to send this thing as a span or as a span event or as an event or as a combination of both you're going to create a child span, and then events against that child span, um but I think the first bits need that need to happen and and get them get the specs going um because, as you say, the logs, you know, there's already discussion about what what is a log event.

E

um I think we we need to hurry up and get to that, so that we're not effectively forced to down some other path that someone else wants.

A

E

um And then yeah like once, we've got them defined, or at least we've got them out there in the in the world. You know, then we can start looking seriously at the implementations.

E

um Looking at the graphs and saying oh, what what events do we want to emit at the different point in time?

E

That's useful uh for where we are now, let's say: okay, we're missing the definition of this event um that we wanted to find, but yeah I, think in terms of we really just need to say: okay, let's just get the shape and the event spec, so that we can actually then start going and implementing, because no doubt it's going to be the normal okay, let's get some proof of concept implementations before anyone's going to sign up the specs.

D

In semantic conventions, uh spec uh no, the.

E

Shape of the event I think is going to be semantic inventions. um I I think that that that's a a good possibility.

D

um In my mind, I'm thinking you know where would this lead lead us to so once we have the the stuff in front of us filled out, and you know we reasonably agree that these are the things that we want to capture. Then the next step is to get it into some spec. You know, I was thinking some medical instance is probably a good place. Yeah.

E

So I was thinking if we can get the shape of the event in semantic conventions. We then should be able to Define and say: okay well, rather than go and Define a separate one for every definition. Here we can then say: okay for the domain browser.

E

These are the events, and this is the values they hold and where they store it. In that shape of that event, um and that way is that in semantic conventions or is that, if the input I, it probably ends up belonging in semantic conventions, especially like the exception one, um but we may be able to effectively get it in a bit faster than going through the spec.

C

Ifically, the last two um uh will will likely go as uh Standalone events right, but whereas all the initial ones uh so far, you know we have been thinking of representing as spans so uh well.

E

You've been thinking representing with Spain, um like the resource timing to me, is an event.

A

E

Page view to me is an event: Ajax I see that as both there will be a span for the for the grouping, um because I think there's two aspects to uh to to the Ajax request, and that is an event to say. The adex request happened and then you've got the resource timings for that Ajax thing and they would be grouped under a span. So.

E

So yeah I I think it's we it's just why I keep coming if we just conceptually think about them about them as events, we can Define the fields um and then we can start having the discussion of how they get represented.

D

uh We're five minutes over. Can we start with this uh discussion tomorrow and then kind of flush out? You know how we all believe we should move our.

C

Work, yeah I think we should list down. What are the um you know, spec uh changes that we want yeah get done, that we can track them and then pursue them. Okay,.

D

We'll we'll start with them tomorrow, then: okay yeah, write them down here, perhaps as part of this top of the document into the steps right. So we can continue those steps and say we'll do this next. Do this next it'll have Clarity in our minds yeah.

C

And also we need uh to follow up on this varying attribute, so um Ram I think we might need your help and you know I think there was there's a past discussion that went dead uh about the female attributes. There was some Hope from Josh um recently, but I. Don't think that's um progressing so well.

E

One of the things that came up with the spec meeting this morning with children pushing the scope being run time stuff- uh that's our build Time stuff- is that we still need some way to pull this for these.

E

um So there is acknowledgment that something like ephemeral resources needs to exist um and by locking down instrumentation scope, that's going to come back to the core again, so it may get revived okay, whether it's arrived as a federal resources or as yet to something else.

E

You don't know.

A

C

A

Yeah, let's list them.

C

Open items tomorrow for this book and then we'll pursue them all.

A

Right thanks, John yeah cheers.