Open Telemetry Uncategorized, 8 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-12-08 meeting

Description

Open cncf-opentelemetry-meeting-3@cncf.io's Personal Meeting Room

A

How's it going good.

B

A

All right, yeah I, was actually trying to um see how I can add additional tests and I I started with your uh repo right and I'm, not sure if I was doing it correctly. uh I can start with that. I mean.

A

B

I think he has vacation today: okay,.

A

uh Let's actually before I start all this, let's uh wait for anyone else joining and yeah she's there yeah okay, we can talk about that and then you know later I'll tell you about my issue.

A

If Santosh does any other topics to discuss about it, yeah.

C

Yeah I I only wanted to discuss Bryce comment on Crash reporting, um so I just need like five five to ten minutes on that. That's.

B

D

B

I, don't think nacho is going to be here today, so we can just hop right into that. Santos Santos.

C

All right um so yeah, um I I, think from outside I, don't know if Stefan is going to join, but I had a discussion with Stephen internally and uh so so I think uh you mentioned that, uh let's open your current thing. You mentioned that some of the symbolication is is not really part of open Telemetry.

C

B

I think is a system here.

C

D

C

On that note, yeah correct.

C

Right so we use a third-party open source software called PL crash reporter I think that's, uh you know commonly used it's a popular one and last time nacho said one more. uh You know similar Tool. uh That is also uh you know, available for the same purpose. Right I, don't know the name yeah.

B

I'm not sure which one you're referring to.

B

C

Okay, so um I think the uh like. You said you know, you also shared another document right. The the link for it is in the top here, so I think uh the. If, if let's say a PL crash reporter is a kind of a de facto or a default. You know commonly used uh in a crash reporting mechanism, a cache reporting tool, uh then does it make sense to uh kind of standardize?

C

uh You know using you know that tool. As uh uh you know, the the data that tools generates. You know it could be in different format. You know the one format is what you shared, which can be. uh You know uploaded to Apple's. uh You know uh websites to to get the uh you know. Symbolicated information, but I am assuming in that case, that you would also provide all these simple. You know mapping files to Apple uh to to get that mapping right to get that yeah.

B

Well, so Apple's gonna have them at least at least they're going to have the one that is for your application itself. They might not have third-party dsims that you are linking with, depending on how those are um uh distributed. I.

C

See so when you submit an app to the App Store, you are supposed to provide them with the DCM files. Well,.

B

They are actually so um you know. That's a good question. um I.

E

Think that the.

B

Way it will work, is they will be uh in the App Store like uh so the way that so, the way that um Apple prefers apps to be submitted is using um a bit code, injector bit code, uh the the Intermediate Language and then Apple will actually rebuild your app for devices internally and then they will generate new DSM files there. So the D7 files that you have won't even be the the uh compatible with the app version that might be running in the wild.

B

So to get the right DCM, you have to go to Apple and download The Right DSM. Oh.

C

I see so so, then there is a possibility of the obfuscation happening twice right like as in let's say you uh you also, so there are two things right. Typically, at least on the browser JavaScript side, you also obfuscate the the names so that the.

B

That doesn't happen for us yeah.

E

B

It, let's look at an Android thing. You know, because the uh everything's, just in the clear yeah in Android right, like there's, no um machine code kind of like addressing right- it's just all in there, so they do like the proguard or Decks card to obfuscate the class names, but that that's not a thing in iOS I.

C

C

I see so the DCM files, that's not something that your build tools, uh you know generated something that Apple will generate and give.

B

It to you well yeah your build tools will generate them. um They're they're, very settings I have to look at it a little bit more closely for like a release, build okay but I. Don't think you're gonna do symbol, stripped stripping um for the release, build um it's just it's going to generate the Intermediate Language and then you upload it to Apple, and then Apple will rebuild it for the architectures that they want to rebuild it. For.

C

And, and is this irrespective of whether you write your software in Objective, C or Swift, yeah.

D

C

Okay, yeah I think so this process of uh symbolication, you know I, think in our case uh we do it ourselves and not submit to Apple yeah.

C

um So we you know, I was just wondering if the initial part where uh you you obtain the report, uh you know that part can be part of the you know the common agents uh and, and then you know, once the data is received and the report is received, whether you know you, you know, take it to the um to the apples uh tooling or you do it yourself. I think that we can leave it to the individual vendors. Certainly.

B

C

um I think we also um do not ship the report in in the format that you shared. We I think it's the same data, but it I think uh we format it in the form of uh Json. We put that same data in a Json form, okay, yep and, and then ship it over.

C

um So it's in that case you know, does it make sense to uh you know Define a data model that we see on the screen here? uh But if, if you're saying that, uh if to be able to submit to Apple, you need it in that other format and not in this Json form yeah correct, um then you know it may not be so helpful to all.

B

Yeah yeah, um and so you were mentioning PL crash. Reporters appeal crash reporter does have an API that that generates this. For um for you, so you don't you don't have to like uh do a lot of work. If we're gonna depend on a PL crash report, you don't have to do a lot of work to put it in this format.

B

um uh If, if that is not something that we want to put directly into the semantic conventions for crashes, maybe we can have like a a an additional, um uh optional, uh key or optional attribute that just takes this whole string in there it's kind of big. Maybe maybe we don't want to do that.

C

um How big can this file be.

C

B

um I, don't know about megabytes. uh That's a good question like how. How big is this.

C

I think um um I think the otlp protocol has limits on on.

D

How much how big things can be things.

C

Can be, they recommend that if things are bigger.

E

C

You split it into you know multiple in some chunks, yeah.

A

Itself is not that big, it's the decent file that that is pretty big.

B

Yeah, the D7 is really big, but you mean it depends, I think the the biggest part. So this part up here is going to be pretty.

D

B

The I guess there's a lot of factors right, so it depends on how many threads you're running when the crash occurs. If you're, if you blow yourself up creating too many threads, this could be really big right. If you create, if you create a stack Overflow, this could be really big.

B

um It also depends on how many binary images you're linking with so, if you're, linking with a lot of third-party libraries. This could be really big. This right here is just with the system. Libraries right, so there's not a lot of third-party libraries that I'm linking linking with here, um and so this is already a pretty big list, so it could be pretty big.

C

So actually, irrespective of the format, if the, if the size of the data depends on the type of the crash, uh then even the Json form could be big and even the splitting, the Json, you know, might be tricky. So one other option um uh you know to consider would be one like outside of the otlp.

C

You know ignore TLP for a second and then just do a file upload uh to some destination and then generate an event uh with the with the with the information where it is uploaded and and some metadata about it, okay um and and where it is to be uploaded, the the tokens required for it and all that can be configured uh and then your your instrumentation could push the file to that and then generate an event uh saying that hey I pushed this crash.

C

You know here, and here is uh the event related to that yeah- that event itself could be smaller, yeah, yeah, um okay, okay,.

A

What is the limit for.

A

B

Yeah, that's a good question.

D

Do I have a second, let me see if I have this file laying around somewhere.

D

D

Let me just make an make a note here.

B

Yeah so that example is 188 kilobytes, so it's not super big okay.

B

But I I think that, like if you created like a stack Overflow, it could be 10 times the size, so it could be like up to a megabyte.

B

I, don't know how common that would be.

B

There's also I think that there are also other ways like um there might be limits in PL crash reported how many, um how many uh stack frames there are in a single thread thread so that could also limit the size.

B

um Let's see so the other thing I wanted to just take a note of is um at minimum. We might want to expand the number of keys um in here. We might need like a uh like uh uh libraries here.

B

Because um that's like this binary image down here this, this is uh absolutely necessary uh for the obfuscating, um the crashes up here. Okay, because these you know these, give you like the library um address here like if I go um and and that's how these are generated in this. These Library names over here is it takes this address and goes in down into the list down here and finds the name of it. Okay, you know here's the range address range for foundation.

B

It gives the good for that for other, more simple. This is more useful symbol. Lookups like if you needed, um yeah the, uh maybe taking a step back like each every single library that is generated for every single version of iOS, for every single.

B

Our CPU architecture has a unique identifier for this and that that's per per build of of uh the for system. Frameworks, that's per build of iOS, so iOS 14 14.1 all will have different Library versions. So this is why it gets really hairy to uh to symbolicate is because somebody needs to manage all these different Library dsim files- and there are. There is a repo online that has that available, but um it only goes up to a certain version of iOS.

B

They kind of stopped, maintaining it at one point, so there's a big gap of more recent iOS versions where, like the the community's missing those uh those files- and so that's one Sticky Wicket and then the other thing is like for each version that is built for a app a customer app it's going to have a different decent file for each version uh or each architecture. That's that's built! So if there's multiple architectures floating around armed V6 arm, they don't do that anymore, but, for example, um arm64.

B

If there's a different architecture that they release, there's going to be a new version, I think right now we only have to worry about arm 64, but you never know with uh with apple. They might release some new architecture, okay, and we might want to add a like a thread. State um crashing thread State as well.

D

C

Given an option for you um to, um you know, get it in. You know in a different data format, uh because if Apple expects uh you know their if their tools expect, you know only this file to be you know given to them, uh then you will have to you know reconstruct this file. Yeah.

B

Correct yeah, if you want to go down that route, if you don't have your own infrastructure for doing this, yeah.

C

So then, I think the question is like among the among the vendors that are at least you know at this point showing interest. You know what what is the format? That's uh you guys are using like I. Think in our case, you know we I uh we're not going to Apple. It's just we're doing it ourselves.

B

Yeah yeah exactly I mean so anybody who wants to symbolicate is going to need these basic elements that are presented in this format. This apple format um I know at New Relic. uh They have their own system for uh decimal or for symbolicating crashes, but they also.

B

um You know that it's a it's a hairy subject, collecting all these data points, so they also provide this format for this apple format. They reconstruct it and provide it for customers as well if they want to access it, um but yeah. So like um the the basis, so I think these things here are are good.

B

So far, like the uh the thread um information you might want to elevate the exception um uh message to like a root attribute because that's gonna be like it's gonna just take up space if it's in Upper thread, because there's only going to be one thread, that's throwing an exception. That's a crash.

B

um You'll need an array for these uh um libraries here and you may want to provide an array for the registry States as.

C

Well, basically, basically, you know everything file, um you know. If we were to represent, you know you will you will have to follow the same. You know, structure that that's uh you know shown here. Yeah.

D

C

So what what? What is the method in your case, like you, you're you're, part of dynatrace, right I? Think, oh.

B

C

E

We're we're just.

B

We're just starting to to add stack um or crash reporting, and so we're still trying to figure it out right now, too,.

C

um So what approach uh um are you going to take.

B

um We haven't quite decided yet I think maybe initially we might just provide this okay, um because you know we need to build the infrastructure around uh actually doing symbolication and that's pretty.

C

Hefty I'm I'm also uh uh I mean without a lot of uh knowledge on this topic. I'm also wondering you know if, if it might be simpler, not easier or not, performance wise, you know any better, but it might be simpler to just you know, generate this file and send it over.

C

um You know that way. You know we don't have to worry about a lot of thing, because this file seems to contain you know everything needed and then the you know uh the backends. Could you know reconstruct? You know the the you know in a different structure that they need, but I think this this file, uh assuming that you know the the pl crash reporter itself is, uh is kind of a standard. Now you know they they won't make any rustic changes. As long as you are also reporting.

C

Let's say at the top uh you reported using this version of PL crash reporter. Then then you know what to expect in this file yep.

C

um So so it might also move some of the work to the back end and not you know, have too much uh on the agent side.

B

Yeah yeah, the the only Pro I guess. um The one thing is is that the back end we'll have to parse this file. If they wanted to do anything with it, and so it might be better just to so pale crash reporter who uses a uh uh protobuf already. So it might be better to just duplicate that Proto buff and then send that to the back end.

B

C

B

Yeah and then yeah and then uh reconstruct it from there. Okay.

C

Okay, so does Apple have any specification around this format um that, let's say PL crash reporter is following to generate this report. I.

B

Mean this is this is uh I, you know, I, don't know if there's a there's like a set in stone specification that they have provided, but they um like, if you're using you know, um maybe like maybe.

D

um Let me see here so.

C

This uh this file was created by PL Crush reporter. Yes,.

B

This file was created by PL crash reporter, so apple apple does present. um Here, let me let me reshare my screen here.

B

So, let's see here so this is xcode. um Let's see devices.

D

And simulators.

B

Maybe, um or can I look at crashes.

D

Or maybe organizer.

B

Yeah, so so, if I go here and okay I don't have any crash logs for anything but like if I plug in um like an app.

B

Maybe this one, maybe that one, maybe this one all right. So this is kind of the tool that you would use. So maybe I can take this file.

D

um How would I do it archive Maybe.

D

B

But I think that so this is the tool that they derived that from so so I would, if I had a crash, it would present it in that format. Here, okay,.

C

B

Trying to see if I can't like convince it to.

D

Will that work? No all right.

B

Well, it was worth the worth the try where'd that even go.

D

B

Here, yeah, so if I, if I got a crash on my app um I might be able to to collect it like if I plug my phone in it might might try to load something out, I, don't know it's been a long time since I've yeah actually built an app myself.

B

That's that's in the App Store. So it's uh it's hard to recreate exactly how I expect it to work. But but this is that this is I, don't think the Apple has like a spec per se. um They might, they might have it in documentation somewhere. I can look into that. But um I think this is how they derived the spec based off of like what they saw here and then recreated it to generate a file.

B

C

um So yeah I think um simultaneously I'm also trying to understand the process for uh the Android side of things, and uh you know, and after that only I. Think, let's uh you know get together and you know take this forward. I think. If, if this process is very different on the Android, then you know we are free to you know model it. The way we want here.

C

But if there are any you know similarities then it makes sense to you know: do things uh in a common structure, yep um yeah so but I don't know at this time.

B

Yeah I mean I, think it'll be pretty similar, um I'm, not sure. If Android has the concept of libraries that that need to get transmitted for you know, um de-offification purposes, uh I think that, basically, that the main thing in common is just like the threads like there's gonna be threads. There's gonna, be you know a crashing uh Inception type exception code, that sort of thing, maybe enough code, but.

C

B

There's there is quite a bit of overlap: there's not going to be like a um a thread. State like a like a registry thread. State, that's going to be transmitted.

B

How useful this is! I, don't really know. You'd have to really yeah you'd have to really know how um the how it, how it Maps, I guess.

C

I think those are the contents of the individual registers at the time of Crash Maybe.

B

Yeah yeah um it, but it's yeah it's it. It would be hard to like kind of figure out what exactly um is getting passed in here like maybe you might be able to get some insight.

B

Like based off of maybe some uh like you'd have to know like. Oh, this is calling this method. I know that, like the you know, if it's saying like it's, this memory address that it's crashing on like a nil value or bad access or something and then say like okay, that might have been the second parameter in my my uh you know in my function.

B

Call and I know that, like the RBP um you know, register is the pointer to the to the to that thing and that's the same object that crashed like then I can figure out that okay, maybe the object that I was passing there, there's something wrong with it, and it gives me a better idea to get where to look but I think it's. You have to be really familiar with um uh the ABI for, for.

D

You know the tool.

B

For the for the for the iPhone okay.

A

Suspentos I think we had discussed this uh on the on the web web uh site as well right there because you're part of the client uh side Telemetry. So do they have something like this for uh the web .js as well, I.

C

C

um It's uh it's a you know, I think relatively simpler. uh Whenever an app it's a first of all the apps uh don't uh the crash is, is uh even though the term crash is used, the app doesn't exist, exit there right, your browser tab, you know it. Never, you know closes on its own, that that will be the browser crash, not the uh web application crash. So typically, uh those are just unhandled errors, so you would see errors on the console and then you you would report those errors uh to the back end.

C

So you will get um so it's a standard and accept it fits into the form of an exception um where there would be an exception message. Actress and uh you know the type of the error so.

A

It's just a JavaScript error.

C

Okay, it will just be a JavaScript error, so.

A

Okay, there is no nothing common between the web and the yeah IOS and Android okay, yeah yeah.

C

The spec, but but even in the case of uh JavaScript, you know you, you know the stack Trace would still be all you know obfuscated. You know you, you can't make out much, so you will still need to. uh You know, use the symbol files to to map it back to the original names to make sense.

A

Oh yeah, the minified stuff, okay, benefits for score, yeah, correct.

C

So that that uh work is there.

A

Yeah, so here here, the map for Android is also you know, has got similar things that a crash file has right or the core that iOS generates, but uh I think the heavy lift is on the all the you know, uploading the D Sim and the.

C

uh But those are like one time thing right, um like once: per version um once for Builder ID yeah yeah one square build yeah.

C

But I I didn't know that you know that decent file, you know, will have to be. uh You know downloaded from Apple, um uh so I don't know. uh How is that process? Typically automated, like let's say.

B

uh uh Yeah sure my screen again so here I have like my archived applications that or my app that has been like uploaded to the app store for release and then I have this button down here to download the debugging symbols. So if I have a crash, I can use that to de-off escape my my um crashes, that I might be seeing or for some reason this one doesn't have anything, but it's I think it's just because I was fiddling around with it.

B

It's not actually a release app okay, so they haven't, they haven't rebuilt it for release, so there's nothing available. Yet.

C

And uh by the way, so the um the service, where you know you upload the crash file, you know for Apple to symbolic it you know, is that a paid service.

B

um I, don't think so, I, don't necessarily think it's uh an upload kind of process. I think it's like I mean maybe they might fetch data from their their systems, but um I think you can just do it uh from your uh from.

C

D

E

From this tool.

B

Right here and I think it might go fetch some data if it's not available.

B

C

It okay um good to know I, think I um I think it might take some time um to understand the Android side of things. So, let's take it a little slowly uh but sure for this info. I think uh what's happening on the Android side is there is currently you know no Android uh instrumentation libraries in open, Telemetry but Splunk has um and which you know they are willing to. You know donate to open Telemetry, so they are, they are working on.

C

uh You know that work today uh said they will do that by you know, giant Feb timeline, so once they do that, then you know we'll have more active discussions uh on on some of these calls with the more Android Folks at that point, I think we can. uh You know close this.

B

Cool yeah we've been, we've been doing some Android development. We have an engineer, but we've been just using the Java.

A

C

Is uh there are a few things yeah yeah? Okay, no.

A

I think the big difference is uh Android doesn't use all of the Java Frameworks right, so so for the Android agent. You just need the ones that are like retrofit yeah.

D

A

C

Kind of stuff yeah, so you don't need all of it. So I think that on the Java side, uh again I'm not super familiar with things, but uh I think they have separated out um the open, Telemetry Java with the open, Telemetry, Java, instrumentation, Auto, instrumentation, so I think the auto instrumentation libraries I think they have a support, for you know tons of libraries yeah so there you know you might uh pick only a few libraries.

C

Yeah I I, don't know how that uh you know built uh so.

A

So it is a custom custom build right. It's not a standard Java agent, Builder, yeah, yeah.

C

You know, like our there's, that Java Auto instrumentation provide one jar file combining all or do they provide separate like on the JavaScript. The information has its own package.

A

um Yeah so obviously right the default Builders, uh it builds with all the They, generate all the demos and so on. Open television contains all of it, but you can do. Custom builds similar.

C

To pull out only specific parts.

A

So that way, you have a light agent right, not the entire instrument right right, foreign.

C

All right, that's all on this topic, I'll! Let you folks discuss anything else. We have all.

B

Right thanks Santos.

A

All right, thank you all right.

B

So I didn't have too much uh um to discuss. um I just wanted to yeah point out that the the logging SDK is ready for review. I was able to add enough um testing to.

A

It okay, okay, yeah.

B

um To get the uh code coverage stuff to pass, so it's all thumbs up. I also discovered a lot of bugs adding the tests.

A

So I fixed all those that's good, okay, okay, that is that's the whole point of yeah.

B

Yeah, that's the whole point yeah, so yeah. If you could give it a review, um if you want to add more uh tests to it, that'd be great too.

A

um Yeah one of the things I was trying to do was I had I had uh uh opened up, let's see when I, when I try to do this oops.

A

Okay, so what I did was okay. What I did here was uh uh actually I should show from here. So I.

A

Picked up yours right, yeah.

D

A

I poked it here, but when I try to uh so so I was gonna. You know use yours and uh and then uh added to that. But then, when I try to open it, it always tries to ask me if I want to Fork it. uh You know I want to change the remote to this open television should I not do that, because.

B

Yeah I mean that should should work. You should just be able to pull from that.

A

D

B

Looks correct to me.

A

So so, if I, because uh I from yours and I come into mine and then I push it to push it as a PIR to yours,.

B

I think so: yeah, okay, that's.

A

All in any case, I think once yours is approved. I can pick it up directly from the open, Telemetry yeah.

B

A

You're done with.

B

Yep, it's all it's all done. It's ready to be reviewed. Okay,.

A

Okay, that's good I'm, going to uh just run the tests and go through this and then yeah I'll approve it I, don't know much was already done it so.

B

No I, don't I, don't think he has I think he's been spent. He hasn't had a chance to look at it. Yet.

A

Okay, so so yeah I I was going through. You know the same thing: the Java logging. uh uh You know it's very similar right. The two are kind of uh yeah, so yeah.

B

I based it off of the Java one, so.

A

That's why yeah cool cool um I think the Java one was done also pretty recently right. They also added logging, like six months back or something okay,.

D

A

Yeah so I'm sure we'll find more things too, that they have missed into respect and so on. You know, yeah.

B

It looks pretty complete to me: um I think you know we I don't have the complete, uh like gamut of um you, know like uh exporters and that sort of thing. So that's the only thing. That's really missing here.

A

They have something, for uh you know, do the compatibility test or something right didn't they have something like that.

B

Oh, what do you mean.

A

For the API, that is, uh you know if you're, if your SDK is compatible or your implementation is compatible with the spec, there was some sort of a compatibility test or something that I think we had it in our project.

B

Oh interesting, yeah I, don't I, don't.

A

Know yeah I think, let's see.

A

A

Yeah I think maybe it's under this.

A

There was something to, or maybe this is something else I I don't recall. There was something to.

A

Okay sounds good, so we'll talk next week and I'll I'll go through the test and uh approve it cool all right, see ya.

B