Open Telemetry Uncategorized, 28 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-02-28 meeting

Description

Instrumentation: Messaging

A

A

Together some kind of agenda items: if anyone has anything top of mind, you go ahead and add it. That'd be that'd, be great.

A

A

New items really, but we can go ahead and get started. So the first thing um that came up was I. Think Tyler. You potentially put this issue in the in the chat but um I think Michael, oh Tristan. Thank you.

A

um Apparently Amazon's making some improvements around cold starts and uh Anthony not sure what you can share, but we love to learn more about what Michael was talking about.

B

Yeah I think most of those changes have been pretty transparent in this repo. Like the there's, there was a change to the Java, uh the JS instrumentation, to allow specifying which instrumentations or illegal for auto instrumentation and largely what we found is that it's the auto instrumentation adding to cold start times um scanning all of the code doing whatever monkey patching your bike, good rewriting, is necessary.

B

Eliminating Auto instrumentation eliminates the bulk of the the Gold Strike penalty, and so we're looking to provide, wherever possible, controls over what Otto's rotation, even attempts to register itself Java provided that out of the box, because it's got environment-based configuration for its Auto instrumentation support was added to the node wrapper recently, I, don't know what other languages have in terms of controls over what sort of Auto is rotation will even attempt to register itself. That is the the password yeah.

A

Okay, so mainly just kind of giving users more control over what packages get loaded and which ones are actually running um seems to be the summary.

A

Is anything needed from the function Sig to support any of those performance and improvements? Your potential controls across the layers.

B

I, don't think so, um like I I, think most of this is related to the auto instrumentation configuration and the languages generally and applies Beyond just the the landowner functions of the service context. It's just more significant here, where you pay that startup penalty more frequently than you might in other systems,.

A

Yeah, that makes sense um Alex. This might be orthogonal to yaws SDK config work, but with the config file also affects Auto instrumentation, or is it purely an.

C

A

C

Right there is a plan to support additional instrumentation, as well as part of the effort.

C

So the there's still I, don't think in that issue. Any much of an explanation of this that it's related to, if at all, auto instrumentation, uh AWS I, think a product manager keeps saying that they'll get back to people about it.

C

um It would be good, I mean obviously I could just reply and say it's the auto instrumentation, but uh they also. He also says that there's work being done in q1 is that the so nodes done job is done. Is that some other languages that are being worked on.

B

uh That I will have to follow up with him on um sorry. I, unfortunately, haven't been super close to what that team is doing right now and spreading a few different directions right now,.

D

C

Just be good to have some sort of feedback to the users, because right now they seem to just think it's slow and not be getting answers.

A

Yeah, if there's any specific data we can share. That would be great but understand Anthony. If you're you're spread a bit then but um yeah, it would be great to get more details here.

A

Okay, um yeah. Well, thanks for sharing the context around that, um so I just wanted to call out kind of our open, spec PRS as well. So we recently had the cloud resource ID one merged, and we also previously had execution and a vacation, ID um or yeah I, guess renamed and then the the max memory updated as well.

A

So um are there any known, work items we'll often do I'm, assuming we'll have to make some updates and the layers to start producing the cloud resource, ID and maybe even you know, give customers a notification of a breaking change. But what are um like? The top of Mind work items coming out of um these two um that PR is being merged.

A

E

I know that Tristan's been making some good progress on a couple of those uh in some of the various repos I. Think Martin uh is also working on some of the node stuff.

A

Yeah yeah I was talking to Martin about that earlier. So so he'll cover updating means for node. um It seems like Python's one. We also want to get updated kind of ASAP since that's another widely used one. Are you working on that one Tristan or what languages have you been focused on Python and go iPhone.

A

Okay, so this leaves What DOT net and Java.

E

Is is dot net really a high enough priority for us.

A

Probably not but I kind of put in the same bucket as Java just based on their probable usage, um since it seems like JavaScript and python uh just based on what Mike was saying and the data from Anthony. Those were the I think by far the leaders in terms of I.

E

I I can uh take the Java one.

A

Yeah I think I mean obviously we need the.net done just so. We have uh consistency across the implementations which is kind of one of the goals of the Sig, but um in terms of prioritization you're right, it's probably not that high, but um still needed so I always follow up and try and find someone for net.

A

um Okay, four or two uh open PRS do we want to say anything specifically about those I think Tyler one's from you and one's from lewd Miller uh he's not here so can we can leave that one for.

E

A

But you have this requirement: yeah.

E

So that one um uh as we've discussed before it's more of a tangentially related to this sig um but uh I I, think it's in a good spot, still waiting for uh some another round of reviews uh Anthony if you've got some uh thoughts there. If you want to, uh uh if feel free to uh take another look- and let me know um but yeah.

A

A

um And finally wanted to give an update on the AWS account I'm, not sure, if anyone's seen it um it does exist, or at least this emcf previously had like an open, Telemetry AWS account, so we're waiting on new credentials, uh hopefully Anthony or I, should have those relatively soon Anthony I'm, not sure if someone paid them to you, um I wasn't aware, but we are we're in process.

B

No I don't have any credentials yet.

A

So, okay, we'll keep paying people I'll keep kind of hassling people, but uh small small progress. I feel like I say that every week, but we are getting closer to actually have it in the account and having access to that account, which is exciting.

B

What is the plan once we have the account like? What do we intend to do there? Is it just testing? Do we intend to have build and release in that account?

B

Did we get multiple accounts? I thought we were looking at one for testing and one for release, or were we going to smash those together.

A

Yeah I think um It's a situation where we take what we can get um I I think Morgan uh potentially mentioned the four accounts, but also I, think uh Wang or whoever was the one who originally put that comment said two would be fine, so if I had to guess or probably end up with just the two, um so I think we do need a China account which I don't care the status of that so we'll have the the uh Sarah the world account sooner um and then I think the intent would be to actually publish um you know Community layers from that account.

A

At least that was my understanding. I'd love to here are some additional feedback. I know one kind of issue people have like with the AWS implementation, for example, is that a layer or like a the layer can be updated in the repo, but not you know, be pushed or published on a specific Cadence or is published too slowly so giving the community the ability to do that. I think is a good thing personally.

B

Yep also um changes to the community layer may not always be reflected in the ADOT layer um for some time uh as we go through some Security reviews and the like for new components.

A

Yeah yeah totally fine I think that's the the kind of intent they say. We can unblock ourselves a little bit and give the community and ADOT a bit more flexibility um and how we run things. But if anyone else has that perspective, Tristan um Alex love to hear it.

A

That sounds good. Okay, yeah silence is acceptance, um but those were all the items I had for this call. Are there any other? You know top of my things or um you know items that people want to draw a specific, take attention to or discuss further in depth.

C

Can I ask a question: that's related to Lambda, but not exactly it's a sick work, well, I! Guess personally, I'm trying to get Ruby layer to work on arm and all splunk's other layers work on arm. Fine, the Ruby blows up uh because it says it can't find a shared Library uh for protobus, but there doesn't seem to be a way.

C

I can't get like a shell environment in Lambda can I, so I can actually look because, if I download the layer, the files there just wondering everybody's ever had to debug stuff like that and if they, if they're able to get I'm looking through the console and there's no like uh shell environment or anything like that available. Is that the case anybody knows.

B

Is is that something that uses like a c wrapper for its bird above implementation and thus might be tied to the architecture it's built on.

C

Yeah, so it is, but the um I download the layer I check the file and it's built for arm it's arm. 64. looks fine and it says it can't find it, but so I'm trying to see the environment. That's actually, instead of just the layer but like what what it's actually running like after, because it's got to put the run time Ruby then it puts the layer and so maybe something's getting messy. It works, um uh AMD 64, just not armed, so it still seems weird that that would be.

C

That would be the layering but uh yeah I, don't know what else to try so I'm trying to see what it's actually.

B

Running yeah, unfortunately, there isn't, as far as I know a real way to get a shell on the Lambda function. Instance. um What I would suggest is uh perhaps adding code to The Collector to to do what you need the collector extension there or to your to your function. Handler, if you can do it in Ruby.

B

um Just add the code there to emit logs yeah.

C

The Ruby I, don't think ever runs, but yeah I was thinking. The next step would be an application that the shell references prints, all the files and I can see if it's actually there and yeah go from there, but yeah thanks.

A

Okay, great um yeah Tristan. If you continue to have issues with the Ruby layer, I know it would be great for the community to have it. But you know maybe we can just post in the channel and find some other people that maybe able to provide some feedback too, but.

C

Yeah after I figured if it get a little further like make sure the file is actually there or something ask Ruby people and.

D

C

After, like the my next step, was getting this working on arm and then, of course, I can work on donating the the layer to because arm is priority for work, but then I can donate to the community. The layer.

A

Yeah one good resource I know is Andrew Hayworth. He seems very passionate about Ruby, so maybe maybe worth sending him a ping whatever he can contributes to the demo. I think he'll write about an essay on his PR, always very detailed.

C

And helpful those are always useful people to have around yeah.

A

Yeah exactly so feel free to take a move, um but yeah great I, think I think we covered all the agenda items uh like any anything top of mind for you before we before we go anything from the field.

D

A

Nothing new, nothing, new, okay, cool.

A

All right, guys, I, think I think we can call it here. I.

E

Actually wanted to throw out.

E

um Was there any uh open concerns on that uh SDK PR that you wanted to bring up now.

B

D

I still think it's saying.

B

B

E

Say that again.

B

I said I still think it's saying too much. We don't need to specify everything that it's saying and I'll comment down up here.

E

Okay, do you understand why I uh why I took the route route that I did.

E

I guess uh from past experience like when I, so when I first uh instrumented uh the AWS SDK, uh it was not intuitive to me uh the implications that uh you know not using x-ray would have, uh and so that's why I think that it's uh important to have a little bit more justification there, just because it is very unintuitive.

B

And there are probably more scenarios that we haven't yet encountered or thought of that we get different uh configuration right, so I I think what we can say is unless the user explicitly configures otherwise use an AWS supported propagation mechanism for instrumenting, the AWS SDK.

E

Right, but why, like so I, I, think it's important to kind of give a little bit of justification for? Why that that's the the requirement.

E

So that way, if you do choose to do otherwise, you understand what the implications of it are.

B

But that can and will change, and it can change without any involvement of anybody. Who's ever thought about this part of the specification.

B

True I don't think we should be writing specifications that is likely to become outdated and incorrect without actually changing what it's trying to accomplish.

E

Isn't that the case with like pretty much any cloud interaction or integration, though.

D

B

I I think that specification around how instrumentation should behave is probably going too far for much of what hotel should be doing. We're we're building a system for instrumenting applications. But what gets done with that? Isn't necessarily something that hotel needs to specify.

E

Okay, so if we don't have that justification uh that explanation of why uh x-ray in this uh in the specification, where should that go because I think it's important.

B

Important to whom.

E

uh To anyone that is, writing instrumentation for AWS SDK is.

B

It important to the people who are writing instrumentation or to the people who are using instrumentation.

C

Writing the instruments writing, because they're going to ask the same questions and they're just gonna have to come, keep asking them.

B

The comments I'm seeing around sqs and in sending data through S3 those are use cases that the instrumentation author doesn't need to consider if they simply use propagation mechanism. That's supported by the AWS Services when communicating to the AWS Services right.

E

Sorry say that again.

B

So if the instrumentation author uses a propagation mechanism that is supported by the AWS Services, when communicating to AWS Services, they don't need to know what the user of that instrumentation or the SDK is using that for right. So the the discussion of the ways that you have to propagate context for through S3 through SNS through sqs, are irrelevant right.

D

They just need.

C

To know that the reason they're using this HTTP header, instead of the attributes and sqs, is because this has been thought about, and it was the most useful method. It's right.

B

That's slightly simpler. What are we trying to do here? We're trying to get the AWS services to participate in the traces any system that's going to participate in traces, a context, propagation mechanism that it supports what it does beyond that out of the callers realm of concern right, it propagates the context.

B

What we need to specify is if you're instrumenting, the AWS SDK, you must use a propagation mechanism supported by the AWS Services. If you want those services to participate in the trace.

E

But if your experience is strictly doing uh HTTP responding lambdas, uh then it's probably going to be relatively unintuitive. Why you should care about some of these other things or if all you're doing is an sqs Handler, it's going to be unintuitive. Why can't you just put it.

B

E

Just propagation in the attributes.

B

That's not someone creating AWS, SDK instrumentation, that's someone utilizing it.

E

For the SDK and all your experience is, is uh using a very narrow portion of that SDK.

E

So, just personally I've used a very little pers, a very small percentage of the SDK and what it provides I, don't know all of the implications for each of the different request types.

B

Which is precisely why I don't think we should be specifying.

E

But I I I will say that I think that it's valuable to understand the the motivation for such.

E

So your main issue is the fact that we're saying hey this is why we're doing this and you would prefer that not be included in the SDK if we removed uh sorry in the in the species.

B

E

Section of why we're doing it then you'd be okay with the rest of it.

B

I think I've made a suggestion of as to what should be removed for me to be okay with it, but we can deal with that dealing with the actual text, I'm, not going to say what I'm, okay and not okay with right here, because we're not dealing with the actual text, we're having a conversation um and I'm not going to get back into a corner based on something that I said. While not looking at the text.

B

We deal with the text.

E

Okay, we'll uh I guess continue this discussion in the pr.

E

I think that's all.

A

Okay sounds good thanks. Everyone for joining um yeah, I, guess we'll work on getting getting the PRS or getting the layers updated um and we'll talk in the uh selection, cheers sure.