Open Telemetry Uncategorized, 8 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-02-08 meeting

Description

Instrumentation: Messaging

A

B

A

I need to read, join I'll, be right back.

A

C

A

Good morning, good afternoon, thanks.

A

So I'm not expecting anyone else to join. uh uh Are you.

C

uh No not that they know of okay.

C

A

Right I can get us started um unless uh you guys have something to cover.

C

I wrote on a few topics: um material can start with a discussing the yesterday's meeting. I guess.

A

Yeah, so the main thing from yesterday uh was that uh you know we're looking for more people to do reviews on uh the specification. So if you guys have interest in you know uh working to be a spec reviewers, uh then let uh I would say make it known in the slack.

C

C

A

Just I I think spec approvers in general. Oh.

C

General, okay,.

A

Obviously, you'd be able to help with the fast, but um so I submitted uh the pr to update the I. Think you guys reviewed it already uh to remove uh the AWS x-ray uh environment, variable environment, variable propagation interference, um so I think that's mostly uh ready to go uh we're just waiting for uh approvers to approve it, and so we can get it merged.

C

Yeah, okay, it looks good, um so I have a question about that specification. I went through the call, um not all of it. um Is there a link somewhere to the uh to the Google Doc sheets uh of the discussion for the justification changes? Do you have it.

A

Yeah I've got that just a sec, uh I believe you're. Referring to this one.

C

Yes, yeah all right thanks.

A

Yeah, so I was actually going to share my screen, so we can talk about that. Next, all right.

A

Which one this one okay, uh so this is the the spreadsheet uh we've all kind of been collaborating on. uh So here we've got uh aws's things, uh items that are fairly commonly available in the context, uh so these are attributes that we can use on instrumentation azures is very limited in in uh so we're trying to get Azure folks to help us identify uh some of these things that we want to share.

A

uh Google has has done so, um and so we have a little bit more to work on here, um so that uh I think that when, for example, um we're ready to to kick off uh Google instrumentation uh similar to Lambda, uh we can use this as a as a resource for that. But the main thing we were trying to do is uh take these things. These terms, for example, that are found in the the various semantic conventions documents uh and and make sure that we're aligned on those.

A

uh So uh these are the the current names.

A

uh And then this is what we want to rename it to and I've bolded the ones that are different so like, for example, uh fast execution. uh This is like, for example, the AWS request. Id uh I, don't feel like execution is very descriptive yeah. So this is the biggest one that I had issue with and wanted to change. So uh we're gonna go with invocation ID, uh so that aligns better that's similar to like what Azure is using uh Google uses event. Id AWS uses request, ID.

A

um Version trigger uh instance, max memory. Those are all the same. The main difference with uh memory is uh currently it was uh in megabytes, because that's what uh AWS reports like memory limit in megabytes, uh but uh just to have more consistent units across uh the different uh foreign Ty.

B

A

Wanted to convert it to bytes.

C

Okay and about the the ID So, the plan is to remove it or yeah.

A

So uh the ID is.

A

So the main thing the ID was was was the uh the Arn for uh Lambda right, and the thought here is that Arn is already just composed of the various uh attributes that are already uh set as semantic conventions, either here or in the cloud spec yeah. So, okay, it felt like there's a lot of duplicate information. There.

C

Yeah I saw that in the python layer it adds the log stream I think or maybe the instance.

A

This is a new one, the the fast instance. So here we added the log stream.

B

A

I, don't think instrumentation currently does this, uh but uh this was already in the specification.

B

A

So is that is that what you're asking about the log stream yeah.

C

Yeah I I confused between uh between the two of them, um but uh yeah I, think it makes sense um uh okay and about the AWS uh tags.

C

So um I think that it could be uh great to separate um the AWS account in the region um instead, instead of using an Arn so.

A

There's already a.

A

A specification for a lot of those things, so the resource semantic conventions for cloud already has account ID and region as separate attributes.

C

Okay, so are we still thinking about adding the RN, in addition to that, for for lambdas.

A

No so uh down here, uh I, you know in the meeting we discussed uh potentially adding the Arn uh to this specification, but then we're like. Then we decided that uh all of that information was already duplicated uh across all of these, so it seemed unnecessary.

C

Yeah also, we also had this uh like the discussion back there in heptagon and I think that it was very useful for customers to be able to search by specific their region or their company. So it's uh usually more useful, I. Think but okay sounds.

B

A

The the main use case I think for uh having Aaron as a uh a single thing would be like if you wanted to copy and paste the elsewhere.

A

Okay, but yeah I, don't think that's a strong enough use case. Yeah.

C

And it's also something that you can probably even add on the UI level for various tools. If it's a if it's proven to be useful,.

A

C

A

Other thing is like we, we said that it's much easier to add than to remove. So if there's enough user demand, you know, uh Amazon can put forth or somebody can put forth a proposal to add air and back in somewhere.

C

Yeah, okay I was also looking on the specification and there are two things that I think. Maybe we can also consider um one is the um there are also specifications for the like semantics for invoked um operation for invoking functions and maybe I can can open it in a sec.

C

um Yeah, um just let me find it.

C

C

Where is it okay, so I, usually right.

B

C

Okay so this part so there's the invoke name and other local providing region which is gets mostly for iws.

A

Okay, so this is like, if you are making a a direct indication called via uh like a client.

C

Yeah, so in this case, I was thinking if it should be just the same resource attributes, and one of the reason is that, for example, the provider and region is relevant also for other Cloud operations. So if I publish an SMS message, for example, so in that case it's probably not going to be an invoke region, I think that that it should be.

C

uh uh You know, standard semantics, because it's like it's a it's quite similar.

A

So I don't have a strong opinion on this. I I would suggest bringing this up in the slack uh Channel, um but my thought there is that uh I think there's some benefit to having a different name, at least for you know the the region uh so like, for example, say you have a Lambda function running in you know uh the EU region right and you invoke a Lambda function uh in it on accident, in the a U.S region.

A

And if you have those uh as the same name, then you know One's Gonna overwrite, the other Yeah.

C

So basically one is in the span attribute and the other is in the resource that you would write and I think that most of okay.

A

C

Most of the systems systems still probably combine them.

A

C

A

Know that a lot of systems know how to keep those separate.

C

Yeah, so I I think it's it's that's also what I was thinking um that it's it's maybe good at it separate, but this issue is relevant for, like uh every SDK operation, probably like in Cloud operations, so um I think we should. um We should have a standard.

C

Idaho wants a separate name, okay, but it still probably should be not unique for uh for fuss, um or we are okay that the same attribute will be both on the suspended resource and then the backends will need to deal with that, um but yeah, it's a it's a good example. I think.

C

um Okay and by the way, also um in the specification, I I'm, not sure, if it's um enough explicitly like saying that um all the trigger data is part of the same span.

C

um So also another point that um I was thinking about.

C

um Because it's it's I, it might seem that the trigger is in a separate span. um So we you understand what I mean.

A

Sorry, no, could you repeat that.

C

Yeah so um I know that in heptagon and I pretty sure that in other sdks as well, the trigger was a separate spam, like a parent span for the spend that um generated for their request. So you have a parent HTTP span: parents, sqs band, for example, um and and here the specification said that it everything on that on one span.

A

I see so I I think that the specification here is mainly just defining the attributes, not necessarily where those attributes should go.

A

Okay, included messaging has got uh gone, a lot more uh heavy-handed on specs, uh specifying the way that the span should be structured and uh which spans should be created, uh but I don't think that that is the case in this semantic convention uh am I wrong.

C

So actually I I, don't know, but I think that it should. It should be stated other like. Even if, if it's you can do either it should, it should be written um and I. I know that, for example, I saw that there is um NPR in JavaScript um that adds HTTP trigger this Quest trigger information and I. Think that it's added on a separate spam um and so I think that it should be like um at least for Lambda, for example, or for each different environment. It should be standard across all the runtimes.

C

So it's something that.

A

So one other thing to consider, though, is that, uh like I said that the messaging Sig is also working on uh updating their specification and I feel like there would be a potential overlap or conflict. If we were to try to do the same because, uh for example, I think, a lot of the messaging uh sigs uh uh design and proposals overlap very heavily uh with uh fast, specifically around cases for like sqs message, listeners or SNS stuff, like that.

C

Yeah yeah, definitely, let's fur, um but still it's it's like a one kind of trigger so like I'm, not sure if it should block us for the order kind Maybe for this trigger it will be, it will be a little different but but I understand.

C

um We should definitely at least take that into consideration.

A

B

A

I mean I would suggest um again if uh I I think there's a legitimate argument that you're presenting here uh that maybe we should be specifying the uh the structure a little bit more I. Think that, uh for example, one might say that hey, uh we need to Define what the span looks like that. uh The uh the Lambda extension is creating, for example, versus what's being created by the applications instrumentation itself, um because I don't think that that's addressed in this specification either.

A

So you mean in that case maybe it's AWS specific I, don't know.

C

I'm not sure why I understood the example.

C

A

I I guess I, don't have a specific point, just saying that uh I think maybe you're right that uh we should specify the expected structure a little bit more.

A

What that is, I I, don't know uh specifically, but uh uh I can certainly, let's take note of that um in the notes, and we can bring that up in the next Sig meeting uh and then also you can uh mention it in slack. If you want to continue carry that conversation there all right, um let's see so.

A

What were the the questions.

C

So I guess it's about the um specifying.

C

um If uh RN trigger span um should be set or I know, maybe we can say it even more generally sure.

A

And what was the other thing that I suggested.

C

Yeah I think it's about uh invoked. uh Oh.

A

Yeah client, stuff yeah.

A

B

A

A

Does that look? Okay, yeah, okay,.

A

A

Spec assessment: um okay, so have you had a chance to look at the uh this? The spec assessment uh updates.

C

Yeah I took a look on the python PR. Have you know if you look at well um looks: okay, I, don't know.

A

Okay uh feel free to uh like And subscribe, yeah.

C

Take a look later today.

A

And then um discussing native support for 11.

C

Yeah I added that okay.

A

C

A

C

Yeah, it's a foreign discussion um about the collector layer, so um so I I I they.

C

They are very Seasons that um that um people gave there about um why it could be useful now or in the future first for uh users, but um I do think that maybe uh we we should decide if we also want to have native support for installing uh land instrumentation without a layer by exporting uh directly from the from the Lambda Handler, um and- and it's true that now you can install the instrumentation but I, don't think it's it's quiet um like clear how to do that for like uh I, don't know, average users and um also I know that there are some some stuff that are added on their layer level, for example, um um resource attributes, at least in Python.

C

So they are extracted from environment variables um on the like on the boot Scripts right yeah. So that's something that we if we go in the direction as well, so we probably want you to change and I just know that from our explicit users having the the most easy way to install this rotation could be very useful in in many cases and I. Think that um adding a layer and also if you need to compile it yourself, um could just add some more moving parts that are um aren't always necessary.

C

A

So I I get the concern about um how the current implementation requires uh blocking right, so uh the Lambda itself is blocked until each span is reported all the way to the back end, whatever back end that is um and depending on you know, Network traffic or whatever. That can be a relatively long time and you're paying for the Lambda during that whole cycle, uh which is unfortunate because you know having the the architecture where you just send it to the collector and then return certainly sounds appealing, but there's certain there's.

A

Obviously the downsides to that of you know the collector could hold on to spans that kind of get stuck there if the Lambda gets suspended or shut down after the Lambda is invoked.

A

um So I was thinking about that, and uh you know the way I would probably try to solve this if I want, if, like, if I had my own application or my own company and I, was trying to optimize uh the Lambda invocation time, I would probably try to have the collector throw instead of reporting to a back end uh outside of uh my network. I'd, probably try to put it into a queue of some sort that.

A

Is now outside of the lambda's invocation context and I I can then take that uh from a a different system collect it and report it asynchronously.

A

Now, what what that looks like in terms of uh processing that might be expensive uh because uh span uh the The Collector data probably would exceed the size uh allowed uh for sqs messages fairly easily right.

A

A

So the other option would be to have a collector write the data to an sqs bucket. Sorry, not a an S3 bucket that another Lambda could listen to and uh collect the the S3 data and report it that way. Instead of uh putting an sqs uh queue in the middle.

C

So how I see it's, it's yeah doesn't necessarily um matters if you export the data to sqs S3, um some backend or um and another auto collector that that you run outside of the Lambda, um because it's like another service that has to synchronously approve that it gets the data or it gets the data. So you can terminate your invocation right.

A

But I guess that's very similar to uh you know a Lambda writing logs right.

C

um Yeah, the logs I think are eventually written asynchronously.

C

So using the like, the AWS environment yeah, you just.

A

Anyway, so the reason I was saying right to like S3 or sqs, for example, is because that's operating that's guaranteed to operate within the uh uh the execution in uh like the the network region.

A

uh So in theory, it should be way faster than blocking on potential Network congestion across uh different regions.

C

Yeah yeah I I uh I can see that it could be like a good solution for uh for users in you know, under some requirements and not sure like how much uh it's different from just running a collector on in AWS, for example, in in eks or ECS, um and just make sure to do it in the same region.

C

um um We can take a look at that in the end like. Potentially it could be. The.

A

300 exporter.

C

A

The reason I would say that that's a little bit more dangerous. Unless that collector is uh running outside of Lambda, uh then you know that HTTP request is, is potentially blocking right, reporting.

C

That yeah, but reporting to sqs is also HTTP right.

A

Foreign, so, but what I'm saying is: uh is that separate collector now also blocking uh or does it let the uh the the data get reported asynchronously so.

C

The receiver is blocking, but then you should have the exporting it further should be as important. Okay, yeah.

B

C

Yeah and it's it's another important Point like at least on Guiding users, to uh to do that to add and as soon currently processor to this kind of uh collector.

C

C

Yeah, okay, so I I think maybe it's we should discuss it um like with the you know, with the rest of the Sig, and it could be good because I think that it may be. If, if we want to do that, it could be a good point to do this um now and not like you know uh um a few months or more now, probably it will means that more work will have to be done if we invest on on the layer and also for like instrumentation.

A

Yeah I mean maybe the better option is to, instead of putting a full collector into the the Lambda, have a like a stripped down version of The Collector. That uh is mainly just collecting all the resource data and then sending to a separate local, uh a separate, collector elsewhere.

C

Yeah I just I, don't think that for collecting research data you need a collector uh I, don't see, see how much it different from running it inside the Lambda, maybe besides needing to code it in different run times, but it should be fairly small, so I guess! If, if you you want to do something like.

A

All of the resource collection, uh jobs that the collector currently has should be pushed into the uh the application and uh the the language specific instrumentation.

C

Yeah I I mean to make sure we're talking about the same thing. It's it's. It's just um parsing involvement right variables.

A

All the things that the collector is doing to be honest,.

C

So I I'm not sure that I do either. But from what I understand it's, uh there is an option to get data from the lambda's Telemetry API, but I. Don't think that it's it's used by default and I'm not sure exactly um what uh what it is used for. But there is this possibility, um but for uh for the most simple use case, um I think that you can just um do it on the on the Lambda code and for for more advanced users.

C

We they should have this option and it's fine, but um but yeah I just think that having both of them could be useful.

A

I wonder if it would make sense to design a collector instance that can be run as a Lambda itself.

A

um Because, like for example, going back to what uh Anthony was saying, some companies don't want to have any they they want to be pure Lambda right. They want to be able to Auto scale uh as much as they can I think that makes a lot of sense. So uh maybe it would uh be interesting to provide a collector instance that can run as a Lambda.

C

The thing that the the Lambda pattern is is not is not based to to be able to aggregate requests, because you can't guarantee that um yeah that you will be able to export the all the data. So I think that this is one of the important aspects of a collector as it Aggregates much data and that's for it.

A

C

Yeah and for for the also for the long term, I think that the best solution is if uh there was a possible, a possibility from AWS to um make sure that you don't lose the lose data without exporting it um right on the Handler end.

C

C

um And if it was possible so a possibility so definitely using Alero was very useful.

A

Is that the question that we're asking.

C

uh Yeah and college slash supports I mean it's probably depend on the on the use case.

C

Yeah and also to I guess to ease instrument, instrumentation installation.

A

I'm, not sure I necessarily agree with that, um because uh you know it's it's a trade-off writing documentation to support a Lambda layer uh and having all of your deployment logic in in there seems a lot easier than okay go and deploy this collector instance to an ec2. You know collect the the information configure it over here, make sure that it's on the same region, if you're deploying across different regions, you have to kind of go through and do this over and over again yeah.

C

It I I, don't think that it's it's it solves one another. It's not either that or either that, because also if you run, uh if you run the collector as a layer and your back end, I don't know whatever it is, is running on another region. You will still pay the extra time. So, if, if you want to save to to have a better latency, you probably should run a collector um anyway outside the Lambda okay, I.

A

Just thought of an idea, so I believe that the issue here is that the collector um will uh will get suspended after the uh Lambda execution is going to shut down right.

A

I believe somebody was saying I, don't remember, who that uh the that Lambda is supposed to send out a um a suspension.

C

A

uh And then you've got like 300 milliseconds to to do something right.

B

A

A

I wonder if it can use that signal to instead of uh reporting and blocking and waiting for the data to be sent out of the network if it can use that 300 milliseconds to write the data to S3 and then the next time it starts up. It looks if there's data in that F3 bucket.

C

uh Yeah but it doesn't necessarily wakes up again.

A

A separate process that looks in that F3 yeah, but the default operation is to send it asynchronously uh until it gets that uh suspension thing and then uh that's when it says: okay, I'll, just dump everything to uh to S3.

C

Yeah I I I agree that it it could be helpful uh to do something like that. um I mean we should probably explore it more and and try it and yeah using S3 or another destination. It really depends on the um on the performance right if Esther gives a better performance, so it could be better than the collector or directly to the back end.

A

um Or or it could uh write it to a collector right yeah that has the benefit of not overloading that collector for most of the use case right. So the the default use cases for each Lambda, uh each laminate collector to send it themselves and then have the fallback being to send it to a a lambda's ec2 backup.

C

A

B

C

Yeah I think it's something that that we can try and and and see.

B

C

It works um I'm, not sure if it's like that, if we do have this option, if users should be able to encourage to use only the layer version because um um I mean I'm, not sure that it will it will, it will mean that you can completely uh export um the traces asynchronously, um but but if it does so that's.

A

Good I guess one question is: how long would it take uh in a you know, production load to send Trace data from one from a Lambda to a collector running in an ec2 instance.

A

Yeah same reason.

C

Yep, that's that's a good question. Ultimately, it's it depends also on how much data you collect and that can could work change. A lot um I mean it's going to be. Thousands of spends, like you know, theoretically,.

A

um Right, even but even in like a high load situation, is it realistic to expect that you can do that before the Lambda gets suspended.

C

Yeah I know that on hexagon we had like uh usual response rate of uh I, don't know, I think it was few hundred uh through like a TENS of milliseconds.

A

Okay, so 300 milliseconds should be totally reasonable. Yeah.

C

But uh that's like the average case. We should also look on the you know the percentiles, um but yeah. It sounds interesting to to check and I also try to see if we can do it on on our team to this kind of PLC and and see like uh how much it helps and yeah like I said if we can count count on that as like as a better solution.

A

Yeah, if you guys want to take that on as like a research topic, I I, totally encourage that I. Think I think this is a a useful area to explore um and uh the the other thing I like about this idea is that the default could be completely blocking right.

A

So the collector could block uh in just the standard deployment- and you say hey if you want to optimize this, go and deploy this collector on a separate ec2 instance configure it in your lambda's layer collector and then uh then it doesn't have to block anymore, and it can just detect that oh I've got a proxy Lambda.

A

That I can send to as a backup and change the logic in the the Lambda collector yeah yeah, because it seems like a much easier uh documentation flow or our user flow than what I've been proposing of having a separate S3 bucket or whatnot.

C

uh Okay, so I I will think uh with uh with our team and see if we can I can I get to that soon and we'll keep on track.

B

A

B

A

C

C

So I think that um next Point, um we think that it could be um much more efficient if we can merge those things to have a single one.

C

um So um I was thinking if we can find a time which is like a bit earlier now it could be. uh It could work for us, um so I will about to just um proposals in the select Channel but happy to to hear your opinion. I.

A

I'm East Coast, so I'm pretty flexible yeah. It's it's not uh as big of a deal for me. It's the the West Coast folks that I think have the earlier morning issue.

A

C

A

Okay early morning for them late night for you, you know uh you're.

C

Both yeah, even if we can do it like uh two hours earlier, it could be um much more. uh You know um it could work much much better for us and then it could be easier to to discuss everything.

A

Yeah I agree: uh I can uh put that down here. Can we find in our earlier.

A

What's your time zone name.

C

uh I guess it yeah.

C

Yeah that work.

A

Anything else we need to add to the agenda.

B

C

A

What's your proposed time slot.

C

So, uh ideally it's either Monday or Wednesday.

C

um At what time it's like, PSD, wise, um it's like to start at 10 A.M uh in maximum.

C

But if we can, if what.

A

Would 10 AM for PST be for you? It's.

C

A

8 P.M, okay and uh the so the current Sig starts at 12 p.m. So you're saying if we can move it ahead, two hours that would be best for you, yeah.

C

And also like a Tuesday or Thursday uh could work, but it uh it's uh less ideal for us, but it's still I think better than and then currently. Okay.

A

B

A

Would recommend writing down some uh some recommended uh some requested time slots uh and see if we can uh change the meeting? Yeah I don't have a strong opinion. So, okay.

A

Anyway, have a good one, it's been a pleasure. Thank you thanks. A lot bye.

C

C