Open Telemetry Uncategorized, 3 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-01-03 meeting

Description

cncf-opentelemetry meeting-2's Personal Meeting Room

A

A

B

Hey how's it going.

C

Yeah, just getting my mic set.

B

C

Right I have my hey: I had Reuben here for a second that came back to an end. Okay,.

B

Give me uh give me a sec, sorry, I, gotta drop off and come back.

A

C

Reuben was here for a minute he could have said hello, but.

C

I, don't know what you're saying.

C

Am I, can you hear me.

C

But can you hear me, can you hear Jimmy, you know.

B

Sam had to drop off.

C

Already I think he's at the same thing as you there's some audio.

B

uh I I got no a.

C

B

Call that I needed to address.

C

Yeah today is my find out which bills I haven't paid because I'm getting emails being like hey, you sure you don't want car insurance. You might want to renew that I'm, like oh yeah right. Thank you.

B

I see uh yeah you gotta I,.

C

B

Mitigate those with like automatic, uh you know, withdrawal or whatever, because.

C

Yeah I'm, just like not very good at being an adult.

B

um It it doesn't get better with kids, I mean.

C

And I was just I I had my baby a second ago. I was gonna, have him say hi, but he had to give him back well.

B

Happy New Year, my friends, I, hope that um this new year is very prosperous for all of you. Thank.

C

You um Sam is that your Shopify calendar behind you that's suddenly free on the wall. There's no looks like there's no meetings.

C

It's a joke because our calendar just got.

B

Wiped uh so my man um Sam, is talking, but we can't hear you.

B

Is your is your like? Did you get like a security update or something, and you need to like reauthorize everything.

C

Did anything good for Christmas.

B

Me yeah um uh yeah got a lot of cool. You know things from my family, so.

C

I know I'm sure you got cool stuff for your family. Did you get anything from you know? Oh.

B

Yeah yeah I got some cool stuff. Okay, my my.

C

B

Stuff, you know got me like a a winter coat which came just in time for the weather to change again, and so now it's like 70s yeah,.

C

Okay, it was like 30s the week before.

B

And then it's like 70s this.

C

Week, the winter coat salad, that'll that'll last you, you know.

B

Yeah yeah yeah, the last one I bought, was like you know, 15 years ago, maybe right so yeah there. He is all.

D

Right, well all this trouble in that I, don't have anything to say: Happy New, Year,.

B

Happy New Year.

D

Everyone I got nothing to say.

C

D

Out there and try your best.

C

Merry New Year.

D

I I woke up this morning. Just wanted to check, get up and I was like Ariel made. 98 pull requests sound.

B

Yeah, you know it's 98 pull requests one line at a time yeah, because you know a lot of it is like a lot of. It is going to be bike shedding, especially with like these RoboCop linter rules um right and it's kind of like I, don't want to make one PR where it was like.

B

We had 30 unrelated conversations because we felt like some rules. Okay and I, don't know, that's why it seemed like a lot really, but it was kind of like I, upgraded Robocop and it's like oh here's. These 97 cops that are whatever um enabled some of the other ones were kind of just like paper cuts where it's like uh good deprecated options that we never got rid of and I'm like I'm gonna, get rid of these now and like rails. Five I'm like that one's a controversial one, I don't want to support Russ 5 anymore.

B

You know what I'm saying uh and then, like all this started, all only because, like this became a yak shaving thing, because the only thing I really wanted to do was make the active support, instrumentation work without rails, because.

C

B

The active support instrumentation, it's like referencing, a rails, constant and I'm like oh, no, that's not good. We can't you know we have to make all the things depend on Rails in order to make it work.

B

So I did that and then I really then I saw warnings in the build output and then I saw this and I was like. Oh wait, a minute Ruby three two is out and oh my God appraisals, don't work with Ruby three two and then it was just like what.

C

B

C

Are you winning son type of.

C

B

I didn't even get to do the stuff that I wanted to do was, which was like look at the SQL instrumentation like I started. Looking at that and I realized to myself, like you know what we did like this subset of copy paste, modify and um I think we need some investment there and be like no. No, we we actually need like database instrumentation and.

C

B

And have like some strategy pattern or something that's like you're, actually injecting this into postgres or you know, MySQL or Trilogy or whatever yeah. um This.

C

Was uh I don't want to spoil the convo if other people come I, don't think anyone else is coming. uh I do have to leave at uh 12, 30 I think uh you know we could do the agenda. I guess but I do wanna I! Think that's a good idea, I think seems like mix-in pattern or something with these like General. The problem applies to http instrumentation as well, but we've been stuck in the mud with like getting that that went through two or three iterations, where I attempted similar things and it just never. The.

A

Pattern never ended up.

C

Being very clean yeah, the pattern was not super clean and.

A

C

Priorities changed um but I think some. That is a pattern worth establishing wherever we put our flag down in wherever the first instance of it is because I think it's something that can be. You know, messaging cue instrument, you know it'll just be a thing we'll want to have in our toolkit. So like I'm, a big fan of that work, I'm not going to do it I'm happy to review someone else's stuff, uh but yeah yeah I'm on the I'm on the same page I just like have zero. You know yeah.

B

It's kind of like we only looked at DD, Trace, RB or whatever, and then kind of like pull some other things in like, for example, like with the with the rails routes thing, the the honeycomb instrumentation has a solution for that already that.

A

B

Cause problems for their existing route, parser yeah. So like that's the kind of thing that we can take, inspiration from and import it or from New Relic like how new Rolex implementation is.

B

um uh It's not an easy dropping copy paste modify thing because their architecture is a little bit different and for me, it's kind of like well. If we want to retain like this notion of uh having any instrumentation per uh Target gem right, um uh we we still need some sort of common package for database instrumentation sure that can look at the SQL query and again like apply a strategy to it. Maybe it's like the PG instrumentation right.

B

It still needs its special hooks, because the PG gem has very specific, um that's very specific API, but the things that are common, which is to uh obfuscate, the SQL and potentially extract metadata from the text. Even though you know the specs like. Please don't do that.

B

um It's something that we can do and- and you know, enriching database drivers, fans with active record events like if we don't want to extract the table name from from there. Maybe that's something that we can I. We can enrich the driver span with active record of metadata I. Don't know if, like the event itself, like the active record event, has um the arrow table metadata in there, for example,.

C

I uh I can't say I'm up on the top. You know like I, don't have context, but I agree with the spirit of what you're saying and like I think it's you know, I think that the work is worth is a high I think it's high leverage work um that will make it easier in the future and also give us benefits now and like I'm, so excited for other people to do it.

B

You are very funny.

D

It's fine, but you two haven't actually worked together at a company.

A

C

Worked in a professional context, if that makes any sense, but I don't know if I can talk about it more.

D

Yeah, don't don't say anything God.

C

B

uh Sorry, what what do you say that Sam, you didn't yeah.

D

B

A

Like we all have the same Rapport right like.

B

Yeah, like Eric and I have New York Sensibility.

A

Yeah right, yeah, Sam's.

C

A New Yorker.

D

C

Not he's Brooklyn you'd spend like a decade in Brooklyn.

D

That's true, that's true: I couldn't hack it in the Big Apple I I had to leave because they chewed me up and spit me out. Like I, went back to the suburbs, Where I Belong. Remember.

C

You didn't sell out you bought in yeah.

B

A

B

I kind of feel like we all got that to some degree right. We all got a little bit of that here. I.

D

B

Know I'm I kind of just try to be that person with everybody. You know what I'm saying like I, never even like I know where Rob Kia, I'm I'm very um very.

A

Playful I mean too, you know what I'm saying.

B

Yes, yes, you people, like my homies outside of my homies yeah,.

C

Rob, the homie uh cool, do we uh yeah I don't have or there should we run the meeting I have I've. Ten I have 15 minutes.

B

um I mean: did anybody attend the sick.

D

I know: I looked at the I, look at the notes, uh I, don't think, there's anything actionable for us merged or anything like over the holidays doesn't look like.

C

um Yeah, let me check the I, haven't even looked at the red thing. I do think uh you know something I think if you had don't have your eye on I think would be of value to you. Thank you, Samuel there's, a I, don't know said: what's your name, um the connector prototype in The Collector is really cool.

C

um It's essentially someone wants to introduce a primitive to allow you to connect pipelines in ways that don't involve and it you know so right now like if you wanna, generate metrics off your traces, Pipeline and then export them through your metrics pipeline you're, basically making like an extra there's an extra like HTTP trip there. You actually have to send it through and it's sort of done on hack, or um this is a component that allows you to connect pipelines, uh okay and sort of Define.

C

What that connection looks like arbitrarily, so you can say: hey I have a tracing pipeline. I want to generate some analytic, metrics or I want to generate some I want to take these span, Events off this pipeline of traces and send them through my logging pipeline because span. Events for me are actually better saved as logs or like same I have yeah I'm. Not you know, hypothetic about you, know, I, don't um or I have some I have some logs and actually there's some traces I want to abstract out of those logs.

C

Like some, you know a test run and ship them through and so like. This is a anyway. It's a cool uh I think it's a cool prototype and it seems like it's getting I attend the collector series.

A

C

The attraction and like a few maintainers separately, uh I forget, who movie store guy, is I want to say I'm movie star guys. A couple folks are feels like it's starting to get done um and.

B

C

Might be something people care about here. uh That's my only like broader Hotel thing, yeah.

B

You know what's kind of curious about these. Things is kind of like how the hotel collector is starting to you know like all this transformation stuff. There's all these other tools, a do data transformation. You know like yeah factors.

C

B

I got here: Reinventing the wheel, Vector Apache, morph lines; yeah some number of ETL programs.

C

Yeah um yeah, no for sure, uh I think it's a little bit of red you it's like there's some folks. You know the meetings are kind of contentious, because some folks have a different feel. The Mandate has a different scope of like. Let's keep this a lead. You know the Unix do one thing well philosophy: let's just be a pipeline for observability data and you know, and then another then there's other folks who want to are just like.

C

Let's make it, let's really bloat this: let's make it the all-in-one everything and do all sorts, and it's like a little bit of yeah.

A

Yeah yeah yeah.

C

um Yeah, so it's some philosophical, philosophizing differences there that I, don't know I, don't know who wins got team with most maintainers when I don't know.

B

We'll see I mean it's like we want to convertible, we don't want to converge kind of thing.

C

Yeah right well, you know, there's some there's, probably some interest that are better understood by people at different. You know we're all end users.

A

C

There's probably other people with different interests and uh yeah.

B

C

B

To me specifically for the two cases, which is the deriving metrics from traces and um because I think there's already this span to metrics, but it's the but the span. Events for example, especially for the exception events right um getting those reported to like uh a century.

B

That's not tracing you're.

C

Right right, that's like.

B

That is a visualization on this stuff, so yeah it's.

C

Like a non-specification thing, you know, but it's a back end, but it's some proprietary back in it's hard to get it. There.

D

C

I was gonna, say the other one. Other thing that popped up randomly is there: is there anything we need to do for Ruby three, two that we haven't like? Are we testing it in Ci or.

B

Are you you might be interested in this pull request? Okay, it.

C

Popped up in my head, because I was like.

B

You didn't, you didn't see it in the 98.

C

Polar Quest that I submitted.

B

This weekend, whatever.

C

I was reading the date. It adds that data class, which is a better it's an improvement on struct and when you struck pretty heavily um and I, was like. Oh maybe we should use that and I was like wait a second to even oh amazing. Thank you. Arielle.

B

Well, this is only the contrary package.

C

B

C

In court screw them, you know.

B

They don't well.

A

B

It's like it's trying to do one thing at a time right, so the challenge that I ran into here is that uh there's a bug in appraisals and there is a what what do I call do you have an affectionate name for Shopify Engineers, not Stewart, my shop stewards anyway.

B

uh So this yes, some shop of folks that opened up uh PRS to try to address this problem with appraisal um and um there hasn't been much movement because I guess you know whatever holidays or whatever I did mention the person from thoughtbot who volunt who's like I, want to work on metrics, so I tag them uh Clarissa, uh carusa, light.

A

B

Or Lima, uh so hopefully you know Clarissa's eyes, you know, Chris could get their eyes on this and then um I try to do something. So basically, what? In order for me to do this like to address this issue, what I did was essentially expand. The functionality from um what appraisal does into a uh a for Loop A bash forward. Loop.

C

Oh, you really engineered I would have just said. Let's wait, it'll get fixed Ruby's a big community.

B

Well, well, you know it's one of those things where it's like um the lead time on those PR is getting merged. Yeah are a problem and I need to address my issues now right.

A

B

Always go back and and and revert it to use the Upstream things, but yeah.

C

No, it's great I done it wrong. I! Think it's good! It's cool to.

B

Be on The Cutting Edge, so let me get on this, so let me let me start on. Let me let me go through some of these. Unless Sam you have anything else, you want to.

D

Add no! No, no, no! No, to walk us through the steps. So we know what order to look at it in please.

B

It's all good I mean I mean uh the one to start with is going to be this RoboCop one or actually. Let me um let me expand these here. uh The one to start with is going to be this RoboCop one, that uh what page is it on it's on here?

B

Essentially, it's the bump and this one says enable the cops, but it's it puts it lists all the cops and they're disabled yep, and then each one of these is a one-off like enable right um and I've got them all in draft mode, because, whatever once the first one is merged, I'll rebase, all these and and we'll look at it. The next one is uh the active support dependency.

B

So basically saying like I, don't want I will no longer reference. Active support. I will no longer reference. The rails constant, but rather use active support.

B

Okay and it added some additional like safety measures in here to record an error, because, because basically, what this is doing is using some private instance variable that got named this versus name that, because the active support notifications doesn't ex expose those.

B

So uh this this will like basically write a lot. An air handler and say, like hey, yeah, open an issue here and let us know like this is no longer working because.

A

B

Upgrade of rails that occurred all right, creative, active support, uh that's high level stuff uh moving on it's kind of like a lot of other things. Little things are like paper cut, so like basically shutting the logger off and test. So we don't see any of those log warnings anymore during the test runs. Those are annoying uh use bundler, instead of requiring gems manually everywhere, so do use. One bundle require I, don't know how people feel about that.

B

One thing, I did add was because I noticed some problems that we had in production um is that I added a noisy air handler by default so like in our instrumentation test now it'll raise the error or message if it recorded an error by default.

B

It wasn't doing that before so, like some of our messaging attributes were incorrectly getting set and they were disappearing like they weren't in production, they were logging errors yeah uh because of a bug in the active record, active uh uh active jobs, notifications so.

A

B

Like okay I submitted a PR for that that got fixed it got merged and now we're back to this so I added these so that we, if somebody's like writing some tests where they're like adding attributes, but they don't actually write assertions for the attributes that they wrote, it'll show it'll, raise an error and be like hey. You tried to add a time object. We don't accept time objects, it's only like you know whatever to end or whatever you're.

C

Saying in the export pipeline like uh if there's a if it, okay, yeah I, think that's a low, helpful I, don't.

B

Yep get us on jruby94 and drop J, Ruby, 9-3 or whatever 939, because this is Ruby two six and this will be two seven and up and three one so I was like uh I, don't want to support, not you know two six, because whatever end of life right, um Ruby, three, two adding that to the builds graph, 2.0 support, uh so the graph we don't have explicit tests for graphql 2.0 uh because of some syntax changes. This one was a little bit funky.

B

So it's a little bit nasty I did one of these things where it's like I'd use, the old API based on the version of The Gem, and it doesn't feel good. Just doesn't feel good. But it's.

C

What I did that's just in the test, though yeah yeah.

B

So add a code coverage for that um to make sure that we were hitting on the right things and.

B

You know uh I kind of feel like there's something else. Okay, these.

C

All feel like I just need to take the time to approve them.

B

C

See anything that I'm always anxious that I will on this call be like wow, looks great and then like some no.

B

No I work with.

C

Will be like you, like you.

B

Gotta, look you gotta, look at them, I'm, just I'm, just briefing people on the kind of that they're gonna run into this is something that's like. Oh my God, the url's wrong.

A

Right, it's silly. It's.

B

Like little paper cuts right little things that are annoying. This is a big one. This is the drop rails. Five yeah yeah controversial, um but uh but but that's it I mean there's not much more to it other than taking.

C

This step, one user who's got an air gap, machine running rails, five and what, if we don't support them? You know.

B

C

D

Arielle, did you find? Was there an incompatibility that like spurred this or were you just like it's housekeeping season like let's, let's not support an unsupported version.

B

It started with I wanted to um uh sorry. uh It started with I wanted to try to do the active support instrumentation, which got me looking at this going. What in the world is this code right here.

A

B

And it's like inactive in real six. It got renamed, oh because in rails, five and under it's this and I'm like. Why do we want to support rails five rails? 5 is end of life.

B

This is nasty anyway right, yep.

B

um And then all the other things were basically when I was looking at the test suite and was failing, there was so much noise yeah that I was like. Why is all this noise? A problem? You know what I'm saying like. Why do I feel this noise, and then um you know and so on and so forth, right and then I was like. Oh man, like here's, an error message: that's like using a deprecated option and I'm like well. It's been deprecated forever, so peace right.

A

B

And then, lastly, the docker dependencies- oh my goodness, I, don't know yeah.

D

A

Up with this there's.

D

Like a zillion, Docker files right or no, maybe there's one you change five files, never mind. There's one dark file.

B

But the thing is like, for example, the big thing is that there's no five image for them? Yes,.

D

This has been, this has been like dirty on my local forever and I, like I. Just did like I'm just using Maria or whatever, um instead, but yeah. This is definitely the right thing to do. Is.

B

It the right thing, I, don't know because, like like we're, not gonna, have support for directly for five something and.

D

B

And GitHub actions is kind of nasty because, like you have to add this native SQL theme, because the trilogy gem doesn't support, um non-native things right now, I'm sorry, the uh the what plug-in is it the shot to Plug-In or whatever, like the authentication uh and so I'm like? Oh, no, so this is a. This is a controversial one, uh but uh uh but this is to try to get this going and it's like you know the the image was doing 265 by default and I'm like wait. Why yeah.

D

B

You know getting us a new Alpine version and using the latest version of bundler and so on and so forth. So I was, like you know, I'm, not so sure yeah.

D

Yeah like okay, well, I'll, look.

B

And I was like yeah I was like I, didn't even know how to use this I like this app thing would run, but it had no dependencies like none of the other services you have to like manually like run them.

D

Back to me, yeah.

B

So I don't know it's a bunch of it's. It's paper cuts really right. Yeah.

A

B

It's like paper cuts and so I, don't even get to do like the real thing that I wanted to do.

B

Right, which was like gets, get active, support instrumentation for rail started, right, yeah, yeah or, like the sorry, the the pending gems, like active record and accurate.

D

B

And stuff like like, move everything, the.

D

Stop monkey patching Quest, yeah.

B

Yeah yeah and I you know, and of course, I was looking at open issues which I don't know why I wasn't getting any like or maybe I was ignoring them. But people have been opening issues and there have been issues that have been open for months.

D

Yeah I, it's yes, all right. That sounds right, I think we're pretty much just like under under uh under people yeah. We need people with that. I.

B

Got you man, here's one that I thought that was interesting, that I never looked at until just now. Okay,.

D

B

Is 127. yeah the SE over a year old right, no.

D

B

A favorite I'm a favorite- this is uh three months old or whatever, and this never came to my attention. I, don't know how why how I? How I missed this but um yeah, but this is a problem with um continuation. I, don't know exactly what the problem is. This is.

D

D

Robert knows more about graphql than me and it's out this week, but and that's not satisfying to be like what about next week, but he might have an interesting. He might please more able to to look at this, but I guess. Maybe the point is that we have this massive backlog of issues and also open PRS. So maybe before the year takes off, we can try to burn that down a little bit. Maybe I don't know.

B

Or burn it down or close it or treat.

D

B

Whatever I'd be like we're, actually not going to fix this, we need some help or tag the ones that are like I really need your help because yeah, like you know like the the graphql one, is interesting, because data loaders is using fibers and uh SDK uses fiber local variables. So that's probably why they're not continuing, because the context is getting yeah.

D

You need to do something with con. Is it fiber? Is it like, but yeah all right, I need to read about what fibers actually are again, because it's it's exited, my brain because I'm thinking about the context class and it uses a ruby class called thread but thread might point to a fiber, maybe yeah.

B

So it's a very interesting I'll. Send you the read.

D

The read the docs.

B

Yeah, what I was going to show you was on you know: fibers are uh the fibers um allocations, for? Why is this so slow.

B

Let's see here, why is this? Can I just look at the red? No, of course,.

A

It's forbidden not allowed not allowed.

B

So threads themselves um thread variables, yeah.

A

B

Look them up here red instance or thread what the heck, where am.

D

I uh like a thread and then open it open bracket, maybe.

B

ah Yeah yeah yeah, you're right you're right uh so thread current, so um these are fiber local variables right. So anytime you create a new fiber. uh It loses scope to whatever's on yeah, fiber method.

D

B

In order for you to gain in order for you to share the attribute amongst fibers, you must use the red variable, get and thread variable set, and now these are going to be thread. Local and because fibers are lightweight uh concurrent uh execution inside of a thread yeah, you know. Whatever um um these things don't have, these things would have to share that yeah.

D

Have implications for performance or whatever.

B

Yeah I mean it might have implication for performance, but I think it's for correctness is the problem, because if a fiber is mutating a threads context stack, then you've got multiple things: scoped changing your contact stack so.

A

B

You change the current span like really it's kind of more like what we want is context. Propagation yeah,.

D

B

A fiber or into a thread that's.

D

Interesting so.

B

So it's like I launched a thread. I want to take the parent context and put it into yeah, something that the thread can access and when the fiber is created, it would grab the context of the pairing thread and.

D

A

I think you're right I.

D

Mean whatever I forgot, what a fiber was until three minutes ago but like uh or until you explained it and maybe I never knew, but it seems like docs on this at the very least would be helpful uh and if the docs results in a proof of concept that we could use in art, instrumentation gems, where it's like. Oh yeah, just use this one cool trick like maybe we can like use that one cool trick everywhere, but I'm not sure that I I don't know what that would be. At this current juncture,.

B

The only thing that I can think of is US, adding a thread: local variable or thread variable yeah.

B

um That is the the the parent, the parent context, but.

D

You could check the current thread for like if you're a fiber, you check the fiber local variable for the you know current span, and if it's not there, then you could fall back to like you know. Current span, finalfinal.pdf, which is set by um which also gets set on the thread. uh I, don't know, I I would need to think a big about it.

B

Yeah yeah, it's kind of it's It's. You know it's a tricky thing. Yep.

D

Yeah. Because because.

B

We don't have access to the fibers yep.

B

So it's almost like we'd have to patch fiber, maybe to be.

A

B

Yeah, that's fiber to get the thread local variable and not mess with it, but then that creates multiple Scopes. It's it isn't fun. No.

D

um But at the very least, I mean if you're, if a program is explicitly creating a fiber, then we could have docs being like set the current span to a variable and then start your fiber and then set the thing on the fibers set. This value on the fiber's context.

B

A

B

Yeah I see I, see what I see what you mean so like. If we did something like uh do, you have a few minutes to look to kind of like.

D

Look yeah yeah I, just I did I had to quit slack so I wouldn't get annoyed, ignoring, sounds.

B

Gotcha man, I'm Gonna, Do, Not, Disturb person, myself.

D

A lot of respect for that turns out. I just grew up on instant messenger, so my brain is wired to just always want to talk um anyway. I'm.

B

A 90s child as well man yeah, so you know.

B

Okay, so if we were to use with current in that case,.

A

B

Yeah we captured the context and passed the context through.

B

Would that end up messing I kind of feel like that, would end up messing things up somehow.

D

Yeah I mean I think you would want to so yeah. You probably need to like duplicate I forget how it there's like a there's, a span key on the context, and so we would want the span key I think this might be stored. Yeah it's in a different file but like we might want to like dupe, dupe the span context or something when you create the fiber.

B

Yeah yeah so yeah.

D

B

A

D

I, don't know what it is. I think you would create a new context object and that new content like because you.

B

D

This I don't well, you might mutate, the span. I was going to say you don't mutate the span. So if you just create a new context, object that points to the same span then you're, okay, because the context will have a new empty. You just create a new context, object with one entry in the stack, which would be the contacts that you just propagated. However, that context might point to a span and then, if you like, if the user tries to run operations against that span, things might go bad.

D

So maybe that's not a good idea.

B

Yeah totally, uh the the context is immutable.

A

B

So when we add new values to it so we're using with current, for example, like we set the current context, yep uh in the fiber.

D

Yep and it's just an array of array of contexts and then each context objects has a span key.

B

So if I do something like uh so if I, if wherever I am right, I have context which is uh Maybe, contacts.current, yeah and then I do Fiverr dot new.

B

Then I have to do context that, with context yeah like this.

B

And then you know your Fiverr code.

D

Yeah, but if someone, if someone online five runs contacts, dot, currentspan.finish.

B

D

I, guess that's fine! That's! Okay! You know what that's totally fine. Everyone.

A

D

Down I think: that's fine, all right, yeah.

B

I mean it's not ideal right so.

D

B

Comes actually if this is open, uh Telemetry tracing.

B

So if, if somebody did this yeah, it's kind of like why, why did you do that.

D

Yeah right so is the response. You must be this tall to write, concurrent programs or whatever uh I forget I forget the joke, but you know what I'm talking about.

B

I understand what you're saying.

D

B

D

Is it your own risk.

B

B

Hotel, if I, for example,.

D

But um I think doing this like monkey patching this into instrumentation gems seems uh scary. It's.

B

Rot with because.

D

You're, just like, uh like I'm assuming that I'm gonna be inside of a fiber like it seems Seems bad.

B

So we could patch this instrumentation.

D

Yeah yeah right.

B

But I don't want to patch fiber.

D

Right but I even patching your instrument, I, don't know it just seems like.

B

Yeah I I know I, know what you mean.

D

A good I mean that seems doable like we might want to prove it out um see if it you know, works and stuff, but like it seems to be like yeah we could just be. We could pass the instrumentation like this. No problem.

B

Right I guess I'm wondering like what is it yielding to so spawn fiber is yielding to something to a block, that's being given to it.

B

B

So like, if we were to patch spawn fiber.

B

Yeah, this is too much.

B

This is this variable should be fresh with each new fiber.

B

Oh boy, uh-huh: what exactly is this doing.

D

I think, like kind of the same thing that we are like, like managing thread variables uh or Fiber local variables,.

B

D

B

So yeah, so so this is copying the thread local variables into fiber local variables. I, don't see the red local variables here. I only see fiber local variables am I reading this wrong, because this is doing thread current Keys. These are fiber locals.

D

Yeah I mean you kind of I, uh you're kind of like. Why isn't this shouldn't the context actually propagate from the parent? Looking at.

B

This yeah, looking at this code.

D

B

As long as it's not the graphql runtime info, so maybe the error isn't what we that.

D

It is, oh, you know what right yeah, because if you go back to the issue, the error is that the the stack, the stack stuff doesn't right. The error is like uh the context, calls don't didn't the calls to detach contexts don't match up correctly and the reason that that's happening is when you detach a context it. When you attach a context, you get a token which is the size.

D

But yeah but then maybe that doesn't make yeah right, because maybe if you have two fibers that are doing things concurrently, then no there's something going on. No.

B

No yeah yeah yeah, so let's think about what that means.

B

um The the fiber exits and the threads are going. The the thread variable is good.

B

You're the fiber variables, the fiber variables are gone.

D

B

It never really detached.

D

Yeah, if a fiber attaches a context and then doesn't detach it, then.

B

Like nothing's happening right because um inside of those like you're like what you're saying is, some work is happening in the fiber and it calls in Span in Span in Span yeah, the fiber that is executing.

B

B

Has the current context in place and it is creating a new context in the fiber that is local to that fiber and it detaches all the way to the one that it copied originally.

B

And then it's up to wherever that context and that span was started that it costs finish.

D

B

So the span is a non-recording span because it was finished before the fiber.

B

Like okay, I have a thread and the thread had a context and it was an in-span yeah and the Fiber goes off and does stuff yeah.

B

But it is a parent thread waiting for the fiber to complete before it exits the spam block right.

D

B

If it calls finish on the span, then it spans not long no longer recording and some instrumentation might be looking for if it's recording, because it's a basically a mutable span at that point, a read-only span.

A

B

Don't know I, don't know we got to look back. Let's look back.

D

At this I think before you hop over on line, 294 I think it's also just doing a straight copy yeah. um So it could be like you know.

D

If you open a span outside of the start of a fiber and then the fiber finishes is to finish a span within the fiber, then the I don't know if parent fiber is the correct terminology, but the the first fiber that existed prior to starting the second fiber um I think the detached call within the child, fiber or the second fiber will actually mutate the array and pop something off the stack um and.

A

Then right the.

D

Tokens won't match up, which is maybe what you were already saying.

B

um Yeah, so this is so. This is more of uh more of what I'm thinking right, so okay, so this was one. That's like this is a a thing so.

A

B

What do we have going on so I've got a thread. That thread has um it's like kind of like a main fiber stack, yeah.

D

B

And on that main fiber stack is the current context.

A

D

Along with, like you,.

A

D

Other contexts.

B

Foreign look at it again, let's just be sure about this: okay, so I'm not talking out of my head here. Oh uh it's API package um I need to learn how to navigate this UI myself right, I, work here, yeah, yeah, sorry, folks, that's not criticism of my employer thoughts and uh whatever are my own, so um with values or whatever is doing the set values attach and yield and detach right that that is a pattern that is happening everywhere: yep, okay, so anyway, um so we have our uh contact, storage, yep right.

D

Which is a stack of cards, an array of contexts, basically yeah.

B

So this is the contact storage and that is, let's just say, like you know, whatever parent context is.

A

B

Okay, so that's a main fiber local variable now from within this thread, I make a new fiber yeah from the graphql.

D

B

So what did uh what did the grack? You all do? Well, the grack you all went and copied this yeah.

B

Okay, all right so what's happening in this thread is that this is doing a you know: um uh let's do Tracer in Span right or right, Tracer in Span sure. So what happens there? So sorry, we've got this.

B

So root context, I'm going to call this sure so in Span right so inspan ends up changing this.

B

And now uh inspan has um a root context and then we have like the you know. The I'm gonna call this. Let's call this so uh I don't know operation get get.

B

um You know what no sorry post, gradual right, graphql right.

B

um That ends up making another context. Nope who's uh current span is equal to post graphql.

A

Yep right, that's right.

B

So now uh we're still in this block.

B

Yeah we're still in this block, so it does a fiber new okay. So this is the uh this is inside of the graphql gem yeah graphql data loader. So this does a fiber new right before it does a fiber new.

B

It creates these. uh This fiber locals uh fiber locals um hash right, yep.

A

B

um What is it doing? It's got that fiber locals hash, which has this in.

D

It yeah exactly right.

B

So this is the memory that we see. This is what's allocated. Sorry.

B

Okay um and now what happens when you're in the fiber you're in the fiber.

B

When you call open telemetry.

B

uh Trace current span.

B

um What is it return?

B

So if we looked at Trace and we'd look for uh Uh current span, yeah.

D

B

Looks for our context, current yep, all right, which looks for current span.

D

No, it looks at the top of the stack. Oh.

B

You're right, you're, right, you're right uh context: current, let's go back to contacts current, what's called stack last.

D

Which should be the right one.

B

Which must return, so this is going to be returning this context here and that's returning. This span.

B

Right yeah now.

B

Now uh so so, once that gets returned, the fiber does something the fiber does tracer.in span right.

A

B

And this does, like you know, operation a or whatever foreign.

B

This should be linked correctly, because this is going to create another Span in this fiber local span context right right,.

D

B

Going to say that that's what's here right um this will this will generate a new, a new set of fiber locals, which will include operation a now.

B

B

So this is correct and what happens when it exits this block. It calls finish on this span and it pops the stack returning us back to what this was.

B

So the only problem is going to be is if we ever finish a span that I didn't make, and you know once this gets garbage collected the fiber locals, go away, I think where the problem lies is assuming that the fiber is still executing within this block.

D

B

So that's my my question is well is the is the thread that launched the fiber waiting for the fiber to complete, or is it detaching from the fiber.

D

B

And then, all sudden, like this, this set is like a closed or finished fan.

B

um You know, whatever, with uh with a set of contacts that are stale.

D

Yeah and then I mean I, don't exactly know how the I forget how it works. But if you spawn two fibers simultaneously and they both have the same, they both point to the same array of contexts. And then each of them is pushing and popping to that array like I forget if it's a gonna be a pointer to the array or if the array is going to get cloned um you're.

B

Talking about for the context when it's yeah.

D

B

So the context as far as I know like anytime, that you do something where you change the values. The set values function is gonna um right.

D

B

Create a duplicate of all the entries in there and return a new instance, but.

D

B

Contents of those can be mutated right.

D

Put the stack itself the stack stored as a fiber local variable, yeah.

B

D

That like when, when graphql the gem goes through and copies the fiber local variables.

A

D

It gonna are all of are all the fibers going to point to the same array of context, objects or.

A

D

A

D

That seems like it could be that because, because that because then, if one of them's pushing and popping and the other one's pushing and popping, then if it you know, depending on when things start and finish like the things might not the attach and detach calls might not match up. If, indeed, that array is shared. Oh yeah.

B

Yeah yeah because it's not duping those values right, yeah.

D

That's right dupe and wouldn't you need to dupe the the about the context? Objects within the array. It just would I think it would just need to dupe the array and then the push and pop on the stack should match up. But this is all like I mean you. You seem to be confident that the the array- it's it's all all the fibers that'll pointer to the same array in memory. So if that's true, then we're pretty host.

B

Yeah because it's effectively making the fiber local variables it's effectively, making them thread local variables. The array has this code. The graphql data loader has effectively made.

B

The fiber local, the the open, Telemetry context, storage, stack into a thread. Local variable right.

D

Which introduces all the problems that I was like scratching my head about before we went down this rabbit hole. We were like what, if you could have like set like a con, you know, but what? If there was a thread, local context, foreign.

B

Okay, Sam yeah.

D

uh What a lot of fun! This is great! It's.

B

A tall drink of water, my man right here, yeah uh I, shared that just with you, you could take a look at it. Thank.

D

B

D

B

D

Yeah, this is uh oh, no I got it, I got it. I got it.

D

Well, um I'm happy to taste I'm happy to to summarize in that ticket if you want, or if you want to, if you want to summarize, um uh but I probably should go, do other other stuff for now. Do you is that what what's your hat.

B

What is my hat yeah.

D

It looks like I feel, like I'm getting high school sports from your hat.

B

You are getting uh Austin city, swim league.

D

um Okay, okay, got.

B

It, uh which is my my kids, used to swim for the Belterra Marlins got.

D

B

Yeah, it was our neighborhood, uh like you know, we have. Our division is pretty large, so we have um a bunch of kids that uh from the neighborhood that compete with other neighborhoods right, yeah.

A

B

Was for it was uh intramural league in the summertime and my kids did it for like three years and then covet hit, and then we never went back.

B

You know my kids start doing other stuff. You know music and Taekwondo and yeah sure you know what I mean so I I should I should actually be rocking. My TKD gear man, you know what I'm saying yeah soccer support, but Sam um I would appreciate it. If you did that.

A

If yeah yeah absolutely.

B

And then, um and then like uh I can show you actually. Let me just share that line.

B

um The graphql data loader spawn Fiverr is the problem.

D

B

I wish it would only freaking copy its own like. Why would it?

B

Why would it be copying somebody else's variables, man, let me just take a look real, quick one more time.

B

Yeah, this is saying everything except for the graphql runtime info. Why is it copying? Everybody.

D

Well, it also wouldn't work if it didn't copy it. So.

D

Right so they would have no no parent contacts, otherwise.

B

Yeah, really, what we want is like the um well. You want this context: propagation, like a text map context propagation in some sense, yeah, because we don't want like the recording span or whatever to be open and closed. What we really want is for the thread to know well the fiber, to know that the it's it's co, it's participating in the trace right and um it doesn't have to be bound to the local variables.

B

It's kind of like it's like it's kind of like treat this execution as if it was a back like a like a job that was getting um I'm pulled off a cue yeah.

D

I mean this is this is like skating point first party instrumentation. uh This is like this is why this is why we might want to do that. So all.

B

Right Sam, you, the man, bro I'm, Gonna, Let, You, Go,.

D

Man, thanks for the time I had I had fun. I appreciate this Fallen King I'll uh I'll try to type something up in this issue right now before it all, because it's my brain.

B

um All right, bro, thank you, I appreciate it. Man yeah.

D

B

Francis know too: yes,.

D

All right, yeah, happy New, Year peace.

B