Open Telemetry Uncategorized, 16 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-06-16 meeting

Description

Instrumentation: Messaging

A

Howdy, hello, hey how's, it going hey it's going well, it's.

B

Going all right.

C

Aaron, I'm feeling your pain uh I'm over here in vegas, and I think it's supposed to be like 115 today, um so I'm getting out of plane to get out of here.

D

Yeah get out of there as fast as you can.

D

115 is when the soles of your shoes start melting on on concrete.

C

Yeah, it's um I think 100 is when I turn into a pumpkin, so uh yeah most people do yeah.

C

It's been pretty brutal, um a lot of entire indoor time. I guess it's all.

D

Yeah, thank god for air conditioning right, yeah.

C

uh Yeah, it's also like, maybe really appreciate oregon, where uh even if it's hot you still go outside um and breathe some fresh air but yeah. It's definitely.

A

Maybe you want to go home.

D

The the trick to it is you do all of your outdoor time between like 5 a.m and, like maybe eight nine before it gets really hot and then again, like 8 30 to midnight.

C

Yeah, uh I've also noticed that, because I normally I'm in bed by like 10 like easy, everyone else is like no. This is the time we go outside so yeah. It's I've noticed that.

D

There's a reason that css were a thing.

A

I totally believe it now. I totally believe it hey damian hi howdy.

C

um Yeah, so uh I know anthony's not uh gonna be able to make it. I guess if four people we could probably get started. I don't think this is gonna, be a long meeting, um so I think we're probably at critical mass. So let's just get into it and um work through it.

E

D

We want to give uh two or three minutes for people that might be straight and uh struggling.

A

uh Yeah yeah, we can do that. uh Definitely.

F

D

Yeah, but I mean I I was going to ask: how has vegas been uh been fun? Are you able to catch any like shows or anything like that? I know.

C

There's a lot to do.

D

C

So yeah there's a lot to do an overwhelming amount of stuff to do here, and it is not my thing: um morgan uh mclean last night and we were kind of joking about how like we were way more into like the open telemetry like conference things where everyone, just like you go to a restaurant like you, hang out with a bunch of other people and talk and like that's way more of my speed um but yeah I mean people are going to all the shows and um yeah there's just tons of stuff to do.

D

I I will say uh both cirque du soleil and blue man group are worth are worth every single penny.

C

They both put on.

D

Really really good shows.

C

I didn't realize it's like it just like is continuously happening. Like I there's like an entire theater dedicated to it. I was like wow, okay.

F

C

Yeah but um yeah, it also is just mind-blowing how much of a different life people live here than I do yeah yeah I like spending time in a garden. They, like you, know, spending time at the the bars and the casinos so yeah. It's definitely different.

D

And if you can get off strip like off the main strip, there's definitely a different style. There, too yeah.

C

That's really true. Actually, first time I came here, I came out here to go to red rocks to go climbing and like it is a very vegas uh out in the desert, let alone um off the strip. Yeah.

D

Yeah yeah all right, I think we probably burnt enough time. Okay, yeah.

A

D

C

I agree uh so yeah, let's just jump into it, um just talk a little bit about our uh metrics sdk uh progress. uh I think we've been doing pretty good on this. I think this number is gonna go down in a little bit as we progress uh into diving into the internals of implementing uh all of the instruments and their aggregations and uh maintaining state of all the measurements.

C

uh I've definitely been playing around a lot um on my laptop with some defining the the sync instruments and the async instruments from the meter creation and how we're going to hook those up with uh the you know the view specified uh transformations as well as uh just maintaining all of the uh measurements. So I do think that this there's a few more issues. There's not I I know there's a few more issues we need to add here.

C

So that's why I'm kind of saying, like this number's, probably going to go down um and probably get closer to I'd, say in the 40 range. uh But that being said, I think we are making some progress, um and so we can kind of talk a little bit about that.

C

On the project board,.

A

uh Cool, so I think the only active, uh no okay.

C

There we go do a little uh housekeeping here um yeah, so I think our our current project boards is uh pretty accurate. Absolutely readers.

A

Is this done uh aaron? I think.

D

So um I can't remember.

C

D

Was something that needed to be from the uh be taken from this.

C

No, I think I think this is done um this video doesn't, let us see it, but let me see if I can find it.

D

Yeah, the the pull request.

C

D

C

A

C

ah Yes, I've noticed this a few times like those directives to actually close things are not um closing things. uh I have seen that as well. Yeah, I don't know. What's going on internally, I've been trying to cycle my verbiage to find out if there's one that works, but I don't think there is um okay.

D

I think the same thing might be uh with uh 2947.

C

A

Okay, let me just close it this. This should be pretty obvious. Yeah.

A

C

And then the internal view state structure. I think we talked about this one last time. uh Wasn't the.

D

C

To break this apart at some point.

D

I think the goal was to uh a lot of this, a lot of the work that would be done, or that was done by the vue state package and the example is probably going to be covered by the aggregator or aggregators right or h-g-g-t-o-r.

C

D

We're not married to that name.

C

By the way, yeah I'm happy to change that.

D

I I get it not wanting to conflict with aggregator or aggregation.

C

D

Honestly, you could probably just put it in the internal package and it might oh, it is.

C

Work, it is an internal. No! Oh.

D

C

Just bring it up one level yeah. I was thinking that as well. um Yeah, there's still some refactoring stuff here to be done. I think uh but yeah I agree. That's the naming in the structure on there. I don't know the exact answer and.

E

Honestly, when.

C

It's internal. I care very little uh just as long as that functionality is there and it's it's like developers, can't figure it out, um but yeah we can. I think I think that's a good thing. Let's take another look at where we come up with that stuff, but um as for this, I think you're right, like this uh like exactly like this aggregator uh interface, I think, is gonna. uh Take over a lot of the view state, which I think is good. I think that is more scope to what the aggregator functionality should encompass.

C

um I do wonder about this pipeline concept, though uh we probably need to split this off to its own uh issue, uh because I have been thinking about this like some sort of yeah, like exactly almost what you have here. This looks.

C

F

C

This is, I think you wrote this. Oh.

A

C

A

C

Ago, yeah, okay, um because I was thinking about that- we were talking a little bit uh for those- I guess maybe not on the conversation this ag door package um is, is some sort of way to uh do aggregations for measurements uh that are, let's just take a look at it, um aggregation for measurements that are scoped by their attributes and but you still have to um handle temporality right like if you have some sort of cumulative state like you need to merge those aggregations.

C

That's gonna return uh back, and I think you also need to have this concept of like um pipelining, which this is starting to get to the place where I'm not really familiar with what the implementation was, but it just seemed like there's some sort of like how do you connect all the way from the reader through the meter provider, to the meter to the the instruments themselves and understand that, like that direct like that connection um and where that state is maintained, so I think that's gonna be separate to this, um and that may still need to come from that one issue, uh but maybe aaron.

C

I could just uh you know if you have time to take a look and maybe yeah.

D

Rename that, I think, was the goal.

C

Or something like that and um yeah.

D

So um my thoughts have changed on this, since I've first put that out honestly um yeah.

D

If we can keep details of whether or not it's cumulative, whether or not it's um brain, fart uh yeah, whether or not it's cumulative or or monotonic- or you know the details of the different implementations of the aggregators contained within the aggregation path package.

D

Really, what I see us doing is having two separate ways of indexing. um Those collection of aggregators uh one is stored in the instrument and it just says I'm an instrument. When I was created, I was created with these n aggregators. Every time I update, I update all of them. I don't care.

D

um The aggregator can be the one that does any of the filtering of day of uh aggregate of uh metro culture.

A

Of attribute sounds right.

D

Yeah yeah um like that, could be details that are at the history level at the at the aggregation level, um because that's where, where it ultimately is stored, the other way you index those aggregators is by reader and then um given.

D

It also has to be tagged at least somewhat way with the instrument name and that's for when you're doing the uh produce from the producer, um because you produce from one reader and then that one breeder will have um essentially another list of aggregators, and each of those aggregators will have a name associated with it um like a name and a description in the unit, because that's what's needed to fill out the uh the uh export data, the export data, but those actually um potentially could like there could be two instruments that create the same.

D

That create uh because of the views, create two different aggregations uh of the same name. Those is an error that isn't considered an error state right. We don't we're not supposed to allow that. That's when uh the view conflicts, but if we put them both in the data pipeline.

D

That is okay like that is not um something that we're supposed to like. That is not explicitly forbidden and it simplifies quite a bit when you do that.

D

That would happen if.

C

You have, we would export two of the same instrument for different aggregation types, yeah.

D

C

And that's allowed.

D

There's no there's nothing that stops us from doing that, but, um and that also preserves the data which is one of the uh the he read my key one of my key readings from the spec, but we're supposed to um present an error when that's registered when we detect that that would happen at the uh instrument. Creation.

C

Oh okay, maybe I gotta go read this again, so we're supposed to present an error, but it doesn't mean that we aren't supposed to still collect both aggregations.

A

C

Oh okay, I didn't understand that.

A

C

Okay, yeah, that's interesting.

D

And, and that also can simplify our export pipeline in that we read through a list of um yeah. The idea is to pass through conflict and data and report the conflict, um and the idea is we are just reading through a list of aggregators.

D

um Ideally, if, if there were no errors reported, they will all be uniquely tagged with a different uh name and description and scope. um But if there are conflicts, then you get two of them. One will be. You know a sum type. One will be a histogram type and it's on the uh the back end to figure that out.

C

Okay, uh yeah, I saved your point and so your idea is that that the instrument itself would store both aggregators and update both of those yeah yeah I mean um yeah. I think that's. That sounds good, so I think that also is still just speaking of the one side right, because we still have the other side of the like you're saying the produce, so the collection of those aggregators and so.

D

That would be part of the the collection yeah.

C

Yeah so one of my questions there is that, like, I think, that's where the temporality should really be sitting right at that level, because um if you have it at the aggregator, I guess you could have it at the aggregator. I mean.

E

C

To be some sort of separation right and like at some level, you need to have some sort of like cycles right and the cycles need to get merged, um and so, whether that sits, you know in the thing that we call an aggregator or it sits outside of them. I I don't know, but wherever that is, it needs to maintain state as well right from one cycle to the next um yeah.

D

C

D

C

D

Have the delta sum and uh continuous sum, and you only need to create one of them when you create the instrument, and you will know all of the information you need at that point right, you can query the reader for what temporality it should be. You can query the view for what the name should be and what the instrument type or the aggregation type should be. You collect all that data and you create the new sum and the difference is.

D

If you have a delta sum every time you collect it, resets the count of whatever it was and uh for every value and the do you have a uh continuous um you. Don't reset right.

C

Yeah I so I I was tooling around with that ag tour package was called something different at the time, but, like I created another uh concept called a cycler and that cycler could either be monotonic or it could be or um sorry, cumulative or delta and the uh the delta one just returned.

C

Whatever aggregation uh returned from the aggregator and then the cumulative one, it maintained the state from the last one, and I think I think if that works, um because it sits outside and essentially just because there's already a merge operation on the aggregation type, that's returned, so it should be able to handle those. One of the things, though, is like um I noticed in like josh's pr like there's a ref count concept, and that is like you know. If you don't see a certain metric after so many iterations like, should you still be exporting?

C

You know empty values for that, and so I don't know where we would keep that. I think in that place. That's maybe at the collection side, uh you know higher up after you get through the cycler.

D

I actually wouldn't recommend that um I'd actually because each aggregator is already unique to a reader um if an instrument isn't updated within you know two or three or however many reference. How many counts of a of a read.

D

You know the the read which causes produce which is cause is the aggregator to aggregate right. um You can do the the ref count there. uh The real difficulty you have is making sure that you're separating um uh you're you're able to do that atomically right quickly, anatomically, which is why the ref count package was used.

D

I have some ideas on how that could be handled, but.

C

C

Yeah, this is um complicated.

A

Building something complicated for them.

C

This one yeah it's it's, but I think we're doing a good job in breaking it out. So, um okay yeah. I think that we're on the the same track, I'd like to get something drawn up uh for some of these levels and structures, uh but uh I haven't yet so look um overwhelmed the switch more to come.

C

Yeah, uh I'm sharing the right screen, yep, okay, cool! um I think with that. Then this looks pretty accurate to what's going on right, aaron, okay,.

D

Yeah um I've been wanting to get a pr out for um the export data.

D

My first swing at it is just going to be kind of a easier to work with version of the oto otlp data structures, just so that we can fill them out manually, um and I think that would be a good thing uh to unblock anybody who wants to try and take on any of the exporters or yeah or readers like prometheus.

D

um I hope to get that out either into today or tomorrow, um but it it's literally just data types. So it's a lot of documentation.

C

More than anything.

D

C

Really so um can you just create an issue to track that or I guess, if you already have the pr, maybe that's not really worth it, but um something here. I think that uh I just want to ask a question. I've kind of got a strong opinion on well, I don't have a strong opinion. I have an inclination as to what the answer is, but like do we want to use the otlp data as the underlying transport.

C

I've been noodling with this. I've been talking with tigrin as well about it.

C

So I um I wish anthony was here because I'm sure he would say what I'm about to say and I don't think the stability guarantees are in a place where we could use it, um especially if we're going to be exposing it as like the export interface, um but that is supposed to be changing uh relatively soon so, like.

C

I was kind of wondering about that because, like taking the otp like our own proprietary, like format passing it through, the exporter then going to the otlp exporter requires a transform and, like I don't know, all the other ones require transform as well, but like it, it seems like we might be able to optimize that in some way. um But I don't know just wondering if you've thought about this david.

D

C

You have any ideas on this one as well.

D

My only concern with that is.

D

That would also mean that we have to at least get the protobufs and because of how otlp the go otlp is is set up. We also have to bring in grpc across the entire project. If that's the case.

C

Okay, that's a good point. um I forgot that yeah.

D

Other than the services as.

C

D

um Yeah, the other, the other thing is um it it. It makes it harder to evolve it um and- and we might have break like- are still potentials for breaking changes um moving forward.

D

I think, which is why I would rather have just a more flat structure that we've controlled and we've set for it, and we can evolve as best. We can.

D

um The one, the one thing that um I would we might actually want to consider with that is actually having a version of it, um because it might be better to version our export data than it is to.

D

Than it is to have the export data, a set of interfaces sort of like the read-only span is in tr and tracing just because we are going to be marshaling and marshaling a lot more sets of data a lot faster so having actual flat data structures is really advantageous to us.

D

B

D

A trade-off there, uh we maybe consider just versioning it and then what how that, how that would look to evolve is when we are ready to introduce version. Two. We, we add up a version, two of the export data structure and we create another interface, uh another export interface that produces or that yeah that produces the uh or that that accepts the version two of it. And then we can do a type check to see if the type accepts the version two and we use that over the version. One.

C

Yeah and ideally these are going to be struts, so we should be able to extend them pretty easily with fields, uh but uh deprecating is always hard, um but yeah. I'm also reading in the chat josh is pointing out that java optimize means not using protocols entirely, which I think is fair. So maybe we should follow suit. um Okay,.

D

A

Completely off the.

D

Wheel off the off hinge and use flat buffs for everything.

C

uh No we'll build our own wire protocol, um just build it up. Yeah.

D

C

So if you do get an issue open, uh could you well, I guess I'll, find it um and I'll just try to comment this or maybe, if you can comment this decision to not use the otlp. For all these reasons, I think it's it's good to capture, uh because we want to make sure that we thought about it. Yeah, okay, uh cool. I think that's it for the project boards uh and talking about everything going on um there's nothing else on the agenda looks like everyone's signed in um so yeah.

C

I'd like I was saying it's probably in a short meeting, um which is good, because I think that I've.

F

Got a question.

C

F

If we've got time for it,.

B

F

um An issue was really recently brought to my attention for the uh metrics sdk. The customer was trying to use it and setting they were trying to set um what was it async counters, the the counter observer to delta temporality, uh and it would throw an error. um I think they got the same error with up down uh observers.

C

Is this the the old sdk or the existing sdk? The ones are right now.

F

uh I tested with it morning using the 1.7 release and was able to reproduce the issue or well.

E

C

It wouldn't be.

F

F

Yeah yeah the 30s.

C

Yeah so whatever.

E

C

Like a yeah, this sounds like an existing uh bug that I've seen before um and there is definitely like incompatibility with delta temporality in the async. If I remember correctly, I may be wrong, um but I do think this is captured in a few places. Josh is on meeting so.

E

I would not call this a bug. This was a debate in the specifications.

C

E

Kind of boiled over, like I didn't think we should ever have to do this and then new relic said we had to do this. We removed this from go around october last year because I didn't feel it was important enough to keep and I thought we had agreed upon it. We changed our mind around december last year, so um the complexity was removed. I could find you that pr- and I mean I believe, tyler and aaron are scoping- that it's got to be done for the spec.

F

Okay, yeah, that I think all I was coming here to ask was: uh is it captured because it's in the spec um and if it wasn't, I was gonna, ask that we capture it? If it is then cool, and I will continue to uh anticipate the uh stable release which we're all excited for.

C

Yeah, uh don't hold your breath, uh but it's coming uh yeah thanks josh uh and thanks for bringing that up, I you're not the first um yeah um but yeah. I do just pause here as well. If anybody else has any other questions.

E

Are there instructions for how to set up gpg so that I can make a release, I'm having trouble and I couldn't find any instructions.

E

uh Like git tag is failing, and I'm I mean I I should be able to do this, I'm not trying to say somebody else, I'm trying to say looking for help instructions wise.

C

Yeah I I can find um yes, there's they're very generic, like linux uh user form gpg um things that I've used. I also use the ub key now, so it's very different, but um I don't know of anyone. Okay,.

E

I've used gpg rarely enough that I'm, I probably just don't, have it set up properly right now, but um the error messages are completely opaque, it just says failed and I'm like, I don't even know how to debug it, because it's below git, that's failing.

C

E

C

I I know I I had that uh there's like some way to turn on the verbosity and um oh man. I wish I had kept that link yeah.

E

Because I know what you're talking about.

C

Yeah thanks, okay, sorry about that yeah! It's frustrating! um Also josh! If you want, I can do the release for you. um That's still on the table. It's a you know. If it's going to take you like one hour, let me try a little longer. I'm almost there cool sounds good. Just ping me if you change my um okay.

C

C

Awesome, uh well, I think, uh are there any other user stories by chance? Maybe we'll we'll end there.

C

um David, I do have to say: I've been talking with a bunch of splunk customers this week and their excitement of open telemetry in kubernetes and fcd, and all those other places is really um overwhelming at times, and I want to say you get to get a lot of that credit. So yeah, okay,.

B

Fun I I was excited this week because I've been having a little bit of trouble getting the some of the gke teams to actually uh put it and use it and give me feedback. But I did get someone to actually connect all the dots in our on-prem product today, so I'm personally very happy, but it yeah. I I'm just looking forward to getting out of the the pickle we're in with being on o20, um because that gives me nightmares, but I yeah, I don't have any updates. Sorry.

C

Yeah, no, I think you and the rest of the community are excited about that. But, okay.

C

uh Cool, well, I think, with that we could probably end it here uh thanks everyone for joining. If you have any more questions, please be sure to um reach out on slack otherwise, we'll see you on issues or same place same time next week, bye.