Open Telemetry Uncategorized, 12 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-07-12 meeting

Description

Instrumentation: Messaging

A

B

Hey how's it going it's.

A

Going all right, oh still, in still at the beach, so my office is a little bit uh in flux here. What beach are you at uh Clearwater in Florida Florida? Oh.

B

Yeah yeah, uh my sister lives there that's funny small world, not my favorite place, I'm, just gonna say it that way.

C

The beaches are great, but I'm happy I'm just visiting here, yeah.

B

That's the way to say it: yeah, uh the hard part is, is that you can't always just spend 100 of your time on the beach there, but yeah 100, ., um well, cool man, yeah I, just got back from the Oregon coast, so the other ocean. Well I, guess that's the Gulf side. So that's not even the ocean, but yeah I, guess I, don't know it's the gulf. The ocean I feel like I, should know the answer to that one after living there, but.

C

I mean it's a c I suppose yeah.

B

C right, yeah, um yeah, okay, I, don't.

C

B

C

Doesn't matter.

B

Yeah I know right um cool. Let me pull this up really quick, so I want to just kind of like bounce some ideas off of you. One of the things also that uh you know uh people are pinging me on they're, like 71, like you guys, are really close and I'm like oh yeah, we've been making some good progress actually, like I, don't know how.

D

B

That is, but uh it's not 10 and I think that's that's correct. It's not 10, um so I think we're actually pretty close uh kind of like what I said to you yesterday I think there's like three large kind of efforts. Well, maybe two large efforts in like one medium size and the medium size I think would be the the aggregators we've already started on um so I I.

B

Think the other two, though, are gonna, be um it's Recreation, because you need some sort of way for like uh registration of that of that instrument, as it goes down the pipeline and then the other side is um the actual, the actual measurement side so like when, when things get recorded like using the aggregator, but also like uh going back up through the readers, maybe more The Collection side, so how the reader actually pulls through that collection and I, don't know if those two are distinct.

B

The the instrument, creation or the collection side, um but I wanted to make sure we have that laid out is, is what I was hoping to do so. The way I do.

C

It um one of those two should be pretty trivial and I think the collection part is actually pretty trivial, uh depending on how we structure, instrument, creation and I've, actually putting together a PR to kind of demonstrate it, um for uh it should should get to the point of being able to demonstrate it. There is one hang up that I found in in all of this, but the idea there is, when you create an instrument it needs to register in essentially two different places.

C

um It needs to go. It needs to go and query all the views and and I ascertain like what are the aggregations that are going to underlie this instrument, and it needs to put those aggregations in the appropriate collection path and.

B

Then it also needs.

C

To hold on a a copy of those right and not not a copy, but a reference to those so that it can update them right when somebody actually does the ad right or the record um and um I was working this morning. This last couple days on how does the registration part work um and I'm I'm like really really really close on?

C

You know just the the proof of concept, the the first pass of this essentially um I'm trying to debate where to like kind of abstract a little bit, because there is some some ugly code in there in that you have like three different ways. Defaults can be applied right.

C

The uh the views question that we were going through in the spec right, um so there's some of that that happens there, but then, if you set it up properly, what you end up doing is when you need to collect you just go uh through your your reader.

C

Your reader goes, um I need to collect, and it you know, creates the res the scope metric with or the uh resource metric with, the the resource attached to it, and it goes for every view I have, but uh for every uh not for every view, but for every um scope that exists from the views that I had but has anything registered into it. Go create a scope metric right. Every scope metric will have some number of instruments in it create a actual metric.

C

um There are the whatever The Next Level under scope. Metric is I. Think it's metric yeah.

B

C

B

C

And then each of those metrics just says from the aggregator give me all of the the data points is what they call it in the export path, but give.

B

Me, the aggregators and.

C

Those aggregators can be converted into the export type right, so that is just a straight like four four four, four.

B

Yeah, no, that that makes sense- and that's essentially that producer API, that we already built and I think that that makes sense. It's just that registration, I, think is kind of going to be the the messy part and I.

B

Don't think that we have that entirely captured in the issues, and so my goal here is to just make sure that, like even if we don't know what we want to do, uh having an issue tracking somebody to go figure, it out, I think it's useful here, yeah uh and so I wanted to kind of, like I I, think we're on the same path as to like uh by by and large like what it could look like. um So what's your opinion here.

B

So um this is all aggregation stuff, so yeah there's this pipeline structure um and this is structure and then there's an implementation uh here right so kind of similar what we did with the aggregation where you define an API more than you build that API. There's. Also these uh two tickets here for implement the stubbed uh sync instrument and the sub async instrument. Lucky I've gone through a few times trying to like work out a POC on both of these, and it comes back to that registration question like getting something.

B

That's really like nothing uh for the instruments. It's not that hard, it's just how they get registered and like like you're saying like it, sounds like they need to get registered in multiple places. It's kind of like the key thing here. So do you think that we have these four issues, kind of encapsulate all of that work, or should we add more or adjust them.

C

um The only thing I would say is like we called that pipeline.

C

The the main pipeline early on um I, don't know if that's still an appropriate name for it, but other than that, like yeah I, think that um that would really kind of be the the work that I'm I'm working on like what I've been working on to this point um and what and then uh because of this work because of what I've been doing, I need to make some kind of adjustment to the aggregation like the internal aggregation layer, just because foreign one of them is because we need to to um one of them is just that.

C

It will simplify things once if we, if the aggregations can actually just export or just produce the the actual export token that we need. So we don't have to do, fill it in the aggregation layer and then convert it immediately over to react. The export.

B

C

You talking about something like this. This.

B

Pr I've been taking a look at this I.

C

B

So this VR just switches from this. It removes this internal aggregation, type and switches it to exporting the export aggregation type.

C

um There is a interface type in the export path that um encapsulate something.

C

B

Yeah, so it's exporting um uh this type right here.

C

Okay, that wouldn't one aggregation produce. One of these, wouldn't one aggregate tour produced one of those just one, not a slice of those.

B

uh uh I guess: what's data point attributes yeah you're right? It would do that. Okay,.

C

So that's that's, essentially the change that I would need um the uh I'll put a comment in there. um The other thing um that I think hasn't been really fleshed out is: where do we do the attribute filtering.

C

B

I I thought it would be at the instrument. um That's.

D

Because it would have the context.

B

And we haven't built it into the so essentially like when you like, you're saying, create the view is allowed I'm. Sorry, when you create the instrument, it knows of all the views that are going to apply to it, so it would already have some sort of mechanism to apply a filter. At that point,.

C

So uh the one the one argument I'd have against that is that filtering is actually a pretty heavy cost operation. It's not it's not super cheap um and we do yeah. We do a lot more.

C

um The assumption is, we do a lot more ads than we do aggregation, so if we could defer it to aggregation time, that would be ideal, but this is something that I'm like uh we could. We could do that. We could wrap the aggregation and uh something that filters that and be okay.

B

So I don't care at this point. I think I'm going to care more at the beta phase or maybe, as we have a ga, we want to validate that we're doing it at the right spot. So yeah I mean if you find that we should do it in one versus the other. That sounds cool I totally saw aggregators like their creation function. You can change those. So if you wanted to change this to, you know, receive a filter and then they would use that filter when they.

B

You know, the aggregations method is called like that: I I, don't care honestly yeah um as long as the public API is, um is somewhat usable and then we have an issue tracking it. So, like you know, if, if we have disagreement, we I are even different ideas, not necessarily just agreement. We should explore it I think in the in the post phase, so whatever you want to do there. That sounds good to me.

C

I started off with assuming that that is that we were going to have to update it for that, but I I'm at your point of like. Let's just do it at one way or the other and just get it over with.

B

Yeah and I think it honestly like doing it. The wrong way is not necessarily like it might be helpful just to show uh how bad it is or how little it can impact us, because it would prioritize I. Think in the next phase is like what we should be focusing our efforts on, um but definitely like. If there's obvious things that we should be doing like um yeah yeah. That sounds good to me too, and mines I I just want something tracking it.

B

Maybe that's the way to say this, um so we already have uh well, it's kind of we can use. um We have a uh another milestone for this and I already started.

D

Oh there's a beta.

B

B

C

B

Okay, cool I uh and that honestly, you reference this um and do whatever whatever is easy right, and you can just say what to do we'll look at this later, uh not many of those okay.

D

Okay, cool yeah; it's a good one to track um so.

C

I was going to put together something so that we so that we can kind of discuss.

C

um The problem is I, think I've gotten a little bit ahead of where the pipeline is just because honestly, it was. It is complex enough that I'm, like I, can't see it just at the abstract level like I. Just cannot understand it at that level.

C

um I can try, and would you rather see this as uh probably something that would cover uh 2834 the Implement and two eight one, four, the implement the sync instruments that I know it's a little bit complicated, convoluted, but I think it's probably a digestible enough patch set that it could be. You can kind of see what's going on here.

B

Well, I, to be honest, I think foreign I haven't seen your patch, it's really hard to like kind of speak to that. But the thing that I I feel- and this is just based on me- trying to do these sort of like proof of Concepts as well as it's going to be a really complex PR, and that's because it's a complex piece of code and I'd. Rather we spend a little bit of time. Thinking about it and and having conversation maybe asynchronously, and that was where I was thinking in this 2833 uh issue.

B

We could put forth like big picture ideas uh more about that structure and like how we want to like set that up and that could be like you know. Just boxes like the media provider creates a meter, creates an instrument and here's the registration path, like it looks backwards through through the media provider through the reader even and saying something like that, I think is, is going to be useful.

B

So we get down to the Bare Bones, because that's that's the key art key thing like I want to make sure that we have something: that's Bare Bones, because I I have found that we've we've started to try to like limit our PR's in this.

B

To you know, I've been seeing a lot of like 300 or less and they're they're possible, like yeah I, can review that it's not immediate, but it's also like their cohesive ideas and like we can kind of like work for it and so I like doing that, and it helps I think do something similar to this, where we're just kind of structuring it. So maybe maybe that's the key is just in uh 2833.

B

We can get like a little bit of an overview um because I think things like this, where we got some sort of API for the aggregator and then we were able to break it apart and do you know it tells the histograms keyboard history of its last value, all these kinds of things how we can break that apart is going to be really key.

B

Having an end to end I think is is useful as a demonstration but having the API is also just as useful and having a well documented API, so that people can then build it up. I think is, is really useful for parallelizing it I am getting questions from jarese over at grafana Labs people internal to Splunk, they're wondering how they can help uh and I'm pointing them to the to do column um it's getting smaller, which is good, but it's also like if you can break this apart into you, know five or six different issues.

B

Once we have a direction. I think it'd be really helpful.

C

I would also I I believe at this point. uh We can actually unblock any of the exporters. uh Just general I.

B

Thought about that. The problem there is that they expect they wanted to test. Values uh is what they originally were doing, um so we can add export it back, but they can't test that it's producing the correct metric values is the problem.

C

They can well I mean they can't test like in a full harness situation, but they can test.

B

C

The either export or the uh produce call right.

B

B

Like the producer will give them like, then mock out the produce, and then they can uh essentially say like if, if I get this sum, then I should expect of this or something like that. Is that what you're saying uh yeah like they yeah? Okay, all right, I, hear you.

C

Okay, yeah, that's a good point. You know they they can test from the reader on, which is what we intended them to do.

B

Why are these the same? Sorry.

B

uh Prometheus, oh example, code and Export.

C

Code, yeah there's a.

B

A

C

B

C

I think the example might be a little bit harder. The example I agree, but the the exporter could easily be created.

B

Okay, yeah I feel like yeah. We could just add that back in um so I'm gonna do this here and then I'm gonna move this.

C

Here and then 807 that one right there yeah okay I, don't know about the bridge code I'm, not as.

A

Familiar with the.

C

Open sub to bridge.

B

I, don't think this is possible yet I, because I think you need to actually get instruments.

D

B

Down down through that, so we need. We need to have stubbed instruments.

C

uh I, don't know no, we don't. No. The providers just returned no right. Now, no.

B

You're right, yeah, yeah, okay, yeah, so I think we need that still uh yeah I think I. Think those three get out of backhand. That's a good point.

C

Do we have anything that covers callbacks because I know that's part of the async, but there there hasn't been any.

B

C

B

Of thought here um in this uh two eight one: five, because you need to handle the registration of those callbacks with the meter at that point, to implement the basic instruments, but uh I, don't have anything specifically lined up. I could definitely use some help if you want to help create issues for work, needs on that.

C

Yeah I think probably just a single issue. Well, I I figure it'd be a single issue that follows in the same Footprints as the other pipeline stuff, but just specifically tracking callbacks, as that is something that is distinct entwined but distinct from the pipeline.

B

Yeah and I think that kind of goes down, like my original questions like I, think that they're I don't know if you can do callbacks just yet. You need some sort of like uh minimal like set up and I'm wondering if we could get that minimal setup defined. So we can, we can add it so then we can break out the callbacks and break out the other implementation of the sync stuff is kind of where I'm at but I'll be honest, I'm focusing a lot more on the aggregate side uh right now.

B

I still wanted to pick up. Probably this Delta histogram next um so I'm wondering. Could you help me in in helping Define what that.

C

B

Is and then so that the issue yeah cool, that's.

C

That's where I was that's where I am focusing like I have: okay, yeah I probably have a 400 line; PR, maybe 500 line um total that that has it's not the minimal it's beyond the minimal, but I'm gonna I'll pull it apart to make it something so that um we can actually have a discussion on how these are structured and how the different components put together.

C

That makes sense. Okay,.

B

um That's going to address part of 203 is kind of the goal there yeah.

C

Okay and give instruction on how the sub sync, uh the sub instruments would be created, essentially.

B

Yeah, okay, that sounds good. um Can we restructure 2833 to scope it specifically to what the pr you're going to add is and then we can also add new issues to consume the missing parts of the pipeline or whatever you're calling the pipeline.

C

I I have a name for it. It's it's literally just the um the collection of aggregators right now so.

B

C

B

C

Have a pipeline or collection.

B

uh Process, yeah, whatever you want to follow it I think that'd be ideal, um I think it's just. It would be really helpful to break that apart because it is I think more work than just a single effort, and so it would help make that a little bit more visible as to like, what's going on there and communicate across to the rest of the step.

C

B

We're doing does that make sense.

A

Yeah, okay, Alpha.

A

How do I suppose.

B

The latter half of this week so I won't.

A

B

Help too much later on, like Thursday Friday I'm gone.

B

C

How do I see it in that that view that.

B

You just had um here: can you share your screen? I'm pretty sure it's just a I'm sure, let's.

D

See share screen.

C

C

No I can't apparently okay.

B

uh um So there's this all work tab here and then there's like this board.

A

B

Up at the top and then there's like new views.

C

And so project board. Okay, there we go.

B

C

Yeah I see the all worked up and I'm, like that's that's great. If I'm making a Gantt chart.

B

Yeah I I, don't know how useful it is. It was useful to start out the project because it just really quickly write tasks but I keep on. Actually. How do you make this like the default.

B

Oh, you can, okay has moved it over.

B

B

I just saw they added this chart, functionality, which is cool, maybe I've, just like nerded out on this, but I was like oh wow, we're actually making progress, but a lot of that progress is just like if I'm reading this right, it's it's uh adding issues as well as fast as we are uh resolving them.

C

So yes, but I mean the idea here is as long as the um purple. The average purple line is greater than the average orange line or yeah the the cfd, which.

B

Yeah at least they're they're trending yeah wait a minute. Oh, this is stacked yeah. This is stacked. Oh, this is yeah. Okay, I, gotcha, okay, good.

A

B

Yeah I think I think that's a good good display, a showy, steady progress, I think if that's useful.

C

What what is this configure.

A

A

C

Okay, well, that's the normal burnout, but on the actual.

C

C

Finding new stuff every day.

B

I know I I'm all about it.

C

I was like: how do we get a burn down chart.

B

There you go right.

C

That's how I'm used to to seeing things.

B

Yeah I I, normally just look at the swim, Lanes uh I'm, just how big the numbers are.

C

I wonder if we could uh exclude done?

C

Oh because if you exclude done you get a burn down, chart essentially right yeah, and then you can very easily see that um the uh which we'll call it the oh.

B

C

How about blocking to.

B

Do or or is going to zero.

C

C

C

I don't know I'd have to play around with their filter thing, yeah filter by status.

C

Group status: oh there, you go.

C

um Yeah exclude status done.

C

So in the last three weeks, three months, we've added a couple: we've added a few, so the thing has gone up, but oh, no, that won't be a burn down chart because it's it doesn't show any of the ones that are done are they're not done are not included on this well dang.

C

B

Yeah, um okay,.

C

Yeah I will I will recapture kind of where I'm going with this and put in my a minimal PR like not one that actually has all of this included with the concepts of what we need to include I might turn.

C

So you know just so that it's said what I've done, like I've, actually done.

C

I think 90 of the work here um in the there I've added not to the meter provider but like a free phone uh free form, function that uses a meter provider to you know iterate through the readers figure out which views um the instrument that you're creating applies or you know, matches and then create an aggregation or an aggregator for each one of those views and then return that kind of list, including the funky error, checking I can kind of, um but and then for the production side.

C

um I've created a provider that basically or and then when you do that it also at the same time registers just in a pool of uh aggregators for the uh for the reader for for the provider portion of the reader and the provider. Basically, just um uh when you, when you, when you call Produce on the provider, it will just iterate through those and fill out the the.

C

And fill out the the corresponding metrics uh at each level. Right, if you want to show that oh no I, I can't without restarting.

B

C

B

Similar I think when I was doing a proof of concept. I had something similar. It I specifically created a new type called like a pipeline that would.

D

B

Exactly so when um you know the uh produce method was called on, the meter provider it reduced or it returns a specific type to attract. Everything is like you needed to keep reference to all the meters which wasn't that hard, because we had that meter registry, um so it just kept a copy of that beta registry or a reference to the meter registry. Technically, not a copy.

D

B

Every producer needs to go through all the meters, and now the meters need to that's also the thing: that's that Dynamic nature right, because you don't necessarily know ad hoc from the producer like it needs to figure it out every single time the produce is called.

C

But it doesn't it, but actually it doesn't need to go through. Every single meter is the the thing that I'm I'm, seeing because.

C

um Technically, because you actually want to filter it by view right and some uh sure I guess when an instrument like doesn't matter like, if you only have one view that only matches a particular meter, even if you have other meters, you don't need to iterate through them. So what.

B

I I mean I. Think that's like the the odd ball out right. Like that's the one case like if you have no views like which is the default, you need to go through all the meters. If you have a view that doesn't restrict based on scope, then you need to go through all the meters right. So it's.

C

B

Need to iterate over all the meters.

C

Well, what the way I ended up doing it is um in each producer. It just has the um the instrumentation library or the instrumentation scope as a key in the map, and that way, as you add more instruments it can, it can fill out um the lower instances of the math in in that sense here uh let me show let me just kind of.

B

Well, yeah I'm, not so I yeah, okay,.

C

So um here is basically where we're scoping, um because so essentially what you, what you do when you, when you do provide, is you iterate through that you iterate through your instruments and you pull out the aggregators for each one of those instruments right and if you look the the code for that, is you know a four a four and then you get the aggregation and it's literally just filling in the data.

C

And you return the resource.

B

C

B

Okay, I mean yeah, but now you have to store all of the instruments here and in the meter. You have to store all the instruments there. It just seemed like I was going to have the meters, store the instruments and then ask the meters to go through each one of them.

C

I I I guess I did this as an optimization, because this also prevents me from having to go and like when we get a scope that doesn't have any instruments in it like either. Nobody added anything or all right like there was a meter created, but no instruments were added or there was instruments added, but they were dropped by The View. uh We don't have to request all of that and then also figure out to drop it as well, whereas this is just a one shot.

C

If if there was an instrument created it's in this, it's the scope also gets created and the instruments are created.

B

Well, I still don't see the difference like the meter itself could do exactly the same thing. You're still just throwing it in two different spots. So now the provider has a list of all the Scopes and the meters are each a instance of a scope.

B

C

B

C

B

I I'd, rather not discuss.

C

B

I haven't looked for code and I'd, rather if we could just making sure that we have all of the issues kind of up to date and a little better structured, so we can communicate our progress as well as, like the active work. That's going on um I'm happy to look over this, but I I, maybe just not in a in a a meeting for a call like.

D

B

It's hard to it's hard to comment on something I haven't looked through. True understand,.

B

Okay, uh cool, okay, so kind of the takeaway from this is uh 2833 like you're gonna go through that and and update the ticket to reflect what its work is and then maybe break it out is the goal yeah and then you said that you were also going to create a shoe to track the Callback work as well or or maybe somehow.

C

B

Two eight one: five, the async instruments: okay,.

C

I I will make sure that if it's, if it doesn't already cover Ed callbacks, um which it doesn't okay, yeah I'll, add a call back um just to make sure that the Callback is explicitly captured there, because right now, there's nothing doing with the callbacks.

B

Okay, perfect I think other than that. Our project board is looking pretty reflective of the work. We still have to do to get this outfit release out right: yeah, okay, okay, perfect um yeah, thanks for joining me, I I'm, getting pinged, left and right to go touch base on other things now, um uh but yeah thanks so much for finding the time. Hopefully, the vacation becomes more of a vacation still.

C

I mean I've already been here, what 12 days now and uh the first, the first time we're actual vacation and the kids are like exhausted at this point, they're all just like a giant Heap on the couch right now.

B

Oh nice I always feel like the first seven days are just decompressing in the next. Seven are the ones that actually are a vacation for me. Sometimes, but yeah yeah happened.

C

B

Yeah, that's always the goals yeah. That's it.