Open Telemetry Uncategorized, 16 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-02-16 meeting

Description

Instrumentation: Messaging

A

B

Hey how's it going.

A

Going all right, nice I see, we got some big changes on the way.

A

uh For jump and jump it into the API a little too early, but oh.

B

Oh yeah, well, yeah, I, think, there's I think there's a good reason there, like um it's funny like I, don't know it's like big changes, but it's also like not at the same time like like I, don't know it's just like it's just it's like you change the return value to not return an error on like the creation methods and then there's 800 lines of changes all of a sudden you're, just like wow. That was okay.

A

I I will say um if those constraints were on us, the the first time we went through this, that's probably how I would have shaped it to begin with.

B

Without the error being returned yeah, where did they come from I'm trying to like figure that out.

A

They come from their um if you have.

B

Like is this like back when the API was supposed to validate names and then, like, yes, maybe validate names, okay,.

A

But both validate names and validate, if you've ever created an instrument that would have conflicted essentially.

B

Oh right, yeah, that's right and I completely moved that to the SDK at this point, so yeah I forgot that was also another thing. I moved, yeah, okay and like I know that there was like there's. Nothing like it literally says like the API should not be doing any validation So like um that's what motivated it but I couldn't remember where the original ones yeah.

A

And if there is no error there, there's no way for you to get any kind of fast feedback that you had you've done. That right, like the only way to get that is asynchronously through the error, Handler or some other way. Well,.

B

Yeah and but it's also like um it's going to be implementation specific too right, because that was the other thing that I was thinking about. It's like okay, so like this is the big problem of information flow like the information flow and it flows back to the instrumentation is always going to be a problem because, like the instrumentation like, how does it handle it like I, like like I, don't know, like so say it uses like the the light step SDK and the light step.

B

Sdk is like 63 characters for a name nah, a dog like: let's do a thousand right like and so like is. Is it really an error that it has to handle at that point that, like so I, was like it's totally dependent on what the implementation is and like and then like the thing is like okay, once you actually have the implementation, give you an error like what do you do with it right like?

B

Do you panic? Do you Vlog it? Do you send it to the error Handler like so it's like there's like all these different ways that you could handle it. None of them are resolve it like none of them are actually going to be like resolve. The instrument name error like go like okay cool, like this is giving me an error, and it tells me that, like this can only be 63 characters. It's like I'm, gonna, add runtime fix the instrument name, it's like that's, never gonna happen right.

B

So it's like yeah I was like oh, this doesn't make any sense actually, but.

A

I I agreed I I thought that was silly, that the get-go but yeah.

B

I know like the more I.

A

Felt like thought.

B

About it, I was just like yeah. This is actually I think the right way to go, but it's still a draft also so I wanted to discuss it today. Yeah, hey everyone.

C

B

Hey David, what happened to Castle Island I was expecting greatness from you. What happened to it? Yeah.

A

C

That's booked okay, yeah, oh no I, just I joined for my laptop, not from the.

B

Oh, are you in Castle Island still.

B

D

There, oh look foreign.

B

I think that's the official uh continuity. It's like um like the head of a state. It's like the state is separate from the actual representative. From now on, Castle Island would just be like the Google representative for the go seg meetings.

B

uh Cool I guess we're like three minutes in, we could probably start um I think we actually have some Quorum here.

B

Let me add my name and then I'll figure out how to share a screen. uh Yeah, if you haven't already add your name to the attendees list, um I've got some items on the agenda just uh from what I've been working on, but if you have more things you want to talk about, please add them there. uh One thing, I didn't add Aaron. If you could add that is in the process. Yeah.

A

Yeah all right gotta go find it yeah.

D

B

There we go um Okay cool, so welcome. Everyone uh just started off. I wanted to touch base on the TC review, so we opened this issue uh in the community, and Riley was able to go through the review um and say you know, everything looks completed, uh looks compliant uh Josh added that he was also able to use the new API. So it looks good. There were two minor things that were noted here, uh just kind of walk through them. One is that uh the attributes.

B

This is something that is um especially with the new specification like it's been clarified uh for the scope, attributes that, like all these parameters, weren't actually like, they were being misinterpreted, I think in the old way the specification written, but it's been updated how the specification was written and the way that it was intended originally, is that we are required to accept these attributes in the API um and so I think that that's something that we need to fix prior to the release here and honestly.

B

I think we need to fix it in the tracing API as well, but I, just like focusing on the metrics one right now, um so I think that's kind of an issue. If you want because like this, we have an open issue here. Talking about uh yes talking about this issue, but there's like those have been on the I think everyone here has like when we tried to do this.

B

The first time like there's this issue with like resolution, so like what happens when there's conflicts, because if these aren't identifying attributes and then we go to like you know, somebody gives you duplicate attributes. Do you use the first ones to use the last ones?

B

Do you merge them like there's, like this whole question of like what happens when there's conflict because they're not identifying for the scope, um so that being said, I ended up just before this meeting put up like this proposal of it's not complete, obviously, but like uh we could just accept them in the API. uh Accept these attributes with some sort of like instrumentation attributes.

B

I don't know if I want to include the name institution here, but anyways like this is kind of a proposal here to include these an API there's, nothing that says I think we have to do anything with them. It does say that we have to accept them uh as a requirement. So like wondering what people's opinion on this one is.

A

I want just a little bit of clarification. These are supposed to be set on the scope or on all the instruments that are created from this.

B

The scope yeah, so this is a this- is the same thing that uh I guess we removed it. Yeah it's on the scope, uh scope attributes were added back in 113 of the specification, and this is on the meter. This is a pass to the the meter as an option, um so this will be set at the scope level.

E

Same thing theoretically plays to Tracer, though, as well. Yeah.

B

It does, uh in fact I like that was the other thing that I was looking at was like, um given we have clarification on how this is actually defined in the specification. We're probably not compliant, not probably we're, not we're not complying in the trace API either um this one's going to be I think a little bit yeah. Maybe kind of go talk about this for a second, because uh not this, um so the problem here is that we could add another option that should be backwards.

B

Compatible, um I, don't know where they're uh I don't know where they're at but like the problem is, though this structure here is comparable as it currently exists.

B

So if we add a slice of attributes, it will not be comparable at that point. um But again, like doing something with this um attribute set is not something we technically need to do in the SDK yet, but we do need to add some option here with that in mind: um I added these attributes as a set to try to do the same thing here uh so that these you know, they'll be stored in some sort of deduplicated fashion, but they're also like.

B

If we do this over here, we could still keep this structure as comparable, so we don't have a backwards.

B

Incompatible change, that's going to be introduced, uh so that was kind of my idea on that one uh I also don't know if we want to include make this comparable, so I was like well, let's just like enforce this- to be not comparable for right now, because I don't know if we want it in the future, fix that fix something specific to this um I don't know if we can, if we're gonna need parity with the tracing but just kind of a heads up. That's why this was kind of defined. This way.

B

um Okay got a thumbs up from Aaron. I can probably clean this up after the uh meeting. If we could actually include this I didn't know what are people's opinions on, including fixes to the tracing API in this PR as well, or should I just do the metrics and then do the traces.

E

I would do them separately. Okay, I.

B

Will I'll take that on time?

B

I really wish. We had some sort of resolution of what to do with these attributes once the past, but I, don't like think it should block the API and I don't want to have the API release being non-compliant, so that was kind of my idea on this one um okay.

B

So that was one thing and then the other thing was just a minor rename for documentation for uh currently the actually I think it's I don't know if it's merged it I might have merged it. No I, don't think it's Merchant, but just rename the um in the meter API uh yeah here right now it says instrumentation name because that's what it is, but in the tracing uh API this is the parameter name is name and then in the specification it's also named name. So there's a PR to fix that as well.

B

It's pretty minor, so I mean it's backwards, compatible, it's just a doc change so, um but other than that, like we have signed off on the TC. So I think that that's uh we're getting close I guess is the way to say that uh then yeah yeah, here's the rename. Oh, this isn't the right PR, but that's a whole other thing, um then I think I'm gonna throw um a wrench into that hole because I wanted to ask this question of uh currently right now.

B

There was a proposal uh made by bogsen afterwards to have this, like meter, must thing, maybe in contrib or in a helper, uh so that the meter error is just panicked. uh If some sort of uh instrument creation doesn't um succeed, like there's a non, no error here.

B

The thing that is, though, that the specification we we just had, this PR emerged uh Upstream by the way so yeah, that's probably something to talk about as well, but that's a tangent, but um this essentially it removes all validation from the API. So there's there's no place in the metrics API that should be validating the no op doesn't validate. So there actually isn't a need for our meter uh during its creation to return an error and we're Aaron and I were kind of talking about this.

B

When we first opened up the um meeting, uh This original error being returned was really coming from this idea that the API needed to Signal back to the instrumentation when um inflicting instrument conflicts. So you try to create an instrument, the same name but with different attributes, or uh you tried to create something that was a conflicting description of the instrument is passed back to the instrumentation. The thing is like we weren't actually doing that in the API, so that was it doesn't matter. It was removed like it's not done in the API anymore name.

B

Validation is not done. No validation of any parameters are done in the API at this point, and then there's like this more uh I think holistic problem of. If you have this creation method, returning an error, this error is going to be returned back to the instrumentation and the instrumentation I like there's. It can use it for testing, but then it really is going to depend on the implementation so like it could test against the default SDK and then it could see like okay. This should work.

B

You know with a valid implementation of the SDK, but if you have another implementation like it could totally have different Behavior right, so that error being returned is only as valuable as the SDK and it how that error is handled in like a runtime situation, so say something for, like uh you know, um HTTP metrics or the runtime metrics, or the process metrics like you know.

B

What do you do with that error when some other user has loaded that package at that point, like there's not a real like you, there's a there's, not a good solution, but there's also many solutions. Right, like you can log it with the logging uh framework. You could log it with a global logger from otel. You could send it to the error. Handler you could Panic, you could drop it like.

B

There's all these different things that you could do so there's going to be a non-uniformity across how instrumentation handles this, and in fact we could you know we could try it really hard to say like well any error from the instrumentation. She should do something with it or we could just do that with it in our implementation and I. Think that's what this is. I know.

B

That's what this proposal here um is trying to recommend is that uh what we do is we take this meter in the API and since it no longer is uh required to actually return anything?

B

Let me see uh this I mean this is like the bulk of the change. It just removes all the errors that are going to be done through the creation and the registration methods so included with all the instrument creation. The register callback doesn't return an error. The unregistered doesn't return to error, because these are all errors that would be passed back to the instrumentation.

B

The interpretation could still pass errors through to the SDK, like that's still something like with a callback or something like that, so that that direction of information travel is still possible, but the information travel going back to the instrumentation is removed.

B

By doing this and uh throughout the SDK, we just update it so that any error that would encounter in any one of these creation methods so for instrument, conflict, name, name, non-conformity which I don't know if we're actually doing right now um and registration errors like those all just go to the air handler as a default, um and so it unifies that into a single source for any operator.

B

So the operator that has already pulled in all the instrumentation has a single place that they know to look for errors at that point, um and so I've got this in a draft State just as a proposal. It's pretty big uh but to be fair. Most of it is just this: we're cleaning up signatures that used to take two parameters, and then you know, uh dropping the error parameter and then using the using some sort of global error Handler and like testing to to verify that no errors have actually been produced. So there's.

D

A little bit of.

B

Like uh you know, an air handler that was added, it's pretty small, um so yeah I kind of wanted to pause there and see what other people's uh opinions about this sort of change.

E

So if an error happens during instrument creation, then what instrument has passed back? Is it a mill up instrument at that point.

B

Yeah, that's a fair question um and it's something the specification talks on uh talks about, and so it depends on the the error that actually happens. So uh if there's a duplicate error or a duplicate instrument, registration and it's through a resolvable conflict, I think is the word uh essentially if it's like you know, and the same instrument is created with the same uh name but a different description, a different unit or a different, uh not a kind I think that actually is an unresolvable error, um but yeah a description or a unit.

B

It's it signals an error, but it also passes back a valid instrument and it's an unspecified. What instrument that will pass back in the API and in the specification uh or I'm sorry, the SDK and currently it passes back the original. uh And so you know it's compliant with the specification because it literally says, like it's unspecified Behavior, but you you know, you should still pass back um a valid instrument.

B

There are situations where, like you, are trying to duplicate register, but you have instrument names conflicting with the instrument kind or the instrument type uh I'm. Sorry, the number type so like. If you try to do the same thing with like a float versus an hint that'll pass back a no op at that point um and then you know it'll still function, but it will not record data and the error will be recorded as well. The errors are more descriptive than just blanket errors.

B

They say whether things you know should be changed or not, or you know, or what's going on in that situation.

E

So from the instrumentation author's perspective, they just continue to use an instrument as if it were successful and on on error than the operators on the hook for figuring out what went wrong and how to resolve it, which yeah involved going back to the instrumentation.

B

Right yeah, it may involve going back to the instrumentation.

A

B

Creating a view and I.

E

B

A few of the situations like this for the unit in the description, the error that is produced or maybe a log message in those two situations- is includes the fact that it tells you to do like a view rename or something like that. It recommends using a view in that situation, but yeah. Otherwise it goes back, and um so this is also I.

B

Think the other thing is like you know, the I think that's that's pretty uh uh valid because, like it also may be that there are other implementations that can just they they decide they're going to resolve it in a particular way.

B

um So I think it's really like it's up to the instrumentation, whether they wanna you know, support the SDK from from go I'm, sorry from the main ocel SDK that they need to validate it using some sort of like testing package that we've done in a very similar function, but yeah. It is that's the resolution path at that point.

E

Seems reasonable to me. I would worry a bit about some alternate implementation coming along an instrumentation building around that and then being a thing for users to use the hotel, instrumentation or implementation or some other implementation. That doesn't do that, but that's a bridged across it another day, yeah.

B

I agree um yeah because you're right, like you, could have another implementation come along and just make a choice uh that we don't. You know like we try to be more um conservative and more inclusive, and so maybe somebody comes along and was just as like. You know like. If you choose to you, know: uh I, don't know how they would resolve the number conflict, because there's literally a static type binary that requires the number come back.

B

As you know, a float or an end but say like the kind and like you try to do a duplicate kind uh creation, and it's like well actually like we're.

E

Not going to hand you back like.

B

A histogram when you've already asked this to be. uh You know an asynchronous gauge, or something like that, but like that, that whole setup is like you know. Maybe somebody comes along is like well I. Don't really care like I'll just always hand back the histogram, but it's like well I. Don't.

E

Know, that's not really a great idea, because.

B

Then, like it makes sense to us why that's not a great idea, like you have you know completely different methodology of different understanding is some syntax and like the data you're going to produce is garbage right, but maybe like they're, okay with garbage coming out and so yeah I mean I. Think that's a that's a fault of their implementation uh and I would did not like the instrumentation built around that, but I I agree like maybe just cross that bridge when it comes.

A

It's a whole discussion has made me like: could we build um libraries that are uh disruptive to instrument collection like and not just in go but like if I create a library that goes to a known namespace and creates a histogram in the init function? Will that prevent the good working Library, like my my uh antique and my antagonistic Library, goes and registers like the the net HTTP things and just turns everything into either gauges or histograms or whatever like something that that they aren't?

A

Can I basically deny collecting of HTTP metrics by doing that, um not a problem that I don't know if we're ready to solve at this point, but.

B

Well, I mean I, think that's a good question, because you have to ask yourself the security implications of that like in that situation, you stop recording measurements I, don't think that that's necessarily like that's just like some and that's an annoyance to the operator and they needed to fix their Imports right like, um but you know is there a way where you can do that and then at the same time like uh maliciously collects data and send it somewhere else or something like that right, like uh I, think that's, fair, I, don't think these changes are going to enable that to happen easier, no.

A

No, like it just brought up that literally side tangent of like can this be used maliciously or denial of service.

D

B

Dave, did you have something to add? Oh.

C

The only thing that makes that I'm nervous about with this is that.

C

I, don't think very many people are going to implement any sophisticated Hotel error, Handler things so I think for most people. This will end up as a log line and I think a lot of people would miss something like that.

C

um It I'm a fan of the like of what Bogan proposed where it was. The must, you know, do food where it panics. If there's an error, um obviously I, don't think we can actually do that in our SDK, because we're not allowed to panic.

B

um So that's actually a really good point and that's something that I was uh I was talking with uh Robert here Splunk as well a little bit about that and in.net. They have this like concept of uh fail fast right so like when you set up an SDK you can. You can provide a flag to the SDK to say essentially Panic right, but we don't really want to do that right.

B

What I was thinking is like we could uh similar to bogdens thing is what we could do is we could provide an error Handler in like contrib, that is essentially like a it has a switch on it right. So if it gets an error during SDK setup, it can panic, and then you can. You know after you're done with the Panic uh section, you could say like okay change, the behavior from Panic to uh I, don't know just log or.

E

Something like that.

B

Go back to the default Behavior, so during the normal collection operations, if there's errors there, your program doesn't crash, it would only fail fast at that point um is something I was thinking you could add. As like a contrib error. Handler uh could be useful.

C

Could could we provide must methods in contrib my convenience methods that panic.

B

Not if we remove this error return right, but then you also get that problem that um the instrumentation author is going to make a decision for the operator of the otel code, so say like uh Hotel https decides that we want to use the must method to set up all of our metric instruments. I think.

C

Ideally, the instrumentation would return the error to the user as part of like, if I'm, using a library, that's instrumented and I'm, calling that new function, like it probably already has an error returned.

C

um So if they want to do instrumentation set up there, that seems reasonable and return. The error.

B

uh uh So say that almost I'm so like you're saying like if the operator is also writing, instrumentation.

C

If, if I'm, if I'm a library, not necessarily an instrumentation library but just a library, um I'll, probably and I, have a new function, like you know, new David's, special cue or something um I'm, probably already returning an error. If you pass me invalid arguments or something right, so thank you and then return it back to the user right. Just like an instrumentation Library would.

B

Well, yeah I think you're you're right, but the problem is is like so um I think, I think when you, when you call that function, you're going to be calling it during a collection cycle right like the new David's Q. Well, maybe not like so like you're, creating the queue then yeah, maybe you're, also creating an instrument to track functionality of that Cube I was I, think I.

C

Was trying to go.

D

C

Some of the instrumentation we have in Cube and see how it's done it seems like the using the must methods is really common if you're defining your metrics with an init function, you're like in it here's my three metrics, so that anyone who Imports the package just gets them um I was trying to see what we do. If, when we're defining metrics like deeper within some random package somewhere as an example, if I find it all I'll pass it back.

B

Yeah I I think that's I'd like to see that um that'd be helpful and going through this change. That's proposed here there's a bunch of like examples that are doing a question as well as the um the tests and like I did.

B

I did notice that again our examples, a lot of the time we panic and then, like obviously in our instrumentation libraries uh we use like Hotel handle, like already um so like it's already kind of making that choice to like just send it back to the hotel, Handler and so, like the I, think I really liked the idea of having it. So that, like you, can you could choose that behavior with the Handler like that you want to set up as the SDK author.

B

It's just that, like you know, maybe we want to make that easier, but providing our own error Handler that does that like very easily so like during the init function, if you want to like provide a library like that, like you ah not really right.

D

E

B

That's the thing it's like: if you have a library, that's like Hotel, HTTP and internet function to create a bunch of instruments like I, really, don't think that you should be panicking at that point right because I think that's that's pretty common right. But what happens if you have an implementation underneath the surface that supports one of the things that you're panicking for right?

B

Like so say, we return an error for an invalid instrument name, but that's only an invalid instrument, name in the otel SDK right so like the next one that comes along, is like well actually like I. Don't care like it's not going to return an error. I guess you won't get a panic but like what?

B

If there's one that's more restrictive like do you want to panic there as well like I I, don't know like it just seems it seems like that decision should be made after they've already done all the Imports and the person who's running. The SDK is set it up and they're saying like Okay like anything anything I, don't I sort of log errors, which is the default Behavior but then, like.

B

Maybe we want to make it easier for them to say like okay, also, the default error Handler can be, you know, switched from panic mode to log mode easily. So you can, you know after setup, go into like a stable run mode.

C

B

The problem that I'm, having, though, is like when I was when I was looking at. This is like it anytime that you're passing that information back to the instrumentation authors like you're having instrumentation author, is going to make a decision for the operators for the SDK operators, which sometimes you're the same person right sometimes like encode.

B

Your like in all of our examples like a Prometheus examples right like we're, both the operator and the um in the instrumentation author right, because we're creating instruments and we're also setting up the SDK but I think that it's going to be a mistake. If, like. If you have a situation where you can have an SDK author, decide how error handle errors are handled and it's going to impact the um uh the actual run. Runtime of of another program right.

E

So, like exactly.

B

Back to that situation, right, like you, have an init setup right and you validate that. It's not going to panic for you if you use the default SDK and then you go along with somebody who says like actually like. We can only handle like names that are of a different type right and it starts panicking for, for somebody, like an operator, says like to switch out the SDK and it starts panicking for them like. That would be a really poor Behavior. But it's also kind of not the problem of the instrumentation author right.

B

You can't go back to the instrumentation author and go like stop complaining about this and stop panicking they're gonna go.

C

Like sorry, I was looking um I, don't think instrumentation should ever panic, uh yeah, so I think instrumentation should always pass the errors back to users. My previous thought was probably instrumentation is already or like libraries and stuff are already passing back errors when they're doing initialization work.

C

um So maybe it's not that hard to pass these errors back just like you would for opening a file or whatever else they have to do. um I. Think if I were the user and I were setting up my application and something like this happened, I would want to fail fast, right and I. Think I feel like. We mostly agree on that point. Well,.

B

I yeah I think so yeah um I mean I could definitely see people that don't, but that's I, think the minority I think I think the majority of people would rather it panic when they're trying to set it up. Yeah.

C

And that's the only reason why I'm hesitant to say yeah, let's use the error Handler, because by default that's a log um I want the easy thing to do to be. Hopefully the error makes it back to me and my main function somewhere and I can have a panic error bar myself.

C

That ends up um capturing this. For me,.

C

And if I switch out my SDK and it starts panicking, then that's like great, because I wouldn't want to switch it out and be like. Oh, it's all going to work and then realize that there's some log line somewhere that I missed.

B

Thank you, yeah.

B

So I'm kind of I'm wondering if uh yeah I don't know how we would do this I'm wondering if we could set up an SDK. Sorry, an error Handler that just does this right off the bat but I, don't know how you would signal the error Handler that both the tracing and the metrics pipelines are done. Configuring.

C

As in, if you make some calls and accidentally those are done by the no op.

B

uh No, not necessarily that but like so so. Like I mean the thing is like I: don't I wouldn't want the air handler to panic during like collection right so I want the behavior to change right. So I only wanted to panic when I'm setting things up like building instruments, right, yep and so it'd be it'd, be nice if well, actually, I, don't even know if you could do this. According to the specification.

D

A

Handler that panics on certain types of Errors yeah.

B

Exactly that's kind of what I'm thinking like if we could pass it like a configuration error, which would only be passed during like the setup of an SDK like a trace or a metrics, SDK and it'll see those and go like. Oh okay, I know how to handle this.

B

It's Panic right and then all the other errors are just logs because, like it's just assumed that those errors are coming from standard like operating uh setups, the problem is that that's going to separate boundaries so, like that's, not important, that's not impossible to like handle that, but like the API itself is I, don't know, I, don't know wherever it is in the specification.

B

If we're allowed to panic at all is my question you know like: should this air handler be the default or does this error Handler live in contrib or something like that is my question.

C

My interpretation of things is that anything we do that panics, whether it be a the the must API or the uh the air handler panicking air handler whatever would have to live in contrib yeah.

B

B

So I mean, if that's the case well, I mean even in the even in the other situation. I think it's kind of the same where like, if we, if we leave it as it is like you're saying like most instrumentation authors in their init function, are just going to pass us back to the air handler anyways or they're going to log it which is going to you can't unless you put a logger in there, that's going to figure out how to panic from the log message like no yeah.

B

Let's just go, let's just say no on that one but like, let's just say like even if instrumentation authors are handed back this error. Currently they don't think I put in the air handler so building this error Handler would benefit them in that situation as well, because that's where these errors are going to end up anyways but.

C

B

Remove sorry, go ahead, you.

C

Don't think that it would just be returned as a GO error.

B

Okay, but then, where does that.

C

Go like how, where.

B

Does that eventually get resolved.

C

I mean I. Imagine right if I'm.

C

I imagine it would eventually get propagated up to someone's main function right where they they've done like uh listen and serve to some rapper or something right. Yeah.

B

But the only the only time that's ever going to happen is, if that is coming from instrumentation, that the operator is running or is writing right is like it's in the same package structure right like.

D

Hotel HD, so an Hotel.

B

Http right like when we create the middle middleware, like the error that comes back from there. Actually I, don't I, don't know if there is an error that comes back from there I'm pretty sure we just dump that all into Hotel like handle. But let's just say that, like the air that came back yeah but like, let's just say it came back to like when we created the middleware like that again, like it's gonna get passed up to the user, but like does the user, then like I guess there they could.

B

They could parse through the errors and they can Panic if they wanted, but.

C

Well, they don't have to parse. They know that, like I got an error during configuration, yeah and like I, feel like for most things. If you get an error when you're trying to instantiate something like something has gone very badly wrong um and you should handle it.

B

Yeah I mean that's fair, but how? What happens for instrumentation? That can't return an error during like some sort of setup like the database creation like if you have like Hotel SQL for like exam or you know other instrumentation, where it sets up the instrumentation on import or import of a package like there's no way to communicate back to the original caller of that.

C

So, like sort of like On in It On in It, rather yeah, yeah I, think the usual at least what I've seen in the Prometheus ecosystem is that you do one of two things: either you have a bunch of inits that all panic when they fail or you um like, pass back the air as part of some new function.

B

Yeah I'm sorry like I'm thinking like an oslc because there's no new function. It's just like you register like through the Import Cycle.

C

B

C

B

uh It's exams or uh Splunk or other Forks of exam. It's all! It's not an official.

C

Oh, it's not in good trip.

B

Okay, no uh yeah I see what you're saying, though like if, if we did update like all of our HTTP libraries, to return errors with our middleware, which we don't currently.

B

Then you could, you could make the decision in line in your main function is what you're saying right. Yeah, okay- and that gives you context for where that was actually aired. Right is the problem.

C

Yes, it means that.

C

Like either way, you'll you're we're trying to get to context right either. I'm gonna go sift through the various errors that could come back from hotel um to try and get some context to figure out what the correct behavior for me is um or I'm just going to use the context of where I'm calling from I go. I'm. Calling this new function and I want this to be successfully set up.

B

Okay, I I got you okay, I'm gonna, think about that I think I've taken a lot of time on this one, so I don't want to take up too much more, but I got some great feedback. I'll, definitely I think it's worth thinking about more um something think about it. More. Okay,.

C

Thank you. If you have any.

B

More thoughts on it uh or you find some links. Could you send them to me in slack? Please? Yes, okay, perfect.

B

uh Okay, cool going back to the agenda.

B

I think we're here, okay, so next up, this is bug fix, So. Currently the um metrics pipeline. You can register multiple readers, but the readers after the first one, don't actually get any data exported for them uh and it was tracked down through I think some people at least another.

B

um The problem is that like right now we're caching instruments at the meter level. The thing is as though we need to be caching instruments at the pipeline or the reader level. So right now, all of the cache goes across all of the readers, so only the first um reader, which is encapsulated by a pipeline is going to have that aggregation that it actually determines added to it. So this PR updates that it's, hopefully not too complex of a PR.

B

It's our caching structure, I think could use some documentation uh and there's a few different places. I think there can be cleanup, but I wanted to get this in first uh before that was actually taken. What this does is it takes this instrument cache that used to be in the meter um and it merges it into the inserter. The inserter is the thing that wraps a pipeline to insert into the pipeline. uh It completely dissolves the instrument cache because it's not being patient anymore and like it's, only used in one place.

B

So it's just directly used here, so it should be seen, there's some uh tests that were added to actually validate the uh bug Behavior, so that should be included there as well. So just a heads up. This is a a bug in the SDK. So hopefully we get this merge for the next release.

B

uh Okay, cool and then Aaron. uh You have this PR for changing the signature of the reader collect.

A

uh Yeah, this was the outcome of what was discussed yesterday. uh This is the first of.

D

A

The first one to to enable the memory use this doesn't actually do the memory we use. This just allows us to to work with it. um I'm doing this just to keep the pr small. uh The next one will update the pipeline, uh the produce functions of the pipeline, and then after we get there, we can actually span spread out and have multiple uh the the different aggregators. You know histogram last value some all be addressed in separate PRS, but uh step one yeah.

B

And uh also PRS would be the pooling functions for, like the periodic reader or the Prometheus exporter right. Okay, I was kind of wondering about that one as well, because, like the Prometheus Explorer might not use a pool, you know it's it's well, actually, I, don't know if it David might know if uh you can concurrently um export in Prometheus but like if it handles multiple requests. At the same time, we may be cool I.

A

Think we lock um uncollect anyway, so no okay, what whatever, um whatever concurrency, would have to be blocked on that the same thing for the uh periodic reader.

B

A

I've tried to keep it simple uh because they can be added afterwards.

B

Yeah uh looks excuse me. I definitely would recommend people taking a look at I think it's great first step. It helps also change the uh exported API. So all these other changes should be less impacting once it's released.

B

Impacting maybe the wrong word it'll be more beneficial once it's released, uh okay, cool I! Think with that we're at the end of the agenda, so I'll just pause here see if anybody else has something I wanted to address, that is on the agenda.

B

B

Any cool use cases of hotel.

A

No, but we have a uh annual hackathon uh light step coming up. So if you have any kind of one week, ideas I would love to hear them, because I'll I'll definitely propose them and build something with it.

B

Yeah, that's a good question.

B

Nothing comes to mind. I feel like I used to have I wrote him down somewhere.

A

That's my problem, but all of my notes are like a uh what you might call it a key value database.

A

Mongodb I, just don't remember where I, where that put.

B

Anthony's posted a desktop viewer.

D

Yeah I haven't used this, but I came across it the other day. On uh a couple places, it looks like it's. It's built using Hotel, go and designed for being something that the developer can use while they're, you know going through the development Loop.

A

I think I saw this too, and it's built off The Collector.

A

It's like a not a receiver, it's like an exporter or something on the collector.

B

So what is it? Is it just like a command line and a URL or.

A

It's it's Jaeger in The. Collector is how I would describe it.

B

A

It starts up a collector and then um it gives you a local host, uh a local host web server that you can go and visualize traces.

B

Okay- and it just is kind of connected with the command line, for what you're doing there.

A

Yeah I think it's, the CLI tool is just to start it up. That was my interpretation of it. I might be wrong.

B

Yeah, that's really cool, it's actually really close.

B

uh Yeah, that's a great idea.

B

um Okay cool, so I think we have like 15 minutes left, maybe kind of just I wanted to touch base on our API release. Timeline I think, is the only thing I forgot to kind of mention so I think with David's feedback that PR for the draft probably isn't the way to go, but we still have the um uh small changes to the API for the docs in the addition of the scope attributes as well, but other than that.

B

I think that we're pretty close to getting an RC out I didn't know what people's opinions on doing that um next week, or maybe the week after, depending on how these PRS land.

A

um Could we put some kind of time box on exploring whether or not we should have an error on the instrument creation, because that seems like a one-way change like once? We have a stable API. We don't want to go back.

B

Yeah, no, it definitely is like that's why it needs to be resolved beforehand. I would say the time boxes, probably today, based on the conversation I've had here like I. Think David's convinced me pretty concretely that we want to return an error, but I wanted to think about it, a little bit more um so I'm, guessing we'll close that uh by the end of today.

B

I do think that bognan's question about like having some sort of I think one of the other things we did talk about was like an error Handler that could do some cool things around panics, but that wouldn't block the API, and maybe it needs to be another ticket, so I think the time blocks on that is like I think by the end of the day, we'll know if we want to go forward with that or not I want to think through it a little bit more but I'm strongly leaning towards closing it right now.

B

um That being said, like the other API change, PRS that so I'm thinking like I, think they'll merge today um so yeah time box was that's. What I was thinking like next week, maybe getting an RC candidate out, given, especially with the TC review.

B

Thumbs up from Aaron Anthony David thoughts on that.

C

I would love to see an RC okay.

B

uh Yeah you hear me both um okay cool, then I will try to shoot for that for for next week it's gonna be a busier than normal one uh weeks for the thought it was rotation and the um config working group is like they're, obviously tough ones, but uh this is specific, uh important Milestone, so I'm excited to try to get something out. So, okay.

A

Cool too bad, we couldn't get those on alternating weeks right, I,.

B

A

I feel that way, but I.

B

Also feel, like you know, get it all out of the way at the same time so yeah, it's like it is what it is. um Yeah cool all right, then I think with that. That's all the questions uh just pause here in case anybody else, something else we'll talk about it. You can end here.

B

Awesome all right, we'll see you all, let's take her to sleep or next week same place. Sometime, hopefully, can I see.

D