Open Telemetry Uncategorized, 1 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-11-01 meeting

Description

cncf-opentelemetry@cncf.io's Personal Meeting Room

A

B

Let's go how to be agenda.

C

State is kind of like a context, but for a like a connection.

B

Yeah basically I thought about using a real context, but.

B

The feedback was not being used for like real context done situation. Maybe it's best not attend there.

A

B

A

Hello welcome hi. There.

A

This is my first time listening in so trying to get a lay of land here, wonderful.

B

There's an agenda Linked In, the um in the meeting invite.

B

A

Imagine Keego will.

B

Be here wait a few minutes to see who else is joining.

A

D

Who's aging, us prematurely. Aren't we the first of November, not the second.

B

A

C

Is this Auto generated I, don't.

A

C

Fixed that's my most recent contribution to Optimum.

D

I guess should I fill myself in here as well.

A

D

Okay, I think we can start. So, let's see the health reporting yeah.

D

So I think Andy. We we discussed this so I think we're thinking about just renaming the up feel to healthy I guess, and that should be good enough. What do you think? Are we good with.

B

That I think that's fine I looked at your PR as well, um just just now um and I I'm happy with the changes. I think it's okay to um I think it's appropriate to still have the the start. Time reflect actual start time. Yeah yeah, um but healthy. You know healthy, doesn't seem to at odds with that at all I think it's fine.

D

Okay sounds good, so, let's then, in that case, if you're, okay with the pr let's match that and then I can make the the changes in the implementation as well. So we should be good with this one.

C

I did just have a question on why I mean I I'm, fine with it, but why a disconnected agent would be unhealthy like. Why do we I mean it seems like you? Could just one duration disconnect it and uh that's not like a bad state for you, um but if that's just our opinion on it too, that I'm fine with it.

D

uh It's it's a bit, I think it's a bit different, it's about an agent that is up and running, but it's on unhealthy. So it's running, but we know something is wrong with it. There is a there's, a situation like that. We we have. We have a health check, extension in The Collector, for example, that we can uh hit the endpoint and ask whether it's healthier or no, and it will tell us that it's not healthy, but but the process is up right. It's it's fine, it's running! So what do we?

D

What do we set the field? The current view to, in that case we have up the field that is named up. It is up right. It's.

A

D

A

It's unhealthy.

D

A

D

The situation is a.

B

Bad config, you know it's probably.

D

Yeah yeah right, config or whatever any anything that will not really don't have to restrict it to any particular situation. We know that there is a concept of healthiness and- and we have an example, that the collector has a health check extension which can say that it's unhealthy, so we have to somehow reflect that state, but we don't have a good way to do that today. With the current set of fields, we could have separate Fields, both Fields like healthy and up, but I'm, not sure we need that. That's that's the thing right.

D

So it's more like we're more interested in healthiness and we're, maybe not so much interested in whether it's up or not up yeah.

B

I would say: no, that's, not healthy and but.

E

If it's not healthy, it doesn't necessarily mean yeah, so I think there's a lot of similarity here to like liveness versus healthy yeah.

D

E

Perhaps that's the right terminology to use I, don't know.

D

Right I guess: uh you're using the kubernetes terminology, yeah.

E

Which is also common from a few other places, yeah.

D

D

So the healthiness in is it it's not healthy. What is it the in kubernetes? They use liveness and what was the other? Is it healthiness, not healthiness, Readiness I think yes,.

E

D

And yeah is it yeah? Is it ready to accept a request, so liveliness is whether it's up and running and and uh and Readiness, and what is whether it's ready to accept requests so.

E

I, don't know maybe yeah yeah, which, which you assume is in a healthy state, yeah.

D

So you're I'm guessing you're, suggesting to call it Readiness. In that case and not healthiness, we would have to rename most of the things that are in the spec. In that case, we'll probably call it health uh what an agent health!

D

What do we call it? An agent Readiness agent.

E

D

Or maybe it stays as an agent Health, but the health is composed of many things, one of which is Readiness or maybe even liveliness can be very part of you. It's really just for now. I think it's the name of the field right, but what we have right now is I think it's not very useful on its own. If it's just what we have the up field, it's not good enough, at least so. We need something else. It's either healthy or ready, I'm, fine, with either name up. It's not it's not good.

A

C

I think health is good, I mean you, you wouldn't get it. You wouldn't get this agent Health message if it was disconnected right. So if you get this message you you know it's connected and it can just it's really supposed to just tell you if it can accept new configs or if it's in a good state or not that.

D

Is true, but if you're using a supervisor it doesn't really mean that the agent process is up and running right. It only means that the reason all pump client that is capable of reporting the health which may be the agent itself or maybe the supervisor. So we don't really specify that.

C

And a supervisor send a message that says this: agent's down or yeah yeah.

D

Yes, yes, that's a possibility too. So I don't know. If we need to do that right now, maybe we can add that later the the up field itself, maybe it's necessary I, don't know right, but it's easy to add later it's not a problem, it's kind of independent field that we can add so I'd like to figure out the name for this, the field that we actually need right now, I mean I'm, trying to implement the supervisor where this comes from in this implementation, I don't know what to set it to well.

D

I I check the health of the of the collector, and uh it tells me it's not healthy, but I don't know what what to do with that right. That's the problem. I have right now, so it's kind of the practical needs of reflecting what I'm, seeing what I'm observing from the supervisor.

D

Anyway, I think ready works as well. If we want to call it ready, that's fine.

D

I guess the this slight disconnect. There would be from what it's called in the collector in The Collector. We call it the health check extension. So it's it's called the health.

B

I prefer Health personally healthy I think it's um directly correlates to health checks in the character and yeah and health checks for kubernetes as well. There's a you know, analogous concept there.

D

Yeah we we can clarify in the description that it's the same semantics as the Readiness in kubernetes, something like that.

C

Yeah I think healthy is good. I, don't feel super strongly, but it matches the collector so I think that makes sense.

E

B

D

D

Let's move to the next.

D

The websocket Heather, so yes, I posted that PR, which shows how the header, a bit in the header, could be used to implement the message. Fragmentation I think. Last time we discussed in the last meeting. We wanted to see something like that: I, don't know if it's helpful or no um I still think that it would be useful to have this header any new thoughts. Maybe since the last time.

B

I'm fine with it.

D

You're fine with it, okay, any anyone, not fine with it any objections. Maybe.

E

I'm fine with it I, yeah I, think I. Think I've come around a little bit more on on the idea of of adding the header and and what opportunities it presents down. The road.

D

Okay, okay, so if let's do this, if I don't hear any objections, let's say in a few days maybe I'm gonna file, a PR that just does that in the in the spec. It's going to be a few changes there uh yeah.

E

And then we all, you already have the implementation at.

D

Least yes, uh partial I mean it's not the full, but it's not difficult to just completely complete yeah yeah.

D

B

I added the last item, which was a PR I created for adding some state um to the connection, so that you could maintain some State between callbacks and op-amp. This is just a go: Library change, not a spec change.

D

Yeah I I remember: we discussed this. I didn't have a chance to look at the pr yet I will.

D

Do we is there anything that we need to discuss right now.

B

um I would just if nobody's had a chance to look at it, then no I guess, but um it's fairly small, so I was hoping that you know. If we, if we, if there's any discussion, we wanted to have that might save on a life back and forth in comments, then we could just have it. But um if we want to just say before discussion outside of this meeting, that's fine.

D

Too, is it just in the HTTP or the websocket as well? It's in both places. It's involved. What I see it's involved? Okay, it's.

B

Sort of less less meaningful in HTTP, because it's not a Long Live connection, but it can still be useful to share information from headers or, for example, a client certificate. Are there other types of connections state um could.

C

B

For for whatever you know, a particular implementation wanted to associate with that connection.

D

So this is just um uh an interface which is I, guess the only limitation there it's supposed to be a comparable, I, guess right, so that you can use it as a key in a map.

B

D

Field on the connection.

B

It's not the connection itself.

D

uh I get it but is: what's uh you you're supposed to be using this somewhere as a key or some some sort of a map of connections? Right? That's the.

B

Thing no. This is just a piece of state that um that the implementer produces as part of the on connecting and then can be used or mutated by the other callbacks.

D

Oh wait so this is. This should be provided by the Callback, the the on connecting callback, the.

B

D

B

To the response.

A

B

The connection response.

D

Okay, so I'm I'm, not sure. Let me take a look offline because I'm not completely sure what I'm that's fine. Thank you. Okay, yeah.

B

The essential issue- and we talked about a couple different ways to accomplish this, but the essential issue is not being able to communicate any state from on connecting where you have access to the HTTP requests to the other callbacks.

B

um No current way reliably. We were able to do it in bind plane because we passed the agent ID in an HTTP error and that's how we can correlate that to subsequent on message calls. But it's not um it's not ideal. It requires that ever be passed, which is not a um you know. It's setting expectations of an agent that shouldn't be there basically for for General agents, but without that there's really no way to know.

B

For example, if the agent identifies itself with agent ID and then you want to correlate that to a known certificate for.

A

B

There's no way to do that.

D

I think we discussed uh an alternate to this as well right, so that the it's not the uh responsibility of callback function to generate the state, we do the opposite. The caller can generate a state.

B

It could generate a key that you could then.

D

Put a key, essentially yeah, some sort of a token or.

B

Something yeah, but that I just thought that that had an extra effort on the implementation side to build the map and to remove things from the map and to protect it with new taxes and when, when it's really in the in the case of the implementation, it's a local variable from.

D

The usage perspective: do you think this is easier to use what you're suggesting.

B

D

Okay, so I think we should aim for that right so that the usage is easier. Yeah.

B

I think I think the easiest case, and this is a a kind of a dumb example because I'm setting a value of a and b in the test. But if you look.

D

B

You could just see that uh you know and on connecting we return a value of a we unconnected. We change it to B and I can actually close. We confirm that it still be, and you know so, it just shows how you could produce State and and let it travel around.

B

um You know we had the discussion and slack about potentially using generics, and it would obviously make this way cleaner, but um I.

A

Also recognize that.

B

That imposes some other limitations so well.

D

I think it's fine to use an interface for this. It's just yeah yeah.

B

Okay, I'm also open to alternative ideas. I I just thought this would be helpful to um you know, set aside some data without having to key it in some external data structure, then again you had to synchronize when when, as far as the implementation is concerned, we're passing a variable around and it's very straightforward.

D

Okay, let me take a look offline.

C

I think the only thing I think is slightly weird. Is that it it since it's an interface it could just be. You know, a string that you could use as a key or it could be like a struct. If you wanted to keep more information in there.

C

um I guess. That's just like up to the implementation just to keep track of like what you're keeping in there. So maybe that's not a big deal and.

B

C

Say that's right.

B

I think Engineers would probably help quite a bit there and um I I implemented using generics just to see what it would look like um you and you end up having to actually have the server specify the type of state that you're going to maintain, so that the callbacks can be of the same type so that everything works well.

B

um It is a little cleaner than for the rest of your code to recognize that. That's what you're going to have as your connection state.

D

Would do would you be able to show in our server example uh how we could use this like? Is there so.

B

I had hoped to actually replace a lot of the like bi-directional map that you have and then I realized that you don't make an assumption that a connection is for a single agent I, don't yeah.

D

Right I try to avoid making that assumptions so I was I was.

B

Very excited to replace all of your maps with State and then um and then and then realized I couldn't.

D

um But the reason I don't is because I hope to add the support for proxy connections. That's why I don't.

B

I I, I I, think that makes sense, um and uh you know that was just an example.

B

Implementation I certainly wouldn't want to limit the you know overall design to not support proxy because that's very important.

D

uh What I'm asking for is what would be some of the use cases that where this would be useful, all right, so maybe I.

B

Mean I think the best. The best example really is the client certificate which you could add possibly to the connection as um another member yeah.

D

um Can Can. We show that in the example that we have, we have the client certificates right in the example we're using them. I think if I remember correctly,.

B

D

B

Can take a look.

D

I definitely did it at some stage. I, just don't remember whether it's in this particular implementation or somewhere else but yeah. If we do, then it would be I mean it's useful to show.

B

It yeah the example: implementation doesn't listen for on connecting at all or on connecting for that matter. Just listens for on message and and on disconnect, but.

D

We are, we are verifying the uh client certificates right, we're doing something about them. You know, am I.

D

One second: we did something about it.

D

Okay, maybe I'll do that offline, I! Guess when I'm reviewing the pr.

B

Yeah, that's fine, yeah, I, didn't I, didn't I, didn't expect you to review it on the spot and I didn't have a chance to write it until last night, so um feedback's welcome from anyone. Thank you.

C

D

So we're good with this one for now.

D

Okay, so I guess one other thing I have is I was a kubecon last week and there was a lot of interest in opamp from people who I met the first time in my life. They they kept asking about all pumped about agent management.

D

Apparently it's a more important problem that I thought it is so I think I'm going to bring this up tomorrow in The Collector meeting as well to see whether we can actually get some sort of commitments from collector maintainers to not to implement it, but to consider that it's an important capability that we want to have in the collector, so that whatever is blocking there. We have a number of things that are blockers. There gets, uh gets implemented or gets approved or had, or we have at least some sort of a solution there.

D

So yeah I'll try to do that tomorrow. We have a collector stick meeting tomorrow, but it's it's. It was pretty exciting, like people come and ask for for opamp, like okay cool, how did you find out about it and they? Some people said that we have similar Solutions we're using something that looks quite similar to what Oppam is supposed to do, and we want to actually change from using our proprietary solution to what you guys are doing there.

D

So I think it's great so, but to do that, we need to at least have a proper implementation or production grade implementation right and the collector is where it probably needs to leave my plane by the way. Also- and we if a few people were asking about whether we're planning to have a back-end implementation like a server side, a coupon and uh I acquainted them to buy plane to take a look at it. uh Yeah I thought, I told them I, don't know whether open Telemetry will have its own.

D

For now, it's unclear, maybe I, don't know at some point we'll.

E

Just wait a little bit longer than and then we'll Fork bind plane and then rename.

A

D

Yeah but uh yeah anyway, maybe you're going to see a few more people asking about buying plane as a result.

B

Yeah sounds good.

D

E

We're buying plane, but without the weird headers.

A

B

That's funny yeah yeah make sure you have both the agent ID and the Outback version set, even though they're not in the space.

E

That's right: that's fantastic news, though I'm glad to hear that there's a greater interest than expected um and yeah I think the right move is definitely just uh for. For open implementations of the back end go to bind plane because I mean that's exactly what I did so I can uh I can recommend that.

B

D

Someone also we're asking about implementing opamp in the operator in The Collector operator, someone actually being the maintainer of an operator. They they the current maintainer of an operator. So they want to do to work on that. So there's anyway, I saw a lot more inquiries and support for opumb in those just two days. Then, in the past several months and people asking for it in person, I was kind of kind of quite motivating, I, don't know yeah, that's great. We should we should finalize this. The the 1.0.

B

And we had quite a few people there, but um but I I didn't make it this time.

D

Yeah I saw I met, Dan I met, uh Mike was there I think uh maybe some others.

B

Yeah I know Dan and danielski. um Yes, you.

A

B

And um and Ryan was there as well so yeah great.

A

The industry in.

C

2019 number: five: what.

D

E

That number five.

D

Okay, uh anything else, anyone any other topics.

B

I guess uh um I I I feel like I'm asking this every every two weeks, but um what what is our we've got a couple things now to the headers in the connection.

B

What I would really like to do is update fine plane to use something I, think we're 0.3 or something it's quite old um I've been reluctant to update because I don't want another incompatible, breaking change, um so I'm I'm waiting for either 1.0 or something that I feel, like we've kind of agreed to say, is stable enough that it's worth the risk of updating again and hopefully not updating another time after that. But.

D

um Are you looking for the stability in the go code as well or just the spec for now.

B

um It's it's both because the yeah you know we're using the go code on both sides, so um they really. They need to be both forward and Backward Compatible. um It's obviously the ideal, because it just becomes much harder when people have real deployments um to tell them to update all their agents. At the exact same time, they update their server, otherwise they won't communicate yeah.

D

B

Not I, I, agree, I, guess the situation right.

D

I get it I. Guess, let's: let's do the header thing if we agree to do it, uh the state is I think optional. So it's not going to be a breaking change. That's.

B

Just a library change to the to the server side doesn't even affect it, doesn't affect Communications and doesn't affect the client.

D

Hey it doesn't break the existing code either right, it's uh yeah, so it's not it's not going to be a problem. Then what else is there I? Don't think we have anything else in the pipeline. That is going to be a breaking change.

D

Yeah I don't see any any open PRS there or anything.

B

The health thing that you're that we have agreed on with healthy I think is.

D

It is yeah I guess if you're using it, yes you're right that one too, but we agreed on that as well: yeah yeah, so the health, the the header and uh there's nothing else, I, don't see anything else. There.

B

And I'd still like to resume the custom message thing that I, as as we have we've talked about previously.

D

Right right, it's not going to be a breaking change. Yeah, it won't be a.

B

Breaking change right, yeah.

D

B

D

All right all right thanks everyone. Thank you to your side, good.

C