Open Telemetry Uncategorized, 2 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-03-02 meeting

Description

cncf-opentelemetry meeting-2's Personal Meeting Room

A

B

Hey everyone, um if you could add your name to the agenda, that'd be great um I, think we'll get started in about like maybe two minutes or so.

B

Okay, I think we can get going um so everyone uh welcome to the March 2nd um operator signing um looks like we have two things on the agenda today, uh starting with Tyler. That's all you want to go for it.

C

Hi everyone, my name, is Tyler helmuth I wanted to come to the Sig today and I guess let everyone know make sure. Everyone was aware that I'm working on this issue to add the Go Auto instrumentation to the injectable languages that the operator knows about um there's an open issue for it. I'm working on it, I just wanted to make sure everyone was aware. I was working on it and make sure that the maintainers and the improvers like are good with that and will be available when I submit a PR. For that.

B

Yeah I'm I'm here for it I, was actually thinking about doing it like last month when I was like looking into the Go, Auto and stuff, um so I'm glad that you're doing it because it'll be awesome to have in thank.

C

You, um the second follow-up question is there's not an official open, telementary image or the Go Auto instrumentation. Yet I noticed that for the other languages, none of them uh like release their own image. So the operator has their own build of the images the Go Auto instrumentation, at least the the original creators like it's not Upstream, but like the key Val version of that code has its own image.

C

um Does the operator Sig? If, if the open, Telemetry Go Auto instrumentation group release their own image, is the operator group okay using that image instead of building their own.

B

I'm fine with that, um but I might defer to other folks who are more in the on the instrumentation side of things.

D

um I've been kind of participating in the Go Auto instrumentation I. Think that, like publishing, our own image, at least, is a really good idea, instead of using the key Val one just because that has been donated at this point from what I can tell I'm, not super familiar with technically how the other languages work, but I think that the go instrumentation image is a little um different in how it works based being based on like evpf and having the launcher and everything too.

D

So it might be a lot easier from the operator side for us to maintain that image and uh I'm. Assuming that there's a level of trust between if they're, both open, Telemetry published images that um you know we'll keep it up to date, and uh so that should be usable.

C

Cool yeah I was having the same thoughts that it seems like the go ones a little bit more uh complicated than the other ones. Not the other ones aren't complicated themselves, but getting EVP up into the mix. I felt like something. Maybe the operator Sig wouldn't want to be involved with so I like the idea of using the go instrumentation repositories published image. So okay I'll keep moving forward with that. Then it.

D

Doesn't there's also the uh it has to generate like a Json file of offset values that are used by the evpf, so that is itself its own tool that, like we I added some automation to generate that regularly on our own. But I think if the operator was building that they would also have to we'd have to do that. Okay regularly, generating. Let's.

C

Push it off to the other, stick and just use their image, but.

B

Yeah that sounds pretty reasonable to me. um Tyler, have you been able to test that this is going to work in a kubernetes environment with like in the containers or something like that.

C

Yeah, so actually I haven't submitted the pr yet, but using the key Val image. I do have a local branch where I was able to build the operator, put it into a kind cluster and then essentially grab the Emoji photo like get started stuff off of the Go Auto instrumentation site. Put those images into my kind cluster and then create the instrumentation object and boom I had traces.

C

um Also it's it's not. uh It doesn't use an in container like the other one, so that is new. It injects a sidecar instead of it in the container I. Think that's a requirement of the instrumentation like the way it works. So that's a little different, um but it did work.

B

Yeah I don't see that being a huge problem, given that we're doing like a mutating web hook anyway, um I think it just won't be able to use the same pattern when you already know this from implementing it, but.

C

Pattern, yeah not as different as I thought. It would be basically in the in the golang.go file instead of the python.go or whatever like. Instead of messing with the init containers, we just mess with the container is proper and there was a little bit of extra stuff. We had to do um you'll see it in the PR, but it really wasn't that much overhead.

B

Great yeah I'm excited to see it.

D

Is there any simple something else to just keep in mind? I, don't know if there's like a stability level to track for the different types of Auto, instrumentation um I know that the Go Auto instrumentation is still really early along um and there might be changes we haven't even I, don't think to find like a compatibility model for how far back uh the Go versions we're going to support yet.

D

But if it can mark this as like an alpha Auto instrumentation, just like as you're merging into operator that might be good for users to know that didn't there may be some changes.

D

C

Know if there's a presence, I don't know if there is that, but we should do that. I agree.

E

There is, as far as I know um something like versions.txt, where.

E

Search for it um so I'm not sure so it's we can have it there as a string, so the API versions- I, can put it here in the chat and um I assume you can go with an API version like Alpha version, one Alpha version, 0 or just yeah, make it implicit and use a zero dot. Something version.

B

Yeah I almost want I. Think the the thing that um it sounds like they're worried about here is that, like the auto instrumentation for go is very experimental at this point and probably shouldn't be recommended for like production workloads. Just yet is what it sounds like to me correct if I'm wrong, but.

D

um Yeah I don't want to make an initial statement for that, but that's kind of what I was Landing towards yeah yeah.

B

And so maybe there's like a uh something that we can do here on the operator side, like some config for the operator itself. That says like enable Alpha feature, and then that's how you get go so that people have to like explicitly state that the operator will allow it.

B

um We would probably honestly we're going to want to do the same for the um op-amp like Bridge projects, um given that that is also going to be an experimental feature. So I think that there is like precedent to make that happen.

B

um Tyler. Does that seem reasonable to you to have that be like an extra step to make to enable this, or do you think it's not worth doing just like you.

C

Have a thing I think it's I think it's important to indicate somehow that of all the instrumentations that are added, that this is the least mature, one, probably or at least the the least ready, I, don't know, I, don't know what it was like. When.Net was added. That was a pretty new concept as well. The way that the.net was doing their Auto implementation right.

C

um If there's not a system in place already, like the collectors, got the feature gate system in place. If the operator doesn't have something equivalent to that, then we would have to build separate from the Go Auto instrumentation um I I, don't like. Is it worth slowing stuff down or is there something else we can do if it already exists like yes, I would definitely want to use something like that to opt in, although the instrumentation stuff already is.

C

It is already only opt-in, like you couldn't accidentally get out of Go Auto instrumentation. You do have to like create the instrumentation object and the go one you you need more at the moment at least, and we can have a discussion on this on the PR, but you need more than just one um annotation you have to annotate, yes, I want injection, and then you have to annotate. This is where my execution path is.

C

um So there's like a double opt-in. There already.

B

Yeah I think, given that we don't have a thing in place for it and you're gonna have to do it, set all these flags regardless. Maybe it's just something we should call out in the readme when you update that with the p for the pr to say like that, yeah.

C

This one's super Greenfield.

B

D

Like something in yeah warning in the um like in the auto instrumentation repo right now at the top, it says like this is Alpha, but just.

F

Saying even the CRT version says it's alpha or beta right, so that should also explain that auto instrumentation is not created.

C

And in the versions.txt we, where we list out the image like I, imagine that the first release, whenever that happens, will be a 0.0.1 or 0.1.0 or something really low and new.

C

Okay, cool um I will keep working on getting that image uh made with the with the go team and then I'll submit the pr once that is finished,.

B

Cool. Thank you thanks, Tyler.

B

um Next, up on the agenda, we have Alex speech talking about um I, think often things.

G

uh Yeah, hey everyone, um I, just kind of had some general questions on like the op amp Bridge Project, specifically um I have an interest in using the operator to basically push configs to a running collector and from my understanding right now. The way that op-amp bridge works is that the op-amp bridge effectively connects to it has an op-amp client that connects with op-amp server and then based on messages from the op-amp server.

G

It'll update, The, Collector, crd and so I'm kind of wondering, like there's several issues that were kind of pointing to this uh extension issue. This op-amp extension extension issue and it's kind of stalled. So there's kind of two things.

G

I'm kind of asking questions about is one uh I'm willing to kind of pick up work on that, if that's necessary and two I'm trying to understand like what the actual role is of the op-amp extension, because it seems like the operator bridge is taken care of kind of the op-amp proxy between the server and The Collector crd.

G

So, like I'm I'm kind of running what the purpose is of of the op-amp extension in the context of an operator because it seems like the operator doesn't need, like the sorry, the op-amp bridge can basically push out configs and handle the um kind of communication to the op-amp server. So I'm not I'm, not quite sure what the purpose is of the extension. In the context of the operator. uh Yeah.

B

Yeah so um I'll answer your questions um in in the order you ask them. um Also. Could someone take notes, while I'm about to speak, because um I don't think my brain will be able to do both at the same time? So someone could write um the notes in here uh that'd be great.

B

um So for the first question, what's up with the op-amp extension and like the things needed on the collector set um the uh Hotel agent Sig the what's the exact name of it, I always forget it's the um Hotel Agent Man working group um that is taking time uh in The, Collector Sig, so like actually push through some requirements in order to make the extension work, um the pr is in a draft State because of that there are two issues linked at the top of it.

B

That um explain why it's in that state and what they're doing to like get it through the reason. Why is that the changes that need to be made are like? Have the significant um I think that they will either introduce some breaking changes into the collector or will introduce some like new behavior? That needs to be better understood, um probably both I'm, not positive, um but so well.

B

Those are in Flight, the actual, like extension, to make op-amp work can't be merged yet because right now like it's unable to even get the existing um uh configuration that the collector is running. So that's that's the reason. That's sort of the status of that um for more questions on that I would point you to their channel in the cncf slack, which is Hotel Agent, Man WG.

B

um They have their own Sig meetings as well um to discuss like ongoing efforts there um to the second question about like what is the point of the extension given the operator, uh the op-amp bridge will be able to do like all of that.

B

um So the reason. Why is that not everyone is going to be running collectors in the kubernetes environment? Nor will they want um to do this type of management, um necessarily so for the first case, it's like you know. If you're, not in kubernetes, you can't use an operator.

B

It's kind of just you know, lay the land, um but the other case is that, like the effect, if we were to uh create a crd in kubernetes and someone's using a something to reconcile we'll assume for a second that the op-amp Bridge Project didn't exist, you have to apply your collectors somehow.

B

So without the bridge, someone would apply their collectors, let's say via Helm via you know, some type of like cicd system. If that collector is using the remote configuration features of the op-amp extension, the actual configuration that's going to be run will be different than the one that the applier is expecting to be there so anytime that their cicd system goes to reconcile those changes.

B

The effective configuration will just be blown away, which is you know, obviously not good um so by creating the op-amp bridge and creating crds from that you have your um you're able to use. uh You know your CI CD to apply the bridge uh crd, so many acronyms I'm.

B

Sorry, um so you are able to apply your crd via CI CD, um and that is the thing that will then go and create all of the collectors which will not be tracked, the ICD cicd, but can be stored in a config map if you wanted, like some amounts of resiliency there.

B

um So that's the reason why it's being done this way. It also means that, like we don't need to wait on this is sort of the side benefit. We don't need to wait on the op-amp extension in The Collector to be available um for us to make progress on the op-amp bridge.

B

um It's the same protocol, but we don't. We don't need the collectors um extension to be available yet so we're able to make progress so.

H

You said if you're a collector running the authentic extension configuration, but your configuration is pushed another way you would lose. That Bridge configuration is that accurate.

B

Sorry uh I'm might have misspoke cool.

H

um And it doesn't make sense.

B

Thank you. That's.

H

What I'm, taking um the Auburn Bridge would allow you to apply your crd and not lose configuration. Is that accurate.

B

Okay, so the reason why this is accurate is that you no longer have uh cicd reconciling a collector configuration um The Collector configuration is only being created by the bridge. So when you go to apply a change to the bridge, The Collector crds are not going to be reconciled; um only the the bridge would be, um which won't do a delete recreate of your collectors so, um but, and that's the benefit of doing it via this external component- did that did that make more sense? Christina I can explain more um if that would help.

H

uh I think it it makes sense to me I, don't know if anyone else has other questions. Also I missed what the original I got. The answer, but I missed the question.

G

So, like there's this op-amp extension uh for the actual collector itself, and my question was related to why that was necessary for the op-amp bridge. But I think the answer was that the op-amp bridge uh doesn't necessarily right now need to have that extension, because the op-amp bridge can talk to both the op-amp server and it can um effectively reconcile changes to the crd and so I believe.

G

What Jacob was saying is that um if you're doing CI CD systems like your CI CD only is going to reconcile the uh the op-amp bridge because, like if you're, let's say that you're updating config maps and you have your CI CD to set to update the config Max for The Collector. Every time you deploy, it would effectively blow away the changes the config map. So the CI CDD system would update the um would update the op-amp bridge and then the op-amp bridge is what basically reconciles the config Maps.

G

So I. Think that answered my question. uh I I was just trying to make sure that there wasn't like a direct dependency between them, but um yeah.

H

Yeah uh that matches my understanding. So it sounds, sounds good to me. I. It sounds like they're, not um there's, no direct dependency.

B

Yeah, the one feature that we have that we want to do as part of the op-amp bread work is to connect the collectors op-amp extension without remote config only reporting health and effective configuration um to the uh op-amp Bridge. So but that's like a future State type project.

B

um So once the bridge is actually like stable and able to do, um you know actually update crds, which we can now, but I mean like we don't have the crgs for the bridge. Yet um a future feature is going to be connecting the collector's op-amp extension to the bridge in order to more effectively report health and effective configuration. That was some feedback that we got from uh tigron from the um op-amp working group.

B

um But it's not required in initial, like running of of the bridge itself,.

G

And I guess, like a follow-up question to that, is there a specific like is? Are we would we be using the op-amp protocol for that Health that health check communication? Where would have to be proprietary, yeah.

B

Yeah up and protocol for that, so the collector would just be configured it. We would. The operator would automatically detect that. Oh, this collector was created by an op-amp bridge I'm, going to configure the collector's op-amp extension to talk to the bridge um directly for just those features.

G

Okay, um and so why is that necessary if the operator is the one, that's reconciling the config Maps like wanted to config the deploy config net? Wouldn't that always be the effective config? Or is there a case where the collector is deployed outside of the operator, and you need to have the effective config.

B

So there are cases where um the operator isn't able to do stuff around like configuration validation in advance. So something that might happen is the effective config. The previous working effective configuration of a collector is not it's not the same as the current one. There are also cases where you don't need to fill in um the default values for processors and receivers, so the actual, like effective configuration, might be different than the specified one.

B

If that, so that's the that's the difference. The other thing is that, because we can't validate it ahead of time, you're able to push uh potentially bad configuration via the crd and so you'd want to know via the health check that is actually like up and available, and we want to do that via op-amp, because it's the you know, sort of again like open protocol for doing that type of thing.

B

We could also do it via like metrics if we wanted, but then we'd have to have the bridge either be a metric server and and then like it becomes a little bit more complicated.

G

So, in that case, like the bridge, basically would have to run an op-amp server, an op-amp client and then like the kubernetes client, effectively right.

B

Sorry say that one more time.

G

So like effectively the operant bridge in order to uh it would have to have it have to have like an op-amp client, so it can communicate at the an outside op-amp server, and then it would also have an OP I have to have an op-amp server that can communicate to the collector's extension. Is that accurate.

B

Yes, that is accurate. Okay, sorry, one second I'm getting a call.

B

Hey sorry, um yeah, uh that's that is right. Alex. Thank you for the clarification.

G

uh Okay, cool well that answer pretty much all the questions I had. Thank you.

B

Yeah no worries um and as far as like contributions go um I think that the someone's already handling this crd uh for the bridge. um We probably should I need to check in with that person, um but uh I think helping get through the collector work would be really valuable right now.

B

um I think that that's what's going to take the longest and probably needs the most amount of people like working on it, um because it's going to be more, it's it's going to be more difficult than the bridge, which is a very simple component with all the code already written for it. um But thank you.

B

Okay. I think we have one more agenda item um from Bene. You want to introduce that.

E

uh Yeah, it's probably short one. So there is the release and it's already approved and I just wanted to ask if we can go ahead here.

B

I can merge it right now, since.

E

I have no permissions yeah uh and then also the branch needs to be Tagged, so I assume I've. Also no permissions for this suggest, with the version number.

B

The branch needs to be tagged.

E

So delete this commit so with the new version and then the bot will take over and create a release. um So.

I

Yeah I think, usually we have to publish a tag once the pr is merged, so I, don't think uh publishing pushing a tag is only authorized for maintenance. At this point.

E

All right so, therefore, this.

E

um Someone would need to take this over and then I can continue with the operator Hub thing and all of that.

B

Gotcha um I can.

B

Yeah advantages, let me know, what's needed of me, to do things I'm, looking at the release, instructions right now and I'll check into that once the pr is done building yes,.

I

Yeah I have done the last release, so if you need anything, feel free to tag them in the operator channel, so yeah yeah.

B

Thank you, um I'm, probably going to need a little help with it so but good practice. So uh thanks, okay, uh I gotta run. Is there anything else um that people have topics on.

B

Oh good great! Well uh thanks! Everyone um hope you all have a great day.

E

Thanks bye, bye.