Open Telemetry Uncategorized, 8 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-12-08 meeting

Description

cncf-opentelemetry meeting-2's Personal Meeting Room

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

B

C

um For everyone here, uh please add yourself to the meeting notes, dot or attendees via chance, foreign.

D

Yeah, hello, everyone and.

A

Welcome to the first kubernetes meeting.

D

Yeah, let's maybe wait for a couple more minutes, foreign.

D

D

I think it would be good to do some kind of introductions. I'm super excited to see you at least like this. Virtually.

D

I'm software engineer and team leader of the distributed tracing team at Red, Hat I'm, based in Europe in Switzerland and yeah I'm. Currently the maintainer of the operator I as well, contribute to Jaeger and used to work on open tracing as well before.

D

Bennett you want to go next.

E

An introduction around um I can yeah I'm and I'm, also working on top and Telemetry, and on The Collector side and I work together with Pavel and that's basically, it.

E

C

Sure yeah, my name is Jake wernoff um I'm, the tech lead for the Telemetry pipeline team at lightstep. We have some other folks from my Superior that will pass you. After this.

C

um My focus has been on the operator's Target allocator for uh the collectors like sharded Prometheus setup, um with a few random operator PR's here and there for features that I've been needing for testing um and yeah. I have like a few things on the agenda for today, but we can get to this later. uh I'll pass to Christina.

F

uh Hi I'm Christina I'm on Jacob's team, the Telemetry pipeline team at lightstep, uh mostly I, work on the target allocator code um and just like fixing things I, find asking questions. I'll pass demo.

G

Hey everyone: my name is Mo I'm, also on the Telemetry pipeline team, at lightstep, with Jacob and Christina and Gustavo um yeah um I'm, relatively new engineer, I'm kind of just trying to contribute where I can uh Target allocator um a little bit on the HPA um and yeah. It's all passed to Gustavo.

H

uh So hey everyone, uh I'm Gustavo, paiva uh I, haven't actually contributed a lot to uh never contribute to the operator. uh I was also a approver at the hotel goal and done some uh contributions to the hotel collector. So I'm, just like looking around here a little bit as well. uh Yeah I will pass through Anthony.

I

Hi uh Anthony Mirabella um MSD at Amazon, working on ADOT uh I'm, a maintainer of the go SDK approver in The Collector, uh also working in Lambda and the operator get my fingers in a little bit of all the buys that are going on in hotel, basically uh and I. Guess I'll pass over to Michael I believe you are.

B

Next hi good morning, guys so uh I'm Michael Tong I'm uh at Apple So, currently I I, just started to uh to learn working with open territory.

B

um So recently, I am working on a idea that, um to create a name chart that works similar to cube premises. So what I mean is that um you click you install the entire stack with one click or one command, and then you automatically get the production ready like um um deployment just.

B

um Yeah so I'm currently stuck in a few um like, for example, the target allocator I, don't find any documentation of how to deploy and use it and manage it.

D

There's some great uh thanks Mikhail and we have a New Journey, the rats Nike. Probably uh so we are running the introductions. Do you want to introduce themselves hey.

C

G

Yeah sorry uh having some webcam issues tomorrow, my name is t I'm, an SRE on the Optometry pipeline team, with Jacob Christina and mo um yeah. That's my introduction.

C

D

Miss any other information, no.

A

That's great I think that's that's!.

D

The theme is quite quite huge, uh yeah, so yeah, let's go to the agenda uh Jacob. Do you want to start with the documentation.

C

Sure yeah, um as uh Michael, was sort of pointing out in his introduction. um The documentation right now for uh the operator is relatively light, especially for the Target allocator, but um also for other features um like Auto instrumentation.

C

um There are like a few examples, but I think that we could do with a really good, like docs push um or a lot of the features that are available, um including like examples um and sort of like behaviors I think would also be really helpful.

C

um I think also like, as we get more people using Target allocator, would be great to do a write-up of like how it works, what the various modes are um and how one can configure it. um We have this chart and Michael I DMD the chart that we worked on. um We have a chart that is the equivalent of the um Prometheus.

C

uh The cube Prometheus stack chart where it's like a like one: click install for all of your infrastructure, metrics um and that'll, be like a thing that I would love to be able to donate to the help chart repo. Maybe um it is the thing that you can install runs the operator um and it automatically connect, uh collects all of your like Coupe, State metrics and then uses the target. Allocators crd functionality to also pull in from service monitors as well.

H

um So that's like another.

C

Thing that would be um great for documentation on how one can use this, um but with all that said, I think I was going to make some issues today or tomorrow on the things to be documented.

C

um Do we think I think that the best place to do it is probably in like the readme's of the individual like subfolders? Does that make sense, or should we put the docs on the docs website.

D

I think that's a great question and we should kind of decide where will be our? Where will our dogs live? um I, don't know what is the direction in other open Telemetry projects? Do they focus on the website or more on the kind of readmies in the repositories?

D

Is there any like overall vision for the project.

I

I think it's been generally fairly split, like a lot of the the technical documentation will stay in the repos close to the code and then on the website. We have getting started guides or fairly General overviews blog posts about how to do certain use cases. Things like that.

B

Yeah um I think what we can do is um so we update the the individual, um the folder, which is with me, and then we put a link in the main main folder with me.

B

It says um to look for details about Target, educator or other stuff, and you click this link, and then it will lead to that individual one. So, instead of now, I have no idea where to find it and there's no link. You have to really go deep into the code base and then find it some documentation there.

A

D

Think it's it's a great idea to kind of split it into uh those quick, starts and examples on the website and then have more detailed docs in the readme's and kind of cross-link them together. Probably yes,.

B

And I think the the card applicator is actually a pretty important feature right. um Otherwise, it's not really. um If you don't use it, then it won't be scalable and reliable.

D

Yeah I'm hearing this a lot that there's not good docs for the Target allocator, although, like in the past I, found some design, docs uh and I'm, not sure if they put them somewhere in the readme uh or between uh yeah I.

C

Don't think they're linked um I think that you have to dig back through like a lot of things to find them.

B

Yes, that's the problem. I think um I can help contribute if um I got I got enough knowledge about it about the documentation so for now, um I'm more interested in how to deploy and manage it, because from the current documentation that um it says it's a standalone and uh when I install the operator I don't see, is installed and also um I I try to extract all the values from the operator chart.

B

um There is no option to configure calculator, whether it like enable it or disabled or any other configuration.

C

So yeah the the target allocator is enabled under the collector's crd. Currently, so you would, um when you're, installing a collector, you would specify that you want the target allocator enabled for that collector pool.

C

C

Think that the only way you would find that right now is either via some examples that we've written or by looking at the types the tech stock go files in the repository which far from ideal um I, think uh Paul I, think that you mentioned a good idea earlier of like maybe we can write some overall getting started stuff in the website or like this is these are just some like basic examples and usage, and then we can link out to the readme's in the repo itself and those can be like independent efforts as well, so that we don't have to wait on one or the other.

B

um If I want to get more information for now, um who can I uh like talk to.

C

um You can put any questions you have in the operator Channel um and then we can like get the discussion going in there and then, if you want to write docs, which would be super helpful I can write some tickets that contain some information on like how to get started on that effort.

C

Does that sound good uh Michael? Yes,.

B

um So also another question I've been um thinking about. Is that uh so for now the Target and allocate her um get a list of targets right so which is actually leveraging the promises kind of style? So what? If people don't want like previous targets, for example, I only use Ubuntu collector.

B

um You guys probably know that there are some of the receivers like um ka's receiver net receiver. So if I only use those, um how can target allocator support that through the screen.

C

um That gets into a much larger um discussion. I think we should probably have that at another time, because I think that it's going to be that's like a lot there's a lot of logic, I think that would go into making that possible.

C

um So I think I think we should maybe set that as an agenda item that we can prepare for for next time.

C

But that's a really good, a really good point, though.

D

Maybe book an issue and with the use case and continue from there probably.

D

I'm as well happy to start working on some examples and I think there's other people as well from App dynamics that were interested in building some examples for the operator kind of use case focused on like The Collector, Auto instrumentation. So on.

C

Yeah we've had a lot of folks come in and ask us questions like internally um about water instrumentation and how Auto instrumentation works so I think it'd be great to get that more documented.

D

There is as well I think some of the official open Telemetry examples. Quite so recently. uh Is it something that we could kind of extend for. The operator have like a profile that you know instead of maybe explicit, like manually instrumenting, it would be deployed with operator instrumented with the operator.

C

Yeah I think the community demo is interested in that um I think they may already have one or two examples that use the Java Auto instrumentation resources.

C

um I, don't know where that is I feel like I did see. That recently, though,.

B

um And so, by the way, there are two places that we can find this um operator. So one is the operator repository itself, but there is another repository, it's a hand chart, so the Hampshire consists of three three folder. um One of them is a operator chart so which one do you guys recommend people to use.

A

C

uh The helm, charts right now, I believe installed, The, Collector and well it's a little confusing. The operator. Helm chart actually only installs the operator um and that's it.

C

um It sort of follows um the operator pattern for a lot of other services, where all that chart does is solely install and manage the operator, whereas the actual installation of a collector or any other crds is on the consumer. So you would have another chart that would install The Collector um rather than installing it in band with the operator itself. Yeah.

B

That's that is that's actually what I'm doing uh now so because to write a hand chart you will put a dependency right. You need a dependency for another hand, chart which is currently I'm using the open, 103 operator.

A

B

Chart um to to directly use the operator as a dependency I will have to need a m chart. You cannot directly put it into a hem chart right, otherwise you are redoing with the operator. Hencher is doing so. What I mean is you will want to implement more complicated use cases with the with hand chart, then you have to need another hand, chart.

C

Yeah you're saying that, like it's, you want to be able to uh install the operator Helm chart as a dependency of your collector installation.

B

Yes, so what I mean is I don't want, for example, I don't want to install the operator uh using like a kubernetes, a couplet Cube controls apply, slash F and then that is pretty much not possible in a in a real world way, and then you, what do you do if you um wrap it with a hem chart, which is what we have already done in the operator hand chart in in the open country.

C

I'm not sure I'd follow the question exactly um I think what we do right now is we install the helm chart for the operator and then we just have another chart that we install the collectors in.

B

Yes, so I was asking: um what's the recommended way: okay,.

C

I think that is the recommended way of doing it.

D

But then your the instant, the the helm chair that is creating the instance, has a dependency on the operational.

H

D

Or or you have to do it kind of manually from the script that is installing those charts.

D

It was a question for Jacob. Oh.

C

Oh I think you're asking Michael sorry, um the yeah I think you could have the operator as a dependency of Detroit that installs the collectors, but I think that there is a um there's, a condition with the operator right now where it actually doesn't. Allow you to do that because it requires the there has to be like um staged applies for the operator and then the collector crds, because then I don't think the operator will automatically reconcile the collectors.

C

That was at least the case when I tried to do that a few months ago, um but I'm not sure if it's still the case. Okay,.

B

By the way, it does the operator hand chart manage targeted allocator, because I did.

C

So the operator is going to manage The Collector crds, which, when you install a collector crd, will create the target allocator, which the operator will manage so sort of know, sort of yes, the operator itself. The helm chart doesn't manage the target allocator, but the home chart manager is the operator which then will manage the target operator.

B

Okay, um I guess we should add Target educator as a crd in the Hampshire folder.

C

So it's already there under the collector one um it's just embedded currently, because that's the only way you could run the target allocator is when you run a collector.

B

C

um Should we move on to Christina's agenda.

D

C

D

There's no other comments on the docs Krishna.

A

D

You want to talk about the versioning of the TV.

F

Yeah, so right now, um as we add, this is the GitHub issue that I made for it.

F

um I made one issue just to for, like a general conversation about um how we want to release and manage the target allocator, but there's kind of two things: one is that the go code like the module um doesn't currently have any versions in it. So when you go to package, go Dev, it's on v0.0.0, um so I was wondering if we wanted to fix that and version it with the operator versions and then the second thing is right: now we build and um when we build the target allocator it's off of me.

F

So when someone wants a specific version, like versions, 0.66.0 they're not actually getting v0.66.0 if V 0.67.0 has been released, vv0660 would be like the last commit before that release. So essentially each version that someone thinks they're getting for an image is actually the next one unless they're getting the latest, in which case it's like Main, um it's not actually the latest released version.

F

D

That mean yeah. Probably let's take it step by step the first about the go module. Do we do you think it's important? Is somebody importing touch the locator.

F

um I think people might if they want to write their own allocator. So we provide a way to um add to write your own allocator and like basically as we provide these options that someone can um build with their own dependencies. It's going to be more difficult for them to do that. If there's no version on the module.

D

I guess this will have to update this version during every ta release. Right kind of manually go to the module ones.

C

I think yeah. We already have the the step in the release, notes for updating the target, allocator version in the version step text.

A

E

G

C

Like pull in dynamically a version from another file in Excel mod, but then we could just pull it in and you change it in one place. But I don't know if that's actually possible or.

E

Not I think in The Collector repository in concrete um I've seen, there's always a commit which goes through all the sub modules and increase. The number I think they have just a small script, which makes this.

I

Yeah in The Collector, they use a tool that we wrote called multi-mod, which has a versions.yaml file that defines the version that each module contained within the repository should be at or sets of versions or sets of modules and the versions that they should be at then. That tool can update any requires that are in other modules and also handle adding the git tags at release time um that may be Overkill here.

I

um I, don't know if we're gonna have as many modules as collector control confirm have, which is kind of what what led us to build that tool, um but ensuring that we do have appropriate tags um and are able to track the versions of the the components there would be beneficial for for the reasons that Christina pointed out about. You know people being able to write their own either allocator components or allocators entirely.

D

H

D

Don't have any objections on putting their version uh just a matter of something a PR actually and changing the kind of the release probably and for the publishing of the image. Is it again like problem how the GitHub action is set up that is running on the merge domain? Yeah.

F

I

I think what we probably need to do is start. You know if we just released 66, we should probably as soon as we do that set the versions, conversions, that text to 67 pre or something like that, so that once the 66 releases happened, that that version is never used again and were pre-released for 67 until 67 happens, oops.

D

A

D

If you kind of deploy the head of main, it's not gonna work right.

I

uh The head of main at that point would be 67 pre right, uh something like that could work.

C

Maybe we can just bring the um Target alligator image, push into the release, action that happens for the main operator image with that I think that might solve it. I like the idea of doing the pre, though, because then you're able to also use well, and at least it's like a versioned latest, build right, like you still have a thing to tie it to, rather than just being tied to weights constantly.

I

Yeah I think, to the extent that we're building and pushing to the the GitHub image repository on every push to the targeted allocator, that's wrong. We should we should be doing it on tag or at release. um This is probably the first thing we can eliminate and then we can figure out how to avoid reusing. The prior released version for subsequent pre-release artifacts.

D

Yeah in the past, in some projects we found it useful to be releasing for every comet in the branch for kind of debuggling, and you need to check something it was. It used to be helpful, but I'm, not sure if, during the operation, somebody is using those images.

F

Yeah I, agree: I think it'd be good to have um some sort of head right that people can get the latest even if it's not uh released yet so I realized. There's one last thing on this issue, which is that right now the change log includes both operator and Target. Allocator changes, um but it's not like I, don't know, I think it's unclear or it's easy to miss. If there's a breaking change or like major changes in the target allocator, does it make sense for the Target allocator to have its own change log?

F

Would it be better to just have like its own heading in the changelog or what do other people think foreign.

E

So it's for me when I see it I like it when there is a change lock and then you see the components and then the breaking changes so which means um we have this breaking changes on top and then maybe the components Target allocator operator, Auto, instrumentation, I, don't know. And then you see the breaking changes for these components, um which makes it quite clear you go to the version to the change lock and it's on top. It was would be my personal preference.

I

Sorry suggesting a subsection for each of the breaking changes enhancements, bug fixes intersections for each of the components that are there as well.

E

Fixes and features I've I've not really an opinion, but it was more about the breaking changes so that I'd see them directly, maybe on top, because that's the things I care about the most.

D

I guess if we go with one that we should kind of, do it for all the for everything for new features and Bug fixes and everything you know just for the breaking changes.

I

The thing that the collector does that could be useful here as well is to prefix the component that is affected by the change to every line in the changelog. So if you look at the The, Collector contribute change. Log it'll, be you know: enhancements host, metrics receiver, added optional, metric, uh Splunk, HEC receiver, added et cetera, et cetera.

I

um We've got a lot fewer components here, but that could also be a way to make clear. This change affects this.

D

Yeah and then kind of serve them based on these prefix, to make it kind of easy to to read.

C

Yeah I like how the collector does the change lock to um and makes it easier with the what's the folder name in it. I don't know, there's a PR check where the author of a PR has to um commit a changelog file as well, which I think there's been a script that will generate the release. Notes with that like formatting as well. Yeah change like Jen.

C

um That thing is really useful. I think we could like use that as well, um because I think we've missed. There have been a few times where the release notes for the operator have missed PRS that were in there, because I think it's done manually currently, um especially if we're going to be uh more diligent about the versioning for Target allocator um I think it'd be great to use that tool as well.

F

That was it from me.

D

Hey thanks, Christina yeah I think. Is it all resolved or.

F

A

D

I can have some consensus.

D

I, just wonder if the change look again, it's gonna be too much overhead. I don't have experience, though, with submitting the PRS on the 2D collector collector country, where the changeable can, but.

I

D

Can try it yeah.

I

But it's actually eliminated a lot of overhead because it eliminates the the conflicts, the continual conflict that happens in the change log for people. I. Don't know that there's enough velocity in the operator for that to really be a problem. But it seems in my experience to be a fairly easy tool to use in low overhead.

D

Yeah sounds good.

C

Yeah, so it's it's been pretty easy. I think the first time that I made a PR there I forgot to do it and then the pr check just says exactly the instructions, some like what you need to make it um and there's like a template in the repo ready to go as well. um It's yeah and it makes it look really nice as well. It gives it good pretty release, notes.

D

Yeah I guess there is benefits, then we, if our format will be the same across repositories.

D

That sounds good.

A

D

Let's go to the remote configuration.

C

Yeah, so um this is going to be uh I, have a document that's in there I'm going to share screen while I go through. It is that okay, yeah um I mean like slide up in my.

A

B

G

G

C

Okay, um can everyone see my screen and.

A

C

Big enough, yes um great so um as there have been a lot of conversations in The Collector um six about remote configuration, um I've been thinking a lot about the remote configuration story for the operator um and how that might look. um I wrote here a proposal for um what it could look like um and I'm. Just gonna walk through it.

C

I don't know if anyone had chances like pre-read, so I'm just gonna give the highlights to this doctor um and it is relatively high level as well like I I didn't go into as much detail. If you read the um collector one, this one is like very detailed.

C

um This one is less detailed, also because it does less and has to do less as you'll see so for those unfamiliar um op. Amp is a really requested capability from previous conferences. I was at srecon in emea and um a PM of ours. Clay was at kubecon in um Detroit this year and a lot of people were asking about it and talking about it um and right now, the operator we haven't like had this conversation about what um the needs will be for the collectors op-amp supervisor.

C

Their plan to summarize briefly, um is to have another container um or another image called uh the supervisor, which is then going to do remote configuration for a single collector, eventually, multiple collectors.

C

um So the first part of this is um what that might look like for the operator and then the second part is adding a new object for remote configuration of operator resources um and not so rather than um the remote configuration for a single collector. This is going to be the remote configuration for all collectors in a cluster essentially, um so um this is the like very high level design where you have the operator, and you have your remote configuration uh container. This application will connect to an external SAS that runs an op-amp operator server.

C

um This the operator server can then push a map of uh collector name to collector crd um in a map uh for the remote configuration to then create the hotel, collector crd instances in the cluster, at which point the operator will go through its reconciliation flow to actually make those things exist. When the operator server API pushes like an update, uh the remote configuration would then update the corresponding collector crd, and then the operator would see that change and then go through the reconciliation flow as it usually would.

C

um This is the real configuration is just essentially like a small agent that runs in your cluster to apply um and modify and delete um Hotel collector crds, um and that that's the initial focus of this um a user might also want to connect um the you know.

C

They also might want to configure uh The Collector to use its op-amp extension, which The Collector to say, is also working on, um which you know is easily done, because you're just pushing the collector config and you just embed it in the config um to reach out to a collector um op-amp server.

C

um So in doing this, there are two main changes required. um The first is a change for the supervisor. Here is a little bit more of a detailed diagram on the supervisor, change, um I, sort of envisioned it initially I was thinking like you might want this as a separate crd, but then I was thinking. It actually is closer to something like the target allocator, where it is one to one with a pool of collectors rather than its own thing.

C

um So here I have it where it's embedded within the collector crd. um um So the flow for this um and again this is no remote config. Yet this is just for the supervisor. The operator reconciles a collector crd, which creates a supervisor config map. It creates The Collector um and then it creates the supervisor. It sets up the connection for the extension for the collector's op-amp extension um and the supervisor reads in its configuration. I actually forgot to include the external connection here.

C

I'll update this diagram, um at which point the supervisor can do its work in uh pushing and configuration and restarting the collector it'll have the fields specified in the original um collector document just one to one pretty much, but we can do some smarter stuff like automatically configure config map as a volume, and we can also automatically configure The Collector to reach out to it, via the op-amp extension, similar to the way that we set up the Target allocator in the config map for The Collector.

C

Currently, um I might stop there before I go to this next one. um Does anyone have any questions just yet.

D

How is the supervisor configuring The Collector? Is it just changing the the config or is it like? Does the collector open some API.

C

Both so what happens here is uh the collector has this extension, which will connect to the supervisor's um server.

C

um The supervisor is able to write to the collector's config, at which point it can tell a collector to dynamically restart the dynamic restart is the functionality that they're building into the Clutter right now, um so this for us would just be. You know this rather than just config.yaml. This would be the config map that we update um for The, Collector and then you'll see here.

C

This is the supervisor config and in my diagram, that is this config map um that we would hook in uh and then this connection is done via this configuration where you specify a server and the endpoint, um and this would just be a local cluster endpoint.

C

uh It would just be you know the uh sorry, uh the Pod IP or the service IP, something like that um for this supervisor to connect to uh did that make sense, and like answer the question, there's a lot of moving pieces here, so I tried to like at least separate out some of the things.

C

um So with that all good uh before I move on to the next part, does anyone have any other questions.

D

Click click. The supervisor is kind of similar to the operator right. It will.

D

Change the conflict and like the.

D

And kind of restart the The Collector uh make kind of new deployment.

C

Yeah, it does seem pretty similar I think the only difference, and so this is one of the options that I considered was embedding the supervisor within the operator, because they do like similar things. I think the reason I went against.

C

That was just a separation of concerns where um I thought that, if we embedded the supervisor within the operator right now, the way that they've designed the supervisor is so that it only manages a single collector, in which case the operator would have to spin up and supervisors for every collector, um at which point you have to manage all those supervisors, and it just seemed like tying. The logic to the operator was going to make the flow of what the operator does more confusing.

E

um What the supervisor as I recover, um you wanted to say something.

D

E

E

There is a conflict map which is configuring, the supervisor isn't it that we could have a config map, so the supervisor is basically just a reconciler and if we would.

E

Add the configuration map, maybe with a specific um labor.

E

We could just add an extra reconcile attitude that stuff. So then it's split it from the current reconcile Loop for the crd, then it's based on the conflict map, which is nothing else than another Cod.

I

The supervisor may be getting its configuration. Your configuration changes from external sources, though that are coming from kubernetes events or anything like that. So it may not be triggered by a normal kubernetes operator, control Loop that may be coming from. You know another op-amp server somewhere else or from polling an S3 bucket or wherever the the operator decides to to make that configuration available.

C

Yeah and I'm actually going to update this graph right now to make that more clear, um because I I don't think it's clear currently.

E

um But then still we could use um another Recon side Loop in the operator, which is then basically the supervisor which just reconsides, based on the configuration map with, for example, a specific label, foreign.

C

Yeah I think my fear with it is um just adding in more logic into the operator. Just do this work, um because then it would need to um I think that, like the way that they're designing, we would probably have to keep our own um supervisor up to date with the supervisor that the op-amp people will be writing, uh because artists would be doing a bit of a custom flow so rather than us having to keep ours up to date.

C

I was imagining we would just use their container and let it do the work that they're that they're going to be doing essentially.

I

I think similar to the Target allocator as well. This could result in an increasing uh usage of you know additional code routines and memory inside the operator. As you add, more and more managed collector crds. So breaking that out enables you to scale the supervisor alongside the things that it's managing independently of the operator controller.

D

Maybe the part I don't understand: is the supervisor configuration the config map, oh yeah, so it's created from the the operator creates that, but then this third parties, how would they kind of change this config map.

C

So I think this should be done in the same way that, right now you can configure the target allocators config map that it expects um via the collector crd So within a collector crd. You would specify that you want a supervisor um and then it would have. The crd would contain each of these fields, so capabilities so annoying to read yaml in Google box um yeah. This looks terrible, but you can see. Capabilities is a key. The exception mode configuration report. Spec did all of those would just be Fields.

C

Under The, Collector crd within a supervisor.

A

In the same way, that.

C

We did the Trigon allocator right and then that would configure the supervisor's config map um I think it would be a relatively it would be pretty static, um but um still I think we would still want it all.

D

Right I was expecting them, but then like some like third-party component, would go and change this conflict map kind of by.

C

Hand, oh I, see I, see um yeah, that's not the case. This is in the config map that would be changing. It would only be um the collector's config map that the third party would then go and change.

D

C

Cool, so uh are there any more questions before I go to the next one, which I think is maybe the more interesting.

A

C

One of these, no okay, so the next one is the remote configuration for the operator.

C

um This one I think could reasonably live within the operator itself, but for the same reason of like separation of concerns, I added as its own container, but this actually like there isn't much of a reason that it doesn't need to be within the operator.

C

um The reason that I wanted it to the reason that I thought it should be done. This way um was that uh so to explain the flow first, um a cluster user would create the remote configuration crd, which is its own thing, separate from The Collector entirely.

C

um That would then reconcile to create a container which runs the remote configuration um the operator remote configuration container.

C

um This would contain an op-amp agent which talks to an op-amp server run by a third party. This remote configuration when it receives anything from its connection to create a new collector. It would then talk to the kubernetes API to actually create that crd, at which point its job is pretty much done. That's all it does is just do the crud operations um against the Kube API. It doesn't need to talk to the operator whatsoever um and then the operator is then going to do.

C

You know what it usually would do, which is just reconcile that crd. um The reason I opted for this was so that there are no actual changes uh in the operator's logic. It just continues to do what it already does and we have an external um objects which can do the creation of these crds um I. Think if we were to move it in band. The thing that I would fear is what the actual creation flow for The Collector crds would look like um I think doing it.

C

This way has the benefit of you being able to query The, Collector crd um in your cluster, whereas I think, if you did it within the operator, if you did that it would create a loop or it wouldn't create it at all. Right, because the operator would need to uh talk to the cube API to create the crd and then it would go and reconcile it and I think that that's kind of a weird like um of of operations, but maybe that's me being too um separation of concern. Anxious.

I

So is the idea here that the remote configuration component would be an op-amp clients whose configuration that it receives from the op-amp server is a list of collector crds to maintain in the kubernetes API.

C

I

And so then, this would be analogous to doing something like using gitups and Argo CD to maintain your crds in the cluster as well.

C

Yeah yeah pretty much. um The only difference is that, rather than something like Argo for get UPS, which is um pull-based, this would be um like bi-directional socket based, so that it would actually receive the update rather than um go against get state to generate it.

I

And presumably also the the op-amp server side would then be able to have awareness of I expect these collectors to appear here. Here's what I've told the operator to do. Here's also the now I'm, seeing the collectors reporting to me and and here's their status as well, or here's a set of collectors that are supposed to exist, but aren't.

C

Yeah exactly so um that would be reported back via the op amp server, the op-amp protocols, effective configuration, um which basically says this is what's actually running, and so um you could use that to report back errors um like validation errors or actual, like runtime errors, as well um for the configurations that you're running or attempting to run.

D

H

D

This scenario, we will totally avoid the extension, the bump extension on The Collector to connect to the reloads.

C

Yeah exactly um this, the idea here and I called it out up here um is the like paired down version of that diagram um is, you might still want the collector to use the op-amp extension, but for read-only operations, where it just reports its status and health, but you would actually disable the supervisor capabilities, so you wouldn't want to use both the remote configuration um and the supervisor at the same time, because they essentially are doing the same thing.

C

It's just you're, letting one you're letting either the operator do all of that configuration work or you're, letting the supervisor do that configuration work.

D

I, just wonder if this extension in the future will support some kind of apis. That will be that we won't be able to support this kind of architecture.

C

um What would that entail? You think.

D

G

One of the things.

A

D

A

C

I couldn't see.

D

So maybe in this architecture the reconciliation will take more time than the other reloads from the previous architecture and I. Don't know if that matters too much in kubernetes, but it might Maybe.

C

I think yeah, that's a good call out around the performance of this um they're. Definitely yeah, sorry or like me,.

D

Like partial configuration change,.

C

So uh one of the things that I think is coming in the future um for The Collector, so separate from all of this remote configuration work, um something that is coming in The Collector is dynamic, restarting um so that you could actually restart like individual parts of the collector pipeline.

C

um I think what we could do in that scenario is just change, because someone who isn't using any of this remote configuration would want that feature regardless. I think that's something that we could integrate into the operator to do where it would just tell the um tell the collector that you have a config change, restart the pipeline.

A

um And then again it.

C

Like looks kind of weird in this diagram um like if I were to put it into this specific diagram but separate from it, I think that that would be good as a feature for anybody using the operator, not just a remote config user.

D

And is then the collector responsible for kind of realizing, which component needs to be restarted?.

G

I

I

What would be the recommended order of operations for building up the system? Should we wait until the collector supervisor op-amp integration, and all of that is well established, or should we be trying to get moving in parallel with them before they've built out? Much of that.

C

um I think we can do this separate from them building out the op-amp extensions, um given that this actual, like remote configuration, um is like relatively simple and can be done using only the op-amp go sdks that already exist, um I think that we could start on the I. Don't know if I put it in here in our internal documents. I have like the order of building um which I can add to this, um but essentially the thing that needs to be built.

C

First is just this remote configuration container, um which can do the op-amp agent connected to a server um and can talk to the Kube API once that is working well tested and we're happy with it, um then we can have that as part of CI CD building, and then we can add in the necessary crds for the operator to actually reconcile and create it.

C

um The benefit of this is that it is isolated from all of the work that the collector is doing, um and it can work totally separate from that as well.

D

Yeah I like this design. Thank you.

C

um Are there any concerns with this model that anyone can think about? Is there anything that, like I, don't know, I, think Paul, but you're concerned about, um like the latency of it or like the time to reconcile essentially is legit, um but given that this functionality is I'd say like pretty different than what the op-amp folks are currently doing um and I think it is, as Anthony said, like closer to a get op style thing and less a um um yeah less of a like Dynamic, reloading scenario.

C

um I think we definitely should in the future, when Dynamic reloading is possible, figure out how we can use that in the operator. But I think that this is like I think that the latency of this is acceptable initially as long as it's not like 30 minutes, because that would be ridiculous, but I don't think it takes 30 minutes.

I

Well, I I also like that. This is a good illustration of how op-amp can be used for more than just managing a collector. You know it that's one of it's design intents, but it's that's. Why I've been fairly closely coupled.

C

Yeah I reading through the spec, um it was actually interesting that they only mentioned the collector a few times, and so that was um what made this idea sort of possible was just rather than having the effective configuration be um for a single collector.

C

um It actually supports doing a map uh for the effective config, so we were able to do any amount of collector crds using that.

D

A

Could maybe like.

D

A

As well, the instrumentations here with.

D

This protocol I don't know how the protocol looks.

C

Like that um yeah, we definitely could um I think the thing that we wouldn't be able to do is the um we wouldn't be able to do the header or the um annotations on pods.

C

um Well, so there is one other feature that uh well I'm, not no not worth talking about it right now. I'm gonna focus.

H

D

Yeah, it's great to see this we're slowly kind of extending the operator functionality. uh We are at the end of the hour, I'm, not sure if we have to finish uh I think we should but I'm, not sure if we have to uh and I don't know if you want to continue Jacob.

C

D

C

Is it um I think my the one thing I just want to do is like next steps.

C

um My plan is going to be open up an issue with the things that need to get done um for this work, um and then we're going to be our team is going to be maybe starting on this uh in January.

C

um So that will leave some time for um other folks to leave comments on the document as well.

D

Good perfect thanks uh and.

D

Yeah I think this is old and from the agenda and the meeting is scheduled to be every two weeks. uh So let's try just to yeah to meet in actually in two weeks. Is it Christmas week or not yet I think it might be.

D

G

D

um I should be working 22 December, but if there won't be many people, we can cancel under meeting January.

D

Right. Thank you very much and uh yeah see you on the on the community. Goodbye.