Open Telemetry Uncategorized, 1 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-08-01 meeting

Description

Open Telemetry Meeting 1's Personal Meeting Room

A

B

Hello, how are you doing good how about yourself I am doing fine.

C

B

To hear learning rust, it's hard.

A

Learning learning rest yeah.

B

It's different, it is it's.

A

Very different, well I have to say uh that the worst is Airline.

B

Yeah um I looked at erlang code I, ironically, I have to look at both codes to do what I'm to do what I'm working on, but I've I've delegated I've asked Christian to help out on all the erlang stuff. um That's his domain. So.

A

Yeah but but rest is um interesting, so that's it.

C

D

Hey everyone good morning or good afternoon, hello.

A

ah I I forgot to open my camera. I need to share my shirt.

C

What's your shirt, it's.

A

The Jager mind stuff.

D

D

Yeah cool I think we have a decent amount of people. Welcome back Miko I hope you had a good vacation.

C

Yes, I did thank you. Yeah.

D

I always kind of assumed when the Europeans stopped responding. I'm like okay, it's the summer, I'll see you some in a month or two.

D

um But this is our agenda for today feel free to add any items or suggestions to it. um We'll start with some feature: updates uh Austin, had an item or two about kind of improving uh the vendor Fork experience in regards to merging um I kind of wanted to discuss baggage as well. So we could come up with either like a some sort of baggage strategy, so we can actually get that implemented or maybe just we hold off on implementing baggage in the one I'm, not sure.

D

We've really had like a strong like use case for it so far and then potentially wanted to cover a couple like build and Dev quality items here, I think erase like an environment variables issue in the slack at least, and then I'm not sure if Michael was able to make the call today, but he's been doing end-to-end Services I'll, see you hey, Michael um he's been doing. Some ended in Services uh for testing and I.

D

Just wanted a quick update on that as well, and maybe if anyone had suggestions or feedback on the testing strategy, all right, sound, reasonable cool.

C

E

uh We want to roll into front end.

D

Yeah, we'll start with that Oscar. Can you give us an update on how the front end's going.

F

Hi guys uh yeah, um so, as you saw last week, we opened uh I opened up VR. uh It has a lot of changes of the front end um and right now, I'm going through the comments.

F

um What I'm fixing right now is the things regarding the docking compose uh ports, so I think yes, I think we should just keep one environment, whatever environment file and we shall not uh be able to require to expose the the ports for each of the um other services. So yeah I'm, going through the comments I'm going to be fixing some of those for others. I will add some responses, but at the moment uh I think there's some work for me to do. The other thing is that um I will be checking.

F

What are the validations that are failing, um since that there are two that are failing, uh so yeah I'll be checking that today, I I guess. The third item is that currently, the the pr only includes the design and in the features like for having the the web store functional, uh but we are not including either instrumentation or testing yet so I guess that will be kind of the follow-up follow-up uh things that we need to do after we make this that they are.

C

D

Do you have any uh specific blockers or concerns you're running into right now, or is it just working through the pr comments.

F

Yeah um I, don't think so. I've I'm just working through that as soon as I find something I will. Let you guys know on the selection today.

D

Yeah um yeah great, uh do you have a sense of you know? Of course the pr review will take a bit of a while as well but from a Dev sense. You know a general idea of when you think you'll have most of this wrapped up by.

C

D

Thanks for that, um let's talk about kubernetes, real, quick um Dinesh. Have you been able to work on that? A bit more I know the helm, charts got merged, I. Think I, see Tyler from the the homesick too, as well.

C

Charts going forward, okay.

D

And now we have I think the home chart merged right now, at least it's in the uh Hotel Helm repo, so Tyler I guess we would take your guidance here. What's our next step or does anybody else um aware of what pengh was going to do.

G

Yeah so I I saw that the helm chart was, you know the pr came through we merged it. I wanted to show up and say hi to everyone, because I haven't been involved in this community yet, but now I'm tangentially involved because there's a Helm chart over there to maintain um I love. That idea like adding the helm chart over. There is awesome. uh We want to support it.

G

um We've got uh whoever the contributor was he's a co-owner on the chart now as well. So we're going to be looking to him to kind of keep that thing up to date. So.

C

G

New versions of The Collector come out or new versions of sdks come out or whatever, and and there's things in the helm chart like default versions or whatever that need updated, we'll be looking to him or anyone in this community community to be kind of the ones that are driving that those updates.

G

It's a similar strategy that we use for the operator um the operator chart.

G

um Okay, there are some CI CD things kind of on our side that will take care of for the uh the new demo chart like getting some examples out there, something that we want to do. We have that for the open, Telemetry collector and we Auto generate examples, and we keep them up to date. Improving CI scenarios. That kind of stuff is something that we can handle, but like the day-to-day like bumping, aversions and whatnot we'll be looking to this community to keep up to date.

G

If anything gets like really out of date, we'll take care of it. But um since this is a community, that's working on the the demo, you'll be the ones that are most knowledgeable on, like versions for all the images and all that kind of stuff.

D

Yeah great makes sense to us thanks for your support and also feedback, um Austin I think a comment came up in this: the Sig um slack channel. So we're talking about adding like the open, Telemetry operator or no operator for V1 I think you had a certain opinion on it, but wanted.

E

D

E

Think the the two kind of I think this relates to like it's a little confusing that the helm chart is in a separate repo.

E

Like if you just go to the you know, my understanding is that Helm chart for the demo was merged right.

D

E

So if I don't know that and I go to the repo right now, I have no indication that this is installable on kubernetes, right, yeah.

D

The dogs for sure yeah.

E

Yeah- and the second part I think is just you know: if we're trying to like the operator is an otel component and if we're trying to if this is trying to be opinionated about um how you should deploy open, Telemetry and kubernetes, then I feel like the operator would be the correct way now I'm totally willing to say that. Maybe we push that because there's like operator features that aren't there yet or we need work to land for, like the kubernetes monitoring side of it.

E

um I just want it to be like we had put in the requirements that you know we were going to be using the operator for this, and so, if we aren't in V1, then I just want that to be an actual decision.

D

C

D

B

I, like the operator rope personally, I think it's. um We should be looking at it doing a future uh and yeah. It allows people to expand upon their collector, install with ease.

C

G

I would say, I agree, uh I would say best practice for running a collector and kubernetes would be the operator Helm chart is like second best to the operator, so there are definitely people who'd rather use a Helm chart than an operator. Maybe they just feel more comfortable with it or whatever, for whatever reason so I'm glad that Helm is an option uh but yeah. If we could like using the operator for the collectors, definitely best practice.

B

Is is there a way and part of my naiveness at the helm chart, but can we have a Helm chart say check to see if the operator crds are installed and if not install them.

C

G

I, don't think helm is that smart, it doesn't have it's just a templating language, so I don't think it has the ability to like check for the presence of something already installed in kubernetes.

E

I mean I think Helen could probably run because installing because installing operator is doing like manifest applications like there's, probably some weird way you can do it I, like I, said my I think we've. It seems like there's a general idea that, yes, we do want to support this, but we also it's- maybe not the top most priority. So that's fine for me. I think, there's also some stuff that's coming like in the next. You know, weeks to months that will really improve, like the ability of the hotel collector to monitor you know, kubernetes deployment.

E

You know cluster itself and support like service discovery.

E

So maybe this is something that we kind of put on a like V 1.1, or we we planned for later in the fall.

D

Yeah I also feel like we can potentially make it a checkpoint item because I feel based on where we're at at the end of August. For that September release, we'll know how much we have remaining and maybe what needs to actually be completed. We might be at a point where it's just maturing the metric story, a little bit and then also maybe add in some finalization, so I think it partly depends on our development Pace too kind of throughout the next month, or so.

D

um Okay, so near future version, if not V1 I'll go ahead and make a point that we'll um kind of just have a decision point I guess.

D

um Okay, uh maybe.

C

Maybe one question still there that that okay, this is fine, but what about then the documentation? Do we have some plan? I mean. Currently it says that um only the docker compose is supported. Foreign.

D

Yeah, we need to make an action item to update the docs to actually point to the home. Chart and I can um probably open a PR for that uh I'm sure um yeah I'll take some reviews on if I put the right information there, but thanks Miko uh feature Flags. If you already have a quick update on the default feature, fog set I've been able to work with Tristan.

B

I have been able to work well, I started off. We went back and forth a few times earlier last week. um I have not heard of him since uh he's also replicating the same issue: I'm replicating.

B

So it's something with the erlang Elixir service, responding to grpc I, put up my PR out there draft assuming it's going to work soon, um so just for an hour to draft PR for the product, uh I I I've got one kind of in works here for the shipping Service shipping is going to break on the same product ID, but uh when it's trying to get shipped to Canada um and actually shipping service is going to induce a latency and instead of breaking so it'll, eventually return it'll just be a very long latency before it returns.

B

Meanwhile, the product flag will return an error when you try to pull up a specific product from the catalog service. If the flag is on so kind of how that's working uh worst case is we communicate the future flight service in HTTP? Only that seems to work I just wanted to keep jrpc going, um so we have a fallback there, but I I was just trying to hope you're interesting to get grpc figured out for her, like that's all.

C

A

A quick question on that uh yeah: uh are we also adding some load on this um on this Canada shipping.

B

um So what I did was uh inside of the locus file, I added I, think there's like eight or nine different addresses there and randomly the the locus file will pick one of those addresses, as well as whatever product was picked, to get added to the cart. So it's gonna be very random on the current load. If you want to increase load, uh you would just change the number of users um to be bigger. We can make that a feature flag as well.

B

If we want to have a feature flag to introduce additional load to the platform and have that be read from the um from the load generator itself, but that wasn't something I was targeting to do.

A

Awesome no, but uh just adding the the address. There is already a good thing. Awesome, yeah,.

B

There's uh I think there's like 10 different addresses in there. I got to go count them out for you I'm staring at it right now. It's uh nine different addresses I, have listed um I, really just picked nine, really popular tech companies to try to keep this in some form of whatever um yeah. That's it.

C

B

Indicating anyone in Shopify, if anybody's wondering.

D

The Interlopers.

G

um Okay, I didn't.

D

See I'm not sure who put this uh otlp export config issue didn't actually.

E

Yeah I put that actually awesome. Okay, um my so I I asked this in this thread, but I do want to make sure. Is there actually an implication to setting metric and Trace export variables independently of each other?

E

Unlike the SDK side,.

B

I think it so I brought this up because there's a little bit of inconsistency going on and the way I always envisioned. This is there's three different. There's four different endpoints or variables right, there's the standard, one that has no we'll call it middle fix and then there's the traces metrics and logs one. uh If you wanted your SDK to send to a different endpoint otlp, you would specify those environment variables.

B

In our case, all sdks are pointing to the collector so I brought up. Why are we using the the independent ones um that introduces a maintenance issue? So eventually, if we decide to add logs we'll have to make sure we have logs environment variable to everything as well, um but that was the only one and there was I think it was the front end was configured differently than all the other services. So I was just let's bring up a consistency issue with that environment. Variable there's, also a few more environment variables that are probably need.

B

Some cleaning like hotel service name is one that really stands out to me.

E

Yeah so I guess my my only point there is I I do think, there's an argument for like being explicit.

E

I just don't know if there's a if there's any actual, like SDK level implications behind that um in terms of performance or or other config things right, um because I do think that the difficulty here is again like. Are we being like trying to be consistent about like if this is a reference implementation um shouldn't we be kind of specifying? Like oh yeah I know you just use one.

E

You try to do the thing that is least surprising, which would be all otlp. Data of all signal types goes to the same environment variable. um Does that make sense.

B

Yeah I I'm, indifferent. If we want to do the individual environment variables or use the generic one, individual.

E

If we did, if we didn't do the catch-all, how many things we need to change.

B

um I, as a stance now I think fewer things would need to be changed now but, as we add metrics we'll have to make sure we add an environment variable to each service for metrics. As we add logs, we'll have to do that again for logs.

E

All right, so are we fine doing a individ, explicit endpoints, then.

C

Let's start at the explicit and see if it doesn't work out, we can again change to one. Nothing is like it's getting started. I'll.

E

Put a note on that issue, then we that was disgusting. We went with explosive.

C

We can come back to it if you feel it's too many videos, I guess.

B

All right, thank you. Yeah, it's not more variables. It's just um Docker compose will reference three now, instead of one variable but environment wise we're only defining them a few times um while we're here, we still need to do cleanup on the environment variables. Otherwise, if you look at the current environment file, um the way it's defined is we Define the traces endpoint, and then we Define the generic one to be based off the trace this one.

B

It should be the inverse if anything and um there's other aspects, I I think we should be specifying service name instead of adding it as a resource attribute.

D

So when I open an issue to track updating the environment variables, I mean it's put.

B

D

Under this action items for now that someone could just link to it, for me, I'd appreciate that.

C

What do you mean by uh studying service names instead of a resource attribute.

B

Right, there's an Hotel underscore service, underscore name environment, variable that you should use to set your service name. uh You can set it using a resource attribute for sure. But okay, um if you specify service name, that's the overrider that overrides everything um I'm all for I'm all for introducing usage, a resource attributes as an environment variable perhaps for something more resourcing. Maybe we just say you know: Source content equals demo and it's or something like that right, like an attribute, just call it demo as a value. So people see its use case.

E

I mean I would imagine that the resource attributes are going to be things that are added on through the like, collector and or like.

B

E

Detectors and stuff like that right.

B

Yeah, when we get to that point, when we start adding more collector things, we'll probably do yeah I.

E

Mean I I, the only thing I would add is that we also need to make sure that we're setting version attributes properly.

B

That should be a resource attribute for sure.

E

Well, isn't the service version also um something you can set through MVR.

E

Was that a yes or a no sorry.

B

um We could we're not today. Oh.

E

I'm actually I'm a little surprised. I thought that service version that I know you can I know that like otlp or hotel service name is a like. That's an actual project level. Environment variable right.

B

Yeah, that's for um defined by the at the hotel, spec level, yeah.

E

B

E

Service version isn't even though no it seems like a spec miss all right. Well, we do need to set both.

D

Yeah we can, we can cover this topic a bit more sounds like um we could probably take it to GitHub and discuss further there yeah um Austin. Could you give us um kind of an overview of what you were thinking on this interfork experience.

E

Well, yeah I, don't actually have a I. This is me looking for Solutions, but this is the problem. I'm encountering um and I think other people will start to encounter it as they make Forks as well.

E

When you are my the thing, what needs config changed is the same thing for everyone um like if you, even if you're kind of in the simplest use case possible and you've got, you know your honeycomb right, and so you can do otlp export um and you so you you need to change. Add a you know. Even if you want to keep everything the same, you have Jager and all that, but you also want to export out to honeycomb. Then you have to add another otlp export with your access tote.

E

You know your API key for light step. You have to have an otop export with your license token.

E

Those changes make it impossible to mer to do a clean, fast forward, merge from Upstream like what I have had to do. Every time for, like the lightstep fork, is actually check out the repo itself um and go through and like do a manual merge, because GitHub can't figure out how to like merge these changes together because, like especially if you're overriding like, if you remove Jaeger and you replace it with like a vendor.

E

So my question is: is there a way that we can make things composable like at least on a config level, so that there's some sort of like build or something that you know like the system of like where you can drop in? You know an override file, or is this something that we just kind of want people? What worries me is that at some point, the customizations to this get significant Enough by Downstream vendors and downstream users that it becomes difficult for them to keep it in sync, with Upstream Hotel changes and I.

E

One would prefer to think about this now, rather than later,.

E

You don't have to come up with solution today, but I want to know people's radar.

B

So a way to specify at least collector configurations.

E

B

That's the most important.

E

C

E

um Although I can see this also being a problem for well I, think as I I think it's maybe some part of this is just saying like hey here's our opinion about how you should do stuff right, like if you're honeycomb and you want to add in some something that shows off a honeycomb feature or your light step, and you want to show off a lightstep feature.

E

Then we can say: okay, cool, then the way you do this is by like creating a new thing and kind of adding it in here rather than modifying existing stuff, and then everything kind of can continue to be updated very easily.

E

um But to do that, we would need the two things I can think of are one The Collector configs and two the docker compose, and they eventually like the kubernetes, manifests or whatever um those are, the the yeah, the config and the deployment stuff seems like what would need to be.

B

I think standard kubernetes manifests are harder than a Helm manifest because we just add a values file for home. That's actually where Helm helps us out.

E

B

E

Like add in another service.

E

B

um Adding a whole new service is something different that that's uh I I'm, just talking about modifying existing yeah.

B

um We would almost need kind of a build that would look at the vendor specified file and merge it with the default file and then run our things.

E

Right that was where my that's, where my head went.

B

um I mean it's: the ammo yeah.

G

There might be a way to take advantage of the fact that the collector can take multiple configs and it can take configs from environment variables.

C

G

B

Can The Collector take multiple configs? That was unaware of this yeah, so.

G

We can definitely take multiple configs, huh so if you pass it, multiple configs it'll, merge them um and those configs can come from the file like we normally do. They can come from an environment variable or they can come from.

G

I forget what the last one's called: it's like a URI path, something like that! It's not it's not optimally used, but in this instance the command might be able to be updated to always try to get a config out of environment variable. If it's not there, I, don't think the collector carries it. Just moves on to this life, but if it is, there it'll merge it internally. Yeah.

E

G

E

Was gonna say you have the collector, you pass it a path to a directory of config files.

G

I, don't think I, don't think the collector knows how to take all the configs in a directory. Here's a dock on with three different types: I, don't think it takes. It knows how to like read all the config from a directory, but if it could, that would be pretty cool and if it doesn't might not be a bad idea to make a proposal to The Collector community to support that kind of idea.

A

Well, what if we keep one default config and also try to pass a different one in uh in.

C

A

Path and then the vendors just say: hey just add this file in your in the path XYZ and then every time the collector starts. If the file is there, it's merged, if not it, ships with the default, it.

E

A

Like a nice solution, actually the.

E

Only question I would have is what happens if you tell it to merge something that doesn't exist? I.

G

C

G

Get upset I'm pretty sure like if you use the dash dash config flag and you point it to a file. That's not there, I think it I.

E

Don't think it likes.

G

E

um What if it's an empty file.

G

I, don't know not tried that scenario. It's a good question could be. It wouldn't be too hard to experiment with.

E

Yeah, no, that's that's one option that might actually be good enough for now option it.

B

Seems like it's just adding an extra dash dash config option, and maybe we just apply an empty enamel that does nothing and um yeah.

G

E

Long as it was kind.

G

Of a valid field, sorry.

E

Yeah really, my only goal here is that we can have an opinion that we can, when the we can say like hey, if you want to customize this, here's how you should do it so that you can continue so that you're not making changes of things that are going to impair your ability to keep this up to date.

D

Yeah we should put together like um uh Fork guidance or something like that, because you already have you know like running it locally and bring your own back end and stuff, but we should kind of mature it to the same level as like our contributing page, where it has a lot of details on like how to get started, and things like that. We should definitely entreat a vendor, as just another user to onboard almost so.

A

Yeah, that's pretty cool and I didn't know thanks Tyler.

E

C

Is yeah no problem.

E

Not aware, thank you of this. This is very handy.

B

Yeah I'd be interested to see what happens with the merging of two pipelines, with the same name, which one wins in that case, the second one.

G

Wins I believe. Okay, if.

C

That's the case.

G

um But I think it. It I think it uses the same kind of templating strategy that Helm uses, which is right to left. Right, takes precedence, uh so it'll merge, dictionaries and then lists are completely overridden, which actually might be a problem uh so like. If you yeah, that's, actually going to be a significant problem.

G

um If you got two configs and the right, one has a preference on the service uh like if you need to in like inject like a a new otlp exporter into one of the pipelines or all the Pipelines I, don't know if the collector will nicely merge those or it'll if it'll just take the list from the second config like Helm, does so that's something to experiment with I'm, not I've not tried it myself. Hopefully it.

B

Knows how to merge them if it.

G

Doesn't know how to merge them will have a problem, probably.

E

Well, this seems like a good place to start at least and I. Think this plus the fork guidance will.

A

E

I'll be useful.

C

D

Thanks for bringing that up, Austin, that's really a thing to call out definitely will help what kind of quality of life going forward. Maybe I think about from the beginning.

D

um Okay, on to baggage, real, quick, so um I know we have baggage, as if you want a requirement. I haven't really heard of any good use case, suggestions so far for baggage and kind of uh for a lot of services like the kind of outstanding item, so to speak, so I love to discuss it a little further and if we want to you know, either develop a baggage strategy up front or maybe push it off until like the 1.1 Plus or something like that.

D

Any any thoughts.

E

um um I thought we had come up with, like the things to put into baggage already.

C

uh Can you linked to something.

E

Yeah, let me look in the issues.

D

That would be super helpful. I must have just missed that one.

E

um Yeah, it's issue number 100, being one zero, zero.

E

And the baggage attributes or session ID and synthetic requests.

C

Oh cool, okay,.

D

Never mind well, this is this is good to have them uh Juliano I know you said you might potentially look into baggage, but you would um potentially need some help on that um any like issues. Have you been able to look into it at all right now or just.

A

Yeah I think with that information that I wasn't aware of it was there maybe I came in there started playing around yeah, okay, yeah.

D

I'm glad it Justified me yeah.

E

It's intentionally like we didn't, we didn't uh come up with a ton of great things, but I think these two. This was kind of what we came up with in the planning.

D

Yeah I think this is a reasonable out of the box and then we can mature from there. I don't know if we need to have a super mature story.

E

Yet I I would say as a just to flag this for um Oscar I. Think the front end does need to set the session ID and baggage. So we should check the front NPR for that.

D

Well, I think you mentioned that the intermentation isn't part of it yet so that would need to be added in a second PR um potentially so what.

F

uh I could yeah that's a a good question. So I saw I, saw the session ID. So is that something that is completely generated by the by the browser side by the front end.

E

A

Id, something like that: I.

E

Can't find that I believe it does.

C

F

So yeah, what I'm doing on the pr is that um I'm trying to read a session ID cookie from from the browser and if it doesn't exist, I we're generating a new uid and setting that into the um local storage and that's what we are using for to show on the on the UI or if I need to do something differently. I can I can just copy the logic from what we have on on the Go version.

A

So and we have the method on.

A

F

Awesome, okay, yeah I can I can uh at this this project as well. uh If it's not required for the initial merge of this uh request, I guess I would prefer having that in a separate PR. If, because it's already too large right so I didn't want to include more stuff, but if you want I can also add it there I, don't know what will be your opinion.

E

uh I just think we need to make sure that if there is stuff, if there's functional aspects of the pr, that's missing like that's fine for now, but we need to make sure they're enumerated.

F

D

Yeah same serious in the voted me I. Don't think it's a big deal to not have it immediately but yeah as long as we call it out, that's something to close and kind of have it on the to-do list. For you know very soon after we should be okay, yep, okay, cool we're going through most of the agenda awesome, uh so environment variables, I, think were already covered by Pierre and all of us kind of further up uh Michael.

D

Do you want to give us kind of a quick overview on what you're doing with our service testing for a quick update.

D

uh You're still muted Michael you'll realize that.

D

Okay, Maya stepped away so um great.

A

I I saw his uh Les PR, actually on the on the tests, and there was there was a a board that was changing, but that part was already being used by the teacher plant service.

B

Yeah yeah, we definitely have poor conflicts that we should clean up. Yep.

A

And uh and uh I kind of uh eradicate the the feedback in the the PR but yeah we should I think I think that was the last uh service that was missing and as soon as this is fixed, then uh I think it's just a matter of adding the tests into the GitHub actions.

D

Cool yeah that sounds great um awesome. Well, I. Don't think we have any other top of line items unless uh someone else has something. I didn't want to point out. We did hit 100 stars on GitHub the other day, I'm, not sure if anyone saw so I thought that was kind of an exciting Milestone many many more to go hopefully, but pretty exciting. You know from the last three months to get to the day so go off essentially.

D

uh But yeah with that uh any any last items I always hate asking those because I know people get a little antsy, but.

D

It seems like a no cool, well, I hope all of y'all have a great rest of your day or night. um Thanks for joining Tyler nice to meet you, we appreciate your help, Mika welcome back and then yeah thanks everyone else too. Oh.

C

D

C

You all thank you very much. Thanks, bye,.