Open Telemetry Uncategorized, 9 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-01-09 meeting

Description

Open Telemetry Meeting 1's Personal Meeting Room

A

B

Hello, how are you fine, how are you good good, happy, New, Year, Happy, New Year.

C

C

D

Hello Juliano good morning, everybody else. Let me go damash James, hello, Centric,.

B

Hello, hello, hello,.

E

We were in the in the AI thingy now here.

D

Yeah and it's uh um I gave that app several dollars of flying, probably about twenty dollars in total, between myself and my partner, a few others so.

C

C

E

We do have a nice agenda, should we get going or.

F

Yeah true, it's a like 10 topics.

E

D

We probably should you want to leave Giuliano.

E

Yeah I can I can share my screen here. One sec, there's Carter.

G

Hey guys, sorry I'm late, hello,.

G

Give me a second.

G

How was y'all's holidays anything fun.

D

I played a lot of video games.

G

D

Yeah I I got the kids of PlayStation 5 and I ended up putting more hours in God of War than both of them. Combined.

E

Awesome all right, I finished it uh two weeks ago, so yeah I understand it.

G

I've been trying to play out of the ring, but I'm not good enough for it.

C

G

E

I heard it's pretty good, but yeah I haven't tried.

G

Okay, um yeah I think we have most people here, um so we can go ahead and the first item I, don't think the calendar switched to bi-weekly. So let me get that updated bi-weekly, um so you won't be meeting next week. That's just a quick FYI um and then just kind of I think we have like 10 agenda items, I'm, not sure what do we want to spend our time on First and what do we think will take more time to discuss um because I would love to talk about resource detectors quickly with their documentation?

G

um I know. Giuliano has a great uh topic on like image size for spill time. But um what are you guys looking to discuss.

F

Maybe the docker topic yeah. It has been mentioned a few times, but never really discussed.

G

Okay, yeah: that's.

F

D

I think I'm at the mindset of we need to keep Docker running for the developer experience, though we may be just forcibly having to limit some of the functionality we get out of the docker experience, and if you want the full-fledged thing you need to deploy this thing to kubernetes um yeah like a local KH thing, particularly if we want to add support for the kubernetes operator.

D

Clearly, that's kubernetes only thing um and I I think what we need to do is say we're going to keep Docker there, but it may not have the full feature so and how we Define that I'm, not sure um I think if.

E

We, if we decide to to to use the operator in technologies that we already have by technology, that we languages that we already have the manual approach. I think it's fine. We just need to state that somewhere, because most of the comments in the in the discussion were in people like um against dropping Docker, so it I've definitely think we should keep it because it's pretty easy to to spin up and play around locally.

E

F

Maybe one thing here: we could discuss that if we let's say provide the instruction that this is how easily you run kubernetes locally on top of Docker or something like really easy can come up with something like that. Maybe this this question is actually easier and it might get forward in the that's. The next step.

D

I feel like there's a million blog posts that do that already. Are we just Reinventing the wheel here.

E

um Yeah we have, we have a lot of tools like kind uh K, zeros, k3s uh mini Cube. They are all easy to set up locally, but uh the problem is that once we do that, then we kind of force people to kind of know how to navigate kubernetes. And it's not something that everyone knows.

F

But I'm thinking, how do we expose all those uis that that I think we we don't have I, don't know those objects for Ingress and load balancer and those things so the users are, they are kind of on their own.

E

Or I think the the envoy would would handle that.

D

Yeah I think moving your Envoy simplifies uh UI exposure for sure.

C

G

Yeah my One requirement for replacing Docker is always, but we must have an equivalently easy experience to talk or compose up and it has to be successful if there is a way to do that, kubernetes I'm more than open to it, but I think you can notice all the PMs to disagree with removing Docker, so so I think for the customer.

G

The customer preference is to have something like that. So if we can't provide it, then I don't know if we should even spend time trying to replace it. It's my thought.

C

D

I think we'll definitely need docs around here to Showcase like right now we're Future parody, but we're talking about adding the kubernetes operator and when that comes we're going to probably remove default. Instrumentation from accounting and fraud for sure or people will ask questions.

G

I think that's fine, there's also um I, don't know if it's on the list right now, but there's a request to be able to disable open Telemetry instrumentation as well. So maybe we could do something through like an environment variable to where the kubernetes version has it disabled and has an operator in place or something like that.

E

Second blast in this list there in this page: yes, there you go.

G

ah Okay, awesome thanks Juliana what.

H

You wanted to say the answer to.

G

Request from Severn.

G

um They may potentially have to do a little bit of work to set this up, though,.

H

Or if we only targeted kubernetes, then we would only then you could only use the automatic the operator.

G

C

I just missed uh yes, I'm sure it's.

G

Just part of that discussion, a lot of people won't vote for docker.

H

A lot of people vote for docker, yeah, well, I, guess this.

C

Is the democracy.

D

um I think what we decided here, often and we'll probably put it in- is um we won't strive for future parity, but we'll make sure that we could at least do the the easy button Docker compose up. You may not have every feature the demo offers, but you'll have the majority of it. It'll be damn near good enough. I.

H

Guess it still feels like oh God hold on.

D

Until um the um our leaders at kubernetes, maybe Kelsey's Focus this year will be on how to make easy buttons for the world's system. So kubernetes is as easy as docker. That should be. That should be the Easy Button of where kubernetes goes next working.

F

D

The minified versions.

C

F

H

um Okay, I guess my not to necro this, because I was like my bad but like we could make an easy kubernetes version like we could have a script that installs k3s like.

E

In the place, the sample- yes.

H

That's true, yeah, like it's k3s is like a single binary. It's super small, you know it's we're not saying you have to.

H

You know, become like a kubernetes master or anything, but I I feel like we could make the invest. I feel like that.

H

Investing a little bit in upfront, tooling, for um making a easy, kubernetes version locally will save us a lot of headache long term in terms of like even fulfilling the basic idea of like.

H

Oh there's, you know we'll strive for parity, but it won't happen right, like the simplest version of this, would be resource detection via you know the collector operator versus not operator like if we have two different deployment strategies, then eventually we are going to have like just a bunch of switches and logic at deployment time and it's going to leak into sort of the application runtime itself um in order to support those multiple strategies right and even if that optionality is literally just it's off or it's on.

H

That's still logic that you're building in that you have to put into deployment artifacts. You have to put into manifest that you have to put even sometimes like leak into the code itself, and it also interferes with any kind of like observability.

H

We want to actually ship with this right, because now, if I have a situation where I'm shipping a grafana dashboard and in some cases, I have um kubernetes resource attributes and then in some cases I don't well now all those dashboards break depending on where I've deployed it, which means I, know I'd, have two copies of the dashboard.

H

Foreign, like I, think we can build the kubernetes, Easy, Button and I think it would make our lives a lot easier, long term to build the kubernetes Easy Button rather than to.

D

Should we build the kubernetes easy button first and come back to this discussion, then sure.

G

Yeah I think I think the challenge has been placed, so we should create an issue to track it and then okay.

H

So, let's do uh yeah in the interest of like asynching. This, like, let's create an issue just figure out like what are the requirements for the kubernetes easy button and then we'll go from there.

G

D

H

Yeah I'm gonna have discussion.

D

On it too, well, there's already a discussion on it. Just link up. That's all.

G

Yeah, there's a yeah there's an active discussion just like the the PFC there and people can get feedback, um but Austin. Can you make the the calendar uh bi-weekly for our meeting, so it.

H

G

It's only every two weeks, thanks um yeah I'm, not sure if we want to discuss this issue.

E

Yeah I think this one we can, we can skip because we already have a ticket. uh Are you sure, okay and we depend on the PHP Community to update or allow us to to configure that? So we can move on.

G

E

Thing I wanted.

B

H

B

G

Oh, is it okay.

H

G

Maybe it was just my calendar being messed up, never mind. That's me.

H

Does everyone else see.

G

H

um The next one should be on the 23rd for people. Does everyone say that.

B

A

That's what I see.

I

I I still see it next week. It could be that I haven't hard-coded in my calendar, though I.

D

G

Next week, as well, potentially a Google Calendar, Sync issue, I, don't know, I've had some weird stuff with the hotel calendar.

H

um Just to be sure, the one you're looking at has the open until Republic calendar as the organizer for next week.

D

How do you find out, who has the organizer obvious.

E

C

I

C

Should be a little checkbox.

I

That says other calendars and then you can check the open, Telemetry public calendar.

D

Yeah no I think like it's on my calendar, yeah.

H

That's probably.

I

H

It then you're gonna get the old one Carter you can. You can show if you.

G

H

I should have.

G

The demo, the 23rd.

C

G

Yeah, it's up to it's updated on this one then.

H

Okay, make sure you update your personal calendars.

H

Last year, anyhow,.

I

I copied it and I didn't get your.

H

Audience yeah, if you copied.

G

H

Over then, the copy won't get it because that's just how it works.

G

All right, oh thanks. Thanks awesome.

H

I was late because I yeah I deleted my copy.

G

Easy to happen, but one thing I want to discuss quickly, is Severance. Adding all this resource detectors to the various Services based on language, I think it'd be great to have that captured and then document somewhere, I'm, not sure where the exact spot would be, but we should probably call out that we have all this resource detection going on person. You know that's just what I personally think so I'm curious. What you all think on that.

B

D

B

Don't we have this.

E

D

Feature yeah: we have the feature Matrix table.

G

That's true: it's not really by service, though yeah.

D

Yeah, go back to demo features MD that had.

H

D

Yeah, so you know.

B

C

D

No, we did it by tracing them by metrics is what we did didn't. We.

G

Yeah, actually, the trade service features and Metric service features. The.

D

Maybe we should put a Telemetry service features and put trace and metrics as their own tables or something like that and call out resource detectors.

H

uh Yeah, wouldn't research aren't resource I, know, resources.

D

Do they apply to all yeah? They apply to a metrics and two um there's a resource attribute, so you try. They apply to everything that comes out of the service.

H

But I thought that resources were detached to a pipeline. The detector had to be attached to a pipeline because no.

D

H

uh Cardinality go boom boom if.

C

You attach High.

D

um You know people should just build better high carnality things, awesome I'm, sorry that was the vendor talking there.

H

Hey I mean uh yeah they're attached to the trait they're attached to the pipeline, so um they would not automatically apply to.

D

You know they're touched the SDK, no.

H

They're attacked right there, node SDK Trace exporter, oh.

C

H

D

Not for now well.

H

No, it is also isn't no different.

D

No they're all like this I think when you, when you bootstrap the SDK, when you create the trace provider or.

H

No wait, is it yeah right, Trace provider gets it, the provider gets it, not the export or not anything else. So you'd have to attach the resources to the.

G

Yeah, except for the cart service, he added it to the Metro provider and for the.

H

um Provider but technically it's attached to each independently. Instead yeah.

D

So should we just add another problem to each sure: should we just add another column, then to the trace and Metric service features, call it resource detectors and just check them.

H

Yeah that works, although I, do like the idea of like consolidating those into just like a Telemetry, especially when we get logs going.

D

Yeah I'm just gonna say when we had logs and you know, there's going to be some overlap and some sdks somewhere. Some funky thing is going to bite us.

G

Okay, um so this is gonna, be uh exciting topic, so Giuliano made some great kind of image, time reductions based on like dependencies and versions that he updated, but that did increase the build time um kind of similar amount. So it's about a 10x reduction in space and about a 10x increase and build time so I think kind of like as a group.

G

Some point discussion is: do you prefer image size or we prefer build time, um and this is probably or yeah also related to some of our performance issues. But storage is cheap.

H

Storage is cheaper than time yeah and um considering that the arm 64 thing keeps coming up, keep in mind that if I mean I, guess it's one of those things where.

C

H

Emulation you know it really doesn't matter that much because they're, the overall build is going to be constrained by whatever the longest single thing is like the jobs, not gonna, be done until rust and C plus plus finish. So you can hide more. You can hide time under that um and from a local perspective, going from seconds to 151 seconds is like a couple minutes.

G

So yeah like two and a half minutes too, like from you, know, half a minute essentially I.

H

G

This is not quite the increased time, yeah.

H

Is there a build that takes longer than two and a half minutes, because your art and builds.

I

Too, less takes about two and a half. If not more minutes yeah do we do we have a ceiling of How High that should go like I mean if it takes five minutes and that's way too long or if it's three and a half minutes it's way too long.

G

I think that's probably a good discussion. One kind of idea I had about here to help us as much on the build times but I know we have like continuous profiling expertise in the community I wonder if we could potentially ask one of the performance vendors to help us. You know get a better understanding of what's going on here and also uh maybe set in some performance requirements.

H

C

H

We have a pretty good um well there's a pretty idea of like where the build time like comes from. We.

G

Need some people to spend time there, though I feel like for like C plus. This is actually I've been kind of tempted to post it in, like the C plus plus slack just to ask if anyone has like free, Cycles just and wants to have some fun.

H

G

H

Mean like that would be a great.

D

Place, didn't we look to see, can we just propose the grpc part of C plus plus? Is that no longer.

G

An option um so I think we had an assessment done of that for us, in which case the answer was no that did not make sense. C, plus plus, has been um but not assessed, earn assessed.

I

Yeah it's worth noting for Russ, but that was highly related to how the open Telemetry collector works. So if the open Telemetry collector in C plus plus, depends on grpc, then it's unlikely that we'll get the time back, because that's what happened in rest is that we were installing the same libraries anyways yeah. um So we could. We could um it'd be worth having somebody look into it for sure what.

D

If you rusted collector was HTTP, would you still need all that.

I

Crap, so we need all that, because the o, o otlp or oltp, uses grpc and has tonic installed by default, which is like the largest part, okay, yeah installs that and you can't turn it off. It does need time. Yeah.

C

H

Should the X4 is everything not just, or it's not split out by?

H

Actually this uh a good piece of feedback to take back to the sigs would be in Rust and C, plus, plus, especially to allow to create um a way to build without grpc.

H

I

Split up the feel free to tag me I'm, happy to make an issue there, because I'd be happy to dig into that and and assess. If the build time gets better, if it gets dramatically better, then that could be a whole through line there.

D

Yeah somebody who knows more about C plus plus than me just double check, with C plus plus whether that exists today, because thank you, I I've watched that build happen far too many times. So I I think I've actually nodded off and fell asleep in the middle of a C plus plus build more than once so.

H

I I seem to recall when I looked at this, that there's no flag for it, but uh install building business.

H

Yeah, there's no target to remove it unless there's like something.

H

Something wild you can do with basil basil. How do you pronounce it um but yeah I think.

H

The can we make flag, compile Flags to not have to to not require grpc is the the fix. I will open a C plus plus issue for this. Do we have a.

A

H

Would be 669.

G

um Not really, no, this is a PR I, don't think we have a linked issue specifically for, like uh the Currency Service, like being unperformant, essentially.

H

Let me make a quick issue, then, that we can link back to um foreign.

E

So, uh just just to to be clear, uh we.

C

E

Prefer the image size the build time and I can Trash the pr okay, yeah, fine I.

H

Mean I think we don't I, think it's a significant Savings in terms of image, size and I. Do think that that's good, especially if, like people that have constrained you know like low respect or like 256, gig laptops or whatever so I actually would be in favor of reducing the image size, especially if the like, if we're not actually increasing the end-to-end wall, clock time right, because if we're already, if we already have long builds and this just kind of slots under the long builds, then cool whatever you know, we didn't actually reduce it.

H

We didn't make anyone's um experience worse.

E

Yeah what what um what I always think is uh in our case, we we constantly build the images, because we are always testing PRS and uh testing changes and doing all that stuff. But for for users there that are trying the demo, they just pull it. And then, if the image reduces from one giga to 200 Mega, then the pull time would be reduced.

H

G

I think yeah again yeah.

H

I think that's valuable.

G

Yeah I think Austin agrees, I think targeted, build or image size reduction. So if there are any other services where we can add or decrease the image size without a drastic increase in build time, I think we're okay with so in, in this case, I think they're like what 10x size, reduction, 5x size, reduction and 3X build time. That's probably okay for me, I, don't know if you want to formally vote in the Sig, but this one seems acceptable, um but if there was probably a bigger Spike, then I would say no.

D

um While you guys are discussing that there is an option that was recently merged, um December 1st by lalit and the C plus plus repo remove grpc from the build awesome. So if somebody wants to give that one, a whirl man, if we could take that 20 minute build and bring it down, I would be very ecstatic.

H

Is that yeah so.

D

It looks like it's yeah I pulled it from an issue asking to do specifically that- and this is the pr that was created for that issue.

G

um uh Does the pr open and just waiting to be merged, whereas.

F

D

C

D

If it's released is a different story, but it was merged.

G

To our tracking issue for now.

C

G

We'll um follow up when I don't have limited time.

C

So Juliana I.

G

Think I think at least I think we're okay with your PR I. Don't think you need to close it I think we can go ahead and get it merged, but in general, we'll just have to kind of do a per image and build time kind of um performance Delta and see what's worth it or what's not.

E

G

E

There is, there is also recommendation service that is also python, then has I. Think 900, megas and I can take a look into it, but.

G

Yeah, besides that, yeah.

E

All the others, I don't know.

G

Yeah that'd be great um before we go on to the next couple: topics Cedric uh and James I know I think either have PR's open or have topics they potentially want to discuss. So guys um thanks for joining us, if you have anything um going on right now,.

A

C

Yeah I have a couple of lines.

A

Full requests that I added around the ad service for adding metrics um but largely I'm here, just to sort of start getting involved.

G

Awesome well welcome thanks James.

D

A

Yeah so yeah I.

D

Do appreciate I'm, just gonna say appreciate the description James and the and the issues exactly what it's looking for great.

G

Let's see uh does this need any active Sega attention, James park or Pierre I haven't actively looked at it, but now we have some hanging PRS.

C

H

I was looking at this earlier and I I everything seemed fine yeah.

D

I thought we were able to merge this one did I not did we not approve it.

H

It doesn't look good.

G

Yeah, oh my goodness, that's.

D

G

Us, oh boy, okay, well, we'll go ahead and get this done too much.

G

Thank you, James great. Thank you contribution.

G

um Cedric! Do you have any topics? I know you've potentially been posting around arm 64, but I don't fully.

B

Remember yeah exactly so, I mean for two reasons: one's also to get involved, um I'm casually contributing to the project. Here. My main concern is actually um I'm 64 support, uh in particular for the feature Flex service.

B

um This one keeps dying uh on me on my M1 MacBook and um it seems that I need to build it natively. So what I did is I proposed that we built images natively enough for on 64 for all the services, but actually that's not so that's not solving my problem and problem solving would be just building the um feature Flex service natively on hype, 64 right.

B

um If that's an opportunity, we can also go that route. um I.

H

Have a quick question on this.

B

H

Have is that feature flag crash happened since, um with the most recent releases.

B

uh Can you quickly tell me what is the most recent when, when did it arrive, um I'm.

D

Sorry um 1.1 I think is when we rolled it, but 1.2 should definitely.

H

1.2.1 1.21.

C

H

Most recent, but any but 1.1 or later, should have fixed the crashing on Apple silicon.

B

Yeah, it's like definitely happened to me uh in December end of December, so it should be in the 102.1 yeah still happening.

A

Confirm it also was happening for me when I first started with the project I had to build it myself.

H

Because that should have the erling thing should have fixed it. um I mean, like I, said we would love. I I would be overjoyed to um build. Both the problem is in a few splunkery issues. You can kind of find the full history is that to build arm 64 in GitHub actions, you have to emulate arm 64 on the GitHub actions Runners and that's fine, except when it comes to building the rust and C plus images, um because they have the the grpc build basically takes forever yeah um and that's why we don't do it.

B

So um I I was working with another window before joining Groupon and what we did there was we had a multi-layered or a multi-stage build that took care of um compiling the uh the grpc bindings, the different build stage, so we could be. We were basically able to Cache um the the GMP build.

B

um Would that be an option for this group, or do you think we should build everything fresh? You.

H

You we could do so, there's GitHub actions, caching, which will do this. The problems I ran into with that are that you still pay the penalty for the first one um to anytime. You know the second one is I, don't know if we pin everything, but any change to Autolite dependencies causes it to be rebuilding. Considering that you know our release, Cadence is so slow or relatively slow.

H

It's not like we're building every day, um then you kind of lose the benefits of the caching, because there's a lot of stuff that we don't pin if we went through and like we pinned a bunch of dependencies and we made sure that we weren't changing like the underlying um image layers and that would make sense. The other thing we could do is publish intermediate grpc images where we've done the jrpc build, but we're still paying the penalty to build those like that's why our discussions are ranging towards.

H

Can we remove the grpc dependency from rust and C plus, because that's the only that's I, don't say the easy way, but that's the way to like remove the problem.

A

H

Now that said, if you want to go- and you want to try um doing a multi-stage like cool the the only thing I could figure out, that would be actually faster would be to have a native arm. 64 Runner, which GitHub actions doesn't provide and I've had a request out to the cncf to use their.

H

They have a thing with equinix metal to do like custom on-demand. You have to do like on-demand whatever or you need um which we could take advantage of, but I haven't heard back from them. I should probably go poke them some more um yeah. Basically, the short version is: if we could, if we could remove grpc, then we could- and we could you know I would say: what's our gut feeling like 10 15 minutes like 15 minutes 20 minutes, maybe is reasonable for um building everything on Free Runners.

G

Yeah we're at 20 right now we're.

H

20 right now, like maybe even you say, we go up to like 30 or 40 right like if, like 2X.

H

If it's twice as bad but we're building arm and um non-arm simultaneous you know in the same process, then I think a 20 minute. An additional 20 minute penalty is fine um yeah. But yes,.

G

That's kind of this man.

H

Yeah, those are the constraints right like.

C

H

You want to take a run at this again, it's very possibly something I missed I am by no means. um You know an expert here, but I did spend a lot of time.

H

Looking into this and that's kind of where we came down to is just we didn't really see a wave path forward that didn't involve where you, you know, with the the following constraints: right not having a extremely long, build at some point in the process that wouldn't that we wouldn't have to redo like monthly, that didn't require redoing a bunch of the docker files and build chains to try and pin everything and then then have to be responsible for like going through and doing security updates or whatever every month, um yeah.

G

Yeah I think uh so, if you want, if you want to help us on that that jerk we love it, but we've been a bit stumped on trying to solve it before so definitely an area. We could use a bit more helping yep.

I

Yeah and um something else to be aware of, if you're going to look into this uh rust is not as bad as C plus plus, but the rust one.

I

um The issue that we've run into at this point is that starting the process of downloading the dependencies and doing the build takes about as long as actually doing the dependency download and build, because it just takes so long for the like uh cargo registry to populate it just downloads, a lot of like where all of these packages are before it starts doing any of the downloading and building and really the only way to Cache.

I

That would be to have an image that has all of that packaged already, and so it defeats the purpose of like we need the pre-created image to do all of the dependency resolution for this image. Otherwise, it'll probably take just as long because it's going to recheck all the packages.

B

Would it be acceptable to have out-of-band build for cashing those that we could have a recurring bill? That would just you know, uh build the cache layer for uh for these images, yeah.

H

Yeah that'll work, I.

B

C

I

C

H

Be great yeah and, by the same token, like if we wanted to do multi like have like every week or something or whatever Cadence like if we can't split grpc out um than having a recurring like a Cron job type thing where every week or whatever it's building those intermediates I, think that would be fine in a very perfect world. I would say we get to the point where we can actually do fast builds like not publish, but at least fast build um for every image. Every PR.

C

H

Because, right now we don't really have a great way to make sure that people haven't like broken thing. You know that things are you're actually working at PR time other than people downloading the PO. You know, then checking out the pr and building it locally. So you know perfect world we're five ten minutes builds and then, when people do a PR, we can either as like a final pre-pr check or as a um just on every push images are being built.

D

If we had the intermediates um and we've built those immediates every week or whatever, we could totally do it on pushes.

D

Because they should be fast at that point right theoretically,.

H

Yeah I mean if I wanted to really be fancy. We could have that so that it only um if all the if everything was self-contained, then it only could just build what changed.

D

Slow down now, one step at a time, yeah.

H

But I mean I think that's like.

D

The I think in terms of.

H

Like vectors right like, where do we want to be going.

D

I want to fix grpc first, let's, let's rip it out, um let's get us some. Let's get an issue up on the rust um Sig ask him to split it. It looks like it might be split already for C plus, but um if we could do that, I I think it all. These discussions become very like they're, almost smooth, because nobody cares anymore. This thing goes in 10 minutes.

E

G

Sense well, I think we're at times we'll have to cover everything else in slack or asynchronously, but um welcome back everyone, uh Happy 2023., a.

E

Couple weeks.

E

C