Open Telemetry Uncategorized, 4 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-04-04 meeting

Description

cncf-opentelemetry@cncf.io's Personal Meeting Room

A

B

C

How's it going.

A

uh Mikey turned into a 90s porn screen.

C

I uh had the blur turned off.

A

Oh yeah you're are you switching something because he keeps switching back into it? That's weird! Yeah. There you go. Oh.

A

I, don't know I'm pretty sure, you're an AI bot. At this point. That's all I'm saying oh.

D

C

Found I'm switching.

D

Are you switching between pal and ntsc or something.

C

Yeah I'll just try rejoining.

A

Oh, it's I mean we can see you now, but it's okay, I, don't don't listen to me. Yeah I can ignore your video. That's fine!.

C

Yeah I just noticed, like I, didn't, have the blur effect on so maybe trying to turn that on real good I'm using the zoom web client too. So.

A

Oh yeah, maybe you're just like a bug and zoom because yeah your background's, not blurred, like that's the other thing, yeah.

C

I just turned it off, causing problems.

A

Yeah I gotcha, who knows who knows okay, um I, think we could probably start where almost three minutes out in yeah I think we've got Quorum too well, then welcome everyone.

A

um If you haven't already, please add yourself to the attendees list and if you have something you want to talk about, add it to the agenda and we can get started here. Real quick.

A

uh Okay, cool so I added, uh just a heads up the Instagram module and the uh the contrib repo was merged last week, um which is you know time for celebration. We got something merged. um That's definitely exciting.

A

I wanted to ask, though, what the next steps are. It looks like I'm guessing um there's a you've responded here, a little bit already right.

E

A

um Do you want to should I hand it over to you and you can kind of just go through each of these points?.

E

Yes, so uh today, I open it two new PRS and the first one is uh simple. Think with uh some small changes in tests right and the other one is, is a fix for for the uh for the code instrumentation. So we, uh the totally incorrectly uh instrumented uh Anonymous functions so I fixed that and the the other points that are listed here are the are my plan generally.

E

So what uh that's the plan? What should we do next? In my opinion, so uh we have to support different exporters, because currently these traces are exported into file only and then uh I would like to start also on hdp instrumentation. We can already instrument HTTP, but, for instance, a context is not propagated uh correctly between uh client and server.

E

uh So that's something that we have to fix and also I would like to start working on other libraries, and the next thing is configurability.

E

I was thinking about instrumentation of selected functions, types and so on, and last thing is: is the validity so now the the Transformations that the tool is doing uh these Transformations are not not checked at the end, so it might be that the AST is not valid and that's something that we have to check. So basically, that's the plan, but there might be other also also things that we should think about any questions uh regarding uh these topics.

A

Yeah uh parallel looks good, I'm kind of surprised, there's not already a label I thought I created one.

B

A

Sorry, one more time, Tanish I couldn't hear you.

B

A

B

A

uh Yeah I, don't! Oh, it's covers.

A

Yeah I don't know, I thought that we set this up so that the I wonder if we set it up wrong.

E

One one question because I cannot add labels there do I need any specific permissions to in order to do that.

E

D

E

That they're gonna do much in this repo.

A

Oh, you can't add labels um yeah, that's kind of I think that might be something we were talking on this. uh The maintainers meeting.

A

I, don't yeah I, don't know what permissions you need to have to add labels, but that's a that's a good point.

B

A

I I can look into that I. If it's right access to the repo, then I don't know. If we can do that, but um we can. We can look into uh trying to figure that one out so.

A

um And then yeah I Dinesh, what's your um handle again.

A

I, don't know why it's not adding the group I thought this was going to add the uh goes to rotation, but um at the very minimum I can I can just do this uh really quick uh yeah, but it'd be great. If these were automated uh instructions.

A

Yeah because I mean.

A

C

A

If the people that need your review aren't getting notified to review it um so that that needs to get fixed, okay, the next thing I would say, though, um is I. Think for a lot of these issues. We'd want to track them in issues and I. Think that's, that's a probably a good way. The uh you know. Obviously the caveat is like we'd want to track them in some sort of way. That can also be grouped um with a label. Another option is a um a milestone or project as well.

A

um We've used in the past in other parts, I, don't know. If, uh if there's not a lot of development effort, it may be just a little bit too much um uh uh bureaucracy to try to handle that. So maybe just a label is probably fine at this point, but yeah. Maybe we can create issues for these okay, okay, um yeah I mean I, definitely know that I can set it up so that the um uh the issue creation window can.

A

Actually, you know, tell you you know: do you want to create a Instagram issue or something like that, and that would automatically add a label, but.

E

A

Don't know why you wouldn't be able to do the the labels right now. I'd have to like I said: I'd have to go. Look into that to add it after the fact.

E

Okay, because currently we can only create these two types of issues right so I, don't know if that's something um that we should do for Instagram.

A

Nice I mean I, think that seems reasonable to me. I mean I. It's like the specification has something similar right like um I, don't know if you've seen it but like it's just it's essentially a way to create something with um a tag or a label. Right like you could go, create a resource uh issue, it's just as a resource here right. Maybe some some text but I think we can do something similar to this right. There doesn't sound reasonable to you, yeah, okay, um yeah.

E

And I think that yeah.

A

Let's, let's do that: okay, I'll! Add that as an action item.

A

Okay, I can. uh We could probably have something like this I think today, uh but if you wanted to create those issues, I can just manually add the the label until we get it all figured out um uh for doing this. Does that make sense.

E

Yes, it makes sense: okay,.

A

Okay, cool um this all seems pretty good as well, uh so.

A

I'm trying to think through the yeah I think.

F

There's a few other I I think.

A

There's a few other issues I wanted to address as well, but I can create those um after playing with it, so I think I think the key here is that we need some sort of way to track this, so having the label applied, is I. Think the right way to do this at least right now so um and then yeah. We need to also let me not lose that um I kind of swear. We got this merged.

A

Yeah I need to figure that out, though I thought this was set up in the um Aaron. Do you remember, like I thought we just set this up, that this was a part of the uh the code owners and that they should have been notified right should be, but I don't know how well that.

F

Actually works.

A

Oh, no, it's not. We've got the go approvers, but okay yeah there you go. There's the issue all right, so there's another PR that we need to change, um so I can yeah. We can take that. Hopefully, what happens is that anybody who's an approver here um should get notified on that one I do want to ask, though uh Mike. Are you an approver on this project for this for estrogen.

C

No okay, not for instrument.

A

um Or or sorry just I was looking at this composition the other day and it was like I, don't know if it's. If it's uh we need to update it or add you to it, because I know you're very active here as well, um and maybe this isn't the right form for it, but I.

A

Sorry, that's just what it's going to be um yeah I guess you can always say like you, don't want to be a part of it, but uh I.

C

Mean I'm happy to help out for the auto instrumentation I haven't done much with the Instagram, but I've have been digging a lot into the evpf side, so yeah.

A

Okay, yeah I, think uh yeah I, don't know um uh what else is something called Tyler's on the call uh uh Tyler. If you wanted to be a part of this as well we're given the early stages, it's a pretty easy one to be joined. This I.

D

Honeycomb 100 wants to it might not be me. I'm gonna try to get I think um Mike involved uh because he's on the he's on the team aligned to this stuff, more I want to do the operator stuff so I'm, showing up here, because I I want to release okay, uh I'll I'll talk to my team a little bit. If Mike can't be involved, then I can definitely contribute more time. I'll.

A

D

A

Yeah I mean the more uh contributors in the produce. The better have found um so I would encourage it, uh but yeah I'll I'll also take that, as an action item uh to add like to this I think that's.

E

A

Community issue.

A

Okay, cool, uh okay, awesome, so I guess moving on under the agenda. Is that sound like a good movie? We want to talk a little bit more about this. Anybody.

A

A

Next up, I have added these three open PRS um in none like particular order. uh The first one is one: that's been open for a little while this is the automated uh offsets. I wanted to ask uh Mike on this one, because uh okay yeah you just.

C

Yeah I just left a comment a little while ago. um I think we can close this one because I I did have to manually generate the offsets for the e2es to pass. Okay and and it's got a conflict, so it.

E

Looks like it worked.

C

So that's really cool, um but it didn't have any tests or anything so we'll see when the next Go version comes out. um If this works and passes and everything that'll be really cool, yeah.

A

Absolutely okay, thanks for doing that too, um that was that was a great PR oud um yeah. That being said, also I have noticed that our uh success rates on those is limited by a timeout, and we may need to adjust the Timeout on the.

C

A

I think I saw that oh.

C

Yeah, the GitHub action yeah.

A

uh I think we've done this in the past for other long-standing go operations um but yeah okay, and that just is a stream of conscious right. There I didn't.

C

um Okay, they are pretty beefy uh tests, yeah building everything from scratch.

A

I mean uh I. Think that's the approach you take here right is you have no testing that you add too much testing and then you kind of refine it as uh as the project evolves, so I think that's the right right, Next Step.

D

Mike, it looks like you got a build working in in the workflow. Is that accurate, like you're, able to build the main command? Is that what that works? Yeah.

C

Yeah, so we should have a we have a Docker build workflow which just builds it um and make sure that it compiles and then there's a uh the kind. E2E workflow is the one that all right: sweet, I.

D

Just had it, basically, it does the same.

C

Thing but then tries to deploy it, make sure that there's a output.

D

When I added the license stuff, we have a mate command for generating and checking the license, but I wasn't able to get it to work in a and a workbook, because I couldn't get the thing to build. But now that we have an example, I could I can Circle back on that and and actually add a workflow that checks that if any new dependencies are added, the licenses are updated. Also, um I'll open an issue for that.

C

D

C

A

E

A

uh Yeah I think that's that's a great idea. um I think the docker step was the key thing there.

A

Okay uh cool: uh what was the next PR I wanted to talk about? So then there's also this reformatting, the offsets that Jason um I. Maybe we could talk yeah I, so the question is I. Guess like this. Does a pretty big refactor here and there was a uh there's, a comment: there's definitely some conflicts now, obviously that I got a result, but my question is I thought this was like.

A

Did we still want to move forward with something like this or is there like a design change for how we're gonna handle these offsets that we wanted to approach instead.

F

C

Specific PR is about like.

F

You're, reducing.

A

F

Of the offsets result, Json um not sure I reviewed it, but uh uh totally agree that maybe we should like rethink and maybe.

E

B

It in non-json.

F

Formatters I, don't know, uh haven't looked at it yet.

A

Okay, yeah I uh I, looked at a very high level. I saw Dinesh commented um I'll, be honest, I had to agree too deep into the offset uh change. I, don't know where it is, um but yeah.

B

I think we should do something like this, but uh I don't know uh yeah, because the current Json file is too big even to analyze or read through it. It's like it makes.

E

B

For you in just like a dependent bot or all of our things, that's right now, it's just repeated n number of times, so it's just reducing the potential so which is good.

C

I think, basically, the change was to like right now the offsets file stores the offset for each version and I think what um this person was trying to do is just look at the diff between each version.

C

um Like know when the position changed and that really brought it down a lot, but I thought I saw them, leave a comment that they like realized. This wasn't going to work or there.

B

Was no I think there was a two different PRS, he created one and then he closed it. uh The first, the first one didn't have like a Min and Max version. This one has Min and Max version.

C

Okay, yeah, because.

A

I saw that I was like: where did that? Go wait a minute yeah. This is 82 lines. Now: okay, yeah 182 yeah. This is okay.

D

B

It's a pretty big.

A

One all right, that's that's worth uh yeah, okay,.

B

A

Right: uh okay, cool yeah, then I will I'll try to prioritize looking at this as well. I thought it was something different um but yeah. Okay, let's take a look at this.

B

Yeah, the change is good, actually yeah.

A

I think the merge is going to be a disaster, but um yeah I think maybe just regenerating is probably all that's needed. um Okay and then the last PR, that's still open. Is this allocate memory without launcher uh wanted to talk about this see where we're at? If we're stuck on anything or what the status is so maybe I'll hand it over to Eden.

F

Yeah sure uh so, actually it's not just I was busy with other.

B

F

But uh hopefully uh fixed most of the problems, it's most of the compilation errors I still have some uh comments here to uh to fix mostly adjustment for the new end-to-end tests, uh but yeah yeah. It's it's uh sorry for the delay, but it will be like it's one to two days and I also like have the arm support, pull request, also uh in the works. So hopefully that's also subjects that I'm going to uh open uh tomorrow. Probably oh.

A

Cool yeah uh I didn't mean to put undo pressure on Avis was wondering status, update, yeah, absolutely cool yeah.

F

Yeah I can just like give a quick overview of what happened in him. So uh uh previously we like we in order to instrument the go Application. We had to invoke it with uh with a specific process with like, uh like sort of a launcher process that allocates the uh to the uh Cisco and allocates memory, and basically that's where the instrumentation process knows where to write, context, propagation information, uh but that's not really ideal, and it also included some weird uh assembly commands.

F

So uh it didn't work very well, and this solution works much better. Let's sort of do a remote for the running process, so we we don't even have to uh restart the process in order to instrument it, we can go into like running, go application and start producing distributed tracing, that's cool um yeah, that's basically it yeah.

A

Yeah, it sounds like a really positive change, so it definitely um is something I'm excited to have merged.

F

Yeah yeah and it is much more stable, I tested it on multiple operating system, multiple CPU architectures. It works much better uh yeah and hopefully it will be merged there. Yeah.

A

F

A

The other thing it's like, since it's using a map, it's gonna, um not rely on uh machine code right to make.

F

These applications right.

B

F

A

Okay, cool yeah I mean that's a huge win um for competitive.

D

It sounds like this is something that would definitely need to be not need to be at sounds like, but we should definitely have this in place before anyone would want to start using this tool in production like I, wouldn't want to get the operator PR merge until this thing was in place and released. That sounds accurate.

F

Yeah yeah, yeah, yeah and like it also should happen very quickly, so yeah.

B

Yeah I think only after this, we can do like a point one or something like that.

D

I would love a point. One release uh should the arm support be in an initial release also, or is that not relevant.

A

D

It's literally.

A

Relevant to some people uh yeah, but I, yeah I, don't know if it's particularly needed as a proof of concept uh for a woman, okay, I.

D

Do want to yeah I was trying to call out yeah.

A

Okay, yeah um yeah, I I, don't know if it's necessarily for the res release, um but I would call it that, like I, don't think we should be recommending this being used in production uh by the way like.

D

Yeah, um that's I mean I, agree, I.

C

A

Problem, if you want to accept the responsibility uh and the you know, the issues that come with it, but I I, don't think that we should be recommending it, at least um at this point.

D

Yeah the operator integration is being handled. Similarly, it will be behind a feature flag, that's disabled, by default, so the operator won't, even let you inject the like the operator won't even acknowledge that go instrumentation is a possibility unless the managers of the operator explicitly enable the future gate like when they deploy the operator itself.

A

Okay, cool, yeah and I think that's the right approach. I'm not like. We want to get there to a 1.0, but.

C

Yeah yeah yeah yeah.

A

Yeah yeah, okay, uh cool and so then uh just kind of circling back on that so I think at the uh generated, offsets stuff uh or not the gender. The allocated memory uh without the launcher stuff is pretty much we're. Gonna wait for that for the 1.0 or the 0.1, and then the reform at the offsets we could probably include as well but I, don't think it's a blocker, um so just kind of a heads up on like timeline for that release. Yeah.

D

When it comes to that release, um do we want to do just a 0.1.0, or do we want to do something like what the demo did and throw on like an alpha tag or a Dev tag, or something that indicates that like hey? This is like truly like new and growing, and.

B

Maturing yeah I think autopilot makes more sense. We don't want people even by mistake to anything otherwise.

A

So the suggestion is to do a v01.10 alpha.

D

Yeah, that's what the Opera or not the operator. Sorry, that's what the demo did for a while at the beginning, I can go look at the exact tags that they used, um but they kind of did that with their first couple releases as like a call and then like hey. This is truly like in progress. Work.

A

Yeah I mean I'm, not opposed um yeah. Sure, uh okay, cool yeah I will I captured that in the notes, uh we're gonna wait on the allocated memory without launcher before the release. uh Oh action animal take an action item to create an issue to track the release.

F

Do we have like a GitHub workflow for releasing.

D

Yes, I added one images.

B

What does it build? The images I.

D

Tested it locally, I can link the pr if you want, but yeah I I reused, a lot of the logic from the key Val repository I added some some stuff that um uh I don't remember the GitHub handle that asked me to do it. um Someone from Splunk who is also asking about the licenses. I can't remember what this is.

E

D

Yeah he provided feedback and whatnot and I was able to test those images in my Fork pushing to both um GitHub and to Docker Hub, and uh it worked as expected.

A

uh Yeah so uh we'll see if it works the first time but uh I got.

D

I got hope, it probably won't, let's be honest, but hopefully it does yeah.

A

We'll make it we'll make it happen. um Okay, cool uh next up on the agenda. There's a question here: I, don't know who put it uh it's about. uh Is the kernel debug symbols, yeah.

D

I put that one Mike had asked this in the um in the slack Channel and it caught my interest, because this is something that I was having the operator add during the injection um thinking that it was a requirement based on the examples and the stuff that I was looking through um I'm looking for trying to understand. If it's, if it's a recommendation or requirement or just not needed it, when you, when you're, like really using the thing.

F

Yeah, so let me answer that quickly, because I really have to drop in just into minutes. uh So basically, the first use that I solved for this is the BPF print K function, which basically lets you debug the application and the right statement into into files that are located in the CIS candle debug as to other use cases, because I agree that, like we should probably not do print K in production, environment um I still need to to research.

F

A little I suspect it also used in the something called map pinning, uh which is something that we are using. uh There is a possibility in ABP, have to share a map between uh different evpf programs, which is something that we do uh so uh like. The bottom line is I. Think the easiest checks is just to remove it and see if everything works like the print K would fail, but it also feels quietly like I, won't stop the evpf program so yeah.

F

We should just like remove it and try yeah I can do it if you want. It will take me a few days, but.

D

I can try I can try to test that um if it ends up being like a really useful debugging tool. Yeah.

B

D

Can also make it an optional thing um like when the operator is using it because optionally add that amount. So, um okay.

B

A

Okay, cool I think, if that's a good question, uh it's a good idea to understand why we have it there uh next up is uh we wanted to try to reproduce the trace and span of these not being updated.

E

Yes, so I'd like to reproduce it with a exam that is mentioned in this issue. There is a simple um server. In fact it's mentioned it in in the other issue, but I I use the same example and I couldn't represent, produce it. However, when I tried to do that, as is described here with the collector uh that that I got the error, that is that is here it's about uh uh that.

E

It couldn't find these offsets, so um Mike suggested to to generate these offsets and when I tried that and then I got another error that there was an error while fetching offsets. So this this process ended without success. So I don't know if you had something similar Maybe.

C

Thank you, but one thing: I'd recommend when generating the offsets is it works best for me when I do it in a goaling container, I, don't know if you've tried that um and just pulling the latest golang like 1.20 or something, but also, if you rebase now it should have all the latest offsets from 120 anyway.

C

um So you can try building that again, but if you can paste the error that you had, that would be really helpful too.

E

E

So I can check. I can paste it on the chat for now yeah.

B

E

So I got something like that.

C

Awesome, yes, lots of information in this error.

C

um Yeah I'm, not I, don't have anything from that. I'd say: maybe try it in a in a container. I usually use the golang like 120 image yeah. It looks like it's just like this is coming from the generator when you're trying to run the offsets.

E

A

Yeah I I do the same thing as Mike uh I can't remember. This has been a while since I've run it, but I did I just ran it in a Docker container, because I was having similar issues and it seemed to work from for me for that. Okay,.

E

A

um What kind of a system are you running or is it? Is it a Linux yeah.

E

It's a Linux, Ubuntu I think.

A

E

Then what version.

A

Of go: are you running.

E

All right so I tried with few versions, but now I have one 18.6, I believe uh yeah.

A

Okay, yeah mm-hmm I.

D

Do understand yeah.

A

It's kind of an older version, but I think this project was originally generated using 118, so I think it should work yeah.

E

I was also checking these newer versions and also I had some problems so.

A

Yeah yeah I, don't know, I'd have to look at the people more.

D

A

Yeah I would probably try the container option just to um yeah and then, if there's still errors, we can, we can try to dive deeper into them, but just I would post them and then into the issue the errors themselves. So we can try to jump right there.

E

A

uh Okay with that we're at the end of the agenda here uh we also touched on the pr. We have some action items, I'm, sorry not compared the release, um and we have some action items uh to hold on to so I'll just pause here. Does anybody else have other things they wanted to talk about.

A

Okay, uh anybody use this in production, yet.

A

uh That's a no for me.

D

um I have I have customers that might if I can get it really operated, so I might use it. Yeah I, don't know, I, don't know about production, but they are very interested in Auto illustrating some go codes. So that's why that's why I want that? I want that release.

A

Definitely done some local stuff, but uh please, if you do, have customers or any sort of success stories or failures, um we'd love to we'd love to hear about them. You know, I think that those are always we've.

D

Definitely found I'm sure if we tell a customer to use the inject it with the operator and then it doesn't work we'll hear about it and I'll share that information back.

A

Okay, yeah, uh if it does work, I'd love to know the scale date they use it at two I think that'd be cool.

D

Yeah I understand definitely I will try to keep everyone informed.

A

A

Okay, um okay, for something else, we could probably end it here. Then thanks everyone for joining uh we'll see, y'all, virtually and otherwise. In two weeks.

E

Thank you, bye.