Open Telemetry Uncategorized, 23 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-02-23 meeting

Description

cncf-opentelemetry@cncf.io's Personal Meeting Room

A

B

Hey good morning.

A

C

D

Good morning, good evening,.

D

Enjoying the snow Jason.

E

I, don't know if I'm enjoying it I mean I I like it, um I need to get out there and and shovel actually.

E

D

Was nice walking around yesterday, though yeah it's beautiful.

E

I just threw that in because I wanted us to acknowledge that a first-time contributor successfully and like with with really not that much feedback contributed a new instrumentation which I think is a pretty solid Testament to what we've got to work on so kudos to the people who help keep this thing improving every day.

D

And the tests I love like that- is probably my favorite thing about the HTTP client instrumentation. Is you just get this huge battery of tests for free.

E

Yeah, let's not talk about the implementation, but yes, I, agree.

D

Yeah speaking of first time, uh contributors, um uh Jason and John brought up. um You know a known sore point about loading, the repo into IntelliJ, um and you know that definitely I mean I. I I, see people here, uh um who've tried to load it and contribute, and I do agree with Jason's sentiment that it is a made at a significant deterrent to contributions.

B

um Recently, we've had an issue. There is a. There was a user who wanted to have AWS research attributes like as part of the Java agent and uh it's already implemented in the country. You can use it as an extension, but you know it's another jar that you have to like control, so that make me think about the uh open, Telemetry Java agent contract idea that we probably have talked about previously a long time ago, so uh yeah I'm, just like throwing throwing it on the table like what do you all think about this?

B

Should we do the same thing? The controller does like release two distributions of the Java agent, like the the core, the base one and the country, one which contains like everything.

D

Does that solve I mean the the main repo is still gonna be monolithic though yeah.

B

But we would be able like to slowly move the instrumentations over to con trip, and maybe we could do some like smart hacks of Gradle, to avoid stopping all those all those instrumentations into a single graded projects. Project I use, Maybe.

B

uh There is a thing when you can import Gradle projects like we, we use it in uh in the Gradle plugins that we have in the minor report.

B

So maybe this like would kind of help manage or all of this in the in the country. Purple.

D

Yeah um I the only kind of thought that I had that oh, like was, if we could and I just don't know like what's possible with cradle somehow to be able for each of these to be its own module that you could load.

B

Yeah so I I'm, not sure if this is possible, but I can imagine it could could possibly work like that. But do you have uh and let's say, instrumentation directory in the country and each of the like subdirectories, it's like a separate Gradle project, and then you have a separate like uh open. Telemetry generation contribute like Java agent release or Java agent contributes, or whatever graded project that just Imports them all. So the the Java agent one could be somewhat heavy to importantly in the Intel J, but you'd like you, shouldn't need to do that.

B

It's just like Gradle script that basically backs them all up in a single jar in the release system and uh all of the instrumentations could be completely separate.

B

I'm, not sure if this is possible. I kind of have the feeling that it should but uh yeah.

D

Would we need to split it into a separate repo um I'm, just wondering from uh um uh at least initially, if we could break it up inside of one repo just have a different folder um from a transition perspective? I think you know we still could you know, move things out, yeah.

B

I think it probably would be a little bit more problematic in the Repository, because uh instrumentations depend on the instrumentation API, the the extension API and like that, some of some of the libraries that we have and published here and.

D

I see so you're actually thinking by having it in a different repo. It's it's been it kind of in a way beneficial, because it's it's it's as if it was like completely external, it wouldn't be yeah. The dependencies wouldn't be on the snapshot versions of things. Yes, I see. Yeah I like that, because um the problem that I've always had with other mono repos, where it's all together is you essentially have to do. Maven install skip tests install first and then you can open up the sub module. You want.

E

Yeah, there's definitely trade-offs, but I think even even doing that. Maven step is probably uh less heavy weights than loading the entire project and refreshing indexes, and mostly because you know night for for Mo, I. Think for most users. 90 of the instrumentations are just not relevant to what you're working on and they require the most downloading and indexing.

E

A

Also, dependency.

E

Resolution too, I'm sure is impacted by that.

F

And what about when we build a project um if we do something wrong on the API, so there's a bunch of projects that we'll complain right away. If we have two different repos.

F

That's not going to happen.

B

On the other hand, we have the J API CMP plugin, that like helps us not to make any breaking changes and uh I mean it's also like a kind of very generic problem right. That applies to pretty much every librarian existence.

D

I could see the like, I mean the the I think that helps I'm, not too worried about breaking the instrumentation API but sort of the internals of the Java agent.

D

um Sometimes there's stuff in there. That's you know related to, like executor, instrumentation, somehow.

A

F

As we can on the pipeline have like a job that will build the the other project as part of the pipeline. If we need to.

F

Put this cause a change in the group ID of the artifacts.

D

Doesn't have to.

B

Even if it did like uh for the Java engine estimation is probably nobody cares, because they are only usable as part of the Java agent.

D

Would uh does it make sense for this to be the repo with all the instrumentations uh and move everything else out into.

D

Somewhere else.

D

Instrumentation infra I, don't know.

B

Instrumentation core I, don't really have an answer to that.

B

I mean I've, thought that maybe country uh fits I, don't know if it fits more if or if it fits at all, but uh for the less popular and like the leader, libraries that we have, uh we would probably like to use the country model of ownership right like that. You would have component owners who are responsible for updating this exotic libraries that we have no idea how they work and uh whatever is left in the core repository like. However, it ended up being banked, then we will basically take care of that.

D

Bruno, uh what was your did? You have a concern about the group IDs.

F

Well, if we change projects um or repos well, we don't need to to change the group IDs, but.

F

They come from different places, so it will get confusing for people that look at things.

D

F

Also, if we move some of these instrumentations to contribute, wouldn't that um reduction in libraries be enough.

D

It would help, depending on how men like, depending on how many I don't know, are we thinking to move them all or thinking to keep some that we think are core I.

B

Think we could probably keep the core ones here and move uh more of the Exotic ones contribute.

B

Like you probably want to keep everything, that's uh like related to jdk, like the HTTP, client, jdbc and stuff, like that, we would probably want to keep spring. uh The open, Telemetry instrumentation is, of course, like Kafka.

E

B

E

This gets complicated very quickly, trying to draw this very delicate.

D

E

All of these, because there's so many different ways that you could group these yeah.

D

E

Does feel like there is some kind of there's, probably something that makes sense, though.

B

I mean it's an arbitrary decision anyway, so like we could just draw a line at some point and whether it's you know a little bit left or a little bit right. It doesn't really matter as.

A

Long as we have.

B

D

Yeah I had done that once I, don't know if I can find it.

B

There was the scope MD file that briefly showed up I.

E

Mean it would be cool if vendors could contribute like instrumentation usage and if somehow we could know like which ones are, you know being used.

E

Like so then they could be tiered right, but.

E

I know it's a big ask.

D

We have some data on that.

B

I'm pretty sure we used to have a file that, like listed actual libraries and I, thought it was that one.

D

Yeah I think I deleted it. uh The listing at one point um but I'm not gonna, find it, but that's: okay, I um I mean honestly in our in the Microsoft just drew. We only enable what we consider P1 instrumentations already by default, just because I didn't want to be on the hook for uh support tickets for tapestry.

C

E

C

All might have you all might have addressed this earlier, um but if these were to move to contrib, uh how exactly would that solve the problem of having to import a a big kind of multi-module project into IntelliJ? Would they be treated as separate projects which you would have to open individually.

E

Yeah I think that was the idea can.

C

That same approach just happen with them still living in instrumentation.

E

I think that we think it can, and that could be a good like transitional.

B

Step Maybe, it could also like be somewhat harder because uh if we still wanna build and publish the Java agent, that, like has all of them, you would have to like uh very carefully with like the dependencies to build, like the shared libraries like instrumentation API. First and all the instrumentations, and some of them are separate projects and like and as the last step, build. The generations.

C

When I have all them be separate projects like all the instrumentations be separate projects, they just happen to live in the same Repository as the agent tooling.

E

I mean I, guess one one downside there is that um if he's the great old rapper, that's a lot of rappers and a lot of genres.

A

C

Also I mean that's that's the same case, though. If they live in contrib, though right totally.

E

F

So what if we published an index that could be used by by IntelliJ.

D

How how do is this something you've heard of.

F

So I have no idea, but I know that they do it for jdk and things like that. Yeah.

E

The shared indexes thing.

F

B

I'm kind of worried that uh the index would be somewhat prone to change, so uh if we publish it once, we would have to like keep it up to date.

E

B

E

We make him do that. You.

B

C

I think that's worth investigating yeah the.

E

Shared indexes is definitely interesting.

C

If it has legs, then you know it could prevent a lot of effort required to reorganize the Repository.

F

So let me try to spin some other ideas, so Quark the quarks project, it's probably bigger than instrumentation, it's way, bigger, yeah and it.

D

May not have as many different transitive dependencies. That's where that's why this project suffers.

F

Well, the version specification: the project has around 300 lines, so just for leap versions, so.

A

F

It has a very large number of dependencies and it takes a very long time to to load. That's true, but uh we we provide some instructions to to build quickly the project and that when you open IntelliJ for the first time it because everything is already tidy, um it take well it loads a bit faster.

D

Can you link those the instructions that you provide.

F

Well, the instructions are tied to build profiles that we have so, even if the the full project takes more than an hour to build on my machine. Just the quick build takes between two and three minutes.

F

Foreign yeah I will I will find the link and put it on the notes.

D

D

So matesh I wasn't quite following what you were thinking was different about like if we just had you know a folder over here um called contrib, um where those things went into from a build perspective.

B

I was worried that maybe it's like unfounded worrying uh that uh building and Publishing the whole job agent would still need to know those of all of these like separate sub projects on like side projects, because if you want to have like one jar, one distribution of the right that contains everything, then you have to build everything anyway.

D

So this is still. This is with the idea that we would have a Java agent, contrib Ute we'd have a Java agent would have two diff distributions in.

A

B

D

Or or across or if you were thinking with the contrib repo that we were I, think I was missing, that in my head of.

B

Yeah, so that the contribut would have like its own distribution of the Java agent kind of like analogous to what we have with the collectiveness like open, Telemetry, collector and open Telemetry, collector country.

B

And so we would have open Elementary Java agent, which just contains the core staff and open telemetric JavaScript country, which has everything.

D

Yeah well I'm, a fan of uh all of the above um I can I can pull out the list I can put together um to propose list uh distributations.

D

um Yeah I'll open an issue and propose an initial list and gather feedback.

D

And then yeah, if anybody has experience with their time to mess around with other things, because still I mean it will help for sure a lot, but we'll still have a fairly good size uh dependency graph in the um instrumentation repo.

G

Boat Moving instrumentations to Country just complicate things for us.

G

Like uh currently it's a refactor or something and everything is in one project, then it works nicely. But if you have multiple projects, then I guess that will get broken and the testing story will also get more complicated. Like.

E

Yeah I think that's the same point that Bruno was asking about earlier regarding the API right. If you made like an API change, you get to see that instantly when it's a mono repo mono project and it's a little more complicated, I think I think the answer is yeah. It is more complicated.

B

No like instantly, you know.

E

I know yeah, that's all yeah I'm definitely overstating.

B

But yeah uh API changes are one thing, but the testing story we pretty much, don't really have one more I mean there's the we have the genuine extensions which are really cool, but uh I mean like take a look at the HTTP test right the services and the client tests. They are still probably not ready to be used outside of the garage and I'd say I.

G

Was I was more meaning that currently being we keep an eye on the on the test failures in our project, but then suddenly we have two projects that we have to kind of manage.

A

D

G

Think the agent depends on the country that there is also the dependency that goes. Vice versa,.

B

Hoping that we could reverse the dependency, because so there's like AWS Lambda I, think an.

D

Instrumentation.

B

That depends on contribution. We could just move it the country because it's like hey that the AWS SDK probably is core, maybe is a core instrumentation, but they're Lambda, um probably not.

G

uh I think some of the AWS and camel instrumentation require the X-ray propagator thermal.

B

Instrumentation is definitely you know the core instrumentation, though.

G

But uh I think it recently came up that uh maybe we should consider including AWS resources in agent distribution.

B

Yeah and like uh so that's that's what I and like started my argument with that. Maybe we could have a like a broader distribution of the Java agent. That includes also components from country revolving. The reverse dependency that we already we kind of have one in London now, foreign.

G

Having multiple distributions for the agent might not also be exactly desirable because it just creates more confusion, I think that's.

A

A

D

So yeah um on my thoughts on the refactoring.

D

Yeah that one has been a concern of mine, also um it's definitely easier to refactor when they're all in one place, I guess it would become a case where we would refactor in the instrumentation repo and then after that, release would have to go through the contrib repo and update things.

G

uh One thing I'd like to point out is that, although making the life easier for the occasional contributors is nice, it is fairly unlikely that we are going to get tons of contributors. So we should still prioritize our own Comfort, like the maintainer's comfort.

C

Maybe we can get some sponsors and make sure all the maintainers have M2 laptops.

D

F

You don't want that, but then you you need to deal with the the docker and two problems.

C

They've mostly resolve themselves I've been on a M1 for over a year, I think and it's it's an amazing experience.

B

But uh Maybe having the maintainers have enough time to work and the project will be even better.

C

Hey Bruno I was wondering uh if you could maybe explain this quick, build profile that workers uses a bit more uh I found that link and and read about it, a bit and I'm, not sure I I fully understand, what's what's going on here, so, um oh sure, actually.

F

So this works like this um yeah, it's it's blunt, but um so basically it will skip tests, documentation, Generation, all kinds of validations and enforcement um and some other things that um are done, but they are not really needed for the build just to have a working build. That should work. If you just check out from the repo everything will work so no need to do uh the whole thing.

C

And then so it is okay, so you build it without executing the tests and the other types of checks. um And then you, you modify some IntelliJ settings to increase the the Heap size and you know, and somehow that makes it yeah. It makes it easier for IntelliJ to index I. Guess that's the part that I'm kind.

F

Of missing yes, yes,.

C

F

C

Intellij tried to like execute the full build with all the tests initially is. Is that the assertion or.

F

um Well, it will try to do wherever is the standard built for intelligent I, don't know what will be, but at the same time it's doing the build it's trying to index and things get conflicted.

F

So if you do the build quickly as quickly as possible and then index, you don't have concurrent access to everything, it will be much faster.

C

Worth testing out like it sounds like kind of like a Gradle, assemble type thing, so you compile the classes without testing them. I wonder if somebody were to try that on a unlike a a clean, get clone and then importing it into IntelliJ, if it would be faster.

C

I'm a bit skeptical, but maybe there's something happening with IntelliJ I, don't understand.

D

What would be the.

D

The testing story, um can we not use I mean if we published the testing Commons stuff from the instrumentation repo?

D

Can we not use that.

D

Or we'd also need to publish the convention, the Gradle conventions.

B

Or copy paste, initially at least.

D

I'm hoping this problem will I mean, maybe we wouldn't want to consider moving things until there's some like HTTP and maybe messaging semantic conventions or stabilized, because that's where currently most of our big refactoring takes place.

B

Or if, if we decide to move things, we could just move them gradually like when we are kind of done with HTTP conventions, we could start moving the uh lesser known, HTTP, clients and servers, and so on.

D

I think it's still worth I'll open an issue and Lori. um Let I think it's good to add the the cons. You know things that will be um more difficult, um so we can. You know, make sure that we, if we do, do something like that, we would address the cons, because I I totally agree that um uh we shouldn't make things painful for um uh frequent contributors, since this project depends so much on them.

D

Cool well um was there anything that anybody wanted to chat about or bring up today.

E

I lost track of this. The discussion last week around uh class litter and caching and I'm wondering if I should close the like I I'm I, intend to or I'd like to close the uh Gradle, not great old, the uh groovy class letter. Pr. That's been sticking out there for a minute.

E

B

You merge it I, don't remember.

D

I thought it was merged a long time ago. Nope. That's.

A

D

G

I think we just went to the second page of the pull requests.

G

E

So I mean I, don't know what we think about this one.

E

If um you know if we, if the caching can take care of this, then I think we don't need to exclude it right.

E

And by the caching I just mean if we had caps on those caches, I think this would not be a problem right.

H

So I I think that the comment at the bottom of you know if, if we know that the class loader is never going to have instrumented logic in it, then ignoring the class loader early on is going to help and I. Think that makes sense.

H

E

I'm, seeing some nodding all right, cool.

G

I'll leave it. It is kind of like a twofold story like the problem. Is that the some of our instrumentations kind of depend on like, but a good assumed that all the Run novels get instrumented, for example?

G

But if you start ignoring stuff, then this assumption is going to break.

D

E

G

It will work out just fine, but there is a slight chance that there's something really bizarre would happen.

G

A

G

Reason why like we might need to be slightly careful when we start ignoring all those class loaders yeah.

D

G

It's definitely beneficial like.

G

If, if there is a ton of class loaders, then uh all like uh the resource lookups that we have to do for each of them, this can be. This can easily add up and take forever to run.

G

I think we are ignoring like some spring type matching class loader because they died in massive amounts of time to application start that.

D

Do we know what um is low like if a user.

D

Yeah, the runnable is a good example, although in that case, I think we wrap the runnable when the X, when it's passed to the executor, I.

E

Forget I think so.

D

I can live with merginess.

E

Well, I wasn't pressuring I, just I I, just remembered it was still out there.

G

Yeah I think: let's merge it and.

B

Beers that are still open, I, just remember about my uh yeah, about redirecting the logging.

G

I think there are a couple of pull requests on the second page, that we should actually look at that some contributors have worked on and we have been ignoring them like uh the Cassandra executor. Active method seems like uh it could be a good addition.

D

G

This planet that maybe I'll try to rebase it and see if I can resurrect it somehow.

D

H

D

Make a list here.

D

G

Ers pull request as trust requested last time. I added a comment there about some of my overhead.

A

D

um The one that I was interested I thought you mentioned something last week about the number of resource lookups.

G

Oh I I actually didn't do that. Oh I thought that as this as the memory usage already increased- and maybe maybe we can do better than this pull request.

D

I mentioned I, see.

D

Okay and I mean this would specifically for a life Ray.

H

I I'm not ready to kind of rehash this one I I haven't had uh time to uh revisit and uh uh sorry I don't have I haven't done what you asked me to do.

D

Okay, um yeah, maybe in the meantime, if you um take a look at this uh this and see, if that um would help in your case also.

D

What else do we have languishing on page two.

G

There was something about the Apache Pulsar I think this.

E

Is mitasha's favorite one right here.

D

Well, sorry, yeah.

D

uh Yes, the HTTP client contacts I mean uh request length.

D

um Was this one ready.

D

D

D

I think this one we were going to close.

B

Yeah I think we can close it. Okay, the person who opened a issue fixed foreign.

B

D

A

D

Not worried I'm not worried about Matthias y'all, so he knows all kinds of ways to poke and protest.

B

Also, like honesty, I might not have enough time to finish, like the the whole story on vlogging right right now, so it can probably wait a couple more weeks.

D

All right, that's good that looks like that covers all right um yeah. Maybe we can uh divvy these up, um matesh and Laurie in chat. Just grab put your name on one.

D

Maybe we can each do one by next week.

D

All right uh anything else.

G

um We wanted to ask about this AWS resource provider. Is there a reason why we why we shouldn't include it inside the agent distribution.

D

I, don't think so. I.

B

Think we could include it probably like at any moment we just didn't because of I didn't even know. Why.

G

B

G

Are there similar resource providers for other Cloud providers that we should include.

C

Don't exist yet in the Java ecosystem.

E

Hey trasks, do you have them for azure, you don't. Okay,.

D

No interesting when we, when we do I, would want to include it here so.

E

Yeah yeah I mean I, think I. Think vendors want that as well.

D

Yeah yeah yeah, so in general I mean the the question of like I. Think. We've generally said that Cloud specific stuff does belong in like the open, Telemetry, Repose but sort of back-end vendory stuff, which there was all kind of a lot of debate about this one like x-ray, um because x-ray is a back-end monitoring piece, but it's also at the same time sort of built into some of the AWS services. So it's a little of a gray area um but I, don't think, there's any reason not to include the cloud provide cloud provider, resources.

B

We are depending on Country anyway, so like, let's just include it.

D

B

No included in the like the Java agent.

D

Oh right, because we're already pulling this in from there yep.

H

Is anyone working in this area specifically because uh I'm I'm actually working on the uh the fast dig right now uh and uh we've got some changes to the spec that I'm probably going to uh Implement at some point on the Java side of things? I just want to make sure I'm not stepping on anyone's toes there.

F

H

F

uh Well, we use it. uh It's especially useful for um the Lambda functions.

H

Right, no so uh I'm talking specifically, is anyone working in this code base, um because I I'm gonna, probably at some point, need to make some changes to align with the updates in the spec. There.

B

We have sometimes not I mean sometimes some AWS people like they were to.

B

You will learn that sometimes joint is sick and I think they were interested of working on this and they I think we even made them the component owners.

B

It was William and uh the other one. Okay, sorry.

H

Interesting, okay, um but you're, saying that you, you think that some of the like, for example, the X-ray propagator, should be brought into the uh the instrumentation repo was that okay, sorry I, misunderstood.

B

It's included, there's a dependency.

C

Got it I I think I didn't prioritize this initially when I moved this I think this was in the the core repository originally and I. Don't think I uh prioritized, including it in the in the agent because uh I figured it was easy enough to include as an extension.

C

um But you know if, if you want to include it by default, that sounds good.

D

I could see I mean I I. Would certain I would support moving the the resources resource providers into the instrumentation repo if there was a desire for that, um as I said they only hesitation. I have around x-rays specifically. Is that it's in these it hasn't been clear. It's been a gray area, although I think some of the work you've been doing Tyler in the the fast group um made clear help clarify that, like that x-ray may be required, for it's not clear to me yet.

H

Right so I guess uh just to give give a little background there. um Currently uh our instrumentation uses.

H

um So when you're working with a Lambda environment, it defaults so uh to use x-ray uh so x-ray kind of overrides uh whatever is there if x-ray is enabled. So even if you have uh otlp configured as your propagator uh x-ray is still going to kind of take priority, and that was something that I fixed in the specification, uh so I need to make that change here. The other thing uh that other languages haven't done so well at is on the AWS SDK side.

H

uh Using x-ray to propagate calls to AWS uh is important, because uh AWS is sorry. X-Ray is the only thing that's actually supported by AWS for propagation, uh specifically, so, for example, if you're making a call to S3, which uh has a uh an SNS listener, that puts a notification on a an sqs topic which has an uh a Lambda listener. X-Ray will actually propagate that chain of calls, uh so uh the the the context is maintained, whereas if you are using uh something else, that context is broken.

H

So that's an example of why, uh when you're making calls from the S SDK it's important to use something that AWS supports.

G

I think there was also something about the billing that, if you're using the X-ray propagation headers, then that's free, but if you're using some other headers, then that will be that those will be built. Somehow.

H

Yeah, so that that does impact it that feels secondary to me, um but uh yeah also something uh that we can consider.

H

Anyway, just a a preview of some work that I'll probably be taking on on the Java side at some point soon,.

H

If you've got any concerns with that, let me know.

D

Materials I'm not really like for a part of me thinks and I mean this is goes against the whole mono repo problem, but like something that we use in the instrumentation in the Java agent. Why wouldn't we? Why does it live in contrib.

B

So uh I just remembered why, if you take a look at the implementation of these resources, it's uh extremely unclear to anybody who's, not working in in AWS. What these actually did? They like call some IP address, and they get some Json and uh like whatever this json's format is, is like I have no idea and uh I take a look at the ec2 resource.

B

I mean I'm, just like kind of like guessing, but.

B

I'm I don't feel comfortable about like maintaining this I have no idea what this does.

A

This is Magic.

E

Yeah yeah I mean it is documented, at least it's not. This wasn't, like you know, internal people like you can go. Look this up, at least at least it's documented.

B

E

D

Know it's hard to tell like um maybe I would I would be okay with this stuff if there was if we had tests that ran in that, you know, ran in the environment and validated things, because then I real.

B

B

Okay, yeah, that would be fine.

D

Because that's my my motto is: if the tests don't break, make the change.

D

Thank you all right. Well, we used up our whole time good chatting with everyone.

C

Thanks, everyone see you take.

E

Care. Thank you.

F