Open Telemetry Uncategorized, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-09-21 meeting

Description

Open Telemetry Meeting 1's Personal Meeting Room

A

Hello, everyone.

B

Hi tom harmark.

C

Hello good morning.

D

Hey good morning and good evening, everyone sorry for being late.

A

C

D

C

D

Okay, you can start specs changes yeah, I think, mark you have added open, telemetry specification release, 1.13.

A

D

Yeah, we do need to upgrade our semantic conventions.

D

And what is this.

B

This was discussed in maintainers meetings.

B

So they said we have to pay attention to this. One.

D

Okay, I think I remember that.

D

This is something which we don't support. As of knowledge, the scope, identity.

C

That's just again.

D

I mean we, we, I don't know.

C

We don't have this.

D

Attribute so we don't really support attribute as part of this scope identity.

D

So probably we don't need to remove to do anything in this I mean that's what I was expecting, so I didn't really think much about that, but just correct me like do we need to do any changes here.

B

Not sure, but this is something we have to go through.

C

D

So like it says that meter, basically, all the meters should be identified using name version, schema, url attributes, and now this.

C

Attribute part.

D

Is removed and at first instance we never added attributes. I think this was added in a separate pr. We never added this.

D

So as of now, meters are identified by name version and schema url, which is what we are following as so, we don't have the attribute, uh as as one of the identity in the scope, instrumentation scope.

D

But let's probably let's go through this, so the code for both meters login traces just to see if anywhere we are supporting it. We should be removing quickly, probably.

D

C

You only have name version and.

D

Schema url, so most probably there won't be anything which we need to do, but I think good to go to once go through go through this pr. If there is something which we need to do.

D

And yeah for semantic convention, I'm hoping that there won't be any breaking change in this 1.13, but good too good. To just see that I mean in the breaking change. I'm just expecting that they there is, should not be any of the attribute name which has been removed, which we are already using.

D

I mean either in the resources or in the traces which it should not be any of those properties which have been removed.

A

D

A

Some change in the level file, format itself and the tooling changed accordingly, and also, I noticed some renaming on of some host ip attribute, but I don't know if it's a breaking change or not, but basically what we need to do is run the run. The code generation again to.

A

To generate venue, uh conversions and see what is going on.

A

D

So this may need, if there is a change in the formula in the yaml file, we may have to do some changes in the way we generate our header files from the handle right.

A

Normally, no because we invoke some docker scripts that should take care of that.

D

Okay, but yes, I think probably we can create one issue on this and I think it's good to upgrade to the latest specification for semantics. Yes,.

D

Okay, yeah anything else in this.

D

Do you want do mark? Do you want me to add some action on you or anyone? For this I mean I just want to be sure that if you know what we have to go in this.

A

Oh, I can. I can look at that.

D

To go through this scope, identity, if there is anything which we need to do.

D

Okay, uh few other specs changes. I think which the discussions are still ongoing. Changing the default buckets for explicit bucket histogram. This is already done. I think it should have been merged. Let me see.

D

It is this is merged here, so we have to change our default buckets.

D

Probably we have to update our default buckets for the instagram and it should not. They have been discussion, whether it's a breaking change or not. It does not look to be a breaking page. uh Okay, I think I didn't this is incorrect. Probably I need to add the correct bucket values, but it did. This adds more even the changes to add more even even distribution of the uh of the events across these buckets. So the number of buckets have been increased. It should not be breaking change, ideally because the existing bucket range is not updated.

D

Just the new bucket ranges have been added, uh but anyway it should not affect us, because we are still into the experimental place for matrix. So you want to break the change. We should be good.

D

I'll write the pr.

B

um Maybe you can assign it to me.

D

Apart from that, in the span exporter also, they have a new method, as we added force flash. It was missing it till now, but I think over the course of a couple of months it has been added and we didn't yet implement that in our trace implementation. So we have to add this span. Exporter force flush as a required method.

D

um It may not have so we may not have. We will not have to do any custom implementation for that.

D

Apart from most probably the http async upload, which we do it, which we do for otlp, um we uh we do support the concurrent upload as a you know, feature flag, so we may have to do some something like this to flush all the uh pending events, which are which are yet to be uploaded, but for all other exporters the upload is synchronous. So, as part of the span exporter export call dx, the actual export operation is happening and that method will return only if the export is successful.

D

So, ideally, for all those exporters, we don't need to do any special implementation for those plus.

D

But I just created an issue for this and probably to start with, I think it's good to add a default method uh in in the sdk default empty button, and then we can see ahead if all the exporters have to do any custom implementation. For that.

D

So now that's apart from that make hotel experimental, sdk disabled as stable. I think this is still going on. There has been some discussion happening on what should the boolean field should look like and that that is already merged? So I think now this should be unblocked.

D

We may see this pr getting merged, but I'm I'm, I think, uh mark you did mention that you had some idea on how to implement this in simplest places. I was thinking that whether we should we may not need to support it or just wanted to understand again. Are you nothing good.

A

So, basically, in the in the api itself, there is the trace provider singleton, which has a setter and a getter, and today everything is considered part of the api, but in in my understanding the api should only be about the getter. So you get the single turn to instrumental code. The setter is to be used only by the sdk.

A

So if we move the problem is that we cannot implement this logic uh inside the api itself because it needs it means to do a get environment and and to do some logic.

A

So if we move the setter of the trace provider from the api to sdk, then we can do a check for this environment variable and ignore any setter. If environment viable is set.

C

A

So it means the trace provider of a matrix provider, analog provider.

A

It's uh it's one of those gray areas. A singleton is right in the in the middle between the two was reacting with this unique singleton uh changes.

D

Okay, I think probably let me let me also see this once. I think I got your point you're saying that if you can move tracer provider into setter and use the environment check the environment wherever and then decide accordingly, if it is not so if, if it is disabled, then we don't need to do any explicit set. Yes,.

A

So so, if we don't set anything, then all the instrumented code will go with the default provider, which is the newer.

D

I just want to check from spec's perspective.

C

D

Okay, I think probably we can start looking into that once this is br is merged, we'll have a look into the specs. If there is it's anything, it says anything specifically for setting and getting the facebook provider. I don't think it should be there, but let's probably check once.

D

D

D

uh Next is more domain. Specific attributes are getting added in matrix, cementing convention. We are seeing lots of pr being raised uh honestly, I'm not be I'm not following all of them.

D

uh That's something which I think should would be used by the instrumentation libraries who would be generating the measurements, uh but this is just just to keep in keep a check on that. What all things? What all changes are happening in the semantic convention for matrix? It's not yet stable, but I think lots of domain specific stuff is already added.

A

So normally cementing conventions are in a yaml file and it's this is generated automatically, although I have not.

D

A

But the same file for.

C

A

So I don't know if they are recorded as a yaml file, yet.

D

Yes, I was, I was also looking and looking for that. I didn't find it.

D

Yeah, I think good good too, maybe maybe once it's moved from experimental to stable, they may do something or- or maybe this is this- this semantic convention would be only used as part of api calls, not the sdk.

D

So I mean just just just giving some random thoughts here, whether it would be there in as part of the amount or not.

D

Yeah, okay, uh on the same I mean I do plan for matrix. I do I mean I just wanted to talk about the performance stats for the matrix.

D

I was going to write some of the benchmark stuff, as I discussed in a couple of meetings back just benchmark tests for the matrix, uh but the problem is our benchmark is not very stable, very reliable, as of now all the benchmark tests, they are running in the ci system and those are actually the ci virtual environment.

D

uh The results are always changing over the course of time, so it's not very reliable, at least for the virtual environment.

A

D

Ci on github.

A

Is totally unreliable? I got some warnings, but the performance broke by changing your comment so.

D

Yes, that's the reason the why we I'm not very relying on the performance benchmark. Yes, I was planning to write an instrumentation library which actually generates the the matrix for the performance stats.

D

That would be the process, stats system, stats in terms of the memory, consumption, cpu and cpu consumption, and that should be uploading all those matrix data to one of the endpoint dashboard injection system, which we want. That will give a end-to-end test for us for the sdk.

D

um So probably, I think I'll be doing it and that that would be a good instrumentation.

C

D

For us, which generates an instrumentation library which is generating a matrix, but that matrix would be basically to collect the cpu and the system strike and the memory statistics, while the sdk is up, is generating the that will creating the matrix out of those measurements, and then it is uploading those to the injection system. So we can use that to calculate the performance stats for any injection ingestion, both for the sdk and exporter, depending on the existing system which we are using.

D

So probably, you can use some of these measurements properties from here cpu memory, paging, what all the style? What all active mentioned here we can use that.

D

The only catch here is that there's no cross plate form secret space library, which gives us memory stats for all the platforms, so probably to start with one of them or something and then we can use. We can enhance it for other platforms. Also.

D

Okay, I just just wanted to share that thought I had and we.

C

D

Not something which is something new dotnet is already doing it. I need to see where it is, but they already have that kind of instrumentation library which generates the measurements for cpu and memory stats.

D

C

Something somewhere here, it is.

D

There I was looking into that. One of the doctor developer did show.

A

If we have code that generates a process matrix, uh would that be into open, telemetry, contrib cpp then.

D

Yes, part of adding it in as part of instrumentation. Here uh we can do that or I mean if, if we really want to do it, we can also put it in the test directory in uh somewhere in text or somewhere. In examples I mean I don't know I haven't thought where to put as of now, but at any one of these places we can put it.

D

D

D

D

Add matrix processes to allow metric to allow sdk automatic providers to incorporate metric data from third party sources. This is just specifically for open sensors.

D

uh It's a new component which would be added, it's still not merged, but I think it's, the pr is still in review a new component metric producer, which will directly take the matrix from third party component and it will directly start it can be exported to the injection system. So there won't be any measurements produced as part of the open elementary instrumentation library, but it would be from external directly. The metric data would be there from the external systems.

D

So it's not very important as of now. I think. Probably this could be something which can be done after ge, but yeah good to have a.

D

I mean I have good good to know that there is. There is a new component which we may have to add in the future and yeah. I think that's all in the specs.

D

Apart from that in the prs anything else we want to talk here, you can directly go to the pdrs.

D

D

At ci test with grpc latest release.

D

Assign this is still in progress right, because I have yeah.

B

So I have the cmake part covered for the latest version basil. It's um I think I have to do the basal part in another and for windows.

B

It I saw that it's using the latest one.

B

I mean we don't specify the version for windows bazel, yes or sorry- for windows with a vc package.

C

B

And so it should pick the latest one, but I couldn't find it in this ui. It's probably coming from the cash.

C

Do you mean the latest visit are, do you sub module red for this package for.

B

It already vc package installs the latest vr pc. I believe, because we are not specifying the version.

C

um Yeah, but this packaging stores the latest one check the ink to to like yeah to that reaper right with the package ripple. Okay,.

B

C

We may need to update some some module hash code for this package.

B

No, no, they have one for eight. What do they.

D

B

I think yeah I I saw one for eight zero.

D

Okay, 143, okay, okay, then, probably I didn't see it properly.

C

Okay, yeah, in that case,.

D

We have to update here because we have to change this to the latest.

B

Okay, so it's good.

D

Only problem here would be that if we update it to latest I'm not sure what all things will get updated, this means, let's see once again, it will update all these libraries all right tom g test, which is fine for details, and I think she'll be fine.

C

There should be no breaking, I think.

D

It actually makes.

C

Sense for other dependencies to update into updated analytics.

D

D

Okay, but I mean the I I'm not really not reviewing it as of now. So probably just let me know once just just define the scope of this, because there would be it's it's getting bigger the decider scope and then we can do it in the incremental manner. As you feel the proper way, and this.

D

Yeah, we just want to add this scorpio next for diesel and then for windows.

B

I'll try the vc package, the new vc package and another branch. If it worked, then I'll bring that to this pr as well.

D

Okay and just let's probably.

C

Create another another.

D

Issue, if whatever is the leftover, probably good, to create an issue and yeah? Thank you.

D

Okay, let me just.

D

C

D

A

So the only thing I did is merge with a recent code so that it only shows.

D

A

Just for windows uh to clean.

C

A

The but there is no activity on that.

D

Yeah, we don't have anything.

C

D

More one more time I have to look into the specs of what exactly it is providing then I'll add more comments on this, but I don't know. I have a very few comments, more of the cosmetic method.

D

Let's click on the cases, because just one of the comments was we can remove the weak, pointer and probably add stores store that will be variants. Fine id of that span.

B

Ctrl shift v.

D

C

D

Yeah, this is your top. I need to do something on this I'll I'll. Remove the conflict from today's picks up here.

C

D

Okay and that's all we we have for the first.

D

This one has to be implemented again. I don't think we can take it even if, even if we fix it and we try to fix the ci for this, probably the tlp dependency is not the right way.

D

So the issue is still there. We have to do it.

D

Let's quickly go to the to the issues yeah this one, we already discussed by export of hostplus at ci, test4 uh yeah. That was exposure.

D

B

Yeah I mean this is for the pr we discussed. Oh.

D

Yes, yes got it.

D

D

Yeah we already discovered the last meeting and toxinophilimetry dot. Io is hey mark. Did you have any issue created on the docks.

A

uh I've not seen any yet. uh I will take a look again.

D

I have to look into this it's on me and try to find the time this week. This is some something and priority payments handle partial success, response from otp exporter service.

D

This is this beer merged and we have to do it yeah. This is also merged.

D

Locks for getting started now we have to handle this matrix, that's repaired in interface. This is something probably have to look for: matrix, clean up hold, matrix implementation, uh remove circular differences, and I I think most of these are from matrix. Probably we can go through the matrix milestone and discuss that again.

C

I don't see anything.

D

New which we have to discuss just let me know please correct me: if there's any issue, we have to discuss explicitly exclusively.

B

There is nothing new, I believe.

D

Probably just quickly go through my matrix and.

D

Yeah, we have lots here and I've been bit slow last couple of weeks and sorry about that. Probably I'll try to pick up most of the stuff which I have already taken on myself and try to fix them and.

D

Have a pr for this uh provide builders to avoid exposing yeah. This is already markers and and in memory exporter. This is not it's a priority v2, so I think once prior to b1 is done. You can click, it add support for pull matrix exporter. This is p1 and yeah. I think sound whenever you can see.

D

So this is also p2 exponential. We don't need to do. We can handle it post g, so nothing don't see it immediately.

D

These two, I have to start looking I'll start looking into these two and just want to remove that weak, weak pointer that circular dependency problem to change some of the design in the code. So let me see this clean up. It will take afterwards. Just before the we are going to release, we will do this.

D

Recordable interface is just a suggestion if we can do uh user whatever interface so that we don't need to copy or pass the matrix data from the uh storage to all all the way to the next to the exporters.

D

If we can just use the recordable interface, which exporter provides so probably something which we can see again prior to pay2 api documentation is p1 now I'll, try to create create some basic documentation for this, and probably you can enhance it on top of that, for both of them.

D

Observable guys yeah, this is something we already discussed I'll. Take it over uh try to fix this by this week.

D

Still we are targeting 15th.

D

D

It's probably four weeks or three three and a half weeks from now.

D

Let's, let's see, I think, probably we'll take a call next week where we are based on the progress we have this week means let's see how it goes.

D

Anything else you want to add a sound on this.

B

C

D

So we already discussed issues and uh switch on w error in the ci. That's a good point um mark right. What we have mentioned.

A

uh Well, I did not write that, but I think we should do it anyway.

D

A

I did a lot of cleanup on the on the warnings that.

C

A

And very there are still more warnings to to fix, also- and I think the best way to address that is to do a build with the latest version of gcc and also the latest version of cdang, because they report different things.

B

um One question: do you use cmake or base? No, I use.

A

B

A

um There is another part also which will be annoying. Is that um when I did the fix to clean the warnings, I did local build fix as much as I could and then submitted the patch.

A

But if we certainly say running error for everything that will also include the generated code by protobuf and this code is not clean. So we will have to figure out in the ci how to check for warnings in the telemetry code and yet ignore the warnings from the portable code. So that will be annoying.

B

We can suppress the warnings for for the externals.

C

C

I remember we already did some of them like before, including photograph generated file. We include some our header to disable such volumes.

D

Yeah we did, it was it for bazel, or was it for cmake will be, which we did.

C

D

D

So this comment is from from you.

C

D

C

D

I'll say for all the platforms for gcc.

D

And obviously msvc is not double w error right, maybe some other flag which we have to see and.

A

So we issue to surprise running for external packages that was not for bazel, but for protobuf and generated code.

A

D

A

D

Yes, okay, this is you're, saying it general for both basil and cmake. We have to do this.

A

So for generated coding.

D

We do it for surely I think we do surprise the warning here. I have to have to check where we have where it is done, but there was there somewhere, it was done not sure in cmake or basil, but somewhere it was genuine. As tom.

B

You, yes, one way is to like mark the library as system library.

D

Okay, everything.

B

Will be ignored.

D

Okay, yeah, probably we have to see more into this.

D

I think it's good good to take it once all the warnings have been removed and, as mark you're saying, there's still some warnings which are coming right, which are there yeah.

A

And it's worth doing it because the the compiler fine already thinks so.

D

Yes, it will fix it, I mean as part of the any beyond I mean.

D

As early as possible, probably I think we should enable it once all the existing warnings are removed and let me see to see all the warnings for msvc probably I'll, raise the issue for that also just check for msbc, whatever ones are coming as of now.

D

Tom should we have a release?

D

What do you think.

C

Yeah in the last week, I'm afraid I try to. I saw there's some like the 80w fix I'd like to include in the fake song yeah I'm creating a new release. Today,.

D

B

And do we have some script for release.

C

There is a script I think already included in the repo. I run the script and I think, most part of the release from your noise manually. Editing the changelog, that's a pleasant, hard hat to imagine for other parts. I have to just run a few scripts yeah.

A

um If we do a new release, we should uh align with the semantic conventions from the spec first. I think.

C

Okay, yeah, yes,.

C

I mentioned you mean the release number or anything else.

A

Well, the respect mentioned the open, telemetry spec released 113, I think, and as part of that there are new semantic conventions so that we should generate the code again. For that part.

C

D

So we should regenerate those header files and then then do a release all right.

A

D

You want me or tom to pick it up. I mean I'm not sure what other things you may have to be doing. So, just let me know, I think if you see that you may not have time, I think one of us can pick it and then either is okay.

A

I wrote some documentation last time to with instruction how to do.

D

A

Would be easy, I guess.

D

Okay, so probably let me take it right and I can I can do generate it and then I can ask tom to pick to do a release.

B

D

B

A

uh Some some code in semantic conventions is a radar file which is automatically generated, so we have to run the code generator again.

D

Yeah, that's one first, one semantic yeah, so these two header files has to be regenerated.

B

D

Okay, yeah, to take into account the new any changes in these.

D

So we use this some converse.

D

This has to be changed to 13.

D

So so how are you going? I mean the mark, I mean as you want. I think, because this part was done by you, so I just want you to take a decision. If you want to do it.

A

Up to you, I can do it or someone can do it or something.

D

Yeah, okay, I think probably he can pick it up. You know better for this. I think I.

C

A

D

If you have enough time to do it, so that that.

A

D

Only concern I was saying no.

A

D

Be fun: okay, thanks.

C

B

C

D

Okay, anything else. We need to talk.

D

If not, I think, probably we're done with the discussion, thanks for all the contributions and have a good rest of the week meet next week,.

B

C

Thank you. Everyone thanks.

A

C

C