Open Telemetry Uncategorized, 26 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-05-26 meeting

Description

Open Telemetry Meeting 1's Personal Meeting Room

A

Everyone is gone.

A

Can you hear me.

B

Yes, hi everyone, hello,.

C

Hey how's it going.

A

Yes, you can't be alright. How about you.

D

Hey everyone hey later.

E

What's going on diego.

A

A

Back to the python meeting I was out last week.

E

Yeah, it feels like I haven't seen you in months, man.

A

Well, you haven't seen me in mudspray yeah, that's true.

A

Oh by the way, next week, I'll be out of office, so I'll be missing. Next week's meeting as well.

E

Okay, no worries: uh do you think it will be the whole week.

A

uh Pretty much I'll be out of office monday, tuesday, wednesday and thursday I'll be back on friday.

E

Can we can um have our triage session the week after that no worries.

A

Yeah, oh yeah yeah. You mentioned that um and also um I'll, be in the european time zone until november. So oh cool, nice, nice, nice yeah everything pretty much stays the same so but just to let you let you know traveling um yeah a little.

D

Bit of it's awesome, man.

E

Looks like aaron isn't here today um kind of wanted to bring up the uh creating the open issues for metrics. um But we could just like leave that until he's here.

A

Sorry, what issues.

E

Oh yeah, I think you missed the meeting last week. um uh The the the point was uh since our metrics sdk rc1 is out now uh we're trying to get more people to use it. So the question was like: how do we do that?

E

um So I think documentation was really important and I think aaron was uh had a few places and points in which he knew we were lacking. uh He was going to create issues for those.

A

Okay, documentation issues yeah all right on related topics. uh There was one feature that we left out of drc intentionally.

A

um It's a shoot requirement of the spec that says that if there is a view conflict, we should print out our review recipe that will help fix the conflict right so yeah. I intend to work on that. I am now a little bit more focused on matrix instrumentation, but uh will be something good to have uh before it will release.

E

uh Like um metrics in instrumentations mean uh yeah cool cool cool.

E

Okay, that sounds good. uh Let me.

E

uh Do you have an issue for that? uh No the feature. Okay, I can open that issue. Okay, cool.

E

E

I remember someone was starting the uh what's: it called the uh collecting metrics in the instrumentations for um requests. uh Sankey ash wasn't one of you who was starting that.

B

Yeah, I am working on requests and some people working.

E

Oh awesome, uh how is that going? Do you any help with the uh with that.

B

He actually, I tried using the console metal exporter, so uh not the otl so that had some issue so uh also mentioned in the channel. It has a bug, so I'm planning to use the export.

B

Actually, I tried the otlt exporter to send the matrix to collector, uh but the metric. What logged was just like a single dictionary type with saying not the whole information about what metrics are sent. So is this as expected, behavior. Actually I wanted to ask that also.

C

um What's the log level uh did you set in the collector config vml file, you, you need to set it to login log level debug. Otherwise it just prints the number of.

B

Okay, I think that might be the problem. I use the basic with the one.

B

Eml file, given with the example directly use that so we didn't check uh what was the log layer I'll check it.

C

Once if you run into any issues still being in the channel, I found some.

B

Locations where to make the code changes so I'll, try it out and close the prs if the blocks are respected.

E

Awesome, I think I'll, take a look at the metrics issue here.

E

um It looks like someone closed it by downgrading it, but I don't think that's the correct thing to do, uh but I will add the issue here so cool all right awesome uh does anyone else have any other topics that are unrelated to pr's or issues before we get right into it?.

E

All right sounds good. I'm just going to share my screen, then all right, cool first up so conti had something about this.

F

C

Yeah some somebody commented that uh that they are moving sleep over this issue, so so.

C

um So basically, the issue is that um our auto instrumentation like if by any chance um our art, auto instrumentation, sets of the logging sdk. um There is any any config changes that people make um like any logging configure people try to do it in their code will be ineffective uh and it does not bring any locks to the console uh like their application logs will not be um uh directed to the std out or stdr.

C

um So that's the issue that they are having, but if, by any chance their application code runs first, our two instrumentation code will not be able, so it's either either that or uh it should be. It will be only one of them uh which which is effective.

C

So this is an issue for many people, so uh some uh this has been there for quite some time. um Probably many are not facing because um their applications have. You know some configuration involved before uh the sdk part that we do. um I think we can.

C

We can do something uh we can either revert the the logging sdk, auto instrumentation we are introduced or we can have a env or a flag which makes the this instrument like a sdk instrumentation disabled by default, and then they should be enabling it like if they want it, they should be enabling it by setting up the env or uh passing the command line up. um So these are the two things are like two solutions that I have right now. um I was wondering uh what what do you guys think about it like?

C

What should we do.

E

Right, um so, if uh let me, I guess see if my understanding is correct, um one is to have the option of giving the option to the user to add the log handler if they want correct.

C

Yes correct um so by default it will be disabled uh if they make sure that uh even enabling it works. It does not conflict with the login configuration in their applications. They can enable it by setting the env. That's the one option and the other option is to revert that all change.

E

What was the old change again? uh Could you reiterate that.

C

Yeah, so this the this, uh this unit- logging it wasn't there earlier. um We introduced it in s uh pr to uh initialize uh like just mimicking what we did with the tracing. um This sets up the whole logging sdk.

E

Right right, um I'm curious that, like if people wanted to use auto instrumentation with logging, um wouldn't they expect like everything, to be like the pipeline to be set up for them without having to do anything. Yeah.

C

So just enabling.

E

An environment variable, for example,.

C

Yes, that's correct. um This is what it's doing uh right now. The problem is uh problem. Is that once we add any handler right, let's say I mean I, I wrote the description. If you can go back, I can. I wrote it here. The this comment like once, we add the handler uh any call to the basic config does not do anything um and there is no stream handler which means that their um error logs are you know, application logs will not be uh written to the std out.

C

I study error, um which means that they are not now they are losing their application locks right, so they don't want to do that. um So what do we do here now with say that um right, right.

D

E

Sorry go ahead.

E

Yeah uh in my opinion, like I prefer um the default behavior to be enabled and then have some sort of option to turn it off. Instead,.

C

So the thing with the default option being enabled is that that now there is a there is a chance that they, their application logs, uh are not being um collected like uh if you, if like, what's that the default behavior right now is it's enabled with this. They have. The issues like uh their applications are logs, are not showing up right right, which is not desirable for them.

E

Yes, correct, um I'm just saying that, like I think, uh if we, if we disable it by default, right um like it, will add more confusion in that like.

E

Why am I not getting telemetry instead of the other problem of why my logs not showing up um like, I think the default behavior for open telemetry instruments should be that, like everything is instrumented, um but if they're you know if if they want, if they're using basic config, and then they have the option of making it so that uh this is turned off, um it's like it's like they're, both they're, both kind of undesirable situations but, like um I think, uh like missing telemetry um um sorry uh having telemetry is like should be the like.

E

The purpose of the the command in the first place right, I'm just switching the default behavior.

C

The correct um I I understand what you're trying to convey.

E

Just my opinion, um like it's, it's I'm free to discuss this. um If someone has a uh someone can argue also that, like we don't want to affect user behavior at all right because that's.

C

Like the nature, that's my point um is that this open, like they do not know that, like I, I'm not sure how many people are aware that we have the login, sdk uh telemetry and then the auto instrumentation, for that, I'm not sure how many people are.

E

Correct correct so.

C

If they, if they have no idea about this whole logging, sdk and then suddenly, their application logs are not showing up. That would be very surprising for them. um So, given that our vlogging sdk is still not, you know stable experimental and they it's important that they they do not miss their application. Metric yeah.

E

This guy's, like losing sleep over it too, so not great. um I think I think I'll be okay with uh well. What you proposed here is.

B

E

uh We kind of like document as part of the instrumentation process, uh like you have to set that environment variable sure.

E

If that makes sense.

C

Yeah sure that makes sense we can also. Meanwhile, we can try to come up with the solution. um You know that makes it work uh in in all cases, even if they're calling the basic country or not.

B

C

But this person losing sleep over. This is something.

D

A

Question here, so it.

B

A

This could be, I mean the user could manage to get their ideal situation if they could configure everything uh in a way that they want right.

A

E

And get telemetry.

A

And get to enter right, okay, so the the problem here is that we cannot offer that configuration mechanism to the user via just uh environment variables. Is that the situation that we bring.

E

Oh, no, no uh for auto instrumentation it. It automatically adds the log handler, but any subsequent calls to a basic config will be ignored in the user's application, because the initialized logging happens first, so the problem is like they're actually missing their because if they add like a stream handler later on to their basic config, they're, actually missing application level logs, um so their their code behavior.

E

The app behavior has actually changed, um which is what circanth is saying is like we shouldn't be doing, uh which is why um we're saying we might want to disable this by default and if they want so that they, you know they get all of this behavior still like the logging uh to stream handler and if they want to enable logs to like otlp or something, then they set some environment variable um explicitly.

E

F

E

It's still, it's not like still not a great solution, because it's like there's not a way to have both.

A

Yeah, I was thinking about how to have both and I.

G

A

The solution, a general purpose solution would be to, uh I think, be able to register hooks uh for artists limitation that allow the users to run arbitrary code.

A

I think we did that once remember. We didn't. We have a pre-instrument hook or something.

E

uh For auto instrumentation yeah, I don't remember something like that, but we, I think that I think the point is like we don't want the user to have to change their code. Otherwise we could just tell them to pass force equals true into basic config right.

A

Yeah, what I'm thinking is that, if this uh configuration can be done via a hook, then the users could run or experimentation with a different.

E

So what what configuration like? The user's basic config.

A

There is a some code that if we run during our instrumentation that affects the user right.

E

uh Yes, it's this right here, yeah.

A

So um the idea that I have is that maybe we can move that code into a hook and let the user decide if that code. If they want to run that code or not.

E

Wouldn't that serve the same purpose as just having an environment variable.

G

A

With uh the possibility of the users running any kind of code because they can make custom hooks, maybe I'm getting into a solution that is over engineered for this problem. So.

C

The thing is that if they have the like, if if this, this issue occurs for them, when our unit logging gets executed, first, let's say: if they, if they have some, you know code that gets executed before this right, then there is no problem for them like it works, because we are only just adding the new handler to what's already existed right. This. This problem occurs when like when this unit logging gets executed like in the whole application.

C

uh You know in third party logging or their application logging in the whole uh process that they are going to spin up. If this, if this is the like login.getlogger.handler, if this gets executed first now they have the issue. Let's say if they have some some sort of code which is running. You know like prior to this, which is calling some basic configure, which is you know, just printing some log warning. They will not see an issue.

E

Hey uh just to clarify, um um if, uh if the user's code has like executes basic config and adds a stream handler before this is executed um like like the the handler is added as like a like in sequence right so right. So how come it's not the same? Vice versa like how come another if the user adds a handler after this? How come it's also not added in sequence,.

H

So they don't explicitly add it. So what.

C

Happens is that um when you people, don't like people, don't explicitly you know, add stream handler in their code bases what they do is they get the logger and then they do log dot, one or log dot um like log dot error, or something like that. So what happens is when you do log dot, error or log dot pawnee.

C

This internally calls the log dot basic config, okay, which adds the stream handler.

E

I say they add the stream handler to the basic config.

C

So when you log it calls the basic config, which adds the stream handler.

C

So this internal operation, which is uh being done right, this only happens when the when there are no handlers at all, like that's the python standard, really logging of code logic.

E

I see god I got it and because they they're this happens first, um a handler. It exists right. Yes, I got it.

C

Let's say for some reason: they already logged and then basic configure is already set up stream handler is already there. Then there will be no problem, because this is just adding the new handler to what's already then. In that case they will not have any problem, which is why I think most of the people do not have any issues like this. This person has this issue, but you know there are many other people who are also using this auto instrumentation right.

C

They probably have some sort of logic, which is like that's the flow that they are having, but this person for some reason and they um our component, is getting like interest fast. So now, they're running into this issue.

E

Would you say that this is a uh a rare use case, not the common use case.

C

Yeah I mean it's uh totally possible, but.

H

Not like not possible, but yeah.

E

Yeah yeah because uh yeah damn like.

C

I mean, uh um like two issues have been created for this so far, um so we can assume that this is something to be addressed.

D

E

um Yeah, uh I can't I can't think of a solution off the top of my head right now.

E

Is there a way to like add, like a like a dummy stream.

C

um Yeah that that was one of the uh solutions that I had also so I mean uh if we initialize like, if we internally call the login that basic config, which sets up the stream handler, they will still, they will be able to see the you know application logs, but the problem is that if, for some reason, let's say I I I as a user, I want to you know, format the log message to be differently right.

E

Right then, I will I.

C

Will explicitly call the login.basic config, but which is then again another issue? If, if I call it once in our sdk, then login.basicconfig will not be, it will be right right.

E

Okay, does anybody have any other ideas right now.

E

uh If not, uh I guess we can like brainstorm offline. um I would have to like kind of see python logging behavior.

C

Yeah, um please please um look at it. If you have any questions.

C

My I think what I, what I propose is that introduce a new flag or env, which disables for now, like I'm trying to give it for now. um We will figure out if there is any way to you know, make it seamlessly work for all cases.

E

Okay sounds good.

C

E

All right, nice awesome, uh cool, moving right along.

E

C

E

C

This earlier.

E

Yeah go ahead: let's go.

C

Yeah, um so this person has a issue where uh so they like the size of the batch, is that too huge and the collector is rejecting it. Our our retry logic only handles it, like you know, trying to retry the same batch again. We do not look at the response code and then try to uh retry the appropriately. We just read right without you know, looking at if it's a resource exhausted or if it's a transient error.

C

um So this person is having an issue where it fails after retrying, because because we are trying to you know export the same batch without dividing it further um so yeah. What do we just want to bring it up that? What do we want to do so? My? What I propose is that we one of the solutions that um iron proposed is that we divide the uh it into you, know the uh chunks in the metric reader and then call um exporter for each of the batches.

C

What I am saying is that any batching logic should be like dividing the batches to the you know. Smaller chunks should be done in the exporter um yeah. Let me take a look at it. Add comments: um what do we think about it.

A

Yeah, let's take a look.

C

At this issue,.

A

uh Notice that there is a submission of uh some discussion on the spec about, I believe how to specify this response. That will tell the explorer that the badge was really rejected, because it was too big. um Is that already defined.

B

A

Mean the the message, the signal that will tell the explorer uh the badge is too big. Is it's already standardizing the expected in the program.

C

um I I am not sure about that. um I'll also have critical. If that's, if there is a there's, a status quo type which indicates that.

A

All right yeah, because I guess that needs to happen first right before we are able to fix this.

E

F

E

To go with this right, if we try to, if you just give a configurable batch size like it, I don't know if it won't matter of what the response code is return.

A

Yeah, but uh I mean if the configurable size is still too big, things will just fail right continuously,.

E

That's true, um I would just think that, like users would create, it would change it to a smaller patch size. At that point,.

A

Yeah but I don't think we even inform the user that the problem is that the batch is too big.

E

Yeah, that's true, that's true!

E

I would assume it would be the same behavior for tracing then too right same problem.

A

C

So in the tracing, what happens? Is that each span which which you know which is ended, gets added to the cube in the processor and then once the either queue becomes a full? You know we reach the timeout, then we trigger, but in the matrix, what happens? Is we collect them? Like you know, uh at intervals, it's not like. You know, they're getting added to the reader where we maintain a queue, and then you know once cube becomes full where we export it. That's not the case for the matrix in the in the matrix.

C

What we do is we eat. First, we wait for certain intervals and then we collect it. It's not that we are maintaining some sort of queue in the reader so which is where it makes the difference between the tracing and matrix.

E

Well, I'm assuming that the if the span processor um has a batch size, that's larger than the the limit. This would also happen right.

E

E

uh Talking about what diego said, then, if that's the case, then we would probably have to see if this is a valid, dependable kind of uh status code to be used uh for logic.

C

Yeah, true, uh if like we, we need to look at if this is a status code that should be handled in the retry logic. We only look at the transient errors right now. uh I I'm not sure if this includes that or if we, if it's out of that list yeah, I think we should do it next ideal.

A

Situation will be for the message to tell what is the maximum batch size that can be handled so that the explorer can adjust and send something.

C

Yeah, uh I believe we already uh like the we already look at the response size. I I I remember looking at some logic where um the we look at the response, and then we see if it's uh if it's a size and then we divide it, uh I don't recall where I looked at it, but yeah that would be ideal.

C

D

All right, cool.

E

uh So khan, you want just like a message here that way like like investigating this possible sure thanks man cool any other comments on this.

E

Like that's pretty, it's pretty good, all right all right last issue here by carol.

E

Ooh, oh, this is from a while ago, is carol here.

F

Yeah, I'm here.

E

F

um Hi I just um joined um xj with nathaniel aws just thought I would say hello.

F

um So I've been working on trying to get to the source of this problem and um thought we would ask for some pointers if anyone has any idea where things would be going wrong with auto instrumentation because seems like maybe the flag that's being put to suppress, http instrumentation might be getting changed somewhere down the line in auto instrumentation um because of the extra span that's only showing up when we use um auto instrumentation for the sample apps.

E

F

It doesn't show up with manual instrumentation, just auto.

E

um Have you done any testing uh like you're able to verify the this output.

F

E

F

Example, apps, like didn't change.

E

Anything I see um and then.

G

The auto is going to.

E

Oh sorry, guys no.

G

Sorry is that is a problem. It's instrumenting. It twice.

F

E

D

A

E

A living request, yeah.

I

Sorry guys, I hope you guys doing well congrats on metrics ga, so awesome, um yeah carol is coming to help on some issues on open telemetry, and this was her first task.

B

I

Yeah and this one I I posted it a few months ago- it's not that anything's been instrumented twice it's that um url is being instrument or creating traces when it's not supposed to be because requests should have suppressed it. So it I think it's been here for a while and yeah. We were able to reproduce it still. So just wondering if you guys know anything changed in auto instrumentation in the last few months, which might be messing with environment, keys and stuff.

E

I don't believe so.

E

Yeah, I don't I don't that's, that's not related. um He only moved. uh He only moved the piece of logic, adding the uh like changing the python path to the beginning. Instead of the final block yeah, um have you guys tried like actually like seeing what the environment variables were like the values.

I

We haven't tried that that's a good idea, basically just printing them out right before url of choice, creating the the spam but yeah. It just seems like a tricky bug like the url lib code, looks totally correct.

D

I

It looks pretty bulletproof but yeah, it's weird that it only happens in auto instrumentation yeah. That's a good thing that we can try next.

E

Yeah yeah debugging online instrumentation is kind of difficult.

B

E

Yeah especially like, because you can't really like break point, I guess um yeah.

G

E

Yeah, like I, I think you could try to like pinpointing exactly where maybe the environment variables are being overwritten if they are um or it could be, something.

B

E

Yeah but at least for what your question is, I don't believe anything has changed for auto instrumentation in the past few months. So.

I

Okay, thanks guys, yeah and we'll keep investigating, and now you guys have the context, we'll ask you questions later. Thank you. Yeah.

A

I

A

The record, I think we don't have any mechanism that will prevent double, uh twice instrumentation to happen and if I remember correctly, the otp one, the one that deals with all instrumentation, I think it says that we should not instrument twice.

A

Just for the record. I think that's if I remember correctly, that's the feature that we are still missing.

E

Wait, diego: are you talking about um instrumenting the same instrumentation twice or just.

G

E

Instrumentations with the same hot path, no.

A

No, uh um it's just for the record.

G

Maybe it's not.

A

Related to this issue, it's just that. I I just remember that uh that says that if we run our instrumentation on a code that already has a manual instrumentation, it should not instrument twice.

E

Right so um we don't have a spec out feature, but I already created a uh I already implemented that feature for all of our instrumentations um there's a flag that we use. Yeah, there's a.

B

E

We use to it's like is open, telemetry, instrumented or something that prevents that so, okay.

D

Right, I remember that.

E

D

E

But that that's uh a bit uh that's unrelated to this problem. This is because two instrumentations share the same code path and if you're instrumented, with both of them um for some, we handle it in the manual instrumentation case in which we use an environment variable so the url. If it sees that environment variable which is set by requests, it ignores all spans created, but for some reason it's being ignored in auto instrumentation. So all right, perfect, yeah, okay, cool thanks. Thanks carol for addressing.

F

E

Yeah please now.

F

E

Us know what you find: yeah cool, um any other comments on that.

E

All right, if uh also carol, are you uh set up in like in terms of like uh um contributors and uh like the slack channel, and everything like was this recently that you started working on x-ray.

F

um I was here last year as an intern, so I'm on the slack channel, but I don't have anything else, I'm not sure what you're referring to.

E

uh Like uh like a github account in the open, telemetry org, so that we can like assign you stuff and like.

F

I don't think I'm on the open telemetry to work.

E

Okay, cool uh nathaniel is that something that you can help her with like um yeah.

F

Totally some stuff- I know I remember watching it started.

E

Awesome cool sounds good. Thank you awesome all right. Let's we'll run along. uh We have one pr here, like diego.

D

All right, yeah.

A

Yeah, that's a it's a fix. I added for an issue. It's a bit complicated.

A

A

Yeah, what happens is that there are some instrumentations where we wrap the connection object. uh I mean no, let me start again, there are some database related instrumentations uh and they use this dbapi instrumentation. That is common for several database exterminations like sqlite radius, I think, and some others and in one part of the instrumentation process we change the connection object for another object that uses a wrap.

A

A wrapped object that acts as a proxy, so this object is not the connection object, but pretty much behaves like that uh for all purposes, and so far that has been working great because it's super transparent and the object acts like the connection object that it's wrapping underlying right.

A

But there is a very specific edge case where there are some instrumentations that perform a type check on that object and that type check is not done in python, but in c at least for the c python implementations and that c check c level check does not uh see, as this wrapped uh object as being the same as the connection object.

A

It detects it as being different and makes things fail. So I refactor the trace connection object. Can you open the files change? Please.

A

um So, let's scroll down a little bit, please.

A

A

um Now, scroll up again, sorry, I I think I missed um it's just a one line of code. Let's scroll up a little bit, just close okay line, three one one, uh the the important part is that the first class is type connection. So these traces connection, proxy object, is, is no longer, as you can see, no longer inherits from wrapped object proxy.

A

It now inherits from the type of the connection object, so it it does. It dynamically defines the parent class to match the connection object uh class so that this will make the it pass. The c check, because this object is actually uh an instance of the of the original connection object, um so so yeah it it.

A

G

A

It's a a fix for a particular edge case, so I wanted to explain why we are doing that type uh connection, inheritance, uh dynamic inheritance, because it's uh quite controllable leaders why we need this. It's just because of this c level check.

E

Cool that makes sense to me and that must have been horrible to debug. It was.

E

uh Is there a way we can add a unit test for this or no no.

A

E

A

E

A

I mean we just compare.

E

A

Yeah, the problem is that.

A

We will need to.

A

Create a unit test in c.

E

Oh no, I meant to simply just like, um like replicate what c is doing.

A

uh No yeah, we can't the problem is that the c function: the okay. There is a c function that has a binding to python. That does this check there. I think I I I added that in the description. In fact, there are two c functions. uh One of them is the the the one that matches this instance, and there is this another one, uh uh yeah right there, pi object type check. Can you click that link yeah?

A

So uh those are the dc functions that that one is the one that that's failing right um and that one is not the same thing as this instance. There is.

G

Another one that matches.

A

This instance, but uh yeah so for us to test down to that level, we will need to create c code.

E

What there's no way to replicate it using python.

A

No because that that has no equivalent in python, uh the other one has an equivalent, which is this instance, but this one doesn't sucks.

E

Let's see I see, okay makes sense to me.

E

All right cool sounds good. uh The explanation makes sense, also curious um um how come we're still extending from a private uh class like what is this? The.

A

Purpose of this yeah that also needs to be explained. um Why did I do that? It was because I needed: uh can you uh control f, underscore trace.

A

Okay, yeah: what? Where are we using, that can scroll down.

A

G

Screws they're just the tests right there.

A

There they're there right there here yeah, so at some moment I needed to make sure that the connection object was this type of object, this new type of object, so to make sure that it was this kind of type.

A

Since the other class is dynamically assigned, and I don't know, I needed a fixed one that I could compare against. So I added an additional fixed class that does nothing just so that I can make. That is instant instance check at some point.

E

So how come we can't use the this class.

A

No, no, the problem is that. um uh Why can't we use uh the tracer connection proxy plus.

A

It was because.

A

A

Something else could be, I don't know, that's a good question now uh when I was uh making the change. I I tried to replicate the the old code and what the old code was doing was checking for the parent class wrap object proxy. So since I did not have that anymore, I added an additional part class, but now that you mention it, maybe we can not do that yeah. If you want to just add the comment in the pr and I'll show you.

E

A

That parent, but.

E

Maybe you, you might have a point there. Actually I actually kind of I've worked on this before and there was a reason why we had to do it in the abstract, like the very abstract um thing, but I don't remember so yeah if you could maybe just test that out again that'd be good, but you could be correct in that. We might have to use this yeah.

D

E

I'll just make a comment: yeah. Okay, thank you, cool awesome. uh I think that's our last pr. uh Does anyone else have any issues prs or topics that we want to talk about.

A

You know there is a metrics related issue, um something is working.

G

With our json representation.

A

Yeah yeah, I already assigned myself.

E

Oh okay, um it's great man.

E

Good thing good thing: we had an rca and not just ga yeah.

B

E

Cool thanks, diego um yeah, and I guess uh you know we'll give this a while. um um Hopefully, aaron will be here next week, so we can talk about the outstanding document issues, so awesome cool all right. Anyone else have anything to talk about or all right, if not I'll see you guys next week.

E