GitLab Monitor Stage, 16 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Gitlab 12.5 Kickoff - Monitor::APM

Description

Review issues for 12.5

A

A

All right, so we can get started. This is the weekly a fear meeting take one of the main things we wanted to talk about today is the can 12.5 issues and the kickoff discussion. So though, if you want to drive that or if you don't know, if you want me to share you're on something all right.

B

No, so this is for the keep of discussion for 12.5 I think we should like make make it like something. The way that the weekly meeting for a p.m. before the kickoff meeting should involve the kick of discussion for that iteration I think this is put into the agenda like once in a month. I think it makes sense instead of having like a dedicated meeting for that, we should leverage this meeting thoughts or comments.

B

I'm going to walk through the 12.5 issues, I'm going to skip the one that out to schedule because we discussed them on the previously thing and security-related, because this is of course with you. Questions comments provide additional explanations. It's the first issue is around vestigation.

B

Is we want to enhance our metric use case and we are using committees? Committees is not suitable to store metrics for long term. We do need to find a database that can store matrix now. Ideally, it will be able to find database that will store both logs and matrix would be.

C

What is prometheus and Prometheus cortex in in previous job? There, like Prometheus, cortex.

B

C

B

Too, but there are two popular projects that provide a scalable solution for Prometheus and provide long term storage. So there is channels. This is one.

B

So thermos is one: it's an open source project project I think we're using panels as well. I mean infrastructure, team and kotas. Cortex is almost similar, but it's hosted solution for per meters. So it's a federal aid highway, lability hi available for me to use as a service- and this is in in a nutshell.

B

So the purpose of this investigation is to find the suitable database that can store matrix, can query the matrix form store the metals from primitives, and while we query the data form poetess printers can go to that database. Query this data and fold it to us. There are several solutions that I'm, aware of and I will list. The hill until for mobile is a talk that I so in in cube con and what they actually discuss this exact. So this is one approach we can take. So this is the investigation and I would like to.

B

And the second one is the infrastructure dashboard and the idea here is. We would like to introduce some sort of a birds of an eye visualization across your underlying infrastructure. Basically, a question of pods nodes containers that, on top of them the workloads, we don't have a similar solution and I, don't think there is one for graph on. Even there are many solutions that have been provided by the other of gendell's metal windows and I.

B

Think I listed some so in the epic actually in the epic I've listed some examples on how we want to how we would like it to look like, and so there is a suggestion sketch already in place. We will keep the dialogue in this issue.

B

And the next one is to deploy elasticsearch has managed up. This is the continuous effort from the previous iteration after the successful, POC and I think we are in the right place to see if we can already introduced in this iteration elasticsearch has managed up in a managed in a managed cluster.

B

This is to store logs to visualize it within. We didn't get lab, you are and we may end up by the way. This is an update that I will put in the issue. We may end up shipping it with cabana, but so I mentioned without cabana, but and maybe if we see that with cabana, we are getting more value, maybe the first second iteration we will enable keyboard users.

B

And next one this is an old issue and I think is important, and this will allow users to change. Choose any panel on the chart, change and customize the dashboard and save it.

B

Next, one is also an investigation, but ideally I would like to make sure it will also be documented and we even maybe publish a blog walk to it, to see how it goes, but basically I want to simulate ourself as a user. That would like to monitor any type of application. Let's say is deploying MongoDB you want, and the user would like to see the metrics on this. So what are the steps that you need to do? Let's make sure that they are documented I foresee that this issue will expand, eventually create multiple issues.

B

Eventually, this entire workflow would need to be created and to have an ability to create a turnkey solution for our users by a push of a button to be able to display any metric in hit lab UI. So this is. This is the first step. This belongs to the epic instrumentation is.

C

This kubernetes focused or not kubernetes focused or how yeah.

B

It's kubernetes focus and if it's not mentioned.

B

Issue should be this.

B

Next one is the ability to load, fake or paste data into an environment. This was an asked from one of our users and I think it will also support some of the work that we are doing here internally.

B

And the next one is the telemetry information that I would like us to capture. There is a big epic about all the dilemma: trees that we want and I'll keep adding items into that, but for now I've scoped it down to only those items so.

B

This is an issue that was opened by Rubin and I. Think basically, and there are several issues that are similar and I- think they're all related and I think that we have some problem about inconsistent inconsistency on how we handle the back end calls.

B

So we are receiving different, all correspond code and multiple issues that I think all of them are related. So there is a skill-shot already and I know that there is a discussion, an ongoing discussion with him.

B

And this is one of the bugs the candy-coated what he was doing, the product workflow about the population of the system, metric dashboard and yeah. So to be good to fix it, a button create to sell to create a self monitoring project for the self monitoring project.

B

So this is an issue that ran across I, think couple of issues and support alone, multiple metrics and it's a part of the bigger discussion.

B

But we adhere close.

C

Related issue for creating alerts on computed metrics completely.

B

Metrics yeah right.

C

B

Is yeah? This is the one, so it's probably not the same, but it's related I'm, not sure if it's the same, but I suspect that those two are related. So I guess we welcome both, but you know just to make sure I.

D

Just something I think the computed metrics issue is about allowing alerts on sort of like a combination of metrics, while the other one is just to allow alerts on the existing methods.

D

Okay might be the same thing: they.

E

Are they you're right Ruben? They are different. Multiple metrics just means we can add multiple metrics that we're getting alerts for and the computer one is just expanding the language we have to set up those alerts. So computed is like, like you see there, my so we're capacity is greater than 95 percent, whereas multiple metrics means you could have requests over capacity is greater than 95 percent requests over capacities greater than ninety seven percent requests over capacity. You know you can take the same thing and just alert it. Multiple different threshold.

B

E

B

C

So multiple alerts on the same metric then.

E

Well, that or I mean it could be as simple as just saying you know, hey alert when you know capacity exceeds that you don't have to do computed to do multiple um yeah. So.

C

Well, currently, we have the ability to add one metric or add one alert to like a custom metric whatever- and this is saying, let's add multiple metrics to each of those panels- right, correct, correct yeah,.

B

Yeah, maybe we need some clarity, I think I.

E

Figured add a little more.

B

E

B

My clothes basically makes up logs available from the operation tab, and today we don't have it easy way to go into logs user. We need to go to environment and then click on the pod and you see the logs and we want. You first create a section for pod load and what we discussed in the last meeting is that those logs will be the one that will come from elasticsearch.

B

So we'll keep this flow for now and we'll have another flow here, form elastic search and then we need to think about how to either combine one or deprecated one. But for now this is just a placeholder for for the logs.

B

So I didn't custom eight weeks for good luck, managed for me to scope and I've also seen a couple of issues around inconsistent behavior when we have for me to set the instance level and how it affect the group and the project level. So apparently it is impossible to add custom metric to a to manage parameters at the group level and, as I mentioned, maybe the following one. The following issue will see similar behavior I. Think.

C

You I think you can add them from the dashboard page, but not from the Prometheus service page, which doesn't exist for the group or the instance.

E

Really is that true, because that's I think that's using the exact same interface, I'm.

C

100% sure that you can add Wow from the from the from the dashboard page. There's a there's, an add metric at the top, and it's only available for arm.

B

Right, oh yeah, but.

C

You can you cannot edit them you, you will not be able to edit those because they don't there's. No there's no Prometheus service object that is over in the over. On the you know, the the integrations tab or whatever, just because.

E

D

I'm I'm opening a bug about this. So when you create, when you add a project level cluster and you install Prometheus, it creates from this service object on the backend. But if you, if you use a group cluster and instance classify it does not create that service. So.

C

Exact, that's where I got that information really good good job on that Chris, Christie and I were talking about that. Yesterday, yeah.

D

So that add button on the UI checks for that sub subject, which is why the button does not appear all group and instance.

B

You think we should link this bar into this issue.

B

D

That'll probably be I'll, find it and I'll link it. Okay,.

B

Is just a simple fix of the name of the file.

B

So this is this is interesting, apparently and so matrix in a mall not display.

B

Not show up correctly, this is what the issue mentioning the thing that I found very interesting is the fact that Josh thought that solving this issue will fix the problem, and you can see that seven month after world there was another user that complain about this issues were so. We need to look into that, but I understand why maybe she was not fixed and we have two users that complain about that.

B

So here another one about Auto deploy committees can only use one project and there is also like two issues that I think are the same. So this is one when you install vomitus and kubernetes. It ends up in this namespace and integration, so mostly in Abel, but other projects no longer install meters, but.

C

Not if I mean, if it's at a group level or a project level, they can use it and the the namespace the application name space is like group or project name ID. You know production or staging or whatever, for the for the different environments. So the namespaces for the applications are unique, but it is certainly a case with instance, level of level kubernetes installed from atheists that they can do. Multiple applications can take advantage of the same prometheus.

B

So they can use it. Oh they.

E

Can't, except if they want custom, metrics cuz, that's not working right now, if they want to edit them. That was all we were talking about before.

C

I will maintain that they can add custom metrics. They just can't edit them unless they want to go.

E

But I don't think they can delete them either. So it's like a one shot. One shot attempt it needs it needs, but it is possible right.

D

This issue doesn't seem to mention whether he, the customer, tried it with project cluster. No.

C

He said if he set up at the top added it to the cluster or the project and a kubernetes cluster for project. One.

B

C

Kind of need to think ahead and do you know, put.

A

C

In a group or whatever it several times, I've wanted to be able to oh I've got this other project. Can I tell this project to slurp up all the you know, the kubernetes information that I've got over some other project. You know I always make my kubernetes clusters of the instance level, which may not be available for many customers, so it would be nice to be able to copy cluster from another project. So.

E

We can it so like, even if they started at the project level, and then they decide. Oh, you know what this needs to be at the group. For the instance level. They can't remove the integration from the settings and then take that same cluster and put it somewhere else right.

C

And it's an advanced topic to get rid of the helm, tiller and stuff. That's inside the that's deployed inside the kubernetes, you have to have some serious kubernetes knowledge. I would love to have a copy cluster. You know feature to copy a cluster from project one to project two like customer, here's one today. That would be really awesome, but that would mean that when, like, if I, if I install from atheists on project one, the project two would need to, it would need to magically show up for project two as well.

C

Yeah I mean it'd, be.

E

Easier to like promote it from like a project level to group level or something if somebody decided hey, you know what I wanna I'm, not sure, what's easier from an implementation standpoint, I don't.

C

Think so, but I don't know if it will be possible to copy it from a project to one from one project to the other I guess, if I, if I would just spend an hour and do some database magic, maybe I could see if I can just copy them over see if it works just copying it over, but there would have to be some kind of synchronization that where each of these projects knows that the other project you know has done something to the cluster like delete, prometheus or a Prometheus or adding grass or something.

C

But this is a this. We force people to think way ahead and, and we make it painful for them to change their mind later, because all their cluster has to go away and that's adding it close, you know, create a new cluster for the most part. This is righteous.

E

So maybe that's what comes out of this, then is we just start investigating that kind of just easing that pain point. If we started in the project level and decide, you know what we want to use this cluster now for multiple projects, so.

B

What should we do? I mean: how will we enable this well.

E

I think I think probably at this point we're at like the feature exploration level like how do we ease this pain point for this specific issue, though I think the answer that we currently have is to set up Prometheus at the group level or at the instance level if they want to use it for multiple projects. But as David was saying you know if we want to ease this user experience, I guess what I'm saying is we do support using auto deploy Prometheus for more than one project. It's just there's some pain points involved.

E

If you already started locked into one project getting that you know to use other places, so we could we could do some exploration like hey. Can we make this easier for our users, because this is happening where they change their mind midstream, but actually I'm, not even sure that this particular user knows that hey we do support this. You can just gonna have to set it up at a group level. This.

C

Was open one year ago as well and might may have predated.

E

C

This level, oh.

E

C

E

So for this issue, we could probably just comment on that spawn a new issue off of this to explore, making it easier and then kind of schedule that issue whenever we want, but it would take some as David was saying it sounds like it'll take some exploration as to what the best way to do. That is, and you know probably just do a spike issue on this. For now.

B

E

And how to make experience.

A

B

Okay, yeah and the other one is the same. Auto-Detected for meters has already been deployed to Phoenix assigned to a cluster and just describe the same problem. You connect a single cluster to multiple project. Unfortunately, you cannot install parameters multiple times. This thing right right.

C

But we don't to deploy prometheus muscle multiple times, because, if you're, if the things that you deploy to the cluster are annotated correctly, they'll them all of the metrics for your deployed ads will show up in Prometheus, I'm 100% sure about that as well. Okay,.

B

B

I get a spike issue out of this one, and can we verify that this is no longer relevant? Oh, that our solution is different than installing multiple times I mean do we do we think that our customers would like to install images multiple times in multiple project? Is it something I.

E

Think this may also is this about like if a cluster already has for me, this, we should simply auto, detect its presence and enable the service. Is this talking about, like the integration settings like seeing that this is enabled that prometheus is enabled when we have I'm.

D

Actually not sure what this is saying.

E

I, don't know that it's the same thing, though I think this is more about like auto, like we know that we have a kubernetes cluster installed, but we should be able to auto detect the presence of the I guess. This would be the pre-existing presence of Prometheus in that cluster, so this would be more for, like a custom, managed Prometheus, maybe well.

A

C

A

E

Kubernetes cluster yeah.

C

I could see that being a ballot thing for detecting a gillip, managed kubernetes but detecting a I mean a Gillett managed prometheus but they're taking a customer's prometheus I. Don't know that would be I.

B

Think a good luck manage one if I remember a long time. I remember correctly, I think it's for the good lab manage one to be able to detect it. If you already have one and you just connect it, nothing to it and.

C

That's this is very similar to what we just talked about about being able to share clusters between you know. Like you, a cluster, that's already been associated with another project, yeah.

E

This seems like a similar or one potential approach is that we always set it up the same way. Maybe and then, as you add to cluster, to multiple projects, it could auto detect that hey. This was already I'm just a viable approach, but.

C

It stores a bunch of keys and certificates and dooblydoo in the in the database, I'm not sure exactly put all they are, but they look encrypted and important, but I think that those things can be just copied between. You know different projects.

F

C

We wanted to use them, as you know, assuming that the permission that's correct between the two projects had the same have similar permissions, but it gets to be. It gets to be complicated if you know if, if one user has access to a subset of projects, that is intersected by the set of projects that another user would have access to, but they kind of intersect on one or two projects, and that means that I could deploy apps.

C

It would mean that and I'm not sure about it, but if there's an intersection of our other projects that we can have, access to and I have and I add one to the the intersected project and to a not intersected project, then the person that would have if they took over my project I think we just think about the permissions associated with it. So.

E

The use case, for this is a little bit unclear is what I'm hearing is that accurate I.

C

Think the use case is very, is, is compelling of being able to you know, but I think the the permissions associated with accessing the cluster. If we, we just need to make sure that we we cover all our bases with.

C

We don't need to allow indirect access to it, someone else's project through a cluster that they share right. You know what I'm saying yeah.

E

Yeah yeah the implementation details have some yeah have some.

D

The deployed applications of both the projects will have to be isolated correctly. I guess so that okay, no one project cannot interact with a deployed application of another project, something about so.

E

I think I think this is a separate feature, ask with a different use case. It seems if I'm hearing it right and then this may actually just still also be in the evaluation stage, where we start looking into like what are the implications for this, and how do we actually make that happen?

E

They made the solution, I mean I, don't know depending on how we do it or how we set it up. One addressing one might actually allow us to kind of better be prepared to address the other one, so they are related, but they're not exactly the same. It's.

B

Like a spike issue from this one and set it up to them, maybe this duration or next one, and this one is something that we need to also create some sort of a spy issue but see if, while we are working on this one, maybe we can ease the way. We are thinking about that when an owl had a chest with Josh to better understand new stencil didn't have time to walk talk to him. Bob.

B

Okay, this is a bug on when you set the condition alert to the dashboard. You cannot see it anymore, which means I mean after you said it. You cannot see it anymore, which means that you cannot edit. It.

B

One one shot basically.

B

You you support for column, chart stacked column, chart.

B

Okay, this was also something weird which was reported I think it's also related to the way we did the inconsistent approach we have from from the codes on the back end to the front end, so the user reported some sort of a CI job that ignore the metrics with tags, and then they said afterward it was able to detect it and, as you read to the issue, it seems that we still have some kind of related to the inconsistency that we have discuss so.

B

Maybe those two issues are related, maybe not, but we need to. We need to investigate it.

C

Sounds like this that might be applesauce angelic, is it I mean? Are there tags?

C

What's that? Is this a you're looking to get laughs, see ya? Okay, let's see how that one L right.

E

Thing is related to.

C

A

More like yeah.

E

I think the original description needs, updated, I think the real issue cuz. He said it does work after like a minute. I think it has to do with the responses we're getting from the API. That's.

B

E

Front end is interpreting that.

B

D

Many of the backend API is use caching, so the thing is any pair that uses caching when it gets in a request. It checks whether this particular request. The result of this particular request has already been cached. If it's not cached, it returns to zero. Four two zero, two old API is returned to zero for I. Think the newer ones will return to zero two, so the front end is supposed to retry until you get a 200 response now here.

D

Sorry now here there's another problem that returning in 2:04 causes the browser to cache the response. So it's one of the reasons that we started sitting to 2:02. Instead.

E

Yeah, so the front end is I, don't know, I mean we need to look into make sure the front. It is actually retrying in the first place, but even if the front end is retrying, it's a significant potential, significant problem, because the browser's just gonna say oh I, know the response to this. This is cash. Then just you're gonna sit there retrying until it decides that cash is over and then you know.

D

E

It may need two pieces, I mean oh I, think the front end definitely needs to double check that we're actually retrying anyway. I didn't realize so you're saying Reuben. That 2:04 is a front end like we need to be retrying on 2 0 for s and 2 0 2 s everywhere within our code base, essentially.

D

So it depends on the API. I mean it's not like. They pay will sometimes return to 0 4 and sometimes to the row to the apparently the return to 0, 4 or not. But.

E

To 0, for in most cases is just means that it's one of our legacy api's and we should be retrying on to zero fours yeah.

D

E

Then there's as far as you know, I know I'm testing. The limits of the code base is huge, but as far as you know, we never used to zero. For it, I mean no data and you're good. Just just stop. There's nothing here. Yeah.

D

I've never encountered such anything. Okay.

E

That's helpful to know because from a front-end implementation standpoint I I really don't think that we're doing that. So we need to change the way work. Yeah.

F

There are a couple of places where we get two or two and we don't. We try again so I think we'll have to evaluate for a lot of these requests. I'll.

D

Check this API specifically and just to make sure that assumptions are correct and I'll comment on the issue. Do.

B

We need to create another issue regarding those inconsistent behavior through the API, with two zero four one, two zero two I mean II think make sense, but to maybe close this one, no not close and create another one that will flexural problem. It.

E

May be an epic worth creating, and then we can gather these little issues where we're actually doing it. I think it's helpful to have them broken down into their specific places like where we're doing any, where we're not so maybe yeah, maybe an epic for addressing this, making sure we're retrying on to zero fours and on to zero twos, and then add this one to that epic and then, as we find them, we can just keep adding them to those to that. Epic, okay,.

B

E

Don't know that's my pick, I just think it's helpful to keep them broken down. As far as you know, if we find a specific instance where we need to do it to keep that as its own yeah.

B

I agree, I, agree, I, think we should work more with epics and can you can you help me create this? People can great. We can tag me in yeah issues to the epic yeah.

E

B

Okay, and apparently there is one metric from parameters that is behind feature flag on comm I. Think it from the discussion doesn't really nobody understand why nobody responded. Why so I think we need to move the future, and this wasn't asked from Miguel to bomb the version of prometheus. We ship with gitlab.

E

I think, actually, that previous one is already being taken care of by this one, this one or no I'm. Sorry, the one after this, the bum prometheus version, I think this is already being handled in order commented on this I think if you scroll down, but this may already be addressed in a separate issue up just a little more yeah yeah there. It is so he's already working on one on those two issues which should address this.

E

So I don't know that we need to necessarily take action on this, because those two should should solve this. There.

C

May be a movement afoot to upgrade to support the latest version of kubernetes I. Think our helm chart version that we have pan is not working so I, don't know if this will be swept up. In that whole it will yeah yeah.

E

Is that they're gonna unpin the version essentially and then it should just you know, stay up to date, a little better.

B

E

Not it's still in it's still in flight, but I just don't think we'll actually have to tackle it. I think their group will end up fixing this for us.

B

B

It's another bug among how to reflect our customers with the narrow says here. The thing is waiting and waiting for performance data, nothing, it's pretty, nothing happens and they'll change, maybe the texture. They know that I've found or something like that.

D

I think there is a error state, but maybe you can see your J s arrow and the image. So maybe that is preventing the error state from this game. I.

F

Think the the failed exit child is being caught properly, but the UI is not updated over there. We do have a retry off I think three times and if it fails all three times- and you should see it here- state- but it looks like this in this bug- push you look at it.

B

B

Yeah, some matrix charts do not load properly.

B

You just need the video granites with the video, when only when you resize them, they will not, hopefully.

F

E

F

This yeah I looked into this one as well, so it is because the collapsible panels when they're closed, it's not rendered on the screen because of which the charts assume that the width is 0. So it's a pretty straightforward fix.

B

Okay, this one, the lodge- you had a comment on that. So, basically, when there is no data on the chart, you have to still you click on it. It doesn't do anything so the lodge your comment is that let's simply remove all the headers with no data so that we populate it. So only when the data is in, we can show this one. My comment here is: we need to make sure history is also relevant.

B

So if there is no data right now, but if we go yes come up days ago, we need to make sure that this.

B

E

Used to also do it that way where we just wouldn't show it. The reason we changed was because we're allowing custom dashboards. So if a user goes in and adds like some custom metrics for you know and nginx ingress or something and then there's no data, it would be an odd user experience or they may think something's wrong, because they're creating a group and then nothing's showing up and there's just no feedback at all.

E

As to why nothing's, showing up there I think what we agreed we were going to do was in these cases we would start with them collapsed and then, when you click it, if there really is nothing configured, we would just have an empty state that would link them to okay. You need to configure nginx ingress in order to see stuff here or you need to configure. You know whatever it is so we can have.

E

We can open that discussion again as to what the best user experience is, but yeah I think initially we had agreed that we would just have group level empty states in these. In this particular case, yeah.

B

It makes sense if you the way you laid out a regardless I think we need to have like a border discussion on dashboard.

B

B

Something that we will probably take on the next iteration, but for now I think at least, let's fix this one with an extra trim of the head.

D

B

We can discuss this condition. Yeah.

F

Because I feel like maybe it leaves this a one line message in that products of the section saying that as you're seeing an empty section of them, because there's nothing configured would be helpful them removing the arrow all together and you're, just seeing the gray section! Well, that's not clickable doesn't open, doesn't have narrow, but.

B

Maybe we will take a discussion there. I think you know the poach is to have some sort of a message. I think I can have I can a finish.

B

Last one, this is a simple documentation. Fakie started there is. There is a major issue for an infrastructure team for the dog fooding. This is why it went in, and apparently there is a minor documentation fix that we need to.

B

Think this is it for 12.5.

B

But do you alright.

A

No, that's good. I, as you were explaining them all. I took some notes, so I'll try to was gonna put them in the document, but I think I'll. I think you'll be.

B

Better, you should go.

A

Update each issue, we're.

B

Doing the issues that me, if needed, I can also alright yeah.

A

Definitely a few of those we had some good discussion on so I'll I tried to take some notes, so I'll try to update the issues today. If I have, if I can't and then you know some other people mentioned, they would add some context to so the more details we can put in each of these issues, the better yeah.

B

The just one comment, I may add additional issue, a design issue and today or two more and I'll update.

B

A

Sounds good cool w the next thing in the agenda, yet you already talked about a little bit at the beginning about the label, the to schedule, label yeah.

B

Yeah yeah, so first thanks for labeling those issues and to better prioritize where to put them I think that we should take also address suggestion about weights a on the issues and I. Don't think it needs to be like to this book. We need to put weights, at least for me, okay, at least for me. If you will be able to put weights when we are moving sliding issues from one generation to another, it will be very, very helpful, so not on all issues.

B

If you want our issues, I don't mind, but at least something one that are to schedule. Let's make sure to put away well, I think you said it's like zero to five right. This is like the scale. Maybe.

E

All the way to 10 I'm, not sure we had talked.

A

E

Not I think Kenny was a little concern that we have like a sliding weight scale that goes down but I, don't know, I mean we can use weights, however, we're not using them at all right now, so maybe even if weights just represent whatever the current milestone is and then it's okay to change it. You know if it does have to get scheduled to the next one, then we know I mean it is weird right: it's like an issue waited ten or twelve or something I.

E

Don't know something crazy if it extends over, then yeah I don't know, but.

B

I'll tell you what what what will help me so when I say that to schedule now, I can look to the issues. Okay, someone walk on that issue or someone maybe didn't walk on that issue and if I say that nobody walk on that issue, there is no comments. Nothing and I think. Maybe this one is not as important for this milestone: I push it to the next one or maybe to the back. That's fine, but then there are some issues where people are actually putting some effort and say: okay.

B

Yes, how much time do you have left until you? It would be in review, and someone may comment like one day one week and it's nice, but it doesn't give me the perspective of how much effort they put into that. Maybe he has like one week.

B

He walk already on that week and like 50% of the work is done, maybe say we say one day, but he didn't work on that either so putting weights and if we can like say internally, zero to ten return is, let's say: Kang is didn't put any at all to that. 550 percent of the walk is already done. You know three, two one and this way how to like things if I see.

E

B

With the weight of one and two okay, doesn't effort left, so I can schedule it to the next occasion right.

E

So 10 being like this will take an entire milestone. Type of that.

A

B

Doesn't have to be like milestone, I think 10 is money to complete, don't want to complete and then oh I.

E

See you talking about it yeah that makes sense in terms of completion. How how done is it.

B

E

Well, when I use them now, so yeah use it for something.

B

Yeah yeah yeah yeah I mean the schedule really helped me in like.

E

B

Well, that's the whole point of the.

E

Way it's yeah yeah.

B

Okay, the words will help me moment before need to tune it.

B

A

So I kind of wrote it down. So if we're saying like a weight of ten or ten means the whole thing it's to still be, it hasn't really been started. Yet nobody.

B

A

Then a one would be it's mostly done. They might be like one is like if you just use percentages, so one is like ten percent laughter, yeah and ten is a hundred percent still means so.

E

We're kind of custom making weights and our own images.

A

E

That's not a problem, let's, let's try it out, I mean if it helps scheduling like I said we're not using them now. So let's do it. Yeah.

B

I mean if something is complete and done. It's done, I mean only when I mean if it's something like I don't want to overload. You know with with more tasks. So if it's like, when we were sliding one from one iteration to another, then make sure they will be awake there. Otherwise.

A

Okay, okay, is that something if we're gonna start using it, that we should be documenting in the handbook somewhere.

E

Yeah, probably especially because we need to have somewhere to point if some have a community member is like what this issue is. Not you know five wait like well, we use them. This is how we use them. This is how which is I, don't know. Well, we may get pushback on that, but let's start, let's just try it and we'll adapt as we go.

A

Okay and then Reuben, you mentioned the we kind of talked about that 202 and 204. Is this issue that we talked about? Okay, so.

D

The third issue are discussing the benefits of switching to two zero, two, otherwise I think we discussed during. While we were going through the issues we discussed, how to zero tools and to zero force are used right.

A

Okay, so this has a little context behind it. Okay, this yeah.

D

So this has been a problem multiple times in the past week. I think so. We need some developer documentation on how we do caching, so I've opened an issue for it. I'll start with the documentation and get some feedback from the documentation team.

A

A

A

Great, are there other topics? People have we're almost out of time or are we out of time? So?

A

Okay, then, thank you. Everyone I think going through. All these issues was helpful. I don't know if everyone else agrees, every one of had nots. So that's good Absalom.

E

A

Yeah we'll plan on doing this once the milestone at least and maybe kind of going through towards EE. Maybe every couple weeks we can kind of alternate like here's go through the whole board again and here's what's left kind of thing. So we kind of did that last week as well during our weekly meeting. So I think that.

B

Yeah I think we should. We should like walk the board in every every meeting we have for the rest of the time, but for that kickoff one we need to make sure we will cover the entire issues for for the elution yeah. Okay, that's great.

A

Okay, yeah well thanks everyone and have a good rest of your day. All right.