GitLab Gitaly group, 27 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 20200527: Gitaly Cluster demo

Description

Looking at read distribution and observability

A

B

How do you, as.

C

B

The wrong webcam on there too many webcams in this house.

B

Who doesn't ask me if I want to show my video when I joined calls that change for you also James.

C

No I've always had just come on turned off by default, so it.

B

Doesn't ask me something changed recently.

A

Hyejeong, hey well, how's it going very good.

C

Everyone had good labor days.

A

Yeah hmm actually Paul I was at Bolsa Chica. Yesterday, wetlands, no, the beach, are you with the.

B

Beach, okay, yeah.

A

So it was nice, but I think people aren't supposed to like sit out and, like hang out, I have.

B

To keep moving no.

A

Everyone was sending out their stuff yeah, like blankets, umbrellas. Everyone was sitting up, yeah.

B

A

Just a question of recreation.

B

A

I um I have friends who are out there: okay,.

D

A

D

My dad worked erectors.

C

D

Good cookie go! Oh yes,.

B

So yesterday my uh my daughter shoved my son and he ended up splitting his eyebrow open. Oh.

A

B

We had to go to Urgent Care and get it sealed up. They put like surgical glue was.

A

Our excitement.

B

For yesterday,.

C

B

A bench that the kids sit at the table, they're the only ones who said that he fell on my boom. He didn't split it open.

C

B

Times yeah remind.

C

Me not to mess with your son.

B

We were not happy about that.

B

My daughter's for my sons a year and a half Oh.

D

B

D

Four-Year in a five-year old now has been doing the same with the the one and a half year old, so I guess it must be a thing.

B

Yep, it's me so I got I.

C

Prepped before this demo, not enough to work out, it was a doing a demo, but to think about what we should try and cover. Okay.

B

C

I think last week we didn't say rate distribution.

B

Yeah I: don't do we.

A

B

To actually to observe that, because I think that was the issue right, we didn't have a reliable way to observe it. So I.

C

In theory, every giddily node should be emitting Prometheus metrics about the number of raid ops. So, just looking at the usual read out repeat, we should be able to see the reads coming in to different nodes if they're getting round-robin around so rather than looking at the Promethean, this special prefect dashboard, we just look at the dashboard, particularly node. That's inside of the prefect cluster.

B

Where are the other dashboards there.

A

Is actually a specific metric for redress distribution as well yeah.

C

So I think we should be able to find a way to do it. I think we saw a only mode, but maybe we can poke around that a bit more and yeah I'd like to look at the giddily dashboard. A little bit more consider is a giddily dashboard that ships with an omnibus that just shows like standard CPU usage. This.

B

One okay right now.

C

I'm, just looking at your Tindall, oh did.

B

I share the window. Only okay, let me share my screen.

C

Those are the things that were, on my mind to cover does anything. Anyone have anything else that we should cover.

B

Yeah, okay, cool and then I added some I have some stuff that didn't make it into omnibus yet, but it's into the Griffin ax repo. So.

D

B

Loaded that in.

D

Grant with the perf team was mentioning that redistribution was bringing Postgres to its knees. I, don't know if this I guess we ready no that's happening. I hope you did demo that I.

C

Was trying to CJ about that last night, because I was under the same impression after reading grants thread ZJ was under a different impression because, from his perspective, Postgres should handle thousands of raid ops per second without much concern. Yeah.

D

It should be I do and.

C

He was saying this configuration or something else going on so I.

C

Don't know that would be good to exercise see if we can see some stats about what's going on, I, don't know if there's any way to observe it, so that would be a good question to work out what we need to better observe the performance of a be confident about it.

C

So where should we start Paul.

B

Okay, so I'm just pulling up where I stopped in the instructions I stopped right before enabling the the failover geez. Where did I put that it's right there.

B

Getting lost in my test.

C

Also gets to an interesting question that I just got it down. We've got a lot of different configuration flags at the moment. Mm-Hmm.

B

C

Either we're doing manually or automatically depending so we've got the Postgres queue as opposed to the dam break. You don't know. If we gained base, we could probably just have the Postgres queue.

C

We've got the failover election strategy that supports manual memory or SQL.

C

There's the read: only option for marking things read only there's the red distribution option. Are there any other options that are missing? The reason I ask is that's a lot, a lot of different feature, interactions and I think we should probably make it a priority to start rewinding those and removing the ones that are safe to remove, because that's just a lot less co-pastor I'll steer.

A

Maintaining yeah agree.

B

Yeah, it's it's an issue with regression. Testing I create an issue about this, because we're only testing certain combination of features, which is a problem because we don't. We don't know that something conflicts with another feature until we remove a feature flag or make it default in the configuration they.

C

Just saying we don't have default on really that we can't adjust have like auto off. We don't really have defaults. I was confused by that I.

B

Would think anything that is off is default because the way the config file works is the zero value of a configuration field is what gets set. If you don't provide anything in a Tom will file it. So if something false is always a zero value, for example, like an empty string, is a zero value, so I'm over here at a automatic failover leader election. So we want to enable Postgres by because I don't think this is in the above instructions right.

C

Afterwards, if you command F, it should be above I say that because I just looked 30 seconds ago, okay.

B

So then that's enabled right now see here. What else do we have.

B

B

This one failover enabled is this one also in the above instructions, yeah.

C

It was the line above okay.

B

In that case, both of those are enabled so we want to start. Oh I forgot to do the last step, which is import. The get lab repo should I. Do that right now then wait, let's see here new project, let's import.

B

Its coms that what it is for the WWE.

C

B

The full path on that does anybody remember now.

C

You say gives lab call that /ww lab call that.

B

C

Yeah don't get.

B

C

Think it is required. Okay,.

D

I think get will follow the redirection, otherwise I think you have to do it when you're doing it by your recollection.

B

Interesting; okay, which is called www.

B

Make it public, okay,.

B

And let's go look at these numbers over here.

B

Take a peek at what get Elise saying.

B

It's not a lot in this dashboard.

B

So the the reads that you were mentioning James well: can you show me.

C

In that instance, top down it's showing all right, so I had this same issue the other day, but it's only showing the local host giddily, but there should be another three Gilley's because we're still scraping them for metrics right. Oh you.

B

Know what well, let me pull up my let's go back and look at the config, so yeah we have prefect giddily.

C

Should we also add those IPAs to the get elite job yeah.

B

That's what I'm thinking.

C

And do we also need the local host 1 as well? I mean not that it actually matters, because we weren't using the default one, but just thinking what the config should be in the docs.

B

I'm sorry I was distracted. You say it. One more time should.

C

We also include the local host 1, the one. That's like listening locally, the default storage.

B

Is enough to do.

C

I'm wondering from a perspective of the docks, since we're presuming that there's default. If we said this, this will just override the Diddley job that might be present or implied.

C

So be localhost or.

B

B

Yeah I, don't know what the default setting is. Is it specified here at all.

B

Yeah I don't see it in this file. Okay, so we have the new job name, get Ally, don't have a syntax error here.

C

C

Because, in theory we should say the requests right coming through the edge of the nodes on that dashboard, yeah.

B

B

Okay, let's restart this refresh.

B

Jobbik Italy.

B

B

Yeah willing to spend some more time fixing this and maybe.

C

B

C

We go back to the dashboard Maggie it's coming through. We didn't give it very long.

C

There's something coming through in that request right, maybe.

D

I maybe drive a damn for the last 24 hours to the last five minutes or something.

B

Do we aggregate all of our Prometheus metrics into one server, this local 1936? This is this is running on the get lab machine.

B

B

What is the what's, the Prometheus configuration for, if I go down to get Lee does even have a Prometheus, listen address, set, see I, don't think. That's even set.

C

Pretty sure it's Potter they get Elaine I could figure out because we're scraping pieces I think.

A

It's part of the default.

B

There's a default.

D

That's on file.

B

Where's the Tamil fall.

A

You know the VAR opt get lab good Lee.

A

Config that tamil but yeah a good way.

B

D

B

Two: three: six: okay: here, it's listening on localhost.

B

Does that matter, no I, don't think that matters so.

B

C

Only listening, that's only listening for the local stuff, like you're, just looking at a good lad, node you're, not looking at the giddily knows, which is where.

C

Or maybe a misunderstanding: what this listen address does isn't that, where we scraped the metrics from yeah.

B

For the the one that's on the get lab machine, the.

C

Best one that's showing up: that's the one, that's showing up right and it's just showing zero, because it's not request going to it.

A

C

B

A

B

Enormous darker.

A

Fauna, I wonder if it's a graph on that thing.

A

Yeah I guess we can also check the interim pili to make sure they're mini metrics, pretty sure they.

C

Are because the confusing bidders were saying instance, just the local heist one, we would be expecting to see the other three gillies, which is not localhost showing up in this dashboard right yeah. So this meant this. This makes sense like this child, as it is, it's just showing the local get early.

A

Also, the issue shouldn't be 807: five, there I think it should be 1936 as well. It'll certifies the gr PC port. Oh.

B

Yeah so then this one didn't even need to be here because it's already set somewhere I.

C

Wonder if that's actually breaking the yeah.

A

I'm not sure how we're fine that works like if it's the bat configure will adjust reject the whole thing.

B

Yeah I don't know it might just be this graph on a variable that has the problem. What.

C

Does the job name actually control like? Could we change this omnibus chart to use the prefect get early job name I.

B

Can change it here.

B

Okay, so I got a duplicate. It.

B

There we go okay, if I go to all this doesn't work on all I guess so this is broken. If you have, if you want to view more than one, it should be like, maybe like a table or something instead.

B

Okay, no data points. So let me create a repository. Oh wait. I was doing an import today in her today. I probably interrupted that didn't I I do.

C

Rates day, yeah.

A

C

Been generating I signed up for an account and credit Ajay Ramsay / ki doli project, its public.

B

You want me import that and said well.

C

No I'm just using it to generate load so like if there were eight events coming through like I'm raising my repo okay.

C

That, essentially, this chat doesn't work.

B

I set the I set it to the prefix storage. Let me just double check it. I.

C

Wonder with a get laid oxide just like the stripe, Gilly dogs.

C

B

It's such a storage, one we're seeing no traffic on any of these wall. These might not be set up for showing more than one.

B

Actually, look at the request right here. So what is it doing? It's looking at started job giddily. This should be John prefect. Now it should be get alee.

B

Instance, 10 150, oh 41, is.

A

That right, that double is like escaping the back space. I mean yeah.

B

Cuz I think it's a regex so like. If you don't escape it, then it's like any character. Okay,.

A

B

Think that's what this tilde here is for is saying. Oh yeah,.

A

B

I, wonder if it's, because that IP address that's not showing any data because I configured it using the private ip's.

B

Why is this taking so long shouldn't? Be that much data on here.

B

Let me just you just create a ad-hoc query here: let's G RPC server started okay, so we do have data here. Let's go to all so we can see everything.

B

Okay, so I can see all of them now and with this showing this is showing the the rate of our pcs started by each giddily over finite.

A

B

Grouped by instance, so we have, we should have three or we chef yeah. We should have three. Why am I seeing so many? Oh here's the default one mm-hmm 1936 and then we have four and then.

B

Here why are there so many.

D

That prefect and quickly it.

B

Must be yeah well, we'd have yeah, because there's six here, that's weird: okay, you know what I need to put the job is probably why so I need to put job equals, prefect or uh giddily I only want the job equals get Li. Let's try that.

C

What about prefect Italy? Because that was the other job ride, I.

B

Have a syntax error, my prom QL sucks I.

B

Guess ago, before the.

B

B

Okay, so we got three of them here. If I take this out, then we should see just the local host.

B

Don't do that there you go. Yes, that's what we want. Okay, so let's go back to our prefect get Alize, so we can observe this now. I'll just save this.

B

C

So if we do read distribution, we should start seeing these spiking up on the other ones right, yeah,.

B

So, let's see here, let's go back to the one that you had created.

B

So just clicking around I should be generating read traffic.

A

So I think to utilize distributed means you need the neighborhood feature, flag, I thought.

B

We enabled that there's another one. Oh.

A

C

You are, this should be the beer and I believe I oughta, not oh, where.

B

Is that flag here.

A

A

A

B

Have to do that on each prefect know.

A

That that's on the.

A

A

So I think you do featured in a ball. Oh.

B

I have to actually open up the rails.

A

D

B

B

Where am I need to get to? Where is the rails console on omnibus install.

A

Dude get lad, rails space see.

B

Get lab rounds or.

B

Yeah I don't know how to actually get to the rails console from here. So.

A

Where is he will and try to get loud? Shrills, oh yeah,.

D

Give that that's real secret yeah.

B

D

Specie known.

B

As space see, it.

D

And then wait 10 months.

B

What is the rails command for doing that? It's.

B

A

D

Feature the capital F.

B

D

On a flag, just featured enable lowercase e.

B

Okay, its mythic Italy, we just put in.

A

You need the googly, you add. Oh yeah,.

B

It's the other way around and get a leave. We don't need a city Italy, okay, which one the.

A

Last one yes yeah.

A

Get mixed up if you use underscore or dashes or matters here, there's a bunch of magic that changes one to the other good yeah.

D

Yeah it'll say no, no matter what you do, whether or not it was right. Okay, you can set any arbitrary value there. You don't know if it's gonna go to get your nut.

A

B

Okay, so let's look at the graphs and let me click around some more oh, and so you also sent me John the the metric for redistribution yeah.

A

That's the first message: I said in Louie underscore psi squared read squared distribution.

B

Okay, let's do see.

B

Okay, that looks good enough for me. Redistribution.

B

A

Okay, yeah make sense. Oh it's a second-grader.

C

A

C

I've that second graph is showing request rate per get early node, so instance, one is still receiving all the read operations. I've got a loop running my bash channel, just hitting a making a blob request over and over and over again, and so that's showing that the request is always going to node one so something's, not quite working, hey.

B

John this metric here, what is this actually telling us? Is it just telling us that it's turned on or that it's currently being selected for a read.

A

That is that, first graph, this.

B

Is a registration on top? Oh.

A

That's the redistribution on top yeah.

B

A

That's just literally: it's logging, which storage gets the read every time an access or RPC is called okay.

B

A

If the top one is redistributed that would signify this is working no.

C

I, sank, storage, one storage, diddly one for all three, so, okay,.

A

Currently, routing.

C

All requests to get early one okay, so have we actually turned on a feature flag? Did I miss that.

A

Just for kicks Paul, can you just do the same thing, but with underscores I always get mixed up, because in chat options you have to use, underscores and yeah.

B

Yeah no change.

C

By just a second I and my script stopped now it's running again.

B

This is a good test to have someone who didn't work on this feature, actually try to run it and observe it yeah splitting up the team and doing it that way.

C

A

C

A

Use underscores in the rails console for future trends.

B

That's weird because we have underscores here: I thought we did a conversion somewhere yeah.

A

B

Confusing okay, Pete you're flying.

A

Okay, that's cool.

B

It's gonna change this to be a little less smoothed out. I.

C

Think for the prefect dashboard.

C

We should think about what the right chart is for observing this, because I'm not sure that, like well I, guess yeah. You probably want to see that at all prefix of diverting traffic, but more useful is, are the request rates kind of balanced to hold the giddily nodes. So really just one HGD node once like this kind of a chart in the prefect dashboard that.

B

Makes sense and.

C

You want to be able to filter it by cluster, so you can say: okay in cluster, one are the reef rates balanced, I, even cluster two other Reed rates balance having all six there is it's only six. It should be nine right. It's kind of do. Yeah,.

B

Yeah we need bowlines c77.

D

Is a weird number.

B

Yeah we're missing a giddily two and a giddily 3.

D

3 from 6 9, 3, 4, 6, 8 and 1 from 3 8.

D

B

Is let's wait for it to refresh a little bit? Maybe it's just.

A

Well, we have to prefix and reinterpret Achilles right, so there should be 6.

B

Me, let me do a curl instead, because I, don't trust the score fauna. Let's go over here and.

B

So we we have giddily, on which one was only had problems with. It was two three. So let's try 2.41 was a 93 92.

B

What is the actual port that we listen to on it's 93, six I think 93, six.

B

And it's read only something I know.

A

It's a redistribution.

B

Why not finding it.

D

If its capital D try doing a case, insensitive.

B

Capital I should.

A

Be lowercase because it should be the actual metric name. So that's weird, that's not finding it. What eyepiece is that this.

B

One is dot 41, which is make sure I'm not on a prefect mission, yeah, it's giddily, so when I look through the metrics I, actually can't find it. For some reason, let me try a different giddily when you try get alley, one which is 67 and let's look for no I, can't find it.

B

A

You know what this metric is on: prefix, oh, okay, I'm, not good, read.

B

So then, let's try looking at it. Therefore it was prefect, two and three I think had the problem so 68 and connection refused. It has a different port I think. Is it 100 101? Oh.

A

Yeah I think so.

B

10 to 15 I know 268. That must be wrong.

B

B

We use a lot of different ports for the same stuff. It's a little confusing okay.

B

There we go so we got one two three and then, if I go to a different machine, one two three and then, if I go to the first machine, which is 38.

B

Only two one in three well I might only seen one in three air.

A

Well, that one doesn't look like it got very much traffic, so maybe just it's a it is a random like number generator. So.

B

It's not gonna pop up on the scrap until we've hit the counter, at least once so. It's a little confusing okay.

B

But it looks like it's working.

C

So here's this for an interesting idea: let's see what happens if I start pushing rides so I've got my one-liner and I'll just start pushing you refs. So we should see because of the replication job. Everything should start pointing back to the primary because the replica will be always be behind. So we shouldn't go if they're behind right.

B

Makes sense yeah.

C

A

What she meant unless the replication job is very fast, let's see what happens all right, they're all caught up, then they will just do a random select between all of them.

A

All you want to do some refreshes on the page, see what happens.

B

B

C

Interesting, so you can see now that now the instance one that one instance is like picking up and the others are slowing down. That's truly like during the replication delay.

B

It's also because the metric is smoothed out over a minute, so you'll see like interference from the previous set of data until that one minute window has passed. Okay,.

C

I'll make sure my scripts keep running then.

C

It's quite interesting to observe.

A

This might be important to note, but right now, replication that will break after 30 seconds, because we have a bug. That's why replications been broken in Prague just figured out what it was.

D

A

Well, it's because we we drop support for HV 1 and H Mike V 2. We do a timestamp check, okay and in the replicator we cache all the connections, and so we said the prior PC token and then it's too old after 30 seconds, yeah.

D

Yeah I've seen it sure that clocks key because in the air, that's funny that took it would expire within the context of a single replication.

C

Stabilize so I guess the the it's such a small repo replication job is very fast, faster, so I guess like some percentage of ready to getting distributed, but then others are during the replication job.

A

Yeah, that's quite.

C

A

Thing we we could turn a couple of Gilley's off and then and then see if we ever yeah right to go to a stale one yeah.

C

Actually, that'd be real interesting. Do you want to turn off one of the secondaries pull so what happens.

A

B

See here so I'll do get away too I.

C

Won't actually say if there's an error, because it's things like making like 20 requests a second like that.

D

Out of curiosity, what's the load that post Christ is putting up right now.

C

Do we have any monitoring all the prefect Postgres, because this would just be the main place for us right.

B

Yeah I think so see here yeah this isn't the one I guess I have to go to GCP to go, find that if they.

D

Were using the omnibus or we're using runners, Postgres or a separate cloud Postgres reason.

B

The managed stress.

D

Yeah, it seems pretty low.

B

D

Yeah ten percent well I, don't know what to expect, hmm but that's no ton of load reporting on it. Yeah.

C

It's probably like 10 to 20 requests per second. This thing should be doing like thousands of requests per second.

C

B

It's a way to make this bigger.

A

Seems kind of high do.

C

We know how many raid ops per second are happening on get live.com.

B

I think we should have a show.

B

Dashboard for that.

B

C

B

C

Postgres levels, I, guess I'm more interested in, like the OP, a state rate ah faces per second, because a tocar respond to one prefect, like one giggly request to say one guilty: making one rowdy I think.

D

I, just going to and 5,000 is that per.

A

D

Answer is four seconds: yeah uh uh-huh, that's not just read. That's aggregate.

C

Okay, it's not too bad.

B

Which dashboard we're looking at well is.

D

Guilty overview.

D

B

I'm trying to find.

B

Try to find how many queries or transactions we're doing in the Postgres.

C

So we're running short of time should we try turning off one of the rate of the replicas, the secondaries and see the request, chart change so.

B

We put down, we took down giddily, okay,.

C

Did anything happen, but we.

B

Did see a change very.

A

Still pushing I am okay.

B

Zoom windows getting away get out of here, zoom window.

B

Yeah we did see it change me back a bit, so I think we were over here in this period when we saw it switchover. Well, so that's it 11 minutes after oh there's.

C

A yellow one, it just disappears yeah on that I.

C

A

You asked it appears and stops. Her pony should should never come back up right.

B

Whoops don't mean to do that.

C

There we go. The interesting thing is now that green one is like gone to pretty much gone to want right. So.

B

Screen one is instance: 39 39 is Italy, 3 yep Italy.

C

3, so the interesting thing is by turning off this one of the nines it looks like redistribution is turned off completely like if we go back to the child like there's. No redistribution happening.

C

B

Everything's cool.

C

Our line is right well,.

B

Let me let me put some traffic in here all.

C

Right, it's coming back again.

B

D

Pushers on that one in written.

C

I'm continuously pushing and continuously reading for my script, like it's just been running non-stop, ok,.

D

So but that there's not too long of a gap for that to have been replication delay, maybe.

C

That was like a brief pause and it just showed up in my script or something maybe my script just stopped itself as I'm crazy, but.

D

Only I'm one kill a server yeah.

B

C

B

Get labs choking a bit here on pulling up some of these blobs.

A

So it's kind of small, my screen, but are we seeing the leaves not being distributed anymore, I.

C

Think they are being distributed again. There was like a brief applause between like 519 and 520. You can see in that lower chart. There's like that trough.

A

Yeah I see okay.

C

Yeah, it is interesting poll that you saw that changelog page was really slow to load. Just then and I mean I'm, making 10 parallel get requests to view the danger file at the same time, so I'm just like curling that rails page 10 times in parallel.

C

Which is like easier, you should actually be hiding the cat file cache like so there really should be like almost zero get elite load on that, because it's just I know the cat file. Cache isn't catching it's more like process pulling, so we don't yeah.

D

We are working a good process and the file itself should already be in the file system. Ram, cache yeah.

A

Take a file cache the way works, it's per Rails request, so.

C

So that won't be here but I think what we'll sang is like. It should be. The hot in the file system, local caching, right I'm, not kidding like random pages on the repo right right.

D

If you hit P and B in in your browser window, we can see where we're spending your time should get the performance bar to pop up Postgres taken a long time. They're weird.

B

Yeah, maybe there is something wrong with the database. That's causing us, because these are small changes being made to the database right James are to the repo.

C

Yeah all the other. The other thing is that that file content is actually coming through the rails application, so maybe canned is actually causing rails problems not guilty problems, because that will be going through a markdown rendering pipeline.

C

That seems acceptable.

B

Pretty small 152nd.

C

Yeah I'm surprised the rate volume so low, so.

D

How many requests per second? Are you doing James 2007.

C

D

Reads or so, per request coming in.

C

Yes, so I'm making yeah about yeah something over 10 10 requests per second to.

D

Every of 5,000 requests per second on github.com we'd have 35,000 reads per second and the Postgres database, or a perfect database.

C

Yeah, it's probably generating a whole bunch of well I mean it's not even going to be triggering the JavaScript stuff, because it's just a curl. So it's not even gonna, hit all the async Java stuff, JavaScript stuff that actually loads bunch of this stuff. Asynchronously.

B

Anything else you want to see I.

C

Think that's pretty useful. We didn't look at read-only mode, but I think that's alright, I think this was pretty useful, so just to wrap up the action items, I think I'll. We need to check if there's issues for all these config options to start removing them, so I'll post a link to all the ones that I can find and then maybe we can create others for the ones that we can also remove. I.

C

Think there's a bunch of fixes that need to be done to this get lab omnibus Gilly brought either to the dashboard itself or to the configuration the scraping config in the cluster set up so that this dashboard works out of the box, and it looks like we're already adding redistribution charts to the prefect one. So that's all right! We just need to consider this one I.

C

Know I still have a guess like questions about performance after this demo of this feature like it seems like it should be fine in some respects, but then it's strange that the giggly changelog was so slow to like feet.

D

You're, the bundled only most database to be the bottleneck there well yeah.

A

You can look at the how long it took in Italy, right, yeah,.

D

It was 80 milliseconds, work, Italy and 600, milliseconds or Postgres feel.

B

Like we don't have a distributed, tracer or something so we can see where these bottlenecks are it'd, be really easy to visualize with something like that.

B

D

It's available: that's when I live for a while, I, never used it, but I've never seen a customer's either. But it's there all.

B

These questions were asking, we could look at it and it would tell us right away.

C

All right, well, it looks like we should try and get and rate distribution up and running like looks pretty good. Let's try and get it on by default. Get some shots in there. So I think again. The key action item is: we've got a lot of the building blocks, but we just don't have observability out of box. So this is just a lot of factoring with getting dashboards working and making sure we have can see what we need to see it be confident things happening so.

C

I'm at Paul do you think, there's issues for these observability things already or do we need more issues again.

B

For the giddily specific ones, yeah I don't think anyone's looked at these dashboards and forever so I don't think there was any issues.

C

All right, I think I'm not have credit upon the other day, I'll double check and we can repurpose that one and then we can go from there and that'd be the only action item. If someone could pick that up as a priority before next demo, because yeah this is.

B

Like what we should be looking at great, let's do some dashboard changes anyway. So I could take that sweet.

C

All right thanks. Everyone thank the demo cool thanks, Paul right back yeah.