GitLab Database Office Hours, 25 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Database Office Hours 2019-07-25

Description

0:00 Efficient counters, usage ping, approximate counting, batched counting
2:10 Migrations: Background, post-deploy, regular

A

And we're recording so welcome to database office always great to see you and we have a lot of topics on the agenda. That's great too I don't know if Victor is on a call.

A

All right he committed his absent absent today, so he couldn't join.

A

It is still sort of an interesting topic because he was pointing out that we have this usage ping database queries and they basically try to count all the stuff and that's increasingly hard when you, when you have like big tables and and for some of those we already use approximate counting and for others. We still count exactly, and there is a proposal for like deficient or transactional counters, but it's still in the back book, so I'm I'm, not sure when we are going to see that happen. I.

A

Created another issue that is for a batch counting and it's very fresh I, don't know I would love to get comments on that. That makes sense, but given that the usage ping is sort of a background job, we can take a lot of time for counting that doesn't hurt. It just hurts when you, when you take ages for one query and if we would instead just batch count, you know count single batches of 10,000 records, or so it would take a long time, but it would actually finish because we wouldn't run into a statement. Timeout.

A

Would that make sense should be also very simple: to implement I guess.

A

Alright, should we move on to mutt requests Mara.

B

Yes, so after reviewing some mutt repairs, I have some questions so I have a question regarding the migrations that are located on their paws, migrate, folder and the migrations are advocated under them. Migrate, folder like if we have a backrub migration. Where should we go? Do we have like Ana strict guideline regarding those or.

A

My impression is that we usually put them to post migrate because they, you know they run in the background, so they would run anyways when when the new code is deployed- and that's probably what you want right when you kick them off in a regular migration that happens before the code is exchanged, so that would that would actually sort of try to run a job that may only become present on the co-pays when, when the deployment finished so I tend to think that's, maybe the reason why we put most of them in the post migrate that make sense, should.

B

We force that were reviewing like hey. You have a background migration that is on the migrate folder, but it should be on the post, migrate, folder.

A

Have you seen instances of that? Yes,.

B

Yes, that is what is making me ask that this, because I'm not sure how to react. Uh-Huh.

A

Could you could you pay some example for that just be interested in what it looks.

B

Like let me try to find it quickly, Thanks.

A

And I can't really think of a reason why we would want to kick of a background migration before we update the code or if there was any difference with him other than that.

C

Well, I do have a an example of that, but I did post it in the post migration because it just felt like obvious. For me: they had a data. Migration should go in there. They post migrate kind of more of a like I, don't know logically, but like we deployed a fix and then in the next deploy we would do a data migration, so that kind of you can think that the code is already there so in the next deploy.

C

You can kind of put it into into the migrate folder, because it's going to just ink you certain jobs. So it's going to be fast and the code is already there so like. There is no reason to have it there like a longer post migration or something like that. So that might be.

C

A

The other end, in that case it wouldn't it would also be totally possible to do that in a post, migrate right.

C

Just like right now, as I see it, it doesn't make any difference where you put it other than just logically separating the fact that this is a kind of a data migration more than a schema migration that needs to happen before before they decode actually changes.

A

What we sometimes do is when, when we do those batch background migrations, where we update records in a batch fashion, is we basically have a post migrate where you know a regular migration, where we sort of build up the batches so that we iterate all the batches that we have and then we schedule jobs for each of those probably doesn't take a long time, but that may also be something that is more heavy than like a normal DDL statement or whatever. Maybe that's another reason to put it in a these.

A

Things are not in the post. My grade, I.

B

Just put an example of background migration that is, on the DB migrate, folder I couldn't find a match of West, but I found, like several background, migrations.

A

You don't mind: I put it on the on the dacha small.

B

Like particularly on this case, how can we tell that it is correct to put like this migration on the on this folder.

C

It sounds to me that there is no like strolling goodness but, like I, think, ideally any data migration should go to me to a post migrate. It sounds like so anything that is schema related that that obviously does need to happen before code can actually start working with that data should go into the my create folder and there pretty much everything else is going to post migrate. That sounds like logical things to me, but it might might not even be that strict as well so or not.

A

There are really the only only reason where that might not work is when, when you actually introduced our background migration or if any change to the back row, migration with a same Amara.

D

A

Then you need to do that in the post migrate, but for database I. Don't think it makes much of difference.

A

Based on, like these questions, I asked the delivery team Yorick answered on that one, whether or not would make a difference, putting migrations and in a regular one or in the post, deploy for the deployment or, if there's any preference, where we should put them. If there's a choice- and you pointed out that for the deploy doesn't really make a difference, because you would, you would have to deploy, takes as long as it takes for executing those it's just the all.

A

The risk is different and we have a sort of a rough guideline for the the post, my grade, where he shouldn't take longer than an hour for for the full deploy.

A

It's probably also something that is hard to tell when you, when you only ship one migration but sort of gives an idea. So everything that would take longer than an hour should not go into any of the regular migration I.

A

Don't think we have to document up somewhere, though.

B

I'm going to add that to the guidelines, so it is easier.

D

A

Thank you sure.

C

Also, another thing is that you can actually skip the post migrations right and then you cannot skip the migration. So this is why I like it makes sense that some data migration should go into into post migrate, because then you can do a deployment and do a code update with just the normal migrations, and you can just run the data migration whenever your server feels like handing the the load.

A

That'll make sense.

A

And the other question is basically how to choose background migration or recognize.

B

An example of this.

B

So I put the match request on the chat, and basically this is a migration that is changing some data on the births table, but the birth table only has like 100 records and they wanted to initially to do a background migration and for 100 records. I, don't think that is necessary because they update where it would be wrote very, very fast, so I'm not sure when we should use background migrations like for updating records. Like do we have like a certain guideline like if the table has less than 5,000 records.

B

Maybe we could use just like an update query and that would be much easier to implement. Instead of writing background migration, because then we will need to work to write the migration and the specs for it.

C

I'm, not an expert in any way, but I'd like my thinking, is that we may have some kind of data in our database and obviously get lappam is probably the biggest instance you can have buddy. You can never know like what other customers are doing so I. Think, like a good rule of thumb, is like to actually just enforce to have all of the data migrations. Perhaps it doesn't take longer I think even it's incude it'll like delay. What 2 minutes later or something like that? Well yeah depends on how big the queue is.

C

Obviously, but it does sound that it doesn't hurt to just enforce it. It will be in them I guess it will be easier to also have some kind of automated rulings and checks on that. So I'm not sure if it hurts or not. Obviously it will be faster if you put it on migration or maintenance.

C

E

Not only faster, not only faster but less fragile, because when you use a radius sidekick of to--for to change something in both keys, you need to keep in mind that something can happen on radius if, in case of just handed records, I would use your simple one batch regular one transaction migration. But of course we need to be sure that other installations have not more than a thousand records.

E

Also I would I would shift from considering amounts of roast or, if, if like, there is a good target for every batch processing time not amount of rows but time every user dealing with website if a latency reach one. Second, it's already very very noticeable.

E

So if we update something, it's better not to exceed one second, better, of course last, but one second is already like if we lock someone with our update up to one second, it's already noticeable so I would estimate when we deal with account migrations, I would estimate using github.com data how much? How long is one batch processing and then I would decide, but back to background migrations. This is the most like most fragile approach, because we split two batches. We put the two register, then we rely on consumers and so on.

E

There are many moving parts here compared to simple, simple migration with ended, right, yeah.

A

I remember an example where we were actually shipping another round of the same migration, but not in a background migration, but in a regular one to make sure that all those batches actually executed. So there's no data left to update. So that was a bit odd too, because you would do the same thing over again to make sure that you know you caught everything so yeah that that's a bit more fragile and yet other person.

D

A

There is, you can always also consider doing batch updates in the regular migration right so for for the the really huge background migrations that we have we we would. We would aim for scheduling a batch every minute or so just to keep the keep. The traffic on the database load spread out the load over time and also make sure that we don't lock end up blocking too many records for for too long period of time.

A

But on the other hand, when you, when you talk about like a few thousand rows, I, don't see a big reason. Why why we wouldn't be able to also do batch updates and in a post migrate, for example,.

E

If it's four thousand rows and the table is very narrow, like no toast involved, it can be just one batch because it will be probably less than one second right.

E

Do you think it's like it's worth, defining some rules here, like thresholds, to make the support decisions like some numbers well, when we prefer these for that yeah.

A

Totally useful, of course, um yeah.

B

I think that will be very useful for reviewers, because that will give us some guidelines to actually measure the tables.

A

Maybe interesting to come up with something that is actually applicable in the end, but I really like the idea of saying that that takes one below one second for a batch on Telecom then should be good. That is something that you can measure easily.

A

You don't have to deal with the number of Records, because that may even be different. Another installs.

B

Now, how can we measure that it is below one second and applaud calm, because I'm not sure if cherubs allow us to do that, because I think Cherokee Street only and it is mostly for selects I- think we will need access to a database production replicas right. Yes,.

A

E

E

New tool, you can check it right now. This is a new slack channel called database lab. Please welcome and see. There are some issues, so this is like production database quo and you can do anything. You want the same way as short hops. You can write out there there and you will see comments, so you can run updates, deletes anything and also create create some indexes and so on, and then this command reset. So you can revert any changes you did, but there are two things to like to remember.

E

First, is it's right now it's single node. So if two people are working as matthias lee, they can probably mix some sound actions and it can affect its proof of concept right now. Only stage and another thing, these databases on the first. There are some issues with some tables. Like you probably will see. If you try, you will see some I also I'm working on it. Something goes wrong, but I will fix it. I hope exactly like exactly this thing is needed to check any type of query on database.

E

You will see detailed plan. Of course, timing may differ from production because this instance is smaller, but be wrong to this direction. Is okay, I mean if you, if you see two seconds on production, it will be one. Second, it's fine, and actually what is most important, also like amount of memory fact. So.

E

Time is very what I think depends on at night at night in Europe. It can be one second, that day in Europe it can several seconds on production because there is more busy buffers like it's very hard. I raise this question: let's define numbers but actually like the most stable thing is number of buffers. We touch right so, but it's hard to define rules based on those numbers, it's better to define them on seconds, but seconds are volatile. So there are like a lot of moving things here, but with defining some.

E

So please check that by slack channel and try.

A

That's really awesome seeing that happening.

E

You can go there and check check execution plan or.

E

It can be much slower, especially with creation of indexes caches on production.

E

My personal data is some cash.

E

Excretion on production will be much faster, but at.

D

E

Most important because you understand okay, for to get these several rows, we need to touch this number of buffers which buffers eight kilobytes. So we understand how much data.

A

Right they, what we did previously was using those one of instances for getting getting a full copy from production and using that and then throw it away, and that suffer from the same thing. Where were you, you have like a different instance size, different resources, but what was really helpful is when you, when you have things like you know, you you compare that to the previous version of the career.

A

You want to optimize the Curie or even just the same Curie and then creating an index for that when you see a speed-up compared to the previous version, that is still so relative to the the old version that is still meaningful may not be the same absolute runtime on production, but it still suggests that there's a good improvement to the Curie in any case, right and.

E

Number of buffers will be the same.

E

This is this thing is based.

C

How, off today,.

E

C

I think even a monthly or even bimonthly should be just fine, I guess like like size-wise, because it would be different, then production that have a very very close to what you have on production. So if we saw what I'm, what I'm trying here to say is that, even if it's not feasible to do it weekly doing it monthly, where every couple months should be just fine, I guess for the engineering team to to estimate stuff yeah.

E

A

The other hand, when you you know when you deploy regularly, um you might even not see all of those tables that have been introduced while, while or since that des natural was taken. So it may complicate things when you have to. You know figure that out what is what is missing and, on the other hand, it may be even possible to just attach that instance to to the production archive and actually keep that updated right.

E

Consuming from archive and when you start working with it, we put it on pause, get a certified snapshot, promote it work with it. In many regards and let's look at about it, maybe it will be feasible, I'm, not sure if you work with it many many hours.

A

Alternatively, we could also like take snapshots from production every day.

E

C

Cool then I have some stuff to try it out. Oh yeah,.

A

This is really awesome benefit from rap because chops. Today's is like read only whenever you want to you want to create something that doesn't work. Well, that's really cool to have when you you create a snapshot. You just promote that you have a read, write copy. You can do whatever you want and then just reset that sort of it's very nice.

C

Cool then I guess my next question is kind of solved.

D

C

Agenda which is like how can we get some of the access where we can try out stuff and then basically come up with these results post it in the Mar, merge request where every issue and get your review, so that should be much easier for you guys to review then go ahead and do it all yourself for them for us to wait for the results. So that's cool.

E

C

Yeah I'll move on and work with some of the migration that I have been working on review now. I guess I can do most of it myself. So yeah. That's awesome. Thanks.

A

The other part of your question was how much what you can do to speed up the reviews. I just wanted to say what you put in there like providing the actual sequels providing the execution plans, and so that really helps to speed up things.

A

Because then you only have to look at a sequel and not not. You know, check out and figure out what the active record produces and what I found really helpful is when you provide an example for the sequel before that change best with an execution plan and then the same thing after that change. So you can really spot the difference in both the the query and the plans and, in the best case, we'd be using a good example for parameter values.

A

For example, when you, when you do have a create that might might be different for the C rep was compared to a smaller project. So picking good parameter values is also great, and that really speeds up things, because you just have to look at that and then that's.

C

Really good, isn't, though, okay to like paste the entire sequel in the comment, or is it just better to link it to somewhere external I? Don't know, I feel like sometimes that's something like the recent Amar's that I'm working on the sequel includes a couple Union statements, and then it doesn't update it's kind of big and then, if you want to leave also some commands and explanations, it just goes like it feels like it never finishes or something like that. So I'm not sure.

C

What's the best approach, there is it okay, for you guys is it is it? Is it rather put it into a poor sequel instance, and just for some, some page being save it there and then I'm just have the link here and have the commands like the explanation and then do this equal or maybe we do have like this snippets or something like that store this equaling, the snippet.

A

It's maybe personal preference, also I, don't know. I I think you can also collapse larger parts of the comment so that you would be able to put your full crew in there and pattern collapses. It doesn't make as much noise I find out quite useful.

A

Snippets not so much I think it's just distracts from you. You have to go somewhere else, but it's just my personal preference.

A

Well, it's what also helps us when you, when curries, are formatted by some kind of standard tool.

A

That's also good, and even.

D

A

When you compare previous and then the current version of the Korea, you can even like make a texture old if the formatting is the same. That's also quite a.

C

Link here and then I guess: I can post it in the document as well, where you can see like I'm, trying to post the sequel and then the query plan and then the new sequel and then the query plan, and just it just drags on and on and on and on, and it's like for me personally, it's kind of well. When does this stop and wonder, I've read what this guy wants. Actually, sir.

A

Yeah I see what you're saying.

D

A

We can put those plans on the in that tool. You know those just explain these.

C

A

Then you just you know you just link that and it's maybe even a better way to look at the at a plan and textural and that reduces the noise a little bit because you don't have like big comments.

A

Mara, do you have a different preference for putting crease and snippets or comments.

B

Not really I mean I think that comment was kind of long, but also very descriptive, so I'm fine, with whatever the developer, chooses I.

D

C

That's a another request for the plan team to make sure you can collapse the. How do you call this? The when you include some the code sections within a comment or something like that? That probably could be very, very helpful because sometimes, like these kind of code, snippets can get long. So then you can shrink them and read the comments in that expand.

C

That would be probably helpful. That sounds useful.

A

This is something that we could make more visible like what to do to make those reduce easier, maybe put that into a documentation page and say and link to that yeah.

C

I think that would be very helpful. Okay, they in the documentation that you've just updated recently for the DB review, and not only for the developer site when they do ask for review to just ask them to put in the generated sequel rather than for you guys to go into the code and generate the sequel yourself. You don't revive the sequel and they explain form so that I guess that'll be helpful for the developers as well to know what you do.

D

A

A

All right, I think we skipped the Robocop topic.

A

Should we dive into that.

B

Yeah so I'm just curious about what would be a limit for the text and for the string columns like if we are going to enforce a particular limit or just ensure that the column has a limit. I.

A

Think where this came from was just to thinking that we should disallow adding columns without a limit, because that opens up for abuse, or it also means we're not really sure how much data we probably going to put in there. So just having a ruby cup that would detect that I'm sort of complain when there is no limit, would already sort of help to spark a conversation about putting a specific limit in there.

A

I thought about the about putting a standard limit in there a bit first I thought it was hard, because what are you going to put in there? But on the other hand, if we made it if we made a default limit of like 200 characters, that would also help. But it is aggressive because you don't you can put like really a lot of data in there and when you actually need that, when your use case asks for that, you can. You can always say, like we allow.

A

More than that, would you rather put a high limit as a default.

B

Yeah, maybe for this iteration we should just enforced the limit like no matter the value, but just enforce, like the columns, have some limit and take it from there.

C

Yeah, it's hard to know sometimes like if you can have a title where you can have a URL or you can have a I. Don't know they shy D so that that they did. The limit really differs a lot for different cases, but just making sure that the developer or whoever is designing. Anything is aware that the needs to be a limit and then they'll have titty they'll need to think about it like and impose something, a nanny trade on that and increase it or decrease it.

A

Right, we could also add another check that says if you or that, actually limits the amount of or that caps to limit that you can put in like. If you want to put more than 10 megabytes into your text column, that's probably going to be an issue anyways.

F

One issue that I ran against was that if you use a text column and you set a limit, rales simply ignores the limit value that we set. It does not actually set a limit on the difference right.

A

I think that's sort of a poster aside thing too, because when you want to limit, you will have to use varchar' instead of text. Thank You Clark, with the limit on text for the data type.

E

E

A

On the back end for postage, there is no no difference between barter and then text, so both both work, the same.

A

Is there another difference.

A

That would mean we would always use watch or if we want to trouble the Medicus.

F

Also, should we be concerned about how long it takes to change the limit on a large table.

A

Right, that's the concern. It comes with watch all right, so when you want to increase that, you will have to rewrite the whole tableau. Although I remember, there was some trick to that too.

A

F

Remember, reading about something that you can use domains or you can set the limit using a function so that when you want to change it, you don't need to retype the whole table.

F

F

Sorry that was a wrong taste I.

F

Piss is what I'm into place.

C

But like I'm not sure how difficult it will be to try to Robocop drooling but, like even for text fills their must outside. There must be a limit within the Ruby code or something like that. I don't know, maybe maybe a red book open just actually people usually ignore warnings.

C

So unless it's an error them not sure it's worth adding it yeah, because we had recently a security shown on a text field bound with the descriptions and undercurrents. So we we did. We need those, even though it's not possible.

A

Do you remember the trick you can use to increase or change the limit later for water I? Remember there was something but I can't can't find it.

E

Well, there is also hockey option to.

A

Like I think your connections, a bit unstable, was that about changing the metadata for the.

E

F

Sorry I didn't get what's not a good way.

A

Victor you wanna rephrase. That word should I.

E

E

This is like, let's ask: how to do it quickly. This is quick way, but this is not for production, unfortunately, and actually actually I'm curious. Why why it's not supported to increase limit without full table scan.

E

So I mean we can directly go to system catwalk, and this is just can be done with simple thing, but this is hack.

A

Maybe this is something we would want to figure out before we put that Robocop in place, so that we have a sort of a suggestion: what what to do, how to enforce the limit and that we have something that works again.

A

Just a little surprised when, with water issues as well,.

B

Should we at least try to enforce a limit on string columns only or who we are also need to wait.

A

Yeah, that's a good idea. We would use string anyway, so yeah, adding that limit totally make sense.

B

A

Cool I see that correctly last topic on the agenda about the increasing the number of database maintained errs Morrow. Yes,.

B

So I put together some numbers about their database review workload for the last milestone and I did it because last week, which was officially the last week to shape things for the milestone, it was a bit crazy to get all review stones because basically my email exploded and my mentions exploded and I was like okay. This is not. This is not going to scale very well, so I just wanted to give an insight of the work that we are doing.

B

Basically, 1991 mergers were created with database labels of those 56 were merged and 31 were approved by a database. Maintainer I think those numbers are not super sacked because I built a script with the API and the API has the limitations.

B

But what scares me ovide is the average of merge request that each that each reviewer has to review, which is 30 and 30 if a lot and the average time it took to approve their database from the Marshall quest from Adam is perspective, which is 8 days, and this is the time that I am measuring this time. From the moment, someone adds the database label until the moment.

B

Other Thomas maintainer approves, which is not very exact either, but it gives you a good idea, so I think or I cannot think on any other way to improve these numbers to decrease these numbers other than adding more regulars.

B

So what do you think I.

A

Agree it's but it's great seeing those numbers, it's very useful, also to see that per release, and hopefully those those numbers are going to go down. I was wondering about the about the time to review a bit because we started to do was using those scoped labels. For you know the review itself in addition to the database level, so I would expect. There is quite a few MRIs out there where you, you put the database level from the start, but you would only request the review at some point later.

A

I, don't know if it was possible with the code, you have to consider the scope labels instead to make it maybe a bit more accurate.

B

You mean to check the time with you between the label, pending review and database, approved.

D

B

I think that is possible to compare what I wanted to do initially is to check the exact time in which the developer asked for a database review, but it is not possible with the API because it has.

B

The limitations like I will need basically to parse all the nodes and try to check for a comment that says more or less like hey can't you do it at our interview, but not all develop all the developers do that most of the developer just assigned someone, and they don't ask, so it is not possible to be that it stuck, but maybe with the labels we could be more exact, maybe yeah.

C

I I didn't know about the label, though I didn't know that you can put on a label to ask for the database review so.

B

The danger both does it for you automatically. You don't need to do well, the danger body supposed to do it for you. So.

C

What what should the label be.

B

Database review pending.

C

I, don't I don't have any of those.

B

The change was introduced last week, so, if you're a metric older than a week, it might not have it. Okay,.

C

F

Had one situation where an M R, which pin touch any database files, the danger bot automatically added the label to that.

B

Well, the database files are not only the ones under their DB folder I. Think if you touch either they leave database folder or some or even some models. They are going to ask for the database. I can search like which are the database exactly. There are pilots.

F

C

Okay, I'm going to sauna I think we are over at the time, but today I learned a lot, especially about the database lab. So thanks a lot a lot, a lot one more time and everyone else for the work have a nice day. Yeah I'll give you some feedback, I. Guess these days, I'm going to try it out the updates, hopefully not the job statements, but you know all right. You too.

F

Right and this, if you have a few minutes, I have one question.

A

Sure should we do that here or somewhere else.

D

A

To do both depends on the question so.

F

On one of my Amar's, you commented that about a locking issue, I'll post the link. So my question was: how do we check whether there would be locking issues? Oh no,.

A

Well, let me take a quick look, so I can remember.

A

Okay, so I don't know much about the metrics to mind and that the features we have there, but it sounded like we would be. When we run that importer for Prometheus metrics, we would basically open a transaction and then run maybe a few hundred updates on those metrics I.

F

Sorry, it's about 20 to 22 rows.

A

All right, okay,.

A

Okay, so let me go my my concern was that when, when you open up a transaction and then you run updates, you basically build up locks as you go until you commit the transaction right. When that takes a long time, it may become a problem for other transactions.

A

That was where my concern come from and that, if that is only like 20 records, every update, it's probably not going to be a problem. Okay,.

D

F

In case it is a few hundred rolls, is there any way that we can verify if it would become a problem.

A

It depends a bit on on how this is all implemented and.

A

And also along those updates, take what I've seen the the importer runs single record updates. Is that right, so it without it would run, would update one record per update. Query yeah.

D

A

Yeah and if those are not heavier than and they get the lock immediately, then that is going to be quick as well like for even for 600 records. That's not going to take forever, but if it was a case where the update itself was maybe a calculated one that would take a while to run them may be different.

A

But it's it's hard to tell. It was just a general concern to look out for I think when you, when you run single record updates that is sort of the most in efficient way of doing, updates and also takes maybe the most time to do them and then, when you wrap them into a transaction, just I would just think about what the what the impact of that is. But it really sounds like it. There is not a problem there.

D

F

Okay, so the longer the update takes and a larger number of updates, then we'll have to start worrying what.

A

Yeah I would think so. On the other hand, it also depends on who's who's using those records right, if that is sort of a highly contended record, where everybody's chasing after a rapid.

E

A

And it's it's a much bigger problem than when you have something that is maybe usually just read. Only because you know read locks they, don't they don't conflict with the write log. You can just continue to read it's no problem, so it really really depends on how that is being used.

F

Okay, so it's a very subjective thing, then: okay kind.

A

Of film well, what's the case here, I mean: how do we use those those records.

F

Those like what they use whenever someone goes to the matrix dashboard, so each of those gets pulled into a chart on the dashboard, so I guess every time someone opens that page.

A

But it sounds like a like a selector. You know read on on that record right, yeah.

F

It's mostly Aradia, the only right is through the importer.

A

Okay I mean, in that case you're totally safe, because you know those reads: I would continue to work, even even when you take ages to update the record right.

D

A

They don't don't conflict with that lock, so you would get it immediately. So it sounds like there's absolutely no problem here. Okay,.

F

Thanks for the info, I'll.

A

Try to capture that on the comment after the call: okay.

D

Thank you sure.

A

All right we're way over time should we wrap it up.

F

Yes, I do brother, no.

A

Problem at all cool what was great, having all those topics on the agenda today, great having that. Thank you for that and yeah have a nice have a nice day and see you next time talk to.

D