GitLab Delivery Team, 25 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Implement a namespace storage limit discussion

Description

Alessio and Nicolas talking about https://gitlab.com/gitlab-org/gitlab/-/issues/209119

A

B

Okay, so we are discussing here the in implementing namespace.

C

B

Lamenting a namespace storage limit, the no namespace is capable of hitting okay as an MVC. This is issue two zero, nine one one nine just here mostly for answering question is Nikolas, is doing, is taking the effort of doing these issues but as part of the delivery team, I have worked on some of this stuff almost one year ago, so I may have some knowledge about it. Thank.

C

You, okay, so just an overview about the issue, so this is MVC. We don't limit it. We just want to implement the limitation itself and we fought for as a starting point. We will limit this on github.com to one petabyte, so it will not affect any namespace. That's not what we want in the beginning, like for the embassy, but afterwards we want to have like elements. So people who, like users, who go beyond the element, need to pay for the storage.

C

That's that's the whole idea and we want to implement this like you can set a limit for all namespaces and right now we have a limit for a repository or project like ten gigabit limit, but you can have, of course, like multiple projects inside the namespace, so we will be limited to ten gigabytes and then you can horizontally add more gigabyte and yeah. So we want to limit this on the top-level namespace.

C

Therefore, on this like root namespace watch what we kept, we call this right now and you then see like a quota like you see it on the CI minutes right now. This is this is the whole idea and yeah. So I left you, your team. Already, a lot of work and I want to talk. We have huge others. There's a database study about a database case, the above-named Space Storage statistics. We made some proposals how you can calculate the project, sizes and yeah.

C

So the solution that you you decided on was every time I proposed a project updates. You schedule something like a chop and then like these cops job gets scheduled in one and a half hours later, I. Think, and then you update the root namespace statistics based on the new sizes, that's correct, and if something within between this one of one and a half hours changes like again, you don't change, you don't create a new job. You just updated the current job, that's what I got so far. This is.

B

C

Aggregations cattle, okay, so my the question like adding this limit is relatively easy and also having like this checking. This limit on one page is all also relatively easy. The thing is at some point like on CI minutes. We want to maybe send out an email that you already have like 80% of your storage used as an example, and the question is: how can we do that and I would like to have some thoughts from you on that, and also this is something that I would like to hear from you and otherwise.

C

I looked through the code like there are a lot of things going on, but it was mostly clear to me. I have just one question, but Mara did this code. This is about I would just yell screen quickly. This is just prune aggregations kettle worker. Let me.

B

Okay, I guess: I resized, my windows working here.

A

You see what should okay so so.

C

This worker is pruning the painting rules for a gracious kettle, every time at 1:00 a.m. and did then yeah I see it a select. If we have not done everything like until this point, we clear all of them and we calculate all statistics yeah.

B

Yeah, it was something like that.

B

B

Actually, don't remember the details: why would this was kind of I? If I remember, we had fear of something getting stuck yeah, cusers say once a day we just clear everything and if we have pending works, we do update all the calculation, but the I think the most of the reasoning.

B

If it's not if something, if you can find something in the write-up, so in the blog post, the original issues has all the discussion about it, because there was me Myra and Yannick involved into this, and we were constantly discussing way to do this in so, if it's not in the blog post, it's definitely in the marecus orientation. Thanks.

C

I don't know, I will check that again. I only looked and the kodan yeah.

B

C

B

I can try to find you why we did this. It's.

C

Just like, but yeah I guess I'm guessing that, for example, if someone performs an update at 1 a.m. and like maybe it gets, it gets to blown up to still you I'm not sure, maybe want to have one point of time. Every day they clear everything, maybe I I.

B

Don't I, don't remember, I, think it was kind of a safety measure if for some reason we because there's this point- that let me try to think of what we did, because we act outside of transactions for scheduling stuff, because we one of the reason why we had to build all those things there too main problems.

B

The one is that the query itself is low because you act on root, namespace level, because you want to aggregate on the paying entity, but statistics are stars, are recorded by project and direct namespace, so not at the root level. So we had to rebuild the tree in order to accumulate the values on every child item. So this takes time as well. The the table is, is big and we're just doing some in calculation say: okay, we can just we.

B

We must save the information somewhere, and this is why we decide if so it gets scheduled, but it's a side of a transaction. So there were, this I think we're the slightly concerned that we may end up writing the schedule on the database, but losing the Redis. For some reason it's may be human because the maybe you restart something or you lose information so say we want to make sure that once a day we have a check point, and we say because this information in database is supposed to reflect something that is scheduled on Redis.

B

We make sure that once they we check this database and we reschedule everything it's something among this line. Okay,.

C

So everything that is left not not all the namespaces again.

C

Everything that is left okay, that sounds good, but you.

B

Had another question: yeah.

C

The question was about the scalability of this and with current that year, that we want to limit it as at some point or show like a banner.

B

C

Yeah I mean Emma's Custis of the scalability team at some point, maybe have also ideas, but I would like to get get your brain at this as well. So the initial proposal had something like daily chop that we run along namespaces and it is basically the same issue that we they have had before like. We would need to calculate everything which is a lot of work and costs a lot and it's not viable and github.com. Okay,.

B

So the the idea that we had about this was that we should introduce the concept of a soft limit because they're one of the reason why I we. This was a shifting focus shift in our team. So we drop, but we completed the calculation and then we dropped the next part of this epics. But there was a lot of UX involved into this and there's no, you external in our team, so we were kind of struggling with that part.

B

So the idea was that that, if you define hard limit in the soft limit- and the thing that you are talking so far is- is the art limits, because you say we want to set a limit that you cannot go over yeah.

C

B

The idea is that if you have a soft limit before which can be a percentage can be whatever you want, then you can update the UI or even send notification email when you eat that value. That still gives time to the user to react. So let's say you have an 80% soft limit so eat.

B

What what you know is that when you change something that effect cirrage in I think up to one hour and a half or two hours, I, don't remember, think one or enough you have the final data that is updated, so the biggest delay you can have is one hour enough, which means that if you, you can do something like this. If you want to enforce a limit, you basically check the size of what is coming in and this this really depends on the upload you're doing right because forgive you know upfront for some artifacts uploading.

B

You don't know the size up front, so you can not. You can not tell if they're at the end of the upload, you will be above the limits, but this is. This is really depends on what you're doing so. The point is that when something comes in you can do it your best to understand if this will bring you outside of the limit and then I do you have to be flexible in this, because you say maybe this will go a bit outside and the next one will not get it.

B

So it cannot be really hard and strict, but it's something like when you are over. You cannot upload anything. So if you have this in mind, you can do the all the uux and the user communication on soft limit thresholds, which means that the same way you daily or in one hour whatever you update, you can also check if that number is below the soft limit threshold.

B

This is something that you can even do when you render the page, because at that point, that the value is already in database and if you have the the limit at the root namespace, which we don't have at the moment. So if you have that limit and the percentage you can just check it and show a big banner in every page of the repo say you know you are above the limit. Send or the first time you go, you send an email things like that.

B

So this was our idea and that's how you can do it do.

C

You think it's an issue if you would check on every page. It's requests like I I. Imagine it is an issue that we check on every page, recursive to namespace that you are in the root. Namespace has a storage capacity that is near the limit. I think this would be too costly right. Do.

B

You think I mean.

C

My I'm, not sure I, just heard like on PS I, think mentioned that this query is quite expensive. One.

B

That gets the root. Namespace is getting the real name. Space notice.

C

Statistics I think.

B

Is not expensive because it's.

A

B

A

B

So here the point is that humilation.

C

Is the expensive part yeah.

B

Exactly but it's hard for you so that that's why we started with all those work, because we need four month way for doing it. So this is not a problem. I was more thinking if getting the root namespace but I. Don't expect this to be an expensive extra query, because if you think about every page that we show you still have the bread crumb or things like that, the root namespace I guess it's already loaded somewhere, because we we display it in a lot of things. So that's wrong!

B

I, don't think, and when you have it you're, just it's directquery, because you have the ID of the root namespace and.

C

B

Is not another thing that we thought about was that when you do the accumulation after it you kicks, you kick another, a synchronous job that updates the status in terms of thresholds. So you didn't want to do the calculation I mean it's not. The calculation is just checking a percentage over a value, but whatever every time, but only when something change. So if something changes, do you update the accumulation value and then you also update the routing space, which may be easier.

B

If you want to do something like show me all the namespace there are over a certain percentage and I, so I should yeah expect this to pay more pay, less change, storage whatever. So this.

C

Is also my ID like to have something that, like a Doolin flex, that on a namespace that says, storage, size exceeded or something like that that we set after updating a project, so the the question I had I was just like means. It could be inconsistent data that is already out this.

B

Is inconsistent we know and there's that's the reason why the root namespace storage, the aggregation table should have I'm quite sure that is there. There is a there's a column to tell you last time that information was checking.

C

Statistics table human.

A

Last we said yeah.

C

No that's last.

A

We just there's.

C

A namespace statistics table yet wow. This is the shared rawness. Sorry just Chevron Oh seconds.

B

Which wish I was looking at the schema, RB and I noticed that we merged them. The microbiome.

A

Still area the.

B

Skin so did happen, okay, so my structure right.

B

Namespace root statistics; okay, okay, this one so updated at OKC.

C

Update; okay, yeah that.

B

Information gives you, when the this.

B

You can it it was we had. We had this discussion when we, because there was one page in the group, set things that show you the usage, there's a back there, so I think it's part they shouldn't marry link to you. We have, if you have more than four gigabytes of storage. The graph QL query breaks.

A

B

Because they integer in graph QL are I thought of picking.

C

B

Yeah, so it basically just break it and that's not so, but there we showed the breakdown. So you know by root namespace how much storage you're using and then you go down each project and you can see the breakdown of LFS artifacts, and so that's something that we have so in the discussion. But I don't know if he made in the final version of the image we said we should write down this overall information is updated up to this date, so it may not be entirely accurate but yeah.

B

This is I, don't think we we, it ended up in the final design, but we have it on the database. So you can.

C

Takeaways, for me, are that it's relatively cheap to calculate the roots, namespace storage and check it on every page request. If you really want to this, is like relatively cheap and that we could use the current drop to execute something like an email which it's that you are above us, so yeah I would say that you should just know.

B

Is the current job to schedule a new one.

C

Like if you, if the top gets executed, it also schedules a job so check this boundary. Yeah.

B

You can see the boundaries and do whatever saving the story, saving the result somewhere sending emails.

C

It's like it's I mean I know that we don't track everything. Yet we track repository LFS wiki size, build artifacts and storage Ison package to size, yeah.

B

C

Also, the question about like uploads image, uploads on comments against these things, there's also the register.

B

Is not tracking exactly.

C

The other packet.

B

It should be part of the packages, but it is not track it, and there is also another issue that I'm not sure if it's closed or not. But this affects your what you're working on so the artifacts has it.

B

Build artifacts asides can be negative because there's a neat there's a box somewhere I mean we I, don't remember the name of issue, so it's well known. So sometimes they agree the pearl project information, so not the other variation. The base value is for checking everything timeout so because it timeouts. It may not write the statistics for that project and we ended up with situation where you had negative information, for instance. So this breaks do the overall effort of checking the values.

B

Obviously, but it's something you should take a look into it because you may have weird numbers like negatives, okay,.

C

I like it is, maybe because people uses overwrite or I've already get commit history and no.

B

No, this is just for the artifacts upload. So what happens is something like this that maybe you have let's say one artifact of 10 gigabytes. Ok, just made up numbers and if you delete it you are, you are removing 10 gigabytes from from the value. Ok, so what happens is that there are the there could be a situation in which, when you add the new artifacts, the storage upgrading query times out so the the project statistics, so not an it, reduces the project, run, don't get the plus value.

B

So you'd say you got another artifacts, one gigabyte so say, plus 1 gigabyte, this time outs. So your storage is still 0 because you had plus 10 minus 10 plus 1. Ok, so it this time also you still have 0. Then you delete the artifacts because the Aspire or whatever, and then you get a minus 1. If the plus 1 timeout it then you get. Your final result is that you have minus 1 gigabyte of storage in use, which is not true. Ok,.

C

And do we know why this timeout happens? Yes,.

B

It's because of all those queries are in in transactions, and so the project statistic table is very busy and what happens is that they just timeouts waiting for the lock on the table.

C

I assume I think I.

B

Can I will find you nation I, can find the issue and sent so that you can do you can cross link this to the thing be aware of that we have this problem, don't make sense, and this is one packages. Okay, we don't know the registry storage. So that's another problem at.

C

Some point we will add this I guess yeah.

B

I mean it's: it's the package group, but I, don't know because we started developing our own Fork of the registry. So I think that there's a there is an effort for having this information and that's not a story. And then we have the uploads, the user uploads to discussion issues that one is completely entrapped. Okay, if I remember, but this we should start tracking it, because at least when we save it, we know the sides we could.

C

C

Be like make makes makes the project statistics table even more PC at all the problems that.

B

We have with the upload is this that there's no way to delete one once you upload. Something is there and the other problem is that when you drag something into the editor right, it will start the upload immediately. So you dropped something it will be uploaded and you and the markdown information will be there for you to link it yeah. But if you, if you decide not to use that information, so you remove the markdown issues. The upload.

C

Can sit there, we never track that on the back end that this exists, yeah so I see, so we need actually like front end codes that we track if this got deleted. We also need consent for the request in.

B

One of the issue that Mari linked there is this: it's an epic aroma issue and remember where we have break down kind of storage and it's all about allowing user to free some spaces, so each line tracks. If we have an issue discussing, how can we allow a user to delete something and there's? No, there is one also for them for the app loads.

C

Okay, cool so there's their ideas, yeah their ideas.

B

C

What it's all about right, yeah,.

C

Thanks a lot for the input like I I, think my questions are answered.

B

If you want to write up what your, what is your understanding about the Scott, so maybe I can just chime in after and say yeah. This is what I meant or no I explain myself terribly so.

A

B

What you we can I can help in the meantime, I will find you all those related problems and issues that may be, but.

C

At least I've in mind, there'll be no yeah.

B

B

C

Put me in the rabbit hole, finance.

A