GitLab GitLab Group Kickoffs - Enablement:Database, 13 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab 14.3 Kickoff - Enablement:Database

Description

Kickoff for the Database Group for the GitLab 14.3 release

Planning issue: https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/188

A

Hello, everyone, my name is fabian simmer, I'm a group product manager in enablement and I'll walk you through the database group's plan for gitlab's next release. 14.3.

A

There are broadly three areas that I'm going to address: infra death issues, um the work we're doing to address uh the risk of a primary flow overall.

A

Excuse me um the work we're doing to address the primary key overflow risks for tables with an integer primary key and then some known issues for our ongoing sharding or more specifically, decomposition efforts of the database.

A

So the first thing is that in 14.3 we are going to take a look at issues that are specifically relevant to the availability and performance of gitlab.com to do with the database.

A

So, for example, this is being worked on already, but if it doesn't finish in 14.2 improving the efficiency of non-strictly, monotonic ids for pipelines, we're also going to investigate why certain sql queries running in humanities are dating back to the primary database.

A

So this may result on more load on the primary or postgres cluster and then some other issues with active record. So these are highly technical database related issues that we need to fix in order to ensure that gitlab.com runs smoothly and they take priority over some of the more long-term longer-term work.

A

The second thing here is the ongoing work to address the primary key overflow risks for a table with an integer. So essentially, these tables have grown so much, but at some point they run out of primary keys because they are simple, integers, so we're working to actually migrate that those two begins so that they don't overflow- and this has been ongoing for a few releases already in 14.2- we're swapping out the relevant columns and then in 14.3 the upcoming release.

A

We are going to actually drop the temporary columns and and triggers, and that will then actually complete the migration, we're doing all of this sequential work in order to avoid any downtime on gitlab.com or any large gitlab installations.

A

That may otherwise be impacted, um as you can see, there's a lot of activity going on in 14.2 and a lot of these items here in 14 point will be in 14.3, and this is mainly about finalizing this and dropping columns and we're all looking forward to having this this resolved.

A

Lastly, there's ongoing work to decompose gitlab's database. The context here is that, especially on gitlab.com, a single database cluster that we're running is sort of bottlenecked by a single primary database server, and it's becoming more challenging to scale this vertically.

A

So we are trying to scale out horizontally and what's happening here is in the first iteration we're going to take all of the ci, so continuous integration, rather than tables and move them into a second database cluster, and so as part of that work, which is done by the sharding group, there's a lot of issues that can be fanned out to other groups, and these are some of those sort of blocking issues that we're going to work on in order to ensure that gitlab can continue to work with several databases.

A

So I'm not going to go into the details for all of those, but to give you one example: here um we have migration helpers, for example, and these migration helpers are not aware that there is more than one database, and so we need to now actually fix them and ensure that it becomes a way of having at least two potentially more in the future, and this is important so we're able to continue with that work and the database group is working very closely with the sharding group on delivering that kind of work.

A

So these are the three big areas. Infra-Def issues are more immediately operationally focused sharding blocker issues are a little bit further out, because this is going to take a few months until it actually gets rolled out and then the primary key overflow risks are important.

A

To avoid that we get a hard block on databases and can grow them, but this is now in the finishing steps of actually swapping and dropping columns, so excited to get that done and that's it. There is a lot going on. I'm very grateful for working with such a television team and I'll talk to you next time. Bye.