17 Feb 2023
A quick guide on how to use the database lab joebot web interface to test the performance of queries against a production clone.
- 1 participant
- 6 minutes
14 Sep 2022
- 1 participant
- 4 minutes
16 Aug 2022
- 2 participants
- 10 minutes
3 May 2022
Pat gives an overview of the decomposition work.
Notes (internal) https://docs.google.com/document/d/1c_NBT-5GSCYhs7CZP9PGQACsP6_wUWCCpDVqVn782dI/edit#
Notes (internal) https://docs.google.com/document/d/1c_NBT-5GSCYhs7CZP9PGQACsP6_wUWCCpDVqVn782dI/edit#
- 3 participants
- 31 minutes
20 Apr 2022
I wanted to take a minute and ask you a question: how often do you deploy untested database code in production? At gitlab, we're using some cool new tools to make database migrations safer and better.
Migration testing project: https://gitlab.com/gitlab-org/database-team/gitlab-com-database-testing
Merge requests that use the migration testing: https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&label_name[]=database-testing-automation
Database team job posting: https://boards.greenhouse.io/gitlab/jobs/5093969002
Migration testing project: https://gitlab.com/gitlab-org/database-team/gitlab-com-database-testing
Merge requests that use the migration testing: https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&label_name[]=database-testing-automation
Database team job posting: https://boards.greenhouse.io/gitlab/jobs/5093969002
- 1 participant
- 2 minutes
9 Apr 2022
This is the regular application database team meeting, where we discuss team matters and review ongoing work.
- 7 participants
- 34 minutes
17 Sep 2021
Kickoff for the Database Group for the GitLab 14.4 release
Planning issue: https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/193
Planning issue: https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/193
- 2 participants
- 16 minutes
13 Aug 2021
Kickoff for the Database Group for the GitLab 14.3 release
Planning issue: https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/188
Planning issue: https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/188
- 1 participant
- 6 minutes
27 Jan 2021
Agenda doc (internal): https://docs.google.com/document/d/1wgfmVL30F8SdMg-9yY6Y8djPSxWNvKmhR5XmsvYX1EI/edit#
We give a demo for accessing database thin clones using psql, talk about Database Lab and our plans to implement fully automated database (migration) testing
We give a demo for accessing database thin clones using psql, talk about Database Lab and our plans to implement fully automated database (migration) testing
- 5 participants
- 34 minutes
4 Jan 2021
Talking through a CI-based workflow that allows us to test database migrations on a thin-cloned production dataset (@abrandl).
For details and to leave feedback, please see https://gitlab.com/gitlab-org/database-team/gitlab-com-migrations/-/merge_requests/1.
Apologies for the screen sharing quality, looks like my CPU has been maxed out in the end while recording this - also hadn't expected my camera stream show up next to the screenshare. Next time... ;)
Here are a few links:
1. The CI job kicking off the automation: https://gitlab.com/gitlab-org/gitlab/-/jobs/943411439
1. The migration job: https://ops.gitlab.net/gitlab-com/database-team/migration-testing/-/jobs/2768158
1. The comment posted back on the original MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50730#note_477762322
For details and to leave feedback, please see https://gitlab.com/gitlab-org/database-team/gitlab-com-migrations/-/merge_requests/1.
Apologies for the screen sharing quality, looks like my CPU has been maxed out in the end while recording this - also hadn't expected my camera stream show up next to the screenshare. Next time... ;)
Here are a few links:
1. The CI job kicking off the automation: https://gitlab.com/gitlab-org/gitlab/-/jobs/943411439
1. The migration job: https://ops.gitlab.net/gitlab-com/database-team/migration-testing/-/jobs/2768158
1. The comment posted back on the original MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50730#note_477762322
- 1 participant
- 14 minutes
16 Oct 2020
Kickoff for the GitLab 13.6 release
https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/94#note_430887473
https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/94#note_430887473
- 1 participant
- 6 minutes
29 Apr 2020
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
- 1 participant
- 11 minutes
29 Apr 2020
Quick demo of using postgres declarative partitioning in combination with foreign data wrappers to push partitions to a foreign server.
We're tackling one table of the GitLab schema here, creating 8 partitions and distributing them on 2 shards. Shard in this example is a local database, which can also live on a remote cluster.
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/53
We're tackling one table of the GitLab schema here, creating 8 partitions and distributing them on 2 shards. Shard in this example is a local database, which can also live on a remote cluster.
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/53
- 1 participant
- 8 minutes
29 Apr 2020
Follow-up to https://www.youtube.com/watch?v=MiZFtM84x44 to checkout how we can do schema migrations.
Very basic examples: Adding and dropping a column.
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
Very basic examples: Adding and dropping a column.
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
- 1 participant
- 6 minutes
16 Apr 2020
We discuss different scaling approaches for GitLab, namely
* database sharding
* application sharding
* service extraction.
Internal doc with notes: https://docs.google.com/document/d/1u9e736mu4SlVtE2gf4HWsqsoV__Dk68qtsrvyfXvky8/edit#
Rough timeline
0:00 Overview
3:30 Service Extraction - Concerns
5:30 Databases on the same cluster
5:58 Praefect Example
7:13 Security concerns when databases share the same schema
8:04 Container registry example
9:10 Data management concerns with single database
11:20 Replaceability of systems - modularity
16:08 What is application sharding Pros/Cons
21:13 Considerations of sharding by namespaces
28:06 What's the strategy
29:56 Management of self hosted databases
31:20 Replication
31:53 Praefect Replication concerns
35:00 Application sharding probably inevitable
36:44 Service extraction details
41:50 Distribution concerns with Service Extraction
43:30 Summary - Service Extraction next steps
48:00 Database guidance and struggles with current structure
* database sharding
* application sharding
* service extraction.
Internal doc with notes: https://docs.google.com/document/d/1u9e736mu4SlVtE2gf4HWsqsoV__Dk68qtsrvyfXvky8/edit#
Rough timeline
0:00 Overview
3:30 Service Extraction - Concerns
5:30 Databases on the same cluster
5:58 Praefect Example
7:13 Security concerns when databases share the same schema
8:04 Container registry example
9:10 Data management concerns with single database
11:20 Replaceability of systems - modularity
16:08 What is application sharding Pros/Cons
21:13 Considerations of sharding by namespaces
28:06 What's the strategy
29:56 Management of self hosted databases
31:20 Replication
31:53 Praefect Replication concerns
35:00 Application sharding probably inevitable
36:44 Service extraction details
41:50 Distribution concerns with Service Extraction
43:30 Summary - Service Extraction next steps
48:00 Database guidance and struggles with current structure
- 8 participants
- 1:00 hours
