15 Jul 2021
Kickoff for the 14.2 release for the Sharding team.
Planning issue: https://gitlab.com/gitlab-org/sharding-group/group-tasks/-/issues/1
Planning issue: https://gitlab.com/gitlab-org/sharding-group/group-tasks/-/issues/1
- 1 participant
- 7 minutes
22 Jun 2020
This video culminates in the conclusion of the Sharding Working group. The timelines below are approximate:
0:00 Intro, read agenda
4:05 Gerir Feedback on sharding approach
5:30 Clarification of Sharding Terminology
7:10 Discussed time and ability to explore other scalability options
8:02 Questioned Audit Events priority
8:36 Requested Lexicon
9:12 Doing Partitioning not doing Sharding - other low hanging fruits
10:15 Garbage in garbage out
13:40 Start talking about alternate solutions
16:10 Talking about Big instances of GitLab (EU, Federal, etc)
18:04 Propose closing Sharding Group and focus on Scalability
20:00 Decision to move forward with Scaling Working Group
21:43 Charter for Scaling Working Group
0:00 Intro, read agenda
4:05 Gerir Feedback on sharding approach
5:30 Clarification of Sharding Terminology
7:10 Discussed time and ability to explore other scalability options
8:02 Questioned Audit Events priority
8:36 Requested Lexicon
9:12 Doing Partitioning not doing Sharding - other low hanging fruits
10:15 Garbage in garbage out
13:40 Start talking about alternate solutions
16:10 Talking about Big instances of GitLab (EU, Federal, etc)
18:04 Propose closing Sharding Group and focus on Scalability
20:00 Decision to move forward with Scaling Working Group
21:43 Charter for Scaling Working Group
- 7 participants
- 24 minutes
29 Apr 2020
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
- 1 participant
- 11 minutes
29 Apr 2020
Quick demo of using postgres declarative partitioning in combination with foreign data wrappers to push partitions to a foreign server.
We're tackling one table of the GitLab schema here, creating 8 partitions and distributing them on 2 shards. Shard in this example is a local database, which can also live on a remote cluster.
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/53
We're tackling one table of the GitLab schema here, creating 8 partitions and distributing them on 2 shards. Shard in this example is a local database, which can also live on a remote cluster.
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/53
- 1 participant
- 8 minutes
29 Apr 2020
Follow-up to https://www.youtube.com/watch?v=MiZFtM84x44 to checkout how we can do schema migrations.
Very basic examples: Adding and dropping a column.
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
Very basic examples: Adding and dropping a column.
Part 1: https://www.youtube.com/watch?v=MiZFtM84x44
Part 2: https://www.youtube.com/watch?v=nt4Khi9Gr3o&feature=youtu.be
Part 3: https://youtu.be/ztQtNmSYmEo
- 1 participant
- 6 minutes
16 Apr 2020
We discuss different scaling approaches for GitLab, namely
* database sharding
* application sharding
* service extraction.
Internal doc with notes: https://docs.google.com/document/d/1u9e736mu4SlVtE2gf4HWsqsoV__Dk68qtsrvyfXvky8/edit#
Rough timeline
0:00 Overview
3:30 Service Extraction - Concerns
5:30 Databases on the same cluster
5:58 Praefect Example
7:13 Security concerns when databases share the same schema
8:04 Container registry example
9:10 Data management concerns with single database
11:20 Replaceability of systems - modularity
16:08 What is application sharding Pros/Cons
21:13 Considerations of sharding by namespaces
28:06 What's the strategy
29:56 Management of self hosted databases
31:20 Replication
31:53 Praefect Replication concerns
35:00 Application sharding probably inevitable
36:44 Service extraction details
41:50 Distribution concerns with Service Extraction
43:30 Summary - Service Extraction next steps
48:00 Database guidance and struggles with current structure
* database sharding
* application sharding
* service extraction.
Internal doc with notes: https://docs.google.com/document/d/1u9e736mu4SlVtE2gf4HWsqsoV__Dk68qtsrvyfXvky8/edit#
Rough timeline
0:00 Overview
3:30 Service Extraction - Concerns
5:30 Databases on the same cluster
5:58 Praefect Example
7:13 Security concerns when databases share the same schema
8:04 Container registry example
9:10 Data management concerns with single database
11:20 Replaceability of systems - modularity
16:08 What is application sharding Pros/Cons
21:13 Considerations of sharding by namespaces
28:06 What's the strategy
29:56 Management of self hosted databases
31:20 Replication
31:53 Praefect Replication concerns
35:00 Application sharding probably inevitable
36:44 Service extraction details
41:50 Distribution concerns with Service Extraction
43:30 Summary - Service Extraction next steps
48:00 Database guidance and struggles with current structure
- 8 participants
- 1:00 hours
