GitLab Sharding Working Group, 16 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sharding Brainstorming Session - 2020-04-16

Description

We discuss different scaling approaches for GitLab, namely
* database sharding
* application sharding
* service extraction.

Internal doc with notes: https://docs.google.com/document/d/1u9e736mu4SlVtE2gf4HWsqsoV__Dk68qtsrvyfXvky8/edit#

Rough timeline
0:00 Overview
3:30 Service Extraction - Concerns
5:30 Databases on the same cluster
5:58 Praefect Example
7:13 Security concerns when databases share the same schema
8:04 Container registry example
9:10 Data management concerns with single database
11:20 Replaceability of systems - modularity
16:08 What is application sharding Pros/Cons
21:13 Considerations of sharding by namespaces
28:06 What's the strategy
29:56 Management of self hosted databases
31:20 Replication
31:53 Praefect Replication concerns
35:00 Application sharding probably inevitable
36:44 Service extraction details
41:50 Distribution concerns with Service Extraction
43:30 Summary - Service Extraction next steps
48:00 Database guidance and struggles with current structure

A

Well, alright, thanks everybody for joining. So does anybody read the donker? Should we have a quick background.

A

No answer the yeah I'll give a quick background. So I was asked to write up a database strategy MRR for the handbook to talk about what our database strategy is, whether or not we allow multiple databases what that means and I'll take up to mr here shortly. So we can share the screen and it came from an issue where they were discussing bringing in different database technology and then that they actually focus back on Postgres sequel. So that was no longer concerned being in a no sequel or some other technology.

A

But then the concern was: should this live in its own database or not, and there was support for having a separate database. But then there was concerns that it goes against our single datastore strategy that we have written down- and this is this- is the Amara and what the disagreement came with whether or not we actually have just a single database or whether we support multiple databases on the same cluster, and we do have some naming classroom clashing we'll get to that here in a second on the dock.

A

So, ultimately, what the question was- and it was raised by Sidwell as if we are pursuing database sharding, then shouldn't we have everything on the same database, so all tables on the same database, not separate databases and if we're not going to pursue database sharding. If we're going to move over to application sharding, and we still need to get a common ground on the definition of what application charting needs, then we could consider having multiple databases so at a super high level. Is that everybody understand what we're talking about as far as the differences here.

A

Okay, I haven't heard any dissenting opinions so tried to put in quick definitions of each and then ondrea's added the service extraction and actually I added an example. That's being discussed right now with the CI CD daemon, so that was very timely. All these conversations, so the goal here is to brainstorm list all the pros and cons of each approach and then come together with what it doesn't have to happen today. But it needs to happen fairly quickly, come together with an approach.

A

What we're gonna go forward with right now, the approach is, we are charting at the database level and everything should be on the same database, since we are already down that path which we need to switch to a different strategy than let's all decide together and move forward with it. So any questions- or you want to jump right into conversations about the implications and the pros and cons of each approach.

B

All right, what do you say? We're not doing both I mean I think you might have just been talking about database versus application, yeah, okay, because service extraction is a new thing. So the question we may actually do service extraction in addition to database and application shard exactly right.

A

It's certainly something we should discuss right because there could be the concern that was raised as if we are taking two approaches like we do we'll go at this example, we do service extraction. What does that look like and does it cause a performance? Nikkor I'm? Sorry, the support matrix nightmare, where we have major database that we shard it across you.

B

Know dozens.

A

If not hundreds of shards and then we're starting to extract services out, and does that mean as we're extracting the services out than their own they're, all living on their own databases as well, so we have a main database, that's charted, and then we have a bunch of other databases that are now extracted and running on their own. And how does that impact customers right com? We manage kind of our own world, but what? What does that mean to our self-managed customers.

B

Right I mean I can argue that, like, for example, giddily we did this with Italy already, so it.

A

B

Happened, it's just you know we need to figure out like what's our strategy for the rest of.

C

Is with Redis we have on kidnap, become we've got three different Redis clusters going to be for at some stage, but then on self-managed. They will it all just goes into the same Redis cluster, and so it's fairly easy to manage, because you know in some places they're all happen to be on the same server and then other places. We can split it apart by service right so that that's a service extraction that the sidekick service we've got, the you know the main assistance tears and their and the cache service, and then for small instances.

C

They just altogether.

D

We would have to same benefit from having multiple databases, logical ones. We could have them. We could run them on the same cluster. If customers want to do that, that's the problem. Easiest real economy may decide to run some data business on a different cluster that make sense. Yeah.

C

Yeah and, as you know, obviously we'll be able to join between them still, but they might still be running together. Another example, which is pretty much the same as Stan's example but prefect as far as I understand, has got its own sequel database as well right, but.

A

Greg- and that was an example that was brought up to that. Yes, it's got its own database now, but it's not. It has not been released and it should be brought back in the main database so that we can then shard it.

D

But there is no like logical overlap between the data that traffic has and the one that we have in the rails database right. So my question is: why would we bring that into the same logical database.

A

Sharding for scaling is the example is the question that raised.

C

Its it what's, the better I think you asking: what's the benefit of of doing that right, I mean it's, it's there's.

A

Advantages and disadvantages right less so like under s mentioned it's it's an anti-pattern right. It's not something! You would normally do to bring all of these separate databases that have no business being together into its own, just so that we can apply charting to it, for our scaling strategy isn't.

E

It also like quite severe security issue that, like you, have multiple applications managing schema because, like kind of them, assuming that every of this application is like a good cities and properly prefix tables and like doesn't monk, will be other tables and it seems kind of risky. This approach to me, yeah.

D

This is the anti pattern element I think every application should really be an owner of its database schema and if we, as soon as we have more than one application, sharing the same schema, it's sort of breaks, everything where it breaks those abstractions and it's it's really hard to manage the schema when you have one application or Nena schemer and then multiple other applications being dependent on the schema that that really it's a terrible place to be in I.

B

Think this came out of a container registry right discussion. A lot of this discussion. It's like well container registry uses all the project. Isn't why can't we just stuff? You know our existing.

A

Database with all the.

B

Tables the container is she's, gonna need right and it doesn't seem like you know it does seem. This might be some benefits, because good interest rate does need to use all the projects, but there's so much data, and there are so many connections that it would. The reason why I didn't think it was a good idea to combine them is because it we are loaded on.

B

The existing database is already very high and registries been a very nice isolated service that hasn't been affected by you know our database going down as much as other things, so in our database internally. You know it's already preloaded with number of connection, so we're limited by number of connections. We can have on the database and having PG bouncer in front of that helps. But then, once you start adding more applications, it gets harder to scale that right, eventually we're already running out of connections to the database.

E

Like in our database or the right, we kind of stop.

E

We actually like now enforce that there is for ranking for like every direction to ensure data consistency, but no happens if we start to tire container registry tarts with the projects and first removes project and cascade date removes all data that is managed by the container Veggie see we kind of have like two brains controlling the removal of the data and studying the same schema like database and forces consistency, but who actually, we must the baking data, it's kind of become probably very grey and tricky area like to actually implement that properly without side effects, because, like now, let's say we start cooking two projects: private ie container registry tax, hooks to private ie bitter price, manages projects table you, remove entry from the products table and they're gone now.

E

You're gonna have cascade in each one of the container registry tasks. So normally it would be the container registry removing data, but is it not longer true? You would actually have to implement a back of the logic into race as part of the product division in order to properly diggities data. Now that you would actually today, you would call the API to actually do that for you, so it's kind of I think it becomes like white spaghetti because, like this consistency becomes it's unfortunately database level, but application level.

E

If you have multiple applications actually magnetic lifecycle, the data become kind of I, think where we are.

C

Camille it's this there's, you know we're getting to the size where this complexity is becoming like a thing and like the number of people that can kind of fully understand.

C

All of these connections is diminishing, ly small and like one of the things that you start having to use to tackle, those complexities is replace ability of systems, and if you have like a module like the container service, which is quite a replaceable module, but then underneath it you're just kind of using the same database, then there's always the the temptation for people, like you, say, to join a cross or run queries against other people's data and and then, as soon as you start doing, that you lose the replace ability.

C

So we can have a pull. A component like replace ability to me is like the reason why you do things might be modular, II, modularity, whatever in modules, because you can, you can pull them out and say you know actually now. This is no longer fit for purpose and and replace them at a later stage. And if you're doing this with, like your entire database, is kind of munged. Together, you can never replace those components, and so you know, if you have an isolated component, it should really have it isolated database, isolated in some way.

C

You know maybe running on the same server, but not tangled up.

E

Like logical database, it could be like the same stuff be given these past, but this is been like really a matter of the configuration for the application yeah. If you see that like this in recent cannot like handle the load, you move, data way to separately starts probably something under search. Yeah.

C

I would like.

D

C

They find API sorry so yeah Jason, yeah.

D

Exactly that and on the opposite side, I would argue that if there's any reason to put the data models in the same database for for two applications, why don't we have just one application implementing that right? You could you could even go that far and say like it. If container registry and rails need to share a data model for some reason and we need to put it in the same database, perhaps the container edges, you should actually live in the rails, application and not be outside of them.

D

I, don't think that's a good approach, but you know I think we should we should separate them. There is no reason to put them.

C

Like like coming from a finance background, you have these huge organizations and everyone has a different view of the data. What you often find is that you will have the same entity with the same ID, but two databases with different data in it.

C

So the risk departments you know, might have risk data on that ID and you know so you you don't have one entity that has all the data, because it's just it's just too much and everyone's trying to change everything, but you use identifiers that are common and then the data is different for different views.

E

Yes, I think going back back to your point like from what could happen, it will basically start raining, then I think a major aspects of the registry in to rise because of the limitation. This is this will be the outcome 3d of, if you would go, if that our process like we serve the same schema, because this would be the only way probably to solve this kind of problem, and it was, we start okay pretty quickly. This is this is my perspective on that.

D

Yeah that kind of feels like if we put everything into the same database, then we basically bet on the on the idea that we can really scale this database nicely and that you know the database sharding solution. Where you you have you don't touch the application, you just you find a way to scale up the database and that works for everything and I. Think that's a really huge bet, pretty risky. Well.

C

Just going back to the Redis example, one of the things that's really advantageous for us on capcom is that because we split it, arts and those workloads are all slightly different. We have different tuning on those different Renison senses right and that can help you get more performance, whereas if you have one big general database, you always have to tune for all the different workloads, whereas if we were breaking it down, you know we could make say, there's one that was difícil, something like that.

C

We could tune it for that kind of big blob kind of workload. For example, I, don't know making things up here, but you know what I mean.

A

Okay, so it's pretty clear, there's some major concerns about having everything in the same database right so I guess what would the alternative be? So maybe we come together on what it means to shard at the application level, and if we need a different term, that's fine. We've just been loosely using applications charting some of the examples. We'd use there, the Shopify pods example.

A

Actually not that's pretty much. The only example we've been using, so everybody have a common understanding of what application sharding means.

A

Yeah, so what would be the some of the pros or cons there? I.

C

Mean the big thing for me with that is, like all the good examples of where I see that working it's very much isolated. Where we've got you know, we don't have like a date. We can't take a database /, namespace or anything like that, because the users are shared, a lot of the data is shared and a lot of the state is shared right, and so it just doesn't feel and maybe I don't understand the methodology well enough, but it doesn't feel like a natural fit like Photoshop afire.

C

It's a really good fit because a shop is very isolated and the models aren't shared between multiple shops. Presumably so it feels like it. You know, and every shop has got a different domain and it's very easy to kind of track down that route and and there's lots of other examples. You know lots of sauce companies that are they don't like that.

B

Right I mean the the flipside is that if we wanted to support organizations right like let's say we had Tier one organization, we really wanted isolate and they have their own instance. This would be beneficial right because, yes, there's no there's no sharing, but they don't want to share, for example, and things that happen on their their system should just only stay within their system. So, but obviously, for, like the you know, public project, something like that, it doesn't really work, but there is a key. There is a use case for it.

B

If we wanted to go this route right, like you know, our major customers right now will actually like that right. They have their own sandbox. So.

C

I mean this is this is possibly a going way out of the scope of this discussion Stan, but one option with that right is if we had- and this is wavy hand. But if we had like a really good kubernetes operator, forget blurb, for example, we could spin up like whole kit, lab instances for these customers using the same code. You know we don't have to worry about the petitioning.

C

You know the application level petitioning and they would literally have their own sort of Postgres inside that pod and that we would run for them, and it sounds a little bit like what was the hosted gitlab thing that we used to have before, but like really well yeah.

B

Yeah get home yeah, yeah yeah, but.

C

Really well run and kind of with good scaling, and you know not just what get host was but um yeah that's a different discussion for a different day right. But what.

B

E

There is you could.

C

Do that with multiple post questions.

E

Actually add that to that, because it's not very clear what those applications are. Do you mean because you could consider starting on two dimensions very like where you separate data I mean across the service? Let's say we have the packages service, it runs into separate database database. It has this own API, but also there is also a point of starting via by cost of customer. Like me, move some enterprise customer to completely separate database. It has its own separate feed money, and this is a completely different beast.

E

So I think that should be evaluated completely separately, because starting by service gives you this ability that you can, in the future kind of move the data into completely separate, might a service, but it's like right below, which is like service extraction, because you're on the data on this actor. So you can move that.

E

We're starting by customer, it's probably more like I'm, assuming we create it create a team, is probably get your own database, your own space, where you work in and probably charting by customer, it's more like creating it refrigeration. It's much closer.

D

What we've been discussing also with partitioning, is doing this group charting so charting by by the namespace that you're in and in this case, I think there are a lot of features, love that only operate in this namespace right. If you think about CI and might request issues most of those features, they they're inside a namespace and perhaps there's there's a way for us to wrap that into your service and and then we can.

D

You can think about charting that on the database level by customer and there and there are other services or other features that are sort of cross-cutting with that, and perhaps there is an opportunity for those to extract them into their separate zones. We would break that up by looking at at features and figuring out, which ones are naturally inside a namespace and which ones are basically cross-cutting and we would treat them differently. Would that make sense.

D

C

One thing: I, don't I, haven't thought about this very much but like if you have a trace coming in it goes like API v4 projects, some ID, you know something and then you still need to look. You know all of our endpoints are not really designed with this namespace, obviously other than weird, but the API really isn't really designed with this namespace approaching. So do you have like a lookup table that then kind of redirects you to the namespace like from the ID, or you know.

D

C

Our API is not designed with with this API approaching, so you have to do lookups and lot. I would imagine so so.

E

Aren't you like, like you mentioned, tries trace, like probably a lot of our endpoints are scoped by the project ID and like. If you look at our tables a lot of the tables they even deprecated project, ID :. Even it's not a like a direct relation to the project ID but like. If we talk about the CI, build and tries it's actually looking up specifically by the given be tidy or like by the given project ID, but the lookup table, we actually have roots-type.

E

We just have a lookup table to like to figure out how to transition from the path. I mean full part to the given scope, which is namespace or project. So in that model like the roost, I would probably be stored somewhere us in a central piece, but they're like if you do partition my private ID. A lot of of the tables are already prepared to the partition by the private ID, basically because they have the profile een in their column, set.

C

Right, so are you saying that I would like it would be fairly easy to do that lookup, then, to to map to like a application partition on the project. Id Camille.

E

Yes, because, like we have routes file that translates a full path VM into into like the entity which could be in Iraq.

B

E

B

E

Complain accrued history for whatever else I'm like we actually use that we started using them once we introduced.

E

Subgroups, because this was like the only efficient way to like to do these transition and I believe that we also use I made me wrong, but I think that we also use routes to figure out exactly subgroups for a given group right.

E

So like what I'm saying like, we really have a really like central routes table which is essential to like to perform routine. So.

C

I could be fairly easy to always make sure like almost like a middle way when you come in to just maps that outs and then sets the context.

F

E

Probably even more interesting because routes tables is not I, don't think that they will stay away. It's pretty big, but it doesn't change very often as well. So you could be slightly inefficient and replicated on all replicas to make what stable very close to the data that you are looking at and it still would be fairly accurate. I, don't know if possible, suppose that this is like the possibility.

E

Really that right you keep routes table like replicated multiple times, assuming that this is like pretty stable and like you, do not create a central contention where you have to go through roads and I can decide or you.

C

Extract the service extraction on the routes on the route service and just have a single one, but.

F

Yeah there is there's, there is multiple ways to actually.

D

While we're saying that application sharding doesn't work for public projects.

B

All the idea is that that those every project, that's in its own database would be a really hard to get this cross database joint right. The idea is that you would just basically have your own sandbox and it would be. You would just query against that. Sandbox.

B

C

Do things like explore yeah.

B

Or you know yeah or any project for that matter. Right now, yeah.

F

A

Think one of the things we discussed there was you'd almost have to have some kind of, like maybe at the database or some other storage mechanism to share those cross database or cross project. Whatever cross join you'd almost have to have like a coordinator, database or coordinator application that aggregates all those together. If we want to continue to support those like public searches across all the different pods.

C

And that presumably, would also have the users in a drive yeah.

A

So it's almost, it would be similar to the database sharding at least the way. I understand that how scientists described it, where you'd have the some kind of data replication. Somehow.

A

So I have to leave in about 15 minutes, so the big.

A

What I need to come up with is database strategy and all of this feeds into it, and what the consigning to answer the concerns about what our next steps are and if we don't have an answer today, that's fine I can buy us some more time. There's a lot of information here, but the big concern still are the support matrix and how does this go again if we're gonna go away from a single database which it seems like we are and everybody support of that?

A

How does this this goes against our single app and our single data store message? Do we need to change that? How does this impact our self-managed folks do? We need to diverge so that what we're doing for dot-com doesn't necessarily ship with self-managed?

A

What's our strategy for scalability and availability going forward, we already know that our database is there's a major concern and we've heard somewhere between a year and two years, it's gonna hit a wall. It's the example that was used, so you figure out if we're not gonna pursue sharding, because we don't want to have everything on one database or if we are gonna pursue sharding, but we're gonna. Allow separate databases. What are what is the impact and how do we? How do we sell that?

A

How do we convince everyone that this is the right strategy and I think having everyone here in heaven detailing a good strategy going forward? People will support it, but that that's what I need to explain in the coming days. I think.

C

You really want to make it clear from the opposite that it's never gonna be like full ticket lab or anything like that. Like I, don't think I, don't I, don't think that's a strategy. We ever want to go down where we have sort of two different versions. We want to make that like clear from the outset that that's not the approach that we'd want to take yep.

D

So if we only allow using multiple logical databases, they could still live on the same host, an omnibus, but we would still ship a post Chris that has like a couple of databases installed logical ones in the same cluster, so in terms of management for self hosted installations that wouldn't make any difference compared to. If we stuck all the data in the same database right, we would still have one cluster. We would still have the same all the data on the same cluster. So that's not really a difference. Is it.

C

Can I just ask a question if people using cloud store cloud database services, they self management cloud, do you get a whole Postgres server or do you get a database and so what they need to have five or six databases in a cloud managed install at five or six times the price? Maybe.

B

Get a whole server. It's.

C

Just you get the whole server: okay, yeah.

B

You can basically create database within that server, so all right, whether it's RDS or a cloud sequel, okay,.

C

Cool then, then, that's like a non-issue just wanted to check.

E

So it means that, like supporting databases would be possible, but then there is like it's probably troublesome to like to set up. Let's say replication of that is somehow semi automated way, because if you would follow that model, you would assume that we gonna add more of these individual databases and there has to be some automation behind allowing you to be automatically supported.

E

D

If you talk about replication in Postgres, we talked about physical replication typically, and that means replicating a whole cluster. So whatever database you add, you don't really carry it out for replication because it just replicates the whole cluster. So in that sense there is nothing need to do with respect to physical replication.

E

B

I think the only Asian considered was a prefect was going. You know, pre-fit was discussing this and they actually said we don't want to replicate all those things in this prefect database and I said well. It doesn't actually cause that much harm you just you know you just generate more load and data on the secondary, because you're replicating stuff took the second. It doesn't really need, but yeah you can just throw it. I create a new database and you'll get it for free, the tío and Postgres. Oh and.

D

The beauty from that is that were if we ever realized that we don't want to be replicating everything we could go ahead and just extract. The database put it on a different cluster. Who could look correct? That makes more sense for us. That's correct,.

C

With the prefect s so stand with the prefect example, though, I thought that the set the giro secondary still a write database, because the prefect which you know so it wasn't actually that it was replicating the primary. It was that it needed to be writing from there, but also.

B

Geo has two databases right: it's replicating everything in the primary, so everything you put in there it's read-only, but then it has a second database that it writes to so we're.

A

C

B

Mono we already using completely different databases right, okay,.

F

Yeah, then that makes sense yeah.

E

Right we seen the application. Do we actually like allocation solving I mean in any model, or we don't hello, because then does the tax point like it's beneficial to do application? Never starting or is it like? You kind of figure out an architecture ie how you should create like a new tables to make to make them possibly sortable? We are post class methods in the future if it needs to. If we start adding in databases- and we start adding in tables.

E

So it's a little hard but, like my question, is very like we have different approaches to application, starting is by customer, it's maybe by service my service allow us to to service extraction extraction later. But if we like keep things as is today, which is like we have Keita price, which is like a big application that adds into its own these database and everyone else into the database. It's very likely that we do not have to start this database anyway or something in the future, it's kind of inevitable.

E

So then, my question is like how we approach this charting in that case, and in future cases, if we allowed to create a logical databases to make this data structure to be actually start able, we are both quest methods. If we need that.

A

So let me see if I understand, you're saying that, even if we were able to do application, char, Denise, I'm, sorry application charting, you still think database sharding would come in at some point in time. Yeah.

F

It's kind of like, what's.

C

Happened so naturally with writters is you know we split it open to these different service extractions and now we're looking at taking the cache one and switching over to Redis cluster, because we still need more. You know: we've got as much capacity as we can over the strategy and now we're going to the next strategy.

C

But the question or the sort of comment I wanted to make was that if you have that service level extraction shiny database sharding it's easier because the scope is more limited because you have this shard and it's not like across everything. Yep.

A

Okay, I added another section in here about the logical databases seems like the best title for it, so which is effectively the path that we're on right now.

B

A

Having prefect in its own database, get laid its own database and container registry wanting to be in its own database, it seems like that's the way folks are leaning right now if they take a quick pull. Is that the preferred path at the moment for folks on this call.

C

So so it would surely it would be a logical database in self-manage, but it would be something that we could pull out on get lab comm if we had scaling issues, so is it different from service extraction or is it just the same thing and that's the implementation detail is that it's a logical database I would.

D

Say it's kind of the basis for service extractions with you, they're allowed to extract the service, and it cannot talk to a logical database that it owns.

A

E

Probably like for services, we would assume at each service has its own database. Basically, it has its own logical database. This is this will probably like the design.

E

So, like it's answer, there's article on permissions: it's on schema that foot controls. Then the question is like: have you? What are the guidelines for this schema? If we want to charge these data, let's say container district on a scale fine for another one year, but maybe in two years is the the database gonna be too large. We gonna have to chart this database how misc image down today to allow us this charting.

E

If we control that decision now, yeah.

A

So then we could potentially, for each service, have a separate charting strategy and that's one of the concerns right. Do we start having the support matrix? It's just unmanageable for our for our dot-com instance and for our self-managed folks, great.

C

Can I ask, though, would it ever be? Is there and neat what stands for the best person to answer this? Do customers have this need to shard, or would it just be that the only setup that we support for customers from support point of view is logical databases and if they want to go beyond that, it's it's really they're kind of going off-piste and drinketh themselves, but we don't supports database sharding for customers only for get live.com.

B

Well, I, don't know if any customer that needs to shard right now right, no customers I'm near what you lot by commas. Doing I think the concern is that eventually, someone or some of them may do that, but yeah there's, no one even close right now, right.

C

Babe, but really from a sort of like a support, matrix concern, point of view. It's probably you know. Maybe we shouldn't tie that in to this and we can just say from a support matrix point of view for now at least the only option that customers have is non Charlotte, so we're not going to kind of complicate the landscape with like different. You know database sharding options as well as everything else does that make sense great.

E

Yes, I do actually it gives you if you did for our customer, which they can move logical databases into different instances and it kind of unlocks a lot of computing power and data based on its own. So I I, don't know if we have ever Konami do not have customer anaconda scale of github. That would not suffice, given the scale of our single Postgres instance. Today,.

C

Like it gives them that option, but we sort of like from an omnibus point of view, we we don't give them that option. The omnibus option is logical, single PG, logical and if they want to go beyond that, they can kind of do that, and you know we've got customers that we know will probably do that. But maybe it's not supported.

A

Okay, so it sounds like the service extraction is the preferred path at the moment and we still need to look into charting the main rails application, but we don't want to add other databases, other contents of other databases into that, simply because we want to have a single data store. So how do we and I gotta run? But how do we answer that single data store, or do we just need to change the wording on that? Well,.

B

I, don't know data store could mean like it could be the same database instance. But just you know they didn't listen different. You know logical databases right.

A

Yeah- and that was a question I asked and the answer I got from Sid was assume. A single database is what we mean by single data store. Okay,.

D

But I think that is the term is really overloaded like. We really have to be clear what we mean by database and yeah.

A

And I tried to clarify to you and we kept going back to assume a single database.

D

We're talking already about the same thing.

C

Single a single database type in Postgres, no.

A

Single single database with all tables in that one database so.

E

So probably it doesn't really matter as long as Andria's. What is saying like we can easy.

E

He replicate our data for the whole instance. So done like this little geek I. Think it's like it's Superman Terry gives you a better separation, but I cannot really have any major downsides. Well,.

A

In the other concern that was brought up was the distribution piece right. If we start allowing databases to be created, then distribution has to be concerned about setting up the database, setting up the connection, managing all those things on self-managed and in production. So I think we do.

C

Any good, I think the second one. We have a fairly good answer, which is you know, it's they're, all just logical databases as far as omnibus and self-managed and support is concerned, and so I don't think that there's a huge amount of complexity on that. Second, one I. Just this is a slightly tricky question. But where is this stuff, this database strategy?

C

When was it written because it's sounding a little bit like dogma, and we all know that get lab doesn't do dogma. So you know obviously strategies work at different times in an organization and.

A

It's linked in the doc above and I have okay, I have no problem, pushing back saying, you know we need to change. Our definition of data store from single database with all tables to our data store needs to be postgis right, where we can have multiple logical databases.

A

So there are some related Docs and I will link them all to the bottom. Here, I.

A

Had the same questions about what does single data stored mean.

A

So I gotta, run and I will come back and read any any updates this doc, but it sounds like we're all leaning towards the surface extraction and logical databases.

A

I'm gonna need to have follow-up conversations and the strategy the mr that I have written up. It does follow that same strategy, the service extraction.

A

So then the following.

B

Question is going to be about charting those specific logical databases right.

D

A

Not necessarily right because we're only considering charting the major the main rails database right now, because.

B

A

So big- and we know we need to scale that out, but how does that complicate things for our customers? Is the big concern like the overhead with distribution which and room just answered and doesn't think it's a major concern? Do we need to change our or dogma for the single data store? How do we define that? How do we get everybody on board and how do we?

A

Yes, how do we manage so not every so we don't have like 40 different databases in a month right. How do we manage the database growth if we're making informed decisions about creating individual databases? How do we make sure that it's not just, though everybody can create a big.

C

I think I think that that's something that I've been thinking about a lot I think that's a big concern. Is people just proliferating?

C

Definitely don't want.

G

F

G

We mean that we have, we are going to to connect to them using the same address and same board, so they are so. We assumed from the get-go that that the resulting the same so it'll be server. I was.

C

Imagining same PG bouncer, but different connections in PG bouncer. You.

B

Have different PG browser, sir, like he doesn't have yeah.

H

C

But but the default would be one right or.

C

B

I mean the beauty, is you can J? You can choose right like if your self-managed chose you can say just connected the same day to be okay and then we'll select, which you know logical database, but for us we can split it out if we need to and I think we're going to stand.

C

Could we, this is a bit of a city like a bit of a crazy idea, but could we have multiple PG bands of pools that are actually behind the scenes, connect to the same database? And so you have the same tables in the same database, but with no connections between them? Do you follow sorry?

C

Imagine you had like if you wanted a single data, so I mean this is kind of just hacking terminology. But if you say you had a bunch of container tables, you actually keep them in the same database and for self-manage in literally the same logical database. But you know we just we just keep them apart and then we go through, but.

E

You can really is certain.

F

C

I think logical databases seems like a reasonable kind of approach. I.

E

Mean like the trophy's right for the container this we, like you, know not have to scale. These I mean star these database, probably next 3-5 years, so we may need to start enterprise, but or this actually databases gonna be just fine biscarrat and slowly as we did, we've kicked our price because web atom.

E

C

About if we had like a strategy that when people shot it off the database, the the noushad had to be sorry, the new the new service database had to be database horrible from the onset, so that.

F

If you would document.

E

E

G

E

Would be like be quite useful in the future if we have to do it, because I think I'd realize what is facing right now with the chiming of the guitar price, that this data structure is only partially stable. So it gives you like very mixed results, and this is export problem of starting enterprise. Today, I.

B

Know like amazon, I think someone told me amazon doesn't allow for keys. They require you to use not actually an RDS. They actually require use like no sequel solution, so that it allows you to kind of break apart. These kind of.

A

B

So I don't think we should do that, but that's the extreme approach that they're they've taken right.

C

From 1996 that any cross-department access has to be API, you can google heard.

B

About that yeah, but I mean that's, that's going to the designer to be startled from the beginning other than you know thinking you know.

D

That we could easily start with with those services with partitioning, for example, so you would have that idea of sharding in mind when you create a new application and from the get-go you start with partitioning along those that dimension and that I think that naturally prepares. It prepares your data model because you already have to have it in that way. Better one, you can. You can dislike to shot that.

B

I think the container registry is probably the first place. We want to try this experiment because they're about to try to do something there.

E

To ask him to love lucky Ruby and lucky Co and basically had like the single like the single implementation for a folder that are based, manipulation and like database configuration because like right, you could. You could start enforcing these decisions about the partitioning as part of the specs, and if you would have that consistent in that out key for Ruby and goal, it would kind of be used.

B

Right away having kids sculpt is increasing quite a bit yeah.

C

That's caught it yeah I mean it's possible. Oh oh, wait! Yeah, it might be, it might be, I mean the the main thing. I wouldn't want to go in there like a.

D

E

D

Creating those applications and from from scratch and I'm already thinking about shouting, that's a that's an awesome way of doing that. How do we go about github rails and that's that sort of the tough spot that we're in at the moment to figure out how we approach the monolith so.

C

What's the number one hot spot that, like you like, keeps you up at night like what can? What would be the thing that, if you have to pull it out and Charlotte, what would it be I.

D

Don't have a really good answer to that, so I think you need to be more systematic about understanding, I'll load in terms of in terms of database size. We can see that CI data, for example, is really a lot. It's like a major portion of the database. um What.

C

About like traces or jobs, yeah.

B

That's that's what it is right. Those rows.

C

Are growing like.

B

Crazy, even though we've offloaded a lot of this stuff, the traces themselves to s3- but you know, there's number of rows increases like crazy, like those dips are actually the biggest, but we move offload of those thought restored as well right and there.

D

D

E

All those missions.

D

Then you know trotting that or expecting that we don't even have favor attention stride she for CIA data and we've been discussing up to. Maybe that's also way of doing it, but it.

C

Old customer yesterday- and they were complaining about the exact same thing that.

D

C

Ran their own scripts to clean it out, but.

D

That aside, how would we also from an organizational perspective like approached us I, feel like from our it seems perspective? It's not possible that we find a a drop-in solution for the database without touching the application ready. So I think this is a much larger scope for us to tackle. So I wonder how we would even approach saying that we want to I, don't know, extract a feature from from gitlab once we had and if I, what what that is I think.

C

Nobody's talking something like the container registry and then have it as a like a demo that people can look at you know and then it'll be.

F

E

As for the CIB, if I'm starting I, don't think that you're gonna be able to do it efficiently today before we changed it like before, we start doing CI demon as a separate service that can efficiently cue bees, because there is very non partition that you could use or this here each table that would be efficient.

E

That's a tricky part because then, like my natural answer, would be trying to partition by the prey psyche, but you how the system works today, it's not charming friendly at all hunters that reflux palate, so it will very miserably, and this from my perspective, until we do until we move these all processing of the queuing into something more optimized and specific, then like I would then say you just start by project like we are always access pipelines in scope of the given project.

E

We don't cross projects ever really when accessing this data right, it happens very rarely, but making very specific cases but or the minority. It's like it's the product, which is like the scope for the pipeline. This.

D

Is that not the case today because he mentioned that wouldn't work with the current database.

E

This is this is the case today, but because of the cueing cueing doesn't care about the price it rules artists at this default system, so you need to come that perspective of the whole system, I kind of, like you really remember that you've had the issue about testing different partitioning for the filtering issues and like the problem was from what I understand you are crossing boundaries between different charts, at least from what I understand deciding to be efficient like you, don't really want to cross boundaries because then, like joining these data, it's like super expensive.

E

So because of the queuing of this guy reads, you cannot do shouting, because these queuing works on the depending on the instance level group level or the private, uh but you cannot figure out efficient partition for that. That's the tricky part. This is how I understand that. So in order to make CI CI char label, we need to move queuing away from get lock. Then, like you could start charging CI using project and it would work pretty much without any application changes.

E

It would be only post questions done, but today's would we get we and I don't think it will work efficiently at all now, I think you'll not have severe performance degradation because of that how we process q today this is the problem.

E

So let you requires the queue requires. You ask to have the full view on the system which is very sweet, so I think that's that's freaky. So.

D

What you're saying is also that don't even try to like change the existing system and make that shot up over you know apply partitioning to it, but extract that wait. Wait until we extract that to the part of the system and make that more scalable from the beginning. Is that what you saying from 107? Yes,.

E

So so what I'm saying like starting on the of the Skype? It doesn't make sense today, because we gonna fight what we can do like is move queueing in two completely separate service site optimized. For that purpose, the current system is not optimized for that car and, on the other hand, basically finish data retention of the civs. It's gonna remove a significant or significantly working data set that we need to keep in the memory today of the CI bees, which increases.

B

E

Pressure on the l1 l2 l3 cache when you are going through indexes so.

E

E

How balls are given charge queuing? We can.

D

So we're basically in a time where we we would already know about the scaling problems. You know that we we hit a wall in like two years over there with the current design and we're so we're in a good spot today to figure out what parts of the system we would extract and redesign.

D

Instead of trying to you know, keep going in that direction without thinking about the about the service design better and then trying to chart a find IHSA charting solution age, a distributed database solution that allows us to keep going in that direction. So we would prepare for, for you know today, we could prepare by by extracting starting to extract services from from the application making them better yeah.

E

E

Time scene for starting.

D

B

Everyone I have to run now, but there's a good discussions. Yep.

H