GitLab Gitaly group, 13 Sep 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Gitaly Training - Architectural Overview

Description

Will Chandler gives an architectural overview of Gitaly.

A

Oh looks like uh we have a good number of folks um and then this is being recorded as well. So I think we can go ahead and get started. So uh my name is John I'm, a yam on The, Daily team and um yeah. We we wanted to do a series of talks, I guess we'll call it training sessions. Originally, the audience was support.

A

Folks, um because yeah giddly is a pretty complex uh topic and uh comes out comes up a lot um uh issues happen uh with customers or uh you know even on getlab.com, and so we just wanted to provide a series of talks to kind of um kind of dive into different aspects of the giddly um code base. I guess so not just the code base just how it how it works and how it works with other parts of git lab.

A

um Just so folks can get more familiar um and so yeah um today uh will Chandler uh he's a senior back engineer on Gili team. He has a support background as well. Actually um so he's gonna give our first session on um like an overview of Italy architecture, so take it away. Will next.

B

A

C

I just make sure we're recording uh that is: okay, we're recording it cool uh all right yeah. So this is just uh supposed to be kind of a a little bit overview of giddly from kind of the admin perspective mostly and how the components fit together. uh All right, can you see my browser.

D

A

Cool sorry well, I forgot to say one thing um sure so we're gonna have q a at the end, so uh oh whoops I just sent the wrong link.

A

um Okay, I'm gonna, send over the uh agenda Doc in the zoom chat, so feel free to put down any questions uh under the question section and then we'll have about 20 minutes um for uh to answer those questions. So sorry about that. Go ahead. Well, of course,.

C

Okay yeah, so basically, my goal is to kind of like walk through. What does this diagram mean in a little more detail and uh give you some more ideas on what gitly is doing under the hood?

C

um So giddly is a grpc server um that handles any git operation, you're doing within gitlab, so anytime, you're looking at a file or looking at branches or anything that's touched in the repository, that's going through Italy.

C

um In the past we had a sidecar called gilirubi that is gone as of 160 I. Don't know! Where uh tone did that work thanks tone um and then, as of like 16 4, there is also another binary launching called uh good to go. Let's use the get two, but that has now been fully replaced with just git, so any git operation you're doing we are just working a git process, basically standard git I think we occasionally will use our own patches on our Git Version.

C

But typically we just try to stay with the Upstream version of git um and if so, we're just working there and reading from standard out typically to get the uh whatever details are uh okay, so there are kind of three main clients that giddly serves so there's rails itself, so like a puma process, so anytime you're actually like looking at a project. That's that's Puma, sending rpcs.

C

um If you're familiar with the performance bar, which you can get with hitting p and B on your keyboard, uh you can just click here and see rpcc's rescinding so find license, and you can see all the parameters we're sending so typically anything that rails, ascending is going to be small.

C

You know, like hey, show me what files just by this name or what are the references that exist and so forth, and so those are all going to be overgrbc and they may or may not be streaming grpc responses, sometimes they're unary or like a we have single response. Sometimes they stream back depending on the size, um but that's that's puma and then next would be Workhorse. So workhorses uh gonna be a longer live connection, because Workhorse is pretty unusual to restart it. So any HTTP get operation which angular.com will be.

C

You know, 95 of CI um or in.

E

Any environment.

C

Really I was using GitHub CI, it's going to be it's going to be HTTP right, so usually you'll see most of your clone or fetch traffic come out over Workhorse, um and so, when we're doing any kind of Clone or fetch, we use What's called the sideband. So originally we were, we were sending the the pack file details over grpc which involve getting the output from git encoding into your grpc buffer, sending it over to Workhorse and then decoding it and writing it up to the client, and that was inefficient. So it was.

F

A year or two years ago now.

C

um Scalability Team added What's called the sideband connection, so we Multiplex the grpc connection to create a very simple pipe on there. That will just send the the pack information and clear so we don't have to encode it. So that saves a lot of CPU time. So that's why you'll see like post, SSH, upload pack or post upload pack with sideband? That's what that with sideband is referring to. Is this this Multiplex connection that we have running um yeah? Thank you side.

C

All right said band is git itself outside channel is the thank you Tom right, um and so so it's like I, said any any kind of git client operation so like a CI, Runner cloning or fetching or someone just on their own terminal, doing a fetch, we'll all use uh uh Workhorse and then the the final connection or client is shell, so that that'll be for incoming estate requests on kitlab.com we're using uh gitlab sshd.

C

So that is, you know stable process, and so that will have an existing connection out to giddly, and so we can kind of reuse that connection, um so, if managed instances that's off by default. So when a new connection comes in we're going to the standard, opensshd is going to Fork a gitlab shell process, which will then establish a new connection. So you'll see a lot more connection churn there. Things will be much more ephemeral when it comes to that.

G

C

It's the same same deal as Workhorse, where any it's just over SSH um it'll be any get client operation.

H

C

That is more likely to be humans. In most environments, some self-managed customers use SSH for their CI. So it's not always, but it depends uh so there is. This is the closest thing we have to a diagram um which is a this page here.

C

This is not mentioned, Shell versus uh well. This is on the Kelly docks. There is also like the larger gitlab.com architecture somewhere in the docs I. Don't know if someone knows where that is offhand, I, don't I, don't have the link handy, but that's uh that's pretty enormous, but you can you can pick out all the stuff I'm talking about there too. If you know what you're looking for it's just there's a lot more stuff going on.

C

um Okay and then occasionally, you also have Italy now to talk to each other. For example, if I am working a project, that's not using an object, pool I might need to go from one giddly server to another, to fetch the details of that project and then clone it over to my own goodly server um and I'm, not actually sure if this, like loopback connection to itself, really happens these days, that used to be for giddly Ruby, where we would like connect over like to the the host address that rails, they would buy um I.

C

Don't think offhand that we still do that, so this might be a little out of date. uh This section here.

I

B

C

uh All right, so those are the three clients that we have um was it going to show you oh yeah. One thing: that's kind of interesting to see. So, if you so here is a standalone, giddly, node.

J

Actually, let's go here.

B

C

If you look at the storage section of the config it which, if.

K

E

C

uh Corresponds with the storage section in the giddly configuration, so you can see here, there's only one storage name listed here, so so, if I wanted to Fork. So let's say: I had a a repository on Italy one here and I wanted to clone it over to giddly Shard to say before, because it's a private project. We can't use object tools, then what you'd have to do is you have to create a connection, but Kitty itself is not aware of any other giddly nodes um so like on.com.

C

Actually, if you look at different figs, you'll see that, like all the giddly stories are listed in the config, but it's on it's only there for ease of configuration um in reality, it's on it's only using whatever storage that node is actually configured for, um so that's slightly confusing, but but basically, how does? How does this giddly node know to talk to this one? Why doesn't the config the way it works is in the rails, node.

L

Actually, excuse me.

C

um Rails will bundle in the addresses it's aware of in the grpc headers um so that we can fetch uh so that it can basically, it tells the Gilly Clans. Here's where you need to go to.

K

C

The other giddly.

K

Node, uh let me show that.

M

Action real, quick right: we moved.

C

I, don't know why they got rid of the wrench. Oh.

N

That's fine right.

B

Repository what.

C

Do I have is the primary storage right now, okay, so my cluster is where everything's going so.

B

Let's work this.

C

Let me also enable.

I

No that's already on turn that off.

C

So this uh goatee buggy should be too debug is enabling extremely verbose, logging.

O

uh uh Giddly, which.

C

You're never going to do in production environment all right, but for our purposes is Handy, uh so you can see it was like writing. Each header, that's receiving on a request.

B

uh All right, so, let's do.

F

N

So now, if I fork.

C

So we see okay. Here we go yeah, so you can see in the uh we get this giddly servers field and then you get the space 64 encoded string.

C

And then, if you expand that you can see that it's a Json blob that says: hey here's, the skilly server Shard, which you can see here in the rails, config and then here's the address, here's his token, so you can authenticate yourself. So that is how giddly servers know how to communicate with each other there's, never any config on the giddly Node itself, as of today at least, to talk between nodes.

C

This a lot of this is subject to change like we're planning to completely overhaul how giddly cluster works and that's going to add some client-side routing which hasn't been mentioned yet um so that may so this. This may be something to change, but right now this is how it works.

A

Could you explain real quick um why, during a forking process ideally would know we need to know about another yearly known.

C

Yeah, so it depends so if you have a public project or an internal project, um Italy will create, what's known as an object pool which is um kind of like a a separate get uh repository which will be the prefix with an at pools path on disk and that will have all the objects in common between the forks. So that way we would do duplicate most of the repository, usually almost all of it, um and we can save a lot of space.

C

So if you have a project like gitlab, which has you know, hundreds or thousands of forks I, don't remember that over top of my head, um we don't have 500 copies of the repository. We have 95 of one copy and then five percent scattered around across the other ones, but they differ um so that lets us be much more space efficient. uh However, if it's a private project, we always create a new Full repository, and so that's what I did in this case, um and so in that case, if you have let.

G

Me go back to the admin page.

C

um Repository storage, so this page here in admin under Repository lets, you set a relative weight for where your new repositories are created. um So right now, I have all of my repositories being created in my defaulty cluster, and none of them go to Shard and I can change that to whatever waiting I wanted.

C

um If I, if I had certain nodes, they wanted to fill up and other ones to not be touched, then I would adjust that value um anyway.

C

So in this case, because I'm creating all new repositories in default and the existing copulate repository wasn't shard the default giddly nodes which are in a gitly cluster, um we'll have to reach out to uh Shard and say: hey, give me this repository, so it'll do internally a little Fetch and get all the objects and create a Zone cut their copy of the Repository uh I'm going to show you what it looks like on disk real, quick.

C

So, by default, this is the directory that, or Anonymous instance, will put the repositories in um you'll see this gitly metadata file, which is historically been used to identify whether or not we've been using NFS. So, basically, the way it would work is rails would validate what this file looks like on its end and then the Gilead server would also say like what this, what the EU ID it has if they match, then we say: okay, this must be NFS and we'd use previously we'd use rugged, which is replica two bindings Ruby, okay,.

A

Started well, I, don't know if the other Focus are experiencing this, but I think it's getting cut off at the bottom. For me, it could use this type thing clear, real, quick yeah thanks.

K

C

um Anyway, so I can't remember at the top of my head: if we've removed record entirely or not I think we've disabled our right job, I.

A

Think we uh yeah well I, think you can well. No! You can still use if you really wanted to through feature Flags, but for all types of purposes. It's not being used by an application.

P

Will be removed in 16.5? Oh that's. The current cloud.

C

Great okay: well, we won't talk about it too much, then um all right and you have this.gilly directory or plus Gately. Excuse me, and this is kind of Italy's scratch directory. Where, when we're leading a repository, for example, we will rename it as a repository name Dash deleted in this directory, so we can atomically move it out of place and then from there we can delete it and we create um reference or archives or archives caches for um the uh the Reps that we're serving.

G

uh That's another thing: that's in there.

C

um It's really implementation detail and then this at cluster directory will have the actual repositories itself and if I had any pool repositories, there'd be a pools directory here too. um This is this is the new.

Q

Format, uh that's interesting.

C

So you can see, we have some very repositories in here: um commit graphs, uh backfill, reverse index index, bitmaps all the different housekeeping things that help keep you efficient, uh and this is a file that guildley creates to say Hey when's, the last time we did a full repack, so we can keep track of that.

G

R

C

As I go here, I think I see yeah. Okay, let's.

K

S

S

K

Yeah sorry, okay,.

C

Okay, so um just to explain a little bit more, so this host here get a triangularly. Two is then a giddly cluster, um and so prefect will be.

T

Use this at cluster.

R

um Format for its paths well,.

C

This is a shirt again so Standalone giddly, which is what almost all of our Ghibli nodes in.com are, and what you'd see in a typical small self-managed instance.

R

C

And so here we use the add hash um format for the path, and so this is. This is the path that you'll see internally in rails as well. So if we.

B

Go to projects.

C

And so like a look here and I can say: okay, this is the path that rails has for the project and this matches up with what we have on The Shard. However, if you look at a product, that's in the cluster it'll show up as hashed, but there is no path at hash 66p86 in in the on the file system here, so we have to do.

U

V

Let's go into the.

C

Prefect, which I have mentioned I'll, get back to that in a second um database to map that between them, um so actually before I go any further. Let me actually talk about what is gitly cluster, so gitly cluster is a way of having multiple copies of a repository that are, you know, kept in sync with each other without uh well, exposing only a single instance to rail.

C

So it's a way of getting redundancy, which is not something that we have with the shared goodly there's no way to replicate a repository to multiple different shards in the current setup.

C

So what that does is typically you'll have a little Bouncer and that'll be the address that you give to rails. uh Let me show you here so like, for example, here this could be my load balancer. In actual fact, I only have a single prefect node in production. You typically have three, maybe more um one. One failure mode is seen somewhat of them with sub managed. Customers is.

C

If you have more than more giddly nodes behind prefecting you have prefect, they can send so much traffic that it overwhelms the network, connection um or network bandwidth. That is available to the prefect node and that will cause silent slowdowns. It used to be it used to cause memory blow down perfect, but that's been fixed.

G

Is it like fifteen five, or something like that.

C

um Anyway, so perfect nodes, what they they're stateless, they have a they have a postgres behind them and that tracks all the repositories in a state of all the repositories, and you know which repository is going to which node. So if we look at the.

W

Repositories table.

C

uh Let me make that bigger, so we've got right now: we've got four repositories, two of them being one keys to them being actual projects, and so you can see this is the path that rails knows.

G

um So if we go back.

X

C

Let's add hashtags v86 Etc, um but if you look here, it actually maps to at cluster repositories, d4732 um and you can see giddly one is the primary node for that which would make gitly two being one of the secondaries. I only have two giddly nodes in this cluster, typically negative. Three, it's two is fine. If a customer only wants to there's no need for Quorum or anything like that, um and this generation counter saying like how many changes have happened. So this is how we keep track, of which node is in which state.

C

So if the generation counter is the same. So if we look at storage repositories, so you can see here they both have the same generation. So in this case a read request for this repository could be read to either gility1 or get delete two because they're both at the same generation.

C

So this lets us an active repositories in particular, distribute reads among multiple nodes and decrease the load on the primary. Any right will always go to the primary and one issue that some customers are seeing right now in general, is that if a right cannot be simultaneously proxied to the secondary and they don't they don't agree on it, for whatever reason so like the secondary fails for an update for whatever reason, then the secondary falls behind.

C

It has to replicate to basically do a fetch from the primary um if it's a very large repository that fetch might take 30 seconds in which case, and if changes are coming in faster than that, the secondary might not catch up to the primary for hours, and so in that case we wouldn't be able to distribute reads: We Do, It, All rights or all reasons have to go to the primary along with any rights and so you're losing most of the benefit other than you know.

C

Having redundancy in terms of you know the primary hard drive failing um in this scenario, you can't you can't spread the load, so that's a failure mode that we see somewhat often it's been improved. Most I think pretty much. All RPS at this point won't do that for a while. There were some that would always trigger a replication and that uh you know that just decreases the efficiency.

C

um Okay. So that's! This is kind of internally how prefect nodes are tracking. You know like how can I route a request. So basically the request comes a little balancer low balancer chooses any of the stateless prefix nodes, prefect, we'll check the database say: okay, I can send this request for repository 6b86 to either get a 1 or giddly 2, because the generation is three it'll route it um and then it'll so it'll proxy.

C

The grpc request down to giddly one of the giddly nodes and then giliwoo do its thing: send it back out to prefect which proxy it adds a little bounce, we'll send it back to the client and so on. So there are a lot of steps there. um uh So how does giddly actually get get data so I'll show that real, quick.

Y

So it's s tracer.

W

Z

Actually now, let's do this chart, it makes more sense.

AA

E

P

AB

Everything I'm gonna, say it makes sense.

AC

Y

AD

O

C

So disability of this page will have triggered.

K

Some yeah all right.

C

All right, uh let's actually cut that ahead.

C

It's kind of hard to read: isn't it sorry um anyway? So let me let me walk through what's happening here, so so the rail is going to requests to pull up a project. So it's going to send a few rpcs so like. If you look here and see it's sent search files by name in this case. We are looking for itself.

C

um And then it's looking for markdown files, then the.

G

Wiki repository, you can see the reg X that we're sending.

C

And then, in addition, we're also to.

G

C

The wiki that's interesting, that's fine um has local branches. So it's saying hey what branches do you have um to this Repository?

C

um And so here you can see this is the the um the file that's actually being forked. So it's it's not opt embedded in Optical have embed and Bin git. um Getting when it launches will extract its own git reposit, um don't copy of the get binary.

H

Z

Let's actually just.

C

Expand that directory, so here you can see everything. That's there, so you've got get to go, which, as I said, should be gone at the next release, um and then you have Kili gpg. We use that to sign and validate uh signatures on commits hooks is used for well, it's used for the pack objects. Cache is kind of one of the primary things which I'll get into in a little bit.

C

um Lfs smudge is used for handling lfs objects and that's kind of a long layer process that we use to make it more efficient and then give the SSH is used to connect internally between guildley nodes, and then we also have look Hooks.

Z

And socket directory in there as well.

C

um Okay and so sorry so I just started printing, it's confusing all right. So then we've got this guitar directory, saying hey! You want git operate in this directory here and we always say: Don't run get GC automatically, which, by default git, will join GC on some operations, just as like a standard housekeeping but cataly controls housekeeping. So we don't want that.

C

um Remember, saying: treat controlled F to whatever you're receiving or crlf the line ending, don't use, replace refs, which has been a security issue. These are the objects, that'll be f-sync, so objects metadata references or fsynced. By get uh whenever we make a change there and then fsync is fsync uh and this I believe has to do with reducing contention on the Pax rest file.

C

uh Okay and so then here's the actual commander running, which is get cat files. This is saying, give me a blob um and then we're running Z, which will give it like null uh null Bank, terminated um and then we're saying patch command, which is saying we're going to send you a number of commands every time. So I don't see if we have.

Z

Any get file processes right now.

C

And we don't um so what gittley does when were these get for these Ruby or rails operations that are often making lots of little requests so, like I, want to look at this file? I want to look at that file um within the context of a given uh rails request, so I think it works out to a single correlation ID.

C

um We will Fork a cat file process and then keep that around, so that we're not for each file that we're trying to view we're not forking a new cat file process. This lets us be a little more efficient um or fewer processes.

C

So when you, when you look at a production, server, you'll probably see a whole lot of cat file processes around those are mostly just their hashed waiting to see if any more requests come in or or that given request for that for that given overall, like web.

AE

Request, that's coming in.

C

So that's that's. Why they're there and then the other thing you'll see a lot will probably be uh upload pack or.

G

Pack pack objects.

C

Related to clones, those will be long-lived typically for a larger clone.

AE

um So there'll be around more.

C

uh Anything else I want to talk about. Well, let me pause right here, first, um anything that any questions that are like immediately relevant that anyone wants to ask.

AF

I have a question just yeah: what's the logic behind creating its own work of the git.

AG

um Executable is there any reason for it or.

C

Let's clearly have more control over what git process or what get uh binary we're running um I. Don't remember the reason that it was difficult to keep it in sync with Omnibus, but this way we didn't have to rely on Omnibus updating what it bundled we could always. We could very you know, control exactly what this is, what we were executing and clients or you admins- can still override the get binary. So I don't believe you have to use this. That's not something we recommend, but it is something that you may encounter in the wild.

C

If you're dealing with a self-managed customer.

A

Yeah, another reason is sometimes: if there's like a security fix or some high priority fix, um we will build a git binary with some of our own changes.

A

um Usually we ship it's the standard, git binary, that's released, but sometimes we'll ship with some of our own changes so um yeah. If self-managed customers use their own git binary, it might break because it doesn't have the features that um Italy needs.

AH

We don't ship the security patches to our customers before they're released.

AI

Yeah, that's right.

AJ

C

Yeah, but one other thing I can talk about within the connective. Gearly cluster is failing over between primaries. So if I go.

H

C

uh So right now we have for this repository Italy one. uh Its primary is going to be one.

K

um So that is the skilly? No! That's not that one. Let's go back.

L

AK

Yeah this one okay.

X

So if I go to giddly one.

C

You stop giddly and then I send a request to view the project we should fail over at this point just to get a lead to.

K

AL

K

I need to go in actually a.

M

E

What am I doing wrong.

C

Can't, oh well I wonder if we need to make a.

H

Z

Because it might just be routing the reads there, let.

H

Me actually try modifying something.

AM

Subscribe, your file.

C

All right, okay, I just couldn't test that I need to update. This is a config problem. It's not right writing out to.

AE

The right address for the internal.

M

Api all right well.

Z

Let's make that work, let me tested this beforehand. Oh well,.

C

um No actually I was just reading it wrong. It did update okay.

A

Probably just took a second yeah.

M

C

Maybe it already changed, I didn't even look right, that's probably it anyway. So you can see now for this repository only. We have now updated the primary to begin League two. So the way failover works is when a request comes in that the primary needs to handle and the primary is not available. Then we will update the primary to be one of the other in-sync nodes that is on the gildling cluster.

C

If there are none, then the repository becomes will become unavailable until the a up-to-date node returns, um but we only do that for repositories that come in the have requests come in so you'll note that this other repository has not updated. Even though giddly one is offline. Originally, we did update all of the repositories and that caused a lot of work in postgres, and it was just it was. You didn't have to do it that way right so now it only it. Lazily updates the primary as needed.

R

K

T

One thing that might be relevant.

K

B

C

Yeah and then I'll just quickly mention um you may see, deadline exceeded errors in the logs. That means that some operation that rails, typically until it's always going to rails, are sidekick um sent took longer to regularly to serve than the timeout configured in the admin.

H

Area, so if we go back again and leave page.

C

Preferences, giddly timeouts, so whatever these values are set and whatever the operation is to develop, the assignment will depend on the uh evaporation that you're doing, but if it takes particularly if it's a large repository and it takes longer than this, then you'll see a deadline exceeded the other area, you'll see a lot is canceled the context canceled. That means typically that there's somewhere between giddly and the client, the connection went away. Maybe the client closed their git process. Maybe you know, load balancer went out.

C

It could have been anywhere along that chain if the connection is lost and you'll see that connection error, contents, canceled, uh error, message or link Italy, so basically just means that the client went away for some reason. We don't really know why. um Just you should look Upstream to try and figure out why that happened.

B

C

Yeah I think that covers everything I wanted to go over. uh Let me stop sharing for a quick second I'll.

G

Pull up the dock.

A

Thanks will so um yeah I know that that was a pretty technical. So if you have any questions or if you want will to go over, maybe one of the parts that he already went over um uh again, uh maybe yeah. So um please uh put those in the questions and um we'll go over them one by one.

B

C

Okay, oh after asked why we use our own, which I think we covered um manual.

AA

C

Oh so so the pooling feature.

C

Yeah Karthik has answered that so so we're using a built-in git feature called alternates. So basically you can say here's a repository on disk and then you'll have an alternate file. Then it'll say: okay and I'll also go to this other directory and use that to get additional objects from it. So we're saying all of you all of you pool reposit all your projects that are using this pull repository.

C

We add that alternate of the pool to them and so then get itself just knows: okay, I need to go to this other directory, which is just a relative path to um to fetch the objects so there's limitation, which is that, because it's a relative path, you know it has to be on the same node like we can't sell git uh like using HTTP request or something to go, do it so that limits any any four cluster to a single giddly node with a negatively cluster.

C

um In theory, you still have the forks spread out or replicated amongst all the nodes that was broken for a long time. I think Justin has gotten that basically fixed at this point or close to it yeah.

C

um So it should start working soon if it doesn't work now for a while, we weren't keeping those in sync accurately. um Does it answer your question at all.

K

C

Okay uh Steve asked: is there a Civic reason why I put things in the cluster directory as a.

AE

C

Are we adding extra overhead of translating the path through uh prefect, um so I think the main answer there is that we wanted to have a a permanent home for a given Repository and but rails was still asking us to move repositories around on disk now, particularly in the context of Geo. So you'd have like a new repository created and G at Geo temporary and then once the fetch hit geofetch and completed, then it would get moved into place.

C

um So the actually, the reason that we have the add hash name original in Italy or on on giddly shards, is the same logic. Originally. The path on disk was the repository names so like including the project so it'd be like fire. Advocate lab, git repositories, repositories, um getlab, Dash, org get lab would have been the path forget that. But um whenever you had someone you know rename their project or rename their organization, they ended up doing a ton of work on disk and just rename all these directories and it was expensive and error prone.

C

So that's why I created that hashed format, and so then this cluster thing is doing the same thing um just at another level of abstraction. Does this way kill the cluster knows that a repository is always at this path, even if rails thinks that it's moved somewhere else. So that's that's why we added that.

AN

um It does add another layer of complication. Cluster is always the same. No matter what you get delay shark, to talk to correct.

C

Okay yeah. So if you look at.

C

C

uh One yeah, sorry that was actually over here. So if you look at the path, so it's actually the hash of the ID, but instead of taking the full hash here like we do with Ed hash, we just put the actual number here. So this is the full hash of number one. This is also hash. One just use one at the end, um but they don't they. Don't always. You know correspond with the ID that uh rails has like this in rails is Project one and cluster it's project or sorry.

C

This is Project one and cluster and project two years project, one in rails, project two in Gilly cluster. So one issue that we see is that, should the prefect database be lost, there's no convenient way to recreate the association between the path that rails has and the path that the cluster had.

C

So there's yeah, there's just no. Unless you're backing up your database, which I don't think we really advised in the docs there's just no way to re-establish that you can. You can kind of fake it if you can go through the logs for so for any traffic, any project that was creating or getting traffic. You could at least look at recent logs and say: okay I can translate between those two because that'll have both, but um if it's like an idle project, then you're, probably not gonna, be able to easily do it. There is.

C

There is one thing you can try.

W

uh If we go into oops.

C

It's looking like a fig. Actually, this one doesn't have it okay. Well, there we go yeah.

AN

So sometimes yeah.

C

Yeah for a long time, I used to write there used to be a gitlab section in this config in the git config. That would list the the project name, so you could relate it back that way. um I think we added that, like 10 version, 10 could get that 10., um but it's that's going away now, so you can't rely on that either.

C

uh Okay, Sean asks so some customers, you can get a independent of gitlab hugging like clogging face I, think is certainly the most prominent. There might be others that are doing it. That I'm, not aware of um so. This, isn't something that we officially support um in terms of like a support team's not going to help them. There's most of the logic in Italy is pretty agnostic there. There is some stuff that is get lastific. You know, like authorizations, I think are hard-coded to go to internal aloud. For example, we do have documentation.

C

I got my prayer, I think it's yeah this. So let's try this Lincoln.

G

A

Yeah and if we're the documentation for RPC or rpcs it's uh there is um good amount, that's missing and that's something that we're working to address. Yeah.

C

This is all just automatically generated from the comments on our protobuf definitions, so it may or may not be entirely useful understanding how it works, but that it is something that's out there, and this is. This is generated automatically if it's part of the CI pipeline, so it'll it'll be up to.

K

P

Yeah, so we do treat Italy as a product, but it's an internal product that we don't currently Market externally hugging face is an exception. They know what they are doing. They are super technical. They understand exactly how git and gitlab and giddly works.

P

They just want to use their own front-end, so they are able to do this, but this is not something we would recommend at this point, however, what we want where we want to be, is to offer a clean, RPC interface with all the implementation, details and internal logic, some of which is currently managed in trails to be moved to gidly, for example, how object pools are created and deleted uh is is kind of a detail that doesn't belong in a client, whereas we want to keep all the business decisions, things that need access to the main postgres database, like customer, tier or customer settings in the repository or or in in their projects.

P

You want to keep that on the real side, so that giddly can be agnostic and just be told what it needs to do.

AO

Make make sense. Thank you William, and address very, very interesting that we are even thinking about having it as a separate component.

P

It may never be marketed outside this is uh this is a product decision, but it does help us to be more independent and develop uh the overall of gitlab yeah with uh less friction. If we have a clean interface between components.

AO

Of course, thank you in.

AP

The briefly mentioned from the product perspective, this is also really important because we formed some very strategic Partnerships through this sort of opportunity. uh We get a lot of contributions back from external people who are you know, interested in giddly hugging face is a great example, but there are others as well that we've seen over the time where we've gotten contributions from other.

AP

You know teams sort of looking at it and it gives us another perspective from a product perspective, and it does ensure that we are continuing to develop giddly in the most independent way, which is part of the reason why we are, as the direction page shows, trying to put the business logic into the rail side of things to make giddly somewhat um agnostic to who is calling in so yeah. It's been an overall win for us, but, as honors has said, we're not actively marketing or pushing it as an independent product.

AP

Just yet, though, that may be something we look at doing in the future.

C

All right thanks everyone, uh so next question from Clement um interested in the cost of serving uh cost of service. What are the most costly operations so I know on.com using fast Storage. Snc storage has been the largest largest cost component of daily. By far, um it really depends on the instance, though so you know.com has is getting. You know, lots and lots and lots of repositories. You know business, you know like a self-managed instance.

C

You might not see as many you might see like a single monitor repository, um which will still be large, but you know not not the scale of you.

G

Know 100 terabytes that you see.com for the you know the.

C

Vast number of repositories there so typically storage is a big expense. Just because you for Reliable and performance service you need to have fast storage um and then, depending on the size of the repository, you may also see significant expenses in terms of compute. So, if you have, you know a truly vast repository, let's say 20 gigs it can easily use 10 or 20 gigs or operation, sometimes depending what it is.

C

So if you're, if you're running multiple operations at once, you know- and you only got 10 gigs of RAM on the box, you're gonna run out real fast right, so you're gonna have to have a really large Post in that case, just a serve the repository of that size, but more typically, storage is larger. Cost.

C

Yeah goodly cluster, you know I would say that you'd expect to spec that are the same as a standalone, giddly node, because you know internally we're still retrieving that as a single node to the world right. So I don't think you'd want to skimp because you're still writing the same operations on a given node, just maybe fewer of them, and any of them could become the primary at any point right.

C

So you can't say like well this one's my primary so I'll upstick that and leave the other ones a little bit wimpier because we could fail over. You know anything we'll let.

C

Skill with usage, yeah I mean so typically, the vast majority of traffic to gitly is going to be CI traffic. So you know a Sierra Runner, fetching or cloning um humans are almost always like. You know, 10 or 20 of traffic from.

Z

What I've seen at least.

C

So you know getting that as efficient as possible. Is the number one priority, so that is why we added the pack objects cache a while back. That was also from the scalability team. uh That's listed down here. So basically, when a fetch comes in or a clone, um we will generate, the git will generate the pack file, it's going to set up the client and then this we're using the gidly hooks binary. We'll then write that pack to disk.

C

So if an identical request comes in, we don't need to go to git to recompute that so we save all the CPU power that would have been used to create another pack file. I can just copy the bytes from disk serve it out to the client. So that saves us a lot of CPU time, but it does add additional pressure onto the the storage, because now we're writing all these packs that are coming into disk, which you know typically they'll, just be a memory and stream Bank out to the client.

C

um So it's usually a win, usually you're, more CPU bound than storage bound. If you have a whole lot of clones coming in at once, um but it depends.

G

On the the instance of the pet traffic pattern, so.

C

Something to look into in general, but maybe not always a win.

B

K

C

That covers that um Terry asks any epics or issues related to sync operation to clarific Italy, cluster replication um yeah. So so I mentioned this in passing earlier. We are moving Italy in the long run to a new architecture which will remove the need for prefect.

C

So that's going to be on each giddly node we're going to have it right ahead, log, basically like what postgres uses or other databases um where a change comes in. We, you know, save its changes to disk and then we apply them one after the other, if you know serialize them and replicate the changes out to any other nodes. That also happen to have this. This given Repository and then the client will be so in giddly will be routing clients without prefect. uh That's pretty well sent to the blueprint for this I.

C

Remember the detail: that's going to work, I think there's going to be a writing table that get a labels served to clients and then some clients will know where they can go and we'll have to be able to like retry to a different Gilly server. If that giddly node goes away.

C

um So there's gonna be a lot more complexity in how giddly talks to clients when this is done, you know right now. It's basically rails has an address. It goes the address. If there's a problem, if you return an error, that's it there's no retry logic! That's that's that! So it's gonna, the client will have to do a lot more work whenever that comes around, but that's uh that's still a ways out foreign.

C

Sorry Terry um but yeah, so we're not we're not putting a a ton of more effort into prefecting and we're still polishing it up, but uh so like, like I mentioned, you know, like other PCS, that previously forced a replication job. We've got that down to like three.

G

I think at this point or something but uh we're.

AQ

Not yeah I was just being um I was just being nosy I've seen that issue, uh that was epic that was linked. Thank you. I didn't get a chance to read through it all the way, but we are working on uh zuk's replication strategy, so I was just curious, as for uh you know, at least it's good not great, to hear the problems that you're encountering, but at least something to think about when we're looking at how the replication is going to work on the zoo side.

C

Yeah I'd be very interested to see uh what your team arrives at.

B

C

I think that's we're at time. That's all the questions in the dock um I'm happy to hang around for another couple minutes. If anyone has anything else, otherwise we can call it.

A

Cool, so thank you so much will for um that session.

A

um So um hope that was helpful to people um so just wanted to announce that in two weeks, we'll be doing our next session and the topic will be diving into get internals um or yeah diving in to get how git internals help debug Gilly issues, so that one should be really interesting.

A

um You can take a look at that issue. That's linked there for upcoming sessions um yeah by the way I haven't sent out. I haven't set up the calendar uh event, yet so I'll do that shortly and send it out on slack, um but that is it. Thank you. Everyone for joining have a good day. Bye, bye.

E

Thanks everyone.

AG

Thank you. Everyone thanks.

A

Will um and then this is being recorded as well, so I think we can go ahead and get started. So my name is John I'm, a yam on The, Daily team and um yeah. We we wanted to do a series of talks, I guess we'll call it training sessions. Originally, the audience was support.

A

Folks, um because yeah is a pretty complex uh topic and uh comes out comes up a lot um uh issues happen uh with customers or uh you know, even on gitlab.com, and so we just wanted to provide a series of talks to kind of um kind of dive into different aspects of the giddly um code base. I guess so not just the code base just how it how it works and how it works with other parts of gitlab.

A

um Just so folks can get more familiar um and so yeah um today uh will Chandler uh he's a senior back engineer on Gili team. He has a support background as well. Actually um so he's going to give our first session on um like an overview of Italy architecture, so take it away. Will thanks.

B

John uh I'll.

C

Just make sure we're recording uh that is: okay, we're recording it cool uh all right yeah. So this is just uh supposed to be kind of a a little bit overview of Italy from kind of the media admin perspective mostly and how the components fit together. uh All right, can you see my browser.

D

A

Cool Sarah will I forgot to say one thing um sure so we're gonna have q a at the end, so uh oh whoops I just sent the wrong link.

A

um Okay, I'm gonna, send over the uh agenda Doc in the zoom chat, so feel free to put down any questions uh under the question section and then we'll have about 20 minutes um for uh to answer those questions. So sorry about that. Go ahead. Well, of course,.

C

Okay yeah, so basically, my goal is to kind of like walk through. What does this diagram mean in a little more detail and uh give you some more ideas on what gitly is doing under the hood?

C

um So giddly is a grpc server um that handles any git operation, you're doing within gitlab, so anytime, you're looking at a file or looking at branches or anything that's touched in the repository, that's going through Italy.

C

um In the past we had a sidecar called gilirubi that is gone as of 160 I. Don't know! Where uh tone did that work thanks tone um and then, as of like 16 4, there is also another binary launching called uh good to go. Let's use the get two, but that has now been fully replaced with just git, so any git operation you're doing we are just working a git process, basically standard git I think we occasionally.

E

C

Use our own patches on our Git Version, but typically we just try to stay with the Upstream version of git um and if so, we're just working there and reading from standard out typically to get the uh whatever the details are. uh Okay, so there are kind of three main clients that giddly serves so there's rails itself, so like a puma process, so anytime you're actually like looking at a project. That's that's Puma, sending rpcs.

C

um If you're familiar with the performance bar which you can get with hitting p and B on your keyboard, uh you can just click here and see rpcc's rescinding so find license, and you can see all the parameters we're sending so typically anything that rails, ascending is going to be small.

C

You know, like hey, show me what files just by this name or what are the references that exist and so forth, and so those are all going to be overgrbc and they may or may not be streaming grpc responses, sometimes they're unary or like a we have single response. Sometimes they stream back depending on the size, um but that's that's puma and then next would be Workhorse. So workhorses uh gonna be a longer live connection, because Workhorse is pretty unusual to restart it. So any HTTP get operation which angular.com will be.

C

You know, 95 of CI um or.

K

In any environment,.

C

Really I was using GitHub CI, it's going to be it's going to be HTTP right, so usually you'll see most of your clone or fetch traffic come out over Workhorse, um and so, when we're doing any kind of Clone or fetch, we use What's called the sideband. So originally we were, we were sending the the pack file details over grpc which involve getting the output from git encoding into your grpc buffer, sending it over to Workhorse and then decoding it and writing it up to the client, and that was inefficient. So it was.

D

C

Two years ago now, um scalability Team added What's called the sideband connection, so we Multiplex the grpc connection to create a very simple pipe on there. That will just send the the pack information and clear so we don't have to encode it. So that saves a lot of CPU time. So that's why you'll see like post, SSH, upload pack or post upload pack with sideband? That's what that with sideband is referring to. Is this this Multiplex connection that we have running um yeah? Thank you side.

C

All right said band is git itself outside channel is the thank you Tom right, um and so so it's like I, said any any kind of git client operation so like a CI, Runner cloning or fetching or someone just on their own terminal, doing a fetch, we'll all use uh uh Workhorse and then the the final connection or client is shell, so that that'll be for incoming estate requests on kitlab.com we're using uh gitlab sshd.

C

So that is, you know stable process, and so that will have an existing connection out to giddly, and so we can kind of reuse that connection, um so, if managed instances that's off by default. So when a new connection comes in we're going to the standard, opensshd is going to Fork a gitlab shell process, which will then establish a new connection. So you'll see a lot more connection churn there. Things will be much more ephemeral when it comes to that.

AE

C

It's the same same deal as Workhorse, where any it's just over SSH it'll be any get client operation.

H

C

That is more likely to be humans. In most environments, some self-managed customers use SSH for their CI. So it's not always, but it depends uh so there is. This is the closest thing we have to a diagram um which is a this page here.

C

This is not mentioned, Shell versus uh well. This is on the Kelly Ducks. There is also like the larger gitlab.com architecture somewhere in the docs I. Don't know if someone knows where that is offhand, I, don't I, don't have the link handy, but that's uh that's pretty enormous, but you can you can pick out all the stuff I'm talking about there too. If you know what you're looking for it's just there's a lot more stuff going on.

C

um Okay and then occasionally, you also have Italy now to talk to each other. For example, if I am working a project, that's not using an object, pool I might need to go from one giddly server to another, to fetch the details of that project and then clone it over to my own goodly server um and I'm, not actually sure if this, like loopback connection to itself, really happens these days, that used to be for giddly Ruby, where we would like connect over like to the the host address that rails, they would buy um I.

C

Don't think offhand that we still do that, so this might be a little out of date. uh This section here.

I

B

C

uh All right, so those are the three clients that we have um was it going to show you oh yeah. One thing: that's kind of interesting to see. So, if you so here is a standalone, giddly, node.

J

Actually, let's go here.

B

C

If you look at the storage section of the config it which, if.

K

E

C

uh Corresponds with the storage section in the giddly configuration, so you can see here, there's only one storage name listed here, so so, if I wanted to Fork. So let's say: I had a a repository on Italy one here and I wanted to clone it over to giddly Shard to say before, because it's a private project. We can't use object tools, then what you'd have to do is you have to create a connection, but Kitty itself is not aware of any other giddly nodes um so like on.com.

C

Actually, if you look at different figs, you'll see that, like all the giddly stories are listed in the config, but it's on it's only there for ease of configuration um in reality, it's on it's only using whatever storage that node is actually configured for, um so that's slightly confusing, but but basically, how does? How does this giddly node know to talk to this one? Why doesn't the config the way it works is in the rails, node.

L

Actually, excuse me.

C

um Rails will bundle in the addresses it's aware of in the grpc headers um so that we can fetch uh so that it can basically, it tells the Gilly Clans. Here's where you need to go to go to the other giddly node.

K

Let me show that action real.

M

Quick right, we moved.

C

I, don't know why they got rid of the wrench. Oh.

N

That's fine right.

C

Repository what do I have is the primary storage right now, okay, so my cluster is where everything's going so.

B

C

This, let me also enable.

I

No that's already on turn that off.

C

So this uh goatee buggy should be too debug is enabling extremely verbose, logging.

O

uh uh Giddly, which.

C

You're never going to do in production environment all right, but for our purposes is Handy, uh so you can see it was like writing. Each header, that's receiving on a request.

B

uh All right, so, let's do.

F

N

So now, if I fork.

C

So we see okay. Here we go yeah, so you can see in the uh we get this giddly servers field and then you get the space 64 encoded string.

C

And then, if you expand that you can see that it's a Json blob that says: hey here's, the skilly server Shard, which you can see here in the rails, config and then here's the address, here's his token, so you can authenticate yourself. So that is how giddly servers know how to communicate with each other there's, never any config on the giddly Node itself, as of today at least, to talk between nodes.

C

This a lot of this is subject to change like we're planning to completely overhaul how giddly cluster works and that's going to add some client-side routing which hasn't been mentioned yet um so that may so this. This may be something to change, but right now this is how it works.

A

Could you explain real quick um why, during a forking process ideally would know we need to know about another yearly known.

C

Yeah, so it depends so if you have a public project or an internal project, um Italy will create, what's known as an object pool which is um kind of like a a separate get uh repository which will be the prefix with an at pools path on disk and that will have all the objects in common between the forks. So that way we would do duplicate most of the repository, usually almost all of it, um and we can save a lot of space.

C

So if you have a project like gitlab, which has you know, hundreds or thousands of forks I, don't remember that over top of my head, um we don't have 500 copies of the repository. We have 95 of one copy and then five percent scattered around across the other ones, but they differ um so that lets us be much more space efficient. uh However, if it's a private project, we always create a new Full repository, and so that's what I did in this case, um and so in that case, if you have let.

G

Me go back to the admin.

K

C

um Repository storage, so this page here in admin under Repository lets, you set a relative weight for where your new repositories are created. um So right now, I have all of my repositories being created in my defaulty cluster, and none of them go to Shard and I can change that to whatever waiting I wanted.

C

um If I, if I had certain nodes, they wanted to fill up and other ones to not be touched, then I would adjust that value um anyway. So in this case, because I'm creating all new repositories in default and the existing copulate repository wasn't shard the default giddly nodes which are in a gitly cluster, um we'll have to reach out to uh Shard and say: hey, give me this repository, so it'll do internally a little Fetch and get all the objects and create a Zone cut. Their copy of the Repository.

O

C

Going to show you what it looks like on disk real, quick.

C

So, by default, this is the directory that, or Anonymous instance, will put the repositories in um you'll see this gitly metadata file, which is historically been used to identify whether or not we've been using NFS. So, basically, the way it would work is rails would validate what this file looks like on its end and then the Gilead server would also say like what this, what the EU ID it has if they match, then we say: okay, this must be NFS and we'd use previously we'd use rugged, which is replica two bindings Ruby, okay,.

A

Started well, I, don't know if the other Focus are experiencing this, but I think it's getting cut off at the bottom. For me, it could use this type thing clear, real, quick yeah thanks.

C

Sure, um anyway, so I can't remember at the top of my head. If we've removed record entirely or not I think we've disabled our right job, I.

A

Think we uh yeah well I, think you can well. No! You can still use if you really wanted to through feature Flags, but for all types of purposes. It's not being used by an application.

P

Will be removed in 16.5? Oh that's. The current cloud.

C

Great okay: well, we won't talk about it too much, then um all right and you have this.gilly directory or plus Gately. Excuse me, and this is kind of Italy's scratch directory. Where, when we're leading a repository, for example, we will rename it as a repository name Dash deleted in this directory, so we can atomically move it out of place and then from there we can delete it and we create um reference or archives or archives caches for um the uh the Reps that we're serving.

G

uh That's another thing: that's in there.

C

um It's really implementation detail and then this at cluster directory will have the actual repositories itself and if I had any pool repositories, there'd be a pools directory here too. um This is this is the new.

Q

Format, uh that's interesting.

C

So you can see, we have some very repositories in here: um commit graphs, uh backfill, reverse index index, bitmaps all the different housekeeping things that help keep you efficient, uh and this is a file that guildley creates to say Hey when's, the last time we did a full repack, so we can keep.

G

Track of that as well.

C

uh As I go here, I think I see yeah. Okay, let's go here.

S

S

K

Yeah sorry, okay,.

C

Okay, so um just to explain a little bit more, so this host here get a triangularly. Two is then a giddly cluster, um and so prefect will be.

T

Use this at cluster.

R

um Format for its paths well,.

C

This is a shirt again so Standalone giddly, which is what almost all of our Ghibli nodes in.com are, and what you'd see in a typical small self-managed instance. As.

R

C

And so here we use the add hash um format for the path, and so this is. This is the path that you'll see internally in rails as well. So if we.

B

Go to projects.

C

And so like a look here and I can say: okay, this is the path that rails has for the project and this matches up with what we have on The Shard. However, if you look at a product, that's in the cluster it'll show up as hashed, but there is no path at hash 66p86 in in the on the file system here, so we have to do.

U

V

Let's go into the.

C

Prefect, which I have mentioned I'll, get back to that in a second um database to map that between them, um so actually before I go any further. Let me actually talk about what is gitly cluster, so gitly cluster is a way of having multiple copies of a repository that are, you know, kept in sync with each other without uh well, exposing only a single instance to rail.

C

So it's a way of getting redundancy, which is not something that we have with the shared goodly there's no way to replicate a repository to multiple different shards in the current setup.

C

So what that does is typically you'll have a little Bouncer and that'll be the address that you give to rails. uh Let me show you here so like, for example, here this could be my load balancer. In actual fact, I only have a single prefect node in production. You typically have three, maybe more um one. One failure mode is seen somewhat often with sub managed. Customers is, if you have more than more giddly nodes behind prefecting you have prefect, they can send so much traffic that it overwhelms the network, connection um or network bandwidth.

C

That is available to the prefect node and that will cause silent slowdowns. It used to be it used to cause memory blow down perfect, but that's been fixed.

G

Is it like fifteen five, or something like that.

C

um Anyway, so perfect nodes, what they they're stateless, they have a they have a postgres behind them and that tracks all the repositories in a state of all the repositories, and you know which repository is going to which node. So if we look at the.

G

W

C

uh Let me make that bigger, so we've got right now: we've got four repositories, two of them being one keys to them being actual projects, and so you can see this is the path that rails knows.

C

um So, if we go back here, let's add hashtags v86 Etc. um But if you look here, it actually maps to at cluster repositories, d4732 um and you can see giddly one is the primary node for that which would make gitly two being one of the secondaries. I only have two giddly nodes in this cluster, typically negative. Three, it's two is fine.

C

If a customer only wants to there's no need for Quorum or anything like that, um and this generation counter saying like how many changes have happened, so this is how we keep track, of which node is in which state. So if the generation counter is the same. So if we look at storage repositories, so you can see here they both have the same generation. So in this case a read request for this repository could be read to either gility1 or get delete two because they're both at the same generation.

C

So this lets us an active repositories, in particular, distribute reads among multiple nodes and decrease the load on the primary. Any right will always go to the primary and one issue that some customers are seeing right now in general is that if a right cannot be simultaneously proxied to the secondary and they don't they don't agree on it, for whatever reason so like the secondary fails for an update for whatever reason, then the secondary falls behind.

C

It has to replicate to basically do a fetch from the primary um if it's a very large repository that fetch might take 30 seconds in which case, and if changes are coming in faster than that, the secondary might not catch up to the primary for hours, and so in that case we wouldn't be able to distribute reads: We Do, It, All rights, or all reasons have to go to the primary along with any rights and so you're losing most of the benefit other than you know.

C

Having redundancy in terms of you know the primary hard drive failing um in this scenario, you can't you can't spread the load, so that's a failure mode that we see somewhat often it's been improved. Most I think pretty much. All RPS at this point won't do that for a while, there were some that would always trigger a replication and that uh you know that just decreases the efficiency.

C

um Okay. So that's! This is kind of internally how prefect nodes are tracking. You know like how can I route a request. So basically the request comes a little balancer low balancer chooses any of the stateless prefix nodes, prefect, we'll check the database say: okay, I can send this request for repository 6b86 to either get a 1 or giddly 2, because the generation is three it'll route it um and then it'll so it'll proxy.

C

The grpc request down to giddly one of the giddly nodes and then giliwoo do its thing: send it back out to prefect which proxy it adds a little bounce, we'll send it back to the client and so on. So there are a lot of steps there. um uh So how does giddly actually get get data so I'll show that real, quick.

Y

So it's s tracer.

Z

uh Actually now, let's do this chart, it makes more sense.

AA

M

Q

AB

Everything I'm gonna, say it makes sense.

AC

Y

AD

O

C

So disability of this page will have triggered some yeah.

K

B

C

Right, uh let's actually cut that ahead.

C

It's kind of hard to read: isn't it sorry um anyway? So let me let me walk through what's happening here, so so the rail is going to requests to pull up a project. So it's going to send a few rpcs so like. If you look here and see it's sent search files by name in this case. We are looking for itself.

C

um And then it's looking for markdown files.

G

Then the wiki repository you can see the reg X that we're sending.

C

And then, in addition, we're also to.

G

C

The wiki that's interesting, that's fine um has local branches. So it's saying hey what branches do you have um to this Repository?

C

um And so here you can see this is the the um the file that's actually being forked. So it's it's not opt embedded in Optical have embed and Bin git um getting Italy when it launches will extract its own git reposit, um don't copy of.

H

The get binary so, let's actually just.

C

Expand that directory, so here you can see everything. That's there, so you've got get to go, which, as I said, should be gone at the next release, um and then you have Kili gpg. We use that to sign and validate uh signatures on commits hooks is used for well, it's used for the pack objects. Cache is kind of one of the primary things which I'll get into in a little bit.

C

um Lfs smudge is used for handling lfs objects and that's kind of a long layer process that we use to make it more efficient and then give the SSH is used to connect internally between guildley nodes, and then we also have look Hooks.

Z

And socket directory in there as well.

C

um Okay and so sorry so I just started printing, it's confusing all right. So then we've got this guitar directory, saying hey! You want git operate in this directory here and we always say: Don't run get GC automatically, which, by default git, will join GC on some operations, just as like a standard housekeeping but cataly controls housekeeping. So we don't want that.

C

um Remember, saying: treat controlled F to whatever you're receiving or crlf the line ending, don't use, replace refs, which has been a security issue. These are the objects, that'll be fsync, so objects metadata references or fsynced. By get uh whenever we make a change there and then fsync is fsync uh and this I believe has to do with reducing contention on the Pax rest file.

C

uh Okay and so then here's the actual commander running, which is get cat files. This is saying, give me a blob um and then we're running Z, which will give it like null uh null Bank terminated um and then we're saying patch command, which is saying we're going to send you a number of commands every time. So I don't see if we.

Z

Have any get file processes right now.

C

And we don't um so what gittley does when were these get for these Ruby or rails operations that are often making lots of little requests so, like I, want to look at this file? I want to look at that file um within the context of a given uh rails request, so I think it works out to a single correlation ID.

C

um We will Fork a cat file process and then keep that around, so that we're not for each file that we're trying to view we're not forking a new cat file process. This lets us be a little more efficient um or fewer processes.

C

So when you, when you look at a production, server, you'll probably see a whole lot of cat file processes around those are mostly just their hashed waiting to see if any more requests come in or or that given request for that for that given overall, like web.

AE

Request, that's coming in.

C

So that's that's. Why they're there and then the other thing you'll see a lot will probably be uh upload pack or.

G

C

Objects related to clones. Those will be long-lived typically for a larger clone.

AE

um So there'll be around more.

AG

C

Anything else I want to talk about. Well, let me pause right here, first, um anything that any questions that are like immediately relevant that anyone wants to ask.

AF

I have a question just yeah: what's the logic behind creating its own work of the git.

AG

um Executable is there any reason for it or.

C

Let's clearly have more control over what git process or what get uh binary we're running um I. Don't remember the reason that it was difficult to keep it in sync with Omnibus, but this way we didn't have to rely on Omnibus updating what it bundled we could always. We could very you know, control exactly what this is, what we were executing and clients or you admins- can still override the get binary. So I don't believe you have to use this. That's not something we recommend, but it is something that you may encounter in the wild.

C

If you're dealing with a self-managed customer.

A

Yeah, another reason is sometimes: if there's like a security fix or some high priority fix, um we will build a git binary with some of our own changes.

A

um Usually we ship it's the standard, git binary, that's released, but sometimes we'll ship with some of our own changes so um yeah. If self-managed customers use their own git binary, it might break because it doesn't have the features that um Italy needs.

AH

We don't ship the security patches to our customers before they're released.

AI

Yeah, that's right.

AJ

C

Yeah, but one other thing I can talk about within the connective. Gearly cluster is failing over between primaries. So if I go back up here,.

C

uh So right now we have for this repository Italy one. uh Its primary is going to be one.

K

um So that is the skilly. No, that's not that one let's go back uh is.

L

AK

Yeah this one okay.

X

So if I go to giddly one.

H

You stop giddly.

C

And then I send a request to view the project we should fail over at this point, just to get a lead to.

K

AL

K

I need to go in actually a file.

M

E

What am I doing wrong.

C

Can't, oh well I wonder if we need to make a.

Z

Push because it might just be routing the reads there. Let me actually try modifying something.

AM

Subscribe, your file.

C

All right, okay, I just couldn't test that I need to update. This is a config problem. It's not right writing out to.

AE

The right address for the internal.

M

Api all right well.

Z

Let's make that work, let me tested this beforehand. Oh.

C

Well, um no actually I was just reading it wrong. It did update okay.

A

Probably just took a second yeah.

C

Maybe maybe it already changed, I didn't even look right, that's probably it anyway. So you can see now for this repository only. We have now updated the primary to begin League two. So the way failover works is when a request comes in that the primary needs to handle and the primary is not available. Then we will update the primary to be one of the other in-sync nodes that is on the gildling cluster.

C

If there are none, then the repository becomes will become unavailable until the a up-to-date node returns, um but we only do that for repositories that come in the have requests come in so you'll note that this other repository has not updated. Even though giddly one is offline. Originally, we did update all of the repositories and that caused a lot of work in postgres, and it was just it was. You didn't have to do it that way right so now it only it lazily updates the primary as needed. uh So.

Z

T

One thing that might be relevant.

B

Q

C

And then I'll just quickly mention um you may see, deadline exceeded errors in the logs. That means that some operation that rails, typically until it's always going to rails, are sidekick um sent took longer to regularly to serve than the timeout configured in the admin area. So.

H

If we go back again and leave page.

C

Preferences, giddly timeouts, so whatever these values are set and whatever the operation is to develop, the assignment will depend on the uh evaporation that you're doing, but if it takes particularly if it's a large repository and it takes longer than this, then you'll see a deadline exceeded the other area, you'll see a lot is canceled the context canceled. That means typically that there's somewhere between giddly and the client, the connection went away. Maybe the client closed their git process. Maybe you know, load balancer went out.

C

It could have been anywhere along that chain if the connection is lost and you'll see that connection error, contents, canceled, uh error, message or link Italy, so basically just means that the client went away for some reason. We don't really know why. um Just you should look Upstream to try and figure out why that happened.

B

C

I think that covers everything I wanted to go over. uh Let me stop sharing for a quick second I'll.

G

Pull up the dock.

A

Thanks will so um yeah I know that that was a pretty technical. So if you have any questions or if you want will to go over, maybe one of the parts that he already went over um uh again, uh maybe yeah. So um please uh put those in the questions and um we'll go over them one by one.

B

C

After asked why we use our own, which I think we covered um manual? uh Oh so so the pooling feature.

C

Yeah Karthik has answered that so so we're using a built-in git feature called alternates. So basically you can say here's a repository on disk and then you'll have an alternate file. Then it'll say: okay and I'll also go to this other directory and use that to get additional objects from it. So we're saying all of you all of you pool reposit all your projects that are using this pull repository.

C

We add that alternate of the pool to them and so then get itself just knows: okay, I need to go to this other directory, which is just a relative path to um to fetch the objects so there's limitation, which is that, because it's a relative path, you know it has to be on the same node like we can't sell git uh like using HTTP request or something to go, do it so that limits any any four cluster to a single giddly node with a negatively cluster.

C

um In theory, you still have the forks spread out or replicated amongst all the nodes that was broken for a long time. I think Justin has gotten that basically fixed at this point or close to it yeah.

C

um So it should start working soon if it doesn't work now for a while, we weren't keeping those in sync accurately. um Does it answer your question at all.

C

Maybe he left okay uh Steve asked. Is there a Civic reason why I put things in the cluster directory as a.

H

C

Are we adding extra overhead of translating the path through uh prefect, um so I think the main answer there is that we wanted to have a a permanent home for a given Repository and but rails was still asking us to move repositories around on disk now, particularly in the context of Geo. So you'd have like a new repository created and G at Geo temporary and then once the fetch hit geofetch and completed, then it would get moved into place.

C

um So the actually, the reason that we have the add hash name original in Italy or on on giddly shards, is the same logic. Originally. The path on disk was the repository names so like including the project so it'd be like fire. Advocate lab, git repositories, repositories, um getlab, Dash, org get lab would have been the path forget that. But um whenever you had someone you know rename their project or rename their organization, they ended up doing a ton of work on disk and just rename all these directories and it was expensive and error prone.

C

So that's why I created that hashed format, and so then this cluster thing is doing the same thing um just at another level of abstraction. Does this way kill the cluster knows that a repository is always at this path, even if rails thinks that it's moved somewhere else. So that's that's why we added that.

AN

um It does add another layer of complication. Cluster is always the same. No matter what you get delay shark, to talk to correct.

C

Okay yeah. So if you look at.

C

C

uh One yeah, sorry that was actually over here. So if you look at the path, so it's actually the hash of the ID, but instead of taking the full hash here like we do with Ed hash, we just put the actual number here. So this is the full hash of number one. This is also hash. One just use one at the end, um but they don't they. Don't always. You know correspond with the ID that uh rails has like this in rails is Project one and cluster it's project or sorry.

C

This is Project one and cluster and project two years project, one in rails, project two in Gilly cluster. So one issue that we see is that, should the prefect database be lost, there's no convenient way to recreate the association between the path that rails has and the path that the cluster had.

C

So there's yeah, there's just no. Unless you're backing up your database, which I don't think we really advised in the docs there's just no way to re-establish that you can. You can kind of fake it if you can go through the logs for so for any traffic, any project that was creating or getting traffic. You could at least look at recent logs and say: okay I can translate between those two because that'll have both, but um if it's like an idle project, then you're, probably not gonna, be able to easily do it. There is.

C

There is one thing you can try.

W

uh If we go into oops.

C

It's looking like a fig. Actually, this one doesn't have it okay. Well, there we go yeah.

AN

So sometimes yeah.

C

Yeah for a long time, I used to write there used to be a gitlab section in this config in the git config. That would list the the project name, so you could relate it back that way. um I think we added that, like 10 version, 10 could get that 10., um but it's that's going away now, so you can't rely on.

G

C

C

uh Okay, Sean asks so some customers, you can get a independent of gitlab hugging like clogging face I, think is certainly the most prominent. There might be others that are doing it. That I'm, not aware of um so. This, isn't something that we officially support um in terms of like a support team's not going to help them. There's most of the logic in Italy is pretty agnostic there. There is some stuff that is get elastic. You know, like authorizations, I think are hard-coded to go to internal aloud. For example, we do have documentation.

C

I got my prayer, I think it's yeah this. So it's for this Lincoln.

G

A

Yeah and if we're the documentation for RPC or rpcs it's uh there is um good amount, that's missing and that's something that we're working to address. Yeah.

C

This is all just automatically generated from the comments on our protobuf definitions, so it may or may not be entirely useful understanding how it works, but that it is something that's out there, and this is. This is generated automatically if it's part of the CI pipeline, so it'll it'll be up to.

K

P

Yeah, so we do treat Italy as a product, but it's an internal product that we don't currently Market externally hugging face is an exception. They know what they are doing. They are super technical. They understand exactly how git and gitlab and giddly works.

P

They just want to use their own front-end, so they are able to do this, but this is not something we would recommend at this point, however, what we want where we want to be, is to offer a clean, RPC interface with all the implementation, details and internal logic, some of which is currently managed in trails to be moved to gidly, for example, how object pools are created and deleted uh is is kind of a detail that doesn't belong in a client, whereas we want to keep all the business decisions, things that need access to the main postgres database, like customer, tier or customer settings in the repository or or in in their projects.

P

You want to keep that on the real side, so that giddly can be agnostic and just be told what it needs to do.

AO

Make make sense. Thank you William, and address very, very interesting that we are even thinking about having it as a separate component.

P

It may never be marketed outside this is uh this is a product decision, but it does help us to be more independent and develop uh the overall of gitlab yeah with uh less friction. If we have a clean interface between components.

AO

Of course, thank you in.

AP

The briefly mentioned from the product perspective, this is also really important because we formed some very strategic Partnerships through this sort of opportunity. uh We get a lot of contributions back from external people who are you know, interested in giddly hugging face is a great example, but there are others as well that we've seen over the time where we've gotten contributions from other.

AP

You know teams sort of looking at it and it gives us another perspective from a product perspective, and it does ensure that we are continuing to develop giddly in the most independent way, which is part of the reason why we are, as the direction page shows, trying to put the business logic into the rail side of things to make giddly somewhat um agnostic to who is calling in so yeah. It's been an overall win for us, but, as honors has said, we're not actively marketing or pushing it as an independent product.

AP

Just yet, though, that may be something we look at doing in the future.

C

All right thanks everyone, uh so next question from Clement um interested in the cost of serving uh cost of service. What are the most costly operations so I know on.com using fast Storage. Snc storage has been the largest largest cost component of daily. By far, um it really depends on the instance, though so you know.com has is getting. You know, lots and lots and lots of repositories. You know business, you know like a self-managed instance.

C

You might not see as many you might see like a single monitor repository, um which will still be large, but you know not not the scale of you.

G

Know 100 terabytes that you see.com for the you know the.

C

Vast number of repositories there so typically storage is a big expense. Just because you for Reliable and performance service you need to have fast storage um and then, depending on the size of the repository, you may also see significant expenses in terms of compute. So, if you have, you know a truly vast repository, let's say 20 gigs it can easily use 10 or 20 gigs or operation, sometimes depending what it is.

C

So if you're, if you're running multiple operations at once, you know- and you only got 10 gigs of RAM on the box, you're gonna run out real fast right, so you're gonna have to have a really large Post in that case, just a serve the repository of that size, but more typically, storage is larger. Cost.

C

Yeah goodly cluster, you know I would say that you'd expect to spec that are the same as a standalone, giddly node, because you know internally we're still retrieving that as a single node to the world right. So I don't think you'd want to skimp because you're still writing the same operations on a given node, just maybe fewer of them, and any of them could become the primary at any point right.

C

So you can't say like well this one's my primary so I'll upstick that and leave the other ones a little bit wimpier because we could fail over. You know anything we'll let.

C

Skill with usage, yeah I mean so typically, the vast majority of traffic to gitly is going to be CI traffic. So you know a Sierra Runner, fetching or cloning um humans are almost always like. You know, 10 or 20 of traffic from.

Z

What I've seen at.

C

Least so you know getting that as efficient as possible. Is the number one priority, so that is why we added the pack objects cache a while back. That was also from the scalability team. uh That's listed down.

G

C

So basically, when a fetch comes in or a clone, um we will generate, the git will generate the pack file, it's going to set up the client and then this we're using the gidly hooks binary. We'll then write that pack to disk. So if an identical request comes in, we don't need to go to git to recompute that so we save all the CPU power that would have been used to create another pack file. I can just copy the bytes from disk serve it out to the client.

C

So that saves us a lot of CPU time, but it does add additional pressure onto the the storage because now we're writing all these packs that are coming into disk, which typically they'll just be a memory and stream Bank out to the client.

C

um So it's usually a win, usually you're, more CPU bound than storage bound. If you have a whole lot of clones coming in at once, um but it.

G

Depends on the the instance of the pet traffic pattern so.

C

Something to look into in general, but maybe not always a win.

B

C

I think that covers that um Terry asks any epics or issues related to sync operation to clarific Italy, cluster replication um yeah. So so I mentioned this in passing earlier, we are moving Italy in the long run to a new architecture which will remove the need for prefect.

C

So that's going to be on each giddly node we're going to have it right ahead, log, basically like what postgres uses or other databases um where a change comes in. We, you know, save its changes to disk, and then we apply them one after the other, if you know serialize them and replicate the changes out to any other nodes. That also happen to have this. This given Repository and then the client will be so in giddly will be routing clients without prefect. uh That's pretty well sent to the blueprint for this I.

C

Remember the detail: that's going to work, I think there's going to be a writing table that get a labels served to clients and then some clients will know where they can go and we'll have to be able to like retry to a different Gilly server. If that giddly node goes away um so there's gonna be a lot more complexity in how giddly talks to clients when this is done, you know right now. It's basically rails has an address. It goes the address.

C

If there's a problem, if you return an error, that's it there's no retry logic! That's that's that! So it's gonna, the client will have to do a lot more work whenever that comes around, but that's uh that's still a ways out foreign.

C

Sorry Terry um but yeah, so we're not we're not putting a a ton of more effort into prefecting and we're still polishing it up, but uh so like, like I mentioned, you know, like other PCS, that previously forced a replication job. We've got that down to like three.

G

I think at this point or something but uh we're.

AQ

Not yeah I was just being um I was just being nosy I've seen that issue, uh that was epic that was linked. Thank you. I didn't get a chance to read through it all the way, but we are working on uh zuk's replication strategy, so I was just curious, as for uh you know, at least it's good not great, to hear the problems that you're encountering, but at least something to think about when we're looking at how the replication is going to work on the zoo side.

C

Yeah I'd be very interested to see uh what your team arrives at.

B

C

uh I think that's we're at time. That's all the questions in the dock um I'm happy to hang around for another couple minutes. If anyone has anything else, otherwise we can call it.

A

Cool, so thank you so much will for um that session.

A

um So um hope that was helpful to people um so just wanted to announce that in two weeks, we'll be doing our next session and the topic will be diving into get internals um or yeah diving in to get how git internals help debug Gilly issues, so that one should be really interesting.

A

um You can take a look at that issue. That's linked there for upcoming sessions um yeah by the way I haven't sent out. I haven't set up the calendar uh event, yet so I'll do that shortly and send it out on slack, um but that is it. Thank you. Everyone for joining have a good day. Bye, bye.

E

Thanks everyone.

AG

Thank you. Everyone thanks will.