GitLab Gitaly group, 21 Feb 2019

Previous Meeting

⏯

youtube image

►

From YouTube: How Gitaly fits into GitLab: Episode 1 – Gitaly client

Description

A 1-hour training video for contributors new to GitLab and Gitaly.

Overview of GitLab backend processes, gitlab-rails deep dive: Gitaly config in gitlab-rails, SQL data model, overview of how Gitaly calls get made via GitalyClient.call.

Recorded 2019-02-21

A

Plans, if you see me type that so what I've got running here, what do we have here?

A

This is a terminal window where I'm in the get early projects deep down in github development kits, and this nested nester directory and up here, I have a window where I'm in the kid lab subdirectory and over here I have a GD k, run process running and I wanted to start by looking at GD k run at that process tree and see what's in there, because it gives us a sort of overview of the backend of gitlab and what the different parts are.

A

So let me look up the bid of that process. So now I need to use you so the bid of four mana stats and five PS tree. That's then we get this. This is readable, I'm, gonna, yep,.

B

A

Okay, I'm gonna assume yes, so here's a process tree of everything that GDK spawns when running his lab and actually there is something missing we'll get to that in a moment. So, what's going on here well, this obviously.

A

Is Postgres actually this whole bit? So let's separate out that and then here's giddily, we sort of know what that is, here's workhorse! That is a thing that stands on its own and there's sidekick here.

A

This note process is unique to development, because in development we have life, we have web back, doing something with life recompilation of the JavaScript assets and, as you don't see, this in production and I won't be talking about that at all, because the github JavaScript front enters and talk to katella directly, so I think we can ignore it in this. In this discussion, then we have unicorn and unicorn sort of belongs with sidekicks I'm, going to move it up here and then down there. We have get lap pages and these things ignore what this is.

A

These are placeholders for docker registry and object storage, which are not in able to dedicate by default, and up here we have read, as maybe I should move that little closer to both gross.

A

So reddish Postgres Italy workhorse and then this is saying this is a real slant and this is database lands.

A

And then then, there's workhorse and what's missing in this picture and was going to other fronts and not let's just call that front ends, but there's workhorse and missing. This is HD, which is the SSH daemon which you run into when you do get clone or get push or get pull over SSH.

A

These two are interesting, but and these two are getting clients but I propose. We don't go into these because they're, just added complexity and most of the interesting client stuff happens over here in the rails, part of of get lap any questions so far as this picture making sense, yeah.

B

Good luck, she'll is that part of the rails app or as I've seen that wait.

A

A

Spawns so sshd spawns one of gitlab shell processes.

A

So you would never see it lap, shell in a process tree, because the processes are one-offs, but they get spawned by.

C

A

And so one way to think of good luck. Shell is that it's an adapter between sshd and the rest of cattle, app I, see.

B

A

Another thing that's missing here is nginx, although it you can mostly ignore the role it plays in get slap so yeah. Let me talk about the HTTP stuff. Unicorn is a ruby HTTP server? It's a wreck server. This is a standard for Ruby web apps and it is the its pluggable.

A

We could be using something else that unicorn but unicorn happens to be what we use in kids, lap and we've been using it for so long that we tend to confuse the rails web server processes with unicorn, because unicorn happens to be the name of the Pacific, the actual process that hosts the rails application but yeah. In some ways the architecture of unicorn is relevant to what we're doing, but most of the time it's something you can ignore, we're doing experiments to replace unicorn with Puma, which is another Ruby web application.

A

Server I, don't know what the stages of those experiments is. That's a separate topic, but yeah I wanted to point this out, because really this should be you should think of this as web, and this is background jobs, but the actual programs we used to do that. They're called unicorn and sidekick, so people tend to talk about unicorn and sidekick around here all the time, but those are I, guess their brand names and does not take the actual generic purpose.

A

It would also be possible for us to replace sidekick, but there are no plans to it. They would do that that I'm, aware of yeah. So one more word about workhorse workhorse is a weird concept. That's a bit unique to Kate lab. It is a reverse proxy that sits in front of unicorn, so it's actually tightly coupled to unicorn and every request that reaches unicorn. Almost every request goes through workhorse, first and workhorse modifies its or does its own little stuff with the request.

A

If it feels like it's and otherwise it just passes it through as a reverse proxy back to you know to the web, the Ruby the rails back ends and then also the response passes back through our course. So there's complex interactions going on between this and this that are a later topic, because gisli actually has a role in this course.

B

Will directly talk to into liens on cases? Yes,.

A

Exactly but workers doesn't know anything about where gates lis is so every time it talks to Gately. It has first talk to unicorn to hear what kill a server to talk to and what to do and and these interaction flows are you're a continued contributor at some point, you're going to run into this, and it's our job to know how this stuff works, but we're going to ignore that for now and go back and focus on the rails app.

A

A

For whatever reasons, I I don't really know how to explain this if you're new to Rails, but these things live out at the same code base. So the kid laps EE repository, git lab org, slash get FCE is the rails code base, plus the front-end code base, and but so in there is a big Ruby application with lots of files, and both of these things are processes that load that entire application into memory, and so they share a lot of codes both of these directly connects to Redis and to Postgres.

A

So all that stuff is clustered in in what we call the rails app.

A

Okay, I'm making this up as I go along. I haven't planned this. Do we want, do you wanna? Do you have any more questions about this high level picture or shall we start looking at the rails code base, I.

C

Think we're gonna move forward. Yeah Emma.

B

A

So this is my window with the rails code base. So how does this talk to grittily?

A

Now, there's two ways to look at it. We can look at the configuration data or we can look at the actual gotelli clients. You want to start with the configuration or with the clients.

C

Configuration okay.

A

Cool so the main configuration almost all configuration of this rails application is in a single yellow file in the config subdirectory, and that is this kit left a yellow file, it's big and actually a lot of configuration. These days is in the database, because this yellow file is stored on this coin. Your application server and the real process is not allowed to edit this file, so I guess nowadays this is more sort of bootstrap configuration, so this is stuff that the application needs to know before it can even boot up.

A

But after that we try to store as much application as we can in the database in the admin control panel of the web application. In case you haven't seen that I have a I have to have running here. So if I sign-in.

A

You it's got to be a little slow as I have an old laptop when this is development mode.

A

It's probably so for everybody, no matter how fast a laptop.

A

A

I want this thing to I, want to show you the admin panel. Just so you know what that is, but because.

A

The other way to think about the different types of config is that some config will be constants.

A

The config, that's in the admin panel is stored in sequel that will be included in a backup and when you restore from a backup you might be in a slightly different environment that looks different, so any config that depends that you can reasonably expect to be different after a restore should not be in that main panel, because otherwise you can end up with a catch-22 where you can't get can't boot the app because it is making assumptions about its environment that are false on your new hit lab server.

A

So, let's see where's that main panel. That's this one.

A

And this thing has okay a little faster. This is a settings area.

A

Where there's some stuff, we can configure with general.

A

Funny way of general and we have preferences- oh, this is general repository I. Don't understand why these buttons are different from Oh general is broken down into these areas. So here is stuff about repository checks and housekeeping. This is actually relevant to get to me and somewhere. Maybe it's in general I. Are there greatly timeouts the default timeouts that can apply to get Li requests made from from within a web context?

A

So this doesn't sound.

A

Account and limits there's no dates.

A

This sounds like it's, so it's preferences.

A

Oh preferences Italy: here we go here, we have a little bit of conflict and this is stored in the database and I yeah. I. Think the way to look at this is that if we can get away with storing it in the database, we do that nowadays, just because it makes it so much easier to apply to conflict changes on github.com, because otherwise you need to change edit, the config file on disk on every application, server and restart the application, which is a major hassle in the case of kids. Little.

B

A

Go ahead. Sorry.

B

Okay, so it can things like this: it doesn't just get loaded during startup.

A

This gets I think this gets done and loaded once and I think this gets loaded during the requests so.

C

The the timeouts here it means that the caller is gonna, cancel the request within that timeout. It doesn't mean that giddily is gonna, terminate the the requesting it senses that those timeouts are exceeded.

A

Actually, both the RPC allows you to set deadlines on calls and do if I've never checked this, but the way I understand it. Those deadlines can then be enforced by either the clients or the server wickedly does not set its own deadlines IRA's by default. There are no deadlines and Gately, and if there is a good deadline, it's only because the client asks for it. So we have a fairly trusting permissive thing going around there: okay yeah, to go back to your question. John I think we reload config from the database on each request.

A

So if, if you make a change, then it should probably start making having an impact from all requests after the change is committed to sequel, ok, I think I'm, not super sure about this, because I'm not active in this app part of the epilogue. So this is one part of config, but for as far as quickly concerned, this is kind of small.

A

Although there is one important part that we can maybe see. If we go to the projects, we can use this as a sort of format, sequel browser, here's a project.

A

When there's a thing called Disney servers here, we've got to check that out next gisli storage name. So this thing is a corresponds directly to a field, just a text field in the database, so each project records then there's a column called storage, name or storage.

A

No, it's called repository storage and that has a string in there in this case called default, and this somehow will get mapped to a Catelli server, which is maybe which I think is one of the things you wanted to know about paul yeah, yeah, so we'll we'll get to that or I want to get to that. While we're talking at this yeah.

A

So each this is an important part of configures far as quickly is concerned, every repository belongs to our projects. This actually goes as far as that repositories used to or still don't really exist in the sequel database anywhere. We just have records of projects and implicitly each project has repository attached to it.

A

So looking up a repository starts by looking up a project and saying project on Tripoli Tory, and then you get your repository and every project actually has two repositories, because it is a normal repository and a wiki repository and these things in the data model. They just hang off projects which is a I, don't know, I, guess that that this happens we hear data model and we look back.

A

Maybe it doesn't make sense, but that's hot, that's how it happened in what it looks like now and I think there are some efforts to actually make repositories. First-Class citizens in the database, but I, don't know how far those are.

A

A

And you can do a transfer repository check, we're not gonna. Look at that now. Let's look at the Grizzly server see what that does.

A

So this is actually the mapping you were. uh We were talking about, pool where I want to know how does the storage name relates to a network address, and this page is generated from data in the config file, so the this is actually data that is written on disk on each grid, lab application server, there's a config file that needs to have this same mapping. If we would have more than one Goodley server.

A

There would be a list here, but by minimal setup, as only one and we mandate that there has to be at least one gisli server, where the storage names default or the application won't boot. So you're likely to see this default thing, one some one place or another.

A

Another thing worth noting is that this thing is on a UNIX socket. That is because UNIX sockets have some some nice properties for this stuff. You don't have to worry about port clashes and it's easy to restrict access using directory permissions, so that only I think this one is not very restrictive, because I suspect this directory is open, but in production in a standard, single server production installation.

A

This socket would be in a directory that has mode seven hundreds and that nobody, except the application user can see its contents and hence no one can access access to sockets. So.

C

Yeah, if someone adds another giddily server address to the configuration file and reloads the server, does it just start, round-robin round-robin rotating between the the get early servers for new repositories, or does it balance the existing repositories? What happens when you add good.

A

Question I think default is that we get into Community Edition for the Enterprise Edition territory, I. Think and good luck, Community Edition, nothing happens. There is so maybe I should have booted Enterprise Edition I. Don't want to do that now, because that we're sitting here waiting for five minutes for the whole thing to boot, but in Enterprise Edition you can designate which repository servers which storages get used to create new repos, and this is a very, very bare-bones mechanism that just does just enough for what we need on github.com and nothing more this. What happens?

A

Don't get out this comb? Is we notice that we were running out of disk on our existing Gately servers? We instantiate a couple new ones, and then we configure the admin panel to say all new repositories will be round robins on one of these three new ones and then let one point to dashboard says: oh, our disk is full on these three new ones, and then we start over again. I think this is in the docs.

A

Just look for moments.

A

A

You yeah so this this little panel is rice edition and they said in the admin panel somewhere, but not in Community Edition and there you can click it. It's a it's a select box where you can click multiple entries and then all new reporters get round robins over those new servers.

B

So so storage is synonymous with get giddily server. Almost almost okay, like.

C

B

Wanted to could I just like have several giddily servers run on the same like physical storage and then the idli server just has its own namespace, for where it stores things.

A

So, historically, a storage was just a mount points, so a subdirectory somewhere on the gate, lap server and because we want its wifely, you see it exactly this. This is outdated, but this picture sort of ignore what get data ders is that's from only bus, but things would have looked like this. Perhaps in the old situation and when we introduced Italy, we wanted to have an option where we say we run grittily close to the app on the same server as the rails application, and we don't worry about these mount points.

A

So then one Gately process would have to serve all these three things. At the same time, so with indigenous config file, you can define multiple storages that it serves, I, say I, it's this I, don't know how to explain this. Clearly, it's it's too much flexibility for our own Goods to be honest, but we were sort of ended up here because of the way we had to transition from the existing situation.

A

B

See so in the in the repository model, there's like the storage field yeah. So does that map to the earlier page, where, like storage, shootin, absolutely server or the Gimli servers storage? Let.

A

Me show you exerting edit maps yeah you find in the config file I think if I search for storage I'll find it eventually, please.

A

Yes, here we go I, don't know why this is schools advanced, but this is where it's it's defined. So there's a subsection repositories storages, and this is an array of hashes. No, it isn't hash. This is hash. Key is it. This is llamo I have to think for a moment about what it means. This is a hash called storages and it has entries- and this one there's only one- there will be another one. It would look like this.

A

This path field is legacy stuff, so that's for the moment just delete it because you don't actually need it.

A

So this is all you really need a storage in server name, and these are the fields that gets stored in the projects table on that column.

A

So if at runtime you would load the projects and it has a value in that column, that is not known from the config file, that will be a runtime error and if you only find out about when you hits that record, because there's no way for us to validates when the app starts up, that all projects in the database and valid storage the again- and this thing has to have at least half a goodly address- which in this case is UNIX sockets, and it is valid for multiple of these entries with different names to be behind the same address but then within giddily itself.

A

Okay, let's maybe contrast that so in grittily itself also.

B

To storage in is, they can actually point to the same same servers.

A

Yeah same same grizzly process on the same grizzly server. Now, if you look at example, get in the config file, this is his own definition of storages, and this is a map from so. Let's see, let's take this example, so we are default and then storage to so then I would define a second one here. I.

B

A

And I think the way things work now and Italy is that we yell at the user or we yell at we abort a replication boot time. If these things are not in it,.

B

A

Think maybe it's a log I, don't remember exactly and and if you look at the picture again, you can sort of see what happened here right, because this was the old situation named bath in get labs, config and now in get labs config. It is named Italy address and then in give me config, we say name bath again right. This.

B

Is actually how we go ahead? Yeah so I was gonna. Ask in the case where, if one giddily server has to like do something in the repository on the separate like honest on the different storage and that storage doesn't live locally, then it has to make a network call that they kind of like figure that out uh these.

A

Files will have to be local on the Grizzly server.

B

Oh, they do okay, okay, yeah.

A

I mean if they're NFS they're, not strictly local but from an application point of view. These are local file system paths.

B

But what I mean is like what, if, in this situation, okay, so with this storage, yeah they're, um the giggly address was something else: yeah I'm gonna do.

A

That now so, let's say.

A

Supposed to be at this right, this implies that there is a second Kootenai process running with its own unique config file. So the first Gately process would have this config file and then the second one would have its complete file. It would have this no come on this with name storage, three, and this can be anything right.

B

Right, so if this giggly server like, for instance, it well it needed to do a clone from a different storage, then.

B

A

Without making that work, all it will make different.

B

Gillies yeah.

A

Yeah, it is one of the assumptions of Italy that each grisly process is quickly. Server can make Network calls to all other gisli servers, Spain right, they have be arranged in such a way.

A

Otherwise, nothing works, for instance, when you make a fork of a repository right now, that is a naive, full clone and because we don't know where the other repository is we just default to making a network call to whatever the quickly server hosts the original rain, we're into a clone like that's, okay and this yeah exactly how this network hole works is very interesting, but we should probably not go into it now. Yes, and unless you both want to, but I think.

B

A

Can move on yeah I think we can focus there's more to explore on how stuff works here both regarding.

C

The and we're using UNIX addresses here, for you, know a local socket a file. So what happens when the Gidley server is on a different machine? Are we forwarding that to a TCP address, or do we replace this config file with a TCP address, yeah.

A

We write different addresses here. Let me go back to the docks.

A

You, oh here's some stuff about network architecture and the storages by the way. So this repeats some of this stuff I was just talking about and let's see, addresses.

A

Okay, this is not very clear, but we we allow addresses that look like eunuchs call on something and we have defined this custom made up scheme where you can say TCP colon, slash, slash, and then you get an unencrypted its well. It really is it's a JDBC connection, so that's HTTP slash, but it's an unencrypted network connection and since 11.8 you can also use TLS colon, slash, slash and then it will use encryption and I think the way things are currently deployed on github.com is that the machines are all firewalls and they use unencrypted variants.

A

There is also a token mechanism where each server may have a token. You can also see that here in the example config file, that's you can set a token Burkett Lee server and at the beginning of each RPC call we transmit the token to prove the client proofs, transmits the token to prove that it's the client that it has access. It's very basic authentication mechanism.

A

So sort of the route of looking at how things work coming in from the convict, so we've now touched on the sequel model. How repositories are stored in the database? Well, not because they're, they hang off of the projects really and the mapping between storages and getting the addresses and the confusing ways they can overlap or cannot overlap.

A

Maybe we can move on unless we know more here, I can show you something about the actual clients codes in the rails, application that makes to your PC equals.

A

Oh close. This close this complain then I'm not saving and I. Don't care. Okay, so some basics of the layouts of the rails, application most Ruby code is in either an app or in Lib I, don't know exactly where it is understanding. What does happen? What is live? It is part of figuring out why the hell rails organized the way it is within app. There is the models directory and these things map to sequel tables for the most part, not necessarily, but almost all of these correspond to sequel tables. So there they are yeah I.

A

Think I guess active record is the actual name of this pattern. So in here you have the project class and the project can have an Associated repository, and it's this actually good example, because projects here is an active record class, meaning that it has a sequel counterparts and repository. It is not, but it is still tightly coupled to the rest of the rails application code. So this is this is the first layer you often hit when dealing with repositories and the kit lab coat days.

A

If you reason from the few, if you start from the point of view of here some application code that wants to do something they first, they have a project instance and then they can say project or repository, and then they get one of these things now repository has a role, repository I, think that's what it's called: let's nope, yes, role, repository and.

A

Repository is an instance of something called gitlab git repository and getalife git repository is in a different directory, so these guys are inept models. I get like git repository, isn't lib.

A

Now this thing started out as a sort of adapter between kids lab the application and, however, we were accessing git repositories before we had kidney, we use rockets a lot or which spawns get processed directly. Before that we used before we use rocket, mainly, we use a gem called grits and good luck. Goods throughout the history of Kislev has been a sort of adapter between the rails application and, however, we really interact with git repositories.

A

So when we did the grisly migration, we kept this adapter, but the this thing is not actually making it nickels because it works easier for us to not write all that code in here, but to write it next to this. So next to this we have lip kit lab literally clients, and this is the actual G RPC client code. So when Caleb interacts with the repository first, it has to fetch project which is AB models project. Then through project access, is a repository which is AB models. Repository then from there.

A

It needs to access lipcott lab git repository, which sometimes happens transparently, which is very confusing because in Ruby you can have transparent delegation and but there's no strict typing, so that models repository typical of git repository and then the actual GOP sequels get made here in this lip get lab get to the client thing. Are you still with me so.

B

A

Customer crisis yeah.

B

We we switched, we swapped out some ghillie calls for rugged calls to the in here. What happen in on the canal. I know I, don't like Italy. We also have like a ruby I.

A

Know that happened here, because this.

A

This directory is where we used to interact with regrets. Rugged calls used to live here, and there were some rogue bits of code. Otherwise, in order we're in the application other place in the application would also call rugged, but officially rugged calls happened here. So these patches that are now being discussed to bring back rugged calls are modifying this codes so that, instead of calling out to easily clients it will call out to records, got it ok,.

A

So cool good question: yes yeah, so ok, I'm, taking the long way around before I, show you the actual to your PC stuff. Forgive me for that. The all all the main stuff happens in this file get to the client RB. This is a generic to your PC clients. That is well. It's generic with respect to all RPC calls that exist with integrity protocol, but it's specific to Italy.

A

These other classes are subdivided by our RPC services as they are defined in the protocol. So these two things like ok, when I keep doing this roundabout thing and let's look at the commit service.

A

This will have a find commit somewhere or find commits. So this is an example of glue codes where we builds. This is us directly using G rpcm protobuf code. This creates a program of objects in Ruby, with some field sets, and then it does Italy client on coal, which is the magic that we have yet to come to and gets response objects, and then it parses the response, and then these things get parsed again in the lipcott lipcott layer.

A

So for each RPC we have some glue codes, that's take some inputs from Ruby and then sets up. The actual are piecing messages needed to make the request response so on to this one, Duke Utley client is cool cool, sure yeah. So here we go. This is a big class.

A

Let's see, let's look at coal first, maybe that's a I haven't thought about how to talk you through this class I. Remember.

B

Looking at this and being confused, yeah.

A

It's a bit weak, so that's the natural reaction, here's cool! So this thing. What let's see what's going on here? It thinks of storage, I, think that's the storage name, the surface and the RPC. So that's abstractly saying we want to use commit service and we want to call the fines, chromate RPC and takes the request, objects and.

A

Forget about this for the moments and it can take a timeout.

A

So then, what we do we do well taking the start time. This is just for monitoring. I have no idea what request hash is for this has to do with tracking the number of skittle equals. Rpc calls that happen within the scope of one web requests and to make sure that we don't make too many.

A

So we can ignore that as well, then we are assembling some keyword, arguments that we will need later based on the storage, the timeouts and the remote storage. That I said we were going to ignore. So what is this thing? Doing? Request keyword, arguments? Okay, it sets authorization as metadata.

A

This is the only way we can inject we're rejecting a header field, which is almost a nice to be header field into the quote, and we have to use a keyword argument hash for that.

A

This is all metric, some sort of ability, so we don't care timeouts.

A

Okay, here we compute a deadline from the timeout and we set that on that hash. So what comes out of this is something that has some metadata, which is there mainly for authorization. It has some stuff for metrics and tracing and it has a deadline.

A

So this thing is doing, and then we look up the stub for this service and storage. And this then we do a dynamic methods goal in Ruby with the RPC and the requests and the keyword arguments. So RPC in this case might be Caleb, so service might be essay surface equals commit service, let's say: storage equals default and it's a RPC equals signs commits. These are ruby symbols, I'm typing, which are the usual way to refer to methods.

A

If you want to abstractly refer to a method, you can call you type it as a symbol and the keyword arguments so keyword arguments was something like there will be hash with metadata authorization, blah deadline.

A

So okay I'm say block, so that's what will be the keyword arguments and this thing is taking care of the reuse of stub objects. So stub objects are how the RPC wants you to make to your piece equals. So if we would make it your piece equal by the book, it would look something like this. It would would say stuff equals, get Li commit service, stub, don't new and then address.

A

If, let's forget about the LS.

A

Credentials, stop don't find come okay, I made a mistake here. This is actually spelled like this find commits requests.

A

This is what things would look like naively. There are more complex because these sub objects establish a network connection to the gate lessor, and we don't want to create a new one all the time, so we catch them globally across the rails process. We're in so this Bess. What this method is doing, this methods is going somewhere up here.

A

Okay, using a mutex, it goes into a hash. It looks a by storage and by service it loops up a stub objects and the deterrents that thing, but there's really nothing to it, except that we don't want to throw away that object, all the time and yeah and then there's stuff in there like what are the credentials, what are the headers and that all has to be looked up, which is just if you look at it in isolation, it's not very complex.

A

It's just that we have to pull all these things together and they're broken down into these methods.

A

So this will be the stub and instead of so this is a method, call that's not something we can do dynamically, oh, and we need to pass the keyword arguments which is the is these headers. So this is what it would look like naively and because we can't- because we want to do this dynamically, for whatever RPC we're dealing with this becomes an under under sends like this.

A

So do that like this and then this fine commit thing is a variable. So that's this variable here.

A

It's this sort of sort of making sense.

B

Yeah so in the stub class method is that where it actually talks to the Ghibli proto gem for defined uh yeah.

A

It's fine. You.

B

Are PC right right.

A

But this is the other problem that we want to dynamically retrieve this Ruby objects, and we do that by calling. Let's just stop class message. Let's look at that for moments, so it goes to the Gately namespace. It says Const gets and then okay.

B

A

Does some magic on the input so actually I got the input wrong. So I wrote that it was this I wrote this commits service, but actually that's wrong. The input is this and then that becomes a string and then that gets camel cased, meaning that this becomes an uppercase. This thing goes away and this becomes uppercase and it looks like this so that's camel case, and then it goes back to symbol and it becomes this.

A

So then, actually you end up with that thing and that constant is then dynamically retrieved here at runtime and then we're in a new name space and that namespace has an object, called stub and that's how we get to stop so crazy, a runtime Ruby, dynamic fishing, whatever stuff crazy. If.

B

Used to Ruby I.

A

C

A

It's the best way. We could come up with making this song what such things without having to do cogeneration, because now we can have those other classes like I was showing you there find commits.

A

A

Is a complex example, but.

A

Okay, this is the real find commits, so this is Dutch coal, and so this would be a string like default and it it can just abstractly name the surface and the RPC and the request. So all this meta programming stuff is to make these lines simple. So we can have just one dog whole function that we can make all the calls with.

B

So this is a bit of a digression, but so with feature flags. Is that something that the client passes through the keyword, arguments or somehow it gets into the header of the ER PC? Call there.

A

Are different kinds of feature flags we usually when we talk about feature flags, we mean client side, feature flags and then the feature flag, just controls the call gets made or not. So that case nothing gets past.

A

We did set up the infrastructure to have server side, feature flags where we can communicate from the clients so from here to get Li that we want get Lee to have optional behavior, and in that case that goes that gets passed in somewhere through the through the dot cool method. Here, okay, I'm.

C

Only like a GRP scene metadata in the header right, yeah.

A

Exactly then, we set metadata here and it gets parsed again, but you can then call a method on the context, request context on the go sites and get the metadata out. Okay, all right and so.

C

You can switch between, possibly the Ruby implementation versus then go process. Some.

A

Process yeah, if we wanted to do something like that, we could do that. We only use this once so it is not completely mature. One thing that is notably missing is that this is not the only get early clients. This is a very big and important to get any clients, but it's not the only one, and so we're course kid lab shell.

A

Elasticsearch indexer are also getting clients, and if we want to have these server-side feature flags, they would have to be propagated from here down to workhorse and then back up into gets me and that propagation doesn't happen yet, so you cannot control the metadata that workhorse is setting on its get equals to get feature flag behavior on the Gateway server right. If we want to, we may have to build that out at some point: it's just not there. Yet that was very complex, so.

C

Is that? Because workhorse is communicating the two unicorn via like a regular web request, it's not using G RPC context that sort of thing where those headers can get relayed back know.

A

Exactly it is, it is using site channels with regular JSON data, so we would have to explicitly in that side tail encode, hey. We want you to set some extra G RPC metadata and that isn't there yet so. Theoretically,.

C

If, like everything used, G RPC we'd be able to just modify the context and that information would get really back to the caller. I am.

A

Not sure if the way the requests flow gets things propagated in the right direction, because unicorn does not make er piece equals to work horse, so you work horses. The initial well depends, you know this is a big topic and I don't think we should get into it because we set aside an hour for this and hours hour is almost up so I think it would be good to try, try and wrap this up and see if there's something else sounds.

C

A

We want to no thanks. We want to know about this file.

A

So I've talked you through somehow most of what getting a client that called us and.

A

There isn't much else going on here, I mean a lot of this has to do with metrics and observability and like putting tags or counters and just keeping track of what's happening with get illegals, and we can do that because all get only calls and rails get initiates, go through this class, which is nice strength to have. But the downside is that this class keeps getting more complex, as we add more metrics and things, which is why it looks the way it does today yeah. So here's the server feature flag thing: I.

A

Don't think I should even start or start looking how that is set up.

A

Yeah, oh and the other thing we do with the grizzly clients class is that it has this feature, enabled methods which is just a really thin wrapper around this generic feature flag thing inside gitlab. So if we make a client sites change like the commit finder we were talking about, we would have a feature.

A

You'll, probably gate, that by saying, if gotelli clients don't feature enabled commit finder and something like that, would then hit these methods and would translate commit finder into italy underscore commit finder, then loop, it up in the sequel database and take it from there and there's there's nice stuff in this class, because you can have features that are enabled the percentage of the time.

A

So you can send the percentage of traffic it gets randomly chosen if the feature is enabled, so you can ramp up the use of a new feature, but all that logic is not in this file. Fortunately, because this file is complex enough already, it closes in here. That is all about counting calls count, counts.

A

Yeah Tomas yeah, so this stuff is interacting with the the currency current settings current application settings- that is, that thing I showed you in the in the database method up settings preferences.

A

Now this thing is cold: no, it's okay! So these fields fast time our default time or medium timeouts. They get queried here.

A

Through this thing and this uses I think this thing gets expired on each request, but then, during the request, its cash, so you can do these config lookups without hitting sequel a hundred times.

A

So that's what this thing is doing and our more coal counts and then that's yet well, it's a five o'clock I think we should wrap it up anything else. You want to ask before I stop screen sharing, that's a no-no.