GitLab Gitaly group, 14 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How Gitaly fits into GitLab: Episode 4 – Git HTTP

Description

A 1-hour training video for contributors new to GitLab and Gitaly.

Intercepting Git HTTP traffic with mitmproxy, overview of
Git HTTP clone steps, code walk in gitlab-workhorse and gitlab-ce,
investigating internal workhorse API messages used for Git HTTP.

Recorded 2019-03-14

A

Hello and welcome to the fourth little training session, where we look at how cuddly fits into a larger picture of gait lab today, we're going to look at skid, pushing it pool over HTTP.

A

We've looked at the SSH case already, so some things I won't be explaining again, and maybe we can get to things like merge, commits and others get created, because that's a third category of doing stuff, that's sort of a natural follow-up to the topic, but I, don't know how much time we're going to spend on get to http before I give an overview of how gate HTTP works.

A

I thought I'd just do a small little demo, because with HTTP compared to SSH, HTTP is very nice tooling, for instance, something like Mitton proxy, which is man-in-the-middle proxy. This is open source projects that allows you to inspect web traffic. If you can control and if you have sufficient control of it and if we can do with git clone through mitten proxy and see what actually happens with the gate HTTP protocol.

A

So, let's try that I'm going to be using a real terminal for this, not my fancy Acme, because bitten proxies and anchors this application- and you can run curses inside like me that just doesn't work.

A

Mitten proxy has a number of modes, and if you want to do stuff where you analyze gate traffic, I've learned, you want the reverse mode, and so you start with a proxy with mode reverse, and you need to say what the upstream is, that the real traffic goes to and for now, just because it's possible I want to show you that you can just intercept traffic real traffic. That goes to get lab. Comm like this.

A

Yes, there we are so this is the mitten proxy app and I now want to do a clone from four great letters come on a public repo. Now Milton proxy is listening on its own ports, localhost:8080, so I'm going to tell gates to clone from HTTPS localhost:8080, um and this is a special gates environment variable. If I don't do this, then gate is going to say: hey, there's a self-signed certificate here and we're not going through we're go. Do it.

A

We don't trust this server, but you can talk it to xscape the certificate, verification and because I want this to be repeatable I'm, going to delete the Kid lab test directory- that's already here. So this should do a clone from HTTP HTTP, localhost 8080, and that will then actually go to github.com.

A

So on this side looks like a club simple enough, but now, if we go back here, we see that two requests happens, it gets in a post and we can see exactly what the resource was. So the first part of our clone this HTTP localhost:8080 that got translated into HTTP GET Lacombe, because we started up with a proxy in this reverse mode, where it targets HP, SK Telecom, and the second part is the URL that gets makes it. It makes its first request to so it takes the base path of the repository and it's ads.

A

/ info / reps with a query, parameter surface equals good upload back and that's it get and we get some stuff back and then it does a second request. The posts do the same base path, but then / get up. Look back now. The cool thing is, we can use VI key bindings, okay, here's normal key bindings, yes, okay, well, I used to fear I key bindings and I hit enter on this and then I can see what happens so it's it gets in this case. So there is no request body.

A

These were the headers, so gate is telling github common what version it is not much else, and this was the query, parameter now the response. In response. We can see all the response- headers, oh by the way it I made sure to clone from a public repo, because otherwise in here would have been my HTTP basic authentication, showing my passwords, so I could to try to remember to not use my show my github.com password here.

A

So the response of the gets has this special content type execute loads, get up, load back advertisements and this what this looks like text data, but actually here you see / /, x0, 0 or there's a little bit of binary data in there, and this is called the ref advertisement, and here a server tells gets all the refs that are in the repo.

A

So that's the response on the first requests and then I think I need to head to hit escape. No I need hit cue to go back then, on the second one we get the posts.

A

I hope this loads.

A

Okay, it's working so the post has a body and a request body and a response body. So this is a custom content type that it looks for, and it is a bunch of these messages where it says that once all these commit IDs and it only wants stuff, it can also say that it already has stuff and then Gatewood calculates a smaller peg file.

A

That only has the difference, but this is a full clone and if we look at these commit IDs, let's look at the first one, there's a special thing going on here that there's some sideband data abandoned here. That's a heck in the gate protocol. So the first thing it asks for is e 56:49.

A

It knows that exists because in the first requests.

A

Here gets set somewhere, let's see where is it a 56? Here we go d 56:49. You need to ignore the first four characters, so that reference was advertised in the get request. So then, in the second request the client comes back and says: I want that commit.

A

I, could oh no I keep going into the source, so that's the request. The response is going to be partially binary data, so not all it will make sense, but you can see that there's progress. Information in here counting objects. All this stuff gets by it set back gradually, as as the response gets prepared or progress, information, more progress, information, compressing objects, okay,.

A

Yeah and here we have to start with a Peck file, so the four letters PA C, CK and then 0, 0, 0 2 means Peck file, version 2 and so that's a magic header and the rest of it is kids back file data and, let's see yes back file data all the way down.

A

So that's, okay, that doesn't work. That is the request, response cycle and I. Think it's a really cool trick that you can do this because you can get. You can look both at the this stuff at the gate, transport level.

A

But you can also see these HTTP headers if you're trying to figure out where something is working or what's going on that's very valuable and just to protect myself from making mistakes I'm going to restarts mithun proxy against local host 3000's, which is my GDK, because my kids, credential helper is not going to send my kid left with calm passwords to localhost three thousands.

A

So that should help, and then in here we no longer I can do is the same clone, because in GDK I also have this repo to same path, but I don't need to SSL no verify anymore. Take that off.

A

It's okay, it's still using HTTP. We don't want that.

A

You not concealed clone. He actually saw some stuff in the backgrounds here. You saw that post request coming in, so you can see it both on the server sites and here in Minton proxy on the client sites that it made those same get them post. So that was my little preview of what happens in get HTTP.

A

Let's try to write some notes about what happens so, okay and we're going ot look at git clone because it push is slightly more complicated, but all the complexity. The complexity of git push is the same here as it was with SSH, and we already looked at it there so to keep this story simpler, I'm going to ignore git push, so what happens? A user runs, user runs, get long gates makes HTTP requests to get requests to, and what you see is a cloning.

A

So, okay copy, there's a I hope you can see. I have the little picture of the video here. But can you see the is anything blocking this panel? Okay, good now.

B

A

Thanks so the first request is to Infosys rifts, and it has this query: parameter.

A

One of the reasons for that is that.

A

No, let's not get sidetracked by that right now, server responds with lists of reps and commits tax. They point to and then get makes an HTTP POST requests to almost the same thing except it's. This and server responds with progress messages and a back pack file.

A

And what so, behind the scenes, this request gets it's intercepted.

A

It lab workhorse, workhorse, three authenticates with kit lap, rails, Gabriel's response with Italy connection parameters, repository details, get elite network address stuff like that and workhorse establishes what workers makes a get Li RPC call.

A

And copies response into HTTP response body and on the second one it looks almost the same except the difference is that Catholic makes an RPC call copies, HTTP requests body to get early copies, Italy response to HTTP response body still with thee cool. So what do we look at next? I?

A

Think one interesting thing: maybe we should just go in order and look at how the request gets intercepted by Hitler for horse, which will force me to say what cater, for course even is, and then once we see that we can look at how this pre authentication happens.

A

So what does get that workhorse getup workhorse is server-side back-end component of gitlab. That acts as a reverse proxy for the Ruby webserver process that the actual bulk of the application runs in and because it is a reverse proxy because it sits in front of it. It can intercept any requests at once and do funny stuff and we use it to offload slow requests from from Ruby, because the Ruby webserver process is relatively expensive as limited throughput losses, lots of resources and by having yeah.

A

So we try to limit how much time we spend in there and get HTTP actually the original motivating example for why we have workhorse, because if you look at a clone, especially this part, the if you're cloning a large large repo, then in the HTTP response, you need to send hundreds of megabytes or gigabytes worth of data and it's very expensive to do that with a ruby application process, so that there's not a good use of the Ruby application process. So we broke that down into. We only use Ruby to do database, queries and lookup.

A

If the project exists, if the users allow to access it, things like that and then the rest of the requests of the users requests right. So the user wants to just get this and we transparently offload the rest to workhorse and that frees up this request response cycle as fast. So after this, the Ruby process can be reused to do something else, and this yeah, like I, said.

A

This is why we made workers in the first place because get HTTP was very unreliable when it was still going relying 100% on the Ruby application server processes, because in particular those have a 60 second timeouts by default. So if your clone would take more than 60 seconds, it would be aborted.

A

You couldn't clone a large large repository over HTTP. So that's what we're course is so concretely, it is a goal application, and so it's hosted on github comm, kickflip org, slash, workhorse, there's also. Let me pull it up for a moment.

A

If you look in the readme there's some background in blog post, we Rhodes yeah, there's blog post a brief history of cattle pork horse, which is that's already more than a year old, but it sort of gives you the story of how this happens. If you want to know more about that, so workhorse is the thing that is making the actual Gately RPC calls and it's it's a web service, so it somewhere has a set of it as a central Rooter, where we try to intercept requests and at the end of the route table.

A

If we didn't intercept the request for any reason, it gets proxies all the way back to github rails. So let's look at that root table and that is in the internal of Stream package.

A

Helpers, pardon we have a bunch of helpers here and then here's the actual roots table and because git clone was one of the first things it's at the top of the roots table. I, don't know if it's actually the most commonly called thing, but it's at the top- and here you see the things we are handling so if it gets four requests that ended him for refs, we have a post for requests that ending get a lot back and we match on the content type.

A

So any request that doesn't have that content type does not get intercepted by workhorse, and then we have the third one. This one is for push so get pushing. It pool make different post requests and where does this go content and calling handler? We can ignore. That's just a middleware in this cause. You can do great dot, upload pack or gated receive back.

A

So let's look for those.

A

And that's an HTTP handler that goes into an inner function, called post RPC handler.

A

Okay, more middleware, it goes with something called repo pre-authorize handler and if that responds with something, this is an internal type, an API dot response. Then.

A

What is the handler? Ok, the handler is an arguments. This is one of these functions and what are these functions do and all the plot back I want to see this function once you see this function so.

A

That function gets a get Li clients and it calls the upload back RPC. Actually, this is an internal function that combines calling the RPC and copying the data can actually find that function. So this is just some glue code. I think we can ignore this one earlier, look for a plot back and turn on this one.

A

Okay, this an actual Gridley RPC call is an RPG RPC stream. We sent a first request where we say what repository want to look at get config options. This is something that gets injected from the github real sides, so we can set extra config on the upload back process. Git protocol is currently disabled, but it's a feature. It's an optimization internal to get when you use the git protocol.

A

How does that work again?

A

V2 I can quickly show you a mitten proxy. What what has to happen for git protocol? Okay, if I put C protocol version equals in my commands, then that will try to use git protocol.

A

So let me put that in there so.

A

Now this client is trying to make a good protocol v2 request, and what happens is that here in the gets? There is an extra request, header that says, version equals, and it's workhorses job to see that header, because workhorse is the thing that sees HTTP. There are two layers to get over HTTP, there's two each to http layer and then there's a gate, transport layer and the HTTP part is invisible inside Catelli itself. Cuz Gately is not serving HTTP requests from the world. So workhorse.

A

Is this a good protocol thing and does some checks to decide if that's allowed and if it's allowed to use first protocol version 2. We need to signal that here in the RPC request, because otherwise that information is lost.

A

Ok, so we sent a first requests and then we send up two set of to go routines, one that copies data from the request body into the stream and the other reads: data back from the Kitley stream and copies it into a response party.

A

So that's really all that all that happens there. This is the this. Is the copying back and forth part that I have just shown you this part makes quickly. Rp is equal, so that's this copies, HTTP, request body! That's this sorry! This one and copies quickly responds to h-2b response, and that's this.

A

So that's the three parts this is. This is roughly workhorses job. There's one thing we still have to look at and that is this one workhorse pre authenticates with reals I sort of crossed over that earlier because, well sorry, any questions about this part. So far.

B

I have one question not yes. In this past you said that the heat conflict option third comes from rails. Yes,.

A

Our workhorses a is it fairly it's a dumb, stateless thing. So when this request comes in it has no idea what has to happen. It has no idea how many kidney surface there are no config about kids whatsoever, but we'll I think maybe the kid config options will become a bit clearer when we look at the previous allocation flow, because that's where that information comes from because they are github rails response with connection parameters, so here I can add git config options for kids up back, that's where that comes from.

B

Okay, okay, cool, because this is the handler that we call after the pre authentication. Yes,.

A

Exactly this happens first, so this has already happens and we can't make these requests because we don't know, we literally don't even have the network address of the Gately server. We can't do anything until this part is happens.

B

Okay, clear yeah.

A

So this part is this, and this is relatively straightforward. So let me close that and let's look at the other parts, so there was the post RPC handler, which returns an HTTP handler and there's this inner function here, repo pre-authorize handler. So what does that do well more layers that uses an api dot pre-authorize handler.

A

So what is that now? We need to go to a different package in workhorse.

A

Hey some of the naming and workhorse is slightly confusing because yeah it's a project that evolved over time and its responsibilities grew over time and well. You know how it goes. We can't always keep up with there. There's probably some technical debts in in the naming here. I actually will get to that, there's actually quite a big chunk of technical debt and how this part of good HTTP works but yeah. So what this thing is doing.

A

This is the API package, and there was a handler in here the pre-authorized handler.

A

This is pre-authorize handler, so what does it do? It takes?

A

It returns, a handler funk, so ender funk is just a function that takes with the signature of HTTP handler. So if you want to understand what this function does you can ignore HTTP handler funk? That's just some Co interface wrapping magic. It really. This thing returns a function that takes a response, writer and a request. It looks at the requests and it builds okay, sorry it takes requests and it makes it course this pre-authorize thing which makes it cool and to get LaBrie's.

A

So this is the actual parts workhorse pre authenticates with github rails and two things come back an HTTP response or well. Actually you get an HP response or an author response, because when we do this pre-authorization two things can happen: either we're on the happy path in the case of git clone.

A

That means an actual git clone is going to happen, but it can also be that there's a typo in the address or the user isn't allowed to access the repository or anything else in which case rails will just respond and say no and then workhorses behavior is to take whatever the HTTP response was from rails and copied that to the clients. So, if there's a failure, if realz disagrees, the response is transparently copied down to the clients.

A

That's why we get two things back here: one is rails talking to workhorse and the other one is real stalking to the downstream clients.

A

I was talking about on the unhappy path. If there's something in this thing- and there is no reals- is not telling us what to do. Then we just need to pass back that response to the client that we're done otherwise.

A

Rails might set headers and we're going to copy those headers this we'll get to this later. This has to do with authentication, which is good to look at, but not about sending username and passwords during get HTTP, and then this thing just goes into the next handler the next handle func.

A

So then the question is: what is the pre-authorized function? Do it takes the incoming requests and it's it modifies the address little bits and it replaces the request body, and that is because, if this is one of these post requests, then the request body is gets transport data that needs to go to get aliy, and we can only read that in the HP request we can only read the body once right: it's a network socket you read once and then either you save it somewhere or the data is gone.

A

So we just leave the request body sitting there. We only look at the header, so we make we derive an empty requests that had the headers of the original request and we forward that to the rails, backends and that then tells us what to do and if it tells us to go ahead, we still have the original rails request from the clients and we can then copy that to kattiline. So that's one of the things that this thing has to do here.

A

So that's what this author request thing is doing and then it makes requests to some Counting and it decodes the yeah. So the case, that's the real understanding us to go ads. Then we get a response body that is some JSON data and that's.

A

And- and that is what that does- that's how we know what to do so. Maybe it would be good if we can see all this earth response. Is these things or what? If now I would just say, log can I log like that?

A

Yes, I can I'm cheating a little bit because I'm not supposed to use the well. We use law gross mostly, but I actually find loggers very hard to read for this sort of stuff. So I'm just going this sort of stuff, so I'm just going to use the standard library printf, but or do we want to say we are looking at Rails API off response.

A

And we want to show this thing I think with the plus. We get some more details. Okay, now, to save this, then go imports will add the log package now I need to restart my JDK processes or I need to recompile workhorse first I think that is make it lab workhorse set up in this window I'm at the top level of kid lab development, kids.

A

A

Why is this so slow? Okay, it's done, and now we reboot the app and that this is the next. Why is this so slow.

A

And let's see if I try to make a new request now it will just fill with a five or two, because why does it feel? Because we're trying to make this pre authentication requests against unicorn, which is a ruby application which takes forever to boot? So we can make requests yet. Let me clean this up because we don't care about protocol version two right now, still not there.

A

It's actually leaking quite a bit of information, but I think that's only because we're in development mode.

A

Still, connect.

A

Okay, there we are now before it gets lost here.

A

Here's the get- and this is a we don't care about this- so I'm- going to delete that here's workhorse here's the post, so here our first did okay, what so? Let's? Okay? What happened on the client sides we have with metal proxy yo. We can even see the 502 s here and here's the clone going through with Catherine posts. So I now want to know when to look at the server side logs for this cat and post.

A

So this is the pre-authorization.

A

For gets- and here you can see the actual gets happening and then this one is.

B

A

Pre-Authorization for the post and my log statements, Rios API author responds and we get a big structs and what's in destruct, MTG led and the username. You may remember these things.

A

This is part of gate push and because we're doing a clone, these fields are not populated right now, here's gate, config options, there's nothing.

A

Here's a whole bunch of things that we don't care about and that's technical deaths because we have one. We have one struck type for different types of responses from reals and they'll get marshaled in here. So there's a bunch of fields that are completely irrelevant to get cloned that are still in struct. So when you log the struct, you get that information, so we don't care about. L, FS, l, FS remote objects. No, this is we don't care about archives or entries or terminals. We care about the Kitley server.

A

So let's keep that and we care about the repository.

A

Where does that end? That's all repository, and then this is also relevant to get so. This is the relevant part of the API response, and that is so that's this. Are you still with me?

A

And yes, it's a lot you can see this is. This is similar to what happens in the grid lab shell case, because when you do a git fetch over SSH there is this preauth nth occation face where gate lab shell makes an api call into rails to try and get the necessary information to be able to establish the gate early connection, so this information looks very similar and get config options.

A

I don't think I want to go into what we use it for because I'm not exactly sure and I, don't think it really matters to the big picture, but this is where it comes from to answer your earlier question friend, okay,.

B

Okay, thank you.

A

So this is the really critical thing where I would lie saying: workhorse doesn't know what goodly servers there are what they're called where they are. It learns it by getting this thing, and this is a repository object. So this is straight-up goodly stuff. You have a storage name relative path. We talk to our alternate object directories. Well, we talked about all this stuff last time and I think this is something we use for geo replication, because there are some refs that you normally can't clone or they get sort of hidden for performance reasons.

A

But when you do geo replication I think we try, we do want to show them to the gate transport. So that's what that is for, if you like, to keep around rafts yeah, exactly I, think this controls, where the keeper on graphs gets shown in the a ticket transport level. So that's not that's not up to the gate, clients that is up to Caleb rails. To decides. Yes, you get to see these or not so yeah.

A

That was some work, but we've seen this part but I was I was promising you some nice technical debt. Earlier, of course, the fact that I deletes 20 fields here that are for other calls is also technical debts, but the cool parts, if you think that is cool the cool part, is when we look at what how this works.

A

So that is where, in the rails, application we are handling these requests, because this is this is a hack that, where you can really see, we took on so technical debt at the time and we never got around to doing something about it.

A

The requests that get made internally, you don't think these catalogs. Maybe if you can bear with me through a recompilation cycle, we can log what the request is. So this is the authentication request. So let's do the same. We did with the other one.

A

So that was the auto response, and here we can read, show the author requests should have thought of that. First time. Let's put that in there and now quit this.

A

A

If you have to do this, Lots there's a trick, let me show you the trick. You can edit the profile and look for workhorse and.

A

Yeah, this is not the nicest way to do it, but but you can do I, can create a new window and take this command. Here. That's used to run workhorse, which is quite long and.

A

Now this boots workhorse in its own window and if I command it it's out in the profile it won't get started here, because we can only run it once and now. The advantage is is that I can stop and starts workhorse without having to wait for 30 seconds, while the Ruby application boots.

B

But always I've done that I always get a verify. Api request, I wrote with the JWT decoder.

B

A

You run it in a separate window like this yeah.

A

So I guess one problem here is that Houston port are empty.

A

I friend I think, if you have the exact same arguments, that should not happen, but there is a problem here in that host and port are missing. So this should be local hosts, 3000th and.

A

This looks better, let's see if I can make a request.

A

A

That looks like it's working.

B

A

I another piece of technical debt and workhorses that we didn't put everything in a config file from the beginning, so some things need to be on the command line and other things need to be in the config file, indicate it just rights. That's why I just copied that line carefully from the proc file, because, whatever apart from host and port, being wrong or being missing its rights, if it works right in the proc file,.

A

Okay, so I just restarted workhorse here and if I want to do another, printf I can restart it quicker. Let's try and do another clone.

A

You that's good happening.

A

Where does it start?

A

This is an authentication requests. This is an authentication response and this is the gets.

A

And then here starts to post.

A

Response as the post itself, okay, so now we can see the request and the response and we can compare it with mitten proxy. So let's look at the post for a moment, because this post got recorded in mitten proxy.

A

It's this one- and you can see here that there is this request body on the post and apparently it was gzipped and the headers are hosts user agents except encoding, blah blah blah. So now, if we look here, then I expect to see the same things here. So we have method posts URL about back.

A

This is some HTTP stuff headers. How far does that reach those other headers body, nil, yeah? So this is the thing why I said we swap out of the request body and we make sure that it doesn't get lost and doesn't get sent to reals and the headers are accepts application. A pro pack user agents get to point to a 2.21. That's also what we saw here. The accept and the content type is where's the content type.

A

We also took out the content type, because actually the request we're making to Rails is not that the content type describes the request body we left out the request body, so the content type would be no longer be a correct description of the request body.

A

So that is the authentication requests and we already looked at the response earlier. Are you still with okay.

B

A

Now the fun part here is that the URL, so this is the URL we're hitting on the rails app- is actually the exact URL. That was the inbound URL, because on the inbound URL here is also slash. Get life works. Let's get up test start kids, let's get up out back. This is the technical that I was talking about. So in retrospect, with the smart thing to do would have been to take to make a clearly identifiable, API, endpoint and internal API endpoints.

A

Where we can go and say, here's the URL, the kid client wants to clone.

A

You tell me what to do with it, but that's not what happens, because the way it lab worked at the time was, if you had one of these get requests coming in, they would actually make it into the rails application and get caught by the real supplication routes at some points, and then there was called there was code on the real sites which would spawn a good process and do the good stuff, and when we introduced workhorse, we did not update those routes.

A

We just changed the behavior of what happens at those routes instead of spawning a gate process there we made it so that reals just answered back with here's, the repo you, you run the gate process, I'm not going to do this, and that means that if you look at the application routes of the rails app, this gets stuff sits there in a very weird random way and it yeah it just makes things it makes life harder.

A

The fact that it's there, oh let's look for that unless you have any questions so far, let's look at where that is in the in the routes of rails.

A

um How do I read this? It's been a while I think the rules are split in two files. Yes, so there's a kids HTTP file.

A

What is this? It tries to match on project paths, and it has roots so I'm, not.

B

A

Exactly what these records are well I know what they should be, but I don't know where they are defined and we define three routes. Get info Rev supposed to get a get a plot back post get received back. So this is we're just mirroring the the real request that cat mates and these things go into the gate. Http controller.

A

Which is this thing which descends from get HTTP client controller? So if you want to understand, what's going on, you need to understand two classes.

A

Yeah, so I think we can leave this for what it is, because this is just a Rooter.

A

Get HTTP controller where we have a method info refs flowed back and receive back, that's everything. It does and.

A

What's access check that either goes into the kid lab great access class or the get access wiki class and what is get commands.

A

Okay, this is probably defined in the parent class.

A

Or not, how can we miss get commands? Oh here we are so get command either the equator amateur service, because you could see you don't see it here here. You see this parts so either eight recognizes surface equals, get to go back or its looks at the action name, which would be getting a score receive back or get to upload back, and all that really happens here. Is this render okay thing? That's a happy path.

A

If the request is denied it gets caught by this before action, but the request comes in and before we even reach this methods, this one gets run which instantiates one of these classes, and then this thing runs a whole bunch of access checks that I think we shouldn't get into now.

A

We can maybe look at them at another time, but they are almost nuts they're, almost not a greatly problem. I would say they're more about the gate, lab access, control features and that that logic, which you can from a kidney point of view. This is the thing that says yes or no, and if, when we make it into this methods, it has said yes by definition, because otherwise this method wouldn't be entered. And now we have. This render okay thing and this returns JSON and.

A

This is a utility class that creates that the specific to workhorse that creates the json data that then gets unpacked here into this into this struct.

A

So this from a real point of view, this is just saying your response body is some json data, and this is what ends up in there but yeah. The infuriating thing is that these roots are all over the application, because if you look at get lab or get lab tests, then this roots is user-facing webpage and it is completely different behavior, because if this would be privates, private repo I would have to be signed in.

A

So this thing would check my cookies to see if I'm signed in right, because that's how user sessions work their cookies but gets HTTP doesn't use cookies, get HTTP, uses basic authentication or maybe Kerberos, but let's not talk about that that still piggy backs on the same mechanism as basic authentication. So we have this whole hierarchy of routes and stuff and controllers where we know like okay, if you make anything like these, these routes, map to a hierarchy of rails controllers and those rails controllers, use inheritance to say at some points here.

A

We do an access check against your cookie to see if you have a proper session and whatnot and then in the middle of that there's this random HTTP basic stuff that returns weird JSON data, that is, that contains secrets because it might contain, will contain an internal access token to reach a Gately server. So that is a secret used for internal next network network traffic that should never ever leaked to the outside world and yeah.

A

This is just in the station response body and we'll just hope it doesn't get dropped on the floor and leaked out to the user would be much easier to reason about this system if it wasn't in the middle of all the other routes. The way it is before you get worried, we do have checks to make sure that this data doesn't leak out. What we do concretely is that we set a custom content type on this JSON stuff and there's a middleware in workhorse, where, before a request, goes back to the user.

A

We check for this content type and if this content type is set- and it is data that should never go to the hearing- we aborts the response. So we do have. We did think about this, but it's it's more complicated than it needs to be, and the reason for that is that before we're course exists, it's this thing. Well, from a user point of view, this thing has to be nested inside the other routes, because as a user, actually, if you clone this, think it lab will redirect to this dot.

A

Kids- and this is what I, as a user I expect my git clone routes to look like and then it's an internal detail that actually gets blown traffic has nothing to do with like this route, has nothing to do with this routes and.

A

The user doesn't know this and because of history also in the application, is hard to tell them apart. I don't know if anybody cares, but me yeah, there you have it. It works and I think that sort of covers the these parts. We also saw how the request code gets intercepted, because that was that brute table in workhorse that I showed you and we're almost out of time. So questions.

B

Well, not really I mean it's. It's interesting that last part I mean that we can. We also use workers to prevent leaking. Some information like this I mean I've, seen it too I-I've seen workers to replace examples on Twitter's, like the blog.

A

B

That we used to replace for real content and stuff like that. So it's it's funny to see this. This legacy behavior and you wouldn't know why, why that chunk of code? Is there if you're not watching this? This video yeah.

A

Exactly so, this is the thing I was talking about, skip before action, verify authenticity token.

A

So that's one that that would just completely break when you get an HTTP basic clients, and this authenticated user is the thing that just HTTP basic check is the is the token signed for a specific project to be accessed.

A

No, it is a very, very, very permissive token, but once once you have that token, you can make get early request to whatever you want on that server for as long as you want until somebody manually rotates that token to become invalid. So this is like a performance thing like we didn't want to issue too many tokens. It's a simplicity thing. We we had to have something, but we made it as simple as we could get away with we. Actually, this is a yeah. This is a complete digression that we did this.

A

We we got complaints about this and we started transitioning to a system where we have time-based tokens with just an H MEK timestamp scheme, where a token, after a certain amount of time is just not going to do anything anymore. But there are some scenarios where we need to delegate the authority to make Italy calls to get one Gately server. So that's it can then make calls to another gotelli server and we can't really put an upper limits. We don't really know what the upper limit is for, how long that delegated token should be fell.

A

It's so I think the conclusion is that we're going to roll back to the old system, because it's more robust, but it is very, very, very basic. It's only suited for internal traffic, sorry, which old system, but also the old system, is that there is a shared secrets and every recruit and every requests has two shared secrets in a in a header and it gets sent across the wire on every request. That is the the first. That's the v1 get a little tent occasion system and to make that more robust. We now have TLS.

A

It's not deployed everywhere yet, but you can put TLS on the Catelli back-end connection. So even though that token gets sent across the wire, then you can snoop it there. These tokens are in the config file right yeah. They are in the config file of each get lesurfr in the config Tomoe and in on the gate web sites. They are in the kettle, a PMO where we have this map.

A

This repositories, this thing, so the token would be here and because the clients are stateless, that the get up shell and workhorse are stateless. They don't know these tokens so every time they need to make a request to get early stuff like this tells them what the token is, and then they use that for one request, they could remember it, but they're not designed to do that.

A

So good question completely orthogonal to the topic of kata HTTP and we're sort of we had one hour now any other last questions.

A

Okay, then, let me see if I can stop the recording properly this time.

A

Good in case this works, goodbye.