GitLab Source Code Group, 22 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: gitlab-sshd Deep Dive

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

And take it away eagle.

B

Yeah hello, my name is currently this will be a walk through a walk through video about GitHub sshd and uh like a documentation that we have for all the information that we have and all the all the comedy, not all the code base, but some of the code base that we have in gitlab Shell project yeah. So uh we have a gitlop shell project that is written in in go language here.

B

Initially we had it written in Ruby, but uh maybe three years ago, we've written it uh to go language due to some performance reasons, and uh quite recently, we've decided that it would be cool to have uh for ah yeah at first it was used as a open, Sage, open, a stage server that handled all the sage connections and a binary that was called for every every SSH connection, but quite recently, we've written it and now it's just a dedicated server that handles all the SSH connections by itself.

B

So uh yeah, let me show here so gitlab. Shell is a product and we also have a package which is called shd that contains all the code and this package actually uh responsible for handling the SSH connections. The reason why we implemented our own SSH solution uh stage, this solution can be found in a blog post uh yeah that we've released quite recently.

B

It contains all the information how all the history pre-history all the opportunities advantages that we uh I wanted to gain with this dedicated sshd server and also it contained the previous architecture that I described here like the one that we have this opening stage.

B

Let me maybe show you that before we had something like this, uh like opensh server, that was calling the gitlab shell as a binary, and currently we have like simplified, simplified architecture that we will return to it a bit later and yeah. Also, this article contains uh uh risks and challenges, problems and lessons that we that we've learned yeah. So let's um have a look, how it works, how it actually works. Now we have, we have gitlab SSH the server that handles all the uh SSH connections, that user performs.

B

Maybe it's uh important to mention that uh gitlop gitlab supports some features that can be performed by SSH, for example the most important one I guess: it's like git operations when we perform a git archive clone, pull and push with the SSH yeah, it can be performed to a HD https and can be performed by SSH.

B

So when we see something like something like this, then yeah we performed we with as say as a SSH connection as an SSH command. Some other comments that can be also also performed, but maybe let's keep it for now so yeah. This.

B

um This package allows us to handle SSH connections and um uh yeah, so when the user, for example, performs a git clone, it's the same as performing uh SSH comment with this uh with this comment is actually looks like like this, so it's returned some of some mysterious cryptic data, but actually get knows how to handle all this one and uh how to how to communicate with the git client and git server.

B

So when this comment is performed, gitlab is ctd actually handles this SSH Connection by performing a handshake or with with a user with a Siege client and also with rails by a fetching authorized keys, and then, when this comment is performed, we also ask gitlab rails where the user is allowed to perform a particular comment, and if, if this comment is allowed, uh a request to gitly is performed and gitly also handles the um the communication with Git on the server and then user reseals or receives all the data that they requested so how it's actually works in the code base.

B

We have a an entry point, which is a mind function. This is actually an yeah, an entry point for for binary this main function here. The most important part here is that it's creates a new server. That is, that is ready to listen, listen for new connections and process them. uh uh This uh new server and uh listening serve functions. They are implemented in the SSH the package actually in this SSH the go file and let's have a look at those comments.

B

Just a second, so it just creates a instance of a struct and then just listens and surf. The connection. Listen and serve works like this. It just calls like private functions, listen and serve and listens. Listen looks like this. It's just uh yeah create a TCP listener and listen for all the requests, but yeah just a quick note. If, if something is uh not very clear but I I'm sure we're pretty sure that maybe it's not very clear, maybe yeah, you can either ask questions now.

B

Yeah or maybe uh it can be skipped for now yeah, because we have like this documentation. Yeah can.

A

I just say something: Eagle um and sorry. I probably should have mentioned this earlier. Yeah um I, don't know how everyone, whatever everyone else, has after this meeting, but if we need to go over, maybe we can go over so that you don't have to rush um so I, don't know if other people have allowed meetings, but um I feel what I'm saying is feel free to take your time uh and the same with questions. I guess.

B

Yeah, it sounds great, sounds great, so I don't have any meetings afterwards yeah. So if you have any questions now, maybe we can yeah discuss, uh um discuss some of the parts of this code. No, but maybe I can continue. What do you think.

B

um Yeah so so request yeah, they are accepted by The Listener and then they are processed by the by the code that we have we have like handle to. The connection is actually handled then um connection it performs. um It's calls like this function and uh actually.

B

Goes um like calls the perform the handshake between the server and an SSH client or a user SSH should perform handshake in order to negotiate the algorithm. The the algorithms and crypto cryptographic models that they are going to use in order to provide like a secure, secure connection between a user and the client. During this.

B

During this handshake, we actually perform the perform the request to gitlop rails, the one that I mentioned before yeah yeah. Okay,.

A

Yeah just a quick question, so um at the client end we just have hit right. We just have the normal gear binary that we haven't touched of course, in any way, but gear labus is HD, is consuming all the good protocols and then providing encryption or whatever that in a way that git can understand. Would that be correct?.

B

Also, actually it's a git git performs like an uh it established, the SSH connection.

A

B

Github is HD, it's uh just performs a handshake to establish a connection, an SSH connection, yeah, and uh so it's just a stage connection and then git starts uh pushing or receiving the data, and it's actually I would say it's actually uh Italy that uh that deals with like with Git protocol and all the data that is related to git gitlab is HD. It's actually responsible, oddly for handling the SSH connection and establish a connection with SSH protocol. So.

A

So, by using a remote which is an SSH remote, didn't get, the git binary somehow knows how to open an SSH connection yeah and which we, which we accept with key lab SSH yeah.

B

Yeah and as I mentioned, uh git is likely performing some something like I'm sorry for this debug message. That's.

A

B

So it's performed something like like this. This is actually an SSH comment to the uh to the my localhost server on this port and it just executes this. This comment on the server and receives receives all the data so yeah.

B

No, so when the handshake SSH handshake is performed, uh this uh request for authorized Keys is performed to to rails in order to authenticate the user because the user it sends like it's a public public key or maybe I'm sorry, I'm, Health, prepared I, wonder I, won't try to show my own public keys or maybe I can I forgot. Yeah like this.

B

So like this, the one that uh configured on gitlab.com SSH uh like sends the public keys that I uh that exist on the server that exists uh locally.

B

Also yeah, that's also exists locally, so a stage actually iterates over the keys that exists in this sh folder and yeah performs a handshake and identify the user that is actually performing this request.

B

So when, when a user is authenticated, we are ready to process the proceed, the sessions, what does it mean or when the connection is established? A user can have multiple sessions. That's something it's something that I learned I would say quite recently that an SSH connection- it's actually can be persisted and uh by specifying this this commands.

B

So if we have a look, if we run our gitlab safety server and perform a simple SSH comment to the server, we see that, like authentication is performed like algorithms are, negotiated and and connection is established when we performed the same uh the same request one more time we receive much less logs and we see we see this line that mentions that multiplexing control connections.

B

So this connection is reused, and so maybe it's now we're not fully approving we're not fully approving this stuff, because uh we have a long run in SSH connections that are established to github.com. So potentially it can be like a lot, but uh we had this behavior before with with opensh, and currently we don't see like performance issues uh caused by exactly this functionality, so we first yeah. We introduced it in order to be Backward Compatible.

B

Now we don't have. We see a lot of like performance issues related to that, so we left it here, but maybe in the future we can like remove this functionality because it doesn't doesn't seem very you know useful because, like git uh because of the specifics of the git yeah, we just perform a git clone or git push. Maybe it's not worth it to keep the sh Connection open.

B

B

So we're ready to handle the sessions and ready to ready to yeah actually process them. This part is tricky because we um we forgot to refactor it, but this function is actually past the parameter and, uh uh let me see yeah handle like a parameter and the actual implementation persists exists in in other another file, which is called like SSH stage. D go yeah, but we forgot to refactor this part so apologize.

B

This yeah excuse me.

B

A

Sorry I've got another question, um so this is something see at the top of the screen. You've got, you've got a deferred function.

A

And then we've got this: is this something similar to how it used to work in jQuery, whereas why is it why we got this function and then it's followed by a pair of empty params afterwards? What does that uh I know? That's a go, laying issue, but what's it actually mean.

B

So yeah it's like line.

A

187, for example,.

B

B

So it is the same.

A

As what her jQuery used to do, it.

B

Actually, I um I, don't remember uh don't remember. This function is in jQuery yeah, but in Golden Beach yeah, since we have time- maybe let's let's discuss it uh like yeah like here, so it's like a simpler example. So defer function is just uh we just specify that we want to. We want to execute this.

B

This code after after the function is completed yeah when, for example, here uh we want to return. We want to execute this function at the end at the end of this block, so we actually will receive like hello and then world yeah. uh Why? Why don't we want to do it like this yeah? We could we could, but uh actually it's possible that we have something like like this.

B

No yeah the fair, uh like here yeah, so uh if, for example, we want to execute F every time they function is completed, yeah after return a and after return B, the defer will actually do it. It will execute this function after this return or after this return, because if we didn't have this like syntaxi sugar, we had to do something like I. Don't know like uh um I mean something like this yeah, but it's actually doesn't work, but yeah with some with some duplication.

B

It would work so this uh what it actually means in our code that we want to execute, for example, this lens. What does it do? It's actually collects some metrics for Prometheus in order to show them on this charts, yeah and actually this function. It's it increments, some Metric! uh It's! It's actually shows how many connections we have in Flight. How many ongoing connections do we have?

B

So we increment some counter and we want to decrement it when, uh when the when the whole function is complete yeah, so we incremented and instantly say that at the end of the function, no matter whether it's returned an error or returned somewhere else, we want it to be decremented.

B

So this is just a syntax, syntaxic sugar or maybe like just a function in go language that allows us to perform this operation. The same with with everything else here, yeah.

A

So say online 174. with.

B

174. yeah so yeah.

A

So, we've closed the go function, notes out on line 171., but then we have this. We immediately have these parentheses. Does that? What does that indicate.

B

Yeah, it's uh it's actually like here or like here, because it's not.

A

It's not the first.

B

Year, yeah, it's a bit different. It's a bit different uh code here, yeah, it's not related to before yeah.

A

Okay, sorry, we can move on so um yeah.

B

It's actually it's actually it's just a way it's written in in the go. That's why I say not to call this function at the end, but to call this block at the end. It's it's not called immediately. It's just the way we say to call cancel at the end of the execution, so.

A

It's a block with no parameters. Oh.

B

A

Yeah something like this: okay; okay, sorry for that Segway, but yeah. Thank you.

B

Totally totally fun yeah. So let me recover this.

B

So yeah we've um as far as I remember, we were talking about the the sessions yeah and uh yeah. So when this session is handled, a session is uh yeah, we actually can process this type types of comment, either the ones types of requests, the ones that describe us. So that gives us uh that give us some information about the environment and the one that contains actual comment. So yeah. Let's talk about this block, this one is actually the block that handles um handle this this request.

B

So when this comment is received by the server, these functions are called and actually this function is called, and so we we receive this comment. Maybe let me show how it looks so when we send this comment to the server.

B

We see that this comment is being execute like executing the comment, and we can see the original original comment, which is called uh guitar and.

B

So depends on which comment we have received: we create a structure, a structure which is defined here, structure of a particular type. um It processes every every type of a feature of a comment of operation described described here. For example, if we perform some Alphas authentication, we wire SSH command, then lfs authenticate object structure will be created when we perform like Gita plot pack or git clone. This comment is created.

B

Some more information regarding how it's implemented can be found in some other piece of documentation, I hope to have it merged like soon, but for every comment we implement this comment, interface, which has execute method so for for the.

B

So when we receive this comment, we create the comment of particular type and execute it and.

B

Yeah, for example, for for Gita plot Park, execute method, looks like this. It's uh it just verifies access actually performs this HTTP request to rails and verifies that the user is allowed to perform a particular operation and then just yeah. Let's skip this block and then just performs Italy call, which is a grpc code, but maybe let's not elaborate. What jab c is in this session?

B

Let's just assume that yeah a request is performed digitally the data is the response is received from gitly and the respond is actually sent to a user yeah. That's probably it just. Some honorable mentions that uh our gitlop shd server, it's uh it supports proxy protocol. Proxy protocol is necessary for us to implement Implement feature, which is called group IP restrictions restriction uh proxy protocol allows to gain a user real IP, for example, when gitlab for com is behind, I mean gitlopcom is behind the load balancer without proxy protocol.

B

We receive for an IP of a load balancer on gitlab shd with proxy protocol. The real IP over user is propagated. That's why we have. We are able to implement this feature, so proxy protocol is just implemented using this open, open source package, and it's it's really implemented. It's really simply implemented. It's just wrapping the our basic listener into proxy protocol listener.

B

Yeah. Some other things that we've also have implemented is a some Open, Stage Alternatives or, for example, an option that allows us to uh to reduce the time allowed time allowed for user to authenticate, for example, if a connection, if a user takes like more than 60 second 60 seconds to authenticate, then we just close close the connection.

B

It's uh like security feature just to avoid having a lot of idle uh unauthenticated SSH connections on gitlab call like this feature, there's also a client a live interval that actually, when the server just sends people live messages to a client in order to keep them alive or in Open Stage. It's called client, Library interval and interval and uh yeah we've implemented an alternative, and one important thing to mention is a is a state machine that we have on the server server, actually implements a state machine and can be in this in this state?

B

Why do we need it in order to implement, for example, graceful shutdown? uh When a termination signal has been detected detected on the on the operation system, we shut down the server when the server is shut down.

B

When it's being shut down? It's uh sorry, yeah. We instantly changed the status of the server to the on shutdown status. In this status. We stop. We stop accepting any new connection and uh give a configurable grace period in order to ongoing connections to complete.

B

So when this grace period expires, we just yeah just hard close. All the ongoing connections and yeah. Also, the state machine also allows us to implement blindness and Readiness probes, and those probes helps. This has just endpoints.

B

Oh, let me show to here.

B

Like these are just endpoints that uh just uh they just tell uh kubernetes whether our about the state of our server, so when the when kubernetes asks uh whether our server is alive in life, we return start like 200 if the server is actually running. If when a kubernetes asks whether the server is ready to accept any new connections, if it's not being shut down, we send 200 if it's unavailable at the moment we answer with 502.

B

and uh yeah. That's that's all yeah I'm, sorry for all this uh yeah complex information, but I plan to merge this uh documentation.

B

I think it will be easier to yeah just reading it and go know it by yourself when you need it yeah. So thank you very much.

A

Yeah, that's great I do have one question about authentication, so when we have rules defining who can push and pull and everything so where in that flow, do we actually check you know if the user can actually pull.

B

Because what happened like during the first authentication step uh so I would say that it happens. On the second authentication step, I mean uh here, a handshake is performed yeah. When we perform the handshake. Our SSH connection is established. We can just uh send uh wherever data we or whatever we want yeah and uh then, when actually a comment is executed. We we verify their access Yeah. We actually perform this additional additional request and uh I better find that a user is able to perform a particular action. So let me see where it's implemented.

B

So like so, it's implemented in Access verify model, and so it's implemented in a different in a different package like this one and allowed a loud request is performed. Yeah after connection is established. Yeah we and we are ready to execute a particular function. Then we verify that a particular user is able to perform. This information is allowed to perform this operation. Yeah.

B

A

There you go, I've got a follow-up question to Joe's question. If you could just go back to the architectural diagram, sure So when you say a handshake, so what we mean is um the first call when we're looking at authorized. Keys is we're just checking whether the sshk is valid right and then yeah, and then, if that's okay, then the second call is then we check. Are they authorized for this? This for this repo or whatever yeah.

B

That's actually uh uh that actually makes me think that in this diagram it would be cool to have like a quick note where uh handshake is completed yeah. So it's it's somewhere here, I would say like it would be cool to have this line to separate the separate that handshake is completed. Ssh connection is established and then some bitlock specific part is started to be started being performed. Yeah.

B

Yeah I think uh yeah. If you have questions I think you should ask them and we can discuss it. Yeah.

A

Does this currently only support um get actions towards git level? Do we also use this for sshing.

B

Into like the CI so actually.

B

So um let me think, maybe I don't know which, which one which comments do we have from CI, that is that are performed, oh so yeah. Let me let me just show or maybe could you please elaborate? What do you mean by Sage actions, so yeah yeah.

A

Like you can you can associate into a CI, Pipeline and debug it right? So you know we had this issue where customers were unable to um allow a range of ips, so I'm just wondering if you know if that, if that exists in RCI as well, when you SSH.

B

Into debugger, so I think I think um CI like boxes instances, they have like, let's say different architecture, I think a simple open SSH is just there, but I can be sure yeah, but exactly this gitlab is safety is just like SSH server restricted to our uh to our particular use. It just supports our features that are listed.

B

Yeah I already showed them so that uh I mean like, uh like git clone git push. So when we, when we deploy gitlop, then we have this sshd server, that that is only used for our GitHub specific commands, but for SSH uh for SSH into the box to the instance I think it's not allowed in gitlop shd, also due to security reasons and uh yeah I think on on instances it's implemented, but with opensh, but I can be sure. Yeah.

A

um So Eagle I think this is also. This is clear on the diagram. I just want to kind of call it out. So um so all the activities that gitlab sshg are performing are basically around authentication and then once once the once, the the user is fully authenticated, both fire keys and then via their um you know their actions. Then then we actually see this uh unload pack request. That's going directly to Italy and at this point there's no more traffic through gitlab sshd. It's giddly replying directly to the good client. Is that correct.

B

um I would say it's uh like yeah. Just let me probably show.

B

B

Maybe it won't won't, explain it a lot, but uh we actually providing a channel to gitly we're actually providing like an abstraction to get lit, something that gitly can write into uh before. Gitlap is HD, it was. It was just uh STD out because open a safety, open SSH was written from STD art, uh but now with GitHub is HD. We provide our channels, we provide channels. These are digital opacity, abstractions, it's something a gitly can write into.

B

So we just yeah gives give bitly the abstractions to write the data into it, and we also propagate this data to the user.

A

And so because we've developed those channels has it made. Those channels are also like you know, instrumented and monitored by our code.

B

uh It they can be they actually, uh this uh their obstructions provided by uh SSH Library Golan SSH library, but uh if we want to monitor them or extend them, it's actually possible because, uh like read uh writer, it's it's just. It accepts not a specific.

B

It's like this is: it's uh accepts, an interface right, a reader and these interfaces they actually just Implement, write and read uh functions if we want, we can um like override them and, uh for example, monitor all the data that comes through these functions. If we need, if we need it actually, yeah.

A

So so, if during the giddly you know, the giddle is downloading data and suddenly something happens right if it goes offline or the Network's down or something does gitlab sshd capture capture that error and report back to the client or does it would just drop the connection.

B

So um if it's it's actually.

A

Like if we lose this for.

B

Example, connection.

A

To Italy, for example, for.

B

Example, yeah if we have a problem with an issue with Italy uh this, this comment actually returns an error yeah and in this case uh this error is propagated like higher higher. So let me just uh show the the actual the final code and then we just yeah capture this error and ah we we track it and currently I'm, showing the place where we check this error in order to in order to show them in order to take them into account on this chart. Yeah yeah uh yeah. But your question was different and.

A

It's kind of related. Actually, so we are I.

B

Guess yeah, but in this function we just yeah returned the message to the to the user. That, like with the reason why this particular uh why this particular particular connection was interacted.

A

So so, even if there's an error in the giddly, we catch it, a GitHub is society catches, it okay, yeah, yeah, great.

A

Anyone else with questions.

A

um I just want to also point out to everyone that we still do have open, SSH and quickly run wrong here, legal in the same code base and we're keeping it because um we originally did not intend to keep it, but for Phipps support. We need to keep it, so we actually have both sets of code in the same binary. Is that correct, eagle.

B

um So it's um maybe.

A

It was two separate binaries yeah.

B

It's a multiple, multiple, like uh binaries that we we have for this is a current, so we have multiple ones here. Gitlab is HD is responsible for the server. uh This binary is responsible for processing all the connections provided by openness, safety and that's true. We we still, we still have like two options: either either our previous approach or this one and the previous approach actually by default on a self-hosted applications and uh yeah due to the hip's concerns.

B

uh We, we also decided to leave open as Sage, because it's like battle tested and keeps approved and keeps it's actually just a standard cryptographic standard yeah. For that we try that we are trying to pass yes, so we have this architecture available as well, and some users actually uh not yet sure whether they want GitHub a CD on their self-hosted applications, because we have like this, this com comments here, hi, what's wrong, it was fixed yeah.

B

A

Any other questions.

A

In that case, I'll ask the last question which is, for example, um if any of the other team members were interested in in contributing to this project. Do we have do we have any low hanging issues that um you know would be? You know a low low entry issues that um someone might be able to look at in our backlog.

B

So those issues can be found to like, like here yeah, but.

A

Do we have a label.

B

No I think we don't have a label yet yeah, but uh it's something here to have a look at the issues list and uh maybe I would even mention that don't be afraid to take something uh more complex. That just simple, because uh quite recently, I've noted that we have and open open source contribution to support like Care, Bears, kerbers or GitHub s HD. So um some complex stuff can be done as well. So yeah we have issues list, but yeah also complex stuff can be done.

A

Yeah that was awesome. Thank you so much that I'm looking forward to seeing the documentation merged. It looks really great. It's um um I think for a long time that project hasn't been well documented. So that's you explained everything very clearly: I thought and um yeah great job.

B

uh Thank you very much. You're very welcome.

A

I'll stop the recording.