GitLab Memory Team, 9 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab Plus MVC - support for instance-level authentication

Description

What we need:
- Introduce the idea of instance level authentication
- An authenticated connection between a GitLab instance and GitLab.com
- GitLab instances are responsible for user-level authentication
- GitLab.com is responsible for instance-level authentication

A

So hi everybody and thanks for joining this session, would be devoted to, like our team, is trying to create an MVC for to support instance, level notification in order to support our self-managed customers to access our AI related features.

A

So this is actually the core problem that we are trying to solve, because currently our self-matched customers are like bring 75 percent of our current revenue, and it would be great if we can allow them to access our AI related features, because this is currently a Hot Topic, I guess- and this represents like existential risk for gitlab, recurring Revenue portfolio.

A

So apparently the situation is that we recommend ourselves manage customers to use uh really ugly and hacky workaround, and that is that we ask each self-managed user to create a.com personal access token in order to authenticate them and get them access. So the idea that we are currently like trying to figure out is first like the short term goal is to support some way to sell to allow self-matched customers to authenticate and uh against gitlab.com, and we can use then gitlab.com as a proxy to access our AI features. So there are actually two topics.

A

First, is to like authenticate the instance itself, or maybe it is an organizational level, because I I saw that we are now like within sales effort. Introducing the organization as well I'm not fully familiar like what this means, but like the first topic is to allow authentication between the instance itself and gitlab.com, and the second topic would be.

A

We will discuss the only first one uh authentification like when we do a proxy, how we authenticate uh users and proceed with like user proxing gitlab.com at gitlab.com, sending the requests to Ai and returning back the response and well I I guess it really depends on the solution we choose. So I already asked on the slack Channel like for some ideas, but I would first would like to hear um where we are with the like. The long-term solution.

A

I I saw that we have instance level tokens issue open like two years ago and I see that there are a recent progress related to the service accounts and I'm, just wondering like. If someone can just briefly comment on that, and is this like feasible in the near future?

A

Should we like try to find a workaround until this is done, or we expect this to be completed in foreseeable foreseeable future and how this related to these sales and organization efforts, and in case that we choose the workaround solution like using open, ID uh or something similar like? Should we considered like moving to service accounts in the near future? I guess.

B

Okay, um so service accounts is, is essentially done back end, at least so today, it's possible to create a service account via the rails console and, for all intents and purposes it works.

B

um The the piece that uh smyrian and maybe Eduardo as well are working on is the uip's allowing um groups and instance level to create and manage these service accounts and the tokens.

B

C

A

Everyone thanks for joining. Yes,.

D

A

Worries and I didn't see you on the wait room.

E

I I was I was just wondering: could you expand a bit more on, so is the service account? Does it um behave like a user? Basically, from the perspective of the Rails app I mean? Does it have permissions tied to it? um Do they expire and uh do you do you then generate generate tokens per service account or like if we want?

E

If we want to do something like because at the end of the day, what we want to do is we want to send a request uh to gitlab, says on behalf of one of these service accounts or users right?

E

um So that's, basically what we want to solve for is that something that would be possible possible with this model.

B

Yeah service accounts are essentially uh no different than a human user, account we've just designated them uh as a service or a bot account. Basically, so they don't have any inherent privileges.

B

They assume the Privileges of you know whatever they're a member of so you add them as a developer, to a group or to a project, and that's the Privileges that they assume so I do have some questions there about how exactly we would give these service accounts the permissions that they need depending on then how within gitlab are you going to proxy those requests to open AI and what accounting do we need to have associated with that?

B

Are we doing billing based on usage or is it just sort of a flat like you have uh git level tonight and you can use as much of these features as you like.

A

Yeah, that's a great question. I'm, not sure about the billing. uh I saw some issue where there is a discussion between those two models. I think that first iteration iteration will take billing as a flat service and not like usage based, but this is not something that I can guarantee that it will stay and I'm not fully familiar.

A

So this is just an open discussion and it seems that, because it's easier to iterate with the first approach, I think this is the approach that we will take, but this doesn't mean that in the future we will not have usage per user and, like billing per user has to say, but but for this iteration and maybe future work. Maybe this is something that we need to ask an address, but my understanding was that all monitoring and.

D

A

Data that we will need to gather will be related to the instance and the self-match admins are self-match owner will handle their usage per user separately. So I guess this needs to support the flat model. I.

D

A

Not 100 sure, but this is my understanding.

E

Yeah I think the whole billing aspect. It sounds like we're not uh not there yet right.

D

E

This kind of questions I think we're struggling with like much more basic questions at the moment and keep in mind like this came up like last week with last week or something so um I think Roger um already did a pretty great job at refining.

E

The initial ask to what we thought was what Roger thought uh was uh a really sensible kind of mvpc to start with, um uh but yeah there's plenty of open questions, uh but before we get to topics like like billing and all of that uh I just I I want to go back to I I'm, still trying to wrap my head around how this would work so these service accounts.

E

Can you describe a little uh like how what was the original use case for this, like how would um outside of what we are looking to build like how? How were how were they cons, conceived to be be used by by a gitlab user or admin.

B

Anyone on the off team can feel free to to jump in as well, um but uh the the vision for service accounts, I think was any sort of integration that the customer had and especially from the SAS perspective, um organizations that that are, you know paying for groups on getlab.com.

B

How do they uh integrate without having you know, maybe I'm the assistant administrator for git lab for this organization in the past, I may have created a personal access token from my own account and and used that in some integration, and that's not good, because then that service has my privileges and not only that if I leave the company now now all these Integrations stop working.

E

Right, so so is this to integrate the third party service with gitlab says when I'm a group admin, for instance on gitlab, says uh and I want this service account to act on behalf of this third-party servers for everyone within my group, for example, is that a wait? Okay got it I think the interesting bit with that one. It's like okay, so I think the interesting bit with our use case is that um we're talking about gitlab gitlab integration right, so it took me a while to wrap my head around like so.

D

E

Have a service account I guess so, let's assume I'm a gitlab self-managed customer and I spit up my own gitlab right, so what I would do is and then we have two gitlabs right. So we have mine and we have Caleb says so as an admin of my instance. I would then go ahead and do what exactly would I would I create a service account and gitlab says instead of a personal access token on and this service account uh token I guess would I would then use on my also.

E

It would also need to be stored on my instance right in in my database, because all of the requests coming to my instance, they will have to authenticate uh against using a service account against GitHub, SAS and then GitHub says, does something using the service account on behalf of the original user right?

E

So I I think we're like one step removed from the original use case that you described so I'm, just wondering how how that would even carry over because I think we need some there's some kind of like stateful uh there's like state to this whole handshake on, like both sides, gitlab says and and the self-managed instance. So I wonder if that would actually be covered in the current model.

B

Yep exactly I I have that question as well, because um you know it is each self-managed instance going to create a group on gitlab.com um and then do they need a subscription, for that too, are we going to give them some sort of deal there um because they're actually using self-managed, and they only need gitlab.com for this? This proxy I'm not sure.

E

I I, okay, that that's fair enough I think we can maybe put a pin in that question, we're very much exploring right uh Solutions here. It's totally fine I think uh it's just good to talk about all these options and see like where there might be obstacles uh of where it might be easy or hard, uh but yeah. So, okay, I think. um So what are the other options?

E

So just assuming, let's just say um uh if that would work the service account um uh approach, because to me so far from a usability point of view- and that was like the original thing we wanted to solve as well is that the whole usability of this integration would currently be very cumbersome.

E

um That would be better because the individual end user of a self-managed instance does not have to go to sales and do that right. So only the self-managed admin would have to do that. um So maybe that's good enough, uh but just to explore other options. I think there was something else being mentioned. um uh Actually was at the point you touched on earlier Nikola, the um yeah.

A

This level token.

E

Is that separate is that a separate concept from service accounts.

A

Or no, it's not I, think the service accounts would or they're like designed to also fit with this instance level. Token I'm, not sure, like I, saw a comment mentioned that, like instance, level tokens will be handled by like service accounts in the future and.

B

D

I saw some comments. There came away.

A

From this uh as a priority because of the organization modeling that like is currently introduced, because this film little change like our top level organizational structure like we don't have organization and then below, we will have groups and products Etc. So I think that Alex poorly commented that in case of sales, we will have multiple organizations, but in case of self-matched, customers actually will have only single organization.

A

So if we use service account on the organizational level, then like this is not a user and it represents actually the instance that we need. So when you create an organization on the, but there are also open version. If you have a group organization with specific permissions on SAS- and you have the same group on your local environment, which one will be used or whatever, but it's not something that I fully wrap around my head.

B

Service accounts also will work at the instance level they're already built that way or self-managed, specifically.

A

So the open question related to service accounts is like how we will manage like permissions and like billing, because I guess the self-match admin don't need to create a group and probably pay for AI feature or something. So we will need to move it and give him some license or whatever we do with open eye stuff or other things that this instance should be approved to use and I guess there are limits as well configured there.

E

So once I con once I create a service account on gitlab says as a self-managed instance administrator. What's next like how do I use that service account for any kind of integration on my self-management, that is, that is something we would have to build from scratch. I assume.

A

So we will get token and then what we always.

E

Yeah, actually is it? Does it work that way like do, I get, do I get a token or how does um maybe that's a good segue into someone mentioned. Odic I'd like to complete uh I, know nothing about all these authentication schemes. I know, oh well, that's as far as my notice stretches.

D

E

So um uh so so there's this concept of a service account uh which essentially, we can think about that as a user. um uh Let's say: let's call it the Headless user right uh so um uh yeah is that connected to a token and um how would I? How would I manage this, and how does the authentication flow actually look like and how does it relate to oidc.

B

Or service accounts each account can have one or more tokens, essentially personal access tokens and um then that's basically simple off essentially password off, then that you can use with the API um versus oidc, which is uh oauth and it's per user per user um end user. So the the problem with that I think in the in the case that we're talking about here is every single user on that self-managed instance would have to authorize their account and then the integration would be operating as each individual user, foreign.

E

Yeah I, don't think we need that right. I want that even I, because um yeah.

D

Because my understanding is that.

E

It's fine that a user! um Let's take the example, the specific example of code suggestions right so I'm an end user of a self-managed instance. um You know an employer at some company that uses gitlab uh and um I want to hook that into up into my IDE, so I connect the self-managed instance API endpoint for code suggestions into my IDE um and I use that using a do I do that using a personal access token issued from the self-managed instance right. That's like the normal user flow.

E

We would also that's what you would do as well, if you were to just use zest directly right um so uh yeah. So as far as that goes, um but that's only like the first leg of the of the authentication Journey, basically right. So then my IDE sends a request with a pat to the self-managed instance and then, and that's the bit like where we probably have to build something from scratch uh that uh that selfish instance would then have to detect okay. This is like a proxied endpoint um and I I.

E

First, first I authenticate the user with the pit like standard stuff uh and and if that checks out, um I basically have like a pre-authenticated request. So this instance can then make a request on behalf of the user, but using these credentials right using the service account, and that then goes to get that says right. uh So so that's how I guess we had imagined it to work so, but we would still have to then um yeah make it transfer completely transparent to the end user.

E

That then this extra handshake happens between the self-mesh instance and and gitlab says so. Ideally, this would only be hooked up once by the Admin and then it would just work. You know uh outside of any token exploration and whatever.

D

E

Would that would that work or is? Do you see anything like why that wouldn't work.

B

Well, the service account is only going to be able to operate as itself. uh It will not allow the integration to operate as uh other users.

E

I think that's fine, I I did I, don't think I meant to suggest it. It should um I okay. So it's we should clarify with Roger and uh the other PMS.

E

If that is okay, but uh I would imagine that um the end user shouldn't have to worry about any kind of I, don't know rate limits or whatever permission limits that were put in place between the self-managed instance, which is our customer right, because because the end user, using that in the IDE they're, not our customers, they're just users of the self-managed incidents and uh whoever runs that uh basically pays US money right to um access these Services.

E

uh So so yeah so I see these as completely separate right and that might be separate permission checks for the end user, um who is actually hooking up the self-managed instance with their IDE. But I would see that as like a separate thing, uh yeah yeah.

A

Because uh so, basically, like we mentioned before, we should consider the self-match instance as something that have I know billing flat billing and it has its own permission to access. You know uh open AI features and probably has some specific rate limiting related to the instance level so like it can request, 200 requests daily or something like that, so not user, specific limits or whatever. So when we create a token or authenticate this instance, everything that is related to this gitlab SAS is based on authorization and authentication related to that instance.

A

So we don't care about their users. So how we imagine this will work. Is we don't like the bottom line of the solution? That we need is that their users shouldn't be aware of any handshakes between self-managed instance and gitlab.com, so admins self-match admins can do uh like one-time handshake and do a configuration, create groups, ask our admins to give them license or whatever is needed once it's configured and we get a token or whatever it is.

A

When user is trying to access some URL, that is proxy Netflix for some API call like it's called suggestions, AI or something else. We should authenticate that user on the self-manage level, like they will authenticate users like and if everything checks out, they will send a token for that instance to the gitlab.com and then gitla.com will just authenticate that service account, or instance, level token or whatever check permissions.

A

If they have paid for AI, they didn't reach the throughput limit of 200 requests per per day or something like that, and we will proxy the the response back to the instance and the instance will handle whatever is needed to be handled.

E

So I mean it creates some interesting scenarios right where, if my self-managed rate limit for API access is greater than what the rate limited might be tied to the service account at the instance level, uh between SM and SAS, uh then it could lead to weird user experiences where, as a self-managed end user right, I I go to the you know, API description or docs on the self-managed instance and see okay, like here's, how many requests this API should be able to service using my personal access token, but if that is greater than the amount of requests the instance is allowed to send to says.

E

Via this mechanism, they might run into tighter limits right before they exhaust their own quota, which might lead to weird user experience, but maybe for an MVC. It's not that important. They would basically have to take that problem to their admin right. Whoever runs that instance to make sure that this is all that's all.

A

It works out, I guess for first few iteration I already saw some discussion related to this, like it should be here, probably similarly to lecture GPT. You will get a message like system is currently like busy weight, and if you pay extra money for super access in the future, we will handle it that you have a priority or, depending on the plan that you are, but this is related more to the open, AI features and their integration.

A

But I guess this currently should be fine, especially if you are talking, on instance, level rate limits or everything else. So I guess like if we are looking for a solution. The first priority is that ends users or self-managed customer shouldn't be involved in any configuration, integration and Etc, and the second one is. We should look it as like. We need to double check and confirm, but I guess, like it's a flat level, like all the uh permissions and like limits are set to the instance level and instance, will handle I.

A

Think that I wrote that here, more or less so instance is responsible for their user level. Authentication, authorization and gitlab.com is only responsible, for instance, level, and that's it. We don't care about their users and how they will use that.

C

Yeah I think that's also true for the rate limiting just you mentioned, because you can Implement rate limiting per user. But let's go to the set managed instance, because in the integration on.com we cannot validate the rate limiting per user.

B

We can only only.

C

Validate the rate limiting or check the rate limiting for the service account.

B

Yeah, that's something we might have to think about on the getlab.com level, though, because we might need a larger rate limit for the for whatever particular proxy endpoints, um but also I'd, say in general, most self-managed instances don't have rate limiting um some number of years ago. We disabled that by default in Omnibus, now I think that you all are also talking about gitlab dedicated, which um that I can't speak to you know we.

A

May be that's completely separate problem that will probably knock out there, but.

E

A

Also unanswered question related, so I will not try to talk about dedicated now, because there is also a Geo problem that where there will not send the data to the other server, especially if it's based like self match, customers is based in New AAU on in Europe, and they are sending data to us, or this will be centralized their problems.

E

Yeah and I think for the embassy. It sounds like it's okay to ignore that I think it was Dylan who responded as well that, um for instance, it's now kind of it's an opt-in thing anyway. So, if you don't want, like your data, you know to penetrate all these different boundaries, then just don't use the feature and.

D

I specifically, as.

E

Is that is that is that good enough- and you said yeah, that's good enough, so so maybe we can just yeah um put a pin in that.

A

E

One more follow-up question because uh and I don't know: if I get the language right and I, don't know which teams are responsible for what exactly, but um all of these third-party Integrations, for which I would use the service account.

E

um uh So if I, if I just look at it again, if I go to gitlab and then I go to the group settings and there's a tab called Integrations yeah, which for me seems to be read only even in my deaf instance, so it lists a bunch of things here, like Asana, adolescent, bamboo, bugzilla and so forth.

E

um Is this like, under your team's purview as well or I'm, still trying to what I still don't understand, is even with these service accounts in place like where would I actually as an as as a self-managed instance? Where would I actually uh connect gitlab says? Is that called an integration and- uh or you know, would that appear on this page or how does that all fit together?.

B

Great question I'm not sure who owns that I, don't believe. It's us there's.

E

A group called Integrations.

B

Like yeah yeah, that could be um maybe that's a good location for the self-managed side. You know, maybe that's where you build that piece, so you ask you know where, when I get the token from jlab.com or do I store that on self-managed, maybe that's where you fill out that that um token piece.

A

Well, just to summarize Matthias previous question so, if uh instance, uh like these service accounts uh in case of service accounts, is it going to work like the way we described without authenticating each user? Like? Can we just proxy the token that we store somewhere once the service account is created without like additional hustle, with asking each separate self-managed user to go through process of like because.

B

A

Don't want that okay service.

B

Account is just a flat, it's just a token personal access token, um and it has no um no per user.

A

So basically, service, like self-made admin, will go, create somewhere like some group and create service accounts get token, and then we will need to store that token to the self-managed, and there is additional confession like who will put that admin in some group and give him a license or whatever but yeah. We will then get a token and we will store that talk in somewhere and we will need a place on UI there, where we will allow that admin to do all those stuff or connect this token with their own instance or whatever right right.

B

Yeah I think you know longer term. We need to talk more about. Where does this service account get created? You know, is it? Is it a group or do we do something at the instance level? Maybe there's an API uh between the the customers app or whatever it is we use for subscriptions now, where you know when we have an ultimate customer? We generate that token. That service account automatically and present that token. There ideas well.

A

Looking at this epics of organization, modeling Etc I saw that they plan to enable, like default, one organization on the instance level like for the self-matched customers, so they will always have one organization that cannot be deleted. So I guess maybe.

B

This is that I would say is currently highly contested. Discussion. I would caution us against relying on that yeah.

A

Right, thank you, okay, and until the service accounts are like, can we use the service accounts without UI builds like at the moment, or we should like continue talking about the proposed workaround solution. That is a little bit hacky, but still would do more or less the same thing using open, ID protocol.

B

We could, uh if you want to start testing, we can help you create service accounts in GDK, just a few commands in the rails console and then uh smarty, maybe your best to get an update on this, but either this Milestone or next we plan to have a minimum UI available.

D

Yeah we are still discussing on the how to uh how to get done with this feature, with minimum security implications, because it's it's gonna, be a very powerful access Tok. Without that, so probably this release, or maybe the next we will- we will be ready, we're gonna. Let me see.

E

Sounds great, uh Nikola I think we don't need to be blocked on this right now, though, I think we can just for now. um I know, that's not ideal, but just to uh I would like to have a POC like as soon as possible, like just something working, even if it's not ideal and for this I think we can just assume for now that a self-managed incident that admin uh would just create would just use it normal Pats.

E

So they would use their personal uh access token, because that would unblock a lot of this work already, because we could, just because whatever token or mechanism is used for authentication at the end of the day, there's still a lot of stuff. That will remain the same, such as, like you know. How do you even set up this integration.

A

um Exactly but yeah I would avoid creating our own implementation of open, ID, blah blah blah.

B

A

Tokens for the POC just to see if the other parts of the solution works, yeah.

E

And it sounds like something we can do in parallel, like.

A

Yeah so and then, when service accounts are ready, we can just.

D

A

B

You can you can definitely use a personal access token today and it's going to be uh functionally equivalent. I think there is going to have to be development on on both sides, though something to configure on the self-managed side, but then something to connect because, like I said, a service account is not going to implicitly have any permission to use these AI services.

B

A

Yeah but I guess for for personal access token, it's the same like the admin will need to create account and someone will need to move it to the proper group with proper permissions, and then we just copy is personal account talking to the self-managed instance somewhere. This story.

E

Well, I think yeah that I guess that's the yeah, that's kind of the the big question we still need to solve right, I think we might have to talk to a different team as well Nikola, because uh so you, if you once you, create your service, account cool. Then that exists from the perspective of says, and it says something like: okay, here's a token or whatever, uh and um but then then what right? So you still you.

E

You still need a way as a self-managed instance uh uh admin to create an integration with SAS, only you know and and that needs to be stored as a record somewhere in the self-managed database. Right.

D

E

Going forward any uh request can resolve against this integration point and then, and then we can, you know handshake with with says uh yes, so this is something we need to figure out, so maybe first The Next Step should be um uh to find out um how this should be done, like it sounds almost like you know, says, will just be another third party integration right and from the perspective of um the self-managed, no.

A

I can I I guess similar to the proposed like alternative solution. We can do it two ways like that. You entered the URL of your submers user and but then, like some synchronization will happen, or vice versa, like in you configure on yourself match customers that you will. We will need some API to return you that token or something but.

B

For sure, even just a a new column in application settings, I mean that's what we've done with with open AI. The open AI key is just an application setting column, and so maybe this is just another one. Gitlab.Com SAS integration.

D

E

Could be a very simple way of doing.

D

A

D

A

Mvc we can just store the manually, the personal look accounts token and that's it for that admin.

E

Sure, but you need to like the admin needs to like how does he do that? Okay, so I think this is we don't need to discuss this now.

A

Yeah I'm just talking about us testing it without delivering to the self-managed customers. So right.

E

I think we can.

A

Focus on other things and then solve this.

E

I think in interest of time you should probably wrap this up, and we have another meeting in five minutes. Should.

A

D

Let everyone go and.

E

Then we can, we can see where we go from here. Nicola yeah.

A

Of course, of course thanks, this was really beneficial to better understand where we are because we don't want to create additional mechanism that will then be tossed away and we just lose time on it. So I guess we are in the better spot. Now, thanks everybody for joining thanks.

D

B

A

Recording from the meeting and thanks again and have a nice rest of the day,.

E

Thanks, thank you. Bye thanks.

D

Thank you. Goodbye.