GitLab Distribution Team Demos, 21 Jan 2020

Previous Meeting

⏯

youtube image

►

From YouTube: Distribution Team Demo - 2020-01-21

Description

The distribution team discusses how we might introduce automatic oauth applications for gitlab pages and grafana into the gitlab helm chart.

A

Okay, so welcome everyone for distribution demo meeting on 21st january 2021.

A

uh I'm essentially gonna use this demo time to discuss on one strategy to implement uh automatic, auth integration with our charts, so this is required to enable uh integrate using gitlab as a identity provider or what provided for different apps that we ship, uh like gitlab pages for access, control or graphana for gitlab, sso and method, most working levels also they in a nutshell the problem is, uh we need gitlab to be up and running for us to register an application s and or the application.

A

uh At the same time, we need these credentials to be present in the service, whether it be gitlab pages or mathematical graphene. We need these credentials to be present in their configuration so right now we don't have a way to intervene between the rails, spot, spinning up and other service ports. Spinning up so we cannot like, after the rail spots, have spin up create this application, then put those values in configuration file then spin up the service ports.

A

We we don't have such an option right now, so we need some way to uh inject these variables somewhere. During the run time and ensure the applications pick up these changes, so uh I tried something and it seems to work.

A

I have linked it in the demo notes: it's charts mr1782, uh essentially what it does is we tap to our existing system of where we run database migrations when we spin up a cluster and we at the at the end of that migrations job, we kind of register every application listed in the values file as auth applications directly hitting the rails console so.

A

This has two parts, one in cng, where we need a new script which will essentially do this like run certain commands in rails context, and we need the charge side to create the secrets. Put it in there, put all of them in the right place and do everything else.

A

So the cng side, uh essentially is uh we assume there are secrets present in the directory they traverse through that directory and pick each one of them run a rails command to register them as earth applications, and we do this in an item potentially so that doing it repeatedly will not change anything and on the chart side, we provide everything that this script needs. We mount these.

A

We create the secrets we mount them in the expected directories in the expected directory structure, and then we rely on whatever we have been using for migrations or that that setup that we already have.

A

So uh let me share my screen.

A

A

This is essentially the cng script. uh We actually tab to the db migrate script.

A

uh We already have some a custom, instant setup kind of thing where we enable a feature flag automatically during startup so similar to that we are running a new new script with the gitlab rails rails, runner command, so that it it can access everyday rails context, and the script essentially does uh traverse through a directory and uh read the secret or everything. Each of them are stardust files. We read them and we run the door keeper command. So this command is what we use in omnibus gitlab.

A

To implement this feature, we will directly hit the doorkeeper gem or use the doorkeeper gem to create an oauth application.

A

uh What I realized today is we can provide uadn secret and we can choose note to ask door people to create them. This means the job of creating the secrets can be delegated to someone else in. In our case, we already have a generate secrets, job which does this, so we can make that job, create the secrets and pass them on to doorkeeper.

A

So right now, this script is well uh three minutes of thought. So it it it doesn't have any guardrails or anything it expects everything to be in place, but we, we need to add guardrails and everything there, but it's essentially the logic is that much and in the chart side.

A

uh Okay, we in the migrations job spec. uh Okay, before that, uh in the general secrets, we create the necessary secrets so uh for all the applications, we technically need three details: one is a application. Id one is a secret name and the other one is the redirect url, where, after all the indication it should redirect to uh so, uh the secret contains both application, id and application secret, and we created uh it's a.

A

It can be anything we we go with the convention of a 32 character, uh alphanumeric string, so cigarettes generation will create that we will mount them in the migration job spec so that it matches whatever that script is expecting. So uh we mount it to slash, etc. Gitlab or sacred spaces, appady app secret and then the redirect url need not be a secret. We, we can compute that, so we put that in a config map and mounts that also.

A

So these three things uh the script will read app edit app secret and redirect the uri, and it will use those details to create the doorkeeper application.

A

So this is the premise uh there are few things we need to discuss and decide. One is uh where should these settings be placed so that if users want they can overrate them like, they can not depend on our general secrets but provide their own secrets and two?

A

How can we hook these to other applications, uh method and graphene, of which we are using the upstream charts, and we currently do not have the flexibility of using templates or anything like that to compute all the information we need so in in the mr, I kind of went with a global setting called gitlab auth, to which we can append pages, mathematics, graph and all those things, but one problem is mathematic. Has its own settings structure for this, like it uses uh config.json.gitlab settings for this. So if we are using gitlab dot mattermost, how?

A

How will we, and how can we pass the values that users set here to where matter most expects them, similarly for grafana, so that that is one thing that I couldn't find an answer to following a global way or general way of doing this like if you have to consolidate everything to one way of doing it, I couldn't figure out how to pass information in one json structure that we use to the json structure upstream, charts use.

A

Next, one is our shared secrets: job can only use globals right now we have an issue open for moving into the root chart, so it can use any sub charts values.

A

uh The then the question, because should we wait for that or should we go ahead with using it globals now and when we eventually move shared secrets, job to the root chart, we can do any clean up or refactor we want. So these are essentially my questions, but I'm happy to hear if anyone has any concerns regarding the method or comments.

B

Yeah, I think the uh moving it into the migrations job seems like a logical uh place to me. I think, on the cng side, it would be ideal if we uh were able to reuse the custom instance runner um and specifically the custom instances uh like rails environment. So it doesn't have to take that extra time to boot up the environment, but that's the the only call out I have on that side.

A

At least so, uh once we open up the rails context, let's reuse that right, yeah, yeah.

B

Okay, basically.

A

B

We can use that same yeah. We can get our this script also into the same uh script that the the rails runner is using in the custom instance setup. Then we would shave off. You know 10 to 30 seconds.

C

B

Extra important, because this is going to run every single time the migrations run, even if there was no migration. So every time you do a healthy soul, yeah or a helm upgrade right. There's.

C

That um the other question is yes this. That is the right place. We would put it in the current format if we have this kind of behavior in two projects, should it be in the rate code base versus in omnibus and in the charts.

A

So I thought of that. One of the problem is it: it creates two sources of truth for applications. It can already be done via the ui. If we add some sort of uh okay here is a list of applications. Do this also, uh we will go to the problem of like what we have with initial root password.

A

uh We can't do that because we want this to be applicable everywhere, not just the first first run or before the database is seeded, because users can choose to flip access control anytime or we will be adding a second way of setting things like not just on the first database sheet, but always look at this file or this location to automatically register applications. So.

A

Generally, I am not a fan of two sources of truth. For one thing it in my mind, it should be either ui or programmatically.

A

So that was my concern for not recommending changing anything in the rail.

C

Okay, um I see where you're coming from. I would point out that that is exactly what we're doing. By putting this script in place, though, um the script is not limited, specifically to the grafana that we distribute or the pages that we distribute it is set up currently in such a way that it will just read that directory and then take every one of them and then go do the registration for that in theory, what happens if somebody alters that registration say?

C

Somebody goes in the ui and changes the app secret and then migrations job comes around or even somebody just runs db migrate or runs custom, instant setup script and triggers that and then reruns this job. Basically, will that overwrite things, or will it behave in such a way that if the database already has the entry don't touch it.

A

uh So right now, if some, you cannot change the secrets directly from ui. You have to be in the console to do those.

A

uh Even then, I'm not sure if it takes an update or just you can delete and recreate, but the faster create method essentially checks for whether an entry already accessed with these settings, if not create them.

A

So what we can do is essentially split this into two first check for an entry with this redirect uri and this name, if not go and create that that will be an additional check because uh maybe not all four of them will match. If someone has changed something somehow but yeah, we can ensure almost item potency.

A

A

My my main concern is essentially this: for pages we control everything. We can tell it to fetch things from a template or compute it. Whatever way we want, but other apps have their own way of doing this. We, how do we hook up our convention with theirs?

A

We can always do this per chart or per service instead of going with this uniform approach, but I wanted to exhaust all possible ways of doing this uniformly before we go. That way,.

B

Yeah, so for other for the other charts, where we've had to do this, which is um things like uh the postgres chart. um Basically, we've had to request changes of the upstream charts to actually um like, typically, we haven't had them change their values. What we've had them do is, uh instead of rendering, like the grafana.ini.off.gitlab directly into their config map, we have them, do it via a template.

A

So we can override.

B

That yeah and then as soon as they do it via a template yeah, we can override the template to introduce our own logic yeah. But that is not not the case. I just checked. It doesn't appear that that's the case currently with the group grandfather.

A

Is doing it like they have a grafana.ni key and they render everything to their conflict back. They don't differentiate anything there. So yeah. If you ask them to change it, it will not be a simple thing for them to do.

A

B

Well, so it looks like they loop through uh they do a a range loop.

C

B

Possibly we could get them in like an extras. Param and the extra spram uses a template potentially.

C

Right then, we also at the end.

B

C

Of actually reaching out to another provider that we use bitnami um and asking for their input on how they're dealing with some of these things, if they're dealing with some of these things, so while the grafana upstream grafana grafana, is where we should look first, we do have the option of considering actually doing something different.

C

If it's truly problematic enough, we can go to the point of considering forking, but I say that with a lot of weight behind it, considering forking, because we need to have a good reason to not be following upstream, um I'm not against 14 for short periods.

C

Do the things prove them out get upstream to adopt them if possible, and then unfork.

C

So now dj did mention, and I'm seeing this in the the link that I provided is. They do have a way to basically provide it via configuration and then have it populate in the problem. Is the extra secrets mount and how that functionality actually works.

A

C

On how that works, it's not something you can actually template. um There are some implementations that actually allow you to basically include a template into your values, and then it will render the template into that section.

C

It's a very touchy behavior. That's super easy for people to.

B

Break for the grafana one uh gerard, maybe you remember, did we did we encounter something like this? Did we get around it by um basically creating our own config map? Obviously this doesn't solve the uh the mounting problem, but did we for grafana? Aren't we already creating our own config map and just pointing grafana at it, so we're not actually using theirs for.

B

A

um My memory is fuzzy on that right now,.

B

A

I don't remember creating a separate config map.

B

For yeah, I'm just trying to think of.

A

B

Already uh wired it up to like our prometheus.

C

To render several uh a data source config map and a secret import.

B

Yeah, so shouldn't the same sort of thing. Work in this case then as well, would be able to do something like that.

C

We may be able to we'll just have to look in into what is actually required to do that in order to get this kind of behavior to function so we'll see, it's definitely gonna take a little more uh look into things um and we do need to reach out to see what's up with matter most, um it's still a scenario right now, where we don't include matter most. We don't currently plan to include matter most and we need to see the state of their chart.

A

Question, if you don't do any of those, should we be thinking of providing out of the box or what, for matter most, because math atmosphere is not coming out of the books? Why should oauth be.

B

Right was was part of the issue for this creating oauth for for matter most uh there.

A

C

No there it's just larger figuring out how to do oauth registration because we we need it for pages security, and we need to do that to be able to directly link oauth for grafana through the gitlab incidents.

B

Yeah, so on the the uh the mattermost side, as long as we can like, we just being that we're not including it directly, people have to install it separately.

B

We should just have them a wave that you know they can provide it it both to matter most and then they also have to duplicate that work and provide it to us as well.

C

Right, like I said, we just need to look into it, I'm not as concerned about making sure we have that in in play and entirely viable here. Our biggest concerns are pages and grafana.

A

A

So if we could get the refiner to support something like uh extra graphina.ini like for for some way for us to provide this thing uh technically, that might solve it for graphite and pages. But the problem is the next step may have something entirely different.

A

A

I'm not sure how how how much work it is to maintain this uniform way of doing things whenever we add a new application to the mix.

C

Well, we don't necessarily have to have it be perfect behind the scenes, and I mean if, if that were the case, then these charts wouldn't be this complicated.

C

um What we need to do is to have a clean interface for the user to be able to actually configure these things, a known state of what will happen for those things that don't already exist and configure whatever is required to get the application behind it to actually do its job.

C

Like the the current state of how we're configuring grafana to consume the prometheus as deployed with the dashboards little that we have, you know that's it's, it's not perfect, but we've made it work in such a way that we don't actually have to alter the upstream chart. In order to get that to function, we control gitlab pages. We can do whatever we need to get that to work, but that is honestly designed to be a very straightforward chart.

C

It's just the application and there should be very little of anything we have to configure to it besides. Here are your secrets? Read this file? Read this file read this file in theory, grafana. We need to get it to that same point of here's. Your secrets read this file. Read this file read this file. How we manage to get from this is the configuration map to now the configuration. These are the configuration properties, and this is the end result. Everything in between is a certain amount of flexible.

C

Every oauth needs the app the app secret and the uri. I mean there's three basic components that are needed there.

A

Okay, so, uh okay, that that uh I'll, I look a bit different with. But uh are we fine with the general idea of hooking this up to the migration script or that part of the life cycle.

C

So I'm generally okay with that, um when using shared secrets that will function, fine, what we have to watch out for is.

C

One we have to clearly document this two. This is something because we're putting it into get lab rails, it ends up being usable by migrations and task runner and web service and sidekick.

C

So, in the event that somebody has migrations turned off, then we need to document how to run this script by hand or trigger this script by hand from the task printer pod. Without the use of the db migrate script.

C

Because they may not want to be running migrations from inside the cluster, so we still have to make it included and important. But.

A

Individual okay, that up.

C

I am happy to see overall, though, that we can actually just preemptively go. Oh, the database is far enough up for rails to access. It.

A

Yeah, I I didn't expect doorkeeper to accept the password and secret like id and secret. I thought it had to be created by its own.

C

But no right that that would make certain people I know in the infrastructure as code block happy, because you can just read all the secret from the file and assure it's in place so that works out well for us. Thankfully,.

A

Okay, we are at the out of time, so uh essentially my next steps will be to think more about how to hook this up with graphara and essentially, uh how can we do this with minimal changes both on our side and upstream side.

A

And then cleaning up the script and or everything I have in this, mr, I have to clean it up and ensure I cover all the edge cases. Does that sound good.

C

Yeah sounds good to me. um I expect this will take probably the better part of a week to get through everything and the amount of items that'll have to be touched for it. So don't think that we expect this light tomorrow.

A

Okay, in that case, let me stop shopping, cool, okay, uh I believe that's it. Then we can wind up the meeting unless anyone has any more questions or comments.

A

C

Thanks everyone.

A

For joining see you later.