GitLab Distribution Team Demos, 27 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Terraform with GitLab and UBI certificates 2020-08-27

Description

[Orchestrator] Support GitLab Terraform backend
https://gitlab.com/gitlab-org/gitlab-orchestrator/-/merge_requests/58

[Charts/CNG] Add UBI-based version of alpine-certificates
https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1796

A

All right good morning, everybody so we're going to be going through a quick, quick demo of two different uh topics. I've been working on the past couple weeks since I'm on triage this week, the first one will be using gitlab as a terraform back end. I saw it mentioned in one of the shout out channels and realized. We hadn't actually gone over it specifically on our team to kind of show how it works I'll.

A

Do that real, quick and then wanted to go into an issue that isn't fully closed yet, but we're working on adding a ubi version of the alpine certificates container so that we can better support the ubi deployments in the charts so I'll start off with the terraform with git lab example. Here, there's a really great documentation, page on the docs site about how to set it up, so we're actually implementing that in orchestrator. Now this was merged in what you have to do to actually leverage.

A

This is just change some of the configuration in the rc file so instead of back end, gcs we'll be using back end as git lab, and then we provide some gitlab environment variables here. If using this ncis becomes even easier, because most of these are pre-provided for you, so you can just kind of map them like that.

A

So to do a quick demo of this, I just created an empty project. uh One note that came up when I was working with robert on this is that a totally empty project can be used as a terraform back end. It doesn't have to be configured as a back end through the ci, it's actually kind of like container registry or any other kind of artifact repository on this project.

A

You can inject it directly there. So mostly, I just need a project id from here so put that in the gitlab project id in my rc file, and I already generated an access token with api scope, and then this is actually the url. You access the terraform backend.

A

If you want to see what it looks like right now, it's empty because I haven't initialized it yet, but it's just one of the api endpoints that you can hit with the project id and the name of the project, and you can actually see the backend state we'll see that populate here in a second.

A

So these are all generic I've just added a quick password file with some random string. I actually don't have a license for this and then google credentials. Json is from my gcp project and these are all default. I just set a different terraform prefix for the resources that will get spun up.

A

I've already built the dev container locally, so I'll just go ahead and run it, and then I will provision the cluster.

A

You've seen on this before this will look exactly like if you had the default gcs background before but actually looks like there's an issue. Sometimes I have to go. Remove all the terraform back end to make sure that's gone.

A

Just going to recreate the project, real, quick to make sure there's no back-end state, that's locked back there.

A

I actually did this before the demo, but it looks like I have to do it. One more time should make a new empty project.

A

So we'll have a new project id now.

A

Api tokens should stay, the same project name stayed the same. This should be good.

A

A

I'll update this link to use the new project id.

A

You can see here it finished provisioning. So if I refresh this page, you'll see all the terraform state here. So this is exactly what would be stored in your gcp bucket, for example, but all the results are kind of stored on the skatelab project now, and so you can use this project to store your back end. Of course you can store your different endpoints, etc, but that's how we can use gitlab as a terraform backend, which is really handy.

A

I tend to do this recently, just to make sure I don't collide different projects when I'm deploying orchestrator, it's a little bit easier for me to just throw up a quick bit lab project than to click around in the gcp ui and create a new bucket, etc.

A

uh So that was just a quick demo of this functionality. Any questions on this one.

B

Pretty handy thanks.

A

I'll jump into the next one, so this issue was opened a little while ago by hussein, so this one is to build the ubi-based version of alpine certificates. Right now, alpine certificates is in almost every pod, that's deployed as part of the charts and but as of right now, there's no ubi specific version of it. So it's a bit limiting if you're trying to deploy a fully ubi version of the charts. So the first portion of this change is to actually go into the cng and create a new docker file for ubi certificates.

A

So you can see here we just add a new docker file, ubi8 and add a new script for bundle certificates.

A

So what we're going to do in here? The the meat of this is I'll blow this up a little bit. So you can see it.

A

Effectively, the new docker file just starts from red hat ubi copies in this bundle, certificates, ubi script, which we'll talk about in a second and prepares the volume that'll be shared, and then that's the entry point command here. So the script is actually really simple. It's a little bit simpler than the alpine version of it, because there's a lot more dereferencing to do.

A

The ubi certificates container actually comes with some pre-configured cas and certificates set up in ssl certs, so we just removed that all the content in there and then we update ca trust is actually just a script. That's built into the ubi base image that runs pki 11 and injects files into this directory.

A

So what we'll just do is copy them out into the sesl search directory. This is kind of the common one. Ncsl search will work between ubi or non-ubi based images. So this will keep it consistent with how the alpine certificates script will dereference all the certificates into that one bundle.

A

And so to save time, I just went ahead and deployed this. This is my custom configuration for the chart, so this is generic store manager. This is my host domain. Here is where I set the certificate image and tag instead of using the alpine certificate default one. I use the repository and tag that jason actually triggered a build to build the changes that I made here into an image, so I'm just pulling that here and then to do further testing, I'm also injecting a custom caa to show you a second.

A

I just grabbed one from a previous job where I knew how to access it, like, I said, there's a set of custom cas that are actually built into the ubi image, but of course we want to test to make sure that any customers could actually inject their own or ones that they're dependent on to make sure that they can reach resources in the containers uh through that custom certificate authority and then just to pick.

A

One deploying with ubi requires a bunch of work at the outset, because you have to set up your own external resources, for example, postgres and stuff, to make sure that all the resources are available and then just the core pods are deployed with ebi8 tags. So to save time, I'm just going to kind of show you a specific sliver of deploying task runner with ubi8 to make sure that the certificates are correctly passed into that final container in the in the task render pod and to prove that this works.

A

I just started uh our vanilla base image uv i8 to coral, this ge endpoint, just to prove that that cert doesn't work that isn't trusted, and so it won't pull. And so what I can do here is now our task. Runner part of the change in the chart which I can show here is to change the mount point where the custom ca would be mounted because it's a little bit different on red hat.

A

So you can see here in the certificates. Template we'll set a variable ubi to true. If the global certificate's image tag has a suffix of ubi8, which you can see here, it does all of our ubi. Images are appended with ubi8 at the end to delete the difference and then, if ubi 8 will actually run the security context run as user 0., I'm still working on this with json. This feels wrong.

A

I'm going to see if there's a better way you can do this, but that directory that the update, ca, trust command actually touches is only writable by the root user being in the root group doesn't help so at the moment we're setting run as user here. We're gonna see if we can find a better way to do this.

A

Also, if it's ubi we'll set the mount path here and if it's not dbi, just kind of debian compatibility will mount here, build up to make sure you can see it so to prove that that worked. I will open up the definition for the pod for task runner and I'll just search custom ca certificate. So you can see the mount path is set correctly because we're in ubi right now, so it's set to user share. Pki see a trust source anchors.

A

And then here are the three containers in that pod, so you can see here. The certificates image is using our ebi8 and the logs show nothing. This might be something I touch before. We actually merge this change. It would probably be helpful to prove some uh provide some confirmation message that the certificate certificates were dereferenced and bundled into that directory. So I think that might be a worthwhile change and then, in this final container here, the search should have been provided in the bundle.

A

Under so here's the bundle, here's also the trust, bundle if that's kind of a more advanced version of getting certificates, but they're valuable too. So the main bundle is here and our custom ca has been bundled into that bundle. So now, if we run the curl command, it'll work, it's binary output, because it's an image, but we no longer get an ssl authority problem.

A

Any questions on that flow.

C

I do actually have one question for you just related to this security problem.

C

Is there any reason we can't just shown the folder.

C

Like inside of the out the the certificates image, um is there any reason that we can't simply tone it so that we have the ability to do it without being root?.

A

You mean in the docker file.

C

A

Yeah that would make sense to me. I think that would be a good way to do it. I don't see an issue having that directory be more open, especially because it's short-lived.

C

Right, I haven't gotten a chance to go poke. My fingers in I've been a little busy with everything going on, um but the the question basically comes down to.

C

Why is what we're trying to do root limited only, and is there any reason that we need to have that constraint, considering what the alpine or what the certificate image, I should say, really does so I mean, if we can get around this just by effectively going well, we may not run as root. We know it here blip then that should be sufficient.

C

This this the question around the second part of that is: why does it work in alpine and what do we have? That's actually different between the two, um because we shouldn't have to run it as root explicitly, but at the same time it should be able to run as root explicitly um it's not uncommon to have it in the container. That does run as root to do something that makes it happy for the containers that run after it, um but we should bump into something or should have bumped into something from the alpine version.

C

If this was going to be a real issue,.

A

Yeah I have to dig into what exactly the alpine version is doing. It seems like it's mostly just creating links under etsy ssl search, which is pretty open to begin with, whereas the ubi version is actually inserting information under this directory, which is writable by the root user, but not the root group. So the permissions that we set on everything is to run as uid 1000.

A

um So it seems like that would explain why you can't right there, but I am curious how it's happening in alpine, but through point yeah, I think, showing in the docker file would make a lot of sense.

C

So for giggles and since it'll be really quick anyways, um why don't you docker run on alpine latest and we'll just go see what that is.

A

Update ca, search right.

C

Right now, when you just do an outright run, I believe you're running as root.

A

Right, I just wanted to see if I think this is actually a binary, so it's a little bit different. We can't dig in and see exactly what it's doing.

C

C

Effectively, yes, they have a a pre-compiled binary version of what everybody else uses a pearl script for, but obviously um pearl's big. So that's why um it's! Basically it's a very, very tiny one and essentially what it does is. It goes back to the trusted routes that are in the ca certificate, so you may actually have to run. You know: apk install ca certificate to get it actually populate that content, but then it basically goes around and makes the sim links from the trusted roots into etc.

C

Ssl certs our alpine certificate image, then basically dereferences all of those and then adds in the ca and then regenerates the bundle and then just again, it's making sure that everything is dereferenced so that when we take it out we're not pointing to some random place from the file system. That may not be accurate, which is why.

B

C

You can use this image with ubi and it won't break the alpine. One won't break the one for ubi images, because it's placing the entire contents of etcetera, ssl.

C

Makes sense so yeah go ahead and run alpine latest as root, and then we can always. You know, swap to a different, um a different id, because now we can just go to apk. You know, add sure, vi works and see a certificates.

C

You may have to do the um update, I don't remember, don't remember the command to tell it to update its indexes.

A

A

That's happening just gonna grab that start.

A

So the directory.

A

A

There's problem one.

C

Yeah in alpine, obviously it's not going to be under pki, because it's not that, basically, it's not the same place.

A

I know it should be the ubi compatible. Oh yeah we're an alpine now yeah easier.

A

So to work as rude, obviously, but then we can switch to so if you do an lstack anal on.

C

Et cetera, ssl certs.

C

You see this. This is why we do all the dereferencing we just go and take all of these siblings and actually just generate a copy of them actually in the file as a in the file folder, instead of all the sim links throughout the rest of the system. So that's what alpine actually.

A

A

So I think, just in the I think it sounds a little bit safer to just shown the directory went right into for ubi, rather than running as the root user. On that container.

C

Because, in the end, open shift just going to come back, I just mean, like you can't do that.

A

Yeah, it makes sense a little bit safer that way.

C

So that works for me, I feel like they'll they'll complain less about us temporarily. You know changing the the right access on a folder than they will literally going. You can't run as root. Stop stopping.

A

Yeah exactly and the benefit is that we're still compatible between either ubi or debian, based because it's all going out. Yes, it's all starts.

B

Could we could we use the same mount point as well? If we just have the script copy it from the one to the other, at the start, most likely.

C

I mean it obviously wouldn't test it, but.

B

I think that might be a little bit nicer just because uh in our case the images are tagged with uh dash ubi8, but people who are building them them themselves may not be tagged in them. That.

B

A

Yeah, I waver between those options a little bit. It felt like the correct way to mount the correct directory for clarity, but that makes sense about end users building the image, so I can change that doc, file to chone and also copy over into a compatible directory. If running on ebi, I guess that script will only run if that ubi compatible directory exists, because I don't think it would exist on it.

A

C

That should work the the alpine certificate script. You mean.

A

Yeah the bundle certificates. I think what dj is saying is instead of this logic here, where we mount different paths right, always mount user local share and then, if we're in a uv, i8 tag image, and this direct this directory will exist. Therefore, we'll copy the certs out from there into here, so that update ca, trust actually sucks it in.

A

Is that retirement dj.

C

Yeah right because the final output from that is the actual et cetera, ssl certs. That's what'll end it right. So this alpine script runs writes this into that and then that ends up getting mounted to the rest of the containers.

C

So we really don't care about the rest. We care about actually being able to get the content.

A

Out, at least from what I saw in uvi eight update, ca trust does not touch at cssl certs on its own. I have to actually copy out the bundles that we want, because what it's really looking at is like etsy pki ssl, something like that and it's linked into that ssl certs. So I actually have to copy this out into the compatible.

C

Directory right, so that's basically the dereferencing script is effectively doing that same thing. It's going! Oh there's a sibling here. Let me go yank the original instead of having to worry about the simulink whatsoever, so you're effectively replicating that same kind of.

C

A

Cool any other questions on this flow.

A

Awesome thanks everybody. The links to these, mrs, are in the demo notes if you're curious about them and thank you for watching have a good day. Thank you. You.