GitLab Verify Group, 12 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to integrate GitLab CI with HashiCorp Vault to retrieve secrets (via JWT or "secrets:")

Description

This tutorial covers how integrate GitLab with Vault to retrieve secrets. The walkthrough examines policies, roles, and the two methods on integrating with Vault (JWT or "secrets:"). This also discusses the 15.0 update on CI_JOB_JWT(_V2) and what changes to review.

Authenticating and reading secrets (JWT): https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault/
Using external secrets in CI (Native): https://docs.gitlab.com/ee/ci/secrets/
GitLab + OIDC: https://docs.gitlab.com/ee/ci/cloud_services/

A

Hey y'all: this is a quick walk through on how to integrate vault and gitlab, along with some of the changes that are coming in 15.0.

A

uh So before I got started with this walkthrough, I used an aws quick start to launch a hashicorp fault environment, so this is uh located under aws quickstarts and I just spun up a hashtag vault on a new bbc, so some instructions on how to do that. There now from a workflow perspective when get lab ci job is fetching out into vault if it's needing to fetch a certain key key value pair, such as maybe a password to production or staging environment.

A

um It's going to authenticate to the fault environment through jwt, so within your chci job you'll, be able to fetch out any sort of secrets that you need. So within that jwt method I'll be able to fetch a default authenticate. Now, there's three pieces: I'm going to highlight and go in a little deeper here. The the three components that I want to show here is the role policy secret secret. This is pretty obvious. This might be a kb value between staging. It might be a db password of some sort.

A

Now the role is going to be an important part here. This is going to be what is the bounds on what this runner or this ci job can access. So, in particular, there are certain claims available within the jd jwt token, such as your project id. Maybe I want to filter my project, id environment, maybe a particular branch.

A

So this is just a couple that you can filter off of now. The policy that's going to be your read and write capabilities so which you can fetch into your actual kv, and what actually has access to that.

A

So a little further into that. So in my example here I have a staging environment with the database password store, so I can quickly see that this is the commands I ran in here and my policy. I only want to give read access to this particular staging directory and then also within this policy. I'm sorry this role, I have a role called my project staging role, so I will use this to authenticate and reach out into the vault environment.

A

um I have a staging role and a production uh make note of the environment here, so the staging this is the the bound claim that I'm filtering off of and we'll show that in the ci job here shortly.

A

So if I go into my vault instance, um I can go into my kde v2 path. Make note if you're, using the native secrets, integration uh and what I mean by that is, if you're going to be using a secrets, colon and then fetching specific key value pairs, we only support the kvv2 engine at the moment.

A

So in this case I can see I have a staging database password um and if I need to review that, I can do that so my database and then also for my production environment.

A

I have a very simple prod password all right, so looking at that, the other part here is, I have a policy created for both production and staging.

A

So in this case this is just a read-only function and then, if I go into my access, this is the way I authenticate out between both the ci job jwt token and also the native secrets integration.

A

So if I view this configuration, I can go in and look that there is one a jwk s url, so these are going to be public keys available to allow you to decrypt that token, so it communicates back to gitlab.com. There from vault, then you'll also see a bound issuer, make note of this for the v2 environment. This will change so I'll highlight that in just a little bit.

A

So if I go into my pipeline here, so I have two jobs set up one using the raw token. So what I mean by raw token is there is a ci job jwt.

A

This is the original uh implementation that allows to set up with vault, there's now a v2 that allows you to hook into different cloud providers such as aws or gcp. um So in this particular java I have a fault image set up within the script here I have my fault instance so link per year.

A

Out of that, for my vault token, so this is allowing me to authenticate out through the gwt method, keep note that I'm passing in a role here, so I just want to fetch out my staging credentials and then I'm also going to go ahead and retrieve that database password, so you'll see that being able to be used in this ci job now. The second job here is this is what I call our native integration, so the secrets function.

A

This is available on the premium tier or above so in this case, we can quickly see that I have a path that I'm going to fetch similar to my staging I'm going to fetch my production database password make note of here. This is where the the kdb2 is mounted so there's documentation on properly using this and then also between both of these jobs. I have environment staging and environment production. This is where that bound claims comes to be when you're, using, in this case your staging or your production role.

A

So if I go into a pipeline, that's already created here, I can quickly look into my raw token see that perform the commands, and I now have access to my database password again. This is using the jwt login and then doing a vault kv through the vault image. There now make note I did get a permissions denied error against my production kb store here, because I did not pass a production roll here.

A

So make note of that, and then, if I go into my secret state of integration, you'll see if you do that you'll see there is a function on line five here using vault seek resolver.

A

So you'll see that if you are using the native secrets integration in this case, I have my prod password available to me in the c head job.

A

So that is the quick walk through on the integration there. Now a few things on the v2 changes.

A

There were a couple of changes or additions to the the jwt token. So if I go back into the instructions here, there is a jwt payload that is injected into the ci job.

A

So if I look at this payload here, this has an issuer and then also there's two things that are not shown here, but one is the audience. So this was needed for our cloud providers and then also um there is a subscriber. So this will be additional information that you can filter if you're using something like aws for gcp.

A

So if I go back into.

A

Function here so one thing on the v2 change: um there will be an https added into the issuer um and then there's audio also an audience added now why this is relevant um to vault is there's gonna, be a couple changes that you need to do.

A

So if I pull up my functions here.

A

I'm going to.

A

Add a bound audience to my role.

A

So if I take a quick look into my staging role, I can see now that I've added this what'll happen in the air is say. If you try to go and use that v2 token, it's going to say that now, there's an audience provided you need to now include the bound audience. So I'm going to go ahead and write that to this function,.

A

And just double check that that went through all right, so that is now part of the gwt token now. The other part I want to include here is the bound issue. So in the v2 change there is https that is added, so I'm going to make sure that that is included in the bound issuer.

A

So if I go back and run this pipeline again, what I should see on that raw token in particular, is there is um going to be a couple errors that pop up.

A

We'll make sure to add these into the documentation.

A

So you'll see one um in the v2 token is an audience claim. So if it's found vault will say hey, you need to add a body bound audience to that to that policy or I'm sorry to that role and then also if the http is not included in this year. You will also see this code 400..

A

So I should see those in my pipeline here.

A

Yep, so issuer is not included now, if I go into.

A

A

Give that a second to run.

A

So if I go back into my configuration here.

A

I should see my bound issue now has https included.

A

Go into my pipeline.

A

I should see now that the v2 can now fetch against that now that I've added those two properties.

A

Now for the native integration at the start of 15.0, we will start to include the v2 token injected in here. So the only side on your part as a customer is to update that issuer and also the the bound audience for your role. So those are the two changes and those will be documented for you to review.

A

That is all for this walkthrough. Thank you.