GitLab Distribution Team Demos, 29 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Distribution Team Demo - Running letsencrypt functional tests for omnibus-gitlab

Description

https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4359

https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5158

A

Hello, everyone welcome to the distribution demo for april 29th 2021.. Today, I'm going to be talking about uh the story. I've been working on about automatically validating, what's encrypt uh integration for omnibus, um and so this is the story that I'm looking here on my web browser on the left.

A

This is an issue that I got open because um our acne client that we use in our cookbooks um for doing lots, lots and stuff got updated, and since we only had specs that um chef specs to validate behavior, uh we ended up pushing a broken, changed production and finding out about it later so um I mean with yeah so that that's the impetus behind this issue um a little bit more about the the current state of things before this. Mr, um so I guess that we did.

A

We do have a let's encrypt spec, I'm looking at here on the right, um which is good, but uh specs are really only helpful for chef when you're trying to understand how attributes are going to be modifying the behavior of your recipes. If you're doing, if else or you know filtering including recipes depending on you know whether a note is red, hat or ubuntu stuff like that, so the specs aren't really meant to catch this sort of issue anyway, but we do have some.

A

So that's good uh building on that, though, so um the way that the lesson crypt works in the omnibus right now I'll come very briefly for those not familiar. um Let me know if I get anything wrong here, but basically we have this. Let's encrypt cookbook, um I'm looking at the enable recipe in it right now on the right side of my screen, and so you can see here, I'm 140 for it well. First of all, it generates a self-signed certificate, so enginex will be working and serving on port 443.

A

But then I go as goes ahead and includes this recipe and it's including it based off an attribute.

A

I believe we only support http 101 right now in omnibus, because there is not, you can see, include, let's encrypt attribute authorization and then, if we look in the folder, the only one there is http. So uh that means you know: a service has to open their port 80 to do the validation instead of dns. So there might be a future uh enhancement here to enable dns. Let's we've got a nice issue for that one it all comes down to now.

A

You have a random node somewhere touching your dns, so be very, very careful about that. um I love it. I didn't actually realize we just made it self-signed, so we do have that functionality.

A

It's kind of interesting how this flow works at omnibus, because, like first off, we regenerate the self-site and then we go into the http authorization and call this let's encrypt certificate resource, and so, if you look at resources certificate, that's what this is calling, and so this requests a less encrypt certificate from the staging endpoint to make sure it works and there's a comment here.

A

We're kind of just helping customers not get themselves into trouble with the rate limits, because if you set up your domains and you set up your configuration- and you run this and it doesn't work- you only hit the staging endpoint- you didn't hit the let's encrypt production endpoint, which is going to start rate limiting you.

A

So I guess, while customer or users of this and figuring this out, this block is here to help them not get into a situation where they have to wait another day to try to deploy gitlab, and so it generates the staging certificate.

A

It resets the private key for acme, which is required to do this again, and then it requests the production certificate which is then used. So that's the flow of it was first looking into this issue. um We had hard-coded well, uh there were a couple of things that made this difficult. So for one, this dur argument is actually the argument for the acme endpoint that you want to hit. um Did I say acme endpoint instead of let's encrypt endpoint, because let's encrypt is just an implementation of the acme protocol and they're.

A

Actually, uh I don't know how many, but it seems like more every month, um services that allow you to get acne certificates that are not that's encrypts, so I I remove the hard-coded values here and move them into attributes that default to let's encrypt endpoints.

A

But I you know this is not the point of this, mr, but this will allow our users to also use those other acme endpoints if they want to, for whatever reasons does that make sense before.

B

Sounds like you're knocking out a couple feature requests here.

A

Well, I mean it's good for testing and it's good for users, so I think that's pretty good. um Okay. So now we have the the endpoints parameterized. Here they default to- let's encrypt, as I said, but for testing um we don't. We don't want to use the production. That's encrypted, endpoint, of course, because of weight limiting, but using the staging. Let's encrypt endpoint for testing is actually not so great either because it also has rate limits, they're higher than production.

A

But so there's a couple things going on here, like there's the rate limits and this failed validations limit 64 hours, probably the one that would hit most people, because the other limits are pretty high. But um you also and I'll get into this more. When I talked about like using pebble and our own acne point for testing, but you can't test it will it will identify you as malicious if you are trying to do tests with the control flow of what's encrypt?

A

So, if you're like um submitting invalid nonces to the endpoint, it's gonna notice that, after a little while and block your ip, so um staging isn't really a solution here either.

A

So what is the solution? Well, uh this was something that mitch actually brought up. I had not been aware of it, but let's encrypt has this project called pebble.

A

It's all up on github, and basically this lets. You run an acme server or docker container, which is great for us, because we can we test github omnibus in a docker container as well, for like qa tests and stuff. So linking those two containers.

A

You know it removes the rate limit problem and it also opens up a lot of possibilities for us to do testing um based on control flows which I'll get into in a couple minutes here.

A

um Okay, so we've got pebble and then we've got this other tool called uh chao test serve, which is not the most inspired name, but it's a challenge test server, and so I'm going to open up the docker compose here. I guess I already have it open. So we've got three images here. This is this: is the testing docker compose and so for the first image.

A

This is the gitlab ee image and then we've got pebble which is configured and then shallow test server, and so the child test server allows you to not only mock dns but mock challenges um to the acme endpoint.

A

And so these are all working together, um because you see here to pebble we're passing the dns server argument, which is the child test server. So pebble is talking to that for its dns, and so when we're running tests, um we've got this function here called start hubble, which starts pebble in the challenge test server and then, once those are up it posts to the challenge tester, where this mock um a record.

A

So we're saying gitlab.example.com, which is what our host name is for the gitlab image um should point to that ip address, and so this is how it there's not. As far as I can tell, there's not a config file for this shell test serve. um You just make api requests to it to set it up in the state. It can do a lot more than we're using it. For here I don't like with a lot of this like infrastructure and tools.

A

The pebble channel, tester folder, a lot of it is, was built by let's encrypt to test their own services and so we're just kind of using smaller parts of it. There's this other tool, they have called boulder, which is bigger than pebble um that it's it's basically like a whole application like you need to have. You need to stand up several services that all talk to each other.

A

um I suppose, if you were trying to develop your own acne fight or something that might be the way you'd go, but for our test purposes you know business work, as expected. um Pedal is more than enough.

A

Okay, so I'm going to run this in the background. While I talk about it a little bit more and the test, so one issue that I ran into that was interesting is that um I wanted. I wanted a way in the in the test to run the mommy must doctor image and then, when it was done, configuring just stop like I didn't want to keep running um and so looking into how we do this a little more.

A

um If you look into this wrapper, which is the entry point for the github omnibus docker image, scroll all the way to the bottom, there's this gitlab post configure script which evals anything you pass to it, and so in the test, we're setting that value to exit, and so when the it starts up it configures git lab um it gets here. It just runs exits, so it's not tailing the logs or waiting for it. So I was glad to see that we had this hook in there.

A

Otherwise, we'd have to like update the entry point and add some more stuff to it.

A

Okay, so um I've got the image here, since this is still at. Mr, um it's got my mr tag. um This is just because I added those attributes for the cookbook and they're only on their branch. Right now uh we've got cleanup which we always do and then the post configured and so by default. When you run this, it just runs all the way through it either passes or fails.

A

That means you either got your let's encrypt certificates or you did it, but for development um we can be a little friendlier, and so, if I set this gitlab post configure script just to an empty string and then say: don't clean up.

A

It'll keep get lab up, so we can inspect it and make sure everything's working as we expected um on the left side. Here, I'm going to uh log pebble and the challenge test server.

A

So we can see those calls going.

A

Okay, any particular part of this anybody has questions comments on right now,.

C

I thought it supports ipv6, not us, but the acme stuff.

A

uh Yeah yep yeah by default. We have um so there's a couple files here. This there's this pedal config file, um and so I changed like by default. Pebble runs its challenges against the port 5000 um yeah 5002, instead of a lower value port because it's made for our docker images that are running with higher port numbers, so they don't have to run as privileged, but for us gitlab runs is 80 and 443.

A

So it's already, you know uh it's running as root in the container, so it's not really a concern, but um so I changed this to 80 and then in the chao tester. um I also set up the port to 80. and so um yeah.

C

A

You you can kind of the only thing we have turned on for this test is http 01, but yeah. If you wanted to do other, um you know test ip6.

A

You can just add that as a flag cool, I don't know if that's something that we've seen or uh I don't know how many customers are running, get our users or running gitlab with ip6. That would be interesting right now.

C

Yeah, I'm not sure we do know that I would suspect a small number right.

A

Yeah, um so the only other file here is this pebble mimika.pen, which is um basically we have to mount it into the gitlab container, so that when we make our https request to pebble pebble doesn't do http, it's got to be tls, um and so this makes it so it's trusted. So you can see over here here on the left that pebble we we made our order. This is during the whole, let's encrypt flow, um and then you can see here issued certificate cereal for order, blah blah blah.

A

So that means this defeated um in general. The logs for pebble are really good like I went through several iterations of all of the config here and it told me what was wrong in a clear way every time, so that was nice.

A

On the right here, the only thing that the child tester is it's just saying we added that stub, which is when I made that uh that post, that's the log for that.

A

Okay looks like we're getting close to the end of our run. We're authorizing rafana.

A

uh While I'm waiting for this to finish shopping to talk about um this, mr that has all this is currently in draft, but the last thing to figure out is really just running this nci, and so I have added this trigger. Let's encrypt test right next to the qa, because it needs the get lab, docker image to be built, and then you pass it to that.

A

um So I'm just working on some some issues right there with like passing the image name and stuff correctly, but once we get that set, then the plan is to get this merged together um and then.

A

Not like uh well, I mean the ultimate plan is to get it running. Let's get get up qa. Instead, we've already got some good examples for how to do this, like for testing and gets matter most, we spin up a separate container. I think- and so this would be somewhat similar to that and that way we can have it running as part of the gitlab qa suite instead of just kind of a one-off functional test.

A

Okay, it looks like git lab is running now, and so, if I go to https localhost 8443., um of course this is not a ca signed, um but let me just set a password click.

A

A

Okay, well, you can see. Oh we're explained there and then, if we go through and inspect the certificate here um view certificate, you can see it is issued by pebble intermediate ca. The alt name gitlab example.com, by default. These are issued for five years, which is, I think, kind of an interesting choice for a test server. But I'm sure that's configurable. If you can.

A

Okay questions comments concerns, that's really most of it here.

B

So you're showing us how you're running it locally was there anything different to how you were trying to run it in ci. Yet.

A

um Well, for ci, I'm running it the same way, I'm cd into the folder and just running that test that shell, the part that's unclear right now, is how I get these image and tag variables, um just because uh it seems like qa gets them without having to know, like maybe there's some logic in there getting the variables the right way, but uh yeah, I feel like once we get these references figured out, it'll just work.

B

Yeah one thing to consider- and I don't know if this will be a good idea or not, but uh instead of using docker compose, we could try just using um the built-in gitlab runner services.

B

So like this in your cin. I don't think we've done this before, but we'd have a job where the main image is our um gitlab docker image that we want to test, and then the two service images would be, one would be pebble and one would be this. uh The certificate handler yeah, that's interesting and you might get away with how docker composed. Perhaps I don't know I we.

B

We definitely haven't done before, like exacting into the omnibus image other like using a script rather than the wrapper, so it would be like you'd be the job itself would be the the exact script and you would probably run the wrapper, but then also run your test in that exact script. Thing.

A

Okay, yeah, I get what you're saying uh I mean that would cut down yeah. I I mean.

B

It kind of sucks it's it's more, I'm not sure if it's super helpful for this, it's more like I don't. I don't know if that's a step closer to what we'd end up doing in qa or not, I'm not sure.

A

Yeah right yeah I mean we could when I, when I was working up this test script I thought about you know we could just run all of this in docker as well to get rid of that dr compose dependency.

A

But then I started to trial out doing that and having this all in one place and readable versus a bunch of doctor args like uh yeah, I think using gitlab services would be an improvement, but I think uh going from documents back to docker like makes this a lot harder to read.

A

Okay, anybody else.

A

All right, fair enough: well yeah, I'm uh hoping to get this. You know once we get this image out, we'll be putting it up for review and then once we get this started then we'll be moving on to uh integrating into qa, and I think that will be the point where we uh might want to look into doing it with services.

A

I I'm not sure how how that's going to look right now, as I'm still learning about how qa works, but um I guess one question I have is: are we doing this for the charts like do we have? Are we validating, let's encrypt this way, with the charts.

B

uh We are not okay,.

A

Should should we.

B

B

B

We should probably should probably set up uh yeah. We should probably set up a let's encrypt test. We don't have to yeah. It would probably be very similar just swapping out the uh like booting up, pebble and swapping out the um staging directory. I guess it wouldn't be staging would be in the case of our charts. We only set one.

A

Yeah well, I I could see that you know being a benefit of moving into qa. Is that, instead of doing it one way and on the bus and one way in the charts, perhaps we could get it set up in qa that it will run against both and validate the same behavior?

A

um Oh one thing I noticed while working on this um is, I don't know if it's a big deal for us, but uh so I I I mentioned the workflow stuff briefly before um as far as the acme workflow, with the requests back and forth, and the nonces and um authorization, so I had to turn so by default.

A

Pebble turns um a few things on like um to test the workflow more specifically and they're, not saying that all of the acme clients out there are implement this right now, but they're saying they should, and I had to turn some of these off for the acme client gym that we're using in our cookbooks, and so this just includes things like making sure you can't do replay attacks against your acme server.

A

um If you get your hands on a challenge, nonce um or sometimes it will like uh sleep for a random amount of seconds between 0 and 15 before it gives you an answer and then the client is supposed to be able to handle that and do retrains and all that, um and so I, when I first was going at this, I just had everything on the defaults and it didn't work, because I had to turn off that sleeping and then I had to turn off this replay once so.

A

um I I'll probably open another issue to do a little investigation into this but um yeah. Ideally, we wouldn't have to be turning these flags off because they are meant to protect uh the acme.

A

A

Okay, well, it looks like we're almost at time here, um thanks for watching the demo, uh if you have any other questions, feel free to hit me up after thanks dawson yep. Thank you.