GitLab Secure Stage, 9 Sep 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Defend - Kubernetes WAF Enablement MVC demo

Description

Quick demo on progress for enabling Web Application Firewall for Kubernetes

Issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/65192

A

Cool okay, welcome everyone to a quick demo of enabling a web application firewall using Ajax ingress on your gitlab kubernetes cluster. So right today, we'll be demonstrating a kind of the first iteration of this, which is currently in an open, merge request and it's not super flexible and so primarily I just want to show what we've got here so get some feedback and decide if we want to ship what we've got or keep iterating on this a little bit so before I get started, are there any questions cool all right?

A

So let me go ahead and share my screen and I'm gonna use my big monitor. So let me know if this is readable, you probably don't need a well okay. Yeah you do need to read the terminal over here. Is everything clear you can make the text a little bit bigger there that'd be hopeful sure.

A

All right so right now this is just tailing, the logs of my GK and kubernetes log. So none of this matters quite yet, okay, so I've got my GDK running here and kubernetes cluster is the GCP. So let's go and create a project first and.

A

What we're going to do is we're going to use the default template here. So I will just create a node.js project.

A

And creatively call it node.

A

Cool so we're going to initialize the project we're using Auto DevOps in this case, so it's gonna kick off and Auto DevOps is enabled by default, but that means that it's enabled to execute no pipeline will initially start it's kind of like an experience baseline here. So this will be fun. I'm also going to speed things along in our pipeline. So I'm going to open up the CICE configuration for my project and to find some variables to disable some jobs.

B

You do this as a public project, but is there any difference if it's a private project, no.

A

It just means I, don't have to worry about permissions as much as far as I can so some of these screens would be less accessible, but that's about it.

C

Cool Lucas you're doing part of the demo, or is that what we intend to have users do as well. This.

A

Right here is just part of the demo, because it takes too long to run a pipeline with all these jobs enabled so I. Don't I don't need any of these things. I need a container skating, pensee skating, license management performance testing code quality and by defining these here, I could also define these in a pipeline run, but when I run the pipeline now, none of these jobs will execute I left sassed enabled because it takes about 50 seconds with this default one, and you can see at least one of them run.

A

Okay, so we're gonna go into our kubernetes page here, and this is assuming that you already have your GCP credentials registered with your instance, which is a separate process but well-documented I'm, going to add a cluster to our project and again, this is just adding the cluster initially and.

A

Call it test we're going to use this project zone doesn't really matter. I'm. Just gonna choose the one closest to me, so things are ideally faster and get live. A tiff cluster is an important one here. This means that we'll be able to install applications. If we do not do this, then you need to install things like helm or chiller yourself.

A

Okay, so it's going to start cleaning this cluster, as you can see here, cluster provision, worker background job kicked off and I believe this part takes about two and a half minutes. So we'll keep an eye on this page and I'm going to go ahead and open up sidekick. Just so I can see when this job starts running now,.

A

Cool, so it's waiting for cluster creation and then the cluster job is gonna, kick off in one minute to ensure that the cluster is created.

A

We could also just refresh our cluster page over here until l charles test shows up, but no big deal there. You, okay, so next steps here. This currently is all standard process for creating a cluster for your project. Once this finishes we're going to install helm tiller, so we can store our applications on the cluster. We're going to install a runner and the cool thing about Auto DevOps here is that you can actually install your runner directly on your cluster.

A

So you don't need to run a runner locally to execute your pipeline and then once the runner is installed, we can actually kick off our pipeline here which will deploy our application to our cluster.

A

A

A

All right, 16 seconds you.

A

A

A

Okay, you can see my cluster is being created down here, and this is really just to keep an eye on things. You should never really have to visit the cloud console, but I'll just go ahead and open this email.

D

Cuz right now, this is nothing unique to the demo that you're trying to show right. This is just the normal cluster creation workflow. Now the whap stuff is being done right now. That's.

A

Right, yeah cool, but even with the laughs stuff you shouldn't need to use Google Cloud console for that as well. Rachel.

A

Okay, our wait. Cluster worker is fired so any minute. Now we should see our cluster.

A

All right, you.

A

A

Okay, so our in our cluster is created we're using this for all environments, so the environment scope is asterisk.

A

Now we need to install our applications this first, one home tiller, is kind of like the the manager of all the applications, so we're going to stall that that should take 30 seconds or so and once we have helm installed, then we will be able to install all these other ones, which is why they're grayed out or as this beautiful text here describes.

A

Now the modsecurity firewall is currently a it's a supported, plugin of the ingress controller here, so so internally. What we're doing, when we're able in this we'll, be setting a configuration on this application itself, so the current way that this works is that we've annotated the helm chart that runs ingress here to install it with this additional study and what it does. Is it basically copies them on security module into the engine X configuration now we need our.

A

As I said, we need our runner, so we're gonna install a runner here to actually push our coat up and we'll install ingress.

A

um In the future, we can also install que native instead here for a another ingress option, so you can assault either ingress or que native to actually have an your initial well ingress point in to your cluster and.

B

It doesn't matter whether you do the runner first or the ingress controller I. Don't.

A

Believe it does because there those two are pretty separate. um There's things like like Jupiter here Jupiter cannot be installed until you install ingress. First, let these Tribune installs every I'm, also get installed prometheus, isn't really necessary for the demo, but it'll give us a pretty charge. So we can see that and as soon as ingress finishes here, we need a base domain here for our Auto deploy work. So we're going to copy the endpoint generated by ingress into our base domain, and then we can fire off our CI pipeline job.

A

And, as you can see over on the left here, this is our kubernetes log. It's basically a subset of the GDK log here but related to jobs associated with clusters, so it just provides an easy way of debugging exactly what's going on in this screen.

A

So are we installed an application here for helm at the end, helm that went okay begin install installed, the runner installed, ingress installed, Prometheus and there's our interest in point, so we're gonna copy that endpoint click here and we're gonna use. Nif I/o for those unfamiliar nip I/o is basically like a it's a public endpoint that someone made that just provides a wild card. Ssl cert you're on two different IP addresses, so it can just be a convenient way to do. Http over random I peace.

A

Okay, so we've saved our base domain here. I'm gonna copy this for use later. Sorry, there is a lawnmower near me, so hope, that's not too loud and we're going to our CD configuration to run our pipeline.

D

And you're running this initial pipeline Lucas just to get the app deployed initially correct, yep.

A

A

A

A

A

Now the initial implementation of this to at least with the merger request, that's currently out wraps the additional changes to the ingress helm chart in a feature flag. So currently we have this on a featured leg in the merger quest, and that means that it's currently off by default, but a user could either enable or disable that feature flag to enable the Z deployment of ingress with mod security enabled that said, it's a feature flag around the deployment of that chart, not the actual feature.

A

So if they slip the feature leg after they deployed, then it's not really gonna do anything unless they uninstall the app and reinstall it right and this production job right here when I mentioned before you need to set the base domain for the cluster. If I had not done that, then this production job here would fail, because there's no base domain set build basically just runs docker build on the container.

A

If there's a docker file present, so it auto built the app again on our DevOps is pretty cool cuz I, don't have to do any of that. Sas is gonna, run like 50 seconds or so I think test is like a minute long and then production should be pretty quick. So we'll keep an eye on that, while we're doing that I'll pop over here get my cute control. Cement command set up I have g-cloud tools installed, so I'm gonna run a quick g-cloud test here, which is.

A

There you go: I'm, ready, G, cog, container clusters get credentials and the name of my cluster, and so it's installing the click credentials for my cluster locally. Just a convenient way of doing this now.

A

Now because the app isn't deployed yet we only see two kubernetes namespaces yell and manage apps, which is a namespace we use for installing those apps we saw before C killer here. Runner, Prometheus and ingress cube system is a Nettie's namespace, that's divine, that's specifically defined for CUDA and internal services. So we don't have to worry about any of those things like DNS proxy is in fluent now. Once this production task runs here, we will see a third namespace for our actual application.

A

Additionally, during this production task and let's go ahead and take a look at what it's doing, it's updating our ingress service with the location of the app so ingress knows how to properly forward requests to our application.

A

A

Now, when this finishes, we can pop back over to our environments, page here and environments page should list our deployed production app. Hopefully this loads quicker. So we you see an empty state screen before this finishes, but we'll see how that goes.

A

Okay, cool, so production no deployments yet, but when this finishes, then it will give us the end point for our application, cool, so I've only finished there. It is I believe that we'll have one yep one instance by default here and if I click this you mate, when I Bo say on the bottom of the screen, this will open the pod logs, so I'm gonna go ahead and open those in another tab and button right here opens our light environment. So you can see that our application is deployed.

A

Any other applications on the cluster will be deployed as subdomains of that. So we're just gonna curl this to make things a bit easier and ignore the invalid search I'm, also going to use I for keeping our headers around.

A

And here's our pod logs, so as you can see, this will go ahead and refresh as I'm making. These here requests see more here, so the original idea was put pod logs here, they're still ticket for that. But currently this is just the it's tailing. The engine, x-axis log I, think yeah cool okay. So let's go and run this again and then we're going to see the new namespace there we go so I'm running coupe CTL get pollinated spaces. This is currently the two pause that were created by our deployments.

A

We have a Postgres one and the actual application, and then this is the one we're really interested in here. So we're going to copy the name of our inject ingress, controller, pod and I'm, going to do a cue control and managed apps.

A

Execute just going to open up bash on that specific pod and let's tail bar logs yep. So this is the default directory where mod set audit logs are stored and by default nothing is happening which is good. These are fairly innocuous requests, but then, let's do something weird.

A

So I'm going to send a request with a little sequel injection here in the params and again passive mode means it will ignore this, but, as you can see, we just got some noise out of our log, so the rather well I've been saying esoteric logging format. They use here uses these roofings here ABCDEF, which I'm going to open this up to actually figure out what these sections are. So our request bodies in C H is our audit log, trailer, etc. So each one of these sections is broken out for analysis here.

A

We'd also determine that this is the same request here, because we look for our eat egg that you take matches the one right here. So this is the current. This would work and you can tail off this log. You don't really have to even log into the container use something like.

A

He'll, if log mod audit.

A

And just keep track of that now we can run some more requests.

E

Can you open their own, the logs of the Express pod instead, no browser check because I did I didn't see this request. That was locked here, follow it because it okay, yeah.

A

Unfortunately, I mean I expected to get for this, but currently of manually refresh part logs. Okay, isn't great, but yeah.

A

Yeah and just to show you where's this.

A

This is the current magic code. That's doing this whole thing, just pull this up quickly, which is fairly simple. It's modifying this default chart here. Well, maybe this is a bit of a digression.

A

Any questions for now.

D

um Couple questions, but first I mean this is this is great Lucas. Thank you for going through the demo. This really helps you know, make it real. What its gonna look like. What the experience is gonna look like so is the only way that a user would be able to view those logs be to essentially use g-cloud to get credentials then use cube control to tail that file me anyway, then yeah.

A

So you could also use. Where is it that's here you can use Google Cloud.

A

Console to logon us there, it is cloud shell, that's that's. What's called so you start cloud shell, you feeling that way. But yes, that's that's pretty much it. You have to open a terminal to your ingress controller and tail. This file, there's a couple ways to modify that, but not with our current ingress charts.

A

So this is this this here on the bottom left here is the default chart we use for ingress, and the modification would basically be this involves, say: config able, mod security. True, unfortunately, there's a lot security and something like it's up here enable mod security.

A

Last month, security RS, which enables the default rule, said there there's like a third option, that's something like mod security snippets and then you can do whatever you want in here and write some Lua butt or nginx syntax, but it's not currently supported upstream by what we're, depending on and I, don't know if we want to fully fork anything to get there. So currently, what I have is I have I'm wrapping the gamal parsing inside the actual model, sticking behind a feature flag, and then it adds these specific values.

A

So again, this could ship 12:3 if it wants, it, doesn't get very far. I've been looking this a while, so I I'm, very curious. Your opinions from everyone on the call is: is this like really a pain to have to like tail this log off like off the controller, or is this standard operating procedure for checking security risks on your kubernetes instance?

A

D

So I mean we're, definitely not done here with the state that the demo is currently in cuz yeah, like you said this, is it's gonna, be some extra steps to go through until that log file, I think before we ever get to a state where we would consider it. You know viable maturity. That type of thing we're gonna need to do some work to make this easier. That said, I mean for our very first iteration of laughs that none of our users have ever seen before. I think this is a great first step.

D

You know to show our direction, and today, users have no visibility into this like granted. This is gonna, be some efforts going to your hard to get here, but there's no way that they can get this sort of visibility themselves.

D

So I think this is actually really really good for a first iteration.

A

Okay, cool um any other comments before I yeah.

B

Lucas, can you just walk through with the current implementation, how you would go about changing those firewall rules like making a whitelist or you.

A

Cannot securely, okay, um the and I guess that that's what's talking about here is the UM there's. Basically, two options that are current fingers controller chart supports, and that is the these two settings.

A

It's either enable mod security here or enable OS modsecurity crs, which just turns the rules on it in passive mode. The idea of adding your own rules. We either need to fork this chart or there's an open, mr for adding a bit more support here, but it's it's been open since March and it hasn't been merged. Yet so I don't know when that's gonna happen, but once I gets merged, then we can update our chart and ship it out with whenever the next releases. For that.

A

Now, that's the to our ingress controllers is currently defined on pre old version. This year's tag, ten, it I think the newest is like tag 28 or 0.28 or something so. We have a quite old ingress controller image that were depending on I.

A

Don't think the rules have changed in that time, but probably a lot of other things have so I'll need to talk to the monitor team, because I think they depend heavily on that chart for monitoring to determine exactly what you fallout is of updating that, hopefully there's none and we've just not been looking.

D

Lucas at the the mo file you're sharing on the bottom left is that something that we would expect users to modify like? Did they look at that file today, or? Is that something that we keep on our back end? Just now, yeah.

A

So we keep this and I'm gonna I'll pull this up because.

A

It's kind of good to get a idea of what the structure of this project kind of looks like. So we could. We currently keep a vendor directory within our code base here of charts, and so we have a Prometheus Chara Jupiter chart the runner charts and, in a few cases, they're customized. So we'll take a look at the runner charts and one depends on.

A

Sexy defined in the model, but it depends on the upstream chart, so there's actually runner charts or charts runner charts yeah. We we actually maintain a chart for this separately, though the vendor directory has this specific ingress chart, which is the one that I have pulled up there, and this is loaded via the controller in our applications.

A

A

So this basically just opens the yamo file and in my case it does this on on the left. Here it is on the configuration file, and this is just a hide behind the feature flag because business logic and llamo sucks. So that's our approach really like modeling, this architecture wise. We have our applications defined within our cluster models here and we can define a separate application that way.

A

The only issue with your finding separate application in this case is that we're essentially modifying an existing application, and that's why it gets a bit tricky in terms of how we got to want to maintain buck.

D

Excellence and.

A

It's it's annoying because, like I feel like it's difficult to do this demo without getting really in the weeds, but that's kind of the whole problem, so hmm I.

D

Mean this is the first step so now that we're in the weeds, we'll work on coming up a level for for future iterations right, yeah.

A

That's good and if anyone has some helm experience I'd be happy to have a chat on letting this out and anyone's opinions would be awesome to get some feedback on okay. um You know those questions.

D

And I think I'm good cool.

A

All right thanks, everyone recording it's.

D

All thanks Lucas.