GitLab Customer Success Skills Exchange, 16 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Container Host Security - Getting Started with Falco

Description

Fern with Technical Marketing presents on the value of adding Container Host Security to your infrastructure and takes questions from Customer Success team members.

Key Topics:
* Intro to and Benefits of Falco
* Installing Falco via GitLab-Managed-Apps
* Creating a Custom Rule
* Examining Falco Logs
* Falco Alerts
* Container Host Security Roadmap

A

All right, thank you again, everybody so much for joining us. We've got fern here to talk about uh container security or adding container security to your infrastructure, so really excited for this presentation.

A

I've got the notes, document updated for you, I'll drop it into the chat, but I want to go ahead and turn it over to fern awesome.

B

Thanks john uh yeah, so today I'm going to be covering container host security and uh pretty much I'm just going to be going over how it works within gitlab how to set it up and I'll do a little quick demo and then have a q a session so um just make sure everyone's pretty well caffeinated. For this everybody, I'm on my like third cup of coffee right now,.

C

B

Cheers cheers well yeah, you can't see with the vikrant um yes, so so I'm going to go ahead, share my screen and then we can start with the presentation.

B

And what ends up happening is, since I have it on the second screen. What I'm going to do is. I can also switch my camera. Let me get a side view hold on.

B

Just got to make it as normal as possible: okay, so share screen, and there we go okay, so leveraging container host security with git lab. So I'm going to talk to you a little bit about what exactly is container host security.

B

So what container host security does? Is it provides intrusion, detection and prevention and monitors our container host to see if anything, suspicious or weird is going on, and then it can either block or alert on that activity?

B

So one thing to note is that we use falco, which is a container host tool for monitoring the container host, and we we leverage that it's a cncf um project and it's customizable via rules and I'll get into that in a second, but just to kind of show you the flow you perform requests using cube, ctl on the cube api server and what that does.

B

Is it modifies our cluster, whether it be adding deployments or you know, adding our application exposing it and what that does is it creates audit logs and these audit logs are sent over to falco, which we install within our gitlab cluster, so it'll be an application running within our gitlab app manage apps namespace and what it'll be doing is it'll, be looking at all these audit logs within all of our uh within our cube api server and what's going to happen, is there's going to be certain rules that it applies?

B

So it's so it searches all the logs and then let's say it finds a log where uh directory is being created and one of the rules says hey alert. If a directory is being created, then what it'll do is it'll go ahead and generate that alert, and then let us know like hey, there's something: that's not supposed to be happening on the system or there's something that we're monitoring for on this system and we've detected that that this has happened. So then we can go ahead and send alerts uh to whatever medium we want.

B

So we can send alerts to let's say pagerduty: we can send alerts to the gitlab dashboard, which I'll show you a little bit about that later today, but something that hasn't been implemented. But it's in the road map it can send alerts, it can send things to a file. It can send um syslog. It can send things to prometheus, pretty much it's highly configurable and can be done. You know in any way that you need so uh just to give you a couple of examples of what the actual rules are.

B

Some example rules are hey, let's alert when a shell is running a container, so it could be something malicious happening where someone got into a container that they're not supposed to have access to. Maybe it's a container that no one should have access to and it should all be deployed through the ci cd pipeline. Well, if a shell is running in, it could be some type of malicious intent.

B

Someone got in that wasn't supposed to so we can alert on that and we can provide even more details on you know what exactly the shell script that was running is um writing to directories that, like slash dev, which shouldn't be written to um unexpected connections, uh non-authorized, namespace changes and, and so much more one of the fun ones, is a process that other than skype or webex that tries to access your camera. So it pretty much runs on the container host.

B

This is something that can probably even run just on your um you know your own mac and detect hey. Is there a skype or web accession happening? Someone using my camera, and I don't even know so, um but the thing is this is containerized and deployed on our kubernetes cluster and waller on these things.

B

So how do we install this on our cluster? So I don't know if you all have known, but before we used to install things with via gitlab manage apps, so there was different ways of installing things there used to be a ui where you would go in and you would actually install the application via the ui.

B

So you'd click install so it has been removed as of gitlab 14.0 and then there's also another way of deploying applications which is through a yaml through get lab, managed, apps, and that has been deprecated and not only has it been deprecated, but it's actually going to be removed in gitlab 15.0.

B

So there is a migration process to moving everything over to the new standard, which is going to be the cluster management. Pro project template and I'll go a little bit into depth on that, but in the future, I'm going to make a video on my grading so I'll make a little video on showing how to migrate. These applications since has a lot to do with the gitlab protect stage, but uh for now I'll, just just give a little bit of details on what the actual um cluster management project is.

B

So let me just um go ahead and show you so the gitlab manage app. So this is this is like the old way of doing it. You would include a template to the gitlab, ci yaml and then you'd. Add a config.yml under gitlab managed apps in the config, and then you just pretty much put the name of each application. Input installed equals true and what ends up happening is for falco. It looks something like this. You just say: falco installed, equals true and then you can add a bunch of other data.

B

So now this has changed and the way that it has changed is. I can show you before and after so.

B

If you look at the before right, you would have gitlab managed apps, you had a falco, sorry you'd have config.yaml, you'd, have uh you'd express everything that needs to be installed, so you'd say: falco installed, equals true, and you set the cluster type and then you'd have specific falco information here in this directory.

B

And within the gitlab ci yaml, you would add this managed uh cluster applications app and then that would deploy falco into our cluster. But the way that that's changed is now you'd have within the gitlab ci ammo. It would just be the regular yamo, where you'd have this supply stage.

B

And you'd have this helm file.yaml, which will tell you exactly what we want to install. So I would uncomment this right here and then that'll go ahead and run the apply job and it'll apply falco within the helm, file.

B

And then, furthermore, there's applications and we can go to falco and then here we can configure what values we want um like the standard helm way, and we would also configure a home file to just give more information on where the actual chart is housed and where the helm files is housed, and this is also done for customizing things different things. So um this is something new within gitlab. That's just been brought up, so you know I'll, be adding more videos and more details on this. As you know, it progresses.

B

But now to go back and if you want, you can start adding your questions to the uh document and I'll review that document after after this presentation, then we start answering things one by one, so custom rules and falco. So you saw some of the rules that you know are already on there and already default. But what if we want to add something specific, so falco has some guidelines on as to the syntax of how you write rules.

B

So, for example, this is a rule where we're going to detect just any directory being created.

B

So you see I had the rule, I had a description of it and you can see that the condition for this row is make dur. So it's any time the make dirt command is run which creates a directory and then what it'll do is it'll output.

B

This message here so file below and known directory open for writing so and it'll provide us information on the username, the exact command that was run, the file, um its parent, a whole bunch of other information. Now this is all syntax that you would have to read from the thoughtful documentation. So you can structure it yourself and make your own rules. You can set the different priority as to how you want it to learn. It could be an error warning info and then you can give it a tag and then for more information on this.

B

You can see what each thing means. So you can set the different priorities. You can set exceptions.

B

You can set different tags um and there's different tags that you can apply so they'll, just there's just more information and the this is what you use tags. So you can use t argument to disable rules having a given tag.

B

So if there's just a whole bunch of information that you can do and and just highly configure falco you would have to you know when writing your own rule, you would have to go through this and just kind of see exactly how you'd want to write a rule, and this is more filco specific, so and I'd be happy to answer any questions later on.

B

So so now that, let's say we have now we have falco installed. Let's say we have a new rule for detecting a new directory. um So what do we do with that? um What do we do with all that information like what happens when a world is detected like um why? Why do we care? Well, what we do is we we create alerts so that way, if any of those rules are ever broken, we can alert something and have some action performed so um so different ways of alerting uh the standard out.

B

It's a file. You can alert to syslog. You can alert to a program to an https endpoint uh to a client via grpc, so pretty much. I can call anything an example would be. I can actually, um you know, send myself an email of hey, an alert was triggered. An alert was triggered. That would be a very simple way of doing it, but I can also go to slack, maybe and just have a slack channel where every time something is um anytime, any rule is broken.

B

Slack will just message me and just you know, pop up and show rules so actually put the wrong link on here, but um and I'll just change that right now.

B

For guidelines on writing alerts.

B

So if I look at how to write an alert, you can see that there is file output, so I just add the syntax, and this will tell me what file name to go to syslog program output. So this is the one that I wanted to show y'all, which is. I can set a program that I actually want to to run uh to run whenever a rule is triggered. So here you can see I just mail, someone sending a mail to someone that there was a focal notification.

B

So it's just very simple, but this can you know, get as complex as you'd like it to get or get as configured as you wanted to get, but pretty much. What will happen is like anytime, a rule is broken. It'll, just mail, you, a focal notification, follicle notifications, so you'll get probably a billion emails depending on your cluster and same thing with slack. So you can configure this to stack.

B

So I couldn't demo this because I'm not a slack admin, so I didn't have access to creating the application, but at some point I'll probably request that. But what you can do is you can just get the json output and then just post that to the slack service, make a little bot or something that or a web hook, that any time that a rule is broken into alerts, it'll, post a slack, so this little bot will go ahead and post.

B

So these are kind of some of the ways where you can see output and you can kind of work with configuring this and so now.

B

What I can do is- and that's that uh that's this is a pretty funny picture. Actually this is the guy he's at the beach, but he's looking at his tv um all right. So now, let's just go into a little demo and just kind of. Let me just show you how this works.

B

So can you still see my screen?

B

B

So if I do keep ctl get pods and get lab managed.

B

B

You'll see that, based off of my deployment, I have three pods running with falco on it.

B

B

Let's go to my namespaces, so this is what I deployed. So I do keep ctl got pods.

B

In this namespace, so these are this is everything that I have deployed on my project. So I'm going to exact into this file from ctl.

B

B

So this pod, I mean.

B

B

So you can see I'm in the I'm in this pod. Let me just.

B

Okay, so now I'm in this pod and I'm gonna go into the slash dev directory.

B

And what I'm gonna do is I'm just gonna start making mr okay make their I'm just going to start making a bunch of different.

B

B

Okay, so now I exit.

B

And now let's go and look at my logs, the falco. Let me see, I don't know if I can pipe this to jq.

B

Okay, so these are all the logs for um for falco, so I just made it nice in this json format. But now, when I'm looking at the different logs, you can see that um there's been some. Some triggers there's a lot of different notices and different things, but there's been some triggers where you can see that there was a new directory detected so detected new directory and um that that's an issue. So that's something that I would have to examine so error file below a known directory opening for writing.

B

So you can see that there's issues with all these new directories that are that are being created. So I would have to add more information. uh Like my um what you may call it. My rules are not configured completely properly where it's showing like the actual file and everything happening, but you can see that there are errors being detected of files being created so.

B

So detect a new directory, you can see that there's been lots of alerts on all these directories created. So I created a bunch of different directories and now they're being shown as detected. So you can see that the rules are being broken within this instance. I don't know if I can grab. I can grab.

B

Let's see if it'll show the files no yeah, so it won't show the files. um I would have to add more to the actual configuration to show what actual files are being created, but at least I can see that there's some tampering being done on the system, so I can always just go and verify that, but you can see that there are uh errors that are applied because a file has been created so um so yeah.

B

Now let me now. I want to show something else. I want to show the alerts, but let me see, let me get back.

B

B

B

B

So yeah, let me see, I think there was a file that I have.

B

Let me um yeah, I think that was. I think that was it right for now.

B

Okay yeah, so now you saw like that they're actually being triggered, so I don't, I, you saw the rule actually being broken, but there's no, I don't have any alerts set up right now, um just because I didn't couldn't get it to work on slack, but I'll, show more detail kind of alerts actually being um sent over to a file. um That's something I had to play around with a bit more, but just to kind of give you an example on what alerts are going to look like in the future.

B

I wanted to show you something in the road map right where you can see here is the container host security, so there's going to be plans for support for the kubernetes agent there's going to be exporting logs to uh to a scene, chs statistics, policy management via ui and default policy set, and one of one of the things where we want alerts to be shown is going to be kind of like this.

B

So if I go into my project and I go into security and compliance and I go into threat monitoring, this is where the alerts would be displayed so anytime that an alert is actually triggered.

B

They would actually be displayed here, as you know, the date and time with the name, the event the incident and you can assign different people to it and provide the status and you can actually manage that status.

B

So eventually, this is where we'd like to see the alerts right now, they're only there for container network security and anytime, a network policy is broken, you'll see the alerts on here, but we want to get that for container host security as well. That's one of the pieces that we're trying to put in and then also have, statistics for um for the container host security as well, but right now everything is just within container network security, but we we want to have everything for our container.

B

We want to have the same things for container host security for container network security applied here, so you would also see instead of just seeing network policies. We'd also want to see um actual rules here and have you know definition between everything and have it done in the ui. So you don't have to go back and forth between the yammels and the gui.

B

So with that being said, if anyone has any questions, I'd be happy to start taking questions now and let me open that document and we can just kind of go over different things. So let me just change this all right.

B

So let me open up that.

B

B

So one question in your introduction: you mentioned the track and prevent from your demo. It seems that falco's alerting when a vulnerability happens any way to be more proactive.

B

So one reason why you may not may not want to block right away is because you don't know what type of what what it's going to cause to your system right, you don't want your service to be completely stopped.

B

What what I've done here is. I put it in um just view mode where I'm just making sure that that we're just alerting on different things. So then we can go ahead and take these steps and further the security of our items. So a lot of times. What's what uh you know, different security teams would do, is they'd actually run it and see what the common things that happened are and then from there and force more policies to actually um stop things from happening to actually block things.

B

So I know how to do this in container network security, I know how to write network policies which block and within the web, application firewall now with falco. I have to look more into details. I know that is something that it does support, um but I'd have to. You know, have to get back to you on that.

C

Yeah thanks, uh I I was writing the the question yeah the idea it was the same from uh from a firework perspective. You start logging what is happening and then you define your final set of policy. There is a specific moment where you are going in production that you want to prevent those kind of factions that just receiving an alert. So I I want you to investigate if we have, if we know something about that stage, but thanks for the for the demo and the answer.

B

Yeah we we had it for the waff which we've deprecated, which was based off of modsec, so I mean it's a good point to have something. um You know something alternative to that to be able to block. I think right now, most customers won't want to block they'll just want to monitor and then kind of enforce, more strict um security policies after after their findings. So so I think for right now. This is something this is something definitely viable, but I'll keep working with y'all.

B

You know to try to figure out more as to like how to actually perform the blocking.

C

D

You hey friend, I got the next one. uh By the way, I love your videos, man you're great stuff, it's fantastic.

B

D

Sure man, um quick question competition, you know: do our competitors offer stuff as far as this goes are folks, usually using just the you know, a free, open source version of of um of um container and post security. Do we know of anything? That's you know, bundled with anything else that we can compete against. There's there's.

B

Different things, honest, there's different, a lot of different companies that actually compete. um What's the name that that actually put different tools on kubernetes to monitor so systig is, uh are the producers of um falco, so that's one of their? They. They offer more solutions around that, um and then there is um what what's that um observability.

B

um I can't man, I can't remember the name of that company. No, a new relic, so new relic also has features like that which can observe everything going on. It's pretty much a lot of uh companies that log different things and then perform events depending on what's found in the log. So like um there's another one called log log dna there's, so it's a bunch, there's a bunch of different tools. Now they all, they mainly offer logging solutions and like debugging type stuff and searching through logs.

B

So it would be mainly those companies because you can alert based off of what's found in the logs, so they don't offer something. Some complete solution like gitlab. They just they'll, mainly focus on that one um part: okay and yeah splunk as well uh table. Just put that so splunk. Does this as well? There's there's a lot of competitors that we have that do this okay appreciate it.

D

B

And if you need, if you want any links to these projects, I mean um so one more thing to add and that I wanted to add is within technical marketing, we're actually working on like different workshops that people can go through independently and that you can you'll be able to send your customers uh on how to do these different steps.

B

Now we do have to modernize them a little bit um before we release them because of the changes within the gitlab managed, apps and different things like that, but we will provide different workshops and different tools so that way, uh customers can go step by step and learn how to use these tools or or anyone within gitlab that wants to learn. So those are a few of a few of the things that that we're working on and just having applications that anyone can use.

B

So I'm going to type it in right here in this chat, but um just going there to the devsecops, you can find some of the you know some of the stuff that I've used today.

B

B

That's but yeah I mean that's, um that's pretty much! um That's pretty much yeah what I have.

A

Thank you so much for the great presentation. Fernando uh does anybody else have any other questions? I don't see any in the document.

B

So yes, uh no more questions, then you know thank you all for attending and you can always reach out to me on slack or reach out in the tech marketing channel. um If you need any more information and then I'm going to try to do a migration video on migrating from get lab, managed apps to this new project management cluster and if I can get some slack access I'll, happily, post the video on you know sending these alerts and stuff.

B

D

C

All right thanks man, thank you.

B

Bye. Thank you all right. Thank you. Fernando.