Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 22 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro: Falco - Jorge Salamero Sanz, Sysdig

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Intro: Falco - Jorge Salamero Sanz, Sysdig

Host intrusion detection (HID) has been around for some time. What if we rethought the problems HID solves in the context of Cloud Native platforms? What if we can detect abnormal behavior in the application, container runtime, & cluster environment as well? In this talk, we’ll present Falco, a CNCF Sandbox project for runtime security. We will show how Falco taps Linux system calls & the Kubernetes API to provide low level insight into application behavior, & how to write Falco rules to detect abnormal behavior. We’ll show how to collect & aggregate alerts using an EFK stack (Elasticsearch, Fluentd, Kibana). Finally we will show how Falco can trigger functions to stop abnormal behavior, & isolate the compromised Pod or Node for forensics. Attendees will leave with a better understanding of what problems runtime security solves, & how Falco can provide runtime security & incident response.

https://sched.co/MPhj

A

Well, thanks for coming this session is about container runtime security question. First, how many people here know Falco? Can you raise hands? Oh so, like half of the audience of great I hope you learned something extra from factor today. My name is Jorge I, run tech, marketing, antistick and I'm very involved in the Falco project, which started Cystic good.

A

Now it's part of the CN CF umbrella, so Falco Verdun, once he haven't heard about it before it's a behavioral activity, monitoring tool that basically means that we can detect suspicious activity defined by a set of policy rules and those rules are created. Businesses like flexible our query, language or filtering language, assisted Falco, can talk to your container, an orchestration tool, docker rocket cryo kubernetes, all of those and when we detect any indicator of compromise and a security event, we can send notifications to multiple places and use those notifications to the more interesting text.

A

How Falco it's fully open sirs. Now there is it's an independent project. We have some external contributors being part of the project. Now so, and that's one of the reasons why we are here today to bring or more people as I was saying before now, it's part of the CN CF umbrella. So that's why we have the opportunity to speak here today and it's the first project for our runtime security. So it's like really unique right now, it's on the sandbox level, but as more people get involved another project we have that we can Clara.

A

That's in this is a slightly abated, but the still wanted to keep it here. We participate on google Summer of Code, and actually this this summer there are a couple of students working on performance of the EVPs filtering part, which is cool so sorry, you're late for next year for this year, but hopefully, if you're interested you can involve I'll get involved next year, I like to compare Falk with home security.

A

So let me give you an example of how we can establish some similarities between implementing security at your home and implementing security on containers so at home, when we do when we want to prevent interships intuition. So we do adult sorry door, locks window, sensors bars on the windows, even exterior cameras.

A

So when you do that on on what you want to see would hope happening inside, you put basically motion sensors and interior cameras and probably maybe not in this room but on the venue you have seen apart from the main doors we have cameras inside. So it's like a very popular approach for home security. What if we wanted to do the same thing with computers? Well to prepend the entry point we use credentials and password key factor authentication. We can do even container images cannon to find known vulnerabilities.

A

We can deploy kubernetes mission controllers, establish kubernetes network policies, but what about those internal motion sensors? What would be the analogy on the container world? Well, we need to detect once the intrusion has happened and what we do from there. So we can look at kubernetes audit locks we'll talk a little bit more about later, but we can also like the container activity, what's happening inside the container, and we can do that using system call instrumentation. There is one thing the differentiates continuous from eum's, which is they share the kernel with the hosts.

A

So we look at the kernel we'll be able to see a lot of what's happening inside. We can use that information of what's happening to detect anomalies, to detect what it shouldn't be happening. Containers are isolated processes. The usually the purpose of a container is just one thing, so it's very easy to detect that that canta specific container, it's not doing what it's not supposed to do. Container images are in multiple, but once they are are running, the image can be changed. They contain not a image.

A

What the container can be changed, so we need to look at that. We need to look at container and what is that the world where they use cases how we can effectively use these kind of approach? Well, we can use for us here day vulnerabilities that they don't have a CV. So in mrs. canning tools, they cannot understand those attacks. We can also look for wrong configuration.

A

Insecure configuration like default passwords that they are being explode to get inside a container and then do unexpected things inside, but also we corley credentials any kind of internal or malicious activity who knows anything else that it's not a known vulnerability potentially can be identified capture years in run time security. So we want to block these kind of things, the approach of Falco. Actually it's not nothing, and it's not known here.

A

We can establish some analogies if we look back in time and we look at the history so back in the day when we wanted to do a connection network connection or network traffic filtering, we could use tools like TCP dump from there more complex tools that allow us to analyze. Network activity were developed like as a real war shark, so cystic, open, sirs and Falco basically take the ideas of these tools and they translate them into the entire system.

A

So if for doing network security, you're using snore to detect the attacks and to block them and war shark to analyze the network activity on looking at the entire system activity and perfectly to containers, we can use for co2. Detecting block those potential attacks, but also suspected a some analysis. Let me tell you a little bit more about how this works. The idea is that we are going to be able to deploy Falco in our container in our kubernetes infrastructure, for example, and it's going to hook into the kernel capture.

A

All the system calls move there into the user space and do all sort of things say. We have an e BPF proof, that's the technology that allows to capture the system, calls through a set of libraries. Those system calls they are put into some kind of a state machine, so we can correlate once with each other, and then they go into a filtering engine where all that activity as much against the the security policy that we have created. If there is something that matches, we can generate an alert and send it to multiple places.

A

Additionally, and we will see that now assists it can ingest events coming from different sources, including cornutus out it, but you might be wondering how we can years like kernel system calls for container security you're talking about something extremely low-level with something very high-level. Actually, if we think a little bit under the hood, we can, we can quickly see that if we see clone or exact via system calls it's because there is a new process that was executed. If we see open, close, read, write our system calls.

A

That means that there is some kind of I/o activity either on a file or on the network. If we see socket connect that means that those sockets or those while the scriptures they belong to network activity. So building from these very low-level comes out, we can come up with high high activity or high-level ideas. That's like kind of magic to define these rules. We use a human friendly language which is demo based something like this, so you either need to have that low-level knowledge?

A

You need to think about system calls all the time there is an ax traction. On top of that, let me get into the details of this language. We have macros that allows us to create expressions that can be referred by a name. We can create lists of binaries or ports whatever, and then we create rules. A rule basically has a name. A description, a condition where is all the magic happening, an output message that goes into the log on the alert and then a priority.

A

Those conditions the year build USA a narcissistic filtering expressions, and we can what's really interesting here is that we can pull in kubernetes metadata. So we could say give me all the system calls of the deployments inside this namespace and from there we can filter for interest in things. So for I, don't know, I want the nether activity where I want to see running processes. All those rules, your combined into that filtering engine and at least one rule- needs to contain a specific system call or a macro that has that system called underneath.

A

Usually we use a macro. There are many things we can filter on so file, descriptor processes, events use groups, our container metadata corn. It is a lot of things. You will see this better with a few examples. So, for example, if we want to detect if our shell was running a container, we can do that first, one container that ID it's not equal to host and then there is a process which name is bash quick example.

A

If we want to see if someone it's over right in system binaries, that shouldn't be the case in our container, because we were saying that in which is, they shouldn't be changed. Well, we can do FD directory in that list and then the operation, that's right. So these are just a few examples have basic basic things you can do with Falcom. So what do we do there? Once we trigger one of these conditions, we generate an event or an alert these.

A

They can be sent to different places like syslog or fireless under output, or they can be piped into a command and start doing more interesting things with them. This is what more real-life alcohol would like. So we have a rule name, the description, detector process that it's not an L, they started in an old container and the condition is going to be something like this so Evan type, exactly e. So we are looking for a new command or process asserted and where well we, the container dub image, start sweet.

A

Now, that's very simple: we use container metadata, not kubernetes meta 8 in this case, and then we do process name, it's not know. So, if, at any point in time there is any process any command, if anything executed in that container there. It's not the know our engine. This will trigger an alert.

A

But in addition to all this information coming from system calls, we can leverage other sources in probably one of the most interesting ones it cuber need is audit log events since kubernetes 111 there is a functionality that provides a set of records of anything change on the cluster. Anything changed through the kubernetes api server, so anything you do with QC TL, for example, calls or the API server and can be potentially are recorded in an audit log. What do you do with that audit log?

A

You can send it into simple log files, but you can also use web hooks to send those events somewhere else place like Fargo, for example. They look something like these like a JSON, so, for example, here we can see how a user called mini key of user, try to delete a namespace which was called foo, and this operation was completed successfully because its response called Chi hundred with these objects.

A

Once we get an inside Falco, we can access to all the different fields and values so time the user, the action research, all this is available, so we can create for Kreutz, and this is a very cool example of the functionality that you can imagine you want to audit if any of the users of your class or the earth storing credentials in a config map in south of secrets, something that they shouldn't be doing so basically, what we do here is, let me see: if I can there we go so we can attack or we can create a marker that contains numbers of strings that can potentially be our credentials like AWS access, key or just passwords.

A

Then we create another marker, the it's called config mark that has that filtering condition of the target resource, it's a config map and finally, the third video. Sorry there is another modifier action where we are create: creating a config map update in a config mark or patch in a config map, which are the three different ways that we can make changes into your config map. So, with all these three macros, we can create the final row.

A

So if you look specifically the condition, basically, what we are doing is just combining the three markers if there is a conflict map- and there is a modification action against a config map and the the payload of that action contains any of the strengths that I can potentially identify as a credentials being pushed into that configure. Bob I'll just trigger and get event, and actually I will be log in the entire country map. That's actually that's a feature, so we force the developers to rotate out credentials. So that's that's very cool.

A

So this is the basic functionality of Falco, but we can actually do a bunch more of things. First, let's see how we would be insulin. Falken Falken can be installed using a system package. Ac gives in a one-liner script that you shouldn't. Do it's also bi-level as a container?

A

A bunch of for the installation methods, but probably if you're using communities you want to use things like our hand, chart or the operator or you can also use inside manually gears in well sorry, here's an a daemon set configuration we'll go into and coughing map and the the Falco Paul will be connected into the kubernetes api. So we can pull in all these metadata that we need for building the rules. This also creates on Arabic permissions under the hood. So Falco has read-only access to see that we need this API.

A

So once we have installed Falco and we have in soul a bunch of different policy rules, how we can do something else other than a little, but for doing that, we created something called respawns engine. The idea is that, with all these events that Falco is going to trigger, we are going to define a set of reactions that can just log out. It would also block potential threats, so the moving pieces in this security toolkit are going to be Falco as the engine or the piece to detect normal activity.

A

We are going to be forward in events in the message broker like nuts and then the way we are going to react to these events. It's using functions as a service that they run in this case in chemistry. So the idea is that we are going to create different topics with the different policies that they can be triggered and with all the different priorities and then different functions as a service. They can subscribe to those topics and you can build your own responses, but we have prepaid.

A

We are providing a bunch of popular or typical reactions that the aircraft code, so, for example, we can kill the path if there is any suspicious activity or Nepal, we kill it governor. This will spawn a clean new path, but if we want our tea, for example, we fear that my have been continued break out on a container on a pot. We can say well Donna schedule any other pots in that node until I do further analysis. I can also isolate the so no needs they go against the specific container using erroneous number of policy.

A

Of course, we can send notifications in these things, like in slack, and we can also take says the capture, since the capture is basically dumping, all the system activity into a file that we can use for forensics analysis and post mortem.

A

One second dinner is long on, so if we wanted to or if we wanted to see all these moving pieces working together, this would look like this, so Falco, detecting all the system activity, all the activity from the containers, the narrative kubernetes nodes, also pulling in the metadata, but also are pulling in events and then forward in any alerts that they get trigger into nuts.

A

Then we have the different functions as a service here subscribe into the different topics, and then cublas can execute call for forwarding events somewhere else like, in this case, a Splunk or to connect again to the core needs api to kill the offending pot.

A

To give you an example, the very same architecture can also be deployed on AWS gears in hosted AWS services like SNS and lambda, but if you're using google, we can also do the same thing year saying: go, observe and Google Cloud functions and even forwarding events to gold classic hit the command center most of the people. They want a record all the events that they were trigger and security people. Probably they want to see this in their cm.

A

We can do that or you can just lock them into our typical solution like elastic elastic search, fluent D and Cabana. So you can create beautiful words like this, so you can get a little bit of idea of what's happening on your clusters. When you have a lot of working containers and now I'm going to show you a little bit of a demo I, don't know how much time is left wait an hour all right, so I have here deployed.

A

B

Is probably not good, so let me change.

B

The profile into something that you can see.

B

I'm, pretty probably 1s transparency.

A

So just very simple: um let's see this.

B

B

That's a good start.

A

A

There we go so that's very simple: here's one, an old cluster Rory, none Jiki I, have previously installed Falco like this I'm, not going to do it again, he'll mean so Falco and then I'm just going to enable forwarding events into nuts and additionally I'm going to install the EVP a prayer, so that will get in soul. Actually we can have a look at it it's running and then the other thing I've done here is to.

A

Deploy these kubernetes response engine.

A

That, basically, is going to install cublas, which is my functions as a service framework and it's going to install Nats, which is a message broker. If I look at my ryan services- oh I see you have phulka here and then well, some cover neatest things and then I have cublas right in here and then nuts, which wasn't solving using the operator you seen. Let me see where's this.

A

On this playbooks folder I have this script that.

A

I can use to deploy these functions as a service, so I can give this deploy playbook and then I give it an action. So what I'm going to when I see the following our policy violation on Falco I'm going to react with the following playbook, so the playbook- it's delete. I have already deployed this, and actually, if I do function list I can see here. Well, this made it a bit how this Falco delete function, it's in Soul and actually can I can also see the trigger list trigger.

B

B

You know thank you, yeah.

A

So that's a the trigger that connects the event or nuts with the function to the service, and what I'm going to do now is to show you there is a default policy, which is that terminal shell into a container. So what I can do now is get pods. Let me see: I have deploy a dummy, nginx, a pot here.

A

So what I'm going to do.

A

Is to run just like on a bash into the container and I didn't even have time to do nothing. The policy was trigger, sent an event, a notes and immediately cublas executed a play. Work to delete the pot.

B

A

That's a quick demo: I can use you, I can show you more things, but well. We would, if you like this and every cue can I come and we talk more about well cue, more and more people. They come and say: hey yeah, every I I deploy Falco on every kubernetes cluster or I, deploy I.

A

Would someone was telling me this morning and I think that's awesome, but now we want to go to the next level and get even more people involved, contributing things and probably one of the easiest way to get the started contributing into a Falco. It's right in policies right in roads we created default, our policies or rule for the top 10 15, most popular continued images on docker hub and that's available on that URL over there.

A

So if you are using other container images, if you can pull requests again, that rapport with your default rule sets that will be awesome. I hear of people doing advanced polish is for detecting crypto mining in their containers, and things like that. So we would be awesome if you could share those and contribute additionally ingesting events from multiple sources. Now it's available as an a small framework within Falco one of the first implementations it's for ingesting our kubernetes audit events, but we can ingest all things.

A

So if we are using a specific database or a different orchestration tool or are now something that generate events, if you can, you can also think about writing that translator to bring those events into the Falco engine which, at the end of the day, just filters for activity.

A

Additionally, another easy way to get started in the project is write in output destinations. So I show you before how far you can send events into some outputs, his blog or just plain logs, but also health can be sent into knots or into a muffin SNS or Google pub/sub, our viewer, for example, using Kafka. Maybe it's good idea to write on even forward in the Kafka also into other login services or web services.

A

A few links that you may find interesting, Falco website, the slack Channel we write off an on the Cystic blog about falco. Kid have documentation docker hub. Just if you want to take a picture or this there's lights are going to be a while all and that's everything I had on my list and I think I have five more minutes. I think we are on time. If you have any questions, raise your hand, Oh John will bring the microphone for a year. So we get your question in the video.

A

We have one question here.

C

Thank you so, basically, my question: if you get the event of violation, a I call, it happens a synchronously, so we might take time until the you kill the pod or stuff like this, and it did a lot of damage, but the too late. Is it possible right now.

A

Yeah, that's possible process in the event can take time, but forward in the event can take time. Falco, it's really really fast, so the detection of the event it's going to be super fast. We cannot give you a nagger aunty's how fast nuts it's going to be or how fast q+ is going to be.

C

To react, but by design it doesn't promise you.

A

We don't make any promise there. Thank you.

D

Hello, do you still require Cystic, Colonel modular? This can work just with BPF filters, so.

A

Falco uses the Cystic libraries for system call collection decoding on all the state machine, the end user binary. They are completely different, so they share a lot of the technology, but you don't need you don't need to so cystic binary to have get Falco running. Okay,.

D

So I could have Falco and configure just BPF as the backend. Yes, so I don't need this colonel.

A

You require a Nero colonel to have all the EVP F things, but yeah you either need a kernel module anymore. Okay,.

D

E

Hi, sorry, if you answered this earlier, but can I use Falco to distribute and execute arbitrary EB PF programs.

A

Now that's of the function Falco, it's quite a big EB, PF implementation and it uses CVP f to do all the filtering, but ye cannot write Evo PF white code to get it quickly did on the kernel. No, that's not the another function.

F

Is there any plan to expose prometheus metrics to be more kind of alert based rather than having this additional tooling around the outside? That you would require in order to be able to know theoretically that something's gone wrong? So obviously you can push it to the logs and you can look there, but that's very that's very kind of reactive, whereas kind of exposing Prometheus metrics on a per rule basis would mean that we could be more proactive without necessarily having to immediately kind of kill. The pod, like you described there, so.

A

That's a few things there, so we are it's on the roadmap to implement metrics, so Falco will be exposing Prometheus metrics on the rules, so you will be able to bring that into Prometheus with carvanha and see trends over there once we have the metrics, even metrics needed ah one of the things we want to explore.

A

If, as if we can connect all these events into some kind of like machine learning, something where rules can be auto-generated based on previous behavior um I, don't know would be great to have more thoughts and on the different things that can be done towards of direction. I know, there's a bunch of people hanging out on the slack channel. The D are working on some PSS, so maybe it's better yes, sir I'm not directly working in anyway. Learning! That's all right now.

A

Does that answer your question? More less, okay, cool.

B

G

There any kind of latency penalties when you are getting the assistant calls, because it's like, if errands.

A

G

A

Correct so we need to identify, or we need to look at performance impact in two different levels. So there is one level which is capturing. The system calls how training the assistant calls it's highly efficient and can be, and that's it's non-blocking and completely asynchronous. It goes into a share buffer moving into the user space and actually the the impact that it has it's a few nanoseconds. Now the challenge goes into analyzing. All those system calls and then that depends on a bunch of factors.

A

So if you create very complex rules, imagine you have a role that looks a bit payload of assistant call every single time, because we are looking at enough for a string into a network socket connection. That's going to have a huge performance impact. So the way you create rules is you try to capture the system called. You know. You are interested as early as possible in the filtering and in process. So if I don't have you saw it before if we are looking for new processes, the first thing in the entire condition- evaluation, it's I'm.

A

Looking for exactly system calls so rules II must be written thinking on performance because you can write terrible rules that will have a huge performance impact, but typically when rules are properly written, it has a very low performance. In fact, it's typically on CPU usage of the near version. The latest version we released on Falco also has some mechanisms to alert if you're hitting performance limits to say well, there are so many system calls that I'm need to drop. Some, that's already an indicator of a compromise, so that and that's not a problem with falco.

A

Specifically this, like is the same thing with snort like what dee dee. We have so much network traffic that he cannot. No I said it's a capacity planning if there is already so when you are do you when you plan your Falco deployment, you should already think what the analysis capacity that you need and prepare accordingly for that and then, if there is way more than expected, that's already an anomalous. Even that should look. You should look at it. Thank.

G

A

Well, I think this is said: Michael do you see who is standing over there? It's the deep dive on all these concepts. When is your session tomorrow,.

A

Tomorrow, sometime right.

A

Falco deep dive.

A

Oh yeah in the morning, okay, well thanks very much, and if you want to see like more use cases or demos, some of the vocal contributors are hanging around here. So talk to us we'll be happy to share more. Thank you.