Cloud Native Computing Foundation KCD UK 2021, 19 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Native Security with Falco — Tom Llewelyn 1.2.4

Description

What if we can detect abnormal behavior in the application, container runtime, cloud & cluster environment using the same method? In this talk, we’ll present Falco, a CNCF project for runtime security.

We will show how to use Falco to tap into Linux system calls, the Kubernetes audit logs, and cloud events to provide low level insight into application and platform behavior, and how to write security rules to detect abnormal behavior.

Falco is also featured in the CKS exam curriculum, so this session should be useful not only for securing your cloud native infrastructure, but also in passing the CKS!

A

So uh so yeah, so it gives it gives me great honor to you know. I got to talk to one of my friends earlier today when when katie chong was talking uh and- and I get the opportunity again to introduce a good friend of mine tom lewellen, he works in our in sorry. The sistig uk team and today he's going to be talking about cloud native security with falco so tom over to you.

B

Okay straight into it yeah. Let me just um share my screen in that case. um Let's hit that and present so are we sharing successfully.

B

Excellent um so.

C

Yes, thanks for the intro chris as yeah as chris mentioned yeah, so I work alongside chris insisting and the focus today is going to be on the falco project, which is one of the open source projects that was originated in systig, but we donated it to cncf, and it's now very much a cncf run. Projects that we still contribute heavily to um and falco is very much focused on uh runtime security in the cloud native space, um so we'll delve into.

C

Essentially, this is an introduction to falco what it is, what it does, um how you can use it, how you can get involved with it. um I will speak a little bit at the end, just a bit of a plug since uh we're sponsoring the event.

C

Just to tell you a bit more about what systig does as well in using falco, but the bulk of the talk will be on uh the open source aspect of it, starting um you know, taking a step back and thinking about approaches to securing systems, though so fundamentally, there are two things you can do.

C

This there's the prevention side of things where you're actually trying to modify the behavior of a system by stopping things from happening, so you know, could be killing processes it could be um just blocking system calls, you know, actual doing things on the system. The other aspect is detection, where you're looking at what's happening, live on a system and then looking to detect things that look suspicious. So you know it could be.

C

A process suddenly starts talking to a new network or accessing files and modifying binaries with you know those sorts of strange behaviors and picking up on those two and they're not they're, two different approaches and you need both in a system. So you'll need prevention, style systems, you know, guard rails and so on.

C

You also need detection capabilities within your systems as well, because you know the prevention is only as good as the rules you thought of um detecting when there is suspicious behavior is a much harder thing to do, but it's also one of the key things that you need to do to find where you have zero day, exploits or possible malicious actors within an environment.

C

Often the only way you'll detect those is by looking for suspicious behavior.

C

So just looking at a few examples of tooling within this, um the linux world uh that deals with this, these kind of security enforcement uh capabilities, you have things like sec, comp and sec bpf, which are involved in you know. Filtering system calls so you're, actually blocking system called se linux, which is very much a you know, a linux security uh suite and you know it's vastly capable in terms of doing mandatory access, control and label-based security profiles and so on app armor, a similar sort of thing, but takes more of an application, wrapper approach.

C

And then you know things like ip tables firewalling, so that all of these tools are there to kind of try and lock things down to, as I say, put guard rails around your workloads, prevent them from doing suspicious things prevent them from accessing things.

C

They shouldn't and you need all of this stuff, but it's quite difficult to get it correct and, as we add more layers of complexity, as we add kubernetes and you know, container runtimes and so on and dynamic workloads getting that balance right of blocking stuff and securing things, whilst also enabling those um workloads to do what they're supposed to do and not slowing down your developers, not slowing down um you know not getting in the way is a difficult thing to do, um and then looking at the on the detection side of things- and we have things like the kernel kernel audit d- um and you know, if you look at what sc linux does he actually sends audit events at the point where it detects denials but they're just that sort of auditing?

C

What's happening, live on your system, just getting that audit log out. um Kubernetes also has an audit logging system and falco, which is what we're going to talk about predominantly today fits into that detection side of the security, so it falco itself doesn't do anything in terms of prevention.

C

It's very much taking a stream of data and looking for anomalous activity within that stream, uh as you can see from there, though, you know the list of tools that do this kind of detection. There's still there's a lot to be done in this space, and you know we're working in inside systing on a lot of things like you know, machine learning, capabilities to be able to piece together the anatomy of an attack across multiple steps.

C

So you know you if you see this behavior followed by this behavior, and then you know this could be a um an escalation followed by a lateral movement. Those sorts of attack vectors, where trying to piece that, together and from a detection point of view, becomes uh extremely important. As and yeah as you see on the bottom of the slide there, you can't rely just on prevention, so assuming that you've put all the prevention in place um and that you're done is never enough. You need the detection rules in there as well um and yeah.

C

Just as an analogy to that, um you know I've locked all my doors, uh but what? If somebody you know, if I leave one open or if somebody manages to break a window, then if I've got a dog in my house, then that dog can alert me to what's going on. So that's you know it's it's the two that you need both are useful. um You're not gonna, rely on your dog to secure your house, but you can you know you can use one to detect. What's going on, you get the idea.

B

C

So moving on to falco and what it does so. The first aspect that falco deals with is the system called stream coming from the linux kernel and the reason that we, this was chosen as an information source is because it's fundamental to everything: that's happening on a linux system. So if you look at trying to secure a container workload on us running in kubernetes on a host, there's a whole lot of layers there, but fundamentally you've got a process.

C

That's running on a linux kernel and in order to interact with the outside world, it has to make system calls and so that system called stream is going to tell us an awful lot about. What's going on in that host and what's that, what that process is up to, um and so then, by starting from that kernel system call we can see. You know what processes are running.

C

What I owe them uh is going on there, what network, they're, accessing and so on, and you know, there's a complete picture that you can build up just from that system called stream. So it's an extremely rich source- and this was one of the fundamental things that systig um started out doing was tapping into that system called stream and making sense of it and tagging that data with all the um the kind of the layers or as you build up. So you know this is a process it's running in a container.

C

That container is running as part of in this pod, which is part of this uh deployment, which is in this name space. All of that sort of complete layering up, but fundamentally, if you've got that system called that's where we start from, um and so one of the things that's just that falco is able to do is instrument at that kernel level, and it does this in two ways, so you can either insert a kernel module and there are, and we've open sourced this kernel module as well.

C

So you can then insert that into your kernel that enables a ring buffer that exposes that system called stream into user space, which is then fed into falco or, conversely, for more modern kernels, where we have ebpf, um which is where we'll be moving in future, is using an ebpf probe, which then gives us that standard interface. So, rather than having to insert modules into your kernel, you can just pull that same data stream.

C

Out of that out of the kernel, either way we're getting the full um system called stream and then feeding it into falcon, um and then you've got a couple of libraries in there. So uh the best cap and lib esp essence, which are the the analysis, libraries that then will um categorize and you know and structure that data as it comes out and then we'll have a rule set that you can give to falco, which you can then use to spot um particular behaviors. So it's pattern matching essentially against that data stream. Within that.

C

The second source that we've um we're able to ingest with cystic er sorry with falcon, is um the kubernetes audio logs. So this is a standard api. That's coming a standard feature of the kubernetes control plane. Now, where all security relevance, data is being dumped out of this stream in chronological order, and so we can see all of the activity generated by users, applications etc for all api interactions that are security. Relevant will be passed out of that kubernetes api, um so the api log.

C

So essentially, this is stuff that is manipulating etcd, which is where the the fundamental state of your kubernetes cluster is being stored. um That's going through cube, ati api to as the front end to that, and then all manipulations that are going into that come out of that audit log and again so this is now a very rich data stream of you know all deployments that are happening any edits to objects, um config maps, creation of secrets, anything that's being deleted.

C

All of those sort of security, relevant data is all included in that audit log and it's being and can be fed out to sys to falco so that it can ingest it and then. Finally, this is a fairly new um capability that we've added recently where um falco is now starting to be able to ingest cloud logs. So aws cloud trail was the first one where we have. You know all of the audit logs from your aws cloud accounts. So this could be things like you know: creating elbs, exposing s3 buckets.

C

uh I am role, creation, etc. um Users logging into your account and then what do they do when they log in all of that kind of um low-level cloud account activity is a new data stream that we can ingest so aws is there. Gcp is coming and azura very shortly and I believe a gcp may already be released um and again what this gives us is yet another data source that you can ingest with falco and then look for suspicious anomalous.

C

Behavior in terms of you know, what's being deployed into your aws account what edits are being made to the resources within that? Are people manipulating policies? Are they deleting logs? You know all of that sort of stuff that you can then look again for writing rules against trying to spot anomalous behavior so bringing this all back together in terms of the falco architecture.

C

So you can see that um if we go kind of bottom left upwards, so we've got data being ingested into a falco instance, so that could be from the kernel on the host where you've got falco running or it could be external data sources coming in via http um via rest apis. So the you know the cloud audit logs or kubernetes audit logs, which are being fed into those um that ingestion service.

C

That's then categorizing that data and making it in usable and that's then fed into the filter, expressions which are built from the falco rules. So the help the falco rules get loaded in at start time that essentially determine your runtime policy and then essentially we're looking for pattern matches of my policy against the event stream and then, when we detect, you know when a policy triggers a rule triggers because the conditions have been met in that data stream. We we can then trigger an alert, and so this is why you know so.

C

Falco is very much on the alerting side of and detection of that security uh structure and then alerting can feed to a number of outputs. It could go to syslog files, shell or grpc.

C

um There's a very good open source project called falco's sidekick, which we'll talk a bit more about which can which you can ingest falco, and then that has integrations with all sorts of third-party uh capabilities, um and you can also do things like um create prometheus metrics. You can push out metrics from it.

C

So in terms of where you push that all of this, then the alerting from falco can then act as the start point for either a response event or some sort of remediation workflow you could trigger, or you know whatever it is, but essentially falco is very much there for this detection of the anomaly and then hook into something else, which will then take care of what happens next.

C

So fundamentally, um falco rules are just yaml, and so you can see there's an example. Falco rule on the screen here and um the rules can be so. The rule set can be made up of uh macros and lists, and so on. So you can create a macro and then call that macro within another rule and so on.

C

So you can see in this rule here we um rule file here we've got a few macros being created, so I'm looking for things that are a container um looking for things, writing to fifos and so on so and each of these macros has a condition. So that's a what are we looking for in the pattern match and one of them is looking for a container id.

C

Other conditions are looking at the process command line, um and then here we've got our full rule for container drift detection. So this is um essentially it's an event. Type is open or create and open. Exec is true. It's a container, it's not writing to a fifo and it's not writing to valid docker and the raw res is greater than or equal to zero. So then, essentially, this is.

C

Can we spot um a new file being uh opened a certain binary essentially created within a container, and if I click on this, hopefully this will switch me into a nice pre-record of this. So I decided not to anger the demo gods and try and actually do a live demo, um but this will um play us a nice little uh execution of that rule that we just saw. So essentially we already have the rules, so we sent falco comes with a default rule set, which is quite large and complex.

C

So for this demo it's just been cut down to just that single yaml rule that we have in our capability so that we can just run falco with that rule and then create a container and trigger the detection of our suspicious behavior.

C

So, let's run falco.

C

And you can see that there we go, we've created a falco instance. That's now there and running, and now we're going to spin up a container and within that container, we're then going to create a new executable file and then try running it so created a file created a container we're now running in the shell inside the container um and my colleague that developed this uh forgot that he doesn't have vim inside his container. So he has a couple of aborted attempts to create the file.

C

Then he realizes what he's doing um so yeah so he's creating a simple c program that is literally just going to generate a binary that opens a file um so drop the code in so you can see what it's doing is opening a file.

C

And then we compile that and then run 8.8 the output of the compiler, and you can see straight away when that executable ran inside the container. It's now triggered my rule in falco, and I've got that detection, as I you know, which has got the variable substitution in it. So you've got the context of what was going on, and so you know it's pretty simple, um so this is just in a single standalone, falco local container running on its own.

C

um But you know in terms of that's, you know sort of my first, uh my first um falco rule, um but you get the idea of you know. This is fairly simple. When you know it's the classic thing of try and do one thing well, which is what falco aims to do, which is that detection piece giving you that rich capability of starting from a very fundamental information source and building up from there um in terms of then developing policy and um having rules to do things?

C

um As I say, falco comes with a default rule set which gives you quite a lot of um additional rules, not just the one we saw there. There's also um the cloud native security hub, which is a systick sponsored uh website where we have policy being developed um and so on there. You can go in and look for specific use cases um and find falco rules amongst other things, um and you can see you know, there's things there for fluently elastic and so on.

C

There's also, quite often um when a new cve, a critical cv is discovered that has a large impact. um There will be a blog post written by somebody assisting who then will develop a specific falco rule to detect the exploit of that cbe, and then that will be pushed out to this as well. So you know with this over time. This resource should be building up, um but you can contribute to this as well. So you see on yeah on the site.

C

There is the ability to send your own rule suggestions if you want to um so that you know you can help the community build up this database of capability in applying falco.

C

um Installing it is fairly straightforward, um so debian or red hat systems, there are rpms and d packages available, so you can just app get or yum install it. um If your distro doesn't have those available, then you can just run uh a shell script that will pull down uh the latest binary and install it for you into locally. um And finally you can you can run it as a docker container as well.

C

So you can just pull the falco security falco image um and there's full instructions on how to do how to install it and run it, and you know just play around with it on that uh on the falco.org website.

C

um If you want to install it onto kubernetes, then we also have home charts available um or you can just go and grab the demon set and apply that directly so depending on which you prefer, um but again um so running falco. As a system d process gets you a better isolation and you know, means you're running low level on the host outside of the container runtimes.

C

Actually, when it comes to kubernetes, um you can then just deploy it out as a demon set onto the kubernetes cluster. That demon set will require privileged access, of course, because it is, it still needs to hook into the kernel on the host. So it's going to require that root level. Access to do that.

C

um And then in terms of additional things, so this this is just an and this slide that will hopefully be made available, but there's a whole load of additional tools that you can use to then help you manage and create and uh deploy falco into your environment, so falco ctl for um just controlling things sidekick, as I mentioned earlier. So this has got a whole load of integrations into third-party things.

C

So, once you've triggered a rule, you can pass it into sidekick and that can integrate into various other um workflows um so that you can trigger other things um and then other clients and so on also a prometheus exporter, which can be very useful in terms of then, if you're using prometheus as your monitoring tool as your monitoring framework, then you can feed those events into prometheus as well and start dashboarding it and so on so yeah, just a quick mention of sidekick. So this is a community contribution.

C

It came from somebody not connected with cystic at all who's developed this as a way to integrate falco into things. Like slack elastic. You know you can see, there's a whole load of lists of uh integrations on there, so it's um extremely powerful and capable and yeah there's a growing list of that, because there's been a nice architecture for adding these integrations into.

C

um So that's extremely useful and you can see that yeah. What just just the explanation of what that falco, sorry that sidekick um capability is going to do is actually then take the events out of falco and publish them into you know an sns topic or a teams channel or whatever else you want to push that event to, um and that could then have you know third part additional workflows triggered from that event.

C

So that could be, you know, execute a reaction like killing the offending pod or so on um and also send on additional notifications elsewhere. So it's it's a first step to building that enterprise level security capability.

C

um If you want to build one by yourself, um another thing you can do so cm, so the security information and event management. So there's a number of products out there um available for doing this kind of event: management where you essentially they'll all ingest, a stream of events from various sources and then help hope to correlate events and respond to things when you have a security breach.

C

If you're looking to do that, building with your own open source stack, then you can use efk for that. So you can push the events out of falco and then use that to construct your own um seam. So if you are, you know of a build it yourself bent. um Then please do that and then you can see that yeah you've got. You can start looking at uh generating those charts and pictures of what's going on, and you know what rules are being triggered.

C

um So you can then start seeing uh threat maps of what's happening, live on your systems and so on and what's going on over time um and there's a whole load of resource out there. um So again, there's a whole lot of links here that we will be able to provide you with after the event um talking about you know the various technologies that falco is built on um things about ebpf and so on um and various other sources of information around falco.

C

It is a growing project in terms of both you know, usage and contribution, and you know the momentum it has is growing. So please, if you want to get involved, um there are a number of ways you can contribute so falco rules. As I mentioned, we have that security hub where we're always looking for new use cases. So if you start using falco, you develop something yourself and you think it's useful to the community, please to contribute it back.

C

If you want, if you have a new integration that you'd like to see in sidekick, well, you know you're free to develop it yourself and contribute it back as well, and then um we're always looking for additional uh use cases, workflows um third-party integrations documentation and so on. So it's. um If you would like to contribute, then there will always be plenty for you to do so. Please contact the project. Maintainers um and they'll be happy to give you something to do um so. How are we doing for time? Yeah, just a few minutes left.

C

um So just a quick word about um so I work for sistig, um as does chris um and with what we the the originators of falco, it's now cncf, but we still use it heavily within what we do.

C

So we start with falco as the foundation for cystic secure, um where we're using it embedded into the cystic agent, um but we're using exactly the same uh three information sources we've just been through, so the system calls the kubernetes audit logs and the cloud runtime audit logs um and what we do from that is we build up from there to create a secure, devops platform.

C

So most of what I've been telling you about, you know the third party integrations and um you know, building your own seam and all of that. Well, all of that integration work and that policy management work comes out of the box with uh systig secure and we also add in additional capabilities like image scanning for vulnerabilities and misconfiguration cloud compliance from a static point of view. So you know things like cis benchmarks and compliance standards for your cloud accounts and for your workload platforms um and then in terms of what we do with falco.

C

We enhance the capability of it. So the policy management is centralized, inner cystic back end, which you can access via web ui or api.

C

um It can also take interventions like kill, stop pause, containers and so on, um and then we also have integrated incident response, so we can trigger captures from falco rules as well, which is another cystic technology which builds on the same capability, where we're just essentially dumping out that capture file, that's generated by the systick runtime to then do post event, forensic analysis.

C

So um if you're looking to adopt cloud native security within an organization, you have the option of building it yourself.

C

um We'd argue that possibly your time might be better spent, actually um furthering your business and that you know systig will give you a whole lot of stuff out of the box that you then don't have to build yourself. But it's building on that same open source capability to run. All of that.

C

So um with that, I think that that will uh cover all I wanted to today I mean there's a whole lot of features in there in terms of what systig has I'd be happy to talk to that talk about that at length. um But at that point I just uh open up and see. Do we have any questions.

A

I don't see any questions on slack or in um youtube, but folks are online. Please feel free to ask questions in either of those portals, uh as as tom had just there. If, if you're watching this on the replay, you can also ask questions in the falco falco slack channel, which is over on the kubernetes uh slack group rather than the cncf1 um but uh yeah. Thank you. Thank you very much tom. It was a great presentation um josh, I'm not sure if you've got anything else to add to that.

D

uh No, no, that was fantastic thanks very much tom. um In that case uh you know, that's the that's the end of uh of this track for today, um just a reminder that we will be back tomorrow uh so um follow us on twitter. Keep on the lookout um uh head over to kcd uk to io for all the information.

D

um If you want to head over to the track one youtube, uh I will post the link in the in the youtube chat. uh You can also find it again on kcduk.io, we'll be finishing off and curry's talk on, kubernetes is doomed um and then we'll be um kicking off with uh with the pub quiz. So we'd love to have all of you over there um to join us for that. um But thank you very much and and that's chris and I are out for today. I think pop.

A

Quiz should be a lot of fun, so do stick around for that, and thank you for everyone for taking the time and energy to to stay tuned um hope it's been a good day for you. It's certainly been a lot of fun for us cheers.

B