Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Threat Hunting at Scale: Auditing Thousands of Clusters With Falco + F... Furkan Türkal & Emin Aktaş

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Threat Hunting at Scale: Auditing Thousands of Clusters With Falco + Fluent - Furkan Türkal & Emin Aktaş, Trendyol

At Trendyol, we are running thousands of production-grade Kubernetes clusters to make our customers always happy. The challenge that we have to achieve is to track every component, resource, user, and team in a timeline manner. This is where we have to collect audit events from almost everywhere! Kubernetes audit logs can effectively track the changes made to our clusters. By using Falco, we consume the kernel events and enrich those events with information from Kubernetes. Enabling Kubernetes Audit Logs feature allows us to scan audit events that forwarded from Kubernetes. By using Fluent Bit, we collect logs from different sources such as containers and Falco; furthermore, we extend them with filters, and send them to multiple destinations. By using Loki, we build a highly-available log aggregation system. We create and manage all of our alerting rules for the log data. In this session, we try to combine pieces and introduce a brand new Audit Monitoring System!

A

Hi welcome to keep calm and sincere. This session is pre-recorded and we are speaking to you from the past. Our present stars will be watching the chat and please feel free to ask any questions. We will talk about threat of hunting and scale by using kubernetes audit logs and current kernel events, and in this journey we are not going to dip that into each technology that we utilize for our purpose, but it's more than that.

A

We want to tell our user story to show the way how we did it and demonstrate how everything works in harmony and our production. Let's get started, I am for country, cal and international. I am maintaining production level clusters as a platform engineer also, I am contributing to various cnc projects and open source projects.

B

Hello, my name is imin. I have been working on containers and the technologies around containers for the last two years. I love working on open source projects and I love watching uh car videos. Okay,.

A

Before we get into.

B

It uh we want to introduce ourselves uh tranquil is a tech company that they focus on e-commerce in turkey and creates a positive impact on our country and the ecosystem. At 10 years we are running thousands of kubernetes elasticsearch, nosql databases and much more. We are highly motivated to include open source project into our systems and be part of open source project community and projects.

B

We are mostly managing our infrastructure on prem that is distributed across seven different data, centers running h8 kubernetes clusters. This is going to be our very first cncf presentation as a trend yo. So, as you see here, this is our infrastructure matrix. You can monitor our metrics at real time by just entering informatics infometrix.chandual.com and before we get into the presentation, we want to thank our platform team for their great work and send special greetings to our squad for the awesome teamwork.

B

Okay, who is this station for if you are curious about runtime security and trade hunting at production and want to know how you can monitor entire system? This session is definitely for you. You will learn how we detect rates automate the process of collecting and analyzing outlooks to monitor of potential security threats, then to edge these threats with appropriate actions. Moreover, this session is investigating this suspicious of this in your infrastructure and create a baseline for monitoring and alerting best of all. We it's all done with open source projects.

B

Okay in today, in this session, uh we'll talk about the threats and our security pipeline related to that will explain the audit logs the runtime security, the log processing and the monitoring. Besides, we will reproduce our mindset in a demo at the end of the session. Furthermore, we want to share some ideas for cluster security monitoring.

B

Okay, what is threat code 19 is a trend to humans. Of course, everyone knows what rate is before we started. uh We want to simplify our subject here with this well-known example. Nowadays, common 19 is losing its effect, hopefully, but we are still suffering from cone 19..

B

This sickness is simply simply uh iterate to us when you become you'll start showing some symptoms like high fever or cough. So it's like. Basically, the trade is coming from outside. It's not, and our body is monitoring itself and starts alerting for us to take action. When we got ill, we are going to follow the same basic for our kubernetes environments, threats.

B

We should focus on our defense to minimize the risk. We have to know what we are surrounded by like here. We have a bunch of examples here which containers reading credentials accessing the file systems, so there will be always actors to intimidate your environment, and you have to be aware of the messages, application and user to protect your environment. So, first of all, you need to define your trust boundaries here. Our transponder is basically our clusters nodes and, of course, containers.

B

The main key point here is to create a focused defense area that we should create fine-tuned rules and alert mechanisms, especially if you work on a high scale. You need to label your logs to mark where the data is sourced. If we are able to find repeatable patterns to detect threats by person looks we can easily optimize our hunting here, our security pipeline, and we call it our second path- security pipeline at runtime. Basically so we're going to cover our threads huntingtop in four different headlines.

B

Each uh each in the each headline will uh introduce a open source projects and will show our configuration and end of the session, like we said, we'll use those application in in a demo if you're not so familiar with these technologies. Don't worry, we'll explain everything in a brief interaction to each one. So here it's time for fulcrum to explain. Audis and falco.

A

Thank you for a great introduction, and here are audit talks. System visibility into events in the container and system give great insight into what a particular or malicious user is trying to do. We can define our audience as answer these questions, such as when did happen, what happened with going and who initiated it and so on, which are defined in the kubernetes documentation.

A

Very well kubernetes audit records to actions and provide an audit trail of users and workloads which are called kubernetes api kernel level. Auditing, allows us to monitor system calls file, accesses and much more things. Eventually, we obtain the information about what's happening in the cluster.

A

Kubernetes point level events for other end, every action that we do when we are interacting with the clusters and using cuba, ctrl or any other things go through the kubernetes api server. What kubernetes audit does is provides chronological set of records that document all the changes to the cluster and each audit event is actually json object. That includes event time response status, who initiated it.

A

Ip addresses user agents, and you name it all that kind of information you can either active before you set up the cluster or you can access the master, not activate the kubernetes audit flake, but you have to restart the kubernetes api server eventually and human is already tested for two weekends and one for log backend and other another one. We probably can't we have can simply send events directly to the top stream server, while log backend is stored in the file system.

A

For our case, we are going to use log backend, and you will understand why, in the next slides.

A

B

A

Be recorded, as you can already see, a simple policy configuration that has two different rules and we might listing the ports, for example, doesn't give you any information secretly right? Getting a config map reading posts, tattoos, whether it is video or not, and this feature actual policy provides you to minimize the variability like changing the lock level or eliminating unnecessary logs and, for example, humanist is logs net, not much critical and for different kind of resources.

A

Falco is a runtime squared engine, it's open source, equal tool for continuous risk and threat detection across kubernetes containers and clouds and function detects unexpected barriers, configuration changes, instructions and data tests in real time. Falco also has support for kubernetes audit and kernel events and uses has different methods to collect the data, which is the kernel model and ebpa probe.

A

Here is architecture diagram of falco. The diagram explains pretty much everything itself when falco engine starts, it simply loads. The rules, the engine then waits for the events entered by lip escape and lip ins complements. Basically, lip escape is responsible for collecting data, while leaping is responsible for enriching the data and what we can collect system events using the kernel, module or evpf prop you can only choose one top active and also supports kubernetes id, and all you have to do is enable the web server and send events to it using flank bit.

A

In summary, it simply takes the events message time with the rules and then, when the event matches one of the rules, falco simply sends it to one of the outputs which is file grpcc your book, and you can also use falco, sidekick, falco client go and falco exporter.

A

B

A

Configuration b eliminating our falco using the hand chart and over affirmation capture is enabled by default, to change the value you can use overwrite this radio and and also it's possible to add new rules or or write rule files, as you can already see here, and these configurations are just tip of diaspora by applying the following configuration.

A

Now we are ready to scan kubernetes id blocks.

A

Fault, gurus follicle has a weaker extent rules with the conditions and macros for more flexibility.

A

It also ships many rules by default, and these rules are stripped by each other contacts, for example, postgres, kubernetes, elasticsearch, etc, and you for example, you can modify conjugate specific conditions or you can add your custom logic with macros. Without changing the rule itself.

A

Okay: here's an example of what the falco rule looks like we are using this use case for detecting a privileged container that launches in the cluster. It's a simple logical expression and combined with some conditions when, first time, when you deploy default rules, which covers many aspects for a variety of situations.

A

However, this might cause some noise in your environment. This noise might mislead you if you are at a high scale and draws false, meaning force positives falca provides a weight or right rules.

A

In this example, on the left side, a rule applies for pitch containers, but you might have some custom privilege containers and this could fire for positive highlights. In this case, we all write. Two is two aspects here. The first one is macro where you can define logic here, and this micro requires a load. These four privileged containers in two steps. We have reduced those positive situations.

A

Follicles also supports rule exceptions and exceptions are defined as a part of the rule. The file property contains one or two more files that will extract a value from the audit events. The comps property, which is you can use, contains, equals greater than less than completion operators and that align with the items in the file property.

A

Each item in the tuple should align one to one with the corresponding file and completion operator.

A

After we tune the rules by enabling the exceptions and for each rule, we finally got rid of a bunch of false positives, as you can already see here.

A

Okay, I mean it's your turn.

B

Thanks fuku, so he he already mentioned about the kubernetes logs, how we write them into a file, so he already talked about the falco, so we are collecting logs from the kernel things to focus uh the kernel module or you can choose epp approved. Let's talk about how we collect it looks and how we can send our looks. Falco flatbeat is a general purpose load processor. It also has metrics collection capabilities for embedded linux systems. You can run flap it in any environment, such vm embedded devices bare metal.

B

For our case, we are going to run in kubernetes clusters in.

A

Our scenario financebeat.

B

Is going to be deployed with, as demons said, since we want to deploy on each node and we want to collect every uh collec every logs in in the node.

B

We decide to go with flat bit over flanked because it is designed to run high scale with low resources, and it is one of the efficient solution for containerized environments pipeline. Let's see inside of the box, this is the data pipeline, which is called in the documentation. It starts with input. Input collects the data in many different ways. For example, you can use kernel plug-in to collect kernel locks. There are other plugins to collect logs.

B

Eventually, you you'll have a lot of data in your hands, but you need to structure the data so parser comes into play here. You can convert your unstructured data into structured data. For instance, your application has a unique log pattern. You can define that pattern and the parser and you can apply it to any lock.

B

So here we have filter. Filter fixes are one of our challenge here. uh Twitter basically enriches and modifies the low records buffer buffer refers to the ability to store the records in memory or option in the file system. Until your output is delivered with routing you can send the data one or multiple destinations take and match parameters are critical here. The great part is always at the end. You can send the data to a service or write to a file.

B

This can be implemented as plugins like you can define elasticsearch or low-key, as endpoints we'll see in the next slides config flat bit based on our needs. As you see here, uh this is how we collect data for kubernetes api logs. Actually, uh we use here uh tail plug-in allies and tech. We defined here is an internal setting that is used in a later stage by router to decide which built which output phase it must go through, and it's it is worth mentioning that we use json password here.

B

You can always find correct values for buffer configuration in the next slides. uh You see the filter here. The key part is to collect out from different location with identity. In addition to the kubernetes audit event itself, we have all this additional information about where all it is coming from, like port name container name- and you can add another custom label here. If you have millions of kubernetes cluster, you want to know who produced this log.

B

These variables are set automatically by our see, I see the pipeline since we store all the flat bits configuration as a code and fortunately there's a built-in pipe named modified plugin. That does this job on the collected data.

B

Lastly, after we give identity for each event now they can be pushed in.

A

B

Remote hosts so that, finally, we can see the out out events and do threat and analysis on them.

B

Okay, why flambeat with locked backhand or republican backhand you? You might want to ask this as that question before you're asking that question we want to answer.

B

As you can see, log processor is very important, but also it's challenging to configure to make it production ready by using flatbed. We are able to modify the event and loads according to our business requirements. Furthermore,.

A

We can easily reconfigure.

B

Our output plugins and start a reload to entire class by clicking a single button in the pipeline, which is our cic the pipeline with flambeat we can send out logs to anywhere. In this case, we see we are simply sending out how it looks to different remotes. One is falco, the other one r is uh the other one. Is our indexing storage in the next slides uh we'll talk about uh monitoring?

B

It's actually how we increased uh the visibility and how we enabled the observing the performance of security, although there are other options around we'll talk about how we used uh loki due to its similarity prometheus and it has uh can easily implement with the graphana. This is another part. Actually, it's very convenient and cost efficient to configure and easily create graphic panels with it.

B

Okay, log query: first, we need to organize the falco logs with loki. We are going to create a log pipeline which is being executed in left right seconds for each lock line. This is how our route logs looks like in graphene.

B

As you see, as you see here in the picture, it is very great to have the logs in json format, uh it's going to make our job very easy, you'll see it in the demo and we'll show it in the next slice slides, but unfortunately, this output is not applicable for panels. For now, however, you can easily filter in the logs here in the next slide.

B

We are showing how you can organize your logs in the left hand, side above you're, seeing we are using json parser here, so that json parser is basically uh can convert json json formatted logs uh and extract the uh logs labels, so those log labels can be usable in the metric queries a little bit below. As you see, the electric field is just detected by graphana, but you can use it in the graph final ui, but you cannot use them in the metric query.

B

So, let's work.

A

B

Let's work on the part log part that is also is in json format. The value is also json format. As you see here, we're going to use in the middle we're going to use the line, format expression to select a part, then we'll use the json parser again to extract labels on the right hand. Side you see all the labels and those labels can be usable in the metric query in the next slides, we'll apply a simple uh metric query. uh Low code has its own materials that can be applied to low queries.

B

You can create panels based on number of entries per second or number of bytes per. Second. Here is an example. How a metric query and a loquer is combined count over time, shows the number of entries for each log within the given range. With combination of some query, we can count long entries based on anything. As you see here. In the example, we are used rules we use the cluster.

A

B

Okay, some time for furkan to talk about challenges and some more metrics.

A

Thank you for explaining flatbeat and global for us and changes at high scale, and actually everything seems easy in the first place right, especially if you're working on high skill, there might be some changes that are worth considering because think might not work as expected for you. Of course, we had various changes in the process, such as eliminating false positives or writing the rules per team per project and making this fine tuned is really time consuming and we are generating tremendous amount of audits every second and meantime.

A

We need to find a way to build efficient indexing, aha backhand storage, which is highly available, and data storage, since our data grows every second.

A

We also would like to thank our teams to make it a highly available next specs, and we are surviving this such a tremendous amount of datas. What happens in seconds a tradition. We are monitoring and collecting audits from thousands of workloads. In a normal days, approximately more than 300k audits is being generated, each second, that's really huge right and for permanent, more than 400k kubernetes audits. Events is being generated and 8k of these blocks are being scanned by falco using flight bit plugin.

A

This is just what happens in a second for our kubernetes clusters. It random.

A

Okay, we we created some dashboards for to talk about all events and these dashboards updates in real time, and we are able to select time range and this one splits, especially designed for falco audits and we group by rule group by namespace clusters, clustering and neonamines.

A

We also active monitoring the users who gain access to running qr spots in the production environment.

A

Another, most importantly, we also monitor the users who read: secure objects, for example secret resource or our custom secret resource objects, inc kubernetes that store sensitive data, that's used by production services or applications.

A

Here's our weekly report, we also getting weekly, diverges for every event. We are actually watching this. A sample screen. Second grab for port exec detection,.

A

Okay, let's put all the mindset into them, I mean here you go. ah Thank you for coming.

B

So we'll put the pieces of the puzzle together in here before we start the demo, uh we actually pre-recorded demo for this pre-recorded video. We didn't want to take much time with the demo uh to reproduce a similar experience. We created an environment on our workstation with minicrypt. Also we published we published the demo on github. As you see the link in the below.

B

Okay. Can you please start the demo.

A

Okay, starting to develop, thank you.

B

So, at the beginning of uh our demo we are creating multi-node cluster.

A

B

We have cluster one and below we have cluster two when the cluster get ready, we'll edit the cube api servers for getting the laws in the host machine. Okay, since the cube api server is static, port, it will restart itself and we save the changes when it's done. Let's start collecting the logs with flat bit since we already mentioned the hand values earlier we'll.

B

We will just look at them quickly here and we will modify the filter to add custom class label and install the hand chart we'll use hem chart here, and you will see it. It's very, very easy to install all of them. Okay, next, uh we'll install falco, basically use the default volumes. Here, you don't need to configure much. It's configure itself with the default rules, so okay, falco ports are getting up. uh Last piece of the puzzle is loki for monitoring.

B

We install it along with graph on which it will come by itself, uh since we are using the loki. uh Look stack hem chart here, but we are going to just install it in the one of the cluster.

A

B

Cluster will define service and end point this way, we'll send the output our logs to locate easily since we defined our output host as a service name and the port number. Let's access the uh grafana, let's see how it looks uh if the logs are coming. Of course they are coming. We already pre-recorded this video okay, we're going to just look at the falco's look and, as you see here, we're getting the first clusters from the first cluster and from the second cluster, we'll we're going to organize our logs.

B

As you see here see here, we don't have much labels for now, but look label has a json format, so we're going to select that part and apply the json parser again and and end of it will have a bunch of labels.

B

Okay time to, as you see here, we have lots of uh labels here now. Finally, we're going to use count over time metric query, which is a special method created for locale, then we'll have a result, but it's going to be messy, since every log is unique. So let's use the rule label to combine everything together. So everything makes sense right now, so we're changing with the cluster, so we're seeing the number of logs based on cluster. Thank you for coming.

A

Thank you for demo. I get really enjoyed for this and, as you mentioned um just before the demo, if you want to reproduce the same mindset you you can clone the repository here and we also add a bonus section section just for you and we drop some ideas here, such as response engine police engine society and uncle alerts.

A

And the idea by using by enabling response engine is that you can take actions against very specific events. For example, ethnic container drift caused by cubicity, electric h or any other interactive kubernetes requests inside the cubanist cluster. You can find really good examples on the follicles block with the policy engine.

A

You can fine-tune your system by enabling custom specific policies that will prevent the malicious action detected by the run time at the very beginning, for example, prevent preventing the deploying privilege, containers and also you can direct your audit logs to secret operational center, so they can analyze and monitor and also take an action for the against event all the time and, furthermore, you can also define specific alerts for emergency call, for example, getting a secret or reading a secret in the production services such as payment api would be an emergency art, and the list can cause likes on this, and thank you for joining and listening us in the session.

A

Thank you for your time.