Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 3 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CSI Container: Can You DFIR It? - Alberto Pellitteri & Stefano Chierici, Sysdig

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

CSI Container: Can You DFIR It? - Alberto Pellitteri & Stefano Chierici, Sysdig

Digital Forensics and Incident Response (DFIR) capabilities are crucial to quickly containing the impact of an incident and preventing the cyberattack from becoming a cyber crisis. Indeed, when criminals get into your environment, it is crucial to adopt well defined DFIR techniques in order to minimize the incident impact. However, identifying and containing an incident was challenging enough in virtual machines, now with containerized applications becoming mainstream it is even more difficult. Following a brief introduction to DFIR, outlining its importance, a comparison between the traditional DFIR approach in on-premises infrastructures and the new way to be taken with containers will be presented. This will provide a better understanding of how needs and challenges have changed, particularly from the Kubernetes perspective. In addition, after a practical demonstration, the audience will get a clear picture of the best practices to adopt during the response phase - such as storing the evidence of a compromised pod remotely, highlighting and extracting the filesystem changes, and much more. To close out, it will be discussed how DFIR is evolving in Kubernetes, talking about the latest Kubernetes features and what capabilities they bring to forensics and incident response.

A

Hi everyone thanks for joining today, so today we are going to talk about um dfir in container, and now we can do the fir in the in container world. So let's start introducing ourselves, I'm, Stefano and I'm a threat, research that edit manager and at sysdig and as you can see from the picture I'm from Milano Italy here is Alberto. A peditary is a security researcher at sisdig and he comes from a tour in Italy as well.

A

Foreigner and Cloud security company uh sisdict donated Falco a runtime security project in today's cncf back in 2016 and is now the sandbox runtime security project at the incubation level, and this is in the process to to become a graduated process talking about Falco. We are both faco contributors and are also active in FICO, community and I'm. Also, if I could reviewer for file core rules.

A

Here is the agenda for today. So we're going to talk about the fir, especially is we are going to see incident response and digital forensic and then we are going to cover and refer to nist um incident response life cycle and then Alberto will go through some dfir container methodology techniques and also tools, specific tools for dfiring containers with some demos, and then we will wrap up the session with some main takeaways.

A

So, let's start, let's start talking about the fir As We Know The Firs put together two different areas: digital forensics and instant response. As we know, digital phrases focus more on collecting and analyzing data, user activities and other piece of digital evidence to to understand, what's happening on a on a machine or what or who may be behind specific threats on or the specific attack or whatever we recorded in our environment.

A

All these activities need to be done following best practice and specific methodology to maintain the chain of custody so that the evidence are are legitimate and can be used and presented to the court in case of legal proceedings.

A

Incident response is that uh are more, are focused on preparing, detecting, containing and recovering from a data breach or or a security incident in early years and early days, as we know, these two processes were completely different and separated.

A

However, I mean, of course, because they have different goals, but tools, processes, methodologies are pretty similar and now with tools like EDR xdr that are evolved. Of course, they empower the incident responder to do further actions like proceeding with investigation and also performing other actions, so it makes sense now to put them together under the under the same head and referring directly to the fir, uh and when we talk about the fir we used to refer to the to to the next instant response.

A

um Life life cycle step- and here are the four main uh Mains that are pretty well known. Of course, today we are not covering all the technical and non-technical aspects from from nist I leave the um the link behind in the in the slide, so that you can check out the the official this documentation.

A

If you are interested- and you want to know more about it- and of course today we are- we are going to cover the technical aspect on how we can do dfir in containers, which are the techniques, the specific techniques that we need to know in order to do that in in container worlds.

A

And so this is the time of the presentation where we start saying the best practices and, yes sure you need to follow the best practice and you need to check for misconfiguration right so I know it's kind of boring, but we need to do that. So jokes apart um when we talk about incident response plan, It's always important to mention that these need to be prepared in advance is fundamental. Attacker aren't going to ask you if you are ready to receive the attack or uh I, don't know we are in so.

A

Are you aware that we are already your environment? So we need to be prepared of that and and we we need, we know we need also to keep in mind that most of the or part of their uh instant response plan are non-technical. So there are some aspects like contacts people contacts, people that need to be involved needs to be defined in advance and people from different teams they need.

A

They specifically need to know which people they need to contact and and exactly what they need to do in specific and which action they need to do.

A

So all these aspects are not are more processes and is not technical related, but it's something that it needs to be set and and of course, if we talk about technical aspect, as we said, uh there are specific tools that that we need to run and and the list of the tool required might be set in advance and also uh all the knowledge about how to use the tool, or uh you know, specific flight that you need to use.

A

uh They need to be all the knowledge should be prepared and to be there before the attack, because, of course, we can't uh spend time on understanding how this tool work or oh yeah I need this tool to uh to do. I, don't know for a a an image, a forensic image of something and we need to understand how the tool works and other stuff so and all this kind of activities.

A

So timing is fundamental and we need to be prepared and of course, even if you have an instant response planning set, we need to make sure and you need to make sure that this is up. To date, I mean people changing companies, uh tools, change pretty quickly, as we know, and so all the list of tools that need to be used should be updated frequently and also the people that need to be involved need to be updated frequently as well.

A

Okay, that's fine! So we covered what we need to what we need to say and of course the fir is a pretty uh well-known practice and well-known methodologies, and today we are not going to cover the well-known tools um that of course, are still valid for containers. So we are not saying they're, not useful anymore. Of course they are, but they are pretty well known and we are just one to cover and we're just to focus on specific tools that can be used on on containers, so to do the fir activities in containers.

A

This happens also in other Technologies, like iot or an ICS. If you think about it, there are specific tools that need to be known and need to be used in order to perform some activities on the fir.

A

So, let's start talking about the fir in containers, so we all know containers and we are pretty familiar with them and we like a lot. It's super easy to deploy code. It's super easy to ship code that we want in our environment and deploy it. So, of course, we know all the advantages that container can bring.

A

But of course we also know that there are some challenging that we need to face, and especially for the fir, we know that containers are uh Dynamic and are ephemeral so doing activities like the fir in such a a dynamic environment could be pretty challenging and, of course we need to respond as quickly as possible, because some activities that Alberto will show now needs to be done when the container is live and need to be there then we know containers are, are ephemeral and and super uh and super quick to to destroy and we might miss some information.

A

So it's really important to gather this information and and then perform all other activities. So a little Alberto Deep dive into the fir and all the techniques and tools to use.

B

Thank you, Stefano I'm, the lover one so for this presentation. We want also to adopt a practical approach covering the previous steps that we have seen about nice incident response plan, and to do so, we are going to cover all of those steps outlining the main aspects that may be useful in the contourization words and along with tools, techniques and so on and so forth.

B

And so for this reason, let's begin from prep operation, so preparation is man is maybe one of the main aspects of incident response methodologies, because a good preparation allows you to not only prevent the incidents but by hardening I mean your environment, but also to enforce the right capabilities so that an organization is ready to respond quickly.

B

We can talk about facilities, Communications Hardware software, whatever you want, but we're going to skip all the organizative and traditional steps like set a word room or plan on site calls and so on and so forth, but we're going to focus our attention on open source tools that can help incident response, team to detect red flags, facilitate attacks, investigations and response.

B

So the first tool um I mean the first kind of tool that I want to um I mean to to stress is contain container runtime security tool.

B

um Falco is maybe the first cncf uh runtime security tool that allows you to spark threats at runtime, um see whenever there are some kind of threats that are running in your Linux environment in your containers and also to spot audits, logs and some kind of resources and requests that are made to the kubernetes API server. So you may want to spot this kind of threats at runtime.

B

You may also want to adopt other kind of Open Source tools and cncf graduated projects like Prometheus, in order to monitor your resources and see if there are some pickup loads, Network changes that may be relevant for your investigations or also some other Solutions, like logging Solutions, because one of the main best practices and that you need to enforce during the preparation step is also to log I mean as much as you can from your applications, so that you can I mean try to have all the kind of information that are needed and then with tools like fluent bits and fluid D.

B

You can try to forward these logs in just them, parse them so that you can have all of them with, for example, follicle alerts in one single login management platform like the arc stack, that is another open source projects like open search and so on and so forth.

B

So, in order to address the fairing containers in a realistic way, we're going to adopt kubernetes because I think that most of you are familiar with it and also because this is one of the main uh orchestration tools used by the open source community and as I mean, as many of you already know, in kubernetes, we can run the we can run pods that embeds containers in there, and so we can also perform the far in this kind of container. So, let's start from the demo that we have set for this presentation here.

B

We have in this case these kubernetes clusters, where we have few nodes where we have demon sets like file core fluent bits and Prometheus. In order to um in order to monitor our resources, we are going to forward this kind of alerts to the single logging management platform like aux stack, but, as you can imagine, in order to perform the far in a container, we need a workload that can be exploited and later analyzed.

B

So, let's teleport to the past- and let's imagine that we are in 2021 I- know that you're you're thinking that I'm going to talk about log4j because I'm not I'm bored about it. So let's talk about another vulnerabilities that he that was his closed in 2021 as well, that is the Apache HTTP server.

B

That was a puff traversal vulnerability that, under some circumstances allowed also to perform remote command executions, and so in our cluster we basically exposed our workloads, our Apache HTTP server, with a kubernetes load balancers that could be exposed by attackers that could be exploited by attackers. So, let's nowhere, the incident, Response Team Hot an incident response team is always monitoring alerts based on Falco detection. I mean alerts like the ones based on Falco detection that we set to be alerted as soon as possible and to promptly respond to incident.

B

Let's jump into our stack, and in this case here we have some logs that are related to one of our Apache HTTP Depot, where we can found something related to launch Ingress remote file copy Tools in container that is kind of speeches. I, don't think that it happens every day that there are core loan duplicate executions that are run at runtime in your container. So this is a red flag. If we go ahead, we can also see some kind of some other kind of alerts like as shallow spawned by the Apache httpd process.

B

There is kind of suspicious as well, and also attempts to steal sensitive credentials, but focusing on our our attention to the uh launching versus remote file copy Tools in containers. We can see that it was executed accrual wget that tried to reach an endpoint. The T is related to the IP address that you can see here, 194 and blah blah blah blah where you, where, basically that was basically used in order to download a malicious payload that we will see later.

B

So, let's now uh go ahead and let's see also the logs that were forwarded by fluent beat in our log management platform that are related to the impacted Port. If we see the logs, we can see that, for example, there were a bunch of get requests on top of this slide, and then there are also some commands that were not performed uh successfully.

B

There are also some other kind of um I mean operations that were not permitted and at the end, if we go ahead, we can also see a post request that is kind of suspicious because it basically tried to it X twice it: it exploited, DC, GI being in order to spawn a shell.

B

Other suggestions by the way may also come from uh open from Prometheus, as I said before, because Prometheus allows you to basically, um for example, query your your code that you have in this specific name space, for example, and you can see, for example, from the memory used that at the beginning there was these normal amount of memory that was I mean used by those ports in that namespace and at a certain point in time there was this speaker float I mean this is just an example also because this also means that I'm a bad programmer because I didn't accept resource, quotes our resource limits.

B

In my workload, but there are other kind of indicators like Network bytes that may be explanatory for your for your investigation purpose. So, let's now, let's now continue come on.

B

Okay, yeah I love the office, and now this is one of the main interesting part, because I mean incident.

B

Response Team um should always be prepared to such kind of events and should be able to respond as soon as possible to the alert that happen also because in your orchestration environments or whenever you have a container that has been impacted, you need to respond as soon as possible because, as Stefano said before, they are ephemeral and these kind of containers can disappear and you may lose the kind of informations about the attack that happened in your environment.

B

So let's now go ahead and let's talk about the real response that is engaged by the response team, so you may want to contain to to evaluate their education and also to recover your infrastructure so contain basically means isolate the attack because you only want to avoid that the affected, pods or container may impact other resources. If you are, for example, in a kubernetes cluster, the code that was impacted may have access to service account.

B

Those service account May Grant, you access to some requests to the kubernetes API server, and so, for example, the attack can generate a mass and, of course, you need also to answer availability and continuity to your uh to your resources to services that you provide, and you must also assess which resources have been impacted.

B

For example, your containers may cause opponent escaping container escaping Grant to the attackers direct access to the nodes, and these, of course, can create some other problems and what is also important for differensic step is to collection all the evidences about the attack. So as always, the first and the main step is to always snapshot the worker node volume, because because these grants um some forensics uh some forensics analysis uh that can be done later during the postmortem phase.

B

But you can also enforce, for example, some other commands like commit and push in order to store the container image that was impacted and also checkpoint. The container that we're going to see later and at the end of course, last but not least, you should also fix at the end of this step of this uh containment eradication. Recovery fix your uh fix the entry point. It was exploited by attackers, um so if it is a misconfiguration or a vulnerability, you need to fix it.

B

Otherwise you have to mitigate if, for example, the patch is not available so uh before jumping into the crime investigation scene, um I will like hold only to stress that there are some kind of useful tools and techniques that you may want to engage Whenever.

B

There are breaches in your containers, but I want also to strike to stress that many of these tools continuously evolve, because there are some kind of tools that, for example, you could use few years ago in Docker, but this, for example, cannot be used anymore in some kind in I mean up up-to-date kubernetes cluster due to the docker shim being deprecated.

B

So there are, of course you can interact, as always, with the container engine or with the kubernetes API server, to do some operations that you're going to see in a while. But you may also want to adopt tools like containerdif in order to check the differences among two different images or Docker Explorer and container Explorer in order to check any kind of differences.

B

uh I mean in order to recover the state of a previously snapshoted worker, not volume, so that you can see which were the previously running containers by.

A

B

Way, that's done, let's start now from the exhalations tab. So in this example, as we said, we are in a kubernetes cluster and we know which is the code that was impacted, and the first kind of thing that we want to do is, for example, to label the the port and then apply some Network policy in order to isolate the port.

B

In this way, if there was an attacker that, for example, exploited a river shell, it will be cut off the body itself, because the network policy doesn't allow any kind of Ingress or aggress traffic later you may want to also to call John the Pod denote where the Pod was scheduled. So in this way, all the new kind of workloads that will be deployed in your cluster will be scheduled elsewhere.

B

So in other notes, and doing so, you are now able to start some other forensics processes like, for example, snapshot the worker node volume or restrict the network access to the um to the worker node itself. You may want to, for example, remove the IP, the external IP that can be used in order to reach that kind of worker.

B

Node and create an intermediate VM, but all these kind of steps are mainly related to traditional step, and our focus is right now to to focus I mean to to dig deeper into the investigation of the container itself. So let's now assume that we have them first, some of these steps that I've just mentioned um and that I'm going into the machine into the worker node that was uh that was scheduled that are scheduled, the impacted Port.

B

So um we are now into the worker node itself, and one of the first things that we want to do is, for example, to interact with the container in time.

B

um Of course, you must be sure that, for example, there there was no pods uh escaping or container escaping, but we're going to see the eradication um their education step later, but one of the main things that we you may want to do, for example, without interacting directly with the kubernetes API server, for example, is to interact directory with the container engine, see which are the logs that we already seen with fluent pits see which are the running processes, and in this example, you can see at the bottom.

B

The kinsing malware- um you can also see that there is, uh we used also the uh CTR snapshot, div that basically allows you to check, which are the changes in the in the file system. So if, for example, the attackers we're able to download some files in the file system- and if there were some changes in the file system itself, you can spot them from there and you can see that in this case we're engaged the TMP path where they downloaded some suspicious file.

B

You can, of course, uh commit the impacted container so that you can push that image in a remote, uh replace in a remote registry or store it as an archive. This kind of, of course, committed and stored image can also be restored later from a sandboxed machine from an analyzer machine that is isolated from your product environment and, for example, in some I mean some strange other cases you may want.

B

You may also want to use a female containers that became stable in version 1.25 and that basically allows you to mainly debug the the container state or some attacks evidences. They can be useful because um they are simply some kind of strange containers that you cannot touch to the to a Radix system, pods accessing directly the same namespace, the same process name space of the impacted container, and you can see, for example, which are the running processes.

B

Of course, there are some all disadvantages, because if there are some threats, they can understand that you jumped into this kind of process, name space, continuing them space, and so they may be dangerous, sometimes but useful. If, for example, you have some this strategy image that don't have any kind of shell or tools utilities and in a postmortem analysis, you may also want to, for example, um pool or extract the images that that you you previously committed and pushed in order to download it in another in into analyzer machine.

B

You can so in this way. You can see, for example, which were the changes in the first system or- and this is one of the best practices, of course, using these snapshotted volume of the worker node that you have previously created. You can use that snapshot that snapshot volume so that you can mount it into an analyzer machine and from the analyzer machine you can use tools like container Explorer or Docker Explorer, if you're using, uh if you have previously used Docker in order to check which were the previously run in containers and hayab.

B

Here also this kind of video where I'm showing how to use container Explorer. It basically allows you to check which were the previously running containers in the previously. It's not shorted volume, and from that you can understand, for example, that there was the Apache impacted container. You can use the mount functionality so that you can mount the previously impacted container into the analyzer machine container and from the analyzer machine container you can run all different processes that you want all different steps that you want So.

B

At the end of all this video, you can see that I was able to successfully mount the container the previously impacted contentful system and then see, for example, the TMP file system that was impacted by the affected container.

B

But okay, let's go ahead and another interesting functionality that Works smoothly in podman. But not in all the other container engine is container checkpoint. A container checkpoint allows you to freeze the container state, so this also means container processes container executions, so the file descriptors are opened by the con by the processes themselves. Also, some also the TCP connections, for example, stored them into an archive, and then you can later restore the container execution in a separated environment.

B

So this is quite useful if you want to understand, also turn time which kind of execution were previously launched into the um into the container. That was impacted by the way, this kind of feature, as I said it works smoothly in podman, but there are container engine that are trying to integrate it as well. I think that there is also the kubernetes checkpoint feature that is a beta, a better feature right now and allows you to checkpoint a container calling the couplet request.

B

It's it's still a work in progress by the way, so, at the end of this containment collection phases, of course you can do all the forensics processes that you want, but it is also important to understand um if the attack spread elsewhere. So containers, as you know, may be run with privileged permissions with cut sensitive mounds, and so these may allow attackers to escape from the container from the Pod to The Host. You should always take care about the security settings.

B

The security contacts that you create in your workloads here, I posted for examples to kind of security context and their analysis that was done by cubescore. That is another open source project.

B

Of course, if we are talking about, pods pods by default are bound to a service account token, but I mean most of the times. These service accounts token are pretty generic and doesn't allow you to do that much. But if you create custom service account, um you can communicate with the kubernetes API server create some new make some requests to the kubernetes API server, create new resources and technically also do some lateral movement to the to the kubernetes cluster.

B

And, of course, if you're talking about, for example, kubernetes clusters, um I mean most of the time whenever we create a Commodities cluster. Its worker nodes are bound to IM roles that have some privileges and if you modify those privileges, assigning um restricted access to Some, Cloud resources. These May Grant attackers to attack your Cloud resources as well and yeah. Of course, the recovery that I think it is one of the most uh implicit kind of thing to do at the end of an incident. So you have, of course, to fix the vulnerability if possible.

B

Sometimes it is not always possible and you want to enforce some mitigation like, for example, workload like, for example, delete the workload if, of course, is possible according to the services that you are providing or the workload that you have, or also restore the container and hope that the attack doesn't happen again in the future or enforce some um some Playbook of actions. Whenever. There is a kind of rule that is detected by, for example, Falco.

B

You can enforce some um some specific Playbook of action in order to say Hey whenever there is this kind of rule that is triggered, kill my container.

B

So in order to wrap up this session, I'm now leaving the floor to Stefan again.

A

So thanks Alberto for all the information shared and, of course, the last step that required in the ascent response plan is post incident activities. So we took all the action needed to contain, eradicate the the threat and we restored and recover the environment, and this is now back to normal and it's all working great. So that's fine and it's now time to do some retrospective to get lesson learned from the incident.

A

Some there are something that probably worked other stuff that probably didn't work, so it's important to understand why they didn't work and what didn't work and perhaps understanding how we can do better.

A

So the output of this of this phase is literally a kind of checklist of of points and a list point of uh of what we need to improve, and this is basically the input for the for the update of the of the incident response that we have So based on the feedback based on what we discover during during the incident, we can update our incidental response plan, make it better and be sure to avoid the same incident in the future.

A

So, of course, what happened we can do much, but we should take these as a good opportunity to do a better work in the future, uh so just to wrap up what we said and just get some main uh takeaways from this talk so, as I said, don't be caught unprepared uh prepare your instant plan as soon as possible. uh As we said, this is fundamental. You need to be prepared if something happened.

A

Of course, um we need to respond as soon as possible. As we said, a container are a female container may not be there. Some activities needs to be done when the container is still alive, so be prepared and act as soon as possible.

A

um The other point, as Alberto said container, are aren't isolated by default right, as we know, so there might happen like container escaping and and the attacks move to the to the host or to the node or in other parts of your infrastructure.

A

So don't just focus on the container that that was exploited during your investigation, but go farther and there might be more to investigate so be prepared of uh not just looking the container but also check all the rest of infrastructure see the lateral movements if there was something and be prepared to go further investigation and also is, is important to have full fully monitoring your your infrastructure so be sure to have all the visibility that you need in order to perform all the activities that we saw before.

A

um As we said, there are, and as we have seen, there are specific tools that we need to use and we need to know how they works and which tool we need to use. As we said, there are a bunch of different tools, but we need to stay updated and we need to know how to use them before the the incident happens.

A

uh Also is actually important and is actually the best practice to simulate breaches to see the response of the plan of the people that are involved to see if it's it's working or if you you need to involve, you need to involve different people, perhaps or update contacts or whatever so check. Your plans is not enough. Just create the plan and say: okay, we have done it.

A

We need to check it see if it's working, if not update it, and make sure that this is working great as you expected and then be ready for the for the real incident and last but not least, um as we have seen in order to do these kind of activities, so especially in the far for containers but same for other Technologies. We need people that knows and have background on kubernetes containers, but also instant response knowledge. Sometimes, of course we don't have the knowledge inside our organization or inside our team.

A

So it's important to hire or engage someone else that has this kind of knowledge and can be and can do all these activities in a good way right, as we said in also in digital forensic, the we need to maintain the chain of cars today, so it's important to do some specific activities in following best practice in in good ways so that we we can use this data in in court or wherever it was. So it's not.

A

We need to have this um uh this knowledge and we need to do the old activities in in a proper way. uh So I get that's it from us. Thanks for joining and I think we are actually running out of time. But if you have any questions we are here, uh we can answer offline. If you have any question, thank you very much.