Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 2 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Finding the Needles in a Haystack: Identifying Suspicious Behaviors... Jeremy Cowan & Wasiq Muhammad

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Finding the Needles in a Haystack: Identifying Suspicious Behaviors with eBPF - Jeremy Cowan & Wasiq Muhammad, Amazon Web Services

As the popularity of Kubernetes has grown, so has its appeal as a target. In an increasingly hostile environment, the ability to quickly flag suspicious behaviors and investigate and identify their source is becoming crucial. In this talk you will learn how AWS is using eBPF to identify a variety of security risks, e.g. communication with known command and control systems, Tor clients, cryptocurrency miners, and other malicious activity. You will also hear why AWS put eBPF above other options and the lessons they learned along the way.

A

So first let me welcome you. My name is Jeremy Cowan and I am a developer, Advocate manager for Amazon eks and joining me today.

B

I'm Muhammad wasser and I'm a principal security engineer with AWS in particular, I work with Amazon guard Duty team.

A

And today we're going to be talking to you about finding the needles in Haystack or identifying suspicious behaviors, with evpf and I'd like to start by first looking at the challenges that you're likely to encounter When developing a solution for detecting threats in runtime events. Now, for starters, the solution should be lightweight and stable. It also should be able to handle a high volume of events, and, assuming that you can overcome those challenges, you then need to separate the wheat from the chaff.

A

So all that you're left with are actionable insights and in this talk, we'll explain how AWS approached these these different problems and why we settled on ebpf.

A

So there are various approaches to uh this problem. uh We could have tried to extend the linguist kernel and the advantage there is that you have enormous flexibility. However, the likelihood that your change is going to be accepted Upstream is significantly, as is significantly small, because when modifying the kernel, your changes typically have to be broadly applicable, and so it's unlikely that you would choose to extend the the Linux kernel to solve this type of problem.

A

You could also write a a kernel module and again that also gives you a lot of flexibility, but a lot of folks are a little apprehensive about installing kernel modules in their system, as they can affect the stability and security of the system of the operating system.

A

um You could also try deploying a sidecar container, and this is good because you have a separation of concerns. You don't have to muddy your application logic with um security code, but this increases overhead and it can be circumvented. So, for example, if you're using an emission controller to inject a sidecar into a container, that's deployed into a particular particular namespace you could you could circumvent that by deploying a pod into a different namespace, for instance,.

A

And yet another option that has come to the fore recently is evpf and evpf like kernel modules, is extremely versatile in that it allows you to capture a rich information about events that are occurring from within the kernel and uh ebpf happens to address a lot of the challenges that I mentioned at the beginning of the presentation.

A

For instance, programs, ebpf programs are typically considered safe to run because they're sandboxed they also only have read-only access to the system called parameters. They can't modify uh the parameters of the of the syscalls um they're, also very performant, because they run within kernel space and they can be loaded dynamically. That is, they don't require you to reboot.

A

The system, which is really nice and ebpf, has really evolved, since it was initially introduced in the Linux kernel, 3.18 I believe it was, and at that time it was largely designed for filtering Network traffic, but it has since evolved, and new capabilities have been added to it. To now where you can uh deny certain syscalls um and the the implementation that we're probably uh a lot of us are probably familiar with is is SEC comp.

A

Okay, so let me uh spend a moment here to explain uh how it works. um So, first, uh the uh the operating system, uh loads, your evpf program or the byte code for your ebpf program verifies that it's safe to run. It's typically run through a verifier looking for things like infinite loops and that your program, exits, gracefully and so on.

A

The program is then, just just in time compiled and then run, and typically there is a an accompanying user space application that is used to load the ebpf program and then reads the output or enriches the output with additional metadata like in a kubernetes environment. uh It's it's valuable to know the container ID or the Pod name, but reading that output from ebbf is completely optional. You don't you don't have to do that um and I feel like uh this is a replica of Liz's slide from this morning.

A

um This basically is depicting how applications in user space uh communicate with uh with the kernel. um You know this happens when you have an application that has to access an area of memory or accessing a file on disk it. uh It interfaces through interfaces with the kernel through these syscalls and ebpf can attach itself or an ebpf program can attach itself to these to these syscalls, and that's where you can. You can get additional information about the the program that invoked that particular syscall.

A

So here's a very simplistic example of how guard duty is using ebpf. In this example, we have a process, that's running in user space, that's attempting to open a file.

A

It calls the system Handler and the the arguments that were that were used to call that Handler is passed to the ebpf probe and then that ebpf probe has the ability to send that information to a user space agent and with Zeke here he's going to run through this in Greater detail a little later when we get to his part of the presentation as for getting started with evpf I'll say this: if you're, if you're, not a seed programmer uh like like me, writing your own ebpf program um might not be very feasible, practical or easy, and, having said that, uh I I'm pretty sure there are plans for ebpf to support rust and other languages in the near future, which will make evpf a lot more accessible to developers and in preparing for this talk, I found three really useful resources.

A

The first is a project from solo. I o called Bumblebee, which I'll say automatically generates boilerplate code. Based on your answers to a series of questions. It also comes with the CLI, which makes loading your evpf program relatively easy. The other resource is a book that appeared on Liz's slide this morning. Called learning evpf um I managed to get through, like the first half dozen chapters uh very, very good resource on ebpf, especially if you're getting started and then the last resource that I'll mention here is the ebpf summit.

A

um All the talks from that event are now available to watch On Demand if you're so interested.

A

And similar to the earlier slide, where I showed you the advantages and disadvantages of different approaches to threat detection I've created a table here, showing the advantages and disadvantages of evpf. We know from earlier that it's extremely versatile and faster to write and deploy than say, Linux kernel module or to change the kernel itself.

A

There are other advantages like memory, safety and and great performance and portability with things like compile once and run everywhere, along with BTF or ebpf type format, yeah, which.

A

uh Which can translate with data structures within the kernel between different kernel versions? Yeah now the downside of evpf is that the tooling, for it is relatively immature and debugging.

A

It is still pretty hard, but I'm sure that, as it becomes increasingly popular that these challenges will be overcome as for common use, cases for evpf security, of course, but it's also found in a lot of networking applications and observability tools like Hubble, which you saw a few pictures of during the keynote, along with Pixie and lots of other networking and observability tools, and at AWS we love evpf. We have several groups that are using it. It's becoming increasingly uh popular at AWS.

A

Here are a smattering of examples of how AWS is making use of ebpf. Lambda is using it today to create pools of geneve network tunnels, and this allowed us to reduce the VPC function. Code start from 150 milliseconds to 150 microseconds, a pretty significant Improvement um and then VPC is actually using it in several different places. They're currently using it to observe TCP TCP flow level performance.

A

We have applications like S3 that can be accessed in a variety of different ways and so they're, looking at the TCP flows for performance and then later they're they're, looking at to looking at dynamically tuning, the performance of the TCP stack for different uh for different services and uh and their access patterns um we're also using it, um or we have a a distributed packet processing pipeline um written in ebpf, and then it's also used to implement security groups and and knackles, and if, if you used Amazon eks Amazon's managed kubernetes offering it has its own vpcc cni today it does not include a policy engine.

A

We make use of Calico policy engine or the psyllium policy engine in the future. We're going to add support for Network policy using evpf.

A

And then, uh why EBP, why ebpf for guard Duty, ebpf elected to use guard Duty elected to use ebpf for threat detection for a variety of reasons? First, it can be implemented quickly, as I mentioned. There's less apprehension about installing evpf programs than there is about kernel modules, because they're, because evpf programs are sandboxed and ebpf programs are relatively easy to install and update.

A

um Finally, ebpf provides Rich information about kernel events, and these these events can be enriched with additional information like container ID and pod name, which gives gives the the user additional context to help identify threats to their environment and ebpf can be used to provide protection at runtime, which also makes it very appealing for threat prevention, which you might talk about later right, yeah. So today, car duty is primarily using it for threat detection, but the appealing thing about evpf is that it could be used to prevent prevent attacks in addition to detecting them.

A

So with that I'm going to hand the Baton to lazique.

B

Yeah, so I'm gonna go I'm going to dive into some more details about how we are using evpf what choices we have made, what type of events we are collecting and we'll also go into a scenario and show you what type of events will will result in that scenario and what type of detections we'll be able to get um currently, we are primarily using ebpf for system called tracing, because it's very effective for threat, monitoring and threat detection.

B

When, uh when you're using system called tracing For Thread detection, there are three main objectives: number one you need to capture the input. Arguments of the system calls so that you could figure out what it is trying to do. Number two: you need to capture the details of the actor process or the process that invoke the system call number three. You may also need to capture the the written value of the system call, so you could figure out if the system call was successful or it returned. An error.

B

Ebpf allows you to, as Jeremy mentioned, allows you to attach ebpf code or ebpf probe to various Trace points or hook points inside the kernel.

B

For system called tracing one such hook point is system called enter hook or system call enter Trace point this Trace Point triggers as soon as the kernel starts processing the system call when an ebpf probe is attached to this Trace point. The kernel passes all the input arguments of the system call to the abpf probe. The ebpf probe can also get the details of the actor process from the task structure, which is an internal kernel structure.

B

Then it can send the input arguments as well as the details of the actor process to user space for further processing.

B

Another option for system call, processing or sorry system called tracing with ebpf is to attach an ebpf probe to an internal kernel function, which is invoked as part of the kernel processing of the system. Call. These type of probes are called K probes.

B

When you attach an ebpf probe to to an internal kernel function, the kernel passes all the input arguments of that function to the eppf probe. The ebpf probe can then take these input arguments as well as the actor process details and send those to user space for further processing.

B

In order to get the return value, the primary option is to hook into the system, called exit, Trace point or hook when you attach a probe to that or ebpf probe. To that the kernel passes the written value of the system call to your ebpf probe. You can collect the return value and send it to the user space.

B

Now I'll also talk about you know solve the security considerations when you are choosing a proper Trace point.

B

um I talked about the system called enter Trace point for act for capturing system called arguments. Although it's an easy and efficient way of capturing system call arguments, it is vulnerable to race conditions or time of check time of use issues.

B

Let's consider the example of open syscall. In order to understand this point, when you hook an abpf probe into open system call, as I said, the kernel passes all the input arguments to your ebpf probe.

B

One of those arguments in case of the open system call is path, name which is a pointer to a user space address that contains the path name of the file being opened.

B

Now, your ebpf probe in this case has to read the path name from from this user space address and some point later, the kernel also reads the same path: name from the same user space address. You can notice that there is a Time window between the time when the ebpf probe reads the path name and the kernel reads. The path name right and an attacker can potentially exploit this time window.

B

So since it's since both the probe and the kernel are reading the path name at different times from user space, a user space attacker can modify the path name between in in that time window. If that happens, your probe is going to read a different path, name than what kernel reads and processes.

B

A safe option or a safe alternative is to attach your ebpf probe to an internal kernel function, for example, in case of open system call one such internal kernel function is security file open.

B

When you hook to your ebpf probe to this internal kernel function, you can read the path name from internal kernel: data structures, for example. The file is struck in this case. This is not vulnerable to race conditions or time of check time of use exploitation, because a user space attacker cannot manipulate the internal kernel function.

B

Sorry, the internal kernel data structure, so we have learned if you are interested in more details of these type of attacks, you can view this Defcon presentation, Phantom, attack, evading system called monitoring very interesting presentation, so we have learned from the research on these attacks and we have implemented our ebpf probe in Secure fashion.

B

In particular, none of our ebpf probe reads: user space memory.

B

Now I'll talk a little bit about the the level of context which ebpf provides us.

B

A significant percentage of our customers now use container workloads on platforms like um Amazon eks when it comes to threat detection. The primary demand is that the detections should contain container level details. In other words, if a detection originated from inside a container, it should have the details of that container. Otherwise that detection is not very useful or actionable for them.

B

Ebpf allows us to not just provide the container and pod level details. It also allows us to provide all the process process level details which is a significant Improvement of on ah on guard duties, existing flow logs and DNS logs based detections, which only provide host level or ec2 level details.

B

A little bit about how we are collecting the context, details which details we are collecting from kernel which, from user space our strategy is, is to collect as much information or data from inside the kernel as much possible because that's more efficient and it's also safer.

B

So we are able to get all the process level details from the kernel, such as the PID of the process, executable path. We are also even able to get the container ID if, if a process happens to be running inside a container from the kernel, of course, there is some information which is not available inside the kernel. We have to get it from the user space so, for example, container image name image digest. We have to get it from the user space.

B

ah Similarly, kubernetes spot IDs, namespace and name is also obtained from the user space, and then the shaw, 256 hash of the executable is also obtained from the user space.

B

Now, what type of events we are collecting Linux supports, or has 300 plus system calls ebpf, can capture all those, but that wouldn't be very efficient. So what we do? We try to collect all the relevant system call events which are valuable in terms of thread detection, ah some of the main ones that we collect include process creation, execution events.

B

These events allow us to provide process level details. They also allow us to identify suspicious process executions. They also allow us to to profile or or track behavior of various executables and pods and containers.

B

Next up, we also collect file system operations such as file, open and file mounts file system mounts. um These events allow us to identify suspicious file system operations. They also allow us to track file system activity of various processes, executables pods and containers, and then another useful category is network connection events.

B

These events allow us to identify connections with known bad IP addresses or known malicious IP addresses. They also allow us to again track network based activity of various processes. Executables, pods and containers.

B

ah Dns requests and responses are also another useful set of data. They allow us to detect when a process tries to look up a suspicious domain name.

B

And then um another category is inter-process interactions or when one process tries to inject into the memory of another process. We try to collect events relevant to this category, because these these these techniques or these system calls are commonly used for in attacks like process injection.

B

And there are some other events that we collect. There's some miscellaneous events system call events that we collect, which are primarily commonly used in exploitation; techniques like um like halting kernel, processing or interrupting kernel from from the user space.

B

And last but not least, we also collect container creation events, because these events allow us to provide container level details and pod level details, and they also allow us to track behavior of various containers and parts.

B

The ebpf agent ah primarily collects events and sends those events out to the back end. It does not Implement any rules within the agent.

B

This architecture allows us to update and add rules quickly. It also allows us to perform more complex processing on the back end.

B

So after the agent sends the events we collect events at the back end and then we pass all the events and we apply threat intelligence to all those events, to identify to identify connections with known bad IP addresses and domains.

B

We then also Pass events to our estate less rules. These are the rules which are which use a single event in isolation, and then we also have more complex stateful rules which depend on more than one events and in future we also plan to pass these events to machine learning in order to profile behavior of various entities in customers, environment and use those learnings for anomaly. Detection.

B

Now, I'm going to go through a simple scenario in order to just illustrate what type of detections that ebpf allows us to generate.

B

This is a simple command injection scenario: a command injection vulnerability, exploitation inside a web application which is running inside a container inside a pod, so the attacker exploits or the threat actor exploits this command injection vulnerability in order to First download a cryptominer and then they download they execute the cryptominer.

B

And then the crypto Miner connects to a mining pool.

B

So, in the first step, when the attacker downloads, the crypto Miner using file system operations, we detect that a new file has been downloaded at runtime, which is kind of container runtime drift right. At this point, we don't generate any detection. We just maintain or store these States state that this is a new file which has been downloaded to The Container at runtime.

B

In the next step, when the attacker executes this new file, we detect that. Oh, this was a new file and it was executed inside the container. So we generate new binary executed detection.

B

So this detection is good in context of containers, because containers are supposed to be immutable at runtime. So there is detections shows that you know something was downloaded to The Container during runtime and execute it.

B

Next up using the execution process execution event, um we are also able to identify that in known binary, which is a known crypto Miner was executed using its short 256 hash, as well as using the the name of the binary and then when the crypto Miner connects to the mining pool.

B

We have the the IP address of that mining pool as a known IP, address or known mining pool IP address when it connects to that IP address. Based on the threat intelligence, we are able to generate a Direction, so these are the types of detections which ebpf allows us to generate the main difference. The the main point here is that that we are able to provide detailed context information in these detections.

B

So this is just an example of one of these detections, so you can see that we are able to provide pod level details and then container level details such as container name and then we also are able to provide process level details all the details of the process and even the lineage of the process or which, which shows you, the the ancestors of the process, like the direct parent grandparent of the process, and we are also able to provide the runtime context which is related to or which is specific to the finding.

B

For example, this is for the new binary executed file finding and in this case, binary path, which is the path of the new binary which was executed and the process that created the binary or modified the binary. We are able to provide the details of that process, yeah, so I'm going to pass to Jeremy to wrap up the presentation.

A

Great thanks, Lucy.

A

Okay, uh so to quickly summarize, what we talked about here.

A

Ebpf is an attractive option for threat detection, because it can capture events from the kernel, and the data can also be enriched to provide additional context, and it's really good for threat detection applications such as guard Duty, because it's lightweight it's portable and doesn't require changes to the kernel, and when it's combined with the power of the cloud, it can be used to find the proverbial needles in the haystack that allow you to focus on the root cause of a security incident uh and with that I want to. Thank you for coming to the session.

A

Today again, my name is Jeremy Cowan. uh This is Muhammad and we'll be here. If you have questions thanks.

B

B

B

um I I, wouldn't I, cannot say if there are any plans or like we are not actively working just to be accurate on those second question is.

B

A typical traffic site, foreign.

A

So is the question: what is the size or what is the performance impact of running an ebpf program.

B

Yeah, that's one of the major considerations right. The volume of the events that you collect on Linux workloads is pretty significant. The size of individual events is not that much in general, um it's typically. What we have noticed is that it's around 1K 1K bytes per event, but the the volume of events is pretty high. So, and this is the reason at the back end, we have to use certain several optimization techniques to bring down the cost.

B

Immigrated this is a feature we are thinking about.

A

This is going to be part of guard Duty Patrol, which was announced at re invent last year.

A

It's looking at the kubernetes auto lock, yes,.

B

B

Yeah, the question is: Falco is deployed as Damon said, um then. What's the demons had said and what we are doing and then how we are managing the resources, how we are making sure that enough resources are are available on nodes and secondly, what's the advantage over open source.

B

So we are working to offer multiple deployment options. One of the primary ones again would be using Daemon set on eks right and we are trying to make it as hands-free as possible for customers, the customers that choose to choose to do it right and then there will be other options as well, where customers will be able to integrate the agent with their CID, CD, Pipeline and then deploy themselves.

B

So we are working with to provide multiple options.

B

In terms of advantages.

B

I believe it's it's it's.

B

The primary benefit is: is the customers just like guard Duty, um the customers, the the the the customers are going to get kind of a ready product where they don't have to really pick and choose what rules they need to enable what type of settings that they need for Daemon set, or you know what um how they are going to set up the back end in order for backend processing of the events, you know, if the volume of the events is too high, then you need lots of engineering even for an individual customer.

B

You may end up. You know, deploying lots of processing capabilities in order to manage the volumes of the events. So that's the customers are going to be relieved from from that work.

B

A

How it reduced the VPC function start time, I, don't I, don't have the details on that um I I didn't have an opportunity to talk to a member of the team before this. This presentation, I did read a paper on it. I have to admit: I didn't understand everything that was in that paper.

A

um But if you come see me afterwards, I can give you my contact information.

B

Yeah as much as we could, uh we, we try to hook into LSM hook points or or the internal kernel functions which are invoked after the kernel has read the input arguments.

B

More than just secure security, exactly it was just an example: okay, okay, there's several LSM hooks that we are working in.

A

Any other questions all right. Well, if there are other questions thanks again for coming to the session, I really appreciate it. Thank you, oh and um don't forget to rate the session.