Red Hat OpenShift Red Hat Summit 2018 | OpenShift, 10 Jul 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Beyond Openshift Monitoring and Security: Troubleshooting and Forensics with Sysdig

Description

Everyone knows how microservices and containers are revolutionizing the way we deploy applications and maintain our infrastructure. But as many teams are starting to find out, containers have a key problem: monitoring and securing them with traditional tools can be impractical, painful, and sometimes plain impossible.

The Sysdig container intelligence platform provides a unified platform to deliver monitoring, security, and troubleshooting in a microservices-friendly architecture. In this session we will demo how Sysdig helps to:
- Monitor and alerting on Kubernetes infrastructure, services and applications
- Unified interface to troubleshoot issues inside containers with full system visibility
- Implement microservices based security policies looking at run-time behavior
- Enable post-mortem analysis and forensics on containers after they are long gone

A

Alright, hey everyone, I'm Knox from Cystic and we're gonna be covering troubleshooting and forensics. So normally in these talks you go into monitoring and security. We want to go a little bit deeper and actually go in and do some forensic analysis today before we get started on that, I want to give you a little bit of overview about who we are at Cystic and what we do. We were founded by this guy named lore stead. Hwanhee, probably most of you guys know from his past.

A

Tools are familiar with the tool Wireshark and a lot of the same approaches that we took in to getting visibility into your network, we're now getting using similar ways to get visibility into your containers. So we launched our open source project Cystic and from there we've got millions of different users of the Cystic open source tool and that's the base of our instrumentation for all of our commercial products as well.

A

We've got 300 plus enterprise customers using Cystic for real-time monitoring and security visibility into their systems and we're part of the Red Hat ecosystem, part of the container catalog and the open innovation labs. All that kind of stuff, so what I wanted to talk about today is how monitoring and security at scale is a different challenge. So, first on the visibility side, containers are really gonna break your legacy, monitoring and security tools. So how do you get visibility into what's actually running inside those containers? The application metric system, metrics any file?

A

That's opened any network connection, those user commands, and then how do you tie that back to your services that are being deployed by OpenShift, so not just protecting a container image, but then going back and scoping policies looking at intrusions and an events add more logical layers like openshift deployments or your different namespaces, and then how do you take that data at scale and manage it without having to add tags all your metrics and running into huge cardinality issues, so I'm going to talk a little bit about the architecture of our platform and we comes down to two different main components.

A

The first is container vision which I'll get into a little bit later, and this gives us visibility into your applications, your networks, the hosts and then service vision, which is where we're going to all the metadata and all the labels from openshift and your cloud providers and tag any security event or any performance metric that we collect with all the relevant and data for your troubleshooting and forensics investigations. This data can be sent to our cloud platform or to be deployed as an on-prem service. So, however, you want to run it.

A

However, you want to consume our services. You can do this and that's single agent and single back in provides full monitoring, security and deep troubleshooting and forensics capabilities. So now I'm going to talk a little bit about how we actually collect data from your systems. So this is a simplified host.

A

We've got a custom app running in a container another classic nginx image running here, and you can deploy our agent as a process or container that's running on the host from there we're actually going to instrument the kernel to see all the different system calls that are running to be able to see any activity inside that container anything that's happening from those processes and then send that back to your back-end from there. What you're seeing on the left is what a lot of your applications look like.

A

So openshift is scheduling your different pods, your different deployments across a bunch of physical nodes, and what we'll do is integrate directly with the OpenShift api's. To then give you a logical view of how those applications are performing, so you can write a security policy that says this deployment as a whole shouldn't have any outbound connections and then that policy will then scale to the hundreds of thousands of containers that rolling are rolling up into that service.

A

So this instrumentation is the basis of our open source projects on the top, so Cystic for real-time troubleshooting, assisting inspect for forensic analysis and then Cystic Falco, which is our behavioral activity, monitor, that's also open source and then our two commercial products of Cystic, secure and Cystic monitor so now, I'm going to talk a little bit about how that metadata changes, troubleshooting and forensics.

A

So if you're looking at something like the HTTP response time of a service, the 99th percentile, it's pretty useful to see, but if that service is made up of a bunch of different containers and pods, you want to know how each one of those individual containers is affecting that service and that's where the labels and metadata come into play. So what we can see now is how that service is now performing based on an individual container and how that rolls up into that aggregate.

A

Another easy example of this is doing something like detecting crash loopback events within OpenShift. So here we can look at the specific pod research ants that are happening within a namespace and then triggering those events before that pod restart starts affecting your service as a whole from a security perspective. You'll want to do the same type of thing so with that outbound connection that I talked about before.

A

If you go in and apply this to a Redis deployment, it will then go and protect all the images, all the different containers that are rolling up into that deployment, and then you can configure the specific ports that you'd want a whitelist and the standard behavior that you expect for that specific deployment.

A

So now, let's go a little bit more into troubleshooting and how you figure out the what the where the who the why? How we did this in the past is with tools like Wireshark, but what I'm gonna do really quickly today is take you through a forensic investigation with insisting secure, so this might be a little hard to do with one hand, so I'm gonna get Troy.

A

A

All right, so what we're looking at now is a topology map of my entire infrastructure. So this is a simple three node openshift cluster we've got a bunch of physical nodes connecting out to remote IP addresses, and here you can see all the different security events that have happened on a per node basis. But if you're, looking at your services and the health of those and the different events that have happened, you're really gonna want to enrich this view with that OpenShift metadata.

A

So now, I can switch this to look at my certain deployments and now we're just looking at the specific namespaces. So now, I can see each namespace in my environment drill into a specific name space where violations have happened and then drill in further to a deployment to investigate further.

A

So if I click on this WordPress deployment, we've seen an event for user spawning a shell in a container, so this is kind of your classic container intrusion that should never happen inside production and if I click on this I'll get further details about where this actually happened from a physical perspective, analogical perspective, so we're getting all the different openshift metadata about the namespace, the deployment, the pod, you don't have to do any additional reverse lookups log correlation every single event is gonna automatically have all this metadata enriched.

A

So you can know if you have to go change application, configuration or go and patch that host so from here using our low level visibility, I can jump off and see the commands that were executed. So, if I click on View commands, we can see all right. A user spawned a shell looks like they curl down a sketchy-looking rootkit. So this is something we're gonna definitely want to do further forensic analysis of and then unzip that rootkit.

A

So, by jumping over into our captures tab, we can open up a capture for further forensics and what a capture is is a dump of all the system, calls that happen, pre and post any security violation in my environment. So this will give me a full snapshot of everything that was going on at that point in time.

A

Let's open one up really quick.

A

All right, tradeshow Wi-Fi, coming through okay, so the first thing I can do is overlay that point in time when the event occurred, and so you can see. I've got all the data from the fifteen seconds leading up to that event and then the 15 seconds afterwards as well- and this is giving me a nice overview of all the different system calls and the activity that was going on so from here I can start to go overlay file, activity, Network activity.

A

Those executed commands that we saw before to spot trends across my environment, and then we can use these filters to only filter the data. That's relevant to the investigation that I'm doing at this point in time. So from here we can jump into those executed, commands again and then now go another layer further to see what actually happened when that rootkit was untoward inside my container. So if I double click on this char command and then because of all our file activity switch over to this files view.

A

What we can now see is every single file that was written inside that container. At that point in time, when that rootkit was unzipped and then because we're seeing all the system calls, you can actually go another layer and, in view and view the content.

A

So if I use our I/o streams button here and drill in make this a little bit more readable, now we're actually looking at the individual contents that were written to the container at that point in time, and the nice thing here is you'll have this full audit in forensics log, regardless of if that container is still running if it was started and killed right away, it's a full snapshot of every single thing that was happening on your system.

A

You can also use all this data for troubleshooting, so if you wanted to jump in and look at, HTTP requests see the errors and all that kind of stuff you'll get full visibility into really anything that was going on. So hopefully this was a quick overview about how you can use sistex data for deeper troubleshooting in forensics, and, if you'd like to stop by the booth to see more about what we do at the higher levels with monitoring and security. Please stop by and then I'll be here for a question.

A

So thanks for joining today, you.