Red Hat OpenShift OpenShift Commons Briefings, 10 Aug 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OpenShift Commons Briefing #90: Network Visibility and Runtime Security for OpenShift and Kubernetes

Description

OpenShift and Kubernetes provide the tools to deploy and manage containers at scale. But how can security be integrated into the workflow? In this briefing, Gary Duan (NeuVector) introduces the container threat landscape and the security requirements for the Build, Ship, and Run phases. Runtime visibility and security is especially difficult and requires automation and built-in intelligence to scale. He shows how NeuVector inspects and visualizes network connections and protects OpenShift managed containers during runtime. NeuVector uses behavioral intelligence to discover the container application stack and network connections, and builds a whitelist-based security policy to protect containers as they scale up or down.

Gary also demonstrates how NeuVector captures network connections for applications deployed with OpenShift and provides multiple security layers for protecting and auditing an OpenShift environment.

A

Well, hello, everybody and welcome again to another openshift common briefing. Today we have a new member to the open ship, Palmer Commons new director and one of the ways I like to introduce new members is to get them to talk about what they're passionate about what they're doing in relationship to open shift in containers and kubernetes and new vectors here and they're, going to talk about continuous security for containers and it's I know. Security is one of the highest priorities for all of us.

A

So I'm really pleased that Gary Doon is here, and the number of one of his colleagues is here as well, so I'm gonna, let Gary introduce himself the way that we do these sessions. As always, is we let them do their presentation and their demo and talking, and you can ask questions in the chat and we'll try and answer them and then we'll open it up for Q&A at the end.

A

So if you're watching this recording the first bit is going to be the presentation, everything and the last bits going to be the really interesting stuff where we ask all the questions so without further ado. Gary, please take it away. Okay,.

B

B

Discussing challenges in the container security and also introduce a new lecture, our solutions, the brief introduction, my name is Gary dawn and the co-founder and the CEO of new vector in your whacker. We provide security solutions for the container environment and our focus is to protect your applications and the services at runtime. So we think that's the most critical time and also the most vulnerable time in the container lifecycle. So today, I'm going to talk about first talk about the new challenges and talk about several use cases we saw in our customers, environment.

B

Also, I will, you know, discuss our solutions and I will do a short demo after that we can go to the cony session.

B

So we know that the container guys so popular, partly because the rise of bankers, this architecture, right in this architecture, the the traditional melodies ik applications they broke down into multiple pieces running as a container, so it's very scalable very easy to deploy and there is a consistent way to deliver. The applications, however, also introduced new new challenges in the sense of how to protect the environment so because the communication used to be contained and within the server now they are they becoming that it becomes.

B

You know the the the network connections between containers, so the attack surface is bigger. So this this kind of traffic is not semis. Those traffic like coming into the data center or going out on out to the external network. So these are the network effects each vulnerable clothes, so we we call them east-west traffic. So those are the something it's very interesting and we want to protect that.

B

Another point is the containers. They use a lot of open, open source code, but I'm not seeing the open source. Kali slice secure it just that the tiger they can reveal the code. They can discover the defects in the code and use that to attack your application.

B

So today, when people they think how to protect your the workload in the data center, they normally they put some security device with them for the firewall and against device at the entrance. They think this probably good enough, but soon they will realize that that's really not a good solution, because this device they are at the age right, so they they don't see the east-west traffic and they don't know the life cycle of the containers and they don't know how these containers are talking to each other.

B

Another thing is very sensitive in you know: application security is the insider insider attacks, so these insiders, normally they have a way to bypass the security check at the edge and intentionally I intentionally. They could bring in malicious, know the province into the datacenter. So so that that's another factor, you need to consider another one. Actually is all these traditional network device, a security device, the the because they don't have the visibility to the container lifecycle and the container itself. They are relatively short short-lived right, so they can scale up and down pretty quickly.

B

So here we capture some screenshots of traditional security devices. The fare was, as you can say, that, in order to keep up the changes of this very small application, all those device devices they have to create hundreds of new policies passing off. If you have hundreds of container holes or thousands of them, I, don't think this is as good solution that has skill.

B

So so then, this make makeup makes us to think about what is the really the good way to protect the container environment? So, in fact, container has some very good features that help us to do the security more effectively.

B

First of all container, they are they're Software Defined. They are very declarative, though the developers and the application owners they you, they use names, use labels, oh this metadata to define their containers and they can define the dependency between. You know the different containers. For example, we have a tight integration with open shift and the kubernetes platform, so we know not only that a lot of runtime information of a container. We also know how others are creates, how they are built and how they are deployed.

B

Another very important principle in micro service architecture is that what my design in Scenario ones, who make it to want to make it too simple things and to do them, while so each individual container day, has no relatively simpler behavior. Because of that, if we can inspect what the container is doing and use machine learning techniques to to build a baseline of this trusted, behavior is more effective. So actually I have worked for this one of traditional security wonders for tonight for a long time.

B

This very long time ago, we we already realized this so-called blacklist model is not working in many situations, but now is in the Pantanal environment, because we have this knowledge, knowledge resolve. You know the metadata and the behavior of containers with actually build the wireless model. We believe that is more efficient and also more accurate thing about all the false positives and false negatives. You know traditional device that rates right. So we believe that, while this model is more efficient,.

B

The way we all know one when we're doing security which to do the protection in depth, but we also think that you know to consider security in all states in the container lifecycle is also very important, so we call them a continuous container security. The in this slide, we listed several steps in the the container life cycle from build to shape to run time, so there are different solutions that covers the security concerns in all these steps.

B

So, for example, you want to make sure your system that container is running on mix, make sure those system ask pure right, so you should consider land auditing, no checks run dog. Her vanity is benchmark at least hard on your system, so OpenShift also provide some very cool features. Security features, for example, the access control secret management, no role based authentication, so those are cool features you should use and to secure the base for the applications.

B

But what about those containers at at runtime when they are running? So we believe that that's the really critical time to protect the applications. It's also very challenging think about. If you want to monitor the network activities in real and to inspect those, if there's a normally, you want to see that and once we investigate and eventually you want to stop those attacks before they reach the application. So those are something that people are looking for to talk about monitoring.

B

So here is a example of our customer in the financial sector, so they are migrating their applications into the you know from the traditional forms into containers, so they have a database application joining in the container. So after some you know application updates all the network change these applications, they stopped working, so they use our tools they found out. There are some unexpected network connections, so they want to understand if this really missed confirmational. Actually there are some. You know security concerns here, so they lock into all the network connections of the database.

B

Eventually, they figure out that some, you know, Isis certificates was misconfigured, but, as you know at this time the customer they they want to have more insights into what's really happening. They want to be able to drill down to you know, network connections. They want to see the behavior at the transaction level, Network transaction level or the packet level. You had a badge level, the traditional tools they don't work in in a very real because they don't know the container next, the container their location, their IP keeps changing way.

B

So these tools are not very useful to to capture the network activities. Also, normally they don't have centralized the unified way to perform all these functions, especially think about see. If you want to correlate these ways, the security bugs.

B

So once we once user, they have a good understanding of the behavior of your of their applications, their containers they want to protect and the enforce the security policies, so they they want to detect the violations. If there is a deviation from the baseline or not right, no matter, this is a network behavior the process behavior or the system behavior, and they also want to know if there anything suspicious happening.

B

They want to see that, for example, if the attacks coming into your datacenter or they make a lateral moves, they try to kill some home or try to break out from the container today they want to be able to see that, but not only that today, most of the customers they want to stop the attack at the real time. So they want to block those malicious connections. Sometimes they also want to quarantine the containers, so they can investigate also food, full context, alerts and logs that that's very important to them. So.

B

Here is another use case so a couple months ago, this is very high profile attacks ransomware. What really happened was that some database applications, for example, the open, the MongoDB and elasticsearch they were misconfigured. The reports are open to the Internet and the password protection was not enabled so attacker. They are just they just scan the internet, try to locate these kind of applications and they can get here and the transit password, so the application now they will be allowed out right.

B

So if, if we have a container behavior-based line already constructed, so this is pretty easy to detect those activities so because these applications they mean to serve internally, they they shouldn't have any inbound connection from from external network. So once we see that this is a deviation from the baseline of the behavior, though we should raise alert right away right. So this is pretty easy to cover. However, there there are the other applications they they mean to open to the Internet, for example your viable applications.

B

So actually we have some honey pots running in the public cloud. So every day we we saw all kinds of attacks. Mostly they are, don't we don't know port scanning or proxy scanning if we only look at the look at them at a network level that looks very similar, they all use HTTP. They owe us for 80. But, however, if we use deep packet inspection to parse those connections, the difference is very distinct, because your normal application logic and those proxy scanning activities, they use very different portals to wince that visibility into the application level.

B

So we can identify more other ones that attacks this is not. This is another very popular popular use case for almost all the enterprise customer we talk to most of them. They have a hybrid environment, which means they have some services or they containerized, and they have other services running as in the virtual machines or human physical servers. So today there's no very good tools for the customer to define the circle egress policy, so they they want to define the policies to only allow a group of containers to access those legacy.

B

Thurber's legacy services running outside the container clusters, so there's a no good way to defund ID policy in our policies. So this is one of the very popular, very popular it requests we heard.

B

Do we talk about the inbound connections and outbound connections? What is more interesting is what happens inside the datacenter, though we all know that, no matter how many security device security measures you put at the edge eventually that hacker, they found a way to tell you once they got in what what do they do? Typically, they will do some hope things. First of all, they try to.

B

You know to the you know, isolate their privilege, so they can have the root access till then they could run some no coupons to to understand where they are and their location and to find out. If there's an a you know the network device of security device between themselves and their their targets, they probably also want to download some software from from internet from the hosts in their control.

B

So as a security device, security solutions, we also want to you know the combine all these different techniques- the, for example, to detect the route isolation and to to see if there is any suspicious process running and also to detect, and they I expected network connections between different network internal services, also identifier tags and the cirrage those especially at the application level.

B

So using all these techniques we can, we can break the kill chain in different steps in different points, so that it's more efficient and effective to to contain and Magic eight the attack, the malware. You know those threats.

B

So now I can introduce what vector offers, and some of our you know feature highlights. So we believe that a security security solution has to be easy to deploy. So we support both screen, fill the deployment and the brunt of the deployment. Do we we ourselves are containers. You can deploy our containers in the same way as you deploy your applications. We have, we have, you know very tight integration supports with openshift and the kubernetes, so you can use the customer, they can use team inside.

B

You know, load balancers or all these constructs to deploy our containers. Once our containers are deployed, we automatically start learning the application behavior and try to establish the baseline. We also support several features in auditing. The will run docker bench we are the first to open source current is as benchmark also the first to integrate that into our product. We also support an ability scanning for the container at one hi, the for protection and the enforcement. We support our seventh segmentation.

B

We can recognize those connection network connections between the containers, also the between containers and external, a legacy services, so we can isolate them at the application level. Also at runtime we may detect the route isolation. We detect suspicious processes and we can detect in authorized exploit between those containers.

B

Once those violations and nostrils are detected, we are able to block the malicious connections, but without infecting the pool traffic the correct, so your service is still serving the legitimate. You know the clients. We can also contain the connection of quarantine, the containers. So in this case the container is running, the network is shut down the mistreat her they can log-in and you know to investigate the incident. So we also support the cab. We can also capture the network sessions Network packets, for forensic hoppers. If the customer wants.

B

So with that, I will do a short demo to show some of our cool features. The way I already have some application deployed, so I'm going to ruin a dirty towel attack. So 30, how is, is a kernel kernel bug right attackers? They can use this to try to get a root access to the system. I also do on another server in in my demo, I'd wonders in my Mac, but this is to simulate the host that controlled by the attacker may be running outside of your cluster running on the inside.

B

So once I get the root. Access using 34 attacked I. Try to establish a reverse shell connections to this server, so whoever control this server now have the root access to your container. So, let's see what we have, so let me switch to the our.

B

B

So, as you can see, we already have the application deployed, and this is the network connectivity graph. Also, the baseline of you know the network activities, and here you can see we run several.

B

You know several contenders I'm going to run this attack from one of these containers. I.

B

Just get into this.

B

Container all the ways I do that there are many ways actually in the Internet, the attacker they can inject. You know this knowledge called into your containers. Oh I, I. Don't want to get detailed. How I did that, but just for this time- oh I, just want to you know, run this dirty cow attack from this console and, as you can see here, on the left hand, side I have already set up a server running. So I just run this. Oh it's connect back to the server so from here.

B

I can do a list insightfully it's the file. There are files inside the container and the if I, 201 I am the root, so I have the root access to the flocking to the container. We go back to our UI and we take a look. What happens so we see a dreadlock here and you can see that this is abnormal behavior. This is a deviation from the baseline, so you can say the connection between this particular.

B

You know container can actually external network that's alert, and if we go to the incident logs, we see the log of the juice, loot escalation event and we also see what's a command that how's. That.

B

Yeah actually use the password to attack. This is a special system program to to attack this container I switch to the violation logs. So this is the violation actually from these clients from this container to the external, and if this is the real, you know external IP, we can use a IP reverse lookup to look at what they are so yeah. Let's conclude my demo.

B

Yeah yeah, thank you. So that's! That's all for this presentation. Awesome.

A

Work we're having a little bit of a discussion here in the background.

A

The kubernetes, cysts benchmark stuff has really been a hot topic lately too, because they did a really good job at what does it stand for the Center for Internet, Security I think getting the the new benchmarks out like incredibly fast from for the one point: seven release all right: it's been really cool and a couple weeks ago, I did a a tech talk with someone from Aqua sex who has started a little open source project around that and I was wondering if, if that was what you integrated to do that, but I think that's probably you know I, probably not that ridiculously hard to do.

A

The project was it's cooked, just called coop bench and aqua, set the depth and I.

C

A

Your this is the first time I've seen someone integrate it in a production way and not just in an open-source, scripting and kind of way. But this is it's really good to see that work integrated into our project and I didn't if anyone who's listening in has any other questions. That's one! Please ask them and I'll do that you did an amazing job. Gary I think that was a really great overview of the whole thing and I really like the idea of application.

A

Behavior, you know monitoring the applications, behave yourself, that's sort of a first medium on the approach. I thought this was very very well done.

A

Thank you, but the other thing that I was thinking was that someday I'm gonna have to do a briefing where I get together a bunch of colleagues and community people like yourself and do a briefing on how to hack a container just to get people riled up, because I think you know it's not like I want to give people the tools to do the hack or whatever it is, but I think people just don't understand.

A

Inherently like some of the issues around giving people rude access inside of a container and- and you know, we have lots of tools for scanning containers and doing things- you know nice things along that line, but on the inherent insecurities in some containers that I've been playing with it scary and to see the networking side of it done so well is very nice, so I.

B

Like to do that, we're done with the dirty politics- it's discovered almost one year ago right, so it still exists in the world because it's a kernel bug it's very hard to replace your whole system to fix that. Well,.

A

I think there's you know, folks, like Dan Walsh and some of the other and the Red Hat team, I think I think there would almost be like a like a timer I. Have this vision, visual of like having a timer to seeing you know, give people a registry of containers and see how fast someone can hack them or find it. You know you just if nothing I'm trying to get your stirrup trouble just I think it would be very educational to do that and then have someone explain you.

A

You have to stand up and explain what the hack was, or in your case you know what the network you know, how they have to network or something where it was running it because I it's we keep talking about security, but we never quite say what it is that it did in the dirty cow and the other things that you're showing a really good example.

A

So um thank you for that and if folks want to get a hold of new vector, there's an info at new vector, comm email address and you can always check out their website or you could hop on to our slack Channel and ask a question there. Because I think the new vector folks are hanging out in there as well as well as all the rest of the openshift commons community. Beyond over 800 people. In the slack channel.

C

A

They're all talking at the same time, thank god, but um it's a good way to reach out and touch and connect with people as well. So Glenn is there anything that you wanted to add. You could unmute yourself if you'd like I'll unmute you there, you go so Glenn if you're still with us, yeah.

C

Here, thanks Dan no I mean I. Think it's pretty comprehensive, so I agree with you that what the community needs is a lot more and insight into some of the threat, vectors and actual. Maybe a hackathon attempt to raise awareness of the vulnerabilities in some of the containers. I think.

A

I could get docker to sponsor that it could be interesting, but long story, short I, think you did a very awesome job today in the briefing I just wanted to. Thank you for allowing us to do the reschedule and apologize anyone watching this afterwards for our BBN issues. Yesterday and next time, we'll see what we can do is if anyone else has any questions, raise your hands and chat. Give you a few minutes here.

A

If not I will be posting this video and if you can send me the slide deck I'll add the slide deck to it as a PDF on blog OpenShift, comm, probably in the next day. So look for that and from at the latest by Friday on evening and thanks again and when you have a new release or a new thing that you're passionate about we'll be happy to have Ben and Gary come back and do another tweaking for soon. So, thanks again,.

B