Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Getting Your Hands "Dirty" in Container Sandbox - Ariel Shuper, Aqua Security

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Getting Your Hands "Dirty" in Container Sandbox - Ariel Shuper, Aqua Security

The session addresses the proliferation of "sandboxing" techniques to isolate containers and improve their security posture. It'll provide a short background on the rise of "sandboxing" technology in the global security space and will drill down into different containers "sandboxing" technologies/projects. It'll examine and compare different sandboxing initiatives: Google's gVisor, Openstack's Katacontainers, Hardware based initiative (containers "enclaves") as opposed to legacy Linux isolation tools applied for Containers (SELinux and Seccomp). It'll analyze the benefit and the challenges of each implementation and will demonstrate the attacks types sandboxing/isolation technologies can mitigate vis-a-vis the attacks which sandboxing/isolation technologies can't mitigate and require additional security layers.

To learn more: https://sched.co/GrZ5

A

So we're gonna make a small game until everything gets off some unexpected challenges, I, don't know what happened, but it's got stack. So it's kind of rebooting or take some time and we'll connect the we're, not gonna play any game, but I'll start talking and I'll. Let you board and then hopefully shortly my laptop will recover from the accident and I give it back on stage. Okay, so everyone again apologize, those slides will be shortly getting on. My name is ariel super I'm, assuming direktor product management.

A

Aqua, is the only here in the audience not familiar with aqua. After three days, we thought to change our name from company to the Liz rice company, but I stupid, I'm late. So aqua is around my oh good, so I cross around micro services, security, providing security solution for cloud native workloads container is several a service containers, etc, etc. All of that period, aqua working, Check, Point, Software, Technologies I, don't work with firewalls wait.

A

Everything is connected: firewalls intrusion prevention system, sandboxing in a more larger space and now in aqua, working on securing cloud native workloads. Great okay, just give me a second.

A

Cpu utilization is high great.

A

A

A

Great thanks, God now all is working. Okay, perfect. So now why am I trying to talk about sandboxing, okay, so look Singh in general and then talk about container saying, because cell boxing fellows are familiar with the security. The larger security space was a huge revolution in the security environment. Why is your evolution? Because all the sophisticated security tools were very reactive, meaning there need to be a vulnerability to discovered and then only when the robbery was discovered or the exploit or the worm. Only then a fix could be applied.

A

Faith could be made now all the victims of the first discovery of this zero-day attack. Where victims they suffer. There was no cure for that, and all the security systems were reactive, only reacting to existing threads, more threads, more security at a cha-cha cha-cha-cha. Now this was very frustrating because you always need to catch up now when sandboxing was first introduced was very nice revolution because from now on, security scene can be very active that can be be before they can discover those zero-day attacks. So how does it happen? Any suspicious file was first emulated.

A

It was tested to see what is trying to exit what is trying to do to see the different patterns that classical malware's or backdoors take, and then you can you can you can reach a conclusion if it's benign or malicious and act accordingly. So in the classical security space, sandboxing was a huge revolution. So when we start hearing customers talking about sandboxing in containers, we start seeing more technologies evolving around several things. The containers we thought great. Now there is a real solution. We can all go retire same piece, everything's gonna, be great.

A

We can use cell boxing also for containers. Now, if look at the timelines, and last year was very very busy somewhere in May Google released, G Weiser in July I be released. Not locked. Containers in August was the first GA release, 1.0 of Qatar containers and even two weeks ago in during the rain event, those there they released, firecracker, which also another member of this family. So there was a lot of excitement in the past few months around containers in boxing, and we thought it's going to be great.

A

We started digging into it, exploring it and we found that the reality is a little bit different and that's what kind of the purpose of my talk we're going to talk about cell boxing understand the reality understand what is trying to do. What are the best cases to use it? So, a quick disclaimer. You can- leave this session with a cookbook of how exactly to use it, but you hopefully going to leave this session with some more knowledge about cell boxing different considerations.

A

What you can do you cannot do and what's the best way to operate with it great. So what I'm not going to talk about in this session I'm not going to talk about security versus performance? Okay, that's a very nice interesting topic, a lot of work about that used to kind of track, an excel file in Google Docs at someone keep putting up information, but last week OpenStack released a very nice study about performance benchmark. You can I have this reference and then of my presentation, some nicer interesting references. You can find it there.

A

It's really very interesting and thoughtful work on different performance. Benchmarking around containers. Now, if anyone in the audience ever worked with performance benchmark, you know this is the biggest lie in the industry. Why it's the biggest slide, because you always people try to pick the benchmark which their product core solution shine in their competition. Of course, look the wars, but I must say when I look at this OpenStack work was very nice very detailed. It was trying to be super objective.

A

Take a lot of real-life scenarios, really environments. It's not like trying to kind of highlight why kind of container is better, so great work. You all encouraged, take a look at it, it's very impressive, so this is covering security versus performance. I'm, going talk about a little bit about the operational aspect, but not kind of a into the details.

A

Also hope on that. No one's going to be frustrated, I'm not going to talk about simple exploitation. I'm, not gonna, show you how to break out of the different unboxing technologies. We explore it. We work it in in aqua. If anyone is interested is welcome to talk about a little bit more about it, but it's not going to be the purpose of my presentation, I'm going to talk about the actual technology, so it's not a child play.

A

Definitely it's a complex, that's so easy to use I'm talking about different security aspects of the different sandboxing technologies, and what about some operational perspective to see how it can be very tiny into real-life environment in turn environment, which contain multiple system, multiple environments with more containers and now what you can do with it great so before digging in into the actual technology, isn't going to the details, you probably want to ask yourself: what are we trying to solve? Okay? What are we trying to do with containers?

A

Why do we really need it right and before we go into the technology, understand the solution? Let's first understand what what's the problem, what are we trying to solve or trying to achieve so I think the ban in like the notable or the interesting attacks that happen in the last year, so I think the first one early in 2018 is somewhere around February timeframe saw some interesting report from red, lock and discover a very interesting attack on Tesla kubernetes environment, so Tesla kubernetes console was exposed on the cloud no password there.

A

Soit Eckerd is, of course, found something interesting thing without a password that went in interesting enough. They found AWS credentials, access key and secret key in this console, and then, of course, they approached the Tesla AWS account they found an f3 bucket, which was not encrypted inside is a third bucket. A lot of juicy information and from there was a very quick way to install their crypto mining. They need some kind of an interesting or smart crypto mining inside was that like utilizing, hello %, CPU was on utilizing.

A

Small portion of CPS was hard to discover. They use a network proxy as you can not kick a detect. Like you know the known IPS for crypto mining. It was very sophisticated until were caught and, of course, deleted. A month later, I saw an interesting way how to hack, ku braces cubelet, so queries cubelet. The API is, if you don't mention anything different, the API is allow any API calls to be made.

A

They don't need a phone to authentic Asian and then, of course, how people can use cubelet in order to install create a backdoor and install another crypto mining technology, and even last months, I receive an interesting note from Juniper labs, and they send me this in the preview. Look at that interesting report about showing how they can use docker API in order to propagate and in fact all the hosts in environment with interesting crypto mining.

A

So all those crypto mining a lot of crypto mining here, a lot about it, not pleasant, not nice, not comfortable. You know my big big call to the board, explain why? But it's not! You know it's. It's not critical right. It's not pleasant, but not so bad. But then last week no one prepared us for last way because last week those of you didn't know there was a huge vulnerability which was discovered kubernetes.

A

A researcher I think in Rancher found a way to reach polluted escalation and get root access to the entire cluster, and you know for attackers privilege escalation. Getting root access is the holy grail of any attack. You don't need more than that. You get root access. You can do anything which you want on the on the cluster. You can take all the credentials all the secrets you can destroy, delete copy, just you know, do whatever you want, and this is something which is really serious. This is not just crypto mining.

A

This is something which is really something that no one really wants. So when you're trying to analyze different attacks, then I think you know what I'll skip that, but it's easy to see that nothing. There is really new. Ok, the attack pattern are pretty much the same. What's new is the environment. There are more entities to attack, there are more entities that we can try to attack and we can try to breach, but the methods are the same are pretty much the same. You know ports, gaming expose for exposed.

A

Api is an authentication authorization, no passwords and no encryption, which is really easily to easily avoid. But there are more entities to look at. There is more entities going to secure and there is nothing we take into consideration now when we try to think about how like holistic container security, look like there are different vectors. We need to look at, so one vector is how we can minimize the attack surface. Okay, we want to minimize what and it the way on the option attacker has into our environment.

A

So how do we minimize the attack vector with like now remove ulnar abilities, will remove sensitive data like credentials or we can encrypt it, and we in trying to minimize any, like you know, create like network policy, creating authorization and password to anything that will prevent attacker to reach our environment. The second thing that we'll do is we'll try to minimize the tech potential.

A

Okay, now assuming someone managed to attack now we want to minimize what it can do in our environment, so we'll work in these privileges so can well require authentication authorization or for every attempt will use promise. Actually, control will encrypt everything so we'll make sure that even someone managed to break in then you cannot do too many things to our environment and last but not least, then, if someone already managed to break in passed everything we want to minimize what we call the attack radius. What are the consequent thing?

A

We can do what it can, access from the environment, what it can reach, what it can do, any additional damage which you can do once you already broke into our environment, and in this context this is exactly worse than boxing incoming. So cell boxing is taking care of the attack radius, we're applying a sandbox environment. You making sure someone managed to break in for some reason he will not be able to access the host. Ok will not be able to reach get into more containers into the host into another elements in your environment.

A

So it's important to take it into consideration. We look about continuous and sandboxing. I understand that sandbox is a good way, is a good thing to do partake part of our security posture.

A

So now, let's dig into more details and get more discerning about cell boxing. So first, what's the motivation? Okay, so we said understand the motivation. We know that, unlike different vm's or different entities that we used to work in the past containers, shame share the same OS same operating system.

A

They share the same kernel, it's great for the apps team, it's nightmare for security team white sniper for security team because zero they exploit or any type of agitation, allow you to break out the container and reach another or either the host or additional containers in your environment, okay. So this is why we need sandbox in science. A boxing in this case has a two phase or two dimension to its operation. On one side it isolate the container.

A

On the other side, it provides the container anything which it needs from the from this from the kernel right because, if contrary will access the current and why do we need two container to begin with? So this is the purpose of containers of sandboxing. But now you can ask me: why do we need that? Because, if I want to isolate my kernel, I already have many options to do so. I can use seccomp, which is a way to create white listing on white white. Lift the allowed system calls, so we stack up.

A

I can decide, define what a system call which Allah, which allow to the container to use black anything else and then I reach the same installation, because it's radically at least I cannot get more from the current and what a container actually can do. Another technology is selinux, which is a nice way to do access access, control to your resources. So it's an interesting way. Every process get a resource, a resource, getting a role and people can access different part of your of the allowed part of the Quran.

A

It's also a nice way to limit what you can do, so a person's can do more than what your lime to do, and this is another way to isolate your your kernel and a power. More is another example which is quite similar to setup, but the reality of all those tools is that they're not so easy to use or to manage. So the reason why I like suboxone, becoming an option, because all those kernelization technologies are great, but when you start using them, you see that they're not so easy to use.

A

So, for example, I tell you that we have an experience with second profile in aqua without customers to create an automatic second whitelisting profile. So we, you know, we took different containers: okay, laughs every container created an automatic way, the white of all the white white list. All the loud system calls, but in reality only few customer using it because it's hard to trace. If there is any issue there is in there is a change or modification. How do you modify it? How do you trace it?

A

If there is, you know, it automatically turn off the. If trying to get a system call, which was that white listed at what o'clock it feels very hard to debug it to find why it is, and if, by chance, weather modification to your container is gone off, just kind of fly, without even understanding why it happened. So it's quite complex to use I think absolutely not we're very experienced with it.

A

No, it's also the nightmare of any IT organization or develop them and why usually disabled by default, and even our mo, which is a more friendly way to whitelist system, calls I found a nice, get a project of get a project of just Raziel, doing an up-armored profile generating called bane, and if you can read the screen, you can see why it's calling this nice way, because it's just for hello, said you know reviewing up our profile.

A

Pull request is the bane of my existence right, so you can see it's not so easy to use, not so intuitive and when you want in that fast environment, both in the keno container environment, large-scale, probably it's not the easy way to use. Actually you either discussion at an interesting discussion just two days ago. During the event, without a big you know, industry giants that has two thousand engineers lot of skilled people, maybe even sit in the audience.

A

You know that have dedicated team which responsible for creating sec second profile a power profile, but in reality you know when your regular enterprise you're trying to apply to have so many other things to do probably only of this skillful team, which is only dedicated to create a power more profile, a second profile- and this is why you're looking for something which is more simply easy to use and doesn't require a PhD to get it done.

A

So, let's get into more details, let's see, take a look from the inside and let's examine the different technologies, so I think the first one, the most notable one or the most I would say immature. One is cutter containers who, due to the kind of container team and kind of container I, took the approach of taking in a hardware virtualization, so they create an isolation in the way that cradle took a neurotic hardware.

A

Virtualization technology and optimize it to fit into containers for every container or every pod is booted as a VM, with its own unique kernel, with no access to the host canal, which is great because now, if I manage, if I leverage a zero-day attack, Oliver G, any type of backdoor which I install in your daughter in your docker image, I can only reach the kernel of this VM I cannot reach the host kernel, which is great it's great for isolation.

A

It's an OC I, comparable runtime, which is really great and easy to use all the known commands very nice. It has an agent inside each VM and machine outside the VM, so you can intercept all the OCI commands or any I/o requests and translate it vice versa.

A

So the agent is coming like the hyper is providing all this calls for the VM and the same communicate with the agent and it's a great way, or at least very nice very nice architecture.

A

Now, let's talk about the security aspects of cata containers, so acara containers leverage qmu as a base for their cue light virtualization, which is great now when I look at all the work- and it is very nice and interesting work on that all of it was around performance, optimization, really optimize the performance, remove devices remove being quicker boot time, but other matter. What of about securities or security, of course, is a byproduct.

A

If you remove like a code room a lot of devices, then you remove that you minimize the attack surface, but I was really wondering what about the security aspects of cata container, what's really becoming more would say curious about it. It's an email which I I saw on the email exchange of cutter containers, email list. It saying you know it was pretty much around the release of 1.0 asking.

A

Are there any tools or available test to test the promise in cata containers? Aka? Did we really test the security of this piece and the answer? The surprising was no, we didn't test it. I said: okay, maybe I'll try to see it I, just kind of like a quick look and take a look at it, because that's all you know QM, you know it's not the best. I would say the technology security practice.

A

So I, look at the Q mu, which is the base for the cumulate and I, saw that Q mu in the last year. Had ten critical vulnerabilities not high, not medium, not low, critical, and it's only in the last year. So he's saying yourself, whoa, ten critical and the last one just few months ago was fixed.

A

So this is a big potential for vulnerabilities and you think about a little bit more is that early in 2017, Google removed, qmu doors to harden their virtualization environment and even let more Digger inside you see that almost any security firm released, vulnerability or discover a vulnerability in the q mu, anything oops a bit, not the best security tool or maybe not the best base for my security and this something which, from my perspective, put kind of a note on the usage of q mu as the base of my of the security level well of this isolation, looking into G visor, so G visor.

A

If anybody here attend the Google I deny very nice presentation about it. Two days ago, it's a really nice and creative way to limit user space to create, and you space, colonel okay. So actually there is a very limited access to the real to the host colonel, because any system calls any attempt the application is knowing is intercept by the century, which is the process. That's responsible for this colonel system calls and it provide any needed resource or support any file system access is done, is traced by gopher, so century use go friend.

A

Gopher is kind of a proxy that let handle all the system calls that the all them file system access at the application in so in this way, they can really isolate the host kernel and really minimize any potential system called access to the host, which is a very nice quote, psycho on steroids, it's very nice. It really limited the tech surface and it's a great work.

A

So it's creative, it's interesting and G visor include the no CI runtime called run SC, which integrates with darker with kubernetes and make it also very nice to use and any good tool or easy to use photo the familiar with darker and or once run with kubernetes.

A

Now the challenge that we see with G visor is that Google took an ambitious approach. Right and ambitious approach is to create a new user space kernel.

A

Now the kernel for many years was kind of tested was bridge, was patch was fixed and when you create your own kernel, you taking the risk of discovery, all those previous old legacy attacks, also in your environment, so I think there was no surprise that somewhere end of October, don't even zone from Google you'll find a way to do kind of breakout and reach the host file system by G visors I was discovering Google. It was coming early, October, fixed by the end of October, two weeks later, privilege escalation.

A

Another researcher found a way you do purchase collation and to rewrite files which stopping up the container and again, both probably I'll, expect to find more and more classical kernel attacks apply to the device or environment, and this brings us to the other is architecture. Cheese knob la flama took a completely unique approach to containers in boxing okay and using unique kernels, okay, using the kernel based on solo 5, using also something which call run from, and there are extras working with it.

A

But now it's kind of using some more higher level programming languages like note Python and go, which is really a very nice way to completely minimize the attack surface using seccomp.

A

There are now only seven system calls to reach to reach the kernel and just like anyone else, a support, noci compliant runtime and provides ability to use it in a nice way and also very secure now, I think it's no surprise that, since using unique kernel and using something which is very, very rarely use- or maybe even not what use before and all I've played experience with, has this operating system unique URLs in the audience, but there were more the rarely it's used. Of course, the less attacks you'll find I mean I.

A

Take us will not waste time on something which is really using. We didn't find any vulnerability, so on a security aspect, it seemed that nablus seemed to provide the desire host or kernel protection. Now, interestingly enough, nabla also created an interesting security benchmark call horizontal attack profile. What is horizontal egg profile? They try to quantify. What is the risk of every technology by trying to see how much code each technology creates a?

A

How many system calls it allow and create some kind of matrix to try to estimate what is the risk of every technology and I? Don't know if you can see the screen, but the lower, of course, is the better and habla is the light blue one which is very low and I think pretty much I think it's counter stood. It could be easily understood. So on the security aspect, they did a great work, we'll talk about operational aspects in a few seconds, and here you'll see it's a bit more challenging.

A

I am another alternative which is relatively new. I must admit. We have too much time to play with it or just released two weeks ago, just before cube corn. So in last we invent it is through this firecracker.

A

So firecracker is the new addition to the team. It's an optimized, virtualization environment and lightweight micro VM, using KVM and at Q mu, which is a good start. Aws uses it for all their service environment like lambda, like for gate and using it. For that, and which is it's a good approach, because we know a lot of customers came to us complaining about how do they make sure that no one is kind of attacking them from the whole star?

A

Can they be secure the natural where the containers or the servers are running, and this is a great way how you address this concern by isolating the initiation of the container or the function from the underlying host, where potentially attackers can hide? So this is a nice way, the optimized for performance very short boot time and also security optimized. The removal devices through those things really left with a bare minimum which made them move very quick on one side on the other side, also minimized the attack surface, which will leverage the previous benchmark.

A

The horizontal attack surface fire cracker also received a high score. I too, didn't update the benchmark. So that's why the inequality, but we applied the same methodology. You can see it also received a fairly low score. Now we don't support an OC I run time per se, so it support container D, but the way support continuity is kind of a in an interesting way. They also have a seam that translated the the OCI commands into the VM em.

A

They have an agent inside and interact with those command and another snapshot or something allow it to create image and I consigned in tiny containers, something it's an interesting way. It's not like sandboxing per se, like all the other technologies, can likely run any continuity or any kind of a runtime command, but it's a good way and they have a word map and I'm sure the Ridgid and it's open source, so everyone can use it.

A

So this leads me to talk about in test the security way, but talk about some operational aspect, because if we create great security tools, but those tools are nightmare to operate, then who's going to use them right. So we need to win, create think about balance between security, but also how easy to use how easy to use those security tools. So when I'm thinking about ease of use, I'm talk about installation, so G visor was very simple, easy to use who to the G visor team.

A

If there are the in the audience, did a great work cut a container slightly more complex, but again nice and easy. An oblong is also simple intuitive. Now, if you want to run those who want to run, you know the daily operation, so we know about the support for the OCR. Is runner has a kind of container?

A

They have a cutter runtime apologized for this cut off so I have the Nexus runtime all the different commands very easy to use again great work, the run the run is C from Google is also nice and easy to use, run and run yes.

A

The double one is a little bit more complex, but again, this is not the only challenge. There is because Knobloch come with love limitation. A lot of challenges you can only that you can only use their their base images very hard to create your own countries like any base image for country that you have like in in docker hub. You create only base image with the unique kernel, they're very limited. You can really create your own, but again the the effort required to initiate and start working with it is much more complex, is much harder.

A

So, on the operational side, there's still a lot of improve and on improvement, its required on the nabla side. Another important aspect of sandboxing is performance. Now. Why did you why it's important, because cell boxing contain or obtain a performance penalty, when using a sandbox or I mean there is inherent performance penalty, whether it's because using a hypervisor emulation?

A

It has, you know an I/o performance penalty or using a different thing. There is the you coming up with performance overhead, so whether it's 15% or 20%, on the iOS or on the CPU CPU. What I saw from all the benchmark are relatively low, but again there is more CP required any classical container, so CPU and memory over this still small boot time is very quick. You can still in less in few hundred milliseconds. You can be up and running and again very this is the source you can look at the OpenStack.

A

They made a very nice benchmark, all different scenarios, I also when using your networking stack in user space. Of course, it has performance penalties or this, while, if you have a system called io performance, all this hypervisor based approach will suffer. So what it means to us is practically what's going to happen is that cell boxing will be implemented selectively. So it's like of implemented all across the environment by only on those words where you will see a bigger risk or high risk.

A

So if it's going to be permitted selectively, then we just ask yourself: okay, so how does it coexist with a regular environment? So how does the coexist with like orchestration layer, artists with other containers, so the interesting thing that I found? That orchestration, of course, is a masked acara thing can really run in a real runtime without orchestration. So that's a must. Kata provide the best support for kubernetes, it's still extruded in G visor table. It's among the other challenges is also, although they they they in November timeframe.

A

They said support kubernetes, but the whole runtime is still experimental right. So it still they do support it. But it's still on experimental phase. Now all the same boxing runtime, interestingly, can coexist on the same host. So there is no need for exclusivity, you can run all of them or you can play with all them on the same host or put some containers with this one, sometimes with other ones and also I, think very interesting and very important.

A

You can run regular containers and sandbox coin on the same host, so you don't need to run only a it's not either all can do some. You can do part and something which make it very, very easy to use and allow you to apply cautiously some boxing on what you think it applied the best or anything it's most required.

A

So now, let's wrap up so in a nutshell, containers you take are on the rise. I think there is no surprise. The more popular container will be. The more attackers will look after this new environment. Same boxing is a good method to minimize the attack radius. It will not solve your security concerns. It will not remove all the barriers it will just minimize the attack reduce. We look at four different sandboxing alternatives: three architectures virtualization user space, colonel euni kernels.

A

Unfortunately, personally I didn't find any knockout alternative, I'm sure there will be some more work. They could be involved, we're going to see more and more coming in, but at the moment there is no knockout alternative in any of those that we tested good security for containers required to address attack surface to minimize it. The attack potential, how we can make sure that what an attacker can access or do is very limited and, of course, then address the radius. Now, interestingly, I saw a very nice blog from just Frisell.

A

The saying when talk about jivas most release was just saying the same thing, but she really want to protect your containers. There is more than one layer of security which is required and, of course in boxing is nice. It's a good way. It's improve your security posture, but you still need to take care of other aspects when you want to create a holistic security for environment. So those are all the resources available in my presentation. Thank you, everyone and if you will be love to hear your feedback, come to our booth.