Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nabla Containers: A New Approach to Container Isolation - Brandon Lum & Ricardo Koller, IBM

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Nabla Containers: A New Approach to Container Isolation - Brandon Lum & Ricardo Koller, IBM

Horizontal attacks are an important security concern for cloud providers and its tenants. Despite its many advantages, containers have not been accepted as isolated sandboxes, which is crucial for container-native clouds. The exposure of the syscall interface directly to untrusted workloads has greatly increased the number of exploits possible to the host. We present Nabla containers, which uses library OS/unikernel techniques to avoid system calls and thereby reduce the attack surface on the host kernel. Using our OCI runtime, runnc (https://github.com/nabla-containers/runnc), we show the running of popular applcations: Node.js, python, redis, etc. permitting use of 9 syscalls via seccomp. In this talk, we will discuss and demo how we have leveraged libOS ideas in a novel way and compare isolation and performance metrics against other technologies such as gvisor and Kata Containers.

To learn more click here: https://sched.co/FuKO

A

All right, hi everyone, my age Brandon and for IBM Research and today I'm going to talk to you about nah brah containers, a new approach to contain isolation. So for those that curious, why it's called nah brah. This symbol over here is in a person bowl and if you see in a later slide our containers look like that and I would say, be very careful when you google search table.

A

If you miss a letter, you may get something very different, all right, so the conventional knowledge today is continuous or not well isolated, they're, not secure right. If I say I'm gonna run multi-tenant containers, people I'll, say no way but VMC like okay. You know we'll do that. So this brings about a few questions. You know. What exactly is this isolation me? We wanna answer this and then we want to go on to see. Why do people consider the ends? Security, isolated, but containers, not right?

A

And once we have to answer to that, then we can go on to answer the question: how do we improve contain isolation?

A

So in this talk, I'm gonna I'm going to go through a few things: I'm going to start with the trap model, the specific share model. We are looking at how we are quantifying isolation and how we can increase isolation by reducing surface reduction. I'll then talk about our approach. Number give you a short demo and talk a little bit about the measurements that we are doing as well as spend a little time on VMs versus containers.

A

Alright, so what does it mean to be isolated? So, let's define dismiss, let's say: isolation is the ability that one container or workload should not be able to access the data or the secrets of a co-located container our workload. So in this case it's diagram over here we have service area, our super-secret service, a and, if I have another container on my host, this container should not be able to access the secret and the other container all right. So why is this important?

A

Consider the scenario where I have my my service over here? That's going to be a NASA static analysis. You know this is military-grade applications, unhackable right and then I so happened to have this legacy tenure application that has $2,000 abilities in it right. If this container gets compromised and I can do something like we call horizontal attack, then all the vulnerabilities and assets all the effort to make this service super secure is reset all right. Another scenario for this is being a cop provider like IBM. This is a big thing, we're faced with multi-tenancy.

A

All right, so what is the state of container isolation today? So what exactly are containers containers are namespace processes right, so that's good namespacing means that de la should be segregated. Everyone should have their own namespace right.

A

So this is great in their respect, but being namespace processes also mean they are processes and whatever cv is what about vulnerability or privilege escalation exploits that can be done on processes can also be done on containers right. So if we consider an attack scenario where we want to get the secret, we can imagine that an attacker may first compromise a shared privilege component, the kernel and for this perform some kind of trampoline attack to the container.

A

He wants to reach all right, so recent CDs in 2018, the most recent ones, is September I think back two years ago, dirty cow was a huge one, very popular, but if you go and look at the CVE database, there are enough of these exploits that. Come up that we should be worried about.

A

So, let's take a quick look at what we're up against right. Let's take a look at 30. huh Yes, I exploit sketch, so it does mmm a peach. It creates a track that does a bunch of em and my M advices calls. It then does a bunch of reading right to the profile system, and what I was trying to do here is. The intervention here is trying to create a race condition and trigger a vulnerability.

A

So, let's take a deeper look and see what's happening, so in this more detailed model, we can look at the kernel and say that it's no longer a black box right. All we have in the kernel are kernel function, so kernels just go all right. So in this case we have identified every kernel function as a box, and we can see that if you have all your abilities in this kernel, they exist in the function. So we can signify the red boxes. Move by kernel functions with vulnerabilities.

A

So the idea is that in the exploit, whenever you did something like M advice, M map, basically what was happening was you were invoking a series of kernel functions and if you are able to activate the right kernel function to exercise the code sequences, you may be able to trigger a bug.

A

So our hypothesis is that if we restrict the number of syscalls possible, the interfaces are Colonel. We basically are reducing the amount of boxes, scalar functions being reached and because of that there will be potentially less funnel. Bertie's with less vulnerabilities. Less exploits will be possible.

A

So to answer the question: you are you happy in your mind having this already been done and yes answer the second right, we all know what's icon. This is basically a white listing policy. The idea is, you can set a second policy that says I want to block certain syscalls and when, as a application, I'm talking to China I say give me give me a new page and he's like no alright, so docker has done this on. The doctor has created the default second policy.

A

So what our updated thing looks like is that we have the second policy and you know 40, for the syscalls, the doc up dogs by default will not be accessible. So what we can imagine here is that now the green are boxes. So this is the kernel functions which are no longer accessible right, so the we can kind of imagine that these disappear from the kernel right. So maybe some red boxes, some blue boxes disappear.

A

The trouble will come, though, is that it's really hard to create a second policy.

A

What people have been doing is to try to create generic second policy side docker, but the problem is to create a second policy that is generic enough for most applications. You end up allowing a ton of things and you don't really get that much security from it.

A

That's a bit of work being done with Cisco profiling, so this is some work adding twist lock and aqua security have been doing this. It's basically running some dynamic analysis and figuring out what sis calls are being used and basically from that create a heuristic profile right.

B

A

So come in Knobloch, so what is not bra so now there is a deterministic and generic second policy. Basically, we are able to create a second policy for generic applications with only seven system costs. So how do we? How do we do this magic?

A

What do you think is a technical library or aesthetics, so this really comes from a lot of work in the unit kernels. So the idea here is that let's look at the functions in the kernel, for example, the tcp/ip stack right what's in the tcp/ip stack, is a bunch of sliding windows, a bunch of algorithms on you know how my window should slide and all that. So the question is: why does deciding window implementation have to be in the kernel?

A

You know it's just a bunch of math and what the LabVIEW technique we're doing is using vaporise technique. What what is happening is that we are taking all this code and then we are bringing it up to user space right. So in this sense, you can imagine that now my tcp/ip stack, instead of being the kernel, is being put in something I Lipsy right.

A

So if I make a soccer call I'm no longer directly talking the kernel, I'm doing some work and use the space and then talking the kernel when I absolutely have to, for example, to talk to hard way.

A

So, just to talk briefly about what exactly were using, so this is taking a lot ideas from uni Colonels we're using a lot of tools and techniques from solar, five and run community.

A

There's a lot of details in this. Basically, what we've done is we've taken única notes and put them into processes. So you get the benefits of a lot of the process. The process benefits your high memory density possible at times, and that's quite a lot detail to this, but we we have a paper. You know kernel such processes that was in SOC, see this year. So if you wanna really dig into the details, you can take a look at that.

A

So, let's talk about Nara in terms of kubernetes in container ecosystem right. So currently we have this limitation that application has to go through a custom, build process to create something called an oblong binary right. So our build process is that we go to this build process which involves ramp, run and so five, and this creates an arbor binary. Now this is something that we are working on to remove.

A

This is kind of the reason for this is due to some artifacts in unit kernels, but we believe that there we should be able to remove this need for build process and then, of course, right now, because we have a novel binary. We have developed a kubernetes runtime valancy, which is ocr-a runtime, and basically, what this does. Is it loads to knobloch images and enforces the second, the 7 Cisco SATCOM policy.

A

All right so demo time.

A

So one of them to look at is we have this repository called up a demo maps. This is where we keep a bunch of applications that we've created so I have this cloned of him in my VM. So the one that we're going to look at today is the node Express.

A

So just to go into applications, but we have a very simple, a very simple application. Basically, if we look at this the package, the tree song and the app basically, if you talk to the root you get Nava.

A

So what we could do in this demo is we're going to create a new tank over here, and we can. Let's say you- know hello from table based on the Yukon path.

A

Alright, so the question now is: how do we build it? Okay, so we're gonna look at both building it from the perspective of building a regular nodejs application and then building a novel application right. So, let's first look at the regular nodejs application.

A

So pretty standard from the known image, we are basically Copic copying our application over and the package over. There are the JSON potential JSON over we're running a npm install and we are basically running node.

A

So now, let's look at the Knoblauch aquifer, as you can see it pretty much stays mostly the same I said we have added this additional stage bill. So what we are saying here is, from this image called a bra containers, not OneNote base. Basically, what this base image has. It has the copy of the note neva binary that we've created.

A

So very simply, we basically copy all the J's files over that's necessary for the application to run, and we tell it to run the application by setting the combined.

A

Now I'm gonna cheat a bit by not building the application now because I tried building it just now and it timed out on getting the npm install so you're gonna just have to take my word or you can download the repository and check it out, but basically I've created two images. One is called legacy and one is called habla.

A

So, let's, let's look at deploying it on kubernetes right. So this is a cluster that I have now no pots currently running, and so the setup here is we have. This is a local couplet discussed on my VM, and this is using container years of runtime and we are using this year, I plug in for untrustable code.

A

So, let's first look at the Department of a regular container right, so fatty straightforward, you know we just have the image over here, which is set as the legacy image right and now, let's look at what the diff of an habla image from the clay so.

A

A

And I'm gonna remove the name because the name is just different: I'd wanna get distracted by that. So the changes are fairly straightforward. One we are adding the annotation over here for the untrustable cloten. So what this annotation does is it's telling this gets passed on from kubernetes to container d and tells it that you start raster workload.

A

That's currently configured, so the container D that we're running is configured with run and C being the workload the runtime for untrustable, and over here we've built our separate Naga image based on the docker file that I showed earlier.

A

So what I'm going to do now is I'm going to create these two deployments.

A

A

A

You all right, so we can see that I have both my my pod funding, both the legacy and the Nara application, and if I look at the logs here, you know I see regular container. Alright. This is what I should see and let's look at the lots of the knob back on Tina all right. That's a lot of stuff going on up here!

A

So, okay, so we have over here we see the solar fire from banner some debug information, which is what we're printing out- and we can see over here some information or from print that's being initialized. So this is the library os that we're using and basically after the after all the initialization and second policies, we have the application running and because it's saying listening on port 8080 is it proves, is it's the same app right?

A

So what I'm going to do now is just gonna test out that you're working and that's scare, the pods.

A

Laughs issue finding my so this is on the same machine. I'm gonna call.

A

8080 and cube con right, so this is hello from legacy container and if I do the one for the number container, which is top 40 I'll, see hello from the Napa container right.

A

But this is weird oops, sorry 140 yeah, but this isn't really interesting because it doesn't really show us much right. I just shows that table is running the same application as the regular container. So let's take a look at one of claims are that we are actually reducing the attack surface.

A

So what I'm going to do in this window is I'm, going to first figure out what processes are actually being run right, so I have two processes here. One of them is the regular node.js application right, and one of it is Dena. Blah did not are binary being run. Okay now that one is all loaded, their abundancy runs. So what we're going to do is we're going to do an S trace.

A

So, while Astrid is doing is basically attaches to the pseudo.

A

Sure so what ashley is doing is attacking the process and basically keeping track of all the SIS calls that are being invoked and that she basically waits and collects here is the information and then prints a histogram after I cultivated.

A

So what we're gonna do here is we're gonna, say, wall, true, OH,.

A

A

A

Oh I'm gonna, three salad.

B

A

So what we see here is the the sis calls are being run. So what what really is interesting to us is the number of lines that are over here. So the number of unique syscalls being invoked right so over here you can see, there's 1, 2, 3, 4, 5, 6, 7, 8, 9 or 10 I, don't know what I can count right. It's a 9 or 10 Cisco's being invoked, and we can see like, for example, over here I'm advice. Remember we saw that in dairy cow all right.

A

This is called as possible, even in the most. If you create a second policy for no gsm advice will still be needed, so you serve to allow that. So now, let's look at this for nah blah.

A

A

Okay, do the same thing we're still above routine.

A

A

And we can see that you know we only use tree syscalls compared to ten of the regular process. You know people write a read, can't possibly be malicious right, there's more to that, but I won't go into details right now, okay, so so we've seen basically that, based on the ash trees, we are getting a smaller sis called profile at these right, which is great.

A

That means, let's just call in this case, if we didn't allow em at first and if an attacker was able to get into this container, they would not be able to perform to the decoy attack.

C

A

All right, so what do we see? Oops.

A

All right, so so what do we see? Just now? What we saw was the graph on the left. So what we're doing was we were performing an S trees, which is basically looking at how many syscalls were perform at this level. Okay, so what we saw was nor the Express we had tree syscalls versus somewhere around eight intensive calls.

A

We have this running on Redis and Python thornado a web server. So Redis is the in memory Kiba you saw Python tornado is a web server and we see very similar results and the number of Cisco productions.

A

But you can argue you know Cisco's, don't really mean anything, because you know a read or write. You know based on the heart descriptor, you can basically read environment, multiple different things, so you can classify them as different functionality.

A

So how we did was we did a more in-depth analysis, so what we did was the F trees. So what actress is doing is a fairly expensive operation.

A

What it does is basically every kernel function that is being invoked in the kernel we are basically talking in, and so what we're doing here is the exact same analysis, we're basically recording the number of unique kernel functions being invoked, and this is the graph that we have. So we can see that this idea that reducing numbers is called translates to the scope being touched in the column.

A

Thank so we did this with a few other runtimes and one of the interesting things we notice was from this graph and I have more of another graph that compares G, Weiser and stuff and color containers and docker to put and the F trace metric. But if you're interested, let me know- and then I can show it later. So the the interesting thing from this was that we saw that look. We found that novel patinas based on our metric was doing a little bit better than color containers so being researched.

A

We asked ourselves a question, you know, what does this mean for isolation, especially PMS does is: what's have we found the the magic secret sauce of hobby ends the isolation, so we spent a couple months in cannot dig dig into this problem and to really understand: why were we seeing similar or better results compared to VMs, and so we wrote this paper. Alcohol say goodbye to virtualization for a safer car.

A

This was a new sex on crop and the main idea of this talk was we basically analyzed virtualization at virtualization as a technology VT and said that if you did the same thing on VT and as a regular process really, while what are you getting from that yeah, this in itself can be a talk on its own. So I'm just going to summarize, like the conclusions that we became to as well as some of the feedback we got from the academic community.

A

So a few things were in the paper we compared against KVM, because it was something that was accessible to us, especially in the unicolor war, and the question was this: is a fairly implementation-specific comparison, you maybe we should look into other hypervisors and see how that performs.

A

Is that the other big question that people brought up was you know the very popular hardware attacks were I, expect a meltdown they'd, be you know, hyper trading, side channels and things like that and yes, our our model doesn't include all these things, which brings us to the last point, which is other metrics. So we we want to gather feedback from the community and all the other container isolation groups, who kind of come together to say how do you really say what is isolation, and how do you measure it?

A

We know we are not helping everything, because we don't come at this point right, so we are currently still ongoing research into seeing what exactly what other things of isolation. You know, data data coverage and things like that.

A

So what snake so currently the team we are about tree or jade tree now we are four people. We want to get more people looking at this and thinking about this there's quite a bit of development. Work still needed to be done in runnin, see at run time as well. As you know, supporting other languages. Most recently, we we were able to support, go tank.

A

So, first of all, you know the need to remove the build process so that you don't have to create a separate container for na blah. You can just launch a regular container and have nah blah kind of transparently be put in the second like I said. You know multiple, and the second thing is like I mentioned earlier. We we also very academic driven, so we want to also try and get more feedback on. Why exactly isolation? Is we wanna hear you know?

A

What are what are the models of isolation that people look at and how they see it and whether it fits into this model? Whether we can you know improve the model to be more holistic.

A

All right, so thank you and that's all I had for today stop. We said some time so I think if you want I, can show you a few more details or you have specific questions, do visit outside Navajo, tiers of github the I/o and also take a look at our runtime and contribute code. If you can thank you.

D

Hey Bren I have a question about performance, so every layer you introduce between your application and the kernel obviously adds some extra code that needs to be executed. How much slower is this than running like on bare kernel? Okay, so.

A

There's two parts of your question: I'm gonna answer the first five first, which is doing this you're running extra code. So if you look back into Yuna kernels, there was actually a performance impetus of doing this library singing and one of it was actually you're doing rest traps I know so some people actually saw performance increases when they use a library operating system. Right.

A

Since we talked about performance.

A

So this is the performance numbers that we got. So this is the register benchmark, and this is the Python tornado and no Express, and so let me just explain this a little bit. So the red one is Dhaka and just ignore the like dark blue one, and then you can see that Kara containers and agua containers have a little bit of overhead over here.

A

So does we believe? The reason for this drop in performance is because of a trick that color containers and lapa containers have to do. Okay.

A

This trick involves that a standard, CNI plug-in provides a rack and because quatre containers and the solo 5 unit kernel interface expects a tag device. We do this really really discussing bridge trick where it's, basically in a network, we have additional hall, so I think that's that's. What we are thinking is attribute into this, like 10% loss here, so this blue one over here is without the loss, so there is still a little bit compared to well depending on how much the work look like.

A

This is pretty close, this one's, maybe about 10%, and then this is the difference between having the additional network hop in the bridge.

B

I was also a divider, so from understanding divisor also to achieve the ISO nation by intercept containers system Co and transmitted to some limited system. Co, so I want to understand. What's the difference between Chi, Bazaar and Napa, so.

A

I'm gonna give my my view or how teams view of this. We have an again. This is our view and I, think people and audience I can talk about G paisa, so so the based on analysis.

A

What we saw that so that was mission is very specific right. Our core belief is that less calls means better okay.

A

So what we want to do is to reduce the number of Cisco's. Now G Weiser, we believe, has a different philosophy. So our belief, based on reading the code and playing around with it, is that the security is from two things. One is the functionality that used to be in the kernel is now in user space and it's written in a safer language. One two is that the segregation of different modules, so you get this the square feet, security from the segregation and these privilege, and that's where we believe the the ideology is bad.

A

That's what we believe so I you know can really say myself. So we basically we looked at the the F trace results and we saw that the average results. Weren't really hinting at this idea that you know it's I, can't you show you so the actress results weren't, really hinting that they were doing exactly the same ideas as us. Given some of these are like Petrie's color function calls, so this experiment, like I, say, is based on one metric, very subjective.

A

You know we love to hear yeah philosophy I mean well. Obviously it's just based on what we see you guys on ahead, something.

A

E

I wanted to ask that is a customer can build as a nebula image by themself and a lawn in public. A cloud. So sorry, Kay, repeat, that is a customer can build a nebula image by themself and the Zen run in public. A cloud Zen like the hypervisor sand run in oh.

F

A

Soap in sauce go down a little bit. Try it out: okay, yeah.

A

F

A

So the depending on your cloud provider I, can give you a hack into how you can do this, which your car provider probably wouldn't like, but it's doable yeah, but basically, in terms of the code, the code said you can download you can play fit. You can just use this as stalker just to play around a bit. Oh, so it's it's pluggable with front C and minus C run V to get the mic to Ian I think he wanted. She rides.

E

G

Yeah, so uh thanks for the talk, who's interesting, it's very similar approach between G, visor and nah, blah I think the idea of moving a lot of the kernel functionality into user space. Yes, in terms of the philosophy, I, think the the main philosophy of G visor is trying to focus on having a second security boundary everywhere, and so, whereas it seems like you're if I understood correctly, your focus is much more on reducing attack surface yes, so.

C

G

Know if you have two security boundaries, a very large attack surface like how much does that really get you, so it's yeah.

A

G

Won't comment on which one is better, but I I mean.

A

With the difference we we want to know if you, if you have an opinion, which is that I mean we can talk offline, obviously, but like we really want to know what you guys think, and we just want to start a conversation so that you know we could be both right. So you create a combined matric yeah.

G

And maybe there's there's room to kind of combine.

A

G

Of these technology.

A

We just added a line of research which is looking into how we can segregate the library os a bit more, especially at BSD, because it's so modular and the idea is we want to do something like a PC but not using IPC, because it's heavy weight, so kind of like think about process is different. It's a very calm research, kind of stuff, yeah.

C

So easy any proof of proof of concepts to show there how secure nominees, not only the number of Cisco city uses, but also we know tiger trying to exploring the column, vulnerabilities and maybe longer survival and other containers just died.

A

Are you talking about a very specific exploit, or are you talking about exploiting genera I.

C

Mean you may I mean me she's one of the from CVE right, okay,.

A

Okay, you see yeah, so so yeah we've looked into this and we haven't played around that. Much of it I mean. Theoretically, we look at the exploit, like dolly cow, and you know it's trying to do. My switch obviously can but I don't know whether you know it takes us in living among them to figure out whether the code paths that were triggered by M advice can be triggered elsewhere. But I do like writing memory really fast or something we don't know, but we do have a blog post. We're writing.

A

There was a don't give too many details, there's a bug in Exe for with extended attributes that causes denial of service. On the hoax. We basically replicated the bug on cattle containers, but it wasn't possible on Nava and G Pfizer, because you guys thought the XS you're pissed yeah I'll answer the question. I will get that later evidence.

H

Hi from a any user perspective about.

B

H

Are the main impact if we shift from darker to nabla, because I think like, if we're going to set up a port security policies and we can set up a stack home and capabilities and all the kind of security measures, security ways and from any user perspective, what we need to do and like? When do you forecast? We can use nabla in a very mature way. Thank you. Yeah.

A

So so to to just preface my answer that don't run this introduction, it's already so with Paul secure. So so the idea is like this provides some way to provide isolation between thoughts based on kernel attack, surface Paul security policy. Also like, if you take a look at you know, we talked about docker. Second right, you're, reducing it a little bit right.

A

So if you have a way to quantify your risk and then it sure is that way, then you know, for example, in the discussions will be having with isolation right if I really want to have a super secret service and I. Had cash I just buy a server and stick that one service on that and there'll be isolated. So mala initial I ideology was: is that you know we are able. So if the use case was juniper, juniper, notebooks or some kind of thing, where people have arbitrary code right?

A

If I can provide a higher amount of isolation and have a really high density, then I can have good isolation with lower costs yeah in terms of actually moving it into like the devil's pipeline and to get it to do it be done in production. We serve a bit of way to go in terms of removing the build process and things of that yeah.

C

A

Oh Oh way all the time, so I always stick around, so just come up with my name again Chet. Thank you.