Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tailor-Made Security: Building a Kubernetes Specific Hypervisor - Samuel Ortiz & Andreea Florescu

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Tailor-Made Security: Building a Kubernetes Specific Hypervisor - Samuel Ortiz, Intel & Andreea Florescu, Amazon

One of the many benefits of the recently introduced RuntimeClass feature is the ability for operators to run hypervisor isolated container workloads in order to build secure multi-tenant deployments.. While projects like Kata Containers allow operators to run their Kubernetes workloads through a growing list of hypervisors, none of them is designed with Kubernetes specific use cases in mind. This session will describe how to improve container workloads performance, security and density by building a Kubernetes dedicated hypervisor. At first we will describe what running a Kubernetes compatible hypervisor requires. Then we will show how the recently formed rust-vmm project allows for designing KVM based hypervisors for very customized use cases, including the Kubernetes ones. Finally we will use the serverless example to show what a reduced Kubernetes hypervisor looks like.

https://sched.co/MPd5

A

Okay, I think we're gonna get started. This is uh welcome to the run times track. I'm Ian, Lewis Alberta, the hosts today. So the first up. We have the building ka kada, kubernetes, Pacific hypervisor, and we have as speakers. We have Andrea Florescu, who is a software engineer at Amazon who works on the firecracker VM and we have Samuel Ortiz who works on at Intel on kada and rust vmn and contributes to a number of open source projects like cryo or creo, depending on how you pronounce it.

A

So with that I'm going to welcome them to the stage.

B

Good morning, everyone thanks for joining this wonderful session. Today, Andrew and myself are going to talk about building a kubernetes custom and specific hypervisor. Well,.

C

Actually, we are not going to talk about that, because I need some tiny modifications to the presentation and it's not working oopsie nope.

A

C

C

B

C

The surprise but yeah so today we are actually going to talk about building a kubernetes, specific vmm and not a hypervisor, and to understand why I'm saying that, let's just look a bit at a typical Linux, virtualization stack, so we have the Linux kernel and inside a Linux kernel. We have KVM and KVM is essentially making the Linux kernel to behave like a hypervisor. But we are not going to talk about this today. So now what we are going to talk about is actually a layer up.

C

So we are going to talk about virtual machine monitors or VMs which are going to be used for managing virtual machines and with that I will let Samuel to get back to his presentation about containers.

B

Thanks Andrea for the correction I appreciate so yeah. We're going to talk about VMs can I call it a Kaiser.

A

B

Going to talk about VMs and containers and well it's a communities conference, it's all about containers in workloads, and you may wonder: what's the relationship between containers and vmm containers and hardware virtualization, and this is a strong relationship if I can make the clicker work. So the reason why we're talking about VM and however virtualization today is because we want to have and define harder, virtualization and VMs in hypervisors in general, as a container isolation layer as yet another container isolation layer.

B

So what I mean by container isolation layer I'm going to just explain that quickly? Typically, when you run your continued work on the hosts, you obviously want to isolate, contain your workload and prevent it from reaching other quickly other workloads and reaching the whole horses and to do that. You're gonna, add isolation layers as many as you need as many as you want, and you would typically add, namespaces see groups second rules to well isolate your continued work from from the host.

B

You don't want you to be able to call any and every Siskel's you want to be restricted on how many resources it can use through C groups and so on. So those are all isolation layers and what we're trying to do with hardware virtualization EMM in hypervisors.

B

We just want to add yet another isolation layer to this story, so we want to be mm to be another isolation, little layer that you can add to your continued work load depending on the need for security and the level of security that you want to reach with your with your application.

B

So this is. This is very well aligned with the with the overall container security architecture and design, which is mostly about defense in depth. The way you build your security story, at least for your continued workload, is by defining how many of those layers you want to add.

B

You want to be able to say: I just want to be able to run my containers through namespaces and nothing else, or you want to add as many as you want, and what we're doing is adding hardware virtualization as another isolation layer and effectively making containers as secure as virtual machines.

B

Okay. So you may think that this talk is all about how we're going to do this. How we're going to add this and define this as an equation layer, but in reality this is already integrated. This already exists. You can already today run your kubernetes cluster and use some vm m as an isolation layer.

B

It seamlessly integrated through the work that we've been doing with with a few other people through signaled and the OCR runtime specification, and today you can, you can define a you, can specify a runtime class and have you container workloads running through the VA and, depending on the runtime class, that you define, even through the OCI specification. The actual run specification, if you, if you want you, can define which virtual machine you're going to use, how which guests colonel you're going to use, which I provides?

B

Are you going to use and and how you're going to configure it? So this is all integrated today in the in the specification that I'm in relating here.

B

Thanks and so there's one well known implementation of those specification on how you're gonna add a VM as a isolation, layer and that's Kara, continuous. How many of you here have heard about CARICOM teams yay we made it I should take a picture of this and send it to my management.

B

It's been long, it's been a long road, so yeah Kara container is a it's kind of a canonical implementation of this of this work and the Kara container team has been working with with the communities six and working group to make that happen, and the way you you do it today is by simply simply adding a runtime class name to your to your definition. So by saying, I want to run karakuri I mean, as my runtime class you're, basically going to run your continual workload inside a QM.

B

You define virtual machine, so you've EMM is gonna, be key mu and your runtime is going to be Kara continue. You can do something similar and instead of using accumulative AMM, you can decide. Okay, I'm gonna use firecracker as a PMF, and this is well another runtime class. It's gotta, FC and callek contain is going to be the runtime. Firecracker is gonna, be the VM and it's as simple as it gets. Yes, okay, so you may think that it's that it's very simple and this diagram make it look that it's very, very simple.

B

But if you look at little deeper into the architecture and how this is implemented, it's it's not that simple, the whole continued ecosystem, a lot of assumption on the fact that the container workload is actually run on the host CPU bare-metal on the host and so to make that happen and the way it's implemented today is through an actual virtual machine.

B

So when you use qmu as UV mm or firecracker as UV mm for for cala containers, you're going to be running your workload and your pod inside a full-blown virtual machine, so your pond is going to do part is going to run on top of the real Linux run. The entire dedicated later kernel to your pod, there's going to be a cat eye agent running on the on the virtual machine to basically handle multiplex and manage the all the commands that you get from from your runtime into into your workload.

B

So even the even the picture on the right is still a simplified description of what it takes you to actually make that happen.

B

So we've been talking about memes.

B

And as of today, with with Kara continues, you can run your workloads through different VMs, as I said, depending on the runtime class that you select, you're gonna have different VMs running your workload and we support three different VMs. We support key mu. We support nimi, which is a stripped down version of kami, and we also support.

C

B

Maintainer she's she can help okay, so the default vml for Kara container is easy. I mean key. Musa is a high-performance vmm. It's it's very stable. It's also very versatile. So it supports all sort of architecture is suppose all sort of platforms it emulates hundreds of devices, so it basically does everything and anything you can think about when, when basically building a virtual machine, the cost for this is that the QA codebase is.

B

Is it's it's more than 1.5 million lines of code and it's footprint is also a relatively large, so you get the performance in the verse of DT and obviously a larger footprint on the other side of the of the VM spectrum. There's firecracker firecracker is much much much smaller than than GM you, but it also has a much narrower focus on one specific use case, which is service. It's designed for supporting surveillance. It's it's running, AWS, lambda, I, don't want to run yes, it's running. It's running, Eddie, Willis, lambda, it's very, very small.

B

So we're talking about 20, 30 line thousand lines of code, not thousand lines of code and well it's very small and, as a consequence, the scope of Phi cracker is also smaller than Jimmy. It's so small that it cannot support all container workloads. So if you continue, workloads needs to mount a volume which is fairly typical, with kubernetes you're, not going to be able to do that, because fire cracker doesn't support shared file system from the host.

B

If your work loss needs to assign your host GPU into your container, you won't be able to do that with firecracker. Firecracker does not support so there's a few things in a continual definition that firecracker does not support. But if you want to do server looks like a work load, then it's definitely the right approach.

B

So I think we have like this big elephant in the room when we're saying we're adding hardware virtualization as yet another isolation layer.

B

Obviously, for security reasons, we want to make containers and secure as VMs, but on the other hands we're adding this new software component here, a vmm that can be large or very large, like um Yui, that can be very small but limited, but basically we're on one hand we're adding we're saying we want to improve security and the security for a story of containers and kubernetes, and then, on the other hand, we're adding a software component. That's yeah can extend the sorry, the the attack surface of your of your container workload.

B

So what we believe is that there is a need for defining a vmm that supports all continued workloads, but only continuous, nothing, less, nothing more, not qmu that supports basically everything and something larger than firecracker to support all container workloads. So this is mostly what we want to talk about today.

B

Finally, so before going into more details about this, we need to understand what it means for vmm to support all container workers, and we need to understand what's what a container runtime needs to do and to do to do that, we're going to look at specific, continued runtime specifications.

B

We could look at the OSI I want, which is a low-level specification for continued runtime, but instead we're going to focus only on the communities. Your eye specification, as a matter of fact, the Sira implementation for most of them end up calling an OCR runtime eventually, so it makes sense to you, look at only only the CRI stuff. So what we're going to do now is look at the CRI specification and try to map that to actual hardware, virtualization requirements and technology to understand what we need to implement in vmm in a cumulative specific vmn.

B

Okay. So if you look at this hero specification, there are a few interesting things, one of them the very first one obviously, is that you're going to need to send commands into your container water you're, going to be able to interact with your continual inside the virtual machine. It seems very obvious. It seems like a natural thing to do, but it's it's actually not. You don't want to run an SSH server inside your virtual machine to do that.

B

Instead, you want to be able to open, like a JRPG socket directly into your virtual machine and so to do that. You're, gonna, you're gonna need a implementation, a virtual circuit kind of implementation, you're, obviously going to need a container image. Your container water is gonna, run inside your virtual machine and to do that, you're gonna, be you're. Gonna have to mount the container image from your overlay implementation on the host inside the virtual machine.

B

That's that's something you obviously do when you're running on bare metal just mount or run your container directly from the host when you're running an inside a virtual machine, it's much more difficult, and to do that you're going to need virtual storage. Obviously an obvious requirement is networking.

B

Cri specifies that you, you may need networking endpoints video container, that's fairly typical, so you're gonna need virtual networking as well.

B

A few other interesting requirements like the device structure inside the year. The CRI specification- that's says: well, you might want to actually mount a device into your container. What you're gonna be you're going to want to mount your GPU to the ml type of workloads inside your your your Cuttino workloads? That's very easy to do with you with a bare metal container. You just basically mount your or by mount your your /dev entries into your continued workload with the virtual machine. It's a lot more difficult.

B

You need to directly assign an entire device into your your virtual machine. It's involves iommu, VF, I/o and so on. So what you're going to need is direct device assignment the mound structure. You want to be able to mount volumes and share secrets with your continued workloads, again very easy to do with a regular container, with a virtual machine that means you're going to have to implement the whole shared file system, paravirtualized implementation for your container water.

B

So you want your virtual machine to be able to share a very tiny part of your filesystem and mount it inside you inside your virtual machine and finally, the sheer ice pack says that you have to be able to update your container resources. Maybe your container is going to need more CPUs or less CPUs or more memory, and to do that well again, very easy on bare metal continues with virtual machine you're gonna have to hot plug things into into the virtual machine.

B

Adding a CPU means you need to make your virtual machine aware that a new CPU is showing up. More memory is the same thing. Removing memory is is very difficult, so you're going to need resource hot plugging.

B

How does that translate into actual virtual machine and hardware virtualization software technologies for the virtual circuit storage, networking well, essentially, you're going to need para virtualized, implementations and the typical way to do this in Linux is vertol, or maybe the host user, which is a user space implementation, while not user space, but a deferred implementation over time where you have separate device simulation for direct device assignment, we're going to need a vfi implementation, so VF I/o is the kernel API for handling iommu based devices, so you're gonna be able to talk to your device through iommu and from user space and take that device and assign it to your virtual machine so that your virtual machine as a process, it's gonna, see directly this device and use it directly.

B

Almost as fast as bare metal usage, the share spy system requirements translate into again a para virtualized implementation. Right now we support nine PFS, we're going to duplicate that for Verta of s, which is a new implementation and finally resource hot plugging. There are many ways to do hot plug into in you want to do memory, CPU and BD, multiply and and other things, and one way to do that is to go for a CPI.

B

So you can use a CPI to do hop like or you can use PCI to do device hot plug, but you won't be able to do a CPU hot plug with PCI, for example. So when you go full a CPI, you can do an entire hop back support for your virtual machine and go as you probably guess, implementing a full. A CPI in your VM is not a small thing. So compare that those requirements to your permit or containers requirements.

B

When you do a by mount in your with your bare metal continues, you can, you can have to do an entire shell file system or an entire vf5 implementation for your virtual machines and so on. So if there are people in the room that are working on CRI specification and runtime specification, please look at this and understand how difficult it is for anything that you add the specification to support that through hardware virtualization okay.

B

So we now have a small set of requirements for our VM m and if you look at GM you, for example, it supports all of this. If you look at firecracker, it supports some of this. Not all of it and what we want to do is build a vm m, a virtualization stack that only supports this and nothing else. So where could we get those components and I? Think Andrea wants to talk about this yeah.

C

This is very exciting process, so yeah. So all these components you can get them from rice, vm m and res vm m is a new open source project that is providing virtualization components, building blocks for building your own virtual machine monitors or hypervisors. If that's your thing and to understand better, let's look a bit at why we decided to start this project.

C

Oh and by the way the virtualization components are crate, which is rust packages, so the whole project is rust now, so, let's see why we decided to start rust be, mmm and the first reason is actually to reduce code duplication. There are already two virtual machine monitors. There are written in rust. One of them is cross VM, which is providing isolation for application in Chrome OS, and then we have fire cracker, which provides isolation for multi tenant workloads in the cloud, so that will be AWS, lambda and AWS party today.

C

But that is not the only reason, and that is not a reason that might be interesting for everybody here, but when we are sharing these components because they are both written in rust and they are similar in some ways.

C

We also have to provide a clean interface in order for them to be so, for these components to be easily used by multiple projects, and then we also want to focus on security and the stability, and we want to provide a high standard for these components, mostly because they are going to not be only used by one project, but they are going to be used by multiple projects. So you want to make sure that they are well tested and they work for every use case out there.

C

Now, with these separate components that we are having, it's also, it gives us benefit, and that benefit is that it's much easier to do security testing in terms of fuzzing. So you now have this component that has a clean interface that you can crawl into and it's much easier to provide fuzzing for it and having this component separate and pluggable and like lido, also gives this new benefit, which is you can develop VM M's faster than before. You don't have to re-implement everything.

C

Again, you don't have to work on k, vm wrappers, you don't want to you, don't have to work on Berta, you and so on, because this components already exist, which means that essentially, you can focus on your on the key features of your product instead of reimplemented everything from scratch, and now that we talked a bit about why we decided to start rescue, mmm let's also look a bit at the recipe mmm development process and I want to start with the people that are already contributing to receive.

C

mmm There are a few companies that are already working on this. We also have individual contributors. There are reviewing the code and asking questions, submitting ideas and so on, and when we are talking about adding these components, whereas hmmm we are currently having two processes, so one of them is to add components that already exist in either firecracker or cross jam, and some examples here are like KVM wrappers, the word: tire, implementation, abstraction for a guest memory and so on.

C

Then we are also adding components from scratch, and these are some components that are interesting for the container vmm and some of them are like B host and a CPI which are component that Samuel talked about before. Okay.

C

So now, let's take an example of a crate that is already that already exists in both firecracker and cross VM and just try to understand a bit what's happening there. So we have the vm memory and the basic component of the vm memory is an abstraction of guest address and in firecracker the guest address is represented as you size.

C

Now we want to make something useful with this guest address, so we want to create a memory region, and the memory region is basically an abstraction that uses a guest address, which is the start of the memory of the guest memory, and then it also has a chunk of memory, an anonymous shared memory and essentially those two represent a memory region now to get to actually have a guest memory.

C

You will have multiple list of multiple non-overlapping memory regions, but this is very specific to how firecracker works, and now we want this great to be used by multiple project. So what do we do? We are converting everything to a trait and the trait is essentially an interface in rust. So, instead of having the gift address as the you size, you are having a trait of guest address which which can be implemented with your size we use 64 and so on.

C

We are also having memory regions as a trait and the guest memory has a trait, and one of the thing that you might notice is that we don't only support the nonnamous shared memory anymore, but we also support file memory back by file, and this is quite interesting because it allows us to support many other use cases like, for example, B host user.

C

Now everybody can submit ideas about crates that they think that they're useful for as vmm or for other projects, and we have a process around it. So the first thing that you have to do is to open an issue in our community repository and in that issue you have to pitch your idea. You have to talk about. How is it going to be used? Who is going to use it? What are the dependencies? What is the design and so on?

C

And after discussions about design and name with naming, which is very hard for us, we actually create an empty repository or named crate and against this empty crate. We are submitting pull requests based on the design discussions in order to obtain freaking development or a useful trade. A crate that has functionality, but now, as I said, these crates are going to be used by multiple projects. So obviously there is a list of requirements that you have to have before. We are publishing the trades on 'pray tile. The people know what Plato is.

C

Can you raise your hand if you know what crates? Okay, so crate al is basically the main I always forget. The word is the main people. Yes, it sort of very positive repo for for packages for us packages, so we basically the place where you get get the packages from so now, let's get back to it.

C

So from a crating development, there are a few things that you have to do in order to obtain a production already crate so prey that is going to be used like byproducts like firecracker, and we really really want to replace the importance of testing. So we have continuous integration and all the crates must have continuous integration and test and make sure that the standard, the quality standard is very hot.

C

Then we are also requiring license and we are having some discussions now in to have a dual license, which will be a patchy to zero and bsd close three, then another important thing is documentation, probably one of the things that I find the most important, especially in a place where you are going to use these components by David, so in a place where these components are going to be used by multiple projects, multiple people that might not have worked on them specifically, so we really want to have documentation that contains like high level design.

C

How are this? How is the public interface going to be used and so on, and then we reach the we file after we did all these things. We reach the my son to you, which is publishing the crate on freight sire to make it available to everybody.

C

Now the current status of crates in in raspy. mmm We have some praise that are still in development. So if you go on the github project, you will see that there are some crates that seem like nothing is happening there they're empty repositories, that's because we are still designing them. We are still working about on the design. We are still trying to merge, pull request and so on.

C

Then we have a crater is in development, which is the VM memory that I discussed earlier, and we currently have only two crates that are actually published on create idle and are used in production, and this ones are the Akkadian bindings and became ukyo's, which provides some pretty or an interface to interact with KVM with the KVM api.

C

So now, let's take an example of how a CRM is going to actually work in practice and because most of the Krait are still in development. We are going to take a paper example. So, first we are going to start with our DMM api, which is basically the bridge between the vmm, the virtual machine and the client that is going to use it.

C

And after defining the VM, a VM mmpi, we can take a look at the components that there are the current that currently exists in grozny, mmm and the thing is that we don't have to take everything, because these are separate repositories, separate components. So the first thing in order to create a continuous VMM, is that we are going to select the components that Samuel talked about earlier, which are specific sort of to the inner use case, but these components by themselves they can't work.

C

So you can't have a vmm only with this component, so you need some other things as well, and so we are going to get the TBM wrappers we are. We also will need a minimal kernel loader that is going to load our kernel image into the guest memory and we are also needing the abstraction for the guest memory.

C

Now. The only thing that you have to do, after all, these components are available, is to write the glue code and essentially the glue code. What it does is it creates the interaction between all these components, and it also interacts with the API in order for users to be able to configure the virtual machine.

C

Now when we are looking at this serverless example, things that will be changed and we don't need. So if we take the firecracker exam well, we don't need a CPI. We don't need Bert RFS, so we are just not going to take them so, but we do need some other things.

C

So one of the most important thing in a multi-tenant environment is also the ability to limit resources so limit the block limit, the network and so on, and we are limiting these in terms of operations per second and bad way, and this whole functionality is offered in the rate limit or crate, which we are going to add to our VM m. And then we are also adding cpuid, which is a great that provides a mean for specifying template for your virtual machine.

C

So basically you're making your virtual machine look like something else, and that is because you want to provide a the same experience across multiple platforms, but the other things are sort of the same. So you will have a different API but you're still going to have tried some vm m glue code and use basically most of the most of the components out there.

C

So now that we we gave this example, let's look a bit in the future. Let's see what is next for ASCII. mmm We obviously have to add all those components that I'm talking about, but we are what we are also trying to do right now is to provide a hypervisor agnostic crate, because there is an interest for people to not only use kb m as the hypervisor, but others as well. So one example is hyper-v, so we are trying to work towards that as well.

C

Now we are also using these components. We are trying to move these components into virtual machines that are already out there. One of them is firecracker and in fact, racket we are already using all the components that are published on create idle and obviously we are trying to create new specialized virtual machine monitors, and one of them that was recently announced was cloud hypervisor.

C

By Intel and in the future, we might as well see container vm m, but the future is set in stone. So if you have ideas or if you think that we are going in the right direction or in the wrong direction, or if you have any suggestions, you can come and decide with us like what what the future of Ravi mmm is going to be, and they can be part of recipe. mmm By subscribing to our email list.

C

We have discussions there and you can also submit if you have an idea about a crate that is going to be useful. You can submit a issue, issue, request, review, issue, review, requests and people will look at at your issue and we will provide feedback.

B

Okay, thanks very quickly as a conclusion: we have one minute left. So what we want to say here is that there is a need for a container and maybe community specific vmm. We think that Russ Bamm is is the right solution for this there's already some some vmm that are being implemented by using Russ vm that go into this kind of direction and yeah.

B

We really think that Russ barium is going to provide the building blocks for for all this so very quickly. We have our github, obviously Russ, vmm surprisingly so go there, there's a community repo where you can submit issue, but it can follow basically what the community is doing around that project which crates we're working on which technology we're adding to this just go there and chime in and we have because we're super hyped.

B

We have a slack channel, no IRC, sorry about that, but yeah there's a slide channel and you can join and ask question or or tell us what you think about it. Thank you very much. Thank you.

B