Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kata Containers: Getting Involved with the Kata Community - Anne Bertucio & Samuel Ortiz

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)

Kata Containers: Getting Involved with the Kata Community - Anne Bertucio, OpenStack & Samuel Ortiz, Intel (Any Skill Level)

Kata Containers is a new open source project that combines the security of VMs with the speed of containers by providing extremely lightweight VMs for each container, while meeting OCI standards. This approach allows for true, hardware-backed isolation between containers -- a must for many multi-tenant or regulated environments. In this session, Kata Containers community leaders will introduce opportunities for newcomers to contribute to the project, including code contribution, community engagement, documentation, and more. Come learn how you can get involved!

About Samuel
N/A

About Anne
I work at the OpenStack Foundation where I work on both the OpenStack and Kata Containers projects. I help organize the communities, work with newcomers to navigate resources and start contributing, and lead release communications.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

ah Excellent, well, then, we're at 3:40, so we should probably will be timely, excellent, um well welcome.

A

This is a an overview of cotta containers and also some information about getting started with cata, using it contributing to the community a little bit about how we work and what we're about I'm, an Bertie Co I work at the OpenStack foundation, where one of the roles I have is being the community manager for cotta containers and with me is Samuel Ortiz he's a principal at Intel he's an integral part of the clear containers team and also on the cotta architecture committee.

A

So he'll join me in just a moment, so cotta containers cotta, is about producing the speed of containers with the security of VMs, we're gonna dive into that. But before that, it's always kind of good to take a step back and talk about what is the problem we're even trying to solve? You know: there's no shortage of containers projects out there, so why did we decide to add another one? This is really the problem here that we're looking at is the issue of the shared kernel in a conditional in a traditional container architecture.

A

Many of you are probably familiar with this setup. You know, containers a B and C all have that shared kernel, and if one is comp, you know container a is compromised. We now maybe have some vulnerabilities and some compromised and B and C, and so for people who are in highly regulated environments, enterprise-grade anyone who's risk-averse. For any reason, this doesn't really do it for them.

A

Socata containers is about addressing that issue of the shared kernel, and so here what we do is each container or pod is isolated through its own VM and its own kernel. These aren't full virtual machines and kernels, but rather it's a lightweight VM and a mini kernel, so there's minimal impact on the performance, we're still producing performant containers, but now we have that Hardware backed isolation between them and here's. A cute clipart slide just to drive a point home one more time.

A

You know virtual machines, great for isolation, runs like a tortoise traditional containers down at the bottom great for speed. A little more risk. Cotta containers were up in the far quadrant there. That's what it's. What we're about kada is the merger of hyper SH is run V technology and Intel, clear containers. So in December we made announcement at the Austin cube con that we were intending to have this project, and the history really comes from both hyper and Intel. We're looking at this problem, but that shared kernel and they were realizing.

A

They were coming to this similar solution, they're both ending up with these virtualized containers so rather than work against each other. I thought you know, that's open sources will come together and they donated the code to the OpenStack foundation to be a vendor neutral home for the governance and just to iterate, because it can be a little bit confusing. So cotta is its own thing completely separate from OpenStack the project, as you might know it.

A

If you're familiar with OpenStack there's, you know 60 some projects that live under that name, so there's compute their storage cotta is not number 61. It's it's completely own thing different set of governance. We just share the same nonprofit, home and I'll. Let Sam you'll give a little talk about thanks.

B

Sam, okay, so I'm gonna talk more about the technical details of Kara. First I want to ask a question so and there's some who de su cuenta you better answer. Well, who knows here what is the main difference between Kara containers and a regular container? Okay? So what's what's the answer?

B

You're running virtual machine? Yes, you just get yourself hoody, you go! Thank you! Yes, that's that's the the main difference and the the whole technical vision around Kara containers is about running a container inside a virtual machine and by a virtual machine we mean we'll literally mean a virtual machine, which is a real full-blown virtual machine. So you might think this is very slow, big, bloated and well what virtual machines are and the the technical vision around Kara containers is to make them light and fast. So we want the security advantages of virtual machines.

B

We don't want them to be bloated and slow and big. So that's the main point about cattle containers. Integration with the rest of the ecosystem is the second most important point. We really want Cara containers to work seamlessly and by seamlessly I mean that from a user perspective, you never notice that you're actually running on a virtual machine or on a regular container. You just can't tell so. We have seamless integration with kubernetes at the CRA level.

B

For those of you who are familiar with the container runtime interface, we also seamlessly integrate with docker, so you can just change or specify which continue runtime. You want to use with docker and say: I want to use cat a container and start all your containers, with, as virtual machines, again seamlessly same for the networking where we just use the regular CNI networking plugins, nothing special there. We try to test with all the major networking plugins out there.

B

We can test with the unknown ones, but so far we haven't hit any particularly issue with the with the existing CNI plugins.

B

We also provide a CRI compatible API, so we both integrates at the OCI level and I'll talk a little bit more about this, but also at the CRI level. So you can use Keller containers not only as a runtime but also as a library, so we can implement your own kubernetes, your I shim, and call into you the cat, a container library to manage and create your containers as virtual machines. So you don't need to use the do CI, a binary that Kara containers provides to use the project.

B

Last but not least, we support multiple architecture, I mean Intel employee, but yeah I want to make sure that everyone understand. This is not an Intel only project, it runs on other architectures, I won't name them. Otherwise, bad things can happen. It also supports multiple hypervisors. So our main focus is k. Vm aqm, u k vm, but the architecture of the project is done so that you can add your own IP visors or if you have your repertory I provider somewhere in your cloud that is not Q or not.

B

Kvm then come talk to us and we'll make sure you can integrate that.

B

So that being said, there's a few interesting use. Cases for for cata containers, the one of the first one is d, where you absolutely need to have the equivalent of virtual machine security for your container were close in some regulated environments. People actually don't have a choice. They either run virtual machine or they don't run, and if they want to go with containers, they need to have something like catacomb teams. Another interesting one is and I also talk about that. A bit later is that you can mix cattle containers and regular containers.

B

As I said, it's completely transparent and seamless. So you can have your regular run. Sea-Based container together running together with wither cata containers, so in environments where you basically throw trusted and untrusted workloads together to your cluster. You have development workloads together with almost production, ready workloads, and you want to run, have them running next to each others, and then you can use something like Hera containers and the one that you don't trust. You put them in a virtual machine and they won't mess with the with the rest of your of your cluster.

B

If you want to run a full multi-tenant kubernetes cluster Cala container is probably a good solution. It doesn't implement the entire multi-tenancy for kubernetes, but it will give you hard multi-tenancy for CPU memory and devices, so there are still some gaps for for having a full multi tenant container cluster. But it's it's a it's. A very good start bare metal. If you want- and this is a as Anne was saying- Carol continues- emerge between two projects: hyper run, V and and clay containers and hyper.

B

Is a company actually providing a bare metal container as a service offering? So when you go to hyper SH Kella containers and you will run your continuous bare metal, so there's no VM created to host your kubernetes notes or anything like this. This runs directly on bare metal as virtual machines. That being said, bare metal is not a requirement for for cata continues. As long as your environment provides nested virtualization support, which most of the cloud providers do, you can run cattle containers. So bare metal is not a requirement.

B

You can run cattle containers on bare metal. You can run cattle containers inside virtual machine as nested virtual machines and finally, an interesting and interesting use case for cattle containers is where you absolutely need some very specific kernel features of very or you need to tune some very specific kernel. Knobs your TCP stack or you need this proprietary kernel module, not that I encourage writing proprietary kernel modules. But if you need one then with Kara containers you can you can actually have your own dedicated kernel inside your virtual machine.

B

So if you need, if you're a DNF provider or deployer- and you need very specific kernel modules to be installed at runtime, when you start your containers, you don't want to do that on your host kernel. Actually, you probably can't do that on your horse kernel, but you will be able to do that on your catechin Taner environment, because while you have access to your own dedicated kernel.

B

One use case that I'd want to highlight, and just I just talked about- is multi tenant, kubernetes and again with Kara containers. You will not get a full multi tenant kubernetes, but it will get you there I'm repeating myself, but it basically gives you real hard murdstone's e for sharing CPU memory and devices across different tenants. So typically as of today, if you want to, if you want to run a safe, multi, tenant, kubernetes environment, you will have to create a cluster pertinent.

B

You want to have you want to have your tenant separately through VMs, at least or well? Maybe you don't, but you know do that at your own risk.

B

So, typically, you will create several kubernetes clusters through several virtual machines, each one representing nodes per per tenant and the way you can do that with with cata containers is just by just having one single kubernetes cluster and with cata containers. All you workloads will be separated the same way. They are separated and the picture on the left because they will be running on full, full-blown virtual machines.

B

So you can have your tenant workers working naturally next to each other and have the same isolation guarantees that you used to have with separate kubernetes clusters running on different VMs.

B

Ok, components I'm going to go through a few of the components that make the Cata container project, so we have a few of them running on the host and we have a few of them running on the guest. Again, your container is gonna run in a virtual machine. The most interesting one on the guests is the agent, so it's basically the the component is going to run your continued lifecycle inside the virtual machine and, as a matter of fact, our agent implementation is based on lib container.

B

For those of you familiar with the Ranji project, lib container is the the foundation library for runs. He runs he being the de-facto container runtime, so inside the virtual machine that runs your cat, a container, you will have an agent that looks like a very trim down runty, so you can apply all the OCI specification settings inside the virtual machine so that when you run your container inside your virtual machine, it actually runs exactly the same way as it would run outside the virtual machine in a regular containerized environment.

B

So that's very important from a security perspective as well, because you can monitor akattak containers the same way. You monitor regular containers. If you apply the same second filter inside a virtual machine as outside the virtual machine, you will see the same behavior. So if, if one of your workload is trying to call into a Cisco that is not supposed to be going in, you will see that with Kara containers. The same way. You see that with a regular continuity and you can kill it proxy.

B

Optionally, we need a proxy, so the proxy is a component that runs on the on the host, maybe actually yeah, that's actually better. So the proxy runs on the on the host and we need a proxy. But it's it's optional. I'll get that on the next slide. We need a proxy because we typically run the agent. The communication between the agent on the guest and the host is going through a serial port. So we have one serial port for each and every process inside your virtual machine.

B

So if you have three containers in running inside, your pod, you're gonna have all UI or going through one virtualized server port and don't be scared with the serial port. A vertel user report is fast. It's not your all the 232 physical server ports, so it's fast, but it's it's still something that needs to be multiplexed and demultiplex somewhere, and we do that through the proxy.

B

We need a few shims. Well, not actually a few shims, but one simple process running inside a container. Why do we need a shim there's already a few shims in the container ecosystem, for continuity and and and so on? But we need an additional one, because the the components running on top of color containers usually assume that they're going to talk to a process and this process is your container process.

B

In our case, it's not talking to a process is talking to to a virtual machine, so we need a shim and the shim is basically a translation layer between whatever we receive or we want to forward to the to the component to the to the ecosystem. On top of Calla containers and darker kubernetes OpenStack translate that between the agent comment and process semantics. So when you want to send a signal to your container process from your cube control, we're going to receive a signal but you're not going to signal your virtual machine.

B

You want to signal something that's running inside a virtual machine, so you need to translate that signal into a command that the agent is is going to be to be able to understand, I hope that makes sense.

B

As I said optionally, we can run without the proxy, and in that case we use a very recent kernel feature. That's called V sock, there's been it's it's an implementation of VMware virtualized socket so with V, so Q gets sockets that are virtualized and you can basically use the socket semantics inside and outside your virtual machine, and you don't need any multiplexer on the host to demultiplex and multiplex.

B

You I/o, you just have a socket semantics, so in that case we simplify the whole thing and remove the proxy I'm just going to finish my technical part, sorry and talking about how we integrate Kara containers with with the rest of the ecosystem, and in that case, with kubernetes I assume.

B

Some of you are familiar with the cryo who knows cryo in this room. Yeah Wow I, don't have enough foodies. Sorry, so cryo is just one example of how we integrate kara containers with with kubernetes. You would have a pretty much exactly the same picture with the continuity today.

B

But and again, this is just an example. So when you run your kubernetes workload today with cryo, you have cubelet talking into the CRI and then this year I talking into the CRA she can be cryo or container e or anything else in the cry own continuity cases. They assume that they're going to have a OCI compatible, runtime below it to actually translate the CRI commands and and specification into actual container lifecycle, commands, run, start/stop, exec and so on, and so they just go into run. T to execute this command.

B

Keller containers is a no CI compatible runtime, and so you can literally replace run C with Kara containers and cryo or continuity will talk directly to the cat, a container runtime, binary runtime and instead of creating namespace bastes containers, we will have virtual machine created for you. So I want to insist the fact that this is a fully transparent and seamless, so it just literally replace runs E and B that you can mix workloads together in that example, is a practical one.

B

We we we do that in production, where you have run t running, regular containers and cata containers running more secure containers, so this is. This. Is very important and a very nice feature from from Kara and I'm going to hand the mic over back to you and now.

A

Okay, so um I'm gonna go kind of quickly through couple these things about contributions and how we work in the community just because I want to make sure to leave time for Q&A and more hoodies the important things. So we are aiming for a 1.0 release and the May, where I'm from May 22nd, so where we are right now clear, containers and run V have been merged to a single implementation.

A

We've surpassed the capabilities of clear containers, 3.0 which came out in January and we're pretty darn close to parity with run V at this point for the release we'll have those CRI integrations that samuel was talking about as well as some documented upgrade paths for existing run, V and clear containers users.

A

If you want to give caught a try, you can try what we have right now, it's on our github and the documentation, repo or there's some documentation about the limitations and some caveats. But it's up there I see a few phones, so maybe I'll pause for a second. If people want that URL.

A

Excellent, but if you're interested in contributing, we would love to have you, you can read the contributor guide on the github, it's in the community repo- and you know this is a kubernetes crowd. Presumably so you're probably familiar with NGO projects written and go, and the patch e 2.0 license is our open-source license so same as kubernetes. One thing we do that some people have not had experience with before is that we use signed off by from the Linux kernel community in the contributor guide. There's some information about that.

A

If you haven't experienced that before I'd, encourage you to just read through it carefully, I would hate, for you know, PR to just sit and fail, because you missed a little bit of the formatting on that one. So just something to call out and pay attention to.

A

When you get to the cotta containers github, you will see that we use the Help, Wanted and good first issue labels other areas that we're really looking for some extra hands are in tests and Docs. So if testing or documentation speak to your heart, we'd love to have you there's a test. Repo Doc's are looking for reviewers and editors.

A

If you think cata containers is going to be the right solution for you, but there's something specific to your use case that you're thinking of that we don't seem to have covered yet or maybe you're. You think you're a corner case of some sort.

A

Please use a github issue to file a feature, request or drop in to one of our community communication channels and talk to us about it. I want to make sure we're meeting everyone's needs here.

A

If upstream code is not your thing, community engagement we're definitely looking for people to kind of spread. The word the good word of secure containers, if you are a meet-up organizer, we're happy to you know, have someone come talk about cata at your Meetup. If you like to make blogs demos, there's there's no shortage of work to be done.

A

If you're interested, we are doing some community planning at the end of this month at the OpenStack summit in Vancouver, BC, May 21st through 24th, we'll be organizing for the 1.1 release and onwards, and also thinking about how we work as a community. What we want, basically our community to look like this is ours to build together. However, we think it should be built, so if you're interested in that, you can drop by and register it's OpenStack org slash summit, slash Vancouver 2018, and this is some information about where we hang out.

A

Like I said we do use github, so you'll see github, comm, cotta containers. We also our communication channels, our IRC, with a bridge slack. So you can kind of pick your poison depending on what you like. We have a developer channel, kata, dev and then just the general enquiries is under kata general. If you do like slack, that bitly will take you directly to the invite. So you kind of bypass that stalled page there um I like to tweet at cotta containers and our developer mailing lists. You can subscribe at lists, cotta containers, dot IO.

A

So all this is also on cotta containers, dot. Io. You can find out more information there.

A

Sorry again, a couple phones, I, don't want to dash off if people want to a photo of our communication channels, but with that we have about 10 minutes or so for some Q&A and I think. What's probably, the the most important part is how we get your questions answered.

C

So one of the things that used to be a pain with vm's is that you have to slice them up yeah before you start them like not give it a 1 gigabyte of memory, and so on. Did you solve this or are you trying to solve it by like demand, allocating memory or disk so on? Well,.

A

That seemed really it's that one so.

B

We we do a CPU memory and networking hot line all the way through so based on the on the demands from from the workload we can add, CPUs and add more memory, basically on demand yeah. But it's not it's not an it's, not an easy problem to solve. Let me put it this way, especially for memory yep.

D

Yes, so what's the typically overhead that you pay for using the isolation and the second question is, can you actually expose hosts paths or GPU devices at this point or is that in the road path.

B

So the the overhead is actually it's not a binary answer where you can say we use 30 mix of memory, overhead, we use all sort of techniques for deduplicating memory, reducing memory conception and what we, what we realize from memory for example, is that when you run very small workloads, the overhead is there so compared to regular containers.

B

You do see a memory overhead, but when we start running the bigger the workload the the smaller the overhead, because with Kara containers, you can actually deduplicate pages across different different different VMs or containers whatever you want to call it.

B

So if you run ten different instances of elasticsearch, for example, you will see memory overhead, that's more interesting with with cata containers than with regular continues, because if you run ten different containers that running elasticsearch, which is very big workload, we're going to aggressively deduplicate a memory across those ten different containers, whereas with your regular containers, you're gonna have a lot lot of memory, duplication so with big, big workloads d, the overhead is, is better with Kara containers than with with regular containers. There's a CPU overhead. That is the least of our concern.

B

To be honest with you, the the VT or the AMD AMD instruction sets are done, so that's the the virtualization overhead is is very small. Nested is another story. Nestle the the the CPU overhead in nested is not is not that that the what what what's really ugly in nesting environment is is memory memory, overhead and the the the worst case that you can hit with in nesting environment is page fault.

B

So when you do page fault, you enter a quadratic algorithm to to feed your pages back into into the page tables, and this is this is very heavy, so nesting has its own set of overheads that are highly dependent on the kind of work that you're doing. So, if it's a it's a very noisy workload, that trash is your 80 lb and and goes constantly across your memory address space. Then yes, in nesting environment, it's it's not very good, but usually those workloads are not yeah. You don't want to have those workers anyway.

B

So was that the yeah more questions? Oh sorry, yes, the host pass on GPUs yeah, so we don't mount their host path. Percy. We do something much more secure. We do device assignment. So that's another advantage of Cala containers where you can, basically you direct device assignment in a safe way by using stuff like vtd iommu, where you can assign your GPU or your networking card, or whatever PCI or even non PCI devices.

B

You want to assign to your to your container and you can do that in a safe way, inside a virtual machine, as opposed to just mounting your /dev into into your container.

B

There was some question there, something.

E

It fixes up with the same page so shared storage containers not having the means of mounting a nice cozy volume. And how do you represent that inside kubernetes.

B

So, for for storage we we can do. Are you talking about volumes? Okay, so for volumes we can either do block based volumes, in which case you get a pretty nice setup. If you, if you can do that, then we go through something called 9p FS and not sure you familiar with us. It's a it's a basically virtio file system.

B

Let me put it this way, which is less performant than doing a blog based device assignment into into the virtual machine, but when you don't have full control over your host environment, this is the only the only solution left. Basically.

C

So qmu is getting fairly old and perhaps a little bloated I. um You is getting fairly old and perhaps a little bloated have you been looking at, say, KBM tool or similar kind of from scratch. Hypervisors.

B

So, there's a lot to say about sorry: there's a lot to say about um you and in yes, it's I think um U is 3 millions line of code. So basically, when you run your regular virtual machine, your VM M is 1/4 of the colonel sighs. So yeah, that's that's all too scary to be fair to qmu.

B

If you, if you configure it properly, are those three millions live love code, you probably end up running three or four hundred thousands, so we've been I mean, as I said, we, our main focuses is KVM TMU because well, this is as much as people ATM you it's the most tested and and the most performance in terms of I/o vmm implementation out there. So that's what people use in production when they don't have their own IP visor, but again in in in Catalan teeners. We implement each hypervisor support as a as a plugin.

B

So if you running your own private cloud- and you have your own hypervisor and you don't even want to have that as a public implementation inside care containers, you can just implement your plugin for cata containers. For your specific hypervisor and I know. Some people are looking at this and maybe running that already we just don't know about it, but the architecture is there specifically for that use case.

B

F

Had this is gonna sound weird, but in a traditional saying, traditional container security world you'd apply like control groups and Palmer and everything to the kernel. Do you need to do that inside the Carter container? Would you would you bother or is the segregation of the VM enough so.

B

I quickly touched on this, and, if you ask me, the the the VM segregation is enough, but we also found some customers and use case that were using tools that were actually monitoring. These kind of things were their regular containers, so they were willing to know if a container was trying to get out of the app are more or Esso Linux contacts or trying to call into a Cisco that was not part of the cell computers and they were they're willing to kill these those containers that are misbehaving.

B

So we realized it was actually used to provide the same kind of filtering inside the virtual machine. So when you run cat a container inside your virtual machine, you have the VM segregation, plus the all the filtering and security context that you can add to regular containers, especially for this for this kind of use. Cases where you want to have very similar behavior when you run your containers as scalar containers or as regular containers.

B

So it to me it's not a real interesting security added value, but it's a it's just for parity between between the two implementations.

G

So I have a question about storage. Do cotta containers have a concept like pool storage, pools or providers, or they can differentiate about storage types and stuff like that, if use them standalone.

B

Yeah, so this, oh you mean the the overlay and the the overlay, the overlay, a driver. We basically use whatever people set their kubernetes or darker engine with. So, if you want to use a device mapper, you can use device mapper. If you want to use overlay, we're gonna use overlay it doesn't. It does have an impact. So when you, when you use a block based device overlay driver, you will get much better performance because we can literally use virtio, scuzzy or vertical block to pass that to the to the virtual machine.

B

If you use other overlay drivers, then we're going to use nine PFS, which is less performant so depending if you can control your environment, then we we strongly advise for using a block based device overlay drivers and we're actually building a reference implementation of cala containers with all the best settings that you can use. If you provided that you can control entirely or your host environment, what would be the best choice for running the the most performance guarantee, I.

A

Think Sam else will have to be our last question of time. So.

H

You said before that the Qatar continents are running inside the full-blown virtual machine, and this comes with a restrictions, and you have to allocate CPU cores memory, storage and stuff so why, for example, they use garden containers extend of having a virtual machine. That's inside I have a rating system where I can run docker whatever. What are the pros and the cons in that case.

B

So so them, the main pro for cata container, is that it's it's seen as a container by all the other components on top of it. If you run docker the darker engine inside your virtual machine, you enter a completely different architecture where you have to have, for example, kubernetes being able to talk inside your virtual machine to your darker engine and your doc engine would launch your container. So you basically back to a kind of regular setup, whereas with carrot containers, the virtual machine behaves exactly as a container.

B

So you talk to the to the cat to Quetta containers at the virtual machine like exactly the same way as you would talk to a container, so you don't need to have specific, setups or specific architecture for using cala containers inside your kubernetes cluster. You just launch them as regular containers and you can specify if you want the isolation de layer to be an HP based or Hardware virtualized. That's the only thing you have to do.

B

You don't need to set everything up besides that, if you run your dagger engine inside your virtual machine, you will need to do a lot of different setups to make it work, and so no more questions. Yeah.

A

Unfortunately, we are, we are out of time. Okay, we have some one-pagers. That kind of summarize what we talked about. Hoodies stickers will stick around for a second.

B

A

B

A

You for coming.