KubeVirt Talks, 8 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Running KubeVirt Workloads with No Additional Privileges - Ľuboslav Pivarč, Red Hat

Description

Running KubeVirt Workloads with No Additional Privileges - Ľuboslav Pivarč, Red Hat

Kubevirt allows running VMs natively alongside containers on top of Kubernetes, enabling faster containerization, serving legacy workloads, security-sensitive applications, and many more use cases. One of the best practices for running container workloads is to not run them as root, drop all privileges and use Linux Security Modules, such as SELinux or AppArmor. Two years ago Kubevirt started cutting down on capabilities that are used to run VMs with the ultimate goal of requiring no additional capability. Simultaneously Kubevirt integrated seamless SELinux support with its own policy for running VMs. The missing part was running VMs as a rootless user and recently this option landed in Kubevirt. I would like to present what it took to achieve it with Kubernetes and Libvirt and show how important it is to have this use case in mind while implementing new features into the Virtualization stack.

A

So my name is I'm working at redhead as a software engineer and I'm working on the cubbird community project about virtual machines on kubernetes I'm about I'm there about for two years in my approver and the name of the talk or the title is running qbr2 workloads with no additional privileges.

A

So let's start so what is coming? What is our agenda today?

A

So I'm going to talk about kubernetes and Qbert, so I'm going to give you a quick crash course on kubernetes and Qbert, then we will have a see uh how is security enforced inside kubernetes because we are running with the kubernetes, then we will have a look on what is actually enforced by these security mechanisms, and then we have a closer look on cubbird and how it stands with this security mechanisms.

A

So what is actually kubernetes so kubernetes is an open source system for men for automating deployment, scaling and management of concurrence applications.

A

So words doesn't tell much to me, so let's have a look on a few of the images and workflows so kubernetes API Pro is API based on on the objects, and these objects are either representing a lot of features, and so when a user wants to run a workload, it needs to find what type of object is representing this and the name of the object is actually pot.

A

So when he wants to run the workload as I said, he needs to find also the schema and API version and so on and then to craft this pack of the of the containers he wants to get from the system. Then he ports it to the kubernetes API kubernetes API can validate the schematics and the schema and we'll store it inside the storage. After that, the like scheduler can pick it up and pick an old for suitable for the containers to run on.

A

So let's, let's have a look on the Node perspective, so we we can see. There is a node agent called cubelet. The cubelet is watching the uh the kubernetes API for the workload and after that, with help of container runtimes. It creates a pot so put this port is actually only abstraction for one or more containers these containers and ensure the resources that are assigned to the pod Cube light is also responsible for a life cycle of the pod, which is necessary for some kind of interaction with the system.

A

So that's what we need from to know about kubernetes. So let's have a look. What qubit actually is so qbuild is kubernetes extension that allows running traditional virtual machine workloads, natively side by side with the quantum workload. So if you are on the move to Containers, but you still know and acknowledge that you need a virtual machines, uh this is the product or project you are looking for and so how? How do we do that?

A

So, as I said, kubernetes is based on objects, and it's really extensible so cuber, just hooks in into the API, provides own object, called virtual machine, and we have some web hooks, an API that can validate the schema and so on. So user just interacts us with any other any other workload type in kubernetes and post a spec of the virtual machine to the kubernetes API. There we pick it up and transfer it into the pot, so the work of workload, specification kubernetes, does some work for us and we end up on the Node.

A

So here are two important things to talk about the first one is that we get a pot. We call it a wheat launcher here we are running components as a live, weird chemo, which you are probably familiar with, and this is this is the component which is around for the user. So we don't trust the component, and there is also second component, which is actually our node agent called wield Handler on the left side, and this is a privileged container.

A

So it allows us to see other processes on the Node, because contents are just processes, we can mess with them and it's really important for micro, because we need to do some privileged setup for the virtual machines to to be able to drop the privileges on the side of the of the user, so the build launcher so great.

A

So how does kubernetes security mechanisms are implemented so back again on the first test uh into the first picture, uh as we can hook into the API, other mechanisms can hook into API, and so, for example, there is a openshift specific one, which is called security, content constraints or a kubernetes or playing kubernetes Upstream project called pod security. It's actually not project, it's integrated into the project and most of the times, this mechanism provides some kind of pod security policies and these policies are like Spectrum.

A

So on one side you have like elasticity policies which is uh like hardening best practices. What you should allow to normal user and on the other side, there is a privileged policies which allows for non-privileged escalation. You can do basically whatever on the system and that should not be assigned to any user. You should know him and so on.

A

So the problem here is that these uh policy and or security enforcement mechanisms are often not enough flexible enough. So if you need some privileges for the cube build for the visual machines oftentimes, you need to give a privileged privilege.

A

Access to the user, so he can then actually craft any arbitrary workload with the with the same privileges or even more privileges than we are using. So this is a real issue and we need to integrate with the kubernetes kubernetes model to allow normal users to run the virtual machines. So what what is actually restricted? At least it's long I, you don't need to run through it.

A

um It's basically it comes down to two things. It is interesting thing, a few of the features that kubernetes provides and the other thing is just plain security stuff which you would use with processes on the Node as as usual.

A

So we are going to focus on three categories here or three uh features and those are capabilities, cellinos and running other route. So.

A

So what was the first step for us? The first step for us was actually to have unprivileged networking, because we used a lot of capabilities for the networking we had to cut down on on these capabilities.

A

So some background background is that kubernetes have some default Network when the workload is submitted. It gets some IP address the container gets interface and to connect the interface to the guest. Chemo, provides few options, for example, sleep, which should be unprivileged way to do it or through. It also provides a way through tab device, but configure configuring. The top device is actually privilege operation or most of the networking configuration is privileged operations.

A

So we need to somehow solve this problem and at the same time, we need to provide the IP address of the workload inside to the guest and we use standard the ACP.

A

So how to address the problem is to actually offload the networking set up to the privileged component, which is called the build Handler, but it also requires uh the well-known management tools like libbeard to get us give us the opportunity, for example, to not manage the networking for us right.

A

So there is a option unmanaged in the library that basically does what you need, and this also means that the existing management tools needs to know that there are some cases when you they are losing their Privileges and should not uh implement the features uh in a privileged way. So one problem stays, and that is the we leave out the net buying service I'm get I get back to it. Why it's the problem, but it's actually a problem.

A

So what about running the workloads as normal? So it can be easier setting the user for the workloads of setting the user on the container and then using Liberian in the session mode. What session mode? What it does is actually tells Libya how you you don't have much privileges. Please don't do a lot and please run the virtual machine as as the user which is requesting it so yeah, and that can be that easy.

A

But for not for us, because some security policies actually dictate you that you need to run as any user, which means you get parallocated ranges of UI IDs and you need to cope with that uh which is good.

A

For example, if you are running human processes on the host, you don't want them to be the same user because they can uh each other other disk and so on, um but it has some challenges with containers, because the first permissions on the on the file system are built uh are set on the build time of the actually in the container image and modifying the content of file system might not be a good idea because there can be copy on rights file systems.

A

So the solution is to actually use kubernetes for this and use a feature called empty directory which basically give you a temp FS. Whenever you need it and the permissions are exposed and then you can manage it on your own.

A

This solves a burden of vendors to actually keep track of the file system structure that they are need to allow to the user or export to the user. So the next feature you would like to use probably is some kind of storage. So kubernetes allows two types of storage so fast and block volumes, and they don't have really standard permissions models. So what basically basically happen when you are running as a non-reducer?

A

You get probably permissions denied, and for this there is a feature inside kubernetes, which is called FS group. This makes sure that the files and directories I are inside a special group, and you are then in in the group as a that that doesn't work well with all the storage providers out there, and this is a feature actually restricted by some of the policies. So we can't use that and the solution is again like manage. The permissions inside the privilege component called rear. Handler.

A

So what about devices needs to access some of the devices, for example KVM for accelerating and we host net for better performance of another networking? What about vgpu, passthrough or srov networking? All these? All these things needs to access the devices so kubernetes have this framework. Our mechanism called device plugins that make sure that clusters is the resources, regular work and schedule and container gets the devices.

A

But what happens when you run as a non-road user? You actually don't have access or it's inconsistent, and why? Because they copy the permissions of the host So based on the vendor, you are using or operating system it. It can varies. So it's not usable, it's inconsistent and so before solution. I need to mention that kubernetes actually acknowledged this and has a follow-up on it to allow the access for the Android HD user.

A

But for us right now we need a solution right now, so uh again offloading the permissions on the build Handler. But these are the drawbacks, because we only know a few of the devices that we can manage. So if you are integrating a new device through our extension points, then you can fail to do that.

A

So I mentioned we left one capability for the networking and we actually introduced one another P trays when we were introducing a vtpm support inside Cube build and what is the problem with uh capabilities and non-road containers.

A

Is that the calculation of the capabilities of the new process that is executing so the container is different when you are running as a root and as a non-root, so you've probably seen this main page and and the trick is that you need to use file capabilities, or the first trick is to use the file capabilities to actually on on the executive binary to get those to to make sure that the process gets the capabilities right.

A

um The second trick, or the second way to do it- is to use embed capabilities, and it is a better way and I will tell you why. But this is not integrating its kubernetes. So again, we need the Upstream super for that, so why ambient capabilities are better. They don't require any changes to the images, so you can run out of the box and some security policies dictate you to run to run with the allow.

A

No, no don't allow any privileged escalation flag in kubernetes, which is actually no new previous bit for the process, which means the capabilities are going or the file capabilities are going to be ignored, and so you don't get them. So you end up being denied access.

A

Two operations you want to do drawbacks or file file capabilities. Is that you need to set the file capabilities and require them, like always for the workload, because if you would not require the capability for the power code, then the process, the creation of the process, will fail because it is not allowed to have the capability that is set on the on the binary. So this allows opt-in approach. For example, if you don't need vtpm, we would probably drop the P trades, but we cannot uh so last I.

A

Think I'm trying to Target is the cell Linux, so uh keep it enabled and uh what's the problem there. So some of the rules were not recognized suitable for a normal container, T type, which means normal container workload, should not have the rules, but the situation is getting better and better, and we have just two pain points there and there is virtual FSD, which requires a lot of a lot of rules, and one more thing is.

A

uh One more thing is a huge table file system which so the policy allows you to create a file, but not the directory and leave your distant like creating the structure inside of a huge PDF file system and it's it gets tricky with Australian example. So that's not good, but probably this rule can be upstreaming, so what's left so arbitrary user needs to be used for the workload to uh to play with some of the policies.

A

uh This can get tricky because the username spaces are going to be thinking kubernetes. So we need to somehow think about that. uh Also, we are trying to remove the custom policy and only set it to any need Upstream all the rules we require or use alternative apis for things we need for, for example, for the huge table uh support you can use the approach of the file system, but also you can use MFD API, which is not privileged and yeah.

A

We are waiting for the ambient capabilities in Upstream, and one more thing we can do all all we can we can do all- is to probably switch to restricted first approach and think about what kind of uh privileges we need for some of the operations. We are doing some management tools or other emulation programs.

A

um That's all thank you and you can get it out with us with Twitter handle and dislike any questions.

A

Thank you very much.