KubeVirt Talks, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: It’s All for the Users. More Durable, Secure, and Pluggable. KubeVirt v0.53 - Alice Frosi, Red Hat

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

It’s All for the Users. More Durable, Secure, and Pluggable. KubeVirt v0.53 - Alice Frosi, Red Hat

If you liked KubeVirt before, you'll like it even better now, and if you ignored it before, it's time you tried it out. In the last year, KubeVirt has added numerous features, improving scalability, security, storage, network options, and usage for specialized hardware and edge applications. Join the KubeVirt contributors for a brief tour of what's new, including: - Additional traditional storage features for hotplug and snapshots - More mature high performance computing with SRIOV and vNUMA availability - Improved security with unprivileged VMs - Foundation for technical workstations using vGPU slicing - Don’t fear updates with mature Live Migrations - Improved automation with new like Argo CD and Tekton integrations - Rapid Kubernetes tenant cluster using ClusterAPI - On the verge to CNCF Incubator? We'll explain and even demo some of the new features. KubeVirt is going places, join our session to find out how it can help you go places too.

A

Everyone, I think, it's time to start so, let's introduce myself. My name is alicia froze I'm from reddit, and today I'm going to tell you what happened in the past years in hubert.

A

So first of all, this is not going to be an introduction about kubert.

A

If you have the chance to attend our virtual office hours, you get already an introduction, but if you are completely new to keyword, you can think about hubert as a kubernetes extension so designed to be kubernetes native in order to run virtual machines using livert and kvm inside containers.

A

So in cuber we focus on many different areas. We add new virtualization feature. We focus on improvement of security, but also we try to be part of the kubernetes ecosystem, so we integrate with various projects in order to offer a more robust and scalable platform.

A

So cubert recently has been graduated as cnc avetine project, and this shows how you can effectively be around virtual machine using kubernetes.

A

So one of the strongest use cases for kubernetes running gpu's workload, and usually this cannot be effectively run with the regular containers.

A

One example is the slicing of a cp of a physical gpus, so with cuber you can create us and slice partitioning gpus using mediated devices, and then you can assign those virtual devices to different vms, for example.

A

So initially the creation of these devices needed to be done manually in kubert, and recently we added the automation and the cuber will automatically create immediate devices based on the configuration. So this has been one of the recent improvements that we did in this area.

A

Usually, to run virtual machine effectively, we need to pass very low level information to the pod, where the vm is running. An example is cpu topology and in this direction we had recently the new affinity for the assigned device to the virtual machines.

A

This, for example, is particularly useful if the vm is using sriv devices or again gpus.

A

So one of the goal of keyword is to abstract the workload definition from the different options in order to tune your vms, one example is the real-time workload. So as a user, you just simply need to specify that you want to run real-time workload. That will be just an option on the declaration of your vms and then cuber, basically automatically pick the best revert option, but it also schedules the vm on a node with the kernel, real-time support.

A

So we focus on different areas. Storage is one of those and in storage area we have. We did a lot of improvement, but we also integrated with other projects in storage ecosystem and we are going to see many examples along the way. So, usually, when we talk about storage, we have two level. So we have the level the storage that we pass to the pod.

A

Then we have the storage scene inside the guest, so this can be particularly problematic if there are, for example, file system changes.

A

One example is the case of snapshot, so we rely on csi in order to take snapshot.

A

However, if, during the snapshot, the vms still continue to write on the file system, your snapshot can be ended up to be not consistent.

A

So recently we added the coordination between the csi snapshot and the qmu gas agent, so the human gas station is basically a process that runs inside the guest. So basically, while we take the snapshot, then the qml gas agent will fs freeze the file system, and so basically this we can guarantee that the snapshot is consistent.

A

um Always in this, this same kind of problem occurs when you try to expand the pvc, so certain storage classes support pvc expansion. However, initially this expansion was not visible inside the guest.

A

So recently we added the support for online resize. So basically we notify the vm that the file system has changed. So you can see also the expansion inside the guest.

A

So backups is one of the strong and needed feature for virtual machine.

A

Unfortunately, csi doesn't offer this feature yet so one of the most popular tool to take backups is velero and in keyboard organization you can find a plugin that helps you to integrate velero with cuber vm and you can use valero in order to backup the disk of your vm um one of the traditionally set tool in virtualization in order to manipulate customize your vm disk, our guest, the first tool.

A

So recently we had a new command versitile versitial is the cuber client with a new command that help you to setting up a guest, the first pod with the pvc attached. So, basically, you are able to use gusty first tool in a containerized environment and, for example, rescue a faulty partition, so another area is of course, network live. Migration is one of the top features implemented by kubert, as kubernetes doesn't foresee.

A

Pod live migration, yet so this is implemented by cubert and relies on levered.

A

Unfortunately, not all the virtual machines are migratable and one of the causes can be that certain devices cannot be automatically detached from the node where the vm is running and automatically attach to the node where we want to migrate.

A

So that was the case for sriv devices, and recently we overcome this limitation by basically unplugging the virtual function device perform the migration and then create an equivalent virtual function device on the target node, where we want to migrate.

A

We have another example of integration in the kubernetes ecosystem. Ico is one of the very popular tool for service mesh. So today, cuber vm can participate in service meshes. So, for example, you can inspect the virtual machine network traffic using the cali dashboard, for example. One recent feature we added is the support for single stack ipv6.

A

This is particularly needed if your cluster support only ipv6.

A

Security has been one of the key areas where we focus our effort in the past year, so initially in order to run virtual machine in a pod. We needed to specify additional additional capabilities, so we try to improve and remove this requirement. So today you can run a virtual machine in a pod with the standard security profile. So we don't need this extra capability anymore, always in security field. We add an initial support for tpms and virtual tpms.

A

This is particularly needed if you want to run windows, virtual machines.

A

One of the emerging area in virtualization is confidential computing, so we have added an initial support for imd, secure, encrypted, virtualization or suv. So, basically, today you can run kuber vm with encrypted memory. However, the vm is not a tested. This is something we are still researching. There are open prs, and this is something coming in the near future.

A

So we want to integrate with many projects and be part of as to the kubernetes ecosystem. So in cuba organization you can find also collection of promethoise, alerting and grafana dashboard in the monitoring repository, and this can help you to understand the keyword, alerting issues.

A

So kubernetes is everything with scale, so we want to be able to create easily a group of yam and also be able to manage this. This is exactly the goal of vm pool. So basically, vmpool allows you to define manage group of vm based on template.

A

One very popular tool in kubernetes ecosystem is stacton or cicd and in kubert organization you can find a collection of tecton tasks that allow you to perform operation on cuber vms, like, for example, creating custom, vmds customization executing command inside the guests, or maybe waiting for a certain state, and there are many more so.

A

As already said, we want to be part of the ecosystem and we focus on many different areas, either from supporting different platform from integrated with popular tools in the kubernetes ecosystem, but also with other projects that have been born with kubernetes, like some kubernetes c project- and here I have a couple of examples.

A

So this year we added an initial support for arm 64.. So this support with the multi-arch images in kubernetes project. There is something called cluster api, so basically, these allow various cluster providers to create kubernetes cluster on top of kubernetes cluster, and this is actually a perfect use case for keyword.

A

So in kubernetes project you can also find a cuber cluster provider and we are going to see that in a demo at the end of the presentation.

A

Again, we want to integrate with many projects as possible and we also added a health check of cuber status in argo cd, so another very popular cicd tool.

A

So what's next, so confidential computing probably will be an area where we will focus in the next year and we want to be, and we want to be able to deploy full confidential and tested virtual machines.

A

As already mentioned, we want to distinguish and separate the vm option from the workload definition and abstraction, and this is exactly the goal of level apis. So basically, you can define the kind of workload that you want to run and then basically, cuber will pick the best option in order to suit and match. Your use case already mentioned the cuber cluster api.

A

We also plan to add more feature like, for example, a better drain mechanism. If, for example, the underlying node has been drained.

A

One of the expected traditional feature for a virtualization platform is the whole plug of various kind of resource, and this is actually one of the main challenges that we are facing with the kubernetes because of the pod immutability. So usually, if you want to change some resources assigned to a pod, this usually means that you need to restart the pod, and this is particularly problematic for virtual machines.

A

So we already had the implementation of volume hot plug, but we also plan to do for other kind of resources like, for example, adding and removing network interfaces from a virtual iranian virtual machines.

A

So that was a quick overview. What happened in the past years? We you can reach us in many ways. We are on slack. We have a mailing list and we meet weekly on zoom, so you can feel free to join us.

A

So I would like to show you a demo and then there will be some q a so here, I'm going to show you um the cuber cluster provider. So basically, I'm going to create a cluster on top of another kubernetes cluster using cubert, and you can see also cube working action. So you can. You can have a feeling how that looks like so here you have a kubernetes cluster with two workers with two nodes, one monster and one worker.

A

I have already deployed cuberdare. You can check the status of kubert using this crd and we can also check the values: q, vert, infrastructure pod, so everything is up and running.

A

So now I want to create a cluster, and this is the definition, so I will just go through very quickly this base on a template that is available in the cuber cluster api repository.

A

So basically I can define a cluster.

A

I have something called cuber machine template that defines the template for my vms and there are some configuration few queue button and then we are interested also in the machine deployment. So this, basically it's a crd that controls the number of the worker I have in a cluster, so I can try to apply this yama and various resources has been created.

A

So, first of all we can check which cuber machine has been created, so I have a cluster with one control, plane and three workers and we can check the avm. So there is one vm starting up and as already mentioned, we deployed virtual machine inside pod. So this starting pod is the pod where my vm will be started.

A

So it's taking a while to start the pod, so you can check the reason. So basically, this is pulling a container image. So with cuber we are using container image in order to deliver vm disk. So this is a very handy way how we can shape vm disk with containers.

A

So it's going to take a while in order to pull this image, but once it's present on the node, the creation of other vms from this kind will be very fast, because we are basically creating a snapshot of this of this disk, so yeah it will take a while to pull. Then the vm need to boot.

A

So yeah we can simply watch the pod.

A

The container this is quite it is quite large, so I have a first pod up and running and after a while I will end it up with the four pod. So let's check it again. I have four part because I have four node.

A

Okay, I can check also the vm status and it's everything up and running okay. So, as already mentioned, all the vm are created from templates, so we can check the template and I have two kind of template. I have one for the control plane and I have one for the worker, so we can inspect, for example, the control plane, it's a very simple vm and we can also inspect that one for the node that looks pretty similar for this example, but you can customize and define and assign different resources.

A

Okay, so, together with the keyword machine template with also okay. Sorry, um first of all, now we just check what the vms had up and running, but now I would like just to access the deployed cluster.

A

So with the cluster creation we have been also different secret has been created and we are focused especially on two of them. So we are going to try to get the ssh key, so the ssh secret basically contain a private key that has been injected in every vm.

A

So I can instruct this secret, save it in a file and use this key later in order to access the vm that has been created.

A

So this is again a simple kubernetes secret encoded in base64. So here we can see the private key. So I will simply save these in a file. Give the ssh permission.

A

In order to be able to use the key, then I have to start, of course, the ssh agent and then I will basically add and the key so together with the key I will I have also. I want also to save the cube config. So here you can see also the cube config of the deployed cluster.

A

So again I'm going to save this and it will be useful uh later. Okay, so I got all the all the secret I needed and I want to access the um the control plane. So, as already mentioned, we have something called versatile. That's the cuber client, and this has different commands that help you to interact with the virtual machine so, for example, vctl as an ssh command that help you to access your virtual machine.

A

So I can simply get again the name of my vm and I want to use the key that I save and got it from the secret and try to access the control plane.

A

Okay, so now I'm inside the control plane, so, for example, I can inspect which container running on it, and I am trying to check what I have inside the my home. I don't have any dot cube, so I'm basically missing the cube config. So that's the reason why I also saved the cube config in in a file, so virtual is also another command called scp that allow me to copy file from my localhost into the remote one, so I can basically copy the cubeconfig I saved previously on the control plane.

A

So in this demo I'm just going to access the control plane, but you can access any nodes, so we can check it yeah we copied successfully the cubeconfig. So now I can use these in order to access the deployed cluster.

A

So, for example, I can get the node and I have a cluster with one control plane, three worker. We can check the part that are also running on this. There are this: is our the boring part that we got we kubernetes installation cool, so everything is up and running now. What I want to try to do is try to scale, so we can try to get the machine deployment.

A

This is something that controls the number of the worker I have in my cluster. This is something a little bit that works like the replica set.

A

So we see that we have three replicas and now I'm going to try to edit this crd and try to modify the number of replicas from three to four and see what happen okay. So this has been updated successfully and we could try to get the vm and I have a new one starting up.

A

So if I get the port, I have already a pod running. This is because I have pulled the container image and is already present the disk, so it's very fast to to create new new vert launcher pod, okay and I already have a four worker.

A

So what we can try now is to access again the control plane and see what happened in the cluster that I have deployed. So if you look now, there is still no, not a fourth worker. This is because it takes a while to put the vms. Then the node needs to access the cluster, but if we try the command after a while, we can see that we have a new, a new node there.

A

There is 20 second each, so everything is up and running, so we have managed to scale very easily using the kubernetes cluster api.

A

Okay. So now I want to play a bit with the storage. So let's try to create a new volume, new pvc and I will try to hard plug so to add this storage at time. So let's create a new one.

A

So um virtual again has a new command called add volume. So, first of all, let's check before adding which kind of volume I have in the control plane. So I have simply two of them: vda and vdb.

A

So yeah that's exit, so they already mentioned virtual as a command called that volume that allow to halt plug resources. So I can just simply use this. I have to specify the vm where I want to add. The new storage need to specify the claim name.

A

And then we will see what happens yeah so.

A

So this the request has been successfully submitted, but we can see that we have a new body here. So this part is actually the trick that we use in order to store the hot plug volume there, because the hot plug volume unplug is not transparent to kubernetes. So we want to protect the pvc from be accidentally deleted, for example, and this is the reason for this hp pod.

A

Okay, so when this part is up and running, it means that we have been successfully managed to add this volume. So we can try again to log into the control plane with ssh and check the disk again.

A

So now you can see that we have a new disk as the aider, so we have been successfully managed to add the node okay. So now, let's try to increase the size of this volume, so I can basically modify the size of this pvc.

A

Let's try from 10 to 20., let's check if this has been already applied to the pvc. That's the case. So let's try to lock it again and check the disk.

A

So you can see you can now sda as a 20 gigabyte size, so just everything I wanted to show you. Hopefully you saw cuber in action and, for example, with the cuber cluster api. You can create maybe on your bare metal server additional cluster, maybe for one for development, one for testing and one for production.

A

Okay, there are any questions.

A

I don't know if you need a microphone.

B

Hi um great presentation, um a question is kubford now fully supported by openshift, so that I can run obscure, bare metal and run fertile machines on it.

A

Yes, so you can install, uh you can have openshift on your cluster. You can install cuber there, but this is still a very early stage. This secure cluster api, so deploying openshift in openshift is something that is not supported yet.

B

Okay, thank you.

C

um So, basically, your nodes were bare metal nodes and they were can be. Can they be provisioned with metal cube.

A

Yeah I mean how you so, where you install kubert, so the um bare metal. Let's call it like you.

C

Say basically, can I have a cluster api with metal cube as provisioner and then over that bare metal? Can I use kuvit.

A

Yeah, if you have kvm yes or even you could use emulation. Of course you should yeah if you have a node with the kvm and virtualization capability. Yes,.

D

C

D

C

Thank you for the presentation.

E

um What are the steps for making a a custom cube, vert vm image for.

A

Like so, can you repeat the question I missed the first part.

E

What are the steps for making a custom cuvvert vm image.

A

um What do you mean with custom keyboard.

E

Yeah, like how do you build an image with like a custom, vm image with some stuff pre-installed on it, for example,.

A

You mean cuber or an operative system.

E

For for cube, verts.

A

E

I can have a vm image with my application, pre-installed on it, yes, and to deploy this with kubert. Yes,.

A

So I mean kubert is just it's like an extension and help you to deploy the virtual machine. What is running in your vm is up to you, so you can prepare, of course, a vm disk with already something pre-installed, and so, for example, when I mention tecton task, that's exactly one very good use case for tecton task.

A

You can use, for example, guest festools in order to add, for example, additional packages on the disk, or otherwise we support also cloud in it. This is something that you can specify in your vm definition and basically, when you boot the first time it's going to install the packages, do some configuration, so it really depends how you want to configure, but it's of course possible.

A

Sorry, I think you need a microphone.

C

In my imagination, we cannot use any image without preparing for kubrick like having cube idm binaries on it, or some prerequisite to be used as.

A

No, you can have a simply a vm disk you, so it's if you can also prepare it, maybe even locally with beard manager, for example.

A

Then you can copy that disk in a scratch container image and then could be used, for example, as a container disk. So the container is basically what I show in the demo when I was pulling so actually that's a good example. So, for example, what I was pulling in during the demo, it was already a fedora vm disk with already kubernetes installed. So basically, when I, when I started it, was already up and running with all the packages there. So yes, for example, cryo and all the the things needed for a basic kubernetes installation.

D

Okay, thank you.

A

So yeah can, I think so when.

B

A worker knows our bare metal. It would be straightforward to provision the vms, but when they're in the cloud or on other vms, do you need nested virtualization.

A

A

Yeah I mean you need to support an asset vms. If you are not environmental or you can use emulation, but of course this is not a suggested way. So necessity is, of course, the best option.

F

Hi in your cluster api demo, I noticed sorry.

A

Where are you right.

F

Here, oh hi, hi in in your cluster api demo. I noticed that you, um you copied the cubeconfig into the control, plane, node and accessed the cube api server from the control plane node. How would you have exposed cube api server outside of the control plane, node? So.

A

F

Could you have run it from the same session that you were running cuttle from.

A

So it's of course running in a vm, so you need to expose the part, and there are a various way how you can do this, for example, virtual is also a command called expose. um Yeah you can forward the traffic, so there are. There are ways how you can do that in cuba. It really depends also on your network solution that you are also deploying with cubot.

A

So it really depends on the use case, but it's it's of course possible to expose a port from from the vm and then be accessible from the outside from the deployed cluster. That was just a demo and it allowed me to just show a couple of things but yeah that's possible.

F

What what does that expose? Look like um in kubernetes terms? Are you creating a service or um how are you how.

A

F

Expo you're just creating a service.

A

Yeah so, for example, ssh we create a service in order to expose the ssh port available inside the virtual machines. That's that's another example. We we are using um kubernetes services in order to access various uh networking inside evm, but the ssh case was exactly the same.

D

A

So we still have three minutes the does anyone have other questions. I think so.

A

No okay done thanks for your attention, enjoy coupon.