KubeVirt KubeVirt Summit, 24 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Introducing Volume Hotplug in KubeVirt

Description

Introduction into the current state of volume hotpluging in KubeVirt, what is possible, what is not possible and what are the challenges.

Presenter: Alexander Wels, Principal Software Engineer, Red Hat

A

Forward to see how to.

B

A

Cd of my vm thanks all right all right, um so my name is alexander wells, I'm a software engineer at red hat and I'm on the storage team, and normally we work on cdi, but every once in a while. We go over to kubert and do interesting things like like hot plugging. um So what is hot plugging?

A

um Well, basically, it's just adding you know additional resources to a virtual machine. You know anybody um is from a you know, a traditional virtual machine management uh system like overt or openstack. You know, everybody knows what it is you just you know. Maybe your virtual machine needs more memory or cpu or an extra network or whatever um you can just hot plug um these resources into your running virtual machine without shutting it down so um we're in kubernetes world right. So why would we even want to hot plug?

A

Because um you know generally, if we want to change resources on a pot or after we just you know, kill the pod and start a new one and we're done there's no! um There's no issue.

B

A

However, with virtual machines, the the um it's a little bit different, uh you actually want to keep your virtual machines alive, usually because it's running something really important and you can't just restart it.

A

You know, maybe the virtual machine takes a while to boot up and and will cause a service disruption. um All the reasons are. um Maybe you, wanna um have cube, verb, be sort of like a service provider for another kubernetes cluster, so you can get you know kubernetes and kubernetes, um and you want to add additional resources to the nodes in the client cluster.

A

um So um you know hot plug is is relatively wrong instead, so so, what's the problem in kubernetes, you know uh kubert vm, um it's basically just a process running inside a pod, and you know that allows the vm to appear like any other pod in the cluster, and we can, you know, manage uh vms just like pods. um It's one problem, though pods are immutable, so I can't just modify a pod and add additional resources.

A

So uh if I want to add cpus, I can because you know the vm um gets the resources from the pod, but the part cannot be changed um all right. I uh went ahead a little bit here, uh but for volumes we we sort of managed to uh think of a a um a way to bypass this problem, and um let me start with sort of explaining how storage works in kubernetes and that will go to the solution.

A

So, storage in kubernetes you know volumes are represented by persistent volumes and if somebody wants to use persistent volume, they make a claim um once the claim is being created, kubernetes will match it up with a persistent volume, bind them together and then, when a spot wants to use the pvc um you know the part is scheduling. Node and.

A

The volumes attached to the node so that the part can use it so the um as you can see here, you know I have two parts running on my node part: one uses volume one and volume two and part two uses volume 3.. This is you know, basic kubernetes storage, so q4 uses kubernetes storage for for their vm disk images.

A

um We you know one time ago decided that we wanted to have uh one disc per one volume, so we could use um any online kubernetes apis like snapchat, shouting and cloning, and things of that nature. You know mike and ryan uh talked about that a little bit earlier um and if something new gets added like the synchronization or whatever you know, we don't have to re-architect our our code because we can just use. You know what kubernetes provides.

A

So um pods are immutable. um We know that a um when a new part is is is started on a node. The volume is attached to this node, but we can't add it to a running path. So what do we need to um make sure that we can hot plug? You know we want to attach the volume to the node.

A

We then somehow need to mount the volume that was attached to note inside the vert launcher part. The vert launcher part is what's running the vm um and we want to use. You know standard lip. Vert attachment calls um to do this and um we don't want to fight the cube.

A

So it is possible for us to outside of kubernetes um attach the volume to the node, but then the cubelet wouldn't know that we did this and it might want to do something with the same volume and we're or we're fine.

A

So let's not do that.

A

Instead, let's create an attachment plot. um So what's the the purpose of this attachment, the um attachment part only purpose is to attach the volume to the note um and we make sure that the attachment pod um has a strict affinity with divert launcher paw. So that we can guarantee that when the attachment part is started, the volume that it uses is also attached to the node that the verb launcher published or not.

A

So we have an attachment part. We have. The couplet has attached the volume to the node. um So now, what do we do? um The first launch of paul has no permissions to do anything.

A

Fortunately, the verb handler has you know enough permissions to take the volume that was mounted on the node.

A

Look it up using the attachment, you know, find the location and then uh mount it into the vert launcher, um and it also you know, can ensure that all the appropriate permissions are being granted so that the verb launcher part can actually access it and read the volume.

A

So that looks sort of like this, so um what has happened? The attachment bar has started. The volume 3 here was attached to the node. The vert handler then sees the attachment pod and takes the volume and bind mounts it into the launcher. Pod. That is running the vm.

A

So at this point the volume has been mounted inside the vert launcher pawn and can now be used by the human process. That's in the verb launcher and attaching it to the vm is, is at this point just you know, calling the appropriate library was a little a little faster than I thought I would be all right. um So let's do that mom. um Let me share a different screen.

A

Here all right um question: is this big enough or do we need to make it a little bigger? I think so.

A

That's better, um I'm not brave enough to do a live demo.

B

A

I recorded this earlier um so just to start off, I um I have a cluster with two nodes and on the cluster, I've started a vmi with a container disk. It's actually the um examples.

A

uh There's a under the keyboard repo there's an examples directory with uh some vmis in there um and I'm going to create a well that's, not good.

A

I can't see my um oh.

A

I don't know why I didn't show the uh the data volume uh yeah, but it's just a blank data volume um oops make it a little bitter okay. So, let's, uh let's pause for a second here, so the uh blank yamo is, is just a data volume and it creates a blank five gigabyte um disk image um on a local storage um and um it created the name. Is volume help plug and, as you can see, the face here is wait for first consumer, so um the actual disk image has not been created.

A

A

So, let's uh log into the the running vm just to show that um there's no, you know additional disks. Yet I'm in depth this directory.

A

B

Right so, as you can see, there's only two disks.

A

One is the container disk and the other disk is the cloud image disk um all right. So, let's um add the volume to the running vm.

A

So it's relatively easy wants the pr that has this add volume command to her control. All you do is say, add volume, name your vmi and then the name of the volume. In this case you know, volume plug.

A

All right so here you can see um because we're in like first consumer, um we need to create a pod that binds the pv to the node that the vert launcher is running on. So this is not the actual attachment part. Yet this is just a dummy part to trick um cdi into creating the disk.

A

So um let's uh here cdi has created the importer volume, so that's actually creating the blank disk image file.

A

All right, so the um disk image file has been created, the importer part is done, and now it's created the actual attachment part. As you can see, this is a different one. From from the first one.

A

So now the disk has been attached to running vm. So let's go back to the console. I was instead disk, as you can see, there's now a buy id directory, so let's go in there and there we go. That's the the disk that has just been unplugged.

A

um To show that you know I can actually work with it. I'm going to create a file system on it by using the the id that that's in the directory, I'm in right.

A

A

All right, so I have a file system creator. I could mount it, but it's not really all that interesting. At this point I just wanted to show that I can actually write to the disk that was attached to the over launcher.

A

B

A

It for a second, and once we moved, you can see that the attachment pod was gone and, um as you can see, the the disk is no longer in this binding. Actually, the by the directory is gone is all normal and, as you can see, if I do have this, the disk is not there.

A

So, let's add it again, but this time add a serial name. um Sometimes it's sort of hard to find your disk inside the vm. So the serial option allows you to put a unique identifier in there to find the disk inside the um inside the vm.

A

So uh just to sort of show, what's happened to the running bmi um right here, you'll see that a new disk was added to the vmi with a scuzzy bus, that's the name and that's the serial that we've passed into the guest. um A matching data volume is added to the volume section, and somebody asked about live migration, because this is a local storage.

A

You know I can't live migrate, this vm at this point and there's a new volume status section and in the volume status section for any hot block volumes, you can see the the part that was created for attaching um the status and then uh the target says sda and right now it actually matches slash def such sda in the guest, but there's no guarantee that the target will actually match the value inside the gas. That's why the serial is important.

A

All right, so, let's.

A

Let's get back into the vm.

A

As you can see, the uh um disc is back, but now the name contains the serial number or you know the serial that I added to the command.

A

So that way I can easily find the actual disk that I added you know in case. I need to find it specifically with that serial number and then demonstrate that it's the same one as I added earlier. If I try to create a file system on it again it will, it will say: hey.

A

um You know, everybody has the faucets tomorrow.

A

All right, oh and I forgot to show that um right here. It actually is an extra disc and it's def sda. But, as I said before,.

B

It doesn't necessarily match what was.

A

Reported in the volume.

A

A

All right, let's just remove it again, just to demonstrate that once it's removed, the extra volume is no longer in the ammo. So now it's.

B

A

Here it's gone from here and it's gone from here and that's it for the demo. Let's switch back to my.

A

Presentation all right, so uh what's next so right now, even if I attach a read, write many volume uh we've disabled, live migration.

A

So we're looking at adding that at some point in the near future, um I actually demonstrated the weight for first consumer support. The current release version does not support this, so if you use a local storage uh with wafer first consumer, you know the odds are pretty good. That uh cdi. Will you know, create your disk on the wrong node and it won't attach to the running. Vm.

A

um Hostpad is a little um painful.

A

There's some questions. I see housepass a little painful, um because um the attachment part doesn't allow me to find the actual path of the the disk image to attach or to to by mount um also working on. I think I've found the solution for that, and hopefully that will have a pr soon and I I showed in the demo the vert control support, which does not exist right now. So right now, you'll have to you know, direct directly call the api, which is also somewhat painful.

A

um That's it questions like about 10 minutes. It's not bad.

A

All right so mike gertz asked on the uh slack channel. um Does it only work with this images or you can use raw devices you can actually use both raw devices are slightly uh more difficult for for attaching because you can actually bind mount them, but once the blood device has been added to the to the host, I can, then um you know update the devices that are allowed in the running launch container and um added that way, so what devices will actually work um and it's slash, dev slash, sda.

A

The s is for for because of discussing yes.

B

Okay, um alice has the question: does the attachment pod need to be deployed in the same name, space of the hot plug pvc or data volume.

A

uh Yes, it has to be in the same name space as both the first launcher and the pvc. So you can't you can unplug the pvc from a different name space into a.

A