Red Hat OpenShift Red Hat OpenShift Virtualization | Red Hat OpenShift, 2 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Demo: OpenShift Virtualization and PCI Passthrough

Description

This video shows how to pass the GPU device of a worker node to a virtual machine in Red Hat OpenShift Virtualization. After the GPU is seen by the virtual machine, the driver is installed and tests are run to demonstrate the functionality works.

Learn more at: https://cloud.redhat.com/learn/topics/virtualization/

A

Hello, my name is john herr and I am a senior software engineer at red hat in this video,. I will be demonstrating the pci passthrough functionality of openshift virtualization.

A

I will do this by passing a physical nvidia gpu in one of my worker nodes of my openshift cluster to a virtual machine running inside the cluster.. Once this is done, the virtual.

A

Machine will have direct access to the underlying hardware.

A

after the gpu is presented to the virtual machine,. We will test that the gpu is being seen at the hardware level and that is working as expected.

A

Please note that portions of this video will be edited for time.. These edits are done to shorten commands and processes that take a long time to run..

A

I will include links to documentation,, software, and other useful urls at the end of this demonstration.

A

As you can see,, the openshift virtualization operator has already been installed in the cluster, and a hyperconverged operator has been created.. I have already created a red hat enterprise, linux, 8 virtual machine.. We will pass the compute nodes gpu device to this virtual machine.

A

The pci vendor and device ids are needed to configure the gpu adapter for pci passthrough.. We will determine the vendor and device id by viewing the pci information on the compute node.

A

We can see this device is an nvidia a100 adapter and it has a pci vendor id of 10de and a device id of 20f1.

A

make note of these ids as they will be needed. Later.

A

The device is using the nouveau driver., we need to have a device bind to the vfio-pci driver, as it is better suited for pci passthrough.

A

The configuration of the compute nodes is done using a machineconfig..

A

We will use the butane utility to generate a machineconfig yaml file. We can apply to the cluster.. This machineconfig will enable iommu on the compute nodes and also bind the gpu device to the vfio-pci. Driver..

A

Here is the butane configuration file.

A

Variant tells butane to generate a file suitable to be consumed by openshift..

A

Name will be the name of the machineconfig as seen in openshift., and the openshift.kernel_arguments section enables iommu on boot.

A

The storage section creates two files..

A

The first file specifies the options used when the vfio-pci driver module is loaded..

A

The options line specifies the pci.

A

Vendor device ids to bind to..

A

This is the vendor device id of the gpu device. As noted earlier.

A

The second file tells the system to load the vfio-pci module upon reboot.

A

When we run the butane utility,, a yaml file is created..

A

This machineconfig yaml file can be applied to the cluster.

A

When we apply the machineconfig yaml to the cluster,, the compute nodes will reboot.

A

After the computers reboot,, we can verify.

A

Iommu is enabled on the nodes by logging into them and viewing the kernel boot parameters.

A

We can see that iommu is enabled on the compute node.

A

When viewing the pci information again,, we can see that the gpu adapter is now using the vfio-pci driver..

A

Now that the compute nodes are configured,, we need to configure the hyperconverged deployment so that it will pass the gpus device to the virtual machines.

A

Let's edit, the hyperconverged deployments, yaml file to add a section that tells it to pass the gpu device to the virtual machines.

A

We will set the pcideviceselector to the pci vendor device id of the gpu device..

A

We also need to specify a resourcename..

A

This is an arbitrary name,, but it is useful to make it something. Meaningful.

A

Checking the virtual machine, we can see from the output of lspci, that it does not currently see the gpu device.

A

Llet's stop the virtual machine, so we can more easily edit its yaml configuration.

A

We will edit the virtual machines yaml configuration to add a section that specifies the gpu as a passthrough device.

A

We need to add a hostdevices section to the yaml file.

A

We will specify dvicename as the arbitrary resourcename we defined in the hyperconverged yaml file.

A

We will again use an arbitrary string for the name entry.. This is not used elsewhere..

A

We will save the configuration and start the virtual machine.

A

Let's check the virtual machine again.

A

We can see that the virtual machine now has a gpu device.. This is the physical device on the compute node.

A

however,. The virtual machine is using the nouveau driver and this driver will not allow the virtual machine to utilize all the features of the gpu device.

A

to utilize all the features,. We must use the nvidia driver..

A

We will install the cuda toolkit from nvidia as it contains the drivers we need and some utilities and tests we can use.. The toolkit needs the gcc compiler, kernel headers, and a few other packages.

A

The cuda documentation also mentions the need for some other repositories to be enabled.

A

And also some packages from the epel repositories.

A

Now we can install the toolkit.

A

I already have url for the toolkits packages because I visited nvidias website earlier.

A

And downloaded the packages for my operating system and architecture. when installing in your environment,, you should also visit nvidias website, since it provides instructions for specific versions of cuda on various operating systems and architectures..

A

The instructions provided for my installation are three steps. first, clean the package managers, cache. second, install the latest dkms.

A

Version of the nvidia driver.

A

Finally, install the cudar toolkit.

A

After the toolkit is installed,, the directory search path and library path must be updated to include the directory containing the toolkits, executablea and libraries..

A

We will set these in the system profile directory for bash, so they will be set upon login.

A

We want to make sure that the device does not use the nouveau driver.

A

so,. We will create a file that will prevent the driver from loading.

A

Rebuilding the initial ramdisk will prevent the driver from being loaded upon boot.

A

Let's restart the virtual machine, so the changes will take effect.

A

After we log back into the virtual machine,, we can see that is now using the nvidia driver.

A

We can also run the nvidia-smi command to make sure the gpu can be seen by the toolkit.

A

Now it's time to see how the gpu compares to the cpu when running a heavy workload.

A

wwe are going to run an nbody simulation that is included with the toolkit..

A

This type of simulation calculates the gravitational pull between bodies or planets..

A

We will first run the simulation on the cpu, then again on the gpu and compare the results.

A

The -benchmark option displays the calculation times without.

A

Actually, rendering anything on the display., the -numbodies option specifies how many bodies to run a calculation against.

A

and the -cpu option tells the program to run the calculations on the cpu and not the gpu.. The grep at the end of the command, just cleans up the output a little to make it easier to compare.

A

running the calculation on the cpu took 6250 milliseconds,.

A

Or just over 6 seconds.. During this time it ran .168 billion interactions per second, or.

A

About 168 million interactions.

A

Now, let's run the same command on the gpu., all we need to do is remove the -cpu option.

A

Wow, running on the cpu is so much faster.. It only took 10 milliseconds to run the same calculation.

A

And it was able to do over 98 billion interactions per second.

A

This makes the gpu about 600 times faster than the cpu. For just this one calculation.

A

As you can, see, passing a pci device through to a virtual machine can easily be done. during this demonstration,.

A

We passed a gpu adapter through to a virtual machine and utilized it directly to take advantage of it's performance.

A

Thank you for viewing this demonstration and I hope it helps you in your future. Endeavorsd with openshift virtualization.