Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Everything you Need to Know about using GPUs with Kubernetes - Rohit Agarwal, Google

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Everything you Need to Know about using GPUs with Kubernetes - Rohit Agarwal, Google (Intermediate Skill Level)

This talk will start by describing the need for making Kubernetes aware of resources like GPUs. Then it will briefly go into the history of GPU support in Kubernetes and the various backwards incompatible changes it has gone through. It will then describe the current state of GPU support in Kubernetes and the best practices for both Kubernetes users (app developers requiring GPUs) and Kubernetes cluster operators (that are managing machines containing GPUs). Specifically, it will touch on managing GPU drivers, device-plugins, advanced GPU scheduling and the newly added GPU monitoring support. Then it will describe what we learnt from running hundreds of GPU clusters in GKE. Finally it will briefly mention what's coming next in Kubernetes with regards to GPUs and other hardware accelerators and devices.

About Rohit
Rohit Agarwal works on Kubernetes at Google. His current focus area is hardware accelerator support in Kubernetes. Prior to Google, Rohit worked on data infrastructure at Cloudflare and Qubole. He holds a B. Tech. in Computer Science and Engineering from IIT Ropar.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Let's chat, hi everyone. My name is Robert Agarwal I'm, a.

B

Software engineer at Google I work in the Cuba native Engine team, which is part of Google cloud, and today I'm, going to talk about using GPUs with kubernetes before we begin a quick show of hands how many of you are already using GPUs on Cuban IDs awesome and how many of you are using GPUs that are not NVIDIA GPUs.

B

A

May be a one single person: okay, yeah, okay,.

B

So in this talk, I'm not going to talk about why to use GPUs or when to use GPUs I'm only going to focus on how to use GPUs and communities I'll start by talking about what makes it hard to use. Gpus from containers I'll go into a bit of history, about the GPU support in communities and then I'll talk about what you need to do as a user and as an operator to get cheap use, working on communities and finally, I'll end with describing what's missing and where the support is going. Next.

B

A

B

Containers and GPUs other things which make containers great and haven't made them so popular is that you can package your application and all of its dependencies in a container image, and then you don't have to worry about missing dependencies or conflicts with other applications. You can run that container image anywhere.

B

However, this property breaks down when one of your application dependencies is a kernel. Module containers use the host kernel. So if one of your application dependencies is a kernel module, then the host on which your application container is supposed to run needs to have that kernel. Module installed and applications using nvidia gpus are examples of such applications. Using nvidia gpus require the nvidia kernel module to be installed on the host using nvidia gpus also require the press of Nvidia's shared user level, libraries to be accessible from inside the container.

B

These libraries are used to communicate with the kernel module and therefore the GP devices. So you need the libraries you need the kernel module on the host and you need the libraries to be accessible from with within inside the container container. So maybe you can package the libraries in the container image and assume that the kernel module will be present on the host accept the version of the Nvidia. Shared libraries needs to be the same as the version of the Nvidia kernel module.

B

So if you package these libraries in your container image, then your container image is no longer portable because it depends on the version of the kernel module on the host. So if you package it for a particular version, it will only run on hosts which have the kernel module of that version right. So this is not good. You don't want to make your container images non portable because that defeats one of the main advantages of containers so in communities. The first thing we decided to do to support GPUs was very simple.

B

We decided to let the user deal with with these dependencies. All communities would do is that it would assume that the Nvidia driver is already present on the host, and then it would see the GP devices present and host and expose them as scheduled level. Alpha dot communities retire, slash, NVIDIA, GPU resources and when a container asks for this resource in its resource requirement, then communities would add these devices to the container. So this is all it would do.

B

It would keep track of the GP devices on the host and attach them to the container when requested. So now you have devices attached to the container. How do you access the device from inside the container so either side before to access the device? You need those user level, libraries, but also, if have those user level libraries than you container image is not portable, so we've recommended the users to install both the kernel module and the shared libraries on the host and then use host path.

B

Volumes to inject these libraries inside the container so install with them and the hosts create a hotspot, valium inject that volume into the container. So this is how it looked like in blue. You see that this container is this. Part of this container is requesting two GPUs, and this is the same place where you'd add CPU and memory at the top right and red color. You see that we are creating a host path, volume called Nvidia libraries and the path.

B

There is the path and the host where these shared libraries are present and at the bottom in red. You see that we are adding this host path. Volume called Nvidia libraries inside the container at the mount path, and this path in the container needs to be a path which is in the shared library search path. So after you do this, your container image will be portable, but and things will work, but this this is still terrible because your, u container images portable, but your pod spec is no longer portable.

B

Your pod spec contains some for specific stuff and we don't want the pod spec to be not portable, so there were some other things which were undesirable about this approach. So this this support for NVIDIA GPUs was in tree. It was part of Cuban Indies, core code base and that's not ideal, because what happens when other vendors want support for their devices in communities right. We cannot continue expanding the community's codebase to support all these things right. We cannot add an MD GPUs, Intel, GPU, Xilinx, FPGA and so on.

B

We at the same time, we also cannot play favorites to a particular vendor, because communities are the open source software and like supposed to work well with everything. As a result of these things, we deprecated the entry support in 1.10 and in 1.11. We are completely removing the entry support, so this resource will stop working in 1.11 completely to replace this entry support. We added device plugins in humanity's device. Plugins are aware to support generic devices in communities.

B

It's an extension mechanism which allows the vendor specific code to remain outside the core community street, but still allows these devices to be available natively through the communities API, and they also enable the pot spec to be portable. The pots, if you use device plugins the pod spec, will not contain any host specific parts which we had in the previous part spec.

B

So this is how it looks like now in in blue. You see nvidia GPU. So all you need to do now is add the resource exposed by the device plugin in your resource requirements for nvidia gpus. That's Nvidia dot-com, / GPU, and once you do that everything will work for you. You'll notice that there's no specific part- and this in this pot spec right, but now like there is the magic like there's no hood specific part in the pot spec.

B

How is the container getting access to the shared libraries present on the host and the magic is in the device? Plugin api's device, plug-in API is allows the device plug-in to set environment variables inside the container or mount volumes inside the container. So, for example, the cluster administrator could configure the device plugin to point to the part on the host where the shared libraries are present and when the device plugins sees a pod requesting GPUs, it will add that volume to to the container so that the container can access this right.

B

So with this, you have your container image, which is portable, because you don't have user level shared libraries in the content in the container image, and your port spec is also portable, because you.

B

And your pod spec is also portable because you don't have any host specific stuff in the Pacific. All the host specific stuff is part of the device plugin, which is which is configured by the cluster administrator who know about the host, so device begins, were introduced as an alpha feature in 1.8. They went beta in 1.10 and if you using GPUs the Kuban it is, you should start using device plugins.

B

So just to recap, what do you need to do as a user at the user? You should build your images. Without the user level shared Nvidia libraries like Lib, Nvidia and l, dot, s o and so on. They should not be part of your container image. However, the container image should still contain the cuda toolkit, so the cuda toolkit does have some dependence on the host driver version.

B

So, for example, each cuda version requires a minimum driver version, but this dependence is not as onerous as the one with shared libraries where you have to match with the exact kernel version. So you build your container image with the cuda toolkit, but without the user level shared libraries and then all you do is request. Nvidia calm, slash, GPU in as part of your resource requirements, and if the cluster administrator has set up the cluster correctly, then everything should work for you at the user.

B

This may be one more thing you may have to do at the user. So let's say your cluster has multiple types of GPU nodes. So, for example, some GPU nodes have Tesla P 100 and some GPU nodes have Tesla V 100. So there is a and you want your application, a particular type of GPU. So you you want to say that I want my application to run on nodes which have V 100.

B

So this, unfortunately, no native way of doing this targeting in Cuban it is communities, does not natively understand different types of GPUs. It only understands that there is you're requesting a GPU, doesn't understand that you're requesting of V 100, GPS, etc, and the workaround. What you can do is you can ask your cluster administrator to label GP nodes with the type of GPUs that a present on the net node. So you can ask your operator to label the P 100 nodes, so P 100, V, owner or even read, and so on.

B

So this is, for example, what we do in GK, where you can say that I want this part to run on a node which has Tesla keys, and then this part will schedule on a node which has just like eighties. Okay, so we learned about what we need to do as a user. What do you need to do as a cluster operator to create a functioning GPU cluster?

B

So, first of all, you need to have notes for GPUs. These things are very expensive and, as I said before, if you have multiple types of GPU nodes you you should label them so that your users can target particular types of GPUs.

B

Then- and this is the one of the most important things need to install the Nvidia driver and I can talk about just this point alone for a long time. Unfortunately, but I will just keep it short here so few things which we need to keep in mind about this parts of the Nvidia driver are closed. Source and Linux is GPL license. So keep that in mind, and the other thing you would want to do is you'd want to keep up with the driver version required by the latest cuda release.

B

So, as I said before, CUDA toolkit has some dependence on the driver version. So you'd want that if CUDA releases a new version that you update to a driver which is required by that version and if you like, because what would happen eventually, is that your users, who start using newer CUDA versions and if you have not updated the drivers on your nodes, those containers will stop running because good a 9.2 requires I, don't know, 395 got something driver which your nodes to under. In GK, we use a daemon set to install the drivers.

B

You could you can look at that and like folk that and do your own thing.

B

You'll also need to install the device plugins. There is a device plug-in for NVIDIA GPUs available from Nvidia there's one available from Google. You can choose whichever one fits your needs. I am hopeful that in the future we will converge them so that you get the best of both worlds. But right now there are two different device plugins and you can use whichever one you want.

B

Okay, so now you have a cluster with GPUs running what else you may want to do? Other things you may want to do. Is you want to set a resource quotas per namespace? You want to say that my marketing namespace or my production namespace, can only use so many GPUs.

B

So we added support for doing that in version one point 10- and this is how it looks like in purple I think you have what this is saying, that whenever this object is created in a particular namespace, the total number of GPUs that can be used by different parts or single part in that namespace is four. So you can. You can set limits like that per namespace using resource coders.

B

We also have so one of the other things which you users, move on to you after you have a GPU lesser is for GPU metrics for their workloads. So communities already supports a couple of important metrics, which users care about for their workloads. So these are memory, usage and duty GPU duty cycles.

B

They are collected by a C advisor which monitoring agent, which is built into the cubelet, and they collected using env ml, which is a closed source nvidia as management library, which can be used to collect GP metrics, and these metrics are can be accessed using SI, Advisors, Prometheus, endpoint or hipster, or Google stack driver, and they were added in 1.9, so you can already use them.

B

This is one other very interesting thing which a lot of you should do if you are not doing it already, is that you you want to because GPUs are expensive, you want to create dedicated nodes for GPU workloads and why? Why do you want to do this right like so? You want to prevent parts that are not requesting GPUs from getting scheduled on nodes which have GPS.

B

This helps to make sure that when a pod which is actually requesting GPU arrives, there's some place to schedule it, because if that GPU node is already busy running on GPU workloads, then you will get into a situation where your GPUs are idle and your pods requesting GPUs are pending and you don't want to be in that situation, because they're just wasting money.

B

Another thing this helps with is to aggressively downscale nodes. If you are in a cloud environment. So if you let's say you don't have any GPU workloads at the moment your machine learning scientists are away on vacation or whatever, and if you prevent parts that are not requesting GPUs from scheduling on GP nodes, then those GPU nodes will be empty when there is no GP workload and the cluster autoscaler can scale those nodes down and save you costs.

B

And how do you do this? The way to do this in Kuban, it is, is to add panes to node. So what you do is you would paint the GPU nodes with some paint and then no pod will schedule on them until it has a corresponding Toleration. So you would tall you'd, add those Toleration to the pods requesting GPUs, and so what will happen is now you have notes with GPUs that have pains someone submit support that does not request GPU. It doesn't have that toleration.

B

It doesn't get scheduled on a GPU node someone, some it support with GPUs adds that toleration, so it gets scheduled on that nodes right in 1.9, we added an admission controller called extended resource toleration, so this extended. This controller recommends that you add a particular tain to your GP nodes. So it tells you what the key of that tin should be, and if you do that, what this control, what this admission controller will do, is that it will automatically add a toleration to your GP pots and so like.

B

So what you do is you add the taint suggested by this admission controller to your nodes. This admission controller will automatically add toleration for that taint in parts that are requesting GPUs, and so those pots will get automatically scheduled right. So you would have like your users would not have to worry about that. I need to add toleration to my part spec that will happen automatically for them, so the pot specs will remain. They will not know that there is a dedicated GPU, node pool or something they'll just submit the GP workloads.

B

Only the workloads which request GPUs will schedule on those nodes, and nothing else is getting on those nodes. Okay, so we we talked about what is already present in communities for GPUs. What's missing, so there is no support for GPU in mini cube. A lot of people have expressed interest in this. A lot of people have asked about it, but the development of this is not done yet, and this is a great area where people can contribute, there's also no fine-grained quota control.

B

Now, what do I mean by that I mentioned that you can use resource quotas to cap the number of GPUs that can be used in a particular namespace, but because humanities does not natively understand the types of GPUs it only in the sense whether something is a GP or not.

B

We cannot say that in my names in this namespace I want to only allow 4k, 80, GPUs or 4 P 100 GPS. You can only say that I want to only allow for GPUs right, so there's no fine-grained quota control and there is some design discussion going on on how to support this and, as I said, there's some user level metrics already present in the in communities for GPUs, but more metrics can be added specifically a lot of metrics which are required by cluster administrators, for example. They may want GPU temperature or power usage, etc.

B

They are not present at the moment and we are working like we're designing how to add these metrics without polluting the cumulative score and like having an extensible way of adding these metrics there's, also no support for GPU sharing. So when a GPU is attached to a part or a container, it's exclusively used by that container, no other container can use that GPU. At the same time, also, you cannot request half a GPU like you get the whole GPU or you don't you get nothing.

B

So this no GPU sharing its exclusive usage right now, and we are thinking about how to do this and like there's just thoughts right now. I think Kuban is also not aware of GPU topology. So if, for example, you get scheduled in a way that the topology is wrong, you may not get the performance you want with your workloads. So that's. This is another thing which we are actively thinking about, but there's no concrete design or anything yet.

B

Finally, the auto scaling support in communities for GPUs is not ideal, so there is auto scaling support. You can auto scale based on the number of GPUs and if their pods pending you can, the autoscaler will know these GP pods are pending, so I need to add norsu GPUs, but it's not ideal, and the reason for this is again. The same community is aware of GPUs. It's not aware of GPU types right, so pod is pending. It requires two GPUs. What type of GPU note should I add?

B

Should I add a KT node or should I add a P 100 node and there's there's some hacks in the autoscaler which work with us, but like that's, not an ideal scenario and we are again working towards fixing this situation.

B

Okay, so I have a shameless plug. Now, if you don't want to do all of this, you can run these two commands in the in GK and you will have a GPU cluster working. The first command creates the GPU cluster and the second command installs the driver yeah. You still have to install the driver, any.

A

Questions yes,.

B

So the question is what, if there are multiple types of GPUs on the same note yeah, so we have not heard a lot of use. Cases and I am not sure if there are any plans to immediately support this I'm hopeful that the design we come for, where we make GPUs or we make humanities or device plugins aware of GPU types may handle that your skills, but, like that's not like that's, not a very common use case. People usually have heterogeneous clusters. Heterogeneous nodes are not that common. So.

A

B

The question is: has there been cases where newer versions of CUDA don't work or older versions of CUDA? Don't work on newer versions of driver I have not seen them like, thankfully, CUDA versions have been backward compatible like the drivers have been backward compatible with the old CUDA versions, so that has been very nice.

A

A

B

Yeah, so the question is what about power metrics for workloads yeah, so we are we're working on making sure that we can add more metrics or we can working on making sure that we can add a way to extensively add more metrics to humanity's workloads. But there's that's I, think that will happen.

B

Maybe in 1.3 like we are hopeful for design in 1.12 and then some implementation, 1.3.

A

B

So the question is: is there a recommendation for a base container image to build your containers on? We recommend Nvidia's CUDA images which are available on docker hub and on get good lab, I think yeah, so they.

A

Are they work really well, I? Think back.

B

Yeah, the question is: how does the Nvidia runtime environment fit into all this so for NVIDIA container runtime? You still need the driver installed. I think there is some talks about how to integrate both of them together, but right now you still need the driver installed for NVIDIA container runtime to work and you so so. As I mentioned, there are two device plugins right now. The nvidia has device.

B

Plugin uses nvidia container runtime, underneath to make your make the shared libraries available to the container the google one does not doesn't do that yet, but like yeah, so that's how it feels like it. It's a way of getting these shared libraries available to the container. It does some other things as well, but.

B

Is the question that you, whether you can use GPUs without end in NVIDIA container and time.

A

B

I see so the question is: what is the relationship between the darker diamond nvidia container n time and the driver right? So so the darker demon allows you to have multiple container n times right by default, you use, run C Nvidia container runtime is run C, plus app restart hook, which adds some things which make your shared libraries available and so on. So the doctor demon is like you, communities does not actually require darker anymore right. There is a CRI API which which can talk to cryo, which can talk to multiple container n times.

B

So docker is basically talking to the container in time underneath to start the container and media container. Runtime is just one of those runtimes and it's very close to run C, which is the default runtime in awe in docker and in cryo and in communities, and the driver is totally separate. Driver is basically a collection of the kernel module, the shared libraries and some debug utilities. So you need to install the driver which will install these things.

B

Then video content, container runtime after that will make these shared libraries available in the container and docker is just one way to use the Nvidia container runtime. You can use communities directly and you can use cryo as well. Yeah.

B

B

So the question is: what are the like the estimated time? Then these the things which are missing arrive, it's really hard to say it's like software estimation right, but like so there's some. So the monitoring thing is one of the top priorities of the resource management workgroup, which works on device plugins. The GPU sharing and GPU topology is in very early design phase and then mini cube is like we're. Just looking for someone to add mini Q support like it should so many cube support. Is you can add it on Linux?

B

You can add it on Mac the Linux one should be fairly easy, because mini cube on Linux can run with a mode called driver equal to none, not the Nvidia driver, but the VM driver, and that should be fairly easy, but no one has just gotten around to do it on Mac, I'm, not sure I think be a hard thing on Mac or Windows. Maybe on Windows it'd be easy, and why else was there I forgot, but yeah like this?

B

All of them are yeah, so the other things were making the autoscaler better and making. Finally encoder control. That's also in the design phase, so hopefully, in one point well, you'll start seeing some sorts of like some. It is some design invent well for all these things, and hopefully later you'll have some alphas.

A

A

Yes, so people.

B

Have used device, plugins, I, think to implement F like provide an FPGA device, but I am NOT very familiar with that like there, there are examples out there, and so there is a you should look up resource management workgroup where we talk about these things and like you'll, have you can talk to more people there and they'll have more sites about this.

A

Okay, all right thanks a lot. Everyone.