Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automating GPU Infrastructure for Kubernetes - Lucas Servén Marín, CoreOS

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Automating GPU Infrastructure for Kubernetes - Lucas Servén Marín, CoreOS (Intermediate Skill Level)

Kubernetes has seen broad interest from the machine learning community and many users are bringing GPUs to their clusters. However, compiling, installing, and updating the NVIDIA kernel modules needed to run workloads on those GPUs continues to be a cumbersome and largely manual process. Furthermore, distributions like Container Linux, which update frequently can require new kernel modules every other week. In this presentation, Lucas Servén explains how to automate all of these operations for Kubernetes deployed on Container Linux and describes his experience running GPU Kubernetes clusters on both AWS and bare metal.

About Lucas
I am a Spanish software engineer currently working for CoreOS in Berlin. By trade I am a electrical engineer, with a Masters in robotics from UC Berkeley. After finishing my degree, I began at CoreOS, where I have been working on Tectonic and Kubernetes for the last year and a half and maintain the Kubernetes incubator project Spartakus. Kubecon EU would be my first time speaking in outside of an academic setting. Outside of work, I am passionate about deep learning and work on several machine learning projects with my twin brother. We are developing deep learning projects for portfolio optimization and AR. I love open source and recently have been filling my free time with: a Kubernetes operator for Jupyter notebooks, a Terraform provider for Vultr, and an application server for the YOLO image detection neural network. Find me on GitHub at github.com/squat
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, hi everybody, my name is Luka Sarah Marian I am a senior software engineer at Red Hat and we're gonna be talking about automating GPU infrastructure for kubernetes for container linux. This sounds like really specific to container linux, but I think that you'll find that it actually generalizes well to Linux in general and we're going to be doing a little bit deeper dive into somewhat like the nitty gritty of getting kernel, modules and stuff set up for your nodes and I think that this will complement well.

A

Some of the other talks we had in koukin so far related. The GPU is like I know. It's talked yesterday and we're here to talk just a half an hour ago.

A

The goal really is to take you from thinking that you know everything about compiling GPUs GPU drivers to doubting everything you know and then to once again believing that you kind of know things as well and if the demo gods, like it we'll have a demo as well, where will bring up a kubernetes cluster accelerate with get people with GPUs and and all that to take under ten minutes.

A

So, let's see what happens first and foremost, the astute observer will see that all of the slides are branded with a red hat, but the shed said that I work at core OS. This is not a mistake. Korra was recently acquired and we very happily join everyone at Red Hat. If you have any questions for Cora, West's related things or red hat related things, please grab me or any other Red Hat person and I'd be very happy to talk to you. So a little bit about me, like I, said my name is Lucas.

A

I live in Berlin I am an electrical engineer: I studied robotics and controls and I work for the last year and a half on the tectonic installer at core OS and now I'm working on bringing everything that we learned from tectonic onto openshift. In my spare time, though, outside of Red Hat and outside of kubernetes I like to work on machine learning, projects and controls related things, I work a lot with this other guy. His name is Daniel he's my twin brother.

A

He also has a funny looking github avatar and we're always working on machine learning projects they got. So one of the issues that we see when we're working on projects for the first time is that since we live so far apart, sometimes collaborating can be difficult, for example, we're sharing our code, maybe on a jupiter notebook or something, but we also want to share our hardware so that we can both train our models in an accelerated way.

A

It makes no sense if I have the hardware and Danny has to wait for me to be up, so you can send me code and I can train something that just doesn't work, but luckily for us we are at coop Connie you and if we just put our work load onto kubernetes, it solves everything and we can go home early right. What we will see is that maybe it's a little bit trickier than this right here.

A

We have the IKEA instructions for how to build a kubernetes cluster powered by GPUs I wish it was this simple you just plug these things in together, but maybe it's actually not so say that you have. You know your CUDA powered work load be at tensorflow or Jupiter notebook or something and you package it up in a container with all of its dependencies. You use kubernetes to manage container its life cycle, expose the service to the internet so that I can log in on a website and be like Jupiter notebook around these commands.

A

I accelerate everything using Nvidia and then personally, I run all of my kubernetes clusters on top of core OS, just because I like container Linux, because it gives me like low maintenance, overhead everything updates automatically, and this is all great. But as we're going to see, some of these components are not very obvious. We know how to plug in a workload into kubernetes we're all at coop con. We've done this before.

A

Just last talk: half an hour ago, ro he'd explained in very great detail how you can plug kubernetes into into NVIDIA GPUs and how to use a device plug-in to accomplish this. But the one thing that's still a little bit questionable for a lot of people out there and is a little bit fragment in the community. Is how do you get your GPU drivers and how do you get NVIDIA GPUs working on Linux in the first place so really quickly? We have a pop quiz. I guess you guys were not expecting this.

A

It should be very easy if you guys were just in the last talk row. He actually covered each of the four points that I'm looking for. Does that anybody know what are the four things that I need to get my GPU working on Linux machine? Can you just call them out? I need drivers. This is actually two of the things. So this could be the kernel modules and the libraries. What else I need I need GPUs. This is the obvious one.

A

A lot of people miss and what's the last thing, I need well I need my CUDA libraries on my workload right if I want to use a CUDA accelerated workload, I need to have CUDA libraries as well. Okay. So here we have an example of what our anatomy of a CUDA workload might look like. So I have my application at the very top. This container comes packaged in with my cuda live libraries as well. My you know: user plays my user level.

A

Libraries I need my Nvidia device files that get created, um and you know these are created once I've loaded, my kernel modules, I, create the device nodes and obviously, at the very bottom I need, might be GPUs as well.

A

So using the knowledge that we just had in the quiz, we can fill in our IKEA diagram a little bit more and we can see that the GPU drivers, sorry, the the CUDA libraries come for free, thankfully, that all the folks at Nvidia are providing docker base images for CUDA that packaged the CUDA libraries for us. So we can build our containers. On top of that, that's very great. That comes for free, assuming somebody in the audience buys me a GPU.

A

We have a GPU, so we can check that off the two things that we don't really know that are kind of still question marks are: where do the Nvidia libraries come from and then also where do the kernel modules come from, and this is actually kind of a trickier question.

A

It turns out that the Linux community is a little bit fragmented here right, like, for example, if I am on Ubuntu I can sudo apt-get install my Nvidia drivers or something if I'm on fedora I can do the same thing. But what happens if I'm on container Linux, where I don't have a package manager right and then even if I did have a package manager? What happens with my OS upgrades and my kernel version changes, and now my kernel modules don't work. What happens then right do I, just have downtime on my server.

A

Do I have to go in and manually fix this again. What do I do so, even though we don't have a package manager, let's not be deterred. Let's get our kernel modules built for container Linux, so the first thing we do all do when we want to do this as we go to the docks right and we go to the dogs to go to the documentation website and we say all right which packages already exist: scent OS, fedora, I, don't see container Linux on here, so I'm, sorry, I wasted everybody's time.

A

There will be no GPUs on container Linux. You can all go home. This is it, but somebody says Lucas don't give up. After all, why don't you just download the nvidia installer and compile the kernel modules yourself? You guys are very smart I'm, not surprised. We are at coop Connie you one of the greatest gatherings of minds in all of southern Copenhagen, so yeah. Let's do exactly that. I'll, take your advice. Let's download the kernel, the nvidia installer sources and compile it ourselves right. What could go wrong.

A

Let's see all right so I download, I curl it down all I need now is to run GCC on my container links machine except it turns out. The GCC is not installed on this minimal OS who would have thunk it okay? What do I do now? This is obviously not gonna work right. This is just a no-go from the beginning. Okay, your guys's next intuition, obviously is. This is a container os? Why don't? We just run the build in a container if I, googled, docker GCC on Google I get like 50 hits right.

A

Okay, so great I'll docker run GCC, download the Nvidia installer and then I follow the doc some more and it says. Okay, now you just have to provide the path to the kernel sources, all right, but where the kernel sources on container Linux right like this, is a minimal OS again this doesn't come packaged in the US. Where does it come from? We shouldn't have been surprised. Of course, we were gonna need the kernel sources, we're at the end of the day we're compiling kernel modules.

A

It makes sense that we have to link against the kernel sources now. The problem is actually even a little bit trickier than this. If you notice we're running a completely arbitrary version of GCC right. Ideally, we should be running the same tool chain that was used to compile the kernel to build the kernel modules as well, and this could provide I mean this could produce completely unexpected results. We don't know this is gonna work or not. Even if we had the kernel sources so I mean this is also just no-go.

A

You guys had very good intuition, but we got nowhere your advice. It was very, very well-intentioned, but we're no closer to getting GPUs on container Linux. So far, the avatars, my github avatar, is getting very impatient and I still don't have my GPUs right. So now, what do I do? Okay, well, I Google, core OS container, Linux, kernel modules. The first result is some docs explaining how to do this on container like so why didn't we do this?

A

In the beginning, it turns out that there's docs that document exactly how to compile these kernel modules so really getting kernel modules working on container Linux should be as easy, and it's just running these steps. In fact, if the steps are very well-documented automating, the compilation of these kernel modules should be as easy as just scripting. These steps right so now I can automatically compile my kernel modules and my Nvidia libraries for containing Linux just with one script.

A

If I look out in the wild there's, actually a few projects that already do this I went to koukin North, America 2016 and one of the speakers, Rudy Kyoto, I, think I said his name right uh from clarify. He gave a speech titled state of the GP Union and the very last slide made a very brief mention to this. One project called aptly named core OS Nvidia, and it does exactly that it just automated the instructions that we saw in the core, OS documentation and it um if I, say chorus, Nvidia build.

A

You know my version of the of Nvidia it'll produce the kernel modules and also libraries. So great is that great. Does that solve all of our problems?

A

It turns out not really ok, so now, I have a way to automatically produce my kernel modules, but how do I Adam ate this script right next week when container Linux releases a new version and I'm on vacation who's, gonna login to my machine and rerun this script and say: hey build a new version for the new kernel or you know what happens if I want to change to a new Nvidia version? Then what happens? How do I tomatoes?

A

Secondly, how can I make this work in a way that's native to kubernetes right, like ideally I, don't want to have to like SSH into my machines or I. Don't want to have to create a cron job to do this or anything like that right I would want to just have a pod something native in kubernetes. That does this automatically for me, so the obvious answer is Lucas. Why don't you just shove it all in a container- and this is the right intuition, putting this build script into a container? It's definitely the right idea.

A

The nuances are a little bit trickier because container Linux the the instructions provided for building kernel modules. They tell you to download a developer container that contains these include the kernel sources, as well as the tool chain used to build the OS. So the nuances of running this container inside of another container is a little bit tricky, but this is definitely the right intuition.

A

In fact, if I run this inside of a container, I can run my kernel compiler sorry, my kernel module compiler in a daemon set, that's scheduled onto all of my GPU nodes if I schedule it until every single GPU, node I no longer have to manually SSH onto my machines and do anything right every single time that my machine updates and reboots, my kernel modules will get automatically recompiled for me. So this saves me a lot of headache. In fact, this is exactly what is done in the wild by the Google platform, Google cloud platform.

A

Folks so wrote he was just speaking about this. They created a container. That is exactly that for cost. The container optimized OS, like other, like container Linux. It is very minimal, but this container is a script. The Downloads, the cost kernel sources compiled the kernel modules and just dump them in a host volume that later pods that need to use CUDA or whatever else can just consume those volumes.

A

We can skip this and come back if we have time so, at the end of the day, if I have a daemon set, this runs this container compiling kernel modules for kubernetes for any of my nodes, it's really as simple as just saying: coop, CL, coop CTL apply this daemon set, and that is like the most kubernetes native way. I could think of doing this really quickly. I'm going to terraform apply something so that by the end of this thing we have a cluster we can play with.

A

Okay, very cool, okay, so this is it um next? What let's see? Okay great? So now we know how to compile our kernel module automatically right. This is great. This gets the long way of the way there, but how do I consume these kernel modules and the libraries once they're on my machine? It turns out that getting these getting these kernel modules and getting the libraries working with any sample workload is actually very straightforward. Here.

A

I have a toy workload that is an nvidia cuda container and I'm just going to run the nvidia SMI utility, which is the system management interface.

A

It just prints out statistics about, like you, know, memory consumption temperature, whatever about the attached GPUs on your machine and the only thing I really need to do is say, tell my container runtime to mount in my device nodes and bind mount my libraries and then that's all I really need one of the issues that we see with this, though, is that I'm leaning very heavily on my container runtime to do this for me, in fact, I'm leaning so heavily on it.

A

That I need to tell it exactly where my libraries are on my hosts and which devices I want to use, and if we try to extrapolate this to kubernetes- and we ask how can we automate this exact same task for kubernetes? It means that we would have to enter pots back, specify I want to use a volume mount with these exact host paths, for my my shared libraries I need to and then my my application needs to know which device nodes to use- and this obviously is not gonna scale. Well right.

A

It means that I have the very tightly couple, the representation of my hosts to my pod spec and now suddenly, if I want to share my pods back across nodes, that I have my libraries installed differently or across different clusters that it's not gonna work anymore. I'm gonna have to modify my pod specs. Oh very good. Let's see.

A

So this is just doing some GPG stuff for our cluster very good. You guys should have told me. Okay, it turns out that things all the efforts that came into the device plug-in system in kubernetes 110, the man again.

A

Let's see, okay, thanks to all the work that went to the device plugin system, all of the work of having to buy mount in the libraries and stuff comes for free.

A

Now, the the device plugins developed for by both Google and also by Nvidia take care of mounting in the libraries for us, and it means that we no longer have to specify in our paw spec explicitly hey I want to use these in video libraries located here on the host and my application doesn't have to know which doesn't have to know which device nodes to use either. In fact, the device plugin for Google, for example, is extremely simple.

A

It's only a couple hundred lines of go and the code that does the mounting of the libraries is also similarly very simple. So like it's only a couple lines, but it means this is the difference between having very tightly coupled pod specs to our infrastructure and having very loosely coupled things where we can just say in a pod. Spec hey require one GPU and that's it okay.

A

So where are we now? After all of this, we know that we can use a GPU installer to generate our Nvidia libraries, get our kernel modules, install the kernel modules, create device nodes. So that would be what everything here right. This is everything that's done by the GPU installer. This is great, so it'll create device nodes that are that have that are have to do with the GPUs that are attached to the host, and then the device plug-in takes care of these other areas.

A

It's actually what's specifying which volumes I want to mount in to my containers, so that my applications at the very top via tensor flow or Jupiter or whatever else don't have to know anything about the on the host. That's awesome right, okay, so now what? If we go back to our Kia, our IKEA instructions? One last time we see that we now have checkboxes on everything we know where our kudo libraries are. We have a GPU that you guys got me.

A

We have our Nvidia libraries compiled automatically and our kernel modules as well, so this is going very good. What else do we need I guess? This is essentially all of the theory around um automating the GPU infrastructure. Now we kind of want to take a look at a real-life situation. So without further ado, let's do a demo I guess: what's gonna happen in this demo? Okay, so in this demo, if you want you can follow along it's on github.com, slash squat.

A

This is my username kook on you, 2018 we're gonna be building a cluster using typhoon. So this is a very minimal self hosted kubernetes cluster we're going to be using gke. It can honestly work on anything.

A

I have a GPU, installer Damon said that I wrote for container Linux. We can use this just last week. I found another one, so there's actually several implementations of these things. Like you know, this is one of the things where the community is getting very fragmented, but we have a GPU installer. They were going to use to compile the kernel modules. We have the device plugin that is in upstream kubernetes.

A

So as of one month ago, there is a device plug-in in the kubernetes repository and the cluster add-ons and then we're gonna run a neural network as our sample workload, and it's going to do some object detection for us. So let's see didn't.

A

Very cool, very cool.

A

Okay, so what do we have here? The if you guys see the terraform here didn't finish running so I'm gonna use my other cluster that I just made this morning that is already built and then we'll apply. All the kubernetes manifests and everything that we need to it. Okay, so let's see if the demo gods are kind to us, okay, very cool, so we already created our cluster now we'll all these think we want to do is: let's get the nodes. Let's see how we're doing here.

A

Okay, great so I have a two node cluster I have one controller and then one worker this worker is has a GPU attached to it. So let's take a look at what manifests we have here. We have our darknet, you know our genetic detection deployment. We have our device plug-in daemon set, and then we have our GPU ins driver installer Damon set as well. So let's coop CTL apply.

A

Let's see what happens, yeah very cool that was extremely fast, then, okay, now what we want to do so we did the manifest. Let's look at the driver, installer, let's see how this thing is progressing. So again, this driver installer is running as a daemon set on all the GPU nodes. In this case, just one it's gonna, compile the kernel modules, put them on the host disk. It's going to modprobe, so it's gonna insert the kernel modules. It's gonna create device nodes that later we can use using a device plugin. Okay.

A

So let's take a look yeah, okay, so it actually finished compiling already these errors at the bottom are noise, but it this should have worked just fine. So, let's see we can check the logs of the device plugin to see if indeed the device plugins saw that this is all registered okay. This is apparently device plug-in on a different node.

A

Let's see on the other night, what happened? Okay, very cool, so the device plugin here is registering a device and video 0 it's in healthy state, healthy state. So this is exactly what we were hoping for.

A

Let's take a look at that node, let's, if we describe the node, we should see that it actually has GPUs on it.

A

Okay, let's see we have one NVIDIA GPU and the capacity here yeah perfect, so we have the GPUs ideal. What else do we see? Ok, let's take a really quick look at what the manifest for the workload looks like.

A

So we have some standard deployed on boiler plate. We have the command, that's just running the the neural network and the interesting stuff is here. We have we're specifying one GPU and we're also tolerating the GPU taint, very cool.

A

Ok, so the last thing we want to see is: let's take a look see if this container is already ready. Sometimes these things take a long time to download, because the the CUDA base images are pretty big, sometimes in depending on the network connection. Ok, this already created the container perfect and let's see the last thing.

A

So, let's check the logs of this ok great, so this neural network is running and we're gonna run. This is just a sample client app that what this is going to be doing is it's gonna run. Some OpenCV is just going to be taking photos of me and submitting it to the to the cluster for some object detection. So this is me: it's just labeling me and seeing that I'm here, and this is accelerated by CUDA. If I didn't have my kernel modules here and I didn't have a GPU.

A

This would be taking like 10 seconds per image, but there's actually taking very little time and most of the lag is actually just latency and uploading the and uploading the images to the server. So this is all working and what's the result in here, so this is our demo. This concludes the interactive portion.

A

How did I make this fullscreen again? Well, okay! So what happened here? We saw that very quickly. If we bring up a kubernetes cluster, we can apply to manifest, have our GPUs and our GPU drivers installed and we can get running with a workload super super fast.

A

What is missing here, one of the things that we alluded to earlier that are missing, is that the GPU ecosystem is still very fragmented, so there is a different, can there's a different GPU installer for every operating system or really that's like the way things are going in the world right now and in fact there may be multiple implementations for every operating system, and this is also not ideal. Instead of us pooling our efforts, it would be great if we had some generic utilities that we could use for more of our more of our infrastructure.

A

What other problems do we have? There's also the issue that whenever I, whenever I update my operating system, I still don't have zero downtime deployments. So if I do an operating system update, my my pod will get my GPU installer part will get rescheduled on to my node. It has to recompile the kernel modules. While this is compiling, so maybe five or ten minutes, my workloads will be unschedulable, they'll just be pending, so I still have downtime and then the last issue is that what about sharing my kernel modules, if I have ten GPUs?

A

Sorry, if I have ten nodes that have GPUs every single one is going to compile my kernel modules. It would be great if I could just compile them once and distribute them everywhere.

A

These are all things that we still have to do, and these are some open questions, but with that I leave you guys. Thank you very much. If you have any questions, please ask me, and otherwise we can just chat afterwards. Yeah.