Cloud Native Computing Foundation Kubernetes AI Day NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Are You Really Out of GPUs? How to Better Understand Your GPU... - Natasha Romm & Raz Rotenberg

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Are You Really Out of GPUs? How to Better Understand Your GPU Utilization - Natasha Romm & Raz Rotenberg, Run:AI

GPUs are in high demand, and we’re always running out of them - there’s just not enough of them in our cluster! But they are also expensive, so we can’t easily buy and add more and more of them to the resource pool. We could learn how to better utilize them, but before we even do that, we need to get a better visualization of our existing GPU utilization. This talk will explore some easy-to-use open-source tools to help you understand the GPU utilization in your Kubernetes cluster, and will help you make better decisions regarding your AI workload assignment on GPUs.

A

All right so hi everybody and my name is.

B

Please raise your hand if you use chicken wings.

A

That's great, okay, now keep your attitude. Please.

A

Wait so I see these two hands came down and you are still alive. So trust me once I always have a feeling that we had enough to do it sooner or later.

A

C

So just so we're on the same page here gpus are highly.

B

C

For deep learning processors, they can dramatically speed up the computer processes. So that's why a lot of people choose to do their deep learning and AI models on GPS. So we want to learn more gpus because we want to do more AI, but gpus are so expensive. We can't just keep buying more and more.

C

So that's why we need to try to bring the most out of our existing reviews and that's a big challenge. So in this talk we're going to talk about GP utilization and how it's often known how we can detect it and we're about to show you some tools that are built that can help in the situation.

C

Thank you, so it's true that when you use.

A

Only one aspect and the problem is how do efficiently manage and provision to your business now I know: GPU. Provisioning is not a term that we use on a daily basis, but the truth is she does that for you and we will see what kubernetes does and also what it does. But first, let's try to understand why GP provision is not an easy task, then we'll do this by an example.

A

Let's say that you are a manager of the team. You'll speak English. Now this is through both the small things of our team members and there are large organizations with hundreds of thousands of different communities, important.

A

To use so each one of the members has.

D

Its own GP, all right, sorry can I hand you a mic. Your um attached one doesn't seem to be working, so people can't hit you online.

A

Thank you so should I go back or just can you hear me now right all right? So let's say we have four team members and we have four gpus. Now each of the members has his own GPU and then comes a new member.

A

Now you as the manager, probably think that you need more gpus and let's go through all the things that are happening while you're out there on the Journey of getting these more gpus that you need. So. First of all, you are sure that you're letting the new member down, because you remember the only last minute that you need these gpus.

A

You start fighting the I.T department to get these more gpus, even though you already have a few. You have to explain your boss why the new project is being delayed and all this time the new member is waiting for her GPU resources. Now trust me this.

A

This is how I look when I wait for my resources, and the truth is that you don't always need more gpus when the new team member comes instead of getting more gpus you just you um I'm sorry, so the truth is that you don't always need more gpus and we will see why so, first of all, not all the tasks require all the resources that a user gets. Some tasks are just smaller now.

A

Secondly, some of the work is being done out of the GPU, leaving the GPU unused, for example, writing code or building the model. Also, some GPU workloads have heavy CPU and I O tasks.

A

In addition to that people take breaks now this could be a short 10 minute, coffee break lunch or even entire days off. During this time, the resources could be unused almost entirely and, generally speaking, not all the resources are being properly used.

A

So instead, when the new team member comes instead of getting more gpus, what you want is just to use this idle GPU resources.

A

Now until now, I I was talking about a team of people sharing gpus, but this is exactly the same case for a kubernetes cluster running different AI workloads, because very similarly to people different AI workloads are not identical. They have different loads and different resource consumption.

A

Now it took us quite some time to fully understand why it's so hard to to fully utilize a GPU, but I think that now we can say that there are two fundamental reasons for it.

A

The first one is that many GPU usage patterns are under utilized by definition, for example, when you do remote debugging with vs code or pycharm, when you write code interactively with a jupyter notebook or just a python shell, when you have idle AI workloads, just waiting for new requests and when you have just not so heavy tasks like small models or small batch sizes,.

A

The second fundamental reason is that gpus are being provisioned, statically and coarse grained. This means that, most of the time, you will see a GPU being provisioned for either a user or an AI workload.

A

In order to overcome these challenges, gpus must be provisioned in a smarter way. They have to be provisioned dynamically, and this means not a GPU per user or a per single AI workload.

A

Additionally, they should be provisioned in a finer grain, for example fractions of gpus, and they also be over provisioned. This means that you should run more workloads than the available gpus, very similarly to other resources like CPU memory and storage devices.

A

Now, unfortunately, for us, kubernetes does not support over provisioning of gpus. It also does not support fractional provisioning in gpus, but luckily for us it does provision our gpus in a dynamic way.

A

Now, if you're not using kubernetes just yet, for example, if you're running on the host machine directly or over SSH, we created the right tool for you. We call it GN and it stands for GPU environment management, it's 100, free and open source.

A

It is highly inspired by existing environment management, software like piant, conda and others, and it works in a very similar way and it helps people to provision their gpus in a smarter way. Now. Gpu gnf, sorry, is a terminal based tool. Just like payan and conda, but we did create integration to Common Ides. We have an extension for vs code.

A

We have an extension for Jupiter lab and we are working on creating a plugin for python as well, so that every GPU user can use their gpus in a smarter way natively in his or her working environment.

A

Gnf and all these extensions are open, source and available on our GitHub page and I recommend you to go check them out and see if it can be useful for your team.

A

Now going back to kubernetes, we said that kubernetes does provision gpus in a dynamic way, but the problem is that it does it in a pod granularity.

A

This means that most of the time, you will see one GPU provisioned for one pod and when this spot gets scheduled, a GPU is assigned to it and as long as the Pod is alive, this GPU cannot be reassigned to other pods, even if the GPU is idle and not used, because any of the reasons that we saw earlier now, this must be changed if we want to fully utilize gpus.

A

Furthermore, even with the smartest provisioning mechanism, the provisioning, smart provisioning is not enough, because GPU resources should not be an afterthought. The point in time in which we deploy an AI workload to production cannot be the first time that we ask ourselves how much how many GPU resources do. We need the GPU resource requirements should be properly defined in the development phase of the project, just like any other project dependency, for example, python packages or a docker-based image and I truly believe that they should be specified in as infrastructure as code.

A

Now, if you use GN for your project, it allows you to save the research requirements as code as part of your git Repository.

A

Okay, so the truth is that there is no magic button for these problems. We have to change our mindset and now, after we have a better understanding of it, I'm going to pass it to Natasha and she will continue with the second part.

C

Thank you, okay, so now that we understand the problem, our responsibilities and the power we have in our hands, but where does it take us? What can we actually do about it? Well, you cannot improve what you don't do so. The first thing we need to do is to get a better View and to better understand that your utilization in our classroom, so we.

E

C

If we have some item gpus see if they have pending workers, so let's drill down and see, are you really confused now going back to the case of AI workloads in a kubernetes club station? We know that different workloads have different lows and.

B

Different resource.

C

Consumption, but how can we actually measure it in reality? You guessed right, it's GP utilization, so this is the official definition of GPU transactions, but the factor just means how much the GPA will be used. So the utilization is low or even zero. It means the GPU is not being fully used.

C

So um in order to detect low utilization in a cluster, we want to detect the organizations. Let's see how we can do this so I'll just say this: if you have a cluster we could use, but you have not installed for brunettes yet and you're, considering it, you can use iron drop with surrounding eyes, open source top like toe or monitoring details and cluster over SSH. We don't have time to look at it today, but it's really cool so you can take over.

C

But for our test is now we do have the brilliant question. So let's say we have two loans for gpus, each eight children. Now, let's get a better look of what's going on on those tubules, so.

B

E

C

B

We're going to go over.

C

Everything here in just a few minutes, but what is the first thing that I want you to notice, so we have eight allocated gpus out of eight total right now and what do we see on the right?

C

The problem we said before we have a problem. Now we can actually see it. We have two pending pods that request the GPU and are not allocated due to lack of resources, but.

B

At the same time, we have.

C

Two gpus that are allocated by costs that are not using them and they're either. So this is a big problem. Now I want us to go a little bit to that and I'm going to I'm going to explain how I built the such position and then we'll go back so to identify an objectives. We need to know the GP utilization of each review at a certain moment.

C

So if the GP utilization is zero and there is a running part on the GPU, it will be identity and probably one of the most expensive gpus you have in the cluster you're paying for it, and you can't even use it.

C

So we need some sort of component to export this database. Luckily, for us we have open source schools that can help us with this, so we have in the videos. This is James Porter. It's a demon set that exports metrics about gpus on the loop. If you are working with gpus and kubernetes, you probably already have it installed as part of nvidia's GPU operators.

C

B

Have videos which ensure you've all.

C

Heard that and it's a monitoring system that collects and stories of metrics, we also have grafana that comes both in with Prometheus and it allows to build the dashboards like the one we said earlier, so they're all easy to install and easy to use. They're.

B

A pretty UI and.

C

Everything I installed them on my kubernetes cluster and I opened the produced now.

B

C

Deep breath we're going to Deep dive into some technology for a few minutes, so this is a screenshot from Prometheus C1, just raw metrics without the font. This is the metric we are interested in. Dcgmfid comes from the dcgm exporter. Each record here stands for a different GPU and the value of digitization is a percentage, so the value will be between 0 and 100., and the labels of the Matrix give us more information about each debut, so we're able to identify them. So, for example, we have the kubernetes node of the GPU.

C

We also have its uuad and GPU index on the on the Node and there's a spot running on the GPU. We'll have the code name and namespace and.

B

If there's no hard running overview.

C

B

C

Be equal now out of all the records here we are interested in the ones that actually had a problem, so we can use chrome2l, which is Prometheus query language to perform some queries only the metrics. So this is how to filter on.

B

The inside results see.

C

If we use a non-empty called label and those records here stand for the gpus with a pod run instrument and also we would like to filter on the ideal allocated. So we want the utilization to be zero, which means the Matrix value equals zero. So this is also from ql and there you have it. Those are the either allocated gpus they have pod running on them and visualization is zero.

C

So I used some more Chrome ql to aggregate all this data on this metric and other metrics as well, and then I put it in this grafana dashboard. It's also open source. Of course you can take a look at it later. You see if the previous queries layers and how I put it in rafana. So now that we understand where this data is coming from, we can analyze what we see so.

B

C

The amount of item allocated to use and below deposites that are running on those GPS and.

B

Here we have the pending defending.

C

Positive requested GPU so.

E

C

Also know how many gpus each photo is requesting for his item now. If each user or team believes a different name space, which is a good next practice uh yeah, um then we would know the owner of those Idol.

B

C

And we can kindly contact them and ask them to delete the reports or maybe delete.

B

C

Ourselves, if we have the permissions for it, and we can also know the owners, the the users submitted attending costs, so we may know who's going to be angry with us, but they're not running on reports that.

B

C

Be executed now I wanted.

B

To look at a slightly.

C

Different case here, the cluster is not full. We have six gpus out here and a store. Now some organizations might not even know this information that they have available. Gpus they're not idle allocated because there's no quad running on them, they're just available. So it's an easier problem and I know it sounds funny, but you should just execute more workloads, use your availability gpus to pay good money for a number.

C

These graphs give us a good visualization of the GPU utilization. So here we have the GP utilization over time. Each line stands for different GPU and you can adjust that time spent on top of the good hand page and so.

B

C

Be like the last five minutes or the last 24 hours, whatever you want.

C

Also, we have the average review utilization out of all gpus in the cluster and we have their running deposit, allocate the GPU for this digitalization.

C

So now that we have all this data, we understand the GP utilization, you can see it and visualize it. What can we actually do about this?

C

Well, the first thing is the thing, and we said that already we can delete the audio GPU box so.

B

C

User or the admin can do that, if it both have access to the dashboard. Another thing we can try to do is to avoid the ideologicals in the first place, so.

B

For example, when you run acrosses.

C

Inside a Jupiter Road.

B

C

You press that Little Rock button and then the process finishes and the notebook stays open and allocates an energy View. So.

B

Instead, you can finish working on your model inside.

C

When you want to execute it long term, you can run it as a previous job in that way, when the process inside finishes the Pod and the job will terminate and the GPU will be available for other ones in that way, depending.

E

B

C

For example, can start running and this is our goal, so, let's summarize, we understood the problem and presented some easy steps. We can do to improve the situation, but it takes more. You need to look behind the scenes, maybe think of a better work process, one that suits you in your organization. We have all this data presented in the dashboard um that can help you get a better understanding of the degree utilization so.

B

You can use the.

C

Dashboard deployment change, it add more information to it, adapt it to your needs, maybe use grafana alerts. The power is in your hands only after we do all that. You can continue to do the next steps, which is, for example, smart provisioning, smart scheduling, increasing.

B

Digitalization.

C

Of existing models, but we don't have time to talk about it today,.

A

Yeah, so thank you, everybody for joining us today. As we said during the presentation, everything we mentioned. Every tool is open, source and available on our GitHub page and we'd love to hear more from you about the way that you use gpus and the problems that you encounter so here are our email addresses feel free to contact us. We will also be at the Run AI Booth booth, number S63, so just walk by in the next couple of days and we'd like to hear from you. Thank you.

D

Brilliant um thank you. Raz and Natasha excellent talk, um yeah I'll, open up to questions. um Okay! Oh there are loads. Oh that's! This I was actually going to ask one but I better better. Let the audience do them. First, when you go first.

F

um I just had a question about.

F

It strikes me that if you're going to do it dynamically, you may you may need to know the length of time somebody's going to use the Pod um in order to so that you're not switching them right or if, if it's idle and then the person who owns it the way you've described, it is very like you take it back so now that first person no longer has access to a GPU, just wondering if you could elaborate on that.

A

Yeah, so that's a great question: yeah.

C

B

Have some idle time, then they go back to being not idle.

C

But if you see like um a pod being idle for a day for a long time- and you maybe you don't have to know that this person went on vacation, you just see its Idol on the dashboard. Then you can delete it and yeah the person will lose access to the GPU, but well they shouldn't in the first place, execute workloads that have idle gpus for 24 hours. So it's like our. Our approach is like their problem.

A

If I, if I may add, so um what you suggested is a is a great idea. It's a bit too simple and naive I mean it would be beneficial. You can specify the amount of time that you think that a pod is going to run so that the orchestration system as a hint to the orchestration system, but this is too simple when you talk about big big clusters with many different AI workloads.

A

So what you would want to have is a system that would automatically detect idle resources and idle pods and we'll take these resources and bring them automatically to other pods. So this would be a great idea, but it's too naive I'm sorry, it might be too naive and we're looking for more advanced ways to even improve it without specifying hints priorly.

G

One Challenge we've had with using the dcgm metrics is that utilization is actually defined as if you've done one Cuda operation in the last one. Second, so you could do a Cuda mem copy and you get utilization I'm kind of curious if you've been able to develop anything else deeper. To actually understand how many Cuda cores are really left and like how much you can push the GPU more.

A

Yeah, so that's a great question because the GPU utilization might be misleading because, as you said, it might be also too naive, because if you do something very small on the GPU every sampling window, then it might seem that you have high GPU utilization. Now this is an inherited problem with looking at GP utilization.

A

uh There might be other metrics that could be created, for example, to understand the size of the Cuda calls that are running, and there are Technologies of Nvidia, for example, MPS that allows you to have a bit more understanding about them, and you have also profiling tools from Nvidia. That will help you to further understand how much you actually do use your GPU. Now GPU utilization is just a very basic metric, but it gives you at least 80 percent into the direction that you're heading cool.

H

So I'm curious what you see as the uh the future of the road map of time slicing or context, switching of gpus. That does seem like the Holy Grail I'm, a system operator, not an a data engineer. So you know what I see as a system operator is well. Why aren't we treating our gpus like CPUs with multi-threading and time slicing and stuff, and what do you guys see as the the future of this this like around the corner? Is it decades away like? Where is the technology going.

A

Okay, so this is another great question, so yeah I think the way we see it is that we believe that we can have a system that automatically detects the idleness and the resource consumption in real time and then just like any other more mature resources, for example, CPU cores or CPU memory that they've been here for decades.

A

Just like that, we can develop technology that would treat GPU resources in a very similar way in which you just submit your workloads, and the system underneath realize understands in real time how much resources every workload needs and if a workload can give it up temporarily, if it gives up, gives it up intentionally. So for this temporary period of time, we can give these resources to another workload. So yeah.

A

This is the Holy Grail and the Holy Grail is to make gpus very similar to CPUs, because nowadays, when you submit CPU workloads, you don't really care about what core. What CPU core is going to run on the operating system? Does it for you? So we believe that you can achieve the exact same thing for gpus and no it's not decades away. It's!

A

It's here and it's going, it's going to happen real soon, I believe it.

E

um So earlier you had described a little bit about looking at like batch workloads and jobs in order to be able to kind of relieve the pressure on the gpus. Is the primary spot that you're looking for to try and relieve this GPU pressure, like looking at being able? How do we move some of what we would know, what that we might do interactively to job-based things or like? What does that Continuum look like.

A

I'm sorry I'm, not sure, I understand the question.

C

C

If you're, debugging or.

B

C

A model, and then you you finished it and you want to train it for a long, um a long time. That's where it's better, that you switch to a job and not a notebook, because when I describe it does that answer your question.

E

I think that's part of it because, like so so like even during that debugging cycle, like so I, have a data scientist working for eight hours a day. They take that break, but they're still interactively looking at that. That is a to me like, even as you described in the presentation, that's a period of time that, like they went for lunch for an hour and a half I could use that GPU somewhere else, but because it's nominally interactive job, that's still holding that GPU yeah.

A

Can relate to that so this is. This is a really great question, so yeah here we presented some basic stuff that people can do nowadays. But if we're talking into more advanced stuff, then your your point is is really correct, because we believe that we that you can have a system that detects this GPU idleness and when you decouple this CPU resources from GPU resources, then you can leave the jupyter notebook running.

A

The user can still use the GPU notebook for visualizing the data for writing code interactively, but as long as nothing is running on the GPU, our system underneath can take the GPU resources to another pod. That is, that is sharing in a controlled Way by this system and it's sharing the GPU resources. So you do I mean like the Holy Grail is that you will not have to stop your job entirely. You just you'll just have a system that manages it underneath.

A

Does that answer all right.

I

uh Did you get a chance to compare schedulers like misos versus kubernetes, so uh some misos has a good scheduler called dominant resource fairness and it has a it, has a pretty good, uh very decent, creating algorithm and you get a chance to compare and so that we can. You can see how which is a better scheduler for resources like GPU.

A

All right so uh yeah, we believe that we understand AI workloads in a in a very intimate way, because we go through all the stack we go through the kubernetes layer and the Linux layer and the GPU the Cuda layers themselves, as well as the applications and Frameworks like tensorflow and pytorch. So we believe that we can create a kubernetes scheduler that is fully dedicated to Ai workloads and we believe we believe that we can build the software that is needed in all these layers, instead of relying on others to implement them for us.

A

So I'm not familiar specifically with the scheduler that you mentioned, but internally we do have a lot of thoughts around existing schedulers and we do. We do have our distinguishment between them and I'm. Sure you can. We can talk about it later, cool.

D

um Time for one more question: if anyone has one I haven't already: oh sorry, yeah yeah, there we go.

J

Have you found that the provisioning and metrics differ by workload type? You know a lot of what you've explained is really good for like batch processing, but you might have inference workloads on the same cluster that want to hold on to a GPU longer, and that might be you know appropriate for for that type of workload.

A

Yeah exactly I mean there are many types of workloads, for example, interactively building a model training the model in inference servers as well, and inference servers can be idle as well, because they are waiting for new requests to arrive, so we can detect it, and while until these requests arrive, you can take these GPU resources to another workload. So this is. This is true also for inference clusters as well.

D

Great um well yeah. Thank you again. Let's have another round of applause for us and Natasha.