Cloud Native Computing Foundation KubeCon + CloudNativeCon + Open Source Summit China 2021, 18 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Beyond CUDA: GPU Accelerated Computing on Cross-Vendor Graphics Cards with Vulk... Alejandro Saucedo

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

超越 CUDA：GPU 与 Vulkan Kompute（AMD、高通、NVIDIA 和 Friends）加速了在跨供应商图形卡上的计算 | Beyond CUDA: GPU Accelerated Computing on Cross-Vendor Graphics Cards with Vulkan Kompute (AMD, Qualcomm, NVIDIA & Friends - Alejandro Saucedo, Seldon Technologies

众多先进的数据处理范式非常适合 GPU 计算提供的并行体系结构，而 Vulkan 和 Kompute 等开源项目所取得的激动人心的进展则使开发人员能够在跨供应商移动和桌面 GPU（包括 AMD、高通、NVIDIA 和 Friends）中利用通用 GPU 计算能力。在本演讲中，我们将从概念和实践方面深入探讨跨供应商 GPU 计算生态系统，以及如何采用这些工具来促进您现有的应用程序。在本演讲中，我们将学习从头开始编写一个简单的几乎能在任何 GPU 上运行的 GPU 加速机器学习算法。我们会对使跨供应商 GPU 加速应用程序成为可能的项目进行概述。我们会向您展示如何利用仅有几行 Python 代码的 Kompute 框架开始使用 GPU 的全部功能，同时也会提供关于如何通过较低级别的 C++ 接口引入优化的直觉知识。

Many advanced data processing paradigms fit incredibly well to the parallel-architecture that GPU computing offers, and exciting advancements in the open source projects such as Vulkan and Kompute are enabling developers to take advantage of general purpose GPU computing capabilities in cross-vendor mobile and desktop GPUs including AMD, Qualcomm, NVIDIA & friends. In this talk we will provide a conceptual and practical insight into the cross-vendor GPU compute ecosystem as well as how to adopt these tools to accelerate your existing applications. In this talk we will learn to write a simple GPU accelerated machine learning algorithm from scratch which will be able to run on virtually any GPU. We will give an overview on the projects that are making it possible to accelerate applications across cross-vendor GPUs. We'll show how you can get started with the full power of your GPU using the Kompute framework with only a handful of lines of Python code, as well as providing an intuition around how optimizations can be introduced through the lower level C++ interface.

A

Welcome to my talk on beyond cuda gpu accelerated python on cross vendor graphics cards with vulkan and compute. My name is alejandro saucedo. I am engineering director at salon technologies, uh chief scientist at the institute for ethical, ai and member at large at the acm.

A

So today we're going to be delving into a very interesting set of topics uh primarily around the parallel and gpu computing ecosystem.

A

We're going to talk about what is the vulcan sdk and how you can use the compute framework um to build uh python. uh Gpu accelerated applications together with a set of hands-on examples, as well as some references that you can actually delve into if you're more interested.

A

So to start with, we want to cover uh the motivations of why parallel processing and uh I'm going to be referencing.

A

A research survey um that I recommend checking out um as well in itself as it basically collects insights from uh 227 papers in the uh parallel deep learning space, which provides a much more uh uh intuitive perspective of of the adoption, uh and this trend of adoption on gpu, for not just scientific computing, but also general purpose computing, and some key observations that are really interesting is to see how the emphasis around a lot of the functions and paradigms in these computational areas that can be abstracted into high highly parallelizable steps and others that may not in themselves be highly polarizable, can actually be reduced into equivalent uh uh structures that that then can be processed uh with specialized hardware like gpus or even you know more specialized.

A

However, like tpus there's also concepts that are continuously evolving. um You know a simple one that may you may have come across is the concept of micro batching, which is basically being able to not just process a single data point, uh but also being able to process uh several more at the same time to ensure that uh the computation is done at a single um or you know, in a way intuitively on a single sort of uh parallel clock, speed uh or clock iteration.

A

Within uh that that perspective, instead of just submitting each one separately as well as different, uh very interesting and innovative ways of breaking down computation that can actually be processed in parallel to then be uh re-aggregated.

A

And the interesting thing about this paper is not only how the trend is moving towards this parallel processing- and you know gpus and specialized gpu uh type hardware, uh but also uh that it's expanding into uh distributed, processing leveraging uh this uh uh parallel, capable type of hardware, uh so really interesting area um now delving into the into the ecosystem um and the motivations of why uh you know we're even having uh uh a new sort of framework.

A

The vulcan sdk, which we're going to introduce in a bit, uh is particularly given that there is this sheer uh range and and heterogeneity when it comes to the gpu and parallel capable processing, hardware like tpus, etc. um That involves multiple different players, multiple different architectures, different drivers and, uh consequently, different uh uh frameworks available to uh take advantages of this parallel capabilities as well as, for example, um you know the increase in very powerful uh mobile, capable uh uh components that now we're carrying in in our pockets as a smartphones.

A

So there is a sheer amount of uh need in this space due to the heterogeneity uh of the different hardware across different vendors, um and from that same perspective, is one of the key motivations of why vulcan came to be.

A

Vulcan is a cross-industry initiative that brings in uh several of the leading industry players to create this open source, and I think that's the the one of the more exciting parts, the open source and open uh uh uh sort of standard that uh uh focuses on not just uh uh interoperability, but also performance, and this is reflected in the interface that is exposed uh by uh the uh vulcan uh sdk. Now, in regards to the the vulcan c, plus plus sdk, there are several advantages and disadvantages, as you would have with anything.

A

Some of the very strong advantages is that it has a very low level interface with rich access to components that comes with this explicit, amber bose c style api within its core. That provides you a you know: no uh language, um sort of sophisticated verbosity or abstractions uh pure direct access into not just the vulcan sdk, but also the hardware underneath right- and this is very important for optimizations that are necessary.

A

There are a broad range of industry, leading players that are contributing to the standards and to the to the sdks and to the tooling, which has been uh very, very uh encouraging to see, um and also there is an emphasis towards this interoperability and high compatibility across different platforms, mobile suppliers, amd nvidia, qualcomm, etc.

A

And you know this is very, very strong strengths right, strong advantages in the disadvantages. You know. We also have that it's very low level, with rich access to components, which means that there is a lot of complexity, and you know a very rich interface that needs to be interacted with and then similarly with the c style api, even though it provides you a w. What you see is what you get with the hardware level.

A

That means that there needs to be a lot of domain specific knowledge to be able to build the foundational layers required to start building the application components right with that, uh there's also the broad range of top players which, even though it's a great thing, always uh with many opinions, many voices uh there will be a a a lot of sort of interactions uh that will be pulling towards different directions, even though they would have the sort of like best uh uh uh in in in in mind and trying to push.

A

For uh uh you know what is what is best and uh again. You know with a high compatibility across multiple different platforms. That means that it needs to deal with a very rich interface that uh and not just rich interface, but rich set of um um flexible. You know back-ends that can actually inter interact with with this sheer number of different uh uh um hardware. uh That is underneath and now, let's see what the architecture of the vulcan sdk looks like.

A

So the vulcan application is the overarching component that we will be uh seeing uh from applications. You can actually spin up instances. These instances are the ones that then allow you to. uh You know talk with your physical hardware, so physical devices there's plus component- that you know actually refers to the to the to the physical graphics card that you have in your computer and then you can create what is referred to as logical devices or windows or views that then allow you to interact with physic with that physical device right so that logical device.

A

You know you can have multiple logical devices for one single physical device and we can have multiple physical devices for an instance and multiple instances for an application right. So this is where it starts getting a bit complex, but you know to keep it simple. You know with this logical device. The way that you interact with it with the uh graphics card is through a cue, and this queue would have multiple commands right.

A

You submit the instructions that need to be executed right and, ultimately, this is how you would be able to uh submit instructions to the gpu once you actually want to run more complex components. That's where the pipeline comes in. You know, there's the compute pipeline the graphics pipeline, but we're going to be just talking about the compute pipeline. This is all just to provide an intuition. You know we. I actually have another talk that you'll be able to reference and check out as well as the documentation that explains all of this in detail.

A

This is just to give you an intuition once we delve into uh into the deeper depths of everything with the pipeline. This is what you're able to say. Well, I want to actually run a specific set of instructions as per a piece of code or an algorithm that is often referred to as a shader, a shader module.

A

This shader is basically going to look like a piece of c code that just so happens to run on the gpu itself and this shader. This piece of shader in the pipeline will also interact with data right, and this data is referenced through the concept of the scripture sets, which again you know, you're not going to really have to like know all of these different things, but uh it's still important uh to get an intuition that this is what's happening under the hood right.

A

So the scripture sets are just like a container that basically says well, I'm going to be using this gpu visible, um gpu, visible data, so that when you run a set of instructions, you can actually reference this. These different things- and you see that, because we're interacting with this queue, the cpu in itself, is able to send uh requests.

A

So you send send instructions to the gpu with its own specific memory address space, and even though there are shared address spaces, which means that you know the cpu may have access to see some of the data in in the gpu memory. There is different address spaces for different uh computational um um um areas.

A

In the in the actual hardware of the gpu, which means that the actual code, your c plus plus your python, will not be able to see that memory space right, and that means that you would be able to interact with your gpu with this sdk akin to what would be you interacting, with a remote service, sending requests for the service to execute, as if it was an api. And you know, even though this is not correct, because this is your your own machine.

A

This is actually a good intuitive way to see how you're, interacting with that gpu, given that it is through this queue with, uh you know, asynchronous command buffers that are executed and, of course, you can then have all the things that allow you to await until the execution has been finished, which we may actually talk about um in later on, and you know being able to build the foundation code required to actually run a simple program in vulkan.

A

You know only takes you know from 500 to 2000 lines of c plus code right- and this is the motivations well, some of the motivations for compute itself, the compute framework. So compute enables uh uh developers to get started interacting with the vulcan sdk, with dozens instead of thousands of lines of code, and the key thing to emphasize here is that the core principle is to augment the vulcan interface instead of abstracting or hiding it. It has a bring your old. uh You bring your own vulcan interface, which plays nicely with already existing vulcan applications.

A

So if you already have a vulkan application to render uh graphics related stuff, you'd be able to actually pass those vulcan components, and it has non-vulcan naming convention and we're going to talk about that. It's just basically to avoid ambiguity, as there are libraries that you know may have uh classes called buffer, and it's like is this buffer from vulcan or for this other application right. So things like that um so now in regards to other features, is that it has.

A

You know a c plus plus interface, but also the python interface that we're going to be using today. It has explicit cpu and gpu memory ownership, and this is important if you're using uh non-compute, vulcan components right if you're already using vulkan in another area of code raw vulcan.

A

It gives you granular access to gpu cues, which is very important for optimizations. You know which we have some material that actually explores that in more detail. um There is you know single header file for the simplest plus development and a pi pi module for easy installation with the python, and it has integration with mobile apps uh through the android ndk, as well as game engines uh such as godot, which is also part of the fosdem conference, um and there is going to be a few links about that.

A

So how does the the the compute architecture look like, um and this is basically uh relatively or conceptually simple, so everything starts with a compute manager and the compute manager is the component that basically oversees and and manages all the uh explicitly relevant uh memory uh that is then created through your uh interaction with this, uh like compute uh application, so the manager in itself would handle this sort of like device and on the queue which we we talked about. You know you don't need to go no into that much depth, but it's still important.

A

You would then have sequences and sequences are basically uh basically single or batches of operations to run on the gpu right. That's, basically what a sequence is, and you can have multiple sequences with multiple operations, each sequence can have one or many operations, and each operation basically performs a specific action. An operation can have one or multiple tensors and tensors are basically abstractions into gpu and cpu memory, as well as the workflows related to move the memory around it and optionally.

A

It can also have what in the compute world is referred to as an algorithm and an algorithm abstracts the concept of the pipeline, the vulcan pipeline, the descriptor sets uh and the specific shader code that then you can actually say. Well, I want to run this this code in the gpu with this specific uh data um structures, and I want to run it in this type of way right. So we will see what that actually looks like in more practical terms, but this is basically it right.

A

There's there's there's not much more around this and there is an almost one-to-one mapping between uh the compute components and the vulcan components, and that is explicit to reduce ambiguity. Now, let's see what that looks like so in python, you would basically create a simple uh manager right. You would then create a set of tensors with you know you can pass numpy, arrays or or python lists uh in this case.

A

It's basically just uh two uh tensors that are gonna, be used in a multiplication and then the output, where we're gonna, save uh all of the results right. So that's what we're gonna basically use. uh You normally would actually initialize the tensors explicitly, but there's a set of helper functions that you know, take your cpu host memory list and then copy it into gpu only memory. So all of this is is handled to you, but again it's not handled through magic.

A

Every single thing can be actually accessed and you can actually call those things yourself if you wish to which is very important for for for several optimizations, you then define the shader code. This is basically the code that you're going to be running in the gpu. In this case, it's just basically a simple multiplication using the pi shader decorator.

A

In this case we have the first buffer, the second buffer, we're going to run a multiplication and store in the output. That's basically it this is going to run in the gpu and then we're going to actually run it through the manager. So we're going to say, we want to run this synchronously uh using these three tensors and using this uh uh shader right, and we can just basically pass that it runs synchronously.

A

We can also run it asynchronously and we can actually uh do a lot of optimizations which we're not gonna delve into this, but you can run it synchronously. Once it's finished running, then you can actually copy the data back right. Make sure that the output is now visible in the cpu and then you can print it right. So you can actually see that our output is two four six right.

A

So that's basically it that's all there is to it, and this is all you need to basically do all of the uh crazy crazy stuff uh that happens underneath, but again, all of that crazy, crazy stuff is really really interesting, and it is also very, very useful, very rich and very relevant for once you're starting to do optimizations so ultimately not to hide but to augment. This is the key thing. So now, let's actually delve into some of those optimizations that I've been mentioning.

A

What we did right now is run a single command, slash operation in through the manager right. So we we we had the cpu running. We submitted that operation, we waited. Then we came back. We then submitted the second one, and then we came back right. What we can do as well is we can actually reuse multiple sequences, um which means that we can pre-record commands, so we can record a bunch of operations and then run them, so the cpu would basically then execute. There would be specific. You know, operations that would run already.

A

uh You know at this in the gpu and then you would actually wait until it's come back. You can actually run a synchronous, um dispatches or submissions of the of the of the commands, which means that you can actually uh not wait for the gpu to finish so. You can actually submit this. You know the cpu would continue doing other things while the gpu is doing other things, and then you can submit something else, asynchronously and then do something else.

A

There's also an await function that allows you to wait for the thing to to finish and then finally, something that we're not covering in this presentation. But it's also very interesting, is that you can also leverage gpu hardware, concurrent queues, to submit multiple uh uh batch operations that would then execute in different gpu cues. That then, would potentially run in parallel, and this is dependent on the uh hardware properties of your gpu and the families you know normally.

A

For example, in my nvidia 1650, uh I have the ability to run hardware concurrent uh batches of gpu loads uh if I submit them to uh one compute, family queue and one graphics, family queue we're not gonna delve into that. But if you're interested you know, there's there's a lot of like really um relevant content in the documentation, as well as in our other talk.

A

But today we're going to be actually covering something um that delves into the world of machine learning and what better thing to cover than the hello world of machine learning, logistic regression. Basically taking in a specific data point and classifying it as either. You know correct or false. In this case, it's just a binary classification and we're going to be letting the machine do the learning in the gpu.

A

What are we going to be doing in terms of intuition we're going to have uh input data that is going to look like two numbers uh that are going to go through our model, our machine learning model and are going to perform a prediction which ultimately should be um what we are expecting. What our data looks like uh at least the training data that we're going to be using the training data that we're going to be using is going to be. Basically, um you know when we see zero and zero.

A

We expect zero when we say zero and one we expect one. We see one one, we expect one and we have, like you, know a bunch of different training data. That looks like this extremely simple. I know, but just to make sure that you know the intuition comes through as opposed to the machine learning I mean it's, not a machine learning talk, um and then you know what we're doing is we want to actually train a machine.

A

Learning model basically learn the parameters that would allow us to ensure that every time that we have these inputs, we provide the respective outputs right, and you know we- uh there is sort of like a more in-depth blog post that covers the underlying. um You know, functions and um the way that everything is broken down, but we're gonna skim through some of that. uh If you're curious, you can actually delve into that we're still gonna talk about. What's actually going to be happening, we're going to be trying to find the parameters of this function.

A

This is going to be the input, basically that sort of like specific, x1 and x2 that you saw in this case, because we want to actually leverage the hardware parallel capabilities, we're going to be able to submit multiple as micro batches so instead of just running one by one we're going to be running like five at the same time, so the gpu actually runs five and then comes back to us. um So that's that's. How we're going to be able to do this?

A

We're going to be learning these two parameters w and b and we're going to be. Basically, you know this is the function that calculates that prediction right we're not going to be delving into too many too much of the depths, but there's a blog post that covers this in a bit more detail.

A

There's thousands of talks that talk about logistic regression in python, so more than welcome to to check that key thing here is that this will be the the shader code that we're going to be writing right and even though we're not going to cover in much detail, we're still going to be looking at what that will actually uh uh uh required to actually write that and in the compute side.

A

So this is the shader, but the compute side is what is going to be running this we're going to have to create a bunch of tensors that represent our input data, our our our parameters, uh our our our predictions, etc, etc. The training data we're gonna have to initialize those sensors using um you know on the sequence, we're gonna initialize uh the algorithm right and then we're going to actually so we're going to record it. So this is just to actually like initialize them and then we're going to iterate and learn.

A

Let the machine do the learning right, so we're actually going to be running uh a multiple, different sort of like iterations to that data, set uh updating the parameters every single time, running micro, batches that are running in the gpu in parallel yeah. So we're going to be basically doing that and then once we actually uh uh iterate those 100 times we're going to be having learned those parameters right. So that's basically what we're going to be doing in this uh high level logic- and you know you know this- is just an intuition right.

A

The key thing here is to see what's happening in the in the compute and in the in in the shader side, so the the shader, what it it looks like is just a much more complex version of what we saw earlier right. So we have all of the inputs so x I, which is x1 x2, is each of them they're going to be an array, because it's micro batches, we have the expected outputs. We have the weights that are coming in and we have the weights that are output and calculated.

A

Remember that the parameters that we're learning are w and b. So those are things that we actually want to take out, and you know we're also taking the laws uh to be able to reuse where we're relevant and the number of of parameters to use right. So in this case, what we're going to be doing is we're going to be uh you know, taking the specific uh input weights that are continuously being updated.

A

These are the ones the parameters that we're going to update in each execution, so we have to pass them every single time we're going to be calculating the function right. So, as we saw uh you know, this is basically the the function that that we just had, and you know in the blog post, I break it down in very minute detail of how um we actually um uh go through each of these steps and and calculate the derivatives, the partial derivatives of each of them, as well as the as the respective loss.

A

But here the key thing is that we are able to ultimately calculate the the the the the parameters um uh that that we now have right. So now, in the compute side, we're gonna first create all of the specific tensors. So, as we saw, we had some training data. This is the zero zero equals zero. One. Zero should be that uh you know one one should be one et cetera, et cetera, um and then you know this is how we start with our weights.

A

So we start with just like a random initialization, which then you know we're going to be iterating towards we're going to then similarly start with, uh you know random initialization for our other parameter, which is going to be zero and uh then the number of uh you know, data points is going to be uh basically um the actual size of this parameter right, so we're going to start with with five right. So that's what we're going to be doing we're going to be storing that in a variable called parameters so that we can reference them.

A

We're then going to just initialize it by creating our manager and initialize the tensors. So what this does? Is it actually initializes them explicitly? Whereas before we didn't do it, we did we did it implicitly with a with a utility function. So here we're saying initialize all of the parameters in the gpu, so they're accessible in gpu memory, then we're actually creating and recording this the the the operations in the sequence. So what are the operations? We're gonna?

A

First, actually uh sync: the data to device for these two tensors, because remember the parameters are going to be updated every iteration, so we're going to put them in the gpu device uh uh memory. We're going to then uh record that algorithm that we just actually wrote.

A

This is the logistic regression shader uh that you saw so we're going to record that execution and then we're going to record a sync to local right, so we're going to record for all of the weights, uh the parameters and the and the and the loss to actually be copied back to the host, so that python can see them right and then finally, we're going to iterate 100 times by you know every iteration running that shader or running the sequence, which is basically all the things that we just recorded and then operating and then updating all of the all of the weights.

A

So I think there's an indentation missing here, but basically, what is happening here is we're just basically updating it by you know the specific learning rate, which is just how fast do we want uh the actual parameters to be updated on each iteration again. The key thing here is just to see all of the features that you're able to use with compute and leveraging a simple machine learning use case as an example that, of course, we're skimming through I'm conscious. Sorry for that, for the people that you know maybe are seeing this and going like.

A

Well, you know. Actually, that's you know 100 correct, but you know in the blog post I break it down in you know much more minor detail. This is just to show how you're able to actually interact with the gpu- and you know, optimize in different areas by you know: pre-recording components using the sequence, etc, etc.

A

So, in this case, we're just iterating once you're finished, you're able to then just print uh the calculated parameters which in this case they are uh the first weight, the second weight, um the the third weight uh and uh the actual um uh b uh bias, which is ultimately what um we ended up with, uh and you know just to emphasize. You know we covered kind of like this high level example.

A

But, as I mentioned, we have a blog post that covers this one in python, one in c plus plus it breaks it down in in my new detail and we have other tutorials other examples that cover you know how to use this, and actually the c, plus plus, uh uh as opposed to the python one for uh integrating with your android apps, as well as to game engines like the godot engine, uh which we would recommend to check out, and you know more than anything what I recommend is to get involved.

A

um You know if you go to uh github.com ethical ml, vulcan, compute uh you'd be able to actually check out what are some of the open issues. uh You know you can take one of the good first issues, uh labeled good, first issues and also, please, you know, there's open, there's uh an issue number 52, which is open for just general discussion, so you have ideas for improvements or questions, and you can actually just post them there and we've had some really interesting suggestions. Some of the key things in the roadmap.

A

uh You know one of the main motivations to build this framework is to actually integrate it as a backend of an existing site scientific computing framework uh that is uh potentially even being used for uh you know, mother, um you know machine learning or for other types of use cases. So definitely really interesting on that. And if someone is, uh you know running a scientific computing library, then you know be open to explore uh also creating more default operations, something like a fast fourier transform or like a parallel sum uh reduction.

A

uh Things like that would be really cool to have like out of the box operations. That are, you know, perhaps even written in c, plus plus, but also exposed as python and then also adding examples. If you try this in a new sort of shader uh or a new sort of like uh algorithm, uh a new machine learning uh type of model, uh you know would love to actually contribute it upstream, and you know added to the repo, because I think that would be very cool.

A

um So with that said, uh I think that's everything that we had sort of to cover today. uh Thank you very much for joining this talk on beyond cuda gpu, accelerated python, on cross, vendor graphics cards with vulkan and compute, really looking forward to explore and hear your thoughts, ideas and suggestions, and if you have any questions, do please feel free to reach out. Thank you very much.