Cloud Native Computing Foundation Kubernetes on Edge Day EU 2022, 9 Jun 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Empower Heterogeneous Edge AI Acceleration with K8s - Tiejun Chen & Zitong Xu, VMware

Description

Empower Heterogeneous Edge AI Acceleration with K8s - Tiejun Chen & Zitong Xu, VMware

As an emerging trend in the area of edge computing, edge workloads tend to be managed and orchestrated by k8s. In the meantime, as the the top one workload of edge computing, edge AI accelerations have been enabled by different vendors' edge AI accelerators quickly, including Nvidia edge GPU series, Intel Movidius VPU, Google edge TPU, etc. Actually you can see many ASIC-based edge AI accelerators and even some high-end CPUs used in edge AI. Obviously, edge users have the challenges around empowering these heterogeneous edge AI on the edge with upstream k8s or those edge k8s versions due to missing a general unified framework on k8s. Here we'd like to introduce our unified framework as a plugin to k8s with the following key mechanisms - 1. Extend the Node Feature Discovery to detect edge AI accelerators automatically 2. Unify different vendors' device plugin to provision work nodes according to it's own edge AI accelerator 3. Introduce transparent backend accelerations to boost ML upstream frameworks such as Tensorflow, Pytorch, etc 4. Attaching remote GPU to edge In our project we provide a unified edge AI framework to help k8s empower heterogeneous AI acceleration on the edge.

A

Okay, thank you for being here for our presentation here. I'd like to start off by introducing the team. My name is: the team also has another member.

A

We are both from vera octo office of the cto outdoor belong to our one group, that's epg for the ones technological group. Our group focuses on compatible new communication through our bad alignment and collaboration with our new product team and even partner customer. Basically, we are trying to include emerging technology solutions back here today. We're going to talk about this empower heterogeneous energy acceleration in kubernetes is about accelerating hai on kubernetes.

A

Now, let's go through today's agenda at first, I will talk some problem areas while running ai to the edge and then we'll talk about how to boost air at the edge over next with our solution. We also have a demo in the end. What's next talking about hdr, I think most people know machine learning area and also heard at the compute. We are facilitated data, processing and or near source.

A

Our data generation, but what's aji, are processing that connected or created on the device at the edge of our networking, but using artificial intelligence as a result for simply put hdr is a combination of edge computing and artificial intelligence.

A

According to some reports, hai in the growing gravity, actually, I believe- or you can find many other kissing around your life.

A

However, there are many challenges in listening for a good implication of hdr architecture, but usually machine learning tasks require powerful ai hardware, but many led users are resource constrained and they have limited, install space or limited power supply. So it's harder to enable machine learning on these edge. Uic variety of hdr accelerators have been introduced, but they are from different vendors, so the architectures are heterogeneous, most upstream machine learning frameworks such as tensorflow patterns and so on, do not support them directly.

A

You have to send out sort of sdk or toolkit to a user or maybe somewhere. You can find some machine learning frameworks can support some edi accelerator, but they are released and maintained by the hardware vendor and it's also difficult to get the best performance on the hdr accelerator, because you should realize there are many technologies potentially can be optimized to accelerate machine learning on this edge accelerator.

A

In the meantime, you know. Air and emotional workload are the most popular workload on kubernetes now. Actually, the edge user also expects kubernetes will be the platform of choice for running ai and machine learning workloads at the edge. How can we unlock with the add ai in our solution?

A

We intend to meet those challenges by building such and to animation and implement service on kubernetes at the well. Mostly, it includes two key parts. One is booster motion influencer with our backend acceleration mechanism.

A

The other is to enable ci based accelerator for selling unknown.

A

Okay, um let's first talk about our transparent backing acceleration. Now here our goal is to build an acceleration 7 system. The system is backing and automated, so this can work automatically. The system also had a unified software work, so you can easily integrate that to any platform.

A

We also enabled a lot of external technology enabled our many.

B

Id accelerator.

A

Here once it was mentioning, is that no any code change to your application written with those upstream motion works? How can we make this happen?

A

Let's move on here's the architecture, but you know we already integrated these two kubernetes, but here I leave this out in order to explain that easily. Overall, as you see here, when you find multiple logic nodes, each node has given you a name now. Basically, we first deploy system agent on your node.

A

This agent can help connect some necessary information, including your edge er, accelerate on your platform and your cpu type, and we also want to know if your cpu can support some special cpu instruction like erx or ex2, or you know in some cases where you will need feature to accelerate motion inputs and the manager of deploy runtime serving with a default back acceleration.

A

But the user can reconfigure that to any acceleration technology and we are supported on this platform through controller, then manager injects interpolator to a niche emotional improvement work on this node once the users call any ai code on this native machining the one hand our interpol can help get some data, including model or model info, and our auto compiler can compare this print trainer model to a one given intermediate reputation specifically to this uh back and forth acceleration technology and even cache and store that, on the other hand, um interpolant can intercept that api.

A

I will do emotionally influence caused by this native emotionally good work. Instead, we use our bacon accessory technology to do real machine influence with that pre-compiled intermediate gravitation and then get that result and output back to our major machinery will work. That's it.

A

Hey I'd like to elaborate a bit of our long-term interposer, essentially, it's targeted to our eps mapping, it's more like a mapping between machine learning upstream works to our backhand. Accelerating technology example is on tensorflow on one typical person code often called some api like load mode under the grids to generate the graph and turn another api predicted to our machining difference. Here we use some passion chips to redirect and know the model to our customer and model predicted to our customized predicts to our work. This out.

A

We also support cr plus plus, in this case we um pre-compile our handler into a shared object library and in the wrong time we hijack the motion, learning framework process to a load data model and operation api to our handler to make this out.

A

A summary of these parts: now we can support several upstream machine learning frameworks, tensorflow patterns, onyx and close to a tensorflow 7 system. There are a few vacant acceleration technologies available in our system, including pvm intel openvino, ah immediate tensor rt. We also enabled those popular ai accelerator intel movies, vpu, google edge tpu media edge gpu. We also leverage some technology like remote cuda, to connect the gpu to any x device to a machine difference. In some cases, are we using cpu to accelerate motion immigrants?

A

Okay, how to enable this transparent, backhand acceleration to our kubernetes ipad tools.

B

Thanks tijin hi everyone, my name is zitong and I'm part of the project team from william we're going to china octo department, as tiage mentioned earlier. In the solution section, I will be providing more details about solutions for deploying end-to-end machine learning, workflows based on kubernetes.

B

When doing machine learning, customers want to have a consistent, kubernetes platform to deploy and management the machine, learning workflow and make use of ai hardware accelerators on the edge for grid performance. Here we mainly combine several popular technologies.

B

The first one is node feature discovery, as name implies. It is used to detect hardware features on each node in the kubernetes cluster and use node labels to advertise these features. We use it to do detection of pci devices and usb devices on the node.

B

The second one is device plugin, which is a framework from kubernetes that you can use to advertise system hardware resources to the kuberleat.

B

We will use them to register and publish hardware resources on the notes to the corporate for scheduler to schedule currently kubernetes already provided some official implementation examples for amd gpu, nvidia, gpu, intel, gpu, vpo, fpga and etc.

B

Openvino runtime plugins also enables inference of deep learning models on some supported. Vpu gpu devices such as intel neural compute, stick intel mobile, ds, vpos.

B

We should note that intel kubernetes device plugins just support many new bpu cards and does not support the old device like ncs1. Thus we did an investigation on deploying on openvino vpo plugins.

B

Then we also use some kubernetes features like node selector, which is the simplest recommended way of node selection constraints, which also is a field of prospect that can make this part work on a specific node.

B

Finally, kubernetes scheduler will assign other machine learning paths to work on the targeted node. According to the registered information, as mentioned before,.

B

By integrating the above popular technologies, we designed an end-to-end machine learning framework solution which greatly reduced the complexity of environment configurations for users when using heterogeneous hardware accelerators and improves the efficiency at the same time, users just to need to use the basic kubernetes command lines to manage edge accelerators with the help of back-end acceleration technologies such as apache tvm, which is an open source machine learning, compiler framework for cpus, gpus and machine learning accelerators.

B

We also exploit intel open window tour, kate or nvidia tensority, as back-end acceleration technologies. By adopting them, customers can boast their machine learning tasks in the clusters without any native code change.

B

Okay. Now, let's move on to the next slides and take a look at the overall architecture.

B

Here assume, customers have a management cluster and they have several hosts equipped with heterogeneous ai accelerators. They can easily manage and take advantage of them by adding them to kubernetes clusters.

B

As you see, we have two worker nodes where the left one represents a general structure where the red one illustrates a specific example for nvidia gpu in general process. Firstly, we adopt node feature discovery and make it works. As a democrat on every node and we know node feature discovery leads pci usb ids in the node labels to represent the device edge devices.

B

Here we can see in the right side. Example. The node label is a number series which is hard for users to distinguish device names immediately. Thus, we have simplified this step by adding a myping function which can automatically translate and translate the discover device ids into human readable classes, wonders and device names.

B

In the example, the digital series has been automatically translated to the specific nvidia gpu cards. As you can see in the left side, which is geforce jt 710.

B

Next, by adding the specific device labels in the node label field, the customers can efficiently deploy device. Plugins demonstrate only on the node, with targeted devices upon the startup. The device plugins will report and register hardware resources to the device. Plugin manager incorporate and then starts the grpc server for kuberlate to excise.

B

Next corporate will establish the listen and watch link to get the device id and provide a health check.

B

In the end, kubulate will update the device information to the node status and wait for further scheduling. I will be providing a clear device. Plugin precise in the upcoming demo, part.

B

After successfully deploying device plugin demon sites, we can start to create machine learning parts. Here we can provide ready to build docker files, which include everything you need to run: machine learning tasks with the backend acceleration interposers, for example, apache tvm, intel, openvino and nvidia tensority and the necessary environment for interposing mechanism, hardware, configurations and etc.

B

It will create an image on the worker node for users to run their native inference codes with different ad devices in the container.

B

Overall. This solution not only provides customers with small edge device selections, but also greatly shortens the user's learning time for wasetor backend interposers.

B

Now, let's head to the demonstration part.

B

Firstly, check nodes in kubernetes cluster.

B

Here we can see, we already deployed the node feature, discovery demonstrate on each node and we have four worker nodes.

B

We can check the node labels by using this kubernetes command line.

B

We have heterogeneous hardware backends with human readable device names in our environment, for example, nvidia gpu intel, cpu, intel, mirade, vpo, google, htpo intel, gpo and amd gpu. Then we can check gpu capacity on each node.

B

Here we can see the nvidia gpu has not been registered.

B

Now checking the node with amd gpu.

B

The amd gpu has not been registered and deploy device, plugin demon, site.

B

Here we add the node selector with the node label.

B

The amd device labels to the node selector field.

B

Then create nvidia gpu device, plugin demon, site.

B

And create amd gpu device, plugin demon, site.

B

Demon sites are successfully deployed on the proper node.

B

Then we can check the gpu capacity on each node again.

B

Here we can see, nvidia gpu has been registered.

B

And the amd gpu has been registered too.

B

Then we can start deploying inference parts.

B

In the demonstration we will use tvm intervals as an example, then we can create inference part of nvidia gpu and our amd gpu, and on an intel cpu.

B

Then we can view the path status.

B

Here the inference parts are assigned to the corresponding nodes.

B

Then we can run tvm interpol, inference demo on nvidia gpu node.

B

This is customer local inference code for machine learning, influence.

B

Then we can enable the tvm interpol's backend server.

B

In another terminal we can run inference demo for resonant 50. We want model here we can see. We triggered cache mechanism.

B

We can run pvm inference demo on amd gpu node.

B

Enable the backend compiler.

B

See the target is rcom.

B

In another terminal, we can run inference demo for resonate 50. We want model.

B

Okay as time consideration here, we just showed the tvm example, but if you're interested in other backend interposers such as openvino, please feel free to contact us. Okay, let's move on to the next sled.

B

This slide shows the performance corresponding to the demo. We have compared the inference time between the tvm interposer enabled and the tvm interposer disabled for different modules such as resonate 50, resonate 101 mobile nate. We want trained by cipher data site and imaginate data site, mobile nate way2 on different hardware, accelerators such as nvidia gpu and emd gpu in our testing. The tvm interpose performs as well as tvm and generally accelerated.

B

The machine learning inference time.

B

Here we can see, in the left hand, side for nvidia. The best case is the mobile network module where, with our backend acceleration mechanism, the inference is eight times faster for amd gpu. The best case is also the mobile network model, which has been accelerated 19 times, and we believe this will even perform better on newer accelerator cards.

B

Okay, that's it for my presentation, part and I'd like to hand over to teejing again. Thank you.

A

B

A

Okay, the last part was our next now we want to support more machine learning works and more commercially solar system in production. We also need to be able to edge network next version um when they are, we will make this adder open source.

A

Okay. Now, that's all, please feel free to reach out to us. If you have any question and feedback. Thank you.