ONNX June 2022 Community Meetup, 13 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Bring the power of ONNX to Spark as it never happened before

Description

Both data processing platforms and deep learning frameworks are evolving in their own fields. Usually, Spark is used for offline data processing, and then various deep learning frameworks are used for data inference. A simplified API for DL Inferencing is very important as a bridge. What does an ideal data and deep learning inference pipeline look like? We'll discuss how to build your AI application using Spark and ONNX, the current status and initial idea of Spark community to improve this pipeline, and also make full use of the capabilities of Ascend Hardware Platform. This topic help you know the latest progress of Ascend Hardware Platform integration in ONNX, as well as the initial idea of the inference pipeline improvement in the Spark community.

Xiyuan Wang and Yikun Liang are Senior OpenSource Engineers at Huawei. Zhipeng Huang is the Director of AI Open Source at Huawei.

A

Hello: everyone, my name is Shivan Wang from Huawei now I'd like to share the topic to you. It is called bring the power of Onyx to spark, as it never happened before. This slide is prepared by equine triple and me.

A

Okay. Let's start this page shows the simplest workflow of inference flow of data and AI pipeline. In general, the user uses the spark a data processing platform and uses the Onyx random us inference platform.

A

The well-defined data frame inference interface is very friendly to data engineer and data engineer can easily to load data and complete the feature engineering.

A

They are very profit proficient in data processing, as the factor standard of the Big Data platform spark can help users easily and conveniently use a simple interface to divide huge data into each executor, especially in recent years. Spark has enabled pandas on spark API since version 3.2, which is also very convenient for users. At present, various AI framework, such as tensorflow, pathology, men's ball, and so on has their own implementations for developers who use AI framework.

A

They are often coefficient in prep parameters. You a turning of AI Frameworks and another internal pre-sample of Frameworks.

A

There are some gaps or effect, friction points between the two, because data Engineers often do not understand the framework very deeply, and they do not know the internal of various framework and the same as AI Engineers, the don't know the deeply about the data Big Data framework.

A

Therefore, the spark Community has also initiated a discussion on spark Improvement proposal and it is called spip hoping to have the complex process process of each framework by providing a simple API and to make the process of spark and AI influence. Small cell, the goal of this spips to Simply simplify the Divide deployment of DL models to spark inference, enable integration with third-part, DL Frameworks and the targets portions are data Engineers, who need to deploy DL models on spark or the developers who need to deploy DL models on spark.

A

You can also search the jira to learn more. This spip is still under discussion. If you are interested, you can join the discussion as well.

A

This is a simple example to show how the developer should complete the influence after DL on spark enabled in the future. We can see that the only thing the user needs needs to do is to import the corresponding framework extension and model URL. Then other complexity, including tab processing conversion and the framework initialization, are hidden in module UDF. What is what each framework needs to do is to complete to implement the corresponding implementation Cloud, according to the X actual situation of the framework, then see the right picture.

A

The spark will help you to execute the Onyx in inference in spark executor and Spark will help gather the final result. In this way, users can easily complete complete Onyx influence on Big Data Onyx Grantham, also influence platform to optimize and accelerate machine learning inter inferencing, especially by providing a well-defined middle layer.

A

It builds a bridge between the underlying AI acceleration, Hardwares and model influence with Onyx format. Huawei has its own AI processor called Ascent. So if we want to complete the spark and Onyx Pipeline on the ascent platform, we should introduce or send support in Onyx runtime. First, first of all, I'd like to introduce the basic concept of ascent: Ascent is the name of NPO AI processors from Huawei around it. Huawei has built the ascent AI ecosystem.

A

Let's take a look at the graph here, the gray rectangles. Here's represent different Ascent processors models.

A

Ascend 310 only supports AI inference and both 710 under 910 support, train and influencers well, based on the processors, Huawei build a serious AI, related Hardware, which shown in the blue rectangle, and they are called Atlas here, I'd like to see more about others, three, zero, zero, zero, it's a kind of PCR card and used widely on data or AI process servers.

A

Our develop and task work is based on it as well. In onyx, then, based on the hardware. Ascend ecosystem also provides a software layer called com. It is the yellow rectangles in the picture. Cam provides apis to help developers quickly build AI applications under service based on the ascent platform.

A

um Frankly Speaking is similar with Cuda in in Nvidia ecosystem for software. Khan is the main point that both developer and AI framework should know. Let's focus on con here. This is the current technical stack view in Ascent ecosystem. Thus they are. My colleague Japan has shared the constantly stack already in the Onyx Meetup, where it was based on kind, 3.0 version, which is out of date. This picture here shows the newest version called con 5.0.

A

As you can see here, there are multiple layer income, it contains service, layer, compilation, layer, execution, layer and the base layer. For example, service layer provides operator, Library, optimization engine and the framework adapter in general developers do not need to know Zone. You may only focus on accent, Computing language, um it's called ACL.

A

It's the apis part to help you control, Ascend Hardware.

A

Currently, the user want to run Onyx model on other Hardware. He should first use the model translation tool provided by con to translate the model from Onyx to ascend.

A

The flow is a little complex and the translated model May lost some precision and the performance May Pro, even in some case the model May cannot work correctly to solve the problem. A better way is to find the solution that Onyx model can work on Ascent directly, so in our next runtime we'd. Like to add con the new execution provider, once it is done, users can use Onyx model on a certain Hardware, while Onyx runtime directly, of course, will add the related CI as well.

A

For example, we can donate VM resources which contains or send Hardware to the community. The orange line below is our road map. First, we'll push the basic code to Upstream, the other two and flow will be done and the rest night model should work correctly at the end of this year. We'll finish all the Onyx operators support and make sure all the models in Onyx model Zoo works well on ascent. In the next year, we will focus on obtaining amazing work like, for example, like performance, Improvement and so on.

A

Now the POC code is almost done, use the POC code. You can now run some basic operation correctly understand using Onyx random, for example, the add operation.

A

If you are interested, please leave us the message we can discuss more in the future. Maybe we can provide a certain resources for you to test as well. Okay, that's all of our topic. Thanks for watching bye, bye,.