ONNX October 2021 Community Meetup, 21 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 010 ONNX 20211021 Lyalin ONNX and the OpenVINO Ecosystem

Description

LF AI & Data Day - ONNX Community Meeting, October 21, 2021

ONNX and the OpenVINO Ecosystem

Speaker: Sergey Lyalin

A

Hello, everybody, my name is sergey lehren, I'm working at intel in openwinner project. I will describe the role that onyx plays in openvin ecosystem, connecting us to deep learning frameworks and what services we are providing for onyx in open winner. Let's start, first of all, what is openvideo openwinner is the end-to-end solution for inference across intel hardware.

A

I use the entry and term here because open bin accepts the model in deep learning framework format, just after the training and can give you the result of the inference on various kinds of supported, intel hardware available on your system.

A

Open openview ecosystem includes a number of tools, but, as we are talking about inference first, the central component is inference engine. uh It provides an api to build python or c plus plus to run the inference inference. Engine as a company doesn't have any code for inference. Instead, it relies on plugins.

A

Each plugin is dedicated to run the inference on a particular hardware, so the user has a command api to run, inference on any supported hardware, just configuring plugin name.

A

Besides pure hardware plugins, we also have kind of meta plugins. They combine capabilities of multiple view. Hardware plugins, underneath the goal for such meta plugins is to simplify selection of the device or use several available devices simultaneously to achieve better performance.

A

Okay, let's extend the picture to opposite direction, describing where the model comes from inference, engine lost the model represented in openwindow intermediate representation; it is produced by a separate company, and that is called model. Optimizer model optimizer can load various types of the original models from deep learning frameworks and convert it to ir.

A

Onyx is one of the formats that we can read running model optimizer offline outside the main difference application. We are minimizing time to the first inference, improving user experience and minimizing binary size for deployment.

A

By the way, there are other techniques available in open winner that speed up first influence time, like caching and offline compilation, the important tool that optimizes models for in-depth execution is post-training, optimization tool which allows you to significantly speed up the inference, maintaining the desired accuracy in the same time, with the help of accuracy checker running on your data set.

A

So what is open winner ecosystem, besides components that we have just mentioned? It contains many others like open model, zoo training, extension, deep learning, workbench tools to annotate and work with data sets and even more part of these components, push us beyond just the inference tasks.

A

You can visit the documentation for more information about openwinner and our github repository where all these components are being developed.

A

Okay, let's take a closer look at onyx support inside openvina onyx is not just another format for model representation.

A

Besides this flow, where we can convert onyx with model optimizer to ir just like other file formats, we have supporting, uh we have something that is currently available for onyx only. We can load linux model directly to inference engine runtime skipping all the offline steps.

A

So if you want to use the model out of the box and don't need any preparation, offline skip the offline conversion step in the code. It looks like this, regardless of the language we use to write the application. The steps are quite similar in c plus and python.

A

First create the core object, which is an entry point for the inference api. It allows to read the models and load them to plugins. Here we are reading model from onyx file. After reading, you can do some model adjustment like setting a new input shape or for model inputs and then load to one of the plugins uh to run the inference. In this example, it is cpu device.

A

These steps are identical to the steps that we are doing for normal, open vnir presented here, where model xml is used uh which is openvin ir file.

A

Thus, openvinayar and onyx files are completely interchangeable from the inference api perspective and user can choose whether to convert from onyx to openvinair in offline step, saving time to the first influence in the target application or to load onyx model directly and spend some time for conversion the later unlocks more rapid prototyping.

A

So what we are doing to make openven a good inference solution for onyx models. First of all, constantly growing support for operators. Our main priority is operators from the popular models and models from the customers.

A

Each new onyx operator can be supported in two ways. First, it can be decomposed to a sub graph of existing operations from official open bin operation set. The second way is when such decomposition is not possible or too bulky and inefficient.

A

In this case, we are extending our operation set to cover this onyx operator. The later case implies more efforts, because such operation should be implemented in all the inference. Plugins.

A

The onyx community can help us to grow faster, to build really complete onyx backend by providing feature, requests and highlighting existing problems and the current tonic support. I encourage to create tickets at openvina github to achieve that, we are constantly monitoring such external requests and it help us to make the right decisions in the development.

A

Okay, let's move to another aspect for us: onyx provides important connection to very popular world of pytorch. Models in openvino v depends on torch, onix functionality to obtain serialized models from pythag.

A

If the original model cannot be expressed in the default onyx offset by using torch onyx functionality, you can still serialize such models in linux file and then using open window extensions.

A

The user can write the inference code for such custom operations in open window plugins, as we can read, onyx models directly in inference application without using an offline step. There's model optimizer. You can call open window inference right in your pytorch application to evaluate performance opportunities.

A

The corresponding code looks like this. Here we are serializing the model to a temporary onyx file and immediately after that, we load it to an open winner in french engine, or this code can be modified to avoid storing a model to a temporary file and make the trick in the memory buffer.

A

So that's all I can highlight in this short presentation.