ONNX October 2021 Community Meetup, 21 Oct 2021

Previous Meeting

⏯

youtube image

►

From YouTube: 016 ONNX 20211021 Anton ONNX WG Preprocessing

Description

Event: LF AI & Data Day - ONNX Community Meeting, October 21, 2021

Talk Title: Working Group (WG) - Preprocessing

Speaker: Joaquin Anton (Nvidia)

A

Hello, everyone- I am jaquin anton from nvidia and I'd like to share with you a few updates about the onyx processing working group.

A

The working group was created a couple of months ago, so I will take this opportunity to share what is a problem that the group is trying to address and what we expect to be our roadmap, as you might be guessing by the name of the group. It focuses on formalizing data pre-processing as a part of onyx.

A

Let us try to understand why we need that.

A

The first big problem we encounter when we deal with non-trivial data pre-processing in deep learning models is that there is no standard.

A

Often we see that different frameworks or libraries have different assumptions of what a certain operation is, and those small discrepancies are easy to miss when porting from one library to another.

A

We usually train models in a very different platform than the one that we run inference on and therefore we not necessarily have access to the same tools for preprocessing discrepancies in the implementations often lead to portability issues.

A

Here we can see a couple of examples of such problem on the left. We have a comparison of how a bilinear filter used in a resize operation looks like in two very popular image: processing, libraries, opencv and pillow. As you can see, the results are obviously different.

A

It all boils down to the fact that what pillow advertises as a bilinear interpolation is in fact a triangular interpolation on the center. There is another example this time, comparing the result of tensorflows and pillows by cubic filter.

A

As you can see, the results are again very different, and this time is because tensorflow 1 uses a different pixel center definition. When resampling on the right, we can see differences in the output of a jpeg decoder, with different implementations for the chroma up sampling step.

A

When such thing happens, the observed result is that we see a degraded performance of the model. The second big problem has to do with the availability of libraries at inference time. We often define and implement the data pipeline with python libraries such as pillow, numpy, opencv, etc, and those libraries might not be available at inference time on an optimized environment and must be implemented with what is available here is a roadmap of the steps that we believe need to be taken to address the issues mentioned earlier.

A

The first step is to figure out a way to make data pre-processing a part of onyx. This way we can distribute the data preprocessing definition together with the onyx models in a format that can be understood by the different backends.

A

This will make deployment of models easier than it's today. The second step is to standardize the definition of the various primitives that we use across data pre-processing pipelines. The definitions should be non and ambiguous so that they can be interpreted by the different runtime back-ends producing equivalent results.

A

We will then add and extend onyx operators to enable the implementation of the data pipeline for most networks and, of course this is a very wide scope. So the plan is to first focus on computer vision, applications and then move to other data domains such as audio processing.

A

As an exercise, we took resonance 50 as an example and tried to replace the data pre-processing pipeline with a second onyx model on the left. We see how a full resonant 50 pipeline might look like implemented in python, using pillow and numpy.

A

It includes resizing the image center crop operation, conversion to float transposition and normalization.

A

Now what if we replace the preprocessing function with the second onyx model on the right, you can see a representation of an onyx model built with existing onyx operators and it should produce equivalent results to the phone to the function on the left.

A

The preprocessing model was saved as a non-x file and loaded the same way as we load the network using an independent, onyx runtime session.

A

Then we just run the preprocessing session first and use its output to fit the model. This example shows that onyx models are capable of representing pre-processing graphs it it's still to be determined. What is the best way to incorporate preprocessing to the onyx models?

A

There are some ideas being currently discussed.

A

First, it's important to keep the preprocessing definitions separate from the actual model, and this is motivated by a couple of reasons. The first is that we want to keep the preprocessing part of the model identifiable.

A

This way we can deliver the data preprocessing to dedicated back-ends or add control flows to handle batch formation, and this is just to name a few examples.

A

The second reason is that we might want to seamlessly change pre-processing pipelines, for instance, between training and inference.

A

This approach is also less intrusive to existing models, and the final model could combine the pre-processing model with the network and apply all the necessary optimizations under the hood.

A

The second concern is the different nature of the data that the preprocessing layer handles data. Preprocessing, typically handles data of different shapes, producing a uniform batch that is then consumed by the model as a single tensor.

A

One idea to approach that would be to use a sequence of tensors as an input to the data pipeline and have a special control flow operator that would apply the same preprocessing steps to each of the samples concatenating their results at the end.

A

At this stage, those are just ideas and more experimentation needs to be done before we decide on a particular approach. For the first prototype, we agreed to focus on a set of computer vision networks, namely resnet ssd and masker cnn.

A

Most of the necessary operators are already present in onyx.

A

The resize operator would need to be extended to support other interpolation types when resampling, and it would also be useful to introduce new resize policies so that we can resize to a minimum or maximum shape, but keeping the aspect ratio.

A

We will also need the new color space conversion operator.

A

This is only the first set of operators and we will extend to support other networks after preprocessing is incorporated to the onyx ecosystem and, lastly, this working group is just getting started. We encourage you to join us on our slack channel and on our monthly meetings and share your ideas. Thank you for listening.