ONNX June 2022 Community Meetup, 13 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ONNX SIG (Special Interest Group) and WG (Working Group) Area Updates

Description

00:00 Arch & Infra - Liqun Fu, Microsoft
05:17 ONNX Operators - Ganesan “Rama” Ramalingam, Microsoft
14:33 Converters - Kevin Chen, NVIDIA
23:08 Model & Tutorials - Jacky Chen, Microsoft
31:49 Pre-processing - Joaquin Anton, NVIDIA

A

Good morning, everyone thanks for coming, and there has been two onyx releases since we last made, and the latest one was, is 112 done by ibm. So, let's give it a round of applause. Contributors have many uh had added many new features to onyx and the kpop making are better cheap influence has removed at operator level. A contact lender able to handle impact, not missing optional error. Checking for incorrect permutation was added for transpose chip, influence for consonant of shape is simplified and the shape influence for expand was improved, such that symbolic shape is used.

A

Data and the shape are now utilizing results from data propagation. This last improvement was made possible by extending chip info shift api to expose results from data propagation. It helps high touch exporter to utilize, onyx chip influence functionality and chip influence for local functions was added in release 111 and the further enhanced in the latest. One power release.

A

Onyx generated test, the data for downstream applications to validate its implementation in this current release know that test data is generated with a deterministic offset version by default, the latest since version is used and it is possible to select a offset version other than default if needed, and we have updated all the text data with this latest behavior and because of this deterministic adventure, we are able to validate test generation in rci pipelines.

A

Ci systems for linux release has been changed to many linux 2014, because earlier earlier version is no longer supportive. We have dropped it. Support for x86, 32-bit linux and the release packages for python 3.10 was add, were added to the release. We no longer produce packages for python 3.6.

A

And there are two important releases I did: model composer can be used through creating and combining models out of several graphs and models, and the function builder can be used to construct function. Operators.

A

It is impossible in this presentation to list all the improvements made by contributors. Here is just a sheltered or only shortened. Release initiators are decoupled with inputs such that the virtual converter can reliably convert models across any offset versions and with a few additional bug, fixed model converter has become more stable.

A

The label a section code is modernized that way the type annotation uh showing our aim for the highest coding standard. The modal pressure has been improved to support.

A

In the next, we plan to provide a better support for our mixed releases. We realized that, from our experience that the major risk factor for release, delay was due to breaking changes that were detected by the runtime are building really so to eliminate this. um We will plan to to provide a ci pipeline for the runtime could be packed to automatically detect any breaks, changes while doing a release, and so that early action can be taken, and we will also, you know, continue with our focus on shape, influence, improvement.

A

And, last but not least, we will continue to provide the infrastructure and the architecture, support for operator data processing and the other special interest groups and.

A

B

A

B

I'm rama and I'll be presenting the update uh on behalf of operator, um so um since some of you may be uh new to onyx, let me give a quick summary um of the focus of the operator. Sig. uh The onyx spec, uh as prashanth earlier said, breaks into two parts.

B

One is the ir and its semantics, and the second key part is the set of operators uh that constitute part of the spec and the operators focus on the second one primarily and the operators are organized into domains and there are two domains: the onyx domain, which works on dnn operators and the on xml domain, which focus on classical ml operators, and these are versioned and the sig mostly focus on uh how to evolve.

B

This setup operators are adding new operators as needed and clarifying uh operator specs and making changes to them as and when needed.

B

So let me give a quick update on the changes that were introduced in the last two releases.

B

There are two new offsets opposites, 16 and 17 were introduced.

C

B

The last two releases, so in terms of the new ops introduced, we have grid sample which is used in spatial transformer networks, uh a layer normalization which is widely used, for example, in language models, lectures but uh signal processing, ops where basically uh dftr fft. They have been widely requested and they are used in audio models and uh sequence map, which was proposed by the pre-processing work group, which enables us, for example, to handle uh batches of images, for example, of varying sizes.

B

If you want to pre-process them and convert them into a format required for a specific model like resnet 50 and most of these are functions except a grid sample and a few others, but there's a plan to promote them to be functions soon.

B

And uh some of the existing ops were also updated. So, for example, scatter op was updated to allow support for duplicate indices.

B

But you can say, the values specified for duplicate indices should be summed up, for example, and support and some of the ops were extended to provide support for types such as b float, 16 and optional types and roi align had a minor update uh to adjust for a variation between different frameworks, but the results were different off by half a pixel.

B

uh So yeah, let me uh uh in the remaining uh few minutes. uh Let me uh describe some of the plans going forward and, uh as I remember, uh the key goal for the operator's seg is primarily to have a clear and unambiguous specification and there is room for improvement uh here and we do need to improve their documentation for their spec for various ops and as to the remaining goals.

B

There is really a trade-off here, because on one hand, we would like to have a compact specification that is a smaller set of primitive ops that makes it easier to develop a new backend, for example, on a new hardware and basically make it much easier to implement onyx and new settings.

B

On the other hand, we also would like to support uh the new kinds of models that come along uh even support, encoding, pre-processing and post-processing logic uh in the model itself, and uh this leads to a uh desire to add more ops uh to the spec, basically and and the desire for efficiency also leads to a uh desire to add more complex, composite tops that are coarse grained but can be implemented efficiently.

B

So uh there's a tension here and uh we basically need to strike a balance between these two and uh one of the key uh techniques we use to deal with this challenge is the notion of onyx functions and an onyx function is essentially an onyx op, whose semantics is specified by in terms of other primitive ops so effectively. This function definition provides an executable specification for the thereby improving uh the clarity, reducing ambiguity, but it also provides a default implementation so that it makes it easy to uh build a new backend.

B

At the same time when efficiency demands it, you can build a specialized kernel in specific backends for these ops, so are allowing us to balance this conflicting requirements. Phase.

B

uh So uh going forward, uh one uh of our goal is to reduce the setup, primitive operators, uh basically by uh promoting them into functions so that they have identified around 25 to 30 of existing ops can actually be promoted as functions, and we plan to do that, and one of the interesting new directions we are planning to take is to enable authoring onyx in python so and have a tool that automatically converts the python function into a onyx function.

B

Proto, which is the serialized ir representation for functions in onyx, and the plan is to also allow uh use this as a way to debug using first standard python debugs, for example, the definitions of uh functions and to understand them, and so the plan is to use a subset of python. We call ionic script to enable all of this.

B

So uh let me give some example couple of a few examples. So yeah uh here you see a function, definition in python for the well-known activation function, and uh so uh the plan is uh so so the we define the function in python this way and the converter will automatically generate the cellular function. Definition and, as you can see, in onyx, uh uh the real function definition expands into a lot more operators and is less readable. But the compact specification is easier to author and also to read and understand so.

B

And so this is another example uh that illustrates the use of control flow. So, for example, a dropout which is a standard up in the python spec can itself be specified as a function on top of a random uniform generation and again, the use of control flow makes it uh python. Control flow makes it easier to naturally and compactly specify uh this semantics of these obs as a function.

B

And so this just illustrates the use of uh standard debuggers to understand these function. Definitions too. So, for example, here you have a tie. Example: python definition of an onyx function and um you executed just like uh you would execute in a standard python by creating some inputs and invoking the function, and you can use the debug to look at the values and understand the specs. So.

B

um So uh yeah, these are just some more examples uh for people interested just to show how various existing onyx ops uh uh can be compactly specified uh yeah. I think uh so. uh That brings me to into the end and thanks for coming and please uh do get involved with the operator sig, as shown here, there's a slack channel and we have monthly sig meetings that uh you can join and uh and we welcome your contributions.

B

D

Okay, great um yeah. Unfortunately, I cannot make it uh to the meeting in person, but it's great to see uh you know all the participants here. um So my name is kevin. I've been involved in the honest community for the past couple of years and today I'll be giving an update for the converter, sig um uh and all the work that we've done uh since the last side. We've met so for the outline of this presentation I'll be going over. The converter update uh so first I'll be going over the front.

D

End converter updates, which mainly deal with dll framework into onyx model conversion and for these front end converters I'll be going over the pytorch onyx tensorflow, onyx and sk learn to onyx converters that we have um next I'll be going over the back-end converter update. So these back-end converters mainly deal with onyx model format, um translate them into a back-end runtime framework where actual inference is run on the onyx model and for these backing converters I'll be going over the onyx tensor rt, as well as the onyx tensorflow converter.

D

Next up I'll be going over the road map for the converter, stick and finally, there'll be a get involved slide for those of you that wish to participate in the converter. Stick anytime after this meeting, uh so the first converter I would like to go over is the pi portion on its converter. So the latest release that will be coming out very soon is pytorch version 1.12, which supports onyx exports up to onyx offset 16..

D

So there has been a bunch of new features added in this release, and some of the highlights include the ability to export neural network modules, specifically as onyx vocal functions, and this allows specific back-ends to target these functions specifically for accelerated inference. There has also been a lot of new off support.

D

You can see the full list listed there and we've also added the ability to export onyx before 16 and optional data types, as well as added support for exporting uh quantized models, along with exporting model, uh train and mixed precision with apexel2, so for any users that are working with these new data types or working with quantized or mixed precision models. We recommend you to try out the new pytorch exporter and see the updated workflow into converting to onyx.

D

Finally, we've added support for a10 fallback for non-cafe to back-end and we could and the and any user can see the full list of a10 supported operators in our documentation online.

D

uh Finally, if this piqued your interest, we have a separate talk by bowen later today, at 1 35 pm, where he will go a little bit more in depth about all the changes we've made in part two 1.12 next up is the tensorflow onyx converter and the latest release here is ts2onyx version 1.11.1, which supports exporting to onyx and offset16 and tensorflow versions up to tensorflow 2.8.

D

So the highlights here is that the default offset export version has been updated to 13 and there are a few new ops that are now able to be exported, including tfla batch map, moles, uh t of light map, mall sensor, scanner ad and random int. In addition, uh we've improved uh model export for q2q model, as well as general improvements for export for models trained in ts lite. So again, if you're, using qdq models or if you're using cs lite, we recommend checking out the latest release for the updated workflow into exporting to onyx.

D

Next up is the sk learn to onyx converter. The latest release here is sk learner, onyx 1.11.2, which supports onyx export up to offset 15.. The highlights here is that we've added a new up support for sgd one class svm. We have also a bunch of bug fixes for uh previously supported ops, as well as improved library input performance across the board. uh So we recommend users to update to the latest version or improve user experience when converting scaler model into on it.

D

Next up I'll be going over the back end converters, so the first up is onyx detector repeat so. The latest release here is onyx tensority 8.4 and we support importing onyx models up to offset 17..

D

There are two new ops added in this release, um so now we can import models using shrink and xor, and there has been improved support for nbox fixes for these following off arc max range, cellu, gem and einstem.

D

Finally, we've improved support for dynamically shaped models and we have general performance improvements across the board, so we recommend any new users and recurring users to upgrade to tensor to 8.4 and start benchmarking with, hence rc, to see what sort of performance improvements you could get when running onyx model next up is onyx tensorflow and the latest release here is version 1.10, which supports importing onyx models up to offset 15 and supports tensorflow versions up to 2.8 uh for new off support.

D

uh Honestly tensorflow actually supports the entire offset 15 offspec at least partially, as well as uh supporting onyx optional data types so again for any users with models that are using offset 15 or lower and are using optional onyx optional data types. We recommend using the latest version. I wanted to tensorflow to import your model um and do any new benchmarking there.

D

As for the roadmap, uh the converter stick had a few short term and medium term uh level goals. So for the short term, uh we wanted to improve our uh onyx operator, uh documentation, support matrix across all of the converters, um and this has been done uh over the past year across august converters. So this is a very good improvement for the medium term goals. There are two main things that we want to focus on.

D

The one is setting a minimal offset support across all the converters, um so there has been discussions that um previous offsets are should be deprecated and we are working on formally proposing uh deprecating um offsets less than nine as a starting point, and we can uh move on with the offset deprecation in the future if this goes through.

D

uh Secondly, uh the other medium term goal we have is to improve community driven, tooling, we've seen across a lot of converted repos, as well as across different companies.

D

There are a lot of ad hoc implementations for a bunch of utility tools, such as buzz testing, shape and type inference for honest models, quantization tools and constant voting across onyx models, and we want to work together with the community as well as the um arc and for the group to help start standardizing some of these tools and getting them contributed back into onyx, so that users and the converter users have one place, have a one-stop shop for all of the utility functions that they may want to use.

D

uh Finally, there is a perpetual goal across all all converters to constantly improve um both in terms of buck fixes and improve operator and offset support and for pythor. Specifically, one big item on the roadmap is that um improved support for pytorch models with lazy, tensor and auto grad functions are in the work, so definitely um uh beyond the outlook for that so finally feel free to get involved.

D

So any quick feedback feel free to join us on slack in the honest converters, channel and drop by a quick comment or start a thread for those that want to get involved. A little bit more in depth feel free to subscribe to our honest converter, sig mailing list where we send out um invites for monthly meetings. uh So please subscribe. If you want to get involved in that, and finally, we have a very um a very amazing community across all the converter.

D

Repos, don't feel shy to open up issues or start discussions um across github, uh because we always welcome new contributions and new discussion, and we are very welcome welcoming to any users um that want to get involved in onyx. um So thank you. That uh concludes the onyx. Converter sticks for uh updates for today. um Thank you all for listening.

C

um Thanks kevin, so um I'm the new sick, lead of onyx model, zoo and tutorials. So today update scenes, that's workshop and I'm jackie from microsoft.

C

So um today I will. um We have two parts. So first I will talk about onyx, monozue and then honest tutorials, so um yeah. So before I jump into details.

C

So let me introduce honest: um what is honest model zoo, so rx mono zoo is a collection of pre-chained um and state-of-the-art, like machine learning or different models, um which is were mostly contributed by honest community and right now there are 40 kinds of onyx models and 168 models in total, with different kind of um rx versions, including 35 vision based on these models like image, classification, object, detection, super resolution and also um there's um five models about machining comprehension like bird dbt2, so yeah and for next, for what is honest tutorial so honest tutorial, plays that have um a lot of documents, notebooks, demonstrating how onyx, in practice for um different kinds of scenarios like um different framework platforms and um how they can work with different kinds of device types.

C

Yeah so I'll um talk about talk, more details about model zoopers, so um we have a lot of new quantized models, which is mainly contributed by um intel um thanks um manny from in intel.

C

She contribute a lot of um in integer, eight models in honest model, zoo and including, like both vision, based and text-based models, and one thing I want to highlight is that um like so we improve the test coverage um at this moment like we have a um routine ci in honest repo, like keep um running the latest, honest tracker and onyx shaping burns with old model, zoom models and like right now, like all of them, are passed and there's only some ancient models which is still using offset three.

C

So they don't have the shaping bands function. So we cannot verify in onyx. Otherwise they are all passed by. The latest release, honest 1.2, and next. um One thing I would like to highlight is that um recently we fixed um a lot of broken test data set in onyx monozu. So um right now that we have um verified that all the models most of the honest model, zoom models can pass on its wrong time with cpu per rider and there's one. Only one model failed.

C

Then let me summarize what um um what we did for the improvements since this workshop. So we did a lot of ci improvements because we introduced a more quantized model.

C

So we need to make the ci to um to take the right machines to run the the test for quantized models and, as I mentioned in the last slide, we fixed a lot of broken test data set and also we collaborate with hacking based team that we um like accomplish a radio app, um which is a python web, app to demo mercenary models and I'll show you in the next slide.

C

So um so radio is a web interface um and then we like host our smart zoom models in hacking phase space, and then we use honest runtime as back-end engine to influence the accurate result and then finally, it will show a great um result in on the web.

C

So in meanwhile, for honest model, zoo like we can also like, have a link to redirect, um like certain rx module models to hawking space, and you can show some demo there and um in they have a tutorial like how to do it in their gradual um websites and in microsoft. We also, um I have also just um published a blog in our microsoft, open source cloud.

C

Next like this, so here I will show a quick demo to you, yeah so like, as you can see it's a website. So after we run some easy python code, we can just upload some image on the left and from the right. You can very easily okay, the yeah.

C

The website will tell me: okay, there's a ted um type cat there and the only thing we need to do just write like a couple of um easy python code here, like do some image processing and then just create the image to honest run time and other strong time will influence the output result for us and it the gradient will show the result um on the web and also they provide some community functions that you can make a poor request or starter discussions about this app.

C

um Next step, please yeah thanks.

C

So look at some um road map about zoo, so, um like kevin mentioned in his slide, we want those like upset um three models, because they even cannot be run by ort and they don't have shaping business function in onyx and also we will keep improving um our ci's to vary by honest, honest model, zoo models and also we like looking for more models contributions, including quantized models and perhaps there's currently there's no training exam model stairs, but I pre, I think we can like have a first um training example models in the future yeah.

C

Then I will talk about honest tutorials, so yeah for online studios. Mainly we um like, we finally introduced the c idea, because there's there are a lot of dedicated urls so like right. Now, if you um publish a new notebook or document there, we will in the ci it will automatically to to um if it will validate the urls for new prs and also that we are weekly to go through the whole wrap pose to check all urls there.

C

Just in case that some url is dedicated, and also we run some python formatters to clean the notebook and yeah for the roadmap. I think here is just like. We want to keep polish, the old um tutorials and in the future I I think we will prefer like uil um redirections to other tutorials, okay yeah, so that that's why I I want to like um say, welcome to contribute and actually the contribution of the new rx module.

C

Is it's quite easy, like you can first get the onyx model like it can be produced by any converter, and then you can use or the test um test data set, including input output and then just write a readme and pass the ci that I onyx.

C

Run by ort impress with cpu and get the right result. Then your pr is ready to go so yeah. If you have any topics you want to bring up, feel free to join us on slack, there's a channel called like um onyx monozu and please help us to review the prs. There are a few openings and, most importantly, I I was looking for more volunteer approvers for sick model tutorials.

C

I think it's a very good opportunities for you to be visible to honest community and also it's a good chance for you to learn, onyx, honest model and ort yeah. That's all. I have thanks.

E

Hi, my name is joaquin anton, and I would like to share with you what we've been working on lately in the onyx pre-processing working group.

E

Let me recap what the goal of the group is and then I'll show you the work we've been doing lately today. Onyx models typically expect the input data to be already pre-processed.

E

The complexity of that pre-processing pipeline may vary depending on the application. For instance, some typical steps in a computer vision, application are cropping, resizing, normalizing and transposing to a planar layout.

E

Those preprocessing steps are usually defined, vaguely or expressed in python with some particular processing library.

E

This lack of standardization creates a problem. Different data processing toolkits can have different definitions of apparently the same operation. In this example, we are comparing an imagery size operation for two popular libraries, pillow and opencv.

E

As you can see, the results are obviously different. It all boils down to the fact that pillow applies an anti-aliasing filter when down scaling while opencv does not.

E

Sometimes small differences in the data pre-processing pipeline using during training and inference can result in a drop of accuracy.

E

The second obvious problem with the current approach is the lack of availability of the libraries on the platform we are deploying.

E

The goal of the group is to standardize the definition of those data pre-processing operations so that they are not ambiguous and include those in the onyx model. This simplifies deployment and avoids the potential accuracy problems we mentioned.

E

Since the group was created, we've been working on different topics that will help us reach our goal.

E

Some of the work is related to the general infrastructure that will allow us to include the data preprocessing in the model, such as being able to combine preprocessing pipelines with existing models or being able to batch process samples, as we typically do at the pre-processing stage.

E

Another line of work is related to the extension and creation of operators to enable our first end-to-end model, which we chose to be resnet 50.. Let's look at those items more closely.

E

We proposed a set of composition, utils that allow the user to merge two models or graphs by connecting some of the outputs in the first model to some of the inputs in the second model, the user can also decide to add a prefix to all the names in a given model to avoid name collisions when merging.

E

The second topic was batch processing, a typical case when doing data preprocessing is to have a batch of samples that are not uniform in shape. In onex, we can represent that as a sequence of tensors.

E

We do want to process each of the samples by applying a series of pre-processing steps. The typical result is a batch of samples that are now uniform in shape before feeding the network. We concatenate all those samples to a single tensor.

E

To enable this pattern, we propose the sequence map operator sequence. Map is a generic sequence processing operator that can apply a graph to each element in an input sequence producing a sequence. As an output, we can use sequence map to apply our preprocessing graph defined on a one sample basis to a batch of samples.

E

Also, we are currently investigating ways to tag the pre-processing part of the model, so it can be later identified by back-ends. The current approach is to use a model local function as a single entry point for the preprocessing subgraph and mark it as such, via metadata properties,.

E

To enable our first example resonance 50, we are proposing a couple of extensions to the resize operator. The first one is an optional anti-aliasing filter to be applied when down scaling the input. This is necessary to match the behavior of some popular image. Processing libraries such as pillow.

E

The second extension is a keep aspect: ratio policy that changes the interpretation of the target size. The policy can be to stretch ignoring the original aspect, ratio, which is currently the default behavior or it can be to treat the target size as either a maximum or a minimum size, while keeping the original aspect ratio.

E

The other operator we need for our first example, is a center crop or path operation. The operator we are proposing offers a convenient higher level abstraction and it's implemented as a function that relies on the existing pad and slice operators.

E

And that's all from us: we encourage you to join us on our slack channel and our monthly meetings and share your ideas. Thank you for listening.