ONNX June 2022 Community Meetup, 13 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: An Overview of the PyTorch-ONNX Converter

Description

This session will present an overview of the PyTorch-ONNX converter, its implementation, and recent improvements to support a wide range of models.

Bowen is a software engineer working on the PyTorch-ONNX converter. He's contributed over 400 pull requests to PyTorch since 2018. He's also contributed to ONNX and ONNX Runtime. https://github.com/BowenBao

A

um Hello guys um I'm bowen and I'm a software engineer working on the pi torch, onyx converter at microsoft. So um today I'm going to give a briefly overview of the converter, basically what it is, what it does, how it does it um some interesting features and the future road map?

A

Okay. So here's the uh like brief overview of the architecture and flow of the converter, so um I think throughout the slide, throughout the talk I will use the word export and convert interchangeably.

A

The converter lives inside pi torch. It's part of the pytorch package. Its source code is in the pythons repository under github. So to start um you have a pi twitch model. You have some sample input and together you pass it to the torch, onyx export api, um the first step we utilize the torch, tracer and torque script to convert the pi switch model to the torch intermediate representation or what we call torch ir.

A

So the second step, the exporter takes the torch ir and through various passes and graph optimizations converted to onyx ir and as a final step, it serializes it to the onyx model in a protobot file.

A

So yeah in this talk I'll try to not get into every tiny bit of detail. However, in the footnote, there's always links to more information and documents, so yeah feel free to check it out if you're interested in certain topics.

A

Okay, so let's start with a very simple and basic example of so this is how it looks like to export an alexnet, a single call to the export api, and you can use the netron tool to visualize the onyx model.

A

So the arguments in the box are optional, but you can provide them if you want to export your model with dynamic, dynamic input and output shapes.

A

Okay, so let's move on to some of the interesting features of the converter, um so first is quantitation. Recently we added support for exporting a quantized model.

A

The quantized operators will be exported in qdq format, so, if like supported by backhand, it can be fused and run as a quantized operator at runtime.

A

Stability people can write models with the python control flow, the loops and ifs, and these are captured in onyx model as the onyx loop and onyx if nodes, together with a composition of other nodes as their subgraph.

A

So one of such example is the export of the autoregressive transformer model. It can capture the entire beam search and or like a grid. Research algorithm inside the single model.

A

And, of course, there's other ways to do that, um for instance, if uh in some cases you might already have a specif specified kernel in your favorite back end, and in this case you might want to capture a subgraph of the pi torch model as a single onyx node in the onyx model.

A

So this is when, like the custom ops coming, there are various ways to export as custom ops. You can write a custom, torch, autograph function and define the forward and backward routine in the static method for and backward respectively.

A

There is additional static message, symbolic, which basically tells the exporter how this custom function should look like in the exported onyx graph.

A

Another scenario would be that you already have a pretty well-defined pi torch operator, be it a custom, pi twitch operator or a standard python, trafficker that somehow the explorer has not yet supported.

A

So in this case, you can utilize this api to register a custom symbolic function um so to tell like exporter, basically how to export it. um And of course you can write it as either exported as standard onyx, ops or like any custom ops in a custom domain.

A

So next I want to talk about onyx local function,.

A

So uh I want to first introduce a little bit of the background and motivation um a lot of times. uh The common uh complaints we receive is for the exported onyx model is that the nodes are all flattened, so in a sense it does not like reflect the hierarchical relationship between layers and modules, so, for instance, for this lure norm since I think back, then it was not a standard onyx op.

A

It was exported as a composition of different nodes and they are in some way sort of like flattened uh mixed up with other nodes in the onyx model.

A

So this makes it like visually uh hard to figure out what was what back in the pie switch model, and it also it's kind of hard, sometimes for back-end to do a fusion and optimization.

A

So what this local function feature does is that now it allows you to specify a certain torch and a module class or.

A

um This node has type function and the previously seen subgraph or the composition of ops now are stored as the function body inside the onyx model locally. So this provides a means for back-end to still be able to run this node or model without like having a specified kernel.

A

Okay, a mixed precision so um yeah. We support both torch, autocast and uh nvidia apex mp. um There's no really like uh extra step needed. You can uh export the decorated model um as how you would with any regular model, and you would have the cast nodes inserted.

A

Okay, so let's talk about.

A

And diagnosing failures, so we want to improve and improve that experience and we want to provide more tools to help make that process easier.

A

And, of course we will keep on focusing the support of more and more models. Export out of the box.

A

Okay, so to wrap up. These are a handful of useful links. There are links to documents of different components. There are links to useful tools and link to a place where you can file issue that is related to exporter.

A

So, thank you all for tuning in and I hope this talk is helpful or interesting.