Cloud Native Computing Foundation Cloud Native Wasm Day North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: High Performance Neural Network Inferencing Using WASI NN - Radu Matei, Microsoft

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: High Performance Neural Network Inferencing Using WASI NN - Radu Matei, Microsoft Azure

The sandbox environment and portability of WebAssembly provide an ideal deployment target for neural network inferencing. But while inferencing engines can be directly compiled to WebAssembly, the current limitations around parallelism and hardware acceleration present real limitations to performance. This is the main reason behind WASI-NN, a new WebAssembly System Interface (WASI) specification that enables near-native performance for neural network workloads with hardware acceleration on a variety of devices such as CPUs, GPUs (and in the future TPUs or FPGAs). In this session, we will present the WASI-NN interface proposal and detail the current state of two specialized implementations -- for the OpenVINO and ONNX runtimes. Finally, we will demonstrate an end-to-end process for machine learning, from building and training neural networks using PyTorch or TensorFlow, all the way to running the same neural network and WebAssembly module from different platform architectures and operating systems using the Wasmtime runtime.This project was developed in collaboration with Andrew Brown, Intel.

A

Hey everyone: my name is radu and in the next nine and a half minutes we're going to talk about webassembly and machine learning and we're going to have a quick look at wes dnn and the new proposal for neural networks in webassembly. I'm a software engineer. I work at microsoft, azure if you've seen a bunch of deus labs projects mentioned today. I work on the team.

A

I also do a lot of work with the bytecode alliance and the cncf and tldr I'm interested in everything that has to do with webassembly machine learning, distributed systems and most often the combination of the of the three I'm a maintainer or contributor for waggy crosslit, hippo bindle, the http extension to azi uh and wes cnn.

A

uh This talk and a lot of wes cnn has been uh contributed by andrew brown and mink- u soon from intel. So this is uh a shout out to andrew who's. I think, on a sabbatical hope, he's enjoying his time off uh we're going to talk about a little bit about webassembly and wazzy and then machine learning in webassembly.

A

I I paid very close attention. No one actually mentioned that webassembly is neither web nor assembly today, and that is absolutely a given at any webassembly conference. So I think a lot of people talked about what web assembly is and what webassembly isn't, but, in short, it's just a binary instruction format for a web for a stack-based virtual machine, but in short, it just lets us run cool stuff on the web. That's not javascript only, but on top of webassembly it's webassembly has no mention necessarily of the web.

A

You can run webassembly outside the browser pretty well and that's where, where wasd cam comes in it's a capability oriented api and essentially it's designed to standardize the execution of webassembly outside the browser and what webassembly and wazi give us outside the web is a portable lightweight, fast and secure way of executing semi-trusted or untrusted code and, if you're, just getting started or interested in getting in depth with uh anything that has to do with wazi.

A

You have to read the lin clark's code, cartoons they're, probably the best resource for anyone getting into webassembly in wazi.

A

Why would we want to run machine learning in wasm? Well, mostly, for the same reasons, we want to run anything in webassembly. We want the portability that webassembly gives us. We want the flexibility to run the same webassembly module anywhere and specifically for machine learning. We want to run the same machine learning model across different architectures.

A

We also want to have language agnostic access to different machine learning. Runtimes and more importantly, we don't want to keep re-implementing the same cpu instructions and machine learning operators, but there are a couple of problems that makes uh that make running machine learning in webassembly today, not ideal. uh First and most obvious is we don't have gpu access in webassembly, runtimes, there's also no multi-threading or other hardware acceleration.

A

There are obvious missing, missing cpu instructions from webassembly that probably no one's going to add for each machine learning framework and then in general, deploying machine learning in production. It's just difficult.

A

So this is where was enn comes in the web, the wazi neural network proposal, it's a way of allowing us to load a neural network model into a runtime and essentially run inferencing on that particular neural network model. It's framework and format agnostic. So if you have an implementation, you can run it for pytorch tensorflow or any other machine learning, framework or model.

A

You want to run it for it's much faster than running inferencing in pure webassembly, and it's really portable, we'll see a recording of the same machine learning model and neural network and runtime on across raspberry pi's and intel cpus and amd and gpus as well.

A

The current implementations are for the openvino model and the onyx runtime, and essentially it's a pretty simple api. It lets us load up an opaque byte array. As a neural network model, we can initialize the execution context and bind some inputs as tensors.

A

uh We can compute the inference and then get the output sensor using a specific get output result. The api is written in widex, which means for anything that you have either code gen or you want to manually, write bindings for widix. You can do it, it's not fun, but in essence, if you want to write the low level implementation, all you have to do is call the four main api functions that was enan gives you and you can implement that for any web assembly runtime.

A

The interesting part is, uh and here we're gonna see uh the same neural network and the same runtime and the same webassembly module run across an insult macbook using the webassembly runtime first, using the wasm time.

A

I hope that's visible.

A

Slides, okay, I'll just let it run. Essentially it takes the same machine learning uh model built with pytorch that does image inferencing and now it runs it. On a on a raspberry pi, the great thing about this: you can cross-compile webassembly from anywhere and I mentioned earlier. We essentially can take the same model and the same runtime and run it in something like wacky.

A

The one of the important things is it's more performant than running pure web assembly, so you can have a look at the uh at the benchmarks.

A

The actual inference is really really fast. It's around two milliseconds tensor preprocessing is the slower part it's around ten times slower than uh the actual inference. The other part is you still have to run pre-processing in non-python, which is not ideal.

A

One of the other really interesting proposals we've seen recently is was a parallel and one of the ideas is running. The pre-processing using wazi parallel.

A

You can find most of the implementations and resources about this on github. Any questions.

A

Yeah, so I'm just curious: uh you mentioned that it's uh faster than running standard web assembly. Why is that? Is because you've been able to strip out some of that stuff and you have a better execution.

A

Go ahead and repeat the question, uh the question is why it's faster than pure webassembly? Essentially, uh there are ways of running inferencing in pure webassembly and the way that usually done is a re-implementation of the machine learning framework that's compiled to webassembly, where znn, what it usually does is. Essentially, you can offload the execution and the inference to the runtime and the runtime has access to hardware, acceleration, gpu or anything like that.

A

So what what you do is through the oasis n api, you pass the model bytes, you pass the input as a tensor input. The runtime does the inferencing using gpu access, and then you get the tensor back.

A

Okay, uh thanks radio, that was great. Thank you.