Cloud Native Computing Foundation Cloud Native Wasm Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Machine Learning with Wasm (wasi-nn) - Andrew Brown & Mingqiu Sun, Intel

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Machine Learning with Wasm (wasi-nn) - Andrew Brown & Mingqiu Sun, Intel

Web Assembly provides an ideal portable form of deployment for machine learning models. Although a whole machine learning framework could be potentially compiled into Wasm, special hardware acceleration is often needed in order to be performant, which is difficult to do within Wasm with the limited support of parallelism. WASI-NN is a Web Assembly System Interface (WASI) module that allows for near-native performance on a variety of devices such as CPU with AVX512, GPU, TPU, and FPGA. In this talk, we will present the WASI-NN interface design and a reference implementation in Wasmtime. We will provide a step-by-step guide and a demo on how to build all relevant components, and write an image classification application with sample code in Rust and AssemblyScript. We will also demonstrate the performance advantages of WASI-NN over alternative approaches.

A

Hello: everyone, uh my name, is min tu sung, I'm a senior principal engineer at intel today. We're gonna talk about machine learning with web assembly, together with my colleague andrew brown.

A

So here's our agenda, we'll start with introduction to wally and the biker alliance, then we'll get into some of the design guidelines for dnn uh such as you know, loader versus builder, and the machine learning model and the virtual I o type andrew then will start talking about how to write a ydn application in rust and assembly script and as a reference implementation architecture on one time and then he would then do a uh image classification demo with what dnn and then followed by performance advantage analysis of our dnn.

A

Then we end the presentation with call for action.

A

So what is wazi why this stands for web assembly system interface? It is a modular system, interface for web assembly and with focus on security and portability.

A

uh So the body that's driving. The specification is a subgroup of the w3c web assembly. Cg and ydnn is a worthy module for neural network and currently we're in phase 2 with the wazi community group.

A

So let's talk briefly about the bico alliance, which is a open source community dedicated to creating secure new software foundation. You know built on top of a technology such as web assembly, web assembly system, interface, interface, type, uh module, linking etc, and the weather implementation is definitely important goal for bicol allies and the first reference implementation for ydnn is built on top of wasm time, which is a michael alliance project.

A

So, let's dig deeper into you know why we want to do whatever then. So. The first question we want to ask is that you know why do we need the webassembly for machine learning and it turned out that people do a lot of machine learning training and then they typically deploy them to a variety of devices with different architecture and operating systems, and that's where the web assembly comes into the picture, because it provides the ideal portable for format for deployment of those models to those devices.

A

So the next question is that you know why do you need a web assembly system interface right? uh The reason is very simple, uh because typically uh machine learning workflow requires a lot of performance and the special hardware support acceleration are necessary.

A

uh You know, example the avx 512 for cpu and access to gpu, tpu, etc, and the current web assembly spec only supports a limit uh set of a parallelism. You know, especially with this 128-bit cmd, so that's clearly not sufficient for near native performance for those machine learning workload on modern highway.

A

So later in the presentation, we'll actually demonstrate 7 to 32x performance improvement. With this ynn approach that we are proposing over a pure web assembly approach.

A

Let's take a look at the web assembly, runtime environment and the use cases associated with each of those environment, so there are three different type of environments for web assembly to execute. So the first one is the standalone environment.

A

Where you know a web assembly application run on top of a standalone web assembly, virtual machine which called into the wazi software stack for system apis, so typically in our standalone application or cloud applications such as the content, delivery network function, service, envoy, proxy type of cloud middleware and the resource constraint environment such as iot or embedded environment.

A

You know those are the typical applications that will uh make use of a standalone web assembly virtual machine. uh So the second one is, you know the current mainstream browser environment.

A

Where that you have access to a javascript virtual machine where the the a web assembly virtual machine is a component of it and have access to web api through javascript environment.

A

That's where the web and then you know, api lives and all the browser, application or pwa will make use of this environment. So the third category is the node.js environment, which is a reuse of the va javascript engine in the standalone environment, and it would make use of a web api as well as a wazi api.

A

So this is a case that when you don't have a lot of resource constraint and you especially when you need to have both javascript- that wasn't web assembly support- and this will be an ideal situation for those type of applications.

A

So, let's take take a look at some of the design decisions we have made for ydnn. uh In this case you know loader versus builder api. So when we started designing uh what was the nn? You know those are the two options that we could take and then we worked really closely with the webinar team and the you know they went through similar exercise.

A

uh So in the end we decided to take the model loader api first and push out the build api to a second phase. So the reason for that multiple flow- and you know influencing, is obviously the main machine learning use case and that's our initial focus. So a load of api is really an easy way to support influencing in terms of future completeness.

A

uh You know, machine learning is still evolving rapidly, with roughly 20 growth of new operation each year, so a builder api will probably take multiple years to be.

A

You know usable and roughly functional complete to cover all the major operations versus a loader api which we will treat a model as opaque object, and we can just define operation related to the machine, learning loader and get done with it. So that's a lot easier pass to to market and the loader api is a much simpler api with a better ip protection opportunity and the next slide. We actually talked about how we treat the machine learning. You know model format so that we can make the machine the the main web assembly program.

A

You know framework and the model format, agnostic, and it's also very easy to support a variety of devices. You know cpu gpu, fpga tpu for the loader api, so.

B

A

uh The model builder api- you know the main advantage of it- is that you can provide like operation specific acceleration. uh You know this is very useful for some machine learning framework that the legs support.

A

You know, buy some model converters or buy some back-end machine learning engine, and in that case you know, if you want to compile that specialized machine learning framework completely into web assembly and have some specific operation accelerated by the hardware, then the builder api will be very useful and that we think we can look into supporting that sort of usage in the phase second phase of world dna and we could potentially leverage the work already done by the webinam team.

A

Let's talk about machine learning model and the virtualized I o type. Our view of a machine learning model is a virtualized I o type, which you know you think of a general y direction.

A

This type is just like a media type with its own data in the metadata and the defined operation associated with it, such as the load.

A

So the approach we take is to you know, to push the mapping to actual implementation to the edges. uh That way, we can keep the main web assembly program portable and the framework agnostic. So there's not really two approaches to to this idea. uh One is that we can perform a model conversion before we do anything else to whatever accept the format for this target platform or, alternatively, we can let the web assembly virtual machine to dispatch to different machine learning backend.

A

uh You know, based on metadata, available associated with this machine learning model, so it could be. You know openvino, tensorflow or onix in this example.

A

So that's our approach to the model uh machine learning model and the next uh andrew will cover how to write one in an.

B

Application hi, my name is andrew brown. I'm a software engineer at intel. Let me tell you a little bit more about the wes enn api. What you see here is the widex specification of that api. It's a five function, api, including things such as load, initializing and execution context, setting inputs, retrieving outputs and, most importantly, computing.

B

The inference using the loaded graph.

B

Here's an example of those bindings in assembly script. What we're trying to do here is make it a lot easier for users to use wazion and from their webassembly applications.

B

Here are the same bindings, but in rust you can see that the same functions are exposed, but you have higher level constructs which should make it a lot easier to use.

B

Though wesleyan is not tied to any specific webassembly engine or machine learning framework, we did have to start somewhere, and so we used wasmtime as that engine and openvino is the machine learning framework.

B

This diagram shows you all the various components in our implementation, starting on the left, you'll see the user application code which, when tied together with the wazing and bindings, can get compiled down to a webassembly file that webassembly file will make use of tensor inputs, as, for example, from an image and the model file or files and we'll pass those on to the engine.

B

In our case, wasn't time wasn't time provides the wazing n implementation, but when it comes to the machine learning heavy lifting it proxies calls down to openvino, which will execute them on a cpu or gpu, etc.

B

These middle boxes can be swapped out the engine and the machine learning framework could be swapped out for different implementations.

B

Let me switch over to my id for this demo.

B

Okay: let's look at an example of classifying this image of a pizza inside wasn't time with wazian in enabled from assembly script. So the first thing we're going to need is a version of wasm time with wazzy nnn enabled that'll look something like the following.

B

What I'm doing here is I'm compiling wasm time with the wasan feature and using some pre-installed openvino libraries as the backing implementation.

B

Next, what I'll do is, I will place wasmtime on my path for convenience and I will tell my path to load up the paths for the openvino libraries.

B

Let's go back to our example.

B

So I've written this assemblyscript code and you can see here that it's using the high level api, the the bindings that we provide in this library, it'll load the graph, both the graph description and the graph weights attempt to use the open, vino implementation on a cpu initialize, the execution context load up the tensor compute the classification and then with the output tensor. It will sort those results and print out the top five.

B

So to compile this assemblyscript example into webassembly we're going to need some tools which are provided in this repository and we will run as build this is this uses the assembly script compiler, to compile the example, I just showed you into webassembly.

B

Let's take a look at uh what it generated here.

B

So if we look at the imports for the optimize.wat file, you can see that it's using ynn, as well as some other wazi apis for, for example, for reading files.

B

Other things that show up in the uh build directory uh are an untouched version of that wasm file and both wasm and wap versions of these uh of these compiled artifacts.

B

I've placed in here the alexnet weights and model files, as well as the image and image converted to a raw tensor, we'll use those in our example to as inputs to the classification.

B

Okay, let's run the classification.

B

So what it's doing here is loading the graph uh files this uh helper function read bytes is probably not as efficient as it could be. I believe it's reading each byte by byte and then we'll see it set up the execution context and actually run the classification okay. So now we see that it's printed out the top five results for the raw tensor that we passed in of a pizza.

B

But what do these mean? Let's take a look at this classes, file that I also place in our build directory and we'll grab for 963.

B

We see okay 963 corresponds to a pizza, so it is identifying a pizza from the image I showed you earlier.

B

So this would be an example of using the assembly script bindings to run wazingnn programs from within wesenton, after proving to ourselves that we could make this work. We wanted to see what are the performance benefits.

B

The intuition was that web assembly couldn't make full use of of the system capabilities, whether that was longer cmd or or threading, or special machine learning instructions, and so to do that, we took as a starting point some work that marat duken had done to run tensorflow models from within the browser.

B

We called that approach, the wasm exclusive approach and we adapted it to work within a standalone engine node.js on the right side, you'll see the wazing n approach, which uses a different engine wasm time and a different machine learning framework openvino.

B

But what we're trying to do is compare approaches here, not necessarily engines or machine learning frameworks. We are trying to use the same machine learning model and the same input, which will then pass down to the machine learning framework.

B

So we ran some classifications using both approaches. What you see here is a chart of us running uh mobilenet classification a thousand times and measuring the average inference time on both a wasm exclusive setup and a wazing n setup. You can see that even across various platforms, the wezzing and approach is significantly faster.

B

We thought oh, this is probably because you know threads.

B

See a significant speed up. We think this is because of longer widths, md and optimizations. That openvino can provide just to make sure that our approach made sense. We tested a more complex model so using the inception model we again ran a classification a thousand times and took the average inference times, and you can see that uh still the wasn't in approach uh is significantly faster, even when single threading uh the the implementation.

B

So we took as a summary that uh there's performance being left on the table and that uh the wazing and approach uh to using the full system capabilities is necessary to get the best possible performance. You can see speed ups of up to 32x from the wazing approach, and we hope that this motivates you know adding adding new engine implementations of wazing in and adding new uh machine learning framework implementations to those engines.

A

Finally, call for action: if you are machine learning practitioner, please go download watson, time time and start using world vienna.

A

Tell us what you like what you don't like what kind of improvement you would like to see if you are interested in the worthington proposal in terms of spec or architecture, please engage us in the worthy community and to help us drive this proposal to final approval.

A

It's a lot easier to make changes right now then later, as this proposal is in phase 2 right now, if you are a web assembly virtual machine implementer, then you have this opportunity of providing a additional reference implementation for what dnn you know own your own virtual machine and perhaps with different backends such as tensorflow or onyx, or anything else.

A

So that concludes our presentation today. Thank you.