Cloud Native Computing Foundation Cloud Native Wasm Day North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Paving the Way of WebAssembly in the Intersection of Machine Learn... Shivay Lamba, Mritunjay Sharma

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Paving the Way of WebAssembly in the Intersection of Machine Learning and Cloud Native - Shivay Lamba, Layer5; Mritunjay Sharma, Hackerrank

Today, Machine learning can be seen in literally every aspect in life. Machine learning has become more than a buzzword. It is used in almost all fields : from education to cloud native, from Deep Learning to DevOps. The lingua franca of machine learning is Python. However, Python itself is quite slow when it comes to computationally intensive algorithms. On the other hand, Rust could be 25x faster than Python for machine learning. Furthermore, the language of the web is mostly JavaScript. While building machine learning solutions for web applications, performance of the machine learning models is really important and that is already a lot of complexity which often results in poor performance. Rust functions can be compiled into WebAssembly bytecode for runtime safety, cross-platform portability, and capability-based security. Developers can then access those functions from an easy-to-use JavaScript API in the Node.js environment or deployed in cloud-native technologies like WasmEdge. This talk shows how you can integrate machine learning models written in Rust and executed in Web Assembly can be integrated with Nodejs and cloud-native WasmEdge to power machine learning based web applications.

A

Hello, everyone welcome to the cloud native awesome day in north america. I hope everyone is having a great time at kubecon and at all the co-located events. I am myself shivai lamba, I'm currently a contributor and mesh meet at layer 5, which is a service mesh, open source community and has some of its project under the cncf landscape.

A

uh Today, I'll be taking over the tutorial on how we can engage web assembly in the path reading to machine learning in the intersection of machine learning and uh cloud native, and with me, I have nitinjay, who will be also explaining some of the concepts it's over to you.

B

Thank you so much shivai and thank you so much to everyone who is here. uh I hope you all are doing well and taking good care of your health in this pandemic. uh Myself, nathan, j, sharma and currently I am a final year- computer science, engineering student in india, as well as I I'm an sd intern at hacker rank having been contributed to many cloud native projects like communities uh built backs.

B

uh I have uh regularly try trying to build more and more uh helping empowering communities and today, with this session, we also hope that we are going to learn something amazing from all of you and help share the knowledge that we have uh so moving on to the discussion that we are going to have about today.

B

The first thing that we wanted to discuss is uh like the lingua franca machine learning is python, of course, but we did not want to do a versus kind of comparison between python and rust, because python has its own importance and nobody can take its place from the ecosystem of machine learning.

B

But here we are today discussing a relative uh study, a comparison between both python and breast and if we observe a few points uh where python uh can like behind rust is uh one, the fundamental reason is was that trust gives better performance and why it is better performance because it is compiled directly into you know: machine code. There is no virtual machine or interpreter required between the code and the computer. So that is one great advantage. That rust has another important advantage of rust or over python.

B

Is that the invest thread and memory management like rust, might not have a garbage collection system like python, but the compiler uh kind of uh it being more a compiler type of language compiled language?

B

It provides better memory, secure memory, memory, reference takes security and thus saves us from other hazardous or irregularities.

B

Another thing is that rust can be compiled into web assembly bytecode, which is going to be one of the most fundamental ideas of our discussion today, and I mean this has been also been observed in various studies and as a matter of fact, a study by ibm already highlights how rust and web assembly together have resulted into 12 to 15 and 15x performance gain over not only plane nodes but also 25 x performance gain from python. And if we look at I mean, there's one library invest, that's called linfa and what is it in spirit?

B

It is similar to sky kit, but it it is written in best. And if you look at this graph, if you look at this graph, then we find that linfa on address web servers handles 25 times more requests per second than sky kit learning, seven times more than lympha a python wrapper on a python grpc server. So this is another. uh You know a result which we can figure out to tell like how we have observed that machine learning, which is uh right now, a hot buzzword in the industry.

B

Everybody is learning machine learning and they are trying to leverage its uh algorithms to build some cool algorithms and products. But if you look at highly standardized algorithms and how to optimize them, then we can definitely try to figure out and see how rust and web assembly can play a major role in them. And today this talk is something about that.

B

Now we will be wondering how web assembly comes into picture, so one of the most important things about web assembly is that web assembly is a compiled target that effectively provides. uh You know, uh executables that run at native speed and that helps in extremely small and efficient containers uh which are virtually anywhere and with the added benefit of being highly highly highly secure.

B

So if you look at web assembly now, they provide highly secure high performance machine, independent byte codes, and another important aspect of web assembly is that it is compatible with multiple languages, like c plus plus rust, and it allows for them to have a compilation packet and since one of the things that we just not discussed that it is a binary instruction format, it enables native decoding making it faster than comparative runtimes.

B

Now. Another important aspect that we are going to discuss is like, of course, we are going to talk about machine learning. How is going to be?

B

You know simplifying our machine learning model conversion deployments, but we need to know that how this security and high performance features of web assembly uh are actually implemented or or or are connected with cloud native and web both and how uh is cloud native evolving with assembly. So not we're not not uh not wasting any time more. Let's move on to that slide so connecting voicem to web and cloud native port, so web assembly is expanding its space and not only in the arena of machine learning, web or cloud.

B

In fact, even in cloud native, so having a it has been already approved by w3c, so it is a language of web only now, but and and thus it's supported by compatible browsers, like mozilla chrome, but it has been been being efficiently leveraged by cloud native technologies and that's why we have cncf sandbox, which is ads and person cloud which are utilizing the strength of their assembly.

B

They are getting connected to the cloud native and web web assembly intersection.

B

Now we, but before we come to cloud native, because that that's going to be a another discussion, we we also need to check how it has also been been. You know playing an important role in connecting to the web.

B

We can see this fact from this uh single point itself like if we see how does this performance web assemblies, security and portability and javascript's ease of use, if they all combine how they can not only result in faster applications faster build times, but also can help us drive us things at a more secure at a more secure space.

B

And how can this be done like this can be done like like the host app can be of node.js dot? Web application written in javascript, which makes web assembly function calls, while the web assembly byte code program can be written in rust and which can run inside the boss image run time. So, if we see in these two points itself, we have covered like the intersection, the intersection of web assembly with rust with web and how it is making the web application arena much more secure and faster.

B

Now, bringing wasn't uh to cloud native wasn't was a major cnc, a project that has enabled serverless functions uh to be embedded into many software platforms. And today, if we are going to like see like like how cloud native is evolving, it's not just verse image. There are many other projects like person cloud. In fact, many applications built on top of communities that are going to leverage web assembly and if you look at this table like this, is a comparison.

B

So this comparison comes from mortis captain from the university of tilleberg published an article comparing web assembly against docker in machine learning, and he ran interference in both docker and web assembly and located as we can see in the results that web assembly is about five times faster and ten times smaller than doctor okay.

B

So this arena is huge, it's it's expanding, it's growing and it's ever evolving and there is a lot of future that we see here and today, if I suggest so, then I have my friend and uh my co-speaker shivai, uh who will be showing us a small tutorial uh that will show the intersection of web with cloud native and machine learning so over to shibai.

A

Thank you so much pretends there for that amazing information regarding the usage of web assembly inside of cloud native and web ecosystems. So as web assembly is increasingly being used in the cloud, it has now sort of become a universal runtime for cloud native applications because of its high performance and lower resource consumption.

A

And what that means is that now, a lot of developers who are building cloud native use cases also want to use javascript to write business applications. That means you need to support javascript in webassembly, and what will happen is that we can actually support calling rust functions from javascript in another smg runtime to take advantage of the web assembly's computational efficiency and the wasm edge web assembly, which is on the fastest or web assembly.

A

Runtimes can actually allow you to do that and again, as I've already mentioned, that what we can do is that to embed javascript within uh webassembly.

A

We can try to actually go ahead and build a web assembly based javascript interpreter program for web edge for was image, and that actually is useful to build with the help of quick js, which is a javascript engine, is really small and madable, and it's actually much more compact than the standard v8 engine that we typically use, and it also comes with version based volume, edge based extensions, like network sockets, and also tensorflow based interfaces incorporated into the interpreter as javascript apis, and you need to install rust to basically build this interpreter now.

A

One of the biggest usage of actually going and using this is that we can now basically run javascript programs inside of the web assembly sandbox and the web assembly uh web uh smd edge, or that is the was image. Runtime provides us that lightweight high performance uh cloud native javascript applications now with this particular application with uh quick js and uh the integration of awesome meds.

A

We can basically allow the application itself to be written in rust and compiled into webassembly, and then it can be compiled and deployed as a single wasn't based by code file and javascript functions can be directly embedded into the rust application, and the javascript source code can be actually included either at compile time all at run time through a file and basically the rest of a program can actually handle combinationally intensive tasks in the application, while the javascript can be used for, let's say, handling the uh business logic, so we are going to be looking at one of those examples.

A

uh I have basically created a simple uh function over here, which is basically the main dot rs. Now this is the following program, which basically embeds a javascript program at compile time. So, as you can see over here, I've defined basically my function. My main function, which has defined a code where we have taken up an example of a tensorflow lite demo.

A

Now, basically, this tensorflow lite demo code will be used to showcase uh the direction of a particular food and it will be able to print its label and the the uh basically the percentage of accuracy, or this is the confidence uh that will print by using a dataset and we'll be using a df lite model.

A

So over here, as you can see that we have included the js code, and we have also given the arguments inside of the quickjs uh which um are set into the quickjs runtime since it's a smaller engine and now, if we quickly go ahead and actually look at um basically, what now we can do is that we have also provided another example, which is basically this main.js file. Now, basically, we can build the rust and the javascript application into a single web assembly bytecode program which I'll just quickly show to you.

A

But basically, what we are essentially doing is that the rust code could actually pass the data into the javascript code by passing the arguments or even modifying the included javascript source code directly and the javascript source code could actually return the values by writing it into a temp file. So basically, what we have done is that we have included inside of our main dot iris. That is our rust file. We have included the code for including our uh tensorflow uh based um tf lite, based uh demo, that we have been we're using.

A

That is the main.js. Now the main.js code itself contains the wasmed tensorflow extension javascript api- that, as I've already mentioned, that the quick js also has tensorflow based uh extensions and over here in this one.

A

We are basically using the wasm edge, tensorflow extension uh in which we are going to be actually reading and classifying an image that is based on uh the imagenet model and, as you can see uh from the code that first we have imported the tensorflow lite over here and then we have imported basically an image that we can change and then we're changing uh the image rgb and we're resizing it.

A

And then we are now starting off a new session in which we are basically calling in our um tensorflow lite file, which is basically a food classifier based on image net that we have created, and then we are going to be printing. Our outputs inside of the predictions that we are getting using the mobilenet view, and we have also created a label file through which we will be able to match the label of the prediction. And finally, then we are going to be printing the label and the confidence that we receive.

A

So basically, this main.js file is having the entire uh code of running the logic behind the machine learning part which is basically running the tensorflow code and being able to detect over here, we'll be able to change the different types of images and I'll just quickly also show you so for this current demo, I'm going to be using the image of a pizza that you can see over here again, who doesn't uh like food, so we're going to be using an image of a pizza and we're going to be running this.

A

So basically, now we can go ahead and build the rust plus javascript application into a single webassembly bytecode, and what we're going to be doing over here is that we are going to be running a command that is going to be using the cargo. So essentially, what it will do is that so, let's try to write that quickly.

A

So we are going to be using cargo, then we are going to be using build because we are going to be using this to actually build the rust plus the javascript application into that single web assembly by port. So I'm what I'm going to be doing is I'm going to be simply just adding my cargo bin and then I'm going to be giving my target as the version 32 bossy and then I'm going to be giving it the release a flag and now um an extension to this.

A

What I'm going to be doing is I'm going to be giving it another flag and then that's going to be the features flag and in the features I'm going to be giving it tensorflow and the reason behind this is that why we are adding this features.

A

Flag is because it asks the rust compiler to basically use the wasm edge, tensorflow extension api, and that is going to be really important to use when we are actually going ahead and building uh this application and uh because we are actually going in and using the tensorflow based uh extension within that is provided within uh the vosm edge, with support for javascript that you're using using big js now.

A

Another thing that you will also uh sort of look over here, which we can sort of see, is that we are also using uh basically the bosom uh 32. Was he and that is also super important to use when we are going to be building? Basically, um when we are going to be building the javascript interpreter for our web voicemage.

A

So um adding that features and adding that wasn't was. He is really important. So once we have done this, we can actually go ahead and execute this command and then simply go ahead. And just you know, run this command and, as you can see, that it very quickly is able to generate uh the build, and this build basically helps to create.

A

It basically helps to create uh the build and for that, what we'll be going to be doing is we're going to be using the wasmets command and we can simply use watermeds and go into the directory and go into that specific path where basically, our executables have been created. For that I'll be providing the link for that.

A

So basically it's under the target and we are going to go into version 32, uh since we are using that, and was he and then we are going to be going to the releases and going to basically the quick js and one thing to keep in mind. Is that I'm missing something over here and the important part is that I'm gonna I'm missing, basically the tensorflow extension and uh we, since we are using the tensorflow extension. uh We also need to actually add that to the washer mag.

A

So, uh as I'll add over here, uh the wall, uh the rust paste was he wasn't. um The thing that we'll also be adding over here is rather than just using the was image we also need to include in this version tensorflow. So we're going to be using that as our uh key over here and let's try to run this now.

A

Let's see, I think.

A

Okay, so now, once uh the target has been created, the next step is to basically go ahead and run this executable with the help of wasmetz. For that I'll quickly, just go ahead and write the versamates command. Now, one thing to keep in mind is since we are using a tensorflow based extension.

A

We are also need need to be adding that inside of volume x, so we will be using uh versamatch tensorflow, and apart from this, we'll be going we're going to be using the directory flag and we're going to be adding the path uh for our target file, so we're going to be using target and then we're going to be using version, 32, aussie and then you're going to be going to the release folder and going to click js and add the rest file. So this will be dot possible.

A

So this should be good enough to now run and give us.

A

So this should be now good enough to give us basically the value, and it should print basically what it finds inside of that image that has been given and it will give out the or it will sort of render the result for you.

A

In the meantime, it actually does that again just to sort of look at what we have sort of uh seen so far is that over here, as you can see, that we are embedding the entire uh function of this javascript code inside of rust, and it's basically helping us to run the business logic through javascript and the business of the over here.

A

Being uh the machine learning program through which we are able to uh you know, go ahead and classify a particular food item, so things like creation of the data and rendering of the data is all can be handled with the help of wasm and while the business logic can be written in javascript, and all of that is being run with the help of the bosom edge runtime and, as you can see, basically, it gives us the label as pepperoni, which is visa and gives us a confidence of 38.

A

Now, let's try to go ahead and change the uh pieces. So apart from pizza, we have a lovely burger. So let's try to actually go ahead and change the pizza to a burger, and for this once we actually go ahead and make this change.

A

We'll have to uh create another uh wasm uh build so we'll go ahead and simply just use the command again, so we're going to be going ahead and creating another awesome build, and once we are done with that, we can simply again go and run our voicemail tensorflow to render uh the next level and conference.

A

So we should likely get a burger with a confidence as well.

A

But I hope, uh with this demo, you're able to see how we are able to actually go ahead and build a wasm edge-based runtime that is able to import or sort of embed uh rust and inside of the rest.

A

We are embedding javascript and that basically allows us to run all these machine learning based applications in um these uh devices, with the help of bosom being that center point of being that runtime wasn't is officially adopted by the cncf and again it's a hugely popular um runtime for building cloud native and also smart contract based applications. So, as you can see, the label is a hamburger and the confidence is 60 percent. So I hope that sort of gives you a quick overview on how we can basically go ahead and create uh these applications.

A

So uh just some of the things to keep in mind is that you can use js. That is a really small embedded runtime um that is provided uh and it can be used directly with vasomet, because it has a lot of different embeddings and you can directly use that with uh different kind of tensorflow models.

A

You can use your own tensorflow model by embedding your own custom uh tensorflow lite model, or you can also run it for standard tensorflow models as well, and that way you can run these applications and build very quick to use applications. Now, of course, wasmeds is one of the fastest uh web assembly based runtimes, and you can even actually make this quicker by using, let's say, was my tensorflow lite, um which can actually help to bring uh at least 10 times the improvement in terms of the overall detection and the prediction that are being made.

A

So thank you so much for watching this demo.

B

Thank you so much shivai for that awesome tutorial, and I hope that the audience would have already loved it, but would have got the intuition to learn more about how web assembly is interconnecting, not only cloud native but also machine learning as well as web. So thank you so much to everyone who joined us over to yeshiva yeah.

A

Thank you so much everyone. As you have said. uh I hope that, with this talk, you're able to understand how these wide range of technologies today can be brought into a single uh functioning application with the help of webassembly webassembly is definitely the future for web for cloud native applications.

B

A

Reach out to us.

B

On these following handles, and now we will be waiting for your question and answers. Thank you so much. Thank you so much.