Cloud Native Computing Foundation KubeDay Japan 2022, 9 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Supercharge Your Kubernetes AI Deployments with Wasm - Rishit Dagli & Shivay Lamba

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Supercharge Your Kubernetes AI Deployments with Wasm - Rishit Dagli, Narayana Junior College; Incoming University of Toronto & Shivay Lamba, Meilisearch

Machine Learning inference (though what we talk about can be applied to other computationally intensive tasks) is often a computationally intensive task and could greatly benefit from the speed of WebAssembly. What if I told you, that you could improve the performance of your application deployed on Linux containers. Enter Wasm and Kubernetes. This talk starts off by introducing the audience to WebAssembly (Wasm) and how they could make use of the speed and security among others of Wasm for their deployments. Another problem we face is that the standard WebAssembly provides very limited access to the native OS and hardware, such as multi-core CPUs, GPU, or TPUs which is not ideal for the kind of systems we target. The talk also shows how one could use the WebAssembly System Interface (WASI) to get security, portability, and native speed for Machine Learning models. The talk then shows how one can use Krustlet to run Wasm on Kubernetes with a demo of deploying a model while also talking about necessary considerations while doing so.

A

Thank you so much for uh coming in, for this talk. I'm super excited to be presenting um on the topic of supercharging your communities, AI deployments with the help of webassembly.

A

um My co-president, unfortunately couldn't make it because of Visa issues, so uh I've recorded a few of his um slides and the demo that he'll be running, so please uh I hope that you're able to learn something uh today from this talk, uh quick introduction about myself: I'm shivai, I'm, a developer Raticate uh at minisource. It's a rough space search engine and also a contributor at layer 5, which is a service mesh community and Richard. Who is my co-presenter who couldn't make? It is an incoming student at University of Toronto uh and yeah.

A

So, of course, the first thing right at the bad leg to cover is what exactly is webassembly right now, one of the most common misconceptions for those who might have heard the terminology or might also have had some experience uh working with or having heard about it in the news or in open source, is that um people often related to web since it's like in the word contains web.

A

So people usually attribute it to just being limited to web browsers or web applications or JavaScript, and they also like to compare it with Assembly Language, but uh it is neither web nor assembly. uh It did start as a browser technology back in 2017, uh but of course it has now more usage outside the browser, especially for cloud native use cases, and we'll also see today how you can use that with the help of kubernetes as well.

A

In order to deploy your applications and, uh of course, the main idea of what webassembly is that it is a binary instruction format. So so it does look very similar to how an assembly language program would work. That means that it runs natively uh and the structure. If you were to see how a DOT wasm file, uh you will see that it's very similar to how any semi language uh is, but of course, it's much more different than how a semi language works and it's primarily used as a compilation Target.

A

So um you can consider webassembly not as a primitive technology but as a compilation Target where and you would basically take a program- a function written in any popular programming. Language like rust, C, plus plus, and then compile it down to the webassembly byte code and your webassembly bytecode is what is interacting with the host environment and uh since, uh like you know, it's a poly Cloud environment. Essentially.

A

So that means that, whether you use functional programming languages or you may use scripting programming languages like python JavaScript or even your proper object, oriented programming languages like C, plus, plus or Java. Those can be compiled into webassembly, so um whether you're, coming from a web background or you're coming from a devops background uh from a golang background, you'll find the ability to actually convert your functions that might have been written in these languages into the webassembly bytecode and uh that helps us to bring some of the main key features about webassembly.

A

So the first one is that it is super efficient and fast. um The biggest reason for this is that your webassembly byte codes are super small, and this allows you to run them very efficiently, and uh this is one of the biggest use cases that we'll also see when we are comparing it against. Let's say our standard Docker images or containers in general, uh the other one is that it's open and debuggable.

A

So, of course, uh the uh entire web assembly uh scenario is it's completely open sourced, so the webassembly is powered by the bytecode alliance. It's the alliance that is kind of the governing body for webassembly, so it's completely open source and all the protocols uh that are surrounding webassembly are regularly discussed at the bytecode alliance meetings.

A

And, of course, the biggest one is that uh that we'll also cover today is that it's also working for non-web platforms, uh specifically in Cloud native ecosystem and and one of the latest uh surveys that is conducted by the cncf right before Cube uh kubecon, North America, and they announced these results at kubecon.

A

North America was that uh the usage of uh webassembly today in server-side applications and Cloud applications is actually two to three times more than in uh web applications, uh so standard web applications that you might see for the web platforms is uh using webassembly byte codes inside of your JavaScript applications and some of the most common uh applications that you might see them being run is like figma Adobe light shot.

A

So these are applications that are running on the browser and they use webassembly to be able to run more computationally intensive tasks that regular JavaScript might not be able to so you run, for example, figma uses uh M scripting to convert your C plus plus codes into webassembly bytecodes and then run them alongside JavaScript to power Sigma, and uh it is also safe. So webassembly comes with a security model, so it's kind of a Sandbox model that uh keeps it super safe.

A

Now, one of the drawbacks that we'll see with webassembly is that in itself as a byte code, it cannot do anything on its own. So if you were to uh make it interact with, let's say your system files or your file system you'll need other dependencies uh to be able to actually manage that and we'll see how that is done in the later part of this presentation.

A

So what we have seen so far is that webassembly is really fast and efficient way to be able to run applications right, and this is also a great benefit that uh you just have one byte code and this byte code can run across multiple platforms. So it is not platform dependent.

A

Usually whenever you are writing any program, uh you might have to install certain dependencies that might be limited to one specific environment, but with with webassembly, uh you can run this across different system architectures and not have to worry about whether it's an x64 architecture or some other architecture as well and again it's safe and portable um and then primarily when we talk about the ability for the webassembly byte codes uh or essentially the entire ecosystem, to be able to interact with your file systems.

A

That is where uh we are going to be talking about the binary compatibility and also what we are going to introduce, which is Wazi, which is the webassembly system interface. So this is the technology that enables you to let interact your web assembly byte code or your modules, with, let's say, system, resources and network resources, and here I'll, basically now forward to the presentation recorded by my co-presenter and I'll just play. The.

B

Vision, this is a very popular tweet. I might have seen it a lot. So this is why the founder of Docker- and he said the voice of physiology existed in 2008. uh We wouldn't have needed to create Docker, that's how important it is.

B

um So this is I. Just uh that's very interesting thing about Diwali and how important was he is so um I I want to uh start talking about. Was he by talking about system interfaces, and uh so this is um so this safety statement? C gives you direct access to system resources, which is more certainly first because it's far too important for stability and security. So what happens is- um uh and this is also a pretty popular diagram- is that applications could be. The kernel are to do all kinds of system calls.

B

Maybe something to redefine delete the file. All kinds of system calls uh the applications.

B

System calls for the applications, and this would probably be in some standards like posix or somebody to do it all applications or programming languages have different ways of doing system calls. uh You do system calls in different way and see then in Python Java, so uh yeah all applications are a different way of doing the system calls, and but these essentially just RC kernel uh and um things about putting the the system.

B

Also uh and- and this solves the problem we were talking about, allowing users should do users and applications to do what they have the right to do without shopping other applications, and this, and this part solves it pretty easily. uh What quality does is see or rust might have very different ways of doing the system calls are very different standards as well, so what boson does is it gives you.

B

Let's say you want to.

B

Addresses native way of doing system calls- and this becomes really helpful, because now you can, when you combine this down to webassembly, you can run this web assembly wherever you want uh on our whatever time you use, uh you can run this web assembly even on the server.

B

So this program is pretty important when you are trying to watch them outside the web, uh because you are not now right to particular kind of system calls and you just use whenever you want to make a system call so uh obviously just gives you uh independent uh methods to do these system calls.

B

So that's what Jose does is, uh and it's pretty helpful to run for some outside the web, so the network we'll be seeing will actually be using Bosley as well, because we want to run awesome on the server side, so yeah uh yeah in those cases was he is pretty helpful. uh So that's what we'll be seeing uh in the demos as well.

B

Different benchmarks over here and I also have the code for this up on my GitHub, but I've done these benchmarks and um uh I I. What I particularly want you to see is that density of python in the Linux container uh and the Boston uh part, so we are using the voice of time, uh which is a very popular version and I'm running the exact same machine learning one uh in all of these, so I wanted to see the version with aot compilation, which is some pretty interesting results.

B

um It's pretty fast, uh uh it's pretty faster than what a Linux container would take to do this and uh yeah. These are just some benchmarks, feel free to explore more on the code for producing these benchmarks. So.

A

um I hope, by the way everyone is able to hear properly okay perfect. uh So what uh we kind of saw was that from this particular demonstration that uh for running a simple inference using one of the most popular machine learning models which is mobilenet uh with the help of wasm, it was a two to a 2.5 times faster than your standard uh container image that we ran with with python uh inside of a Linux container.

A

So these kind of showcases the excellent approach when it comes to being able to do more highly computational tasks with the help of webassembly. Of course, this was just one example, um but now we'll be moving on to the demos and within the demos um and within the demos.

A

What we'll do is that we'll focus on and we'll show you that, how you can basically create a web assembly based uh apis and microservices and then uh deploy them on side of your kubernetes clusters uh on running them as kubernetes nodes so and the example that we're going to be showing you is that, um inside of this specific example that we'll run uh and again, we'll show you in the this demonstration as well. That will have a few pools of Linux containers, so there'll be nodes that are running your standard.

A

Next containers and there'll be nodes that will run your webassembly container as well, and then how you can interop operate between them and make system calls between them as well. So first I'll just set the context for the demo that we're going to be showing.

B

So what we'll be using is either spin or slide uh so Spin and slider. We used to just run voice very easily on the server side, and what we also use is the container assumption. So the container D was on shim is pretty helpful and kind of giving you approach between kubernetes and container deal. It was just none was a as a library and allows you and makes it very easy to run version Maps.

B

What do I and uh talking about this like, if you see how the overall uh thing looks like right now, uh you have cubed uh and the container you uh and one was he can be managed using a CRI plugin as well, which is what you probably want to do, and then we have the container new version dimension.

B

Containerdy, so this is how our setup for running awesome work use in kubernetes would essentially.

B

This is essentially what you want to do so yeah. This is uh overview of the demo, we'll be showing what we'll do is um we first want to start by making a runtime class a sense name. uh So it knows that whenever I have a slight acquisition or a spin application, I I need to use an entire class.

B

It's pretty uh easily supportable for both, but you can also have, and then what you do is whenever you make.

B

Slider spin container applications I understand class and with that you workload will run on uh and without your permission please will be done, I'm being managed in a kubernetes cluster uh yeah. So that was the overview of this, and if you kind of took something back from this, then you'd know that I'm trying to have peanuts and wasn't containers side by side I'm not trying to replace leaders.

B

So I I tell you what I have and but this is what I'll be showing at the demo as well and having the next notebook and I will have both of these side by side. Some of my requests are probably comfort in terms of the question being handwritten by the onesie notebook and um the rest of the things being done by the Linux node group. So I'm not saying that you should have wasn't replaced enough containers altogether, but just have them run side by side. Let's go to the devil.

B

So uh so now we'll set up.

A

So uh basically, what you saw was that uh what we are trying to intend is that we are not trying to replace the docker containers with your webassembly containers, but instead uh be being able to run them side by side so that your Docker container can actually because of the extensive ecosystem that and the tool chains that come inbuilt with Docker.

A

uh Those can be managed with the help of your Docker containers, whereas the high computational tasks, those specific tasks, can be managed with the help of your uh with the help of the web assembly containers and the demo that we are going to be showing is, with the help of Microsoft Azure and uh I'll, quickly, walk through over the demo and then, uh of course, I'll be open to questions as well later. On.

B

I want to be showing the demos and what we wanted in the demo is a map and run it and manage it with kubernetes. uh So that's what I'll be sharing in the demo, and this will be very similar to what we have been talking about. uh So what I have over here is a single kubernetes cluster and right now just look at the system. Nodes I have so I have the system nodes, which is pretty standard and right now just look at that. uh Those are the system nodes. I have.

B

You want so without what I want to do is come to uh come to making a web assembly image and what we try is just making a sample was an image.

B

And slide this allows you to easily run your voice or map, so share version apps and uh it actually runs, and it actually uses Dragon Ball Z as a library to very easily reduce the capabilities.

B

On a uh when you make an HTTP call to it and that's what we'll be looking at uh just something simple, but showing the HTTP capabilities, making a request handling the request version. uh That's what we are actually interested about.

B

B

But you might have to install the version 32 minutes so uh uh now that I've had the version. 32, minus 4C, Target I'll just go ahead and build this. So if you, if you actually see what we are building, uh we are building a slight image, um so slightest short for spring lightning and if you don't know uh as I was just talking about it, helps you uh essentially on your watermark series.

B

Is a single file and less is equal to the 100 hello function, and this is returned to our hello. With a um with the response. Okay, the history is of 200 exactly what we want and that's all this uh piece of code does so yeah. That's what will be now building so I'll just do cargo print. I did Target.

B

The build was very fast and that's because I had already builted. So you don't waste time in that debug.

B

And we have a slide, that was a fight, and this is the version file that we wanted. This is what we have compiled uh when you convert this uh into a container using the docker using the docker file we have over here. So uh this one is to know it are version 5 version here it is.

B

This will allow us to convert this into a container image, and then we want to use a container image and run this application.

B

But it is actually pretty straightforward makes it very easy for you, uh so uh yeah. If you want to know more about site, probably give the official documentation I tried and um what I also wanted to show is. uh There are some examples on spin as well. uh So, oh by the way in this example, I'm using site has taken from the official slide examples so yeah.

B

This is building an application and then running it with kubernetes, but you can also use, pin and I'll soon talk about how you can use spin in your kubernetes clusters to run webassemblies but yeah. You can also use spin very easily, and so now that we have a version. uh Fine. What we'll do is go to the kubernetes cluster and try running it.

B

And so what I'll start by doing is I I have my editor will appear I'm running all of this.

B

And whatever privacy you want I'm just running so first off what I do is I'll note one. So what this node pool does is actually playing the kubernetes cluster that this uh this node that runs on additional depends on uh this. Node should be running, wasn't buzzy.

B

Which I have so uh yes, I already have the nodes and, if I actually just to, um uh if the phrase is to just try to see the power will be showing the premise and what we want to do in the demo is okay. So first, uh this node runs in all four nodes. So that's what I'll be doing I've already created this node pool. It takes quite some time so I've already created this node pool for you, and if you see this is where I have the notebook uh I have in all four nodes.

B

Three of them are the system modes, and one of them is the my voice, which I have so uh yes, I already have the nodes and, if I actually just to, um if I change, to just try to see the nodes that I have so, let's just start, this okay I have an interesting character at the start.

B

So, let's okay, so if I actually see this is the node I have, and this is actually running it was on Wazi, okay, great, so now that I have that what I'll try to do is just here are the notes. I have I will again see over the the node naming and how it is named. So I have three agent tools uh and I have a my voice as you guessed uh okay, so uh that's the setup we have until now and what I will do next up is like we talked about.

B

We first want to make a runtime class.

B

And this code is actually are the same, the one same as the one we were talking about. So this is a runtime class for uh slide. We also have one for spin because you can essentially enjoy it the same way this demo. If you want to use, pin and said uh it says it's just lock and replace that's all you need to do so. This uh creates these two runtime classes, foreign classes. So let's go ahead and apply and apply so this is with that. I should have yes, I have created two runtime classes.

B

Next up what I want to do is actually deploy my application, so here I have the deployment, and um this is also pretty similar I'm just telling it what container it needs to run, and this is the fishing container we had built out with the version they have it hosted. So that's what I'll be using and um I'll also tell you that the runtime class it needs to use is version time slide. So what's the time is the name of the uh is the name of the.

B

And uh what's the time is the one will be using? You can, of course, use others like boss image, but right now it's awesome time very well. So we'll try using version time.

B

A

So, just for uh keeping the time sake, but I hope that, with this demo, you could see that uh what we did was that in the demo we took uh four different node pools. The first three were your standard, uh Docker uh containers and the last one that we deployed, and you can see the yaml structure and I also shared the link.

A

If you want to see the default Yamin structure uh for uh the deployment uh yaml that has been created to deploy our uh node node, specifically for our was he uh node uh and the other demo that I wanted to quickly showcase is with cave awesome. So what cave awesome is that it's a kubernetes operator that allows you to directly run your webassembly workloads on communities now.

A

One great thing is that it comes out of the box with multiple different types of ways in which you can run communities so, whether it's mini Cube or its micro kids right. It has a really great support for multiple communities distributions and also with multiple webassembly runtimes, and um in case you want to get started with this. uh You can install cave Awesome by using help.

A

So if you have help you can install Helm and install cave awesome and the cave awesome operator and then just run this Quebec till command uh in order to install the webassembly example and over here, I've already done that. So this is my terminal, and this is what I'm going to be doing so over here. I'll display this out and I'll run this so I'll apply this over here and then what I'll do is I will uh go ahead and test out this particular job.

A

So what we are doing over here is that um we ran a sample, webassembly job, and you can see the result over here and, if I take a look at all of my uh nodes that I have so let me go ahead and do that.

A

Let me bring up all my notes, so you can see that uh all my pods over here there is one that is awesome test. So this is running and this runs successfully and of course my kposium operator is also running perfectly fine and uh just to kind of summarize what we have covered. So, of course, there's a two-way relationship. Why communities needs wasm is because of the fact that wasn't has tiny containers.

A

It is a much faster startup time and, of course it is platform independent and why wasm needs communities, because of course we know that communities today has a golden standard in terms of how you can scale up applications with the help of communities.

A

So the main idea for today's presentation was to Showcase that um that balance and how you can run your webassembly workloads inside of kubernetes and especially if you are having more highly computational tasks, such as machine learning or even outside of machine learning, you can run them very easily on webassembly because of the small size and the faster processing and then, of course manage those workloads on the help of or on the top of kubernetes. And, of course, please take a look at the slides and the code snippet that we covered in today's demo.

A

This might be useful and yeah with that. uh Thank you so much and you can connect with us on our Twitter uh in case uh you have any questions and, of course, I'll be here to answer any of your questions. Thank you.