Cloud Native Computing Foundation KnativeCon NA 2022, 1 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building Machine Learning Inference Through Knative Serverless...- Shivay Lamba & Rishit Dagli

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Building Machine Learning Inference Through Knative Serverless Framework - Shivay Lamba, Meilisearch & Rishit Dagli, University of Toronto Student

Deep learning is gaining tremendous momentum in certain academic and industry circles. Inference—the capability to retrieve information from real-world data based on pre-trained models—is at the core of deep learning applications. Deep learning inference can be used to classify images when they arrive in object storage, whether it's hosted on a public cloud, such as Amazon S3 or Azure Blob, or on-premises using an interface such as Ceph RADOS Gateway (RGW). The conventional workflow for this use case is that when an image is updated, it triggers an event, and the object storage publishes the event to its subscribers. The subscribers then download the image and send it to an inference service. Such an event-triggering mechanism can be found in AWS's Simple Queue Service. In this talk we are going to show how to leverage KNative serverless framework to create serverless solution for deep learning inference to move away from using on-prem storage.

A

Perfect, so welcome everyone. Thank you so much for attending our lightning talk. We're really excited to be presenting today at Canada con and it's lightning talk on building machine learning and Friends through the K native service infrastructure and architecture, so without wasting any further time. Let's get started so quick introduction. uh Thank you so much for doing that already, but I'm sure my developerations engineer at millisecerts and I primarily focus a lot on Rust and webassembly, and machine learning inference with ML Ops.

B

B

When I was submitting, I was in high school I just completed it. Okay.

A

All right so, first, let's go over the standard uh infrastructure through which we operate and create a machine learning. uh Tech stack right. So, of course, you'll have your general envelopes life cycle. Where you have your model uh preparation of your data, then you go ahead and train a model.

A

You evaluate your model and then you go ahead and, like you know, productionize it and then you're also running a lot of things such as promises metrics to be able to monitor your model post uh the deployment and when, when once it actually goes into the production, so there's a lot of things that goes into uh not just looking at once.

A

It has been like you know, pushed into production, but there is this entire Tech stack that you'll need to use in order to evaluate- and you know like you know- understand how is the machine learning inference going ahead, and so, basically, that involves deploying your machine learning models, scaling up your model and then also updating your model.

A

In case uh you require to change certain parameters, so you can see over here this kind of the standard text app where you can uh refer to any of the different types of machine and model uh from coming from tensorflow, PI, torch or Escalon, and then you'll need all these things, such as, like you know, kubernetes istio for traffic management and creative, which is the serverless infrastructure for being able to run serverless functions.

A

And if we take a look at how machine learning deployments are done on kubernetes, uh you need to ex you need expertise in all of these different things. Right.

A

That means being able to configure your HTTP and grpc requests, then deployments and how you'll Define your yaml files, where you'll be defining your services, your pods and then uh you'll also need a persistent volumes for any kind of model, updates and model changes, and also, of course, where you require to save your model files themselves and then you'll need things such as CPUs, tpus or gpus for doing that. Inference on the go and you'll need dedicated model servers. So just to kind of deploy your machine learning model on communities.

A

You would require you basically need to set up all of this things and, of course it can take a lot of time, and especially if someone who is just getting into the machine learning in uh world right and were primarily focusing and using data science uh as data scientists, you don't want to really focus a lot on on all of these right.

A

uh All of these are primarily focused on devops, so that's where uh we are going to be like you know, going ahead and taking a look at how we can add a layer on top of our existing inference stack and how we can reduce all of these dependencies that we just saw with the help of KF, serving which relations we'll talk about.

B

B

Thank you so I'll be.

A

Talking more about KF serving.

B

Which is also bundled with Q flow and uh and what you can do with KF serving is simplify this process of deploying your machine learning workloads uh easily on K native and both of us have actually been contributing quite a lot to kubeflow and uh we have been using KF serving for quite some while so uh so. Some of the things that KF serving allows you to do right off the back is uh brings all the pieces together. So you have an intuitive and consistent environment.

B

uh It's simple enough to deploy a machine learning model without having to worry about all the logistics or how something like scheduling or or hard or Hardware would be. Hardware would be given to run the machine learning jobs, which is also a big problem at designating getting hardware for your machine learning jobs. What kind of Hardware do you need? What what kind of processes in your model need?

B

What kind of Hardware you might need to use a mix of gpus and CPUs in a way that a lot of your data processing or data ingestion happens in the CPU? And all of these are problems even when you are running the model inferencing with the model, not just the training part where it is pretty apparent. So all of these are definitely problems when you are inferencing with the model as well, and uh and what we'll try to do is because this is a short talk.

B

I'll just try to give a quick overview of how KF serving works and I'll also share a couple of demos which are pretty easy to run, but will not run them right now due to time constraints.

B

So uh we just show a sample example of running a scikit-learn model and that's as basic as it can get, but you can also apply by the same ideas or how or if serving, to buy torch models to tensorflow models or to probably more complex models where, uh where you don't use your, uh where you don't use custom, where you don't use pre-built layers at all too, but right now, we'll start with a simple example.

B

So uh we start with the scikit-learn example and I I want to highlight some things, because we don't have time to go through all of this. So uh so some some of the things uh this allows you to do, and if you see this is very similar to The Experience, you have uh running workloads on kubernetes as well, um so it is so. Many of the ideas are pretty similar to that as well, and over here, um I just specify the minimum amount of uh a minimum amount of Hardware.

B

My model would need I, also specify the number of replicas I need and I also have the canary deployment options and all of this in a simple yaml file. So you now you can do all of this with kubernetes as well, uh which is interesting right. If you can do all of this with kubernetes as well uh in some way, then why do you, then? Why do you want to bring KF serving into the picture? And why talk about all these yaml files, newer uh uh and new and a new uh Tech stack over there?

B

So uh so a lot of the things that I talked about. Even if you see uh See the previous example, they were machine learning, focused ideas which you would have to go through a length to implement them with kubernetes, some of them uh something like replicas. You can do it very easily, but some of the machine learning specific ideas uh which are probably not very applicable for other kinds of softwares.

B

Those are pretty well supported uh with KF serving, and uh this is also an example of using tensorflow or pie torch and all I want you to see from your because this is quite some code is that we are simply changing the models we have uh over here and um uh another thing I showed is: um you can make Canary deployments just as easily in the context of machine learning with this, and this is an example where I do it, uh while also changing some of the, uh while also changing some of the uh things for the canary deployment customizing?

B

How I want my deployments to be sure? Would you like to talk a bit about explainability sure.

A

And what you can also do is that uh when it comes to writing your models and then once you, of course, we are talking about inferencing. So another part of the inferencing is also the model explanation you want to describe what what is? What exactly does your model do and in terms of once, you start to infer, you can do that with the help of Alibi, explain, which is by Selden, so Cube flow and the care serving was built with Google, Engineers, Bloomberg and uh Selden as well.

A

So you can use I'll buy, explain to understand what are the model model metrics and how does the model perform when you run these differences, and you can also do post processing and pre-processing of your model to understand more in depth about how the model behaves under a given a load of data that is being sent to? So all of these can be defined with the help of these simple yaml files, with the help of KS serving so what you're seeing is that it supports multiple types of machine and models.

A

It is able to do model explanation. You are able to uh do post, processing and pre-processing of your data with this and, of course, not having to worry too much about setting up all of your infrastructure. That you'll normally require with your communities, architecture and then to kind of summarize. Okay.

B

So uh we are almost at the end of our talk, uh as you might have seen the time. So uh so we didn't go a lot into depth of how KF serving works, and we just wanted to give you a quick overview of what KF serving does and how it can be integrated. uh If you want to deploy machine learning on K native- and we also talked a bit about explainability, which we think is pretty interesting, how easily it can be done with KF serving and some of the aspects of KF serving.

B

uh We also talked a bit about Canary deployments, and that was uh oh. What we did in the talk.

A

So, just to kind of summarize the ideas that KF serving has been built on top of K native framework, so where uh K native allows you to create these serverless functions and for your machine learning inferencing uh the KF serving adds a layer on top of it to kind of give you an ability for very easy deployments of your Machinery model and infer from it with all these different functionalities that are provided to you with the help of kf7.

A

With that, we'll conclude and thank you so much, you can connect with us and we'll be open to any questions. Thank.

B