Cloud Native Computing Foundation PrometheusDay NA 2022, 1 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Prometheus in the MLOps Lifecycle - Rishit Dagli & Shivay Lamba

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Prometheus in the MLOps Lifecycle - Rishit Dagli, Narayana Junior College; Incoming University of Toronto & Shivay Lamba, Meilisearch

MLOps is widely talked about and used to make the practice of deploying, managing, and monitoring ML models in production easier. Monitoring ML training or evaluation jobs is obviously very important however it is more important to monitor once an ML model is deployed. This talk first starts by giving a gentle introduction about how ML deployments should be monitored, briefly talking about edge cases in production, data drift, concept drift, model metrics as well as the standard system and resource metrics. We give the audience an overview of observability and monitoring in the context of MLOps. This monitoring could also provide valuable results in terms of whether a model should be retrained, if more data should be collected, if different kinds of data should be collected, and more. We show how one can handle the very important task of monitoring and performing the aforementioned tasks in the context of MLOps with Prometheus. We also show how one could take their existing deployments and add the power of easy and useful monitoring with Prometheus. Finally, we also show demos about how one could use Prometheus paired with their Flyte or Seldon Core, or FastAPI ML deployments.

A

Sure, hello, I'm Richard.

B

And I'm sure and first of all, sorry for keeping you waiting because we have just been juggling around today with a bunch of.

A

B

So I hope you're able to gain from today's talk. Yeah.

A

So in the next 30 minutes we'll be talking about Prometheus in the mlops life cycle. uh We want to talk a bit about things that have worked for us using Prometheus. Both of us have been working closely with machine learning things that have worked for us and uh that's what some of the things we'll talk about monitoring, especially in the context of machine learning, so um I'm Richard I'm, a student at the University of Toronto.

B

And hi I'm sure I am a contrader and maintainer at layer 5, which is a service mesh community.

A

Okay, so uh when we are talking about monitoring uh the the traditional software monitoring, a lot of you might know about monitoring and softwares, or you might have done it a lot earlier, using Prometheus or maybe some other tool. But if you don't uh usually monitoring in the context of software uh includes that you think about slo's, monitor, uh monitor if your slos are being made, monitor, system failures and so on. There's a lot of things that monitoring means.

B

And if we talk in the perspective of machine learning right, because today's talk is all about ml Ops. uh So if you take a look at this diagram for the ml Ops or the machine learning life cycle, there are, of course, multiple steps right from getting your input, data and cleaning it up, and then using that data to train your Machinery model and going ahead and evaluating your model testing your model and then finally, productionizing it and post the productionizing of a model.

B

You are also looking at looking to always test your model from time to time to ensure that the model performance does not degrade as the model is being used in production and uh post deployment here also conditionally monitoring the model, and all of these are steps that are part of the envelope's life cycle and that's where specifically, when, when it comes to from uh the monitoring and observability side of things, we'll see how Prometheus plays a really key role.

A

uh So we uh so we want to talk about monitoring in the context of ml sure. Would you like to talk a bit about that sure.

B

So uh if we take a look at the important use cases, as I mentioned that once you productionize your model, uh there are a lot of hidden variables that might prop up, because, when you're working with the production data, there might be cases where you might run into edge cases where the model does not perform. uh Then over time.

B

There's also a possibility that the data Distribution on the base of which originally your model was streamed on that might shift, and there are we'll basically will be explaining two different ways in terms of the time and in terms of the data drift. So we have particularly two terms that if you are in the ml ecosystem concept, drift and data drift, a couple of items that you will come across from time to time and there could be situations where a model might not be configured properly.

B

So the idea is that model will still make a prediction. But of course it's not uh always going to be making a good prediction. So that's what we're also trying to monitor from time to time that the prediction uh does not go like the prediction rate does not go down.

A

So uh so so two things I particularly want to mention is the aspect of different challenges, uh the monitoring and context of machine learning, and what we want is pretty different from how you might have been using Prometheus earlier. uh So so these are the different challenges I particularly want to highlight on, and so the ideas are a bit different here. What we want to achieve is a bit different than traditional software and uh I also want to highlight that the system goes through properly. There is no error.

A

As such, you don't have any system failure, you don't have any SLO failure, nothing at all the model still makes a prediction, but you are still doing something wrong. It could be. You are still doing something wrong. The model does not make sensible predictions, so this is also a very interesting use case. Everything works perfectly. There is no error, but you still need to be able to monitor in those situations.

A

So uh so the ideas are a bit different here uh in context to uh in contextual software.

A

uh So what we want to do is not just uh so not just many monitor, system, metrics and resource metrics, which we already do. We introduce a third term over here called Model Matrix, which is trying to figure out all of this. The model still uh the model, is still giving you predictions. There is no error, but you will have to figure out something make monitoring work just by getting the model Matrix.

A

So that is what we want to do and yeah in this presentation also we'll particularly be stressing on model Matrix part because system Matrix resource metrics. You might have been uh doing that since since, uh since a long time or some other talks even before us, uh some other talks even before us uh do cover this.

A

So uh we want to talk more about the context of ml, uh so we have uh so we have more in this presentation, we'll talk about monitoring model metrics, so uh so so two of the main things we want to monitor uh and I just set your context or give you an example of why uh you might still want to monitor if your system works well, there is no error, uh so it could happen that environment changes affect the model.

A

So, for example, uh here uh there could just be a different environment from when you had created a model, and now, when you're deploying the model, the environmental variables are all different. In that case, your model works. Well, your model works, but it does not work as we expected it to be. It does not give you the right predictions.

A

There could also be a change in data distribution, so the idea of the so the thing we want to do with Prometheus in the mL of life cycle is uh have a model uh trained on some data and then apply it uh and then make predictions from it. Now. A lot of you might tell me: that's not exactly what we want to do. That's a very simplified way of saying it.

A

You would also iterate your model and do all that, but let's just stick with the simplified definition for right now and one thing that could majorly go wrong and still have your model work, but we want to get that each case in our monitoring aspect is that the data Distribution on which you trained your model is different from the data Distribution on which your model is working, so uh so so think of it as the data on which your model is making predictions should be in the same distribution or it should be in the same likelihood of distribution as your original data, and there are multiple mathematical models to measure this uh multiple ways.

A

You can do this and will not go into the machine learning specific aspects in the stock, but these are the two ways uh very popularly known as concept drift and data drift. So we also want to monitor these well, we'll think of them as edge cases. We we still want to monitor all of this.

A

So, can you talk a bit about how Prometheus comes into play? Sure.

B

So this is where, like you know, we're talking about Prometheus so so far, what we saw was that the system metrics are something that probably was already covered in some of the earlier talks, and some of you who might be Prometheus users might be already doing that. So uh this is a quick sample of how you could potentially use Prometheus, because it's one of the most uh promising, uh like you, know, tools to be able to monitor your model performance and also the model health.

B

So you can use a combination of both parameters and Griffin and that's what we're going to be showing in today's demo as well. So primarily, what we are doing is that Prometheus will allow you to scrape your data. So even if you are having a Time series uh database um or times data that that's what you will be able to capture with the help of Prometheus and then uh you you'll be able to see the Prometheus logs and using that at least you'll be able to see how does the model performance vary over time?

B

And that will give you some insights to understand whether the perform with the performance of a model is actually uh degrading over time, as you kind of analyze these logs, that you get from Prometheus and, of course over that, you can put in some alerts with pay the duty or email, and then you can also go ahead and view them on a grafana dashboard that we'll be also covering in today's demonstration.

B

So it makes it easy for, as a data scientist or as an ml Ops engineer, to integrate Prometheus to see how the model performance works over time, especially because we might have some edge cases that might not be considered when deploying the model initially. So as part of the model serving process. It plays a really important hand in determining how and what changes? Will you be making inside of your model post uh going into production.

A

Great, so next up uh we'll actually, oh, let's go one slide back and uh next up we'll actually take a look at two demos and we'll get more context for these demos. uh So uh so we'll try to show how how do you do this different kind of monitoring or monitoring in a different context, like we talked about monitor all the aspects that we talked about uh so Shiva, would you like to start out with a demo of uh showing Prometheus on Sheldon.

B

Me this quickly connect.

B

A

B

All right, so just a quick introduction about Selden itself, so Selden is a bunch of tools provided to you. It's an open source Tool uh specifically for model serving model performance model monitoring model deployment, so it works very closely with Q flow, so you can definitely check it out, but what we are going to be covering is uh the demonstration for uh how you can use perimeters. So it's going to be little difficult for me to kind of stretch to go to the screen or let me actually try to monitor.

B

Let me try to just go ahead and um mirror my screen. The stone skin.

B

I think so now it should be better yeah perfect. So what we're going to be covering is an example for being able to view the Prometheus metrics with the help of Selden, so uh some of the prequencies that you will require in case you also want to use that you have to install the Selden core. uh So for that you need probably a kubernetes cluster. You can use kind if you're running it locally or you can also use services like civo or ziki or any other cloud provider to just have a kubernetes cluster running.

B

So in my case um and I'll just open up my terminal as well. So in my case I have- and let me just quickly show you my Cube config. So let's go to my DOT Cube.

B

All right, so, if you're able to see that uh right now, I'm running my kubernetes cluster, it's a gke cluster. So it's running on Google cloud and what I've done is that I've already configured my and I'll just quickly show you my quebecile pods as well. So let me just go ahead and do that.

A

A while he does that, uh what's the right way to say Cube CTL, any anyone wants to go first, I say it as cube cattle. If you say it another way, please don't hate me or hit survive, go on yeah.

B

So, as you can see that uh currently I have a number of different services that are pre-configured in my quebecile I I hope that everyone is able to see the screen right yeah. So you can see that I have uh of course issue because istio Gateway is used for monitoring and monitoring and kind of like looking at all the traffic that is going on.

B

So we have an SQ Gateway, and apart from that, we have a bunch of Selden related, uh like you know, pods that are running so we have the course held in and then we have a few monitoring ones as well. So these are essentially what is running so we have a Selden Prometheus operator.

B

So if you want to go ahead and run any particular command and run any machine learning model, you can use the Prometheus operator to that works on top of Selden to monitor all your requests and you're able to see that in a dashboard. So since I already have my monitoring running, you can see that this is my uh dashboard, Prometheus dashboard and over here. So far, I've just run a few sample requests.

B

So the idea is that if you follow along this documentation and we'll share the link for this documentation, um first, we are setting up Selden core and then we install the Prometheus operator and we set it up with with Selden and once we basically do that, then what we're going ahead and doing is that we are just deploying this example model that you can see it's just a simple uh model that Echoes and you can kind of then see the results as we will run this and uh as you go ahead and check this particular docs out um you'll see that how, as you uh like, let's say from time time, you run your machine run model you'll, see the changes coming in this live time.

B

Series data, so over here, I already have one of my windows. Open and I have a call request that is heading to my endpoint, which is running on localhost 8003, so I've used port forwarding to kind of run it locally and you'll be able to see it uh in that GK cluster as well. So as I run this um you see that it run it successfully. Oh okay and now, let's take a look at our Prometheus dashboard.

B

So let me just quickly refresh this so as you can see that we have now a new request that has come up as compared to the last time, so we were up to 12 requests and now we have the 14 requests. So this way what you can do is that, um of course, since we are using, like you know, you can put in any query into your Prometheus dashboard. To can then also monitor uh to get a summary version in this case.

B

What I'm doing is that I'm, looking at on a per a second request basis, that how is my model working right so how many requests have been sent so far, but you can also create your custom uh dashboards as well to kind of monitor the actual live prediction rates that that are going on for a particular model. In this case the example that we showed is a simple model, but you can run it on any model that you would want.

B

Sql and tensorflow Pi torch space models, and this we can monitor your live machine learning performance uh on on the Prometheus dashboard and you can further expand this expand this by also, uh like you know, going ahead and creating a grafana dashboard to kind of monitor your logs that we'll be seeing in the fast API demo that Richard will be showing to all of you. But yes, apart from this, I, also wanted to quickly showcase another one.

B

So um how many of you are aware of machine learning pipelines or might have heard about this term ml pipelines? Anyone has previously probably heard of cube flow. So there are a number of different data, orchestration platforms like you flow flight and which allow you to basically take your entire envelopes cycle and then convert them into specific workflows. So you can think of like let's say the model training as a separate workflow.

B

uh You can have your model uh testing and model deployment as specific workflows right, so you can you're, basically dividing them into various tasks. So another example that I would just want to quickly showcase is uh with the flight so over here. uh Basically, let me go back over here to the docs section so and I'll just quickly, search for Prometheus. So as we showed an example for Selden, if you were like, let's say using a flight for an example for being able to run your workflows, machine learning workflows.

B

So Prometheus is directly in embedded into your machine learning workflows with flight, so you get metrics out of the box and then you'll find that there are some published dashboards to monitor the flight deployments again. This is related to your envelopes, with the help of Light by using the flight workflows. So you can take a look at uh this is well and again, we'll be more than happy to share some examples. But what we just want to showcase through these examples is that today, like Prometheus, plays an extremely important role in the mlops life cycle.

B

So if you are looking to embed that you have a number of different Integrations out of the box, that you can set up locally in your systems and start to monitor the performance and we'll also see an example for a native, fast API that will be showing you to all of you.

A

Okay, so I'll show another demo and uh yeah. Let me get set up.

A

Okay, I think it's up now. Yes, it is so uh now we'll take a look at the first API demo, and the idea of these demos is actually to show a very, very popular ways of deploying machine learning, uh uh of deploying machine learning or doing ml Ops, and we just want to show you how we have integrated Prometheus into that. But again you are not limited to what we of course show in the demo. You can work it make it work with other platforms.

A

We just want to share how we have done it with these platforms and apply these ideas to other platforms. So we will take a look at fast API, which is probably one of the most popular ways to deploy a machine learning model as well, and uh so so setting some context for this demo. What we want to do is create a rest service to expose the model, not something we'll be doing in this talk. uh Let's just say it's been done for us.

A

What which we want to do is instrument the server to collect the metrics, which might probably be exposed by a separate Matrix endpoint um from the rest API. We have created for our model and we'll use the Prometheus fast API instrumentator, which allows us to which allows us to collect metrics uh from uh from a fast API deployment.

A

What we'll also be showing will be how to how to show the data distribution. Your model is right now working under the data distribution. The model was trained on so essentially all the aspects which we just covered in the slides, so we'll also be showing that and then, of course, we want to use Prometheus to collect 10, store, metrics and, we'll just add a layer of rafana to visualize a collected Matrix. Not something I will focus a lot on in this talk talk but uh yeah.

A

We do have for that and we'll just be showing it, uh and if you want to try this out for yourself, so especially some of the things uh like data distribution, what data distribution does your model operate under it's pretty hard to see with the single request, uh so we also have some code I've actually been. So it's actually uh I've been running some code since morning to uh which is essentially using Lucas uh to simulate requests.

A

Hundreds of requests it is, it has been simulating since the morning, uh so that will also allow us to kind of see uh some of the other aspects of machine learning in action which we, uh which we particularly wanted to see with this talk, ah which we particularly wanted to show with the stock. Sorry, no, uh not thank you. uh uh Let's go to the demo, yes, not yet great! So, let's start with the model.

A

uh We don't want to talk a lot about the model, but we'll start with the model and then work our way up. I'll directly start with the docker file. uh I have I, have a fast API deployment up and I also have a model up. My model is trained uh and I and I have a Docker file up to uh to make an image out of it and uh and and now I'll be using this fast API deployment.

A

uh So because we don't really want to talk about training, a model and all that in the demo, so yeah uh then I. uh So then, what I want to talk about is the dashboard. Then what I want to talk about is the model.yaml file. We have a deployment here uh which, uh which uh makes a rest API uh for our model, uh which is uh pretty standard, and we also have a service here. uh The service allows us to expose the rest, API and then use it.

A

So uh those are two of the things we are doing and then we also have a service monitor. So um so, of course, this must be. The service monitor should be in the same space that namespace that Prometheus is running, so Prometheus can automatically get get and collect the data from it. uh So that is uh what will be um so that is another uh service, another, so another resource. We have uh we'll also deploy a service monitor. So, okay, uh where are these three resources deployed?

A

So, uh oh, no, of course you can use something else, I'm, just using sivo, because I've been using it um so I I've deployed I've made a kubernetes cluster I deployed the resources. I talked about the rest, API, uh a service Monitor and a service I've deployed all of that on this civil cluster. uh Of course you can use whatever Cloud you like, but just for this demo we'll be using CEO.

A

So with that, let's go full screen and okay, so we have our model.aml file and then we actually have the dashboard itself. So so some of the things we'll be doing in the dashboard to make sense of the data given to us by Prometheus, for example, is identifying uh what identifying what distributions these data falls falls under now in to do this we will actually use uh and we'll actually leverage some of the functionality by grafana to do this and which is pretty simple.

A

You are making a heat map out of it and then you can simply see- uh and you can then simply compare it with the original distribution. The model was trained on, so uh these are so. This is one of the things that uh that we are doing uh on on our grafana dashboard. uh So uh with that now we have set up now we have set up a rest, API deployment for our model.

A

We also have a service Monitor and uh we I've already deployed Prometheus uh I'm using Prometheus Jack here, so I've already deployed Prometheus uh for us in the cluster. uh All of this is pretty intuitive uh up until now, at least a deployment part and the part which was in I just showed that on the screen, but uh at least deploying all of these is pretty intuitive. uh So uh so that's what you have done until now.

A

uh We have also created a dashboard uh using the model.json, which is, uh which is what grafana will be using uh to make sense out of the metrics collected by uh Prometheus, so I already have uh so I'm just forwarding put here um uh to uh to get the grafana dashboard and let's actually take a look at the grafana dashboard I particularly want to see one of the demos. uh This is uh all of this is by the way open source.

A

uh So you can feel free to try out everything, see how we have implemented other things. But right now uh the part which I talked about which distributions are the requests under uh is something will be showing. But we also have some other things which we'll show in the grafana dashboard, but not go into everything right now, uh due to time uh so another thing before so we have deployed all of this until now. What we have also deployed just to better understand uh is locusts.

A

So, let's close this and let's open our load test, so we actually have a locust file, uh which is.

B

A

Which is another um which is another pod. We are running to make continuous uh to make continuous requests, uh so we can just understand and see and see some data on the grafana dashboard see the heat map being made for different distributions. So uh that's what we have until now and let's actually go to our grafana dashboard, which which I was talking about, and you can actually see- requests have been coming in since quite some while it's actually on since morning- and uh uh this shows what the model score is, especially the model prediction distribution.

A

All of these are like random requests. I'm sending so I can see the different bins. uh uh All of the uh the data is in I can see the different bins. The model has been predicted. uh The data is in so uh as we were talking about in the slides. This is something pretty useful for machine learning use cases, uh which is why we want to see it so so some of the other things we. So this is, of course, uh what we would call the model metrics, and then we also have the service.

A

So, in the model Matrix, we are also showing model score. um So model score is um so the model score is essentially a way to say how good my model is performing, get an estimate of it. Of course, we haven't seen the data earlier or trained on it, but it at least allows the model score at least allows us to get an estimate of how this works. We won't be going into the mathematical models right now. We just saw it for model prediction distribution, but uh uh all of this is open source.

A

You can take a look at it, uh okay, so with that, uh so those are the model metrics. We have also. uh uh We have also monitored uh service metrics and uh resource metrics, uh which are pretty straightforward, so I'll not talk a lot about them, uh but uh with that uh I, but with that demo I come to the end of my talk and this time for real. Thank you and.

B

And this kind of summarize one last point is that uh you can use Prometheus as a very effective tool in all the different life formats of uh during especially the time when you're training model and then post the training when you have put into production to view the live, metrics and then, of course, monitor the overall health of your machine learning models. Yeah any questions we love to answer them now.

A

A

If not right now, then you can always find both of us on Twitter and thank you.