Cloud Native Computing Foundation PromCon North America 2021, 1 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Machine Learning Observability with Prometheus - Shivay Lamba, Layer5

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Machine Learning Observability with Prometheus - Shivay Lamba, Layer5

Evaluating the metrics of a Machine Learning system is obviously a critical task during research & development phase. However, once a machine learning model is deployed in production, it is also critical to know how this model is performing. Having good instrumentation and observability practices is needed. Prometheus is a platform for monitoring application metrics and this lightning talk will share how you can use Prometheus to monitor ML Pipelines.

A

Hello, everyone, I hope everyone is having a great time at prom, con north america 2021 and at coupon, and all the other co-located events for this year's flagship, cncf event hi, I'm shiva lamba and I'm going to be giving a lightning talk on machine learning, observability with prometheus, a very quick introduction about myself.

A

I am shivai, I'm a contributor and a mesh mate at layer 5, which is an open source organization, part of the cncf landscape, with its products like measuring and the smi uh performance, and it is all about service measures, and I've been also a google search code specifically for service missions under cncf.

A

First, let's talk about the machine learning life cycle. So whenever we talk about any kind of a machine learning algorithm, we always start with the model building through which, once we have built the model, we undergo the evaluation and basically the training of our model, and we do experimentation with all the different kind of um with the model to whether see whether it's a good model or not.

A

Once we have been consent with or we content with the model evaluation, we productionize the model, and then we put it to testing to see whether the model is giving us good accuracy or not, and once uh we are able to test it out completely. Then we'll go ahead and actually go ahead and deploy the machine learning model and once the model is under deployment and has been used for production use, we will continue.

A

We set up a pipeline through which we'll be able to actually monitor and observe our machine learning model, and that's what we're going to be talking about today is specifically what exactly is observability, and how can we do that for our machine learning models now evaluating the metrics of machine learning system is really important for research and even for production as well, because once the machine learning model has been deployed into production, it is also very critical to see.

A

How is the machine learning performing at all times, because having of great observability practices is needed to ensure a smooth running of our machine learning model and the machine and metrics through which, on the on the basis of which we can evaluate the performance of our model, can be interface-based, which could include latency model based with good, include the performance of the machine, learning model or even infrastructure, based that includes the cpu utilization?

A

Now, some of the other metrics that we need to also keep in mind include the latency whenever there is some kind of a uh ml based on machine learning based api that has been called, then how much amount of cpu bandwidth memory bandwidth is actually being used uh when performing any kind of prediction. How much amount of disk utilization is actually being used if it is applicable, then what are the prediction values and how do they change over a given time frame?

A

What are the minimum and maximum prediction values that are actually going and we are getting? Let's see what is the standard deviation that we get over a period of time and also like? Let's see the changes to the statistical distribution of all the predicted values, all of these need to be measured in order to maintain the observability of a machine learning model, and how can we do that?

A

We can do that with the help of prometheus now prometheus helps to basically scrape data or metrics from instrumented jobs, either directly or indirectly, as well um with the help of short-lived jobs. Now it is, it is capable of actually storing all these script samples locally and runs rules over this data to either let's aggregate or record even new data uh from the existing data or to help generate alerts and one of the most popular stacks to actually use for monitoring.

A

The metrics is the uh combination of prometheus that actually helps to measure the metrics and graphene that helps to actually create alerts.

A

So from this diagram, as you can see that within the promise server we are pulling in the metrics, we are evaluating those metrics and then let's say, if those metrics are concerned, we can actually create alerts relevant alerts so that we can be notified if certain metrics are falling behind and this diagram sort of shows you an example of that where we are basically creating a dashboard in grifana, but we are actually going and uh looking at all the different metrics, uh we're sort of evaluating the metrics and fetching the metrics, uh for example.

A

This particular example showcases an example of the average house price uh prediction and we are evaluating all of that and all these metrics are getting measured with the help of from atheists and alerts can be generated with the help of grifana. So that's pretty much it for this lightning talk. uh I hope if you have liked this presentation reach out to me on twitter and on my github.

A

uh The main idea again behind prometheus is to be able to monitor and machine learning models which are in production and deployed need to be monitored and prometheus can play an important role in monitoring the different kind of metrics, on the basis of which you want to adjust the performance of our machine learning model. With that in mind, thank you so much for watching this lighting talk and I hope to see you soon in a cubecoin.