Cloud Native Computing Foundation Kubernetes AI Day EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Enhancing the Performance Testing Process for gRPC Model Inferencing at S... Ted Chang, Paul Van Eck

Description

Enhancing the Performance Testing Process for gRPC Model Inferencing at Scale - Ted Chang, Paul Van Eck, IBM

Performance testing is a critical part of software development that helps us to identify bottlenecks early on and avoid costly crashes that impact operation. When it comes to thousands of machine learning models of many different formats and sizes, ensuring that users can perform inference on these models in reasonable time is paramount. In this session, we show how a Kubernetes cluster is set up with KServe's ModelMesh to enable the high-density deployment of models for gRPC inference. Then, we demonstrate how we load test several thousands of models, and how Prometheus and Grafana are used to illustrate and monitor key performance metrics.

A

So hola buenas tardes hello, thank you for coming to our talk, enhancing the performance testing process for grpc model inferencing at scale. Yes, that is a mouthful that's like 20 plus syllables. So, yes, I am paul vanakk we're- and this is my colleague ted chang, we're both from ibm silicon valley, labs in california.

A

uh Still a bit jet lagged very tired, but we're here to talk to you about kind of what we use for production model model deployment, uh k-serve and model mesh, which there was a wonderfully artistic talk, introduction today earlier by alexa, so a little bit of overlap, but it will be a good recap and then some performance, testing and monitoring setup, which ted will talk about then we'll kind of go into a brief demo, so yeah first, let's just talk about model deployment in general.

A

It's of course an integral part of the ml ops pipeline of the ai lifecycle. You do your training that you need to. Obviously, you need to host it somewhere for your applications to consume the model right so deploying a model as a microservice is probably the most common model serving strategy, especially in production settings right.

A

So in this in this paradigm, the models are exposed via an api endpoint, whether it be rest or grpc, then clients make a inference request with their payload to receive inference response to get what they need right and, as we speak about microservices, of course, kubernetes is the kind of deployment target of choice right so kubernetes, as you probably all know, solves a whole host of problems for you right from scaling your service up and down.

A

You know launching and spinning up new new model server pods, as uh maybe request load, goes up and down. Then it allows for zero downtime model upgrades. You know using rolling deployments and, of course, the aspect of resiliency where you know model serving pod. Is you know it restarts automatically right if there's a failed pod, but you know with all this: even just getting production grade model serving is tough. There are some additional complexities you have to be wary of.

A

You know so yeah complexities everywhere there are, you know quite many things to consider, especially when you're dealing with a variety of frameworks. You have to know how to containerize them for kubernetes um how you should deploy it, how how should the inference or inference request be formatted and, of course, making sure inference. Response times are acceptable, especially when deploying at scale, and so this is. These are some of the things that a user would have to navigate when considering production grade model serving on kubernetes so back to k-serve.

A

So for those who were here earlier in the morning, alexa did give a a good introduction to k-serve. So I'll just keep it brief here.

A

So it's a k-serve is an incubating project in the lf ai and data foundation. This was recently added. Originally it was under the kubeflow umbrella and it allows highly scalable and standard based model inference on kubernetes for trusted ai. So the key features are the serverless inferencing workload. um You can serve your models in a serverless manner, using k native um you don't necessarily have to use k native or ndsto with with uh with k-serve. You can also use what we have, what we call raw deployment mode.

A

um This just uses the native kubernetes resources, like deployments, ingress and services, just to kind of deploy your model, then one good thing about k-survey provides the standardized inference protocol, what we call the v2 protocol- and this is something that's been collaborated and developed in the community, especially with folks from seldon and nvidia.

A

It's a good thing about this protocol is that it'll work.

A

Your inference request will work across all these different cloud providers and I guess model serving runtimes like triton or selden's ml server, and I think even pi torch's torch serve supports the v2 protocol and, of course, casey provides an easy to use yaml where you just supply a storage uri, which is the endpoint to your model file, and then you can actually just in case you will handle the rest and will determine where that model of what type of container or image that model should be loaded into.

A

So you apply yaml and then ksrib will spin up the kubernetes deployments necessary to serve the provided model for inference. So, as mentioned earlier in alexa's talk, there are scalability concerns with this pod per model. Paradigm right, you know. Each inference service has a kind of has a resource request and, of course, how kubernetes is resource allocation you'll you'll quickly hit the limits of your node.

A

Then, of course kubelet has a maximum. I think the default maximum power limit is 110 and I think kubernetes best practice kind of dictates or suggests that you shouldn't have more than 100 pods per node, and you know there's also the aspect of the maximum maximum ip address limitations. You write each specific pod and replica of the pods uh each has its own ip address, so you might quickly. If, especially, if you're dealing with thousands of models, you might quickly exhaust your ip addresses.

A

So all of this leads us to the notion that need to be able to serve multiple models in a single pod per container.

A

Oh sorry, you need to serve multiple models in one singular pod right, so we want to reduce the overhead and avoid hitting all these limitations.

A

So this leads us to model mesh. This is something that I work very closely with, and this is something that was open sourced last year um as a part of the as a part of the case of organization.

A

It's it's intended to serve as a multi-model serving back-end for k-serve somalo mesh has been used in production for several years, um underpinning a lot of the watson services, like watson, assistant, natural language, understanding and discovery, and for those for stuff, like that, you have maybe hundreds of thousands of models, and not all of these models are being used right. Some people create the model and might not perform inference on it for several weeks, and so you don't necessarily have to keep it um available in memory right.

A

So model mesh has the capability to page out dormant models right, pods, not receiving any or models not receiving any traffic and can load it just in time if it does receive traffic. This kind of falls in line with the k native serverless approach, but just uh using in one singular container so yeah so model mesh does handles the intelligent loading and unloading right he's trying to find an intelligent trade-off between responsiveness to users and their computational footprint, and so the overall architecture, I'm not sure.

A

If yeah you can see that so a user would yes, as multiple predictors or our models are deployed, compatible serving runtime deployments are scaled up to host these models, so you might see several run serving runtime deployments um default, it's two deployments or two replicas per serving runtime.

A

So in each pod you can see that there are three containers: the model mesh, container, adapter or puller container for retrieving models from the external object store like s3 or google cloud store, and then, of course, the model server itself which are third third-party serving runtime are inferent services, inference servers like triton or ml server, and all of these support loading, multiple models and two into one container.

A

So the model mesh sidecar they're all connected together. They help route traffic um which is performed through the singular kubernetes service at the top.

A

um So external inferencing requests are made via this service and the ingress smaller mesh pod, which could be any of these routes them to other pauses needed in order to yeah, and so that that kind of creates kind of like your model mesh and in order to coordinate operations and persist, model and instant states and an xcd cluster or instance, is used yeah. So so briefly, some of model measures key features are the cache management, so modemesh treats the set of pre-provisioned serving runtime pods on kubernetes as an lru cache.

A

So malamesh decides what models are loaded or unloaded based off of the usage recency and current request volumes.

A

Next model placement will be done in such a way to balance both the cache across the pods, as well as the request load. So this just means more commonly used. Models are placed on pods that perhaps aren't getting as much traffic in models that may not be getting much traffic or traffic is sporadic will be placed on less utilized pods then there's resiliency, um you know if, for some reason, object store, is unavailable, it'll keep retrying, then there's operational simplicity, so rolling hand. Rolling updates are handled you can.

A

Traffic will continue to be routed to an older model as you're deploying a newer model, and once the newer model is loaded in the cache and available to use, then it'll switch traffic to the newer model, so yeah so model mesh does have quite a lot of moving pieces, and you know, especially when running something like model mission production for thousands and thousands of models, um and there are several aspects that can really impact the effect effectiveness of the platform, and you know from varying configurations- um environments, runtimes and you know the different types of models you might load right, and so this is why performance, testing, automation and monitoring are critical.

A

So as we develop model mesh and as pieces change, um you know performance testing. Automation becomes very important. So you know this is a platform that serves hundreds of thousands of models. So we need to be able to validate performance with different config and different code bases. You know different dependencies models and runtimes.

A

Then we also need to ensure that new code doesn't introduce performance regression and we need to keep doing this periodically throughout the development cycle right. We don't want to test at the very end of the release um and just be caught caught by surprise by some performance degradation.

A

So you know this is all viable and you know by creating automation for this and adding performance testing as part of our ci cd process and by creating tooling for effectively monitoring our model mesh performance. You know this really helps with finding you know costly defects sooner, and so this saves a bunch of time and money. So here I'm going to pass it off to ted, who will kind of show what we do, what we, our performance testing setup and what we use for our performance monitoring.

B

So, as paul mentioned, um there are many critical reasons for running performance tests right so now I will talk about how we actually run some of the performance tests, so it all boils down to three things, at least for us. So the first thing is that the automation framework should run repeatable. Tests should be able to run repeat repeatable tests for every code release and then second is the load driver. It should be very flexible in writing.

B

The workload and the same workload should be able to run tests in both http and grpc, and it should be fast and it should have low overhead and we are currently using k6.

B

We have tried other low driver before and k6 is what we're currently using.

B

Metrics monitoring should be able to monitor cpu and memory utilization from worker nodes and containers inside the cube cluster. Right now we are using prometheus operator.

B

So this is kind of like our environment, the performance test environment. So we have three nodes right. So the first node is the automation in the automation we are, we use uh kfp tekton. This is the kubeflow pipeline and then we have k6 inside the keyflow pipeline. As you know, part of the task and then on the second note we have our prometheus operator, which has the grafana and prometheus and also influx db.

B

And then the third node is our model mesh. This model mesh deploy only one worker now, because this is uh I'm using a very small deployment here.

B

So let's talk about the kefla pipeline, so q flow, as you may know, is an umbrella project for many other smaller projects, a kubeflow pipeline is one of them. This particular q-flow pipeline is running on top of the tecton back end.

B

So next the k6 tools, the k6 load tools, is, you know, an open source load testing tools, you support, sending payload in http and grpc protocols and it's written in gold lane. So it's pretty quick and you you can you can define your workload in javascript.

B

uh It also provides a cli which you can run the script using k6 run, and then it also provides some functionality for you to check whether your test has met any threshold to pass or either fail, and then it also provides output integrated to sending output to influx db so later on. We can use the grafana dashboard to visualize. It.

B

Now we've been talking about grpc right, so grpc is the main uh api for model mesh. So that's why we keep talking about jpc, but the grpc plus the protocol above is usually faster than the rest plus json interface, because uh the grpc is is built on http 2 and it can send payload in binary format.

B

So last thing I want to talk about is a prometheus operator. This operator provides many things you know by default, such as prometheus db, and also the no exporter which collects the linux matrix on your cube cluster and also they provide grafana which can visualize the matrix, and you can view it on your browser.

B

So right now I'll probably show you some demo.

B

Let's switch to.

B

B

B

So, let's see take a look first, we have three notes and then all three. This is because this is so slow.

A

B

So yeah we have three notes, so one note for and then this particular note is for the model mesh deployment, uh so our model match deploy is running on a relatively small worker. Now it has.

B

Two tritone inference pot only tried on two and then the note itself has eight cpus and 32 gig of memories, and all the model match component is running on just one worker node.

B

And this is the grafana dashboard, so this grafana dashboard is set up to monitor the model mesh matrix and as well as the cluster level system, metrics such as cpu and container utilization, cpu memory utilizations and as well as the k6 matrix from the load driver.

B

So at this point let's start a test.

B

And this is the flow queue for pipeline ui.

B

So I'm going to start my test using the q flow pipeline ui.

B

So this is a pipeline. This pipeline has two components so first the pipeline will deploy 20 models and then it'll run some load tests with the k6 driver.

B

So let's start tests real, quick.

B

Okay, uh so while it's running test, let me show you the number of models on my system, so I have 10k models already deployed and uh on the system there are 5 000, amnest, onyx models we can see and then another 5 000.

B

Cfar pie torch models. Oh these are very small small models.

B

And while the test is running, you can see the log here, so you can actually see 20 more amnes. Onyx model has already been deployed.

B

Now it's running the performance test using k6, so the k6, while it's running it's sending a matrix to influx db, which we can graph using the grafana dashboard later.

B

Okay, so let's switch back to the dashboard, so these are on the dashboard. uh These. These two numbers are the same, because it's basically saying that. Okay, I have 10k model registered and 10k model, also loaded in the cache, and these two numbers here are the number of models on each triton runtime pods.

B

So there are roughly 5 600 and four 5400 here.

B

uh You may be wondering huh this don't add up to 10k right it's because uh some models are replicated across the two parts and right now, as the tests running, you can see that each triton is taking a 1000 request currently and the request latency is single digit below, like 10 milliseconds or so, and also here you can see the number of total requests coming in from the model mesh server side, and so this is all the server side data and, as the test is running, you can see the cpu also increased.

B

The serving runtime containers also have higher cpu. Also the tritone containers cpu becomes higher, and this is the memory, so you can see. uh Tritons uses the most memory. Okay, so I think our test has finished. Let me just zoom in here, so you can see a little better all right.

B

So this is uh this. Top three charts are from the k6 driver, um so the test I was running. I was running roughly 100 second load test. The first 30 seconds are like a ramp up, so you'll see gradually ramp up and then run for 60 seconds and then ramp down. So on the client side, I'm using 40 virtual users to drive this test and these 40 users are running sending payloads as fast as they can.

B

Sometimes some. If you like, to emulate real users, you may want to add some delayed or timers for each user, but for this test I'm only curious about what's the maximum capacity of my model mesh deployment. So, as you can see 2k it can take 2k requests per second for this particular model. I'm testing and it's very similar to what the server the model mesh server metrics as well, but the grpc request. Duration is a bit higher like here. It's reading about 15 milliseconds, 20 milliseconds, but like down here the single digit.

B

So that's because the client side may have some network delay and maybe spending some cpu time for processing the response data. So it's it's always good to look at the metrics from both sides.

B

Now the arrow we didn't have any arrows or the arrow is empty. uh The error rate is nothing.

B

So some this chart here shows that when you know initially, when we load 20 more models, it showed that model loading and unloading activity here and also the loaded model size. Here we don't have any activity here in the cache miss because, basically, all the models are already in the cache so to test this.

B

I have some tests also made for this corner case, and I think we can run it just to show.

B

Some cash miss action.

B

Okay, so I'm going to run uh this pipeline only has one test in it. That's running the k6 stress and for this test, I'm going to uh wrong for five minutes and I'm going to send.

B

3000 send a payload to 3000, tie torch, sci-fi python model and 3000 amnes onyx models.

B

uh So I'm here doing all I'm doing here is that if I hit enough uh I'm trying to hit some models that have not been used for a long time and then to see, if that we can get some cash missus action.

B

B

B

So while this is running, I guess we can take some questions because we're almost at the end of our time, yeah.

C

Sure so uh so we had a question on uh slack. That was, the question was: uh do I have, should I run model mesh if I just have a few models? I think that was the question. Let me just double check. Yeah. Should I consider model mesh when I used to have a few models that is before running into cadence limits on pods, ips, etc? What would you recommend.

A

Yeah, oh it's my mic. Okay, so I think if you just have a few months, so the main, the main I guess use case for model mesh is for the high scale, high density right. You have thousands and thousands of models. If you only have a few models, I think just using this case, your single model serving is fits the bill.

A

Well enough, you can easily plug in transformers and explainers just using case serve a single model serving, but it's when you really need to kind of load in and out of memory, multiple models just to make room and well utilize. Your cluster is when you want to use model mesh, okay, it's like so, if you're not already hitting limitations- and I think just using k-serve is fine, it's perfectly fits the bill.

A

Awesome so take down. If you want to explain.

B

Yeah sure so, if we don't have any more questions, so let me show you so yeah, because we are sending tests to so many different models. Now we are actually getting some cash misses now in real life. This may not be a good thing to do to your server, because this actually break the lr, the l ru cache, but for this for the test, we want to test this. This is how you can test and here, because I'm sending so many payloads to two different models.

B

So I'm getting two different response time here for two different models here, so I can actually toggle different models here. Both.

B

So that's it for our talk. So today we have show you how what model mesh is, and then we have show you how to run performance, test model mesh and we're actually using some of those tests in the public model. Mesh github ripple for our nightly build process, and we do have a few more talks coming up. That's related to this collocated event so feel free to participate. Those talks. Thank you very much.

C

Thank you. I think we have time for one or two more questions. Is there anyone that has questions in the audience?

C

There's a question in the back.

D

Thank you very much for the presentation. In the beginning of the presentation you mentioned, that you could post models or to save to save resources for money. So I'm curious how? How do you do this? How how how do you post models which are not in use, and then you recover them.

A

All right so so model mesh, so one of the main components of model mesh is the third-party um inference, services or servers. So we have nvidia's triton. It could be intel's, open, vino or selden's ml servers. All of these servers support loading, have a load and unload api where you can load in models and unload models and so model mesh acts, as kind of like a management management layer on top of that to load and unload, based off of usage or usage demand and based off of usage recency.

A

So pretty much model mesh handles the loading and unloading. Just it's a it treats all of the serving runtime deployments, and these servers as an lru cache right and it'll.

A

If there's, if it's already used of all of the memory capacity allocated for these models and it'll bump out the least recently used model using the unload api for the model, servers then we'll load in or scale up the model being requested, and so that's kind of the kind of the notion behind model mesh and kind of making making use of all the cluster capacity as much as possible.

B

A

B