Cloud Native Computing Foundation PrometheusDay NA 2022, 1 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keda with Prometheus: Scaling Your Kubernetes Application with... - David Solanas & Jesus Samitier

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keda with Prometheus: Scaling Your Kubernetes Application with Custom Metrics - David Lorite Solanas & Jesus Angel Samitier, Sysdig

When you're working with a complex application, for example an e-commerce and one day, let’s say the Black Friday, you get 1000% request more than on a normal day, your application stops working, and you lose a lot of money. It Is At that moment when you start thinking that it would have been nice to have an autoscaler in the first place, but you were always afraid to use one. Native Kubernetes HPA can scale your application with some system metrics (cpu, memory). However, you cannot use custom metrics from your application. For example, if your application is exposing saturation metrics, such us the number of connections, latency or buffer length, it would be amazing to use them to trigger the autoscaler. In this talk, Jesus and David will show how Keda uses your custom metrics to scale your application, even with a secure Prometheus remote endpoint.

A

uh Today, I'm going to try being an emcee, this is my first time so bear with me. Thank you and we have David and Jesus with us from systic.

B

A

And they're going to talk about scaling your kubernetes cluster, with custom metrics using Tata and Prometheus.

C

Hello: everyone! How many of you have the nightmares when you have to deploy an HPA in a kubernetes cluster, any of.

B

You come on really, we are the only ones I can't believe it.

A

B

Had several nightmares using HPA, especially in life environments in life, communities, clusters.

C

And how many of you have afraid to Deploy on HCL to in a live kubernetes cluster? Because it's it's a scary too.

C

Yeah, so uh my name is David.

B

My name is Jesus.

C

We work at cystic. We help to our customers to understand what our home their cluster, is working, yeah, yeah again.

A

C

An Spa, an xpa is an original part of this color that allows us to scale our workloads by set on memory CPU and it's great in many scenarios, but HPA is only for word loss. So if you are going to scale your cluster, you have to use another thing: that's it kubernetes cluster of this color um eight is for horizontal. So if you want to scale your warlock vertically, you have to use a better garlic both of this color that comes with kubernetes.

B

So we're talking about our applications kubernetes, and there are a few tools we can use depending on each scenario we'll be using one or another. Just for summarize so right here.

B

So everyone is on the same page here we can use HBA vpa and cluster autoscaler, we'll be using HPA and vpa for auto scaling workloads and we will be using cluster autoscaler for Auto for scaling your cluster. Your your machine uh HPA uh is for pots, vpas for limits and requests, so we'll be using HPA if our application can be easily distributed.

B

For example, a microservice, on the other hand, we'll be using a vpa if our application needs just one single core thread with higher resources, for example a monolith and we'll be using cluster autoscaler in both privacy scenarios when we run out of physical resources.

B

So here's the thing HPA has some limitations. We all use HPA or every day, but it have it has some limitations. Hp doesn't allow combining metrics and it exposes just uh A reduced number of metrics by default, which are basically CPU and memory. This metrics come from The kubernetes Matrix server. If you want more metrics there, you have to make a custom implementation of that server.

B

So sometimes we we need more. uh For example, imagine that you want to monitor your application uses them using the golden signals. You might need to monitor your traffic, your saturation, your your errors or your latency, or imagine that you want to combine those metrics in formulas or make mathematical operations with them. We were talking earlier about how Prometheus, let us do this right uh or make aggregations.

B

So what if we could use Prometheus and from ql to feed our HPA? That would be great.

B

So, let's see a couple of scenarios to to show uh what we want to to say here. Imagine we have an nginx server and we want to scale it up to five parts based on saturation.

B

What's the problem here in DNX doesn't increase memory or CPU when it's saturating, so with the out of the box Matrix that HPA can use this. Is this isn't enough? What would be the solution here? The solution here would be to use the fantastic and Zenex Prometheus exporter, which has this metric, which is uh the connections waiting we could use. We could calculate the average of the connections waiting and set up a threshold, for example, if the average connection is waiting.

B

Foreignty next are above 20, for example, we'd like to create another pod, but let's see a more complex example.

C

Now let's say we have three Apache tongaat Sports and we want to scale it up. A five Bots base it on the memory usage versus the memory limits.

C

So the problem here is we can not get the limits in the HPA that kubernetes give us, because the metric is not there.

C

But if we have to change the respect or we have to change change, the requests or the limits in this case, we will have to change the hph2. So the solution here is to use the coupon container resource limits that giving us a DSN exporter and use the container memory usage that give us the C advisor that is already running in cubelet having them contain more reduces divided by the container limit. We are getting the percentage of the memory usage by.

A

C

Workload, so we can get say that we can use as.

A

C

Of 85 percent- and we will scale it with that- so one of the solution here is use using parameters. Adapter the producing the premises are up. There is a custom implementation of the kubernetes metric servers. This adapter use Prometheus as a storage, The.

A

C

We can use all metrics that the supporters give us, but uh no also sorry. We can use all math that bronchial give us, we can create operations and we could use all the power that promises give us previous adapter is great, but you cannot use it in all scenarios.

C

Imagine that you have a Prometheus server outside of your cluster, or maybe you have a Prometheus that is authenticated.

C

So, let's make.

B

Let's meet Kayla, we, let's pick it up. Kira is a cncf incubating project. It's an open source project that you can deploy on your kubernetes cluster. Basically, what Kita does is more or less the same as Prometheus adapter it. It creates a custom implementation of the kubernetes metric server, so you can feed it with metrics from different scalers. One of these are scalers that that Kida brings us is Prometheus, but if you take a look at the website, there's tons of them.

B

um So what Gita does is? It brings us a custom resource that we can use to create easily an HPA. The the main benefit of using Kita is how easy it is to deploy an Spa. It's just a couple of steps. It's easy, peasy, really and also you can use Keda to Auto scale your applications with metrics from a Prometheus server outside your cluster or property server that it's behind unauthentication.

B

So this is a summary of the case scenarios. If you just need the CPU and memory that comes out of the box with the default HPA, you can use kubernetes HPA. If you need a more powerful, more more sophisticated metrics, you can use primitive adapter if your Prometheus cluster, if your Prometheus is inside your cluster and it's it it ascends, it doesn't have Authentication and you can use Kita in the other cases. Like any other case, uh I mean you can use either in all the cases foreign, it's pretty straightforward.

B

You just create an image space and you install the hand chart there, and then this is where the magic starts. This is the custom resource I watched I was talking about earlier. This is a scaled object in a skill object. You uh you have to set a few things. For example, what is the Target in this case the the workload Target that you want to Auto scale uh in this case is the ntx server, which is a deployment.

B

Then you have to set up the mean Maxi the mean replica count in this case. One and the max replica count in this in this case is five. And finally, what is the type of the scalar that you're using in this case we're using Prometheus, we'll we love Prometheus uh and what is the query that we are using in this case? This is the query we were explaining earlier. In the first scenario, we want to use the average weighting connections of nginx. We also have to set a threshold.

B

What in this case is 20., so basically this is the first. This is a solution for the first scenario we were talking earlier. If the average welding connections are above 20, uh the HP will the HPA. We will create a new port for our engineers and when we deploy this scaled object in our kubernetes cluster. If we install Kita, uh this will generate automatically the HPA in our cluster and and that's it that you, you have your your HP working based on your on your on the data you want. So let's see a demo.

A

A

C

A

C

Yeah continue to start again.

C

Okay, so we are here: we have an engine, X server with a nginx supporter running and we are going to apply the HPA object, the Kira, the object scale, sorry that we saw before in the last slides- and this is the the object. So we have the same we saw so this uh object did a scaling. Object has created an hp8 automatically, so we are going to see it. How it's look like.

A

C

So that is the HPA we have now two replicas we hold automatically.

C

um The spa have autoscale two two replicas and we have the mean ports, one mean port and five ports, and now we are going to scale the generator. We have a generator to create more connection, so we can see how the scales works. So now we have five traffic generators and now we are going to see the pots that we have too so right now we have to still like we saw in the HPA, and now we are going to show the traffic generator pods that are defined, that we escalated before foreign.

C

So if we wait 10 minutes because the scale of a no no it's not they scale have a threshold. So we want to wait some time. We will get yeah that one.

C

So we will get all the all the pots like we are expecting so right now the HPA has the max spot. You can see five replicas of pipe in the Max and we can see the ports are escalated, so we have five and the metric is there. So we have nine uh waiting connections all right.

B

B

On our kubernetes cluster, but we need to monitor it. So I was wondering: do you guys know any cool monitoring tool to use to do this? No, maybe not just no. uh Fortunately, Kira exposes Prometheus metrics, uh not by default. You need to set it up in the uh when you install it just use that that options over there Prometheus metric server enabled and then Kira will start exposing four Prometheus metrics.

B

uh The first three are four errors which are great for creating alerts and have a picture of what is happening on your HPA or on your Kita and but the the last one is very interesting. This this metric gives us the values of its metric that are are automatically generated by Kira in The kubernetes Matrix Explorer. So you can see the picture of actually what is happening and why your um your HPA is doing whatever.

B

So, with this Prometheus metrics you, you could create a cool graphana dashboard like this one to have the Kira, Errors, aggregated, By, A, scaled object or just all errors. You could have what are all the metrics that are automatically generated and also use some HPA metrics to to see uh the the health of your HPA in governance is not not just Kira.

B

So that's, that's all. You need to know to start using Kira in your kubernetes cluster. Let's, let's summarize a little bit uh with the the main goal of this talk is to show how easy it is to deploy an HPA in kubernetes with Kida is really easy.

B

uh It just exposes a custom kubernetes metric server with Prometheus metrics this this these metrics can be from Prometheus, but there are a lot of scalars. So maybe, depending on your use case, you, you can't find any other scalar that that that fits into your into your business.

B

uh You can use Kira in almost any uh real life scenario, because it's highly configurable and uh it's it also exposes Prometheus metrics to to to expose its internals so monitoring it. It's also straightforward. So that's all we wanted to show you guys today. Thank you.