Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Kubernetes Metric APIs using Prometheus - Matthias Loibl & Sergiusz Urbaniak, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: Kubernetes Metric APIs using Prometheus - Matthias Loibl & Sergiusz Urbaniak, Red Hat

Kubernetes traditionally uses metrics for its core scheduling decisions - in the beginning all of this started with an opinionated internal stack. Since then Kubernetes has introduced 3 orthogonal standardized metrics APIs. As of today many implementations exist - i.e. for cloud providers and on premise. In this talk we will first show the community process around metrics in Kubernetes, how the Special Interest Group (SIG) for instrumentation works and how to get involved. We will do an overview and deep dive in all 3 metric APIs, with a concrete fully open source Prometheus based deployment example. Once we have Prometheus running we will show how to bridge the gap between Prometheus and Kubernetes to use these APIs. Finally, we will conclude the talk with an example on scaling your deployments based on custom metrics served by your Prometheus with the Horizontal Pod Autoscaler.

https://sched.co/MPc1

A

A

Welcome to kubernetes map, kubernetes metrics, api's, past present and future of the Prometheus adapter, I'm sergey romanyuk, don't try to spell my name, just call me search sir or whatever works for you. I work on different things and kubernetes since around five years and currently I'm a member of the monitoring team at Red, Hat, integrating Prometheus into communities and many satellite in projects and around it.

B

So I met a matter or Matias, as most people call me, but that's my github panda I'm also part of the same team working on integrating Prometheus and kubernetes yeah. So today we are going to take a deep dive through the cooper, kubernetes metrics api and take a look what's actually going on. So first a bit about the history. The metrics in kubernetes 1.1 were yeah CPU and memory.

B

So that's about it and they were coming from the node reporting to the kubernetes api, so that was actually used by the by the scheduler to know what's going on in the kubernetes cluster and that's the way it was back and back in the days. That was the architecture we had like he'd, sir, in the middle of it all, and that would communicate to the to the api server and it had its own kind, small storage, back-end and it would communicate to the cube latency advisor. So yeah I think that's like the rough idea.

B

So there were a lot of problems with hipster and it's been deprecated for one or two I think like two versions of kubernetes already there was a push based model, it has Sync plugins, and that meant that, basically, everything that needs to be supported needed to be put in into hipster and there was kind of a feature creep so to say, like everything needed to be actually implemented in that code, it was an opinionated too no abstractions and yeah. What if you wanted to use Prometheus instead? So we have some goats to improve the situation.

B

It was about decoupling, the scheduling and scaling decisions from hipster for the reasons we were just talking about and introduced an abstract API scheme to do the same.

B

Just to point out, we are not talking out the about the Prometheus exposition format. We are talking about metric API types and those are real types like go. Struct definitions inside the kubernetes code and as you can see, it's yeah, it's a struct and that's it. You can find those types by using curl on the API server endpoint and the ones we are talking about are in bolt and the slash metrics, API or endpoint is actually what gives back the Prometheus exposition format and there's scraped by Prometheus. But we don't really care about this today.

B

So yeah there are three types of metrics API or types in kubernetes. One is the resource, metrics API, and there was the one we were talking about earlier in kubernetes 1.1 already, so it was CPU and memory and that's it. Then we have now custom metric API and that is actually doing taking an object inside of kubernetes and turning that into a metric. And then we have external metrics API, which is what we can use inside kubernetes to do something to talk or to get metrics from an external provider like a cloud provider.

B

For example, because it's only an API and it's decoupled, we can have several implementations and those are I think mostly it there's the metrics the metric server, which is most likely shipped as a default for for most kubernetes clusters. But it only gives you the resource metrics. Then we have the kubernetes Prometheus adapter that is talking to Prometheus, and then there are a bunch of club provider, specific adapters and yeah out.

A

Of interest yeah, who a few uses metrics server exactly like in production, okay, who a few uses any of the other adapters or knows about it, that's quite okay, that's way less! That's actually expected yeah cool all.

B

Right, so what about Prometheus right so say yours is gonna, tell us about the Prometheus adapter and about integrating Prometheus and kubernetes. Now yep.

A

Before I go into the details into the parameters that have to, let me point out that, since these are three separate api's and kubernetes resource metrics, custom, metrics and external metrics, what you can do actually is to deploy a mix of those tools here. The most canonical example would be- probably many of you have by default, the metric server deployed. It's a quick win, it's deployed by default by many distributions. The drawback, though, is that the metric server only serves the resource, metrics API.

A

So the thing that you see when you invoke cube cut all top parts or Q capital Q couple top notes, um but you can, in addition to have custom metrics in order, for instance, to introduce your own metrics. That communities doesn't know about anything. You need another piece of software desta. That does that, so you could deploy a metric server.

A

In addition to that, if you're, for instance on Azure, you could deploy the asura metrics adapter, which hosts the other two endpoints or- and this is what I'm going to talk about here, you can use the Prometheus adapter, which, as of today, can host all the three metric API endpoints inside Cuba Nettie's. Note, though, external metrics was merged in just recently, it's still a little shaky, but already quite usable and just out of interest. So Leros are you here, no Salyers.

A

Unfortunately, he I saw him on the attendee list, but a big shout out to Sally Ross, because she was the initial creator of the Prometheus adapter project. He went to other duties inside the kubernetes ecosystem, the API machinery, but we are actively maintaining or continuing to maintain that project.

A

Okay. So what our prometheus? Let's say, you have a kubernetes deployment that has prometheus is a first-class metrics backend, and you just want to use that. So you have configured to Prometheus to scrape all the resource metrics. These are mostly coming from couplet see advisor. You would like to add additional custom metrics canonical example here being HTTP requests per second by workload and potentially also external metrics.

A

That means the metrics that are coming totally outside of the cluster, which are not related to any objects, and all of those metrics are being scraped into your Prometheus instance. In this case, you could use Prometheus, adapter and host all of the three API endpoints. All that works internally is that the Prometheus adapter and by the way, all the other tools do it. The same way register asks what kubernetes calls an aggregated API server and that's actually a pretty straightforward approach.

A

As you may know, kubernetes is a huge API of many things and it's actually an API tree and whenever kubernetes wants to serve metrics, API endpoints, in that case it simply dispatches to the premise of the adapter or metric server or any any tool that you like in order to host those metric endpoints, what the Prometheus adapter internally does it simply converts those API requests into real Prometheus theories and gives back the results and how this is being configured.

A

I will show you in very much detail here and other of interest who have you configured Prometheus adapter already in the past, Steffie, okay, who of you had issues with configuring? It?

A

Okay, so hopefully I'm gonna, clear things a little bit up in this talk, make it more approachable for you to configure Prometheus adapter, but also we're going to give an outlook on the future of of the project resource metrics else Matias mentioned. These are actually hard-coded metrics into kubernetes itself, currently, most famous ones. These are node metrics and plot metrics, CPU memory. You will find out if you actually have this API enabled by calling cube cuddle, get API services.

A

It's a good way to check if this returns true- and that means that your aggregated API set up is actually correctly configured and then there is another need: sub command of Q capital called API resources. Here you can grab four metrics and you will get exactly those two metric names that are interest for for resource metrics, namely notes and parts. Note notes on namespace name spell notes are North namespaced parts or names based metrics.

A

Obviously, you know- probably most of you know: Cube shuttle depart. This is exactly the API that is be invoked behind the scenes, but the same way you do cue cut, will get part or cue cattle get deployment. You can dupe you, capital, get part of metric static, ASIO and then the pot name, and you will get a DC realized version of the metric and can inspect the values.

A

So what you see here, we have the pot metric in this case Gravano in the namespace monitoring, and it actually contains two containers inside this part, one with the named refer nor the other, continuous unnamed, and there you have the CPU and memory matrix.

A

So the question is: how do these values get in here by the means of prometheus.

A

Prometheus adapter ships with a simple config map that you can configure and to configure resource metrics. We have a section that is called resource rules and the first thing you simply specify here is CPU and a corresponding container theory that will give back the container CPU usages for for this constraint. There's this metric here container CPU usage seconds total comes directly from C advice. In this case. The nice thing here is the whole.

A

Theoretically, this metric could come from anywhere right, the most canonical deployment and also the sample deployment that we ship in the Prometheus adapter comes pre-configured with three advisor metrics, but theoretically you could grab those values from anywhere else. In addition to that, you see you have those templates variables called label measures, a label mature and grew by and I will give you also a table on what those mean exactly.

A

You have the container theory net con next comes the knot, theory for CPU metrics, literally the same same thing, and then, since these are Prometheus metrics, you have certain labels on those metrics attached and you want to associate those labels with a resource type that I available in kubernetes for resource metrics. Obviously these are nodes and parts in addition to namespace, and this is exactly what you do in the resources section of the promisee setup, the configuration what you say here is okay, Prometheus adapter.

A

Please take the label node and associate that with the note resource type in kubernetes and when you have deviations from that, you would say: okay, whenever we have a label called pot under screw name, please associate that with the resource type pot, pretty straightforward. In addition to that, we also since for the pot resource metrics. We also have a distinction between containers. You also specify explicitly the container label inside inside your metrics little tip for debugging. This Prometheus is a very nice API endpoint the series API endpoint.

A

So if you are unsure, oh, what labels does my metric have and what do I associate with? Simply take the label at take the metric name stitch it into this slash, API, v1 Col, request against the Prometheus and there you will have a nice listing of all the series that are available, and then you can the labels that are of interest for you and associate them with the resource types in communities. So in this case, obviously container name is associated with the container. Namespace is associated with namespace pot name and node are pretty obvious.

A

Same goes for memory, the one that I showed you here were CPU metrics the same exactly. The same applies for memory metrics with just a little different metric name coming from prometheus.

A

In addition to it, to this just be careful to have a match whenever you have an aggregated function like in this one, the rate function and you have a window specified. Please set also the window here, because that is the same window here, because that is exactly what communities also expects you to incur in figure. So please be aware to set this to the same value.

A

Custom metrics, that's actually like the second gear, which those metrics are pretty simple. They are just to metric types, no, not metrics and pop metrics for custom metrics. These are metrics that are coming from you, like abilities, doesn't know which one you would like to expose inside kubernetes, for instance, for the horizontal pod autoscaler.

A

But there are certain constraints which you have to fulfill before. Exposing those custom metrics, the biggest differentiator matías explained between custom metrics and external metrics, is that custom metrics are always associated with the concrete kubernetes resource, a pot service, a job deployment whatever there is where, as external metrics, the only Association they have is the name space. So there there is no connection to concrete artifacts inside Cuban Ares. Therefore, custom metrics are a little bit more complex to configure again the same thing.

A

If you want to see if custom metrics on abled correctly in your deployment check out the Q cattle gate, API services v1 better one custom, metrics, KS IO, and then you should see if it's available or new or not to see which metrics are available inside your cluster. It's not as nice as with the resource metrics, because again you define them yourself. So there is no automatic discovery mechanisms of the metric types that you pre configure, but you can still use the cute couple.

A

Rock command and say cube cut, will get raw custom, metrics, Kate, Sao and then kubernetes will print out all the known custom metrics that are configured in this case. In this example, we configured HTTP requests per second, which we want to expose and feed this value back into their horizontal, put out a scalar to scale up and down your parts.

A

So how does this work? How do we expose those things? Well, there are four steps in the parameters adapter which you have to configure. First, you have to somehow specify a/c risk theory that will let the permutation have to discover all the custom metrics that you want to slurp into kubernetes. That's the first step. Second step I showed you already and then resource metrics how that works out. You want to associate because these are custom. Metrics, all certain labels with kubernetes resource types, parts services deployments. Second step.

A

Third step is you have to define the name of the exposed metric inside kubernetes? You may, for instance, apply certain aggregation functions like the write function to a raw metric, so you maybe want to add the suffix per second to the metric exposed inside communities. So there is another configuration section for that and finally step four. You want to specify the actual theory that is being executed against Prometheus whenever you get a concrete custom metric.

A

So let's go through the configurations, one by one discovery: series query: as I showed you we want to expose HTTP requests total per second, so you could say: okay, I have a Prometheus and I have certain metrics inside it that I'm already stripping HTTP requests total. Maybe not all of them are related to parts that are being scraped, so you would add another constraint, namespaced not empty, and by doing so you say the parameters. Adapter.

A

Okay, please execute this serious query against the Prometheus API that one will return a list of metrics and all of these metrics will be considered the ones that you are interested in to be present inside Cuba Nettie's, as I show you before what we did essentially to debug these things and configure Prometheus adapter ourselves.

A

You can take this exact series theory and execute it yourself against this end point, so you see exactly the same result that the primitive promises adapter will observe next step Association and the resource metrics I showed you one way to associate the labels with kubernetes resource types, namely explicitly say: okay, node corresponds to the node label. Pot, underscore name. Port underscore name label corresponds to pot resources, but you could say well my labels match exactly so. My metrics have pot name space service, whatever labels are known also to kubernetes resource types.

A

In this case you could simply use this template variable dot resource. Beware, though, this is dangerous. We were bitten by it, and I will show you exactly in an example how, namely, when you stitch this to the serious matching endpoint of Prometheus. You will see the name space part and service labels are correctly associated with the corresponding types. Well, we have to jump label which is pretty ubiquitous inside primitives, and this one will be simply associated with the jobs resource type inside kubernetes.

A

So please be careful if you use this dot resource templates template because you might have too many associations.

A

Third step naming once you know which metrics you want to fit into the Prometheus adapter. We have a reg X based configuration scheme where you say: ok, whenever these series come in take whatever is that starts and ends with the underscore total, have a reg, X group defined and use that that one and suffix it with per second so HTTP requests, total becomes HTTP requests total per second right. That's that's a concrete example here, final step, the actual theory, then that is being executed.

A

Well, in this case it's a simple sum over rate and pretty much all the values that are needed to be feed it into this theory already come from the configuration above like the series name, label match isn't and the like. So you can use those template variables down here to configure the key fury, what you get as a result, when you involve cube, cut' will get raw custom, metrics, KS IL for a certain namespace and a certain part and HTTP requests per. Second, you get exactly what what we wanted to achieve a concrete value.

A

In this case, it was 5,000, Milly's, HTTP, requests per second associated with this concrete part when you bump up the verbosity level of committed after you can also copy paste, this query and execute it yourself against Prometheus and recheck. If the values are the same so which template variables are available inside the parameters, adapter well, first, the series name: that's the one that you essentially also specify and the series query so in this case is HTTP request total. Then you have a pre rendered version of all the associated resources like in this case.

A

Namespace come apart. It's a comma delimited list. Then you have all the label matches which is the same as group by but includes also the values so namespace equals default pot equals some name, and then you have two additional template variables, which one is a map, the label values by name and the group by slice which you could use in standard goal. Template range queries to do more elaborate, like query definitions,.

A

Finally, external metrics good news here is the configuration, fortunately, is nearly exactly the same as with custom metrics, with one biggest differentiation. By the way again, the same same procedure is above to get the API services and check. If this metrics endpoint is available, you have exactly the same steps as with custom metrics. You have for discovery. We have the Association naming and clearing.

A

The Association is different, though, because external metrics are not associated with communities, resource type types with one exception, the namespace external metrics are per definition in cuba, Nettie's namespaced, so you would have still to associate some namespace label that you get, and this is a concrete example. Again, this feature was was merged very recently. This is some external metrics. I took literally the same example. Http request total, which will be in this case associated with the cube system namespace coming from the control plane.

B

So yeah we merged the external metrics support. I, don't know like three weeks ago same.

C

B

So it's it's pretty new, but we are in the future kind of already know. We want to move the Prometheus adapter somewhere else. It's still in service namespace, which is fine, but we want to like make it a bit more obvious. That's a really supported thing by us. At least. We want to really take out the config complexity, as you saw it's quite a lot yeah. So that's like going to to be improved by savior's, probably yeah and.

A

For this, I would like also to do like a little survey. The whole configuration story that I showed you who liked this configuration hands up. Very, very ok, so I agree with you personally, I also don't like the way we currently have parameters set up to configured. Please if you would like to use Prometheus as your first class metrics back-end. Please tell us or tell me or Matias, you stole how you would like to have the configuration to happen for you as a best case.

A

That's that's the input that would I would love to get here as a QA after this presentation. So.

B

Yeah, next to the attack of the scalability issues, flatten the library, dependencies and yeah, we are thinking about actually creating a CR D to kind of wrap everything that we just shown in the configuration and make it a bit more high-level, a bit more approachable yeah. So that's the future plan. So I said you said. Please approach this and if you want to talk about specific use cases, here's a sample HP a configuration just to mention this.

B

We have this HTTP requests per second and there's actually the metric that was defined in one of the earlier examples. So we can now reference these metrics in the horizontal pod, autoscaler yeah, so yeah the target average value is 1000, and if that threshold is crossed, we're gonna like scale up the number of parts that we have and the minimum replicas is one and we add most have 10 replicas for for that specific deployment.

B

So if you want to actually try the Prometheus adapter or everything we just talked about, we are also the maintainer of the Prometheus operator and the Q Prometheus project, which we finally moved out of the old repository. So it's a separate project now and yeah. The Prometheus adapter is part of the Q Prometheus project, which assembles all the good things like cube, set, metrics Prometheus adapter, not export or like yeah graph Anna. Even so, we basically ship a full class time.

B

Monitoring stack that you can just do a cube cut and apply on the manifest folder that is pre-rendered for you and you already have the Prometheus adapter once final. One single note on this do make sure that you don't have the metric server installed on your casa. Otherwise it were failure cuz. We have this matrix API and only one thing surf.

B

This metrics API at one time so make sure to not have the metric server serving that API and then you are good to go if you're interested in this topic, please to get involved, come to us talk about your use cases, but even better if you want to really yeah develop this kind of stuff upstream with the kubernetes community. We are part of the sig instrumentation, and parts of this are also interesting for the auto scaling. For example, it's used by the horizontal pot autoscaler and both have a mailing list and be weekly meetings.

B

I assume. So thank you and questions.

B

Please show your hand again: yeah yeah,.

C

Hey one question: is it possible to use several Prometheus instances with the doctor why I have to put the rate metric to 1 and then only link it to one so yeah.

A

It is definitely possible to use separate from it either. I think I should don't have to repeat the question because it was done with microphone. It's definitely possible but per metrics API endpoint. So that's essentially also the same thing that we are doing in the commercial version context that we are shipping in OpenShift. We have a Prometheus adapter that is being deployed that points to a dedicated cluster monitoring Prometheus and then what you could do.

A

In addition, on top of it, have a separate primitives, adapter deployed as completely separate deployment and pointed to a Prometheus that is hosting workload, metrics and that's I would even say even recommended and consider the best practice.

D

Ok in the previous version when, for example, your matrix was not present in Prometheus at some time, you have to restart every single time the the adapter going to talk. To this thing, sorry, you have to restart so when you do discovery for the matrix and the matrix are not developing parameters, because the compound has not been started and so and so on. So how do you tackle this thing? So.

A

As I understand, your problem is what happens if Prometheus adapter is not able to discover all the metrics that you are interested in right? You have to simply kick it. Yes, exactly so there are retries being made as part of a regular HTTP client retries. But if you have a Borg Prometheus that is serving wrong series and point, then, yes, you would have to Rick again.

E

Thanks for the presentation and for your work on the project, first of all, could you go over how to configure the kubernetes echo great aggregation layer in terms of certificates and how to have the Prometheus adapter generate the correct certificate so that even system anonymous requests get accepted? Oh that's.

A

That's a good question. Yes, so there's actually a question on how to configure aggregated API endpoints and there is an API servers resource type inside kubernetes that you essentially it has a couple of sections inside that that you tell the kubernetes api hey. I have an adapter, it may be the permutation adapter, but it may be something else deployed and it's reachable under a certain service and it uses this certificate right, but that's only part of the story. They're.

A

Actually, three methods on how to configure like this mutual trust, relationship between the API server and the Greg Righetti API server. So TLS is one of them they are actually.

A

There is also an example in the deployment of the Prometheus adapter I'm, not sure if it uses already certificates- or it still uses unsecure skip verify, but I can, if you want to come afterwards to me, I can definitely punch it to our repository inside the cluster monitoring operator, where we have all of this setup, which can give you a guiding path on how to set up TLS certificates for API services by the way. This is not really nothing to do with like the metrics adapters. This is ubiquitous topic like across all API services.

A

I can give you a concrete example. We have some open source yeah.

B

I think Q Prometheus doesn't have us, but we have it in OpenShift yeah, exactly yeah. Any more questions raise your hand. I happy to run around I need some exercise.

B

Now I need to run No.

F

So in a multi-tenancy cluster- and you have multiple teams in different namespaces can say, configure the adapters for their own metrics themselves, or does it have to be done by the operators? That's.

A

Actually, a very good, very good question. Unfortunately, this is not 100% possible, as if today, like multi-tenancy is definitely something we want to tackle also in the context of OpenShift, but this also will need more involvement of the API machinery, because currently API services are not names based once that is possible and you can define an API service, for instance, custom, metrics, /, namespace, and let them point to different services. Then you would have nice tenancy model and you would deploy a Prometheus adapter per tenant.

A

But currently we have a couple of hurdles essentially yeah implied by the way. Api services are architected any.

B

Other questions: how are we I'm talking by the way? No.

A

Two minutes all right: two minutes.

A

No questions: okay; okay, thanks, okay,.