Cloud Native Computing Foundation PromCon North America 2021, 1 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Who Are You? A Heuristic Approximation to Service Identification - David de Torres

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Who Are You? A Heuristic Approximation to Service Identification - David de Torres, Sysdig

Standard Prometheus annotations allow scraping applications via service discovery. This works great when all the jobs are similar, but what if we need to apply some special relabeling, drop metrics or add new labels? Sometimes, it is possible to tag the workloads with new annotations that can help to identify a specific application. However, most of the time, restrictions on permissions or restarts of critical services can make this hard to implement in production environments. In this talk, David will introduce techniques to identify the application running in a pod using the information provided by the service discovery. To do that, he will present the use of heuristic filters and its different applications and use cases.

A

Hello, everyone- I am david, the turtles from cystic, and I will talk about service identification with prometheus.

A

First of all, let's talk a little bit about service discovery. Service discovery is just great. It allows us to scrape almost anything with just a couple of annotations and just a single job. Here there are some samples for sure you're familiar with in this spot. For example, we added the scrape annotation equals true and the port number here. We added the path, for example, and all of them will be a script with just single job. That for sure you have seen before this is great.

A

But what if we want to make something special with some of the jobs? Some of the services like, for example, adding some extra labels for enrichment or whatever we need or drop in some labels of a metric that has a high cardinality to reduce that or just directly drop some metrics that we don't use or adding some relabeling that we need for some metrics. We just cannot do that if one job makes everything we need to identify the pots, but we still want to use the default prometheus. Notation, so what can we do?

A

Well, here's the first idea: we can use a new annotation here. We use, for example, prometheus.ios service and in this spot, this new annotation equals service a to make a new job that identifies and select the bots within the annotation. We just add the action keep with the source label, the new annotation that will create a service dot. I service equals service a and this job will select these spots and scrape only these spots.

A

After this we can add the drop in labels, metrics re-labeling, whatever we need also remember that you have to remove these spots from the generic job, because else it will be scraped more than once. How would we do that? Well here we added a new rule that says: drop ultimate, all the bots that has the annotation not empty.

A

This is okay if we can use the annotations, but this is not always true. Sometimes we cannot just annotate thoughts, because maybe they are in restricted environments that we don't have access to that. Maybe we don't even have access to the pots of deployment, because we are not the owners that or maybe we have access, but they cannot be just restarted because they are critical services.

A

So if we cannot annotate the bots how we can identify that onepot belongs to a application or a service. Well, here is the idea. We can use the information that the service discovery offers and make some heuristic rules. For example, we can use the container name inside the board or the port name to say: okay, if there is a port where it has a container with the name kita operator, metrics api server.

A

I can say that this is a kid operator or kida api server, and I can identify that and I can make a specific job for that, or maybe I know that the service is exposing the port 9113 and it is annotated. So I can say: okay, I will select all the pods annotated with the prometheus.io port equals 9113, and I will use that for making a special job for that.

A

Also. This has a limitation that remember that you have to add this same rule with the drop action to the generic rule, because else, as we saw before it will be scraped more than once so as conclusions we can say that service identification rules are easy to implement and they are powerful because it allows us to make special thanks with these spots, but it has some limitations like, for example, as we saw, there is no standard annotation for that right now.

A

I used, for example, prometheus.ios service, but maybe you can use prometheus.ioslash application or service name or application name or integration type whatever you can come up with, so it depends on the implementation.

A

Also, as you saw, one of the problems that we are having is that we are still using the prometheus annotations and the pot will be selected more than once and we will describe more than once making this metric duplicated. So how can we prevent that here? We use another rule to drop metrics, but maybe we can use a new option in prometheus to prevent the body or the endpoint to be scraped multiple times, even if it is present in different jobs.

A

Also, we are using some information that the kubernetes server discovery offers, but maybe there can be more information available to make better heuristic routes like, for example, the image name of the container.

A

I hope you enjoyed this talk and you saw is useful. Enjoy the prom com see you later.