Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Prometheus Introduction - Julius Volz, Prometheus

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Prometheus Introduction - Julius Volz, Prometheus

Prometheus is an open-source monitoring system and time series database. It features a multi-dimensional data model with a powerful query language and integrates many aspects of systems and service monitoring: from the instrumentation of services over the collection and storage of metrics data, all the way to dashboarding and alerting. Native support for various service discovery mechanisms also make it particularly suitable for dynamic cloud-based environments like Kubernetes. In the introduction, Prometheus co-founder Julius Volz explains the architecture of Prometheus and highlights its key features.

https://sched.co/Zex4

A

All right, hi, I'm julius, I'm the co-founder of the prometheus monitoring system and I also recently created a company around prometheus, called prom labs and in this session I'm going to just give an introductory talk about what promises is, what are some of the main features and why you might want to use it.

A

So first of all, prometheus is a monitoring system that is based on purely numeric metrics.

A

It cares about the entire chain of monitoring, meaning from getting metrics out of things you care about instrumentation, then collecting those metrics, storing them and making them useful.

A

You know for dashboarding for alerting and so on, and it really doesn't focus on any single part of the stack, but is useful from networking devices up to the application layer, anything that can expose metrics. Basically, it also works, especially well for dynamic cloud environments. um You know, especially in the kubernetes environment. This is very relevant because both kubernetes and prometheus have mutually good support for each other.

A

We also explicitly try not to do everything, so we don't do logging or tracing which are important in their own rights, but prometheus only focuses on numeric values that change over time.

A

We allow you to define, alerting rules that might be complex potentially, but they have to be explicit, so prometheus also doesn't do automatic machine learning, style, anomaly, detection and, while prometheus's local storage is pretty efficient and good, it's still tied to a single node. So it's not a horizontally clustered complex, a durable system.

A

It's really geared towards live monitoring.

A

So prometheus started at in 2012 at soundcloud when matt, proud and myself came from google to soundcloud, and we found already a cluster scheduler at the company there. That was in-house built, and this was before docker before kubernetes very early on, and there were no proper oss monitoring tools that could deal with that situation. Well, so we started building prometheus first in our free time and then more and more on soundcloud time and we're really inspired by google's pokemon. When we did that.

A

So you know once it was somewhat successful at soundcloud we fully published it in 2015 and then a bit more than a year later we actually joined the cloud native computing foundation just after kubernetes. The first project- and you know by now prometheus is an independent project where many companies are working on and basically you know many many companies are using it all kinds of small startups to huge corporations and you can find prometheus the project itself at prometheus, io.

A

So, let's look at the overall architecture of prometheus for a second. So imagine you have some. You know web applications or api servers where you can control the code yourself, and you want to monitor these with prometheus.

A

Somehow you need to expose metrics to prometheus over http, so prometheus actually wants to actively go to the different things you care about and pull metrics over a format that you that prometheus defines so the things that you care about that you want to monitor. We call targets in prometheus and the best case is when you can actually um modify the code of these things include a prometheus client library into the code that tracks metrics, like counters, gauges, histograms summaries and then exposes those metrics every time that promises comes by over http.

A

Now there are some other things where you can't actually do that. Necessarily, for example, you know you might have an actual linux virtual machine or the mysql daemon c code base, or so things where you can't easily just add a direct prometheus metric server too. So we have another pattern for that. That's the exporter pattern. Where the you run an extra process next to the thing you actually care about, that does the translation, from whatever the backend metrics format, is to the prometheus metrics format.

A

So, in the end, then prometheus knows about all these targets and scrapes these targets in a regular interval, and you might be wondering what does the actual format get looks like that gets transferred over the wire here? So let's have a brief interlude about that.

A

It's a simple text-based format, where every sample of a time series is on one single line, giving you a metric name and then uh labels that further specify the sub-dimension of what you're measuring and then the actual current sample value. So this only always transfers the current sample value that is either being counted or tracked or whatever, and as you can see, this format is pretty easy to produce and we'll talk about the data model in a second more so now.

A

Another question is: how does prometheus know where all these targets are, and this is where service discovery comes in, where prometheus knows how to talk to different sources of truth that tell prometheus what is supposed to be where and how to gather data once it has gathered the data in a local time series database.

A

You can then point tools like grafana, the dashboard builder or edit to create dashboards or you can use prometheuses built in web ui, or you could also build automation against prometheus's query apis to actually do things like autoremediation or so.

A

Finally, you can also use prometheus to calculate alerts based on the collected data for you and then send those alerts to actual humans or machines via a separate service called alert manager, and typically, you would have maybe many prometheus servers in your organizations, but just one alert manager and all the different alerts in your organization come together in alert manager and get grouped over time across dimensions and so on, and then get rooted to the right teams and the right notification mechanisms.

A

So I would say the the four main selling points of prometheus are these four.

A

The data model of prometheus allows you to collect data in pretty good detail about all the different moving infrastructure and service parts in your org, and then we offer you a query language to work usefully with that language to actually ask interesting questions.

A

The architecture of prometheus is also relatively simple, and a single server is also you know it can store. It can process and store a lot of samples and then the service discovery. Integration is the part that makes it work well in dynamic environments.

A

I will now dive a bit deeper into each of these four parts to flesh them out a bit more. So first, the data model prometheus, fundamentally tracks, numeric time series, so numeric values that change over time- um and you know they have some kind of identifier and then they we just once we have indexed this identifier.

A

One time um we just track timestamp value pairs on an ongoing basis for this identifier, so, for example, how the temperature in a given room, develops over time and all the time stamps in prometheus are millisecond in 64 timestamps and all the sample values are float64s, even if you're, just tracking integers.

A

This actually compresses really well under the hood and is tracked nicely so a big difference to other monitoring systems or time series databases that came before prometheus is how time series get identified so that we can better query and filter them in the query language.

A

So what does this look like the identifier of a time series, rather than going for a flat single metric name or just a hierarchical data model in prometheus, we identify each time series first by its metric name. In this case, http requests total and then you know this would be the total number of http requests handled in a given type of web server, for example, and then we subdifferentiate the different dimensions in which specific process this happened, on which specific path, an http request, got handled and so on. In things, we call labels.

A

So we have these key value pairs called labels that help us create the sub dimensions in the metric name. That tells us the central aspect of a system. We are monitoring.

A

Now this data model has the benefit that it's pretty flexible.

A

You know in a hierarchy, you, a hierarchical data model, is a bit harder to change after the fact, if you add a new dimension, for example- and you have to know which component in a hierarchy actually is, for example, the path or the status or the instance, um and in this case it's completely not order based, not hierarchical, and you see very explicitly, which dimension actually means. What so, now that we know this data model, we also have a query language that goes on top of the data model called promql.

A

The prometheus query language- and you know this is a completely new query language. It's explicitly not a sql style language, which sometimes irritates people, but we still think that the way promql works is better for numeric time series computations that you typically want to do in systems monitoring than a sql style language.

A

So, let's just look at a couple of examples without diving into deeply imagine you have a node exporter running on all the different nodes in your infrastructure, giving you file system metrics, for example, the total size of all your mounted partitions in your infrastructure, and now you would one might want to know. Give me all the give me all the partitions in my infrastructure that are not mounted on root and that are larger than 100 gigabytes of capacity.

A

So you can even ask these kind of questions against prometheus, which might not directly be relevant for alert, but just help. You look into your infrastructure, so you know in this case you would just start by the metric name. That gives you the size of all partitions, then filter it down with a negative matcher to all all partitions that are not mounted on root.

A

These are still in bytes, so now we're dividing by a billion to get to a gigabyte and then filtering it down to the ones that are actually larger than 100 gigabytes.

A

And what you get out of this is an actual labeled list of all the different partitions with their dimensions. So you actually know all the metadata about them.

A

Another example, that's pretty common is that you might want to know the ratio of errors to the total request, so your error rate. Basically um in this case you could just select all the 500 status quo requests form a per second rate over them, and then you know some sum all these requests over all the different dimensions, path, method and so on to get a single number and divide that by all the requests, so not only 500 ones.

A

This would give you a single output number for the ratio of errors, but often you might want to preserve some dimensionality, for example, you might want to see this whole result for every path in your infrastructure or in your service.

A

So in this case, you could just take the same expression and add a bypath modifier to the sum which preserves the path dimension in the sum on the left hand, side and on the right hand, side and then prometheus knows how to automatically join the left-hand side with the right-hand side based on identical label sets, so they will have the same path label and give you the ratio for every path now- and you know, there's more customizability around this possible. But this is just a simple example.

A

um You can also do things in prom ql like um without diving, too deeply into it, like collecting latencies, request, latencies in histogram and then estimating, for example, the 99th percentile latency, based on that histogram aggregated over all instances, but still preserving other labels like the path and the method.

A

So you know once you actually learn this language. What can you do with it? So you might just start using the expression browser in prometheus that is built in to kind of introspect, what's currently happening in your prometheus server, so you can show in your infrastructure. You can show things in the table uh by the way. Excuse me for some of the fonts that are bit off this happened, while uh converting to powerpoint from google slides.

A

um You can also graph uh the expressions that you that you learned and then once you want to build a bit more like serious dashboards, you would typically go to grafana and um you know create dashboards with all the bells and whistles that you can save and you can share with your colleagues.

A

um Another thing, though, that prometheus allows you to do is actually base alerting on the collected data. So you know, whereas previously you might have had a separate system for alerting from your time series database prometheus integrates all that with each other. So first collect everything as time series and then you can use promql to formulate alerting conditions.

A

In this example, we want to alert when there's a path in our service that has a larger than five percent error rate. So we take the expression that we saw earlier.

A

uh That gives us this ratio of errors for every path and we multiply it by a hundred to get to percent, and then we filter it down by only the ones that are larger than five percent.

A

And then we add another condition here to say only paths that are in a bad error rate for at least five minutes should alert us and in the end, for the paths for which this is true. We will then get a single alert output element for each of these paths and then an alert manager later on. We can actually choose how to group these into single notifications or multiple.

A

Now prometheus itself is operationally pretty simple, at least to start out with, so it only writes into local storage directly on the file system. You don't need a special database or so it's all integrated. It doesn't have clustering features, and you know if you want to have high availability for alerting. You just run two of the same prometheuses that are configured in exactly the same way and they will calculate the same alerts. Send the same alerts to alert manager and alert manager then will actually deduplicate them on their label sets.

A

So you will only get one notification in the end. It's also written in go which makes it easy to deploy in case you're, not anyway, using something like docker nowadays, where that doesn't matter so much anymore. But it's a nice language to run in the cloud.

A

A single server can also get you quite far. um If you have a big machine, you might be able to get to something close of a million samples ingested per second, and I typically see big prometer servers with many millions of series and the local storage also compresses these the samples that it collects very well. So every numerical sample that it actually collects ends up on disk, typically using maybe one to two bytes with all the overhead around it.

A

So the local storage is good for keeping maybe a couple of weeks or months of data, and I do have to mention that you know some people actually use it for keeping years of data. But then, if you want to do that, you have to be really careful uh not to overload your prometheus to back up your data regularly. You can create uh consistent snapshots that you can back up and so on, um but prometheus is at least initially was more designed as a live monitoring system.

A

With you know a couple of weeks or months of data.

A

So for people who do want to keep data more durably and long term, their prometheus gives you an integration called the remote read and writes protocol where prometheus can send any sample or a subset thereof, that it collects to a remote endpoint to some kind of bigger, clustered durable database, uh either through an adapter or nowadays there's more and more products and projects that actually support this protocol out of the box.

A

And then you know on the remote end. It could really be anything and prometheus can also read back the data from the remote end and do promql on it again, and so we have many many different integrations offering this kind of protocol already, for example, cortex is an open source, one or influx db, or time scale, and many others.

A

There's also a long-term storage solution that integrates in a quite different way, which is called thanos. This is pretty close to the prometheus project as well: large overlap with developers and pretty popular in the community.

A

This works not using the remote right protocol. Nowadays it actually starts to support that as well.

A

But the original idea is that you don't use that protocol, but you add a thanos sidecar to all of your existing prometheus servers and then the thanoside car ships, any older, persisted data on these prometheus servers to an object, storage like s3, gcs or mineo, or so, and then you have other thanos components like the store gateway and the querier which allow you to get an integrated view over all the recent data in multiple different prometheus servers, all in one query and the object, storage, which has potentially very very long-term data and can be durably replicated and the thunder's querier also implements prometheus's query api.

A

So you can just point grafana at it as if you're pointing at a prometheus server and run the same kind of queries all right. So the last point is about dynamic environments and making a monitoring system that works well with them. So nowadays you know we have more and more stuff happening in the cloud with dynamically created vms, and then on top of that we put cluster schedulers like kubernetes, and then on top of that we put rapidly moving microservices that are just being deployed and rolled out and taken down again and so on.

A

So many many processes that are just moving around a lot and are living on different hosts and ports all the time. How do you still make sense of this kind of environment as a monitoring system?

A

So this again is where service discovery comes in uh prometheus supports talking to many different kinds of service discoveries, whether they're, more particular to a cloud provider or cluster scheduler or more generic ones, but in all cases prometheus uses service discovery for three distinct purposes. The first one is I'm a monitoring system and I want to know what should be there.

A

So you know so I can even tell you that something is broken right. The other thing is technically okay. Now I I know something should be there: how do I actually go out actively and pull data from it? um This information typically comes from the service discovery in the form of a host and a port, for example, and sometimes more options, and then the third thing. If it's a good service discovery like the kubernetes one, it will know certain details about the the target, the object that you discovered.

A

So it can tell you you know a particular part is in environment, equals production part for example, and then it allows you to map that metadata into the actual time series labels that you collect, so you can use it later on in the query language, so prometheus has built-in support for over 10 different service discovery mechanisms by now, yeah some are for vm providers, aws, azure, google and so on to actually discover instances.

A

Others are, you know, for cluster managers, for example in kubernetes. You can say stuff like give me every endpoint give me every service give me a parts, give me ingresses and so on, and then you have more generic ones like dns and so on, and even a custom plugin, where you can build your own service discovery if it's not one that is built into prometheus yet, and this works over a file based watcher mechanism.

A

So, in conclusion, prometheus is a monitoring system that works well with these dynamic environments and can tell you what's broken and also allows you to make nice dashboards and with the data model in the query language, it allows you to get really good detailed insight into your infrastructure while being able to use that both for dashboards and alerting- and you know it's implemented in a pretty simple architecture with good efficiency and the service discovery. Integration really makes it work well, together with dynamic environments all right. Thank you.

A

All right, hi now we'll get to the live q, a part some people have submitted questions really built in chat in the platform. um So, let's start with the first one, so someone asked: can it also be used for snmp monitoring and also monitoring the devices via sorry via rest, api interfaces? Is it fully open source or also enterprise?

A

um So, first of all, yes, you can monitor snmp devices using prometheus. There's a an exporter called the snmp exporter. If you google, that it's part of the official exporters that we have in the prometheus github org and yeah, you basically provided with a configuration of the network devices that you want to scrape and then prometheus, basically scrapes those network devices through the snmp exporter. The snmp exporter is the thing that then actually runs snmp against your network devices and translates metrics back into prometheus format.

A

A

A

So, generally, prometheus itself can only.

A

A

A

Easy to convert any rest based interface into this format, so you will need to use some kind of exporter.

A

The first thing that I would recommend is going to prometheus.io and in the search bar search for exporters and integrations. Very often, you will already find a custom exporter integration to scrape exactly the type of system that you want to get metrics out of if it doesn't exist. Yet there is also um something called the json exporter.

A

If you search on google for a prometheus json exporter, this allows you to define to use like it's basically a generic exporter, which you can configure to look at any json endpoint, which might be part of a rest api and then translate some of the components in there into prometheus metrics.

A

And of course you can always build your own custom exporter um and then is it fully open source or also enterprise uh prometheus. The main project is fully open source, there's no company behind it. It belongs to the cncf. uh But there are many companies offering different. You know hosted monitoring services around it, that you know, do long-term storage and sometimes do prom ql or sometimes doesn't don't, and sometimes they integrate alerting in their platform and some some of those interfaces are maybe compatible with prometheus and others are their own.

A

You can find out more about commercial stuff. If you go to prometheus io, there is a section called support and training and there's at least some companies listed there, but there's many more companies. You know offering support around prometheus.

A

Can you recommend any resources to learn from ql? So I think I mean this is a shameless self plug, but a long time ago or a while ago at least I wrote a two-part tutorial, which I many people told me, is very helpful if you search for querying prometheus digitalocean on google there. So if you just say, crank prometheus digital ocean on google, you will find two parts of a tutorial that is named how to query prometheus on ubuntu 14.04.

A

So you'll see it's not 100 up to date, but if you need to understand the language from the ground up, I think that's a good resource to start with and then of course, in the commercial support section on our website, you will also find trainings.

A

My company does trainings brian's company, robust perception does trainings the linux foundation does trainings um so yeah many many places to learn uh in youtube videos and so on.

A

um Let's see brian wanted to send me a comment about savix.

A

Okay, so there was one question about in your opinion: what what's the best way to integrate prometheus with old tools like zabbix, please, if possible, show me where I can find out about this. um Okay, brian just sent me a link. There is okay brian. Do you want to put that in as a public answer to that question um that would be cool, so brian has a link on zybax.com. That mentions something about prometheus integration.

A

A

Because because I don't know much about zavik itself, but there's many ways to integrate with prometheus um what about the semantic of value? In your time series I come from the networking world where we have data models which have strict semantics, like mibs and y-a-n-g, netflow, ipfix, etc. By just having a metric name such as http requests total. To take your example, we lose the semantic definition defined somewhere else how to use the prometheus metric name while not losing the semantic definition defined somewhere else.

A

Yeah, so I mean prometheus itself, has a pretty loose schema and data model. um Prometheus itself doesn't really have a programmatic understanding of what exactly a metric name means it does allow on a slash, metrics endpoints in the in the transfer protocol to attach a type of the metric and the help string and the unit now nowadays, um so the the help string would be a human, readable interpretation of a metric name.

A

um Basically saying this is the count of http requests served by this binary or so, and then the type could be a counter metric or gauge metric or histogram summary or unknown, um and then the unit is kind of a future thing which which we're not really using yet um and currently so for the longest time. Prometheus hasn't done anything with that.

A

But currently we do ingest that data uh into the in-memory portion of prometheus and there's an api to retrieve it four give metric names for a given target that it scrapes it from so you eyes can start working with it so, for example, in grafana it I think it in the explore mode. It already shows you the help strings for a given metric name when you ought to complete it and stuff like that.

A

um But it's not you know deeply integrated into any correctness checks yet or so we're we're hoping to uh thread that kind of information more and more through to also remote storage systems. There's currently work going on on that and then making more and more use of metadata like that in the ui and the query language. At some point. Maybe it's going to take prometheus 3.x or so, but we can do some uh stuff already in the current major version, so yeah, that's the developing area.

A

And yeah, do you have any specific reasons? Oh, no, that I got that one! Sorry! um Oh yeah! There was another prom qr question and I already talked about prom qr. What would you say about prometheus operator? So the prometheus operator is a tool originally by core os. Now red hat now ibm to run to automatically run prometheus on kubernetes clusters, and I think it's a good thing. um I would recommend. I mean it's very popular, it's written by people who know both kubernetes and prometheus as well.

A

So there's a large overlap with maintainers there and yeah, but I think the majority of people running prometheus on kubernetes, who I know of run that using the prometheus operator and it's you know it works very well.

A

It takes a lot of details of the day-to-day operation and rollout, and so on and versioning of the config file from you as an operator and otherwise you know you know it does introduce another layer of abstraction and another tool that you need to understand and, of course, if it breaks, then you need to understand the operator as well, but it. On the other hand, it really integrates a lot of the operative knowledge. You need for running prometheus on kubernetes, so I like it.

A

Hi, do you have a suggestion for a tool for automation or structured queries of data in prometheus other than the web ui or grafana the odd for automation or structured queries? So I mean so so. There's one thing that would be plugging my own uh tool at the moment of my new company prom labs, it's called promlenz.comlens.com, which is kind of a power tool at the moment for helping people understand the structure of a complex, pronql query, and you know you can run this.

A

You can use it the preview version on promnents.com, but you can also run it in a docker container, either with a license or the free version locally against your own prometheus servers, and at least you know it helps you understand, a bit more, um which node sub node in a query selects what data where exactly an error is happening and stuff like that other than that, like grafana, is pretty much the standard, dashboarding tool that everyone uses and it does have besides the main dashboarding mode, it has an explore mode, which already is a bit better for just exploring from ql data.

A

So you know looking at metadata and stuff like that: um okay, is it possible to send snmp trap, alarms and brian sends me a chat message saying: snm pretraps no, but I put links in the question for am webhooks put links in the question: ah okay, brian. I guess, then you can publish this answer right and turn off the private mode.

A

Or I will just turn off the private mode.

A

Let's see oh okay,.

A

I turned off private on that answer. Hopefully it will show up now. Okay, next question mine, there's many questions: how would you deploy prometheus in production um roll it onto the kubernetes cluster next to the application? It is monitoring or on a separate vm.

A

I mean typically, if you want to run if you want to monitor stuff on a kubernetes cluster, it's better to run prometheus in the kubernetes cluster itself, as well, simply because it's easier to reach all your targets in the kubernetes cluster you're part of that clusters network and you automatically get the right service account tokens mounted into your prometheus.

A

um So it's you know it can reach all the right things, and can it can do service discovery uh easily against the kubernetes api server?

A

Of course, then, you might be wondering like what is what happens if that entire cluster has an issue, then you, I would still want to have at least some kind of meta monitor outside somewhere that monitors that this entire cluster isn't completely down, of course, but for the applications on the cluster yeah. I would I would have it. I would have the prometheus in the cluster.

A

What are the scalability solutions to scrape from thousands of targets streaming or message queues?

A

um Well, I mean a single prometheus server can get you quite far up to maybe like a thousand or a couple of thousands of targets um if it's a very large server, but ultimately we recommend uh in the prometheus project to start at some point functionally sharding uh either.

A

You know sharding your prometheus installations by the type of service, they're monitoring by team by organization or so find some kind of natural subdivisions to create multiple prometheus servers that each scrape a portion of your infrastructure there's also ways of doing kind of clue: g, horizontal charting where, where you have let's say, 10 pro meter servers and they all have the same targets configured, maybe 10 000, but each of them only scrapes uh one-tenth of those targets.

A

But then you need to build stuff on top of it to kind of find the right one of those to query for a subset of the data and so on. So that's not the super common use case. um There's also approaches like the grafana agent, for example, which is and there's similar tools by other monitoring vendors which you put on every host, and then they scrape exactly only the targets on that host and but that's like the grafana agent, for example, isn't a fully fledged prometheus server. It can only forward.

A

It can only forward data that it collects via the remote write protocol to some kind of prometheus, compatible, long-term storage, um but yeah. So you know, there's there's different ways of functioning charting and then using federation and building um there. If you search for scaling and federation and promises on google, I think you'll find in some ways as well um and then, if you want to get a you like a unified query view over multiple prometheus servers. um Thanos is, for example, a good candidate for for establishing that again.

A

Okay, um what is the major difference between using kubernetes client library for kubernetes api and prometheus.

A

I don't understand the question exactly the kubernetes client library for the kubernetes api. I guess I mean that's just a general client library to do to interact with kubernetes in arbitrary ways and prometheus is a monitoring system that collects metrics on an ongoing basis from the different metrics and points um yeah. I don't. I don't. I'm not sure if I understand that question completely.

A

um What is the? What is the best practices to secure the scraping of prometheus endpoints?

A

So for the longest time there hasn't been any built-in security on the serving side of the prometheus metrics endpoints like if you have an exporter somewhere and the x, the official exporters that we offer uh they wouldn't have.

A

You know a facility to say, listen on https and with a given authentication setting, and if you wanted that inside your organization, we always told people- hey just put an engine x in front with your preferred tls and auth settings and now finally we're getting to the point where, in the node exporter the one that gives you linux and unix metrics we're starting to try out integrating that directly into the exporter, because, obviously a lot of people want that and it makes things a lot more convenient.

A

So you will have in the future once that goes well in all of our official exporters, you will have built-in tls and basic authentication settings, but you know for any support any exporter out there in the world that doesn't support that we still just say: yeah secure it in any way. You want, for example, put an engine x in front of it and configure that the way you want and then prometheus itself on the scraping side can it does support tls.

A

It does support various scrape authentication options, bearer tokens, client, certs, basic auth and so on, so that site is already solved. All right. um Can I use prometheus to scrape the data from kafka, I'm pretty sure, there's a kafka exporter yup. So if you go on promises, io exporters and integrations page, you can find it through the search. There's a kafka exporter which you could use for that.

A

What protocol does prometheus use normally to get the metrics? Is it http? Yes, it is http um so early in the talk I showed the payload and that is transferred over http. This simple text-based format, thanks for your presentation, could you kindly explain the best practice of prometheus federations yeah, so federation is a way for one prometheus server to pull over via a normal scrape or roughly normal scrape.

A

At least some of the collected times used data from one prometheus server into another prometa server, and it's not meant to work as a full replication mode thing where you pull over all the data from one server into another.

A

It's rather, you like should be used to transfer only specific metrics from one server to another. Like aggregations, for example, are very common use case, so let's say you have 10 data centers and then each you have one prometheus server monitoring, only the local stuff and fitting all the 10 data. Centers data wouldn't directly fit into one prometheus server, so you could have a global prometheus server, though that federates only the aggregations for each of the 10 data centers into a global level.

A

So each of the 10 per data center or per cluster prometheus service would record aggregations over this cluster.

A

So it's fewer time series, because it's not anymore for every process and so on, but really for the entire cluster, some kind of sum over everything, and then you could pull that into a global level, and this is also one of the ways you can scale prometheus by building these kind of trees, where you have a more global level and a more local level and the local level has all the detail and the global level uh has the entire world view, but it doesn't have all the detail anymore.

A

So this is one one example for federation, but don't use it to replicate all of the data from one prometheus server to another, because it's not built for that. It's not efficient for that you'll. Basically, it will fail or you will kill your prometa server, at least for very large prometheus servers that have millions of series.

A

Okay, is it somehow possible to integrate prometheus with? I singer?

A

Yes, brian sent me another link, search for nrpe, exporter and brian. If you want, you can add that again as a public answer to that question and make the make the answer and the question public, but yeah nrpe exporter for prometheus is what you want to look for, and this nrx nrpe exporter exposes metrics on commands, sent to a running nrpe daemon.

A

Okay, is it somehow possible to integrate? Oh, we had that question. Sorry. Is it enough to work with prometheus with python background or is golang preferred for working with prometheus itself? You don't need to program at all, so prometheus and all the most of the official prometheus components are written in go.

A

But if you just want to monitor things with it, you don't need to be able to code at all, there's a lot of existing exporters, and if you want to build your own exporter, you can also use any programming language that you prefer. So that doesn't really matter. Of course, if you want to start developing on prometheus itself, then at some point, you'll probably need to learn, go and oh yeah also there's this client, libraries, official client, libraries for tracking metrics and then exposing them in for multiple different languages.

A

So, for example, you know python, there's one and there's java and there's go and so on does prometheus work out of the box with open, telemetry, so open telemetry is another effort to create client libraries uh for tracking metrics, logs traces and so on, and I only know that richard hartman, one of the prometheus people from our team has been communicating with them and working with them and the prometheus or well it's it's.

A

The prometheus format is evolving into a standard called open metrics not to be confused with open telemetry, but open metrics is basically the prometheus transfer format of the future, and it will be one of the supported output formats for open telemetry. So, yes, they will work together.

A

Does it make sense to collect metrics directly via kubernetes api? What are the drawbacks.

A

So again, I'm not sure exactly what it means to collect metrics directly via the kubernetes api.

A

The kubernetes api does expose certain information that you might want to track as metrics, and there are already exporters that translate kubernetes api calls directly into prometheus metrics, for example, if you search for cube state metrics, this is one of these exporters that does that it talks to the kubernetes api server and then tells you like how many replicas of a given deployment there currently are, how many there should be and so on, and translates that to prometheus metrics um and yeah.

A

That's that's one example of that, so that totally works um four things that the kubernetes components don't already just expose as an existing prometheus endpoint. You know if there's already an existing prometheus endpoint, um then that is usually easier to scrape directly all right. That is the last question and I will go over to slack and answer some more questions over there. Thank you.