Cloud Native Computing Foundation PrometheusDay NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Boost Your Logs with Prometheus! From Logs to Metrics - Jaime Yera Hidalgo, Sysdig

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Boost Your Logs with Prometheus! From Logs to Metrics - Jaime Yera Hidalgo, Sysdig

Sometimes, finding useful metrics from applications is not easy. Prometheus is the Cloud Native standard for monitoring, but some applications still have more info in their logs than what they provide as metrics. Logs have always been used to track application problems and warnings, but with observability technologies like Prometheus and all those metrics, it could be tough to open that logs and gather that information. Using open source projects, such as Fluentd, mtail, Grok exporter, and others, we can generate new information as Prometheus metrics from these logs. This useful information can be combined with actual Prometheus metrics collected by exporters or directly from the application itself. This way, we have Prometheus as a complementary tool to the logs for troubleshooting problems and alerting on issues. In this talk, Jaime will explain how to create Prometheus metrics from another pillar of observability: log files. The audience of this talk will learn: - Scrap useful information from logs - Turn information from logs into Prometheus metrics - Combine scrapping tools with monitoring applications

A

Good morning, everyone.

A

's engineer assistic and also I'm a trainer of the open source project data today and here to talk about different ways to take advantages of your useful information inside your locks and don't all that into a proper Matrix for your civility systems.

A

So listen to it. The first question: why metrics from logs? Well nowadays it's very common that we have an environment, kubernetes cluster, for example, and it's very common that we are already instrumented that application. We make a port forward to sonofin all done and as a type slash Matrix and recover or that useful data.

A

But sometimes we try to deploy its own application. That is not interested instrumented or don't have any exporter either, and we try to make the same things type, slash, Matrix and recover the data, and we can't find anything. But maybe the other reason is even with all the data inside our metrics.

A

We can enrich all of that with information in our logs, because we have the bike size, that we have the Response Code Source destination AP, and we can turn all of that into metrics because pick up the information make sure an operation in that log traces and to reacting to metrics and in order to explain different ways to do that, we have to split the chalk in two contexts.

A

The first one is the Scorpion loss from containers, because it's a smaller one, for example, we have an Apache application that is gonna write these logs in such some volume, and we have to share this volume with a specific application that is gonna, write these same logs and turn that information. In this case, we are going to explain the grogasporter.

A

uh First of all, we configure the source of information.

A

It's very simple: we have to deploy our Apache application and specify the volume that we want to share.

A

And after that, we deploy our Glock exporter.

A

The only two things we have to apply here is the growth configuration and the same volume that Apache logs are using for its logs okay, so we have our logs in our application and, as you can see, we have data information that we can take advantage of, and this is the final result that maybe we are looking for. We have our metric description. We have labels inside this metric.

A

As we can see, we had a goals or the method or the path that we are requesting and Grog used a useful fin for that. That is patterns, as you can see. A simple lock line here with the time series or the log level and the rest of information in the lock line, and we can use in this case. Pattern is another thing that says yes, expression, but user suppressing could be very hard to read when we configure something. So in this case, girl save this information in some kind of level and.

A

For two reasons, the first one is easy to read and the other one is. We can use several patterns in order to create another one bigger, for example, Apache already have several patterns and create another one for a whole Trace line, and the other thing that we can make with this pattern is save it in two levels.

A

In this way, we can save our information in those labels and use it after that, for in order to build our metric.

A

So the word configuration with this pattern is very simple and, as you can see, we have the patent directory here we will have all our regist Expressions, save it in our labels and obviously we can compose on other patterns with this patterns already created and use it. As we can see here in the metric session, we are going to build this or metric and created with these labels methods, but who we can take.

A

This information is with the combine, Apache log pattern, and maybe it's very crazy, because we only have we only see this IP or host or user, but in the other side of the label we have the information that we are saving. In this case we can see this double rocket and birth request and response and I already have in this pattern called combine Apache log. So using these patterns we can recover the information and we can create this metric from the log information.

A

So it's very simple to use the clock exporter and, as I said, it's very easy to configure the growth exporter be customizable. The metrics are very custom. We can create from scratch. We can create our regress Expressions assign this label this value to some label like in our own patterns, but obviously we don't have only the growth exporter. We have another application that we can use.

A

For example, Google has the M Tech and is kind of different, because obviously you're returning the information as education or a Prometheus.

A

um Use entity programs is more synthetic, we can have conditionals or constants. We can never use it. I won't explain how to code some example with mtel, but only take into account that there is not only one choices for our context of work.

A

The other context um is a very common environment is scrapping from our kubernetes clusters.

A

In this case, we won't read for any volume and we can use the growth exporter and obviously we we are our Environ environment is a node kubernetes cluster.

A

We obviously deploy several applications because it's a common environment and in order to write our logs, we are going to write from the standard output to a standard error. This way, kubernetes will pick the information and save it into the no log.

A

This way, if some application go off and go over again, we save it the the status and, let's see save it, of course- and maybe one of the application that we can use for for this case is one application that is deployed in every node, as a demon set and for this example we are going to use fluentd fluent is very easy to configure and deploy in our clusters. If you don't want to make any specific configuration, we have our hand chair and with barely a few options, we can deploy this application in our clusters.

A

So, as I said, with the growth exporter, in this case, this Contour metric with several label in order to get more information in our metric and the source of information. Is this basic line of nginx lock?

A

We have the the method we have the endpoint request, the agent timestamp the apertures, but for this KO we only have we only want to to make the level for the method or path or status code.

A

So, as I said, with Grog exporter um fluently obviously use reass expressions, but for that purpose, fluently have something called plugins use. Plugins for everything for reading the source of information in order to filter that information make some combination or return that information in in Prometheus or other patterns.

A

So this is the kind of radius expression that has fluently for an engine is lock, lock, Trace because we're very, very scared at the beginning, but obviously we can select our specific values that we want for this case, as I said, the method or path or the status code. For for that reason we only have to select this label for this case for select the the post method or recover the path with this 3DS expression, and we only this status code.

A

We recover this value, so this is the Syntax for in order to retrieve information for any source and, as I said before, fluently use and and useful thing called the plugins in this case for the source recovery, we are going to use the tail under yes, plugin. The first one is in order to read from the kubernetes node and the other one is uh in order to use our yes expression.

A

You will have to locate our registration that we saw before in this context, and we have to attack this source of information with the attack we want in this case is engine X. This is very important because we did test after that. We filter that information. We indicate that we are gonna use this filter for that log, Trace and obviously again in this case, in order to turn that information in a Prometheus pattern metric.

A

We use this already plug-in created by the community for promises and after that we create our custom metrics for the information we recover for the lock line.

A

In this case, obviously, we have the name of the of the metric. We say that it's a counter time metric the description and the labels, as we can see here between this Dollar Arm Embraces. We have the method, path and status code, that this is the same label that we have in the previous slide and win this. We have the the material we already create.

A

It's very simple, less as I see here with with you and it's the same, it's kind of similar to rock, because we are using the radius expression, but obviously we have other tons of plugins because fluently have a great community and is increasing day by day um in order to summarize a little bit fluency, um it's kubernetes native, because it's very simple, as I said, we have a home chart that we have Bali to configure in order to deploy in our clusters.

A

um As we said, we deploy, as a demon said. So um we have a look aggregation very centralized and talking about plugins is the more authentic or useful thing that have fluency because has a great community and have very tons of plugins that we can configure for several sources or several tone of information or many other filters.

A

Obviously, in for kubernetes clusters, we only have we don't have only fluency, obviously- and in this case the the same guys that build this fluency are brilliant right now uh flowing a bit well right. Now it's already wielded and it's increasing day by day. You don't have the same amount of plugins that have through MD, but it's increasing day by day.

A

We can select this kind of fluid before our contest with low resources, because is the difference with fluently. You don't need the same memory that had the that. Obviously we had to select between this this both applications and obviously there is another applications for in order to aggregate lock information.

A

In this case, we had obviously prone tail I, let prontel here, but it's not alone, because neat from Loki in order to recover information, is deployed as an agent that recover all the information inside our nodes, aggregate order, labels and sending back to to the lucky main server and it's very powerful. Maybe it's a difference between other aggregators is located in the information use labels for kind of aggregate information, and so it is after that in grafana and it's very powerful, um usually in order to maintain the same thread for this talk.

A

I already talked about grogasporter or or the Intel friendly or friendly and prone tail for for kubernetes cluster. That I want to talk about that. We have different choices for our contest or our environments. We don't have to stick together with with one of them, because sometimes our contextual environment changes, and we have to know that uh we have different applications to choose and don't stick with one of them.

A

So that's all. Thank you.