Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to Export Prometheus Metrics from Just About Anything - Matt Layher, DigitalOcean

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

How to Export Prometheus Metrics from Just About Anything - Matt Layher, DigitalOcean (Intermediate Skill Level)

Prometheus exporters bridge the gap between Prometheus and systems which cannot export metrics in the Prometheus format. During this talk, you will learn how to gather metrics from a wide variety of data sources, including files, network services, hardware devices, and system calls to the Linux kernel. You will also learn how to build a reliable Prometheus exporter using the Go programming language. This talk is intended for developers who are interested in bridging the gap between Prometheus and other hardware or software.

About Matt
Matt Layher is a Senior Software Engineer at DigitalOcean, a member of the Prometheus team, and a regular contributor to a wide variety of open source networking applications and libraries written in Go.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi everyone, my name, is Matt leer, and this is how to export prometheus metrics from just about anything. Just to start. Here's a little bit about me I'm, a senior engineer at digitalocean I'm, a member of the Prometheus team. You can find me on github and twitter at md lair and all the content available in this talk will be linked to from this repository here. My talks, repo and get up so just to start. Let's talk about a crash course on Prometheus, so how many folks here have deployed Prometheus in their environments?

A

Quite a few. It's awesome, that's really great. So what is from ETS anyway, prometheus is an open-source systems, monitoring and alerting toolkit. It uses a pull based, metrics gathering system over HTTP and a simple text-based format for actually exposing those metrics to a Prometheus server over the network. It also features prom QL, which is a powerful query. Language built right into Prometheus.

A

The Prometheus architecture looks a little something like this, but for the purposes of our talk, all we really care about is the Prometheus server in the middle, a display such as graph on on the right and jobs and exporters and other sources of pull based metrics on the left.

A

This is an example of the Prometheus text format. So if I curl the node exporter running on my local machine, we'll see a little bit of information about metrics here, so a metric has components such as a name, possibly some labels that describe different dimensions of the metric, a value which is a raw float64 value and some metadata, including help text that helps you identify what a metrics purpose is and also the type such as counter guage, summary or histogram.

A

This is a super basic example of prom QL, so I just say: Prometheus. Give me the value of this time series from this instance over the course of let's say a 24 hour period, and this shows the hard drive temperatures on some of the machines at my house.

A

So what is a Prometheus exporter anyway? An exporter is a system that bridges the gap between Prometheus and systems that do not speak the Prometheus metrics format, so you might have some service and expose those metrics to something like stance, D or perhaps a head like its own custom, JSON endpoint or axe far from go, but sometimes those systems aren't compatible with from a theist. So that's where exporters come in and tip, you would run these on the same machine as the service itself, but not always as we'll talk about later. So.

A

Some examples of some extremely common exporters include the node exporter, which exposes system metrics from unix-like machines such as Linux and FreeBSD. The mysqld exporter exposes metrics from a MySQL server and the black box exporter, and this is a unique one. It actually dials out to remote systems using things like HTTP or ICMP ping to probe these black box systems.

A

So, let's imagine rolled with a cloud and native future. You've got your Prometheus and you've got your cloud and the clouds up. It exposes a metric that says everything's all good right, so who has an environment that looks something like this Oh.

A

Nobody I was gonna, ask Mike to come work with you, but some of us live in a bare-metal reality. We've got all these switches and core routers and spine switches on top of rack switches. We've got these wrecks of fine racks of bare metal servers, so naturally I log into the router and I say you know. Please give me Prometheus metrics seems like a very polite thing. You would ask her router. Oh uh no such luck. What gives ok so you know Linux is pretty open-source friendly, so there's probably like a Prometheus metrics device file.

A

All right, oh no! No such luck! So where can I find Prometheus metrics for these systems? If you want to find Prometheus exporters, your best bet is the Prometheus website or you can scour the internet, the mailing lists, the Prometheus wiki, I, github and etc, or as a last resort. You may just have to roll your own and that's going to be the focus of our talk today. So, let's start with some of the basics of building an exporter in the go programming language.

A

So to start your function main is going to build your types wire up dependencies and start an HTTP server. So first we create this collector type, we'll talk about shortly and we make the Prometheus client aware of the collector. We set up an HTTP handler and expose metrics over the standard, / metrics endpoint, and then we finally start listening for HTTP connections for Prometheus.

A

So let's talk about the from atheist, go client a little bit. The most important interface is probably the Prometheus collector interface and Prometheus client uses this to actually expose metrics over the network.

A

So our collector structure here is going to be our implementation of the Prometheus collector interface for some service, and let's say we want to expose a metric called requests total, which is just the number of requests that go to some arbitrary system and also we're going to explicitly pass a function dependency here so that we can swap this out for testing. But this function is going to return an integer. That is the number of requests that have occurred and an error. If anything goes wrong. Okay,.

A

So, let's create our collector, we have this constructor here we explicitly pass. Our dependencies makes things much easier to test such as our request function, so we can swap it out and also this is where we actually create the meet of our metrics. So we have our request total metric. Here we give it a name such as exporter requests, total a help text such as the total number of requests that occur meant for humans and also, if we want some label dimensions.

A

But in this particular case we don't need any variable or constant label dimensions, so we go ahead and omit those. So the first method of the Prometheus collector interface is describe and describe accepts a channel of metrics and our channel of metrics descriptions. Excuse me and then it gathers metadata about each metric. So basically you can collect them in a slice and iterate over that pass. The descriptions on the channel and you're pretty much done collect is a little more interesting, and this is the second method of the Prometheus collector interface.

A

So I collect accepts a channel of Prometheus metrics that you can send on and what we need to do first is take a metrics snapshot using our function that we passed in and this must be concurrency safe. So, if you think about it, you might have one Prometheus server, but you could have two or five or ten, so you need to make sure that you lock appropriately, so that nothing can run into like a nasty data race.

A

So we retrieve the value or, if it fails, we can send an invalid metric to notify Prometheus of the error and we can alert on that later on. But if it succeeds, we take that request value we use this. Must new cosmetic constructor passed the name of our metric, the fact that it is a counter type and the raw value of the request will talk about Casta metrics shortly and why those are important.

A

So if you want to build an exporter and go here, my recommendations build reusable packages. You don't want to mix low-level details of some file format or binary Network protocol or filesystem traversal with actually x40 metrics. If you separate these things, cleanly you're going to do yourself, a lot of favors in the long run write unit tests. If you think about it, a Prometheus exporter could be a pretty critical part of your production environment.

A

You need to make sure this thing functions appropriately, so my recommendation is to set it up in tests and perform HTTP GET using suck some fixed set of inputs and a fixed set of outputs and compare the two and finally use prompt to we'll check metrics for linting your metrics. So what this does is if I curl some exporter and pipe it through this tool, it will give me recommendations on how to make my metrics more standard.

A

So in this case we have this metric called X gigabytes, and this is a counter, so counters by convention should have a total, suffix and also you'll notice that we use a unit of gigabytes. Instead, you should use the base unit of bytes and leave that conversion up to your display systems such as griffons.

A

With that being said, let's go get some metrics, so some of the sources of metrics we're going to talk about today include files. Hardware, devices and system calls and there's definitely some overlap between the three to start. Let's talk about files in particular gathering metrics from proc stat on Linux soap, rocks that contains kernel and system statistics. That's a fun one to say so.

A

The numbers indicated here are the amount of time the CPU spent in various states, such as user system, idle, etc, and these are in a kernel, internal unit called user Hertz, but you can convert it to seconds later on. So the top line shows us a summary of all these times added together, but we don't need that. So we can skip that and, like I said, for the purposes of our talk, we're going to look at user system and idle, even though they're I believe 11 different values here.

A

So to start, let's focus on creating a clear and concise exported API. We create the CPU stat structure and this contains stats for an individual CPU. We give an ID, which is in the string format, so we can have CPU 0, 1, 2 or 3 and etc, and then we export the values that we care about as integers here, user system and idle.

A

So next we're going to create this top-level function, scan and scans job is to read and parse CPUs that information from I/o Reader are if you're familiar with go. Io reader is one of most fundamental interfaces in the NGO standard library.

A

It essentially represents something like a file or a network stream or a byte buffer, and this is great because we can accept a file in our production code, but we can also swap it out for a byte buffer or another source later on for our tests, so use interfaces, they're, very powerful and also, as it turns out, the buffaio scanner type takes an I/o reader our and we can use that to easily scan over text-based input. So we create this Bluff I/o scanner.

A

We're gonna skip the first summarized line because we don't need it and then we enter this inner scanning loop. If, for some reason, our scanning loop exits early, say, for example, we're reading from a network stream and there's a Oh F or something similar, we need to be sure we check the error from this scanner type. This is important and often overlooked, so be careful.

A

So within the inner loop we need to carefully handle our slice boundaries of this file. So within the loop we have these CPU stat lines. We know that each one should contain a CPU, prefix and exactly 11 fields, so we explicitly check for both of those things. If a line doesn't have a CPU prefix, it's not related to CPU stats and we don't care, but the lines that we do expect.

A

We know they should have 11 fields, so we make sure we check for that because otherwise, if you access your slice later on, you could run into a nasty out of range panic. That'll take down your program and nobody wants that.

A

So now that we've gathered these values into this string, slice we're going to parse the values we care about user system at idle from indices, 1, 3, & 4, respectively, and we're going to parse them into an array of 3 values, because we know exactly how many we need. So we iterate over the indices. We convert the raw string into an integer and pack it into our array. And finally, we unpack it all into a top-level CPU stat structure at the end of our loop.

A

So now that we've put this all together, let's build an example to try out our API and just see how it goes. So we open a handle to proc stat. We defer closing it to make sure that we clean up after ourselves and we pass the file, which is an I/o reader directly to our CPU stat scan function.

A

If it succeeds, we're going to iterate over all of those structures and print them to the screen and as you'll see, it seems to work just fine and, like I said all this code will be available in two slides, as well as my tox repository so check that out later, if you'd like to see full examples.

A

So now that we've put this all together, let's build a prometheus exporter. So to start, we need to wire up our dependencies appropriately in our function main, so we're going to create this stats function, and this is a closure. It has no arguments and it returns a slice of these CPU stat structures and within each call to this function. We're going to open a unique handle to proc stat, defer, closing it to make sure that we clean up and then pass it to our CPU stat scan function. So this fulfills two purposes one.

A

Every time this function is called by collector, collect we get a unique handle to proc stat, so we're totally concurrency safe and two. This is nice because we can swap out this function right here. For one that returns say, a fixed set of values for our unit tests makes your life much easier. So we passed this function to our collector type and then we register it with Prometheus.

A

So this is our collector type, we're going to export a metric called time user, Hertz total and we're also going to pass that function explicitly as a dependency of the collector.

A

So here's a tip when you're working with fields that have are structures that have repetitive fields, such as user system at idle integers. You can use anonymous structures to simplify your code somewhat, so we call our stats function here. We get our stats, we're going to iterate over each entry and what we want to do here is we have this slice of anonymous structures.

A

If you've ever done table driven tests and go- you may be familiar with this, but essentially we're going to associate each of these mode strings with the raw value, so the user string is associated with the user value and so on.

A

Next we iterate over all these tuples and unpack them, and then we create the metrics using this must new, constant metric constructor and we're going to associate two labels, the idea of the CPU and the current CPU mode. So let's talk about cosmetics while we're here, let's say, for example, I have a system with two CPUs and they're hot swappable, so your exporter is running along you're, getting metrics from those CPUs and suddenly one of the CPUs dies or you take it out of the system.

A

If you don't use the con symmetric constructors, the Prometheus client will continue to export the last value for those time series forever. So you need to make sure that if you're implementing the Prometheus collector interface, you use the Const metric constructors because they allow time series to come and go as time goes on.

A

Alright, let's put this all together and give our egg for exporter a try it with curl and just see what happens so we curl the exporter. We run it and, as we'll see, we have some metrics available with our given labels, such as the CPU ID and the mode and these raw integer values.

A

So if you want to gather metrics from files with go here, my recommendations use IO reader whenever possible. This is much more flexible than say. Accepting a file path in your API. The buffaio scanner type is super useful for all the files that reside in proc insists because, typically, these are text files that are pretty easy to parse and that's a great type for doing so always check your slice and array boundaries. You don't want to run into an out of range panic in your exporter.

A

It's no fun for anybody and also check out the Prometheus proc efest library. We've done a lot of the work for you for these types.

A

Alright, let's move on to the second part of our talk, so let's gather metrics from some hardware devices, in this case the silicon dust HD homerun, so who here still pays for cable, TV, wow, just a few of us I love, my cable TV, ok, I, love, NFL and HBO, and all those things. But this is a pretty cool device. If you have a cable subscription, I understand it also works with over-the-air, but this one's specific to cable. So you get your cable from your provider.

A

You plug it in you, get a cable card to decrypt the signal and you plug in an Ethernet cable and what this becomes is essentially a network TV tuner. So you can use this to watch, live TV on different devices or also record to something like a plex media server.

A

So they also offer a linux utility, so I started poking around and I can discover the device of my network and use its ID to ask it. Questions such as give me your tuner debugging information and, as you'll, see, there's a lot of different statistics available. Here we have information about the tuners, such as the current channel and the channel.

A

It's trying to lock to the signal strength, the signal-to-noise ratio, various bit rates it passed within the device and finally, the network, packets per second and error rates, so naturally I bust out, TCP, dump and start looking around at all these packets. But it turns out I didn't ever need to do that, because silicon dust actually has an open source library called Lib, HD homerun written in C.

A

So we can use that as inspiration to create a go client. So, let's start by building a go network, client API, we create this client type, it has a mutex, so we can serialize access to the device we're going to embed a connection in there and we also give it a timeout if you're, making a network connection and a timeout. Next we have this dial, constructor and dials job is to dial a TCP connection to an HD homerun device.

A

So first we need to build up our low-level communications types. The HD homerun speaks using these specialized packets. So this is what a packet looks like and go. A packet has a type that specifies the type of message it carries and tags, and these specify optional attributes such as debugging information or information about the device and lots of other things.

A

So then we create this little function called execute. An execute stub is to send a single packet and receive a single packet. So we lock the connections to the vice-marshal. It's a binary format write it out, wait for a response, unpack it and then we're all done and you'll notice, I've omitted the error checking, but don't ever do that in production code. Ever on top of that, we can build a higher-level friendly API, because who cares about like what the packets for this thing look like right?

A

So we create this high-level function, called query and queries. Job is to perform a read-only query to retrieve information from the device. We accept this query string, we put it into our packet and then we send it to the device and while I'm here I want to mention this really cool tool called C forgo.

A

Given a C header, you can output pure go constants, so you don't have to like write down enumeration to my hand, super useful if you ever work with a thing from the kernel would recommend, but we can do even better than that query function. So we're gonna create this tuner debug type that contains debugging information about a tuner, and then we can add this tuner debug method to our client type. We query for the debugging information from the device for tuner 0, for example.

A

We retrieve this raw bite, slice and hey, as it turns out a raw bite slice. That's a lot like a text file. So it's pretty much this the same routine as we did last time. We wrapped the bite slice and it bites reader. So it adapts to the I/o reader interface. We can put an I/o reader into a buff I/o scanner scan through the file and hey we're all done so now. Let's build an example program and just give this a shot and make sure it all actually works.

A

So I build and run my program give it. The my device asked it for the tuner zero debug menu and, as you can see, it seems to work just fine.

A

So, let's move on, we want to export HD home run metrics using Prometheus. So how do we do that? Well, the HD homerun device has this network API, but we can't actually run our own code on it. It's a lockdown little third-party device, so what we have to do is enable our Prometheus exporter to dial out to a remote device.

A

This is where things get kind of interesting, so we create this function. Dial and dial is used to connect to an HD homerun device. Given some address string, it's going to create the HD homerun client for us and within dial we're going to call our HD homerun dial method or function and then we're going to make sure we set a timeout. You want to always set a timeout on your network communications, because what happens if something goes wrong? The device is down. Your network is down something in the middle of town.

A

You don't want to necessarily leak file, descriptors, otherwise, you've run out of file descriptors. Your exporter stops working, you get paged nobody's happy, ok, so we have our dial function. We pass it as a dependency to our handler type and the handler is going to look something like this, so our handler is going to implement the HTTP handle, handle our interface from the go standard library and we're going to do that now. So this is where this gets interesting.

A

We need to actually configure Prometheus to send a target parameter with each scrape request, so we're going to retrieve the target parameter here, we're gonna, validate it and make sure it's got a valid port, and if it doesn't, we had the default.

A

We joined it back together and we tried dialing out to whatever was passed as the target. If it fails, we return in HTTP 500, some Prometheus can know something is wrong, but if it's a seat, if it succeeds, we're going to defer closing the client, we're gonna, wrap it in a small interface for testing and we're gonna actually serve the metrics for it.

A

So what might that interface? Look like, as it turns out, our HD homerun client can do what's most of the work for us actually, so we create this device interface here and it's going to wrap the HD homerun device type or the client type, and it has the same method signature as our HD homerun client. So by the rules of go, the HD homerun client type implements our device interface.

A

So putting this all together, let's give the exporter to try with curl and that target parameter and just see how things go.

A

So we curl our exporter running on, let's say my local machine and it dials out to the remote HD homerun device. So we exposed a couple of metrics here. We have network packets per second that shows the packets per second rate for a given TV tuner, and we also have this interesting tuner info metric, and this contains metadata about each of the tuners available to a device. It exports a constant value of one more on this shortly.

A

So if you want to configure Prometheus to actually use this thing, you have to make from athiest pass a target parameter. So this configuration is mostly taken from the black box exporter. So definitely look at that repository for more information, but the basic ideas are you have this targets list and you pass a list of like say, for example, your HD homerun devices.

A

You create this reliable in configuration, and this tells Prometheus to pass the target parameter and also to replace I believe the address of the HD homerun exporter in the metrics, with the address of your HD homerun device. So relabeling is a super powerful concept. I definitely recommend reading more about it, particularly the robust perception blog is super useful.

A

Putting this all together, it seems to work. Just fine Prometheus is up and running. It is scraping our HD homerun exporter, running on some machine which reaches out to our HD homerun device itself gathers the metrics exports them in Prometheus format, and everything is good. So, let's go talk about that information metric again that metadata. You can use these synthetic information metrics for super powerful, prom cue while queries, so we construct it like this. We have these labels here that are the tuner the channel.

A

It is trying to lock to and the channel it's actually locked too and we're going to export this metric. It has a constant value of 1, it's a gauge and it just has these labels. So why is this useful as it turns out? Prompt QL is super powerful and you can use this to effectively perform a relational join between different time series.

A

So, if I want to answer a question, something like what is the packets per second rate for a given channel, this query seems to work so I'm effectively joining my network packets per second query on the channel ID with my HD homerun tuner info metric and as we'll see on channel 3 for my HD homerun device on tuner 0, the packets per second rate for that channel was 840.

A

So this is another super-powerful prom qaul concept and in particular, if you learn more about this technique, check out Bryon's blog and how to have labels for machine rolls.

A

So if you want to gather metrics from hardware devices, hear my tips set timeouts for network connections, always just do it, it won't take you very much time just do it. Use interfaces for testing makes your life much easier. You shouldn't have to dial out to an actual hardware device to run your unit tests right, create these synthetic information metrics so yeah.

A

You could add these labels to every single one of your Prometheus metrics, but then you can result in some very high cardinality time series and that's no good for Prometheus, so take advantage of prime ql export your labels as metadata on a single metric and then join them together using from ql and finally learn a little bit about relabeling. It's a super powerful concept and one I'm in admittedly not very familiar with, but it's super useful.

A

And that brings us to the home stretch. So, let's talk about gathering metrics from some system. Calls in this case stan FS on Linux, so state FS is used to get filesystem statistics. So, given some mount point, the function, stat FS returns information about a mounted filesystem.

A

So to start we want to build a high level OS agnostic API, because that's the friendly thing to do and go so. We create this file system structure.

A

It has a path or mount point, a type which is an enumeration of filesystem types like ext4 or NFS or XFS, and the number of files that reside within that filesystem we create this table all function, get with a capital G, and this retrieves stats for the filesystem mounted at path within get we're going to call get with a lowercase G, and this is going to be our operating system, specific implementation of get.

A

So if you want to make use of system calls and go, you must always guard them with build tags. Your code won't compile on other platforms if you're using system calls unless you're using build tags appropriately- and this is one of Rob pikes go proverbs.

A

So let's talk about the Linux implementation. First, we have our file stat FS underscore Linna that go, and we have this Linux build tag at the top. You don't necessarily need the build tag just because of the file name but I like to add it just because it's more explicit, and instead of using the standard library syscall package, we're going to use going org, exorcist unix. This is effectively the modern replacement for syscall, because cisco had to be frozen after a certain period in time, so get once again lowercase G.

A

This is going to be our Linux implementation of that function, get the accepts a path and then we're going to create this UNIX stat FST structure. We call out to this UNIX stand. Fs system call and provide it some mount point and we pass a pointer to the structure and, as a result, the kernel is able to populate that structure. So when our function call returns successfully, we have a structure with all the information that we need. Finally, we unpack it into the OS agnostic structure.

A

Remember system calls can't cross the boundary outside of where your build tag is so we unpack the structure here within our Linux type to make sure that it's friendly for all operating systems.

A

So next up we're going to talk about code, that's not on Linux, so we have this file stat FS underscore others that go and at the top you have a build tag that says for every operating system. That is not Linux build this code, and here we're just gonna leave the function get unimplemented. Perhaps there are similar api's for other platforms, but for the time being, leaving it on influence.

A

It is fine and I definitely like to return an error that says something like you know, whatever this is, is not implemented on your current platform, such as windows, Darwin, etc.

A

So next, let's build an example program and give this a shot and just make sure it all works. So we create our program, we call out to it. We get the first argument and that's going to be our Mountain point path. We've passed it to our Stanton FS gate function, and this should compile successfully in all the platforms, even if it doesn't run, and then we print the information to the screen and as you'll see, it seems to work just fine on Linux.

A

Of course, you can implement more platforms if you're able, but for the time being, Linux only is okay. If you want to export metrics for this well I'm a little short around time that I might have thought. So this is going to be an exercise for the reader.

A

If you want to gather metrics with system calls and go hear my recommendations build a high-level, easy to use API your caller should not have to care about system calls or pointers or any of that kind of unsafe things, so make sure that you make it as easy to use as possible and take away as much of the complexity as possible from your color always use build tags with system calls.

A

If you don't, your code will not compile on all platforms and go is a great cross-platform language, so we should keep it that way and also be really careful about using elevated privileges with system calls. It's pretty easy to do something like I, don't know, overwrite your main disk or reboot your machine or something nasty, so just be cautious about using root.

A

So in conclusion, if you're gonna build from easiest exporters and go it mostly comes down to a set of typical software, best practices avoid global and package level state as much as possible and pass your dependencies explicitly as parameters. It's going to make your life much easier. In the long run, it makes your code easier to read easier for your co-workers to review and easier to maintain in the future focus on creating simple and reusable package api's.

A

When building exporters, like I said you don't need to mix the details of exporting metrics with crawling a file system or some binary format or some network protocol separate these things, create nice reusable packages and then import them into your exporter and finally, read up on the Prometheus metrics best practices from Prometheus I. Oh, these are really great, there's a lot to learn but apply them judiciously, and your exporters and you'll be better off in the long run and finally, I have a special unrelated announcement.

A

Digitalocean is working on a kubernetes product, so if you're interested there are some folks in here, work on that or otherwise come see us on our booth and that's it. Thank you very much for your time and I look forward to answering your questions.

A

All right, well I'm, not sure where we're at on time right now, but I'd be happy to take any questions if anybody has I otherwise, just come, find me in the hall: that's just fine to 7:00, okay, sure.

A

I'm sorry I think I missed some of that could just pick up a little bit.

A

A

Yeah, so the question was: if you have a bunch of metrics in a push format, how can you best convert that to a pull format to start I believe there are lots of Prometheus exporter adapters, essentially for things like stance, D, graphite, etc.

A

So, if you wanted like a stopgap solution, you could push to those exporters instead and then those export native from a theist metrics, but if you want to actually swap it out, it kind of depends in your application, so you might have to go through and like I guess, I, don't know which language is using her in your environment, but many yeah sure so it's kind of tough. But yes, my recommendation would be definitely look at the adaptor exporters. Things like the stance, the exporter there are.

A

There are several under the Prometheus organization, so check those out and those will probably give you a good start anymore. Yeah, okay, nothing uh yeah, any more, any more questions, all right cool! Thank you very much. Yeah Goan. Sorry,.

A

Yeah sure, so the question is: how easy is it to deploy Prometheus in a Cooper dies environment? And if so, do you have like Hamill files prepared for deploying Prometheus in a kubernetes environment and truth be told I, don't deploy for meeting communities? I have co-workers here who do I'm sure there are folks in the room. I run this at my house, I using system D and ansible I'm bare metal native right.

A

But yes, so, unfortunately, I'm not the best person to answer, but I'm sure there are folks here would be happy to help any more questions cool. Thank you very much for attending my talk and I hope it was entertaining.