Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: What You Need to Know About OpenMetrics - Brian Brazil, Robust Perception & Richard Hartmann

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

What You Need to Know About OpenMetrics - Brian Brazil, Robust Perception & Richard Hartmann, Grafana Labs

The OpenMetrics format intends to standardise metric exposition, making it easy for both those developing and operating systems to monitor them. While it is based on the Prometheus format, it is however a new format. Will it be supported by your monitoring system? Will you need to rewrite your existing instrumentation? What's needed to transition? What about 3rd party systems you don't control? How does this differ and expand, and improve on the existing Prometheus format? Do you have to use Prometheus?

https://sched.co/ZevQ

A

Hi and welcome to what you need to know about openmetrix, let's start with who we are.

B

I'm brian brazel, the founder of robust perception, one of the developers of prometheus and, of course, one of the people working on openmetrics.

A

And the author of the prometheus bible prometheus up and running, and I'm richie, I'm the community director at grafana labs, also prometheus team member, the chair of the sick observability within cncf and the founder of openmetrics, and we couldn't really be talking about openmetrics without also talking about prometheus. So, let's look at promises for a minute before prometheus.

A

Historically, the closest we had which which we assembled a common standard within the space of monitoring, was snmp, and many of the old solutions are based on what is today ancient technology like, for example, asn 1, which is which is front and center in snmp. And while this has it's a great piece of engineering, but it's it's not matching the design and engineering traders of today.

A

Many of those protocols are chatty and slow. They tend to be, or they can often be proprietary hard to implement or even both and many of the underlying data models. For example, snp also modbus also others tend to encourage vendor-specific variations which follow the letter of the law, but not the spirit and one thing which is my my one of my largest pet peeves hierarchical data models because they almost never fit your need.

A

If you have a thing where you have, I don't know region customer deployment, but now you need to look at your data by customer or per deployment, and this particular data model is basically it's broken as soon as you're done defining it.

A

So, let's look at the word after prometheus prometheus, as of today is the defective standard in cloud native, metric monitoring and way beyond and by extension. The same is true for the permissive exposition format, the ease of exporting this data has led to an explosion in compatible metric endpoints, and we have thousands and thousands of exporters and integrations which people wrote on their own for their own stuff, just because they wanted to get their stuff into prometheus because it was so simple and so powerful.

A

We also have standard exporters and we have libraries which make integrating easy and we even going beyond this by creating a standard set of basically a scaffold where which you can use to write your own exporters and not focus on writing http endpoints and making that stuff work, but actually focus on what you want to expose about your own stuff and, of course, label sets.

A

Allow you to bring up this n-dimensional matrix of of how you look at your data and if you need to slice and dice according to by customer by deployment by region by version of a thing it doesn't matter, you can always access your data precisely as you need it as long as you have the labels for it, there's always tons of politics in any of in any open standard, and, to be honest, quite a few vendors and projects have been torn about adopting something which which carries the prometheus name, and especially the more traditional vendors like, especially in the networking space, which is many moons ago, where I'm coming from.

A

They want to have an official standard whatever that means. But obviously we wanted to reuse. The install base of prometheus, because a for the ease of adoption and b for the shear installed base, but also we want to still reject that kitchen sink approach which some standards have where they try to call all use spaces. We want to follow the unix mantra of doing one thing.

A

Well, so we really remained focused and also, to be honest, quite opinionated, opinionated about how to lose things just because of of a lot of experience, if I do say so myself in in how to run monitoring and observability and many many different companies have collaborated within open metrics and helped shape this. So I I dare say we actually achieved a neutral standard.

A

We have a ton of commitments and this list is partial of people who actually are of companies who want to adopt openmetrics, but we also have a lot of people or a few people behind this. The marathon runners, who have been at this for quite a few quite some years now are ben koji of gitlab brian who's. In this talk, myself rob skelton, formerly uh m3db, now chronosphere and honorable mention for someone who dropped relatively recently, but did a ton of useful work and gave a ton of useful input within within open metrics. That's sumir.

A

We mainly work by consensus in forthright, because um so we we tend to not have overrides or anything, but we tend to have very long discussions about all the details and we had quite a few attendees from many different companies over time so over to you, brian.

B

Yeah so in terms of the actual openmetric standards itself, the good news is that it is largely the same as prometheus format. There's been a few cleanups, you know warts removed and a small few new features, as uh richard said, we're trying to avoid the second syndrome or kitchen sink effect and try and keep things small and any additions are small still and so so far like it's quite possible.

B

If you are using the python client official, prometheus python client that you've already been using openmetrics for a year and a half because well the reference implementation is there and the general plan is that beyond the python client that the other clients at some point will transparently migrate you to open metrics without you noticing the general idea being that, in theory, everything will just work.

B

There are, however, breaking changes which are likely to affect you. The big one is that the convention for counters to end in underscore total, is now mandatory. So if you are already following that convention, it will be a seamless change. If it isn't, then, when your client library switch, your metrics name are going to change. So this is something to get ahead of if you can so, for example, if you had a metric called cpu seconds, it would end up being called inside prometheus cpu seconds total once it migrated true.

B

The other thing is the time. Second, stamps are now in seconds for consistency, because we use seconds everywhere in prometeus, but historically prometheus format uses milliseconds for the actual timestamps, and so we just changed that for consistency. But the thing is that extremely there's, a very few good use cases for exposing timestamps directly, so this is unlikely to actually affect anyone, at least not too many people. So I wouldn't worry about this. It's the total on the counters.

B

That's the big thing you need to think about, there's also a whole pile of other things like there's two different types of escaping there's. Now only one there's a way to take these scrapes there's a higher resolution, timestamps. That was something that was asked, for example, talking of collaboration with other companies.

B

That was a request from influx data, and it's got 64-bit integers units are now in there in first class, created something a request coming from google and stackdriver, which handles some corner cases, and now prometheus is pull, but there's also considerations in there for push as well to make sure it works for everyone, and the text, format is still there historically prometheus also had a protocol format that went by the wayside, basically with prometeus2, but has reintroduced openmetrics as an optional thing.

B

So what's new, probably one of the big features is exemplars because, as metrics are just one part of your mod monitoring solution, you're also going to have logs and traces and the idea. What example is you can say hey. I had just had this request that took a second in this particular histogram.

B

Can I go and find out in my tracing system, so here we're saying: hey def, that particular trace id took 290 milliseconds and you can go find that- and this is already supported directly in cortex and tunnels, and will likely support information itself at some point as well to access this data.

B

So the current status in terms of the prometheus project, as I've mentioned, the prometheus kind python client, is the reference implementation for openmetrics and uses the openmetrics data model internally. uh Go so far has just limited support because uh go wanted to support at least the go. Client wanted to support, exemplars and prometheus has, since version 2.5 will negotiate, openmetrics preferentially um and info and enum are now first-class features.

B

These are conventions that came up over the years as different ways to represent basically strings, and you can use these and handy thing with all this stuff is all the grades gracefully. So even if you're using say info and enum- and you end up not negotiating open, metrics you'll still get them looking like gauges as they would with prometheus today. So it's all nice and transparent, as far as we can make it.

B

Now, of course, we want to be an open standard, so other companies are also ingesting it. uh So today I think pretty much all the main metrics monitoring vendors, both commercial and non-commercial support, media's text format, there's already companies as well like datadog, for example, is able to ingest open metrics and in fact they also contribute quite a few performance improvements to the official reference parser.

B

In addition, open telemetry is planning to support openmetrics as a first class wire format, and indeed some of the discussions we've had within openmetrics and communities generally are helping to uh shape open telemetry by our experience, uh just one thing to watch out for because you know, openmetrics has been the word's been around for a while. Now, if someone says open, metrics there's a good chance, they actually mean the prometheus text format, and so there is some confusion there, unfortunately, but here's a simple way to help spot it.

B

As I mentioned, the total counter change is to maintain so here. The first example you'll see, there's an underscore total in the type line and also on the actual um time series, whereas in open metrics you'll see it has unit, which means the format and there's underscore total is not present.

B

There's also an example of underscore created, and it's also that hash eof at the end. So we know that the exposition wasn't just interrupted, mid transfer.

A

And, as you can see, we since 2018, you were actually already able to speak, open, metrics and it already or happened automatically in the background. But standardizing stuff is hard, so sometimes the implementation is a lot further than the standard. So let's talk about that a little bit um the text format and the protobuf specs are done. Obviously, because else we we couldn't have implemented those or brown couldn't have implemented those in the in the python library.

A

The long form prose version is done um and it's currently being worked with and compressed. But there's there's uh the saying within itf that something isn't done when you can't add anything anymore, but when you can't take anything away anymore- and I think that's also an old chinese saying so we started with roughly 52 pages of just pure spec.

A

Eight pages of those or 16 pages of those have already been compressed down to roughly half of this, and we have 36 pages to go to also just compress more and more because else uh yeah it wouldn't go through itf anyway. If, if we didn't compress it like a lot, um we do have a test suite which is ready to use um it's based on the python client library, and you can already use this for compliance testing to get ready for, for when this actually hits the road or it's the dirt or whatever.

A

You know what I mean. What are the next steps completing this internet draft actually submitted to the rfc process within, hopefully, the ops wg within itf we already establish established contact. There is good, or there is active interest in adoption by that wg call out to warren komari, who has been helping him on the end of itf.

A

um Obviously, we all support openmetrics. In all the official client libraries we have ex-employer support still to go into mainline premises. It's already in the branch and downstream projects like, for example, in 3db or chronosphere uh grafana loki and such also should use make use of all this new metadata.

A

So how can you transition to open, metrics well again for the counters? The underscore total thing is is important, and if you add this now and if you do it properly now uh you actually control when you make that change. So we highly suggest you do this, and just for the record, all the client library integrations they'll handle this automatically. So even if you call your counter underscore total, it will not be called underscore total underscore total. That will all happen for you.

A

um Anyone who's who's, currently emitting data, as in being scraped, ensure that you set the correct content type. So a prometheus can see if this is actually open, metrics or the permissions exposition forward series 304 and for anyone who is writing scrapers or ingestors. Please set your headers accordingly, so you can negotiate either prometheus or open metrics, as you need thanks a lot, and now we are open for questions.