►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
What You Need to Know About OpenMetrics - Richard Hartmann, Grafana Labs
OpenMetrics is now an Incubating project withing CNCF. It's also a required part of the Prometheus Conformance Program. What does that mean? What are the updates to be aware of? How can you leverage all of this today?
A
A
A
If
you
have
your
region,
you
have
your
data
center,
you
have
your
customer,
you
have
your
whatever
and
then
you
need
to
select
by
customer.
All
of
a
sudden,
you
need
to
reshuffle
your
tree
or
you
need
to
walk
up,
go
over
need
to
go
down
and,
like
you
can
do
all
of
this.
It's
not
exactly
hard,
but
it's
just
overhead,
which
is
not
nice
looking
at
the
time
after
prometheus
by
now
permit.
This
is
the
defective
standard
in
cloud
network
cloud
native
metric
monitoring
and
way
beyond.
A
A
We,
as
in
prometheus,
even
have
an
exporter
scaffold,
which
makes
it
super
easy
to
just
write
new
exporters,
so
you
can
focus
on
the
metrics
and
what
actually
adds
business
value
instead
of
focusing
on
writing
yet
another
http
endpoint
and
again,
I
probably
don't
have
to
convince
anyone
who
is
using
kubernetes
or
prometheus
that
label
sets
are
the
way
you
want
to
use
to
to
access
your
data.
Of
course,
this
is
just
so
much
more
powerful
than
hierarchical
data
models.
A
Some
vendors
were,
let's
call
it
torn
about
adopting
things
which,
which
are
from
a
competing
product
or
project
which
literally
have
the
name
in
prometheus
exposition
format,
especially
the
more
traditional
vendors
or
the
super.
Large
companies
prefer
to
support
official
standards
just
for
level
setting
and
everyone
playing
equally
ideally,
and
we
wanted
to
not
lose
the
installed
base
of
prometheus.
For
obvious
reasons,
we
wanted
to
retain
that
ease
of
adoption.
We
wanted
to
make
upgrading
basically
invisible
to
to
pretty
much
all
the
users.
A
We
also
wanted
to
reject
this
kitchen
sync
approach
where
you
have
to
support
every
single
last
use
case,
which
you
can
possibly
conceive
and
by
by
this
basically
dilute
and
overburden
your
standard,
your
implementation.
Here,
what
have
you
on
quite
on
purpose?
We
decided
to
do
one
thing:
do
it
well
and
remain
focused
and
opinionated
about
how
to
do
matrix-based
monitoring
right?
A
A
lot
of
competing
companies
have
helped
shape
what
has
become
open
metrics
today
and
the
result
isn't
actually
neutral
standards
standard
which
takes
pretty
much
everyone's.
Who,
who
spoke
up
concerns
into
account,
there's
thousands
and
thousands
of
people
and
companies
who
helped
us
get
there.
We
have
a
few
marathon
runners,
benkochi
brian
brazil,
myself,
rob
skellington
and
honorable
mention
samia
baller,
who,
until
relatively
recently
drove
drove
the
standard
forward.
A
A
Still,
while
we
were
finalizing
the
standard
and
everything
without
impacting
any
end
users.
Most
of
you
might
be
surprised
right
now
that
you're
actually
already
using
openmetrics
at
scale,
but
that's
a
distinct
design
goal
of
what
we
did.
We
didn't
want
you
to
be
forced
to
re-re-implement
against
the
new
api
every
three
weeks
or
six
months,
or
something
should
just
keep
working
and
be
largely
invisible
because,
frankly,
you
care
about
so
much
about
the
details
of
how
open
metrics
evolved
and
how
it
be
and
how
the
promises
internal
works.
A
A
That
being
said,
we
have
breaking
changes,
but
there
are
very,
very
few
you
can
see
the
complete
list
on
the
screen
counters
require
underscore
total
in
the
time
series
name,
it's
already
a
common
convention.
It's
part,
it
said
it
has
been
part
of
our
naming
scheme
and
on
or
of
our
recommendations.
Again
we
with
the
permittee
set
on
for
basically
years,
I
think,
even
before
prometheus
joined,
cncf
or
before
cncf
was
even
founded.
We
already
had
that
in
as
a
as
a
convention,
but
now
it's
actually
enforced.
A
So,
for
example,
if
your
metric
used
to
be
called
cpu
seconds,
it
will
automatically
be
renamed
to
cpu
seconds
total,
which
is
not
a
huge
change,
but
you
need
to
be
aware
of
this
and
we
consider
it
breaking.
Ideally,
your
your
counters
all
end
in
underscore
total
anyway.
A
Of
course,
anyone
using
standard
prometheus
stuff
won't
be
impacted.
It's
already
happened
in
the
background
or
you
never
even
use
those
those
bits
and
pieces
on
purpose.
We
also
have
improvements
for
interoperability
and
generic
cleanup.
A
We
tightened
up
a
few
specs
or
bits
of
the
specification,
so
a
little
bit
of
spacing
escaping
and
such
to
just
make
the
parser
easier
to
write
and
thus
make
parsing
quicker
and
easier,
because,
obviously,
you
run
quite
a
lot
of
scrapes
in
in
any
given
environment
and
so
being
able
to
to
just
improve
the
parser
is
going
to,
or
has
led
to
quite
impressive
overall,
like
at
scale
improvements
in
in
scrape
time.
A
A
A
We
allow
for
64-bit
integers,
not
just
floats,
there's
new
metadata
unit,
which
tells
you
what
base
unit
that
counter
or
that
that
metric
is
in
underscore
created
for
when
metrics
were
created
and
reset,
which
allows
some
some
deductions
for
for
certain
rates.
And
such
we
extended
open
metrics
for
considerations
for
push
and
for
pull
and
in
the
consideration
section
of
openmetrics,
you
can
find
quite
some
some
text
about
how,
if
you
need
to
do
push,
you
can
do
this
in
a
compatible
way
with
prometheus.
A
It
leads
to
sometimes
even
or
commonly
you
even
needing
to
replicate
parts
of
service
discovery
in
your
scraping
layer
or
in
your
metrics
pipeline
layer
to
to
be
able
to
have
a
status
of
upness
of
liveness,
of
or
stainless
of
your
metrics
just
something
to
be
aware
of.
Yet
we
have
all
those
considerations
in
and
openmetrics
explicitly
supports
push.
Of
course,
that's
something
which
users
wanted,
and
so
we
support
it.
A
Prometheus,
doesn't
support
push
open,
metrics.
The
text
format
is
still
mandatory
to
have
a
baseline,
where
you
know
that
things
are
working
and
also
it
means
that
the
debuggability
is
really
easy.
Of
course,
you
can
always
just
connect
with
a
web
browser
and
start
reading
stuff,
if
you
need
it,
but
we
re
introduced
protobuf
for
the
people
who
like
it
like
for
prometheus
itself.
We
text
format
is,
has
even
been
quicker
than
protobuf
under
under
high
load,
but
there
are
scenarios
where
people
prefer
prologue
and
that's
completely
fine,
something
which
is
completely
new
is
exemplars.
A
Ex-Employers
is,
if
you
haven't
heard
of
them,
they're
absolutely
ingenious.
Basically,
they
allow
you
to
attach
information
about
traces
to
your
logs
and
to
your
metrics.
In
this
case.
Obviously,
I'm
focused
on
the
matrix,
but
it's
also
for
your
logs
because,
usually
in
traces,
you
have
this
needle
in
haystack
problem.
You
need
to
go
into
a
into
a
trace,
but
you
don't
know
if
it's
actually
relevant.
So
you
need
to
search
through
all
the
stuff
which
is
attached
on
the
traces
or
to
the
traces
to
find
things
which
match
certain
properties,
as
opposed
to.
A
A
Exemplars
are
already
widely
supported,
prometheus
cortex,
thanos,
loki,
grafana,
others,
there's
tons
of
software,
which
already
works
with
exemplars
and
from
what
I
can
see.
Others
are
also
adapting
this
concept,
because
it's
just
insanely
powerful
current
state
in
prometheus.
A
The
prometheus
client
library
is
the
reference
implementation
uses,
openmetrics
data
model
internally,
hundreds
percent
blah
blah
blah
blah
blah
has
done
this
for
years.
So
if
you,
if
you
want
to
look
at
open
metrics
like
the
dirty
details
or
the
reference
implementation,
look
at
the
python
client
again
on
purpose
course,
python
tends
to
be
easy
to.
Reach
is
used
for
teaching
and
such
so.
We
chose
this
as
the
reference
implementation
go
and
java
also
support
it.
A
So
that's
basically
most
of
the
prometheus
ecosystem
right
there
already
prometheus
will
preferentially
negotiate
open,
metrics
and
has
done
so
for
years.
So
again
you
are
most
likely
already
using
it
without
being
aware.
Info
and
enums
are
first
class
features
in
prometheus,
so
you
don't
need
to
implement
stuff.
A
You
don't
need
to
need
to
implement
them
by
hand
in
your
in
your
client,
libraries.
You
actually
have
the
support
in
your
client
libraries.
You
don't
need
to
do
this
by
hand
and
if
you
scrape
through
the
prometheus
exposition
format,
like
you
actively
negotiate
this
it's
exposed
in
a
backwards
compatible
manner,
which
is
nice.
A
Other
implementations
datadog
supports
ingestion
of
openmetrics
has
been
for
quite
some
time.
They
even
contributed
performance
improvements
to
the
python
parser.
Thank
you.
Open
telemetry
supports
openmetrics
as
a
first
class.
Wire
format
and
openmetrics
is
part
of
the
prometheus
conformance
program
and
that
part
is
important.
So
now
I'm
going
to
put
on
my
promises
head
again
the
prometheus
conformance
program,
officially
launched
or
launches,
depending
on
when
this
is
published
on
october
14th,
because
that's
when
the
promises
conformance
program,
talk
also
by
me
is,
is
going
to
be
published.
A
So
we
have
several
test
suites,
which
test
four
promises:
interfaces
in
the
permissive
org
and
anyone
any
vendor
any
project.
Any
product
which
wants
to
get
an
official
promises
compatible
mark
needs
to
to
be
compliant
to
all
the
interfaces
which
we
define
as
relevant.
For
that
thing,
so,
if
you're
scraping
data,
you
need
to
be
100
open,
metrics,
compliant
to
be
to
be
able
to
get
the
prometheus
compatible
mark.
A
A
A
You
can
see
that
we
also
have
a
unit
metadata,
which
tells
you
that
four
seconds
is
in
seconds.
You
can
also
see
that
the
type
is
now
not
four
seconds
total
anymore.
The
type
and
the
unit
is
four
seconds,
of
course,
by
definition,
as
it
is
a
counter,
you
get
the
underscore
total
attached
to
the
value
by
if
you're,
using
one
of
the
client
libraries
automatically
else,
you
have
to
do
it
by
hand,
and
you
can
also
see
that
you
have
underscore
created
forward
every
time.
A
I
I
think
I
put
a
joke
in
there,
but
I
forgot
honestly,
you
can
revert
it
and
see
what
what
joke
I
made
back
then
it's
also
in
the
official
spec,
of
course
obvious
yeah.
But
anyway
you
see
that
that
the
name
of
the
matrix
is
now
without
that
tail.
Without
the
suffix
for
consistency,
reasons
for
the
internal
data
models,
if
you're
not
using
the
prometheus
client
libraries,
then
you
need
to
do
stuff
by
hand.
It's
relatively
easy.
A
You
add
underscore
total
for
your
counters,
even
if
you're
using
the
client,
libraries-
and
you
don't
do
this
now,
just
edit
and
if,
if
you're
using
your
client
libraries,
you
will
not
see,
I
mean
in
neither
case,
will
you
actually
see
a
change?
Just
add
it
and
either
the
prometheus
client
libraries
will
do
it
for
you
once
you
upgrade
or
you
do
it
by
hand
and
either
way
you
you'll
not
see
an
impact.
A
If
you
write
stuff
yourself
like
by
hand,
please
send
the
correct
content
type
both
if
you
want
to
explicitly
do
prometheus
exposition
format,
04
or
if
you
want
to
do.
Openmetrics1.0
either
is
fine,
but
please
make
sure
that
you
that
you
explicitly
set
the
content
type
correctly
and
if
you
write
scrapers
or
in
gestures,
please
set
your
accept
headers
to
negotiate
previous
or
open
metrics
format,
depending
on
on
which
you
prefer
and
how
far
you
are
in
your
own
story
of
adoption.
A
My
recommendation,
or
our
recommendation-
that's
with
both
heads
on,
would
obviously
be
openmetrics,
because
that's
the
future
there's
more
resources.
We
have
to
get
repo
for
openmetrics.
We
have
the
compliance
suite
for
prometheus.
We
have
the
part
of
the
open
of
the
prometheus
compliance
suite,
which
is
living
in
the
open,
metrics
repository
and
that's
it.
I
hope
you
have
plenty
of
questions
for
me.
Thank
you
very
much.