►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
OpenMetrics; the State Of 1.X And the Plans For 2.0 - Richard Hartmann, Grafana Labs
Speakers: Richard Hartmann
OpenMetrics is the open standard for transmitting Prometheus-style metrics. It's also part of OpenTelemetry's compatibility efforts with Prometheus. We will go through a 101 of OpenMetrics, the current state of 1.0, what could potentially go into 1.1, and look at what 2.0 will look like.
A
Anyway,
so
welcome
to
an
update
on
open
metrics,
if
you
could
close
it,
oh
perfect,
so
how
we
structure
this
is
relatively
easy.
We
do
a
speed
run
through
the
history
for
those
who
don't
know,
and
then
we
we
come
to
the
current
state
and
then
we
go
towards
the
future.
At
the
end
of
we
have
a
room
for
questions.
Cayman
is
going
to
run
around
so
looking
at
at
observability
data.
A
Historically,
you
had
basically
one
standard
in
this
space
which
is
SNMP
any
one
of
you.
Who's
ever
dealt
with.
Networks
is
probably
familiar
with
this.
There
is
a
little
bit
of
a
lot
of
love-hate
relationships,
and
one
of
my
major
pain
points
with
SNMP
is
actually
that
it's
based
on
asn1,
which
is
like
super
old,
seven
bit
encoding
highly
efficient
and
absolute
mess
to
get
right,
and
it's
really
really
really
I
see
people
laughing
yeah,
it's
really
a
mess
and
it's
kind
of
risky.
It's
often
chatty
and
slow.
A
The
data
models
are
vastly
different
between
vendors,
sometimes
even
between
just
versions
of
software
and
the
thing
about
hierarchical
data
sets.
Is
it
almost
never
fits
your
needs
because,
like
if
you
have
region
data
center
customer-
and
you
want
to
group
by
customer?
Congratulations,
your
your
data
is
wrong.
So
those
are
just
pain
points
which
which
existed
before
Prometheus
after
Prometheus.
A
A
Tens
of
millions
of
users
who
are
who
are
using
these
these
formats
directly
or
indirectly
every
single
day
to
keep
their
stuff
running
and
we
have
standard
exporters.
We
have
standard
libraries
in
particular.
The
go
library
is
one
of
the
most
goal:
libraries
on
Earth,
so
there
is
substantial
access
success,
but
yet
again
this
is
no
official
standard
and
yeah
label
sets
I
mean
I,
don't
have
to
convince
your
labels
are
better
than
hierarchical
data.
A
There's
also
some
politics
involved
in
particular,
in
former
times
I
mean
cncf
is
getting
bigger
and
bigger
every
year,
comparing
this
to
2016
the
first
Cloud
native
con.
This
is
awesome,
What's,
Happening,
Here,
but
still
a
lot
of
vendors
and
projects
were
very,
very
torn
on,
should
they
actually
adopt
something
which
has
a
different
name.
This
thing
called
Prometheus
and
it's
gotten
better
and
better
over
the
years,
but
still
there
is
some
some
smell
attached
to
a
single
thing
in
particular
back
then
a
lot
of
traditional
vendors.
A
They
just
respect
standards
like
you
write
it
down.
You
say
this:
is
the
specification
implement
this
and
you're
going
to
be
fine,
and
that
makes
a
lot
of
conversations
easier
and
obviously
we
wanted
to
reuse
the
complete
installed
base
of
of
Prometheus.
We
didn't
want
to
have
something
which
which
diverges
too
much,
because
else
you
split
the
community
and
that
that's
going
to
hurt
every
single
person
you
need
to
maintain
long-term,
long-term
stability
and
long-term
compatibility.
A
So
many
many
different
companies
chipped
in
with
making
this
happen,
and
the
result
is
an
actually
neutral
standard
like
there
are
several
things
in
there
which
Prometheus
didn't
even
really
need,
but
which
others
needed
in
particular
like
Google,
with
the
underscore
started
for
their
Monarch
installations.
These
are
things
which
aren't
there
on
purpose
to
to
have
a
better
general
standard
yeah.
This
is
the
shout
out
to
to
the
Core
Group
I'm
here
alone
today,
but.
B
A
So
what
does
this
mean
for
you?
If
you're
using
Prometheus,
Exposition
format
and
there's
still
quite
some
large
installations
which
do
this
and
that's
completely
fine?
It
is
largely
the
same
and
again,
this
is
on
purpose
for
quite
some
time.
Most
of
you
in
this
room
have
probably
already
been
using
open,
metrics
without
even
realizing
it
or
without
caring
about
it,
because
it
just
keeps
working,
because
we
spent
so
much
time
in
in
making
it
working.
A
There
are
breaking
changes.
You
need
an
underscore
total
if
you
have
a
counter,
and
sometimes
there
are
collisions
the
libraries
which
have
migrated
handle
this
transparently
for
you.
But
if
there
is
a
naming
Collision,
obviously
you
you
can't
do
much
underscore
total
at
the
end
was
already
an
anti-pattern
for
non-counters
in
in
standard
Prometheus.
So
anything
who
did
who
did
things
by
the
book
does
not
run
into
this
issue.
A
But
the
fact
is,
there's
a
lot
of
people
or
there
are
a
few
people
who
ran
in
there,
and
the
other
thing
is
that
timestamps
are
now
in
seconds.
Of
course,
one
of
the
things
which
are
really
really
hard
in
Prometheus
is
to
always
have
base
units,
so
some
of
those
non-obvious
things
which
happens
if
you
just
count
Joule
as
a
counter
as
a
measure
of
how
much
energy
you
use
just
by
putting
just
by
putting
a
Time
component
onto
this.
This
turns
automatically
in
Watts.
A
Just
because
that's
how
the
SE
system
is
designed
and
all
the
physicists
like
years
and
years
ago,
put
a
lot
of
effort
into
into
making
those
those
different
units
really
seamless
and
and
transposable
between
each
other
and
that's
why
we
believe
so
strongly
in
base
units
and
that's
why
we
made
this
change
and
don't
have
milliseconds
anymore,
there's
also
a
ton
of
cleanups
relative
to
to
the
premises.
Exposition
format,
it's
a
lot
cleaner.
It's
a
lot
tighter
with
permitted
Exposition
format.
You
can
do
more
or
less
endless
amounts
of
white
space.
A
A
It's
all
64-bit,
as
in
Prometheus,
we
have
a
new
metadata
type
for
for
the
unit
underscore
created.
I
already
mentioned
this
and
we're
going
to
see
it
in
a
second
to
last
slide.
We
support
push
explicitly
in
open
metrics,
even
though
Prometheus
again
does
not
really
support
it.
I
mean
there's
been
work
on
this
and
and
it's
getting
there
where,
in
particular
with
remote
right,
you
can
do
a
lot
of
stuff,
but
open
metrics
again
goes
beyond
what
Prometheus
can
do
here
and
the
thing.
The
other
thing
is
one
of
the
really
nice
things.
A
A
If
you
don't
know
what
exemplars
are
so
when
many
many
moons
ago
when,
when
I
met
with
a
few
people
at
Google,
and
we
were
considering
to
merge
open
sensors
with
open
metrics
way
way
back,
they
mentioned
that
for
them
searching
for
traces,
didn't
scale
and
when
Google
tells
you
that
searching
for
something
doesn't
scale,
you
probably
better,
listen
and
figure
out.
Why?
A
A
If
you
see
those
things
like
in
the
in
the
header,
you
can
see
you
can
like
in
the
HTTP
header,
you
can
even
see
the
format
but
as
if
you
see
like
sorry
brains
like,
for
example,
when
you
see
an
underscore
created,
you
know,
but
it's
pretty
certain
to
be
open
metrics.
If
you
see
an
eof
at
the
end
of
the
file
written
like
this,
you
know
it's
open
metrics.
A
It
should
not
matter
to
you,
of
course,
again
Prometheus
and
everything
handles
this
completely
transparently
and
don't
even
notice,
but
if
you
do
hand
debugging
or
something
those
are
the
really
the
really
detailed
tale.
Science
and
the
unit
of
course,
as
mentioned
earlier,
is
one
of
the
well-breaking
or
new
things.
A
A
There
are
a
few
known
issues
in
1.0
again,
we
sometimes
have
this
issue
reported
that
people
have
clashes
in
their
namespace
when
they
have
existing
underscore
total
there's
not
a
lot.
We
can
do
about
this
because
it's
just
how
how
the
new
standard
works-
and
it
is
actually
a
good
thing
where
we
just
made
a
previous
implicit
requirement
and
strong
recommendation
mandatory.
So
at
some
point
we
had
to
eat
this
pain.
A
If
you
have
counters-
and
you
have
underscore
started
in
there,
this
can
double
the
amount
of
counters
and
the
cardinality,
and
that
can
be
a
surprise
to
a
few
people
and
that
also
can
also
lead
to
pain.
We
might
make
this
optional
in
openmetrics
1.1,
that's
not
yet
clear,
I
guess
we
will
probably
do
it,
because
it's
just
okay
I
see
people
shaking
their
heads
in
their
own
Prometheus
team,
see
it
it.
It
needs
some
more.
It
seats,
some
more
no
more
discussion
and
the
other
thing
is.
C
A
What's
up
for
2.0,
the
Highlight
feature
is
a
high
resolution
or
native
histograms,
where
you
don't
have
this
Le
bucket
thing
anymore,
where
you
need
to
set
your
own,
your
own
boundaries
for
your
buckets
and
everything,
but
you
can
actually
just
toss
it
at
a
thing
and
in
most
cases
it's
just
doing
everything
as
it
should
in
the
cases
where
it's
not
doing
the.
What
it
should
you
can.
A
You
can
set
like
the
the
basic
computations
of
how
the
bucket
boundaries
are
Auto
computated,
but
it's
it's
really
a
game
changer
for
for
Native
histograms.
The
other
thing
which
this
does
is.
It
introduces
for
the
first
time
complex
data
types
in
Prometheus.
Previously
you
needed
one
single
metric
for
every
single
thing
which
you
wanted
to
to
talk
about.
Now
you
can
actually
have
more
than
than
one
single
data
point.
A
A
These
days
the
landscape
has
massively
changed.
People
have
basically
stopped
fighting
with
Prometheus,
because
most
people
just
accept
this
exists,
and
it's
it's
the
thing
which
one
yeah
so
maybe,
but
this
is
another
point
of
discussion.
Maybe
we
release
a
2.0
and
then
we
fold
it
into
Prometheus,
or
maybe
we
fold
it
into
Prometheus,
and
then
we
release
a
tool
that
oh
or
whatever
we'll
see
so
there's
a
few
there's
a
few
resources
here
for
the
ones
who
want
to
take
pictures.
A
A
A
E
Thanks
Richie,
so
I
mean
thanks
for
the
you
know,
kind
of
concept
of
the
roadmap.
It
feels
like
exciting
to
kind
of
like
improve,
improve
this
and
I
was
just
asking
like.
What's
the
historical
reason
for
like
creating
another,
you
know
metric
for
this
created
timestamp.
Why
not
just
immediately
putting
that
into
metadata?
Is
there
any
blocker?
Maybe
you
remember.
A
We
had
many
many
discussions
and
the
short
version
is
that
Google
really
really
really
wanted
to
have
this
because
they
needed
for
monarch
and
honestly
back.
Then
we
like,
we
did
some
back
of
napkin
math
and
we
were
like
okay.
This
should
be
fine
these
days
with
more
and
more
metric
data
exploding,
because
people
keep
adding
more
and
more
and
more.
Arguably,
it's
a
little
bit
of
a
victim
of
its
own
success
here
and
yeah.
That's
why
it's
directly
in
there.
A
The
other
reason
is,
we
didn't
have
any
mechanism
within
Prometheus
to
persist
any
of
this.
This
type
of
metadata
there's
a
good
reason
why
we
could
do
this,
because
unless
you
do
something
fundamentally
wrong
and
and
update
created
all
the
time,
it
doesn't
actually
change.
So,
on
the
cardinality
side
on
the
index,
it
actually
imposes
some
cost
on
the
storage
and
the
actual
block
storage.
B
F
A
A
A
So
the
thing
is,
as
you
can
see,
the
type
of
who
here
is
histogram
not
counter
but
same
difference,
and
the
thing
is,
as
you
can
see
here,
it's
only
Foo
and
then
you
add
those
other
extensions.
You
add
the
bucket
boundaries
again,
not
high,
not
high
resolution,
not
native
histograms.
This
is
the
1.0
Style.
You
see
the
count
you
see,
the
summon
you
see
the
underscore
created
and
the
underscore
created
is
literally
just
a
timestamp
when
you
did
create
the
thing.
H
Hello,
I'm
I
had
a
question
about,
let's
say:
I'm,
upgrading,
a
metrics
library,
for
you
know
language
and
I
want
to
move
it
over
to
open
metrics
from
the
the
existing
format
and
I'm
a
little
bit
worried
about.
You
know
you
know
changing
the
names
of
the
metrics
with
the
you
know,
underscore
total
suffix.
H
A
First,
you
should
look
at
your
data
if
this
is
actually
a
concern,
because
chances
are
if
you
followed,
Prometheus
best
practices,
it
shouldn't
be
a
concern
because
you
wouldn't
have
any
underscore
total,
because
you
are
supposed
to
put
the
unit
at
the
end
and
something
meaningful
and
that's
usually
not
the
underscore
total.
So
the
likelihood
of
this
existing
is
not
super
high.
If
it
exists,
you
need
to
find
some
plan.
A
If
you
write
your
own
library,
because
that's
how
I
understood
you,
then
you
can
in
theory
just
not
do
it
and
not
be
fully
compliant
like
that
works.
But
it's
not
nice
if
you
use
a
real
Library
you're
going
to
run
into
problems,
and
ideally
you
already
start
this
migration
before,
like
the
naming
migration
before
you
do
the
open,
metrics
migration,
the
reason
being,
if
you
have
two
lifts
at
the
same
time
and
something
goes
sideways,
you're
going
to
have
a
really
big
bedtime.
So
it's
better
to
do
like
one
step.
B
I
Hello
earlier
earlier
this
year,
I
was
looking
into
this
for
a
project
and
and
saw
oh
there's
protobufs,
that
I
could
use
which
I
would
have
been
happy
to
use,
and
then
it
was
not
clear.
I
never
found
out
kind
of
what
the
state
of
things
was
with
that
and
I
just
moved
on,
but
is
that
something
it's
just
an
idea?
Is
it
implemented?
Is
it
specked
out
or
is
it
you
find
the
protobots
in
the
spec?
I
A
Who
didn't
pay
attention?
Do
we
already
ingest
the
protobus
for
of
open
metrics
directly
into
Prometheus?
Okay,
then
the
answer
is
currently
no
yeah.
So
the
thing
here,
complete
transparency
and
honesty.
With
my
Prometheus
head-on,
we
were
kind
of
waiting
for
the
native
histograms
to
settle.
Of
course,
we
knew
we
needed
to
to
actually
have
this
like
larger
change
back
because
initially
we
support
it
like
in
Prometheus
1.x
we
promote
supported
protobuf,
and
then
we
took
it
out
because
it
didn't
have
any
any
efficiency
benefits
at
that
time.
A
For
us
and
we've
kind
of
been
waiting
and
I
mean
open,
metrics
mandates,
text
format,
it
does
not
mandate
the
protobuf,
but
for
the
2.0
you'll
have
it
then,
and
then
you
also
have
the
native
histograms,
but
it
will
still
mandate
the
text
and
not
the
Proto.
Of
course
again.
This
debuggability
with
just
your
web
browser
is
something
we
highly
highly
value
and
we
don't
want
to
lose
this,
but
there's
more
efficient
ways.
I
agree,
of
course,.
C
Thank
you,
I
just
wanted
to
ask
your
reference
like
remote,
fright
and
effort
in
standardization
of
that
interface
is
actually
one
of
the
interfaces
used,
the
most
light
to
plant
like
Prometheus,
with
like
long-term
storage,
and
things
like
that.
Can
you
please
comment
more
on
how
this
is
going
to
be
impact
in
the
standardization
and
what
is
going
to
happen,
especially
on
the
parameters
side
here.
A
So
those
are
different,
of
course,
when
you
have
your
exporters
and
your
your
instrument,
It
Whatever
here-
and
you
have
your
Prometheus
here,
then
this
is
where
open
metrics
is
being
spoken.
If
you
have
your
Prometheus
here
again
and
you
have
other
Prometheus
instances
or
you
have
a
long-term
storage
here
is
where
you
speak.
Remote
read
write.
A
So
it's
a
fundamentally
different
use
case.
In
theory,
you
could
hack
together
a
library
which
uses
Prometheus
remote
right
to
push
from
those
workloads
down
there,
but
that's
not
really
what
the
what
the
whole
system
is
optimized
for.
We
do
have
discussions
about
what
how
we
are
going
to
evolve.
The
remote
read,
write
format.
We
might
actually
just
steal,
otlp
and
shove
that
into
Prometheus,
but
that's
an
ongoing
discussion.
We
don't
yet
know
yeah.
C
Yeah,
okay
I
was
asking
really
like
also
in
relationship
to
what
is
happening
on
the
hotel
side.
Where
do
you
have
like
this
agent,
which
is
essentially
like
scrapping
and
then
like
pushing
things
whether
there
was
any
idea
on
the
Primitive
side,
only
scope
to
metrics
to
do
something
similar
and
use
a
standard
interface?
Thank
you.
I.
H
A
H
H
A
Oh
so
yeah
the
format
of
so
the
format
of
the
exemplars.
We
deliberately
did
not
specify
this
within
Prometheus,
but
what
we
did
in
the
specification
is
all
the
examples
which
you
can
find
in.
The
specifications
are
taken
directly
from
the
WC3
tracing
standard
on
how
to
propagate
tracing
context,
morgue,
McLean
and
such
were
working
on
this
within
the
context
of
WC3.
A
B
A
F
F
A
Yes,
yeah
yeah
yeah,
you
you
can
only
by
Design
per
exposure.
You
can
only
expose
one
single
exemplars
because
we
don't
overwhelm
the
storage
or
anything
and
also
we
want.
We
don't
want
to
give
people
to
to
build
a
large
foot
gun
and
turn
promises
into
an
event
monitoring
system,
because
then
you're
going
to
have
a
really
bad
time.
So
you
can
only
put
one
in
the
other
thing
which
you
see
behind
there.
A
But
buckets
are
nice,
but
I
mean
we
are
almost
back
at
Native
histograms.
It's
also
nice
to
have
like
more
specific
hits
and
that's
why
here
we
we
showed
how
you
can
have
your
Trace
ID
and,
at
the
same
time
give
the
user
exactly
the
information
about
what
specific
runtime
in
that
one
latency
bucket
or
whatever
so.
A
J
A
Grafana
Labs,
of
course,
we
run
mimir
I
mean
I
I,
don't
know
if
I
mean
AWS
is
running
cortex
internally
or
as
as,
like
user
facing
I,
don't
know
if
they,
if
they
actually
enabled
the
users
I
honestly,
don't
know
so
at
least
there
is
grafana
Labs.
If
data
talk,
I
don't
know
if
they
have
it
in
a
service,
I,
I
honestly,
don't
know.
I
I
only
want
to
do
do
any
of
you
know
of
others
which
no
okay,
so
at
least
grafana.
Okay.
We
have
five
more
minutes
for
questions.
K
A
Also
be
I
mean
I,
I,
think
the
pr
which
which
I
saw
merged
I
think
today
was
from
the
branch
native
histograms
and
in
the
pr
description
says.
No,
so
in
the
branch
name,
what
Spar
system
comes
and
the
pr
said
native
histograms,
so
just
to
give
you
an
idea
of
how
much
of
a
mess
this
is
but
going
forward.
We
are
having
one
name.
We
also
have
one
name
which
is
synchronized
between
Prometheus
and
between
open
Telemetry.
To
just
have
we
have
two
names:
okay,
we
have
okay.
B
D
G
A
C
A
Standard-
okay,
that's
basically
it
like
we've
been
we
even
before
we
cut
the
1.0
release.
We
already
had
consensus
that
2.0
would
be
oh,
that
exponential
or
as
far
as
histograms,
would
be
the
reason
why
we
cut
2.0.
So
we've
been
waiting
for
this
quite
some
time.
I
can't
give
you
any
specific
timeline,
but
we've
been
waiting
for
this
for
quite
some
time.
We
want
to
get
this
going.