►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: OpenMetrics - Richard (RichiH) Hartmann, SpaceNet AG
OpenMetrics is taking the Prometheus format and transforms it into a truly open standard while remaining compatible with the hundreds to integrations and dozens of data collectors that grew within our ecosystem. We will go through the current state of OM, the challenges we faced getting there, and an outlook for next steps and allies would found along the way.We will have an extended open Q&A at the end to enable you to ask questions.
To Learn More: https://sched.co/I1kk
A
First
things:
first
I
made
open
Metrix
ribbons.
If
anyone
is
interested
in
a
ribbon
grab
on
their
rainbow-colored,
it
will
be
a
little
bit
of
back
and
forth
course.
I,
don't
have
a
copy
screen
here
and
I.
Don't
have
a
clicker,
so
I
kind
of
have
to
lean
over
to
see
what
we're
talking
about.
So
this
is
the
interim
into
to
open
metrics.
My
name
is
Richard
I
do
a
few
things,
amongst
others,
have
a
permit
his
team
member
show
of
hands.
Oh
no
sorry!
A
So,
first
thing
we
will
be
talking
a
little
bit
about
promises,
even
though
it's
an
open,
metrics
talk.
This
is
because
there
it's
so
much
intertwined
it's
more
or
less
impossible
to
talk
about
open
about
open
metrics
without
also
mentioning
promises
to
some
extent
so
who
has
heard
of
Prometheus
or
is
using
Prometheus
that
it
is
pretty
decent.
I
would
say,
I
think
that's
a
hundred
percent
like
a
clear
hundred
percent,
so
I'm
just
going
to
do
it
like
this.
Before
Prometheus
we
had
something
of
a
monitoring
landscape
where
everything
was
extremely
fragmented.
A
A
I
see
people
nodding
so
yeah
and
a
lot
of
them
had
hierarchical
data
models
in
as
much
as
you
have
region,
you
have
country,
you
have
data
center,
you
have
whatever,
and
then
you
might
have
a
customer
who
is
in
one
data
center
and
then
all
of
a
sudden
he
buys
more
services.
So
now
he's
in
a
region
and
all
of
a
sudden,
you
have
a
rocket
data
model
where
first,
you
had
your
your
stuff,
your
customer
hooked
on
their
data
center
and
now
they're
in
the
region
that
hierarchical
data
structure
needs
to
be
changed.
A
Thank
you,
so
the
one
good
thing
about
SNMP
is
it
works
and
it
exists,
and
most
things
have
something
which
which
implements
SNMP,
but
it
still
sucks
everyone
who
ever
implemented
ace
and
dot.
One
knows
the
pain
of
actually
doing
things
with
SNMP
a
depth.
Of
course
it
was
built
for
a
different
time.
Obviously
we
are
now
talking
about
a
new
standard,
so
yeah,
let's,
let's
see
if
we
can
get
there
so
after
Prometheus,
this
has
become
the
de
facto
standard.
It's
yeah,
it's
especially
when
you
use
kubernetes.
A
You
can't
really
get
around
using
perimeters
at
all,
even
if
in
non
cloud
native
environment
it
has
massive
massive
adoption.
We
are
talking
more
than
300
exporters
written
by
people
who
are
not
on
team
tens
of
thousands
of
companies
using
primitives.
So
there
is
a
huge
installed
base
for
using
primitives,
and
one
of
the
issues
we
are
facing
is
a
lot
of
people
were
kind
of
worried
about
adapting
something
from
a
competing
project,
even
though
we
don't
have
any
profit
motive
at
all
cause
we're
just
a
few
people
doing
stuff.
A
They
were
kind
of
worried
about
adapting
this
name
of
something
else
and
coming
from
the
networking
space
myself
and
also
currently
building
a
datacenter
I
interface
with
a
lot
of
more
traditional
vendors
and
those
guys
believe
in
official
standards.
They
believe
in
inspects,
written
in
POS
and
in
pretty
much
nothing
else,
so
to
speak
their
language.
You
need
to
somehow
have
something
official,
switching
back
to
cloud
native.
A
We
are
currently
in
this
magic
buzzword,
gold,
rush
phase
for
8,000
people
stream
into
one
single
occasion,
just
to
talk
about
low
native
stuff
and
every
single
company
likes
to
stake
their
claim,
because,
obviously
they
intend
to
make
money,
but
staking
your
own
claim
means
fragmentation.
I,
don't
have
any
profit
motive,
so
I
just
decided
to
okay.
Let's
try
to
use
this
thing
and
just
make
it
into
a
standard.
Of
course.
We
can
basically
and
prevent
part
of
this
fragmentation.
Of
course,
you
always
have
like
this
explosion.
A
Then
this
contraction
again
and
to
prevent
part
of
this
of
this
explosion.
Why
not
try
and
contract
early
few
acknowledgments
I
said
I
started
this,
but
I've
not
been
doing
this
myself.
There
are
several
people
working
this.
The
main
work
has
been
done
by
ben
kochi,
brian
brazil
and
myself
from
prometheus
team
Sameer
Bhalla
from
google
jerome
Furley
who's
actually
here
at
the
venue,
but
he's
not
no
he's
not
here
in
the
room
and
drop
skeleton
both
from
uber
there's
tons
of
companies
who
already
committed
to
supporting
open,
metrics,
there's
tons
more.
A
It's
it's
easily
three
times
that
in
the
meantime,
it's
there
there's
a
huge
demand
for
this
thing,
where
you
have
like
a
standard
which
people
can
agree
on.
So
my
approach
is
pretty
simple:
I
take
promises
exposition
forward
as
the
basis
we'll
be
seeing
it
later
on
later
slides
how
this
looks
like
for
a
very
simple
reason.
A
Where
know
that,
except
for
a
few
corner
cases,
I
have
full
backwards
compatibility
and
something
which
ingests
open
metrics
data
can
just
ingest
promises
data,
and
if
you,
if
you
took
to
heart
the
best
practices
of
Prometheus,
you
will
not
run
into
those
corner
cases
of
of
Prometheus
exposition
format
versus
open
metrics
and
those
changes.
So
by
having
this
this
kind
of
umbrella,
where,
where
everyone
comes
again
under
a
a
neutral
brand,
basically
the
same
what
CNC
F
does
on
a
larger
scale.
A
People
are
more
willing
to
actually
engage
on
this
and
to
work
on
this.
So
when
I
approach,
people
with
the
proposition
of
hey,
why
not
happen?
I
have
two
premises:
exposition
formative,
I,
don't
know
now
that
this
is
something
which
they
know
will
become
a
standard
which
is
not
tied
to
anything
else
based
on
something
else,
but
not
tied
to
it.
A
People
are
coming
in
droves
also,
we
have
tons
of
implication
or
assumptions
in
our
stuff
and
there's
tons
of
operational
experience
on
how
we
do
things
in
premises.
But
still,
this
is
like
a
dozen
people,
nearly
20
people
in
the
mean
time,
but
the
initial
design
has
been
done
by
maybe
about
maybe
half
a
dozen
people,
so
there's
tons
of
stuff
in
there,
which
is
good
stuff,
but
it
has
not
taken
into
account
all
those
other
use
cases,
and
this
is
something
very
especially.
A
A
A
If
you
want
to
play
with
this,
you
also
have
the
Python
client
library,
which
also
supports
emitting
open,
metrics
and
appear
using
the
current
version
of
the
Python
client
and
the
current
version
of
premises.
They
were
already
starred
or
the
negotiating
between
themselves
and
start
using
primitive,
open
metric.
Sorry
without
even
noticing
just
so.
We
can
push
this
into
the
wild
without
breaking
any
any
backwards,
compatibility
as
to
the
full
upgrade
path
we'll
have
to
see.
For
that.
A
There's
certain
things
we
have
to
take
into
account:
the
full
migration
will
probably
take
a
year
or
so
just
to
make
sure
that
people
have
time
to
move
over,
maybe
even
more
than
a
year,
but
it
doesn't
matter
because
all
the
new
stuff
you're
putting
out
will
just
start
speaking
up
in
metrics
as
soon
as
it's
able
to.
If
you
want
to,
let
me
move
the
mouse
cursor.
Sorry
I
didn't
realize
if
you
want
to.
If
you
want
to
test
all
this
and
start
implementing
your
own
stuff,
this
is
a
URL.
A
This
is
something
again
coming
from
the
networking
space
you
have
an
RFC
or
think
everything
is
defined,
its
people
start
implementing
things
and
there's
bits
and
pieces
which
diverge
so
by
having
two
separate
entities,
write
it
from
scratch.
We
hope
to
weed
this
out,
we'll
see
if
anyone
else
is
interesting
to
working
and
suspects
and
implement
a
third
one
be
my
guest.
A
Next
steps
follow
up
a
metric
supporting
Prometheus.
This
is
already
basically
done
just
as
open
metrics
evolves
at
the
last
few
bits
and
pieces,
but
this
is
basically
done
in
flux,
open
census,
M,
3
DB.
All
these
are
committed
to
doing
this.
Also
sec
driver
obviously
spreading
the
word,
which
is
why
why
I'm
here
to
just
get
more
people
to
use
it
and
move
from
sandbox
to
incubating
within
CN
CF?
Basically,
sandbox
is,
let's
see
if
it
works,
so
if
it
dies
incubating,
is
actually
they're
putting
some
sort
of
trust
in
the
project.
A
But
this
is
not
just
about
cloud
native
stuff,
myself
again
I'm
building
a
data
center.
So
it's
very
much
away
from
all
the
cloud
native
stuff
or
a
different
type
of
cloud.
I,
don't
want
to
just
go
into
this
space.
I
want
to
go
way
beyond
this.
I
want
to
have
the
traditional
vendors
I
want
to
have
Hardware
people
who
actually
implement
open,
metrics
arista,
if
you
know
them
they're,
a
pretty
large
network
vendor
they're
already
committed
to
doing
this.
A
They
already
have
working
images
for
for
primitives
and
I
will
change
them
for
open
metrics
to
also
support
this
virtus,
which
is
the
largest
company
for
for
data
center
stuff,
like
cooling
units
and
power
distribution.
Ups
and
such
I
showed
this
to
them
last
week
and
they're
really
interested
to
just
replace
their
own
stuff
for
their
own
lab
course.
A
It's
horrible
what
they
have
if
it's
90s
level
of
metrics,
it's
horrible,
you
select
a
line,
and
then
you
select
the
marker
and
then
you
can
move
the
mark,
and
then
you
know
the
value
of
whatever
you're
looking
at.
So
you
take
20
seconds
to
just
look
at
a
value
on
the
line.
It's
ok
anyway,
but
it's
not
only
just
about
metrics.
It's
also
going
beyond
metrics,
encompassing
more
than
just
a
name,
open,
metrics
and,
of
course,
for
example,
which
something
which
is
really
useful.
That's
coming
from
Google
Ads.
They
have
this
requirement.
A
Open
measures
allows
you
to
attach
exemplars.
So
if
you
listen
to
the
talk
by
David
Koch
myths
from
your
father
this
morning,
correlation
of
stuff,
for
example-
a
trace
to
histogram
packets
through
labels
is
really
really
powerful
at
scale.
At
some
point
it
starts
breaking
down,
so
exemplars
is
basically
a
trace
ID
which
attach
to
your
to
your
metric
to
your
histogram.
A
A
Obviously,
promises
doesn't
support
us
I.
Don't
expect
us
to
support
this
within
parameters.
To
be
honest,
so
open
metrics
allows
you
to
just
ignore
this
data
ordering
it
ingest
it
like,
for
example,
stackdriver
would
do
both
is
fine,
then
the
other
thing
I
already
got
a
little
bit
into
this
earlier,
with
the
with
the
state
claiming
I
like
to
say
that
infrastructure
or
architecture
breathes
where
you
basically
go
in
and
out
all
the
time
you
have
your
machine,
you
have
your
virtual
machine,
you
have
your
container,
you
have
your
micro
service.
A
Basically,
you
go
from
one
extreme
of
centralization
to
decentralization
or
something
else
back
and
forth.
You
might
raise
the
state
of
the
art,
then
you
hit
new
problems
and
you
go
a
little
bit
back.
There
becomes
a
bus,
you
go
all
the
way
you
hit
the
wall
and
you
go
back
after
raising
the
bar
a
little
bit
more
so
Prometheus,
basically
breathed
in
in
as
much
less
today
as
we
decided
consciously
to
just
focus
on
metrics
and
ignore
everything
else
on
purpose.
A
To
do
one
thing:
do
it
well
yet
for
the
wire
format,
this
doesn't
make
sense
for
the
project
of
Fermi's
didn't
make
total
sense,
but
for
the
wire
format
it
does
not
make
sense.
So
I
want
to
go
beyond
just
metrics.
I
want
to
have
the
same
format,
the
same
wire
format
for
other
stuff.
If
you
look
at
stats,
T,
it's
it
works,
but
it's
very,
very
unstructured.
So
what
we
are
looking
at
is,
for
example,
creating
something
you
called
open
stats,
which
looks
exactly
the
same
like
open
metrics,
but
it's
push-based
system.
Why
don't?
A
We
just
have
both
of
them
in
the
same
standard,
for
one
simple
reason,
or
for
two
simple
reasons:
a
there
is
confusion,
if
you
say
I'm,
open
Becker
is
compliant
and
you
only
do
push
on
the
other
side.
I
am
I'm,
optimistic
implying,
and
a
little
spell
does
pull
that
kind
of
doesn't
work,
and
the
other
thing
is.
A
few
working
assumptions
are
different.
For
example,
sparseness
with
open
metrics.
A
You
have
to
scrape
all
the
data
all
the
time,
every
single
time
in
push
situations,
it
can
make
sense
to
push
sparse
data
and,
if
I
don't
update
something
it
just
didn't
change,
that's
a
huge
difference
and
it
has
tons
of
implicit
implications
down
the
wire
functionally.
It's
the
same
and
it's
trivial
to
write
a
proxy
which
translates
one
to
the
other.
A
But
the
the
systems
at
the
back
end
have
huge
implications
on
how
you
have
to
write
them
and
that's
why
we
decided
to
just
have
split
the
two,
but
you
can
use
the
same
label
sets
and
the
same
label
concept
for
other
stuff.
Again,
if
you
looked
at
Loki
this
morning,
it's
log
files
with
premises,
labels
basically-
and
you
can
do
the
same-
for
traces
for
chorim's
for
whatever
so
by
having
this
one
wire
format
or
family
of
wire
formats,
which,
basically
all
look
the
same.
A
This
correlation
between
different
data
becomes
a
lot
easier
because
at
the
moment
it's
a
mess.
It's
a
huge
mess,
so
I
talked
a
lot
about
how
all
this
works.
Now,
let's
look
at
how
this
looks?
You
have
your
metric
names.
You
have
your
label
set,
you
have
your
value.
This
is
prometheus.
So
look
close
now.
There's
the
change
to
open
metrics.
This
is
open
metrics.
A
So
again,
if
you,
if
you're
doing
good
stuff
and
you're
taking
care
of
doing
things
properly,
you
can
just
reuse
your
old
stuff,
which
is
like
one
of
the
core
goals
of
what
we're
doing.
But
there's
one
breaking
change.
Well,
actually,
there's
two
promises
uses
microseconds.
If
you
do
explicit
time
stamps,
which
you
probably
shouldn't
be
doing
anyway,
but
it's
microseconds
in
parameters,
we
change
it
to
seconds
because
we're
pushing
everyone
to
always
use
base
units,
so
we
obviously
have
to
make
you
space
units
here.
A
The
other
thing
is,
you
can
use
exemplars,
you
just
have
a
hash
pound.
You
have
your
label
set
again.
The
label
set
is
same
for
events
or
whatever
you
have
the
value
you
have
a
timestamp
of
when
that
one
trace
or
would
ever
happened.
So
this
is
the
ID
of
my
trace,
and
this
is
the
timestamp
when
it
happened.
This
can
obviously
also
be
sub
second
position
and
they
don't
have
to
agree.
A
So
obviously
it
has
to
happen
in
the
past
relative
to
this
grape,
which
this
number
is
lower
than
this
timestamp
obvious,
but
else
just
works.
We
can
have
explicit
or
implicit
timestamps
worse
and
that's
basically
it
these
two
changes
and
you're
not
free
to
here.
You
can
do
whatever
with
white
space
here,
you're
not
allowed
to
have
endless
white
space
because
it
makes
the
parser
quicker
to
not
anticipate
random
white
space.
So
there
is
tons
of
further
reading
again
these
slides
can
be
downloaded.
A
A
A
Okay,
the
question
is
why
we're
switching
to
seconds,
basically,
we
had
microsecond
time
stamps
before
and
now
we
have
second
time
stamps.
We
are
encouraging
everyone
to
use
base
units
and,
as
I
may
be
exact
for
kilogram
course.
Kilogram
is
the
base
unit
for
whatever
weird
reason:
you're
just
cutting
off
these
last
three
zeros,
and
if
you
want
to
have
microsecond
position,
that's
what
you
do
so
you're,
not
actually
changing
too
much.
Obviously
those
are
float,
so
you
lose
position
at
some
point,
that's
true,
but
else
there
is
no
change.
A
It's
just
to
have
everything
in
the
same
because
we
can't
keep
telling
people
that
you
should
always
be
using
seconds
always
bites
always
bits
and
not
like
kilobyte
or
anything
and
then
use
microseconds
within
the
standard.
So
that's
why
we
changed
it.
It's
a
breaking
change,
but
we
decided
that
it
actually
makes
sense
going
forward,
because
if
this
still
exists
in
like
20
years,
everyone
will
hate
us
for
this
one
yeah.
A
Okay,
the
question
was:
if
we
define
a
language
how
to
describe
labels,
we
do
not
we
have
so
in
the
RFC.
We
will
have
consideration
sections
which
basically
spell
out
best
practices
and
a
lot
of
things
which
have
been
done
in
Prometheus
for
a
long
time
will
be
spelled
out
and
there
will
be
a
rationale
of
why
it
makes
sense
to
do
this,
but
it's
not
going
to
be
mandated,
so
you
can
do
whatever,
but
still
we
will
try
to
push
people
into
the
right
direction.
A
Obviously,
naming
is
hard
and
people
are
opinion
opinionated
about
names.
We
definitely
deliberately
have
constraints
on
what
you
can
put
here.
You
can,
basically,
you
have
to
start
with
an
S
key,
and
then
you
have
Eskie
and
underscores,
or
you
have
chars.
You
have
numbers
and
underscores
and
nothing
else.
So
you
can't
put
a
daughter
or
something
like
that
for
just
for
to
force
people
to
not
just
cargo
cult
from
one
thing
into
the
other,
classic
example
would
be
just
people
using
graphite
style
and
metrics
and
just
poaching
them
over
there.
A
A
A
Okay,
if
I
were
to
have
this
discussion,
my
main
point
is
labels
labels
and
also
labels,
curse
by
by
not
having
this
hierarchical
data
model
and
just
having
this
n
dimensional
matrix,
where
you
can
slice
and
dice
your
data.
However,
you
need
it's
just
genius
and
it's
just
so
more
powerful.
So
that
is
my
main
point
about
why
I
like
this?
If
someone
really
complains
about
this,
there
is
probably
options
to
make
UI
better
to
hide
labels
and
to
put
them
back
into
structures
or
something,
but
the
underlying
data
should
be
as
versatile
as
this.
A
A
So
the
question
was
when
you
are
familiar
with
Prometheus.
If
you
scrape
this
data,
you
will
also
have
an
instance
label
in
a
job
label,
and
how
are
we
intending
to
do
this
within
open
metrics
and
also
if
the
exemplar
is
relate
to
this?
The
exemplars
don't
relate
to
this
at
all.
They
are
just
to
attach
an
event
most
likely
a
trace,
but
it
could
be
a
log
line.
It
could
be
a
core
dump
to
a
metric
bucket.
A
That's
all
it
is
for
for
open
metrics
again
we
will
have
considerations
and
best
practices
where
we
state
that
you
should
attach
information
about
where
this
is
coming
from
to
your
data
in
open
stats,
we
will
mandate
that
you
put
that
data
there
yourself
course
in
a
push
scenario.
You
cannot
always
be
certain
where
the
data
is
coming
from.
So
this
is
one
of
the
again
in
theory.
A
It's
absolutely
the
same
thing
when
you
go
into
the
specific
implementation,
it's
hugely
different,
so
open
metrics
will
work
like
the
Prometheus
or
suggests
that
people
work
like
Prometheus
will
not
mandate
it
for
open
stats.
We
will
either
strongly
encourage
or
mandate
that
you
actually
push
put
put
instance,
information
or
whatever
information
to
your
to
your
label,
set
to
avoid
this
problem.
A
A
Yep,
so
the
question
is:
if
we
are
just
limiting
ourselves
to
dis
or
if
we
go
further
in
and
specifying
it,
we
do
go
further.
This
has
to
be
served
over
HTTP.
There
are
some
headers
which
you
have
to
set,
so
you
can
do
auto
negotiation
order,
detection.
It
will
not
only
be
text,
there
will
also
be
a
proto
version,
and
these
are
then
basically
41.0
the
two
sanctioned
ways
to
do
this.
A
Of
course,
those
make
sense
in
in
this
day
and
age
yeah.
So
there
will
be
more
basically
the
scope
and
the
limit
of
what
we
do
is
we
want
to
make
certain
that,
whatever
solution
you
have
which
claims
to
support
open,
metrics
and
you
have
something
else
which
also
supports
to
claim
claims
to
support
or
metrics.
They
can
talk
to
each
other
usefully
and
that's
thus
the
exact
scope
and
also
the
limit
of
what
we
are
doing.