►
From YouTube: Deep Dive: Prometheus – Richard "RichiH" Hartmann, SpaceNet & Ben "SuperQ" Kochie, GitLab
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Prometheus – Richard "RichiH" Hartmann, SpaceNet & Ben "SuperQ" Kochie, GitLab
Join Prometheus maintainers for an open Q&A about all the nitty gritty details of what makes Prometheus tick. We won't give a programme in advance on purpose; instead, we encourage you to shape our time together by pre-filling our form : https://goo.gl/forms/jjn4s0YIfMS2M7N63 or asking during the session.
To learn more click here: https://sched.co/HMiC
A
B
Doing
the
first
half
alright
so
back
to
the
same
questions,
who's
heard
of
Prometheus
yay
who's
who's,
considering
using
Prometheus,
alright,
so
who's
already
using
it
as
like
a
proof
of
concept
or
in
testing
few
people,
good
good
and
then
who's
who's
shifted
all
the
way
to
production
awesome.
So
we're
gonna
have
a
lot
of
questions.
Aren't
we
so
back
to
previous
101,
it
was
inspired
by
a
Google's
internal
monitoring
system.
It's
a
numeric
metric
time
series
based
metric
system.
It
basically
stores
a
series
of
data
non
numeric
data
points
over
time.
B
More
than
3-hundred
it's
not
it's
a
it's
a
single
tool
for
a
single
job.
It's
not
a
fix
everything.
It
doesn't
do
tracing
it
doesn't
do
logging,
it's
not
an
event
store.
It's
just
for
the
first
level
of
marring
and
our
favorite,
our
favorite
plug-in
tool
for
Prometheus
is
go
fauna.
It's
usually
the
second
thing.
I
install
after
I
install
a
Prometheus
server.
The
main
selling
points
over
other
metric
space
monitoring
systems
is,
it's
super
highly
dynamic.
It's
super
efficient.
We
have
a
very
powerful
query
language
and
it's
super
simple
to
operate.
B
A
lot
of
people
come
to
us
and
say
well,
I,
don't
want
to
host
my
own
monitoring,
it's
a
lot
of
work.
You
know
I've
used
Nagios
and
it's
always
been,
and
many
many
engineers
worth
of
time,
running,
Nagios
or
or
other
older
metric
systems
like
graphite
prometheus
is
super
easy
to
operate.
It's
not
as
big
a
deal
as
classic
monitoring
systems.
A
Do
it
so
now
we
are
here
cloud
native
con
and
everything's
everything's
in
container
everything's,
a
micro
service.
Everything
has
sidecars
and
all
of
a
sudden.
You
have
this
one
thing
which
is
kind
of
a
big
thing
in
in
in
CNC
F
land
and
it's
one
monolithic
application,
and
this
is
Prometheus,
and
why
are
we
doing
this?
Of
course?
A
A
Yeah,
of
course,
what
you
really
need
for
operations
is,
if
you,
if
you
look
at
what
you
actually
provide
on
the
physical
level
and
and
a
little
bit
above,
you
need
power
and
cooling
for
a
data
center.
You
need
network
connectivity
to
connect
to
your
services
and
to
debug
and
stuff
and
need
observability,
AKA
monitoring,
and
that's
about
it.
The
rest
you
can't
fix
so
because
this
is
in
the
absolute
critical
path.
B
So.
Where
are
we
at
now
with
Prometheus?
So
we
have
the
storage
back-end.
About
a
year
ago,
we
developed
a
new
storage
time
series
database
to
significantly
reduce
the
overhead
of
running
Prometheus
from
a
computer
resource
perspective
and
also
it
significantly
improved
the
performance
of
Prometheus
when
you're
running
kubernetes
clusters,
where
you're
launching
and
creating
and
destroying
pods
continuously.
So
if
you've
got
a
CI
pipeline,
the
old
prometheus
had
lots
of
problems
with
that.
We
also
improved
staleness
handling.
One
of
the
old
problems
with
prometheus
was,
as
pods
came
and
went.
B
You'd
have
that
you'd
still
see
the
pod
for
five
minutes
until
it
would
finally
disappear.
We
fixed
that
it's
now
much
better
and
yeah.
We'll
have
12
links
to
the
specifics
of
that
at
the
end
of
the
talk
and
also
we've.
Finally,
stabilized
a
remote
read
and
write
access
to
the
database
so
that
you
can
take
a
Prometheus
server
and
get
access
to
the
data
or
ship.
The
data
outside
for
more
in-depth
analysis
or
long-term
storage.
B
So
Prometheus
1.0
had
a
bunch
of
problems.
It
had
a
single
index
for
the
entire
lifespan
of
the
Prometheus
server
and
this
would
grow
and
grow
and
grow
as
paas,
and
that
was
a
big
problem.
The
original
database
was
actually
quite
clever,
but
in
terms
of
like
how
fast
we
could
we
could
spin
it
up
and
get
it
going,
but
it
was.
It
turns
out
that
creating
a
file
for
every
single
time
series
in
a
kubernetes
cluster
just
didn't
scale.
You
know
we
had
at
Sound
Cloud.
A
Okay,
so
basically,
this
is
the
test
we
did
back.
Then,
when
we
introduced
a
new
storage
engine
for
2.0,
we
had
one
cluster
with
their
imitated,
Prometheus
notes,
sitting
aside
of
the
cluster
and
what
we
did
is
we
had
800
microservices.
We
had
100
to
120
km,
birds
per
second
300
K,
active
time
series
and
every
10
minutes.
50
percent
of
those
part
died
and
were
put
in
a
new,
and
this
was
the
test
setup
and
the
result
was
drumroll.
A
A
15-time
reduction
in
memory
usage,
6
times,
reduction
in
CPU
usage
up
to
a
hundred
time,
reduction
in
disk
writes
5
time
reductions
in
on
disk
size,
four
times
reduction
in
latency,
and
we
actually
reputed
service
results.
If
anyone
doesn't
trust
us
course,
people
tend
to
not
trust
us
on
these
numbers,
but
they
are
actually
really
real
numbers.
Yeah.
B
One
of
the
other
things
that
we've
introduced
recently
is
as
part
of
our
our
code
review
pipeline.
We
actually
built
in
prom
bench
so
that
we
can.
We
can
automatically
spin
up
using
a
github,
slash
command
to
spin
up
a
benchmark.
So
anytime
we
introduce
a
time
series,
database
change
or
an
integration
change.
We
can
just
spin
up
a
whole
Prometheus
benchmark
to
see
if
we
have
a
performance
degradation
because
everyone's
about
we,
we
we
make
a
change.
The
time
series
database
things
get
worse
so
now
now
we
can
basically
spin
up
this
benchmark
anytime.
B
We
want
so
also
with
the
remote
readwrite
API.
This
is
much
basically
stable.
Now
we
have
a
whole
bunch
of
integrations
for
external
time,
series,
databases
and
other
related
long-term
storage
methods.
So
if
you
have
a
very
have
a
very
dynamic
cluster,
like
you're
running
your
Prometheus
inside
your
kubernetes
cluster,
you
usually
only
want
to
keep
a
few
days
or
maybe
a
couple
of
weeks
of
data
in
inside
the
kubernetes
cluster,
and
you
can
use
the
remote
write
API
to
ship
all
that
data
out
to
an
external
storage
for
long
term.
B
We've
we
wanted
them
to
focus
on
like
making
sure
that
we
were
had
a
good
data
model
and
didn't
have
any
really
obvious
security
problems
with
Prometheus
the
biggest
thing
that
they
came
back
and
said.
Is
you
really
should
have
endpoint
protection?
So
we
we
are
actually
state.
We
took
a
vote
at
the
last
developer
summit
and
said:
yes,
we
should
probably
start
doing
endpoint
protection
so
over
the
next
sometime.
It's
now
in
our
to-do
list
to
actually
start
implementing,
TLS
encryption
and
authentication
for
export
or
endpoints.
A
A
A
After
six
weeks
we
start
a
new
RC
cycle,
also,
if
there's
any
really
bad
bugs,
what
we
decided
to
do
is
we'll
call
a
moratorium
on
any
new
features
to
re-stabilize
course.
That's
something
we
didn't
do
very
well
in
two
point:
X
like
maintain
stability
for
for
end-users,
and
we
definitely
needed
to
get
better
at
that.
So
also
there's
quite
some
nice
field
and
2.4
2.5
are
already
under
this
new
release
cycle,
which
is
why
they're
in
this
section,
so
Brian
did
something
really
nice.
A
A
Then
also
there
is
this
recurring
thing
of
long
term
storage,
so
most
people
who,
if
they
have
to
voice
one
concern
with
Prometheus,
it's
that
they're
not
quite
certain
how
to
do
long
term
storage,
myself,
I
just
kept
all
my
data
and
my
servers
have
data
since
the
late
2015.
So
that
works
still,
it's
not
the
ideal
working
model.
So,
as
we
fundamentally
designed
fermitas
to
have
distinct
data
islands,
that's
not
something
we're
about
to
change
within
permeated
itself
by
definition
and
by
design.
A
But
what
we
have
is
we
have
our
new
storage
which
allows
you
to
actually
take
snapshots
and
backups
which
wasn't
possible
with
premises
one
not
X.
So
that's
already
quite
good.
If
you
need
to
restore
something
and
also
this
remote
readwrite
API
allows
you
to
export
data
to
two
other
things,
and
there
is
a
ton
of
projects
which
integrate
with
our
remote
readwrite
API.
There's
a
total
of
12
in
the
meantime,
which
came
from
permeated
itself.
A
Our
cortex
antennas
cortex
basically
distributes
the
index
and
the
ingesting
on
auto-scaling
parts,
while
tannaz
does
a
totally
different
thing
and
it
puts
all
the
other
storage
into
object,
storage
like
SEF
or
something
which
is
quite
nice,
and
we
actually
expect
those
two
to
merge
back
again
in
some
time.
And
then
you
have
something
like
a
distributed
premises
like
a
truly
distributed
communities,
but
there's
others
as
well,
and
we
definitely
do
not
want
to
say
okay,
we
endorsed
this
one
thing
and
you
shouldn't
use
any
other.
Of
course
we
don't
really
want
to
be
kingmakers.
A
We
want
to
see
what
the
community
comes
up
with
and
whatever
is
best
should
just
will
course.
We
don't
have
any
profit
motivation,
so
we
can
just
do
whatever
and
would
see
what
works
best.
Looking
beyond
just
2.5
and
beyond
one
thing,
which
is
still
missing,
we
thought
we
would
have
that
earlier
is
like
a
proper
asset
database.
We
have
the
automa
see
we
have
the
consistency,
we
have
the
durability,
we
are
still
missing
the
isolation.
There
is
a
patch
set
which
has
been
carried
since
early
to
dot
X.
A
It
hasn't
been
merged
yet,
but
basically,
what
we're
going
to
do
to
have
proper
isolation
as
and
you
only
get
one
full
scrape
in
your
data
set
or
you
get
or
you
don't
get
it,
but
you
never
see
a
partial
scrape
result.
That's
what
isolation
means
in
this
context
is
basically
we
just
take
a
monolithic
counter
or
monotonic
counter.
A
We
count
it
up
and
every
every
ID
or
every
every
scrape
gets
one
one
ID,
and
unless
that
is
committed,
you
don't
return
that
sample
and
no
matter
in
which
context
it's
relatively
easy
in
at
least
on
this
layer,
but
there's
more
beyond
just
parameters.
Of
course,
there
I
mean,
if
you,
if
you
know
the
story
of
permeate,
is
how
how
the
Titans
stole
the
light
from
from
the
gods
and
gave
it
to
humans
there's
quite
some
ambition
in
the
name
of
Prometheus,
and
this
is
by
design.
A
So,
yes,
we
do
actually
want
to
change
the
world
and
actually
change
how
the
world
does
monitoring.
We
firmly
believe
that
any
hierarchical
data
structure
in
monitoring
is
doing
it
wrong
by
default,
so
labels
we
consider
labels
one
of
the
most
powerful
things
which
we
have
within
premises
and
labels
are
encoded
in
our
exposition
format.
A
So,
while
there
is
a
ton
of
political
considerations
about
people
supporting
something
called
Prometheus
exposition
format,
we
actually
want
to
push
the
system,
because
that
means
people
will
start
using
labels
more
and
more
so
we
came
up
with
open,
metrics
yeah,
that's
basically
it
we're
spending
on
permissive
exposition
format.
It's
already
a
CNC
F
sandbox
project.
We
are
aiming
to
have
an
IT
IDF
RFC
publish
the
full
test,
suite
blah
blah
blah,
so
we
have
something
actually
official
to
to
run
against
I'm.
Currently.
A
Actually,
writing
the
internet
draft,
which
will
hopefully
become
the
RFC
and
2.5,
already
has
open
metric
support
for
me
so
2.5,
but
it's
experimental,
it
might
change
it
might
break.
We
might
do
away
with
it,
but
it
already
works
and
also
our
pile
Python
library
already
is
able
admitted
it's
able
to
emit.
2.5
sorry
emit
open,
metrics
metrics,
so
open
matrix
is
just
metrics,
but
when
we
could
already
do
is
we
can
attach
exemplars
in
open
metrics
to
certain
histograms
to
a
certain
buckets.
A
So
basically
you
have
a
trace,
and
you
know
this
ran
for,
let's
say
more
than
60
seconds,
and
then
you
attach
one
of
those
traces
from
that
bucket
of
latency
to
your
data.
So
you
have
a
tire
link
between
your
latency
packet
and
your
trace
and
if
you
have
really
high
latency
in
some
cases,
you
already
know
what
trace
I
need
to
look
at
there's
some
integrations
like
open
census
and
stack
travel,
for
example.
They
want
to
have
this.
They
want
to
support
this.
They
want
to
ingest
this
data.
A
Our
are
free,
for
example,
promises
to
just
drop
this.
This
reference
to
the
exemplar
course
currently
within
premises.
We
don't
really
have
a
use
for
it.
This
might
change,
it
might
not
change,
but
so
basically,
open
metrics
is
designed
to
allow
people
to
do
this,
but
not
force
people
to
support
this,
and
we
are
currently
only
looking
at
open
metrics,
but
we
also
want
to
have
markers.
Metrics
is
usually
the
start
of
your
monitoring
story
or
your
observer
ability
story,
but
there's
tons
more
to
it.
So
we
also
want
to
have
events.
A
We
want
to
have
traces,
we
don't.
We
want
to
have
all
these
things
within,
something
which
looks
like
open
metrics,
where
you
have
labels,
where
you
have
your
well
defined
data
structures,
basically
to
redo
a
new
level
of
observability
across
the
whole
ecosystem.
Tom
Volk
is
actually
already
working
on
something
which
is
basically
for
like
log
files.
A
If
you
have,
if
you
imagine,
having
premises
for
log
files,
that's
what
he's
working
on,
which
is
pretty
awesome,
and
just
by
happenstance
it'll
support,
open
metrics
or
whatever
we
end
up
calling
the
done
stuff,
there's
already
quite
a
few
people
who
are
companies
who
are
committed
to
supporting
open
metrics.
These
are
just
the
first
adopters
because
they're
in
the
meantime,
too
many
to
list
all
of
them,
and
yes,
we
do
want
to
change
the
world
like
seriously.
A
We
will
always
commit
to
having
simple
and
resilient
operation.
That
will
always
be
the
case.
Our
most
important
thing
will
always
be
to
get
from
raw
data
through
computation
into
alerts.
Code
step
is
what
what
enables
you
to
keep
your
services
up
and
running,
we're
talking
to
hardware,
vendors,
about
intermitting
premises
and
or
open
metrics
data.
Some
of
them
are
already
working
on
that
and
we
always
try
to
it's
a
little
bit
in
your
face,
but
still
we
are
also
being
serious
about
that
one.
A
C
Thanks
for
sharing,
we
use
promise
use
in
production
and
I
found.
Sometimes
we
calculated
the
CPU
usage
right
is
no.
It's
not
correct,
because
I
found
the
route
KC's
primitives
collect
metrics
with
the
10
step.
Is
the
scrapped
scrapped
to
time
note
the
real
time
step,
but
if
the
couplet
has
a
delay
matrix
and
the
temperatures
will
correct
that
time,
the
third
high
is
not
not
correct.
That
will
cause
the
matrices
over
the
limit.
So
we
try
to
find
a
solution.
C
C
B
Yeah,
so
so
yeah,
so
there's
a
lot
of
indirection
there,
and
actually
the
best
thing
to
do
is
to
have
your
prometheus.
Instead
of
going
through
the
couplet
go
directly
to
the
node
exporter,
go
directly
to
see
advisor
and
that
way
the
the
timestamp
for
when
Prometheus
Connect
collects
the
data
from
the
target.
It's
getting
it
as
direct
as
possible
and
actually
that's
part
of
the
part
of
the
design
goal
of
previous.
Is
we
didn't
what
we
intentionally
don't
have
a
single
server
agent?
B
Whereas,
if
you,
if
you
grant,
if
you
create
a
granular
set
of
metrics,
where
you
talk
to
your
app
and
you
talk
to
the
couplet
and
you
talk
to
the
see
advisor,
you
talk
to
the
node
exporter,
each
of
those
happens
in
milliseconds.
So
it
allows
Prometheus
to
collect
that
data
and
really
get
it
fine-grained
and
much
more
accurate.
D
B
B
So
if
you
have
something
like
there's,
Prometheus
billed
info
contains
the
the
go
compiler
version,
the
get
commit
of
the
server
and
the
actual
Prometheus
version
string,
and
so
all
those
strings
are
all
as
labels
on
a
single
metric
and
the
value
of
that
metric
is
one
and
because
of
the
Prometheus
compression,
storing
one
over
and
over
and
over
again
in
a
database
seems
like
it
might
be
a
lot
of
overhead,
but
because
of
the
Prometheus
compression,
that's
only
20
bytes
of
storage.
For
what
every
half
an
hour
for.
B
It's
every
hundred
and
twenty,
but
20
bytes
for
120
samples.
So
if
you're
doing
15
seconds
it's
half
an
hour,
so
so
you're
you're
only
talking
like
a
few
hundred
bytes
to
store
that
one
over
and
over
again
for
an
entire
day.
So
the
the
overhead
there
is
much
smaller
and
we
store
all
those
strings
in
our
database,
and
so
this
actually
leads
to
a
bunch
of
really
interesting
use
cases.
B
For
example,
if
instead
of
storing
the
kernel
version
like
or
the
hardware
version
or
the
VM
type
in
as
a
label
on
every
single
metric,
we
put
them
into
these
info
labels
and
then
you
can
use
joins
and
prom
ql2.
If
you
want
to
correlate
based
on
kernel
version
or
vm
type
or
hardware
type
or
some
other
one
of
these
other
strings
or
versions
like
if
you
have.
B
I
actually
did
that
once
in
production,
where
we
had
a
JVM
app
and
we
some
of
the
JVM
s
on
the
cluster,
were
using
two
CPUs
and
some
of
them
were
using
one
CPU
and
it
was
the
same
binary.
It
was
the
same
JVM.
It
was
the
same
app
and
they're
all
running,
and
we
we
couldn't
figure
out
why
half
the
VMS
or
half
the
JVM
s
were
using
twice
as
much
CPU,
and
so
we
started
just
using
prom
ql
to
correlate
like
well
what,
if
it's,
the
hardware
type.
B
So
we
had
an
info
metric
that
had
the
CPU
type,
because
we
had
some
versions
of
these
xeon
cpus
and
we
had
some
versions
of
these
xeon
cpus,
no
correlation,
so
I
threw
in
I,
did
group
left
on
kernel,
name
and
so
I
brought
the
kernel
name
in
and
boom
perfect
correlation
that
the
older
kernel
version
made
the
JVM
twice
as
slow.
So
guess
what
we
have
to
upgrade
the
kernel
and
all
these
servers
just
happens.
So
that's
that's!
That's
the
power
of
prom
cool.
E
Okay,
thank
you
so
previously,
I
use
the
provisions
with
the
remotest
origin.
I
establish
a
link,
friend-friend
DB,
so
I
found
those
performances
not
at
courtesy
and
now
I'm
using
our
craft
now
to
query
the
primitives
and
as
a
problem
for
Patriots
refresh
the
slowly
I'm,
not
sure
why
so
this
is
a
puzzle.
Question.
B
Yeah
so
yeah,
so
Prometheus
is
a
very
fast
time
series
database
and
many
of
the
remote
storage
options
are
actually
slower
than
Prometheus
and
you're
better
off
just
using
Prometheus
instead
of
using
remote
storage,
which
is
why
we're
like
you
know
we're
we're
we're
experimenting
with
many
different
remote
storages.
So,
whichever
remote
storage
you're
using
which
one
did
you
say,
you
were
using
in
flux,
I,
don't
know
that
one.
A
E
A
B
And
Thanos
yeah
there
are
two
there's
two
different
projects:
Quartet
cortex
is
a
Prometheus
time
series
data
scalable
time
series,
data
base
and
Thanos
is
a
different
kind
of
scalable
time
series
database.
So
these
are.
These
are
the
two
that
I
like
the
most
but
yeah.
Many
of
that
many
of
the
remote
storage
options
are
interesting
but
are
actually
slower.
A
E
And
so
I'm
I
turn
back
to
suss,
local
storage,
so
I
think
a
local
storage
is
a
good
and
it
can
better
from
performance,
but
is
it
bring
and
that's
a
question
so
after
several
days
I
mean
one
man
sees
that
this
car
is
a
flow,
so
I'm
beans
I
wanted
to
delay
the
some
of
our
data.
It's
the
possible
for
the
lovers,
the
load
header.
You.
A
A
This
comes
with
performance
penalties,
so
it's
best
to
just
cut
off
and
just
drop
the
data
at
the
end,
and
basically
you
need
to
scale
your
your
storage
in
a
way
that,
if
you
let
let's
say
you
want
to
have
two
weeks
or
two
months
off
data,
then
you
just
need
to
do
some
calculations
maybe
give
give
some
extra
room
and
that's
your
storage
need
and
everything
which
is
all
your
just
drop.
That
is
a
command
line
flag
for
Prometheus
or
or.
D
A
F
Thank
You
Jimmy,
I've
reviewed
the
codex
and
the
saddles
and
but
I
think
all
those
projects
are
very
complicated
and
the
brewing
is
great
operation
burden.
Yes,
Fran
so
I'm
wondering
if
Francis
will
plan
to
support
local
long
term
durability
and,
most
importantly,
promise
this
town's
gambling,
because
if
we
have
long
term
durability
storage
we
have
one
year
or
two
year
date.
We
can't
kill
them
without
some
sampling.
So
I'm
wondering
if
from
this
203
have
a
plan
so.
B
So
weak,
if
you
have
a
bunch
of
data,
we
can
just
insert
actually
not
here,
but
we
can
just
add
a
whole
bunch
of
new
time
series
as
backfill
and
the
two
reasons
we
want
to
do
that
first
is
one
so
that
we
can
do
backfilled,
recording
rules.
So
actually
the
best
way
to
do
down
sampling
right
now
is
to
do
recording
rules
that
create
down
sample
data
and
then,
instead
of
querying
the
the
live
data,
because
that
Prometheus
2.0
time-series
database
is
totally
durable
all
on
its
own.
B
It's
actually
a
really
good,
durable
time
series
database,
but
it
doesn't
support
mutation.
It's
it's
a
right
once
and
so
using
recording
rules
is
a
good
way
to
do
down
sampling
because
you
can
court.
You
can
set
your
query
to
query
a
bunch
of
data
and
you
can
record
it
only
once
a
minutes,
because
in
Prometheus
2.0
we
allowed
her
recording
rule
group
intervals.
So
you
could,
if
you
want
to
create
a
once
every
one
minute
or
every
five
minute.
You
can
create
those
recording
rules
that
only
record
every
five
minutes.
B
Also
remains
for
a
long
time
yeah
and
it
just
takes
up
a
lot
of
space.
The
other
thing
you
can
do
is
you
can
use.
Federation
is
a
simpler
solution
for
on-prem,
so
you
can
create
your
recording
rules
and
then
and
then
use
a
federated
endpoint
and
have
another.
Have
one
Prometheus
server,
that's
dedicated
to
long
term
storage,
and
this
is
what
Ritchie
does
for
his
company
and
and
also
we
do
this
for
the
same
thing
at
git
lab.