►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: Prometheus – Richard "RichiH" Hartmann, SpaceNet & Ben "SuperQ" Kochie, GitLab
Join Prometheus maintainers to learn about the design, project goals and history of Prometheus. We will cover basic concepts of Prometheus to enable you to get an understanding of what all the hype is about. We will start at what time series are and what makes them different, look at our non-hierarchical data structure, our exposition format, and tie it all together with our query language, PromQL? As the end, you will have a solid understanding of Prometheus, enabling you to start using it in production.
To learn more click here: https://sched.co/HMhy
A
B
A
I'm
going
to
hand
over
for
Ben
for
the
first
half,
oh
just
for
the
structure
of
this
talk,
we
decided
to
keep
some
concerns
and
best
practices
about
operations
and
observability
in
here,
even
though
it
is
permissive
specific
course
chances
are,
if
you're
new
to
prometheus,
you
hugely
benefit
from
the
observability
stuff
as
well.
Alright,.
C
C
They
had
used
Boardman,
which
is
Google's
one
of
Google's
internal
monitoring
systems,
and
it
was
really
well
put
together
to
do
container
monitoring
and-
and
it's
a
little
different
in
it,
from
something
like
Nagios
for
monitoring.
Is
it
started
with
this
time
series
database
and
it
uses
that
uses
a
time
series
database
as
its
core
Prometheus
is
very
simple.
It
uses
only
metrics
that
are
numeric.
C
It
started
with
the
instrumentation,
so
actually
the
first
thing
that
was
developed
when
they
were
developing
corinthians
was
an
instrumentation
library
so
that
they
could
start
collecting
the
NIR.
They
could
start
producing
the
data
in
order
for
it
to
be
used
for
monitoring
it's
not
an
event
logging
system,
it's
not
a
replacement
for
elasticsearch
or
Splunk,
or
those
kind
of
things.
We
consider
that
a
separate
a
separate
system
is,
and
so
Prometheus
is
like
a
very
specialized
single
tool
to
make
it
really
like
powerful
without
needing
to
handle
every
use
case
and.
C
Okay,
so
the
main
selling
point
is
it's:
it's
very
simple:
it
has
very
dynamic
built-in
service
discovery.
It
has
a
very
powerful
query
language
for
extracting
data
from
the
metrics.
It's
super
simple
to
operate.
It's
super
efficient.
You
can
monitor
many
many
many
thousands
of
processes
with
with
one
Prometheus
server.
It's
pull
based.
It
uses
it.
It
knows
your
network
topology
and
collects
the
data.
You
can
do
blackbox
monitoring
by
configuring,
a
blackbox
exporter.
It
can
do
white
box
monitoring,
so
it
can.
C
It
can
go
directly
into
your
application
and
pull
data
out
so
that
you
know
you
can
expose
things
that
are
are
more
granular
than
you
could
get
out
of
your
event
logs.
So
one
of
the
things
that
I
talk
about
when
developing
is,
if
you
have
things
that
you
might
want
to
have
from
debugging,
you
can
have
metrics
for
those
without
having
to
actually
have
the
cost
of
debug
level
logs.
C
It
has
some
stability
guarantees
were
we
don't
have
a
very
good
security
model
yet,
but
that's
a
work
in
progress,
so
one
of
the
things
that
we
talked
about
is
time
series.
So
what
is
a
time
series,
because
this
is
kind
of
fundamental
to
how
Prometheus
works?
It's
a
a
regularly
collected
data
point
with
a
timestamp
over
a
long
period
of
time,
which
could
be
hours
or
days
or
months
or
years
and
in
Prometheus
we'd
like
to
collect
really
raw
data.
C
So
if
you
have
an
event
in
your
system,
you
get
a
request
to
an
API.
Your
API
generates
an
error.
You
make
a
database
request,
whatever
you
count,
those
and
just
we
store
the
raw
counts
in
Prometheus,
but
some
things
aren't
really
countable
how
many
things
are
in
a
queue
or
the
temperature
of
a
sensor.
Those
things
are
what
we
call
gages
is
they
they
go
up
and
down,
and
you
can
only
you
you
have
to
sample
them
instead
of
with
a
counter
you
sample.
C
When
you
sample
a
counter,
you
know
it
was
one
at
time
a
and
10
at
time.
B
Prometheus
is
super
efficient.
We
just
did
some
had
some
new
benchmarking
recently,
where
the
important
key
metric
for
us
was
actually
not
how
fast
we
could
ingest,
because
it
turns
out
that
a
million
samples
per
second
is
more
than
plenty
for
just
about
every
situation
we've
run
across
what
we
really
wanted
is
we
we
want
to
get
the
the
cost
to
ingest
down,
so
we
got
it.
It's
now
about
200,000
samples
per
second
per
CPU.
C
So
if
you
don't
want
to
waste
resources
on
your
monitoring
Prometheus's,
this
is
a
super
super
efficient
there.
Actually,
this
is
a
typo,
it's
actually
not
1.3
bits
per
sample,
it's
actually
1.3
bytes
per
sample
and
that's,
of
course,
data
dependent.
So
the
Prometheus
compression
can
actually
do
a
little
better.
C
How
do
you
get
data
into
Prometheus?
Well,
we
have
a
very
simple
text:
representation
and
basically
the
metric
name
represents
some
line
of
code,
so
HTTP
requests
total
represents
the
request
Handler
in
your
application,
and
then
you
can
tag
it.
So
some
labels
come
from
the
code
like
the
status
code
or
the
method,
and
some
labels
come
from
your
discovery
like
the
environment.
So
one
of
the
key
things
about
prometheus
is
we
have
two
two
basic
kinds
of
labels.
We
have
labels
that
come
from
the
target.
C
We
have
labels
that
come
from
discovery
so
that
you
can.
The
application
developer
doesn't
have
to
know
about
the
environment
that
the
code
is
running
in
because
you
could
take
a
piece
of
code
from
kubernetes
or
other
some
other
random
piece
of
software,
and
if
your
company
has
different
hierarchical
concepts,
they'll
you
can
apply
those
labels
differently
than
another
company.
Would
so
one
of
the
interesting
things
about
prometheus
is
we
went
with
a
a
vector
style,
programming,
language
or
a
query
language
versus
SQL,
because
it
makes
some
things
much
much
simpler.
C
So
when
you
do
a
very
simple
query,
like
average
average
temperature
by
city,
that's
pretty
simple-
it's
also
pretty
easy
to
do
in
in
SQL,
but
if
you
want
to
take
and
two
different
metrics
and
do
math
between
them
in
SQL,
that
gets
super
complicated,
super
fast
and
there's
even
worse.
Examples
than
this,
where
a
single
line
of
prom
ql
would
be
hundreds
of
lines
of
SQL
for
visualization.
We
start
out
with
our
own
dashboard
system,
but
most
of
us
working
on
Prometheus
are
we're
all
back-end
developers.
C
We
don't
have
a
lot
of
UI
people,
so
we've
said
we'll
just
delete
that
code
and
let
gravano
be
our
our
front-end.
So
most
people
who
are
using
prometheus
are
using
Gryphon,
it's
beautiful.
It's
nice
I
like
it
a
lot
and
then
now
I'm
gonna
hand
off
to
Richard
to
talk
about
the
second
half.
Second,
half
yeah.
A
One
of
these
single
most
things
to
realize,
and
actually
you
have
at
the
front
of
your
head,
while
while
designing
systems
and
working
on
systems
is
the
concept
of
toil,
which
is
basically
manual,
work
which
you
repeat
endlessly
and
which
doesn't
generate
lasting
benefit.
Also,
it's
scales
roughly
linearly
with
your
workload
or
the
amount
of
users
or
whatever.
A
Why
is
this
such
important
concept?
Basically,
while
you're
always
putting
out
fires
course,
there's
an
outage
here
or
a
need
to
provision
new
service
there,
bla
bla,
bla,
bla
bla,
you
don't
have
time
to
actually
sit
down
and
think
about
the
whole
system
to
actually
engineer
a
good
system,
basically
by
getting
rid
of
that
kind
of
work
which
does
not
provide
any
lasting
value.
You
have
time
to
focus
on
stuff
which
actually
benefits
you
and
your
company
or
your
team
or
whatever.
A
So
it's
really
important
when
you
try
to
to
do
anything
any
change
at
your
company,
but
especially
when
you
want
to
reduce
work,
you
kind
of
have
to
explain
that
you
increase
work
for
a
short
time
until
you
reduce
the
total
amount
of
work,
so
having
immediate
benefits,
which
you
can't
wake
in
xocai
people
look
here.
This
is
good
stuff
that
what's
good,
that's
what
gets
you
buy-in,
then
we
have
the
concept
of
sleep
most
humans
like
to
sleep,
and
how
do
you?
A
Basically,
it
depends
a
little
bit
or
a
lot
on
on
agreeing
within
your
company
how
you
want
to
run
things
and
if
something
is
not
actionable,
like
you,
don't
know
what
to
do
with
an
alert,
then
it
cannot
be
an
alert
by
definition,
you
should
not
be
vocal,
it
should
not
be
escalated
unless
you
know
what
to
do
and
how
to
fix
and
how
to
approach
it.
If
it's
not
urgent,
it
should
also
not
be
an
alert.
A
It
should
be
persistent
in
your
ticket
system
or
whatever,
and
you
can
do
it
during
business
hours
next
business
day.
There
is
no
need
to
be
woken
at
3:00
in
the
morning
for
something
which
could
easily
be
handled
next
morning.
Also,
if
you
have
good
users,
user
prediction
or
even
auto
scaling,
that
also
means
you
don't
get
welcomed
by
random
stuff,
just
going
1%
over
whatever
your
baseline
is
and
again
try
to
do
this
during
business
hours.
A
Then
something
really
important
if
there
is
no
by
the
people
who
build
it
playbook
documentation
how
to
run
a
thing.
It
should
never
ever
go
into
production
chorus.
Basically,
if
you
do
any
any
other
way,
what
happens
is
the
on-call
engineers
are
the
ones
who
then
get
to
document
the
system
they
didn't
build,
which
is
not
a
good
thing
to
have,
and
also
if
there
is
no
clear
concept
which
is
agreed
upon
by
management.
What
the
actual
targets
of
availability
and
of
latency
and
of
all
these
things
is
for
that
product.
A
It
should
also
not
go
into
production
course.
If
you
don't
know
what
you're
actually
running
you
shouldn't
be
running
it
or
you
should
think
more
about
how
to
run
it
also
really
important.
While,
while
having
these
discussions
in
your
company,
every
single
role
will
think
a
lot
differently
about
what
you
actually
do
and
some
people
care
deeply
about
certain
aspects
of
what
you
do
and
none
at
all
about
others,
and
this
is
really
important
to
keep
in
mind
the
managers.
A
They
will
think
about
revenue
and
that's
about
it
and
about
process
actually
execution,
but
they
don't
actually
do
the
process.
So
that's
the
architects.
They
care
about
good
process
definition
and
about
a
good,
clean
design,
which
hopefully
also
gets
implemented.
Then
others
would
like
to
have
just
the
high-level
overview
of
yes:
things
are
fine.
Teammates,
obviously
want
their
team
to
be
motivated
and
to
have
energy
while
working
and
not
be
suddenly
demotivated
and
the
actual
operator.
But
this
is
us
being
beer
200.
B
A
A
So
when
talking
about
how
to
implement
something
new,
hopefully
prometheus
at
your
company,
you
should
you
should
really
try
to
to
tell
people
what
they
want
to
hear
and
what
they
need
to
hear,
instead
of
just
telling
everyone
the
same
story,
but
also
I'll,
obviously,
never
ever
lie
course:
that'll
come
back
to
you
and
it'll
not
be
pretty
so
post
mortems
are
also
huge
in
importance.
Course.
Mistakes
will
happen.
No
system
is
perfect.
Something
will
break
so
you
have
to
design
the
system
in
a
way
that
braking
is
fine.
A
You
have
to
learn
from
those
mistakes
course.
If
you
just
keep
making
the
same
mistakes,
you
will
lose
a
ton
of
users,
and
also
this
is
like
the
textbook
definition
of
toil,
but
there's
a
very
important
aspect.
If
people
fear
attribution
when
they
make
a
mistake,
they
will
not
be
honest
about
what
mistakes
they
made.
A
So
unless
you
have
a
culture
at
your
company
in
your
team,
in
your
whatever,
where,
where
it's
actually
okay
to
say,
okay
I
made
a
mistake,
and
this
is
how
this
mistake
folds
into
the
whole
big
picture,
and
everyone
else
is
also
honest
about
their
mistakes,
about
their
actions
when
they
did
what
and
what
they
didn't
do
and
such
then
you
can
actually
write
those
blame
free
post-mortems.
Where
you
write
explicitly
someone
that
acts
you
don't
have
to
name
them,
but
you
write
engineer
of
database
team
drop
database.
A
That
was
not
a
good
idea
and
everything
was
gone.
You
write
this
down
curse.
Then
people
can
read
it
later
and
they
can
learn
from
that
and
you
can
build
institutional
knowledge
in
your
company
based
on
those
past
mistakes
and
not
repeat
those
mistakes.
But
it's
really
really
important
to
build
this
trust
course.
A
If
someone
assumes
that,
if
they
document
that
it
did
something
wrong
that
they
only
pick
daily
be
kicked
out
of
the
company
or
something
they
will
never
ever
document
this
properly,
so
this
trust
internally
is
hugely
then
leveraging
this
one
pistol
system
again,
hopefully
prometheus.
If
you
have
one
combined
system
as
opposed
to
a
ton
of
different
systems,
you
can
actually
correlate
that
data.
You
can
take
different
aspects
of
your
whole
state
of
the
system
and
you
can.
A
You
can
actually
make
deductions
if
you
have
both
your
services
and
your
data
center
in
the
same
monitoring
system,
you
can
actually
graph
the
power
usage
against
how
many
users
you
have
on
that
system.
Myself,
I
started
in
networking
and
we
had
weird
issues
with
a
line
where
the
out
a
quick
change
in
outside
temperature
up
or
down,
took
down
a
certain
long-haul
line
or
a
certain
dark
fiber
which
took
ages
to
find
out.
A
We
didn't
have
premises,
but
even
that
such
things,
you
could,
in
theory,
just
correlate
and
just
put
into
context.
If
you
have
the
data
and
the
same
in
the
same
system
and
there
is
tons
and
tons
and
tons
of
other
things
and
as
you
now
have
this
good
way
to
do
math
to
do
actual
math
on
your
metrics,
you
have
the
ideal
starting
point
to
jump
into
your
logs
into
your
traces
into
whatever
course.
Metrics
are
usually
the
start
of
something
course
they're
easy
to
aggregate
and
easy
to
work
with.
A
So
you
can
go
from
there
and
you
can
jump
into
other
parts
of
your
whole
observability
system
to
get
the
full
picture.
Sometimes
metrics
are
already
enough,
but
sometimes
you
just
need
to
look
somewhere
else,
and
this
is
almost
always
the
best
starting
point.
And
when
you
have
this
one
source
of
truth,
which
might
be
one
system
or
several
systems,
then
you
have
everyone
agree
on
that
state.
You
have
the
same
technical
overview
for
your
current
state.
You
have
the
same
data
in
your
drill
down.
C
E
C
There
are
there:
are
some
companies
out
there
working
on
external
correlation
systems
that
pull
the
data
from
Prometheus?
So
the
idea
that
we
had
with
Corinthians
is
it's
the
data
collector
and
it's
the
first
level
of
monitoring
and
we
wanted
to
keep
it
simple
and
robust
and
not
require
tons
and
tons
of
CPU
time
so
that
a
single
Prometheus
server
is
very,
very
a
very,
very
small
piece
of
the
overall
system.
So
people
are
building
external
correlators
that
will
use
the
Prometheus
data
and
then
spend
the
extra
CPU
time
outside
of
the
act.
D
A
Several
so
for
just
normal
committees,
you
can
just
run
to
as
there
and
they
can
do.
Basically,
they
both
do
exactly
the
same.
So
if
one
fails
you
have
the
same,
but
you
also
have
a
remote
read/write
API,
where
you
send
data
to
two
other
in
jester's,
there's
two,
which
are
basically
modified,
modified
promises
which
are
tennis
and
cortex
and
there's
tons
of
others.
A
We
have
a
total
of
12
different
ones
listed
in
the
meantime,
and
those
can
actually
do
a
JN
per
system
and
such
so
currently,
you
should
probably
be
looking
at
cortex
if
you
need
more
availability
than
then
standard
parameters
can
give
you.
But
again,
this
is
basically
modified
from
easiest.
So
it
comes
from
Prometheus
team
members,
yeah.
C
A
We
got
into
scaling
issues
towards
the
end,
but
2.0
really
is
changed
all
that
so
just
long
term
retention,
if
you
take
good
care
of
your
machines,
already
works
within
parameters
and
also
I
mean,
if
you
do
the
pairing
you
basically
average
a
it
kind
of
depends
on
what
you
want
to
have
here
or
your
availability
for,
if
you
want
to
have
it
for
alerts.
H.A
like
just
having
a
pair
is
already
way
enough.
A
C
So
CentOS
is
a
different
way
of
doing
prometheus
scaling
where
basically,
instead
of
having
a
separate
database
cluster,
it
takes
the
data
and
puts
it
into
a
cloud
storage.
So
if
you
have
any
kind
of
cloud
storage
like
s3
or
Google,
compute
or
even
Seth,
you
can,
you
can
just
Dennis
will
take
your
data
and
put
it
into
the
external
storage
and.
A
A
Okay,
so
the
question
was
how
to
reduce
memory.
Usage
of
permeate
fermitas
is
designed
to
use
as
much
memory
as
it
gets,
but
it
doesn't
need
that
memory.
So
what
it
does
is
it
just
keeps
everything
in
REM
to
be
quicker
when
it
answers
to
you,
but
you
come
on
lines
like
where
I
can
tell
it
the
target
value
of
how
much
RAM
it
should.
C
So
so
changed,
but
it's
know
it's
gone.
There's
there's
no
merit
so
Prometheus
use
the
Prometheus
2.0
uses
as
much
memory
as
is
needed
for
storing
the
ingestion
and
handling
queries,
so
we're
currently
working
on
improving
our
internal
metrics,
so
that
you
can
tell
is
my
memory
being
used
by
ingestion
or
is
my
memory
being
used
by
queries?
So
one
of
the
things
that
we've
I
discovered
was
I
had
a
bunch
of
Gryphon
on
queries
that
were
written
and
those
graph
on
a
queries
used
a
lot
of
memory.
C
So
we
went
through
and
created
recording
rules
to
optimize
to
reduce
the
amount
of
memory
for
display
and
that
that
significantly
reduced
the
amount
of
memory.
So
you
can
use
recording
rules
to
pre
aggregate
data
for
display
and
then
the
other
thing
is
you
might
just
have
to
it
might
be.
You
have
to
shard.
So
if
you
have
one
know,
one
single
Prometheus
server
may
not
be
enough.
C
Once
you
get
a
certain
amount
of
ingestion,
because
you
need
a
significant
amount
of
memory
to
buffer
all
of
that
ingestion
and
it's
it's
just
kind
of
it's
one
of
those
things
where
Prometheus
does
need
quite
a
lot
of
memory,
but
not
a
lot
of
CPU.
We
are
working
on
a
bunch
of
optimizations,
so
we've
been
going
through
over
the
last
few
releases
and
optimizing
our
code
to
reduce
the
overall
memory
usage
for
things
like
compactions.
So
look
in
the
next
few
releases,
the
memory
usage
should
start
to
come
down.
C
G
C
Yeah,
no,
it's
it's
actually
totally
fine
and
normal
to
run
Prometheus
inside
a
kubernetes
cluster.
It's
totally
fine
to
run
it
outside
the
coop
it
it
really
doesn't
matter
it
mostly
it's.
It
depends
on
how
much
durability
you
want
and
how
durable
your
storage
is
inside
kubernetes.
If
you've
got
a
good.
C
If
you've
got
a
good,
durable
storage
layer
for
your
kubernetes
cluster,
then
it's
totally
fine
to
run
Prometheus
inside
the
cluster
and
actually
one
of
the
things
like
the
way
we
deploy
it
or
the
way
we
recommend
deploying
it
is
actually
deploying
a
one
Prometheus
per
kubernetes
namespace.
So
if
you've
got
multiple
applications,
you
can
deploy
a
Prometheus
per
application,
and
we
can.
We
can
maybe
talk
about
that
more
in
the
deep
dive,
but.
A
There's
one
important
consideration:
if
you
want
to
use
primitives
for
monitoring
the
availability
of
that
cluster,
you
have
to
have
something
sitting
outside
of
that
cluster,
looking
into
it
so,
depending
on
how
important
it
is
to
you
at
least
what
we
do,
we
have
physical
machines
which
are
dedicated
to
be
used
as
prometheus
instances
and
those
monitor
other
Prometheus
servers
to
make
sure
that
we
know
if
a
certain
server
goes
down
course.
If
that
server
goes
down
course,
the
cluster
is
going
away.
You
won't
have
any
alerting
on
the
cluster
anymore.
A
A
This
is
a
religious
question.
Quite
sorry,
so
the
common
path
currently
would
be
to
take
good
care
of
your
premises.
Servers
ever
since
2.0.
We
can
do
snapshots
so
take
good
backups
of
that
server
and
then
to
use
either
or
cortex
or
Athena's.
There's
people
putting
stuff
into
other
databases
as
well,
they're
also
happy,
but
if
you're
asking
someone
from
premises
team,
if
the
answer
it
probably
be
either
cortex
or
tennis,
yeah.