►
From YouTube: Prometheus Deep Dive – Julius Volz, Prometheus, & Max Inden, Red Hat (Intermediate Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Prometheus Deep Dive – Julius Volz, Prometheus, & Max Inden, Red Hat (Intermediate Skill Level)
This deep dive will include a panel of Prometheus developers answering your questions around Prometheus.
About Julius
Julius co-founded Prometheus and lead the project to success at SoundCloud and beyond. He now focuses on growing the Prometheus community and is the main organizer of PromCon, the first conference around Prometheus. In a previous life, Julius was a Site Reliability Engineer at Google.
About Max
Max is a software developer at Red Hat and member of the upstream Prometheus project, working both on Prometheus and Kubernetes. Previously hacking on data quality analysis, he decided to stop suppressing his interest for distributed systems at scale and joined CoreOS (now Red Hat).
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
should
we
start?
Let's
do
it
hi
everyone
welcome
to
the
Prometheus
deep
dive
session,
which
is
we
you
know
just
a
group
of
Prometheus
maintain
errs
on
stage
answering
your
questions.
We
don't
have
any
other
program.
So
if
that's
not
what
you
were
looking
for,
now's
the
chance
to
leave,
but
basically
the
idea
was
really
just
you
know.
We
have
so
many
Prometheus
people
at
this
conference,
it's
probably
outside
of
prom
con,
which
is
the
actual
prometheus
conference,
the
biggest
gathering
of
all
of
us
I've
ever
seen.
A
So
that's
really
cool
and
I
just
thought.
Maybe
that's
a
good
opportunity
for
people
to
ask
random
questions
about
the
project's
direction,
the
community
about
technical
things
to
us
and
then
hopefully,
we'll
find
someone
who's
the
most
qualified
to
answer
that.
But
if
you
have
to
ask
questions,
please
try
to
ask
questions
that
are
not
like.
A
A
Just
as
an
intro
I'm
Julius
I'm,
one
of
the
initial
two
creators
of
Prometheus
back
then
nowadays
I
freelance,
completely
independently
helping
companies
use
Prometheus.
Then
we
have
matte
layer
here.
He
works
for
digital
ocean
Tom,
Wilkie
Griffin,
alabs,
Richard,
Hartman
space
net,
one
of
the
oldest
German
internet
I
his
pieces,
always
what
you
say
right:
yes,
Brian
Brazil,
robust
reception,
Prometheus
consulting
company.
We
have
Mac's
inland
now,
Red
Hat,
formerly
core
OS
same
for
Frederick
branch,
shake.
Let
me
Fabian
who
just
freshly
joined
Google.
He
was
on
stage
yesterday.
B
B
A
At
least
you
know,
and
so
we
don't
really
have
the
one
Prometheus
company
or
so,
which
is
a
good
thing
and
terms
of
Independence,
but
also
means
we
don't
have
as
many
full-time
staff
and
community
members
working
on
it
getting
paid
by
companies.
You
know:
Google
has
many
many
companies,
for
example
many
many
people
working
on
kubernetes
and
so
does
Red
Hat
and
other
companies
so
yeah.
That's
that's
also,
sometimes
a
challenge
but
yeah.
That's
that's
kind
of
what
sets
us
apart.
They're
a
bit
cool
and
I.
D
A
Not
so
so,
okay
depends
what
what
you
mean
was
difficult.
So
in
prometheus
itself
we
decided
not
to
integrate
long
term
storage
because
we
want
to
keep
Prometheus
relatively
simple
single
node
kind
of
setups.
We
do
have
you're,
probably
aware
of
the
the
remote
right
integrations
and
several
backends
existing
for
that
are
those
too
difficult
to
use
currently
or
are
they
not
working
well
enough
or.
A
D
There's
there's
a
bunch
of
systems
except
in
this
long-term
storage,
remote
API,
one
of
them's
cortex,
which
is
I
work
on
and
the
other
one
stano's
which,
having
talked
about
a
sec
cortex,
is
too
hard
for
you
to
run
on
your
own,
like
so
generally,
it's
mainly
aimed
at
SAS
vendors
who
want
to
sell
you
storage,
which
is
why
I
work
for
labs.
But
it's
you
know
it's
kind
of
cool,
just.
E
F
Now
I
think,
like
one
one
thing
we
see
was
cortex
antennas
that
it's
very
different
approaches
solving
the
same
problem
in
a
way
but
catering
to
different
use
cases
so
like.
If
we
had
some
canonical
answer
in
Prometheus,
probably
someone
good
would
get
lost
along
the
way
and
tennis,
for
example,
is
a
different
approach.
It
doesn't
use
even
more
api's.
Instead,
you
have
a
sidecar
that
you
basically
then
use
to
cluster
up
together.
F
Multiple
Prometheus
servers
and
then
you
have
sort
of
horizontal
is
capable
instead
to
scrub
you
layer
and
then
can
do
long-term
storage
based
on
object,
storage,
which
is
entirely
based
on
the
new
storage
from
what
we
have
as
of
promises
to.
So
that's
a
very
different
approach,
and
that
probably
works
much
better
for
on
Prem
and
then
the
question
is
like
what
is
what
is
prometheus
right?
The
query
language
is
kind
of
you
can
have
those
elsewhere.
Cortex
has
it
has
it
and
it's
kind
of
the
same
system
you
access
right.
F
If
you
have
the
same
query
language,
you
don't
probably
don't
care
too
much
more
user
perspective
like
where
this
data
is
coming
from,
and
then
this
is
the
operation
perspective
yeah.
It
sounds
relatively
simple
and
it's
not
the
same,
but
it's
relatively
close.
So
if
we
had
sort
of
some
promises
built
in
tool,
if
it
also
have
some
additional
complexities
in
terms
of
operations,
so
whether
it
has
a
different
name
or
not,
that's
probably
not
paying
too
much
of
around
the
end.
Yeah.
A
B
Another
reason
that
we,
we
specifically
don't
have
clustering
in
Prometheus
as
we
wanted
to
make
the
the
ability
to
deploy
it
and
the
ability
to
adopt
it
as
easy
as
possible,
and
if
we
put
clustering
in
the
Prometheus,
it
would
make
it
much
harder
for
new
developers
to
adopt
it
and
start
using
it
and
just
getting
getting
it
tried
out
to
see
if
it
would
actually
fit
for
them.
So
by
making
clustering
and
external
storage
an
optional
feature
with
Thanos
or
cortex
or
whatever
other
database.
You
want
to
send
your
data
to.
B
It,
allows
the
the
users
with
more
complex
problems
that
probably
already
have
a
better
idea
on
how
they
want
to
do
it,
whereas
if
you're,
just
a
small
group-
and
you
only
have
one
Prometheus-
and
you
only
really
need
two
weeks
of
data-
it's
fine
or
to
do
it.
That
way,
you
and
even
Prometheus
itself
is
totally
fine
storing
six
months
or
a
year
of
data.
Internally
and
that's
you
know
at
gitlab,
we
started
out
simple.
We
started
out
with
one
Prometheus
server
and
we
just
store
six
months
of
data
in
there.
B
G
A
G
A
H
D
E
I
E
Up
and
running
is
a
personal
project
and
not
endorsed
by
the
fruitiest
team.
That's
so
we
always
have
to
be
careful,
because
we
are,
as
Ben
mentioned,
a
project
without
any
one
company
backing
ish.
We
asked
we
care
about
who
we
don't
do
and
don't
endorse
just
so
we
can
keep
ourselves
open,
even
if
it
would
be
nice
right
business,
but
you
know
III
do
believe
in
keeping
trying
to
keep
searching
state
reasonably.
F
I
B
Yeah
then
I
mostly
work
on
exporters
and
integrations,
and
some
community
stuff
around
integrations
so
like
trying
to
encourage
projects
to
adopt
previous
metrics
format,
not
for
Prometheus,
but
just
because
it's
a
good
way
to
integrate
with
other
systems.
So
I
I
tell
people
I,
don't
care
if
you
implement
Prometheus
metrics
for
Prometheus,
because
somebody
could
use
the
Prometheus
metrics
within
flux,
DB
or
any
of
our
competitors
I.
Just
we
want
to
get
a
good
standard
data
format
for
for
getting
data
out
of
apps.
A
Maybe
me
and
last
one
so
in
the
past.
You
so
also
touch
pretty
much
almost
on
everything
in
prometheus
in
some
way
organized
the
last
two
years
of
prom
con
as
well
the
Prometheus
conference.
Now
that
I'm
freelancing
around
it
I'm
doing
more
like
what
companies
need,
which
sometimes
is
upstream
work,
but
more
often
than
not
is
just
something
internal,
so
yeah,
if
you
need
any.
If
you
want
to
sponsor
upstream
development,
talk
to
me.
A
B
I
work
for
gate,
lab
and
I
do
some
of
the
management
of
get
labs,
alerting.
Of
course,
I
teach
the
production
engineering
team
how
to
make
their
own
alerts,
but
we
do
everything
we
just
have
a
git
repo
and
we
check
in
our
changes
into
the
git
repo
and
it's
auto
deployed
into
our
production
environment.
So
it's
just
check
it
in
wait.
Wait
for
it
to
get
pushed
important.
A
B
A
And
it's
maybe
important
to
say
there
is
no
API
in
premises
itself
to
update
a
file
on
disk,
because
we
want
to
have
one
path
of
configuration
to
enter
Prometheus
and
not
like
Prometheus
modifies
a
file
on
disk,
but
then
it
might
also
get
overwritten
again
by
contact
management
system
or
so
alright.
So
that's
why
everything
comes
through
the
file
and
if
you
want
to
make
changes
to
that,
you
have
to
build
your
own
API
that
changes
that
file.
A
C
D
With
Frederick
on
a
new
way
of
kind
of
expressing
alerts
to
make
them
agnostic
to
how
you
label
things
in
your
infrastructure,
which
hopefully
should
make
them
more
redistributable
and
collaborative,
and
so
with
Frederick
and
I
have
published
a
set
of
like
humanities
alerts
and
dashboards
and
recording
rules
and
we're
calling
the
mix-ins
and
we're
talking
about
it
tomorrow.
At
2
o'clock,.
B
A
A
It's
the
Prometheus
conference
will
have
maintainer
there,
but
also
we
want
to
sort
of
the
talks
from
users
of
Prometheus
to
share
their
experience,
how
you're
using
Prometheus
what
you've
learned
best
practices,
the
CFP
is
still
open,
I
think
until
May
14
prom,
condyle
you'll
find
it
there
submit
talks
buy
tickets.
The
registration
is
open
as
well
at
the
moment,
and
it's
in
Munich
again,
which
is
a
lovely
city
to
visit
in
August
and.
E
Just
in
terms
of
talks,
we
aren't
just
interested
in
us
lecturing
everyone.
We
would
like
user
stories,
people
using
it
in
different
ways
like
an
good
diversity
of
talks
and
attendees.
That's
like
last
year
only
like
eternity
to
talk
swear
by
us
and
you
know
we
want
to
keep
it
that
way,
whereas
dominating
is
actually
a
community
event
organized
by
both
Richey,
mostly.
A
E
E
I
Okay,
so
in
terms
of
being
able
to
horizontally
scale
I,
don't
think
this
is
actually
very
kubernetes
specific,
but
there
is
a
possibility
to
shard
Prometheus
with
a
with
the
hash
mod
function.
That's
like
a
relabeling
rule
in
and
Prometheus,
and
then
that
way
you
can
distribute
the
targets
multiple
instances
of
Prometheus.
If
that's
really
what
you
what
you
need,
but
what
you
could
also
do
in
terms
of
because
you're
you
said
that
you're
scraping
15
clusters.
I
You
could
also
adopt
Thanos,
for
example,
where
you
then
have
a
Prometheus
server
per
cluster
I.
Don't
know
if
that
actually
makes
sense
depends
a
bit
on
how
large
each
individual
cluster
is,
but
with
Thanos.
You
could
then
globally
query
all
of
those
and
make
it
look
like
one
but
decrease
the
sample
memory.
Overhead.
B
Yeah
so
part
of
the
part
of
the
original
like
where
Prometheus
came
from.
It
was
designed
with
the
idea
that
a
previous
server
should
be
small
and
it
should
be
attached
to
the
the
thing
that
it's
monitoring.
So
if
you
have
15
kubernetes
clusters,
you
really
want
to
have
15
Prometheus
servers,
one
for
each
cluster
and
then
you
can
either
you
can
use
one
of
any
of
our
techniques,
whether
it's
cortex
Thanos
Federation,
to
to
pull
in
a
global
view.
But
then
you
you
get
the
date.
B
B
B
A
A
Yeah,
obviously
it's
trade-offs
like.
If
you
have
data
spread
over
many
Prometheus
servers,
then
you
cannot
create
craft.
One
prom,
ql,
query
that
touches
all
of
the
data,
but
yeah
the
classical
way
of
using
promises
would
be
really
like
one
Prometheus
per
cluster.
You
know,
and
then
you
have
a
global
one,
that
federates
and
it
doesn't
have
all
the
details
anymore,
but
it
would
have
the
global
view.
Just
the
aggregations
of
each
cluster.
A
If
you're
heading
run
issues,
then
your
queries
and
for
your
time
series,
especially
like
the
number
of
current
concurrently
active
time
series
that
your
primitives
is
managing,
scraping,
that's
probably
incurring
the
most
memory
cost.
So
theoretically,
so,
for
example,
you
could
say
you're
not
so
interested
in
some
of
the
metrics
that
either
you
drop
entire
types
of
targets
or
put
them
in
a
different,
primitive
server
or,
as
a
last
resort.
A
A
A
A
K
L
L
F
F
B
Yeah
so
so
we
forget,
forget,
lab
we
paid
you
loose
to
develop
an
actual
upgrade
tool
that
will
take
the
old
time
series
database
and
create
the
new
time
series
database.
But
of
course
you
know
it
was
manie's
a
bit
of
memory
and
it
takes
a
long
time
and
currently
the
way
it
works
is
you
have
to
do
it
all
offline,
so
during
the
offline
window,
you're
not
collecting
any
metrics.
So
really,
my
my
answer
to
this
is
just
deploy
a
new
Prometheus
server
wait
30
days
and
throw
away
the
old
one
yeah.
L
I
Question
so,
just
last
week
we
released
a
version
of
the
Prometheus
operator
that
has
all
the
requirements
for
tannaz
merged,
so
I
think
Donna's
needs
external
labels
on
a
Prometheus
server
in
order
to
discover
its
identity
and
potentially
do
some
optimizations
for
queries.
So
that's
that's
down,
and
you
can
also
now
inject
arbitrary
containers
next
to
your
Prometheus
server
through
the
Prometheus
operator,
which
means
you
can
inject
the
ethanol,
sidecar
and
necessary
to
do
this,
and
we
actually
have
some
example
manifests
in
the
Prometheus
operator
repository
as
well.
A
Cool
okay,
yeah:
let's
keep
maybe
two
questions
first,
so
first
one
is
water.
Stannis
was
awesome.
Second,
one
was:
are
we
planning
on
adding
more
ml
kind
of
machine
learning
type
functions,
functionality
to
promises
itself
by
the
way?
If
you
could
use
the
remote
API
is
currently,
but
you
can
also
query
Prometheus
and
get
it
out
that
way
right.
F
So
Thanos
essentially
tries
to
solve
your
problems
that
you
might
encounter
when
you
first
scale
out
prometheus
first.
Is
you
may
end
up
as
much
of
a
permissive
service?
I
know
you
want
to
ask
query
across
the
data
of
all
of
them
and
what
you
have
to
do
currently,
essentially,
is
you
have
to
use
Federation's?
You
have
to
decide
a
certain
subset
of
metrics
that
isn't
federated
in
that
server.
F
Finally,
like
long
term
storage,
so
Thanos
essentially
tries
to
soften
all
the
simple
passes
you
deploy
a
side
car
next
to
Prometheus,
which
then
will
sort
of
cluster
together
side
cars.
And
then
you
have
a
horizontally
scalable,
stateless
crew
layer
which
essentially
just
implements
he
promises,
API
and
prompt
you,
and
this
corrida.
Then
clusters
who
gathers
all
these
side,
cars
and
sort
of
fans
out
queries
to
all
these
servers,
and
then
you
have
basically
one
entry
point
for
liquor.
Is
that
can
hit
all
servers
at
once
and
it's
trying
to
be
smart
about
it.
F
So
if
you're
sort
of
label
selection
your
career
and
can
exclude
certain
Premiership
servers,
it
will
just
not
go
to
them,
and
so
it's
sort
of
functionality
routing
in
a
way
it's
open
source,
and
it
also
detects
when
to
promises.
Servers
are
sort
of
HMS,
are
correcting
the
same
data
and
never
then
apply
some
heuristics
to
merge
this
data
back
together.
So.
A
M
A
F
Good
thing
is,
you
can
kind
of
do
a
gradual
rollout,
so
the
simple
passes
you
just
use
a
sidecar
and
the
COBE
layer.
That's
already
only
so,
it
cannot
really
break
anything,
just
state,
the
stuff
that
you
deploy
essentially
and
then,
if
you
want
to
sort
of
extent
this
and
have
long
term
storage,
you
can
have
the
sidecars
update
the
database
files
as
I
written
to
disk
to
an
object
storage
which
can
be
cooked,
so
it
as
three
or
something,
and
that's
so
yeah
whatever
and
that's
kind
of
like
a
simple
backup
strategy.
F
So
it
does
like
really
sort
of
direct
and
range
queries
against
the
object,
storage
and
consolidates
these
range
requests
together
and
so
that
you
basically
almost
get
like
real-time
response
times,
even
though
your
data's
like
really
cheaply
stored
in
object,
storage
and
lastly,
what
we
added
as
well
was
down
sampling
and
simply
because
the
computation
gets
at
some
point.
Really
expensive,
if
you
have
like
really
long
data
sort
of
date,
ranges
that
you're
accruing
and
doesn't
if
it
was
just
a
crispy.
F
E
The
first
thing
we
need
to
be
careful
about
is
the
word
anomaly
detection,
because
it
is
largely
a
marketing
term
you're
using
it
in
the
way
it's
actual
anomaly
detection,
some
other
products
might
not
it'll
just
mean.
Oh
we've,
an
alert
if
the
JVM
is
spending
a
lot
of
times
in
garbage
collection
and
which
is,
it
is
technically
an
under
normal,
a
level,
so
people
really
mean
so
just
took
that
for
that
one.
E
So
the
thing
is
that
using
any
form
of
LnL
and
it's
my
experience
on
engine
bar
toy
examples
is
extremely
difficult
and
someone
actually
works.
Google
and,
like
one
of
machine
learning
team,
said
yeah,
it
takes
18
months
pay,
wait
to
come
up
with
a
model
that
you
can
use
for
something.
So
if
you
spend
18
months,
you
can
come
up
with
a
model
for
a
single
nurse
and
spend
your
3
or
4
engineers.
E
Doing
that
simple
static,
stress
rules,
work
really
well,
you
know,
and
just
that
kind
of
trade-off
was
also
the
question
of
inside
parameters,
how
heavily
droite
to
get
into
stats.
It's
my
personal
belief
that
you
know
Holt.
Winters
is
already
slightly
over
the
line,
that's
largely
because
you
don't
actually
have
hold
winters.
You've
got
double
exponential
rather
than
triple
so
there's
a
limit
of
how
much
CPU
Baretta
manage
and
how
much
history
you
need
like
read.
He's
only
nominally
has
three
months
of
data.
E
Is
that
enough
to
do
this
all
and
we
all
have
two
CPU
to
keep
things
reliable,
but
you
can
do
that
externally.
There
are,
however,
I
know
that
fresh
tracks
are
investigating
DML
stuff
for
Prometheus
and
integrating
it
there.
So
they'll
have
some
I,
don't
know:
where
does
commercial
open
source
option
there
at
some
point
and
what
I
would
see
it
as
generally?
This
is
something
you
do
outside
with
Prometheus
and
then
use
that
to
inform
thresholds,
and
then
you
can
feed
the
data
back
in
via
and
read
or
something
if
you
want.
B
So
the
question
is:
is
how
do
you
manage
the
resource
usage
of
these?
These
Prometheus
servers
in
space,
1:30
I've
heard
her
name
space
Prometheus
servers.
That's
a
good
question.
We
don't
actually
have
a
very
good
plan
for
this.
Yet
so
it's
just
basically
like
Prometheus
the
Prometheus
2.0
server
model.
B
Is
it
uses
what
it
needs
to
scrape,
so
you
can
kind
of,
like
you
can
kind
of
guess,
based
on
how
many
metrics,
like
there's,
cut,
there's
kind
of
a
rule
of
thumb
of
like
8
kilobytes
of
memory
per
per
metric
that
it's
collecting
simultaneously
in
memory.
So
you
so
you
can
kind
of
get
like
how
much
data
you're
collecting
informs
your
your
scale.
B
B
Yeah
or
just
leave
it
unlimited
and
and
set
a
set
of
set
of
razor
in
kubernetes.
You
just
set
a
request,
but
not
a
limit,
because
you
don't
want
to
own
the
you
you
don't
you
want
to
avoid
ooming
the
Prometheus
circles,
you'll
corrupt,
your
data
or
you
or
you
shouldn't.
You
won't
corrupted
eya,
but
you'll
lose
data.
A
B
Open
you
know,
you
know
once
when
someone's
people
start
the
basically
like
when
we
see
somebody
who's
contributing
quite
a
lot
and
they
seem
interested.
We
have
a.
We
have
a
process
in
our
our
our
governance
to
to
invite
them
to
become
a
team
member
and
then
they're
invited
to
join
the
meeting.
I
think
it.
A
Would
be
cool
in
general
to
have
everything
open,
I
think
at
the
moment
the
one
blocker
there's
that
we
sometimes
discuss
like
things
like
sponsorship
for
a
conference,
or
you
know
things
that
shouldn't
details,
that
that
shouldn't
leak
out
to
the
entire
world
I
like
the
way
prometheus
kubernetes
is
doing
it,
but
they
also
have
like
professional
resources
behind
it,
which
we
can't
quite
yet
have.
But
I
think
this
is
an
area
where
we
should
really
improve.
Yeah
I
think
we
have
to
wrap
it
up.
A
A
A
A
G
Question
was
basically,
if
there's
any
plans
or
any
support
for
integrating
with
different
monitoring
solutions,
something
we
saw
said
a
few
companies
actually
started
supporting
prometheus
stuff.
This
is
kind
of
a
hot
political
topic,
because,
if
you're,
actually
building
and
selling
a
product
and
all
of
a
sudden
you're
plastering
a
different
name
all
across
your
own
product,
this
might
not
come
across
too
well.
G
So
what
we
actually
did
is
there's
an
effort
called
open
metrics
which
actually
splits
out
this
standard
into
its
own
thing
to
actually
get
new
to
a
vendor
support
it's
in
the
planning
that
we
also
have
an
RFC.
So
you
can
just
point
your
vendor
at
RSC,
1,
2
3,
and
please
support
this,
and
once
they
accept
a
tender
and
don't
check
the
RFC's
they're
legally
obliged
to
to
support
from
uses
and.
F
A
I
just
say
like
we
have
a
ton
of
ways
which,
in
which
we
do
integrate
with
other
monitoring
service
and
in
the
right,
get
getting
metrics
out
of
existing
sentient
systems
into
Prometheus,
getting
Prometheus
metrics
by
writing
our
nog,
your
Sun
Prometheus
metrics,
sending
from
easiest
metrics
to
a
different
long-term
storage
like
we
integrate
pretty
much
in
any
way.
You
can
imagine
because
all
these
points
integration
points
are
open.