►
From YouTube: Declarative Multi-Cluster Monitoring with Prometheus - Matthias Loibl, Loodse & Frederic Branczyk
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Declarative Multi-Cluster Monitoring with Prometheus - Matthias Loibl, Loodse & Frederic Branczyk, CoreOS (Beginner Skill Level)
Loodse and CoreOS run many different setups across multiple heterogeneous public and private clouds. In this context, they faced some challenges with deploying the monitoring infrastructure. Matthias and Frederic will show you how declarative monitoring helps their teams to work consistently in many different environments. They will show you how to apply these techniques to Prometheus, the Alertmanager and Grafana. Finally, they will give an outlook on what the community is going to improve in the distant future.
About Frederic
Frederic is an engineer at CoreOS contributing to Prometheus and Kubernetes to build state of the art modern infrastructure and monitoring tools. He discovered his interest in monitoring tools and distributed systems in his previous jobs, where he used machine learning to detect anomalies indicating intrusion attempts. He also worked on projects involving secrets management for distributed applications to build sane and stable infrastructure.
About Matthias
Matthias is a Software Engineer at Loodse. He loves working on Distributed Systems with Go, Docker, Kubernetes and Prometheus. In his free time, he contributes to numerous open source projects related to Prometheus and Drone.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
A
A
In
this
case,
we
have
a
request
counter,
so
you
atomically
incremented,
whenever
there
is
a
request
and
then
Prometheus
comes
and
scrapes
those
metrics
over
an
HTTP
server,
which
is
how
you
expose
those
metrics
to
Prometheus
and
scrapes
those
in
some
interval
by
default.
That's
happy
15
seconds
and
when
Prometheus
grabs
those
metrics.
A
It
then
writes
that
in
to
its
internal
time,
series
database-
and
in
this
case
then
we're
at
T
0
and
there
has
not
been
any
traffic
to
our
application,
so
it's
zero
and
then
we
actually
create
a
load
balancer
or
something
or
there
load,
balancer
picks
it
up,
and
user
traffic
actually
hits
our
application
and
our
request
counter
increases.
And
then
the
scrape
interval
has
passed.
A
It
just
goes
through
all
the
alert
definitions
that
you've
given
it
evaluates
them
and
if
it's
an
alerting
rule
and
it
triggers
it-
actually,
it
fires
it
against
the
alert
manager
and
yeah.
That's
that's
pretty
much
how
how
Prometheus
works
so
now.
I
already
mentioned
some
more
of
the
I
already
mentioned
what
a
target,
so
what
do
I?
Actually,
when
I,
when
I
say
a
target?
What
I'm?
What
do
I
mean
that
by
that?
Well
really
anything
that
has
metrics
endpoint
by
default.
A
Prometheus
wants
this
to
be
the
slash,
metrics
and
point
on
your
HTTP
server,
but
it
could
really
be
anything
it's
configurable
and
for
this
to
actually
be
picked
up
by
Prometheus,
maybe
it's
obvious,
but
it
needs
to
actually
be
discovered
and
Prometheus
has
a
number
of
mechanisms
to
do
this.
It
can
just
be
a
static
list
of
targets.
It
can
be
one
of
the
service
discovery
mechanisms
like
DNS
or
because
we're
at
coop
con
kubernetes
discovery,
and
it
has
all
of
these
built
in.
A
So
that's
really
neat
and
kubernetes
discovery
works
in
a
way
where
you
can
say
you
want
to
discover
pods
directly
or
you
want
to
discover
nodes
or
something
that's
actually
most
commonly
used
is
the
endpoints
discovery
so
because
in
within
Prometheus
or
within
kubernetes,
we
already
structure
our
applications
in
a
way
and
service
discovery.
Mechanisms
are
meant
to
reflect
how
you,
internally,
in
your
organization,
structure
your
applications.
A
So
this
is
a
very
common
way,
because
that's
that's
how
we
already
do
this
in
communities
and
thus
we
want
to
be
able
to
display
this
reflect
this
on
two-on-two,
Prometheus
and
what's
really
cool
about
the
built-in
discovery.
Mechanisms
specifically
with
kubernetes
is
that
we
can
automatically,
depending
of
our
life
cycle
within
kubernetes,
add,
remove,
update
our
targets
without
having
to
reconfigure
Prometheus
all
the
time,
because
in
kubernetes
all
of
these
things
are
super
dynamic.
They
can
happen
within
sub
second
time
intervals,
and
we
we
want
to
be
able
to
that.
A
Our
monitoring
always
picks
up
everything.
So
this
is
what
kubernetes
discovery
is
super
useful
for.
So
let's
look
at
a
more
more
concrete
example
of
this,
so
we
have
our
number
of
pods
and
I
think
this
case.
We
just
have
our
random
web
application
and,
as
so
often
it
is
backed
by
some
database,
and
the
way
we
do
this
in
criminales
is
that
we
then
have
a
set
of
labels
that
identify
our
pods
or
a
number
of
pods,
and
that
selector
we
use
in
a
service
object
and
the
way
a
service
object.
A
Actually
works
in
kubernetes
is
that
the
service
is
what
you
specify
right
with
the
selector
and
then
respective
to
that
service.
Aaa
endpoints
object
is
automatically
created
and
the
endpoints
object
actually
has
the
concrete
part
information
in
it,
and
this
is
actually
where
Prometheus
then
goes
ahead
and
grabs
this
information.
So
because
we
because
we
want
to
know
we
want
to
actually
do
white
box
monitoring
right.
We
want
to
go
to
our
targets
and
grab
the
metrics
from
our
targets.
A
We
need
to
know
how,
where
to
actually
find
that-
and
this
is
exactly
how
endpoints
discovery
works
in
kubernetes
and
then
for
some
additional
metadata.
We
can
also
go
and
refer
to
the
service
object,
because
we
know
most
endpoints
objects
are
associated
to
a
service
object
in
kubernetes.
So
that's
how
we
can
have
some
useful,
additional
meta
information.
A
When
we
have
such
a
such
a
setup,
it
turns
out
that
managing
a
configuration
for
your
entire
infrastructure
can
get
quite
large,
so
I'm
just
showing
skipping
through
a
couple
of
slides
of
configuration
here,
and
this
was
just
66
lines
out
of
613
line
configuration
and
we
used
to
actually
maintain
a
configuration,
that's
huge
and
it's
kind
of
crazy.
This
is
the
kind
of
thing
that
we
want
a
configuration
management
system
to
take
care
of
us
for
us,
so
we
thought
to
ourselves.
A
So
we
want
to
be
able
to
offer
Prometheus
as
sort
of
as
a
SAS
offering
to
the
rest
of
our
organization,
so
that
people
who
actually
care
about
maintaining
those
prometheus
servers
can
do
that.
But
do
it
independently
of
the
people
that
use
Prometheus
so
that
the
people
who
actually
just
want
to
use
Prometheus
to
monitor
their
applications.
They
can
just
say
this
is
where
to
find
my
application.
Please
monitor
so.
A
This
is,
this
is
what
we
did,
and
this
is
cool
because
you
don't
need
to
learn
the
Prometheus
configuration
paradigms,
which,
at
least
in
the
start
can
be
quite
complicated
to
get
into,
but
extremely
flexible,
because
Prometheus,
we
have
to
remember,
came
out
of
an
environment
that
was
not
strictly
built
for
kubernetes.
It's
extremely
fact
flexible.
We
can
do
pretty
much
anything
with
it,
but
that
also
means
that,
with
flexibility
there
comes
some
complexity,
but
specifically
the
Prometheus
operator
case.
A
We
know
we
are
going
to
Bria
be
running
in
kubernetes
and
we
are
going
to
be
monitoring
targets
in
kubernetes,
so
we
can
make
paradigms
that
are
that
people
within
communities
are
already
used
to
like
label
selectors
available
to
configure
their
monitoring
system
in
the
exact
same
way.
So
we
built
use
useful-
or
at
least
that's
what
we
think
abstractions
for
this
and
the
Prometheus
operator.
You
can
find
in
this
repository.
If
you
aren't
already
familiar
with
it,
and
because
we
did
this,
we
could
we
be.
A
We
could
encode
of
this
knowledge
that
we
as
developers
who
also
work
on
Prometheus
like
a
lot
of
us
or
actually
all
of
the
maintainer
of
the
Prometheus
operator,
our
core
developers
of
the
Prometheus
team.
So
we
can
encode
this
knowledge
that
we
have
of
running
Prometheus
in
various
environments,
mostly
kubernetes.
A
And
in
terms
of
this
talk
because
we're
talking
about
declarative
monitoring,
because
we
have
now
applied
all
of
these
concepts
that
kubernetes
has
which
are
declarative,
our
pods
are
a
declarative
definition
of
how
we
want
to
run
our
application.
Art
deployment
is
a
declarative
definition
of
how
many
pods
there
should
be,
and
now
we
have
a
declarative
configuration
that
is
cribben,
andes
native
and
that
describes
how
we
want
to
monitor
our
applications
and
alerting
and
kubernetes
and
prometheus
is
also
declarative.
A
You
say
the
alert
name,
the
expression
and
Prometheus
just
goes
ahead,
regularly
evaluates
that
and
when
it
triggers,
we
send
an
alert,
and
really
this
boils
down
to
just
declarative.
Everything,
and
the
only
thing
that
is
now
left
is
the
clarity
of
core
fauna
and
something
that
the
graph
on
our
folks
have
been
working
on,
which
is
really
cool,
and
actually
we
used
to
write
this
really
really
ugly
hack,
where
we
ran
a
sidecar
next
to
grow
fauna
and
it
deleted
the
entire
graph
on
a
database
and
then
reprovision
stuff
from
files.
A
A
So
now,
with
Griffin
r5,
we
can
just
drop
a
couple
of
dashboard
definitions
next
to
our
graph
Hana
instance,
and
it
will
just
provision
itself
itself
from
these
from
these
files
at
startup.
So
that's
really
awesome
and
something
that
has
just
started.
This
is
a
an
initiative
that
Tom
and
I
have
been
working
on,
and
maybe
you
were
just
in
his
talk.
I
know,
Matias
was
where
we
want
to
be
able
to
share,
alerting
rules
and
dashboards
more
and
still
have
this
declarative
nature.
A
B
So
we
run
multiple
data
centers
and
what
you
don't
want
to
go
into
each
data
center
and
have
to
do
that
manually.
So
we
want
to
have
something
that
is
the
declarative,
the
configure
rule
and
we
can
just
upload
configuration
to
each
cluster
and
then,
in
the
end,
that's
going
to
be
applied
and
yeah.
That's
it
so
we
can
apply
the
same
configuration
always
slight
differences
to
each
cluster.
B
B
We
also
want
to
monitor
these
kubernetes
cluster
classes
that
are
running
in
inside
the
name
space
with
a
new
Prometheus
to
have
that,
like
as
close
to
the
cluster,
that
we
are
monitoring
inside
there
as
well
and
then
in
the
end,
everything
should
bubble
up
to
the
company
Prometheus
so
that
we
as
I
said,
have
the
global
view.
So
how
do
we
actually
deploy
the
Prometheus
in
into
the
namespace?
We
are
running
a
controller
for
our
cluster,
so
inside
our
code,
we
basically
just
say
hey
into
this
namespace
that
was
just
created.
B
Please
also
deploy
the
CRVs.
So
there's
a
Prometheus
object
as
an
alert
manager
object
and
there
are
the
service
monitors
which
are
the
declarative
way,
as
Frederick
told
you
earlier
to
to
find
the
targets
in
in
the
namespace,
and
we
do
all
of
that
with
an
controller
and
yeah
within
that
namespace.
The
alerts
for
that
specific
class
are
being
shipped
to
the
alert
manager
for
in
which
is
running
in
the
data
center.
B
B
So
we
don't
want
to
grab
everything
that
is
in
there,
but
only
the
things
that
I
use
for
one
level
above
so
this
is
one
example
of
a
service
monitor
that
we
are
using
and
that
per
data
center
Prometheus
and,
as
you
can
see,
there's
this
path,
federal
rate,
yeah
key
value
pair-
and
this
just
has
Prometheus
to
go
to
the
Prometheus
running
in
inside
the
namespace.
To
then
in
this
example,
get
metrics
which
are
which
are
starting
with
a
machine
controller.
B
So
we
only
want
in
this
example
to
to
fetch
those
metrics
and
then
like
at
the
very
top.
We
are
only
like
getting
very,
very
specific
data
with
a
high
retention.
We
want
to
solve
that
very
for
a
very
long
time
and
yeah,
it's
mostly
for
dashboarding,
so
we
want
to
have
like
a
long
term
view
of
everything
that
is
going
on,
and
that's
I
mean
it's
it's
good
for
SLA,
SSL
laws
and
so
on
yeah.
B
A
B
B
Okay,
so
in
this
Prometheus,
which
is
running
in
the
namespace,
we
have
this
metric
and,
as
you
can
see,
we
are
currently
running
three
notes
in
our
cluster
and
that's
basically
just
to
show
that
this
metric
is
available
and
yeah.
This
is
like
my
private
cluster
ID,
so
we
are
going
to
see
that
across
all
different
Prometheus's.
So
this
is
one
level
above.
This
is
running
in
the
data
center
and
we
are
again
like
scraping
every
service
that
is
running
in
there,
so
yeah
I,
like
the
couplets
and
so
on.
B
And
then
again,
we
are
also
scraping
the
Prometheus
that
are
running
inside
the
each
namespace.
So
this
is
the
Federation
approach
right
there
and
this
my
my
Prometheus
for
my
class
at
that
we
looked
into
earlier
so
again,
I
can
specify
the
namespace.
Please
give
me
the
number
of
notes
that
we
are
using
in
in
this
cluster
and
yeah.
We
get
the
same
metric
again,
so
this
is
now
the
Prometheus
which
is
running
at
the
company
level.
B
So
at
the
very
top-
and
this
one
is
federating
into
the
different
data
centers
and
this
example
we
have
to
yeah
and
this
only
fetches,
the
Machine
controller
and
some
other
very
specific
metrics
that
we
actually
care
about,
and
this
is
like
the
park
where
the
sticky
sessions
come
in,
and
it's
not
that
really
nice,
but
it
kind
of
works
but
yeah.
There
are
better
solutions.
A
B
As
you
can
see,
the
retention
on
that
prometheus
is
only
one
day
and,
as
I
said
before,
on
that
very
top
level,
we
want
to
have
a
very
long
retention
to
to
see
what's
going
on
for
the
last
half
year
or
something
like
that,
and
this
actually
were
a
new
project
from
the
core
maintain
as
fabian
is
involved.
He
is
running
a
project
called
tonneau,
snow
and
Frederic
is
going
to
tell
us
a
bit
more
about
that
in
the
next
slides.
B
So
there's
like
an
overview
of
everything
that
is
going
on
inside
our
cluster
or
between
the
different
data
centers
and,
as
you
can
see,
even
though
in
the
Prometheus
itself,
we
only
have
a
retention
of
one
day
with
tunnels.
We
we
are
able
to
get
a
retention
of
four
weeks
now,
which
is
kind
of
nice
and
there's
like
we
could
do
that
for
month.
B
A
A
For
quite
some
time-
and
probably
some
of
you
are
asking
themselves
as
well
and
Thomas
is
just
one
of
the
solutions
out
there,
but
we
think
that,
particularly
for
this
use
case,
it
would
make
sense
to
use
something
like
tano's
just
to
mention
the
other
solutions.
There's
also
we've
cortex,
which
is
more
of
more
more
built
to
be
like
a
SAS
offering,
so
that
some
company
runs
it,
and
we
think
that
tano's
really
works
really
well
for
like
an
on-premise
solution.
A
So
the
goals
that
tano's
wants
to
solve
and
just
to
make
sure
fabian
worked
on
this
together
with
someone
from
improbable
and
this
project
actually
is
under
the
improbable
organization
under
github.
So
there
are
the
two
people
that
primarily
created
this
this
project.
So
sadly
fabian
is
not
in
this
talk
to
represent
this,
but
we
think
it's
really
awesome
and
we're
thankful
for
them
for
building
this
and
really
that
the
things
that
we
want
to
solve
with
this
is
a
long-term
storage.
We
want
to
be
able
to
view
our
entire
infrastructure
globally
at
real-time.
A
We
want
to
have
down
sampling
so
that
when
we
do
queries
over
a
year,
we
want
to
get
a
good
idea
of
of
the
actual
data,
but
we
don't
need
to
actually
query
15-second
granny
granularity,
so
something
like
one
hour
averages
or
whatever.
Actually
it's
good
enough
and
then
when
we
zoom
in,
we
actually
get
higher
resolution
and
tannaz
does
exactly
this
and
how
this
is
built.
Maybe
you've
seen
fav
Ian's
keynote.
He
explained
how
the
new
Prometheus
storage
that
we
that
was
built
for
Prometheus
2.0.
A
That
is
exactly
what
is
the
core
of
this
project.
Basically,
what
Prometheus
uses
on
disk
now
is
just
mapped
into
a
remote,
remote
storage
and
really
how
this
works
is
that
we
run
vanilla,
kubernetes,
vanilla,
prometheus,
always
mixes,
though,
but
apparently
other
people
have
problems
with
this
as
well.
So
you
run
Prometheus
unmodified,
no
fork,
no,
nothing!
A
Now,
if
you
know
how
the
Prometheus
storage
works,
you
know
that
the
actually
persistent
data
on
disk,
like
the
immutable
chunks,
are
only
written
at
a
by
default
to
our
interval,
and
so,
if
you,
if
we
just
had
a
setup
like
this,
we
could
only
scribe
read
all
of
the
data
that
is
at
least
two
hours
old.
So,
in
addition
to
that,
the
courier
can
also
go
out
and
query
all
the
life
Prometheus
servers
too
and
then
merge
all
of
this.
A
This
data
together
and
then
have
the
global
view
over
long-term
storage
and
active
servers,
and
this
is
really
neat
because
in
the
past
we
wanted
to
create
Prometheus
servers
that
have
really
large
disks
so
that
we
could
have
long-term
storage
on
like
normal,
hard
disk
or
large
SSDs,
or
something
like
that.
But
it's
not
really
necessary.
Here
anymore.
We
can
have
small
local
disks
and
just
ship
off
into
long-term
storage
and
have
whatever
is
needed
for
alerting
purposes
local
to
the
Prometheus
server.
A
Because
in
this
model
will
we
keep
all
the
reliability
in
the
alerting
mechanisms
that
we
know
and
love
from
Prometheus,
which
is
that
the
alerting
happens
exactly
where
we
ingest
our
information
and
then
is
sent
to
alert
manager
like
we
used
to
and
the
problem
that
matthias
mentioned
about
the
j
problem.
So
whenever
we
accessed
Prometheus
in
this
model
and
we
had
Prometheus
H
a
because
we
we
want
to
be
reliable,
the
problem
was
that
we
never
knew
which
Prometheus
should
we
be
requesting.
A
And
maybe
if
we
were
flapping
because
Prometheus
may
not
be
exactly
synchronized
when
you
do
scrapes.
So,
even
though
they
are
configured
to
do
scrapes
every
15
seconds,
they
might
not
do
this
at
the
exact
same
time,
so
cow
for
example,
or
really
any
other
metric,
may
not
be
exactly
synchronized
for
alerting
purposes.
This
is
fine
because
they're
good
and
similar
enough,
so
that
they
produce
the
same
alerts,
but
in
per
dashboarding.
A
This
is
strange
because
every
time
when
you
refresh
you
get
a
different
graph,
so
this
is
something
that
tonneaus
also
solves,
and
there
are
some
heuristics
that
it
applies
in
order
to
merge
this
data.
But
then
you
can
also
say
that
you
want
to
disable
this
in
case,
you
want
to
say,
I
want
to
see
exactly
the
data
from
this
replica
and
I'm
happy
to
say
that
we
have
recently
implemented
all
necessities
for
Thanos
to
be
run
with
the
Prometheus
operator.
A
A
A
B
A
B
So
we
want
to
do
that
on
the
per
namespace
level,
just
because
there
might
be
the
case
or
there's
actually
the
case
of
running
hundreds
of
clusters
inside
one
kubernetes
cluster
again,
and
we
just
want
to
have
like
the
very
raw
data
right
next
to
the
cluster
itself
and
only
like
federal
rate.
The
data
that
is
really
important.
What
level
of.
A
In
terms
of
the
the
API
objects
that
we
that
you
have
here,
you
have
the
Prometheus
object,
which
has
a
selector
for
the
service
monitors.
It
should
select,
and
it
can
do
this
within
the
names
within
the
same
namespace
as
the
Prometheus
object,
as
well
as
a
namespace
selector.
So
you
can
select
the
namespaces
from
which
you
want
to
select
service
monitors
from.
A
So
the
point
of
why
we
do
this
is
so
that
users
who
want
to
monitor
their
application
can
set
can
specify
how
to
grab
those
metrics
but
don't
need
to
care
about
running
the
Prometheus
server
itself.
So
that
can
be
something
that
an
infrastructure
team
offers
to
the
rest
of
their
organization.
Then
they
just
need
to
say
these
are
the.
This
is
how
you
monitor
my
thing.
I.
A
Okay,
so
the
question
is:
have
we
thought
of
writing
a
graph
on
ax
operator?
Basically,
yes,
we
have
thought
of
that,
and
actually
one
of
the
first
issues
that
we
had
on
the
Prometheus
operator
was
to
have
graph
an
ax
tightly
integrated
into
the
Prometheus
operator.
We
decided
against
this
because,
with
the
new
model
of
how
graph
Ana
works,
it's
actually
super
simple
to
do
this
without
an
operator.
There's,
no,
really,
there's
no
stateful
application
that
we're
managing
here.
That
needs
extra
care.
The
only
the
only
way.
A
A
A
So
the
question
is:
every
time
we
change
the
configuration,
the
sidecar
goes
out
and
reloads
it
or
grabs
the.
So
the
the
prometheus
operator
already
makes
sure
that
only
actual
changes
get
propagated
down
to
the
Prometheus.
So
only
if
you
actually
check
if
someone
if
someone
updates
the
service,
monitor
object,
for
example,
and
doesn't
actually
doesn't
actually
change
anything
or
doesn't
change
something
that
actually
influences
the
Prometheus
object,
then
nothing
happens.
Is
that
was
that.
Does
that
answer
your
question
or
okay?
A
A
So
a
time-series
in
prometheus
is
uniquely
identified
by
its
set
of
labels
right
and
then
Thanos.
A
requirement
for
every
Prometheus
server
is
that
every
Prometheus
server
has
a
set
of
external
labels
configured
and
actually
created
a
pull
request
to
make
tonneaus
fail
entirely
like
like
error
out
of
it.
The
external
labels
it
once
wants
to
register
itself
with
are
not
unique,
so
yeah.
There's
that
there's
definitely
a
unique
time
series
at
this
point.