►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Adopting Prometheus the Hard Way - Tim Simmons, DigitalOcean
Deploying Prometheus is simple. Successfully adopting Prometheus as part of your company’s Observability story is not. In this talk you will learn how to scale the adoption of Prometheus, how to train your organization to know and love Prometheus, and the importance of fostering an Observability culture. If you are interested in building a great Observability culture at your company, this talk is for you! An extended outline of this talk is: -How DigitalOcean uses Prometheus successfully -The subtle art of scaling Prometheus and tradeoffs to consider -How to teach people to use Prometheus correctly -How to integrate Prometheus with Kubernetes and other software you use -How to utilize Prometheus to measure everything you do -What to do when you find Prometheus’ limits and how to push past them -Prometheus’ friends that you’ll also love -The value of your new Observability culture
To learn more: https://sched.co/GrXX
A
A
My
primary
job
I
do
is
caring
about
metrics
and
for
basically
all
of
our
internal
and
external
products.
My
team
runs
centralized
logging,
metrics
tracing
and
exceptions
in
404.
For
all
of
do
you
can
follow
me
on
Twitter.
If
you
want
I,
wouldn't
recommend
it,
though
cuz
that's
kind
of
a
general
stuff,
you're
gonna
get
so
a
few
goals
that
I
have
for
this
talk.
I
want
you
to
walk
away
with
an
understanding.
What
prometheus
is,
if
you
don't
have
that
already
I
want
you
to
understand
the
value
of
a
healthy
observability
culture.
A
Again,
if
you
don't
already
how
Prometheus
and
Friends
can
can
help
you
achieve
that,
and
maybe
some
of
the
nuances
of
scaling
Prometheus
and
its
adoption
so
really
quick
kind
of
blazing
through
an
intro
on
prometheus.
If
you
don't
kind
of
know,
it's
a
time
series
database,
which
means
it
stores
lots
of
series
of
time
series
data.
That's
just
data
points
that
have
a
value
at
a
time
at
which
they
were
observed.
Often
called
samples.
Prometheus
provides
a
very
rich
query
language
for
this
time.
A
Series
data,
so
you
can
do
a
lot
of
kind
of
cool
stuff
at
you
know,
applying
mathematical
functions
to
data
and
that
kind
of
thing,
metrics
and
Prometheus
have
labels.
So
this
allows
you
to
define
different
dimensions
for
foreign
metric
and
then
you
can
filter,
manipulate
and
kind
of
aggregate
data
based
on
those
dimensions.
Prometheus
is
primarily
used
for
gathering
machine
and
application
metrics.
So
you
can
imagine
how
this
might
be
useful.
If
you
have.
You
know
a
web
app
for
example,
and
you
want
it
to
have
I,
don't
know
page
view
counters
for
each.
A
Metrics
in
Prometheus
are
collected
using
HTTP
polls
from
endpoints
that
are
actually
exposed
by
applications,
sometimes
called
exporters,
so
pull
based
metrics
are
kind
of
good
because
it
simplifies
the
kind
of
metrics
process
for
instrumentation
at
your
application.
Basically,
you
are
incrementing
counters
and
doing
these
kinds
of
things
directly
in
your
and
then
somewhere
you
open
this,
this
port
and
then
Prometheus
goes
and
finds
it.
You
don't
end
up
with
kind
of
complicated
pipelines,
we're
pushing
metrics
to
a
central
source.
A
You
know
my
those
metrics
might
get
dropped
or
mangled
or
I,
don't
know
any
number
of
terrible
things.
Because
of
this
kind
of
standard
format,
metrics
exporters
exist
for
basically
everything.
So
if
you're
running
a
lot
of
open
source
infrastructure,
you
can
kind
of
get
metrics
for
all
of
your
things,
for
free
with
you
know,
or
at
least
with
very
little
effort
and
then
creating
metrics
for
your
applications
that
are
custom
is
possible.
A
A
Maintenance
is
less
exciting.
I!
Guess
it's!
It's
not!
You
know
it's
not
so
visible,
it's
not
so
new
and
important,
but
I
think
it's
actually
kind
of
more
important
than
you
know
doing
the
new
thing,
because
well
you
you
can't
keep
doing
new
things.
If
you,
you
know
your
old
stuff
kind
of
breaks
and
dies
like
it,
it's
just
a
very
important
kind
of
thing
for
people
to
do.
You
can
make
it
a
lot
more
exciting
by
calling
it
revenue
projection
protection,
which
is
a
cool
tweet.
A
I
saw
like
two
days
ago
and
I
think
successful
revenue
protection
means
that
you
can
measure
the
performance
of
your
system
debug
it
when
that
performance
is
poor
and
support
the
customers
of
that
system,
whether
they're
paying
customers
or
not.
Right
it
doesn't
matter.
If
you
haven't
some
sort
of
internal
system-
you're,
probably
you
know,
transitively
supporting
you
know.
External
customers
and
observability
tools
are
literally
built
for
this.
A
For
the
thing
right,
the
the
idea
is
to
kind
of
keep
the
lights
on
long
enough,
so
that
people
can
see
to
build
the
next
kind
of
big
thing.
I
guess
a
great
way
to
encourage
revenue.
Protection
is
by
having
a
healthy,
observability
culture,
and
this
is
roughly
what
I
think
that
looks
like
I
think
you
should
always
know
kind
of.
What's
going
on,
I'm
gonna
go
through
kind
of
a
few
outcomes
of
what
this
kind
of
attitude
and
philosophy
you
know,
hopefully
bring
so
that
we
can
understand
how
that
manifests
itself.
A
So,
hey
dashboards
are
good.
A
big
surprise
here
in
an
incident
like
clarity,
is
kind
of
awesome
being
able
to
see
what's
wrong
on
one
page.
Hopefully-
and
you
know,
have
a
faster
diagnosis
and
resolution
loop
means
happier,
customers
happier,
leaders,
you
know
less
kind
of
revenue.
Loss
and
dashboards
mean
that
more
people
can
kind
of
understand
the
situation
because
it
can
be
visualised.
You
know,
leaders
and
support
teams
can
have
that
understanding
of
when
something
is
working
better
if
they
saw
where
it
was
broken
on
on
a
dashboard.
A
It's
also
just
a
really
good
way
to
make
new
engineers
productive
members
of
a
team.
A
good
dashboard
can
kind
of
help.
You
visualize
an
architecture
and
the
flow
of
data
through
a
complex
system,
and
you
know
diagnosing
issues
again
for
for
a
new
engineer,
seeing
big
big
bad
thing
on
a
dashboard
means
like
okay,
I,
can
kind
of
understand
this.
That's
really
cool
and
powerful
right,
because
holding
the
architecture
and
debugging
skills
and
the
heads
of
senior
engineers
really
kind
of
sucks.
A
You
can
take
your
dashboards
to
the
next
level
by
adding
things
like
annotations
and
explanations
on
on
the
graphs
and
what
they
mean
and
linking
to
kind
of
the
next
step
in
the
process.
Be
it
like
a
you
know,
some
Doc's
or
play
books.
You
know,
cuz
people
are
really
good
at
pattern
recognition,
so
you
can
see
on
a
graph
where
something
might
look
kind
of
odd.
A
Now
it's
time
for
a
recurring
segment
in
this
talk
called
Friends
of
Prometheus
talked
about
Gravano,
because
it's
really
good
and
helps
you
create
beautiful
dashboards
and
arranging
them
is
a
dream
and
doing
things
like
arranging
or
arranging.
Graphs
in
web
interfaces
usually
is
bad,
but
in
Griffon
it's
not
so
thanks
core
fauna,
the
Prometheus
data
source
plug-in,
is
really
really
good.
There's
a
lot
of
just
really
other
cool
features
and
overall
I
think
it's
the
best
kind
of
single
pane
of
glass
for
all
of
your
observability
tools,
because
it's
not
just
Prometheus.
A
If
you're,
not
refining,
you
definitely
should
be
hey.
Did
you
know,
there's
a
lot
of
value
in
knowing
what
things
are
broken
like?
This
is
another
just
real
obvious
hot,
take
coming
out
here,
but
detecting
and
alerting
on
failures,
reliably,
I
think
it's
actually
harder
than
you
think.
Alerting
tools
often
try
to
exercise
kind
of
bad
admin
interfaces,
and
you
know
tools
to
introspect
state
and
then
maybe
tie
those
things
together.
A
Prometheus
exporters
can
kind
of
unify
all
these
sources
of
information.
You
can
push
the
bad
admin
tool
poking
down
to
some.
You
know
well
defined.
You
know,
exporter,
that
does
that
thing
and
exports
the
best
kind
of
metrics
that
you
can
get
out
of
there
and
because
alerts
are
just
prom.
Ql
queries
they're,
really
easy
to
kind
of
define
and
tweak.
Once
you
do
have
those
metrics
as
an
engineering
team,
you
can
save
your
sleep.
A
Your
sleep
schedules
by
having
you
know,
good
alerts
with
playbooks,
so
a
good
alert
kind
of
only
fires
when
something
is
actually
actually
broken
and
then
it
is
actionable,
meaning
something
can
actually
be
done
to
fix
this
and
then
hopefully
it
has
some
context
about
what
is
actually
happening.
What
is
broken,
what
this
alert
firing
means
you
know.
Oh
customers
are
in
pain.
The
website.
A
What's
going
on
and
fix
it
lets
you
handoff
the
pager
between
more
people,
which
gets
more
people
up
to
speed
on
a
system
which
is
always
good
and
enable
to
put
you,
you
can
put
your
manager
on
call
which
is
great,
because
hopefully
they
will
give
you
the
time
you
need
to
fix
something
if
they
get
paged
at
3
o'clock
in
the
morning.
Having
ubiquitous
alerts
inside
an
organization
means
that
you
can
kind
of
build
a
hive
mind
around
solving
larger
issues.
A
So
if
eight
teams
alerts
all
go
off
at
once,
you
know
when
the
common
thing
is
whatever
that
can
kind
of
help
you
get
to
a
root
cause
faster.
Another
awesome
friend
of
prometheus
is
alert
manager,
so
alert
manager
is
a
separate
application.
That
often
runs
you
know,
kind
of
int
and
with
Prometheus
the
way
the
Prometheus
does
alerts
as
it
evaluates
those
queries
that
are
alerts
on
occasions
and
then,
when
they
kind
of
have
something
alert
able
they
send.
A
Those
alerts
to
alert
manager,
alert
manager
can
deduplicate
alerts
from
many
Prometheus
instances
evaluating
the
same
alerts
and
then
group
them
together
and
make
sure
to
send
one
good
notification
to
slack
or
page
or
duty
or
email
or
whatever
alert
manager
also
has
a
high
availability
mode,
which
is
really
nice,
because
you
can
send
alerts
from
many
Prometheus
instances
to
many
different
alert
managers
and
only
ever
get
one
notification,
and
if
you
know
you
have
some
sort
of
outage
where
you
know
your
only
thing
that
is
alerting
goes
down.
That's
not
a
problem.
A
If
you
have
many
many
of
them,
so
you
won't
miss
any
alerts.
This
is
kind
of
critical
and
a
major
incident,
because
if
you've
lost
a
data
center
and
all
of
your
alerting
is
there,
you
might
not
know
until
you
get
a
really
angry
phone
call,
just
in
general
kind
of
developing
software
with
an
observability
culture
in
mind.
You
know
some
really
cool
stuff.
Hats
ends
up
happening
so
with
experience.
You
will
end
up
building
kind
of
better
systems
that
are
easier
to
revenue,
protect
because
you're
visualizing
how
supporting
and
debugging
that's.
A
That
system
will
look
like
because
kind
of
what
you've
known
in
the
past
you'll
end
up
with
clearer
code,
because
you're
gonna
make
it
obvious
where
those
critical
measurements
end
up
happening.
So
you
know
metrics
can
kind
of
be
a
shortcut
to
say
like
oh.
This
is
where
this
is,
where
the
important
thing
happens
done.
A
If
you
aren't
meeting
your
SLO
s,
maybe
you
should
focus
on
meeting
those
before
you
do
new
feature
work
and,
of
course
this
is
the
constant
struggle
right,
but
balancing
new
features
versus
maintenance
can
be
done.
That
decision
can
be
a
little
bit
easier
if
you
can
actually
quantify
your
arguments
so
now
you've
become
a
framework
for
what
Prometheus
is,
and
hopefully
the
ethos
of
what
we're
trying
to
achieve.
I
want
to
talk
about
what
it's
looked
like
for
us
at.
A
Do
we
don't
know,
do
is
a
cloud
company
like
the
big,
but
not
quite
as
big,
but
it's
still
very
good.
Also
I
have
a
bunch
of
t-shirts
up
here,
so
maybe
come
get
a
t-shirt
after
the
talk
like
this
one.
It's
this
one
so
I
want
to
see
some
general
stories
about
how
we're
using
Prometheus
that
I
think
are
useful
and
kind
of
demonstrate
what
I'm,
what
I'm
on
about
here.
So
we
deploy
a
node
exporter
on
every
machine
to
gather
physical
resource
metrics.
This
is
stuff
like
CPU
memory
disk.
A
You
know
all
the
basics.
We
scrape
this
on
all
of
our
hypervisors
at
a
regional
level.
This
kind
of
dashboard
shows
how
a
hypervisor
is
performing
it's.
This
is
the
kind
of
thing
that
gets
embedded
in
a
lot
of
different
support
tools.
Being
able
to
alert
on
certain
conditions
of
this
kind
of
thing
is
really
cool,
so
you
know
if
hypervisor
is
running
out
of
memory,
or
is
you
know
we
have
identified
some
situation?
That
means
a
crash
is
imminent.
We
can
kind
of
alert
on
that
being.
A
You
know
before
it
actually
happens
as
a
support
or
security
person.
Having
this
data
available
to
drive
automation
and
build
tools
on
top
of
is
really
really
powerful.
Just
like
having
this
data
in
general
to
track
fleet-wide
patterns
and
answer
questions
like
what
was
the
95th
percentile
of
bandwidth,
it
helps
to
make
data-driven
decisions
and
not
just
kind
of
do
things
based
on
hunches.
We
also
keep
a
lot
of
metrics
about
every
droplet
or
VM
that
runs
at
do.
This
is
tremendously
useful
for
our
support
teams.
A
You
know
there
are
a
lot
of
ways
to
get
this
type
of
information.
When
you
know
you
were
running
a
cloud
company
and
you
need
to
know
you
know
a
customer
as
a
support
inquiry
about
it,
you
know
their
VM,
but
this
is
kind
of
the
thing
that
is
reached
for
first,
a
lot
of
the
time.
You
know
another
one,
that's
kind
of
embedded
in
support
tools.
A
The
way
we
do
this
is
we
wrote
a
custom
exporter
that
exports
these
metrics
on
each
hypervisor
for
all
the
droplets
that
are
on
the
hv,
and
we
scrape
that
again
regionally.
This
data
actually
powers
a
number
of
really
powerful
tools
that
it's
not
an
exaggeration
to
say.
We've
used
that
data
to
solve
problems,
help
solve
problems
that
were
costing
company
millions
of
dollars
a
year
right.
This
is
critically
important
data
that
is
used
a
lot.
A
We
have
a
centralized
kind
of
bus
of
in-flight
or
completed
events
at
do,
and
these
are
kind
of
the
events
of
VMs
being
created
or
destroyed.
Backups
happening
stuff
like
that.
This
is
a
visualization
of
the
events
that
are
currently
in
flight
or
completed
or
failed
and
how
long
they've
been
running
or
how
long
they
took.
This
is
powered
by
Prometheus
histograms,
from
a
custom
exporter
that
I
wrote
and,
of
course,
poor
fauna,
huge
shouts
to
Agra
fauna
for
literally
the
sexiest
visualization.
A
You
will
see
today,
luckily
we're
the
only
cloud
company
out
there
that
never
has
incidents
but
like.
Let
me
tell
you
what
that's
hop
graph
in
in
an
incident
boy.
How
do
you,
let
me
tell
you,
it
gets
red
and
angry
when
something
theoretically
would
ever
be
going
on
when
some
of
these
senses
kind
of
a
disturbance
in
the
force,
maybe
being
able
to
look
at
overview
graphs
like
this.
That's
like
one
panel.
This
is
kind
of
the
state
of
you
know.
Our
world
is
really
powerful
and
cool.
A
Also
like
it's
just
it's
so
pretty
all
right.
Our
internal
service
level
management
project
is
also
extremely
good.
Most
products
can
define
being
healthy
in
terms
of
a
Prometheus
query.
So
what
we
do
is
we
run
air
flow
jobs
that
query
that
Prometheus
on
a
cadence
and
then
persist
these
key
slis
to
HDFS,
because
Prometheus
only
keeps
about
fifteen
or
thirty
days.
Those
means
we
can
query
this.
You
know
these
key
performance
indicators
going
back
forever
and
that's
really
cool.
So
the
first
thing
that
we
did
this
for
was
with
droplets.
A
A
We
do
we
have
this
for
products
all
over
our
business
now
and
it's
really
really
cool
it's
kind
of
just
a
good
North
Star
to
orient
yourself
towards
you
know
to
make
sure
that
you're
constantly
tracking
that
you're
improving
customer
experience,
our
scheduler
team
has
written
an
exporter
to
give
metrics
on
the
capacity
to
place
different
types
of
VMs
and
our
hypervisor
fleet.
So
having
this
data
from
the
view
of
the
actual
scheduler
means
we
are
as
close
as
we
could
possibly
be
to
the
real
state
of
the
world.
A
You
know
bugs
that
exist
in
in
the
scheduler
will
result
in
these
graphs.
So,
even
if
you
know
the
capacity
you
know
physical
capacity
they're
saying
there
is
capacity
there.
It's
like
well
sorry
bud,
like
you
know,
there's
something
wrong
with
the
schedulers
and
that's
really
really
powerful.
You
know
being
able
to
alert
on
this
stuff
in
real
time
and
sense
trends
and
say:
oh
wow,
we're
heading
for
disaster
can
be
really
really
good.
A
I
want
to
point
out
that
that's
four
stories
in
a
row
where
we
have
cotton,
this
kind
of
uncommon
level
of
insight
into
a
system
by
writing
a
custom
exporter.
You
know
these
were
all
kind
of
highly.
You
know,
tailored
solutions
to
our
architecture
and
software.
There
was
no
off-the-shelf
solution
that
would
have
given
us
this
type
of
insight.
A
You
would
have
had
to
have
done
some
custom,
instrumentation
I
think
this
is
one
of
the
coolest
things
about
Prometheus,
because
none
of
these
things
took
more
than
a
couple
of
weeks
to
actually
achieve,
but
they
kind
of
give
us
these
that
perfect
thing
that
it's
like
I
wish.
I
could
know
that
about
my
system
and
it's
literally
there
it's
on
a
graph,
it's
useful.
A
So
this
is
the
slide
with
the
numbers
on
it.
I,
don't
know
how
productive
this
actually
is,
but
I
think
it
could
be
a
good
starting
place
for
conversations
so
I
guess
if
you
want
to
do
that,
come
talk
to
me.
For
example,
if
you
want
to
know
my
samples
per
second
is
so
low
compared
to
the
other
things
or
if
you
want
to
come,
tell
me
about
how
a
single
big
server
could
scrape
that
many
samples
like
yeah.
Let's
come
come,
come
up
here
and
fight
me,
I!
A
Guess:
I,
don't
talk
to
me
after
this
I'm,
not
gonna
I!
Don't
care
like!
Let's
just
chat,
it's
fine,
so
I
want
to
give
you
some
tips
on
scaling.
Fermitas
from
an
operational
perspective
is
just
like
a
couple
of
things
that
I
think
you
need
to
know
because
I
really
don't
think
this
is
it's.
It's
not
an
easy
problem,
but
this
is
not
operating
kubernetes,
for
example,
here
like
this
is
a
much
more
kind
of
helpful
friendly
system
from
that
perspective.
A
But
that
said,
this
is
the
slide.
If
you're
gonna
take
a
picture
of
my
slides
with
your
iPad
or
something
this
is
the
time
to
do
it.
This
is
really
important
so,
when
you're,
creating
metrics
for
an
application
in
Prometheus,
remember
that
each
unique
permutation
of
labels
creates
a
new
time
series.
So
when
you
eventually
query
those
metrics,
you
need
to
know
that
if
your
query
needs
to
process
thousands
or
tens
of
thousands
of
time
series
to
complete
you're,
probably
going
to
have
a
bad
time,
it's
important
to
point
out
again.
A
This
is
not
samples
like
the
data
points
and
time-series,
but
this
is
the
number
of
different
time
series
you're
actually
accessing
samples
in.
So
this
isn't
just
a
graph
with
a
thousand
lines
on
it,
which
is
probably
not
useful.
Anyway,
say
you
have
a
metric
with
a
user
ID
label
in
your
system,
and
you
want
to
know
how
many
users
have
that
metric
greater
than
n
or
you
want
to
know
some
sort
of
quantile
of
where
what
the
metric
is
for
all
of
your
users.
A
A
I'd
like
to
acknowledge
this
kind
of
sucks,
because
a
lot
of
the
time
there's
some
really
cool
information
that
you
could
get.
You
know
per
user
metrics,
for
example,
in
your
system.
If
you
could
do
something
like
this,
but
it's
just
straight
up,
not
what
permit
Prometheus
was
meant
to
do.
However,
you
can
get
away
with
this,
depending
on
what
you're
doing
you
just
kind
of
need
to
understand.
The
limitations
understand
how
the
thing
is
going
to
be
queried
and
you
can
actually
do
some
cool
stuff
because
of
this
kind
of
limitation.
A
You're
going
to
hear
people
say
that
metrics
are
a
dead
end
from
an
investigative
standpoint.
Don't
listen!
It's
fine!
You
might
not
be
able
to
go,
find
the
specific
container
ID.
That
was
having
a
problem,
but
you
can
probably
find
the
pod,
and
if
not
that,
then
you
could
find
the
application
and
then
you
can
go
find
the
container.
It's
fine!
It's
fine
like
this!
Isn't
it's
at
Google?
Yes,
like
they!
Don't
there
is
no
way
to
do
cardinality
like
this
and
they've
invented
in
crazy
systems
for
most
people.
A
You
shouldn't
have
to
worry
about
this
too
much
unless
you
do
something
that
has
cardinality
and
millions
the
easiest
way
to
figure
out.
If
your
query
is
going
to
be
terrible
before
you
actually
do,
it
is
to
do
it
in
a
console
view
in
prometheus.
So
in
the
Prometheus
UI
there's
a
graph
and
a
console
view,
the
console
evaluates
your
query
at
one
point
in
time.
The
latest
point
in
time.
A
A
Prometheus
probably
needs
some
extra
RAM,
especially
if
you're
using
kind
of
custom
metrics,
where
cardinality
might
be
an
issue
during
normal
operations,
it's
actually
pretty
chill
about
general
resource
usage
and
even
under
you
know,
moderate
to
high
query
loads.
But
then
someone
will
do
a
query
that
uses
50000
time-series
over
seven
days
and
stuff
like
this
can
happen,
so
this
server
was
running
happily
for
days
at
six
and
a
half
gigs
of
memory
usage
and
then
quickly
instantly
essentially
ran
out
of
memory
at
16.
A
Gigs
I
see
stuff
like
that,
like
this
happened
fairly
often,
but
you
know
what
Ram
is
cheap
and
Prometheus
is
actually
pretty
good
at
restarting
after
it.
You
know
runs
out
of
memory,
so
just
something
to
keep
in
mind.
It's
fine,
no
have
another
one.
So
eventually,
there'll
be
a
set
of
metrics
that
you
want
a
query.
That's
gonna
get
really
big
like
too
big
for
a
single
instance.
A
You
probably
don't
want
to
have
more
than
a
few
million
time
series
on
the
single
Prometheus
instance
again,
unless
you
know
kind
of
why
or
what
you're
after
you'll
want
to
stand
out
more
from
atheists
in
shard
the
metrics
collection,
so
you
should
always
be
sure
to
shard
based
on
something
that
is
kind
of
a
logical
query:
boundary
something
like
a
region
or
a
service.
You
don't
want
to
split
up
scraping
by
taking
some
hosts.
A
Metrics
in
one
instance,
and
some
hosts
in
another
because
querying
across
multiple
Prometheus
instances
is
kind
of
a
pain
and
it's
sort
of
non-trivial.
You
can
do
it
and
getting
that
global
query
view
is
possible,
but
there
is
rarely
a
situation
where
I
think
it's
super
necessary
again.
You
can
do
it,
though
it's
fine
for
metrics
that
you
really
care
about
which
is
hopefully
all
of
them.
You
should
deploy
h.a
pairs
of
Prometheus
instances,
so
this
is
multiple
Prometheus
that
scraped
the
same
metrics,
preferably
in
different
geographic
locations.
A
To
isolate
yourself
from
those
kind
of
failures.
This
is
cool
because
then
you
can
use
Engine,
X
or
H
a
proxy
to
sit
in
front
of
those
instances
and
do
things
like
load,
balancing
queries
or
you
know
doing
an
active/passive
to
have
a
situation.
You
can
also
do
things
like
rate
limiting
and
log
your
queries.
A
You
know
it
works,
really
really
well
getting
people
to
adopt
it
and
use
it
to
its
fullest
potential
from
an
organizational
perspective
is
a
lot
harder.
In
my
opinion,
I
think
this
is
the
good
part.
So
if
you're
asleep,
wake
up
or
something
remember,
I
talked
about
why
we
observe
our
systems
in
the
first
place
earlier,
it's
about
revenue
protection,
which
is
you
know
how
you
add
value
if
you're
not
adding
new
features.
So
if
you
want
to
provide
maximum
value
which
gets
you
maximum
paid,
you
should
build
observability
into
your
systems.
A
There's
a
problem
with
that,
though,
because,
basically
no
matter
what
happens,
you
are
going
to
be
doing
that
under
some
sort
of
pressure.
You
can
do
it
in
the
beginning,
under
pressure
to
get
back
to
building
more
new
features,
or
you
can
do
it
when
something
is
broken
and
you'll
have
to
spend
extra
time
retrofitting
it
in
later,
possibly
into
an
architecture
that
wasn't
really
built
well
for
that
type
of
thing
either
way.
A
You're
gonna
have
a
limited
amount
of
time
to
spend
building
observability
into
your
systems,
so
engineers
do
what
engineers
do
they
copy
the
patterns
that
they
see
and
these
patterns
end
up
proliferating
through
organizations,
because
everybody
is
short
on
time
right.
So
that
means
the
impact
of
a
good
or
a
bad
pattern,
actually
gets
multiplied
out
over
time
by
a
fair
amount
and
with
Prometheus
more
so
than
something
like
logging
or
tracing.
You
are
doing
custom
instrumentation.
So
this
is
kind
of
a
real
issue.
A
You
are
writing
you're
deciding
what
kind
of
metrics
to
create
in
your
code.
You
are
actually
incrementing
those
things
in
your
code,
so
the
quality
of
your
you
know,
outcomes
and
your
data
is
much
more
dependent
on
the
decisions
that
you
make
rather
than
something
organizational
like.
Hopefully,
you
know
tracing
is
kind
of
integrated
into
your
whole
system,
and
this
can
kind
of
bite
you,
because
at
first
everything
is
everything
is
fun
and
easy.
A
Like
you
installed
an
exporter,
you
created
a
couple
of
metrics
without
reading
the
docs
and
you're
like
hunk,
like
I,
can
see
inside
my
app
now
the
world.
You
know
your
galaxy
brain,
but
but
doing
things
like
actually
measuring
your
availability
and
quantifying
your
customer
experience
is
a
little
bit
harder
if
you
didn't
follow
best
practices
when
you
were
creating
your
metrics
and
or
maybe
you
don't
understand,
some
of
the
theory
standing
up
that
Prometheus
server
and
then
scraping
those
metrics
is
super
easy
too.
It's
like!
A
Well,
you
know
you
pull
down
the
binary
you
edit,
the
mo
file.
You
start
it's
great
and
honestly
for
a
long
time.
It's
gonna
be
totally
good,
but
then
you
might
have
a
lot
of
metrics
or
a
time
series
illegal.
You
might
want
to
start
sending
alerts
and
make
gravano
dashboards,
then
all
of
a
sudden
you're
operating
three
different
applications
and
your
ability
to
alert
on
failure
and
measure
the
performance
of
your
system.
This,
these
critical,
critical
efforts
are
built
on
that
playground
that
you
stood
up
a
year
ago
and
haven't
touched.
A
When
you
go
on
to
kind
of
define
alerts
that
make
on
call
good
or
try
to
create
those
good
dashboards.
If
you
don't
have
patterns
that
are
easy
to
follow,
maybe
because
you
can't
find
those
other
team's
core
fauna
instances
that
they
stood
up
because
they're
down
or
hidden
somewhere,
you
know,
that's
gonna,
be
bad
and
then
eventually
you're
gonna
start
writing,
queries
that
don't
come
back
and
that
could
be
for
a
lot
of
different
reasons
and
you're
gonna
have
to
go
and
either
figure
out.
A
Why
we'll
just
take
again
taking
some
of
that
limited
time
or
just
you
know,
give
up
or
you're
very
talent
and
forget
it?
What
I'm
saying
is
that
each
of
these
steps,
the
choices
that
you
make
impact
the
next
and
they
all
build
on
each
other,
every
decision
actually
matters
and
they
are
multipliers
on
your
ability
to
effectively
maintain
the
system
in
the
end,
when
you're
short
on
time,
making
those
good
decisions
is
actually
hard.
A
So
this
is
why
I
think
that
you
need
people
dedicated
to
this
problem,
and
this
is
what
I
mean
by
adopting
Prometheus
the
hard
way.
That's
the
air,
getting
let
out
of
the
balloon
of
my
sexy
talk
title
because
it's
not
it's
obvious!
You
know
the
naive
path.
I,
don't
want
to
scare
you
off.
The
naive
path
will
get
you
a
lot
of
value
with
your
pain
but
I
think
the
hard
way
is
kind
of
where
the
real
magic
happens.
A
I
think
you
can
multiply
your
gains
by
you
know
a
hundred
times
by
establishing
good
patterns
and
iterating
on
them
over
time
and
making
sure
that
people
are
kind
of
protecting
that
that
value.
So
what
are
these
people
actually
going
to?
Do
it's
important
that
everybody
kind
of
make
good
use
of
their
limited
observability
time?
Obviously,
everybody
needs
to
have
a
basic
understanding
of
how
to
use
those
tools.
A
The
tools
need
to
be
operated
well
so
that
they
are
actually
available
and
scalable
so
that
you
know
people
can
keep
getting
that
value
out
of
them.
There
needs
to
be
patterns
and
documentation
in
a
centralized
place
that
people
can
go
and
copy
if
they
need
to
and
people
somebody
needs
to
be
around
to
consult
with
teams,
while
they
are
kind
of
have
that
one
week
or
two
weeks
to
actually
start
making
their
stuff
observable.
A
Eventually,
that
kind
of
momentum
in
an
organization
can
sustain
itself
when
everybody
is
kind
of
seeing
what
it
looks
like
to
have
your
stuff
be
easy
to
maintain
you
know,
and
once
everyone
wants
to
be
on
that
train.
You
know
this
is
the
kind
of
thing
that
is
desirable,
at
least
so,
and
what
I
mean
by
operating,
prometheus
and
friends?
You
know
it
should
go
without
saying,
but
you
shouldn't
stand
up
tools
without
somebody
to
be
responsible
for
them.
A
Keeping
teams
from
standing
up
bespoke
you
know,
installations
of
Prometheus
alert
manager
and
Ravana
and
more
not
caring
for
them
is
important,
because
that
is
a
perfect
waste
of
that
limited
time
that
I
keep
talking
about.
You
know,
instrumenting,
you
could
be
instrumenting
applications
rather
than
you
know,
breathing
life
into
a
fragile
observability
tool.
A
Somebody
needs
to
be
on
call
for
that
Prometheus
and
Friends,
and
that
gets
more
complicated
as
you
make
that
stuff
highly
available
right,
like
you
shouldn't,
have
all
of
your
dashboards
in
one
kubernetes
cluster
in
one
data
center,
because
then,
if
that
kubernetes
cluster
goes
down,
not
only
can
anybody
see
graphs
about
their
own
applications,
they
can't
see
graphs
about
the
kubernetes
cluster,
so
you're
kind
of
really
in
a
bad
situation,
making
sure
that
everybody's
metrics
and
alerts
stay
healthy
is
also
really
cool.
So
over
time
you
know,
metrics
themselves
actually
end
up
rotting.
A
They
stop
being,
you
know,
stop
being
as
useful
or
they
just
kind
of
disappear,
because
you
know
applications
move
and
you
know,
alerts
are
based
on
metrics
that
changed
at
some
point,
and
you
know
the
alert
is
still
off
trying
to
fire
like
hey
but
I'm
doing
my
best,
but
you
changed
metrics,
so
I
don't
know
anything
anymore.
Having
somebody
that
keeps
an
eye
on
that
stuff
is
really
valuable
because
you
can
kind
of
you
know
prop
up
and
maintain
that
value
that
somebody
that
somebody
made
when
they
had
that
limited
time.
A
They
also
need
to
decide
when
it's
time
for
more
Prometheus
and
be
able
to
justify
that
expense.
You
know
if
you're,
if
somebody's,
adding
a
bunch
more
metrics,
and
that
is
too
much
for
the
current
system
to
handle
it's
important
that
somebody
can
say
like
ok,
it's
gonna
cost.
This
much
are
these
metrics
that
you
made
worth
of
this
much
money.
It's
a
good
thing
to
know.
A
It
needs
to
be
easy
to
copy
those
good
patterns
again
I'm
driving
it
home,
but
having
some
sort
of
config
file
and
chef
or
ansible,
or
a
docker
image
with
that
config
file
and
source
control,
where,
where
everybody
can
play
with
it
and
kind
of
you
know,
add
to
it,
you
know
self-service
is
a
really
good
first
step.
Eventually,
this
ends
up
getting
too
big.
Nobody
wants
to
edit
a
20,000
line.
Config
file.
A
You
need
to
then
provide
some
sort
of
abstraction.
Let
people
declare
that
they
want
their
metrics
scraped
rather
than
you
know,
making
it
happen
themselves.
It's
important
when
you
do
this
kind
of
abstraction
that
you
don't
obfuscate.
The
core
concepts
still
try
to
enforce
some
basic
level
of
understanding
of
the
system,
because
ultimately
it
will
become
a
black
box,
and
people
will
do
mean
things
to
the
black
box
or
worse.
If
they
don't
understand
the
black
box,
then
it
can
disincentivize
them
from
kind
of
owning
their
own
availability
and
they
won't
use
the
box.
A
This
is
kind
of
a
hard
problem.
So
if
you
come
up
with
something
cool,
I
guess
let
me
know
because
I
would
like
to
know
you
need
to
be
able
to
point
people
at
a
place
where
they
can
learn
about
what
they
should
be
measuring.
You
know
they
should
go.
Read
the
golden
signals
chapter
in
the
SRT
book
or
use
the
use
method,
or
something
like
that.
Helping
them
be
able
to.
You
know
quantify
that
user
experiences
is
good
stuff.
A
You
should,
as
a
you
know,
person
doing
this,
create
common
metrics
patterns
for
things
like
RPC
apps
and
these
apps
and
things
that
are
similar
so
that
you
can
apply
general-purpose
alerts,
and
you
know
general-purpose
dashboards
to
make
it
easy
for
everybody
to
kind
of
understand.
All
of
the
systems
that
you
know
has
share
these
characteristics.
A
Make
them
go,
read:
Cindy's
blog
post
about
how
to
do
alerts
right
like
that's.
That's
the
thing
and
you'll
need
to
document
all
your
other
various
internal
tools.
So,
having
contextual
Prometheus
query
examples
for
systems
that
people
are
probably
familiar
with
explaining
how
alert
manager
works
so
that
people
set
up
alerts
that
kind
of
fit
well
into
that
flow
and,
however,
you
end
up
getting
people's
metrics
scraped,
you
should
kind
of
document
that
so
that
people
can
hopefully
self-service
that
the
other
big
part
is
consulting
with
teams
when
they're
adopting
this
tool.
A
So
when
a
new
team
starts,
they're
gonna
have
a
lot
of
questions
and
need
a
lot
of
help.
Having
someone
there
to
kind
of
answer,
those
questions
saves
people
time
from
you
know,
being
their
head
against
a
wall
and
for
highly
expensive,
and
you
know
smart
engineers
that
opportunity
cost
is
real
because
whatever
they
could
have
been
doing
is
probably
really
valuable.
When
the
tools
become
semi
ubiquitous
in
an
organization,
people
will
start
reaching
for
it
to
do
more
and
more
things.
A
A
You
know
losing
time
and
because,
like
my
team
ends
up
knowing
where
a
lot
of
the
you
know,
kind
of
the
metrics
and
the
observability
data
are
we're
able
to
come
jump
into
incidents
and
help
track
things
down,
just
because
we
kind
of
know
where
things
are
and
in
the
tools,
and
we
can
also
kind
of
help
with
a
lot
of
other
kind
of
important
cross-functional
efforts.
So
stuff,
like
standardizing
operational
readiness,
review
criteria
and
implementing
error
budgets
is
something
that
these
people
can
kind
of
be
the
owners
of.
A
So
these
were
kind
of
some
of
the
problems
that
I
had
set
out
to
solve
when
I
kind
of
started
doing
this
kind
of
thing.
You
know
we
had
things
that
needed
to
be
discovered
in
chef
and
prometheus.
Didn't
spork
chefs
I
wanted
to
do
that.
I
wanted
to
kind
of
centralize
alerts.
You
know
basically
do
these
things
to
centralize
knowledge,
centralized
configurations,
other
people
can
copy
the
stuff.
A
My
attempt
was
to
what
I
did
was
create
a
git
repository
that
acted
as
a
configuration
directory
for
some
code
that
turned
these
Yambol
abstractions
into
Prometheus
configuration
my
team
reviews,
pull
request
to
this
repository
and
provides
feedback,
and
what
people
are
doing
from
me,
theas
wise,
so
checking
that
their
metrics
are
good,
checking
that
their
alerts
will
or
won't
fire
stuff
like
that.
This
is
kind
of
a
double-edged
sword,
because
what
it
done,
what
ends
up
happening
is
that
people
do
have
better
metrics
and
better
observability,
but
it's
not
completely
self-service.
A
So
if
you
have
kind
of
limited
time-
and
you
see
that
you're
gonna
have
to
wait
for
somebody
on
that
team
to
review
things,
then
you
might
choose
to
do
something
else.
Instead,
I
also
wrote
a
lot
of
documentation
that
most
of
the
time
people
don't
read,
because
they
would
rather
just
ask
the
question
directly,
but
overall
I
think
is
still
important,
because
it
enables
the
best
type
of
folks
to
actually
go
and
learn.
A
It's
important
to
have
kind
of
a
centralized
short
version
of
the
stuff
that
you
constantly
shove
in
people's
faces,
so
that
hopefully
they
get
those
basics.
If
I
did
it
over
again,
I
would
probably
build
on
top
of
the
kubernetes
prometheus
operator.
I
think
there's
a
lot
of
potential
and
having
some
sort
of
application
on
top
of
the
operator.
That
kind
of
verifies
that
metrics
are
good
and
places
them
somewhere
and
verifies
lots
of
other
resources
and
then
actually
creates
necessary,
kubernetes
crts.
To
make
that
stuff
happen.
A
But
I
don't
know,
maybe
someone
from
the
audience
can
come
up
and
tell
me
if
that's
a
bad
idea.
So
that's
what
I
think
you
need
I
think
that
you
need
people
to
kind
of
do
these
things.
I
think
this
function
fits
naturally
and
into
a
team
of
people
that
do
this
for
other
tools.
Also
having
this
kind
of
influence
for
centralized
logging
tracing
exceptions,
I
think
is
extremely
valuable
and
I.
Don't
think
you
need
to
hire
any
sort
of
observability
expert
for
this
type
of
stuff.
So
what
happens
in
the
end?
A
You
know
you've
you've
done
all
of
this
work.
What
what
is
the
result?
Some
people
will
really
love
it.
They'll
get
involved,
they'll
build
tools,
you
know,
they'll
they'll,
make
you
know,
build
tools
on
top
of
you,
which
is
always
a
really
cool.
Most
people
will
just
copy
pasta,
the
stuff
they
see.
They'll
get
a
couple
alerts,
get
a
couple
of
dashboards:
fine,
that's
totally
cool
right
because
they
have
got
value
in
that
limited
time.
Some
people
want
to
push
beyond.
A
This
could
be
an
entire
talk
in
itself,
but
what
people
will
want
to
do
is
stuff
that
can
sometimes
be
solved
by
some
of
the
other
kind
of
newer
friends
of
prometheus
out
there
and
again,
all
of
these
use
various
different
approaches
to
do
some
combination
of
down
sampling,
long-term
storage,
global
query,
views
and
better
handling
large
cardinality,
but
the
thing
they
should
keep
in
mind
is
that
any
of
these
things
are
going
to
be
a
lot
more
work
operationally
than
good
old
prometheus.
So
that's
all
I
got
come,
get
a
shirt.