►
From YouTube: Beyond Nagios: Modern Monitoring of Bronze-Age Applications with Prometheus by Ben Kochie
Description
Beyond Nagios: Modern Monitoring of Bronze-Age Applications with Prometheus - Ben Kochie, SRE/Systems Engineer
"Prometheus is a high-performance, time-series-based monitoring system. We wanted to improve the monitoring and alerting for a mature Rails application that relied on a variety of tools, which each had their own limitations. Using Prometheus, we were able to consolidate these tools while gaining deep, user-focused insight about the application performance.
With minor changes to the legacy codebase, we gained insights about Rails performance that were not possible with the existing tools. This allowed us to consolidate metrics collection, improve the quality of alerting, and reduce our dependency on expensive third-party monitoring services.
We also improved the state of the art for MySQL monitoring, which allowed us real-time visibility into database performance problems.
But the story doesn't end here. There are still monitoring challenges to overcome. In the last part of the talk, I will cover future plans and problems that we haven't solved yet."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right
welcome
to
beyond
Nagios
modern
monitoring
of
bronze
ADA
applications
with
Prometheus.
My
name
is
ben
kochi
I'm,
an
s3
at
SoundCloud
on
our
production
engineering
team.
We
manage
all
of
the
infrastructure
software
required
to
run
the
sound
cloud
platform
for
all
the
product
teams
that
make
the
actual
software
for
for
SoundCloud
the
history
of
me,
I've
been
in
ISP,
I
Tisa
said
men
s,
serene
changes
over
the
years
for
about
20
years,
I
got
into
high
performance
computing
in
the
early
2000s
and
then
I
got
into
the
sra
thing.
At
Google.
A
A
A
A
The
some
of
the
biggest
downside
is
it's
really
expensive.
I
did
a
bit
of
a
math
to
figure
out
how
we
were
using
my
new
relic
and
like
hey.
Why
are
we
only
running
new
relic
on
a
small
5%
sample
of
our
servers?
Well,
it
turns
out
that
that
5%
sample
was
costing
my
salary
every
month,
so
it
was.
It
would
be
economically
infeasible
for
us
to
actually
deploy
new
relic
to
the
entire
mothership.
A
So
we
only
had
a
small
5%
sample
of
servers,
instrumented,
which
was
just
like
wait,
a
minute
we're
only
looking
at
when
I
look
at
New
Relic
I'm
only
looking
at
a
5%
sample
of
our
of
our
app.
What?
If
all
the
other
90%
of
our
servers
are
down?
Well,
we
don't
know
so
we
needed
we
needed.
We
needed
other
things.
Other
problems
with
New
Relic,
it's
data
aggregation,
is
by
minute,
and
you
know
I
see
this
there.
Everything
is
in
New
Relic
US
by
minute,
so
it's
pretty
low
resolution.
A
Right
was
a
yeah.
I
was
the
top
of
the
hour
and
they
were
all
hit
all
Android
devices
in
the
entire
planet.
We're
hitting
our
servers
simultaneously
or
maybe
not
all,
but
a
pretty
big
chunk
of
them
and
with
with
New
Relic.
We
couldn't
see
that,
but
with
Prometheus
we
could
the
other
problem
with
New
Relic
was
the
the
user.
Interface
is
they're
using
your
interface
and
you
only
get
to
see
it
the
way
that
they
want
you
to
see
it.
A
Another
thing
that
the
mothership
had
is
stats
T,
because
the
rails
model,
the
rails
app,
is
basically
single
threaded,
there's
no
way
to
get
to
it
other
than
it's
HTTP
interface,
but
that's
blocked,
responding
to
requests
so
early
on
before
I
worked
on
them.
On
this,
there
had
already
been
some
work
to
instrument
the
mothership
with
stats
T,
but
there
are
downsides
to
this.
A
In
the
original
version,
all
of
the
rails,
binaries
were
sending
their
metrics
over
UDP
to
a
single
collector
in
production,
and
it
turned
out
that
70%
of
the
samples
were
being
lost
but
because
of
UDP
drops
over
the
network
and
one
of
the
one
of
the
Prometheus
developers
actually
fixed
this
by
adding
stats
the
aggregators
on
every
node,
but
so
now
we
had
and
then
building
forwarders,
but
we
were
still
losing
data.
It
wasn't
accurate.
So
there
is
there's
a
big
downside
to
the
way
stats.
These
UDP
protocol
handles
data.
A
And
then
stats
T,
we
would
take
all
the
data
from
stats
T
and
send
it
into
graphite.
We'd
also
take
note
level
metrics
from
ganglia
plugins
and
send
that
into
graphite,
and
that
was
actually
not
too
bad.
I
got
a
lot
of
really
good
instrumentation
done
of
mothership
by
building
new
views
into
the
graphite
data,
because
we
had
all
this
data,
but
nobody
had
really
put
the
time
into
thinking
about
how
to
visualize
that
and
I
got
us
about
half
of
the
way
to
a
really
nice
setup,
just
by
using
Graphite's
data.
A
A
Handling
tens
of
thousands
of
requests
per
second,
so
graphite
couldn't
keep
up
with
the
amount
of
data
we
wanted
to
collect
to
really
get
good
insight
into
into
what
was
going
on.
Plus
the
data
model
is
a
bit
limiting
and
if,
if
you
compare
the
Prometheus
data
model
to
the
graphite
data
model,
everybody
knows
that
the
labeling
is
just
way
better.
A
Another
one
that
instruments
mothership
is
that
air
brake
air
brake
is
completely
unrelated
to
Prometheus.
For
me,
theist,
it
simply
forwards
exceptions
that
so
code
crashes
within
this
within
mothership.
It
forwards
that
off
to
air
brake.
This
is
an
unsolved
problem
for
us,
because
Prometheus
doesn't
do
this,
it's
not
what
Prometheus
for
so
it's
still
actually
quite
useful
to
use
air
brake
for
monitoring
but
I.
A
Finally,
some
people
really
like
it
I
find
the
user
experience
as
an
engineer,
highly
frustrating
the
the
most
hilarious
thing
is:
if
you
get
a
spike
in
exceptions
on
their
graph,
you
can't
click
on
it.
It's
completely
useless.
It
won't.
Actually
you
can't,
you
can't
even
select
a
time
range
to
get
that
data
to
find
out
what
happened
in
that
air
break
window.
Like
there's
a
spike,
but
what
did
what
is
it?
No
I?
A
Don't
know
you
can,
and
it
only
shows
you
by
latest
exception,
which
could
like
the
small
low
frequency
exceptions,
can
just
scroll
off
the
high
frequency
exceptions.
You
don't
know
you
don't
actually
know
what
happened
and
you
end
up.
Tate.
You
end
up
spending
three
times
the
amount
of
work
chasing
down
was
this.
Was
this
the
one
that
was
in
the
spike?
Was
this
the
one
that
was
in
the
spike?
A
Was
this
one
that
was
in
the
spike
you
just
you
just
don't
know,
but
unfortunately
that's
not
something
we
can
with
fixed
with
prometheus
and
then
to
actually
get
alerts
out
of
the
mothership
we
have
Nagios
and
Nagios
is
everybody's
favorite
thing?
No,
it's
a
pretty
terrible,
broken
data
model
for
alerting.
It's
super
slow.
It's
super
annoying
to
configure,
but
we
were
able
to
get
some
of
things
like
using
check
HTTP
to
monitor
the
passenger
engine
ex
passengers.
A
A
We,
you
couldn't
actually
get
information
about
what
was
broken,
so
you
you
would
have
to
go
and
tail
logs
and
and
look
at
things
or
go
to
New
Relic
and
try
and
figure
out
from
from
from
logs
and
new
relic.
What
was
actually
going
on
so
prometheus
is
what
we
wanted
to
get
the
mothership
back
up
and
floating
so
Prometheus
for
the
people
that
don't
know
as
a
time
series
based
monitoring
system.
A
The
first
thing
that
we
instrumented
in
the
mothership
was
we
want
to
instrument
the
database
that
runs
the
mothership.
So
the
my
sequel
database,
we
wrote
the
my
sequel
exporter
and
this
gave
us
really
really
deep
insight
into
what
was
going
on
on
the
database
side
so
that
we
could
improve
performance
and
latency
from
the
databases
perspective.
A
Data
mine,
but
it's
really
hard
to
look
at
because
the
only
way
it
presents
it
is
in
SQL,
so
you
have
to
do
select
States
minutes
to
get
it
against
it,
and
this
is
really
not
useful,
so
Prometheus
what
collects
that
data
turns
it
into
a
time
series
and
now
I
can
do
stuff
like
this.
So
this
is
a
performance
schema
example
where
I
can
take
and
look
at
the
average
number
of
rows
examined
versus
the
average
number
of
statements.
A
So
now,
I
can
get
a
graph
of
how
many
rows
per
query
are
required
for
a
particular
query,
and
this
tells
you
an
approximate
cost
for
that
query,
and
now
I
can
look
at
a
graph
and
see.
Oh,
these
are
the
top
10
select
statements
that
cause
me.
A
super
expensive,
I.
Think
the
number
over
there
is
something
and
they'd
like
400.
Rows
per
query
are
needed
to
resolve
that
query
and
that
could
that
can
be
and
and
there
there
are
periodic
problems,
there's
constant
problems
and
there's
periodic
problems.
A
A
The
next
thing,
I
did
is
a
total
hack.
A
Google
engineer
wrote
this
really
nice
tool
called
M
tail
and
M
tail.
Just
tails
your
log
file
parses
it
with
reg
X
matching
and
turns
those
reg
X
match
matches
into
counts.
So
now
I
can
count
every
log
line
and
but
with
Prometheus
I,
don't
just
get
the
count
of
log
lines.
I
can
pull
out
little
bits
of
data
from
each
log
line
using
the
regular
expression
pattern-matching
and
put
those
into
labels.
A
So
now,
with
a
little
bit
of
tweaking
of
the
rails,
log
I
can
now
say
well,
I
have
the
info.
Well,
sometimes
it's
info,
sometimes
it's
worn
so
now
I
can
have
counts
of
log
lines
by
info.
Worn
I
can
take
the
time
out
of
the
time
stamp
there
and
actually
have
a
count
of
time
spent.
So
now
I
can
get
average
Layton
sees
and
then
a
little
bit
and
plus.
A
Of
course
we
have
the
status
code,
so
we
know
whether
we're
sending
a
two
hundred
or
a
four
oh
four
or
a
five
hundred
or
whatever
the
HTTP
status
code
is
returned
by
the
rails
controller
and
then,
with
a
little
bit
of
tweaking
to
the
rails
code,
we
appended
the
rails,
controller
and
action.
So,
basically,
what's
what
function
within
rails
was
being
called
and
what
mode
it
was
being
requested
in
and
so
now
we
have.
A
We
have
latency
errors
responses
by
action
by
controller.
So
now
we
have
all
of
these
dimensions
and
we
are
now
we
have
time
series
for
every
single
one
of
those,
and
we
can
now
take
these
with
this
M
tale
configurations.
This
is
a
contrived
version
of
what
was
actually
in
our
production
configuration
and
we
pull
all
those
things
out
and
now
we
have
all
these
labeled.
A
Values
and
we
can
actually
start
writing
alerts,
and
now
we
can
get.
We
can
actually
get
an
alert
for
a
single
controller
and
a
single
action
from
every
instance
of
our
rails,
app
and
and
know
immediately.
What's
going
wrong
instead
of
going.
Oh
mothership
is
returning
errors.
I
get
but
I
better
spend
half
an
hour
investigating
now.
I
just
get
an
email.
That
tells
me
exactly
what
controllers
malfunctioning.
A
Here's
a
dashboard
example
of
looking
into
a
single
specific
controller,
so
you
can
see
the
responses
by
control
by
action.
You
can
see
the
response
codes
by
coming
from
the
controller
and
we
know-
and
we
have
a
latency
graph
down
on
the
bottom
here
that
tells
us
the
latency
by
action.
So
I
can
see
that
Oh
index
index
actions
are
nice
and
fine.
There's
a
little
bit
of
noise
in
the
in
the
create
and
there's
actually
quite
a
bit
of
noise.
A
A
There's
200
requests
per
second
to
post,
a
comma,
so
to
look
at
comments
for
track
and
how
and
maybe
a
few
per
second
of
people
posting
new
comments,
and
then
we
have
a
a
DIN
error
and
that's
actually
a
different
prometheus
query
that
tells
us
the
increase
of
errors
and
over
the
last
five
minutes,
and
then
it's
blocked
in
five
minute
blocks.
So
you
can.
You
can
really
see
if
you
have
a
very
low
error
rate.
You
can
now
see
that
super
low
error
rate,
because
Prometheus
allows
that
flexibility
in
query
language.
A
Perspective,
it
costs
about
10
to
15%
of
motherships
CPU
capacity
to
just
tail
and
process,
all
those
logs
so
for
for
every
one
of
our
every
one
of
our
servers.
If
it's
my,
if
it's
running
10
cores
of
rails,
its
running
one
core,
just
flat-out,
reg
xing
logs,
because
we're
generating
tens
of
megabytes
of
logs
per
minute
per
node,
so
200
nodes
is
we're
generating
gigabytes
per.
You
know
order
order,
magnitude,
gigabytes
per
minute
of
logs
from
these
things,
but
because
it's
M
tails
distributed
on
every
node,
it's
actually
pretty
scalable.
A
Stastny,
actually
is
a
totally
great
way
to
get
data
out
of
it
out
of
something
that
is
a
single-threaded.
So
if
you've
got
PHP
code
or
rails
code,
that's
single
threaded
or
some
other
thing
that
can't
isn't
a
persistently
running
app
or
purse.
It
doesn't
have
any
persistence,
threads
stat,
the
sending
out
to
that
local
host
aggregator
is
actually
pretty
nice
because
we
can
take
that
those
stats,
the
metrics
use
the
Prometheus
stats
the
exporter
and
turn
those
into
counts.
That
Prometheus
can
come
and
collect
and
then
graph
in
time
series.
A
But
the
trees
are
a
little
messy
and
you
have
to
be
very.
You
have
to
be
much
more
diligent
in
your
code
to
do
this,
so
I
had
to
do
a
bit
of
code
cleanup
to
actually
make
some
of
these
stats
D
metrics
useful.
So
we
had
some
met
metrics
for
memcache
usage
and
there's
one
piece
of
code.
That
would
count
the
number
of
hits
and
count
the
number
of
misses.
Then
there's
a
different
piece
of
code.
That
would
count
the
number
of
lookups
and
the
number
of
misses.
Well.
A
Lookups
is
the
sum
of
hits
and
misses
so
I
had
that
we
had
to
fix
that
plus
there's
not
enough
context
in
here.
It's
like
wait,
user
or
user
whitelisted.
What
is
it?
Where
is
that
coming
from
so
it
turns
out,
it
was
actually
cut.
Well,
one
was
coming
from
a
rails
model
and
one
was
coming
from
a
rail
service
and
it's
that's
that's
real
specific
things,
but
now
I
started
cleaning
up
all
this
stats,
D
names
and
move
the
the
hit
to
actually
count
lookups.
A
So
now
we
have
consistent
lookups
and
mis
metrics
and
then
I
use
the
stats
T
mapping
in
the
stats,
the
exporter,
to
turn
that
into
a
key
value,
labels
for
Prometheus
to
collect
and
I.
Don't
know
if
I
have
any
good
graphs
of
that
one,
but
it
it
it
turned
out
to
be
really
useful.
So
we
can,
we
could
actually
start
alerting
in
Prometheus
on
cache,
miss
problems.
A
Here's
the
example
of
a
New
Relic
thing:
it's
actually
quite
a
nice
interface,
but
you
can't
customize
it
as
as
much
as
I'd
want
and
then
here's
this
similar
like
application
overview,
and
it
gives
me
the
things
that
I
really
actually
care
about,
which
is
which
can
which
top
controllers
are
having
errors.
So
I
can
immediately
look
at
a
graph
and
see
which
controllers
are
causing
any
problems,
and
then
I
can
also
looking
at
latency.
A
So,
whereas
New
Relic
tends
to
focus
on
their
apdex
score,
the
overall
throughput
of
the
entire
thing
and
then,
but
it
really
does
have
this
nice
stacked,
latency
graph
for
different
subsystems.
So
that's
that's
one
thing:
I
haven't
been
able
to
reproduce
in
Prometheus
yet,
but
I
probably
could
if
I
actually
went
into
the
rails
code
and
and
added
some
stats,
D
metrics
on
this
latency
to
get
that
data
out.
A
So
what's
coming
up
in
the
future
for
mothership
I
right
now
we
collect
latency,
just
average
latency,
but
improvements
to
the
M
tail
code
now
now
allow
us
to
actually
add
histogram
support.
So
we
can.
We
can
now
bucket
eyes
the
latency
response
for
different
controllers.
I
want
to
continue
doing
the
stats
D
cleanup.
A
A
No,
so
the
stats
D,
so
that
is
there
a
library
to
swap
out
stats
D.
Well,
so
there's
the
Prometheus
library.
So
if
your
code
is,
if
you
can
load
the
Prometheus
library
in
your
code,
you
can
swap
out
stats
T
that
way,
but
of
course,
you'll
have
to
it's
not
the
same
exact
semantics
depending
on
your
programming
language.
It's
not
it's
not
exactly
it's
it's!
You
would
use
it
the
same
way.
You
do
increments
and
you
do
sets
so
you
set
a
metric
or
you
increment,
a
metric
or
you
or
you.
B
A
A
What
what
things
matter
to
your
users
and
what
things
matter
to
your
engineers
and
you
and
just
drop,
put
draw
a
line
in
the
sand
and
if
it's
over,
that
line,
that's
an
alert
so,
for
example,
high
persistent,
latency
high,
persistent
errors,
large
spikes
and
errors,
those
those
kinds
of
things,
that's
a
lot
of
that
it
is
actually
really
well
covered
in
the
Google
s
or
ebook.
Is
there
are
lots
and
lots
of
things
you
could
alert
on,
but
you
really
only
want
to
pick
the
things
that
matter
to
your
users.
C
C
A
A
And
yeah
so
yeah,
so
the
question
is
open.
Tracing,
yes,
I,
open
tracing
is
probably
the
answer
there,
or
some
some
kind
of
tracing
or
something
like
air
break,
but
yet
having
having
having
the
logs
is
and
what
were
what
we're
working
on
right
now
is
just
making
really
easily
searchable
logs.
So
we
try
and
just
up
the
logging
verbosity
and
put
as
much
information
into
the
logs,
but
only
keep
logs
for
a
very
short
amount
of
time
so
that
we
don't
overwhelm
our
log
in
cluster
and
that
way
we
get
that
way.
A
Oh,
my
god,
oh
my
god,
how
is
managing
prometheus
compared
to
graphite?
That
is
entirely
why
we
wrote
Prometheus
in
the
first
place.
Was
it's
super
easy
to
run
a
Prometheus
server?
You
give
it
a
config
file.
You
start
it
up.
You're
done
that!
That's
it's!
It's
one
binary!
It
reads
its
config
file
and
you
just
let
it
do
its
thing.
It's
it's
just
hilariously
easy
to
run.
So
we
we
end
up.
We
now
in
it,
because
it's
so
hilariously
easy
to
operate
prometheus.
A
We
have
30
of
them
monitoring
different
applications,
so
there,
instead
of
having
one
big
giant
graphite
that
does
everything
and
it's
like
nobody-
wants
to
touch
it.
Nobody
upgrades
it.
It
just
sits
there.
No,
we
just
have
this
Prometheus
monitors
this
app.
This
one
does
this
app!
This
one
does
this
app
and
if
they
break
well,
we
don't
care.
We
just
hit
a
button
and
a
new
one
pops
up.
A
Yep
yeah,
so
the
yes,
you
couldn't
can
automate
graphite,
but
it's
it
was.
It's
still
a
little
bit
more
work
from
what
I've
seen
and
previous
we
don't.
We
didn't
have
like
the
Prometheus
itself,
didn't
have
to
do
much
work
and
it
can
handle
significant
amount.
More
data.
You
know
the
the
Prometheus
server
that's
handling
the
mothership
is
ingesting.
I,
forget
I,
think
it
was
about
it's
it's.
It's
actually
only
about
600,000
time
series
for
that
server,
but
and
it's
but
its
handling
about
50,000
samples
per
second
and
it's
idle
yeah.
A
A
A
A
A
A
A
A
That's
a
good
question:
what
are
what
is
the
bulk
of
all
that
data?
It
is
yeah
a
lot
of
a
lot
of
time
series
for
for
those
endpoint
metrics,
because
I
think
there's
something
like
30
or
40
controllers
with.
However
many
actions-
and
you
multiply
that
because
we
have
a
time
series
for
every
single
rails
server-
that's
running,
so
it's
actually
quite
a
lot
of
granularity.
A
A
Basically,
anything
that
we
put
on
a
dashboard
eventually
becomes
a
recording
rule,
so
the
dashboard
just
pulls
from
that
that
pre-recorded
data,
which
makes
the
dashboard
super
fast,
but
really
600,000
time.
Series
seven
is
not
actually
that
much
for
Prometheus,
it's
totally
an
ax
normal
normal
load
load
level
for
Prometheus,
although
we
do
run
on,
we
do
run
our
Prometheus
on
bare
metal.
A
So
we
have
pretty
beefy
32
core
servers
with
64
gigs
of
RAM
to
run
that
from
efya
server,
but
even
if
we
didn't
have
that,
it's
still,
we
kind
of
we
just
leave
it
totally
on
tuned
and
it's
still
only
using
17
gigs
of
ram.
So
if
we
were
to,
if
we
were
to
squeeze
that
down,
we
probably
could
cut
that
resource
usage
in
half
it's
just
we
have.
We
have
there's
no
motivation
there.