►
From YouTube: Alerting and Anomaly Detection – Best Friends Forever? - Björn Rabenstein, Grafana Labs
Description
Alerting and Anomaly Detection – Best Friends Forever? - Björn Rabenstein, Grafana Labs
When Prometheus became publicly known starting in 2015, the Prometheus developers expected many questions. But one surprisingly stuck out: “Can you do anomaly detection?” Somehow, everyone expected a next-generation monitoring and alerting system to venture into anomaly detection. PromQL is powerful enough to support fundamental building blocks of anomaly detection, but the general direction of Promethean alerting is, in a way, exactly the opposite: Towards confident, non-noisy alerts based on your SLOs. In this talk, Beorn will share a few stories from the receiving end of the pager and why it is almost always a bad idea to put anomaly detection at the other end. He will talk about the “proper” Promethean way of alerting (including its limitations) and where anomaly detection (or even machine learning) might have its place in it after all.
A
I
should
say
that
julian
was
supposed
to
be
here.
If
you
have
read
the
program
in
advance
on
short
notice,
he
unfortunately
couldn't
make
it
here,
so
I
just
made
up
a
talk
that
I
always
wanted
to
give,
but
never
found
the
right
occasion
for
it.
So
this
is
something
I
wanted
to
talk
about
for
a
long
time,
but
the
talk
itself
is
completely
improvised.
I
just
hacked
together
some
slides
yesterday
in
the
hotel
yeah.
So
that's
essentially
it
julian
wanted
to
talk
about
new
stuff
in
the
prometheus
world.
A
We
will
still
have
a
very
compressed
version
of
that.
In
the
maintainer
session
on
thursday
2
30.,
it's
called
prometheus
intro
and
deep
dive
that
will
be
an
intro
and
then
there
will
be
deep
dive
with
the
news
it's
very
compressed,
but
julian
also
told
me
he
will
record
his
talk
anyway
and
publish
it
on
his
blog
or
somewhere.
So
just
follow
him
on
twitter
or
something,
and
you
will
enjoy
the
talk
anyway,
okay
cool,
so
my
topic
alerting
and
a
normal
detection
best
friends
forever.
A
short
answer
no
and
talk
done
right.
A
I
guess
we
have
20
more
minutes.
I
should
be
more,
I
should
have
a
more
nuanced
view
on
the
topic:
let's
go
back
to
2015.
what
happened
in
2015,
you
should
be
fans
right.
This
was
when
prometheus
got
publicly
announced.
Remember
prometheus
was
always
in
an
open
source
reaper
on
github
ever
since
november,
2012,
meaning
we
are
kind
of
approaching
a
10-year
anniversary,
which
is
pretty
incredible,
but
2015
was
the
year
when
we
make
a
bit
of
noise
and
prometheus
became
very
popular,
quite
quickly,
which
we
didn't
really
expect.
A
Can
you
do
anomaly
detection
because
back
then
in
2015
the
hotness
was
anomaly
detection
and
whenever
somebody
claimed
to
create
a
next-gen
monitoring
and
alerting
system.
Obviously
this
must
have
to
do
with
anomaly
detection
right
and
we
were
like
what
we
were
kind
of
surprised
and
I'll
explain
why
one
thing
is
seven
years
later
and
normally
detection
is
still
the
hotness.
A
I
have
to
read
this
that
way
so
marketing
language.
Imagine,
for
example,
a
food
delivery,
app
that
has
lots
of
usage
at
lunch
and
dimmer
times,
but
it's
pretty
quiet
in
the
early
hours
of
the
morning.
The
same
threshold
doesn't
work
well
for
both
scenarios
and
could
lead
to
missing
incidents
and
or
noisy
alerts.
What,
if
we
could
learn
from
our
metrics
in
the
past
and
create
alerts
that
adapt
to
our
data
and
context
over
time?
Wouldn't
be
that
great?
A
So
I
I'm
carefully
navigating
the
competitive
landscape
here
I
don't
want
to
put
blame
on
any
of
the
competitors
of
my
employer,
which
is
cofounder
labs,
as
you
might
have
seen
on
the
first
slide.
So
I
took
our
own
marketing
language,
so
my
honest
answer
is,
then
you
would
almost
certainly
miss
incidents
and
get
noisy
alerts.
That's
the
sad
truth.
A
Why
is
that
the
case?
First
of
all,
outages
are
not
necessarily
looking
like
anomalies,
especially
nowadays
you
might
have
a
high
availability
service,
three
nines,
four
nines.
If
you
just
have
fractions
of
a
percent
of
an
error,
you
are
already
out
of
your
slo
but
patterns.
I
mean
it
looks
normal,
so
your
normal
detection
system
might
not
even
see
that
you're,
like
you're,
burning
your
arrow
budget,
slowly
yeah.
So
you
will
miss
outages
if
you
rely
on
anomaly
detection
for
alerts.
A
A
There
is
my
favorite
example:
the
world
cup
final
here
in
spain,
if
I
say
world
cup
final,
you
know
what
I
mean
right
in
most
countries
of
the
world.
You
know
what
I
mean.
There
are
a
few
countries
where
world
cup
final
means
something
else.
I
mean
this
year
right,
this
sports,
which
personally
I
don't
really
like,
but
most
people
like
it,
so
the
world
cup
final
happens
and
most
of
the
world
is
watching
that
and
you
might
be
the
engineer
on
call-
and
you
also
want
to
watch
it.
A
But
since
your
alerting
is
based
on
anomaly
detection,
every
world
cup
final,
every
four
years
again,
you
get
paged
during
the
world
cup
final,
because
your
traffic
drops
like
70
of
your
traffic,
drops
away.
Everyone
is
watching
the
world
cup
final.
You
want
to
watch
the
world
cup
final.
No,
you
have
to
act
the
page
and
realize
this
traffic
drop
is
actually
fine.
People
are
just
not
using
my
service,
it's
not
an
outage,
so
really
bad.
You
will
never
watch
the
world
cup
final,
there's
even
a
little
anecdote
internally
at
google.
A
I
don't
know
if
it's
true,
but
it's
so
nice
that
I
like
to
tell
it
because
where's
google,
from
it's
a
country
called
the
united
states
of
america.
They
don't
watch
the
world
cup
final
right
and
so
they're
engineers
and
at
least
initially,
when
they
were
like
more
euro
centric.
They
didn't
know
there
was
a
world
cup
final.
They
just
saw
70
traffic
drop
watch.
Something
must
be
wrong
for
sure
right.
A
So,
even
if
humans
do
the
anomaly
detection
and
they
are
supposed
to
be
smarter
than
your
machine
learning,
they
will
still
have
false
positive
and
panic
because
the
traffic
has
dropped
yeah.
So
another
thing
is
that
anomalies
are
just
anomalies.
They
don't
judge
you
normally
right.
So
if
you
are
a
crappy
startup,
you
are
not
making
any
profits.
Your
product
is
unpopular.
A
Suddenly
your
product
becomes
popular
traffic
goes
up.
You
should
be
happy,
but
it's
an
anomaly
right.
So
you
get
paged
because
you
have
success
or
you
were
just
like
duct
taping,
your
side
together,
you're
most
of
the
time
down
and
suddenly
you're
up,
because
you
did
proper
work.
You're
up.
It's
an
anomaly.
You've
never
been
up
for
like
longer
patches
of
time
and
normally
normally
normally,
but
no
right.
It's
not
really
an
outage.
A
We
kind
of
knew
that
anomaly.
Detection
is
problematic
for
alerting,
but
we
really
didn't
have
it
on
our
radar.
We
had
something
else
on
our
radar,
which
is
this
philosophical
alert,
philosophical
discussion,
symptom-based
alerts
versus
cause-based
alerts.
That
was
the
hotness
for
us
in
our
little
prometheus
bubble.
I
mean
it's
still
like
you
have
might
have
heard
about
this
discussion.
A
Traditionally,
you
would
monitor
packet
loss
on
the
network.
I
don't
know
servers
that
go
down
this
that
fail
right.
This
is
traditional
monitoring
and
this
is
what
we
call
cause-based
alerting
in
distributed
complex
systems.
First
of
all,
you
have
so
many
nodes
that
this
will
happen
all
the
time.
Luckily,
your
systems
are
designed
to
tolerate
that
which,
in
turn
means,
if
you
alert
based
on
causes,
you
will
get
very
noisy
alerts
and
you'll
get
woken
up
in
the
middle
of
the
night
because
of
something
your
system
can
cope
with.
A
So
what
you
should
do
is
symptom-based
alerting,
so
you
should
alert
on
conditions
that
actually
mean
that
your
users
have
a
bad
experience.
If
you
have
an
sla
signed
or
you
have
set
lclos,
you
actually
know
what
your
service
should
do,
so
you
can
alert
directly
on
that
which
is
called
slo-based,
alerting,
that's
another
new
hotness.
A
Our
discipline
has
the
habit
of
like
riding
the
hype,
so
nowadays
I
mean
still
an
only
detection
is
a
hype
but
also
slo-based
alerting,
so
people
have
like
overblown
expectations
there,
but
in
principle
it's
a
very
good
approach
to
do
like
symptom-based,
alerting
as
a
low-bass
alerting.
So
if
you
get
paged,
it
should
be
a
real
issue
which
has
to
be
dealt
with
now.
This
is
why
you
got
woken
up,
so
we
were
fighting
kind
of
this
battle,
symptom-based,
alerting
versus
course-based,
alerting
and
now
normally
detection
comes
into
the
game.
A
Where's
that,
even
here
anomalies,
could
be
symptoms.
In
rare
cases,
we
will
talk
about
this.
A
bit
later.
Anomalies
could
be
causes
or
correlated
to
causes,
but
most
anomalies
are
neither.
There
are
neither
symptoms
nor
causes,
and
why
should
you
base
your
alerting
on
them
like
this
was
for
us
this
was
so
off
the
radar
that
we
didn't
even
think
about
that
and
the
rest
of
the
world.
They
ask
us
how
we
do
normally
detection
prometheus
now,
there's
hope
there
is.
A
You
could
argue
there
are
cases
where
you
can
use
anomaly
detection
for
alerting.
Strictly
speaking,
that's
the
case.
If
anomalies
are
the
symptoms,
so
there
are
very
rare
systems
were
anomalies,
are
a
symptom
of
a
failure.
There
are
systems
where
you
just
want
them
to
run
very
normally
all
the
time,
it's
just
hard
to
come
up
with
really
good
examples.
I
always
think
like
of
medical
stuff
like
if
my
heart
rate
is
too
low
or
too
high.
That's
an
anomaly
and
it
shouldn't
always
happen
whatever
like.
A
However,
the
more
like
the
worst
signal
noise
ratio
you're
willing
to
accept
because
the
stakes
are
so
high,
the
more
anomaly
detection
is
something
that
might
make
sense.
So
stakes
are
really
high.
If
you
run
an
airplane
or
nuclear
power
plant
right,
you
could
kind
of
argue,
you
should
have
an
error
budget
and
in
reality
you
have
like
nuclear
power
plants
blow
up.
Planes
fall
from
the
sky,
but
you
shouldn't
really
tell
your
like
passengers
or
the
users
of
your
electricity
yeah,
there's
a
kind
of
0.1
chance.
A
This
will
blow
up
or
fall
from
the
sky.
You
kind
of
try
to
keep
this
really
really
low
like
so
in
this
those
cases,
you
also
want
to
know
if
something
is
going
to
happen
before
it
happens,
and
it's
it's
so
expensive
if
it
happens
that
you
are
willing
to
accept
noisy
alerts
and
then,
of
course,
if
anything
is
weird
in
the
airplane,
you
want
to
know.
If
anything
is
weird
in
your
powerplant,
you
want
to
know.
A
Even
if
90
of
those
alerts
are
false
positives,
you,
you
don't
want
to
operate
your
whatever
search
engine
in
that
way
right
because
nobody
dies.
If
your
search
engine
throws
a
few
errors,
I
also
think
about
like
fraud,
detection
of
something
where
normally
detection
might
be
cool
because,
with
like
intrusion,
fraud,
you
have
an
attacker
that
deliberately
tries
to
not
trigger
you
alerting
right.
So
you
kind
of
yeah
you
you
have
an
intelligent
antagonist
and
then
you
might
say
anything
that
is
not
normal.
A
That
looks
weird
should
ring
a
bell,
and
maybe
that's
a
good
thing,
but
I
also
remember
I
was
a
couple
of
years
ago
whenever
I
travel
to
another
country,
first
transaction
with
my
credit
card,
it
got
blocked
because
anomaly
right,
I
never
travel
to
france
and
suddenly
I
do
and
the
whole
point
of
a
credit
card
is.
I
can
travel
to
another
country
put
in
my
credit
card
without
cash
and
get
money
and
the
bank
thought
that's
an
anomaly
and
it
should
block
my
credit
card.
So
even
there
you
have
to
use
it
responsibly.
A
Okay,
there's
another
thing,
and
this
is
way
more
relevant
for
most
of
us,
which
is,
we
should
actually
think
of
alerts
in
a
wider
way
than
just
pages
and
for
prometheus
people,
which
you
probably
all.
Are
you
are
familiar
with
that,
because
what
we
call
an
alerting
rule
in
prometheus
isn't
necessarily
necessarily
a
rule
that
creates
a
page.
We
have
all
kinds
of
alerts
right
and
but
the
lingo
is
really
different
and
everyone
uses
different
terminology
and
then
there's
the
sre
book.
A
The
famous
google
blue
s,
rebook,
which
has
a
terminology
established,
which
I
find
a
bit
weird,
but
they
use
alerts
synonymous
synonymous
with
pages,
and
I
call
this
soundcloud
lingo
because
that's
where
prometheus
was
first
used-
and
this
is
what
I
personally
try
to
establish
like:
let's
not
call
pages
alerts,
because
that's
the
general
term,
let's
call
it
pages
right
and
then
you
can
in
prometheus.
You
can
attach
labels
free
form,
so
we
use
the
severity
label.
You
can
use
something
else.
You
can
use
a
different
terminology.
A
For
me,
this
is
really
inclusive
here
and
open
minded,
but
it
comes
at
the
price
that
terminology
might
change
so
severity
critical.
This
is
something
that
you
act
on
immediately.
It
should
only
ever
be
based
on
symptoms,
and
it
goes
to
your
pager
now.
Another
alert-
and
I
like
the
word
from
the
sre
book,
which
is
tickets,
so
I
also
try
to
establish
it
at
soundcloud,
but
most
people
call
it
an
email
alert
because
they
send
it
to
email,
although
it
should
be
in
an
issue
tracker.
A
A
Here
you
should
act
on
it
eventually,
but
you
can
do
it
during
work
time.
If
you
have
packet
loss
in
your
network,
you
should
not
ignore
that
it
will
eventually
create
something
or
some
real
outage,
or
it's
definitely
something
bad,
but
nobody
has
to
be
woken
up.
The
network
people
who
are
dealing
with
that
they
can
get
an
alert
during
their
work
hours.
So
that's
what
you
should
do
here.
It
could
even
be
based
on
courses,
and
then
this
is
less
known.
What
I
like,
I
call
it
information
alerts.
A
A
I
like
those
alerts,
informational
alerts
because
they
make
sense
if
you
have
a
real
outage
and
you
want
to
find
the
cause
contrary
to
common
belief,
a
monitoring
and
alerting
system
is
not
only
there
to
tell
you
if
something
is
broken.
It
also
helps
you
to
find
out
what
is
broken
and
those
alerts
help
you,
because
they
tell
you
if
something
is
weird
which
smells
like
anomalies
right.
So
this
is
many
alerts
actually
that
you
might
have
in
your
system
and
they
are
even
paging
or
creating
a
ticket.
A
But
if
you
see
that
something
else
is
going
wrong,
you
should
know
right
so
information
alerts-
and
I
think
anomaly-
detection
has
a
great
place
here,
and
this
is
actually
what
I
think
this
whole
like
machine
learning,
blah
blah
blah
blah
blah
blah
buzzwords
where
this
actually
makes
sense
and
is
a
billion
dollar
market,
because
that's
really
what
you
do
if
you're
troubleshooting
right,
you
want
to
see
what's
wrong,
you
get
a
page
because
you
have
a
real
outage.
It's
not
a
normally
based
it's
based
on
real
measurement
against
your
slos.
A
But
what
what
do
you
do?
Usually
I
mean
we
all
have
this
intuition.
What
has
changed
ten
minutes
ago
and
the
audit
started
was
the
new
software
version
pushed?
Did
the
traffic
spike
did
something
the
network
go
wrong
and
it
would
be
so
nice
if
you
could
like
star
trek
style.
Ask
the
computer
please
list
all
the
things
that
have
changed
in
the
last
10
minutes
or
perhaps
in
the
hour
running
up
to
it
and
that's
what
we
humans
do
by
what
we
call
staring
at
dashboards
like
this
is
an
anti-pattern.
A
You
just
look
through
all
your
dashboards
and
you
might
find
something
would
be
so
cool
if
machines
could
do
this
for
us
and
that's
what
machine
learning
can
actually
do-
and
this
is
also
in
it
inverts
the
requirement
for
signal
noise
ratio.
If
90
of
the
pages
you
get
are
false
positives,
you
will
hate
your
alerting
system
if
you
are
fighting
an
outage
and
you
have
no
clue
what's
going
on
and
your
machine
learning
system
presents
you
10
theories,
what
could
go
wrong
and
one
of
them
is
right.
That's
awesome
right.
A
I
only
have
to
test
10
theories
and
I
will
find
the
real
cause
of
the
outage
great
right.
So
I
think
this
is
where
we
really
have
value.
Okay,
let's
come
back
to
2015..
Everyone
is
asking
us:
can
you
do
a
normal
detection?
We
slowly
understood
what
people
actually
want.
We
try
to
explain
why
you
shouldn't
use
it
for
alerting,
but
we
also
told
them
listen.
Prom
ql
is
super
powerful.
Of
course
you
can
do
a
normal
detection.
All
the
building
blocks
are
there
and
brian.
A
Here
you
can
see
2015.
He
wrote
this
blog
post
on
the
prometheus
blog.
It's
still
there
explaining
how
you
can
do
a
normal
detection
with
prometheus.
So,
finally,
we
could
claim
we
can
do
a
normal
detection
and
it
might
even
make
sense.
This
is
like
the
example
query.
In
the
blog
post
and
you
can
see
there
are
the
buildings
block,
we
can
do
an
average.
We
can
compare
to
the
standard
deviation
of
latencies.
This
is
between
instances.
So
that's
a
reasonable
thing
right.
A
You
load,
balancer,
shoot
load
balance
stuff
reasonably
and
if
a
single
instance
is
more
than
two
standard
deviations
away
in
its
latency
from
the
others,
probably
something
is
wrong.
It
might
not
be
a
problem
that
you
should
page
on,
but
if
you
get
paged
based
on
your
nice,
slow
base,
symptom-based
alerts-
and
you
see-
oh
yeah-
one
instance-
is
much
slower
than
the
others.
Great
right,
perhaps
noisy
neighbor.
I
don't
know
broken
note.
A
This
is
helpful,
then
right
in
the
blog
post.
This
query
is
refine
the
law
to
make
it
less
noisy,
and
all
of
that
you
can
look
it
up.
There's
also
a
lightning
talk
on
this
during
this
prometheus
day
later
today
about
anomaly
detection,
where
you
probably
see
all
those
tricks
issue,
so
you
can
do
this,
but
all
of
this
is
rule-based
right.
What
the
people
back
then
were
all
looking
for
buzzword,
machine
learning
right
so
I
want
to
close
this
with
cartesian
coordinates.
Quadrants,
that's
always
cool
right,
so
this
is
a
normally
detection.
A
The
method
you
use,
which
could
be
more
on
the
machine,
learning
part
or
more
on
the
rule-based
part
and
the
purpose
which
could
be
alerting
or
what
I
call
troubleshooting
right
and
prometheus,
is
here
right.
It's
in
this
quadrant
prometheus
is
always
rule-based,
and
it's
for
troubleshooting.
It's
what
we
want.
Like
you
could
use
it
for
alerting,
of
course,
there
are
prometheus
rules
for
normal
detection,
but
we
kind
of
philosophically
don't
want
this.
There
are
again.
This
is
all
with
exceptions.
There
are
ways
where
this
might
make
sense.
A
For
example,
I
would
never
just
alert
on
a
traffic
drop
because
world
cup
final
right,
but
you
could
be
pretty
sure
that
there
are
limits
where
you
say.
If
my
traffic
drops
99,
that's
weird,
even
during
the
world
cup
final
like
you
could
make
up
rules
where
you're
reasonably
confident
that
something
is
wrong.
But
then
you
could
also
say
it's
not
even
an
anomaly,
it's
something
where
you
just
know.
This
must
be
wrong.
It's
kind
of
more
symptom,
but
anyway,
like
this,
is
careful
alerting
with
machine
learning,
just
don't
right,
dangerous
territory.
A
It's
hardly
ever
useful
exceptions,
power
plants,
fraud,
detection,
maybe
right
so.
But
this
is
an
interesting
area
here.
You
want
machine
learning,
and
this
is
essentially
star
trek
computer
style
right
where
captain
picard
still
has
to
ask
the
computer
the
right
questions,
but
the
computer
will
do
all
the
hard
work
of
analyzing
and
correlating
and
everything.
A
This
is
just
nothing
you
can
do
in
prometheus
and
I
profit
like
my
prophecy
is
that
prometheus
will
never
have
those
machine
learning
parts
not
just
because
we
want
to
be
focused
on
our
prometheus
as
well,
but
also
because
your
machine
learning
should
look
at
all
signals
and
prometheus
is
looking
at
metrics,
but
you
have
so
many
other
signals
in
your
modern
monitoring
systems
that
you
should
put
them
all
into
your
machine
learning
system.
So
what
machine
learning
systems
do
they
query
prometheus?
A
They
can
retrieve
the
the
data,
there's
a
query
api
right
and
then
they
can
process
this
and
do
this
and
then
maybe
it
will
help
you
troubleshooting.
I
think
there
there
we
will
see
the
the
real
progress,
the
real
money
and
the
real
convenience
coming
from,
but
not
not
here,
right,
okay,
I
think
that's
it.
We
have
a
few
minutes
for
questions.
B
A
Can
you
hear
me
now
you
can
hear
me,
I
don't
know
of
any
in
particular,
but
I
talk
to
the
people
like
I'm,
not
following
so
much
the
marketing
side
of
this,
but
I
talk
to
the
engineers
who
are
doing
machine
learning,
including
the
machine
learning
experts
in
my
own
company,
because
I
always
I'm
always
keen
on.
Please
don't
tell
people
that
they
get
automatic
alerting.
They
have
never
to
think
about
thresholds
again,
because
this
is
just
wrong
right
and
they
all
tell
me
yes,
we
understand.
A
A
Troubleshooting
part
they
want
to
support
you
and
I
think
they
can
I'm
just
not
sure
if
somebody
has
something
that
has
already
in
the
market
that
actually
works,
but
I
know
that
the
people
are
working
on
it.
So
if
it's
not
already
on
the
market
soon
it
will
be
there,
and
I
guess
multiple
vendors
will
offer
that,
and
I
hope
it
will
be
good.
B
E
A
Yeah,
you
should
do
alerting,
of
course,
but
not
based
on
anomaly
detection
right.
So
the
the
usual
like
the
marketing
story
about
is
that
with
which
you,
which
you
could
see
from
the
example
from
grafana.
But
everyone
is
telling
you
that
that
if
you
just
have
thresholds
static
thresholds,
it
won't
work.
This
is
a
bit
unfair
because
modern
monitoring
systems,
they
also
teach
you
to
not
just
use
a
static
threshold.
A
They
tell
you
you
do
something
smarter,
look
at
your
slos.
So
and
even
then
you
say
you
get
like
an
error
rate
in
your
slo,
which
is
already
a
bit
smarter
because
it's
like
relative
to
traffic,
but
even
that
isn't
really
what
you
do.
You
do
like
error.
Like
you,
you
alert
on
the
speed.
You
burn
your
arrow
budget,
there's
like
an
essay
workbook.
The
second
essay
book
chapter
five,
I
think
and
there's
I
myself
did
something
back
at
soundcloud.
A
There's
a
blog
post
then
did
the
same
thing
at
grifana
labs
and
it's
in
podcast
and
all
over
like
this
is
what
we
call
slo-based
alerting.
It's
like
it's
real
science
like
rocket
science
in
a
way,
but
it's
very
elaborate
and
it
has
its
problem.
There
are
also
a
lot
of
talks
these
days.
Why
slos
aren't
like
the
silver
bullet
for
everything.
Nobody
ever
said
that,
but
it
helped
so
much,
and
there
was
a
reason
like
sorry,
shameless
blog
grafana
has
a
new
podcast.
A
It's
called
big
ten
there's
an
slo
episode
where
they
interviewed
me
and
matthias
libel
right.
Who
is
purpose
in
the
room.
I
don't
know
so.
Matthias
also
has
a
lightning
talk
on
prometh
today.
I
think
about
this
as
a
low-bass
alerting,
so
watch
this
water
all
the
day.
All
the
talks
today,
of
course,
but
this
is
so
kind
of
ingenious,
because
it
really
alerts
you
in
a
non-noisy
fashion
and
tom
wilkie,
which
is
like
my
boss's
boss
at
grafana.
A
He
he
said
this
so
nicely
like
how
they
had
very
noisy
pages
initially,
and
then
I
came
along
and
said.
We
have
established
procedures
do
this
and
it
was
like
it
was
day
and
night
right.
Of
course
there
are
shortcomings
and
it's
not
the
perfect
solution,
but
it's
so
much
better
than
what
you
initially
and
if
you,
if
you
ever
get
into
this
direction,
you
also
realize
why
did
I
even
ask
for
like
automatic
alert
thresholds,
push
the
button
and
never
worry
again
like
it's?
You
should
know
your
system
better
right.
A
D
I'm
wondering
you
said
that
we
supposed
to
avoid
noisy
systems
no
noisy
alerting,
but
sometimes
it
depends
on
context.
Context,
like
you
said,
with
this
example
from
google,
I'm
wondering
if
you
have
any
advice,
how
to
avoid
noisy,
alerting
in
the
same
time,
keep
the
context
of
averting
and
keep
other
things
that
are
important,
which
depends
on
time
or
any
other
things
that
are
outside
of
the
system.
A
Yeah,
I
mean
it
really
like
there's
no
silver
bullet.
You
cannot
stop
thinking.
You
have
to
what
you
call
context
right.
You
have
to
think
about
your
system.
I
think
most
of
those
like
online
serving
system.
Most
of
us
run
probably
read
chapter
five
in
the
essay
book,
go
through
the
blog
posts
and
podcasts.
This
is
probably
what
you
want
right,
don't
do
it
don't
apply
it
without
thinking,
because
if
you
run
a
nuclear
power
plant,
you
don't
want
to
have
a
0.1
blow
up
rate
per
month
or
something
right.
A
Obviously,
but
if
you
run
just
a
normal
service
that
might
be
just
fine,
it
really
depends
what
what
you're
doing
and
then
you
have
to
pick
the
right
method,
and
it's
like
can't
like
give
you
the
one
three,
whatever
rules
for
that
to
find
out
what
it
is.
It's
it's
a
nuanced
decision,
of
course,
but
most
of
the
time
don't
do
a
normal
detection
for
your
paging.