►
From YouTube: Alerting in the Prometheus Ecosystem: The Past, Present and Future - Josue (Josh) Abreu
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Alerting in the Prometheus Ecosystem: The Past, Present and Future - Josue (Josh) Abreu, Grafana Labs
One of the most important functionalities of Prometheus is being able to alert based on your metrics. The Prometheus Alertmanager is a critical piece of cloud native observability, and in this talk, Josue wants to share a bit more of its past, present and future. About a year ago, he set out on a path to improve scaling in the Cortex Alertmanager component, then he decided to include the Alertmanager within Grafana to continue fostering open source collaboration. For the future, his plan is to take all the good parts of what he learned on this journey back to the Prometheus Alertmanager thus going full cycle. He’ll cover: The Prometheus Alertmanager and the benefits of its modular architecture (past) The benefits of the new architecture of the Cortex Alertmanager: Like Cortex but for Alerts (past) Inclusion of the Prometheus Alertmanager within Grafana (present) The future of the Prometheus Alertmanager (future).
A
A
All
right:
well,
the
good
thing
about
this
is
there's
not
going
to
be
any
questions.
So
lucky
me
good
afternoon.
Kubecon,
I'm
super
excited
to
be
here,
welcome
to
alerting
in
the
prometheus
ecosystem
the
story
of
the
past
present
and
the
future.
If
we're
going
to
spend
the
next
30
minutes
together,
we
might
as
well
just
get
to
know
each
other
a
little
bit
better,
I'm
josue
commonly
known
as
josh
or
god.
A
Josh,
yes,
is
a
reference
to
the
1993
advertisement
of
got
milk
when
I'm
not
coding,
I'm
usually
snowboarding,
which
wasn't
the
easy
sport
to
catch
on.
Given
I
come
from
the
caribbean,
I
work
as
a
software
engineer,
leading
the
alerting
team
at
grafana
labs
with
a
bunch
of
other
folks
from
the
prometheus
community,
which
you
might
know
about,
such
as
like
tom
wilkie,
bjorn
ravenstein,
ganesh,
bernakar
kanesh
has
to
talk
just
after
this
one
in
this
room.
A
A
Before
we
begin,
I
want
to
see
a
show
of
hands
who
here
is
familiar
with
prometheus
and
the
alert
manager,
cortex
and
mimir
cool
cool,
a
sea
of
hands
for
the
first
one
and
not
so
much
for
the
second
one.
Hopefully
I
get
to
explain
a
little
bit
about
what
mimir
and
quartus
handle
alerts
today
and
you
leave
the
room
with
that
knowledge.
A
Okay,
great
I'm
going
to
be
talking
about
the
past
the
present
and
the
future
of
the
alert
manager,
and
I'm
going
to
do
that
with
four
topics.
Has
groundwork
I'll
review
the
alert
manager
and
remind
you
of
its
features.
Then
I'll
guide
you
through
how
we
achieve
horizontal
scalability
for
the
alert
manager
in
cortex
and
mimir.
A
Moving
to
moving
to
the
present
I'll
show
you
why
grafana
now
includes
the
alert
manager
to
power
its
alerting
system
and
with
eyes
on
the
future,
I'm
going
to
share
my
alert
manager,
wishlist
and
what
I
hope
to
contribute
back,
and
if
I
can
do
it,
so
can
you
so,
let's
begin
with
the
past
originally
started:
has
a
one
file
program
by
julius
boats
and
matt
proud?
The
alert
manager
is
a
binary,
that's
in
charge
of
receiving
alerts
generated
by
prometheus
or
any
other
client
application
there.
A
A
This
makes
it
really
simple
to
use
them
as
libraries
within
your
projects
before
we
delve
into
this
function,
let's
break
down
what
an
alert
is,
as
you
can
see,
from
what
we
have
here
on
screen.
This
is
the
alert
reception
endpoint
of
the
alert
manager
on
alert
it's
composed
of
a
start
timestamp
to
determine
when
it
should
dealer
be
active,
an
end
time
stump
to
indicate
when
should
the
alert
expires,
two
sets
of
key
value
pairs
labels
which
identify
the
alert
and
then
annotations
that
provide
a
way
to
include
additional
information.
A
Think
of
like
the
place
where
you
would
put
a
runbook
or
a
better
description
of
your
alert.
Finally,
a
link
back
to
the
query
and
the
graph
that
generated
the
alert
called
generator
url,
with
this
definition
in
mind.
Let's
move
on
to
the
juice
in
the
case
of
an
alert
storm
and
tens
of
thousands
of
alerts
being
generated,
you
certainly
don't
want
a
one-to-one
mapping
and
notification
time.
Instead,
the
alert
manager
is
capable
of
grouping
alerts
based
on
label
criteria.
A
Moving
on
to
the
duplication
by
design,
the
alert
manager
is
a
it's
a
highly
available
mode
and
it's
highly
available
mode.
It's
an
ap
system.
It's
meant
to
be
available
and
partition
tolerant.
To
achieve
this,
it's
expected
that
alert
generators
send
all
alerts
to
each
alert
receiver,
even
if
duplicated.
A
However,
under
normal
operation,
you
don't
want
duplicate
notifications.
The
alert
manager
uses
a
hash
of
the
alert
labels
to
determine
if
we
already
notified
for
that.
Moving
into
routing.
We
all
know
it's
possible
to
have
label
based
granularity,
based
on
where
you
want
to
send
the
alert
on
a
three
like
structure.
A
But
we
reached
a
point
where
vertical
scaling
was
not
enough,
and
this
is
what
the
upstream
version
supports
we
had
to
bring
heavy
machinery
into
play.
Before
I
begin,
I
want
to
clarify
that
when
I
say
alert
manager
instance,
as
you
see
up
here,
I'm
referring
to
the
importer
alert
manager
that
belongs
to
a
single
tenant
and
when
I
say
replica,
I'm
referring
to
the
multi-tenant
alert
manager,
the
cortex
and
memory,
one
attendance
alert
manager,
the
one
on
the
corner
right
here
right.
A
Gossip
be
a
hashicorp's
member
list,
it
gossips
silences
and
a
notification
log
between
replicas
to
achieve
eventual
consistency,
ensuring
your
alerts
won't
fire
two
times,
but
I
must
note
that
alerts
themselves
are
not
part
of
the
state.
That's
gossip
alerts.
Generators
generators
must
not
load
balance
traffic
and
instead
deliver
alerts
to
all
the
different
replicas
of
the
alert
manager.
A
A
In
this
example,
tenant
number
one
it's
present
with
all
of
their
data
on
each
alert
manager,
instance
and
replicas
same
tenant
end
times
I'll
admit
that
vertical
scaling,
the
three
replicas
that
we
ran
at
grafana
labs
worked
pretty
well
to
us
for
a
very
long
time.
But
as
the
rafana
has
a
graffana
cloud
offering
kicked
off,
we
started
to
a
point
where
reaching
10x
the
current
load
was
going
to
be
a
heavy
challenge.
A
It
was
the
right
time
to
bring
horizontal
scaling
to
the
alert
manager
when
tom-
and
I
designed
this-
we
started
off
with
a
set
of
requirements,
as
every
engineer
should
do
survive.
Machine
failure
make
sure
that
a
single
replica
crack
crashing
or
exiting
abruptly
should
not
cause
any
external,
visible
time.
Downtime
or
failure
to
deliver
notifications,
eventual
consistency
for
reads
and
writes
user
users
should
have
an
eventually
consistent
view
of
all
the
alerts
currently
firing
and
under
active
notification.
A
With
that
we
decided
to
split
this
grandious
design
into
four
areas:
routing
and
sharding,
persistent
and
state
replication
and
consistency
and
the
service
architecture
of
itself
for
routing
and
charting
all
of
the
cortex
cells.
All
the
cortex
and
memory
services
rely
on
hash
rings.
Hash
rings
are
distributed
consistent,
hashing
scheme
that
is
widely
used
by
cortex
and
mimir
for
shorting
and
replication.
A
A
You
already
know
that
it
persists
notification,
state
and
configuring
silences
to
dig
to
this
after
after
a
certain
amount
of
time.
This
is
mostly
to
avoid
what
we
call
right,
amplification
right
with
horizontal
scaling.
We
needed
to
move
the
state
around.
Has
the
number
of
replicas
either
grew
or
shrunk,
and
we
also
needed
to
say
that
this
this
state
across
restart
or
or
has
we
rolled
out
new
replicas?
A
A
The
alert
manager
replicates
the
notification
state
between
the
replicas
to
ensure
notifications
are
not
sent
more
than
once.
Remember
that
firing
twice
that
I
said
before
before.
Moving
to
moving
to
a
model
where
the
alert
manager
instance
for
a
given
tenant
are
only
present
on
a
subset
of
the
replicas,
we
have
to
decide.
How
are
we
going
to
keep
this
replicas
in
sync?
I
don't
think
anyone
here
enjoys
getting
paid
twice.
A
Managing
cluster
membership
is
the
jobs
of
hashicorp's
memberless
in
the
upstream
implementation.
However,
in
cortex
and
mimir,
the
hash
ring
is
the
one
responsible
for
that
function.
We
settle
on
keeping
the
same
notification
log
and
read
silence
replication
algorithm,
while
that
the
upstream
alert
manager
was
built
upon.
But
the
difference
here
is
that
we
were
going
to
rely
on.
Grpc
has
the
transport
of
the
replication
messages
and
the
hash
ring
for
knowing
where
to
send
what
effectively
dropping
the
direct
implementation
of
member
lists.
A
I
got
to
admit
that
this
worked
out
pretty
well
for
us,
despite
seeing
it
in
paper.
At
the
end
of
the
day
we
didn't
have
to
rip
any
of
the
internals
of
the
alert
manager
and
the
only
required
change
that
we
had
to
make
upstream
was
turn
the
clustering,
the
clustering
mechanism
into
an
interface
and
the
code,
so
that
we
could
override
it
within
cortex
and
memory.
I
would
call
that
a
massive
win
for
open
source,
the
last
piece
of
the
puzzle
was
the
service
architecture
to
implement
the
sharding
strategy.
A
A
The
idea
here
is
to
have
this
sort
of
alert
manager,
distributor,
which
is
the
same
as
kind
of
the
cortex.
This
cortex,
remember,
distributor
at
the
end
of
the
day
has
the
first
step
in
the
request
reception
once
received.
The
component
is
in
charge
of
sending
the
alerts
in
parallel
to
all
the
other
replicas
and
instances
to
fulfill
the
request.
A
A
So
now
with
everything
in
place,
this
is
how
it
looks.
Rights
of
alerts
highlighted
in
red
are
first
sent
to
my
to
any
alert
manager.
Replica
alerts
are
then
replicated
in
parallel
to
each
other
replica
that
hosts
the
alert
manager
tenant
out
to
the
alert
manager
using
what
we
call
dynamo
style
quorum
for
consistency.
Do
note
that,
because
the
alert
idea
replicas
have
access
to
the
same
hash
ring,
we
no
longer
have
the
requirement
of
the
alert
generator.
A
As
for
the
state
that
needs
to
be
rep,
speaking
of
reads,
we
have
multiple
strategies
depending
on
the
api
call,
but
the
gist
of
it
is
that
we
can
either
merit
responses
depending
if
you
want
to
talk
to
multiple
of
them
or
you
can
send
to
any
if
the
request
only
requires
a
single
response
of
any
of
them.
As
for
the
state
that
needs
to
be
replicated,
which
is
highlighted
in
purple
at
the
end,
it's
constantly
communicated
between
replicas
directly
using
grpc.
A
At
the
end
of
the
day,
we
have
been
successfully
running
this
architecture
for
over
nine
months
as
part
of
grafana
cloud,
with
huge
success,
as
we
didn't
observe
any
major
problems
or
any
major
increase
in
our
costs,
but
enough
of
cortex
and
memory.
I
now
want
to
come
into
the
present
and
show
you
my
most
recent
project
in
the
alerting
world
fast
forward
to
where
we
are
now
and
after
working
with
the
alert
manager
on
such
a
complex
project.
A
A
While
grafana
supports
a
high
number
of
data
sources.
The
examples
I'm
going
to
provide
are
based
on
prometheus
data
for
the
sake
of
simplicity,
let's
start
off
with
support
for
queries
using
template
variables
and
multiple
alerts
per
graph,
even
though
this
evolved
into
two
separate
issues.
As
you
see
here
in
reality,
they're
the
same
thing,
you
have
a
single
query:
you
want
that
query
to
produce
multiple
alerts.
You
want
individual
control
over
all
of
those
alerts.
A
Number
one
and
number
two
have
already
been
a
part
of
rafana
for
a
while
now
and
it's
baked
into
grafana
data
model,
and
it's
exactly
how
you
visualize
time,
series
data
number
three
would
be
the
new
concept
here.
Imagine
grouping
all
the
alerts
that
come
from
my
production
environment
and
delivering
them
to
slack.
A
I
think
you've
heard
that
that
word
grouping
before
exactly
the
alert
manager
does
grouping
and,
more
importantly,
they're
routing
of
notifications
as
well.
With
that
in
mind,
we
can
have
grafana
use
this
existing
data
model
and
turn
any
sort
of
data
into
a
set
of
labels
that
we
can
use
as
the
alert
definition.
A
Then
I
can
send
the
definition
to
the
alert
manager
just
like
prometheus
and
take
advantage
of
all
the
features
I
mentioned
before.
Grouping
routing,
silencing
and
notification
templating
next
are
silencing
and
time
of
the
day.
Restrictions
for
silencing
grafana
supported
a
simple
form
of
do
not
deliver
me.
This
alert.
It
was
called
pausing
and
halted.
The
alert
evaluation
completely
the
passing
feature
was
per
alert,
meaning
there
was
no
way
to
pause,
multiple
alerts
at
the
same
time
that
match
a
single
criteria.
A
Let's
say
that
you
want
to
pause.
All
your
alerts
are
coming
from
a
developer
environment
because
you're
having
a
game
day
or
you're
testing
something
there
was
no
way
of
doing
that.
Personally,
I
think
this
was
not
great.
Why
would
you
keep
an
alert
around
that?
You
don't
want
to
be
evaluated
silences
with
an
alert
manager,
semantically
work
very,
very
differently.
First
they're
based
on
label
matches,
meaning
you
can
met
a
set
of
you
can
match
this
a
lot.
A
You
can
match
a
set
of
alerts
that
come
from
completely
different
rules
and
second,
they
have
an
expiration
date.
They're
meant
to
be
ephemeral
so
that
you
don't
forget
about
this
you're,
not
ready
to
unpause
the
silence,
just
extend
it
at
the
end
of
the
day.
Right
now,
time
of
the
day,
restrictions
is
kind
of
a
hard
problem.
If
you
think
about
it,
consider
not
having
your
alerts
firing
during
weekends.
A
Time
intervals
at
the
end
of
the
day,
allow
you
to
specify
multiple
criteria
for
time
selection
that
could
then
be
reused
against
multiple
route.
This
is
exactly
what
I
needed
for
grafana,
so
I
think
it's
pretty
obvious
at
this
point
right
using
the
alert
manager
within
grifana
gets
us.
The
both
features
that
we
wanted
and
we
can
continue
to
collaborate
with
the
prometheus
community
by
upstreaming,
any
improvements
that
we
make
the
absolute
beauty
of
open
source
open
source.
It's
all
about
giving
back,
and
in
my
next
section
the
future
I'm
going
to
cover
exactly
that.
A
What's
my
plan
for
giving
back
to
the
community,
the
first
order
of
business
was
to
improve.
How
do
we
gather
feedback?
We
have
already
created
a
discussion
section
within
the
alert
manager
repository
that
allows
us
to
prioritize
just
a
tiny
bit
better.
If
you
have
a
feature
request
or
something
that
you
would
like
to
see,
please
share
it
with
us.
A
We
have
excluded
an
experience
within
grafana
that
enables
you
to
test
how
an
alert
would
look
like
on
a
particular
receiver,
such
as
slack
or
paid
your
duty
or
grifano
alert,
no
more
guessing
with
those
always
firing
alerts
and
require
them
for
and
require
them
for
them
to
fire.
As
you
make
a
change
now,
you
make
a
change
and
you
get
instant
feedback.
A
A
Another
challenge
that
our
users
face
is
the
lack
of
visibility
in
the
notification
pipeline
with
the
most
common
being
fail
notifications
to
do
templating
issues,
others
such
as
like.
Why
is
my
deliver?
Why
is
my
alert
not
being
delivered?
I
have
already
unsilenced.
It
are
also
present
prometheus
offers
information
about
the
state
of
your
alert
rules.
As
you
can
see
here,
you
can
see
their
evaluation
time
the
error
what's
their
state.
I
want
the
same
thing
for
the
alert
manager
as
examples
directly
from
the
ui.
I
want
to
be
able
to
know.
A
A
When
did
a
notification
for
this
alert
fire?
How
long
did
it
took
to
notify
and
for
a
particular
receiver?
Had
there
been
any
problems
in
here
recently?
Well,
the
current
logs
are
are
extremely
useful
to
debug.
Some
of
these
problems
we'll
need
access
to
them
in
the
first
place
right.
I
want
to
help
more
the
case
where
the
operator
is
not
necessarily
the
end
user
to
end
my
moonshot
ambition
of
a
historical
unstateful
view
of
the
alert
received
a
notify,
the
answer
to
the
question
what
was
received
and
notified
in
the
last
48
hours.
A
The
alert
manager
is
already
semi-stateful,
as
we
know
silences
and
notifications
stored
to
this.
We
already
saw
this.
Perhaps
improving
the
premier,
perhaps
importing
the
prometheus
tsdv
or
any
other
database
with
a
low
retention
period
and
similar
query
interface
to
prometheus
is
an
idea
here,
I'm
going
to
be
thinking
long
and
hard
about
this
for
the
next
few
months
and
hopefully
bring
something
to
fruition
that
can
help
tackle
this
problem.