►
Description
Understanding and Extending Prometheus AlertManager [I] - Lee Calcote, SolarWinds
The AlertManager component handles alerts sent by client applications such as the Prometheus server, carefully de-duplicating, correlating, and routing their notifications to their appropriate receiver (e.g. email, webhook, etc.). Current behavior of this component is only to display actively firing alerts.
Contributing to Prometheus is no different than most other open source endeavors, which, like many projects, welcomes community contributions. Let’s gain better familiarity with the process by augmenting Prometheus’ AlertManager with a new “history” view.
About Lee Calcote
Lee Calcote is CNCF Ambassador & an innovative thought leader, passionate about developer platforms and management software - building best-in-class teams and offerings, with a solid record of leveraging advanced and emerging technologies for competitive advantage, improving processes and speed of delivery on the way to driving revenue and market share. With nearly two decades of combined technical, management and leadership experience with globally distributed, agile teams throughout full software development lifecycle, I aim to profoundly impact by leading strategy and development of product and services at the intersection of clouds, containers, infrastructure, applications and their management.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Given
you're
welcome
to
go
through
those,
if
you
want
to
suffer
through
more
than
you
suffer
today
a
little
bit
about
me,
you
can
generally
find
me
within
this
space
and
actually
with
great
pictures
I,
like
though
the
one
was
taken
by
this
gentleman
I
think
at
the
last
conference.
Oh
yeah,
no,
no
in
case
he
takes
good
pictures.
Now
it.
B
A
I
trust
me,
my
wife
says
I'm
Bonnie
too
so
anyway,
if
the
thing
is,
is
yes,
so
you
know
we
tout
poked
at
me.
I
enjoy
a
good
third
talk,
good
conversation,
so
don't
be
shy.
I
end
up
getting
involved
with
some
local
meetups
and
some
local
conferences.
If
you're
coming
to
the
next
iteration
of
this
conference,
see
you
in
Austin,
I
may
even
hand
you
some
barbecue
and
a
little
we'll
have
a
breakfast
taco
together
or
something
to
say
my
full
time.
Role
is
over
at
SolarWinds.
A
How
many
guys
are
familiar
with
SolarWinds
software
monitoring
software
business,
I'd
yeah,
very
good.
Do
you
guys
running
that
software
as
well
so
yeah
if
you've
got
complaints
or
feedback
or
if
you
got
someplace,
you
can
just
keep
you
if
you've
got
feedback,
okay,
you've
got
feedback
or
just
want
to
talk
about
the
products.
Come
see
me
I'd
love
to
talk
about
them.
I
I
also
love
to
talk
about
what
we're
doing
in
this
space.
A
A
You
know
so
I
think
a
number
of
the
talks
have
been
about
Prometheus,
the
monitoring,
the
white
box
monitoring
system
that
you
know
today
that
scrapes
metrics,
its
beloved
components
in
companion
is
the
alert
manager
so
two
separate
systems.
Sometimes
people
ask
why
those
are
separate.
A
couple
of
benefits
there.
One
is
the
two:
these
two
sub
components
can
be
advanced
independently,
so
they
can
move
at
their
own
pace
so
to
speak,
also
as
you
go
to
deploy
them
and
kind
of
scale
them.
A
There
are
other
binaries.
There
are
other
components
to
the
project
that
you'll
find,
which
is
exporters.
There's
a
long
list
of
exporters.
Exporters
are
valuable
to
the
extent
that
you're
trying
to
monitor
a
system
that
doesn't
natively,
isn't
directly
instrumented,
meaning
it
doesn't
natively
export,
prometheus
metrics
and
the
format
that
Prometheus
wants
to
scrape
them.
A
There's
another
component
to
push
gateway
which
is
really
intended
to
be
used
for
scrape
metrics
cache
for
jobs
that
are
shorter,
lived
and
aren't
necessarily
always
available
to
be
scraped.
I
suspect
some
folks
are
maybe
using
the
push
gateway
in
the
Farias
sort
of
unintended
ways
for
maybe
breaking
through
firewalls
or
that
kind
of
thing.
But
so
the
purpose
of
alert
Manager
is
as
a
few
things
it's
as
an
alert
in
gesture,
so
most
prominently,
an
adjuster
of
alerts
from
Prometheus
doesn't
have
to
be
from
Prometheus
alert
manager
as
a
separate
component.
A
Does
you
have
all
of
its
functionality
through
API?
And
so,
if
you
have
a
separate
alert,
client
I
want
to
send
an
alerts
from
some
other
system.
You
could
so
long
as
it's
adhering
to
the
API
specification,
sending
in
JSON
of
of
the
right
type
and
at
the
right
frequency
alert
manager,
ingests
alerts,
but
of
a
couple
of
different
states,
so
obviously
alerts
that
are
firing
as
well
as
alerts
that
are
cleared
or
ones
that
have
transitioned
back
to
inactive
alert
manager,
also
acts
as
an
a
alert
grouper
or
D
duplicator.
A
So
it
does
some
correlation,
both
time
base
and
kind
of
label
based
correlation.
It
does
deduplicating
as
well
of
alerts
or
talk
more
about
that
it's
a
silencer,
so
you
can
go
in
and
mute
alarms,
active
alarms,
prevent
them
from
being
notified.
It's
an
alert
throttler
so
to
the
extent
that
you've
got
an
alert,
that's
been
that's
been
notified
and
continues
to
be
a
problem
for
you
can
the
extent
that
alert
manager
would
continue
to
send
notifications
about
those
alarms.
You
can
tell
it
how
frequently
to
do
that.
A
So
that's
rotted
back
those
notifications
and
very
clearly
alert
manager
is
a
notifier,
so
it's
notifying
third-party
systems
of
alongs
that
have
transpired.
So
let's
do
a
bit
of
it.
So
no
that's
not
a
german
word
as
that's
not
that
deutsch
out
there,
but
it's
just
nomenclature.
That
is.
I
should
have
been
more
clever
and
used
a
german
word,
but
so
let's
talk
a
little
bit
about
some
nomenclature.
Prometheus
a
couple
of
core
constructs
we're
going
to
talk
about
routes
and
receivers.
These
are
described
within
the
singular
configuration
file
that
you
will
hand
alert
manager.
A
As
you
start
up
alert
manager.
You
describe
them
inside
of
a
configuration
file
which
is
yamo.
Routes
are
used
to
match
alerts
and
group
them
and
align
them
to
a
receiver.
Also
they're
used
to
identify
how
frequently
those
alerts
should
be
notified.
Receivers
are
the
where
and
how
to
notify
alerts
so
aware
being,
maybe
maybe
when
you
send
an
email
to
Gmail
and
the
how
being
smtp
over
a
certain
port
with
TLS
and
and
your
credentials.
A
Other
constructs
that
you're
going
to
find
inside
of
the
alert
manager
configure
are
have
to
deal
with
muting,
suppressing
and
correlating,
and
since
I
have
the
mic.
I
get
to
take
liberty
with
a
couple
of
twists
on
words.
There
aren't
the
exact
same
ones
you
might
find
from
from
those
that
are
actual
the
actual
Prometheus
authorities.
A
There's
such
a
thing
as
assignments
and
I'm,
calling
it
a
silencer
here,
but
so
you
know,
Sciences
is
where
you
can
go
into
the
alert
manager
UI
and
for
an
active
firing
alert
just
you
know,
mute
it
prevent
it
from
being
from
notifications
going
out
about
it,
I'm
also
taking
Liberty
with
the
word
inhibitors,
you
can
define
inhibition
rules
to
control
suppression.
So
really
a
common
use
case
here
is
that
there's
an
alert,
that's
fired,
maybe
you're!
A
Maybe
an
entire
rack
went
down
in
your
data
center
and
you
you
go
out
and
match
a
source
alert.
Maybe
it's
a
critical.
You
know
all
systems
down
your
hair
on
fire
thing
and
you'd
like
to
suppress
all
the
subsequent
warnings
and
others.
You
know
their
messages
that
come
through
for
the
500
other
systems
that
are
down
and
so
you'd
use
an
inhibition
rule
to
to
match
match
and
but
both
of
those
what
you'd
like
to
match
against
and
what
you'd
like
to
suppress.
A
So
you
describe
that
inside
of
the
alert
manager,
configuration
and
then
lastly,
I'm
really
grouping.
Is
this
notion
that
I
like
to
think
of
in
terms
of
correlating
both
time-based
correlation,
as
well
as
label
based
correlation,
and
so
you
use
these
group
by
functions
again
within
the
your
configuration
file?
A
So
there's
a
there's
a
group
by
here
where
you
get
to
describe
where
you
get
to
correlate
what
alarms
you'd
like
to
put
into
the
same
group
based
on
labels
or
you
you
can
also
alert
manager,
also
will
allow
you
to
define
things
like
a
group
wait
period,
which
is
when
an
alarm
is
firing,
and
you
receive
the
first
notification
for
that
alarm.
For
that
group,
it
may
wait.
A
The
group
interval
interval
here
or
in
the
case
of
five
minutes
and
so
once
you've
received
an
alarm
for
a
particular
group
and
you've
notified
on
that.
To
the
extent
that
that's
continues
to
be
an
open
alarm
for
some
time
and
you're
receiving
additional
correlated
alarms
inside
that
same
group,
you
can
you
know
kind
of
define
how
your
throttle
here,
how
frequently
alert
manager
will
continue
to
notify
external
systems?
A
Okay,
so
we
talked
a
little
bit
about
so
muting
suppression
correlation,
some
other
kind
of
approaches
to
suppression.
So
we
talked
about
silences
and
your
doing
you
know
creating
silences
in
the
UI.
You
can
you
know
anything
you
can
do
in
the
UI.
You
can
do
through
the
API
and
so
there's
a
use
case
here.
Maybe
for
creating
silences,
as
you
were,
to
have
a
maintenance
window.
A
Maybe
you
kick
off
your
maintenance
script
and
one
of
the
things
that
does
is
goes
out
and
creates
silences
using
the
API
and
then,
when
it
maintenance
maintenance
window
is
done.
You
go
in
and
use
that
same
script
to
remove
those
silences.
That's
a
use
case.
So
I
think
you'd.
Hear
folks
argue.
Maybe
you
shouldn't
be
creating
alarms
during
your
maintenance
window
anyway,
but
so
we
talked
about
innovations
and
the
fact
those
are
globally
described
that
that
use
case
of
you
having
a
critical
alarm.
That's
firing!
A
You
know
that
you
don't
need
to
be
flooded
with
a
million
other
alarms,
so
you're
suppressing
other
alerts
that
are
related.
Another
way
of
kind
of
approaching
suppression
is
on
a
per
route
basis,
so
using
the
repeat
interval,
and
that
would
be
to
the
extent
that
you've
alerted
or
has
received
an
alert,
its
notified
on
that
alert
to
you
know
for
that
route.
A
The
repeat,
interval
controls
when
the
next
time
is
that
alert
manager
would
notify
so
that
same
set
of
grouped
alerts.
So
conceptually,
if
you
had
this
repeat
intervals
like
you
know,
for
24
hours
or
something
some
really
extended
period,
you're
in
essence,
kind
of
suppressing
the
rest
of
those
alerts
that
would
come
through
kind
of
a
couple
of
different
ways
to
you
know
reduce
your
pager
fatigue.
So
probably
last
construct
here
is
obviously
very
central
to
what
we're
talking
about
not
set
alerts.
A
A
Those
alerts
can,
then
you
know,
go
into
these
different
states
and
are
notified
to
alert
manager.
The
interesting
state
here
is
this
pending
one
I
think
you
know
I
think
it's
fairly
obvious
what
inactive
and
firing
do,
but
this
pending
state
is
to
the
extent
that
you
get
an
alert
defined
if
for
a
given
metric
and
that
metric
begins
to
match
this
criteria,
it
will
move
from
inactive
to
a
pending
state
depending
upon
the
duration
that
you've
described
here
under
the
four.
So
if
it
was,
you
know,
matching
CPU
over
a
certain
percentage.
A
For
you
know,
you
begin
to
move
from
and
your
CPU
spiked
over
that
Demma
you
move
depending
for
and
you'd
hold
out
in
pending
for
a
minute
or,
however
long
you
described
a
duration
and
assuming
that
your
metric
is
still
crossed
that
threshold
after
the
duration
has
ended.
You
transition
into
firing
and
Prometheus
would
send
a
firing
alarm
to
alert
manager.
A
Also
when
those
alerts
then
move
back
to
inactive
a
Prometheus
will
let
alert
manager
know
that
as
well
and
so
there's
this
notion
of
you
can
control
whether
or
not
alert
manager
sends
out
resolved
or
clear
notifications
out
to
the
third
parties,
third-party
alert
management
systems,
which
are
really
these
systems
here.
So
the
systems
that
you
see
with
the
pretty
icons
or
have
kind
of
first-class
integration
into
alert
manager,
the
email
and
kind
of
webhook
icon.
If,
if
that's,
what
is
because,
because
you
know
whatever
that.
B
A
Really
represents
a
web
bellcranks
in
it
anyway.
Those
are
the
more
generic
ways
of
integrating
the
systems
used
to
be
that
slow
doc
was
up
here
as
well
in
the
dot
five
release,
that
kind
of
fell
by
the
wayside,
I'm
sure
to
come
back,
so
you
it
is
possible
to
have
alert
manager
to
notify
to
multiple
destinations
that
maybe
you
want
to
send
a
Nena
and
webhook.
So
a
couple
ways
to
do
that.
Probably
one
of
the
more
conceptually
easier
ways
here
is
just
define
multiple
destinations
under
the
same
receiver.
A
Maybe
that's
when
we
call
in
the
18
yeah
I
didn't
think
it
was
finite,
so
a
little
bit
about
of
the
alert
manager,
architecture
or
really
kind
of
the
system
flow
for
how
an
alert
comes
into
the
system
and
kind
of
the
things
that
happened
to
it
and
actions
get
taken
on
it.
This
is
really
the
non
AJ
version
as
I
understand
it.
So
again,
you
know
all
things
come
through
the
API,
whether
it's
the
UI
or
let
maintenance
script
I
talked
about,
but
others
come
into
the
API
they're
ingested.
A
Really
it's
the
dispatcher
that
takes
on
the
job
of
grouping
and
aggregating
kind
of
assigning
the
correct
notifier
to
a
given
alert
that
alert
gets
put
into
a
notification
pipeline.
We
check
for
those
inhibition
rules
and
check
for
the
silencers
that
you
may
have
or
them
sorry
the
silences
that
you
may
have
defined
and
whether
or
not
this
note
of
particular
notification,
this
group
has
already
been
notified
previously,
so
you
know,
try
to
be
duplications
and
then
send
out
to
your
your
destination.
A
The
H,
a
version
ends
up
looking
I
think
you
see
a
few
more
things
in
this
notification
pipeline
I'm.
One
of
those
is
that
after
an
alert
has
been
notified,
then
you'll
see
some
gossip
happening
here,
where
the
other
alert
managers
in
that
quorum
are
in
that
same
group,
would
exchange
state
and
understand
who's
notified,
which
alerts
whether
or
not
they've
already
been
notified.
So
speaking
of
high
availability,
Frederick
David
great
gave
a
great
talk
earlier
on
high
availability.
A
The
nice
thing
about
high
availability
with
alert
manager
is
that
you,
this
sort
of
eliminates
the
need
to
monitor
the
monitor
so,
to
the
extent
that
you
had
a
singular
alert
manager
there
before.
If
it
went
down,
you
didn't
necessarily
know
unless
you
were
monitoring
the
monitor,
and
so
with
a
che.
You
don't
need
that
anymore.
The
other
way
that
this
system
works
kind
of
behavior
here
is
that
it
guarantees
you'll
guarantees.
It
will
receive
notification
at
least
once,
which
is
good
all
right.
Let's
crack
open
the
history
book.
A
Let's
crack
open
a
use
case
around
how
to
add
a
feature
into
the
into
alert
manager.
Let's
just
open
up
the
code,
a
little
bit
so
you're
all,
probably
very
familiar
with
the
alert
manager
UI
in
this
case,
we've
got
a
screen
shot
of
a
singular
alarm.
That's
that's
firing!
This
is
an
instance
down.
What
we
don't
see
is
a
bunch
of
context
about
what
else
has
necessarily
happened
in
the
system.
You
know
for
this
in
this
case
is
an
instance
down.
A
So
how
many
times
does
this
instance
bend
down
or
have
we
received
this
alert
in
the
last
little
while
or
what
additional
context
can
I
use
to
troubleshoot
here?
And
so
let's
look
at
what
it
might
mean
to
bring
in
a
history
of
you
and
put
it
up
there.
So
before
we
can
start
coding,
we
obviously
have
to
have
a
user
story.
A
Otherwise
we
can't
go
write
the
code
so
as
an
operator
I'd
like
to
not
only
see
a
list
of
firing
loads,
but
also
a
list
of
all
transpired
alerts,
so
that
I
can
better
troubleshoot
as
a
little
bit
of
a
background,
you
can
pet
a
troubleshoot.
If
you
can't
begin
to
define
what
is
normal
or
understand
hey.
Do
we
have
a
problem
area
here?
Is
it's
just
happening
every
Wednesday?
A
What's
going
on,
maybe
to
the
extent
that
you
can
see
how
these
things
have
happened
in
the
past
and
a
certain
order,
you
can
some
crude
the
root
cause
analysis.
We
do
want
to
identify
that.
We've
got
a
limitation
here,
that
the
version
of
alert
managed
that
we're
talking
about
isn't
is
not
the
H
a
dot
five
version,
but
is
our
prior
versions,
which
we're
using
a
sequel
Lite
as
their
database
to
persist,
alert
notifications
and
so
and
that
wasn't
intended
to
be
on
long
term
storage.
So
so
this
is
the
limitation
acceptance.
A
Criteria
for
the
story
would
be
once
once
an
alert
is
fired.
We
should
be
able
to
see
all
of
all
of
those
transpired
alerts
on
a
history,
page
optionally.
We
should
probably
see
that
the
slack
channel,
which
probably
put
some
stretch
in
here
and
say
that,
if
we're
looking
at
history,
you
know
to
the
extent
that
we've
got
thousands
of
alerts.
Maybe
we
should
have
some
pag
nation
or
a
date
picker
or
some
kind
of
filters.
Of
course
I
think
we
all
know
what
stretch
means.
A
That
means
out
of
scope
right
ain't
going
to
do
it.
So
we
probably
we
probably
aren't
nice
thing
about
this
user
story.
Is
it
only
talks
about
affecting
changes
inside
of
alert
manager?
So
it's
also
kind
of
nice
that
it's
a
separate
component
to
that
end,
we
can
go
in
and
begin
to
set
up.
Our
I
meant
with
Prometheus
and
some
random
sample
targets
there
as
part
of
the
Prometheus
project.
There's
a
small,
a
small
target
written
in,
go
that
you
can
go
out
and
deploy.
A
Okay,
so
it's
probably
horrifically
small
I,
don't
know
why
anyone
ever
asks.
Can
you
see
that
like
hey
you?
Should
you
never
can
see
it?
I
I,
don't
know
if
I'm
the
only
one,
that's
blind,
but
it's
like
you
know
it's
always
do
small.
It's
just
yeah.
You
guys
feel
the
same
for
that
and
okay
good.
So
we've
set
up
a
target.
A
A
Get
this
into
you,
get
it
get
them
up.
The
we'll
take
a
quick
look
at
our
Prometheus
config,
which
is
just
here,
is
just
you
know,
essentially
the
vanilla
stuff,
we're
going
to
add
an
external
label
of
cloud
native
Colin,
most
appropriate
and
then
also
just
some
very
simple
alert
rules
or
actually
just
one
alert
rule
nuts.
A
Okay,
so
now
that
we've
got
our
done,
so
we
let's
just
confirm
that
an
environment
is
up
okay,
so
we've
got
Prometheus
here.
It
is
pointing
to
itself
and
scraping
itself,
and
it's
also
scraping
that
little
sample
target
that
we
put
up.
As
you
can
see,
it's
trying
to
assign
some
alerts
right
now,
but
not
finding
alert
manager
because
we
didn't
set
up
a
litter.
So
let's
set
up
alert
manager.
In
this
case,
we
need
to
kind
of
set
up
our
development
environment,
cuz
we're
going
to
get
in
there
and
make
some
changes.
A
So
we
go
out
clone
the
repo
I
won't
spoil
all
of
our
bandwidth.
By
doing
that,
we'll
also
need
to
go
and
grab
a
tool.
That's
used
within
the
Prometheus
project.
It's
called
Gobind
data.
This
tool
helps,
helps
you
since
we're
going
to
be
messing
and
modeling
with
a
UI
and
adding
a
new
history
view.
We
need
to
go
use
this
tool
to
help
us
stuff
in
some
static,
HTML
and
CSS,
and
an
angular
into
the
alert,
binary
alert
manager,
binary
I'm.
A
A
A
Interestingly,
as
you're
defining
your
alert
manager
configuration
things,
can
get
kind
of
gnarly,
with
the
how
many
routes
you've
got
set
up
and
and
who
you
know
what
should
get
matched
who
should
get
notified?
One
where
there's
a
visual
editor,
that's
available
on
Prometheus
tayo
that
where
you
can
go
and
paste
in
your
llamo
configuration
and
it'll
visually,
show
you
what
what
routes
you
take
and
let
you
kind
of
do
some
test
matching
to
see
which
messages
would
go
where
so.
A
A
Okay,
so
you
have
a
load
manager
up
as
good
some.
A
couple
of
other
tips
for
folks
who
stumble
through
things
like
I
do
to
the
extent
that
you've,
you
update
your
prometheus
configuration
or
your
alert
manager
config.
These
are
a
couple
different
ways
to
reload
that
config
on
the
fly,
also,
to
the
extent
that
you've
made
changes
in
your
committee,
s
config
or
the
alert
rules,
you
can
use
prom
tool
to
just
check
and
verify
that
they're
syntactically
correct
before
you
reload
and
and
start
missing,
alarms.
A
So
do
have
a
silence
defined,
got
our
config
loaded
that
very
simple
one
that
we're
looking
at
Prometheus
and
then
get
into
the
changes
that
we
would
make
inside
the
code.
So
yeah
I'm
really
only
a
few
about
six
files
that
need
to
be
changed
a
little
bit
ago,
some
angular
HTML
and
some
sequel,
so
not
very
much
six
files
in
total
to
add
in
that
a
new
history
view.
A
A
We
go
down
and
find
what
list
all
what
that
function
means
and
tell
it
that
it
would
be
a
new
method,
call
a
new
alert
iterator
on
the
alerts
object
by
the
way
the
yellow
is,
to
the
extent
you
guys
come
back
and
look
at
this
later.
A
couple
of
notes:
one
is
all
the
yellow
links.
All
these
yellow
texts
are
links
to
where
this
code
is
where
this
tutorial
is
so
so
then
having
to
find
that
API
endpoint
go
and
define
what
that
get
all
alert.
A
A
A
Okay,
so
we
defined
them.
How
it
is
that
we
retrieve
data
from
our
database.
Then
we
go
into
our
angular
into
the
web
UI
aspect
and
create
a
new
menu
item.
Called
history
we'll
go
over
and
define
what
service
that
history
menu
item
should
call.
What
HTML
should
we
should
use
to
format
the
record
set:
that's
retrieved,
so
just
about
four
spots
on
our
in
our
angular,
which
ends
up
looking
like
this.
So
we
do
need
to
go
over
and
then
create
a
history
HTML
too.
A
A
Okay,
let's
go
see
if
that
happens,
so
so
we
do
have
a
recognizable
cat
kind
of
pre
baked
the
cake,
a
fair
bit
in
here
just
to
try
to
make
it
through.
But
anyway
we
do
have
a
new
history,
tab
click
that
and
we
go
over.
And
fortunately
this
is
a
very
small
system
which
doesn't
have
a
lot
of
alerts,
but
we're
seeing
our
full
history
here.
So
you
do
have
our
cloud
native
con
alert
that
has
fired
a
bunch
of
other
instance
down,
but.
A
Hopefully,
what
this
does
is
begin
to
serve
as
kind
of
this
simple
enhancements
and
serves
as
an
example
of
how
to
navigate
part
of
your
way
through
the
various
aspects
of
alert
manager.
What
it
might
take
to
add
something
new,
certainly
communicating
with
the
core
maintainer
x'
is
probably
the
great
place
to
start.
If
you
can,
you
know,
go
and
add,
add
something
like
this
make
sure
it's
kind
of
lined
with
the
direction
of
the
project.
A
We're
going
to
point
out
some
resources
here
in
a
minute
about
that.
But
I'll
say
that
again
to
the
extent
you
want
to
come
wade
through
what
we've
done,
there's
a
fork
on
github.
You
can
go
look
at
as
well
as
there
may
be
a
better
written
tutorial.
That's
all!
It's
not
a
slide
format
that
just
kind
of
written
out
for
you
to
look
through
here
are
just
a
list
of
resources
and
how
to
engage
some
of
the
folks
that
have
already
presented
today
and
some
of
the
people
that
are
helping
advance
prometheus.
A
B
A
Project,
you
know
I,
think
you
know
the
the
caveat
that
I
think
I
made
earlier
is
that
the
project
has
since
moved
away
from
using
sequel
light,
and
it's
now
moving
towards
an
H,
a
system
based
on
the
framework
that
I
mentioned
before.
But
you
know
as
I
understand,
the
team
does
consider
the
need
for
your
this
type
of
a
view
and
considering
how
they
might
facilitate
for
that.
At
some
point,.
A
B
A
A
Okay,
I'm
not
finding
the
slide,
but
anyway
it
was
a
small
utility
globe
in
data
that
Kenneth
I
stumbled
over
initially
I
went
through
made
some
angular
changes
like
a
dot
very
clearly
that
thing
is
going
to
show
somewhere.
You
know
like,
and
after
the
damnedest
reason,
I
couldn't
figure
out
why
it's
not
showing
hey.
This
is
compiling
I'm,
getting
a
new
binary,
but
there's
no
changes
in
the
uija.
What's
going
on.
In
my
you
know,
I'm
dumping,
my
cache,
like
you,
wouldn't
believe
seriously
anyway,
it
was
come,
come
to
learn.
A
I
came
to
be
educated
on
that
that
utility
itself,
so
that
was
a
little
bit
of
a
gotcha.
For
me,
all
the
ones
moving
back
and
forth
I
think
I
was
just
for
me
pleased
to
see
that
something
like
see
something
sequel
like
was
being
used
just
given
my
familiarity
with
that
that
query
language,
and
that
certainly
made
it
easy
of
the
type
of
functionality
that
was
delivered
here.
It
was
really
easy
to
just
use
other
other
methods
and
other
functions.
A
As
examples
of
you
know,
the
right
way
to
you
know
to
what
was
to
be
done,
so
that
was
really
helpful.
All
in
all
it
was
about,
it
was
a
little
less
than
a
day
to
do
the
to
do
the
work,
and
so,
even
though
I
stumbled
through
it's
about
how
long
it
took
or
just
the
write-up
tickle,
you
know
three
days
but
but
making
it
through.
It
was
pretty
short.
So
one
of
the
things
I
would
say,
like
maybe
even
answer
that
question
further.
A
Is
that
once
you
get
into
like
pulling
back
100,
you
know
a
couple
hundred
records
and
things
by
getting
into
pag
nation
and,
using
you
know,
I'm
an
angular
newbie
as
well,
so
getting
in
and
using
that
would
be.
Yeah
I
can
start
to
see
what
I.
It
would
take
me
longer
figure
that
out,
oh
by
the
way
on
that
the
team
is
considering
moving
from
angular
over
to
elm
kind
of
a
proposal
on
the
table
that
maybe
I'm,
probably
talking
to
an
insider
I
know
with,
is
a
Elmo
or
react
and
I.