►
From YouTube: Prometheus Data analysis and Event Notifications for Progressive Delivery - Ravi Hari, Intuit
Description
Prometheus Data analysis and Event Notifications for Progressive Delivery - Ravi Hari, Intuit
Prometheus is a defacto monitoring tool in kubernetes. Argo Rollouts is an open source kubernetes controller provides ways to perform analysis to drive progressive delivery in kubernetes using Prometheus. While it is crucial to do the analysis it is also important to send the notification of analysis status to the user in near real time. Argo Rollouts uses notification engine which will trigger notifications based on a successful or failure status of analysis using Prometheus data. In this talk we will walk you through an example of how an application can be configured using argo-rollouts by using analysis templates that rely on Prometheus and use notification templates to send notifications in real time to the user. This will also show how this can be integrated to multiple notification channels, destinations and recipients on analysis status.
A
A
A
A
A
Argo
adopts
this
gitops
paradigm
for
continuous
delivery
and
progressive
delivery,
as
well
as
it
enables
to
run
mlaps
on
kubernetes.
So
there
are
multiple
projects
in
argo.
Like
argo,
cd,
argo
rollouts,
argo
workflows
and
events.
Today,
we
are
primarily
going
to
talk
about
our
good
rollouts
for
progressive
delivery.
A
A
So
let's
delve
a
little
bit
into
why
we
needed
argo
right.
The
out
of
the
box
deployment
controller
that
we
get
in
kubernetes
provides
two
type
of
strategies.
One
is
recreate
the
other
one
is
rolling
update,
so
with
neither
of
them,
we
will
be
able
to
do
a
blue,
green
or
canary
or
progressive
delivery.
That
is
one
of
the
reasons
why
arguable
stepped
in
and
it
provided
all
these
different
functionalities.
A
A
You
also
call
it
as
stable
and
desired
and
both
point
to
the
first
version
initially
and
as
we
bring
in
the
newer
version,
the
preview
points
to
the
revision
2
and
if
we
test
it
out
and
everything
looks
good,
then
we
can
promote
the
revision
to
be
the
stable
version
and
then
the
traffic
from
service
and
stable
also
goes
to
the
latest
version.
That
is,
revision
2
and
then,
once
we
are
comfortable
with
the
revision,
do
we
can
delete
the
whole
set
of
parts?
So
this
is
how
blue
green
generally
works
right.
A
A
Let's
look
at
generally
in
canary,
we
have
say
revision,
one
and
both
stable
and
desired
services.
Point
to
reason,
one
initially,
but
as
we
start
deploying
revision,
two
in
canary
the
traffic
percentage
is
get
split
into
revision.
Two
based
on
the
number
of
parts
that
we
create
in
the
revision
two
say
in
in
a
set
of
four
parts:
if
a
new
pod
gets
created
in
revision,
2
25
of
the
traffic
goes
into
that.
A
The
the
difference
between
this
one
and
the
rolling
updates
here
is
that
rollout
provides
you
with
a
lot
of
features
to
control
this.
You
know
revision
2
at
this
particular
point
and
do
your
analysis
and
testing
and
only
when
you
feel
comfortable
you'll
be
able
to
promote
it,
whereas
in
deployment
it
continuously
keeps
updating
the
newer
deployment
newer
parts
as
the
newer
parts
are
stable.
A
It
terminates
the
older
versions
and
it
brings
up
the
newer
ones
continuously,
whereas
a
lot
with
canary
it
provides
you
with
the
ability
to
pause
here
and
do
your
analysis
now.
Once
we
are
comfortable
with
this,
we
can
promote
more
percentage
of
traffic
and,
after
a
certain
point,
we
can
delete
the
older
version
and
completely
step
up
with
the
newer
one.
So
how
do
we
specify
the
canary
deployment?
We
qualify
that
with
a
canary
strategy
and
then
we
give
the
weights
to
it,
and
the
weights
can
have
can
be
distributed
into
different
steps.
A
Each
step
can
have
a
different
rate
percentage
and
we
can
qualify
the
pause
duration.
However
long
we
want
at
every
percentage
of
weight
distribution
that
we
are
interested
in
now.
An
extension
to
this
canary
is
a
alb
canary.
We
have
seen
that
you
know
the
percentage
of
traffic
is
dependent
earlier,
only
based
on
the
number
of
replica
parts
that
are
coming
up
in
the
canary.
A
However,
if
you
want
to
do
traffic
routing
as
we
are
doing
canary,
what
we
could
do
is
we
can
configure
alb
canary
and
we
can
split
the
percentage
of
traffic
that
we
need
to
send
to
the
newer
version.
That
is
division
two
and
we
can
control
it
by
configuring.
This
in
the
traffic
routing
with
our
allots
using
alb
canary.
A
So
initially
we
can
probably
send
five
percentage
of
the
traffic
and
later
once
we
are
comfortable,
we
can
completely
send
the
traffic
and
completely
shift
the
stable
revision,
also
to
send
the
traffic
to
division.
Two
after
everything
is
stable.
How
we
can
specify
this?
We
can
specify
this
with
the
traffic
routing
configurations
in
the
argo
reloads,
as
we
have
seen,
and
the
analysis
templates
can
also
be
used
here
to
analyze
how
your
traffic
is
behaving
with
the
newer
version
of
the
parts
right
and
in
the
alb
ingress.
A
What
we
get
is
that
if
you
see
that
there
is
an
annotation
that
is
introduced
as
the
percentage
of
the
traffic
gets
moved
into
the
new
revision
based
on
the
set
weight
percentages
and
the
steps
that
we
give
the
weight
here
in
this
lb
ingress
for
the
desired
service
gets
updated
and
accordingly,
that
person
that
percentage
of
traffic
gets
routed
to
the
revision
too.
So
this
is
how
alb
ingress
with
canary
works.
We
can
configure
this
with
various
types
of
other
ingresses,
like
steel
or
nginx
and
other
things.
A
A
So
let's
understand
how
this
works
internally
in
arguable
a
lot.
Argo
rollers
itself
is
a
controller
that
manages
this
deployment
objects,
as
well
as
the
replica
sets
and
the
services
ingresses
and
stuff
like
that.
On
top
of
it,
argo
reloads
has
a
an
embedded
analysis
controller,
wherein
we
can
define
an
analysis
template
as
it
goes
through
the
list
of
the
steps
that
we
defined
in
the
analysis
in
the
analysis
portion
in
the
canary
strategy
or
in
the
blue
green
strategy.
A
It
would
take
that
analysis,
template
and
convert
that
into
an
analysis
run
and,
at
the
time
of
analysis,
run.
It
goes
and
fetches
the
data
from
the
desired
metric
provider.
It
primarily
we
use
prometheus
and
kubernetes,
so
it
goes
and
fetches
the
data
from
prometheus
and
then
ensures,
if
you
know
the
conditions
that
we
specify
for
success
and
failure.
For
example,
the
total
number
of
requests
that
are
successful
or
failed
or,
for
example,
the
cpu
usage
on
a
container
or
the
memory
decision,
a
container
should
be
less
than
some
80
percent
stuff.
A
So
how
we
define
an
analysis,
template
an
analysis.
Template
can
be
simply
put
can
be
defined
like
this,
wherein
we
write
some
prometheus
query.
We
just
declare
the
prometheus
url
that
we
want
to
look
for
in
the
cluster
as
an
environment
variable
our
analysis.
Template
can
be
as
simple
as
this.
Primarily
we
just
have
to
write
our
query
and
the
condition
at
which
we
want
to
qualify
it
as
successful
right.
A
So
we
can
add
these
pre
promotion,
analysis
and
post
promotion
analysis
with
analysis
templates,
and
we
can
query
the
data
from
prometheus
and
then
ensure
our
applications
are
stable
after
promoting
to
the
newer
version
right.
This
is
in
blue
green.
How
about
canary?
There
are
multiple
ways.
One
is
in
background
analysis
when
we
qualify
the
with
the
strategy
canary
and
write
the
analysis
on
top
of
the
steps
it
keeps
running
continuously
for
all
the
steps
that
we
have
defined.
A
For
example,
we
have
defined,
set
weight,
forty
percent
and
set
weight,
60
and
stuff
like
that.
So,
at
every
step
it
continuously
runs
this
analysis
and
then
ensures
that
the
newer
version
of
the
part
or
the
application
is
stable
at
that
percentage
of
canary
deployments
right
and
in
any
case
it
fails.
It
automatically
roll
backs
to
the
previous
version
that
we
already
have
and
arguably
already
keeps
in
memory
the
hash
value
of
that
replica
set.
A
How
about
the
other
approach?
The
other
approach
we
have
is
inline
analysis
in
terms
of
inline
analysis.
What
it
does
is
it's
like
a
blocking
step
after
every
step,
every
step
that
we
define
it
stops
the
rollouts
there
and
then
it
does
this
analysis
at
that
time
and
then
gets
back
data
and
qualifies
whether
that
analysis
run
is
successful.
Only
then
it
proceeds
further
right.
So
this
is
how
an
inline
analysis
can
be
done.
A
This
is
primarily
used
if
we
want
to
do
some
kind
of
quick
checks
where
you
know
more.
Some
heavyweight
processes
like
if
you
want
to
run
some
benchmarks
at
40
of
the
traffic
or
80
of
the
traffic
and
ensure
everything
works.
Fine
with
your
newer
version.
This
can
be
done
with
the
inline
analysis,
so
we
have
seen
analysis
right
when
we
want
to
bring
observability
to
the
application
developers.
It's
not
enough.
If
we
run
the
analysis,
we
also
need
feedback
right.
A
Let's
say
developers
update
their
github
repo
and
they
want
to
fire
and
forget
right,
but
it's
also
good
to
notify
them
if
everything
went
successfully
fine
or
if
something
had
a
problem
and
stuff
like
that
right.
So
events
in
this
case
is
really
helpful.
So
how
does
events
work
with
our
cure,
allots
right
so
again
extending
to
the
previous
architecture
that
we
have
arguable,
also
embeds
a
notification
engine?
It's
a
controller
in
argo
project.
A
Now,
basically,
what
this
events
does
is:
let's
say
if
we
run
analysis,
we
generate
an
event
whether
that
analysis
is
success
or
fail.
Let's
say
if
we
complete
the
rollout.
That
is
an
event
that
we
generate
or
if
we
update
a
step
in
the
dollar,
that's
a
given
that
we
generate
all
these
events
that
we
generate
and
write
into
the
event
recorder
will
be
pushed
to
notification.
Engine
and
notification
engine
can
then
send
based
on
our
subscription
to
any
one
of
these
channels.
A
Right
it
could
be
email,
it
could
be
slack
telegram,
pager,
duty
or
obscenity.
All
these
are
integrated
and
many
more
are
also
available.
So
we
can
choose
which
notification
service
we
want
to
be
get
qualified
with
and
alerted
for,
based
on
the
type
of
the
trigger
that
we
are
interested
in.
So
let's
look
at
what
it
takes
to
configure
events
in
our
rollouts
right,
so
the
first
thing
that
we
need
to
set
is
this
is
an
example
for
slack.
A
We
need
to
set
the
slack
bot
token
and
the
secret
for
the
arguable
arts,
and
then
we
need
to
set
the
config
map
in
the
config
map.
We
consume
that
token,
and
then
we
write
a
trigger
for
one
of
the
conditions
that
we
are
interested
in,
for
which
we
want
to
get
alerted,
say,
roll
out
completed,
and
then
we
want
to
send
a
message
to
the
slack
with
all
the
details.
So
we
we
write
the
message
in
the
slack
format.
A
If
it
is
an
email,
we
can
write
the
message
in
an
email,
format
and
stuff
like
that,
so
we
configure
our
config
map
with
this
data
and
then
in
the
rollouts
we
qualify
with
annotations
that
we
want
to
get
notified
on
a
specific
channel
right.
So
we
we
subscribe
to
specific
triggers
and
those
figures.
We
can
qualify
this.
A
This
was
until
recently
we
were
doing
like
this,
and
we
realized
that
if
we
have
to
subscribe
for
multiple
channels
for
same
set
of
triggers,
the
cardinality
here
is
high
and
our
rollout
object
becomes
very
clumsy
right.
It's
not
pretty
neat.
So
what
we
have
come
up
with.
We
have
come
with
a
new
feature
in
notifications
with
a
new
annotation,
where
we
can
call
that
annotation
as
subscriptions
and
here
for
a
common
set
of
triggers.
A
If
you
want
to
get
notified
for
multiple
slack
channels
or
multiple
services,
we
can
define
them,
for
example,
in
the
second
case
here
in
the
first
case,
we
are
getting
notified
for
rollout,
updated
and
rollout
completed
on
a
slack
channel,
but
in
case
of
failures
we
not
only
want
to
get
updated
on
the
slack
channel,
but
also
email,
then,
in
the
same
annotation
we
can
write
it
with
the
new
service
description
here
right.
So
with
that
we
get
notified
in
case
of
failure,
events
and
stuff,
like
that,
let
me
do
a
quick
demo.
B
A
The
arguable
arts
object-
this
is
similar
to
what
we
have
seen
earlier
in
case
of
rollout,
updated,
completed
or
rollout
paused.
I
want
to
get
notified
in
this
channel
in
case
of
analysis,
successful
or
rollout
aborted
and
other
kind
of
failure
scenarios
on
analysis.
I
want
to
get
notified
in
the
data
analysis
channel
and
I
want
to
get
emailed
on
this
email
right.
A
A
A
B
A
A
You
know
completed
and
in
case
of
analysis,
as
we
have
seen
just
now
here.
If
the
analysis
is
successful,
we
got
a
notification
here
that
the
analysis
job
the
analysis
that
was
successful.
We
got
a
notification
in
the
data
analysis
channel.
We'll
also
show
you
what
happens
in
case
if
there
is
a
failure
right
so
on
paused
events.
Again
we
are
getting
notification
in
another
channel.
A
So
when
we
go
through
the
rest
of
the
steps,
there
is
a
failure
step
that
we
have
defined
at
that
step,
it
would
fail
and
there
we
would
see
a
notification
coming
primarily
for
us
on
the
data
analysis
channel
there
you
go
so
here.
If
you
see
the
note,
the
analysis
failed
and
we
have
aborted
the
rollout
and
we
switch
back
to
the
previous
versions.
Essentially,
we
are
getting
notified
with
different
color
patterns.
All
these
things
you
can
configure
in
your
config
map
on
the
colors
and
schemes
and
stuff.
A
Like
that,
you
know
we
have
qualified
two
notification
service
channels.
One
is
slack
the
other
one.
We
should
also
get
in
email.
Let
me
quickly
show
you
that
yeah,
so
I'm
getting
notified
in
my
email.
So
at
this
point
this
was
for
the
previous
one
and
the
latest
one
is
here
where
initially
the
first
analysis
was
successful
and
then
we
run
into
an
analysis
failure.
So
we
got
a
event
here
and
then
the
rollout
got
aborted.