►
From YouTube: Ups, let's revert! 🔥 Automating Kubernetes Rollouts with Argo a... Bartlomiej Płotka & Anais Urlichs
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Ups, let's revert! 🔥 Automating Kubernetes Rollouts with Argo and Prometheus for HA - Bartlomiej Płotka & Anais Urlichs, Codefresh
"Thanks to Kubernetes, deploying and running applications in the cloud was never easier. Unfortunately, with the capabilities, desire for highly reliable services with strict minimal downtime SLO grows too. And we all know what increases risk of incidents: Upgrades!
Vanilla rollouts functionality works great for basic needs, but it lacks many advanced deployment strategies which can drastically reduce distruption time during critical production promotions, especially those failed ones. That’s were Argo Rollouts comes in, solving this need by introducing tiny shim over Kuberentes Deployment object. But with the power, comes the complexity and without proper observability and monitoring it can become unmanagable pretty quickly.
In this talk you will see demo showing an efficient way of setting up critical Prometheus monitoring and alerting for advanced Kubernetes deployment operations Argo allows. You will learn how to observe the progress of your rollouts, Blue-Green or Canary deployments, how to perform rollbacks without human intervention and how to only alert when humans have to be called!"
A
B
Hello,
everybody
really
excited
to
be
here.
My
name
is
anes
orlese,
I'm
a
site,
reliability
engineer
at
sivo.
Sivo
is
a
cloud
computing
company
based
on
kubernetes.
Before
that
I
was
working
at
code
fresh,
which
is
a
devops
automation
platform,
I'm
also
a
cncf
ambassador
and
I
run
a
youtube
channel
around
a
challenge
called
100
days
of
kubernetes.
If
any
of
this
got
you
curious,
you
can
check
out
my
twitter
at
holy
sonny's,
also
linked
there.
A
Yeah,
my
name
is
bartek
platka
and
I'm
principal
software
engineer
at
red
hat,
I'm
part
of
the
team,
which
is
you
know,
responsible
for
the
observability
and
monitoring
for
red
card,
including
openshift
as
well
and
yeah.
We
are
maintaining
lots
of
projects
around
that
part
of
the
infrastructure,
I'm
maintaining
prometheus,
I'm
co-author
of
the
thanos
project
and
I'm
maintaining
other.
You
know
go
like
repositories
all
around
around
the
github.
A
B
Awesome,
so
what
do
we
have
prepared
for
you
for
the
next
20
minutes,
or
so
we
have
a
little
ping-pong
application
of
which
we're
going
to
show
you
different
scenarios,
and
in
addition
to
that,
we
have
a
client.
That's
gonna
request,
requests
information
from
that
application
and
those
are
both
micro
service
application
and
then,
in
addition
to
that,
we
are
going
to
use
prometheus
and
argo
rollouts.
B
Now
aggro
rollouts
are
going
to
be
used
to
automate
our
deployments
and
prometheus
is
going
to
be
used
to
out
like
basically
to
help
our
worlds
to
know
what's
happening
within
the
rollouts
to
in
case
something
happens
wrong
to
revert
our
wallets
as
well.
So
let's
jump
right
into
our
demo
and
show
you
what
we
have
set
up.
B
As
you
can
see
here,
we
are
in
katakota.
Katakura
is
an
online
platform
that
allows
you
to
spin
up
demo
environments
such
as
this
one.
You
can
also
try
out
the
scenario
at
the
end,
we're
going
to
give
you
all
of
the
information,
so
you
can
check
out
the
repository
and
the
code
and
also
the
scenario.
Now
we
have
our
cuban
news
cluster
set
up
here.
We
have
nothing
on
it
right
now
and
we're
going
to
just
go
ahead
and
create
a
namespace.
A
And
now,
to
start
with,
every
infrastructure
has
to
have
some
monitoring,
so
we
start
installing
our
resources
from
the
observability
components.
So
first
we
start
on
the
right.
You
can
see
prometheus
components,
so
we
install
promote,
use
one
binary
and
it's
supposed
to
gather
metrics
and
exemplars,
and
I
will
tell
later
what
example
means
and
why
we
need
that.
We
also
can
do
alerting
on
this
on
this
node.
So
it's
kind
of
super
super
useful
and
promote
use
will
gather
those
metrics
and
examples
from
any
components,
including
other
components.
A
You
can
see
on
this
diagram.
So
we
have
also
graphana,
which
is,
will
be
our
single
point
of
access,
so
it
will
be
our
ui
our
front
end,
and
we
can
you
know
we
statically
configure
some
dashboards
and
that
will
allow
us
to
see
the
progress
on
our
rollout
and
you
know
the
health
of
our
system
and
how
it
impacts
or
not
our
customers,
and
we
have
our
secret
agent
here
temple
on
the
left
and
it's
it's
like
a
cool
project
really
with
similar
architecture
than
than
other
systems.
A
B
Now
we
went
ahead
and
we
applied
those
resources.
Now
those
resources
will
take
a
few
seconds
to
spin
up
completely.
In
the
meantime,
we're
going
to
take
a
look
at
our
rollouts
and
going
to
apply
our
wallets
resources
now,
what
is
aggro
rollouts,
exactly
it's
an
operator
and
custom
visas,
definitions
and
both
are
used
to
then
deploy
other
resources
for
our
rollouts
to
make
our
wallets
happen
and
basically
use
advanced
deployment
strategies
such
as
progressive
delivery,
blue
green
deployments,
as
well
as
canary
deployments
now,
in
our
case,
we're
going
to
focus
on
canary
deployments.
B
So
let's
take
a
look
at
the
aggro
resources
at
the
yamafest
that
we're
actually
deploying
and
using
agro
rollouts
with
at
the
beginning.
We
have
our
rollout
yammer
file.
Now
the
roller
demo
file
is
similar
to
a
deployment
resource
that
you
might
be
familiar
with
if
you're
at
this
conference-
and
it
basically
adds
additional
information
to
our
deployment
resource.
So
we
can
see
over
here
that
we
also
specify
the
container
that
we
want
to
that.
We
want
to
use
and
the
replica
set
of
our
deployment.
B
So,
in
this
case,
we're
going
to
use
a
replica
set
of
five.
If
you're,
not
using
a
service
mesh,
you
will
have
to
use
a
minimum
replica
set
of
five
so
for
aggro
rollers
to
be
able
to
distribute
the
traffic
and
move
traffic
between
your
deployment
resources,
so
that
is
specified
in
the
steps
section
we're
going
to
start
off
with
a
new
deployment
with
a
rate
of
traffic
rate
of
20.
B
So
that
means
whenever
we
roll
out
a
new
deployment
of
our
application,
it's
gonna
first
just
spin
up
one
part
and
scale
down
one
of
the
existing
five
parts
of
our
initial
deployment
and
then
with
an
interval
of
first
60
seconds
and
then
30
seconds.
It's
going
to
gradually
roll
out
the
new
deployment
with
this
resource
based
on
this
resource.
B
B
If
we
don't
have
the
section,
agar
bullets
will
just
gradually
shift
the
traffic
and
roll
it
like
it
is,
and
as
soon
as
long
as
the
pods
are
running
correctly
within
our
cluster,
we'll
just
go
ahead
and
roll
it
out.
Now,
that's
obviously
not
what
we
want,
because
that
would
require
us
to
manually
check
whether
or
not
our
roller
is
okay.
Instead,
we
want
to
utilize
the
analysis,
template
and
prometheus
to
automate
this
process.
A
Right
so
on
a
rollout
section
above,
we
specify
certain
template
that
we
used
and
our
template
is
called
low,
error,
load,
latency
template
and
we
specify
here
the
exact
you
know,
data
what
it
means
for
rollout
to
be
successful
or
not,
and
we
kind
of
base
those
criteria
right
now
in
our
demo,
based
on
the
two
metrics
we
have
one
metric
is
lower
than
20p
error
rate
and
second
thing
is
latency
90p
lower
than
one
second.
What
that
means
well,
essentially,
first
of
all,
our
error
rate
metric
is
supposed
to
give
us.
A
A
Now
we
do
that
much
very
often
like
we
just
we
do
this
check
every
20
seconds.
Normally
you
probably
you
you
like
to
do
this
every
minute
or
every
five
minutes,
and
but
we
wanted
to
kind
of
you
know,
speed
up
this
demo
and
we
have
a
very
aggressive
scrape
interval
on
pro
materials
five
seconds.
Normally,
you
should
have
probably
15.
now.
This
is
only
one
metric
second
metric
called
latency
90p
lower
than
one.
A
Second,
we
are
observing
the
tail
latency
of
our
application
again
for
all
the
customers,
and
we
expect
the
tail
latency
to
be
lower
than
one
second,
so
most
of
the
users
90
of
the
users
should
have
that
latency.
If
not,
then
we
claim
this
operation
to
be
to
be
failed
and
rollout
to
be
reverted,
and
we
can
see
also
the
failure
limit.
This
is
kind
of
important.
A
You
know
additional
info
that
you
can
provide,
which
tells
argo
rollout
to
wait
for
free
failures
for
this
metric
and
only
three
failures
actually
trigger.
You
know
the
failure
of
the
whole
deployment.
This
is
to
avoid
some
spikes
or
some
fluckinesses
as
well,
and
now
you
can
see
our
diagram
expanded
to
those
new
two
new
resources.
We
are
just
rolling
out.
A
We
have
argo
rollat,
which
is
our
rollout.
You
know
operator
actually
controller
in
in
details,
and
it
is
you
know,
managing
those
applications
and
distributing
traffic
and
and
as
and
I
said,
we
don't
need
to
have
service
match
for
that.
We
don't
need
to
have
a
special
ingress,
although
those
are
supported
as
well.
A
B
Now
that
we've
deployed
those
resources,
we
can
check
our
rollout
client,
whether
or
not
our
initial
rollout
has
happened
successfully.
So
in
this
case
we
can
see
that
five
parts
are
currently
up
and
running
of
our
initial
deployment
of
our
initial
image
and
it's
called
stable,
and
we
can
see
that
any
further
rollouts
will
use
the
canary
deployment
strategy.
B
So
in
this
case
we
have
several
different
sections.
We
could
be
precise
and
different
panels
within
each
section.
Now,
let's
reduce
the
time
for
the
past
five
minutes
and
have
a
look
at
the
first
section
or
user
experience.
What
does
it
actually
tell
us?
So
this
checks,
our
ping
application,
makes
sure
what
is
the
percentage
of
our
successful
pinks
from
that
application,
and
in
this
case
we
are
right
now
at
95
percent.
Now,
additionally,
we
can
see
in
the
next
panel
the
overall
latency
per
pinx
from
our
paying
application.
B
So
we
can
see
that
this
is
the
50
percent,
like
how
many
users
are
seeing
that
latency
below
0.4
seconds
and
how
many
are
seeing
about
around
0.5
seconds
latency.
So,
basically,
how
long
does
it
take
for
a
ping
to
be
received
by
the
fighter
pinger
from
the
punk
to
be
received
by
the
clinger
from
the
pawn
application?
B
A
Yes,
so,
as
mentioned,
the
very
first
row
is
for
the
client
experience,
so
we
take
those
metrics
directly
from
the
client
to
have
like
you
know
the
closest
experience
they
have.
But
now
there
are
kind
of
useful
information
we
can
get
from
our
infrastructure
as
well,
so
the
below
metrics
are
from
our
servers.
So,
first
of
all,
we
have
a
certain
metrics
from
the
server
app
directly
the
same
error
rate.
To
make
sure
we
see
the
similar
error
success
rates
between
the
client
and
the
server.
A
We
see
the
latency
as
well,
and
the
difference
is
that
we
now
visualize
that
in
a
second
type
of
visualization
you
can
do
for
for
latency
histogram
metric.
That
promotes
allows
so
you
can
do
percentiles,
but
you
can
actually
show
this
with
the
heat
map
and
the
heat
map
is
kind
of
very
natural
and
easy
to
to
read
from
essentially
the
brighter
the
color.
The
more
amount
of
users
are
experiencing,
the
latency
within
certain
bucket,
so
we
can
see
most
of
the
users
are
experiencing
latency
between
300
milliseconds
and
600
milliseconds.
B
A
Good
experience,
overall
there
are
some
requests
you
can
see
which
are
even
faster
but
but
small
amount
of
those
and
then
on
the
right.
It's
kind
of
the
similar
thing
we
can
see
on
the
panel
above.
So
it's
an
error,
error
rate.
Just
we
hide
the
success
rate
because,
from
the
you
know,
sre
perspective,
self-reliability
engineer
or
devops.
A
You
really
want
to
focus
on
what
is
important
for
you
and
right
now
we
want
to
care,
you
know
how
many
errors
each
version-
you
know
kind
of
exposes,
so
we
have
some
separations
on
the
version
which
is
in
our
case
like
a
docker,
docker
image
tag
really
and
then
last,
but
not
the
least.
We
have
a
rollout
state.
So
we
to
take
various
metrics
from
argo
rollout
controller
directly.
A
First
of
all
we
see
what
are
the
rollouts
in
progress.
Then
we
can
see
the
rollouts
per
our
version,
so
there
will
be
number
of
replicas.
Essentially
sorry,
it's
not
rollouts,
but
number
of
replicas
per
each
version
where
we
have
been
rolled
and
and
deployed
and
then
on.
The
right
last
panel
is
really
about
the
you
know.
Our
rollout
analysis
runs
if
they
were
successful
or
not.
Based
on
our
argo
analysis,
template
which
you
remember
was
measuring
the
latency
and
the
error
rate,
and
you
know
what
are
those
checks?
B
Awesome
so
now
that
we
have
a
general
understanding
of
what
our
grifana
dashboard
will
tell
us
over
time
from
our
deployments,
we
can
go
ahead
and
we
can
trigger
a
new
deployment
through
aggro
rollouts,
so
we're
just
going
to
deploy
a
new
version,
and
we
named
in
this
case
errors
really
fancy
now.
First
off,
we
can
already
see,
within
our
argo
rollout
section
that
one
part
is
spin
up
and
our
initial
deployment
is
down
to
four
parts.
B
Then
we
can
also
see
the
analysis
that
is
being
performed
and
basically,
whether
or
not
it's
hitting
an
error
rate
can
see
here
in
our
errors,
dashboard
per
version
that
right
now
here,
our
new
deployment
is
actually
is
already
showing
some
arrows.
But
at
this
point
it's
it
was
really
just
now
up
to
just
now.
It
was
around
the
same
as
with
our
initial
deployment.
So
there
was
nothing
worrying
yet,
but
then
we
can
see
that
our
overall
success
rate
that
we
want
to
have
above
80
is
slowly
going
down.
Now
it's
below
90..
B
So
let's
see
how
this
is
going
to
develop.
Now,
in
terms
of
latency,
we
can
actually
see
that
the
latency
of
both
deployments
are
fairly
similar.
Nothing
has
really
changed
there.
Now
now
we
can
see
that
actually
something
happened
ago.
Rollouts
realize
that
something
is
something
is
going
wrong.
Something
is
off
and
has
terminated
our
rollout,
so
we
can
see
it
just
basically
stop
and
right
at
instead
of
slowly
scaling
down
our
new
deployment,
it
just
cut
it
off
and
scaled
back
up
our
initial
deployment
to
five
replicas.
B
A
B
B
We
want
to
deploy
a
new
version
which
does
not
cause
as
many
errors
for
our
users
and
overall,
that
the
error
rate
here
is
staying
low
for
everybody
from
our
client
side
right,
so
we're
going
to
go
ahead
and
we're
going
to
deploy
a
new
version
in
this
case
we're
going
to
call
our
update
or
fix,
slow
and
we're
going
to
deploy
this
image
and
going
to
observe
our
rollouts
again
now
it
will
show
us
our
previous
revisions,
so
we
can
see
here.
Revision
2,
which
was
our
error
deployment,
has
actually
failed.
B
We
can
see
that
that
fail
of
our
of
our
the
checks,
our
errors,
how
many
errors
we
have
has
caused
the
immediate
termination
of
this
rollout.
We
can
see,
additionally,
that
our
new
deployment
is
spun
up,
there's
one
port
running
and
it
has
already
passed
two
checks
from
our
analysis:
template
now.
Let's
go
back
to
our
grafana
dashboard
and
see
how
everything
is
developing
overall,
here's
our
error
rate,
that's
slowly,
showing
on
in
the
dashboard
in
the
panel,
but
what's
happening
over
here
so.
A
It's
like
a
tail
latency,
so
you
know
most
of
the
users
are
not
impacted
at
all.
It's
like
you
know,
50
percentiles,
it
stays
stable,
but
you
know
the
tail
latency.
So
the
unlucky
users
are
really
unlucky
now,
because
it's
not
like
only
a
couple
of
milliseconds
hundred
milliseconds
more
is
suddenly
two
seconds
or
more
to
res
to
retrieve
the
punk
when
you
are
pinging
right.
So
something
is
really
wrong
with
this
deployment.
Hopefully
our
gorillas
will
notice
that.
B
At
the
same
time,
we
don't
see
many
like
much
of
change
within
our
error
section
it's
about
the
same
for
both
of
our
deployments,
and
now
we
can
see
right
away
life
that
aggro
wallets
actually
was
not
happy
with
that
deployment
and
it
didn't
pass
our
checks,
so
it
scaled
down
right
after
two
parts
being
spun
up
from
our
new
deployment.
It
con
reverted
the
rollout
and
it's
been
back
our
initial
deployment.
We
can
see
now
that
our
rollout
reverted
to
our
initial
deployment.
B
A
Yeah-
and
we
are
super
frustrated
like
what's
going
on
with
our
coding-
I
don't
know
we
are-
we
are
bad
programmers
or
what
like,
how
how
all
those
rollers
are
failing
and
to
know
more.
I
would
love
to
know
what
what
exactly
wrong
with
this
application
and
with
the
errors,
it's
usually
easier
to
find
the
root
cause
of
the
error
like
on
the
previous
deployment.
But
now
we
have
kind
of
tricky
situation.
We
have
a
low
latency
normally
for
most
of
the
users,
but
some
users
are
experiencing
enormous.
A
You
know
wait
time
for
the
request
and
you
know
imagine
that
you
we
cannot
really
locally
test
this
out
and
and
and
find
the
root
cause.
We
cannot
reproduce
this
locally.
So
what
we
do
this
is
where
exemplars
in
pro
for
promote,
use,
exemplars
and
actually
the
whole
story-
is
that
it's
open,
metrics
exemplars,
because
prometheus
is
using
open,
metrics
data
model
which
designs
the
exemplars
and
the
metrics
of
different
types,
and
we
kind
of
scrape
those
information
directly
from
the
applications
from
both
pink
and
finger
and
the
application.
A
Now
this
exemplar
allows
us
to
specify
additional
information
that
happened
during
this,
for
example,
latency
observation.
So
with
the
latency
observation
for
every
request,
we
can
put
something
else.
What
we
put
we
essentially
put
in
our
case
the
trace
id,
because
I
didn't
mention
that
before
our
applications
are
instrumented
with
tracing
as
well,
so
every
ping
is
actually
traceable
and
the
trace
is
essentially,
you
know
a
certain
information
bound
into
the
bound
to
the
request
flow
across
different,
multiple.
You
know
microservices.
A
So
now
we
have
the
link
between
the
trace
to
and
we
put
between
the
metrics
and
the
trace
within
the
form
of
the
exemplar.
So
we
can
see
a
different
metadata,
accompanied
by
the
by
the
observation.
So
we
can
see
that
this
direct
unlucky
user
has
the
you
know
that
the
request,
where
we're
kind
of
above
two
seconds,
but
we
would
like
to
know
the
root
cause.
So
let's
go
and
and
kind
of
go
to
the
tempo,
which
is
our
tracing
solution
to
see
what's
going
on.
A
So
now
we
have
a
tracing
ui
visible
and
we
can
see
the
trace
so
essentially
the
the
the
flow
of
the
request
between
different
components
and
the
co
and
the
code
itself
right.
So
we
can
see
that
finger
is
there
is
the
green
line.
It
is.
You
know
our
client
that
seen
what
two
zero
2.01
seconds
of
latency
and
it
experienced
this
latency
based
on
some
other
things
that
happened.
So
let's
go
to
the
child
process,
which
is
essentially
it
hits
you
you
can
see
down.
A
Now
we
have
a
different
service.
It's
called
up
it's
our
service,
which
is
responding
to
our
request,
and
we
can
see.
It
also
reported
that
long
duration,
which
is
two
seconds
something
within
the
server,
maybe
introduced
this.
So
let's
go
further.
Our
pink
handler
also
see
two
seconds,
so
it's
like
a
span,
a
different
function
that
happens
in
the
code
itself,
different
phrase
and
now,
when
we
go
below,
we
can
see
that
all
of
those
are
reporting
two
seconds,
including
the
adding
latency
based
on
probability.
A
That's
that's
a
suspicious
name
by
the
way,
and
then,
but
you
know,
maybe
that's
not
the
thing
that
actually
introduced
the
latency.
Let's
go
and
see
other
requests
in
this
trace.
We
can
see
that
the
last
request
right
status,
which
is
essentially
just
responding
with
some
kind
of
bytes
to
the
user,
actually
happened.
You
know
pretty
fast,
like
almost
immediate,
it
was
only
like
0.02
milliseconds.
A
A
So
what
we
did,
we
we
essentially
simulated
some
latency
based
on
probability
and
we
can
see
the
unlucky
one
was
was
this
and
it
reduced
exactly
two
seconds.
So
we
literally
sleep
for
two
seconds
and
we
kind
of
you
know
we
choose
some
random
number.
In
this
case
it
was
like
30
33
and
he
was
within
some
probability
that
introduced
this
latency.
So
now
we
know
where
to
go
in
the
code
where
exactly
the
code
we
need
to
go
and
and
fix
it.
A
So
that's
pretty
powerful
technique
of
finding
the
root
cause
of
the
problem,
and
just
maybe
to
you
know,
show
you
exactly
how
we
did
that
we
can
jump
super
quickly
to
the
code.
It's
a
glock
application,
but
you
really
do
the
same
thing
in
other
languages.
This
is
what
is
super
powerful
with
instrumentations
and-
and
I
use
like
from
to
use
the
most
popular
promoters,
client
and
then
I
use
open
telemetry
go
instrumentation
to
combine
those
two
together.
A
A
In
our
case,
everything
is
sampled,
but
we
check
anyway,
and
then
only
then
we
put
a
special
trace
id
into
the
exemplar,
and
then
this
is
kind
of
observing
the
the
the
the
kind
of
duration
of
the
request
in
the
histogram
of
of
from
to
use
with
the
trace
id.
And
then,
if
we
don't
have
the
stress
id,
then
we
just
observe
without
trace
id.
A
So
this
is
the
whole
code
that
has
to
be
created
like
not
not
like
very
complex
thing
to
have
exemplar
being
provided
in
the
metric,
and
then
it
is
automatically
chosen
by
the
graffana.
If
you
go
to
back
to
grafana,
it
is
automatically
discovered
that
this
metric
has
some
traces.
Has
some
sorry
has
some
examples.
A
A
B
Awesome
so
now
that
we
know
what's
actually
going
wrong
with
an
application,
we
can
deploy
another
update
and
make
sure
that
we
that
we
actually
fixed
it.
So
in
this
case
we're
going
to
deploy
our
our
update
called
best.
Now,
it's
currently
spinning
up
our
new
deployment
of
our
best
application
and
scaling.
Now,
on
the
other,
let's
see
how
this
translates
into
our
dashboard
and
whether
or
not
it's
actually
improving
things.
A
B
And
now
we
can
see
our
initial
deployment
has
actually
completely
scaled
down
and
our
new
deployment
is
up
by
our
entire
replica
set
by
five.
So
it
seems
that
our
entire
robot
has
actually
passed
and
is
now
up
and
running.
Our
new
deployment
is
now
running
and
has
passed
our
checks,
and
we
can
see
that
there's
no
new
new
errors
from
our
analysis,
template
being
introduced.
This,
this
red
block
is
just
basically
showing
our
or
the
previous
errors
from
previous
rollouts,
but
our
new
analysis
has
passed
successfully.