►
From YouTube: Leveraging Service Meshes for A/B, Blue-Green, and Canary Deployments (aka "Progressi... Paul Curtis
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Leveraging Service Meshes for A/B, Blue-Green, and Canary Deployments (aka "Progressive Delivery") - Paul Curtis, Weaveworks
Do you want to maximize your service mesh capabilities by enabling canary deployments, A/B testing, Blue-Green, etc. that have come to be called “Progressive Delivery”?
Paul will discuss progressive delivery and its considerations, and show how - using the open source Kubernetes operator, Weave Flagger - you can leverage service meshes to enable progressive delivery.
We will cover:
- talk about the principles behind service mesh technologies
- give an overview of the current state of service meshes for Kubernetes
- explain how you can leverage service mesh capabilities to implement progressive delivery
- show how, by doing that, you can limit the impact of a failed deployment in production
A
Okay,
so
my
name
is
Paul
Curtis,
something
with
weave
works,
and
you
heard
Michelle
talked
about
SMI.
You
heard
Jodi
talk
about
service
mesh
and
ways
that
they're
getting
deployed
I'm
going
to
talk
to
you
about
what
Michelle
mentioned,
which
was
flagger
very
specifically.
So,
let's
get
going
so
flagger
is
a
progressive
delivery
operator.
A
The
good
thing
about
flagger
is
is
that
it
will
work
with
pretty
much
every
mesh.
That's
out
there.
It
will
perform
a
bunch
of
different
progressive
delivery
functionality,
meaning
it's
not
just
canary.
You
can
do
other
things
with
it
and
you
can
not
doesn't
necessarily
need
a
mesh
actually
to
do
some
of
those
things.
So
you
have
engine
X
in
place
or
glue.
A
Does
work,
we're
all
good
with
that,
but,
as
Michelle
mentioned,
it
implements
a
control
loop
in
the
same
way
that
kubernetes
does
in
order
to
control
how
the
traffic
is
moved
between
two
sets
of
services.
So,
as
we
go
on
I'm
gonna
ask
everybody
to
raise
their
hand
progressive
delivery.
Okay
is
a
term
oh
come
on
this
service
mesh
come
on,
eat
really:
okay,
Wow,
okay.
So
where
does
flatterfit
in
the
world?
A
So
flagger
is
the
operator
does
two
things:
it
needs
to
talk
to
the
kubernetes
api
and
it
needs
a
source
of
metrics,
so
flagger
by
default,
uses
Prometheus
metrics.
In
order
to
do
this,
you
can
see
in
the
picture
there's
a
lot
going
on
here,
but
basically
flagger
is
controlling
the
ingress
and
it's
moving
the
traffic
between
what
Stephan
calls
the
primary
and
the
canary,
so
Michelle
called
inversion,
one
inversion
to
primary
and
canary
same
terminology,
everybody
good
with
that.
So
far,
okay,
let's
talk
about
the
different
strategies.
A
Is
everybody
familiar
with
all
four
of
these
there's
actually
a
fifth
one,
but
this
makes
sense
canary.
Yes,
everybody
knows
what
a
canary
does
do
you,
where
the
term
came
from
ya,
see
some
people
do
a
be
the
difference
between
a
B
and
Bluegreen.
Do
you
have
those
handles
and
one
that
you
don't
see
that
often
it's
traffic
mirroring,
which
actually
has
a
lot
of
benefits,
especially
in
a
big
production
world,.
A
Sorry,
I'll
just
want
to
keep
my
time
all
right
so
Bluegreen.
This
is
the
kubernetes
CNI
way
of
doing
things,
okay
by
the
way
you'll
get
all
these
slides.
So
these
are
Stefan's
pictures
too.
So
the
idea
behind
Bluegreen
is
very
simple:
it's
a
rolling
update!
Okay!
It
says
that
if
I
have
five
pods
of
version,
one
I'm
going
to
cross
over
and
roll
in
five
pots
of
version
two
and
the
way
that
typically
happens
is
over
time.
You
can
control
it,
but
fundamentally
it's
a
manual
process
to
you
a
B.
A
This
is
kind
of
the
old
insiders
trick,
so
you
deploy
version
two
or
the
current
the
secondary
and
you
let
some
people
try
it
out
right.
So
this
half
of
the
room
gets
a
and
this
half
of
the
room
gets
B
for
some
period
of
time.
When
you're
everybody's
happy
that
everything
is
cool,
then
I'm
going
to
switch
and
make
version
one
where
the
old
version,
the
new
one
so
version,
one
to
version
two
and
then
I'm
going
to
get
rid
of
my
old
pods
a
lot
of
times.
A
This
is
done
with
special
features
like
I
added,
a
URI
or
I
added
a
service
and
I'm
going
to
test
it
with
some
a
small
segment
of
my
user
population.
So
far,
so
good
everybody
makes
sense.
Does
everybody
understand
the
difference
between
this
and
a
Bluegreen?
Please
raise
your
hands
wow,
that's
much
better!
Okay.
That
makes
me
a
little
happier.
So
now
we
get
on
to
the
fun.
Everybody
loves
the
Canary
all
right.
A
The
idea
behind
the
canary
is
very,
very
simple:
a
canary
is
going
to
progressively
move
traffic
from
the
primary
to
the
canary
over
a
period
of
time
up
to
50%
when
it
reaches
50%
and
will
then
be
able
to
make
a
decision
to
move
all
of
the
traffic
over
and
make
version
2
the
new
primary.
So
everybody
get
that
no
problem
with
that
question
right,
good,
so
two
things
you
have
to
keep
in
mind
with
a
canary
deployment
versus
some
of
the
others.
First
off.
A
In
order
to
make
this
work,
you
need
to
have
some
metrics.
You
need
to
have
a
general
idea
of
how
you
want
this
to
work
and
what
the
time
is.
Okay,
you
also
need
to
be
able
to
see
what
happens
as
you're.
Doing
this
progressively
see
an
a/b
or
a
Bluegreen.
It's
like
a
Frankenstein
switch.
You
turn
the
switch
and
it's
either
gonna
work
or
it's
not
gonna
work
and
you're
gonna
know
real
quick.
You
still
should
probably
have
some
metrics
okay
and
the
rollback
is
basically
a
one-shot
thing
right,
just
turn
it
off
canary.
A
A
The
last
deployment
I
want
to
talk
about
is
mirroring
this
one's,
not
as
common,
but
this
is
a
very,
very
simple
way
to
do
this,
because
it
gives
you
an
advantage
that
a
B
and
blue/green
don't
alright
same
idea.
You
have
a
primary
and
you
have
a
secondary
or
a
version,
one
inversion.
Alright,
so
what
you're
doing
is
you
basically
deploy
version?
A
Two,
the
green
you
test,
okay,
make
sure
it
works,
and
then
you
mirror
the
traffic
so
that
both
the
blue
and
the
green
version,
1
and
version
2
are
getting
the
exact
same
traffic
all
right
now.
That
means
that
that
is,
like
you
know,
trial
by
fire
for
version
2,
because
he's
gonna
get
the
full
load
of
everything
that
the
current
running
version
is
getting
okay.
A
The
good
part
about
this
is,
is
that
you're
gonna
know
real
quick,
whether
version
2
is
gonna
hold
up
or
not,
because
it
will
either
run
or
it
will
die
horribly.
Okay,
you
also
collect
metrics
across
there,
so
that
you
can
keep
track
your
for
whatever
reason.
The
reason
you
deployed
version
2,
but
you
want
to
keep
track
of
the
metrics,
make
sure
it
performs.
Okay
and
there's
no
big
problems.
A
A
Yes,
thank
you.
Okay,
good
I
know
it's
afternoon
and
it's
about
snack
time
and
everybody's
starting
to
fade.
Ok,
so
what
is
flag
or
do
alright?
So
flagger
is
an
operator
as
I
mentioned,
and
it's
been
simplified
over
time.
Flaggers
configuration
and
operation
is
using
get-ups,
so
fundamentally
thing
that
we've
works
pushes
into
is
using
get
for
all
configuration
and
source
of
truth.
A
Alright,
so
think
about
it
assume
you
have
a
pod
deployment
out
there
you
have
to
create
the
canary,
you
have
to
create
the
IP
service
for
it.
You
have
to
make
sure
it's
deployed.
Okay,
it
has,
and
the
now
the
canary
has
to
have
its
own
set
of
IPs
its
own
HPA
if
you're
going
to
use
it
and
its
own
deployment.
So
now
what
you
basically
have
to
do
is
create
all
that
you
amyl
to
do
that
or
copy
and
paste
it
or
whatever.
Now,
let's
go
down
a
service
mesh
at
minimum.
A
You
have
to
have
two
sets
of
routes:
okay,
because
you've
got
to
completely
sets
of
two
deployments.
You
have
to
have
the
rules
that
you're
going
to
implement
for
how
to
do
this
and
the
service
mesh
has
to
understand
how
to
do
it
because
the
canary
is
not
like
all-or-nothing.
It's
like
one,
two,
three
four
one,
two
three
four
like
that,
so
this
mesh
has
to
be
able
to
handle
the
fact
that
it's
going
to
shift
traffic
under
control
back
and
forth.
A
You
need
all
the
verge
virtual
services
to
do
this
right,
so
there
are
going
to
be
a
primary
and
a
secondary
service.
So
all
of
this
stuff
here
on
the
left,
gets
to
be
timing
out
on
the
screen
display
right.
There's
all
these
things.
Now,
if
you
go
over
to
the
right,
flaggers
canary
consists
of
two
things:
you
need
a
deployment,
some
kind
of
deployment
and
by
the
way,
flagger
uses
the
name
of
that
deployment.
A
Okay,
so
keep
that
in
mind,
so
you
do
have
to
tag
and
do
proper
naming
in
your
deployments
and
it
needs
the
flag
or
canary
object,
and
that's
it.
That's
all.
You
have
to
do
pretty
magical
okay.
So
what's
the
CR
D
look
like
alright,
so
this
is
gonna,
be
a
little
hard,
I'm
gonna
actually
step
down,
because
it
makes
it
a
lot
easier.
If
I
do
that,
so
there's
two
parts
of
what
you're
looking
at
on
the
left
hand
side.
You
have
a
very
good
idea
of
what
this
looks.
Like
does
everybody
understand.
A
What's
going
on
when
I
said,
the
name
of
the
deployment
is
critical.
It's
because
the
metadata
pot
info
name
is
how
it
does
the
hook-up.
That's
how
it
finds
it.
Okay
and
the
target
is
the
other
way
it
needs
to
have
them
bolt
the
bottom.
If
you
use
an
HPA,
does
everybody
know
what
an
HPA
is
horizontal
pod,
auto-scale
right,
fine,
okay,
you
is
not
is
an
option,
don't
have
to
do
it?
Okay,
if
you're
pretty
confident
that
you
ploy
him
it's
gonna,
be
cool
and
it's
not
gonna
go
running
wild.
A
Then
it's
up
to
you
flip
side
of
this
on
the
other
side.
Okay,
these
are
the
beginnings
of
the
things
that
you
need
to
make
sure
that
how
the
Canaria
gets
run,
what
it
is
looking
at
and
the
routing
for
it.
We're
gonna
talk
a
little
bit
more
about
how
the
threshold
in
works
in
the
next
slide
I'm
trying
to
keep
you
awake.
A
Okay,
this
makes.
Is
this
clear
to
everybody
what
we've
done
here
so
we've
defined
the
targets
and
we've
defined
the
routing
for
those
targets
now
what
is
happening
underneath
is
very
nice,
so
Stefan
has
made
this
such
that
this
CR
will
go
ahead
and
do
all
of
those
things
in
this
list
on
the
left
on
your
behalf.
A
Okay,
so
if
you
think
about
what
has
to
happen,
is
the
only
thing
that
you
have
to
tell
it
is
the
routing
and
the
pot
info?
Basically,
the
only
other
requirement
for
this
to
work
is
you
have
to
have
prometheus.
We
have
to
have
some
way
to
get
metrics.
Is
that
clear,
no
questions
we
like
that?
Okay,
so
now,
how
do
you
do
the
analysis?
This
is
even
more
fun
and
I
did
ask
a
question
of
Stefan
when
I
did
this
so
canary
works
on
a
time,
okay
and
a
step
interval
all
right.
A
The
idea
here
is
is
that
over
a
certain
amount
of
time,
I'm
going
to
shift
the
traffic
by
some
fixed
amount
from
the
primary
to
the
canary
or
from
blue
to
green
or
primary
to
secondary,
whatever
your
terminology
is,
and
it
is
your
choice
as
an
application
developer
or
an
operator
figure
out
what
those
numbers
are
now,
how
many
have
done
this
in
production?
How
many
of
done
canary
progressive
deployments
one
two:
three:
okay:
five
in
a
room
of
a
hundred
people?
A
Okay,
so
this
is
not
cutting-edge
idea,
but
it's
something
you
have
to
think
through
and
I'm
going
to
give
you
a
reason
why
I
want
to
talk
about
this
on
the
left.
From
my
background,
is
a
developer,
so
I
need
to
think
about
something
very
simple:
what
interval
do
I
feel
safe
to
ramp
up
my
application,
because
a
blue,
green
or
an
a/b
I
mean
it's
pretty
much?
It's
all
or
nothing.
It's
a
binary
thing
right
in
doing
progressive
delivery.
In
this
way,
I
get
to
be
able
to
do
two
things.
A
It's
kind
of
like
everybody's
flown,
a
jet
right.
Okay,
you
know
what
they
do
right
at
the
beginning
of
the
take-off
right
before
they
actually
floor
it
do
you
know
what
they
do.
They
take
the
engines
to
50
percent
and
they
let
them
you
hear
them.
They
spin
it
up.
You
know
why
they
want
to
make
sure
everything's
okay
before
they
hit
it
and
make
it
go
to
a
hundred
percent.
This
is
the
same
idea
and
it
has
the
same
results,
but
you
got
to
know
what
these
numbers
are.
A
What's
a
reason,
what's
a
reasonable
criteria
for
you
now
there
was
a
production
one,
a
couple
of
people
in
production.
Can
you
just
tell
me
what
your
what
your
interval
time
was?
I'm
just
curious,
dad,
there's
somebody
down
here:
what
was
the
interval?
How
long
did
you
go
from
zero
to
50%
near
canary?
Do
you
remember.
A
A
Right
there
was
somebody
else
down
this
side.
What
was
yours
two
and
five
minutes
one
week,
two
in
five
minutes.
Okay,
do
you
think
the
application
services
they
were?
Testing
are
exactly
the
same?
Probably
not
I
came
out
of
the
big
data
world.
We
would
do
this
for
prediction.
Models
would
be
an
hour
an
hour
to
two
hours
so
that
we
would
know
that
if
the
correlation
rate
was
going
off
in
an
hour
like
we
were,
you
know
telling
the
lady
to
buy
carrots
when
she
was
looking
for
pantyhose,
probably
not
a
good
recommendation.
A
A
The
max
weight
is
remember
our
diagram,
how
much
traffic
goes
to
the
primary
and
how
much
goes
to
the
Canary.
That's
what
that
number
is.
Fifty
percent
goes
to
the
canary
we're,
never
going
to
turn
a
hundred
percent
over
all
right.
This
is
pretty
much
a
given
the
step
weight,
every
increment
that
it
increments,
that
progressive
delivery.
That's
that
number
so
over
a
week,
here's
your
steps,
probably
going
to
be
pretty
small,
like
a
percent,
maybe
two.
A
Right
so
there's
another
one:
where
you
do
this
in
testing.
There
may
be
a
reason
where
that
interval
is
very
long
and
the
step
is
very
small.
But
when
you
get
more
confident
with
your
testing,
you
can
shorten
the
interval
and
up
the
step
so
that
it
happens
quicker,
but
each
step
is
something
you
want
to
monitor,
because
you're
gonna
want
to
see
what
happens.
Is
each
step
goes
on,
so
don't
make
the
interval
too
short,
because
otherwise,
you're
not
gonna,
have
enough
time
to
get
any
metrics
out
of
it.
A
Okay,
now,
on
to
the
other
side,
flagger
has
a
couple
of
nice
features
and
a
couple
of
ones
I
really
really
like.
Okay,
the
first
thing
is,
is
that
flagger
will
look
at
splat
thresholds
and
thresholds
can
be
based
pretty
much
on
any
standard
service
mesh
metric
all
right.
The
service
measures
report
a
lot.
A
Okay,
the
key
one
just
on
these
two
is
right.
Does
everybody
know
what
red
metrics
are
ever
heard?
The
term?
No?
Okay?
It's
a
couple
over
here
so
service
match
reports,
request,
delivery,
error
rate
and
duration,
red
okay,
that's
two
of
the
three
you
can
put
the
third
one
up
here
as
well,
alright,
but
that's
a
very
standard
way
to
view
a
web
based
service
or
a
REST
API
service,
or
something
that
uses
standard,
networking
connections,
so
red
metrics,
very,
very
simple,
but
in
my
world,
coming
out
of
big
data
and
model
testing.
A
I'm
red
is
okay,
but
it
doesn't
tell
me
if
the
model
is
actually
doing
anything
useful
alright.
So
there
may
be
application
specific
metrics
that
your
developers
put
in
that
are
things
that
they
know
about
their
application
that
are
critical
and
in
the
model
world
it's
called
correlation
rate
or
error
rate
hit
rate.
Okay.
A
Things
like
that
that
says
you
know:
I
want
to
make
sure
that
my
correlation
rate
always
stays
above
90%
and
my
errors
are
below
two
right,
because
I
don't
want
the
model
to
do
something
really
stupid
like
send
carrots
to
the
pantyhose
blind,
lady,
not
a
good
plan
right.
So
to
do
that,
you
can
put
in
other
types
of
metrics
here
anything.
A
What
are
these
do
so,
just
to
give
you
a
general
idea,
request,
duration,
request,
success
rate.
Let's
do
the
first
one
says:
I
have
to
have
a
success
rate
of
99
percent
over
the
course
of
a
minute
if
it
goes
below
that
if
it
goes
to
98
percent
in
160,
second
interval
I
fail
and
I
begin
to
roll
back
now
understand
what
this
does.
Flagger
reverses
the
process
goes
backwards,
so
if
you're
at
45%-
and
you
hit
an
error
rate-
that's
too
high.
It
goes
40,
so
it
moves
it
back.
A
A
Yep
got
my
clock,
so
the
last
thing
here
with
flagger
as
well
as
Michelle,
mentioned
somebody
who
really
likes
us,
as
you
can
actually
put
in
web
hooks
to
have
it
do
stuff.
So,
for
example,
if
you
want
to
actually
have
it
start
up
a
load
test,
if
you
have
a
service
that
isn't
under
great
load
or
you
have
a
very
specific
set
of
corner
cases,
you
want
it
to
do
right.
When
you
start
the
canary,
you
can
say:
hey
go!
Do
this!
This
should
be
very
familiar
syntax
to
anybody.
Who's
done
like
pods
back.
A
A
Custom
metric
checks,
Prometheus
queries,
so
anything
that
you
can
get
impromptu
out.
You
can
turn
it
into
a
metric
and,
if
you
go
to
the
how-to
for
flagger
in
the
documentation
actually
has
a
nice
little
example,
so
you
could
do
things
like
error
rate
or
percentage
error
rate
or
a
rolling
average,
or
you
know
any
number
of
those
kinds
of
things.
Last
thing
is
the
web.
A
Hooks
are
actually
kind
of
cool,
because
what
it
means
is
that
you
can,
in
a
test
environment
where
you
may
not
be
running
a
kubernetes
test
or
dev
cluster,
that
has
real
traffic
like
it's,
not
the
real
system,
but
you
want
to
still
do
a
progressive
like
a
canary
test.
You
can
do
this
in
QA
right
and
all
you
do.
Is
you
use
the
web
hook
to
generate
traffic
right?
A
Remember,
traffic
mirroring
back
in
in
the
beginning.
Qa
is
a
perfect
place
to
use
traffic
Mehring.
How
do
you
get
traffic
mirroring
right?
You
can
just
turn
it
on.
So
there's
a
lot
of
ways
that
you
can
use
web
hooks
and
get
sample
traffic
generated
traffic
or
mirrored
traffic
in
order
to
do
your
testing.
A
Obviously,
flagger
has
a
bunch
of
built
in
stuff,
but
for
the
most
part,
alerting
is
kind
of
done
nicely
emails.
Obviously
one
of
the
choices
but
the
real-time
alerts.
Slacking
teams
seems
to
be
pretty
much
the
common
so
now
I'm,
just
gonna
for
the
last
minute
I
have
is
to
talk
about
what
this
looks
like
in
a
whole,
get
ops
pipeline
so
get
ups.
For
those
who
don't
know
is
a
method
method
of
using
it
as
your
sole
source
of
truth
for
declaring.
A
A
Everything
you
declare
for
flagger
to
do
is
to
clear
it
and
get
ok,
you
can
roll
it
back.
You
can
use
the
past
one
and
once
the
canary
is
deployed,
flagger
will
update
the
git
repo
with
the
correct
new
primary
images,
so
it'll
actually
keep
it
up
to
date,
so
it
makes
it
very
simple
to
do
if
it's
unsuccessful.
A
Obviously,
it's
not
going
to
do
much
with
that,
but
if
it
is
successful,
you'll
be
able
to
know
that
you'll
be
able
to
see
it
up
front
very,
very
quickly
and
I
wish
I
had
that
clicker
thing
this
also
works
with
helm,
so
flux
works
with
standard
manifests
any
kubernetes
object.
You
can
throw
home
charts
in
there
as
well.
We
kind
of
like
using
helm,
ok
process
is
the
same.
The
only
difference
in
that
picture
from
the
one
before
is
flux
is
using
helm.
Now,
here's
the
key
slide.
A
Alright,
everybody
get
your
camera's
out,
okay,
so
the
repose
at
the
top.
Ok,
a
nice
discussion
of
Sto
as
an
example,
a
nice
one
with
flagger
and
linker
D
and
down
at
the
very
bottom,
with
AWS
at
mesh.
As
I
said
in
the
very
beginning,
the
at
mesh
is
new.
Ok,
that
was
just
added
to
flagger.
So
that's
something
that's
fairly
new
to
what
we're
doing
Stefan
protein,
who
actually
wrote
this
is
actually
around.
A
He
is
not
here
today
because
he
is
over
at
AWS,
giving
the
exact
same
talk.
Ok,
so
two
things
our
boo
number
come
by.
We
have
this
interesting
little
contest
that
you
can
do
on
your
own
time
and,
lastly,
I
recommend
for
people
who
are
interested
in
the
get-ups
methodology
and
building
declaring
using
declarative
source
code
control
for
controlling
your
clusters.
That
is
to
talk
to
go
to
ok,
it
is
on
Wednesday.
Now
before
we
go
since
I've
got
exactly
30
seconds.
Does
anybody
recognize
that
ID?
That
is
my
ID
one
of
my
IDs?