►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Continuous delivery has been a driving force behind software development, testing and deployment for years, and CI/CD best-practices are evolving with the advent of new technologies like Kubernetes and Istio.
Progressive delivery, a term coined by James Governor of Redmonk, is a new approach to continuous delivery that includes “a new basket of skills and technologies… Such as canarying, feature flags, [and] A/B testing at scale”.
Andrew will demo a canary deployment combining service mesh and Flagger, a progressive delivery operator. Traditional canary deployments shift traffic, so bad canaries still partially degrade user experience - Andrew will show an enhancement that mirrors traffic before shifting, which can prevent bad canaries from being visible to any users. He will compare traffic mirroring to other techniques, exploring when you might choose what kinds of progressive delivery.
A
First
I'd
like
to
thank
everybody
for
joining
us
today.
Welcome
to
today's
webinar
delivering
progressive
delivery
with
service
mesh
I'm
Taylor
Wagner,
the
operations
analyst
for
CNCs
I'll
be
moderating
today's
webinar.
We
would
like
to
welcome
our
presenters
today,
Andrew
Jenkins
CTO
and
Zak
jewelry,
head
of
marketing
for
ask
the
mesh
before
we
get
started.
There's
a
few
housekeeping
items
we'd
like
to
go
over
during
the
webinar.
You
are
not
able
to
talk
as
an
attendee.
There
is
a
Q&A
box
at
the
bottom
of
your
zoom
screen.
A
B
Great
thanks,
Taylor
actuary
here,
had
a
marketing
looking
forward
to
spending
a
few
minutes
up
front
and
the
presentation
talking
about
press
and
delivery,
some
of
the
concepts
behind
it.
Why
we
think
matters
and
why
we
think
the
service
mesh
helps
with
that
and
then
I
will
wait
to
the
fun
bit
after
that,
we'll
Andrew
we'll
do
a
couple
of
demos
showing
how
you
can
actually
use
service
mesh
to
deliver
progressive
delivery.
So
progressive
delivery
is
an
idea.
B
I
should
give
credit
where
it's
due
that
James
governor
of
Redmond
introduced
a
year
and
a
half
or
so
ago
there
were
a
couple
of
companies
that
were
doing
progressive
delivery.
We
works,
launched
darkly
the
engineering
team
at
Target.
We're
using
a
stack
of
tools
to
I
would
say:
do
a
new
new
method
of
CIC
be
around
kubernetes
things
have
kind
of
changed
the
way
we
deliver
applications.
B
B
In
this
way,
it's
really
cool,
because
progressive
juggler
delivery
means
software
deployment
does
not
necessarily
equal
software
activation,
which
means
that,
just
because
your
dev
team
is
doing
deploys
doesn't
mean
you
have
to
roll
out
those
features
to
all
your
end
users
at
once.
This
is
very
similar.
You
know
these
thoughts
from
CI
I
like
to
think
of
this
as
CI
2.0
kind
of
the
new
requirements
for
how
you
deliver
software
in
the
micro
services,
world
CI
was
really
great
for
the
business
because
it
gave
the
business
away
to
limit
the
blast.
B
Radius
around
software
deploys
so
deploys
weren't
as
scary
I.
Think.
Progressive
delivery
takes
that
a
step
further
by
not
only
allowing
you
to
turn
hand,
grenades
into
firecrackers,
but
also
being
able
to
put
those
firecrackers
out.
Very
quickly,
if
there
are
a
problem
and
even
put
a
band-aid
if
they
have
one
of
those
firecrackers
causes
a
wound.
B
One
of
the
really
neat
things
about
for
instant
delivery
is
the
business
actually
cares
about
it.
This
controlled
empowerment
idea
is
around
the
sense
that
the
business
can
say
when
it's
acceptable
for
new
code
to
hit
the
end
user.
So
dev
teams
are
not
constrained
by
the
business.
They
can
still
do
rapid
iterative
development,
but
the
business
can
say
we're
gonna
package
that
all
up
and
we
can
wait
for
six
ten
twelve
deploys
until
we
activate
that
for
the
end
user.
This
is
really
cool
because
it
gives
the
business
psychological
safety
around
deploys
and
I.
B
So
there
there
are
a
couple
different
pieces
here.
One
is
psychological
safety
for
the
business
I
talked
about
in
the
last
slide.
The
other
one.
That's
really
neat
is
the
fact
that
this
provides
a
much
better
developer
experience
decoupling
releases
from
rolling
out
any
features.
The
customers
is
a
really
good
way
for
the
business
to
be
satisfied
without
forcing
dev
teams
to
comply
with
business
controls
and
concerns.
B
The
part
we're
going
to
focus
on
the
second
half
of
this
webcast
is
about
how
service
mission
navels,
presses
delivery,
feature
flags
and
rolling
deploys,
or
some
things
launch
dark.
That
I
mentioned
earlier
is
a
great
tool
to
use
feature
flags
for
a
progressive
delivery.
One
of
the
things
that
we
have
been
using
with
our
customers
is
a
service
mission,
and
it
allows
us
to
do
progressive
delivery.
B
Things
like
canary
and
traffic
mirroring,
which
are
really
cool
because
you
can
roll
out
an
experimental
service,
see
how
that
service
is
performing
with
a
set
of
users
and
use
those
metrics
to
decide
if
you
rolled
out
further
or
roll
it
back,
and
so
with
that.
I
think
that
I
will
turn
it
over
to
Andrew
here
to
deep
dive
a
little
bit
more
into
how
you
can
use
canary
and
traffic
mirroring
and
do
a
couple
demos.
B
C
So
you
know
is
actually
mentioned
right:
you've
got
the
you've
got
a
bunch
of
sort
of
components
to
progressive
delivery.
It's
all
about
decoupling,
the
the
deployment
of
software
from
the
way
that
users
for
the
counter,
so
that
way,
businesses
would
make
trade-offs
and
that
can
involve
kind
of
independently
of
a
decouple
and
let
software
teams
go
as
fast
as
they
as
they
can.
C
However,
you
still
need
some
stuff
to
manage
how
you're
going
to
actually
activate
it
for
new
users,
how
you're
going
to
deploy
it,
and
so
part
of
that
story
is
around
sort
of
these
news
with
deployment
techniques
for
deciding
how
you're
going
to
shift
traffic
and
expose
it
to
users.
So
what
we're
going
to
talk
about
today,
sort
of
demo
is
canary
deployments
and
traffic
mirroring,
which
is
a
kind
of
a
new
thing
that
a
service
mesh
lets.
C
You
do
that's
as
some
interesting
sort
of
opportunities
for
improving
the
way
that
you,
you
might
have
used
or
heard
about
Canaries
before
so.
What
we're
going
to
demo
here
is
using
a
progressive
delivery
operator
called
flagger,
and
so
this
is
kind
of
the
flagger
architecture
and
so
flaggers
job
is,
to
you
know,
sort
of
employee
of
canary
and
then
shift
traffic
to
that
canary,
based
on
different
criteria.
For
this
demo,
we're
going
to
step
through.
D
C
Know
ramping
some
random
subset
of
traffic.
There
are
other
options
as
well
and
the
and
then
from
flagger
observes.
What
happens
to
that.
You
know
that
new
canary
service
compares
the
performance
of
the
canary
to
the
primary
service
and
continues
with
its
progressive
deployment
and
activation
of
the
canary
service
as
long
as
that's
healthy
and
if
it
becomes
unhealthy.
C
If
the
canary
is
now
performing
out
of
spec
relative
to
the
primary
or
relative
to
absolute
markers,
then
you
can
say:
okay
wait
hold
on
this
activation
is
failing
and
so
we're
gonna
we're
gonna
roll
back
to
the
primary
we're
going
to
cancel
this
activation
and
we're
going
to
let
somebody
figure
out
what's
wrong.
Wait
for
a
new
candidate
version
that
we
can.
C
We
can
try
activating
along
those
same
lines
so
and
the
the
service
mesh
tie
in
here
is
that
for
actually
choosing
what
traffic
goes
to
the
primary
and
what
traffic
goes
to
the
Canary.
You
can
use
a
service
mesh
to
do
that
kind
of
traffic
shifting
and
then
later
we'll
see
how
you
can
also
do
something
called
traffic
mirroring.
That
will
let
you
do
a
little
bit
of
both
before
you
sort
of
activate
the
canary
at
all
and
we'll
talk
about
why
that's
that's
good
or
bad,
so
this
is
a.
C
This
is
an
example
of
a
canary
deployment
through
a
time
line,
and
so
what
we're
looking
at
here
is
we've
got
an
old
version
of
service
deployed
and
fully
activated.
100%
of
traffic
is
going
to
the
old
version
of
the
service.
We've
got
a
new
version.
It's
passed,
you
know
our
continuous
integration
tests.
C
D
C
Shift
100%
of
traffic
to
the
new
service.
So
this
is
this
kind
of
like
shift
measure
shift,
measure
or
sort
of
loop
and
the
the
service
mesh
is
involved
in
shifting
traffic
to
the
new
service,
as
well
as
collecting
the
measurements
in
a
way
that
we
can.
We
can
compare
so
at
this
point,
I'm
going
to
shift
over
to
the
demo
and
go
through
we'll
sort
of
see
this
in
action.
So,
let's.
D
C
C
So,
of
god,
make
sure
Prometheus
is
up
and
going
alright
and
I'm
gonna
shrink
this
so
that
we
can
see
sort
of
just
the
current
amount
of
time
here.
So
right
now,
I've
got
my
old
service
running.
This
is
it's
taking
all
the
requests
currently
and
nothing
is
shifting
over
to
the
new
service
and
what
we're
going
to
do
is
I've
got.
So
this
is
a
table
we'll
watch.
C
D
C
So
this
this
canary
is
going
to
say,
hey
every
30
seconds,
we're
going
to
move
one
more
step
through
our
progressive
delivery,
our
canary
deployment
and
what
we're
going
to
do
is
we're
going
to
compare
the
request,
success
rate
and
the
request
duration
between
these
two,
the
primary
service
and
the
new
canary
service.
These
are
handles
for
cans
from
Prometheus
queries
inside
a
flagger
that
go
and
look
at
the
service
mesh
underneath
like
Aspen
mash.
That
is
collecting
the
metrics
from
the
scenario
deployment
and
then
here
we're
going
to
say
we're
going
to
do.
C
C
What
we've
got
right
now
is
we're
running
a
basic
primary
service
version.
Two
point:
one
point:
one
we've
got
our
canary
configured,
flagger
is
sitting
there
ready
to
go,
and
so
what
we'll
do
is
we
will
try
to
update
this
image
to
be
a
new
version?
Two
point
one
point:
two
and
I'm
going
to
do
this
manually
and
directly
in
the
demo,
but
you
know
in
production
usage.
This
is
driven
by
whatever
sort
of
your
deployment
scenario
is.
If
you're
using
you
know,
get
ops,
then
it's
your
get
ops
operator
for
configuring.
C
This,
maybe
it's
just
you
know
a
CD
system.
That's
driving
this
for
you,
whatever
it
is,
but
for
the
purposes
of
this
demo,
we're
just
going
to
play
around
with
with
updating
this
image
directly,
so
I'm
taking
the
place
of
you
know
your
good
ops,
operator
and
I've
just
said:
okay,
great
we're.
We're
now
saying
that
we
want
to
try
this
candidate
image
pot
info
two
dot,
one
dot
two
and
we've
got
the
little
canned
Watchers
that
we
can
kind
of
see.
C
What's
what's
going
on
here,
so
flagger
has
noticed
the
new
revision
detected,
and
so
it's
deploying
some
two
point.
One
point
two
pods
and
it's
going
to
wait
for
those
to
get
started
and
show
up.
But
right
now
it's
still
saying:
hey:
all
the
traffic
is
going
to
go
to
this,
this
primary
version
of
the
service.
C
C
C
Two
point:
one
point:
two
version
is
our
new
primary
version
and
complete
the
the
deployment
to
go
back
over
here
and
a
little
bit
more,
the
traffic
continuing
to
step
up
on
the
the
canary
version
of
the
service
and
starts
with
marching
down
on
the
primary
version
of
the
service
keep
shifting
traffic,
and
this
it's
for
the
purpose
of
the
demo.
You
know
I've
got
a
pretty
quick
ramp
here,
30
seconds
only
on
on
each
sort
of
stage.
That's
to
make
the
demo
go
fast
enough
that
it's,
you
know
interesting
to
watch
in
production.
C
You're,
probably
looking
at
you
know
letting
this
soak
for
quite
a
bit
longer,
depending
on
how
rapidly
you
sort
of
do
these
these.
These
deployments
right
he's
continuing,
is
from
March,
more
and
more
traffic,
shifting
to
the
canary
version
of
the
service
and
less
and
less
the
primary
version
of
the
service.
C
And
we're
finally
at
5050,
and
so
this
means
that
you
know
flagger
has
now
decided
that
hey
everything's
good
and
we're
ready
to
go
ahead
and
make
two
point
one
point
to
the
new
versions.
This
is
a
successful
canary
deployment
from
flagger,
and
so
it's
going
to
update
the
primary
to
be
our
new
two
point.
One
point
two
version,
and
it
says
you
know
it's
a
finalizing
this
canary,
getting
everything
else
done
yeah.
So
here
goes
shifting
all
traffic
over
to
primary
and
then
it'll
start.
C
You
know
the
canary
is
now
ready
to
be
used
again
for
the
next
sort
of
Kanak
canary
to
plan,
and
so
we'll
see
here
take
a
little
while
for
the
Prometheus
metrics
to
soak
through,
but
then
we'll
go
back
to
the
stage
where
all
the
traffic
is
going
to
the
primary.
You
know
the
primary
is
now
new.
It's
that
two
point.
One
point:
two
version
of
the
service
there
you
go
so
now,
it's
all
shifted
over
traffic
is
is
totally
shifted
to
the
primary
okay.
C
So
that's
a
successful
canary
canary
deployment
with
traffic
shifting
I
can
show
you
the
next
one.
Here
is
what
happens
on
failure,
and
so
what
happens
on
failure?
That's
pretty
cool
is
so
we
can
shift.
We
can
measure,
you
can
say,
oh
wait,
this
traffic
is
not
not
as
healthy
as
the
old
traffic,
and
so
we
should
actually
roll
back
and
sort
of
step
through
that
scenario
now
and
the
one
thing
that
I
know
I
could
watch
out
for
here.
D
C
C
This
is
a
busted,
build
that
somehow
slipped
through
CI
and
it's
got
some
bug
where
it
responds
with
failures
to
just
you
know
some
some
subset
of
requests,
and
so
you
can
now
detect
this
with
the
canary
and
we'll
see
how,
by
doing
it,
with
traffic
shifting
or
some
other
sort
of
thing
you
can
limit.
Just
as
Zack
was
saying
you
can
limit
this,
the
impact
of
letting
one
of
these
broken
deployments.
D
C
Let
me
flip
back
to
my
launch
loop
here
and
we
can
see
okay,
I've
told
you
know:
I'm
pretending
to
the
CIA
or
get-ups
operator
again
here
and
I've
said:
okay,
we're
going
to
try
to
playing
this
new
image,
and
so
once
Liger
notices,
this
guy
he's
gonna
go
ahead
and
start
scaling
that
up
there
you
go
so
flagger
bugger
notices,
flaggers,
saying:
okay
cool!
I'm
going
to
execute
this
canary!
Now,
let's
go
ahead
and
scale
up
that
container,
get
it
to
be
available
and
then
start
shifting
some
traffic.
C
C
C
So
this
this
top
plot
will
show
only
successful
traffic
that
didn't
have
500s
and
down
here.
I'll
show
you
traffic
that
does
have
500,
so
we'll
see
a
some
amount
of
traffic
shifting
from
the
primary
over.
So
here
it
goes.
You
see
a
little
dip
there.
You
don't
see
this
count
up
in
this
view,
because
this
is
all
these
are
all
errors,
but
what
you'll
see
here
is
there
are
now
a
high
rate
of
errors
coming
out
of
out
of
this
service.
C
If
you
go
over
to
flagger,
we'll
see,
oops
halt
pot
info
test
advancement
success
rate
is
less
than
99
percent,
so
this
is
not
happy,
so
we're
not
going
to
progress
and
fire
will
sit
there.
It's
going
to
try
a
couple
times
and
then
say
that
this
is
all
busted
and
it
will
roll.
It
will
just
sort
of
shift
traffic
back
100
percent
two
primary
zero
percent
to
canary,
and
it
will
mark
this
canary
as
failed,
and
so,
if
you're
an
operator,
you
can
go.
Look
at.
You
know
why
what
happened
to
this
build?
C
B
C
This
is
no
good
routes,
you
know
100
zero
back
and
then
left
us
a
note
that
the
canary
has
failed,
and
now
we
can
go,
take
action,
but
we've
automatically
rolled
back,
so
our
production
deployment,
all
of
our
users,
are
seeing
responses
from
2.1.
The
healthy
two
point,
one
point
two
deployment,
and
so
now
we
can
pause.
You
know
we
don't
have
to
panic.
We
can
take
our
time
to
figure
out
exactly
what
home.
C
We
want
to
talk
about
something
something
that
you
know
we
when
we
saw
this,
you
know
we
thought
that
service
fish
an
opportunity
to
do
something
pretty
cool,
and
so
we
did
an
experiment
as
adness
to
flagger,
and
this
is
not
committed
to
upstream
flagger.
Yet
because
it's
not
ready,
we
haven't
figured
out
enough
about
how
other
service
meshes.
Do
this
sort
of
thing
there's
some
caveats
that
we'll
go
through,
but
this
is
the
the
sort
of
traffic
mirror
in
case
the
source
code
is
available.
C
Look
if
you
go
to
that
demo,
you
can
see
the
source
code.
It's
just
this.
This
part,
this
mirroring
part,
is
not
really
probably
ready
for
production
usage
yet
and
we'd
like
to
get
it
upstream
into
flagger,
so
that
it's
available
for
everybody,
so
traffic
mirroring
is
cool
because
it's
a
little
bit
of
a
difference.
So
in
this
mirror
increase
stage
we're
going
to
actually
duplicate
the
request.
One
copy
is
going
to
go
to
our
primary
and
one
copy
is
going
to
go
to
the
canary.
C
The
user
sees
only
the
response
from
the
primary,
so
the
service
match
is
going
to
take
care
of
copying
these
requests
and
it's
going
to
take
care
of
taking
the
response
back
from
the
primary
and
sending
it
on
to
the
user,
for
the
duplicate
that
it's
sent
to
the
canary.
All
it's
going
to
do
is
collect
the
response,
drop,
the
response
on
the
floor,
but
still
collect
a
metrics
and
statistics
about
how
long
it
took
and
about
whether
it
was
a
success
or
a
failure.
C
But
then
it's
not
going
to
send
that
it's
not
going
to
send
the
actual
response
on
to
anybody
like
the
end
user.
It's
just
going
to
kind
of
drop
it
on
the
floor,
so
what
this
enables
that's
pretty
cool
is
that
if
the
canary
is
a
bad
release,
you
can
detect
it
without
actually
sending
any
bad
responses
back
to
the
end-users.
C
So
in
that
traffic
shifting
case
you
know,
we
said:
okay,
we're
going
to
shift
10%
of
traffic
to
the
to
the
canary
bill
right,
and
so
that
meant
that
while
we
have
that
deployment
going
on-
and
we
were
in
that
90
but
10%
of
our
users
were
getting
those
bad
responses
and
that's
a
lot
better
than
a
hundred
percent
right.
This
is
part
of
the
progressive
delivery
story
where
you
can
limit
the
activation
scope.
C
But
you
know
10%
is
still
more
than
zero,
and
so,
if
you
can
improve
on
that
even
more
by
saying,
hey
we're
going
to
do
some
free
testing
where
we're
just
going
to
like
try
to
filter
out.
You
know
bad
builds
in
advance
of
this,
then
you
know
we
think
mirroring
is
a
cool
opportunity
to
do
that
kind
of
thing.
C
So
that
means
that
your
system,
if
it
handy
if
it
receives
a
copy
of
the
request,
it'll
still
be
okay
and
so
idempotent
requests
cover,
gets
like
all
sort
of
read.
Operations
are
naturally
item
potent
and
also
special
kinds
of
rights.
So
if
you
have
requests
that
are
rights
like
puts
are
posted
that
are
semantically
kind
of
like
this
thing
on
the
left.
Hey
did
docked
$100
from
my
account,
then
those
are
not
item
potent
because
you
might
have
you
know
like
when
we
duplicate
the
request.
C
You'll
get
two
requests
that
say:
deduct
$100
from
Andrews
account.
That
will
mean
then
I've
been
deducted
$200
I'm,
not
happy
about
that.
So
you
know
I,
don't
that
that
traffic
is
not
compatible.
It's
not
idempotent,
so
it's
not
compatible
with
this
very
the
kinds
of
requests
that
all
right,
impotant
are
all
gets
like.
Hey
get
my
account
balance
if
you
duplicate
that
it's
fine!
It's
a
read,
also,
here's
an
example
of
an
item.
C
Unimportance
e
is
a
good
thing
to
have
in
your
system
anyways,
because
since
we
are
dealing
with
unreliable
networks,
we
have
a
hard
time
guaranteeing
that
that
you
know
there's
only
one
copy
and
it's
reliably
delivered.
This
is
pretty
interesting
to
a
lot
of
people,
so
we're
publishing
a
blog
this
week.
C
D
C
This
in
action,
alright,
so
I'm
gonna
go
over
here
and
canary
info
and
I'm
just
gonna
turn
on
this
mirror
property
theory,
and
so
we're
just
telling
Fagor
hey.
We
want
you
to
do
a
mirror
stage
before
you
do
all
the
rest
of
the
normal
canary
deployment,
stuff
and
I'm.
Going
to
deploy
to
point
I
always
do
is
I
get
the
wrong
one.
Okay,.
C
To
point
one
point:
two
and
we'll
see:
hopefully,
I
didn't
talk
so
long-
that
Sturm
got
disconnected
all
right,
update
that
image
and
we
will
go
ahead
and
wait
for
for
flagger
to
notice.
Oh
whoops
I
did
something
wrong
which
is
I,
try
to
deploy
the
same
image
again.
So
that's
vigor
is
going
to
ignore
that
because
you're
not
really
doing
a
very
interesting
deployment
if
you're
deploying
the
same
thing,
so
I'll
go
ahead
and
step
forward.
One
more
version
here
and
so
now
we've
got
a
canary
deployment
that
we're
proposing
hey
Fagor.
C
D
C
Flagger
noticed
and
is
waiting
for
our
new
version
to
be
available
to
do
the
traffic
mirroring
we're
gonna
see
down
here.
These
are
this:
is
the
command
to
the
service
mesh
for
how
to
split
traffic?
It's
going
to
have
a
mirror.
Pre-Staged!
That's
going
to
run
for
a
bit!
That's
going
to
do
the
the
mirroring
stuff
that
we
talked
about.
C
Much
like
aspen,
mash
or
sto
that
have
they
are
built
on
envoy
envoy,
a
mirrored
traffic
has
the
authority
or
host
header
as
an
addition,
where
it
says
shadow
at
the
end
and
that's
how,
if
you
have
other
tracing
systems
or
logging
systems,
you
can
differentiate.
That
shadow
is
the
clue
that
hey
this
was
actually
mirrored
traffic,
and
so
you
can
kind
of
compare
cool.
C
So
this
the
pre-staged
just
went
successfully
and
mirrored
traffic
completed
here
and
flagger
decided,
hopefully,
okay
and
so
now
it's
gonna
start
shifting
traffic
over
to
that
canary,
clip
back
and
see
yep
canary
says
it's
happy,
it's
it's
progressing
through
right.
It
went
90
and
then
80
and
it's
all
the
way
up
to
70.
That's
got
a
70/30
split.
We're
now
under
that
normal
sort
of
canary
traffic
split
scenario.
C
C
C
One
point
three
is
the
new
primary
once
that's
finished
we
will
flaggers
next
step
is
to
go
ahead
and
ship
traffic
100%
to
the
new
deployment
and
be
ready
for
you
know
the
future
other
other
deployments
that
we
want
to
do
routing
all
traffic,
primary
100,
zero,
flaggers,
finalizing
and
I.
Think
at
this
point
we
don't
need
to
watch
the
tail
end
really
here.
So
that's
that's
a
flagger
progressive
delivery
in
in
a
nutshell.
C
You
know
I
pasted
it
into
the
the
slack
the
links
here
for
if
you
want
to
try
this,
you
know
at
home
all
the
commands,
yeah
Mel's
toast
will
reproduce
the
same
demo.
You
can
see
right
there
great
and
so
at
this
point
you
know
I
think
my
demo
is
finished
and
I'll
go
ahead
and
I
think
open
it
up
for
questions.
Yeah.
B
C
Yeah
so
I'm
pretty
sure
that
this
can
be
to
put
an
open
shift
as
sort
of
a
primary.
You
know.
Open
shift
has
a
lot
of
kubernetes
stuff
inside
right,
and
so
the
way
that
flagger
works
is
it's.
It's
taking
normal
kubernetes
workload,
descriptions
like
deployments
and
services
and
then
commanding
one
or
more
service
meshes
underneath
to
do
the
traffic
shifting,
and
so
the
you
should
go
check,
flaggers
page
for
any
of
that.
The
details
around
compatibility,
but
basically
those
are
the
two
pieces
that
you
need.
C
C
A
good
question,
so
this
is
this
layer,
it's
independent
of
CNI,
so
the
trick
is
that
your
again,
your
service
mesh
is
going
to
be
the
thing
that's
doing
the
traffic
shifting,
and
so
your
service
mesh
has
to
be
compatible
with
whatever
C&I
you
have.
Underneath
this
demo,
I
happened
to
use
flannel,
which
is
like
VX
land
underneath,
but
you
know,
most
service
meshes
are
compatible
with
most
seeing
eyes.
Try
to
operate
a
layer
above
that
Aspen
mesh
does
operate
at
a
layer
above
that,
and
so
we
don't
really
care
about
this.
A
C
C
The
metrics
request
success
rate
and
request
duration.
These
are
handles
in
flaggers
code
to
say,
hey.
If
the.
If
the
user
is
running
on
top
of
an
sto
or
an
sto
service
mesh
like
Aspen
mesh,
then
you
should
query
Prometheus
this
way,
so
you
can
get
the
the
corresponding
service
match,
metrics
back
out
and
so.
C
That
users
don't
have
to
bother
with,
with
all
the
details
of
what
query
owner
we're
going
to
run
against
Prometheus
and
where-
and
you
know,
all
that
sort
of
stuff-
it's
designed
to
make
that
pretty
easy,
but
you
can
do
that
yourself
and
you
can
go
check
sort
of
the
flagger
web
github
repository
and
documentation
for
how
you
would
make
your
own
metrics
here.
You
could
point
them
in
previous.
You
could
point
them,
as
you
say,
some
other.
You
know
APM
tool
whatever
you
want,
you
can
list
more
and
more
metrics
here.
C
A
Ok
great.
The
next
question
is
from
Jim.
He
actually
asked
two
quick
questions
right,
no
row,
so
I'll
just
give
you
both.
The
first
is,
which
metrics
are
you
using
to
make
the
canary
decision?
How
are
you
determining
if
the
app
is
good
and
ready,
and
then
right
after
that,
he
asked
seems
like
failure
rate
based
on
response
codes,
yep.
C
So
there's
there's
a
couple
of
parts
here
and
so
I'll
sort
of
go
through
all
the
details.
The
the
first
is
that
when
I
said,
hey
deploy
this
new,
you
update
the
image.
Here's
our
new,
build
that
we
want
you
to
run.
That
was
adjusting
a
plain
old,
kubernetes
deployment,
and
so
that's
the
glue
between
whatever
CD
system
you
have
and
flagger,
and
so
that
plain
old
kubernetes
deployment
in
production
usage
should
have
all
the
normal
things
that
you
should
do
for
one
of
those
kubernetes
deployments.
C
If
you
need
more,
but
it'll
just
say,
hey
I
want
you
know,
I
want
this
to
be
available
and
it
will
let
the
plain
old
kubernetes
machinery
take
care
of,
deploying
that-
and
it's
not
you
know,
just
like
all
other
kubernetes
deployments,
they're
not
available
to
be
part
of
a
service
until
aliveness
and
readiness
checks
have
passed
and
so
that
that
first
stage
before
plugger
can
even
start
mirroring
traffic
or
shifting
traffic.
That
first
stage
is
a
plain
old,
kubernetes
deployment
and
that's
great
because
it
means
that
you
can
use
all
the
stuff.
C
You've
already
got
for
managing
that
for
rolling
deployments,
and
you
know
those
the
ones
that
I
recommend
really
are
yeah.
You
want
health
checks,
you
want
aliveness
checks
and
readiness
checks
before
you
even
start
shifting
traffic.
So
that's
that's
the
first.
That's
just
getting
the
deployment
up
and
running
then
the
next
one
is
going
to
be
how
you
start
deciding
that
it's
it's
ready
to
take
traffic,
and
so
that's
where
your
service
manager,
flag
or
progressive
delivery
canary
deployments
start
coming
in.
So
the
first
scenario
I
showed
is
that
okay,
we've
got
one
that's
deployed.
C
It's
past
you
know,
readiness
checks
and
so
now
we're
going
to
ship
just
start,
shifting
some
traffic
to
it
and
see
if
it's
healthy
or
not,
and
then
the
extension
that
we
showed
was
hey.
Maybe
you
want
this
pre
stage
where
you're
gonna
mirror
some
traffic
in
advance
and
that'll
give
you
that
you
know:
okay,
it's
good
and
ready
to
even
start
taking
some
production,
some
production
traffic.
C
A
C
So
for
TCP
traffic,
the
the
trick
is
that
you
have
to
have
some
way
to
decide
that
it
is
healthy
or
not.
So
in
theory,
this
would
work
for
hey
as
long
as
there's
some
Prometheus
metric
that
we
can
go
grab
somewhere,
it
may
be
a
different
metric
than
you
know:
you're
an
application
layer,
sort
of
computes
and
reports
like
the
app
could
be
trusted
to
say
how
successful
or
family
it
is.
That
would
mean
that
you'd
have
to
go
write
your
own
custom.
C
A
A
A
C
D
C
Rush
has
service
graph,
as
so
much
as
you
know,
telemetry
computing
things
like
that.
We
didn't
show
that
off
in
this
demo,
we
focused
just
on
progressive
delivery.
You
can
head
over
to
our
website
and
get
all
lunch
more
details
on
that.
We've
happy
to
show
this
off
to
you,
set
up,
set
up
a
call
with
us
or
check
out
our
demo,
our
open
beta,
that
just
wasn't
the
focus
really
of
this
webinar.
A
C
A
great
that's,
a
really
great
question
and
let
me
talk
about
what's
going
on
inside
of
there,
so
the
question
I
think
is
so
like
if
you
have
a
failed
deployment
and
it
might
take
a
minute
but
I'm
gonna
trying
I'll
try
to
set
this
to
do
this
and
then
we'll
talk
about
it.
While
it's
going,
you
can
come
see.
What's
going
on.
D
D
C
So
when
so,
if
you
have
a
bad
deployment,
a
bad
canary
deployment,
then
what
happens?
Is
you
roll
back
and
so
the
and
then
the
get
code
state
might
be
different
than
what's
running
in
in
production?
And
so
there's
a
couple
of
pieces
here
with
so
one
is
the
get
ops
sort
of
half
of
it,
and
so
that
is
where
what
you're
trying
to
do
is
you're,
trying
to
say
I
want
to
manage
the
state
of
my
cluster.
C
All
the
applications
that
are
running
in
it
I
want
to
manage
that
like
how
I
manage
source
code
in
game
so
that
I,
can
you
know,
sort
of
code
review
it
I
can
merge
it.
I
can
review
history,
all
that
sort
of
stuff,
and
so
that
that
is
still
supported,
because
the
key
is
that
when
I
did
my
deployment
of
the
broken
image
here,
I
touched
this
pot
info
deployment
and
what
happens
when
you
set
up
a
flagger?
Is
that
the
first
thing
it
does
is?
C
It
doesn't
creates
a
second
deployment
for
you
that
flagger
owns
that
is
not
respected,
reflected
in
your
dip,
ops
and
it
or
you
get
repository
or
declarative
view
of
the
cluster.
No
human
has
configured
this.
The
operator
of
the
flagger
operator
took
care
of
it,
and
this
is
the
special
pot
info
primary
deployment,
and
so
the
contract
is
that
flagger
gets
to
own
what
potica
primary
is
running,
and
that
is
always
the
version
of
pod
info.
C
The
most
recent
version
that
has
passed
the
canary
deployment
scenario,
and
so
you
do
have
to
get
used
to
the
fact
that
what
you
have
in
version
control
for
pod
info
may
not
reflect
what
you
have
in
version.
What
you
you
are
actually
running
for
pod
info
primary
and
that's
why
you
have
to
let
flag
or
manage
this
pot
info
primary
deployment.
Leave
it
out
of
your
your
sort
of
your
world
view
of
your
declarative,
config.
What
you're
putting
into
your
get
ops
operator.
C
Hey
the
primary
may
not
be
the
most
recent
one
that
we've
published
a
past
CI,
it's
actually
the
most
recent
one
that
passed
CI
and
also
passed
this
progressive
delivery
scenario,
and
so,
if
you
notice
a
flag
or
canary,
has
failed
that
should
make
you
uncomfortable.
You
should
go,
make
code,
changes
and
fixes
and
try
to
push
a
new-build
through
that
moves.
It
forward
to
be
so
that
pas,
tempo
and
potential
primary
are
both
running
the
most
recent
version
of
your
code
and
you
sort
of
reconciled
that
difference.
So
I
think
it's
totally
corrected.
C
It
should
be
a
little
bit
uncomfortable
that
you
have
a
canary
build
that
failed.
The
only
other
options
you
have
are
you
could,
you
know,
revert
like
you,
could
have
a
robot
go
and
revert
the
commits
that
caused
the
build
to
fail,
but
that
can
be
kind
of
hazardous
because
you
know
there
may
be
other
commits
pending
that
are
changing
things.
What,
if
that
robot
can't
actually
merge
things
so
I'm
not
aware
of
a
lot
of
people
doing
that
what
I
usually
see
is
people
say?
Okay,
we
understand.
What's
happened,
this
canary
has
failed.
C
A
Okay,
last
question
is
a
little
bit
of
a
long
one
from
Alexander.
One
of
his
services
is
using
WebRTC
connections
between
client
and
service.
Terminating
connection
will
cause
bad
user
experience.
The
session
is
typically
going
on
for
one
to
two
hours.
Is
it
possible
to
gradually
prevent
new
connections
to
be
established
on
the
old
version
and
shipped
new
connections
to
the
new
service,
to
allow
him
to
remove
the
old
version
when
there
will
be
no
more
WebRTC
session,
is
open
any
ole
information.
C
So
that's!
This
is
a
question
again.
That
kind
of
depends
on
your
service
mesh.
So
so
for
a
spin
mash,
you
know
it's
you
an
envoy
again,
the
you
know
envoy
will
last
at
least
for
a
configurable
amount
of
time.
Old
connections
to
old
services
stay
open,
and
so
you
can
sort
of
gradually
shift
that
over.
But
you
do
face
this
tension
of
well,
but
if
you're
not
gonna
shift
over
any
old
stuff,
if
you're
not
going
to
shift
over
anything,
you've
got
connections
that
last
one
or
two
hours,
then
the
soak
time.
C
In
order
for
you
to
get
enough
new
traffic
to
your
new
connections
might
be
pretty
darn
long,
and
so
you
know
you
might
be
looking
at
hey
how
long
of
a
window
do
I
have
to
have
to
get
enough
new
stuff,
but
you
know
most
I
think
most
service
measures
support.
This
sort
of
you
know:
hey,
let's,
let's
tail
off:
let's
not
disrupt
existing
connections,
I
know
as
to
mesh
does
and
the
others
you
just
have
to
check.
A
A
All
right:
well,
it
looks
like
we're
with
questions
I
thank
Sandra
and
Zach
for
a
great
presentation
and
thanks
everybody
for
joining
us
today,
and
the
webinar
recording
on
slides
will
be
online
later
today
on
the
CN
CF
website
and
the
link
I
posted
earlier
in
the
chat,
we're
looking
forward
to
seeing
you
all
at
the
future.
Cf
webinar
have
a
great
day
thanks
so
much
Zach
and
Andrew
thanks.