►
From YouTube: Leveling Up Your CD: Unlocking Progressive Delivery on Kubernetes - Daniel Thomson & Jesse Suen
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Leveling Up Your CD: Unlocking Progressive Delivery on Kubernetes - Daniel Thomson & Jesse Suen, Intuit
Kubernetes Continuous Delivery methods have continued to evolve to more advanced strategies such as canary, A/B testing, and blue-green. Progressive delivery is the next step of CD, enabling service promotion for a subset of users in an automated fashion backed by metrics. There’s no one-size-fits-all on what are the appropriate metrics to drive promotions. Often, the four golden signals (latency, traffic, errors, saturation) are used, but what if this isn’t enough? More sophisticated techniques might use algorithmic or even AI-driven analysis. The Argo Experiment and Analysis CRDs provides simple constructs to drive automated promotion in an extensible fashion. This session discusses how Intuit leverages experimentation and analysis, the challenges in providing an automatic but generic approach to analyzing experiments, and envisioning the future of declarative progressive delivery.
https://sched.co/Uaaj
A
A
A
Okay,
a
third
of
you,
how
many
of
you
do
progressive
delivery,
okay
and
what
it
does
enhance,
how
many
we
do
progressive
deliver
and
kubernetes
yeah
like
dozen
all
right,
so
those
gentlemen,
the
name
Jesse
actually
do
progressive
delivery
on
kubernetes
that
deserves
a
standing
ovation.
Everybody
up!
I
kid!
You
not
everybody
up,
everybody
up!
Yes,
yes,
all
right
and
now,
when
you
are
awake.
Welcome
to
this
talk
about
progressive
delivery
on
kubernetes
and
one
thing
the
slides
are
already
in
the
sketch
on
this
scheduling
application.
B
B
B
B
So
let's
go
back
to
that
original
CD
pipeline
and
see
how
progressive
delivery
might
change
the
delivery
process.
So
we
still
have
this
initial
same
three
steps,
but
what
comes
after
the
test
phase
becomes
a
lot
more
sophisticated.
So,
instead
of
deploying
to
your
entire
user
base,
you
only
deploy
to
a
small
subset
and
the
way
you
can
do
that
to
me
in
many
different
ways.
B
A
common
way
is
to
just
serve
a
small
percentage
of
the
traffic
to
that
new
version,
and
then
next
you
wait
and
you
see
how
well
that
deployment
went
through
through
analysis
and
by
analysis,
you're
measuring
metrics,
which
you
care
about
for
your
application.
Oftentimes
Google's
for
golden
signals
are
used
which
are
latency
air
traffic
and
saturation,
but
not
necessarily,
then
you
determine
was
my
rollout
successful
or
not,
and
if
it
was
you
progress
and
you
increase
that
subset
of
users
that
you're
exposing
that
new
service
to.
B
If
not
you
bore
it
and
you
roll
back
to
that
previous
version,
all
right,
so
there's
four
basic
building
blocks
to
progressive
delivery.
The
first
is
user
segmentation,
that's
being
able
to
isolate
a
set
of
the
users
that
you
wish
to
target
and
expose
them
to
that
new
version.
Basically,
this
is
how
you're
controlling
your
blast
radius
examples
of
this
might
be
to
dial
in
that
traffic
percentage
to
a
small
amount.
It
could
even
be
something
more
coarser
grain
like
like
deploying
to
a
geographical
region
or
a
city.
B
Observability
are
things
like
tracing
logging
and
metrics,
and
the
metrics
are
what
you
are
going
to
be
relying
on
to
measure
whether
and
not
that
update
is
going
smoothly.
And
finally,
we
have
automation
and
automation
is
now
that
you've
been
collecting
those
measurements.
You
can
now
automatically
progress
or
abort
that
update,
based
on
what
was
observed.
B
So
now
that
we
have
that
thousand
foot
view,
let's
kind
of
narrow
down
in
how
this
applies
to
kubernetes.
So
with
deployments
we
have
rolling,
update,
rolling
update
is
all
about
trying
to
get
you
from
version
a
to
version
B
as
quickly
and
safely
as
possible.
However,
it
provides
very
few
controls
over
that
speed.
B
You
also
can't
use
external
metrics,
like
if
the
metrics
you
care
about
our
and
Prometheus
you,
it's
gonna,
you're
gonna
have
a
hard
time
getting
readiness
probes
who
connect
and
Prometheus
and
while
rolling
update,
can
halt
the
progression
of
your
update.
It's
unable
to
reverse
that
update.
You
can't
automatically
rollback
when
your
metrics
are
starting
to
fail.
B
So
we
talked
to
many
of
our
developer
teams
to
understand
their
use
cases
and
when
you're
supporting
thousands
of
developers-
and
you
get
there's
a
lot
of
staying
in
ideas
and
use
cases
that
come
up-
and
here
are
some
of
the
common
ones.
Obviously
there's
this
need
to
automatically
roll
back
when
your
metrics
are
failing,
but
we
also
got
a
lot
of
other
interesting
requirements
such
as
the
people
want
to
customize
their
success
and
failure
criteria,
or
they
still
wanted
a
human
manual
gate
somewhere
in
the
deploy
process.
B
So
we
came
up
with
a
set
of
requirements
that
we
of
what
we
wanted
to
build
number
one.
The
solution
needed
to
be
robust,
not
early
into
the
process.
We
tried,
we
had
a
lot
of
false,
starts,
trying
to
use
things
like
Jenkins
the
power
canary
deployments
or
canary
analysis,
and
it
ended
up
being
extremely
fragile.
B
A
second
we
wanted
to
build
on
top
of
industry
standard
tooling
things
like
Prometheus
client
service
meshes
and
have
great
integrations
with
those
existing
tools,
and
after
talking
with
all
the
developers,
it
was
obvious
that
we
needed
a
very
flexible
solution
in
order
to
support
all
of
their
use
cases.
And,
finally,
it
needed
to
be
declarative
and
get-up
focused,
there's
a
lot
of
numerous
benefits
that
you
get
with
the
curve
declarative
approach.
B
So
about
a
year
back,
we
started
the
argo
rollouts
project
and
at
the
time
the
scope
was
quite
simple:
it
provided
a
drop-in
replacement
for
a
deployment
and
just
give
two
additional
update
strategies:
blue
green
and
canary
and
I'll
just
quickly
go
through
whether
we
rolled
out
resources
looks
like
to
give
you
guys
some
background
of
what
this,
what
this
is
okay.
So
this
is
a
rollout
specification.
Rollouts
work
exactly
the
same
as
a
deployment.
B
It
manages
the
creation,
scaling,
deletion
of
your
replicas
sets,
and
so
the
spec
itself
should
be
mostly
identical
to
the
deployment.
As
I
mentioned,
it
was
meant
to
be
a
drop-in
replacement
and
we
wanted
that
experience
to
be
exactly
the
same.
But
what's
different
about
the
rollout
is:
are
the
new
strategies
and
he
I'm
showing
a
canary
strategy,
and
what
this
is
specifying
is
that
during
a
deployment
it
will
first
set
a
canary
weight
of
40
percent.
B
C
Thanks
Jesse,
so
in
order
to
enable
progressive
delivery,
we
need
to
introduce
two
new
concepts,
analysis
and
experimentation,
and
with
each
of
these
concepts
we
realized
that
they
could
have
their
own
C
or
DS.
We
didn't
want
to
kind
of
add
them
into
the
rollout
object
if
we
added
them
to
the
rollout
object.
We'd
have
this
really
long,
Amal
that'd
be
kind
of
hard
to
parse
and
each
of
these
concepts
can
kind
of
stand
alone
on
their
own
and
so
to
that
end,
I'm
gonna.
C
When
you
know
your
application
is
healthy
or
rolling
back
if
you
notice
errors
and
so
to
that
end,
we
integrated
it
with
rollouts.
So
here's
a
rollout
that
has
analysis
you'll
notice.
It
looks
almost
the
same
as
the
previous
world
that
you're
looking
at
earlier,
except
now
we
have
an
analysis
section
and
within
this
analysis
section
we
have
something
called
a
template
name.
So
this
template
name
is
referring
to
a
new
CD
called
a
analysis.
Template
I
will
talk
to
the
analysis
template
in
a
minute,
but
here's
what
you
need
to
know
for
now.
C
So
when
you
make
a
change
to
your
pods
back
and
the
rollout
is
progressing
through
the
steps
we
have
listed
and
under
the
Canaries
or
the
canary
strategy,
we
will
also
create
a
analysis
that
analysis
is
checking
the
success
rate
of
requests
to
our
service.
If
that
success
percentage
drops
below
90%,
we
consider
the
new
version
bad
and
we
need
to
rollback
and
so
to
that
end,
I
will
give
you
a
quick
demo
of
a
rollout
that
uses
analysis.
C
So
the
first
thing
I'm
going
to
do
is
pull
up
my
terminal,
so
we
also
in
addition
to
adding
analysis
and
experimentation
to
our
rollouts.
We
created
a
coups,
ETL
plugin
for
our
go
rollouts
and
we
did
it
for
two
reasons:
one
we
want
it
to
be
able
to
visualize
our
rollouts
and
then
two.
We
wanted
to
be
able
to
automate
some
of
our
common
operations
on
the
rollout
object.
So
what
you
can
see
here
is
a
rollout
object.
C
You
can
notice
that
right
now,
it's
in
a
steady
state,
it's
executed
all
the
steps
and
then,
under
this
metadata
we
have
a
tree
view.
This
tree
view
shows
all
the
children
resources
created
from
the
rollout,
so
you
can
see
a
repla
set
and
a
bunch
of
pods,
and
so
now
let
me
pull
up
the
demo
app.
So
this
is
our
demo
app.
What
you'll
notice
is
that
there
are
a
bunch
of
floating
dots
going
across
the
screen.
Each
of
those
floating
dots
represent
a
request
to
our
back-end
service.
C
Then
our
back-end
service
has
a
predefined
color
that
it
will
return
and
that's
the
color
that
the
dock
takes
you'll
notice.
This
little
bar
graph
kind
of
slowly
incrementing
across
the
screen
that
represents
the
percentage
of
of
that
color.
That's
the
back
ends
are
returning.
So
in
this
case
right
now
we
are
a
hundred
percent
blue.
Another
feature
of
this
demo
app
is.
C
In
addition,
we
have
this
new
concept
called
an
analysis
run
that
runs
alongside
the
new
replica
set.
That
is
the
analysis
that
we
were
talking
to
you
earlier.
So
if
our
success
rate
drops
below
90%,
where
this
new
version
is
bad
and
we
need
to
rollback
so
to
actually
demonstrate
that
I'll
change,
the
yellow
version,
error
rate
to
a
hundred
percent,
and
so
either
on
this
measurement
or
the
next
you'll
see
a
little
red
X
here,
indicating
that
the
measurement
was
a
failure
and
we
need
to
rollback.
C
So
it
looks
like
this
one
just
missed
it.
So
give
me
another
5
seconds
and
there
you
go.
We
had
a
failed
measurement,
and
now
we
realize
that
this
version
is
bad
and
we
need
to
rollback,
and
so
we've
scaled
up.
The
old
blue
versions
first
and
now
we're
starting
to
scale
down
the
yellow
ones
and
you'll
notice
that
now
the
blue
dot,
there's
only
blue
dots
floating
across
the
screen
and
the
bar
at
the
bottom
is
all
blue.
C
C
So
similar
to
before
it
is
creating
those
two
new
pods
to
achieve
that
forty
percent
that
you
were
looking
at
earlier
and
it
now
created
another
analysis
run.
So
this
analysis
run
is
checking
the
same
thing
again
of
if
the
percentage
of
healthy
requests
or
service
is
above
90%
we're
good,
and
since
the
area
is
at
zero,
it
will
continue
to
make
those
measurements.
So
right
now,
the
with
this
demo,
our
rollout
will
late
an
hour
before
we
fully
progress,
so
I'm
gonna
run
a
command
to
skip
the
rest
of
this
wait.
C
And
so
now
we'll
start
to
progress
through
the
next
steps.
In
case,
if
you
don't
remember,
it
will
increase
the
ratio
to
60%
wait
10
seconds,
let's
increase
it
to
80%,
wait
10
seconds
and
then
progress
all
the
way
so
I'm,
probably
around
10
15
seconds.
We
should
finish
rolling
out
the
new
version
and
during
this
time
we're
still
checking
the
analysis
run
to
make
sure
that
we're
above
that
90
percent.
C
So
the
next
question
is:
okay,
that's
really
cool,
but
how
did
you
actually
do
it
so
I'll
kind
of
dive
into
the
analysis
template
now?
So
the
analysis
template
consists
of
one
or
more
key
metrics
that
you
want
to
monitor
while
you're
deploying
a
new
version,
and
so
each
metric
has
a
different
provider
that
queries
a
back-end
service
to
take
that
measurement.
In
the
case
of
our
demo,
we
were
using
Prometheus,
but
we
support
a
bunch
of
other
providers
and
the
one
I
just
want
to
call
out
in
case.
C
If
you
don't
know
what
it
is.
Kai
anta
is
a
canary
analysis
tool
that
lives
under
the
spinnaker
project,
but
is
can
be
used
as
a
standalone
service,
so
to
actually
kind
of
dive
into
the
template
a
little
bit
more.
Let's
first
start
with
the
Prometheus
provider,
it's
relatively
straightforward.
You
specify
a
Prometheus
address.
You
want
a
query,
but
you
also
specify
a
prom
QL
grade
that
you
want
to
execute
when
it
executes
it
grabs
a
measurement
from
prometheus.
But
then
that
raises
the
question
of
how
do
you
evaluate
that
metric?
C
And
so
that's
where
success
conditions
come
into
play?
Success
conditions
are
a
way
of
evaluating
the
results
of
your
measurements
from
your
providers,
and
you
might
notice
that
we're
using
an
expression
instead
of
a
threshold.
Well,
since
we
have
different
providers,
we
realized
that
certain
providers
might
return
different
payloads,
and
so
we
needed
flexibility
in
how
we
interpret
these
results
and
so
to
that
and
not
everything
can
just
be
a
simple
equality
statement,
and
so
at
this
point
you
can
say:
ok,
I
can
take
a
measurement.
I
can
evaluate
that
metric
measurement.
C
How
frequently
do
I
do
it?
That's
where
interval
and
count
come
into
a
play.
So
interval
is
how
frequently
we
measure
take
a
measurement
against
our
provider
and
count
is
the
total
number
of
times
we
want
to
execute
that
provider
and
so
from
there
there's
one
last
concept:
I
want
to
for
analysis,
and
that
is
arguments.
C
So
not
only
can
you
use
these
analysis
templates
within
your
organization.
You
can
also
use
them
across
communities.
To
give
an
example
of
that,
the
demo
we
are
using
today
use
a
metric
from
the
nginx
ingress
controller,
and
so,
as
a
result,
anyone
here
who
uses
that
controller
for
their
ingress.
They
can
leverage
to
this
analysis,
template
and
so
that's
a
high-level
overview
of
analysis
and
I'm
going
to
hand
it
back
to
Jesse
to
explain
experimentation.
B
It's
most
useful
when
you
couple
that
with
analysis,
experiments
are
something
you
can
start
standalone,
but
it
can
also
be
something
that
you
start
as
a
blocking
step
in
your
rollout
and
some
use
cases
that
we
see
for
experimentation
include
things
like
a
be
testing
baseline
versus
canary
analysis
or
even
things
like
machine
learning,
machine
learning,
model
testing.
The
thing
I
wanted
to
talk
about
a
baseline
versus
carried.
B
This
is
actually
a
good
best
practice
for
anyone
who's
doing
canary
analysis
that
you
actually
want
to
test
your
canary
and
compare
it
against
a
baseline
in
not
your
existing
running
prods
in
production
and
the
reason
you
want
to
do.
This
is
because,
when
you
start
up
a
brand
new
baseline,
you
eliminate
all
the
skewing
factors
in
your
in
your
measurements
for
things
like
Java
heap
size
or
cache,
some
cashews
that
I
already
warmed
up.
B
B
Then,
on
the
bottom,
you
see
that
same
similar
analysis
section,
but
this
time
it's
also
a
list
here,
I'm
using
a
template
called
HTTP
benchmark,
which
I'll
show
in
a
minute
in
a
second
and
I,
just
wanted
to
point
out
here
that
the
same
template
is
being
used
multiple
times
just
passed
with
different
arguments,
one
with
the
purple
hostname
and
the
other
with
orange
hostname.
And
finally,
this
is
an
example
of
another
style
of
analysis
which
we
provide,
which
is
our
job
based
metric
and
instead
of
a
Prometheus
query.
B
Okay,
so
what's
happening
now,
is
that
this
experiment
launched
two
replicas
sets.
Both
replicas
has
had
two
replicas
each
and
what's
going
to
happen
is
as
soon
as
these
pods
have
become
ready
and
available.
It'll
kick
off
the
analysis,
and
this
is
analysis,
as
you
recall,
was
using
the
job-based
metric,
so
give
it
just
a
second.
There
goes
okay,
so
it
started
an
analysis.
It's
running
this
job,
I
think
every
five
or
ten
seconds
and
the
success.
B
All
right,
so
to
summarize,
what
we
showed
was
how
you
can
leverage
these
new
concepts
that
we
introduced
analysis
and
experiments
and
integrate
to
like
a
rollout
strategy
and
tailor
it
to
your
needs,
and
we
designed
this
to
be
very
flexible
and
loosely
coupled,
so
that
it
could
be
used
not
just
with
rollouts
but
in
other
ways
that
we
might
not
anticipate.
We
will
also
wanted
to
point
out
that
progressive
delivery
is
multifaceted.
There
will
never
be
this
one-size-fits-all
solution,
for
example,
in
to
it.
B
It
makes
heavy
use
of
things
like
feature
flags,
so
that
developers
can
experiment
and
canary
feature
at
a
feature
toggling
level.
And
finally,
this
is
just
the
beginning
of
progressive
delivery
and
just--how
CD.
You
made
the
lives
of
developers
easier.
We
think
that
progressive
delivery
is
gonna,
have
that
same
effect
for
operators.
C
So
here
are
some
links
that
you
can
check
out.
In
addition,
Jesse
and
a
couple
of
our
colleagues
have
been
writing
a
book
on
kubernetes
and
get-ups,
so
you
can
scan
that
QR
code
on
the
left
for
a
preview
of
the
book.
In
addition,
Intuit
is
hiring,
and
so,
if
you're
excited
by
progressive
delivery
or
just
solving
kubernetes
at
scale
feel
free
to
scan
that
QR
code
or
scan
them
both,
and
so
with
that
we'll
open
it
up
for
some
questions.
Thank
you
for
attending.
A
Yeah,
that
was
that
was
amazing
that
just
just
well
done.
Thank
you
a
couple
of
items
before
we
go
to
questions.
First
of
all,
you
don't
need
to
rush
for
lunch
because
we
have
like
a
private
lunch
area
here.
We
don't
need
to
go
with
all
the
crowd
downstairs,
whether
it's
right
here.
Second,
don't
forget
to
rate
this
talk,
that's
like
five
stars
in
your
schedule.
Application!
Please
do
that,
and
we
do
have
like
five
minutes
for
questions.
I'll
try
to
run
the
mic.
I
need
to
get
my
steps
in
today,
hello.
E
B
So
hopefully,
I'm
answering
your
question,
so
one
of
the
in
our
delivery
tool,
Argo
CD,
we
actually
have
a
concept
of
health
of
a
resource
and
so
for
things
like
deployment
a
healthy
deployment.
It
actually
is
the
same
exact
health
when
you
run
cube,
CTO
rollout
status
on
the
object
and
you
basically
meeting
your
available
replicas
and
they
all
are
you.
B
Your
count
is
desired
and
all
that
stuff,
so
in
our
in
our
pipelines,
there's
this
gating
step
that
it
won't
continue
past
that
Jenkins
step
until
the
application
itself
is
in
a
healthy
state,
because
before
that
is
actually
in
they
say
what
would
you
which
we
call
progressing?
Hopefully,
that
answered
your
question
also.
C
E
B
The
robots
yes,
so
these
are
ends
so
I
guess.
Another
question
with:
how
do
we
use
version
deployments
because
were
less
are
basically
the
same
as
deployments?
Yes,
we
use
get
ops,
I!
Guess
it's
the
answer
to
that,
and
so
as
part
of
our
pipelines,
a
step
in
the
pipeline
is
actually
to
commit
a
new
image
tag
to
the
git
repo
and
then
a
call
is
made
out
to
our
go
CD,
which
then
synchronizes
the
live
State
with
what
was
just
committed
and
they
get
pipeline.
Yeah.
D
B
So
that's
also
something
that
is
partly
covered
by
the
Jenkins
pipelines
we
rent.
So
the
pipeline
looks
like
okay
commit
that
the
new
image
tag
wait
for
that
application
to
be
available
and
ready,
and
then
some
people
will
do
actual
innovation
test
like
a
run
as
test
script
from
the
Jenkins
pipeline,
but
also,
we
think
that
things
like
the
job
based
metric
that
I
demonstrated,
will
have
like
a
more
tightly
coupled
test
like
a
more
smaller
scope
to
test
which
maybe
they
can
access
things
that
aren't
accessible
in,
say
from
the
Jenkins
namespace
like.
D
C
B
If
you
get
anything
from
this
talk,
the
one
thing
that
we
wanted
to
highlight
was
that
the
analysis
template
was
basically
gives
the
developers
control
of
what
they
consider
interesting
metrics
for
their
specific
application,
and
so
it's
we
use
the
prompt,
well
query
and
they
could
write
whatever
prompt
well
query.
They
want
to
extract
the
metrics
that
they
care
about
yeah.
Thank
you.