►
From YouTube: How Intuit Does Canary and Blue Green Deployments with a K8s Controller - Daniel Thomson
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Intuit Does Canary and Blue Green Deployments with a K8s Controller - Daniel Thomson & Alex Matyushentsev, Intuit
GitOps has taken Kubernetes by storm, but the default kubernetes deployment is missing strategies such as canary and blue green. How do you run two versions of one application at the same time in a GitOps manner? Do you need to make multiple commits to orchestrate those strategies? Isn’t that redundant for a single deployment? The onus is on the user to orchestrate these industry standard strategies often using error prone scripts. To address this need, Intuit open sourced a deployment controller called Argo Rollouts that codifies these industry standard strategies into an custom resource called Rollout. This session will discuss how Intuit leverages rollouts of hundreds of apps in a GitOps style to implement these missing strategies, the challenges in converting inherently imperative strategies into a declarative configuration, and envisioning the future for GitOps based deployments.
https://sched.co/MPZD
A
A
A
B
Hi
everyone,
my
name,
is
Danny.
This
is
my
colleague
Alex
and
thank
you
so
much
for
attending
our
session
on
how
into
it
does
canary
and
blue
green
deployments
with
a
k-8,
c-can't
Roller,
so
Alex
and
I
are
both
and
software
engineers
at
Intuit
and
for
the
last
six
months
we've
been
working
on
our
guerrilla
outs
and
we're
really
excited
to
give
you
a
demo
of
it
today.
B
So
just
a
brief
overview
of
the
talk
today.
Just
it's
a
classic
presentation,
we'll
kind
of
talk
about
into
it
and
some
of
the
engineering
problems
that
we
have
challenges
that
we
have,
especially
in
the
context
of
deployments,
talk
about
kind
of
some
of
the
options
that
we
came
up
with
or
really
failed.
B
The
tents
I
should
say
and
then
talk
about
our
final
solution,
which
offers
a
declarative
way
to
do
blue,
green
and
canary
strategies
so
to
jump
into
it
so
into
it
is
a
company
of
there
is
a
company
with
a
revenue
of
six
billion
and
we
have
over
50
million
customers.
We
provide
financial
and
accounting
software
to
our
users
in
order
to
simplify
their
lives.
To
get
that
software
to
those
users,
we
have
around
4,000
engineers.
B
As
a
result,
this
kind
of
creates
a
there
are
a
lot
of
different
needs
that
all
of
these
developers
have
and
in
order
to
help
move
into
the
cloud
we
really
adopted
kubernetes
as
our
underlying
platform
to
deploy
into
the
cloud.
So
here
is
our
adoption
of
kubernetes.
Over
the
last
year,
it
was
initially
a
little
bit
of
a
slow
start,
as
we
were
kind
of
learning,
getting
a
lot
of
learnings
and
building
up
that
muscle
memory.
B
But
we
really
started
to
pick
up
pace
and
started
to
address
a
bunch
of
different
needs
and,
as
the
number
of
users
kind
of
go
up,
we've
got
a
lot
more
knowledge
and
I've
come
across
a
lot
more
problems,
just
some
more
stats
on
into
it's.
In
general,
we
have
four
different
business
units
and
right
now
we
have
around
1200
engineers
using
kubernetes
with
out
of
our
4,000
employees,
some
of
our
fund
kubernetes
stats.
We
got
around
120
clusters.
Each
of
those
are
around
200
to
300
nodes.
B
We
want
to
keep
them
kind
of
medium
sized
and
then
also
we
do
around
1300
deploys
a
day
with
those
deploys
we
are.
Most
of
them
are
using
the
standard
deployment
object
as
a
result,
that
leaves
us
with
two
deployment
strategies
recreate
and
rolling
update.
Most
of
the
engineers
are
using
the
rolling
update,
but
that
doesn't
really
solve
all
the
use
cases
that
our
developers
have.
B
For
example,
we
have
some
legacy
applications
that
really
can't
do
a
rolling
update,
and
then
we
have
more
cutting-edge
applications
that
want
to
do
some
more
interesting
strategies
so
result
we're
looking
to
solve
for
blue
green
strategies
canary
and
a
be
testing,
and
so
one
of
the
core
principles
of
our
kubernetes
platform
is
that
we
manage,
or
we
really
take
into
account
the
get
offs
methodology.
So
what
does
get
ups
its
using
it
as
a
single
source,
the
truth
of
your
desired
state
for
your
infrastructure
and
application?
B
So
what
does
that
mean?
We
store?
All
of
our
kubernetes
manifests
in
a
git
repository
and
we
use
a
tool.
We've
developed,
called
Argos
an
open
source
tool.
We've
developed
called
Argos
CD
to
take
those
manifests,
compare
them
to
what's
in
the
cluster,
identify
what
the
differences
are,
show
that
to
the
user
and
then
kind
of
reconcile
that
difference
and
apply
those
changes,
and
so
this
kind
of
introduces
a
problem
with
the
strategies
I
talked
about
earlier.
B
How
do
you
do
a
Bluegreen
or
or
canary
deployment,
while
still
leveraging
get-ups
since
there's
no
native
support
for
Bluegreen
and
canary
and
kubernetes?
We
can't
just
use
an
off-the-shelf
solution
and
there's
this
kind
of
conflicting
approach
to
get
off
sand
Bluegreen
and
canary
deployments,
since
get-ups
is
inherently
declarative.
Where
you
say
this
is
the
state
where
I
want
to
go.
B
Here's
where
I
am
just
get
me
there,
while
Bluegreen
and
canary,
are
more
imperative
where
you
have
a
defined
list
of
steps
that
you
want
to
execute
to
get
to
the
next
step,
and
so
we
really
wanted
more
of
a
declarative
approach
and
we
wanted
to
kind
of
integrate
this
with
our
CI
tools
so
that
our
developers
could
have
a
simple
clean
pipeline
from
commit
to
deploying
a
master.
And
so
we
had
a
problem
and
we
were
thinking
all
right.
B
How
could
be
kind
of
approaches
to
provide
a
solution
for
our
users
who
want
to
use
these
different
strategies?
Our
first
approach
was
jenkins
and
it
works.
It
had
a
bunch
of
problems,
though
it
didn't
really
fit
our
model
of
get-ups,
because
we
were
taking
a
lot
of
state
and
putting
that
into
our
Jenkins
pipeline,
which
meant
we
had
two
different
sources
of
truth,
your
Jenkins
pipeline
and
your
git
repository.
B
B
The
pipeline
would
fail,
and
if
you
try
to
execute
it
again,
it
fail
again
and
that
would
require
our
users
go
in,
and
change
early
modify
the
cluster
to
get
it
back
into
the
right
state
in
order
to
run
their
pipeline,
and
since
he
was
extremely
brittle
that
wasn't
really
the
solution
we're
looking
for.
Additionally,
we
didn't,
we
really
don't
like
putting
kubernetes
credentials
on
Jenkins,
and
we
found
that
taking
our
that
solution
and
distributing
it
across
multiple
pipelines
was
a
lot
more
work
than
we'd
really
like
to
do
so.
B
We
took
step
back
and
said:
okay,
how
can
we
do
this
better?
Well,
the
next
step
was
trying
to
push
a
lot
of
that
logic
from
the
Jenkins
pipeline
into
deployment
hooks
executing
them
before
and
after
the
initial
Huber
Nettie's
apply,
but
that
also
still
took
still
had
a
lot
of
those
underlying
assumptions
that
we
had
with
our
jenkins
and
it
really
wasn't
following
the
get-ups
model
that
we
want
to
give
users,
because
we
didn't
have
a
declarative,
spec
saying
this
is
where
we're
going.
B
We
just
said:
okay,
here
are
a
bunch
of
steps
and
you
still
need
to
execute
so
this
worked,
but
then,
but
we
weren't
happy
with
it.
So
we
took
a
step
back
again
and
we
were
like
all
right.
Both
of
our
first
two
approaches
were
very
imperative.
We
very
much
said
do
this,
do
that?
Instead,
we
want
to
kind
of
really
take
advantage
of
what
makes
kubernetes
one
of
the
things
that
makes
kubernetes
so
great
its
declarative
nature,
and
so
we
decided
to
create
a
custom
resource
with
a
custom
controller.
B
With
this
custom
controller,
we
took
these
five
design
considerations
into
account.
First,
we
really
wanted
to
take
all
of
that
logic
that
deployment
orchestration
logic
and
kind
of
just
package
it
into
the
controller,
so
that
our
developers
don't
have
to
worry
about
the
deployment
logic,
really,
they
should
be
focusing
on
their
application
code
and
it
should
be
a
seamless
deployment.
Second,
we
really
wanted
to
make
sure
it
was
get-ups
friendly
and
regardless
of
the
state
of
the
cluster,
it
should
be
able
to
handle,
handle
that
stay
and
get
back
into
a
solid,
steady
state.
B
We
wanted
it
to
run
inside
the
kubernetes
cluster
so
that
we
didn't
have
to
provide
credentials
to
some
outside
source
in
order
to
orchestrate
the
deployment
and
also
in
terms
of
a
migration
pattern.
We
really
wanted
people
who
are
coming
from
deployments
to
have
a
seamless
story
to
switch
roll
out
and
not
lose
any
of
the
functionality
that
they
had
with
these
rods,
and
so
with
that
and
six
months
of
work,
we're
really
excited
to
introduce
our
go
roll
outs.
It
is
our
advanced
open-source
k,
its
deployment
controller.
B
B
B
B
So
if
a
user
wants
to
run
some
automated
tests
and
then
immediately
deploy
that's
possible
or
if
they
want
to
have
some
more
manual
tests
that
they
want
to
run
and
then
give
a
manual
thumbs-up,
that's
also
possible
and
we
wanted
to
stay
in
line
with
the
deployment
functionality
by
integrating
with
existing
tools
like
HP
a
so.
This
is
still
pretty
high
level,
and
why
don't
we
kind
of
dig
down
into
some
manifests?
So
here's
the
deployment?
B
Nothing
special
nothing
too
exciting,
but
when
we
want
to
change
it
to
a
rollout,
the
first
thing
we
need
to
do
is
change
the
API
version
and
kind
to
the
rollout.
Second,
we
need
to
add
a
bar
guarulhos.
If
extraor
castrates
Early's
define
the
orchestration,
you
want
to
do
declaratively.
There
are
two
currently
available:
blue,
green
and
canary,
so
I'll
start
with
blue
green
first.
So
here
is
the
blue
green
strategy.
Spec.
The
first
thing
I
want
to
point
out
is
the
active
service
in
the
preview
service.
B
These
are
references
to
services
in
the
same
namespace
as
the
rollout.
The
active
is
in
10
to
be
used
for
live
production,
consumer-facing
traffic.
Well,
the
preview
is
more
meant
for
kind
of
testing
like
just
an
endpoint
that
you
can
kind
of
test
before
you
switch
it
to
the
active,
and
so
the
rollout
controller
will
be
responsible
for
modifying
these
services
to
direct
traffic
to
specific
replicas
sets
that
were
created
by
the
rollout
and
so
in
order
to
do
that
similar
to
a
deployment.
B
We
will
do
a
demo
of
this
in
a
bit,
but
we
also
offer
functionality
to
control
the
size
of
the
preview
environment.
So
if
you
want
to
try
to
save
some
resources,
while
you
do
some
testing
and
we
can
do
manual
and
automated
or
promotions
up
next,
we
have
the
canary
strategy.
So
the
blue
green
strategy
had
a
bunch
of
different
knobs
to
kind
of
control.
The
behavior
of
the
blue
green
deployment
with
the
canary
strategy.
B
B
And
so
with
this,
we
don't
modify
a
service
because
we
want
to
be
able
to
send
traffic
to
both
versions
and
at
the
moment
we
don't
use
a
service
mesh,
we're
using
just
pure
replicas,
the
replicas
count
ratio.
So
we
are
going
to
scale
both
the
canary
and
the
stable
replica
sets
to
reach
the
ratio
listed
in
the
canary
steps.
B
C
So
since
it
is
get
ops
all
the
manifests
in
get
github
repository
in
this
case,
and
as
you
can
guess,
there
is
a
bunch
of
llamo
files
and
not
all
not
all
of
them
are
very
interesting.
Service
is
just
a
service,
but
we
have
roll
out
the
Tiamo,
and
this
one
I
want
to
show
you.
So
here
is
roll
out
it's
as
promised.
It's
pretty
much
deployment
except
it
has
different
IP,
inversion
and
kind,
and
it
has
a
strategy
and
I'm
going
to
start
from
canarian.
C
In
this
case,
I
have
just
two
steps.
First
step
just
tells
about
controller
to
go
ahead
and
replace
20%
of
my
application
with
the
new
version
and
then
wait
for
me
to
say
it's
ok
to
continue
and
that's
pretty
much
it
now.
We
just
ready
to
deploy
the
application
and
to
do
that
I'm
going
to
use
our
goal
CG
since
Intuit
use
a
server
CD
for
github
deployment.
This
is
our
two
of
choice
to
to
get
ops,
so
this
particular
instance
has
applications.
C
C
It
will
figure
out
which
sources
of
applications
there
are
in
this
case,
it's
a
whole
repo,
just
one
customized
application
and
second
piece
is
where
I
want
to
deploy
it,
which
is
my
cluster
I'm,
going
to
deploy
into
the
same
cluster
where
our
NGO
CDs
running
I
also
have
to
tell
which
namespace
application
supposed
to
be
deployed
into
which
is
rollouts
demo
I
created
it
before
the
presentation.
I
have
to
give
a
name
to
my
application
as
a
reference
plus.
This
is
optional
field.
I
can
enable
outer
thinking.
C
C
There
is
a
project,
so
once
application
is,
there
is
created.
You
can
see
that
something
start
happening.
So
to
get
more
information
about
the
application,
you
can
just
click
on
the
app
icon
and
you
would
see
all
the
resources
which
we
were
created.
So
this
for
are
ones
which
are
defined
in
git
repository
and
each
one
created
some
child
resources.
So
the
most
important
thing
one
is
roll
out.
It
creates
a
duplicate
set
replica
set
created
five
ports.
All
of
them
are
running.
There
is
two
pieces
of
information.
C
I
want
to
highlight
the
sync
status
and
the
hub
status,
so
sync
status,
I
think
you
can
guess
what
it
is.
If
it's
green,
that
means
we
have
in
the
cluster
the
same
thing
which
we
want
and
what
we
commit
it
in
to
get,
and
the
second
is
health
status,
so
health
status
is
again
self-explanatory.
If
it's
progressing.
That
means
your
applications
to
have
a
chance
to
be
healthy.
If
once
it
is
healthy,
it's
up
and
and
let's
go
ahead
and
prove
it.
So
this
is
the
shortcut
which
our
Peugeot
gives
us.
C
C
It
keeps
sending
high
pay
queries
every
like
few
milliseconds
and
each
have
a
request
solves
a
string
which
is
treated
as
a
color,
and
it's
used
to
produce
one
of
these
little
circles
so
we're
getting
all
the
red
circles
continuously,
which
means
the
version
of
application,
is
only
serving
like
a
red
color
and
great.
That's
that's
it.
Now.
We
are
ready
to
go
and
change
something.
So
as
a
step
one.
C
Usually
you
would
make
a
code
change
committed
and
then
your
pipeline
would
pick
it
up,
make
a
build
new
version
and
committed
a
change
into
your
deployment,
repo,
which
I
am
going
to
do
manually
so
and
of
course,
I
already
built
bunch
of
images
before
the
demo
so
happens
to
use
customize
to
manage
my
manifests
and
so
I
have
a
convenience.
I
can
just
go
and
change
image
tag
in
customize
docume
file,
so
lets
me
change
the
red
color
into
green
and
committed
so,
and
I
want
to
switch
quickly
to
the
algal
UI.
C
So
let
me
just
explain
what
just
happened.
Agro
noticed
there
was
a
change
in
repo
and
the
things
that
has
changed
switched
into
outer
sync
and
then
algo
decided
to
fix
it
and
immediately
updated.
My
rollout.
If
we
took
like
a
longer
time,
we
would
notice
that
robot
was
out
of
sync
as
well,
so
once
robot
is
updated,
it
inspect
the
spec
of
its
own
spec
and
decides
to
you
know,
follow
the
steps
of
a
canary
deployment.
C
A
traffic
and
I
want
to
highlight
kind
of
two
things
here,
one
than
you
already
mentioned.
So
it
is,
if
you
using
a
rolling
update
and
you
want
to
switch
to
a
rollout,
you
can
at
the
first
step,
just
change,
IP,
I
version
and
kind
and
great
you
keep
using
crawling
update.
Nothing
happened
at
all.
As
the
next
step.
You
can
decide
to
introduce
canary
strategy
and
great
you
had.
C
You
cannot
easily
migrate
it
from
rolling
update
to
canary
and
second
I
did
a
single
change
in
a
get
and
I
can
just
define
the
target
state
and
it's
very
good
opt
friendly,
because
that's
what
you
want
to
do.
We
just
want
to
make
one
change,
even
though
the
process
takes
two
three
or
more
steps
and
yeah.
So
what
we
did
we
just
took
all
of
that
logic
and
put
it
into
controller
inside
of
a
cluster
and
I
promised
to
mention
how
would
you
automate
it?
So
you
have
choices
you
can
either
use.
C
C
One
thing
we
have
not
done
yet
is:
we
have
not
completed
the
deployment
yet
so
what
else
is
changed?
Is
the
application
status
its
fished
from
the
progressing
to
suspended
State,
and
this
is
the
signal
to
UCI,
where
you
actually
want
to
execute
that
CLI
command
to
to
tell
they're
a
lot
controller
to
continue,
and
as
I
mentioned,
there
is
CLI
I'm
going
to
just
use
UI
and
I'm
going
to
tell
to
our
guchigi
okay.
Send
the
signal
to
controller
at
all.
All
controller
and
signal
means
just
update
the
spec
overall
out.
C
Robot
start
doing
his
job,
yes,
and
we
can
change
that.
We
suddenly
start
getting
all
the
green
traffic,
which
is
great.
That
means
we
kind
of
we
slowly
migrated
to
the
next
version
of
our
application.
So
now
we
are
ready
to
you
know,
switch
back
to
deployment
and
I
can
demonstrate
to
one
more
kind
of
advantage
of
a
rollout.
You
can
easily
changed
your
deployment
strategy
and
switch
into
blue
green
strategy.
So
let's
go
ahead
and
do
it
so,
as
you
can
imagine,
all
I
have
to
do
just
go
ahead
and
change
the
deployment
strategy.
C
C
Application
fish
from
in
sync
to
out
of
seeing
how
our
controller
decided
to
go
ahead
and
execute
the
strategy
which
is
blue,
green
and
blue
green
means
create
just
a
full
copy
of
my
production.
We
can
see
the
two
replica
sets
fully
kind
of
scale
up
to
the
max
number
of
replicas.
Let's
make
sure
we're
still
serving
green,
because
I
have
not
yet
approved
to
you
know,
promote
my
blue
to
green
and
I
can
promote
it
in
the
same
way.
I
can
just
do
the
UCLA
I
mean
if
you
use
Jenkins
this.
C
Is
you
che
chance
to
test
the
blue
version,
and
that
is
okay?
There
is
one
more
agro,
CG
feature
which
I
want
to
demo,
which
is
network
view.
It
is
meant
to
just
visualize
your
traffic
flow
and
I
can
use
it
to
figure
out.
You
know
the.
Where
is
my
blue
version?
This
on
the
I
can
see.
There
are
two
external
points
where
which
can
sort
the
traffic
so
as
they
mentioned,
blue
green
has
preview
service.
In
my
case,
it's
a
rollout
demo.
C
Preview
I
can
get
its
IP
address,
open
it
in
a
browser
and
then
I
can
make
sure
it's
blue
and
almost
see
a
light.
See
a
drop
can
do
that
for
me
testing,
yes,
that's
it!
So
it's
just
tested,
it's
really
blue
and
now,
let's,
let's
confirm
the
deployment
using
Qi,
which
is
confirmed
and
so
yeah
once
it
is
confirmed,
we're
going
to
scale
down
all
of
them
green
version
right
in
green
colors
and
just
keep
blue
and
as
you
can
see,
that
switch
happened
kind
of
instantly
because
we
didn't
want.
B
So
that
was
our
go
rollouts
and
from
here
we're
gonna
talk
about
what's
next,
so
we
definitely
want
to
add
a
surface
mesh
integration.
That
announcement
with
the
server
Smith's
mesh
interface
is
definitely
exciting,
because
that
will
enable
us
to
service
a
wide
variety
of
service
mesh
providers,
and
it
also
enabled
us
to
do
more
fine-grain
traffic
shaping,
so
you
don't
have
to
expose
nearly
as
much
traffic
for
two
year
canary
without
having
to
run
large
numbers
of
replicas.
B
B
We
didn't
want
to
introduce
any
analysis
on
into
the
rollout
controller
because
we
feel
like
that
could
be
a
separate
problem
that
we
could
really
solve
with
a
separate
promotion
CRD
that
will
enable
us
to
have
more
flexibility
with
the
promotion
process
and
integrate
with
a
bunch
of
different
vendors
versus
just
having
it
all
kind
of
bundled
into
one
say
your
need.
Finally,
we
want.
We
are
also
working
on
adding
a
be
testing
where
we
could
run
multiple
versions
at
the
same
time
so
that
we
can
really
enable
experimentation
throughout
the
deployment
process.
B
So,
if
you
have
any
questions,
please
feel
free
to
check
out
any
of
these
links.
The
Argo
rollout,
so
the
Argo
CD
web
pages
are
a
great
resource
and
also,
if
you're
interested
by
this,
please
feel
free
to
check
out
the
into
a
careers
page
we're
always
hiring
and
yeah
with
that.
Thank
you.
So
much
for
your
time.
D
Thanks
for
the
demo,
that's
great
ideas:
I
was
wondering
what
would
it
make
sense
to
maybe
have
all
these
ideas
being
merged
into
the
deployment
from
communities
like?
Why
did
you
build
your
own
way
of
deployment
instead
of
going
to
the
communities
project
and
contribute
to
having
this
supported
by
them?
First
questions.
Second
questions:
just
out
of
curiosity,
people
do
stupid
things.
What
if
someone
have
like
one
replica
and
try
to
make
percentages?
How
are
you
dealing
with
this
Thanks.
C
B
C
E
C
That
this
is
the
point
of
a
controller.
This
is
something
we
could
not
do
in
Jenkins
controller
keep
watching
continuously,
no
matter
what
you
do,
I
mean
it
will
try
to
get
you
back
into
the
kind
of
state
it
thinks
it's
supposed
to
be.
So,
even
if
you
try
to
you,
know
engineer
applica
set,
it
will
recreate.
If
you
try
to
scale
up
and
controller
would
see.
Oh,
the
template
is
not
supposed
to
have
that
many
number
of
particles
kept
down
and
you
can
do
you
know
interesting
things.
C
So
if
you
in
the
middle
of
blue,
green
or
canary
deployment-
and
you
don't
like
what
you
see-
you
can
just
go
ahead
and
roll
back-
let's
say
your
Oba
committee
and
get
and
control
would
know
what
is
it
and
it
will
treat
it
in
a
right
way
like
it
would
figure
out
that
oh
I,
just
I
was
supposed
to
deploy
canary
new
version
of
a
canary
I'm
now
supposed
to
run
the
previous
version.
So
it
would
just
cancel
the
canary
and
kind
of
BEC
go
back
into
active
State
and
it
was
hard
to
do.
F
Stadium,
thank
you
for
a
great
talk
and
what
I'm
assume
will
be
a
very
successful
tool
as
well
having
been
to
earlier
talks
on
pod
readiness
gates
and
knowing
that
you
guys
also
work
on
for
Mars.
Are
there
any
plans
to
integrate
those
into
the
canary
release
process
to
have
automatic
canary
analysis
and
promotion
throughout
environments.
C
E
C
Kind
of
question
about
algo
CT
right,
so
this
is
I,
think
yeah.
So
idea
of
Argus
eg,
it's
a
multi
cluster
management
tool.
You
can
use
it
in
two
ways:
you
can
just
manage
objects
in
the
same
cluster,
where
your
application
is
running,
apps
algis,
it
is
running
or
you
can
just
add
credentials
to
an
external
cluster
and
application
has
a
target
cluster,
URL
and
namespace.
So,
yes,
and
that's
why
Argo
is
able
to
connect
to
that
external
cluster
and
you
know,
do
the
changes.