►
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Canary Deployments for Infrastructure Upgrades - Stephen Atwell, Armory
Canary and Blue-green deployments have become more commonplace when deploying end-user applications because they can decrease the risk of frequent deployments. This talk explores why these deployment methodologies matter for infrastructure tooling as well.
I will tell the story of why I started managing my prometheus upgrades via canary and later blue/green deployments, how I use them to ensure that none of my application monitoring breaks anytime I upgrade prometheus, and the challenges I conquered along the way.
benefits:
Understand the benefits of applying canary deployments and automated canary analysis to an unusual problem space, and how doing so can increase the stability of your infrastructure platform.
A
Awesome
so
I'm
gonna
start
by
telling
you
guys
about
a
problem.
I've
had
a
few
times
this
year.
So
three
times
this
year
when
I've
gone
and
upgraded
pieces
of
my
infrastructure,
it's
broken
me
completely
and
every
single
time
I
haven't
discovered
it
until
a
couple
days
later,
when
you
know
I'm
in
a
time
crunch
and
I
need
things
to
work
and
then
I
discover
they
don't
and
you've
got
to
roll
back
and
it's
late
enough
that
it's
hard.
A
A
Why
did
these
break
these
broke
because
there's
a
lot
of
different
moving
Parts
there's
lots
of
different
pieces
of
configuration.
These
are
complex
systems
and
they
don't
have
my
configuration
to
test
with.
So
they
can't
really
guarantee
everything's
going
to
work
and
I
need
a
low
effort
way
to
have
a
stable
environment
and
I
really
don't
want
to
be
running
software.
That's
two
years
out
of
date,
so
I
got
to
be
updating
regularly.
A
How
did
I
detect
these
issues?
Well,
every
single
one
of
these
issues
was
detected
because
something
else
I
had
in
place
already
broke.
So
you
know,
I
had
other
applications
that
were
using
Prometheus
for
the
monitoring
at
some
point,
I
look
at
a
chart
realize
the
chart's
not
working
in
grafana.
Similarly,
when
when
kubernetes
broke,
I
had
certain
applications
that
were
using
Vault
secrets
and
they
just
wouldn't
start
so
what
I
realized
was
issue?
A
One
many
of
my
applications
are
using
Canary
deployments,
they're
querying
Prometheus
with
health
checks,
while
they
deploy
and
hey
number
two.
If
I
redeploy
an
application,
it's
going
to
restart
the
pots.
So
if
I
redeploy
my
applications,
I
would
have
detected
the
issues
in
all
three
of
these
situations.
A
This
got
me
thinking
about
a
way
that
at
a
prior
SAS
company,
we
handled
a
complex
systems,
testing
challenge.
We
didn't
have
enough
unit
test
coverage.
It
was
a
very
complex
system
and
every
customer
looked
different.
So
how
did
we
test?
We
set
up
an
infrastructure
to
automatically
run
every
customer's
configuration
through
test
validation
as
part
of
our
release
process.
So
before
we
shipped,
we
validated
every
customer.
Hey
I've
got
all
my
applications
I'm
choosing
when
to
upgrade
Prometheus
I'm
choosing
when
to
upgrade
kubernetes
I
can
do
the
same
thing.
A
I
can
think
of
my
applications
as
test
cases
for
the
infrastructure
that
is
supporting
them.
So
you
know,
I
didn't
really
have
time
to
build
much
tests.
I
didn't
want
to
manage
anything
extra.
So
you
know
I
didn't
want
to
set
up
an
extra
staging
environment,
but
you
know
for
my
purpose:
I
could
also
handle
schedule
downtime.
A
So
you
know
I
start
going.
How
do
I
use
my
existing
monitoring?
How
do
I
use
my
existing
deployment
pipelines
for
apps
in
order
to
make
sure
that
everything
else
works
so
in
order
to
do
things
in
place
and
and
kind
of
deal
with,
this
I
decided
to
try
leveraging
Advanced
deployment
strategies
so
in
case
anyone's
not
aware,
Canary
deployment
is
a
process
where
you
stand
up
a
new
version
of
code
alongside
the
old
version
and
then
slowly
send
new
traffic
to
the
new
version
in
between
every
increase
in
traffic.
A
You
run
some
automated
validations.
You
know
maybe
you're
querying
Prometheus,
maybe
you're
running
a
web
hook,
Etc
just
to
make
sure
everything's
still
healthy.
Once
everything
seems
healthy,
then
you,
you
know,
increase
it
to
the
next
level
and
finally,
when
everything's
at
100
to
the
new
version,
you
shut
down
the
old
one.
A
Another
Advanced
deployment
strategy
that
I
considered
using
is
blue
green
deployments
it's
fairly
similar
only
instead
of
incrementally
increasing
traffic.
You
kind
of
move
it
all
over
in
bulk,
but
you
validate
service
Health,
both
before
and
after
you
move
traffic,
so
similar
Concepts
and
I
actually
landed
on.
For
this
use
case,
doing
a
bit
of
a
hybrid,
so
I
was
originally
going
to
use
a
canary
strategy.
I
started
setting
that
up
and
I
went.
You
know
for
my
use
case:
I,
don't
actually
care
about,
like
limiting
traffic,
didn't
care
about
limiting
blast
radius.
A
I
just
cared
about
running
those
automated
tests
in
place
when
I'm
upgrading
it-
and
you
know
Bluegreen-
does
that
well,
but
you're
testing
before
you
route
traffic,
you're
testing
after
I
just
wanted
to
test
once
so.
What
I
did
is
kind
of
a
little
bit
of
a
hybrid
I?
Still
did
it
using
a
canary
strategy
and
a
CD
tool,
I,
basically
just
spun
up
the
new
version,
send
all
the
traffic
to
it
and
then
ran
all
my
automated
validation
didn't
write
any
new
automation
I.
A
So
all
of
this
was
done
by
basically
just
adding
like
one
new
deployment
pipeline
in
a
off-the-shelf
deployment
pipeline
tool.
So
the
strategy
I
wound
up
using
was
basically
deploy.
100
of
traffic
run
run.
My
existing
Prometheus
monitoring
through
Prometheus
make
sure
every
query
is
giving
me
back
a
value.
Make
sure
that
value
seems
reasonable.
A
A
We
then
the
the
other
thing
I
wanted
to
do,
though,
is
I
was
talking
about
how
my
apps
are
our
test
cases
and
how
like
Vault,
breaking
I,
only
noticed
because
my
pods
wouldn't
start
so
in
order
to
catch
that
class
of
issue
I
then
went
and
cool
we
validated
Prometheus,
we
validated
my
queries
are
happy.
Now,
let's
go
validate
all
of
the
applications
that
use
Prometheus
in
their
deployment
pipeline
still
validate
so
I
have
a
bunch
of
other
apps
that
are
doing
Canarian
blue
green
strategies.
Do
those
still
work?
A
If
they
do,
then
I
can
probably
safely
upgrade
Prometheus
if
they
don't
well,
I've
got
a
problem.
I
need
to
go
investigate
that
and
I
can
roll
back
and
take
a
look
so
when
using
other
applications
as
a
test
case,
how
did
I
do
this
so
I
started
by
updating
Prometheus
I
then
query
the
existing
Prometheus
monitoring,
which
was
you
know,
making
sure
that
everything
I
knew
should
work
was
working
and
then
I
simply
triggered
cicd
pipelines.
I,
basically
just
told
every
other
app
go
redeploy
yourself:
they
redeployed
themselves.
A
They
ran
where
their
logic
was.
You
know
many
of
them
are
made
an
analysis
against
Prometheus,
which
is
again
validating
the
health
of
Prometheus.
If
those
apps
failed,
they
were
automatically
rolling
back
so
when
they
roll
back
their
app
I
roll
back.
My
Prometheus
upgrade
suddenly
I
know
what
app
breaks.
I
can
go
and
take
a
look
and
once
I've
upgraded
I
know
all
my
apps
can
deploy.
A
Having
done
this,
what's
my
experience,
so
you
know
I
I
talked
earlier
about
how
Prometheus
broke
me.
So
every
time
Prometheus
broke
me,
it
was
causing
one
of
my
existing
known
queries
to
fail.
So
the
first
one
was
a
Helm
chart,
change
that
caused
it
to
no
longer
ingest.
My
annotations,
hey
I,
have
queries
analyzing
annotations,
they
all
broke.
The
other
one
was
a
out
of
the
box
Prometheus
metric
that
was
renamed.
A
How,
when
I
upgrade
I
can
easily
see
which
metric
broke
I
can
click
in
I
can
get
the
Prometheus
query?
I
can
go
run
that
against
Prometheus
and
then
I
can
decide.
Okay,
cool
I
know
it's
breaking.
Let's
roll
back.
Let's
go
read
the
Prometheus
release
notes.
Hopefully
they
told
me
what
to
do
about
it.
Then
I
can
fix
it
and
then
finally
upgrade
so
I
want
to
do
this
to
more
of
my
in
five.
A
So
far,
I've
only
done
Prometheus
and
open
cost,
but
I
believe
it
can
help
me
with
more
and
there's
a
couple
benefits
that
I
got
that
I
wasn't
expecting
when
I
started,
so
one
of
them
is
now
I
can
easily
see
when
I'm
deploying
what
broke
before
it
was.
You
know
all
a
bunch
of
shell
scripts
and
you
know
now
I
know
exactly
which
system
broke.
I
also
was
using
a
declarative
deployment
tool.
A
So
when
I'm
provisioning,
a
new
cluster
I
run
the
exact
same
logic,
but
you
know
Vault
I
want
to
run
it's
broke
before
before
I
upgrade
it
next.
I
need
to
flip
it.
Over
to
this
console
is
used
by
My
Vault
installation
I
also
run
Spinnaker.
So
when,
when
Vault
broke
me
the
thing
that
actually
detected,
it
was
the
next
time
I
updated
Spinnaker,
because
I
was
using
Vault
secrets
in
my
Spinnaker
Pipelines.
A
So
in
order
to
you
know,
detect
Vault
I
need
to
also
apply
Spinnaker
to
this.
Some
of
my
apps
that
are
using
Canary
are
using
Linker
d
as
a
service
mesh.
So
you
know
if
I'm,
using
those
in
order
to
Route
traffic
during
deployment
again
I
can
validate
that
those
Linker
d,
the
Linker
D
infrastructure,
is
working
properly
through
the
same
means.
A
So
how
do
we
validate
these
different
use
cases?
So
you
know
these
are
kind
of
different
use
cases
for
different
things,
all
of
which
simply
running
my
apps
pipelines
and
redeploying
will
test.
So
you
know
redeploying
Spinnaker
gets
vaulting
console
Spinnaker
and
open
policy
agent
I
use
in
my
to
deploy
the
apps
that
I
would
deploy
with
Spinnaker.
So
you
know
if
I
redeploy
those
apps
I've
just
validated
the
health
of
both
of
those
systems.
Linker
D
I
mentioned
Jenkins,
so
my
integration
testing
could
have
action.
Some
are
in
Jenkins.
A
How
do
I
validate
Jenkins
well
I
run
all
I
redeploy
the
apps
that
use
Jenkins
revalidate
their
integration
tests
work
and
for
kubernetes,
so
I
use
eks
I,
don't
know
if
you
guys
have
ever
tried
to
roll
back
at
eks
cluster,
but
the
last
two
times
I
did
that
it.
It
made
me
hate
my
life.
So
my
current
intention
for
kubernetes
upgrades
is
it's
a
declarative
deployment
tool
rather
than
doing
it
as
upgrade
testing
I'll
stand
up.
A
A
So
I
have
yes
so
on
in
the
case
of
Prometheus,
it
uses,
it's
got
multiple
different
ways
to
deploy
and
I've
I've
done
two
different
versions
of
it.
This
way,
so
my
main
Prometheus
implementation
is
using
Prometheus
operator
and
is
using
stateful
sets,
and
one
of
the
things
which
made
me
flip
to
sending
100
of
traffic
is
the
Prometheus
operator
doesn't
like
having
two
copies
of
itself
running
in
parallel,
but
it
Prometheus
upgrades
and
downgrades
on
that
stateful
set
cleanly
so
basically
spin
up
the
new.
A
A
Destructive
intrusive
activity
that
you
want
to
avoid,
it
depends
on
your
applications
and
your
risk
profile.
So,
in
my
career,
I
have
seen
a
few
different
types
of
applications,
so
at
a
at
a
prior
company
that
shall
remain
nameless.
We
had
a
product
that
took
you
know,
depending
on
the
company
you
know,
ran,
ran
pertended
architecture
took
four
to
like
48
hours
to
start
redeploying
something
like
that.
As
part
of
this
would
be
a
really
bad
idea.
A
Most
of
the
applications
I
have
today
are
microservices
based,
and
you
know,
take
10
to
20
seconds
to
start
and
it's
a
relatively
low
risk
thing.
I
would
definitely
be
mindful
of
what
applications
you
choose
this
approach
on.
If
you
have
something
that
is
going
to
take,
you
know
hours
to
start,
it's
probably
not
redeploying.
It
is
probably
not
your
best
bet
at
that
point.
I
would
recommend
doing
something
like
maybe
kill.
One
pod
make
sure
that
one
pod
start
restarts.
A
So
I
have,
but
most
of
my
data
migrations
have
been
fairly
simple
and
I
haven't
done
them
in
my
deployment
strategy.
I've
done
them
in
my
app
and
basically
had
it
like
check.
A
current
database
schema
version
make
sure
that
you
maintain
schema
compatibility
for
a
while
I
have
a
customer,
so
armor
Armory
makes
deployment.