►
From YouTube: Progressive Application Delivery with Consul Service Mesh and Kubernetes - Zan Markan & Nic Jackson
Description
Even with a high level of test automation and coverage, deploying applications to production carry a particular risk. Differences in infrastructure, configuration, application software, and data stores can mean that an application that performed flawlessly through the development and test cycles can fail when introduced to the production environment.
To counter the risk involved with deploying to production, we developed a technique called canary testing. Rather than immediately replacing existing software with a new version, you gradually introduce it while monitoring its behavior.
In this talk, you will learn how to implement a fully automated progressive delivery pipeline for canary testing using Consul Service Mesh, Flagger, Circle CI, and Kubernetes. We will look at the end-to-end process from code commit to production release, and by the end of the talk, you will understand the core configuration of all the tools involved so that you can replicate the workflow in your environment.
A
All
right,
okay,
so
what
we
want
to
talk
to
you
all
today
about
is
progressive
delivery
and
specifically
how
how
the
kind
of
the
concept
of
delivery
has
has
has
been
dramatically
enabled
by
technology
over
over
the
last
couple
of
years,
and
it's
you
know
it's
a
completely
different
landscape
from
when
I
was
deploying
software
applications
into
production.
Just
you
know,
four
years
ago
before
I
I
joined
hashicorp
as
a
as
a
developer
advocate
so
progressive
delivery,
but
I
mean
zlan's
going
to
start
us
off
we're
going
to
kind
of.
B
Yeah
hello,
so
our
story
begins
with
the
tool
called
cloud
native,
translate
or
cloudnate
translate
as
I
and
nick
like
to
call
it.
So
this
snap
we
wanted
to
build
for
our
global
audience
and
essentially
it
lets
you
make
translations
from
any
language
into
any
other
language,
so
we're
making
the
communication
over
the
internet
truly
a
global
thing
and
accessible
to
all
everywhere
and
such
an
innovative
app.
This
is
right
so
yeah
we
were
working
on
this
app.
We
developed
it
and
we
deployed
it.
We
had
we
had
an
mvp
running.
B
It
was
pretty
good.
It
translated
everything
from
english
into
slovenian,
so
as
as
as
as
usable
as
you
might
imagine,
this
might
be,
and
as
we're
like
really
following
the
best
practices,
we
were
good
cloud
native
citizens.
We
had
a
lot
of
automation.
Set
up
for
kind
of
deployment
pipeline
was
all
kind
of
in
a
great
shape,
and
we
had
automated
tests
to
kind
of
make
sure
that
our
software
was
always
working
and
our
ci
would
always
run
them,
which
is
also
great.
B
But
yes,
despite
that,
all
the
tests
we
had,
we
still
ended
up
with
a
broken
application
in
production,
and
this
was
actually
a
particularly
bad
way
because
our
users
were
seeing
errors
and
our
translation
app
most
certainly
did
not
translate.
And
this
is
the
tale
of
this
application
and
what
went
wrong
and
how
we
improved
our
deployment,
pipelines
and
lives
in
the
process.
A
B
So
once
the
code
is
committed
and
pushed
to
github,
then
this
is
then
picked
up
by
our
ci
cd
tool,
which
is
circle
ci
for
us,
and
this
essentially
runs
our
continuous
integration
or
ci
pipeline.
This
builds
our
app
runs
all
the
tests
that
we
have
and
after
they
all
pass,
so
they
all
pass
for
us.
So
all
good
here-
and
this
is
great-
because
we
don't
really
have
to
rely
on
running
these
tests
locally
on
our
developer
machines.
We
kind
of
offload
this
to
a
nicely
scripted
and
safe
and
reproducible
environment
in
the
cloud.
B
It's
also
everything
is
automated
and
all
we
need
to
do
now
is
go
to
github
and
show
the
next
slide.
Please.
Yes,
you
go
to
github
and
we're
about
to
merge
our
pull
request
here,
which
essentially
puts
our
branch
work
into
the
into
the
main
which
which
we
let
nick
do
and
once
nick
nick
made
that
merge.
B
A
Then
it
all
went
wrong
because
actually,
what
we
started
to
see
from
the
logs
was
a
an
elevated
level
of
500
errors
coming
from
the
api
server
immediately
we'd
gone
from
something
which
was
very
stable
and
had
been
for
a
long
time
to
something
that
would
every
request
was
was
failing.
So
of
course
you
go
into
that
kind
of
usual
usual
process
of
oh
dear.
We
need
to
fix
this
everybody's
running
around
offering
help
the
pointy
head.
Boss
is
sort
of
coming
over
asking
what
we
can
do.
A
We
tell
the
pony
head
boss,
please
just
you
know,
go
and
go
and
sit
quietly
over
there
and
kind
of
leave
us
leave
us
be
to
fix
this.
The
point
your
head
boss,
then
takes
offense
at
that.
Then
you've
got
to
deal
with
two
problems:
a
broken
build
and
an
offended
boss,
but
ultimately
you
you
kind
of
get
through
the
process
of
let's
start
to
to
do
the
diagnostics
and
debug,
where
this
is
going
wrong
and
wrong.
It
was
really
going.
Oh.
B
Yeah,
this
is
not
what
you
want,
your
translation
app
to
look
like
when
you
wanted
to
make
translations
by
the
way.
Essentially,
this
is
the
end.
This
is
the
kind
of
bugs
that
our
end
users
ended
up,
seeing
and
to
understand
why
we
had
to
look
through
the
through
the
logs
and
our
logs
would
essentially
tell
us
a
story
and
the
problem
existed
between
keyboard
and
chair.
B
A
B
They
can
actually
exist
in
many
parts
of
your
well
actually
in
all
parts
of
your
software
delivery
life
cycle,
for
example,
your
infrastructure
might
change
what
you're
running
the
physical
hardware
that
you're
running
or
your
software
might
change,
or
you
might
introduce
an
upgraded
dependency
which
brings
a
slightly
changed,
behavior
or
kind
of
down
the
line.
It
has
a
dependency
that
breaks
something
or
has
a
or
has
a
or
has
a
bug
bugs
happen.
Obviously,
and
that
can
happen
or
there
are
also
scenarios
where
your
tests
don't
really
work.
B
That
well
and
one
example
is
where
you
don't
have
plan.
We
haven't
planned
for
the
scenarios
and
that
obviously
means
that
you
don't
have
tests
for
it.
So
example
the
scale
that
you're
thinking
about
the
your
application
might
operate
might
change,
or
maybe
you're,
seeing
spikes
in
usage
that
you
really
haven't
foreseen,
and
maybe
there
is
a
change
in
like
shift
in
the
payload
pattern,
what
your
users
are
actually
sending
or
like
using
your
app
for
so
exactly.
A
B
B
We
mentioned
a
few
of
them
already
so
circle
ci,
but
also
kubernetes
and
and
flagger,
for
monitoring
and
console
for
for
the
service
mesh
for
all
the
routing
magic.
So
that's
what
we'll
be
covering
today,
so
there's
also
two
patterns
we
want
to
cover.
So
canary
is
the
first
one
of
those.
The
idea
is
to
split
your
traffic
between
two
running
two
deployed
running
instances
of
the
app
or
api.
B
B
B
If
our,
if
our
app
hits
the
canary
and
that
holds
a
bug,
then
or
that
would
crash,
essentially
the
application
as
it
was
in
our
case,
this
would
essentially
retry
the
same
request
behind
the
scenes
before
that
kind
of
500
failure.
Response
reaches
the
user,
and
hopefully
this
will
end
up
hitting
our
primary
api.
The
one
that's
already
stable.
So
so
that's
the
canary
pattern
and
the
highlight
level
process
is
just
that.
Like
you,
you
push
some
bugs.
You
don't
catch
those
bugs,
because
your
tests
haven't
thought
about
every.
A
A
You
push
that
code
to
github
circle.
Ci
builds
your
code
and
it
also
executes
your
tests.
If
the
tests
are
passing
in
the
build
is
satisfied,
so
you
know
you're
going
to
get
that
sort
of
github
thing
all
the
all
the
checks
are
passing.
You
can
then
raise
a
pr.
You
raise
a
pr
pr
is
merged.
After
a
code
review,
you
go
back
into
that
sort
of
build
just
quick
sanity
check,
and
then
you
deploy
the
code
now
when
you're
deploying
the
code.
What
we're
doing
is
we're
literally
just
using
a
kubernetes
deployment.
A
A
Is
it
detects
that
there's
a
change
in
in
the
deployment
and
flagger
then
will
will
start
to
modify
the
traffic
routing
using
console
service
mesh,
because
the
way
that
things
work
with
console
service
mesh
is
all
of
your
traffic
between
all
of
your
services
are
flowing
through
the
service
mesh,
they're
intercepted
by
the
data
plane
or
the
proxy
before
they
hit
your
destination?
So
it
means
that
the
service
mesh
can
take
complete
control
over
what
the
shape
of
the
traffic
looks
like
and
to
where
it
up
it
goes
so
we
create
a
service.
A
Splitter
flagger
is
doing
that
automatically
for
us.
It
then
starts
to
monitor
the
metrics,
so
flagger
is
looking
at
the
the
metrics
from
from
our
application
in
prometheus
or
datadog
or
whatever
you
use
as
a
as
a
kind
of
a
metric
store,
and
then
it
started
like
trying
to
infer
some
information
about
that.
A
If
the
metric
meets
your
required
standard,
so
if
there
are
a
number
of
errors
which
are
acceptable
to
you
or
no
errors
at
all,
then
what
it
can
do
is
it
can
start
to
keep
increasing
the
percentage
of
traffic
that
goes
through
to
the
traffic
splitter.
So
you
start
low
start
10,
20,
30
40,
whatever
you
know
and
keep
going
up
because,
as
we
mentioned
earlier
on,
sometimes
bugs
only
manifest
at
a
particular
level
of
traffic,
you
can
push.
A
If
you
just
push
stuff
into
production,
you
give
it
a
couple
of
quick
refreshes
or
you
you're,
running
very
low
levels
of
traffic.
Through
it
you
can.
It
can
behave
correctly,
but
maybe,
when
you
hit
a
particular
volume
of
traffic
on
the
service,
you
start
to
run
into
issues
could
be
like
a
mutex,
a
database
lock
or
something
like
that.
A
So
this
is
my
my
my
currently
running
code
in
production.
Now
the
problem
with
this
code
is
that
I'm
not
I'm
a
go
developer,
I'm
not
actually
a
javascript
developer,
and
it
was
only
when
I'd
been
going
through
this.
This
investigation
that
I
realized
that
you've
got
this
async
await
thing
and
that
stuff
doesn't
necessarily
happen
in
a
standard
linear
fashion.
So,
even
though
we
our
unit
tests
were
checking
that
we
were
setting
the
languages,
we
weren't
checking
it
truly
in
an
asynchronous
nature.
A
So,
let's,
let's
just
sort
of
box
box
that
up
a
little
bit
so
I'm
going
to
do-
is
I'm
just
going
to
create
a
an
async
function,
I'm
going
to
call
it
fetch
languages
and
inside
of
fetch
languages,
I'm
literally
just
going
to
do
my
my
fetch,
but
what
I'm
going
to
do
is
I'm
going
to
use
the
keyword
wait
so
that
I
can.
I
can
ensure
that
I'm
not
returning
languages.
A
Fine
here
what
we
can
do
is
we
can
call
const
and
we're
gonna
do
await
batch
languages,
and
now
everything
should
execute
asynchronously.
So,
let's
just
before
I
forget
call
law
set
up
at
the
bottom
of
our
index.js
file
here,
but
this
now
should
should
work
and
everything
should
be
should
be
fine,
so
again,
usual
usual
kind
of
process
from
my
git
status.
Of
course.
Oh
of
course,
sorry
run
my
tests.
A
A
A
So
whilst
this
is
running,
what
I'm
going
to
do
is
I'm
I'm
going
to
cut
a
corner
for
the
sake
of
of
brevity
and
I'm
just
going
to
merge
this
so
we'll
come
back
to
this
later.
Just
pretend
I
didn't
do
that,
but
I
don't
want
you
to
have
to
wait
around
whilst
we
we
kind
of
discuss
more
about
what's
going
on
here,.
B
B
Is
defined
in
this
configure.tml
file,
which
is
in
this
dot
circle
ci
folder
at
the
top
level
of
your
repository,
and
that's
where
that's
where
everything
is
defined?
What
needs
to
happen
and
in
this
file
we
have
a
couple
of
things
or
actually
three
interesting
things.
One
is
orps,
that's
jobs
and
workflows,
so
jobs
are
essentially
a
collection
of.
B
So
you
don't
have
to
like
re
reinvent
the
wheel
and
the
way
our
kind
of
build,
docker
and
create
deployment
jobs
are
orchestrated,
is
in
this
thing
called
workflow,
which
is
essentially
a
place
where
you
arrange
your
jobs.
For
example,
we
want
this
node
test
job
to
run,
which
again
comes
from
this
node
orb,
so
you
don't
really
have
to
tell
it
anything.
B
You
just
give
it
a
node
version
and
that's
what
it's
going
to
do
it's
going
to
run
in
this
in
an
environment
that
has
this
node
version
installed,
then,
obviously,
we
want
to
run
the
build
docker
image,
which
kind
of
requires
a
node
test.
You
can
tell
it.
Okay,
these
these
jobs
are
kind
of
a
kind
of
requirement
for
this
latter
one
to
continue.
B
For
example,
you
want
to
run
your
tests
first
and
then
do
any
kind
of
builds
or
deployment
after
which
we
go
into
the
deployment
stage,
and
you
might
notice
that
for
one
we're
we're
doing
this
only
on
the
main
branch,
so
it
didn't
it
didn't
trigger
this
deployment
pipeline
portion
unless
we
merge
this
into
the
main
branch,
so
you
kind
of
have.
This
is
your
ci
part,
which
is
the
continuously
integrating
just
verifying
that
everything
builds
all
the
tests
are
working.
B
Everything
is
is,
is
fine
and
then
we're
obviously
jumping
into
this
deployment
workflow,
which
first
we
have
to.
We
have
this
approval
step,
which
is
like
a
manual
thing
saying:
okay,
is
this
really
what
you
want
to
do?
Yes,
you
have
to
click.
Yes,
I
want
to
approve
this
deployment.
Can
I
continue
and
then
we're
going
into
create,
update
deployment,
which
is,
does
all
the
gte
and
kubernetes
work?
So
nick
mentioned
this
customized
thing?
So
I
suppose
we
can
talk
about
this
a
little
bit
now.
A
A
This
this
step,
where
we're
looking
at
next
is,
is
the
deployment
step
so
in
the
deployment
step,
what
we're
going
to
be
doing
is
we
want
to
be
updating
that
deployment
and
we
want
to
be
pushing
that
notification
to
so.
Flagger
can
set
it
up.
So
how
we
do
the
deployment
is
is
is
again
very
standard,
it's
just
kind
of
kubernetes
setup,
so
let's
have
a
look
at
our
application
first,
so
our
application
is
just
a
standard
kubernetes
service.
A
Here
now
the
service
is,
is
because
we're
using
the
prometheus
operator
and
we're
using
service
monitors.
So
we
just
want
to
tell
prometheus
that
it
needs
to
scrape
metrics
from
from
this
application
service,
and
then
we
have
a
standard,
kubernetes
deployment,
and
there
is
literally,
even
though
this
is
going
to
be
part
of
the
service
mesh.
There
is
nothing
different
in
this
deployment.
Apart
from
these
two
lines
here
and
these
two
annotations,
what
they
do
is
the
the
controller
that
works
with
console
service
mesh
will
will
automatically
create
this.
A
It
picks
it
up
as
a
mutating
web
hook
and
it
will
modify
my
deployment
and
it's
going
to
add
all
of
the
the
service
mesh
data
plane
and
the
various
authentication
pieces
which
are
needed
in
order
for
this
part
to
take
part
in
service
mesh.
But
you
know
I
don't
have
to
configure
anything
different.
I
just
got
my
my
standard,
api,
etc,
etc.
A
A
So
what
I'm
doing
is
I'm
defining
a
canary
I'm
specifying
just
named
metadata
standard,
coupe,
stuff
and
then
the
target.
So
I
I'm
saying
I
want
you
to
monitor
the
api,
the
deployment,
a
deployment
called
api.
You
can
set
some
some
various
sort
of
settings
here,
I'm
setting
600
seconds.
If
there's
an
instance,
it
will
continue
to
try
and
monitor
errors
for
600
seconds
before
it
rolls
back.
A
We
have
the
the
port
that
the
service
is
operating
on
and
then
sort
of
the
analysis
criteria.
So
I
want
to
to
check
and
look
at
my
metrics
every
every
30
seconds.
I
think
the
the
kind
of
the
cardinality
that
I'm
running
on
my
prometheus
is
is
probably
no
greater
than
30
seconds.
So
if
I
set
that
to
five
seconds,
I'm
not
really
gonna
get
anything
better
out
of
that.
The
threshold
is
how
many
times
does
a
metric
check
fail
before
we
start
instantiating
the
rollback
and
then
the
the
weights?
A
A
Up
until
I
reach
a
maximum
weight
of
80
percent.
If
I
hit
80
of
traffic
going
through
my
canary
I'm
going
to
assume
it's
good
and
I'm
going
to
put
it
into
production
in
order
for
flagger
to
to
understand
how
like
what
what
the
criteria
for
success
or
fail,
is
you
you're
generally
going
to
be
looking
at
a
a
query
in
an
in
our
instance?
That's
going
to
be
a
query
which
is
a
prometheus
query,
so
I
specify
these
metrics
here
and
I'm
saying
I've
got
a
metric
template
called
console
requests.
A
I
can
again
I
can
set
some
some
discord
alerts
and
things
like
that,
as
well
or
slack
alerts,
so
the
metrics
are
defined
in
another
flagger
configuration
and
because
I'm
using
prometheus
as
my
my
matrix
store,
I
just
write
a
prometheus
query.
If
you
were
using
datadog
or
you
were
using
any
any
of
the
other
sort
of
providers
that
that
flagger
supports,
then
you
know
you
use
whatever,
whatever
you're
just
going
to
write
your
own
query.
A
So
here
I'm
writing
a
sum.
I'm
getting
the
the
envoy
cluster
upstream
requests,
so
this
is
a
metric
which
isn't
actually
coming
direct
from
my
application.
It's
coming
from
the
the
service
meshes
data
plane
envoy
in
this
instance,
and
it's
going
to
get
the
pod
based
on
a
replaceable
that
flagger
will
will
pop
into
there
and
where
we're
just
checking
some
things
like
the
the
envoy
cluster
name,
to
make
sure
that
we're
looking
at
the
right,
the
right
metric
and
then
we're
just
checking
the
response
code.
A
We
could
also
look
at
duration
because
we're
looking
at
just
kind
of
like
has
the
request,
passed
or
failed.
What
we
could
also
look
at
are
things
like:
how
long
does
it
take
for
a
request,
so
in
the
instance
of
I
could
have
introduced
a
bug
which
just
made
the
application
really
slow?
Again,
I
can.
I
can
define
a
metric
template,
which
is
using
things
like
the
sort
of
the
the
time-based
histograms
and
stuff
like
that,
and
look
at
things
like
my
99
percentiles
to
it's.
A
They
actually
built
the
first
version
of
the
s2
adapter
for
rasamay,
so
super
big
props
out
to
to
kinvoke
on
that.
So
what's
next!
Well,
next
is
rooting,
so
flagger
is
going
to
to
kind
of
configure
those
traffic
splitters.
But
there's
there's
more
to
it.
There's
there's
a
couple
of
extra
pieces
of
configuration
that
we
we
need.
A
What
we
actually
need
in
terms
of
console
configuration
is
the
the
configuration
around
what
our
retry
is
going
to
look
like.
We
need
our
service
splitter,
which
kind
of
allows
us
to
configure
the
the
division
of
traffic
that
goes
between
the
canary
and
the
the
the
primary
and
we
need
a
service
resolver
and
a
surface.
Resolver
is
a
construct
in
terms
of
console
which
allows
you
to
kind
of
create,
I
suppose,
virtual
services
for
for
one
of
a
better
thing
and
we
can
create
those
services
based
on
a
lookup
in
in
the
service
catalog.
A
A
The
service
resolver
first,
so
the
service
resolver
is,
as
I
say,
think
of
it
in
terms
of
like
a
virtual
service.
So
I
have
a
virtual
service
called
api,
which
contains
a
number
of
subsets,
so
I
have
a
subset
called
api
primary
and
a
subset
called
api
canary
now,
if
I
look
at
console
and
what
I
have
as
my
service
catalog,
so
here
we
can
see
our
api
service
and
if
I
look
at
the
instances
these
are
the
pods.
I've
got
three
instances
of
three
pods
running
of
my
api.
A
So
in
my
my
service
resolver,
what
I
can
say
is,
I
can
say,
hey
for
for
create
me
a
virtual
service
for
the
api
and
create
a
subdivision.
If
the
service
id
contains
api
primary
put
that
into
the
api
primary
subset.
If
it
doesn't
contain
api
primary,
then
it's
then
it's
connect,
it's
part
of
the
canary.
A
So
at
the
moment
we
only
have
the
primaries
running
when,
when
we
trip
this
in
a
second
you'll,
see
the
the
canaries
come
into
there.
But
this
is
this:
is
automatic
configuration
and
automatic
routing
that's
going
on
inside
of
the
service
mesh?
I
don't
need
to
reconfigure
any
load,
balancers
or
nginx,
or
anything
like
that.
This
is
service
to
service
traffic,
it's
internal,
and
then
I
have
my
my
service
router
and
my
service
router
allows
me
to
configure
things
like
the
the
number
of
retries.
A
So
we
said
that
you
know
we
want
to
kind
of
abstract
the
error
message
from
the
end
user,
so
we're
gonna
we're
gonna
retry,
because
we're
gonna.
If
we
have
a
bad
instance,
we
will
then
retry
it
and
hit
the
good
instance
setting
a
number
of
retries
to
five
for
the
service
api
and
the
status
codes
that
we
constitute.
Failure
are
going
to
be
500,
501,
502.503.
A
Now
the
service
splitter
service
splitter,
this
is
created
by
flagger
automatically.
I
never
have
to
create
this,
but
it
looks
like
this,
so
we
create
the
the
server
splitter.
We
just
define
a
weight,
so
what
percentage
of
traffic
do
I
want
to
go
to
the
subset
api
canary
of
the
service,
api,
10
and
90,
etc?
A
A
So
if
we
go
over
to
to
our
workflow
what
this
is
the
the
pr
that
I
created
earlier
on.
What
I
have
is
this
this
approved
step,
because
we
want
to
do
a
kind
of
a
manual
approval
process
don't
have
to.
I
think
you
kind
of
need
to
look
at
this
sort
of
thing,
which
is
based
on
the
importance
of
the
risk
associated
with
your
application.
A
I
think
when
you
get
really
good
and
you've
got
a
very
sort
of
robust
testing
policy,
combined
with
a
sort
of
a
very
robust
monitoring
and
canary
deployment,
you
can
automated
deploy
a
lot
of
your
applications,
but
this
is
handy
to
have
the
approved
job
in
this
instance.
There
I
go.
I
hit
the
button.
A
Blame
is
a
terrible,
terrible
thing,
there's
no
hiding
anymore,
which
is
really
bad
for
me.
So
so
I'm
doing
my
deployment
now.
So
what
circle
is
going
to
be
doing?
Is
it's
just
going
to
run
that
kubectl
apply
with
my
my
customized
configuration
now,
I'm
I'm
running
everything
here
in
in
google
google
cloud.
I've
got
a
cluster
in
gcp,
three
nodes,
six
cpus.
A
A
Now,
I'm
looking
at
console
here
and
console's
just
giving
me
a
dashboard
of
what's
going
on
in
my
system,
but
immediately
you
can
see
that
we've
now
got
api
primary
and
then
this
thing
just
called
api-
and
this
is
my
my
canary
so
flagger.
What
it
actually
does
is
the
first
time
you
make
a
deployment
of
an
application.
A
A
A
We
are
looking
here
so
we
can
see
that
flagger
has
has
deployed
everything.
You
can
see
that
we've
got
the
the
canaries
and
the
primary
running
in
production.
This
horrible,
yellow
red
line
here
was
because
everything
was
broken,
but
what
you
can
start
to
see
here
is
you
can
start
to
see
now
that
we
are
we're
starting
to
go,
go
back
into
a
healthy,
a
healthy
phase.
A
This
is
starting
to
to
go
and
deploy
into
into
production
you're.
Seeing
this
kind
of
crossover
flagger
is,
is
monitoring
this
and
detecting.
That
everything
is,
is
healthy
and
is
is
slowly
starting
to
increase
the
traffic.
I'm
just
grabbing
the
logs
here,
but
you
know
you
can
see
that
every
sort
of
30
seconds
the
weights
are
getting
increased
from
from
the
traffic
there.
A
This
is
going
to
run
up
to
200
percent
and
eventually
flagger
is
going
to
flip
the
switch
it's
going
to
promote
the
current
canary
into
production,
and
it's
going
to
remove
the
old.
The
old
primaries
basically
switch
them
over
in
the
instance
that
this
wasn't
wasn't
working
correctly,
then
you
know.
Obviously
we
would
we
would
roll
everything
back
and,
and
that
again
would
have
would
happen
as
a
completely
completely
automated
automated
process.
A
A
A
A
A
Flagger
is
monitoring
this
and
you
can
see
by
the
flat
line
there
that
flaggers
like
well.
This
service
is
not
healthy.
It's
it's
exhibiting
errors,
so
it
isn't
increasing
the
canary
steps.
In
fact,
what
now
has
happened
is
flagger
has
completely
rolled
back
that
that
deployment.
So
again
the
end
user
is,
is
completely
unaware
of
that.
A
We,
you
know
we're
going
to
get
it's
going
to
look
a
little
bit
small
and
I
apologize,
but
we
can
see
you
know
the
things
here
like
the
the
error
message
that
we
would
get
back
from
from
flagger
that
we've
got
these
failed
checks
and
we've
got
the
sort
of
that
information
coming
in.
So
we
can
do
the
whole
kind
of
git
ops
thing,
but
it's
it's
just
a
really
sort
of
powerful
way
and
really
really
simple,
because
once
you've
got
this
set
up,
I
don't
need
to
touch
it
again.
A
It's
just
gonna
kind
of
work
like
this
forever
it
gives
me
gives
me
as
a
developer
a
little
bit
more
confidence
like
there's
that
old
thing
where
people
are
like.
Oh,
I
don't
want
to
deploy
on
a
friday
and
and
then
you
get
half
the
people
who
are
like
adamantly.
You
should
never
deploy
on
friday
and
then
there's
half
of
people
who
are
like.
Oh,
you
should
have
all
of
this
like
fantastic
stuff,
and
you
should
never
be
afraid
to
deploy
on
a
friday.
It's
true,
you
should
never
be
afraid
to
deploy
on
a
friday.
A
The
reality
is
we're
human
and
and
the
reality
is
we
we
kind
of
work
hard
during
the
week
and
we
want
to
enjoy
our
weekend.
So
you
know
there's
a
very
good
reason
why
we'd,
rather
not
deploy
on
a
friday,
but
but
the
nice
thing
about
being
able
to
do
this
canary
process
is
that
if
we
deploy
on
a
friday
and
we
break
the
the
application
and
we
break
the
system,
it's
fine
because
we
can
just
we
can
just
roll
it
automatically.
It's
just
going
to
roll
back.
A
We
can
pick
up
the
bugs
on
monday
morning
and
we
can
we
can
find
out
what's
going
on
our
end
users
and
our
customers
are
are
completely
completely
abstracted
from
that.
We've
got
some
source
code
here.
There's
an
example
of
how
this
is
all
set
up
on
gke
and
some
some
terraform
config
for
setting
it
up.
There's
a
lot
of
stuff.
I'm
not
gonna
sort
of
lie
to
you,
but
you
know
that
it's
one
of
those
things
it's
it's!
A
You
take
the
first
hit
and
then
you're
kind
of
good
to
reuse
it.
The
application
which
I
don't.
You
know
what
I
think
I
actually
wrote
some
different
bugs
into
the
app
I'm,
not
even
certain
that
I
made
it
work
I'll,
be
I'll,
be
amazed
if
I,
if
I
did
testing
no,
I
did
I
broke
it.
I've
got
to
to
set
the
the
correct
end
point,
but
it
half
worked,
but
that's
that's
something
little
carry
that!
A
And-
and
I
think
the
key
thing
is
that
a
lot
of
this
wouldn't
be
possible
without
the
ability
that
the
service
mesh
can
kind
of
intercept
your
traffic
and
you've
got
these
great
integrations
with
your
see,
icd,
workflows
and
and
kind
of
the
whole
sort
of
modern
approach
around
taking
a
sort
of
git
flow
and
things
genuinely
it's
the
sort
of
stuff
that
I
I
could
only
have
dreamed
of
10
10
years
ago.
But
thank
you
thank
you.
A
So
this
is
really.
This
is
really
difficult
and-
and
I
think
the
answer
is
you
you
can't,
but
but
I
think
what
you've
got
to
do
is
you've
got
to
to
look
at
the
way
that
you
you,
your
db
structure
changes.
So,
for
example,
if
you
need
to
change
a
field
from
let's
say
a
a
string
to
an
integer
or
something
that
would
cause
like
a
a
completely
breaking
change
that
there's
a
sort
of
a
question
of
whether
you
can
do
things
in
two
steps.
A
A
It's
okay
to
add,
add
fields:
it's
not
okay,
to
take
stuff
away,
because
what
you
have
to
have
is
you've
got
to
have
a
situation
where
both
versions
of
your
application,
the
the
kind
of
the
primary
and
the
canary,
can
both
safely
read
from
from
the
data
store,
and
they
can
they
can
both
do
so.
You
know
exactly
the
same
time
when
they're
running
in
production.
A
So
it's
it's
very,
very
possible,
but
you've
got
to
get
a
little
bit
creative
about
the
way
that
you
do
your
database,
migrations
and
and
in
some
ways
some
some
folks
might
might
decide
that
they'll
do
a
a
multiple
phase
thing.
You
deprecate
a
field
as
opposed
to
removing
it
and
then,
in
a
later
release
you
you
kind
of
do
the
removal.
So
it's
it's
kind
of
that
step
by
step
version.
A
One
version
two
can
share
the
same
database
version
two
to
version
three
can
share
the
same
database
version
three
and
version
one
can't,
and
if
you
kind
of
follow
those
steps,
then,
even
if
your
canary
deployment
fails,
your
database
migration,
which
has
been
already
applied,
can
still
work
with
the
version.
One
of
the
of
the
application,
unless,
of
course
it's
your
database
migration,
which
is
the
root
of
your
problem
and
then
in
which
case.