►
From YouTube: Deep Dive: Moving is Hard – Migrations without Impact (or back pain) - Stacy Kirkland & Tom Mitchell
Description
Deep Dive: Moving is Hard – Migrations without Impact (or back pain) - Stacy Kirkland, Liberty Mutual Insurance & Tom Mitchell, Stark & Wayne
The thought of moving from one place to another is daunting. Whether you’re moving to a new city or next door, how and when do you make a snap decision? While working remotely across multiple time zones, Liberty Mutual migrated thousands of applications between Cloud Foundries across twelve foundations and five regions in the public cloud and on-premise. In this session, we will discuss how we approached the effort to migrate a Cloud Foundry platform. We'll share with you the successes and some of the challenges we overcame at a technical, organizational, and personal level and how important it was to have a partner like Stark & Wayne there to support us. Please join us if you’d like to learn how we migrated Cloud Foundry with zero developer effort and minimal impact to the business.
To learn more about Cloud Foundry, The Proven Developer Experience For Kubernetes click here: https://www.cloudfoundry.org/
A
Welcome
today,
we
will
be
discussing
our
migration
of
a
sizable
and
live
cloud
foundry
in
the
public
cloud
over
the
past
year,
how
we
approach
the
migration
and
the
lessons
we
learned
along
the
way.
Hopefully
you
can
relate
to
some
of
the
challenges
we
were
faced
with
and
you're
able
to
leave
the
presentation
with
an
understanding
of
some
of
the
technical
decisions
we
made.
A
Liberty,
mutual
is
the
sixth
largest
property
in
casualty
insurer,
and
our
purpose
is
to
help
people
embrace
today
and
confidently
pursue
tomorrow
by
promising
protection
for
the
unexpected
delivered
with
care.
Liberty,
mutual
has
nearly
45
000
employees
across
the
globe,
5
000
of
which
are
in
technology.
A
Our
cloud
foundry
platform
team
has
eight
engineers
across
two
time
zones.
Some
of
us
are
located
on
the
east
coast,
a
few
are
in
wisconsin
and
one
is
in
indiana.
We
support
about
twelve
thousand
applications
running
on
cloud
foundry,
with
three
thousand
average
daily
deployments
and
about
two
thousand
developers.
B
Starting
wayne
is
a
consultancy
that
specializes
in
cloud
foundry,
kubernetes
cloud
native
and
serverless
technologies.
We've
been
consulting
for
close
to
eight
years
now,
and
we've
worked
with
many
fortune
500
companies
to
better
enable
their
cloud
native
devops
teams.
I
was
the
primary
consultant
with
liberty,
mutual
and
I've
worked
with
them
for
the
past
year
or
so.
A
We
also
manage
about
1500,
rabbitmq
and
redis
service
instances,
and
we
support
12
foundations
across
five
regions,
of
which
seven
are
in
the
public
cloud
and
five
are
on
prem.
We
also
have
a
lab
environment
that
is
not
customer
facing
within
each
one
in
public
cloud
and
one
on
prem.
Our
largest
foundation,
contains
over
50
percent
of
our
workload.
A
A
B
Together,
we
worked
on
getting
the
platform
team
up
to
speed
and
on
the
toolkits
required
to
do
just
that
by
working
with
them
on
each
aspect
of
the
build
out,
allowing
each
team
member
hands-on
keyboard
experience
and
collective
troubleshooting
when
needed.
We
also
wanted
to
work
with
the
team
in
order
to
construct
a
migration
plan
that
fulfilled
the
requirements
for
application,
downtime
and
developer
impact,
with
as
little
complexity
as
possible.
B
Early
discussions
about
the
migration
plan
began
with
what
our
overarching
strategy
to
perform
migration
would
be.
The
least
complicated
strategy
from
the
operator
perspective
would
be
to
simply
build
a
new
cloud
foundry
and
enable
cloud
foundry
users
to
migrate
their
applications
to
the
new
cloud
foundry
on
their
own
time
as
easily
as
possible.
B
B
A
We
felt
conflicted
on
this
decision
in
the
early
discussions
we
had
and
the
team
had
several
discussions
around
the
realities
of
each
the
new
city.
Option
seemed
safest
but
difficult
to
accomplish
within
a
reasonable
amount
of
time
and
the
snap
migration
felt
overwhelming
and
intimidating,
especially
if
we
had
no
room
for
error,
but
it
seemed
more
likely
to
be
completed
within
a
specific
time
frame.
A
We
had
some
additional
decisions
to
make
internally,
like
which
foundations
do
we
want
to
tackle?
First,
do
we
want
to
start
small
where
there's
least
impact
and
try
to
learn
and
understand
all
the
new
concepts,
or
do
we
want
to
start
in
our
largest
foundation,
with
over
50
of
all
cloud
foundry
workload
and
figure
out
all
our
problems
at
scale
right
up
front?
These
are
all
questions
we
tried
to
answer
by
understanding
our
customers
and
their
needs.
A
A
They
likely
support
additional
applications
outside
of
the
cloud
foundry
runtime.
They
may
have
developed
for
and
supported
different
application
teams.
In
the
past
they
may
even
no
longer
support
an
application
that
they
were
the
last
committer
on
and
likely
still
considered
the
subject
matter.
Expert
of
this
customer
likely
does
not
have
time
to
prioritize
and
understand
each
app's
needs.
At
the
same
time.
A
Another
example
of
a
cloud-founded
customer
is
a
developer
that
supports
legacy
applications
and
migration
efforts
to
the
public
cloud.
Their
existing
running
app
applications
in
cloud
foundry
are
not
their
priority
and
they
don't
want
to
have
to
shift
priorities
or
worry
about
their
running
apps,
being
impacted
of
all
the
running
applications
in
cloud
foundry
only
about
62
of
them
are
deployed
frequently.
A
A
Our
priority
on
the
cloud
foundry
platform
team
has
and
always
will
be
to
put
our
customers
first.
This
means
no
impact
to
the
applications
or
the
business
and
because
they're
likely
busy
working
on
application
specific
priorities
at
any
given
time,
we
don't
want
to
ask
them
to
add
work
to
their
plate
to
help
us
migrate.
A
B
B
We
used
a
stock
and
wayne
tool
called
genesis
to
pre-package
and
distribute
pre-tested
configuration
options
for
bosch,
manifests
and
custom
configurations,
but
because
we're
using
a
new
tool,
the
team
needed
to
be
taught
how
to
use
the
tool
effectively,
as
well
as
how
to
add
any
custom
configurations
that
would
be
needed
to
suit
liberty.
Mutual's
specific
needs
for
the
platform
the
build
out
required,
deploying
more
than
just
cloud
foundry
itself,
though,
we
deployed
several
additional
software
stacks
alongside
cloud
foundry
to
aid
in
its
operation
and
augment
its
capabilities.
B
B
One
goal
he
wanted
to
achieve
in
order
to
build
out
a
new
cloud.
Foundry
was
to
deploy
our
cloud
foundry
foundations
to
the
same
subnets
that
the
existing
cloud
foundry
foundations
that
we
were
migrating
from
were
in
with
the
hopes
that,
by
doing
so,
we
could
avoid
firewall
security
group
related
issues
for
app
traffic.
A
And
because
of
some
internal
security
requirements,
we
needed
to
separate
our
new
foundation
components
from
our
existing
foundation.
So
we
accomplished
this
by
creating
a
new
aws
account
to
deploy
all
of
our
new
foundation
components
into
and
since
we're
using
the
same
subnets
as
our
existing
cloud
foundry,
we
paired
the
subnets
across
the
counts.
A
One
of
the
initial
foundations
we
built
out
with
stark
and
wayne
was
a
lab
environment
that
we
could
learn
and
play
in
without
worrying
about
interfering
with
any
of
our
cloud
foundry
users
using
all
the
new
components,
tools
and
learnings.
We
also
needed
to
practice
a
migration
end-to-end
from
our
existing
lab
environment
in
aws
to
our
new
lab
environment
in
aws,
while
practicing
and
learning
in
lab.
We
also
spent
months
configuring
and
validating
the
newly
built
out
cloud
foundries
in
our
customer
facing
foundations.
B
We
made
the
decision
that,
while
we
were
migrating
the
existing
cloud
foundry,
we
would
not
yet
migrate
the
existing
data
services.
We
saw
an
opportunity
to
decouple
the
migration
of
data
services
from
an
already
complex
application.
Migration.
The
data
services
would
remain
managed
by
their
brokers
in
the
existing
foundation
for
weeks
after
the
application
migration.
B
So
we
needed
to
ensure
that
the
newly
migrated
applications
were
still
capable
of
provisioning
new
services
instances,
as
well
as
maintaining
connectivity
to
existing
service
instances
because
we're
staying
in
the
same
network,
no
application
security
group
changes
were
needed,
but
because
we
were
in
a
different
aws
account,
we
had
to
be
sure
to
configure
firewall
rules
to
enable
traffic
infrastructure.
Configurations
aside,
there
were
additional
boston
cloud
battery
configurations
that
needed
to
be
made
in
order
for
the
new
platform
to
be
able
to
locate
service
instances
and
brokers
in
the
source
platform.
B
Therefore,
we
had
to
come
up
with
a
solution
for
continuously
merging
the
bosch
gns
hostname
entries
from
the
existing
bosch
director
to
the
target
foundation's
diego
cell
vms.
To
implement
this.
In
the
original
foundation,
we
deployed
an
nginx
server
that
served
the
records
file
containing
all
the
redis
dns
host
names.
We
then
created
an
application
that
periodically
checks
the
file
served
by
the
nginx
server
for
changes
and
can
merge
the
remote
dns
names
into
a
local
dns
records
file
that
is
managed
by
the
target
bm's
bosch
agent.
B
B
The
routemq
management
api
in
both
the
ravdomq
and
redis
service
brokers
register
their
dns
host
names
with
the
cf
go
routers
using
a
component
called
route
registrar
in
this
configuration
traffic
to
these
components
enters
through
the
cloud,
foundry
go
router
and
the
go
router
in
turn.
Reverse
proxies
this
traffic
to
the
appropriate
component,
based
on
the
subdomain
of
the
host
name
of
the
request,
without
any
additional
work,
the
go
router
of
the
new
cloud
foundry
foundation
would
not
know
where
to
reverse
proxy
the
traffic
to
for
these
service
components.
B
Normally,
a
route
registrar
assumes
that
the
ip
that
it's
mapping
to
the
host
name
it
registers
is
the
ip
of
the
vm
that
the
route
road
star
is
installed
on.
However,
this
can
be
overridden
and
it
had
to
be
because
we
were
installing
these
route
registrars
in
our
new
foundation
to
map
the
components
in
our
old
foundation.
B
Additionally,
any
given
route
or
your
star
route
registrar
can
only
map
routes
to
a
single
ip
and
as
such
we
had
to
deploy
a
new
registrar
for
each
component.
We
wanted
to
map
our
route
to
this
means
we
had
to
deploy
three
new
route
registrars,
one
for
the
rapid
q
broker,
one
for
the
redis
broker
and
one
for
the
rabbitmq
management
api.
A
While
we
built
out
our
new
cloud,
foundry
foundation
and
configured
all
the
missing
links
between
our
old
foundation
and
new
foundation,
we
attempted
to
communicate
and
document
our
efforts
while
practicing
the
migration
we
attempted
to
script
and
document
the
entire
process.
In
order
to
be
repeatable,
we
created
a
runbook
from
this
in
which,
as
we
learned
or
discovered
new
practices
or
configurations,
we
were
to
document
the
steps
that
were
taken
for
us
to
later
reference.
A
We
plan
for
our
migration
window
to
consist
of
four
major
phases,
prevent
any
customer
changes
to
the
foundation
clone
cloud
foundry
state
to
the
new
foundation
verify
that
everything
is
ready
to
go,
live
and
then
actually
flip
the
switch
based
on
our
learnings
from
lab.
We
estimated
that,
if
everything
went
according
to
plan,
then
the
migration
in
its
entirety
should
take
us
about
four
to
six
hours
because
of
the
risks
involved
with
the
migration,
we
were
to
make
any
impactful
changes
to
dns
during
a
coordinated
window
at
2am.
A
This
gave
us
several
hours
in
our
change
window
before
the
impactful
change
time
frame
to
migrate
the
data
and
perform
our
platform
validations
once
the
dns
change
occurred,
we
allotted
two
hours
for
application.
Teams
to
validate
their
own
applications
are
running
as
expected
on
the
new
cloud.
Foundry.
B
The
change
window
needed
to
be
long
enough
for
us
to
take
a
snapshot
of
the
data
and
then
import
it
into
the
new
cloud
foundry
and
then
cut
the
dns
over
changes
to
the
platform
consist
of
app
deployments
creation
of
new
artifacts
and
really
anything
else
you
can
do
to
the
platform
as
a
user.
We
prevented
these
changes
by
stripping
away
users
authorization
to
make
changes
through
removing
space
developer
access
to
the
source
foundation.
B
While
this
is
arguably
the
entire
data
migration.
The
process
in
total
really
only
took
about
15
to
20
minutes
the
time
it
takes
to
start
up
the
target
cloud
foundry
and
the
attention
to
detail
we
put
into
validating
the
target
platform
is
where
the
bulk
of
our
time
was
spent
during
the
migration
window.
A
Once
the
data
was
successfully
migrated,
we
started
cloud
controller
and
diego
backup
and
then
restarted
all
the
diego
cells.
This
took
a
considerable
amount
of
time
because
of
how
many
diego
cells
we
had
once
the
target
foundation
was
started,
we
could
start
validating
the
platform,
the
team
divided
and
conquered
the
various
validations
of
the
platform.
A
A
Cutting
over
also
required
us
to
shorten
dns,
ttls
and
cut
connections
to
the
old
load
balancer,
so
that
all
connections
would
be
forced
to
reconnect
to
the
new
load
balancer
by
having
the
dns
team
help
us
change
every
host
name
to
be
a
senior
record,
pointing
to
a
single
a
record.
Prior
to
the
change
window.
We
were
able
to
perform
the
cutover
independent
of
other
teams
because
of
how
many
urls
we
managed
this
simplified.
A
The
work
needed
to
be
done
during
that
actual
impactful
change
window
at
2
am
which
significantly
reduced
the
risk
in
time
for
our
team
once
dns
is
cut
over
our
new
cloud,
foundry
is
live
and
we
can
begin
to
shut
down
our
old
cloud.
Foundry
is
what
I
would
have
said
had
everything
gone
according
to
plan.
A
However,
during
our
first
attempt
to
migrate,
we
had
a
few
failing
validations
smoke.
Tests
were
not
consistently
passing
after
the
initial
migration,
which
we
knew
meant.
There
was
potentially
a
problem
at
the
platform
level,
and
we
also
noticed
a
large
discrepancy
in
running
apps
or
lrps
between
the
source
and
target
foundations.
We
couldn't
identify
the
specific
apps
that
were
crashing
with
the
monitoring
we
had
in
place,
at
least
within
the
time
our
change
window
had
left.
So
it
was
because
of
these
two
impediments.
A
A
We
were
emotionally
invested
in
completing
the
effort
that
we
had
spent
so
much
time,
configuring
and
coordinating,
but
ultimately
it
was
the
right
decision
to
make
every
time
we
chose
to
roll
back.
We
had
a
go.
No
go
decision
point
identified
prior
to
migration
day,
which
allowed
us
to
do
validations
prior
to
any
of
the
impactful
changes
that
the
dns
cutover
would
introduce.
This
allowed
us
to
roll
back
without
any
impact
to
the
applications
or
the
business.
A
Had
we
not
communicated
the
change
as
widely
as
we
did.
Our
customers
would
have
had
no
idea
anything
even
occurred.
We
actually
received
feedback
that
we
should
have
communicated
the
rolex
better.
Some
app
teams
assumed
we
were
successfully
running
an
open
source.
The
monday
following
our
rollbacks.
A
We
wanted
to
be
as
transparent
as
possible
with
customers
and
stakeholders.
So
the
week
following
each
migration
attempt,
we
posted
a
lessons
learned
post
to
our
internal
blog.
We
actually
received
lots
of
positive
feedback
on
the
decision
to
roll
back
from
customers
and
management,
calling
the
failed
migration
attempts
a
success.
A
A
A
B
B
Liberty,
mutual
has
a
service
that
shuts
down
applications
and
resources,
such
as
databases
in
our
non-profit
environments
off
hours.
In
an
effort
to
reduce
costs,
we
assumed
that
some
of
the
migrated
applications
could
not
start
without
connectivity
to
these
resources
that
were
brought
offline.
So
we
would
later
reach
out
to
the
public
cloud
team
to
disable
the
service
during
future
migration
windows.
A
Now
that
we
have
the
metrics
to
identify
the
crashing
apps,
we
were
well
positioned
to
dig
into
the
application
logs
for
these
failing
apps
and
what
we
saw
across
the
board
were
timeouts
connecting
from
the
target
cloud
foundry
to
aws
resources
such
as
rds
instances
from
here.
We
dug
into
the
application
cfts,
where
these
rds
instances
lived
and
identified
some
of
their
ingress
access
rules
we're
using
a
security
group.
A
A
In
order
to
successfully
migrate
all
cloud
foundry
applications
to
the
target
foundation,
we
decided
to
temporarily
deploy
the
diego
sales
into
the
old
aws
account
prior
to
our
third
attempt.
This
allows
the
external
aws
resources
using
security
group,
ingress
access
rules
to
successfully
be
accessed
by
their
respective
cloud
foundry
applications.
A
Today,
we
are
working
with
application
teams
to
update
their
external
aws
resources,
re
access
rules
to
use
a
new
security
group
that
is
defined
in
both
aws
accounts
we
are
using
in
the
near
future.
We
intend
to
deploy
the
diego
cell
vms
into
the
new
aws
account
with
the
knowledge
that
all
cloud
foundry
application
dependencies
are
using
access
rules
that
will
work.
A
We
had
nothing
but
support
from
application
teams
and
management
throughout
the
weeks
and
months
of
the
migration
attempts,
our
own
team
doing
the
work
truly
supported
each
other
throughout
the
weeks
and
long
weekend,
nights
as
well.
Every
member
of
the
team
played
a
key
role
in
successfully
migrating
cloud
foundry.
A
A
A
We
also
now
have
all
of
our
configurations
as
code,
which
means
all
configs
and
change.
History
is
managed
and
get
it's
easier
to
stand
up
new
cloud
foundries
in
order
to
offer
new
foundations
and
regions
to
our
app
teams
now,
and
we
also
have
access
to
override
individual
platform
components
to
suit
our
requirements
at
any.
Given
time
we
both
kind
of
touched
upon
the
platform,
automation
and
monitoring
that
we
configured
and
learned
prior
to
migrating.
A
Not
only
did
it
help
us
during
the
migration,
but
we
continue
to
benefit
from
it
today,
we're
using
concourse,
which
allows
us
to
easily
and
more
frequently
upgrade
cf
patch
amis
recreate
container
instances
and
rotate
credentials,
amongst
other
things,
and
with
prometheus,
grafana
and
alert
manager.
We
now
have
better
alerts
and
visualization
of
all
our
platform.
Metrics.
B
A
Join
cool
hi.
I
should
start
out
by
saying
overall
it
took
us
about
a
year
to
migrate
all
of
our
foundations
to
open
source
and
immediately
following
that,
we
were
able
to
stand
up
a
new
foundation
for
our
users,
with
quick
turnarounds
as
well
as
start
running
through
the
upgrade
process
and
seeing
how
everything
works.
And
it's
been
a
really
fun
year
with
starkinweed.
A
B
A
B
B
A
Well,
for
instance,
we
were
recently
going
through
a
cf
upgrade
and
we
realized
that
the
version
of
go
router
that
was
with
the
cf
upgrade,
was
causing
a
few
applications
to
have
problems,
so
we
were
able
to
individually
go
in
and
update
the
goal,
router
and
roll
back,
just
the
go
router,
but
keep
cf
and
everything
else
in
place,
as
was
so.
We
roll
back
just
the
go
router
which
we
wouldn't
have
been
able
to
do
had
we
still
been
using
a
vendor.
A
Another
good
question,
oh
yeah,
so
I
would
say,
with
we
treated
each
foundation
as
its
own
migration,
and
I
can
confidently
say
that
if
we
were
to
migrate
one
more
foundation,
we
would
still
learn
something
new.
It
was
a
process,
a
learning
process,
every
single
time
I
feel
like
we
got
better
and
more
confident
with
ourselves.
Each
migration,
but.
B
B
B
B
We
could
migrate
all
of
the
v
host
and
q
configurations
by
essentially
adding
more
nodes
to
the
cluster
on
the
destination
that
we
wanted
to
migrate
to
and
then
cutting
out
one
by
one,
the
old
parts
of
the
cluster,
so
that
effectively
we're
just
like
using
the
quorum
to
just
have
rev
mq
fix
itself.
Every
time
we
made
a
node
until
all
that
was
left
was
in
the
migration
target
and
redis
was
pretty
conceptually
easy
because
redis
is
a
cache
and
for
the
customers
at
liberty,
mutual.