►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Performing Infrastructure Migrations at Scale - Melanie Cebula, Airbnb
Everyone is excited to adopt the latest technology trends, but how do you migrate from one technology to another when you have high availability requirements? How do you separate hype from a technology that actually works at your scale? This talk aims to solve both: 1. the business problems of justifying and resourcing infrastructure migrations and managing the migration lifecycle 2. the technical problems of initial prototyping, identifying and solving blockers and gotchas, giving best practices for rollout, and automating migrations with infrastructure as code. This talk will be backed by our case study migrating hundreds of legacy services with strict latency and uptime requirements to Kubernetes at Airbnb.
https://sched.co/NrmL
A
So
yeah
I'm
Melanie
I'm
here
to
talk
about
performing
infrastructure
migrations
scale,
particularly
like
everybody's
kind
of
scale,
so
like
a
pretty
large
company
and
so
I'm
really
happy
to
be
here
at
keep
Kron
China.
Oh
it's
my
cooker
working,
yes,
okay,
so
I'm
only
I
work
as
a
software
engineer
on
the
service
orchestration
team
at
Airbnb
and
our
goal
is
to
empower
our
own
engineers
to
create
maintain
operate
their
own
services
and
we
use
kubernetes
as
like.
The
orchestration
layer
for
our
services
and
I
want
to
demonstrate
best
practices
for
migrations.
A
Yes,
all
right,
yeah,
there's
like
not
that
many
seats
in
here,
but
I
wanted
to
demonstrate
the
best
practices
for
migrations
based
off
my
first-hand
experience
of
migrations,
Airbnb,
so
I
think
one
of
our
best
migration
is
case.
Studies
discriminatees
itself,
but
also
talked
about
a
few
more
lated
migrations
that
we
worked
on
and
like
I,
demonstrate
the
best
practices
from
there.
So
today
about
70%
of
our
sources
are
in
kubernetes,
mostly
business
facing
services
versus
the
legacy
system,
and
we
have
about
300
critical
services
and
kubernetes
running
in
production.
A
A
Technically
it's
fairly
a
low
effort
because
you're
just
bumping,
you
know
like
you're,
a
bunch
of
version
or
something
like
that,
whereas,
like
a
very
high
effort,
migration
would
be
like
going
from
non
cloud
to
cloud
because
you
have
to
somehow
like
we
write
everything
to
work
in
a
cloud
environment
or
containerization.
So
like
now,
everything
has
to
run
in
containers
and
so
like.
The
difference
here
is
like.
A
Another
dimension,
I
like
to
think
of
for
migrations,
is
whether
it's
virgin
or
discretionary
so
again,
like
a
security
patch,
is
fairly
urgent,
because
you
want
to
be
more
secure
immediately
as
soon
as
the
vulnerability
is
found
and
a
more
discretionary
migration.
It
tends
to
be
things
that
can
take
months
or
years
to
unfold,
and
you
can't
really
have
a
lot
of
time
to
do
them
and
so
like
a
very
urgent
migration.
A
So
there's
reassurance:
you
need,
regardless
of
scale
like
upgrading
your
language
or
framework
version,
and
then
there's
migrations
that
you
don't
really
need
to
do
until
you
hit
a
certain
scale
and
then
once
you
get
that
skill,
you
like,
we
really
need
to
do
that
migration,
and
so,
like
some
examples
of
things
you
need
to
do
at
a
high
scale,
is
like
update
your
storage
layer
or
your
orchestration
or
your
service
smash.
These
are
things
that
like,
when
you
already
have
a
lot
of
load.
A
A
lot
of
traffic
you'll
start
to
realize
that
you
need
these
migrations.
So
these
are
the
three
interventions
I
think
of
it,
so
different
dimensions
and
you'll
kind
of
realize
that
all
of
these
dimensions
are
a
little
bit
related.
So,
for
example,
the
higher
scale,
your
it's
more
common
that
the
migration
becomes
more
urgent
and
also
more
common
that
the
migration
becomes
higher
effort,
so
they're
kind
of
all
multi-dimensional,
so
that's
kind
of
how
I
think
of
migrations
or
what
they
are
different
dimensions.
A
But
why
are
they
important
because
they're
kind
of
a
bit
of
an
effort
to
do
so
when
you're,
a
very
new
company
and
there's
actually
probably
quite
a
few
new
tech
companies
here
in
Shanghai?
You
don't
really
have
to
worry
that
much
about
migration,
but
as
our
company
matures
my
patients
can
become
a
fact
of
life.
So
everybody
is
ten
years
old
now.
So
that's
like
a
lot
of
things
that
I
mean
like
a
decade
of
technology.
A
It's
really
out
of
date,
so
you're
you're
sort
of
replacing
all
of
the
pieces
as
you
go,
and
so
you
find
that
as
your
company
grows,
you
accumulate
a
lot
of
tech
debt.
It
becomes
important
to
reduce
that
tech
debt
to
maintain
your
velocity
and
your
competitive
edge.
So
migrations
are
a
way
to
reduce
this
tech
debt
so
like
the
biggest
tech
debt
for
large
companies
is
low,
developer,
velocity
or
like
low
developer
productivity
developers
are
less
productive
and
less
efficient
because
of
the
amount
of
tech
debt.
But
there
are
more
kinds
of
tech
debt.
A
That's
a
little
bit
more
subtle
so
like
when
you're
scaling
exponentially
it's
like
a
hyper
growth
company.
You
accumulate
tech
that
much
faster,
and
so
you
need
to
keep
up
with
the
tech
debt.
So
here's
a
few
other
examples
of
tech
debt
that
you
will
run
into
I'm.
Just
like
a
few,
you
know
scaling
limits.
Networking
issues,
systems
that
have
now
been
end-of-life
excetera,
see
how
migrations
are,
but
where
they're
important
they're,
basically
the
sole
lever
to
systematically
create
technical
leverage
at
scale.
A
So
now
that
we
know
what
migrations
are
and
why
they're
important
I
want
to
kind
of
spend
most
of
the
time
talking
about
how
do
you
do
migrations
well
and
like
what
are
the
strategies?
You
need
to
succeed
at
doing
migrations
and
also
think
like
I'd,
like
some
examples
of
things
that
don't
go
that
well.
A
Here's
an
example
for
us,
like
we
do
all
these
kinds
of
migrations
components
like
upgrading
systems,
upgrading
Oh
operating
systems,
languages,
deprecating
things
yeah,
as
I
said,
replacing
like
CI
CD
system
or
like
new
service
mesh.
Networking
is
kind
of
a
system,
level,
migration
and
then
infrastructure
for
a
lot
of
companies
is
like
moving
to
the
cloud
or
containerization
or
capes
or
like
serverless.
Any
of
those
kinds
of
things
is
really
like
a
big
infrastructure
change.
A
And
so
the
strategies
here
is
to
know
which
type
you're
working
with
and
like
that's
really
the
first
step
and
and
also
know
that
for
each
you
know,
as
you
go
up
in
this
hierarchy,
you
have
exponentially
increasing
complexity
for
each
type,
like
it's
really
more
complex
to
do.
In
a
structure
we
write
than
a
system
move
and
it's
more
complex
than
a
component
update,
and
so
that
you
should
also
know
that
for
each
of
these
you're
dealing
with
a
lot
more
complexity,
a
lot
more
overhead
and
therefore
it's
a
lot
riskier.
A
So,
for
me,
one
of
the
most
important
migration
strategies,
other
than
just
identifying
the
type
of
the
migration
is.
How
are
you
going
to
sequence,
this
migration
and
so
I'm
gonna
spend
a
lot
of
time
talking
about
this,
because
I
think
what
is
really
complex
about
migrations
at
scale
is
that
they're
all
interacting
with
each
other
and
so
they're,
not
in
isolation,
they're,
basically
all
affecting
each
other.
A
You
know
in
kubernetes
it's
like
what
are
all
the
things
I
need
to
do
to
get
there,
and
if
you
don't
really
like
plan
it
and
think
about
it,
you'll
have
a
lot
of
dependent
things.
You
need
to
do
first
and
so
like
one
way
to
think
about.
It
is
if
you're,
if
you're,
very
good
and
careful
about
planning,
you
don't
have
to
do
a
rewrite,
but
most
people
like
if
they'd
wait
too
long.
A
So
you
don't
really
want
this
to
happen
and
I
was
also
kind
of
what
happened
at
Airbnb.
So
like
we
were
running
a
very
old
configuration
management
system.
You
know
at
this
point
it's
probably
five
to
nine
years
old.
So
in
order
to
get
to
kubernetes,
we
really
had
to
make
a
lot
of
changes
very
quickly,
so
we
were
already
in
cloud,
so
it
wasn't
as
big
of
a
jump,
but
you
can
imagine
that
like
if
you
want
to
go
to
kubernetes
and
you
don't
you're,
not
containerized,
yet
you're,
not
in
the
cloud.
A
A
Basically,
if
you're,
inefficient
and
migrations
you'll
likely
be
performing
a
lot
of
them
at
the
same
time,
so
inefficient
migrations
can
cause
your
overall
velocity
to
slow
down
because
you're
doing
all
of
them
at
the
same
time,
and
they
don't
necessarily
depend
on
each
other.
So,
unlike
cascading
migrations
were
like
you
have
to
do
this
one
and
this
one
to
get
to
the
ideal
migration
state
of
all
these
migrations.
Don't
necessarily
have
to
do
each
other,
but
they
still
kind
of
add
to
the
overall
complexity
of
the
system
and
then
juice
introduces
additional
risks.
A
So
here's
one
example
basically
moving,
and
this
is
like
a
real
everyb
example,
except
we
do
way
more
migrations
at
the
same
time,
but
here's
three
we're
doing
at
the
same
time
right
so
I'm
working
on
our
kubernetes
migration.
Similarly,
our
CI
team
is
trying
to
replace
our
CI
system
because
the
old
one
doesn't
scale
anymore
and
our
CD
team
is
trying
to
introduce
deployment
pipelines
so
that
developers
can
like
have
a
canary
environment
and
tester
changes
better.
A
So
the
first
thing
you
want
to
ask
is:
are
these
migrations
really
independent,
like
how
you
like
what
CI
system
you
use
and
what
CD
system
you
use
actually
kind
of
depends
on
kubernetes
so,
for
example
like
if
you
want
to
use
spinnaker
or
something
like
it
like
that,
then
that's
like
having
any
specific
to
put
a
CD
system
and
see
I.
Also,
you
probably
want
something.
That's
containerized.
So
actually
these
migrations
are
somewhat
related
and
is
each
migration
making
assumptions
about
your
system.
A
So
if
you
start
the
kate's
migration,
you
know
at
time
a
and
then
start
your
deployment
pipelines
migration
slightly
after
are
you
assuming
that
you're
still
in
that?
Previous
state,
like
will
it
work
once
the
other
migration
is
done,
and
so,
if
you
have
really
slow
and
fast
migrations
all
happening
at
the
same
time,
you
may
realize
that
you
didn't
account
for
the
finished
state
of
another
migration,
and
so
does
your
migration
actually
support
a
mixed
state
from
another
migration.
A
A
So
what's
the
strategy
here,
those
are
kind
of
things
that
can
kind
of
go
wrong
so
like
for
me,
I
call
it
sequencing
migrations,
like
let's
plan
and
think
about
how
we're
going
to
do
this,
and
so
you
want
to
avoid
the
infrastructure
rewrite.
So,
let's,
let's
do
a
lot
of
minor
migrations
early
on,
so
we
want
to
make
migrations
themselves
happen,
a
lot
of
the
times
for
frequent
and
very
efficient
so
that
we
can
kind
of
like
production
line.
A
Do
them
really
quickly
and
then
finally
like
when
you're,
when
you're
doing
the
migration
and
try
to
keep
it
not
too
open
end
it
so
like.
Let's
talk
about
very
specifically
what
we're
going
to
build
and
and
not
like
sort
of
like
you
know
so
very
large,
like
let's
move
to
the
cloud
like,
can
we
break
it
down
into
small,
sequential
steps.
A
So
yes,
strategies
lower
risk
migrations
and
it
requires
more
planning
in
time
to
get
to
the
states,
and
so
one
other
thing
you
can
do
is
paralyze
migrations
that
don't
have
dependencies
kind
of
similar
to
a
topological
sort.
So
one
example
I
have
with
our
communities.
Migration
is
like.
We
have
several
things
we
need
to
do.
First,
one
is:
we
were
running
a
really
old
version
of
Java.
This
version
of
Java
was
not
container
aware,
so
you
know
if
you
try
to
run
containers
or
kubernetes
using
this
old
Java
version.
A
A
So
yeah,
then
the
continuation
effort,
and
one
other
thing
I
want
to
talk
about
here-
is
prioritizing
migrations.
So
let's
say
you
have
a
bunch
of
migrations
that
you
want
to
complete.
What
can
you
do
to
decide
which
ones
to
do
first,
and
one
way
I
like
to
think
about
this?
Is
you
should
part
I's
migrations
that
make
the
other
migrations
easier?
So
one
example
is
we
migrated
to
infrastructure
as
code
first,
so
we
had
a
lot
of
infrastructure
settings
that
were
not
in
code
like
things
like
just
like
a
basic
Amazon
web
console.
A
So,
like
you
didn't
you
had
no
way
of
really
storing
that
information
in
a
way
that
developers
could
edit
it
so
like
emphasis,
codes
could
be.
Kubernetes
could
be
terraform
puppet
chef.
Anything
like
this,
it's
better
than
not
storing
it
as
code,
and
once
you
do
that
you
can.
Actually
your
migrations
could
be
code
refactor.
Once
you
have
Cobie
factors,
you
can
actually
build
tooling
to
automate
it.
So
that's
really
useful
if
you
decide
to
migrate
a
bunch
of
infrastructure
before
doing
this
one.
A
A
Cool,
so
that's
some
basic
migration
strategies.
One
other
thing
I
want
to
talk
about
is
like
what
makes
migrating
at
scale
so
especially
difficult
like
even
if
I
know
to
do
all
these
things.
Why
do
we
still
have
some
trouble
so
I
kind
of
want
to
come
back
to
this
graph
so,
like,
let's
say,
you're
thinking
about
replacing
your
service
mesh
and
you
sort
of
like
forecast
that
you
probably
have
you
know
a
few
months
or
a
year
to
replace
it.
So
you
kind
of
want
to
do
it
right.
A
You
kind
of
want
to
like
blend
out
and
slowly
start
migrating,
your
edges
and
then
kind
of
sort
of.
Suddenly
you
realize
that
you
have
like
a
lot
of
failures
that
are
unacceptable.
That
suddenly
start
happening,
and
now
like
this
migration
is
very
urgent,
whereas
before
it
was
not
urgent,
and
so
this
is
kind
of
why
migration
at
scale
is
tough,
which
is
like,
because
you
have
increased
scale
and
you're
experiencing
peak
traffic
and
peak
load.
A
Your
systems
may
suddenly
like
hit
their
limits
and
not
scale
anymore,
and
that
kind
of
causes
it
to
be
urgent
so
like
because
it
wasn't
urgent
before,
but
now
that
you're
at
the
scale.
It
is
urgent
and
like
what
I'd
like
to
call
that
is
sort
of
forced
urgency.
Where
is
if
you
are
able
to
properly
forecast
or
plan
for
this
then
and
like
and
finish
the
migration
before
it
becomes
urgent,
then
you
kind
of
have
all
this
time.
A
When
you're
like
when
you're
working
for
a
company,
you
know
that's
out
of
scale
that
not
that
many
other
companies
are
at
where
you
sort
of
have
to
do
a
bit
more
due
diligence
on
like
planning
for
these
events
and
then
finally,
the
other
thing
that
makes
it
difficult
when
you're
at
scale
is
just
like
the
sheer
effort
involved
so
like
for
us,
the
COO
Benes
migration
has
been.
You
know
at
least
a
one
to
two
year
effort
and
we're
not
done
yet
like
right
over
at
70%.
A
So
why
does
it
take
so
long
to
migrate?
And
the
part
of
that
is
because
of
just
the
surface
area.
So
imagine
if
you
only
have
one
service
or
like
one
database
or
whatever
it
is
that
you
need
to
migrate.
So
you
do
spend
some
amount
of
time
making
it
possible
to
do
the
migration,
and
then
you
do
that
flip
switch
and
then
you've
migrated.
A
So,
but
for
us,
it's
like
well
what
if
you
have
ten
services
or
a
hundred
services
or
a
thousand
services?
And
then
you
really
start
getting
into
like
you
can't
just
manually,
be
switching
these
things
over,
because
it's
just
going
to
take
too
long,
and
it's
just
there's
more
effort
in
migrating
and
then
the
other
thing
is
it's
sort
of
like
when
you
have
that
many
cases
you're
more
likely
to
have
weird
services
or
special
services
and,
like
you
start
getting
into
sort
of
edge
cases.
A
A
A
You
basically
have
way
more
services.
All
of
those
services
talk
to
each
other.
Mc
have
this
very
complex,
like
graph
of
services,
all
like
all
these
edges,
talking
to
each
other
and
now
like
because
you're
having
problems
with
your
permit,
like
you're,
replacing
your
proxy
because
you're
having
problems
with
your
previous
proxy.
A
But
as
soon
as
you
realize
that
you're
like
have
a
short
term
time
to
do,
it
then
realize
you're
on
fire
fighting
and
now
you
need
to
move
very
quickly
and
so
there's
like
ideally
you're
doing
the
forecasting
in
the
stress
testing.
But
once
once
you
realize
you're
having
problems,
you
might
need
to
have
a
very
aggressive
timeline.
A
A
So
we
basically
wanted
to
move
source
configuration
to
each
service.
So
this
is
in
my
bunk
service
and
since
we
have
exponentially
more
services
being
created,
we
actually
created
a
generator
that
generates
the
new
services
using
the
approach,
and
so
when
we
had
new
service
is
being
created.
They're
already
using
this
thing,
and
then
now
all
we
have
to
do
is
we
can
kind
of
like
like
take
a
sigh
and
just
migrate,
all
of
the
old
things.
A
A
One
other
thing
I
want
talking
about
was
migration
overhead.
So,
basically,
when
you
do
a
migration
I
talked
about
how
it
reduces
tech
debt,
how
it
creates
technical
leverage,
but
while
you're
doing
the
migration
itself,
it
actually
introduces
complexity
to
your
system,
because
you're
dealing
with
a
mixed
state,
so
migrating
is
actually
an
explicit
trade-off
between
taking
on
overhead
now
to
reduce
the
worst
overhead
later.
A
But
it's
like
it's
a
lot
so,
like
you
know,
imagine
if
you're
trying
to
use
all
of
these
features
and
so
you're,
basically
trying
to
for
us
we're
trying
to
patch
how
we
use
H
epoxy,
while
building
out
the
new
thing.
So
there's
a
lot
of
overhead
and
maintain
the
next
state,
and
so
this
overhead
kind
of
looks
like
this,
where
you
want
to
reduce
the
tech
debt
via
migration.
A
So
developers
want
this,
they
basically
wants
to
spend
some
time
reducing
tech,
debt
and
the
rest
of
the
time
actually
working
on
what
they
work
on
and
what
developers
actually
get.
If
you're,
not
careful,
is
making
progress
on
their
thing
and
then
us
saying:
oh,
can
you
migrate
all
these
things
and
like
actually
at
Airbnb?
A
That's
like
why
they're
hitting
it.
So
you
don't
want
your
infrastructure
to
look
like
this.
So
basically
for
me,
the
biggest
thing
here
is
unfinished
migrations
actually
make
teched-out
worse.
So
you
really
need
to
finish.
Each
migration
and
I
have
a
few
strategies
to
end
on
with
that
which
is
basically
how
we
make
it.
So
migrations
are
finished,
so
developing
abstractions
over
the
intra
infrastructure
is
very
important,
so
you
want
to
make
the
current
migration
easier,
but
you
also
want
to
avoid
leaky
abstractions
so
that
the
future
migrations
are
also
easier.
A
So,
like
one
example,
I
have
here
is
like
the
kubernetes
files.
So
we
have
all
these
kubernetes
configuration
files
and
we
built
an
internal
abstraction
over
them,
because
we
know
that
in
the
future
you
know
five
years
from
now
we
won't
be
using
kubernetes
will
be
using
the
next
technology.
So
how
can
we
make
it
a
little
bit
better
now?
So
these
are
all
concepts
on
the
left
that
Airbnb
engineers
are
mostly
familiar
with,
and
so
we're
telling
them
like.
A
You
know
we're
trying
to
make
them
not
think
too
much
about
whether
they're,
using
kubernetes
or
not,
and
so
this
is
our
abstraction,
but
one
thing
I
realize
is
like
actually,
this
abstraction
still
leaks,
several
things
like
the
containers
and
like
some
some
of
the
kubernetes
settings.
Actually
we
didn't
fully
abstract
them
away
so
like
what's
the
better
abstraction,
and
so
this
is
actually
what
we're
working
on
now,
which
is
like
what?
A
If
we
have
like
a
manifest
that
just
says
you
know,
this
is
my
service,
it
uses
a
lot
of
resources
or
a
few
resources.
You
know
it's
a
web
service
or
a
cron
job
or
like
a
workload
and
like
these
are
the
services
it
calls.
These
are
the
other
services
it
calls
and
like
can
we
just
limit
it
down
to
that,
so
that
like
when
we
might
get
to
the
next
thing,
there's
actually
no
effort
at
all
on
the
service
developer
side
and
like
actually,
we
migrate
the
whole
thing
in
service
developers.
A
Don't
know
that
they're
using
kubernetes-
and
they
don't
know
that
they're
using
the
next
thing,
so
we're
kind
of
talking
about
building
a
like
a
whole
platform
on
top
of
this,
and
so
this
is.
This
is
like
something
I
kind
of
want
to
build
internally,
and
we
also
want
to
do
this
for
our
for
our
service
discovery
layer
so
like
as
we
migrate
to
envoy
and
as
we
migrate
to
this
thing
after
envoy,
we
want
to
make
it
as
easy
as
possible,
and
so
this
is.
A
This
is
a
strategy
for
from
me
for
Airbnb,
but
I
think.
Actually
this
is
a
strategy
where
the
old,
the
whole
industry
is
going.
So
today,
when
you're
configuring
kubernetes,
you
probably
get
frustrated
by
just
the
amount
of
configuration
that
you're
doing
so
like
I
think
in
the
future.
This
will
become
a
lot
easier,
but
we're
still
working
on
building
platform,
and
so
that's
sort
of
like
how
can
we
make
migrations
that
are
as
like,
a
kubernetes?
You
know
community,
and
so
that's
something
I've
been
thinking
about
a
lot
and
so
here's
some
other
strategies.
A
Basically,
when
you're
thinking
about
migrating,
can
you
standardize
on
something
that
90%
it's
like
90%
correct
for
most
cases,
and
can
we
automatically
migrate
that
case
and
so
then
like?
Can
we
migrate
under
an
abstraction
layer,
so
developers
aren't
aware
that
we're
migrating
and
then
finally,
can
we
migrate
programmatically?
So
can
we
do
code?
We
factors
to
migrate
people,
so
we
have
something
called
a
factor
rater
to
perform
three
factors:
it's
basically
because
we
have
infrastructure
as
code
now,
we've
unlocked
it
automated
automatically
factors,
and
basically
this
process
is
a
bunch
of
scripts.
A
That
kind
of
cover
the
whole
refactor
itself.
So,
if
you
think
of
like
how
every
factor
should
look,
basically,
we
have
kubernetes
cron
job
that
runs
and
it
can
look
at
a
look
at
a
code
base
and
know
what
stated
is
in
and
what
state
we
want
it
to
get
into
like
run
the
V
factor.
So
it
has
this
logic
where
it
basically
can
check
out
the
repo
find
the
project
in
the
repo
run,
the
V
factor,
job
that
gets
it
to
state
B
and
then
tags
the
owners
of
the
service
and
creates
the
PR.
A
So
no
developer.
Does
this
the
developer
writes
the
refactor
job,
so
it
has,
you
need
to
be
able,
to
add
input,
say,
go
from
state
a
to
state
B,
but
the
rest
of
this
is
automatic,
so
the
refactor
job
will
run
and
create
the
PR,
and
then
the
refactor
job
also
can
like
comment
on
the
PR
telling
hey
like.
Can
you
please
look
at
this
owners?
Can
you
please
edit
or
merge
the
PR
and
then
as
well?
We
have
the
V
factory
job.
A
It
can
just
merge
the
PR,
so
we
actually
have
some
many
factors
with
no
developer
involvement
at
all.
We
just
run
it
create
the
PR,
merge
the
PR
so
like
you
can
imagine
so.
A
lot
of
our
services
use
stonker
files
they
inherit
from
some
base
image.
If
we
realize
the
base
image
needs
to
pick
up
a
security
fix,
so
we
can
run
our
refactor
and,
like
I'm,
immediately
give
everyone
the
new
security
fix
without
them
being
involved.
A
So
that's
really
powerful.
We're
not
asking
the
developers
to
do
it,
we're
just
doing
it
automatically,
and
so
what
can
we
do
automatically
so
that,
like
what
communities
version
you're
using
that
can
be
automated
based
image
upgrades
like
for
security
patches,
we
can
do
that.
Programmatically
security
patches,
changing
the
CI
CD
system
is
a
little
bit
tricky,
but
we've
done
that.
We
did
that
about
like
90
percent
automatically
for
our
CI
immigration
like
two
or
three
weeks
ago.
So
we
basically
ran
a
V
factor
that
created
a
new
created,
the
new
CI
job.
A
And
then,
if
the
new
CI
job
passed,
then
we
would
merge
it
and
if
it
failed,
that
means
that
developers
had
to
kind
of
like
look
into
why
the
new
CI
was
failing.
Some
other
things
also
and
I
wanted
to
end
on
the
migration
program.
So,
basically,
when
you
think
about
your
migration
strategy,
what
are
what
is
your
migration
strategy?
Do
you
make
one
person
do
all
of
it
and
I
think?
A
Maybe
if
you're
a
very
small
company
you
can
make
one
person
do
all
of
it,
and
what's
nice
about
that,
is
it's
a
very
tight
feedback
loop
because
they're
doing
the
whole
thing,
but
what
doesn't
work
is
that
if
you
have
a
huge
company,
you
can't
do
the
whole
thing.
One
other
thing
we
do
is
make
devs
do
all
of
it,
so
we
basically
ask
them:
hey
so
I
asked
developers.
Can
you
migrate?
This
problem
with
this
approach?
Is
that
everyone's
always
asking
developers
to
migrate?
Are
they
actually
going
to
do
the
migration?
A
How
do
we
make
sure
that
we
finish
the
migrations,
because
I
just
talked
about
how
bad
it
is
if
migrations
are
left
unfinished?
So
if
we
want
to
actually
finish
the
migrations,
we
need
a
migration
team
that
owns
the
migration
end-to-end
and
figures
out
how
to
make
it
as
efficient
and
frequent
as
possible.
A
So
in
our
case,
like
we
have
a
kubernetes
migration
team
that
actually
is
running
this
effort
and
you
know
I'm
part
of
that
team.
Oh
sorry,
there's
one
other
thing:
I
wanna
talk
about,
which
is
the
life
cycle.
How
do
you
know
whether
you
actually
want
to
do
a
migration
or
like?
How
do
you
do
it?
So
you
start
with
a
design
document
and
a
prototype,
and
you
stress
test,
but
one
thing
you
should
make
careful
here
is
that
you
need
to
make
sure
that
the
technology
are
using
really
works
for
your
hardest
cases.
A
So,
like
high
load,
high
traffic,
don't
start
with
the
easiest
case
and
assume
that
you
have
validated
your
migration
so
make
sure
that
it
works
with
like
low
throughput
high
latency.
You
know
all
these
requirements
that
you
have
so
really
make
sure
that
it
works.
Don't
don't
just
like
you
have
a
simple
prototype
and
assume
it'll
work
when
you
get
to
really
complex
cases.
A
So
we
out
we
just.
We
have
like
10
years
of
services.
So
it's
a
lot
so
yeah
the
finish
phase
and
I
wanted
to
leave
time
for
the
10
takeaways.
Basically,
you
want
to
identify
a
migration
type.
You
want
to
run
frequent,
efficient
and
tightly
scoped
migrations.
You
want
to
do
migrations,
sequencing
like
prioritization
and
planning
them,
and
you
want
to
actually
stress
tests
in
forecast
migrations
before
they
become
urgent.
A
You
want
to
make
the
new
approach
the
default
so
that
time
is
working
for
you
not
against
you,
and
you
want
to
fully
finish
the
migrations
to
reduce
tech
debt.
You
want
to
develop
abstractions
over
the
infrastructure
and
run
migrations
as
code
be
factors,
and
then,
when
you
actually
think
about
the
migration,
we
need
to
run
a
migration
program
with
a
life
cycle,
and
then
you
need
to
iterate
on
the
migration
until
it's
fully
vetted
enabled
and
finished
and
I'm
out
of
time.