►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Keynote: Switching Horses Midstream: The Challenges of Migrating 150+ Microservices to Kubernetes - Sarah Wells, Technical Director for Operations and Reliability, Financial Times
The FT content platform team put our first containers live in mid-2015 and migrated the rest of our services over by April 2016. At that point, we weren't using Kubernetes - and much of what we were using, we built ourselves. At the end of 2016, we decided we wanted to benefit from the work other people were doing and switch over to Kubernetes. But it's not easy to do that kind of move when you have 150+ microservices and you need to keep the existing platform running in parallel while you do the migration. I'm going to talk about this migration and the challenges we faced.
About Sarah
Sarah Wells has been a developer for 15 years, leading delivery teams across consultancy, financial services and media. Over the last few years she has developed a deep interest in operability, observability and devops, and this has recently led to her taking over responsibility for Operations and Reliability at the Financial Times. Before that, she lead work at the FT on building a semantic publishing platform, making it easy to discover and access all the FT’s published content via APIs in a common and flexible format. That project meant a focus on Go, microservices, containerisation, and how to influence teams to do the right things.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
We
often
give
conference
talks
as
though
we're
talking
about
a
journey
as
though
we
had
a
destination
and
we
set
out-
and
there
were
a
few
problems
along
the
way
and
we'd
got
there.
I,
don't
think,
that's
really
how
it
works,
because
we
might
have
some
idea
of
what
we
where
we
want
to
go.
We've
got
this
picture
of
a
city
with
the
streets
paved
with
gold,
but
we
don't
actually
know
where
it
is.
But,
more
importantly,
we
don't
start
out
from
home.
A
It's
really
rare
to
start
a
project
from
scratch
and
even
when
you
do
start
a
project
from
scratch
very
soon,
you're
carrying
lots
of
baggage.
The
decisions
that
you've
made
along
the
way
when
those
decisions
are
made
for
the
right
reasons,
but
they
can
come
back
and
hurt
you
later
when
you
make
some
other
change.
A
So,
as
I
said,
I
work
for
the
Financial
Times
and
we're
one
of
the
world's
leading
news
organizations,
and
we
don't
describe
ourselves
as
a
newspaper
because
most
people
who
read
the
Financial
Times
do
that
online,
which
means
technology
is
really
critical
to
us
to
our
success
and
a
couple
of
years
ago
we
decided
to
give
teams
a
lot
of
power
to
make
their
own
decisions
about
technologies
they
use,
which
means
we
have
a
very
diverse
stack
and
they
use
very
different
technologies.
But
I'm
going
to
talk
about
one
team
in
particular.
A
Well,
this
is
the
content
platform,
so
the
F
T's
content
platform.
It's
very
simplified
view.
We
take
content
from
multiple
source
content
management
systems.
We
transform
it
into
a
common
format.
We
have
lots
and
lots
of
concepts
that
we
load
into
our
system,
companies,
people
topics
and
we
annotate
that
content.
So
we
can
say
this
article
is
about
Apple,
it
mentions
Tim
Cook,
and
then
we
make
all
of
that
information
available
through
api's.
A
But
this
is
what
it
really
looks
like
I.
Don't
expect
you
to
see
the
detail.
That's
actually
the
point
of
this
slide.
It's
a
really
complicated
architecture
diagram
because
we've
got
a
Mike,
reservist
architecture
and
boy.
Have
we
gone
for
that?
It's
150
mike
services,
it's
around
650
instances
running
in
production
and
in
those
circumstances
migration
is
not
a
small
thing
to
attempt.
So
the
first
question
is:
why
did
we
migrate
at
all?
A
Well,
we
started
running
docker
in
production
in
mid-2015
and
I'd
say
that
at
the
time
that
was
pretty
leading-edge,
I
mean
there
were
things
you
could
use,
but
you
probably
had
to
do
quite
a
lot
of
work
to
glue
them
together
and
I
know
Simon
Woodley
speaking
at
coupon
tomorrow,
his
work
on
value
mapping
is
fantastic.
This
is
just
talking
about
the
maturity
curve
for
technology.
Things
start
off
as
being
something
for
experts,
and
you
do.
A
Custom
builds
for
it,
but
as
it
becomes
more
ubiquitous
people
start
building
products
and
finally
it
becomes
a
commodity,
and
a
great
example
of
this
is
electricity.
We
don't
build
power
stations
now
we
just
plug
something
in
and
then
more
recently
example
would
be
about
computing.
Sure
lots
of
companies
here
did
what
we
have
t
did,
which
is
have
their
own
datacenter.
Then
build
their
own
private
cloud
and
then
move
to
a
cloud
provider,
so
in
2015,
docker
and
associated
technologies
were
definitely
closer
to
the
left
hand.
Side
than
the
right
hand.
A
Side
of
this
Etsy
talk
about
innovation,
tokens,
the
concept
that
there's
a
limit
of
the
amount
of
innovation.
You
can
do
at
any
one
time.
I
mean
it's
not
really
a
limit,
but
if
you
want
to
be
successful,
you
should
probably
try
and
limit
it.
So
you
have
to
imagine
you
have
a
couple
of
tokens.
What
are
you
going
to
spend
them
on?
You
want
to
use
a
new
database
that
no-one's
used
before
you're
spending
a
token
I
think
building
your
own
container
cluster
orchestration.
It's
definitely
spending
an
innovation
token.
A
So
why
did
we
do
that?
Well,
because
the
benefits
outweigh
the
risks.
Before
we
did
this,
we
were
running
every
micro
service
in
its
own
VM
and
that's
pretty
inefficient,
because
they
you,
don't
you
don't
use
up
the
CPU
or
the
memory
of
even
the
smallest
VM.
We
ended
up
running
our
entire
stack
on
a
extremely
large
VMs
per
region,
as
opposed
to
300
plus
small
VMs.
A
It's
about
an
80%
cost
reduction,
but
it's
not
just
the
reduction
in
costs,
because
before
we
did,
this
migration
we'd
have
to
set
up
those
VMs,
and
we
have
two
instances
in
each
region.
In
production,
two
regions
staging
as
well
development
you'd
basically
go
through
the
process
of
provisioning,
a
VM
1216
times
and
even
if
that's
a
very
short
amount
of
time,
it
adds
up
and
now
after
we've
done
this,
we
just
created
a
single
service
file
and
listed
that
service
in
a
yeah
mall
directory
effectively.
A
But
the
gains
you
get
can
be
offset
by
other
things.
If
you
build
it
yourself,
you've
nowhere
else
to
go
for
help
documentation,
particularly
in
terms
of
decisions
that
were
made,
and
why
is
very
rarely
a
priority,
and
in
our
case,
most
of
the
containerization
work
was
done
by
a
core
group
of
people
pretty
much,
all
of
whom
left
the
FT
in
a
couple
of
months
in
2016
and
when
that
happens,
you're
supporting
stuff
that
you
don't
really
understand,
and
you
can't
ask
anyone.
A
So
the
concept
of
innovation
tokens
is
actually
mentioned
in
this
blog
post
from
dama
Kinley.
As
the
flip
side
of
using
boring
technology
and
boring,
doesn't
mean
bad
boring
technologies
are
things
that
lots
of
people
are
using
successfully
letting
them
get
on
with
focusing
on
the
stuff,
that's
a
differentiator
for
them
as
a
company
new
and
innovative
technology.
Although
it's
exciting
has
capabilities
that
you
probably
don't
understand,
and
in
particular
the
failure
modes
are
not
likely
to
be
well
understood.
You
really
want
boring
in
software
that
you're
operating
so
by
late
2016.
A
We
could
see
that
tools
were
maturing
so
2015.
We
didn't
really
think
we
had
a
boring
alternative
to
choose,
but
by
2016
we
could
read
stories
about
people
successfully
doing
things
in
production
and
we
felt
those
things
were
moving
up
through
this
maturity
curve.
It's
important
to
take
advantage
when
things
move
up
of
this.
You
don't
want
to
be
maintaining
something
yourself
once
it's
available
as
a
product.
Unless
it's
something
that's
absolutely
core
to
your
business.
The
FT
is
not
a
cluster
orchestration
company.
A
We
are
a
news
organization
and
that's
where
we
should
focus
our
innovation.
So
in
late
2016
we
had
a
workshop
and
we
started
thinking.
Well,
what
could
we
use?
What
should
we
move
to,
and
the
first
thing
we
did
was
decide
some
metrics
for
the
success
we
spent
quite
a
lot
of
time,
keeping
our
cluster
healthy.
We
had
a
lot
of
resilience,
so
actually
it
didn't
affect
users
of
our
cluster,
but
it's
painful
for
the
people
that
are
doing
that,
and
actually
a
very
good
metric
for
developer.
A
Satisfaction
is
how
many
sarcastic
comments
they're
making
in
slack,
because
they
won't
necessarily
complain
to
you
all
the
time,
but
you'll
see
it.
We
opted
for
Cuban
Nettie's
B,
two
reasons.
First
of
all,
we
preferred
it's
the
other
things
that
we
assess,
but
secondly
we
thought
this
is
looking
like
it's
going
to
be
the
emerging
standard
and
we
kind
of
had
a
big
appetite
for
being
able
to
ask
other
people
what
they
were
doing
and
then
in
February
of
2017
I
saw
Liz's
poll
and
I
thought.
Yeah
I
think
we're
making
the
right
decision
here.
A
I
think
we
actually
are
pretty
comfortable
with
having
been
at
doing
taking
advantage
of
some
leading-edge
technologies,
because
you
can
very
much
benefit
from
it.
So
when
we
went
to
microservices,
we
moved
from
doing
12
releases
a
year
to
doing
2200
and
they
were
much
lower
failure
rate
as
well,
but
you
had
to
be
prepared
to
stop
and
say.
Is
it
time
to
give
up
on
this?
A
We
tried
something
innovative,
it
doesn't
work
or
is
it
time
to
move
to
the
product
that
is
now
available
and
I
think
you
should
expect
that,
and
you
should
build
that
in
to
your
plans
for
your
work,
but
I
don't
think
you
should
be
too
scared
about
making
the
wrong
decision
if
you're
picking
up
something
new
and
Jeff
Beezus
says
that
there
are
two
types
of
decision
type.
One
is
not
reversible
and
you
have
to
be
very
careful
when
you
make
it
and
for
the
Brits
in
the
audience.
A
Like
to
is
like
walking
through
a
door,
you
can
always
go
back
and
bezel
says
people
often
use
the
heavy
weight
decision-making
process
for
type
1
for
all
kinds
of
decisions.
You
can
actually
just
decide
to
do
it
and
then
change
your
mind,
but
even
having
good
reasons
to
to
do
this
migration
and
feeling
quite
happy
that
we
should
do
it
still
a
major
challenge,
you
are
trying
to
change
horses
in
the
middle
of
a
roaring
river.
You
have
a
lot
of
other
stuff
going
on.
A
At
the
start
of
this
migration,
we
had
150
services
running,
live
in
production
and
under
active
developments
when
I
say
active
development.
This
is
an
attempt
for
us
to
work
out
the
dependencies
between
different
teams
trying
to
do
work
on
our
stack
in
2017,
so
the
migration
team
had
to
try
and
do
the
migration
without
affecting
all
of
that
work.
A
If
you're
building
something
from
scratch,
you
can
quite
easily
build
up
a
little
trial
platform
and
put
a
few
services
in
it,
but
the
things
that
we
really
wanted
to
understand
whether
kubernetes
worked
for
us.
We
needed
a
lot
of
our
services
to
be
running
those
things
around
replacing
our
hosts
based
routing,
and
that
meant
we
had
to
run
in
parallel
and
we
ran
in
parallel.
Quite
a
long
time
did
about
2000
code
releases
while
running
at
least
part
of
the
stack
in
parallel.
A
If
it
takes
an
extra
10
minutes
to
do
the
deployments
to
both
of
those
stacks,
that's
actually
47
working
days
to
do
all
those
releases.
So
you
know
you
pay
a
cost
when
you're
running
stuff
in
parallel
as
soon
as
you
have
a
lot
of
services,
and
then
we
had
to
make
some
decisions
about
how
we
were
going
to
run
in
parallel.
A
We
started
off
by
thinking
about
how
are
we
going
to
commit
changes
and
in
our
proof
of
concept,
we
had
a
cube
Letty's
changes
on
a
separate
branch
and
I'm
sure
anyone
that's
ever
been
stuck
in
merge
hell
will
realize
that
that's
a
bad
idea.
We
very
quickly
decided
that
that
was
just
going
to
diverge
too
much,
and
it
would
be
too
painful
to
try
and
maintain
it.
A
A
If
you
have
completely
separate
deployment
mechanisms
for
two
stacks,
you
can
absolutely
guarantee
things
will
get
out
of
sync,
because
someone
will
forget
to
do
the
deployment
for
one
of
the
two
and
if
you've
got
150
services,
it's
actually
quite
hard
to
scan
and
see
that
you're
out
of
sync
but
running
a
single
pipeline
is
risky
because
you
don't
want
to
block
and
emergently
emergency
deployment
to
your
live
stack,
because
someone
got
something
wrong
on
your
new
experimental
stack.
So
in
the
end
we
kind
of
compromised
both
deployments
were
kicked
off
by
tagging.
A
A
release
in
github
deployments
to
lower
environments
are
done
automatically
and
we
had
a
push
button.
Okay,
go
policy
for
putting
it
to
the
kubernetes
stack,
and
then
we
wrote
a
service
that
would
basically
compare
the
versions
and
report
on
the
differences,
and
we
just
ran
that
manually
every
couple
of
days
and
then
said:
look
you
forgot
to
deploy.
This
testing
is
another
area
where
you
can
get
really
sucked
in
you
don't
want
to
have
to
test
everything
in
two
places,
so
we
focused
on
the
live
stack.
A
We
did
very
little
testing
as
we
went
on
the
fibonacci
stack,
but
we
had
some
things
that
really
helped
us
here
that
we'd
built
for
our
staging
environment.
We
just
used
them
now
for
the
new
stack.
First
of
all,
we
had
every
event
about
publishing
content
that
goes
through
our
production
stack.
We
can
also
send
it
off
to
any
other
place
we
want
so
every
published
went
to
both,
and
that
meant
that,
if
it
failed,
we
would
know
immediately
because
we
have
monitoring
for
every
publish
to
say,
did
it
make
it
through
the
whole
stack.
A
And
secondly,
we
have
a
load
replicator
that
replays
API
requests
and
we
could
play
that
against
the
new
stack.
So
we
knew
if
we
had
a
problem
there
with
something
not
quite
working
and
then
at
the
end
we
did
a
big
enter
end-to-end
testing
effort
so
running
in
parallel.
It
is
costly.
Second
thing
I
wanted
to
talk
about
was
just
the
fact
that
you
have
a
hundred
and
fifty
services
doing
anything
a
hundred
and
fifty
times
it
takes
time.
A
A
They
were
all
very
similar,
I
didn't
take
very
long
and
we
had
a
small
amount
of
hook
in
code
to
be
able
to
deploy
to
a
templated
Jenkins
pipeline
and
to
avoid
having
a
single
team
working
for
days
and
days
and
days
on
the
same
repetitive
tasks.
We
got
everyone,
everyone
from
five
development
teams
involved
in
this
and
we
had
helm
day.
A
So
every
single
person
basically
migrated
some
services,
and
this
was
great
because
you
want
everyone
on
your
team
to
understand
the
technologies
you're
moving
to
and
have
a
chance
to
give
you
feedback
about
them.
The
problem
was,
it
wasn't
a
helm
day.
It
was
helm
days
because
we
discovered
a
lot
of
broken
things,
several
reasons
for
that.
A
You
always
have
bits
of
your
codebase
that
change
a
lot
and
other
bits
that
don't,
but
in
micro
services
that
ends
up
being
some
service
that
hasn't
been
built
or
deployed
for
ages
because
it
has
just
worked
and
then,
when
you
do
need
to
make
a
change
to
it.
You
have
to
build
something:
that's
not
been
built
for
two
years,
and
that
is
particularly
bad.
If
you
wrote
the
servicing
go,
didn't
have
vendor
and
so
you're
picking
in
two
years
worth
of
changes
that
have
happened
in
any
libraries
that
you
depend
on.
A
We
now
have
vendor
in
all
our
go
outs,
but
we
had
to
do
that
as
part
of
the
work,
so
we've
improved
our
code
base,
but
it
was
painful.
One
of
the
other
teams
in
the
FT
actually
builds
every
service.
Every
night
they
don't
deploy
it,
but
they
build
it
and
I
think
that's
probably
a
wise
idea,
because
if
you
suddenly
have
to
do
something
like
this
for
security
reasons,
you
really
don't
want
to
have
to
spend
ages
fixing
everything
up.
A
We
are
lucky
enough
at
the
FT
to
have
a
very
widely
complied
with
health
check
standard.
So
almost
every
service
has
an
underscore
underscore
health
endpoint.
That
will
say:
I
am
healthy
in
a
in
a
well-known
JSON
format,
and
we
use
that
to
have
an
aggregation
service
that
can
tell
us
the
health
of
the
entire
cluster.
A
We
wrote
this
when
we
move
to
kubernetes
and
we
rewrote
it
taking
into
account
all
aspects
of
the
health
check
standard,
but
it
turns
out
that
we've
never
previously
cared
whether
the
health
check
timed
out
it
within
five
seconds,
and
we
discovered
that
as
a
result,
quite
a
lot
of
our
services
didn't
do
that,
and
that
meant
that
we'd
have
things
were
Porter's
being
down
when
in
fact
they
were
just
slightly
slow
in
telling
you
whether
they
were
healthy.
So
we
had
to
change
that
and
that's
another
improvement,
but
it
takes
time.
A
Similarly,
we
have
a
good
to
go
endpoint
and
we
thought
that's
great.
We
can
use
that
for
the
readiness
probe
in
the
previous
tank.
We
don't
use
it
at
all,
which
means
we
found
quite
a
lot
of
services,
didn't
have
it
and
what
was
worth
quite
a
lot
of
services
had
it
but
returned
a
200
regardless,
so
traffic
always
went
to
the
service.
So
again
we
had
to
fix
that
up,
and
this
is
just
you
know
unless
you're
using
it,
you
don't
necessarily
know
that
you've
got
it
right.
A
The
final
thing
was
actually
caused
by
lots
of
our
services
having
been
written
before
we'd
containerized
at
all.
So
you
know
we
are
a
big
fan
of
12
factor
apps,
but
before
you're
in
containers
it
didn't
matter
if
it
expected
the
days.
Basically
there
with
our
previous
stack
things
didn't
get
moved
around
that
much,
so
we
could
kind
of
get
away
with
it.
A
But
now
Kuban
notice
is
moving
services
around
all
the
time
and
you
need
your
services
to
be
able
to
recover
if
the
things
they
depend
on
aren't
there,
so
we
had
code
changes
to
do
for
that.
The
final
comment
on
this
is
that
developers
don't
like
to
leave
something
in
a
bad
state
and
if
you
pick
up
some
code
that
you
wrote
two
years
ago,
when
you
were
just
learning,
go
and
you're
just
supposed
to
be
fixing
some
stuff
up.
A
So
there
are
lots
of
small
things
that
we
could
have
done
differently,
but
the
major
thing
that
I
would
change
if
I
did.
This
again
is
I
would
have
tried
to
do
it
all
together,
because
we
had
a
very,
very
small
team
working
on
this
and
it's
very
hard
on
them,
because
they're
doing
a
lot
of
repetitive
work
over
a
months
and
months.
A
It's
also
the
overhead
that
you
pay
when
you're
releasing
stuff
to
two
stacks
and
the
high
other
costs
so
running
in
parallel
increased
our
AWS
costs,
which
is
fine.
We
we
were
allowing
for
that.
I
do
recommend
mentioning
it's.
The
people
who
pay
the
bills
before
they
get
a
chart
like
this
I
think
it
caused
some
surprise.
A
A
So
how
did
it
all
go?
Well,
we
went
live
just
before
a
key
part
of
our
old
stack
when
end-of-life
just
I
mean
technically
probably
a
couple
of
days
after
it
went
to
end
her
life.
The
last
few
months
were
quite
intense.
We
had
to
get
everyone
involved,
but
that
was
good
because
it
felt
like
we
were
really
invested
in
this
new
platform.
The
migration
went
very
smoothly.
The
interesting
thing
is
everything
that
went
wrong
in
the
migration
was
because
people
were
using
our
stack
in
ways
we
didn't
understand.
So
we
had
a
small
bug.
A
We
were
on
our
api's,
where
we
weren't
complying
with
the
HTTP
spec.
We
thought
was
fine
to
fix
that
up
turns
out.
Clients
were
relying
on
the
fact
we
weren't
complying
with
the
spec,
and
we
had
some
very
we've
got
some
really
well-defined
integration
points.
So,
if
you
publish
you
hit
a
particular
URL,
if
you
come
into
our
API,
so
you
come
in
through
an
API
gateway,
except
the
people
who
weren't
and
the
only
way
you
can
find
them
is
when
you
migrate
stuff,
and
then
you
look
at
what
load.
A
You've
still
got
on
your
existing
stack
and
then
you
try
and
work
out
who
that
might
be,
but
but
actually
this
took
so
little
time
to
fix
and
it
was
incredibly
smooth
when,
in
terms
of
the
results,
did
we
get
the
results
that
we
were
looking
for?
Yes,
I
mean
system
stability.
We
had
three
production
instance
and
when
I
say
incidents,
it
didn't
affect
anyone.
We
just
had
some
nodes
going
down
in
the
month
after
we
went
live
compared
to
17
in
the
same
month,
in
2017,
we've
had
two
out-of-hours
calls.
A
In
both
cases,
kubernetes
has
recovered
the
system
before
the
developers
even
opened.
A
laptop
and
I
actually
haven't
seen
any
complaining
in
slack
about
the
new
stack,
which
I
think
is
a
major
achievement,
because
I
don't
think
it's
really
rare
to
have
a
developer,
not
complaining
about
about
something
I,
don't
know.
If
anyone
does
the
Spotify
team
health
checks,
but
I've
never
seen
a
developer
happy
about
the
health
of
their
code
and
we
have
something
we
can
learn
from
others.
A
So
that
was
our
journey
and
we
didn't
start
from
the
easiest
place
and
the
interesting
thing
is,
we
probably
didn't
end
up
where
we
thought
we
were
gonna,
go
and
I.
Think
it's
a
good
thing
to
stop
occasionally
and
say:
is
this
a
good
place?
Maybe
we
could
maybe
could
pause
here.
We
have
a
much
more
stable,
cheaper
system
and
a
happier
team
who
are
able
to
spend
more
time
on
functional
stuff
rather
than
supporting
our
platforms,
so
I
believe
I'm,
just
gonna
take
the
opportunity
to
say
we're
hiring
in
both
London
and
Sofia.