►
Description
Migrating Legacy Infrastructure to Kubernetes: Olark’s Lessons From the Trenches - Brandon Dimcheff, Olark
When you start building new infrastructure from scratch, it’s relatively easy to take into account the constraints that Docker and Kubernetes impose. Unfortunately most of us aren’t starting brand new projects, but are maintaining and migrating legacy infrastructure that may not be well suited to run on Kubernetes. Over the past several months, Olark has migrated a number of services that were never designed with Kubernetes or Docker in mind from over 200 puppet-managed Ubuntu VMs to Google Container Engine. Brandon will share some of Olark’s successes and failures, so that hopefully you can have a good starting point and avoid making the same mistakes that they did. He’ll also answer some questions like: How can I set up DNS and VPN so that I can route between my legacy infrastructure and Kubernetes services? I have a stateful application, can (or should) I still use Kubernetes? What are some things I can do to reduce the risk involved in a large-scale migration?
About Brandon Dimcheff
Brandon is a software engineer born and raised in Ann Arbor, Michigan, where he still lives with his wife and two cats. He’s now the Director of Engineering at Olark, a largely remote company that provides live chat services for your website. When he’s not working, he’s probably skiing or trying to plan his next ski trip. Brandon has previously spoken at Ruby Midwest, jsconf, and has organized and spoken at Great Lakes Ruby Bash.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
my
name
is
Brandon
dim
chef
I,
usually
look
like
that
on
the
Internet
I
work
for
Oh,
lark
and
I'm,
going
to
talk
to
you
a
little
bit
about
our
migration
from
sort
of
puppet
managers,
legacy
infrastructure
to
kubernetes.
We're
not
done
with
this
yet.
But
we've
already
learned
some
important
lessons.
I
think
that
hopefully
will
be
helpful
to
other
people
as
well.
A
B
A
Is
pretty
much
just
a
huge
meeting,
so
it's
always
good
to
have
an
agenda
right.
So,
okay,
we're
gonna
talk
a
little
bit
why
we
decided
to
embark
on
this
kubernetes
mission
when
I
talk
a
little
bit,
how
how
we
did
it,
you
know,
have
a
few
Pro
tips
littered
throughout
and
also
some
of
my
cats
and
cats
from
the
neighborhood,
because
I
feel
like
people
could
use
caps
today
anyhow.
So
why
well
clearly,
because
it's
100%
certainty
solve
all
of
our
infrastructure
problems.
It's
gonna
be
a
silver
bullet,
a
panacea.
A
B
A
So
let
me
just
tell
you
a
little
bit
about
what
a
lark
is
first,
so
we
have
a
basis
to
go
on
because
we're
a
live
chat,
provider,
software-as-a-service
live
chat
provider.
You
copy
a
little
bit
of
code
onto
your
website
and
when
you
log
into
our
like
admin
chat
panel-
and
you
can
talk
to
visitors
that
show
up
on
your
website.
So.
A
And
legacy
is
not
like
we're,
not
this
like
we
were
a
start-up
in
2009
YC,
so
you
know
if
you
were
expecting
like
how
to
move
your
backs
thing
to
kubernetes.
That's
not
gonna
happen
and
honestly,
our
architecture
fits
kubernetes
idea
of
the
universe
relatively
well
already,
like
our
previous
architecture.
We're
got
some
micro-services
and
all
those
fun
things
got
dozens
of
them.
Some
of
them
might
be
a
little
more
macro
than
micro
they're,
mostly
in
Python,
but
there's
some
some
Ruby,
node
and
Erlang
here
and
there
most
of
them
are
stateless.
A
We
use
rabbit
and
thrift
really
heavily
in
our
infrastructure,
and
most
of
our
state
is
in
MySQL,
with
some
in
Redis
and
memcache
and
other
various
things
here
and
there,
and
there
aren't
a
huge
number
of
single
points
of
failure
that
we
have
and
we're
currently
have
hundreds
of
single
purpose.
Vms
at
Rackspace
by
single
purpose,
I
mean
it's
the
VM
that
one
runs
one
process.
Basically
yeah
it
doesn't.
You
know,
there's
no.
We
don't
really
have
things
that
are
like
trying
to
manage
utilization
or
anything.
A
It's
all
managed
with
puppet,
and
we
have
a
few.
A
few
pets
that
like
if
something
bad
happens
to
them,
these
guys
over
here,
are
gonna,
get
paged
and
then
run
out
of
the
room.
So
it's
not
perfect,
but
it's
sort
of
close
to
what
kubernetes
might
expect
from
services
now,
but
we
have
really
really
bad
utilization
right
now,
I
mean
this
is
like
you
know.
We
have
this
one
server,
that's
using
maybe
a
quarter
of
a
CPU
most
of
the
time,
and
maybe
it's
using
a
little
more
memory.
It's
just
an
example.
A
I
pulled
that
immune
in,
but
thinking
about
trying
to
manually
allocate
processes
here
and
there
to
like
put
a
CPU
intensive
process
and
a
memory
intensive
process
on
the
same
machine.
It's
just
like
that's
like
that.
I
keep
me
up
at
night,
so
we're
burning
like
kind
of
a
lot
of
money
with
bad
utilization,
we're
also
tired
of
babysitting
these
individual
machines.
If,
if
one
of
them
gets
whacked
in
the
middle
of
the
night
and
pages,
someone
there's
still
some
manual
intervention,
we
have
to
do
and
we
want
better
velocity
right
now.
A
A
A
Anyways
I'm,
just
like
whoever
someone
someone
in
the
audience
make
our
Redis
work,
but
in
the
meantime,
basically
we
had
because
we
needed
to
use
Redis,
Sentinel
and
Pat's
pet
stats
are
pretty
new
and
we
were
trying
as
I,
trying
to
figure
out
how
to
get
that
all
working
correctly
turned
out
to
be
more
effort
than
it
was
worth
at
this
time
for
us
to
try
to
migrate
stuff
off
so
maybe
later.
So.
A
A
The
first
thing
that
is
sort
of
maybe
obvious
to
everyone
here
is
that
you
have
to
containerize
things.
It
turns
out.
It's
really
easy
to
containerize
cats,
you
just
put
a
container
and
then
they
go
in
it.
It's
not
quite
that
easy
for
other
things,
but
that's
sort
of
like
a
different
talk
and
a
different
problem.
So
I'm
not
going
to
talk
about
that
anymore
and
actually
for
us
our
step
negative
one
was
to
apply
a
little
bit
of
twelve
factor.
A
I,
don't
know
if
everyone
has
seen
this
but
take
a
look
at
12
factor,
dotnet,
it's
sort
of
principles
for
building
micro
service
applications
that
apply
pretty
well
to
kubernetes
stuff
too,
and
it
made
us
it
made
it
easier
for
us
to
move
our
stuff
to
kubernetes.
Once
we
had
like
environment
variables,
configuring
our
applications
and
logs
logging
to
standard
out
and
stuff
like
that,
that
kubernetes
expects.
A
A
You
can
get
a
lot
of
value
just
by
like
taking
the
first
part
of
the
I
guess,
that's
like
the
cat,
but
part
of
it
before
it
gets
to
the
tail
but
and
we're
sort
of
we're
sort
of
like
still
here
in
this
process.
So,
as
I
said,
I'm
still
work
in
progress,
I
don't
know
if
we'll
end
up
getting
all
the
way
down
there.
We
may
end
up
deciding
that
you
know
some
of
our
legacy
stuff
will
stay
legacy
but
we'll
see.
So
that
brings
me
to
another
tip.
A
So
we
saw
we
came
up
with
a
strategy
to
avoid
this,
and
the
first
thing
was
to
move
a
less
important,
the
ancillary
service,
something
we
could
work
the
kinks
out
on
and
to
answer
as
many
sort
of
unknown
unknowns
that
we
could
and
then
to
move
like
kind
of
the
most
risky
service,
because
if
we
do
the
hardest
thing
and
it
works,
then
everything
else
is
is
easier
and
it
reduces
the
risk
of
like
alright.
We
saved
the
most
risky
thing
for
the
end
and
then
we
moved
it
and
it
didn't
work.
A
As
you
move
and
you
might
need
to
do
extra
work,
but
just
like
having
good
tests
and
stuff
like
that,
it's
probably
probably
worth
it
in
the
end,
because
you're
more
likely
to
succeed
a
little
secret,
it's
not
really
specific
to
kubernetes
I,
think
like
any
sort
of
migration
or
a
sort
of
large
undertaking,
is
best
done
in
a
piecewise
fashion
that
you
can.
You
can
check
as
you're
going
along,
so
we
were
looking
around
for
our
less
important
service
that
didn't
really
matter
that
much
so
naturally
chose
our
billing
system.
A
It's
not
as
crazy
as
it
sounds.
It
happens
to
be
a
little
bandwidth.
Unfortunately,
I
mean
you
know
it's
like
a
handful
of
requests
per
second
or
something
it's
like
I
mean
I
wish.
It
was
like
we
had
like
serious
scaling
issues
with
our
billing
system,
but
unfortunately
not.
It
also
happens
to
be
representative
of
the
rest
of
our
infrastructure
because
it
uses
the
same
databases.
It
uses
thrift,
which
we
use
heavily
elsewhere.
It
uses
rabbit,
that's
what
a
rabbit
might
look
like,
and
these
are
things
that
we
use
all
throughout
our
infrastructure.
A
A
A
A
It's
pushing
a
couple
gigabits,
a
second
back
and
forth
between
the
rest
of
our
infrastructure,
and
if
it
goes
down,
then
our
chat
doesn't
work
at
all.
So
you
know
it's
not
super.
It's
not
like
the
payment
thing
where
things
will
still
like
mostly
work
if
it's
gone
so,
but
it's
it's
still
pretty.
It's
is
pretty
well
suited
to
kubernetes,
because
there's
just
stateless
services
that
are
running
any
user
can
hit
any
back-end
and
get
their
chats
routed
properly
it.
A
A
A
It
gathers
a
bunch
of
performance
metrics
just
for
free
when
you,
when
you
use
it,
and
so
we
decided
to
use
this
both
in
kubernetes
in
our
and
in
our
legacy
infrastructure,
so
that
as
we
migrate,
we
can
like
make
sure
all
of
those
numbers
are
staying
in
the
green
and
I
know
that
at
least
one
person
from
buoyant
is
here.
So
if
you
are
interested
in
in
talking
to
someone
about
link
or
Dee
Williams
around
here
somewhere,
so
it
gives
us
these
kind
of
metrics.
There's
like
you
can't
read
that
at
all.
A
But
there's
like
99th
percentile,
95th,
percentile,
50th
percentile
max
Layton
sees
request
for
second
failure
rate
stuff
like
that
and
it's
all
it
exposes
Prometheus
endpoints
as
well
as
stuff
for
other
monitoring
things.
So
it's
easy
to
aggregate
these
from,
like
all
of
your
link
or
the
instances
or
whatever
in
whatever
you're
using
so
that
you
can
monitor
it,
and
then
we
were
using.
This
is
that
we
have
so
like
in
our
legacy
data
center.
We
have.
A
So
as
we
move
like,
if
we
move
service
three
over
to
Cuba
Nettie's
we're
just
going
to
update
some
config
in
the
various
linker
DS
to
point
it
to
the
to
the
new
one
in
kubernetes,
instead
of
back
at
the
old
one,
and
so,
instead
of
having
to
like
push
a
configuration
change
to
your
extra
application
to
point
at
a
different
thing,
it's
sort
of
like
a
load
balancer
right,
you
can
just
like
cut
over
the
load.
Balancer.
A
A
Like
migrating,
anything
I
think
like
link,
link
or
D
is
worth
a
look
if
you
want
to.
If
you
have
like
you,
know,
service,
oriented
architecture,
kind
of
stuff,
and
you
need
or
want
better
metrics
without
having
to
instrument
of
applications
yourself,
because
you
get
you
get
that
for
free.
It
also
does
retries
and
some
other
cool
things
so
yeah.
A
You
still
get
all
the
service
discovery
and
stuff
that
services
give
you
and
also
to
use,
link
or
D,
so
that
we
can
always
just
basically
talk
to
link
or
D
all
the
time,
but
because
we
have
in
our
old
puppet
stuff,
we
have
a
list
of
where
all
the
where
all
the
services
are
pointed
anyways.
In
our
legacy
infrastructure,
we
can
use
that
to
generate
services
and
endpoints.
That
are
that
point
that
allow
kubernetes
communicate
back
with
our
regular
infrastructure.
A
C
A
Yeah,
so
in
some
cases
our
services
in
kubernetes
are
talking
to
back
to
services
that
are
running
in
our
old
infrastructure
outside
of
kubernetes,
so
the
services
are
just
like
we're
just
using
how
much
pointers
are
like
I
mean
it's
like
DNS
kind
of,
except
like
fancier
like
we
can
use
it
as
service
discovery.
When
we
switch
it
all,
we
like
the
service
name,
stays
the
same.
Everything
is
connecting
the
same
thing.
It's
just
getting
kubernetes
now,
instead
of
their.
A
External
name
is
like
one
thing
that
you
can
use
to
kind
of
do
this.
We
constructed
our
own
endpoints
like
this,
and
this
like
resolve
thing,
gets
processed
through
a
make
file
to
resolve,
like
our
internal
DNS,
not
actually
from
DNS
but
from
our
the
stuff
that
actually
produces
our
DNS
records
to
begin
with,
so
that
just
like
it
creates
all
these
endpoints
for
some
service
manually
and
then
other
stuff
can
talk
to
that.
And
so
this
is
come
in
really
handy.
A
A
A
So
we
ended
up
hacking,
service,
load,
balancer
and
some
termination
grace
periods
on
pods
to
get
like
kind
of
what
we
wanted
to
do
and
we're
not
gonna
actually
set
it
to
a
day
but
like
if
this
pod
sticks
around
for
a
while.
People
will
eventually
like
sign
off
from
that
thing
and
will
not
just
sort
of
you
know
after
a
day
or
something
that'll
be
like
six
people
still
attached
to
it
or
something
and
like
they
can
just
be
reconnected.
A
And
so
sort
of
what
we
figured
out
here
is
that
you
can
gain
a
lot
of
flexibility
by
deploying
some
custom
pods
or
like
hacking.
Some
things
that
are
that
are
internal
to
kubernetes
service
load,
balancer,
isn't
actually
internal
to
kubernetes,
but
it's
a
contributed.
Some
minor
modifications
to
do
this
and
it
it
works
pretty
well.
My
recommendation
is
just
to
pick
your
battles
wisely
in
this
case.
Unlike
the
Redis
case,
we
didn't
really
have
another
good
option,
for
we
did.
A
We
definitely
didn't
want
to
just
run
this
the
old-fashioned
way,
because
we're
not
gonna
be
able
to
get
any
like
auto-scaling
benefit
and
deploying
additional
ones
of
these
is
relatively
common,
and
so
we
don't
want
to.
You
know,
leave
all
that
behind
just
for
a
little
bit
of
a
little
bit
of
fight.
A
Yeah,
so
that's
pretty
much.
What
I
have
here
really
the
takeaway
is
I.
Think
here
are
you
know,
control
your
change.
Try
not
to
do
too
much
at
one
time
now
go
with
the
flow,
and
you
can
have
some
baseline
measurements.
Occasionally
you
might
need
to
do
something,
a
little
crazy,
like
a
little
stateful
thing
and
I
like
to
put
cats
on
my
slides.
A
B
D
A
So
service
load
balancer
is
it's
like
a
I.
Don't
know.
Example
thing
in
the
kubernetes
contribute:
oh,
that
is
just
H
a
proxy
that
is
kubernetes
aware,
so
you
can
use
it
for
internal
load,
balancing
of
your
services
and
stuff
like
that,
and
you
can
use
it
for
external
ingress.
If
you
want
it's,
not
an
ingress
controller
but
like
you
can
use
it
to
it
to
like
route
traffic
from
outside
into
your
cluster
and.
B
A
B
A
Didn't
we
didn't
really
do
any
of
that
we're
using
gke
for
our
stuff
than
I
mean
I,
don't
know
if
you
can
like
do
that
with
inside
of
there,
but
we
were
just
trying
to
so.
We
have
a
VPN
from
Rackspace
to
like
g
c
e,
using
their
like
software
as
a
service
load
balancers
on
both
ends,
but
that
just
gives
you
access
to
the
node
IPS
on
the
kubernetes
ahead
right
so
like
you
still
need
to
like
get
into
the
cluster
somehow
and
then
going
the
other
way
around.
It's
like.
A
If
you
are
running
your
own
kubernetes
cluster,
you
can
add
to
cube
dns.
You
can
add
a
configuration
for
an
upstream
DNS
provider
that
you
would
use
your
internal
DNS
or
something
like
that
for
that
right.
But
since
we
were
using
gke,
if
you
modify
that
pod
and
then
upgrade
your
cluster,
it
will
smash
it,
and
we
didn't
want
the
risk
of
us
that
inadvertently
happening
at
a
bad
time
and
like
all
of
our
kubernetes
stuff,
not
knowing
about
our
old
stuff
anymore.
So
that's
sort
of
how
that
all
worked
out.
C
A
A
That
wouldn't
like
our
private
DNS
does
is
not
available
inside
of
our
kubernetes
cluster,
so
that
wasn't
that
wasn't
really
an
option
for
us.
But
if
you
have
like
public
dns
or
if
you
have
DNS
accessible,
you
can
use
external
name
in
the
service
to
just
point
at
some
random
DNS
somewhere
as
like
the
service,
but.