►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
From Data Centers to Cloud Native - Dave Zolotusky & James Wen, Spotify (Intermediate Skill Level)
Spotify’s infrastructure is undergoing a drastic transformation from data centers running a large amount of proprietary services to public cloud hosted, cloud native services. Two years ago, all of Spotify’s services ran on on our own hardware. Today, they are running in Google Cloud Platform, and Spotify is in the process of replacing many of the proprietary systems with cloud native solutions like kubernetes, gRPC, and istio. James and Dave will talk about the work that went into moving services into GCP and how Spotify got to where it is today. They will then cover ongoing work and plans to modernize the service infrastructure and migrate to cloud native services. This includes running service in kubernetes on GKE, migrating from a proprietary messaging stack to gRPC, as well as integrating the new infrastructure with existing service discovery and monitoring systems.
About Dave
Dave Zolotusky is a software engineer on Spotify's Infrastructure and Operations team in Stockholm, Sweden. His team builds and maintains the core platform that the rest of Spotify runs on. Before joining Spotify, Dave worked at Amazon Web Services, VMware, and Microsoft.
About James
James Wen is a Site Reliability Engineer at Spotify. He's currently on a Spotify task force focused on revamping Spotify's infrastructure and adopting Kubernetes and co. Prior, James was the Team Lead (Anchor) of the Cloud Foundry Buildpacks team at Pivotal and a core contributor and maintainer of Bundler. James has previously spoken about buildpacks at Cloud Foundry Summit Europe and Spotify's infrastructure journey at QCon New York. In his free time, he absolutely loves climbing, whether it's on gorgeous Fontainebleau slopers or nasty plastic crimps
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
A
So,
let's
start
at
the
beginning,
let's
start
with
what
we
had
on
our
data
centers
and
what
our
services
looked
like
when
we
were
running
in
data
centers,
we
had
a
whole
bunch
of
custom
things,
but
we
also
picked
up
a
fair
amount
of
interesting
new
technology.
We
were
a
fairly
early
adopter
of
docker
containers
and
we
were
running
a
lot
of
our
services,
our
Java
services
in
docker
containers
in
like
2013
2014.
A
We
built
our
own
container
expression
engine
because
at
that
point
that
really
weren't
great
and
engines
available
elsewhere,
and
we
eventually
open-source
to
staying
this
link
goes
to
our
github
page.
We
open
sourced
it
right
on
the
time
cover
IDs
as
open
source
or
announced,
but
obviously
crazy
things
at
the
time.
So
we
couldn't
use
it.
A
We
have
our
own
custom
monitoring
platform.
We
have
a
time
series
database,
we
call
heroic,
which
is
open
source.
It's
based
on
an
open
source
time
series
database
called
open,
TS
DB,
but
has
some
differences
that
we
needed,
and
then
we
have
an
internal.
We
have
an
open
source
that
dashboarding
system.
We
call
iliyan
that
does
our
dash
boarding
our
alerts
and
gets
us
paged,
and
the
internals
of
all
of
this
stuff
is
based
on
things
like
Cassandra.
We
use
elastic,
search
for
metadata
and
it's
all
kind
of
globally
federated
through
our
data
centers.
A
So
you
can
get
single
dashboards
for
all
of
your
services.
On
top
of
that,
we
have
our
own
proprietary
message
protocol.
This
thing
we
call
Hermes
it's
based
on
zero
MQ,
it's
mostly
protobuf
/,
0
mq
and
the
biggest
drivers
for
having
our
own
protocol
were
that
we
needed
message
multiplexing
over
single
TCP
connections
and
back
when
we
were
starting
to
do
this
around
like
2006-2007.
A
A
We
have
our
own
custom
Java
framework.
This
thing
we
call
Apollo,
which
is
open-source
but
has
modules,
some
of
which
are
open
source
on
which
or
not-
and
you
can
think
of
this
thing
as
something
kind
of
like
spring
or
Spring
but--,
but
obviously
on
an
internal
thing
and
not
nearly
as
kind
of
full-featured,
and
it
has
built-in
clients
for
things
like
Hermes
protocol
and
for
heroic
metrics
and
all
of
our
other
kind
of
custom
pieces.
A
It's
really
a
way
to
bootstrap
a
service,
give
all
the
kind
of
initial
boilerplate
code
that
all
of
our
services
need.
Most
of
our
storage
was
either
in
Postgres,
with
my
sequel
for
kind
of
very
early
Spotify.
It
was
almost
entirely
Postgres.
Then,
as
we
grew
enough,
we
started
switching
over
to
Cassandra
and
then
really
more
recently.
Cassandra
has
been
our
kind
of
one
database
for
everything
if
we
can-
and
that
means
that
we
have
an
incredible
number
of
Cassandra
clusters
that
cover
almost
every
use
case.
We
have
huge
ones.
A
A
We
probably
have
a
Cassandra
database
somewhere
in
our
deployment
that
youth
has
that
rough
use,
and,
of
course
that
means
that
we
have
a
lot
of
experience
throughout
the
entire
company
on
kind
of
operate,
needs
Cassandra
clusters,
and
we
have
a
team
in
our
infrastructure
team
that
is
kind
of
really
Cassandra
specialists
for
all
of
these
different
use
cases,
the
other
really
big
thing
we
used
a
lot
of
is
Hadoop.
We
use
so
much
a
dip
in
fact
that
we
had
the
largest
single
tenant
to
do
cluster
in
Europe.
A
This
thing
ran
I
think
it
was
just
over
20,000
jobs
a
day
when
we
were
kind
of
at
his
peak
it
had
around
3,000
nodes.
It
ran
kind
of
hundreds
of
different
types
of
events.
It
got
hundreds
of
thousands
of
events
per
second
in
some
specific
streams
and
many
many
more
than
that
when
you
combine
all
the
streams
across
all
the
clients
and
many
of
the
data
pipelines
that
ran
of
those
20,000
jobs
were
kind
of
complex,
chained
pipelines.
A
B
A
A
documented
way
to
tell
a
developer
within
our
team
if
you
want
to
create
a
service,
follow
these
steps
and
you'll
have
a
blessed
kind
of
official
Spotify
way
of
creating
a
service,
and
we
start
having
a
few
of
these.
We
had
one
for
kind
of
how
you
could
a
data
pipeline,
how
you
create
it
back
in
service
and
so
on,
how
you
create
a
mobile
feature,
etc,
and
the
a
lot
of
these
things
come
out
of
our
tech
platform,
or
so
we
have
now.
A
We
have
close
to
4,000
employees
and
we
have
a
500
person
or
so
type
platform,
org
of
which
2
to
300
is
just
infrastructure
and
operations
and
data
infrastructure.
So
people
that
run
all
of
these
things
and
just
ensure
that
all
of
our
back-end
services
are
using
kind
of
reasonable
new
technology.
We're
doing
things
like
moving
over
to
cloud.
One
plan
go
it
now
and
then,
like
the
data
in
front
of
team,
is
things
like
maintaining
all
the
tools
that
we
had
for
the
Hadoop
cluster
and
transitioning
all
of
that
to
cloud.
A
So
we
had
all
of
these
things
in
our
data
centers.
We
had
all
these
proprietary
tools
and
eventually
we
realized
that
kind
of
way
too
much
work
to
maintain
all
this
stuff.
There's
so
many
things
that
just
are
not
our
core
business.
We
are,
at
the
end
of
the
day,
a
music
streaming
service
and
even
though
we're
a
fairly
large
company,
we're
not
big
enough
to
kind
of
have
enough
people
to
really
focus
on
running
data
centers
and
running
all
these
things,
so
mostly
for
the
sake
of
focus,
but
also
for
all
the
other
benefits.
A
Everyone
talks
about
the
elasticity,
the
managed
services.
All
these
other
great
features
we
knew
we
wanted
to
move
to
cloud,
but
at
that
point
we
really
had
no
idea
kind
of
where
to
start
how
to
do
it.
What
to
do?
We
just
knew
that
we
wanted
to
do
it,
so
we
decided
that
the
right
thing
to
do
would
be
to
figure
out
kind
of
stages,
to
figure
out
milestones
to
do
it
like
any
other
software
project,
and
our
first
really
big
goal
was
just
move
out
of
our
data.
A
Centers,
like
set
the
goal
as
stop
paying
rent
kind
of
stop
having
data
centers
and
then
we'll
figure
out
the
rest
later
and
that's
exactly
what
we
did
so
our
platform
teams,
infrastructure
and
operations
teams,
the
datum
structure.
All
these
teams
then
kind
of
got
to
work
on
figuring
out
what
that
even
meant
for
everyone
else,
because
everyone
else
in
the
company
is
building
features
of
building
their
services.
They
don't
want
to
have
to
think
about
what
is
under
what
is
running
their
services.
A
Ideally,
that's
our
problem,
so
we
went
through
and
figured
out
the
network
connectivity
between
our
data
centers
and
our
cloud
providers.
Data
centers.
We
figured
out
how
to
kind
of
extract
away
machine
provisioning
so
that
the
tools
that
all
of
our
developers
used
to
use
the
provision
machines
now
could
provision
VMs
in
a
cloud
data
center.
Things
like
deployment
same
thing
kind
of
they
could
just
say,
I'm
deploying
the
service
and
it
ends
up
in
the
cloud
instead
of
in
one
of
our
data
centers,
but
it
feels
the
same
to
them.
A
So
we're
not
teaching
them
all
new
things.
Service
discovery,
of
course,
also
had
to
work
across
cloud.
All
of
these
things
kind
of
came
out
of
the
platform
we
could
before.
We
did
anything
after
that.
We
really
had
to
go
on
kind
of
a
tour
across
the
company
to
get
every
service
moved
and
get
all
of
that
working.
So
the
platform
team
at
that
point
kind
of
went
around
the
company
working
with
services.
A
The
next
big
thing
that
giant
Hadoop
cluster-
and
this
really
was
the
harder
part
because
with
services
it
was
a
lot
easier
to
just
say:
move
you're
like
you
have
bare-metal
in
furniture
here
you
have
a
VM
there
kind
of
looks
and
feels
it's
the
Linux.
It's
still
running
for
container
still
using
the
same
container
orchestration
system.
We
can
fake
it
with
data
jobs.
A
A
The
initial
thought
is
kind
of
we
have
to
start
either
at
the
beginning
of
all
of
these
pipelines
or
at
the
very
end,
and
then
move
the
other
way
and
that
was
really
hard
to
because
the
jobs
of
the
beginning
or
the
jobs
at
the
end
may
be
owned
by
one
of
many
different
teams.
They
look
and
feel
very
differently.
A
It
would
have
been
very
difficult
to
kind
of
coordinate
everyone
on
either
end
of
this
kind
of
complicated
data
pipeline
to
really
start
and
flow
across,
and
then
like
one
team
has
to
move
their
first
job
and
then
they
might
own
the
fourth
job.
So
we
come
back
to
them.
Two
weeks
later,
it
just
wasn't
going
to
work.
So
we
kind
of
came
back
with
in
the
data
team
thought
about
what
else
we
need
and
created
a
bunch
of
tools
to
make
this
easier.
Obviously,
we
needed
better
network
connections
between
the
data
centers.
A
But
on
top
of
that,
it
lets
you
pick
with
a
command
line
flag
if
you
want
to
schedule
to
the
on-prem
one
or
in
the
cloud.
So
at
that
point
we
did
have
to
have
the
feature.
Teams
change
some
things,
but
all
they
really
have
to
change
was
add
a
flag.
That
said,
like
we've
tested
this,
and
we
think
it
works
in
clouds
so
now,
like
spider,
kick-off
job
cloud
as
opposed
to
just
leaving
it
by
default.
A
If
you
move,
if
your
job
moves
and
the
job
after
you
didn't,
then
then
you
would
just
have
to
copy
their
back
so
the
job
after
you
can
run
what
can
run
normally
and
then,
when
that
job
moves,
you
no
longer
do
the
disk
copy.
You
just
run
next
to
it,
and
this
was
really
the
biggest
thing
to.
Let
us
start
moving
our
data
jobs.
At
this
point
we
now
could
just
go
to
individual
teams
and
say
hey.
A
We
want
to
work
with
you,
but
let's
move
all
of
your
jobs
and
it
didn't
matter
where
they
were.
Let's
work
with
you
guys,
we'll
move
all
of
your
jobs.
The
biggest
problem
with
this,
of
course,
was
we're
constantly
running
distributed
copies.
We
maxed
out
the
pipe
we
had
between
our
data
center
and
our
cloud
providers
constantly
and
kept
growing
it
all
the
time
through
this,
but
once
it
was
actually
done,
it
was
amazing
and
we
had
moved
twenty
thousand
jobs
a
day
and
this
enormous
new
cluster.
A
So,
despite
all
the
pain,
it
was
absolutely
worth
it.
So
now
we're
effectively
moved
out,
there's
kind
of
there's
always
a
long
tail
of
things,
but
the
intense
majority
of
the
things
we've
done
have
moved
out.
This
was
overall,
probably
what
a
three
and
a
half
year
effort
from
deciding
me
to
do
it
to
being
where
we
are
now,
but
the
biggest
push
was
2017
the
at
the
beginning
of
the
year.
We
really
had
a
lot
of
the
infrastructure
for
moving
some
of
the
network
connections,
the
ideas
for
how
we
were
gonna.
A
Of
course,
at
this
point,
we're
not
we're
now
running
a
replica
of
our
on-prem
data
centers
in
someone
else's
data
centers,
so
sure
we
don't
support
the
hardware,
but
every
one
of
those
proprietary
things
I
just
told
you
about.
We
still
have
and
we're
not
getting
great
relation
with
hardware,
we're
not
getting
great
use
of
all
the
managed
services
and
we're
just
not
still
not
completely
focusing
on
streaming
music.
A
We
still
have
a
whole
lot
of
stuff
to
do
so
now
we
need
to
figure
out
what
it
means
to
be
kind
of
to
be
cloud
native
to
feel
like
we're
born
in
the
cloud.
But
we
didn't
really
even
know
what
that
meant.
We've
been
running
in
data
centers
forever,
like
sure,
there's,
auto-scaling
and
there's
open
source
communities,
there's
managed
services.
A
But
what
do
we
do
so
we
figured
we'd
start
somewhere
relatively
obvious
and
just
start
experimenting
with
things
like
kubernetes
and
G,
RPC
and
figuring
out
how
to
do
the
kind
of
the
simple,
the
right
sizing,
the
Auto
scale
and
all
this
stuff
I'll.
Let
James
tell
you
about
the
committee
side
of
that.
B
B
One
with
procedural
stages
in
each
stage
is
very
succinct
goal
so,
for
our
very
first
stage
was
kind
of
an
experimental
one
and
we
sought
to
just
run
one
service
on
a
single
cluster
and
have
that
service
received
production
traffic
for
just
one
hour,
and
so,
despite
that,
being
a
very
like
a
small
goal,
I
drove
out
a
lot
of
things
for
us,
for
example,
we
actually
needed
to
create
cluster
at
this
point.
It's
manual
process,
since
it's
a
single
cluster
we
needed
to
set
up
DNS
networking
route
ability,
for
example,
we
needed.
A
B
Set
up
the
correct
upstream
name:
resolvers,
IP,
msq
agents,
so
that
the
infrastructure
stuff
for
clusters
and
the
service
could
resolve
the
correct
host
names
for
our
internal
Spotify
stuff.
We
needed
to
set
up
integration
where
there
is
a
staying
service.
Discovery,
which
is
an
in-house
system
called
nameless,
so
that
other
back-end
hosts
and
services
would
know
to
talk
to
this
service
running
on
kubernetes.
We
needed
to
set
up
integration
with
our
existing
metrics
metric
system
and
logging,
so
that
the
we
knew
that
the
service
was
operating
correctly.
B
It
was
receiving
traffic
and
was
responding
to
requests
properly
so
that
all
worked
as
expected,
and
then
for
our
next
experimental
phase.
We
sought
to
run
three
services
on
a
shared
cluster.
So
now
we
introduced
the
element
of
multi-tenancy
I'll,
be
on
like
a
very
small
scale,
and
so
with
multi-tenancy.
B
Because,
up
to
this
point,
we
were
dealing
with
a
single
team.
It
was
easy
enough
to
jump
in
there
slack
channel
or
schedule
a
Google
hangout
meeting
and
talk
to
them
directly.
But
now
we
don't
want
to
have
to
repeat
the
same
exact
message:
two
or
three
different
teams
and
later
multiple
teams.
So
now
we
start
to
write
developer
documentation.
So
after
that
we've
ended
the
experimental
of
phases,
because
at
this
point
we've
gained
enough
confidence
to
say
that
we
can
probably
run
most
of
our
services
eventually
on
communities.
B
B
B
So
we
want
to
isolate
the
clusters
that
we
experiment
with
resource
manifests
on
away
from
the
production
clusters.
So
we
have
no
with
the
test
clusters.
We
now
have
multiple
clusters
that
we
want
to
have
mostly
the
same
configuration
on.
So
we
script
out
the
creation
of
those
clusters.
We
start
to
automate
that
process
and
then
the
we
integrate
with
the
existing
a
secret
system.
So
we
have
an
in-house
system
called
a
CLO
and
a
lot
of
services
rely
on
this
system.
B
So
we
basically
write
integration
that
automatically
puts
their
Co
secrets
as
secret
resources
in
their
namespaces.
So
they
don't
have
to
worry
about
that
kind
of
stuff.
Note
that
for
this
it's
kind
of
a
stopgap
solution
in
the
future,
we
want
to
move
to
something
more
like
vault
or
something
more
robust.
But
for
this
it's.
B
We're
only
dealing
with
a
small
amount
of
services
and
the
next
thing
was
initial
deployment.
So
for
this
we
wrote
a
pretty
small
wrapper
around
cube,
CTL
that
and
manually
invoke
in
their
pipelines,
just
like
a
one
or
two
line
thing
that
would
deploy
to
the
clusters
that
the
production
clusters
that
we
have
set
up
for
them
and
right
now.
This
is
like
good
enough
of
a
solution
because
we're
dealing
with
say
less
than
20
teams
less
than
20
Jenkins
instances,
and
in
this
phase
we
got
a
lot
of
learning
from
learnings
from
running
these.
B
Issues
we
found
some
unexpected
behavior
with
readiness
probes.
We
found
this
control
arm
n
max
issues
with
the
like
packet
buffer
size
on
nodes.
That
kind
of
bit
us
I've,
seen
some
people
nodding
actually
so
looks
like
we're
not
alone
there.
We
also
ran
into
the
limitation
with
cube
DNS,
where
it
didn't
support:
reverse
DNS
lookups
for
upstream
name
resolvers,
but
props
to
the
cube
DNS
team,
because
just
two
weeks
after
we
created
that
issue,
they
app
they
added
that
feature
and
then
close
that
the
issue.
B
So
we
were
really
impressed
with
the
really
quick
turnaround.
They're
an
important
part
of
this
kind
of
alpha
phase
was
that
we
really
wanted
to
run
to
high
traffic
services
on
communities
and
have
them
the
two
we
chose
were
fan-out
and
mediator.
Poxy
fan-out,
you
can
think
of
as
a
an
internal
pub/sub
system,
meditator
proxy,
think
of
as
the
gateway
to
all
of
the
metadata
as
part
of
Phi.
B
So
all
that
mission-critical
song
and
artist
info
in
terms
of
load
fan-out
gets
about
1.5
million
requests
per
second
and
that
works
out
to
about
15k
requests
per
second,
for
instance,
and
permitted
at
epoxy.
It
receives
about
500k
requests
per
second
and
500
requests
per
second,
for
instance,
services,
and
so
what
this
drove
out
for
us
was.
We
started
to
use
an
experiment
with
auto
scaling,
horizontal,
auto
scaling.
B
At
this
point,
it
really
forced
us
to
understand
the
need
of
Grady
of
our
network
setup
because
of
the
low
latency
requirement
of
these
services
and,
most
importantly,
is
served
as
a
great
reference
for
other
teams
who
were
migrating.
We
had
direct
feedback
from
some
teams
during
migration
that
they
were
initially
worried
about
their
services,
how
they
were
going
to
run
on
kubernetes.
They
thought
they
had,
but
we
would
point
them
to
the
example
of
these
two
services
that
were
able
to
run
on
the
communities
clusters
that
we
set
up
and.
B
Services
were,
for
the
most
part
like
less
complex
and
had
lower
performance
requirements
than
these
services.
Thank
you.
So,
the
next
phase,
after
the
alpha
phase
that
we
recently
wrapped
up,
is
the
beta
phase.
What
we
consider
the
beta
phase,
and
in
this
we
have
self
service
migration,
and
so
in
here.
Teams
that
are
interested
can
follow.
Infrastructure
provided
Docs
guidelines
tooling
to
self
service
migrate,
either
partially
or
fully
their
capacity
from
the
old
Helios
runtime
to
this
new
kubernetes
runtime
that
we're
providing,
and
so
the
two
elements
that
we
introduced
here
are
as
operators.
B
B
This
no
longer
becomes
as
easy
and
we
need
to
introduce
a
lot
of
reliability
on
the
part
of
our
clusters.
So
we
start
to
look
at
reliability.
We
start
to
set
up
on-call
for
clusters
and
the
associated
alerts
in
playbooks.
We
do
like
dirt
testing
for
disaster
recovery
and
backups
on
the
service
side.
B
We
start
to
think
about
how
do
we,
let
developers
create
their
own
namespaces,
but
have
those
names
to
be
set
up
with
the
proper
are
back
permissions
and
saying
resource
quotas
so
that
they
don't
have
to
really
worry
about
these
kinds
of
things?
On
the
deployment
side,
we
start
to
look
past
manual
command
vacations
in
Jenkins
pipelines
and
look
at
a
more
robust
deployment,
orchestration
layer.
We
chose
spinnaker
for
this
and
assigned
note
that
we're
kind
of
on
the
leading
edge
of
spinnaker
and
that
we're
using
the
v2,
the
new
v2
communities
provider.
B
So
if
you're
using
this-
and
you
have
any
thoughts
or
want
to
compare
notes-
were
highly
interested
in
in
that
we
needed
to
make
a
decision
on
how
to
store
manifests
in
terms
of
version
control.
What
we
decided
on
for
the
time
being
is
to
store
all
infrastructure
related
stuff
in
a
mono
repo
and
have
developers
resource
manifests
and
co-located
with
their
application
repos.
B
But
we
designed
everything
in
a
way
that
we're
not
fixed
to
this
and
that
we
can
migrate
to
other
paradigms
if
needed
later
on,
and
so
the
phase
after
that
is
the
golden
path
phase.
So
this
would
be
GA
general
availability,
and
this
would
be
where
any
new
service
at
Spotify
would
be
deployed
to
only
kubernetes
and
have
be
able
to
leverage
all
the
native
kubernetes
benefits
like
auto
scaling
I
via
the
vertical
pod
autoscaler.
So
that
would
take
away
some
of
the
capacity
planning
concerns
from
developers.
B
Horizontal
node,
auto
scaling
would
take
away
some
of
our
operator
capacity
planning
concerns
the
one-click
migration
is
kind
of
the
Golden
Goose
here
and
that
we
would
like
to
provide
either
a
button
in
our
like
UI
console
for
everything
called
System
Z
or
a
simple
script,
or
a
CLI
tool
that
would
be
able
to
convert
developers
running
services
on
Helios
to
kubernetes
for
them.
So
they
don't
have
to
follow
some,
like
you
know,
long
series
of
Doc's
or
guidelines.
Ideally
the
everything
else
falls
into
this
category.
B
So,
following
like
the
80/20
paradigm,
the
this
would
entail
us
investigating
the
20%
of
kind
of
tail
end
services
that
have
additional
requirements,
for
example,
IT
GC
services
like
IT
general
control
services,
at
about,
if
I
have
additional
security
and
auditing
requirements,
for
example,
all
SSH
activity
currently
needs
to
be
tracked,
monitored
and
audited.
So
increment
Eddy's.
We
need
to
figure
out.
What
does
this
mean?
Do
we
track
like
cube
CTO
exit?
Do
we
just
not
allow
them
to
do
that?
B
These
are
things
that
would
need
to
go
into
GA
the
we
would
also
do
migration
tracking
as
part
of
this
phase,
so
we
would
actively
track
via
metrics
how
many
services
are
still
running
on
Helios.
How
many
are
migrated
to
kubernetes
the
rate
of
change
from
one
to
the
other
and
hope
to
boost
that
through
things
like
the
migration
road
team?
So
this
is
a
strategy
we
use
during
our
GCP
migration
with
a
lot
of
success,
and
this
was
a
temporary
team
that
was
spun
up
that
visited
office.
B
All
the
offices
that
had
developers
in
them
and
these
this
temporary
team
held
lunch
and
learns
like
GCP
trainings
and
sat
down
with
developers
to
actually
help
them
migrate,
their
services
to
GCP
and
then
at
the
when
we
were
all
migrated
to
DCP
this
team
spun
down.
So
we
hope
to
do
something
similar
for
that
for
kubernetes
and
so
the
takeaways
and
like
lessons
we
had
from
the
migration
process
we've
done
so
far.
B
Is
that
scoping
the
scoping
this
into
stages,
really
helped
us
avoid
like
a
very
monolithic
like
migration
movement,
when
we
were
initially
thinking
about
things
at
the
very
beginning
of
migration,
especially
with
our
long-standing
legacy
stack.
We
were
really
thinking-
oh
my
god,
there's
so
much
stuff
that
we
need
to
integrate
and
adjust
fix
or
like
there's
just
so
much
stuff,
especially
because
we've
been
around
this
legacy.
Instructure
has
been
around
since,
like
2005,
2006
and
whatnot.
It's
a
lot
to
deal
with,
but
designing
each
stage
around
the
type
of
service.
B
There's,
like
scope
of
the
service
that
you
want
to
migrate,
really
helped
us
in
terms
of
discovering
all
these
like
hidden
unknowns
that
we
knew
were
out
there
at
a
very
controllable
pace,
and
so
the
when
we
discovered
these
unknowns.
What
we
would
do
is
basically
consider
does
this
need
to
be
part
of
this
migration
phase,
for
example,
like
secret
secrets
integration
at
the
experimental
stage
when
we
were
running
three
services
like
do
we
need
to
solve
that?
B
No,
we
don't,
let's
put
it
on
a
backlog
and
remember
to
address
that
in
a
later
stage,
the
one
that's
relevant,
and
so
the
caveat
to
that
is
to
when
in
doubt,
leverage
time
box
investigations
to
reduce
risk.
What
we
mean
by
that
is
that
when
we
were
throwing
these
things
on
the
backlog
very
often
some
team
members
would
have
fears
that,
when
we
later
need
to
implement
this
thing,
we're
going
to
have
to
rewrite
a
lot
of
the
existing
stuff
that
we
just
did
for
this
stage.
B
This
current
staging
that
we
kind
of
did,
and
maybe
a
hacky
or
quick
way
so
to
alleviate
those
fears.
We
would
have
a
team
member
basically
take
a
time
box
investigation
and
she
alleviating
that
fear.
So,
for
example,
in
our
spinnaker
integration
work
for
the
beta
phase,
we
were
seeking
to
do
a
simple
deployment
pipeline
and
Gi
phase,
we're
looking
to
support
Canaries
Bluegreen
deployments
more
complex
stuff,
and
so
we
were
worried
that
the
integrations
we
were
doing
with
spinnaker
for
beta.
B
We
would
have
to
rewrite
for
the
GA
phase,
so
a
team
member
took
a
day
really
dug
into
spinnaker
pipeline
design.
They
manually
created
a
canary
and
blue
green
appointment
pipeline
and
they
vetted
that,
okay,
what
we're
currently
doing
with
spinnaker
well,
that
was
to
easily
support
complex
strategies
later,
so
we
greatly
emitted
a
mitigated
that
risk
there,
and
so
that's
our
equipment,
IDs
migration.
So
far,
that's
the
runtime
side.
Dave
is
going
to
talk
about
the
protocol
side
of
things
so.
A
A
None
of
the
great
l7
features
we
can
get
from
cloud
features
like
cloud
load,
balancers
work.
We
can
use
a
service
mesh,
but
we
don't
get
any
of
the
cool
l7
any
of
the
protocol.
Inspection
features
of
that
we
obviously
don't
get
any
free
features
from
someone
else
doing
work
on
the
project,
it's
harder
to
onboard
new
people,
it's
harder
to
onboard
new
languages,
it's
harder
to
get
new
P
to
get
kind
of
debugging
deep
into
the
protocol.
If
you
see
some
messages
need
to
figure
out.
A
What's
going
on,
you
have
to
kind
of
dig
through
the
code
as
well,
just
because
there
really
aren't
great
tools
aside
from
the
ones
we
built
when
we
had
some
issues,
so
we
needed
something.
So
we
really
want
to
switch
protocols,
but
on
the
other
hand,
it's
really
hard
when
you
have
say
a
thousand
production
micro
services
that
are
all
communicating
instant
protocol
to
just
flip
a
switch
and
make
them
all
speak
a
different
protocol.
A
So
the
similar
to
the
to
the
kubernetes
migration
we're
trying
to
figure
out
how
to
stage
this-
and
some
teams
have
just
said-
we're
gonna
use
G
RPC,
because
it's
just
better
and
even
if
the
interactive
team
isn't
providing
it,
we
don't
care.
We
want
G
RPC,
we're
working
with
those
teams
to
kind
of
understand
how
they
did
it
to
understand
what
they're
doing
to
understand
how
they're
interacting
with
the
existing
Hermes
services
and
to
create
a
model
for
the
rest
of
the
company.
So
we
can
move
across
there.
A
Obviously,
if
a
whole
bunch
of
our
existing
services
are
still
speaking
Hermes,
any
new
service
will
steena
at
Hermes
client.
But
ideally
we
don't
need
Hermes
servers
anymore,
because
they'll
all
just
serve
in
G
RPC
and
connect
to
the
older
services
or
the
not
yet
migrated
services
over
Hermes
but
still
leaves
a
whole
bunch
of
big
questions.
For
us
things,
like
kind
of
do
we
migrate
all
the
old
services.
Do
we
migrate
some
of
the
old
services?
Do
we
kind
of
not
migrate?
A
Anything
just
give
tools
to
create
new
ones
and
see
what,
if
the
feature
teams
care
about
their
old
ones,
we're
picking
so
far
we're
kind
of
leaning
towards
a
middle
path
of
trying
to
migrate
the
most
critical
services
but
kind
of
give
a
help
in
supporting
the
others,
but
not
necessarily
be
personally
involved
because
they're
over
200
teams
we
have,
and
only
so
many
of
us
at
infrastructure
but
kind
of
I'm,
happy
to
hear
anyone
else's
stories
of
similar
migrations
to
see
what
they've
done
and
what's
worked.
The
other
really
big
thing.
A
We're
trying
to
figure
out
with
my
guruji
RPC
is
schema
management
like
protobuf
schema
management.
With
our
existing
protocol,
we've
found
that
some
of
our
more
critical
services
have
over
a
hundred
different
clients
spread
across
all
of
our
internal
code
bases.
We
don't
really
want
to
have
over
a
hundred
different
versions
of
the
same
protobuf
spread
across
all
of
these
repos.
A
We're
happy
to
again
kind
of
hear
what
other
people
in
a
similar
situation
do
here
and
potentially
work
with
them
to
build
something
good,
and
the
other
thing
thing
everybody's
talking
about
at
this
conference
is
this:
do
we
have
clearly
want
to
use
this?
Do
we
see
a
lot
of
benefits
from
things
that
it
does?
We
have
done
a
few
pocs
but
they've
been
very
simple.
A
A
Then
we
moved
on
to
testing
services,
service
authentication
and
we
still
have
thousands
of
VM,
as
we
will
probably
for
a
very
long
time,
we'll
have
services
in
multiple
different
clusters,
so
we
were
figuring
out
service,
the
service
off
with
services
across
clusters
and
across
VMs
and
found
that
we
can
make
it
work.
But
there
are
a
lot
of
pain
points
so
where
it's
gonna
still
figuring
out
how
to
smooth
those
over
or
try
to
get
it
so
that
it
by
use
do
one
or
by
the
time
that
we
have
time
to
really
deploy
us.
A
Do
we
can
get
them
fixed,
and
the
next
thing
we
really
want
to
investigate
with
this
do
is
tracing
kind
of
how
to
do
a
good
distributed
tracing
with
it
and
then
we're
trying
to
figure
out
kind
of
what
our
next
thing
we're
working
with
the
end
user
community,
we're
looking
at
other
projects,
whether
it's
prometheus
or
some
other
things
that
we
haven't
listed
here.
Looking
at
non
CN,
CF
things,
and
then
there
other
big
things
like.
We
have
no
plan
right
now,
for
it
see
how
to
replace
our
framework
for
Apollo.
A
But
we
don't
really
want
to
have
our
own
framework
once
we're
and
if
kubernetes
and
gr,
pc
and
sto
when
running
with
all
these
community
tools,
so
we're
still
figuring
that
out
and
again
kind
of
we'd
love
to
work
with
people
in
similar
places.
Rather
than
invent
this
on
our
own
and
that's
it
and
we're
happy
to
take
a
few
questions.
If
we
still
have
time.