►
From YouTube: How Spotify Migrated Ingress HTTP Systems to Envoy - Erica Manno & Vladimir Shakhov, Spotify
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Spotify Migrated Ingress HTTP Systems to Envoy - Erica Manno & Vladimir Shakhov, Spotify
Erica and Vladimir are on the team responsible for perimeter systems that sit between Spotify’s clients and its backend services. They started unifying those systems from a range of different technologies and protocols to a solution based on Envoy proxies and a unified control plane. This talk introduces Spotify’s vision for the next-gen perimeter. However, it will mainly focus on the migration of all HTTP ingress traffic, handled by a brittle, custom Nginx/HAProxy setup to an Envoy-based solution. The speakers will discuss how they’re migrating multiple high volume web services, serving millions of requests/sec, with minimum disruptions and zero-downtime for the feature teams that maintain Spotify’s backend services. This talk will also illustrate how Spotify’s engineering culture of loosely coupled but highly aligned teams has informed the decisions taken during the migration.
https://sched.co/UaZH
A
So
my
name
is
Erica
and
we
are
software
engineers
at
Spotify.
We
work
in
the
ATC
team
and
that's
the
team
that
owns
and
operates
Spotify
perimeter.
The
perimeter
is
on
the
path
of
each
and
every
communication
between
clients
and
Spotify
back-end,
and
so
for
that
reason
it's
a
very
critical
piece
of
infrastructure,
as
you
can
imagine,
Spotify
is
adopt
an
opt-in
teams
model,
meaning
they're,
a
team
that
developed
a
service
is
also
responsible
for
operating
it
24/7
in
production,
and
our
team
is
no
different.
A
A
We
are
here
today
because
we
want
to
share
with
you
how
we
migrated,
apart
of
Spotify
perimeter,
to
an
employee
based
solution.
Of
course,
we
will
explain
how
we
did
it
and
also
share
with
you
what
worked
well
during
this
work.
What
didn't
so
the
lessons
that
we
learned
along
the
way
and
also
we
will
briefly
talk
about
our
vision
for
Spotify
perimeter
moving
forward.
A
A
Spotify
has
248
million
monthly
active
users,
making
it
the
most
popular
audio
streaming
subscription
service.
As
you
can
imagine,
all
those
users
generate
quite
a
bit
of
traffic,
but
just
to
give
you
a
more
precise
idea
of
the
volumes
we're
talking
about
at
peak
Spotify
Perimeter
handles
more
than
8
million
requests
per
second,
as
fortifies.
Beckon
is
a
distributed
system
and
it
has
a
micro
service
architecture.
We
have
more
than
a
thousand
micro
services
deployed
in
production.
A
A
Of
course,
some
of
these
services,
like,
for
instance,
the
login
service
or
search
or
playlist,
have
to
be
exposed
to
the
public
internet
so
that
your
smartphone
or
whatever
other
device
out
there
can
actually
talk
to
these
services
when
you're,
using
it
Spotify
and
that's
exactly
where
the
Perimeter
comes
in.
It
allows
exposure
of
back-end
services
to
the
internet
so
that
clients
can
communicate
with
them,
but
it
also
acts
as
kind
of
a
wall
protective
wall
between
Spotify
back-end
and
the
Internet.
But
let's
now
zoom
in
a
little
bit
more
on
the
perimeter
itself.
A
This
diagram
is
somewhat
of
a
simplified
view
of
our
perimeter,
but
it
still
kind
of
serves
to
illustrate
the
main
use
cases
for
the
perimeter.
The
first
one
is
represented
by
the
vertical,
on
your
left
hand,
side
and
it's
to
basically
allow
HTTP
clients
on
the
Internet
to
communicate
with
back-end
services
that
only
understand
hermès.
A
Now,
if
you've
never
heard
of
or
miss
before
that
totally
fine,
because
it
proprietary
protocol
that
Spotify
developed
back
in
2012
and
though
it's
now
being
phased
out
in
favor
of
JRPG,
we
still
have
a
lot
of
services
that
use
it.
So
the
perimeter
has
an
HTTP
to
Hermes
gateway,
just
dedicated
to
this
use
case,
but
spotty
voice
back
and
also
has
quite
a
bit
of
services
that
talk
HTTP.
So
in
that
case
the
perimeter
has
a
web
gateway
that
you
can
see
in
the
middle
of
the
diagram.
A
For
these
HTTP
clients,
HTTP
services
use
case,
and
finally,
one
more
use
case
by
the
perimeter
is
to
allow
persistent
connections
between
Spotify,
back-end
and
clients
and
that
to
allow
state
changes
that
originate
in
the
backend
to
be
pushed
out
to
all
the
connected
clients.
So
a
typical
example
of
this
would
be
if
a
new
track
is
re
to
a
Spotify
playlist.
A
For
example,
this
talk
is
going
to
focus
on
the
part
of
the
perimeter
that
is
allotted
by
the
red
box,
so
the
HTTP
clients,
HTTP
services
use
case
and
it's
going
to
cover
how
we
removed
or
si
replaced
this
part
of
the
perimeter
with
an
ember
based
solution.
Now,
Wladimir
will
zoom
in
a
little
bit
on
this
red
box
and
explain
what's
inside.
B
B
B
Each
pod
pepper
an
updated
configuration
file
for
each
proxy
with
NPS
and
ports
for
the
backends.
When
sports,
which
were
per
wanted
to
expose
a
website
or
an
HTTP
upstream,
they
made
a
poor
request
to
a
puppet
get
rapper
in
this
single
configuration
file.
So,
let's
see
if
there
was
something
to
improve
the
parameter
had
some
flaws
and
those
became
apparent
as
in
the
past
speed
5
prioritized
mobile
traffic
over
the
web
traffic
and
that
made
up
a
perimeter
kind
of
ops
kind
of
an
afterthought
compared
to
the
traffic
which
walker
was
coming
from.
B
B
A
B
B
Our
perimeter
was
cosmetic
rate.
It
was
like
a
lot
of
instant.
It
gets
to
28
in
the
role
it
was
mentioned
in
30.
It
was
difficult
to
troubleshoot
because
of
poor
observability
on
the
SSP
upstream
we
could
evolve
the
all
primitive
fix.
The
service
cavalry
and
I
kept
the
relevant
features,
but
that
would
mean
that
we
would
invest
only
in
the
SSP
to
access
LeapPad
part
of
the
parameter
that
Erica
told
us
about,
since
it's
obvious,
where
are
we
going
with?
B
This
looks
like
we're
a
migration
from
it,
which
also
in
five
perimeter
and
the
one
awesome
gateway
doing
all
things
perimeter
and
based
on
Android
and
coded
edge
all
our
protocols.
This
video
here
means
pure
SB
and
JC
would
need
the
similar
features
like
authentication
organization,
security
features
and
metrics,
and
so
on
and
so
on.
It
makes
a
lot
of
sense
to
implemented
once
so
immigration
sure,
but
like
why
I'm
going,
which
way
grow
is
becoming
more
and
more
cognitive.
B
B
The
perimeter
is
far
from
specific
or
business,
so
it
makes
a
lot
of
sense
to
use
a
cloud
to
use
a
CNC
F
baked
one
which
has
taken
off
as
a
standard
already
like
it's
kind
of
an
easy
choice
to
make
exist.
Protocol
allows
us
to
configure
our
invoice
with
hello,
Java
control
plane,
which,
with
lots
of
tests
using
abstractions
and
using
Spotify
domain
terminology,
this
will
allow
us
to
provide
a
bliss
set
of
use
cases
to
the
users
between
developers.
B
B
So
this
is
edge
of
new
perimeter
system.
Drone
is
a
set
of
boxes.
The
bigger
blue
box
is
the
hvm
on
the
VM.
There
are
like
a
few
docker
processes
and
the
first
one
and
the
main
one
is
anyway
this
into
space
traffic,
asbestos
abyss,
traffic
coming
from
the
Google
HTTP
a
load
balancer
handles
both
PC
traffic
and
participate
in.
Why
issue
incoming
requests
is
caught
by
invoice?
B
Extol
filter
which
passes
to
ex-old
service,
which
runs
on
the
same
host,
which
owns
service
checks,
the
authentication
we
currently
only
support
dealer
with
tokens,
and
also
it
does
decoration
of
the
incoming
requests
with
like
geolocation
date
it
much
as
I
purposes
of
the
of
the
users
to
the
countries
and
cities.
Where
are
they
coming
from
o
in
what
instances
are
configured
by
our
control
plane,
cluster
visual
PC
control
service
which
we
implement.
B
A
Right
so
pretty
early
on,
we
identify
the
core
major
requirements
for
this
migration
work.
First
of
all,
it
shouldn't
create
any
additional
work
for
the
owners
of
the
upstreams
being
migrated.
The
main
idea
here
is
that
product
teams
in
Spotify
should
be
able
to
stay
focused
on
developing
new
product
features,
and
the
switch
to
a
new
parameter
should
be
completely
transparent
to
them.
A
Third,
the
migration
should
happen
gradually
and
that's
for
mainly
two
reasons
up
until
the
point
edge
actually
served
virtually
no
traffic.
So
while
we
were
pretty
sure
that
it
was,
but
already
we
didn't
actually
have
any
metrics
to
prove
that,
so
we
kind
of
needed
to
build
some
evidence
in
the
new
perimeter
if
you
will
and
also
we
wanted
to
minimize
impact
on
end-users
in
case
of
unforeseen
problems
with
edge.
A
And
lastly,
we
should
have
a
quick
fall
back,
so
if,
during
the
migration
at
any
point,
something
goes
wrong
and
the
users
are
impacted,
then
we
want
to
have
a
way
to
revert
back
to
a
working
state,
and
that
must
happen
quickly
within
minutes.
So
we
kind
of
want
a
big
panic
button
that
we
can
push
any
time
we
even
suspect
we're
in
trouble.
Given
these
poor
requirements,
we
came
up
with
a
migration
in
three
different
stages.
So
now
we're
going
to
talk
about
each
of
the
stages.
A
A
These
Pacific
gateway
serves
on
the
traffic
for
experimental
features
or
features
that
are
under
development
and
makes
no
guarantees
about
its
availability
or
reliability.
So
it's
kind
of
less
critical
compared
to
similar
gateways
that
serve
production
traffic.
Also,
we
ourselves
own
that
that
gateway.
So
what
kind
of
stakeholders
of
this
first
stage,
so
we
don't
have
to
coordinate
or
interact
with
other
teams.
A
For
this
reason,
during
this
particular
stage
of
demography,
decided
to
deviate
a
little
bit
from
our
requirements
and
to
optimize
for
speed
over
safety,
we
decided
to
switch
traffic
over
to
add
for
a
global
DNS
change
and
yes,
clearly
that
was
a
bit
of
a
risky
move,
but
we
thought
that
being
able
to
move
quickly
through
this
stage
was
actually
worth
that
risk.
This
initial
stage
proved
useful,
so
the
degree
kind
of
showed
that
edge
was,
you
know
good
enough
for
for
traffic
with,
but
already
it
also
uncovered
the
bug
in
our
ex
tops
implementation.
A
So
we
kind
of
fix
this
before
moving
on
to
the
next
stage.
The
main
objective
of
this
second
stage
was
to
put
some
real
production
traffic
on
edge.
We
settled
on
migrating
the
HTTP
upstream
for
Spotify
Web
Player.
If
you're
not
familiar
with
it,
it's
hosted
at
open,
dot,
Spotify
comm
and
it's
what
you
would
use
to
browse
or
play
your
Spotify
music
on
Chrome
or
Safari,
for
example.
A
A
A
After
some
discussing
and
brainstorming,
we
came
up
with
an
iterative
strategy
that
consisted
in
shifting
traffic
to
edge,
shifting
progressively
more
and
more
traffic
to
edge
the
monitoring
and
making
any
adjustments
along
the
way,
if
necessary,
and
then
repeating
until
100
percent
of
the
web
player,
traffic
was
moved
to
edge
since
there
were
external
stakeholders
impacted
by
this.
We
also
had
to
take
into
account
some
coordination
work,
but
let's
see
how
this
happened
in
practice,.
A
So
this
first
diagram
illustrates
the
set
up
for
the
web
player
on
the
old
web
gateway.
So
before
any
of
this
migration
happened
and
we
call
the
old
web
gateway
well
being
wobble
beam.
So
that's!
What's
in
the
so
in
the
top
right
corner
we
have
the
web
player
upstream,
which
is
configured
to
be
exposed.
A
A
A
So,
as
I
said
earlier,
our
strategy
was
to
progressively
shift
more
and
more
of
the
web
player
traffic
to
edge,
but
this
original
setup
couldn't
really
get
us
there.
It
needed
some
tweaking
so
what
to
do.
Well,
we
considered
several
different
options,
but
in
the
end,
the
one
that
kind
of
fit
most
naturally
with
this
setup
that
we
had
made
use
of
the
Google
cloud,
the
HTTP
load
balancer.
A
So
let's
see
how
we
use
that
in
this
second
drive
diagram,
I
have
colored
in
blue
the
things
that
are
either
new
or
different
compared
to
the
original
set
up.
So
the
very
first
thing
we
did
is
that
we
stood
up
an
HTTP
load,
balancer
in
front
of
the
web
will
be
and
proxy
in
traffic
to
it.
It's
the
blue
cloud
that
you
see,
on
the
right
hand,
side
of
the
diagram
to
minimize
risk.
Initially
we
modified
our
DNS,
just
a
big
configuration
so
that
only
Sweden
would
be
resolved
to
the
new
load.
A
So
at
this
point
note
that
we
still
don't
have
any
traffic
going
twedge
what
I
just
described,
was
kind
of
a
preparatory
step
in
that
direction.
So,
let's
see
what
we
did
next
so
first
off,
we
added
a
new
back-end
for
edge
behind
the
HTTP
load
balancer,
and
so
at
this
point
we
had
to
pay
backends
behind
them.
We
had
one
for
the
old
web
gateway,
the
web
Alby
and
a
new
one
for
edge,
but
no
traffic
was
sent
to
edge
just
yet.
A
A
So,
to
recap,
the
strategy.
First,
we
stood
up
a
new
HTTP
load
balancer.
Second,
we
gradually
resolved
all
of
the
web
player
traffic
to
this
new
load
balancer
and
lastly,
we
added
an
edge
back-end
to
sit
behind
this
new
load
balancer
and
use
the
capacity
scaler
to
gradually
ramp
up
traffic
to
the
new
edge
back-end.
A
Initially,
we
routed
traffic
to
edge
only
during
office
hours,
we
monitored
the
performance
of
edge
under
load
and
we
tuned
their
alerts
so
that
they
would
have
sensible
thresholds
for
things
like
resource
usage
or
latency.
We
also
created
the
Kingdome
alert
that
would
probe
open
dot,
Paulo
Filho
comm
for
low
latency
and
for
availability,
so
that
in
case
of
issues
with
the
web
player,
we
would
be
notified
and
page
pretty
quickly.
After
a
few
days
of
this
dialing
up
and
down
the
traffic
on
edge,
we
felt
confident
to
live
here
on
24/7.
A
A
A
If
we
found
that
there
was
no
strong
use
case
for
a
particular
configuration
option,
then
we
just
simply
dropped
it.
Our
main
objective
in
this
case
was
to
keep
the
configuration
API
for
edge
as
simple
and
streamlined
as
possible.
To
give
you
a
practical
example,
the
old
web
gateway
allowed
the
exposure
of
both
HTTP
and
HTTPS
endpoints.
However,
we
decided
to
drop
support
for
HTTP
endpoint
in
edge
as
a
way
to
encourage
better
security
practices
moving
forward.
A
Also,
since
we
had
a
lot
of
upstreams
to
migrate,
we
wrote
a
Python
script
that
would
parse
the
configuration
on
the
old
perimeter
and
translate
it
to
edge
configuration
the
script
covered.
75
percent,
roughly
of
the
upstream
for
the
remaining
25,
we
relied
on
manual
translation
as
a
process
of
peer
review.
Once
the
configuration
were
ready,
then
we
were
good
to
go.
We
just
simply
repeated
what
we've
done
for
the
web
player,
but
this
time
on
a
bigger
scale,
because
we
were
shifting
the
traffic
to
edge
for
all
of
the
130
app
streams
at
once.
B
B
We
sent
agent
Jay
Jay
email
just
before
specify
how
quick
that
meant
that
we
encouraged
Oh,
which
way
teams
to
expose
the
high
peak
services
on
our
new
system
as
much
for
her
quick
is
deterrence
which
we
were
held.
Innovations
like
the
discovery
weekly
is
the
thing
that
was
implemented
during
her
quick
and
the
social
listening
is
the
thing
that
allows
you
to
share
the
play
cue
with
your
friends
to
listen
together.
It's
it's
also
started
as
a
quick.
B
So
using
Android
turned
out
to
be
a
good
thing
for
us,
there
were
bugs
and
the
sisters,
but
we
never
blocked.
There
was
always
a
way
forward.
So
we
didn't
have
a
lot
of
experience
before,
but
the
dogs
are
good
and
we
read
the
source
and
how
team
members
committed
the
poor
requests
and
pushed
what
is
more
important
than
successes
failures.
Obviously
we
had
few
bugs
and
missed
a
few
things.
The
zero
downtime
promise
was
not
fulfilled
completely.
Examples
would
be
our
better
script
missed
a
few
as
edge
cases,
and
few
services
were
overexposed.
B
The
pin
that
for
a
few
days
we
were
it
so
instead
of
returning
in
four
threes
from
the
perimeter
for
some
parts,
we
let
the
don't
stream
back
up
stream
back
and
handle
it.
Nothing
bad
happened
through
and
it
was
later
caught
by
a
scanner
and
humans,
I
guess
the
the
example
would
be
to
be
mr.
protocol.
Somebody
in
the
old
gateway
was
easy
WebSockets,
but
it
was
like
completely
not
clear
from
the
reason
the
configuration
and
our
first
session
didn't
support
no
seconds
at
all.
B
So
there
is
a
software
in
trippy,
it
doesn't
makes
things
easier
and
the
whole
system
had
lots
of
entropy.
There
were
sites
that
were
exposed,
the
people
not
used
and
there
were
like
30
of
them,
I'm,
pretty
surprised
at
the
number
of
sizes.
So
well
also
there
was
officials
that
were
used
by
tiny
fraction
of
websites
and
also,
for
example,
officials
like
WebSockets,
were
used,
but
they
will
not
explicit
advertised
that
they're
being
used
so
I
guess.
The
takeaway
here
is
to
design
for
food
entropy
design
against
the
unused
things.
B
B
B
B
B
B
B
A
Just
before
concluding,
we
wanted
to
share
the
four
key
takeaways
from
all
this
migration
work,
so
the
first
one
would
be
expect
things
to
go
wrong.
Sorry,
I'm,
president
second
I
expect
things
to
go
wrong
and
be
prepared
for
it.
Most
importantly,
so
if
you're
migrating
at
scale
and
adopting
a
new
technology
that
you're
not
familiar
with
as
well
as
our
case
with
envoi
most
likely,
you
will
bump
into
problems
along
the
way.
A
If,
on
top
of
that,
the
stakes
of
the
migration
are
high,
then
definitely
you
want
to
have
a
way
to
roll
back
to
a
working
state,
so
the
takeaway
would
be
invest
in
a
good
panic
button
and
also
think
about
how
granular
should
be
with
a
global
one.
Be
enough
for
do
you
need
a
per
service
one.
A
second
key
takeaway
would
be
work,
iteratively,
start
small
and
possibly
non-critical.
Maybe
fail,
quickly,
learn
and
adjust
and
then
go
progressively
bigger
and
more
critical.
A
This
helps
a
lot
when
you
are
migrating
a
big
number
of
services
over
the
span
of
a
few
months.
It
allows
you
to
quickly
learn
about
the
new
system
you're
migrating
to
and
also
to
uncover,
bugs,
while
minimizing
the
impact
on
users.
A
third
key
takeaway
is
look
for
opportunities
to
reduce
the
migration
surface.
This
is
especially
relevant
if
you
are
migrating
away
from
a
legacy
system
that
is
been
around
for
some
time
and
has
evolved
in
weird
ways
over
time.
A
And,
finally,
when
we
started
all
this
migration
work,
we
thought
it
was
very
daunting
and
scary,
but
it
turns
out
that
giving
yourself
room
to
fail
and
having
a
reliable
on
it
button
to
get
you
out
of
trouble
makes
the
whole
thing
remarkably
pain-free
and
weirdly
enough
kind
of
fun
all
right.
So
this
is
out.
Thank
you
very
much
for
taking
the
time
to
listen
to
us
today,
we're
badly
out
of
time.