►
From YouTube: gRPC Proxyless Service Mesh by Gijs Molenaar
Description
As one of the first streaming companies, Spotify has always been at the pioneering frontier of inventing and adopting new technology. But in a changing world and growing as a company, the requirements of our stack require constant improvement and rethinking. In 2022 Spotify started migrating to a gRPC Proxyless Service Mesh architecture, which we reverted due to 'issues'. Currently, we are in the process of rebooting this effort. This talk will be about why the original in-house developed protocol named Hermes and service discovery technology called Nameless are not suitable for our current scale anymore and what we are working on to replace them. I will also address the benefits and risks of migration at such a scale.
A
B
B
I'm
Dutch,
if
that's
not
obvious,
I
work
for
about
10
months
at
Spotify.
It's
a
system,
engineer
and
I'm.
Also,
since
beginning
of
this
year,
Tech
lead
of
the
Hermes
deprecation
I'm,
going
to
talk
a
bit
more
about
what
is
Hermes
and
and
all
the
the
pain
points
we
have
with
this
deprecation
project
before
I
was
a
research
engineer.
B
Also
a
contractor
and
I
work,
mostly
in
science,
specifically
in
astronomy
and
I've,
been
a
lot
in
South
Africa
I
worked
in
radio
astronomy
there
and
in
South
Africa
they've
been
building
a
huge
radio
telescope
called
the
square
kilometer
area
specifically
doing
research
in
large-scale
data
reduction
pipelines.
Containerized
data
reduction
pipeline
so
I
have
a
I,
did
a
PhD
in
that
that's
a
good
idea
to
build
these
pipelines.
That
way
is
a
different
question,
but
that's
why
it's
research
right
so
so
many
talks
have
a
have
have
some
quote
from
a
person
in
it.
B
So
I'm
this
Tech
lead
migration.
Now
so
I
was
looking
around
at
migrations
that
didn't
go
flawlessly
or
or
didn't
go
very
smooth
and
in
science.
There's
many
but
I
don't
want
to
bore
you
with
some
obscure
radio
astronomy,
packets
that
we
try
to
migrate
away.
So
I
want
to
focus
a
bit
on
Python
and
the
migration
for
I.
Don't
know
if,
if
everybody
here
is,
is
old
enough
to
remember
that,
but
I
think
so.
If
I
look
around,
but
the
migration
from
python
2
to
Python
3
didn't
go
very
well.
B
B
So
often,
if
you
might
grade
do
something
you
migrate
to
something
that
has
improved
features
or
new
features
you
want
to
use,
it
could
be
that
there's
better
support,
paid
support
or
there's
a
community
with
people
that
you
can
ask
questions
about
this
new
thing
that
you're
using
it
could
be
compatibility
that
this
new
thing
that
you're
using
is
way
more
compatible
with
all
these
other
things
that
are
also
new.
Now,
your
community
could
be
larger
again
that
you
just
have
more
people
that
you
can
ask
questions
to,
but
also
you
can
hire
people
easier.
B
So
if
you
have
new
people
coming
into
your
company,
you
don't
need
to
explain
all
this
old,
obscure
technology
to
them.
They
already
know
this
technology
and
they
can
start
quicker,
but
there's
also
a
national
resistance
with
within
every
company
that
has
an
older
stack.
It
could
be
that
your
software
is
very
complex
and
it's
very
hard
to
adjust,
and
that's
very
much
the
case
in
radius
front
of
me,
where
you
have
these
very
old
packages
from
from
the
70s
written
in
Fortran,
and
you
need
to
modify
that
and
there's
not
very
many.
B
Many
people
that
can
do
that.
It
could
also
be
that
you
have
a
lack
of
resources
in
money
or
in
humans
or
in
time.
That
is
just
not
a
priority
now
to
migrate,
and
we
see
that
a
lot
within
Spotify
where
we
know
it's
important,
but
it's
just
there's
a
lot
of
other
important
things
happening
that
so
this
migration
does
doesn't
flow
to
the
top,
and
it
could
also
be
that
it's
a
lot
of
people
just
think
it's
good
that
it
is
now.
You
don't
need
to
change
because
it
just
works.
B
You
have
all
these
cool
new
features
and
but
it's
very
hard
to
to
to
to
migrate,
because
it
requires
a
lot
of
changes
and
I
would
argue,
python
is
somewhere,
Python
3
was
somewhere
there
and
on
the
left
there
is
technology
that
is
actually
might
be
easier
to
migrate
to,
but
it
has
similar
features
and
it
just
not
that
interesting
to
migrate
to
and
there's
no
need
to
migrate
to
because
we're
already
sort
of
there.
So
you
want
to
be
somewhere
in
the
middle.
There.
B
Sorry
so
now
I
move
a
bit
back
to
Spotify.
So
Spotify
was
the
first
paid
streaming
platform
and
they
were
pioneering
in
this
field
in
that
sense,
and
they
were
one
of
the
first
that
solved
this
problem,
or
they
were
the
first
that
solved
that
problem
and
they
needed
tools
that
didn't
exist.
Yet
so
I
had
to
make
these
tools
with
that
comes
a
great
creative
freedom
right.
You
can
do
it.
You
can
make
this
because
there's
no
there's
nothing
there
yet
and
very
smart
Engineers
before
us
working.
B
There
made
all
kind
of
very
cool
things
in
encryption
in
online
streaming
compression
and
it
was
all
very
much
state-of-the-art
technology.
What
they've
built
and
the
tradition
within
the
company
was
always
to
adopt
the
latest
theoretical
things
available
to
us
and
and
and
utilize
that
to
its
full
potential,
and
it's
the
same
with
the
microservice
architecture.
Right
as
soon
as
Docker
was
invented
or
became.
Popular
Spotify
was
one
of
the
first
to
actually
go
full
on
on
this.
This
kind
of
Technology.
B
So
Hermes
is
a
protocol
invented
by
Spotify
long
time
ago,
and
it
behaves
a
bit
like
HTTP.
But
HTTP
was
just
not
good
enough
and
HTTP
2
didn't
exist.
Yet
hp2
solves
many
of
the
problems
that
Spotify
saw
with
HTTP
for
back
backend
to
back-end
communication.
So
it's
home,
built
built
on
xero,
mq
and
protobuf
Proto
buff
back
then,
was
also
just
released
by
Google
version
two
and
it
supports
bundling
multiple
requests
in
out
of
order.
So
this
gives
way
less
overhead
and
gives
better
performance
and
and
saves
money
and
and
data
usage.
B
Another
technology
we
made
ourselves
is
called
nameless
and
this
is
a
service
Discovery
service
and
it's
based
on
DNS
server
records
and
DNS
service
records
are
not
that
popular
but
they're,
very
similar
to
cname
records
that
they
create
a
list
of
of
host
names.
If
you,
if
you
resolve
a
service
but
also
include
a
port
number,
so
all
internal
Spotify,
Services
register
per
region
with
a
nameless
instance
and
then
other
services
can
query.
This
nameless
servers
to
to
get
all
the
hosts
that
offer
a
specific
service,
and
then
we
utilize
client-based
load
balancing.
B
So
the
client
decides
which
service
is
going
to
be
picked,
which
is
usually
around
Robin
nameless
operates
per
region,
so
we
have
multiple
regions
around
the
world
and
but
if
some
services
for
some
reason,
don't
run
in
a
specific
region,
then
we
can
manually
redirect
traffic
from
one
region
to
the
other.
But
this
is
a
manual
task
semi-manual.
B
It's
also
important
to
note
that
nameless
Works
across
different
environments
and
that
I
mean
it
works
with
classic
VMS
and
kubernetes
hosts
at
Google
or
at
AWS.
It's
completely
independent
because
it's
DNS
and
the
third
one
is
Helios,
which
is
our
own
Homebrew
container
platform,
and
we're
very
proud
to
to
mention
that,
and
we
actually
recently
discovered
this
that
a
colleague
of
mine
did
some
archeology
in
in
programs
of
Docker
conferences,
and
we
discovered
that
Helios
was
actually
announced
one
day
before
kubernetes
was
and
back
then
functionality
functionally.
They
were
very
similar.
B
They
are
a
platform
where
you
can
schedule
containers,
you
can
schedule
them
together
and
you
can
expose
them
on
the
network
and
they
have
an
API,
but
quickly
communities
just
became
way
more
popular
and
took
over
so
but
up
up
to
today,
Helios
is
still
used,
but
we're
migrating
away
from
it.
So,
but
these
Technologies
brought
us
where
we
are
now
we
have
a
a
worldwide
company
leading
the
streaming
business
and
we're
everywhere,
but
still
the
all.
This
technology
just
cannot
keep
up
with
with
the
growth
we're
seeing.
B
Another
problem
is
that
all
these
things
come
with
overhead
right,
so
you're,
the
first
mover
you
create
something
you
make
it,
but
then
you
have
other
people
learning
from
your
mistakes,
so
they
create
something
that
is
better
and
other
people
start
using
it.
That
has
more
momentum
and
then,
before
you
know,
you
have
your
own
do-it-yourself
technical
depth.
Also
the
company
grows.
It
keeps
on
growing
and
all
the
technology
that
we've
built
ourselves
only
go
so
far
and
keep
on
adjusting,
so
it
can
keep
up
with
the
scale.
B
Many
of
the
tools
that
we
created
doesn't
have
proper
third-party
support.
For
example,
the
protocol
there's
no
package,
never
that
understand
that
offer
great
support
for
for
something
like
Hermes
and
many
new
developers,
including
myself,
are
just
not
familiar
like
I
joined
Spotify
10
months
ago
and
I
have
to
learn
all
of
these
things
a
reverse
engineer,
my
understanding
of
something
like
Helios,
so
I
want
to
move
away
from
Helios
and
it
I
mean
I
already
explained
it.
B
B
B
Another
problem
we
have
is
the
crosstown
traffic
problem.
So
if
you
go
to
your
favorite
cloud
provider
and
you
create
a
new
kubernetes
cluster,
they
advise
you
to
just
have
three
zones
and
fire
up
some
VMS
in
that
in
those
zones
and
and
don't
really
worry
about
that
and
just
randomly
deploy
your
putts
over
it,
and
there
is
some
cost
involved
for
interzone
traffic,
which
is
about
like
one
cent
per
gigabyte
and
which
all
sounds
fine,
but
this
becomes
really
a
problem.
B
So
we
want
to
save
money
also
and
so,
and
we
need
sonaware
routing.
So
this
was
hard
to
do
with
nameless,
but
then
I
had
a
colleague
who
was
very
smart
and
he
hacked
Hermes
to
hack
in
zone,
aware
routing
and
we're
actually
deploying
this
as
we
speak,
we're
saving
money
as
we
speak.
We
immersed
a
couple
of
pull
requests
yesterday,
but
we
kind
of
keep
on
doing
this.
We
cannot
keep
on
hacking
these
protocols,
it's
it's.
While
there
is
more
robust,
future-proof,
Technologies
available
so
now
to
service
mesh.
B
So
for
people
who
don't
know
what
is
service
mesh
service
mesh
is
a
different
way.
You
can
manage
traffic
going
through
your
cluster,
where
traditionally
I
mean
you
have
your
kubernetes
clusters
and
your
pods
and
they
have
one
service.
You
want
to
communicate
with
another
service,
they
could
just
do
that.
You
say:
okay,
there's,
it
has
a
a
name
you
can
resolve
and
then
maybe
there's
a
couple
of
parts
service
Mass
introduces
another
abstraction
layer
in
this
whole
Contraption,
where
it
has
a
dedicated
infrastructure
layer,
typically
manifested
in
the
form
of
a
proxy.
B
So
traditionally,
a
service
mesh
has
been
realized
with
a
proxy,
but
the
dream
is
the
the
the
the
proxylist
service
mesh
and
that's
the
where
we're
heading
to
so
what
you
can
do
you
can
just
get
rid
of
this
proxy
and
but
for
to
do
this,
you
need
a
client
library
that
is
able
to
communicate
with
a
control
plane
now.
Luckily,
all
the
modern
grpc
libraries
basically
have
support
for
this.
B
So
now
it's
just
a
matter
for
us
to
start
switching
to
grpc
fully
so
traffic
directory
is
the
managed
service
for
service
mesh
by
Google
in
case
that
you
didn't
know
that
an
alternative
is
istio,
I,
hope
I
pronounce
it
right.
I
always
get
a
bit
confused
there,
and
so
you
might
ask
why
don't
we
use
that
so
the
problem
with
istio
is
that
it
has
multi-cluster
support.
So
we
have
many
clusters
per
region,
but
we
need
better
support
for
that.
B
More
advanced
routing
techniques
that
traffic
director
actually
has
support
for
also
we
have
Global
routing,
so
we
need
certain
services
are
really
for
legal
reasons,
for
example,
bound
to
certain
regions
and
traffic
needs
to
be
redirected
in
between,
and
also
we're
still
not
fully
converted.
Yet
we
still
have
significant
infrastructure
running
on
Helios.
B
So
why
would
you
want
to
go
proxyless?
Well,
you
just
have
less
moving
parts
right,
you,
you
don't
have
your
proxy,
you
don't
have
this
mental
overhead
and
we
actually
already
do
this.
We
already
do
it
as
I
mentioned
before
we
do
this
client-side
load,
balancing
where
we
have
our
Hermes
client.
That
gets
these
nameless
records
in
and
decides
who
to
talk
to
it's.
Just
it's
a
bit
stupid
this
way
we
can
make
it
a
bit
smarter.
B
So
this
whole
architecture
is
philosophically
compatible
with
what
we've
already
been
doing
and
everybody
is
reasonably
comfortable
with
it.
So
what
is
Stopping
Us
in
this
whole
story,
so
the
Hermes
grpc
migration
is
something
we
are
slightly
underestimating.
It's
actually
quite
hard
to
migrate.
A
protocol.
The
I
I
mean
I've
not
been
involved
with
the
Helios
deprecation
effort,
but
that
seems
way
more
easy.
Where
you
just
have
a
container,
you
move
it
to
a
different
platform.
B
While
here
you
need
to
modify
the
content
of
the
container
and
also
the
protocol
that
that
the
containers
talk
in
between
and
so
there's
there's
reasonable
resistance
within
the
company
natural.
It's
not
that
people
don't
want
to,
but
it
is
just
it
takes
time
and
effort
to
move
and
and
help
all
these
teams
and
all
these
services
to
integrate
a
new
protocol
in
there
in
their
infrastructure.
B
In
combination
with
that,
there
was
a
bug
in
a
library
that
we
used
that
propagated
the
error
even
further.
The
result
was
that
basically,
every
service
within
Spotify
started
returning
404
and
it's
a
security
measure.
Ever
all
the
clients
were
locked
out,
so
we
have
like
a
perimeter.
Security
check
and
people
start
accessing
things
that
they're
not
allowed
to.
They
get
locked
up
so
I,
don't
know
if
you
ever
realized,
but
one
of
the
things
with
Spotify
is
that
you're
never
locked
out.
B
So
now,
afterwards,
we
sat
down
with
Google
and
we
said
like
look,
we
really
like
traffic
director.
We
want
to
use
it
again,
but
we
need
to
prevent
the
happening
there.
So
Google
sat
down
and
looked
at
what
happened,
and
they
came
up
with
a
couple
of
solutions
to
solve
to
address
these
issues,
and
one
of
that
is
just
make
it
harder
to
make
this
mistake
not
to
push
a
faulty,
faulty
faulty
configuration
or
break
the
configuration.
B
B
Gpc
is
a
big
pickle
and
we
should
not
underestimate
the
migration
and
just
asking
people
internally.
It
doesn't
work
either.
Spotify
is
a
community
of
people
and
just
asking
nicely
it
doesn't
help
you.
You
need
to
create
a
bit
of
an
incentive.
So
that's
what
I'm
working
on
now
I'm
not
expecting
Miracles,
but
I
just
want
to
make
it
the
obvious
next
way
and
we're
working
on
that
with
certain
teams.
We
slowly
migrating
very
important
services
and
hoping
that
has
a
sort
of
network
effect
within
the
company.
B
B
B
C
Hey
guys,
thanks
for
the
talk
back
in
the
days
before
service
mesh
existed,
Netflix
had
a
library
called
histrix
and
I
moved
away
with
it,
because
it
was
forcing
them
to
do
everything
in
Java
and
use
that
library
and
cause
tight
coupling
now
you're
moving
in
exactly
the
same
direction.
What
has
changed.
B
Right
yeah
grpc
is
a
protocol
with
implementations
in
almost
all
languages,
so
we're
not
tightly
coupled
to
Java
and
I
mean
we're
investigating
other
languages.
Also
we're
fully
Java
based
now
with
some
python
here
and
there,
but
the
beauty
with
glpc
is
that
you
can
write
these
product
buff
definitions
and
you
can
generate
code
in
almost
every
language.