►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Story of Why We Migrate to gRPC and How We Go About It - Matthias Grüter, Spotify
At Spotify, we have historically built services based on our own proprietary messaging protocol and framework. Last year we finally kicked off the daunting multi-year task of migrating everything to gRPC. Migrating over 1000 services to gRPC poses interesting challenges, many of them are not only technical in nature: they boil down to questions of engineering culture and leadership at scale: How do you get 200 autonomous engineering teams to align on something as fundamental and cross-cutting as a new RPC framework? How do you roll-out gRPC at scale whit minimal disruption to both the organization and to the end-user? This presentation will address these questions alongside more technical discussions of advanced gRPC concepts such as interceptors, deadlines, and effective schema management and how they are essential in large distributed systems.
https://sched.co/MPY3
A
A
Well,
of
course,
we
are
infrastructure
developers,
so
we
have
to
care
and
also
it's
always
interesting,
to
talk
about
the
like
the
nitty
gritty
details
of
technology.
So,
and
what
also
happens
is
that
those
kind
of
fundamental
things
they
kind
of
leak
out
and
the
abstraction
breaks,
so
we
do
have
to
care
in
the
end.
So
I
would
like
to
talk
about
that
a
bit.
A
My
name
is
Matthias
and
I
work
at
Spotify
I
work
as
an
engineering
manager
in
the
infrastructure
group,
and
in
that
group
we
try
to
migrate
the
organization
from
from
something
else
to
cheer
PC
as
a
manager
by
trade
and
engineer
by
heart,
I
love
to
work
in
the
intersection
between
technology
and
leadership.
So
in
that
talk,
I'll
talk
about
those
two
things,
because
it's
not
only
technically
challenging
to
move
everything
over
the
GRP
see
it's
also
a
big
cultural
change
that
has
to
happen,
and
that's
that's
where
leadership
comes
in
yeah.
A
So
why
are
we
adopting
share?
We
see
it.
It's
modified,
there's
really
a
first
half
of
the
talk
I'm
going
to
talk
about.
Why
exactly
we're
doing
that
and
then
the
second
one
about
how
we
go
about
it
yeah
so
do?
Why
really
is
two
has
two
parts?
Why
do
we
need
to
change
from
what
we
have
today
and
yhe
RPC?
So
why
are
we
changing?
Why
do
we
need
to
change
whatever
we
have
today?
So
today,
a
little
bit
about
our
infrastructure.
A
We
at
Spotify
we
serve
quite
a
lot
of
people
wanting
to
listen
to
music
or
audio,
and
we
have
a
lot
of
server.
Is
that
interact
with
each
other
in
the
backend
at
the
time
of
I,
wrote
those
lights
they're
about
two
and
a
half
thousand
services,
interacting
with
each
other.
We
do
hand
a
lot
of
a
lot
of
requests
coming
into
our
ingress
and
we
also
have.
We
are
a
lot
of
developers
working
on
it
right.
A
For
into
communication
between
the
services,
we
actually
have
our
own
protocol.
It's
called
Hermes,
it's
actually
not
that
Hermes
I
thought
it
was,
but
it's
not
future
I'm.
Sorry
miss
it's
a
Greek,
god
Hermes,
and
it's
like
it's
named
after
the
Greek
god.
That
is
the
emissary
and
messenger
of
gods.
It's
also
funnily
enough.
Hermes
is
also
the
Greek
god
for
the
divine
trickster
and
where
we
are
right
now
it's
qualified
with
Hermes
our
own
homegrown
protocol
and
technology.
A
We
feel
a
little
bit
tricked
a
lot
of
our
own
kind
of
homegrown
solutions,
actually
all
:
named
after
Greek
gods,
so
you
might
have
heard
about
us
talking
about
Helios
our
or
contain
a
crustacean
system
about
a
polo
our
service
framework
and
so
on.
But
we're
not
going
to
talk
about
that
and
Hermes
was
started
in
2012
was
initially
based
on
CRM
q,
the
framing
of
space
in
CM
IQ,
it's
the
payload
is
either
choice
and
chasing
a
pipe
protobuf,
and
it's
actually
not
an
RPC
framework.
It's
just
a
protocol.
A
So
it's
maybe
it's
easier
to
compare
to
it
with
HTTP
and
the
good
thing
with
Hermes.
Is
it
works
and
we've
been
using
Hermes
throughout
our
back-end
since
then,
and
I
hope
many
of
you
use
Spotify
and
you're
able
to
listen
to
music.
So
that's
a
testament
to
that.
Hermes
actually
works.
So
why
are
we?
Why
do
we
need
to
change?
The
problem
is
not
Hermes
in
itself
the
technology.
The
problem
is
that
it
the
fact
that
it's
our
own
thing
that
nobody
else
about
slows
us
down
in
multiple
ways.
A
First
of
all,
if
so
its
modify,
we
constantly
hire
new
people.
If
you
join
Spotify,
you
need
to
her
learn
about
Hermes.
We
can't
assume
that
people
know
about
Hermes
joining
Spotify
well
good
day.
It's
also,
we
shouldn't
be
in
the
business
of
trading
protocols
and
as
something
as
fundamental
as
an
RPC
kind
of
technology
at
a
company
like
Spotify,
and
that's
a
reason
why
we
moved
to
the
cloud
because
we
won't
don't
want
to
build
beta
centers
and
that's
likewise
with
protocols.
A
It's
it's
a
cry.
It
forced
us
to
write
a
lot
of
boilerplate
and
kind
of
repetitive
code
and
there's
another
aspect
and
that's
the
Hermes
ecosystem.
The
Hermosillo
system
doesn't
exist.
We.
How
could
it
right,
so
you
want
to
use
a
load.
Balancer
sorry
doesn't
exist.
Your
news,
nginx
nope,
the
common
denominator
for
everything
is
TCP
and
that
doesn't
get
you
that
far,
especially
now
that
we
start
talking
about
it
service
measures
and
things
like
that.
The
same
is
also
with
tooling,
so
you
want
to
do
debug
your
API.
A
A
Okay.
So
now
we
know
why
we
need
to
move
why
cheer
BC,
there's
a
lot
of
factors
or
pull
factors
that
won't
that
make
us
want
to
use
to
your
PC
right.
It's
open,
that's
already
a
better
better
than
what
we
have
now
it's
a
open
source.
It's
performant!
It's
based
on
HTTP
too!
So,
there's
a
lot
of
standards
that
we
can
rely
on
it's
backed
by
C
and
C
F,
which
gives
us
some
kind
of
confidence
that
it's
gonna
stick
around
for
a
while.
A
Also,
it's
used
by
a
lot
of
heavy
weight
and
weight
and
Industry
already
that's
also
a
good
testament.
It
strongly-typed,
which
is
nice
and
there's
a
slew
of
other
features
that
really
won't
make
us
want
to
use
geo.
Pc
I
would
like
to
talk
about
three
area
focus
feature
areas
that
that
really
make
a
good
case
that
excite
us
about
you.
Have
you
seen
that
and
made
us
convinced
us
to
to
go
all
in
the
first.
One
is
about
the
proto
protobuf
definition
files
I
mean
if
you've
used
with.
A
If
you've
worked
with
to
your
proceed,
proto
file
itself
is
kind
of
the
Holy
Grail.
It's
that's
where
you
define
your
API
as
one
one
source.
You
define
your
API
and
also
your
message
kind
of
format,
and
from
that
you
can
generate
code
in
multiple
different
languages.
So
I
think
at
the
time
of
writing.
There's
a
lot
of
different
languages,
supported
by
G,
RPC
core.
A
A
We
didn't
have
that
before
and
many
of
you
previously
ever
different
companies
use
h-2b
rest
that
doesn't
really
exist
either.
You
have
to
write
a
lot
of
Chasen
boilerplate
and
derealization
and
series
ation
code
and
it
always
breaks
you
have
the
deal,
it's
kind
of
a
constant
pain,
sure
there's
schemas
for
those
JSON
payloads.
But
it's
not
the
same
and
few
things
we
learned
once
we
started
adopting
G
ABC
is
okay.
How
do
we
handle
those
profiles
if,
given
that
they
are
those
central
source
of
truth?
What
do
we
do
with
them?
A
The
first
challenge
we
had
is
kind
of
convince
people
that
they
should
embrace
those
pro
two
definition
files,
because
we
were
so
used
in
to
kind
of
distributing
client,
char's
client
implantation
for
our
services
to
different
consumers,
or
also
just
considering
our
API
as
kind
of
a
wider
protocol.
So
we
we
can't
have
to
convince
people
that
way
the
proto
file,
that's
your
first-class
citizen
and
from
there
everything
can
be
generated.
So
don't
we
initially,
we
went
in
to
decide.
We
had
that
work
planned
out.
A
We
would
basically
write
our
own
schema
registry
and
we
would
generate
jar
files
for
you
and
Python
the
client,
client,
libraries
and
everything
everything
automatically,
but
in
the
end
we
realized.
Nobody
else
is
doing
it.
So
maybe
we
shouldn't
either
otherwise
we're
just
going
to
be
in
a
Spotify
cherries,
land
which
is
not
much
better
than
Hermes.
A
A
I
think
the
best
approach
for
now
that
we
have
and
a
lot
of
other
people
do
is
we
have
a
shared
repository
where
everybody,
every
certain
service
owners
puts
their
profile
and
from
there
you
pull
the
files
that
you
need
to
to
properly
check
to
generate
the
client
code
that
you
need
to
call
that
service,
and
on
top
of
that
repo,
you
can
write
tooling
right.
So
we
have
a
tool
that
helps
you
download
the
proto
files
and
dependencies.
Also,
this
is
also
the
place
where
you
can
enforce
guidelines
on
how
you
write
profiles.
A
A
Actually,
a
quick
shout
out.
Goober
has
a
nice
tool
to
handle
Oh
produce
files.
Is
called
the
proto-tool
that
we
heavily
rely
on
and
we
have
our
own
tool
that
way
it's
called
proto
man
that
we
looking
into
how
we,
if
we
should
just
merge
our
stuff
into
proto-tool?
Well,
that's
and
we
do
that
in
the
open,
though
that's
a
change,
cool
and
then
the
third
thing
that
we
learned
is
versioning.
A
Just
don't
do
it,
as
in
of
course,
you're
going
to
have
worsens,
but
don't
try
to
version
control,
don't
write
in
yeah,
don't
try
to
kind
of
distribute
different
profiles
with
different
versions
in
their
file
name
or
anything
like
that
or
in
your
client.
Implementation
just
rely
on
the
proteome
of
being
backwards
compatible.
If
you
need
to
do
breaking
changes,
you
create
a
new
package,
proto
package.
So
the
example
here,
let's
say
we
have
a
player's
service,
the
Xplosive
service,
with
a
v1
version.
A
A
Okay,
so
a
few
features
that
another
feature
when
I
talk
about
it's
around
resiliency
and
Cheer
PC
has
a
really
nice
features,
they're
different
patterns
that
you
can
enforce.
So
the
first
one
is
retries
in
hermes,
for
instance,
we
don't
have
automatic
retry,
so
you
don't
have
that
general
HTTP,
client
libraries
either
in
the
world
of
her
and
Cheer
pc.
A
There
is
actually
proposal
on
going
to
kind
of
get
that
work
into
G,
RPC,
core
and,
and
one
of
those
aspects
in
in
the
proposal
is
hedging,
which
is
an
advanced
way
of
retries.
So
hedging
means
it's
also
referred
to
me.
You
might
back
up
requests,
what's
called
in
the
cassandra
land
or
people
also
refer
to
as
respectively
speculative
retries,
so
they
yes,
the
DA's.
A
You
sent
the
first
request
and,
after
certain
delay,
if
you
don't
get
a
response,
you
set
the
second
request
and
after
a
delay,
you
said,
the
third
requests
basically
fanning
out
your
request
to
all
the
backends.
Until
you
get
some
response
as
soon
as
you
get
the
response,
everything
is
canceled.
Every
are
every
outstanding
or
busiest
cancels.
So
this
helps
you
to
kind
of
cut
the
tail
latency
drastically
in
certain
cases.
A
The
problem,
the
downside
is,
of
course,
that
you're
gonna
spend
more
resources
on
GL
RPC
and
that's
a
really
a
great
way
to
kind
of
create
cascading
failures,
and
that's
where
what
you
end
up
is
kind
of
a
thundering
her
and
heard
kind
of
situation
right
where
everybody
hammers
the
same
back-end,
that's
already
down
and
yeah.
You
know
it's
really
hard
to
get
out
of
it
so
to
prevent
those
cascading
failures.
Your
busiest
other
features
that
help
you
talked
about
two
things.
A
One
of
them
is
called
retry
throttling
and
pushback
they're
also
part
of
the
same
proposal
so
or
retry
throwing
basic
noise.
You
have
it,
you
have
a
good
of
retries
I
can
use,
and
if
they're
used
up
you,
you
can't
use
anymore.
So
the
hedging,
for
instance,
won't
issue
more
our
pcs.
If,
if
you
only
used
up
your
retry
bucket
program,
pushback
is
on
the
server
side,
so
the
server
can
actually
signal
to
clients
took
like
white
pulled
on
read,
sent
me
a
request
in
set
amount
of
time
again
and
try
it
and
again.
A
To
propagate
yeah,
I
lost
I
forgot
about
the
term
anyway,
that's
a
good
two
ways
to
kind
of
prevent:
cascading
failures:
okay,
the
second
thing
that
we
like
about
in
ng
RPC
and
that's
its
deadlines.
So,
instead
of
setting
time
lights
time,
timeouts
manually,
you
can
define
a
deadline
on
a
request
and
the
request
will
be
automatically
cancelled
once
the
deadline
runs
out.
So.
A
A
So
since
many
of
the
many
solutions
in
Google,
ferns
and
other
drivers
as
well
used
to
your
PC,
we
can
actually
use
the
same
kind
of
way
to
call
out
into
GCP.
Let's
say
we
connect
a
big
table,
so
we
can
use
deadlines
there
as
well
and
can
enforce
it.
The
same
way
also
we're
looking
into
how
we
can
extend
that
the
other
way
into
the
edge.
So
we're
looking
at
how,
if,
if
we
should
use
to
your
PC
in
the
mobile
client,
for
instance,
and
that
will
give
us
a
lot
of
benefits.
A
Okay,
another
thing
I
would
like
to
talk
quickly
about
its
load,
balancing
routing
where
Jo
PC
brings
a
lot
of
a
lot
of
features
to
traditionally
you
will
have
server
side
load,
balancing
you
have
a
proxy
and
that
elder
request
goes
through
and
the
kind
of
the
proxy
the
server
distributes
traffic
in
the
back
through
the
backend.
The
problem
with
that.
Well,
it's
very
simple
advantages.
The
clients
aware
you
don't
have
to
care
about
load
balancing
at
all
the
problem.
Is
you
get
a
latency
hit
because
you
have
an
extra
hope,
that's
might
be.
A
A
Remember
we
use
Hermes,
so
there
isn't
a
load
balancer
for
Hermes,
so
everything
we
do
at
Spotify
is
client-side
load
balancing
today
with
Hermes
and
and
we
actually
have
a
quite
sophisticated
policy.
We
use
expected
latency
selectors,
which
is
not
today
available
in
CC,
but
it
gives
us
pretty
good
load
balancing
across
the
backend
and
we
don't
have
this
extra
help
anywhere
now
GBC
introduces
another
kind
of
load
balance.
It's
called
the
leucocyte
load,
balancing
it's
fairly
new.
The
idea
isn't
that
new,
but
it's
I
haven't
seen
many
implementations
of
it
yet.
A
But
the
idea
is
here
that
you
get
the
benefit
of
your
basically
client-side
load,
balancing,
but
the
logic
of
load.
Balancing
your
client.
You
extract
that
into
a
central
load.
Balancer,
that's
not
part
of
the
data
plane,
so
you're
not
gonna,
have
extra
hop
the
load.
Balancer
sits
to
the
side.
Think
of
image.
Yes,
you
sits
to
the
side
and
basically
just
feeds
the
client
with
updated
lists
of
healthy
backends
upstreams.
It
also
at
the
same
time
considered
of
course
monitors
the
back
tendencies
which
services
and
endpoints
are
healthy
or
not.
A
That's
also
the
perfect
place
where
you
could
introduce
advanced
routing
or
rollout
strategies,
and
if
you
squint
hard
enough
a
bit
and
consider
that
this
is
all
just.
We
extract
logic
from
the
client
from
the
data
plane
into
a
central
place.
It's
only
looks
very
much
like
a
service
mesh,
because
what
the
leucocyte
load-balanced
really
is
just
a
control
plane
for
the
GBC
clients.
So
there
is
currently
very
early
work
on
going
on
actually
supporting
the
XDS
routing
api
from
envoy
in
g
RPC.
A
So
you
could
actually,
in
the
future
sometime,
maybe
use
a
control
plane
like
sto
to
control
your
Chi
OPC
mesh.
So
you
will
actually
have
a
mesh,
that's
proxy
less
and
can
do
at
least
a
subset
of
what
a
full-blown
proxied
word
service
mesh
could
do
for
you.
So
that's
I
think
is
very
exciting.
Nothing
that's
available
today
and
we
don't
use
it
currently,
but
I
think
that's
something
that
we
are
definitely
looking
into.
If
you
could
help
us,
okay,
there's
one
more
reason
why
G
RPC
is
great.
A
Xie
pc
has
by
far
the
best
mascot.
The
mascots
name
is
pancakes
for
those
who
haven't
heard
about
it
and
pancakes,
because
Qi
RPC
is
a
anchor
name
for
golden
retriever,
pancakes
and
I.
Think
if
you
have
that
kind
of
mascot,
you
don't
need
much
more
reason
to
to
adopt
Ripa,
see,
there's
more
versions
of
the
same
mascot
logo.
The
proto
can
serve
proto
kits
the
proto
compiler,
I
love
it,
and
thanks
to
a
colleague
of
mine
and
work
actually
and
help
from
Google.
We
have
our
own
Spotify
pancakes
logo.
A
So
if
you
would
see
that's
what
I
have
on
the
one
single
app
sticker
have
my
laptop,
we
have
a
limited
manner
amount
of
stickers.
So,
if
you're
interested
in
that
sticker
come
talk
to
me
afterwards
cool
how
we
going
about
it,
how
do
we
migrate
to
G
ABC,
because
something
as
fundamental
as
changing
a
protocol
is
a
lot
of
work
right,
and
we
have
to
consider
that
everybody
at
Spotify
all
the
developers.
They
have
a
lot
of
important
things
to
do
right.
They
need
to
kind
of
deliver
value
to
the
customer.
A
A
Well,
the
first
thing
is:
we
have
to
get
me
embrace
the
fact
that
this
is
a
journey
right.
This
is
going
to
take
time
and
what
we're
really
doing
is
we're
not
necessarily
moving
just
to
cheer
PC.
We
just
move
to
I'm
sorry,
so,
right
now
we
had
10%
migration.
We
have
10%
of
services
used
to
your
PC
a
half.
A
year
ago
we
were
around
a
little
bit
more
than
0,
so
we
already
made
some
headway
and
what
we
really
want
to
achieve
the
destination
is
not
just
your
PC.
A
We
want
to
get
rid
of
Hermes
because
that's
one
of
the
big
point
points,
but
we
also
really
like
to
encourage
people
to
use
to
your
PC
because
there's
there's
a
lot
of
benefits
to
it.
So,
yes,
how
do
we
do
that?
And
in
order
to
do
talk
talk
a
little
bit
about
how
we
get
the
teams
to
that
migration
because
we
don't?
We
can't
force
anyone
at
Spotify
to
do
those
things
and
we
kind
of
pride
ourselves
that
you
and
say
that
we
have
autonomous
teams.
So
how
do
we
get
autonomous
teams?
A
To
kind
of
you
want
to
migrate
to
cheer
PC?
How
do
we
get
them
to
join
the
journey
before
and
I?
Think
I
need
to
talk
a
little
bit
about
what
that
entails,
because
a
big
migration
like
that
is.
It
is
a
big
kind
of
a
change,
a
technical
change,
but
also
cultural
organization
will
change
and
change
is
hard.
A
Change
is
hard
because
it
threatens
what
you
find
is
familiar
and
it's
kind
of
challenges,
what
you're
just
already
hack,
maybe
planned,
and
that
change
kind
of
breaks,
your
expectations
and
familiar
things,
feeling
nice
and
comfortable
right.
So
how
do
we
build
up
momentum
to
move
in
the
right
direction
and
then
how
do
we
convince
people
to
move
or
migrate
in
on
their
own
free
will,
unfortunately
not
Jedi's?
We
can't
use
mind
tricks,
so
we
employ
three
different
strategies.
A
First
of
all,
we
kind
of
need
a
culture
of
trust
in
order
to
to
have
autonomous
teams.
We
trust
engineers
to
do
the
right
thing.
We
empower
them
and
giving
them
access.
Let's
say
to
the
production
environment
too,
and
telling
them
my
dad
that
we
know
that
you
will
act
in
the
best
interest
of
the
company
empowers
them
to
do
the
right
choice
right.
A
So
what
we
need
to
do,
we
don't
just
tell
them,
use
you're,
perceiving
to
tell
them
well,
you
have
to
explain
to
them
why
this
makes
sense,
and
even
if
something
goes
wrong,
we
don't
want
to
blame
anyone.
We
have
to
kind
of
see
that
as
a
learning
opportunity-
and
so
the
first
thing
when
you
talk
about
autonomous
teams
is
autonomy
is
not
freedom,
but
it's
very
often
perceived
like
that.
A
It's
not
the
idea,
isn't
that
everybody
can
do
whatever
they
want.
The
idea
is
that
you
ally
you
align
around
your
autonomy
and
I
think
that
diagram
shirts
are
very
nicely
if
you,
if
you
look
at
autonomy
on
the
horizontal
and
alignment
on
the
vertical.
What
you
want
to
you
want
to
end
up
in
that.
In
the
corner
to
the
top
left,
a
top
right,
where
we
have
high
alignment
and
high
autonomy,
if
you
just
have
autonomous
teams
without
alignment,
they
just
everybody
does
whatever
they
want
and
we
don't
get
anywhere.
A
If
we
have
high
alignment,
but
no
autonomy,
people
just
do
what
they're
told
top-down,
which
is
not
very
empowering
and
the
hard
to
scale
and
well.
If
you
have
low
autonomy
and
low
alignment,
I,
don't
know
what
that
looks
like.
So
what
we
really
need
is
aligned
autonomy
and
how
to
achieve
alignment.
A
We
do
it
in
multiple
ways.
One
of
those
is
that
we
make
what
we
have
in
order
to
get
alignment
around
tech.
We
have
something
what
we
called
it:
the
golden
stack
at
the
Golden
Path,
it's.
What,
if
I
basically
golden
past,
is
like
a
document
that
helps
you.
It
tells
you
if
you'd
follow
these
kind
of
practices
and
use
those
technologies,
things
will
be
laid
out
for
you
kind
of
it's
the
path
of
least
resistance.
A
It
all
points,
and
now
it's
just
that's
a
screenshot
of
a
developer
portal
internal,
where
the
different
disciplines
have
their
own
golden
path,
and
it's
kind
of,
if
you
see,
if
you
can
think
of
it
as
the
blessed
stack,
but
you're
not
forced
to
walk
the
golden
path.
If
you
don't,
though,
it's
probably
not
it's
gonna
be
as
easy
as
it
could
be,
and
hopefully
somebody
will
have
hold
you
accountable
that.
Why
are
you?
Why
are
not
using
the
golden
path
if
it
would
speed
you
up
X
times?
So
what
we
really
do?
A
Is
this
v8
the
right
choice
to
be
the
easy
natural
obvious
choice
for
people
to
take?
So
the
first
thing
we
did
or
we're
doing
right
now
is
adding
to
your
PC
to
the
gold
path
and
removing
Hermes
from
it,
so
that
this
is
clear
signal
to
everyone
that
from
now
on,
if
you
use
to
your
PC,
we
will
do
everything
in
our
power
to
make
it
as
easy
as
possible
for
you,
if
you're,
not
in
the
golden
path,
you
might
need
to
do
some
extra
work.
So
that's
one
way.
A
Second
thing
is:
we
need
carrots
right.
We
need
incentives
for
people
to
want
to
switch.
Carrots
can
be
things
like.
We
addressed
pain
points
or
it
can
be
features
that
they're
looking
forward
to
or
you
can
just
be
overall
improved
in
performance
or
because
they
are
attending
cube
corn
and
cloud
native
content
thing
if
your
PC
is
cool.
So
this
is
really
an
exercise
of
gamification
and.
A
And
again,
if
you
remember
Hermes,
there
is
no
echo
system
so
pretty
much
everything
out.
There
is
a
good
incentive
to
to
switch
so
and
in
a
case
of
emergency,
if
you
feel
like
you're
getting
anywhere-
and
you
have
this
long
title
migration
and
we
just
end
up
with
yet
another
protocol,
which
you
definitely
want
to
avoid.
We
want
have
kind
of
finalized
the
migration.
In
the
end,
you
have
the
counterpart
of
carrot,
which
is
a
stick.
The
stick.
Of
course
we
haven't
used
that
much
and
we
haven't
actually
used.
A
You
can
look
like
you
set
arbitrary
deadlines,
you
say:
ok,
Hermes
will
be
end
of
life
by
I,
don't
know
December,
2020
or
June
2019
to
kind
of
Fort
force
people
to
migrate.
We
really
don't
want
to
do
that,
but
I
mean
it's
one
tool
that
we
have
in
case.
You
have
to
it's
really.
If
the
migration
takes
long
enough,
you
kind
of
force
it.
A
So
right
now
we
are
just
starting
our
journey.
We've
did
a
lot
of
pre-work
to
kind
of
get
off
the
ground.
We
hope
now
that
the
flight
is
going
to
be
smooth
sailing
and
we
hope
it's
not
going
to
look
like
that
once
we
land
and
we
hope
that
most
of
our
developers
didn't
have
to
care
that
much
about
GA
BC
just
that
they
have
a
new
RPC
in
general.
So
I
would
like
to
change
this.
That
developers
don't
care
about
the
RC
technologies
to
their
words
that
don't
have
to
care
about
RPC
technologies.