►
Description
How Niantic switched Pokémon GO to use Envoy - Renana Yacobi
Niantic are the creators of Pokemon GO. As one of the world's most popular mobile games, Niantic needs to serve ##'s of players all across the world, concurrently, necessitating the need of a truly planet-scale solution. In this presentation, Rennana Yacobi, Server Core Infrastructure Lead, explains why Niantic made the transition from NGINX to Envoy, starting with the most important question of ‘Why Envoy?’, then reviewing Niantic’s journey with extending Envoy to support our proprietary protocol which includes websockets and player leasing, using xDS to minimize disruptions when scaling, load testing to ensure Envoy can handle millions of QPS at Pokemon GO's scale, where things fall apart, all the way until consolidating everything to the final launch.
A
Hi
everybody
and
welcome
to
how
niantic
switch
pokemon
go
to
using
boy.
My
name
is
lana
nayakovi
and
I'm
a
server
core
infrastructure
tech
lead
at
niantic
who
are
we
niantic,
is
a
leading
ar
technology
company
focused
on
exploring
new
applications
for
advanced
hardware
wearable
devices,
in
particular
that
marry
the
digital
world
and
the
physical
world.
A
Our
mission
is
based
on
three
key
product
principles,
exploration
and
discovery
of
new
places,
exercise
and
real
world
social
interactions
with
other
people.
This
mission
has
been
our
compass.
It's
paved
the
way
for
our
consumer
ar
games,
ingress,
pokemon,
go
harry
potter
and
that's
what
we
are
arguably
best
known
for,
but
we're
a
lot
a
whole
lot
more
than
that
underpinning
these
games
is
a
powerful
technology
platform,
the
niantic
real
world
platform,
which
we've
been
investing
in
and
evolving
from
the
very
beginning.
A
Niantic
real
world
platform
also
supports
social
features,
mapping
and
advanced
ar
all
the
components
needed
to
develop
and
publish
a
real-world
ar
game,
as
our
platform
grew
to
offer
additional
components
and
new
games
were
published.
We
needed
to
adjust
infrastructure
to
support
the
new
workloads
and
complexity.
A
A
In
the
beginning,
our
configuration
was
simple
engines
deploys
our
edge
proxy
routing
traffic
to
the
game
servers,
but
as
new
services
were
developed
and
deployed,
different
routing
configurations
were
created
and
defined
by
the
new
by
the
dev
teams.
Most
connections
were
passing
through
the
front
door
using
regional
ips,
hoping
not
to
hit
the
public
internet.
A
A
The
flexibility
provided
by
despair
enabled
us
to
support
our
different
models
from
duplex
mode
to
either
pull
or
push
add,
unified
security
with
federation
for
games
and
services.
By
creating
a
pool
of
gateways,
we
were
also
able
to
shift
all
services
from
using
the
edge
proxy
to
using
their
dedicated,
more
secure
endpoints.
A
A
A
One
question
you
are
probably
wondering
about
was
why
we
didn't
adopt
service
mesh
implementation
for
orchestration
and
management
for
the
mesh,
in
one
word:
maturity
when
looking
at
cmcf
landscape
map
under
service
mesh.
These
are
the
graduated
projects.
These
are
the
incubating
projects,
but
most
of
the
projects
are
still
in
send
back
stage
or
not
yet
added
to
the
map.
A
Looking
at
our
next
steps
with
a
future
of
increasing
numbers
of
back-end
services
and
games,
it
is
obvious
for
us
that
we
will
need
to
find
a
solution
for
orchestration
and
management
at
scale.
What
we
liked
about
shifting
to
envoy
was
the
most
of
the
tools
under
development.
We're
using
envoy
as
the
proxy,
so
migrating
to
them
should
be
an
incremental
step
and
we
still
had
the
option
of
using
xts
in
the
meantime,
as
our
management
tool.
A
I
mentioned
before
that
our
game
cluster
deployment
is
simple,
but
while
the
deployment
is
simple,
there
were
several
things
you
wanted
to
address:
improving
user
experience
when
scaling
was
one
of
the
goals,
and
I
will
elaborate
more
about
what
discovers
and
why
in
a
minute.
But
besides
this
issue,
we
also
wanted
to
improve
our
monitoring
and
alerts
unify
as
much
as
possible
our
tools
and
data
plane
and,
lastly,
support
the
company's
goal
in
enabling
external
developers
to
create
new
games
without
exposing
rrep.
A
I
would
like
to
focus
for
a
moment
on
the
goal
of
improving
user
experience
when
scaling
mainly
because
this
part
of
our
our
architecture
caused
us
some
trouble
here.
The
big
question,
of
course,
is
why
scaling
created
bad
user
experience
problem
lies
in
the
proxy
configuration
and
config
map
limitations.
A
Our
proxy
configuration
is
tied
very
much
into
our
ip
for
every
incoming
request.
The
proxy
is
checking
if
the
request
url
has
a
prefix
of
an
opaque
id.
If
one
is
found,
the
proxy
matches
that
id
to
a
game
server
removes
the
prefix
and
route
the
request.
Accordingly,
we
crest
without
id
a
round
rubbing
between
the
available
game
servers.
A
A
A
This
step
can
be
disruptive
to
clients
that
are
currently
connected
through
websockets
or
pending
a
response
from
the
server
when
all
pods
are
updated
and
healthy.
We
update
the
backend
with
the
number
of
available
game
servers
and
update
again.
The
proxy
configuration
creating
a
second
wave
of
disruptions.
Scaling
down
process
is
very
similar
only
in
reverse
order.
A
Well,
next,
yes
to
the
rescue
using
a
custom
xds
operator,
we
were
able
to
scale
up
and
down
the
clusters
without
the
need
to
restart
or
redeploy
them.
The
operator
receives
the
requested
cluster
size
through
a
kubernetes
custom
resource,
update
the
configuration
and
pass
it
to
envoy.
We
are
currently
in
the
process
of
deploying
this
new
operator.
Our
next
steps,
after
verifying
the
operator
is
stable
and
working
as
expected
will
be
to
integrate
it
with
the
game.
Server
existing
drain
feature
for
a
seamless
scale
experience
this
overview.
Our
new
deployment.
A
A
We
started
with
load
testing
using
a
pokemon,
go
load,
testing,
setup
and
runners.
We
used
the
latest
invoice
version,
which
was
112
and
started
with
a
basic
environment
of
x,
game
servers,
x,
caching,
services,
one
proxy
and
a
load
of
hundreds
of
thousands
of
requests
per
second,
the
result
looks
great,
except
some
pesky
five
or
three
errors
that
surface
from
time
to
time.
A
We
had
to
make
a
couple
of
changes
to
our
production
environment
to
be
able
to
shift
the
traffic
from
engines
to
enjoy.
First
change
was
to
separate
the
traffic,
the
public
traffic
from
the
backend
services
I
mentioned
before,
since
we
didn't
onboard
the
service
gateway.
Yet
we
took
the
approach
of
simply
duplicating
the
engine's
configuration
and
creating
a
dedicated
pool
for
them.
A
A
A
We're
becoming
very
good
in
writing.
Those
well
the
big
day
arrived
and
we
started
shifting
the
traffic
from
engines
to
invoice.
Well,
we
started
started
deploying
the
traffic
shift
plan
was
to
first
increase
the
number
of
envoy
replicas
in
the
pool
when
reaching
full
capacity
decrease
the
number
of
engines
replica
until
hitting
zero.
A
A
This
is
just
a
short
version,
but
after
several
days
of
investigations
and
digging
up
in
envoy
and
gclp
documentation,
we
found
the
following
results.
The
main
reason
for
the
invoice
crash
was
because
we
were
hitting
it
with
20
times
the
expected
load
and
even
more.
It
was
already
deployed
on
a
pod
with
limited
resources
which
might
have
been
a
contributor
factor,
but
unfortunately,
after
correcting
the
traffic
distribution
to
2000
requests
per
second,
we
never
encountered
this
problem
again.
A
A
A
Over
the
next
two
weeks.
We
slowly
slowly
scaled
up
the
envoy
fleet
from
two
to
five
from
five
to
fifty
percent
and
so
on.
Until
we
hit
one
hundred
percent
when
hitting
one
hundred
percent
of
it
void,
deployments
and
after
verifying
all
is
okay,
or
at
least
that's
what
we
thought,
we
started
scaling
down
the
engine
split
parallel
to
this
migration,
we
had
new
features
and
services
also
added
to
the
environment,
creating
problems
of
their
own.
So
when
problems
started
hitting
us,
we
didn't
recognize
what
is
the
source
of
them?
A
A
Like
any
other
team,
we
were
busy
and
with
little
information
or
access.
This
turned
to
be
the
status
quo
for
four
months
until
goldfaced
arrived.
If
you're
not
aware
go.
Fest
is
our
biggest
event
of
the
year
when,
starting
to
scale
the
number
of
game
servers
in
preparation
for
the
event
we
started
receiving
reports
of
an
increase
in
five
or
three
errors.
This
happened
immediately
after
deploying
envoy
images
with
the
new
configuration.
A
A
We
also
started
noticing
an
increase
in
player
complaints
on
the
down
detector.
We
investigated
that
configuration
wrong
health
check
everything
but
found
no
clue
except
an
interesting
decrease
in
coop
dns
errors
that
part
confused
us
more.
But
then
we
found
a
hint
and
just
a
reminder.
Our
deployed
void
version
was
112,
and
this
was
fixed
only
in
114.
A
Back
to
our
low
test
environment.
We
started
with
reconfiguring
it
to
match
production.
We
scale
it
up
to
much
one
tenth
of
production
and
configure
to
have
x10
services
pointing
to
the
game
servers
in
help
to
recreate
the
conditions
that
caused
the
problem
with
sound
production.
It
was
easy
we
got
the
ns
errors
already
when
hitting
one
30s
of
the
production
environment.
A
A
It
was
our
configuration
that
caused
the
loading
dns
queries,
our
stateful
back-end
configuration
was
causing
the
proxies
to
query
two
dns
four
times
for
each
game.
Server
now
multiply
that
by
the
number
of
proxies.
Add
to
that
the
number
of
game
servers
and
the
number
becomes
too
high
a
solution
we
tested
was
google
cloud
new
dns
caching
operator,
which
made
the
errors
disappear
completely,
but
will
require
an
update
to
the
jke
version.
A
A
Unfortunately,
I
can't
share
a
lot
about
this
work.
I
can
tell
you
that
included
request
validation
for
http
and
websockets
requests,
shifting
the
responsibility
to
block
this
traffic
already
in
the
proxy.
It
also
manipulates
the
incoming
traffic
based
on
the
received
message
and
provide
us
with
additional
metrics
related
to
the
games.
A
A
Thank
you
for
listening,
and
I
would
like
to
thank
everyone
in
niantic
that
helped
me
in
this
work
and
for
those
of
you
that
find
this
interesting
and
would
like
to
learn
more.
We
are
hiring
feel
free
to
reach
to
me
and
check
out
our
career
website.
Thank
you
and
I
will
be
very
happy
to
answer
now
any
question
you
might
have.
A
A
A
So
as
as
sweeney
that
trinity
mentioned,
we
have
seen
an
issue
where
void,
retains
high
throughput
connections
constantly,
whereas
engines
releases
them
priority.
Did
you
see
such
issue
during
your
migration?
Well,
we
didn't
see
this
as
an
issue.
This
was
part
of
the
behavior.
We
already
knew
that
this
part
of
your
boy,
as
I
mentioned,
the
main
difference
that
we
had
between
envoy
and
engines
was
actually
around
the
dns
and
the
dns
query
and
jenks
are.