►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Game Server Networking with Envoy - Christopher M Luciano, IBM
Game servers can be incredibly difficult to scale and operate within cloud environments. Latency management is paramount to ensuring that game players' experiences to vary widely even on less reliable broadband. Game developers often operate at Layer 4 TCP/UDP and implement their own custom protocols atop. As such, many popular libraries that operate at the HTTP layer are useless for assisting the developer with common traffic shifting patterns like circuit-breaking and retries. This talk will introduce Envoy and demo how it can be leveraged by game developers to assist in their network communication between hosted or on-prem deployments.
To learn more: https://sched.co/Graj
A
So
Who
am
I,
not
a
game
developer,
but
this
talk
is
based
on
a
lot
of
research
and
talking
with
individuals
who
are
actual
real
game
developers.
People
who
have
worked
on
various
layers
of
the
stack.
What
I
do
actually
do
is
I
contribute
to
upstream
projects
like
sto
like
kubernetes,
most
recently
a
bit
of
Envoy.
A
My
contact
information
will
be
available
at
the
ends.
I
have
a
few
business
cards
if
you
really
want
one,
but
not
very
many,
so
catch
my
email
at
the
end
of
the
presentation.
If
you
want
to
talk
more
on
the
flip
side,
my
tank
font,
you
on
the
Bonjour
realm
I
play
Decarie
and
here's
the
storms
and
lega
legends
Protoss
for
Life
agenda
wise,
so
I've
swapped
out
a
lot
of
slides
I've,
put
in
more
slides,
taken
out
again
talked
to
a
few
people
ripped
out
a
lot
of
slides
put
more
in.
A
A
So
a
few
of
the
goals,
we'll
start
with
just
a
few
high-level
goals
of
you
know
some
of
the
interesting
things
within
game
engine
code
and
our
first
kind
of
goal
with
a
lot
of
this
is
is
scale
and
scale
means
a
lot
of
different
things.
Depending
on
the
type
of
game.
We
have
very
smaller
kind
of
things,
more
of
an
older
networking
type
of
land
party,
some
other
thing,
maybe
more
familiar
to
the
analog
like
Dungeons,
&,
Dragons,
sort
of
scenario.
A
There's
one
Dungeon
Master,
there's
multiple
players:
if
the
dungeon
master
dies
happily
through
the
campaign
needs
to
go
home,
you
need
to
elect
another
game
master.
So
a
lot
of
these.
A
lot
of
your
network
problems
in
this
scenario
are
really
based
on
the
game
master
there,
and
if
that
players
connection
is
particularly
bad,
everyone
else
suffers
as
well.
A
One
kind
of
benefit
to
this
is
you
do
have
the
ability
to
migrate
hosts,
but
it's
going
to
be
quite
obvious
for
everyone
as
they
go
through
the
game.
If
everything
just
kind
of
stops
and
then
in
transition
to
a
new
host,
then
we
have
the
dedicated
game
server
model,
and
this
is
what
I've
noted
as
one
of
the
more
popular
ones
very
familiar
for
people
working
with,
maybe
like
an
instance
in
the
larger
game
like
a
dungeon
that
you're
running
through,
perhaps
or
even
just
a
normal
first-person,
shooter
type
of
scenario.
A
A
Engine
running
and
everyone
connecting
from
either
the
Xbox
or
their
PC
is
connecting
to
this
one:
a
game
server
engine
is
there
a
load
bouncer
in
between
normally
not
you're,
going
to
expose
this
IP
from
your
general
list
of
all
the
IPS
that
your
cloud
provider
is
going
to
give
you
or
maybe
you're
hosting
this
yourself
and
they're,
going
to
be
connecting
literally
directly
to
that
IP.
So,
there's
nothing
in
between
here.
A
A
Important
note
in
here
you're
not
going
to
get
your
host
migration.
So
if
one
process
the
one
process
dies
everyone's
affected,
the
game
just
kind
of
ends,
no
winner
ever
that's
the
good
news
is
this
generally
doesn't
happen.
But
again,
then
you
have
another
scenario
where
you
have
like
a
massive
sort
of
world.
This
would
be
more
like
your
large-scale
mmo's
different
parts
of
the
map,
maybe
being
served
from
a
majority.
A
large
majority
of
servers
dedicated
to
that
specific
purpose
that
specific
serving
scenario,
but
with
inside
of
this
world
as
well.
A
A
So
performance
wise,
this
is
kind
of
the
thing
that
comes
up.
Always
it's
the
number
one
thing
on
everyone's
mind
when
they're
working
with
game
servers
and
it's
difficult
to
prioritize
performance,
especially
in
the
game
scenario,
when
there's
so
many
factors
that
go
into
just
one
message:
when
you're,
when
you're
thinking
about
a
kind
of
the
the
messages
and
the
pieces
in
the
game,
you
could
think
of
them
sort
of
like
a
large-scale
chess
map
and
as
we
know
what
each
piece
on
the
chef's
map
has
its
own
details
about
it
and
its
own
attributes.
A
This
timeslot
doesn't
even
work
for
them,
so
we
need
to
reschedule
the
meeting
and
deliver
everything
back,
that
sort
of
pause
and
act
and
nack,
so
I
prevent
and
sequencing
just
doesn't
reliably
work
for
fast-paced
scenarios
like
a
game
as
well
as
to
a
person
playing
the
game.
This
might
ends
up
being
them
running
across
the
map,
suddenly
stopping
and
then
ending
up
like
50
feet
from
where
they
imagined
they
were
supposed
to
be
in
the
first
place.
A
The
game
becomes
really
popular
and
now
you've
got
thousands
and
thousands
of
players
connecting
to
an
area
that
you
did
not
intentionally
think
that
they
were
going
to
play
in
then
you
end
up
deploying
servers
there
to
actually
support
them,
and
then
people
never
want
to
migrate
off
those
servers,
because
now
they've
developed
a
community
and
they
don't
want
to
switch
to
one
where
the
friends
aren't
playing
on.
So
this
is
difficult
balance.
A
Security
was
a
heel.
They
also
face
interesting
problems.
It
turns
out
people
can
be
really
terrible
if
they
die
in
a
game.
They
might
feel
it
necessary
to
DDoS
your
game
and
ruin
it
for
everybody,
because
if
they
died,
everyone
else
should
die
or
not
be
able
to
play
in
general.
On
the
other
hand,
people
take
a
different
approach
and
decided
that
they're
smart
enough
to
reverse
engineer
your
binaries
and
figure
out
how
to
get
all
the
great
weapons
and
then
they
just
wreak
havoc
on
everybody
in
general
fleet
upgrades
is
another
interesting
scenario.
A
I
was
emotionally
dedicated
game
server
model.
It's
pretty
easy
to
spin
down
that
instance
at
the
end.
But
what
happens
if
you're,
constantly
playing
with
the
same
sort
of
maps?
How
do
you
relatively
deploy
new
instances?
River
game
such
as
you
know,
maybe
something
like
adding
voice
features
to
your
game
so
that
you
can
interact
in
a
scenario
that
you
didn't
allow
something
like
multiplayer
in
before.
So
in
this
instance,
we
have
the
Star
Trek
bridge
computing
game.
You
guys
have
played
that
before
in
the
game
you
could
they
recently
added
some
well
not
completely.
A
Recently,
but
they
there
was
support
added
for
playing
without
connecting
with
four
people.
So
how
can
we
actually
deploy
this
sort
of
feature
out
without
affecting
people
that
are
currently
playing
and
making
sure
that
when
people
actually
do
connect
that
they
can
use
the
sort
of
voice
feature
and
they're,
not
just
left
alone,
trying
to
figure
out
what's
going
on
with
their
game?
So
we
want
to
maybe
selectively
growl
people
to
more
of
the
quality
assurance
type
of
machines
to
test
these
features
out,
or
else
only
silently
release
this
to
some
specific
group
of
individuals.
A
What
are
some
things
that
we
can
do
in
the
game
community
for
that?
So
this
is
kind
of
where
we
come
to
like
the
some
of
the
basics
of
of
envoy,
which
I've
targeted
at
this
solution.
Now,
there's
gonna
be
different
use
cases
of
fondle
and
talking
to
people.
They
don't
really
have
the
appetite
to
place
envoy,
potentially
directly
in
their
game
of
server
path.
If
I
give
you
a
bowl
and
a
spoon,
that's
how
you
can
use
it
to
make
ice
cream
and
fruit
salad.
It's
not
very
useful
to
you.
A
If
you
already
have
an
ice
cream
maker-
and
you
only
want
to
make
ice
cream
so
I
don't
expect
this.
Some
of
this
will
be
applicable
to
someone
that
will
want
to
immediately
rebase
all
of
their
game
engine
code
on
top
of
envoy,
but
at
the
end,
I'll
describe
a
few
other
scenarios
that
I
think
are
still
gonna
be
applicable.
But
what
envoy
really
is
is
its
its
target
IOC.
A
So
it's
hell's,
Jim
proxy,
but
normally
deployed
as
either
a
sidecar
or
perhaps
it
runs
on
the
host.
And
what
it's
going
to
do
is
going
to
intercept
the
traffic
coming
from
surface
one
in
this
instance
running
on
the
node
and
tossing
it
over
to
at
the
other
node
by
way
of
talking
to
the
other
nodes
envoy
proxy.
So
it's
sitting
directly
in
the
middle
here.
A
The
listener
proxies
events
whenever
connections
are
made,
and
you
can
tweak
various
filters
to
kind
of
augment
different
connection
aspects
prior
to
handing
it
off
to
its
next
service.
There's
different
methods
like
the
weird
different
binding
options
like
free
binds,
as
well
as
options
like
the
transparent
mode,
where
you
can
kind
of
represent
things
that
aren't
immediately
on
your
server,
but
you
kind
of
know
where
they
are
it's
awesome
over
there
and
there's.
The
notion
of
a
cluster
clusters
within
envoy
are
a
collection
of
normally
similar
hosts
that
I'm.
A
We
can
make
connections
to,
for
example,
each
instance
of
a
web
server
or
other
type
of
a
game
network.
That's
just
processing
events
for
a
specific
event
will
be
represented
within
one
envoy
cluster,
and
you
can
dynamically
configure
clusters
of
these
services
through
various
api's
or
statically
assigning
the
through
things
like
file
lives.
A
One
other
good
aspect
of
clusters
is:
if
you've
got
some
preparation
before
you
can
actually
start
adding
and
removing
different
instances
and
putting
them
back
into
your
little
pool.
There
is
some
warming
and
the
options
that
you
can
have
so
that
you're
not
going
to
immediately
accept
a
connection
and
fail,
because
you
haven't
properly
warmed
yourself
up
and
then
received
all
the
information
you
need,
and
then
the
last
small
component
is
filters
in
the
filter
aspect.
A
Here:
connections
that
are
passed
between
envoy
transition
between
various
filters
representing
different
OSI
layers,
so
you
can
have
filters
at
the
listener
level,
as
mentioned
the
network
level.
There's
a
few
options
up
there,
as
well
as
more
higher
level
feature
filters
like
HTTP
filters,
so
we're
just
applying
gaming.
Well,
this
is
where
it
comes
to
custom
that
filters.
A
So
this
is
kind
of
an
area
that
might
be
an
interesting
use
case
for
the
kind
of
layer
three
for
transitions
within
your
game
engine,
not
to
say
that,
like
I,
said,
you'd
want
to
go
off
and
immediately
try
this
out,
but
here's
the
options
that
we
have.
So
we
have
Reed
filters
for
parsing
information
about
the
event
or
the
intention
of
a
player.
Like
my
player
is
now
dead
or
here's
the
position
of
my
player.
A
That
would
be
interesting
for
someone
working
at
the
3/4
level
is
the
rate
limiting
feature.
So
we
can
imagine
in
this
case
having
a
ton
of
players
connect
to
one
instance
and
then
going
over
the
limit
that
we
have
targeted.
That
instance
of
a
server
app
and
onwe
exposes
some
useful
metrics,
there's
a
small
typo
in
there,
but
so
after
hitting
at
a
certain
threshold,
we
want
to
start
shedding
traffic
potentially
off
of
a
server.
A
A
One
of
the
examples
that
are
built
to
into
on
point
now-
and
this
would
be
once
it
hits
a
certain
threshold
on
the
host-
maybe
there's
too
many
processes
taking
up
memory
and
we
were
gonna
start
we're
gonna
track
that
note
it
and
maybe
start
shutting
off
features
of
things.
So
the
use
case-
that's
brought
up
in
envoy
at
the
moment
is
certain
keepalive
metrics.
So
maybe
we
don't
want
to
keep
alive
and
we
want
to
shut
down
that
particular
components.
A
Another
interesting
aspect
to
kind
of
that
would
be
interesting
for
someone
that
has
persistent
servers
is
envoys
kind
of
management
of
itself.
So
in
this
instance,
you
can
safely
drain
connections
by
a
majority
of
methods,
some
more
simple
ones,
such
as
just
posting
to
the
admin
health
check
API,
as
well
as
configuring
health
checks.
Cipher
can
take
a
picture
as
well
as
having
health
checks.
Currently,
these
are
only
mostly
targeted
at
the
HP
example.
A
However,
we
can
think
of
extending
these,
perhaps
for
a
given
use
case
and
I'm,
where
the
layer,
4
level,
there's
also
outlier
detection
for
making
sure
that
these
aren't
a
little
too
aggressive.
So
in
the
outlier
detection
mode,
it's
more
of
a
passive
model
whereby
you're
removing
unhealthy
hosts
based
on
conditions
that
happen
along
the
way-
and
this
is
important
for
envoys
hot
restart
mechanism
whereby
you
can
fully
reload
binaries
or
your
envoy
binary
without
dropping
connections
easily.
And
this
happens
by
utilizing
UNIX
domain
sockets
and
starting
up
the
new
process.
A
A
So
these
were
more
of
the
targeted
things
if
you
wanted
to
put
on
void
directly
in
the
path,
but
I
think
there
are
also
a
couple
of
interesting
use
cases
that
could
be
made
for
just
general
game
developers
that
don't
want
to
modify
their
game
engine
code.
So
a
lot
of
these
come
from
kind
of
some
of
the
more
favorite
things
that
I've
heard
from
like
Mack
line
mentioning
how
much
he
loves
the
statistics
coming
out
of
envoy,
and
then
this
is
a
use
case.
A
I
imagined
a
using
these
statistics
in
a
QA
testing
sort
of
mode.
In
this
instance,
you
would
have
envoy
only
running
in
your
queuing
environments
and
you
can
utilize
the
access
logging
to
determine
and
check
how
you're
determining
player's
position
within
the
world
gathering
metadata
that
you
might
find
useful
like
their
connection
rates,
as
well
as
just
all
of
the
general
stats,
as
mentioned
before
connection
speeds,
player
values
from
their
previous
connections,
so
that
you
know
how
to
intelligently
match,
make
them
for
the
next
game.
A
The
other
example
is
kind
of
like
I
said
before
the
QA
testing
example.
In
this
example,
I
can
imagine
using
Envoy
to
simulate
some
of
the
poor
connections
that
you
might
notice
on
particularly
slow
bandwidth
speed
without
actually
physically
traveling
to
that
location.
So
you
can
use
envoy
to
inject
in
some
sort
of
a
fault
or
slow
down
the
connection.
Speed
notes
what
it's
like
when
players
that
have
a
particularly
terrible
connection,
are
playing
your
game
and
react
to
that.
A
Some
other
future
enhancements
and
that
are
coming
up
at
non-void
I
thought
would
be
particularly
interesting,
for
the
crown
would
be
a
UDP
support
which
has
been
is
still
in
progress.
There's
been
some
enhancements
recently
to
work
towards
the
goal
of
like
a
UDP
listener,
as
well
as
just
general.
You
to
be
proxying
I
feel
the
issue
up
there,
as
well
as
quick
and
one
ish
example
that
maybe
think
that
it
was
actually
possible
to
use
this
within
your
game.
Engine
was
a
particularly
library
from
valve
that
I
found
for
the
game.
A
Okay,
as
well
as
performance,
is
always
being
worked
on
with
inside
of
envoy
to
both
lessen
the
amount
of
memory.
That
envoy
is
taking
up
particularly
large
clusters,
as
well
as
just
making
sure
that
things
are
stored
in
a
very
a
performant
way.
Webassembly
a
port
support
might
be
particularly
interesting
as
well
for
people
that
are
running
across
different
platforms
or
they're
using
different
languages,
and
Matt
mentioned
this
on
Monday,
which
I
thought
was
very
interesting,
is
some
of
the
use
case
for
mobile
support.
A
I,
don't
have
any
more
information
about
that,
but
I'm
sure
Matt
will
share
it
with
us
in
the
future,
and
some
of
the
other
last
bits
is
just
general
platform
support
with
inside
of
a
game.
Things
like
proxying
voice,
proxying
messaging,
as
well
as
excited
tracking
metrics
for
different
events,
so
that
is
all
I
really
have
for.
A
How
far
away?
Well
it's
Matt,
it
is
Matt
in
here,
so
the
the
tenets
of
a
goal
was
q1
of
next
year.
It's
kind
of
lengthened
out
partially
doing
my
fault,
partially,
just
it
being
a
complex
thing
in
general.
There's
the
stateful
vs.
stateless
UDP
types
as
well
that
we
want
to
make
sure
to
handle
the
big
driver
of
it
as
well
as
quick
and
I
believe
that
at
least
some
examples
are
supposed
to
be
out
within
q1
of
next
year.
For
quick
and
UDP
is
actively
being
worked
on
at
the
moment.
A
Sure
so
we
can
move
on
to
this
giant,
backup,
slides
that
I
have
here
where
you
can
follow
along
for
the
XDS
api.
So
this
is
the
common
one
that
a
plot
that
a
control
plane
like
sto
would
utilize
for
programming
envoy,
it's
very
fine-grained
at
the
level
of
you
know
the
listener
the
clusters
figuring
out
each
individual
endpoint.
So
you
can
use
a
lot
of
these
source
of
api's
to
program
it
as
well
as
maybe
just
dropping
down
to
a
more
static
configuration
if
necessary
as
well.
A
A
Yeah,
so
this
kind
of
is
related
to
some
of
those
features
that
I
mentioned
towards
the
front
about
rate
limiting,
as
well
as
the
the
overload
manager
I
myself
have
not
started,
because
we
need
the
EDP
that
support
there.
First
I
haven't
started
to
experiment
with
how
that's
going
to
work,
but
they're
all
within
the
UDP
proposals.
They're,
these
sorts
of
things
have
been
brought
up.
A
A
A
Other
questions:
okay,
the
worst
offender
for
having
latency-
oh
that's,
a
good
I
mean
the
the
most
obvious
one
to
me
is
the
mixer
filter
with
inside
of
this
Jo,
and
essentially,
what
this
is
doing
is
every
so
often
it's
bundling
up
and
setting
it
on
the
wire,
but
it
kind
of
introduces
another
component
within
the
data
plane.
Every
so
often
things
like
that,
where
you're
directing
through
envoy
as
well
as
yet
another
thing
are
the
big
ones.
Specific
features
within
vanilla,
envoy,
I
haven't
done
any
measurements
at
the
moment
to
know
the
biggest
players.
A
Other
questions
right
well,
I
hope
this
was
used
for
everyone.
I
have
a
very
limited
set
of
business
cards
to
give
out
if
we
really
want,
but
please
copy
down
my
email
here.
If
you
want
to
connect-
and
they
said
a
lot
of
this-
was
just
a
bunch
of
research
that
I've
done
in
some
simple
cases
that
I
thought
where
this
could
be
useful,
but
I
think
there's
a
lot
more.
That
could
come
up
with
us
talking
thanks
a
lot.