►
From YouTube: Intro to Agones: Scaling Multiplayer Game Servers with Kubernetes - Mark Mandel, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro to Agones: Scaling Multiplayer Game Servers with Kubernetes - Mark Mandel, Google
Kubernetes provides an amazing toolset for running processes over potentially thousands of machines. However, Dedicated Game Servers for real time multiplayer games, such as Fortnight, Overwatch, etc, have their own challenges that don’t directly line up with the solutions that Kubernetes natively provide. Game Servers are neither stateless (no Deployments), or ordered stateful (no StatefulSets) - but sit somewhere in between the two. They require direct connections (no load balancers), and can’t be turned off once players are connected to them (no traditional autoscalers). In this talk we’ll discuss and demo the open source project Agones, developed by Google Cloud Platform in conjunction with Ubisoft. Utilising Kubernetes native extension mechanisms, it attempts to solve this problem by providing a batteries-included solution for running Dedicated Game Servers at scale on Kubernetes.
To learn more: https://sched.co/GrU3
A
Hi
everyone,
my
name,
is
Marc
Randall
I
do
develop
our
advocacy
I,
do
developer
advocacy
for
games
at
cloud,
so
I
do
a
lot
of
infrastructure
stuff
for
games
I'm.
Also,
the
founder
of
the
iguana's
project.
We're
talking
about
today
and
I
lead
the
open-source
side
of
Google's
contributions
to
the
project.
I
usually
give
this
talk,
game
developer
conferences.
So
this
is
a
bit
funny
I'm
interesting,
so
who
here
works
in
the
games?
Industry,
whoa?
Okay,
so
that
was
a
lot
more
than
I
expected.
A
That's
awesome,
who
here
just
like
this
seems
like
a
really
interesting,
like
extension
of
kubernetes.
That's
also
really
totally
cool
I
expected
more
of
that
which
is
totally
fine
who's
here,
because
they
didn't
like
mark
talks
about
this
a
lot
and
I'm
friends
with
him.
Yes,
some
people,
okay,
cool,
that's
mine
too,
all
right,
awesome,
all
right
cool,
so
we're
gonna
have
some
fun
and
we'll
see
some
cool
stuff
and
we'll
talk
about
Game,
Servers
and
I,
gonna
and
I
kind
of
stuff.
A
So
I
want
to
sort
of
set
the
time
for
some
people
who
may
not
be
as
familiar
when
we
say
multiplayer
games.
That
can
mean
a
whole
lot
of
things.
It's
a
wide
and
varied
sort
of
thing,
and
when
we
talk
about
multiplayer
games
in
this
particular
instance,
what
I
want
to
talk
about
is
very
fast-paced,
competitive
action,
multiplayer
games
right,
so
we're
talking
about
like
your
fortnight's.
Your
overwatch
is
your
rocket
leagues
things
where
latency
really
matters
really
it
just
doesn't
even
go
back.
Latency
really
matters
competitive,
fast,
paced
action.
A
A
Has
anyone
played
synodic
suite,
that's
actually
more
than
I,
usually
good
fantastic,
and
what
we're
gonna
talk
about
is
actually
a
model
for
doing
this,
which
is
what
was
commonly
referred
to
as
dedicated
game
servers,
and
there
are
a
few
different
ways
we
can
do
this,
but
for
the
matter
of
time,
I
just
want
to
focus
on
this,
and
this
is
kind
of
become
I
feel
like
the
prevalent
way
of
doing
big
scale.
Multiplayer
games,
especially
it's
a
triple-a
audience.
A
This
is
a
thing
that
happened.
You
should
be
aware
of
it,
and
so
you
can
see
it
now.
There
are
a
couple
of
very
good
reasons
for
doing
it.
This
way,
one
of
which,
having
control
over
the
box
that
actually
is
authoritative
and
does
well.
The
processing
in
memory
is
kind
of
important,
because
people
cheat
and
they're
mean
and
horrible,
and
so
having
control
over
that
mitigates
map.
The
other
side
of
this
is
these
sort
of
games
are
very
latency.
A
Sensitive,
ten
milliseconds,
20,
milliseconds
30
milliseconds,
either
way
can
mean
a
big
difference
in
player
experience
and
can
mean
that
you
know
people
won't
come
back
and
play
your
game
because
it
kind
of
just
sucks
so
having
dedicated
game
servers
means
that
you
have
a
lot
of
control.
You
can
place
dedicated
game
servers
in
particular
geographic
locations
and
depending
on
where
you
put
it
in
that
geographic
locations,
gives
you
some
control
over
the
sort
of
latency
that
your
players
are
going
to
play
so
we're
from
a
role-playing,
a
game.
A
For
example,
we
would
probably
want
a
server
to
be
here
in
Seattle,
but
maybe
if
I
was
playing
a
game
with
a
friend
in
New
York
or
something
like
that.
Maybe
we
would
want
to
put
that
in
central
u.s..
So
it
gives
us
some
control
and
gives
us
some
things
we
can
do
so.
That's
the
dedicated
game,
server
right
super
high
level.
So
what
does
the
traditional
architecture
for
this
look
like,
and
it
looks
kind
of
like
this?
Some
of
you
might
be
familiar
with
this
right,
so
you've
ever
played
an
online
game.
A
You
have
a
couple
of
people
who
are
like
I
would
like
to
play
a
game.
It
seems
like
a
fun
thing
to
do.
They
usually
connect
to
some
kind
of
matchmaker.
Matchmakers
job
is
to
group
people
together,
usually
based
on
skill
level
Mabel.
They
maybe
their
social
graph
things
like
that,
and
then
usually
they
talk
to
some
kind
of
game,
dedicated
game
server
manager
and
it's
the
job
of
the
dedicated
game.
Server
manager
to
look
at
a
large
pool
of
machines.
A
Go
hunt
through
them
see
if
they
have
resources
like
CPU
and
memory,
start
up
a
process
for
that
game.
Server
allocate
a
port
for
it.
Some
of
this
might
sound
familiar
basically
spin
up
one
of
these
instances
of
a
game
server
put
all
the
assets
in
it
all
that
kind
of
stuff
and
then
eventually
get
back
an
IP
import
that
we
want
to
actually
make
it
do
a
rec
connection
here.
A
A
A
This
server
manager
that
manages
processes
over
a
huge
number
of
machines
does
that
sound
familiar
at
all
we're
at
a
kubernetes
conference,
cool
all
right,
I
think
you
might
know
where
I'm
going
all
right
sweet,
so
I
was
doing
a
lot
of
work
in
the
guys
industry
and
I
kind
of
realized
that
this
is
a
problem
that
a
lot
of
people
had
and
there
I
didn't
seem
to
find
a
an
open
source
solution
for
running
sort
of
dedicated
game
servers.
So
we
came
up
with
one
so
this
project
has
been
going
for
about
a
year.
A
It's
still
technically
an
alpha.
It's
0.6.
We
do
releases
every
six
weeks,
but
it's
getting
quite
full-fledged
and
we
have
people
who
are
using
it
now
for
games.
We've
got
one,
that's
come
out.
That's
still
I've
tried
to
go
on
it.
Now.
I've
got
a
couple
more
coming
out
early
next
year
and
we're
working
on
it
pretty
heavily,
but
basically
yeah.
It's
an
open
source
platform
for
dedicating
hosting
and
scaling
game
servers
on
top
of
communities.
So
it's
an
operator
much
like
anything
else.
We'll
have
a
look
kind
of
how
it
works.
A
But
what
has
been
super
neat
about
this?
Is
we
didn't
start
this
by
ourselves?
I
have
some
wonderful
friends
over
to
be
soft
or
any
of
them
in
the
audience.
They're,
not
that
wonderful
friends.
No!
They
are
wonderful
friends,
I,
just
great
no
they're,
just
not
here
today,
but
we
have
some
wonderful
friends.
I
do
be
soft
that
we've
been
building
this
in
collaboration
with
them.
That's
been
really
great,
so
we
can
bring
our
experience
with
kubernetes
suite
and
they
can
bring
their
experience
of
running
really
large-scale
multiplayer
games.
A
But
since
then,
we've
also
had
contributions
from
about
25
different
people
from
a
variety
of
backgrounds.
So
it's
been
really
good
to
get
that
diverse,
set
of
skill
set
and
the
experience
coming
through,
which
is
this
week
all
right.
So
let's
actually
look
look
at
how
this
thing
works
and
you
can
see
how
the
cool
things
happen.
Unsurprisingly,
we
put
game
servers
in
containers.
Okay,
that's
not
likely
interesting.
We
will
do
that.
The
only
fun
thing
about
this
is.
A
We
do
have
an
SDK
integration,
it's
G,
RPC,
based
which
is
kind
of
neat
right
now
we
have
SDKs
for
C++
and
go
and
one
with
rust
that
mostly
works,
and
we
also
exposed
things
by
arrest
short
answer.
If
you
want
to
integrate
a
game
server,
rather
our
system,
you
can
find
a
way
of
doing
it,
which
is
good.
A
A
I'm
gun,
hey
I'm,
healthy,
just
making
sure
I'm
regularly
healthy,
there's
some
stuff
about
health
state
that
it's
kind
of
interesting,
being
able
to
get
configuration
information,
and
if
now
gets
me
that
kind
of
stuff
and
sitting
configurations
and
whatnot
I'll
talk
a
little
bit
about
this
too,
because
I
didn't
mention
it
before
health
status.
You
might
look
at
that
me
like,
but
what
about
and
Eddie's
health
checking
like?
That's
that's
or
you
solve
problem
game
servers
are
kind
of
special
in
that
they
do
this
migration
from
stateless
to
stateful.
A
So
when
you
have
a
game
server,
that's
running
in
newer
players
are
playing
on
it.
Nobody
cares
if
you
shut
it
down
as
soon
as
you
have
players
playing
on
it,
you
do
not
want
to
shut
it
down.
Players
get
mad
when
you
shut
off
their
game
in
the
middle
of
their
game.
Who
knows
what
I
had
understand
and
it's
not
like
game
players,
have
any
issues
in
toxicity
at
all?
A
Like
that's
another
thing,
so,
yeah
generally,
a
lot
of
the
stuff
inside
a
gun
is,
is
really
just
making
sure
that
game
servers
don't
get
shut
down
and
people
get
that
cool.
So
we
have
this
SDK.
So
it
does
some
neat
stuff
around
that
and
manages
health
checking
and
some
interesting
and
special
ways.
But
essentially
we
have
you
know
CR
DS,
which
are
pretty
great.
So
at
the
core
level
like
the
base
building
block,
is
we
have
this
thing
called?
Unsurprisingly,
a
game
server?
A
It
looks
a
lot
like
a
lot
of
other
CR
DS.
We
have
a
name
cool,
but
one
of
the
things
that
we'll
do
for
you
is
dynamic
port
allocation,
so
we
use
Nettie's
host
ports
to
expose
game
servers
and
one
of
the
things
that
agonist
does
is
it
exposes
it
dynamically?
So
you
can
basically
say
when
you
set
it
up.
A
This
is
the
range
of
ports
I
would
like
to
expose
and
it
will
assign
one
to
you
and
make
sure
it's
open
and
then
it'll
give
it
to
you
at
the
end,
which
we'll
have
a
look
at
in
a
minute
from
there.
We
specify
what
container
we
want
to
have
as
a
game
server.
Nothing
too
special
there.
This
is
a
full
pod
spec.
You
could
have
multiple
containers.
You
can
have
all
your
mounts,
all
the
other
usual
stuff,
so
you
can
do
whatever
you
like
there
really
at
the
moment,
that's
the
default.
A
So
if
there's
only
one
container,
we're
like
that's
the
game,
server
container,
we'll
expose
that
awesome,
there's
more,
you
can
just
specify
it.
It's
pretty
straightforward!
So
that's
a
game
server,
but
that's
not
that
interesting
much
in
the
same
way
as
we're
all
familiar
with
pods
pods
are
great,
except
without
them
without
deployments,
things
aren't
perfect.
So,
unsurprisingly,
we
have
this
thing.
Called
a
fleet
and
a
fleet
is
a
big
group
of
game
servers.
That's
really
it
so
much
like
you
would
have
an
employment.
A
We
have
a
set
of
game
servers
and
we
can
specify
how
many
replicas
we
want,
and
we
can
scale
this
up
and
down
and
do
all
the
usual
sort
of
stuff
from
there
and
from
there
under
that.
Just
it's
out
game
server
configuration
so
again
if
you're
familiar
with
pods
and
deployments.
This
Oprah
should
look
pretty
familiar,
but
you
may
have
noticed
that
we're
talking
earlier
about
like
how
we
make
direct
connections,
we
don't
use
load
balancers.
When
we
talk
to
game
servers
at
extra
latency
trying
to
route
it
to
a
very
specific
machine.
A
You
can
get
tricky
like
it's
not
ideal,
really
and
usually
you're
sending
UDP
packets,
to
which
is
also
some
other
shots,
the
pain
once
we
want
to
make
a
direct
connection.
So,
if
we're
doing
that,
how
do
we
like
scale
this
out
across
well
a
game
servers
short
answer
is
what
we
actually
have
is
what
we
call
an
allocation.
So
basically
you
need
an
atomic
way
of
saying:
hey
I've
got
this
big
pool
of
game
servers.
Please
give
me
one,
and
here
we
can
actually
say
cool.
A
What
I
want
you
to
do
is
create
a
fleet
allocation.
Again,
it's
another
CRT.
We
give
it
a
name,
and
we
tell
it
what
fleet
it
is.
We
want
to
pull
stuff
out
of,
and
then
it's
gonna
create
that
atomically
and
then,
when
you
return
this
we'll
see
it
in
a
minute.
It'll
give
us
some
game
server
state
we're
actually
working
on
a
selector
based
version
of
this.
That
will
actually
be
really
cool
and
I'm
actually
really
excited
about
it.
So
it's
fun.
A
Alright,
let's
actually
show
you
some
stuff
actually
working
and
then
maybe
it'll
break
and
you'll
love.
We
should
be
good.
So,
for
the
sake
of
time,
I've
already
installed
a
gun.
Is
it's
a
home
chart?
Can
everyone
see
that
okay,
awesome,
perfect
awesome
cool,
so
I've
got
that
installed.
So
what
I'm
gonna
do
is
create
a
fleet.
A
A
If
I
want
to
get
a
little
fancy
with
my
cube,
CTL
commands,
which
I
can
do
here,
I
don't
make
it
look
like
that
which
is
much
nicer
to
look
at
I'm
just
running
this
mini
cube
locally,
primarily
because
I
know,
conference
Wi-Fi
is
awful,
but
so
I
have
to
and
I
can
do
all
the
kind
of
standard
things.
I
can
say
a
cube
CTO
edit
my
fleet,
let's
go
down
I,
would
like
5
replicas,
please,
yes,
bro!
A
So
if
I
do
this
again,
we
can
start
to
see
that
we
get
some
more.
Nothing
special
here
right,
little
kind
of
looks
like
deployment
stuff,
nothing
very
incredible!
So
you
throw
up
and
ready
that's
lovely.
So
there
already
they've
got
player
connections.
The
SDKs
worked
they've.
All
been
assigned
ports
sweet,
so
now
we
want
to
allocate
right.
I
want
to
be
able
to
grab
one
of
those
atomically
out
of
that.
A
A
So
we're
gonna
run
that
and
we're
getting
a
bunch
of
stuff
back.
I'm
gonna
make
that
look
a
little
prettier
and
just
run
this
again.
So
now
we
can
see,
we
have
one
of
these
in
allocated
state.
So
remember,
I
was
talking
about
earlier
right.
The
game
servers
go
from
stateless
to
stateful
allocated
is
the
marker.
That
says
there
are
players
on
this
thing,
and
this
is
important
because
it
means
if
we
want
to
update
this
fleet,
we
want
to
scale
this
fleet.
We
basically
want
to
do
anything
to
this
fleet.
A
Those
allocated
game
servers
are
sacrosanct
right,
like
do
not
touch
them.
Players
get
mad
and
they
post
things
on
reddit,
and
then
you
just
it's
all
very
bad.
So
we
can.
We
can
sort
of
play
with
this
a
little
I
suppose
don't
do
that
we'll
play
with
it
in
a
minute,
actually
yeah.
So
let's
actually
well,
let's,
let's
get
some
more
here
and
here
you
can
see.
Okay,
so
actually
what
happens
when
we
do?
We
do
our
allocation,
there's
a
bunch
of
stuff.
A
Here
we
actually
get
back
the
full
game
server,
but
the
most
important
bit
is
the
bit
here
down
the
bottom.
You
can
see
we
get
an
address.
We
get
a
port,
we
can
actually
assign
multiple
ports
if
we
want.
So
let's
do
like
three
of
them.
So
so
we
have
three
that
allocated
so
if
I
actually
say
keep
CTO
edit
my
fleet,
and
if
my
Scrolls
goes
down,
let's
make
that
like
replicas,
for
example,
we
actually
have
a
look
at
our
pods.
A
We
should
say
that
they're
still
running
right,
they're
still
three
of
them
there
and
eventually
we'll
end
up
with
like
one
of
them's
gone,
unhealthy
right
and
so
there's
allocated
ones
never
get
touched
because
we
have
players
on
them
and
then
we're
waiting
for
them
to
shut
down.
So
let's
actually
do
the
cool
thing
and
fire
up
a
game,
because
that's
what
you're
all
here
for
right!
Oh.
A
A
So
let's
join
that
game,
so
black
screen
is
good
for
whatever
reasons,
because
whenever
I
do
demos,
it
just
likes
to
give
me
a
heart
attack,
so
wait
for
that
to
load,
dododo,
awesome,
so
actually
yeah.
So
right
now
on
my
trackpad
and
I'm,
not
gonna,
say
that
I'm
any
good
at
this
I'm
playing
against
a
bunch
of
bots,
and
this
is
running
on
my
creepy
Nettie's
cluster.
So
this
could
be
running
anywhere.
It
doesn't
matter
if
it's
on
my
mini
Q
cluster.
It
doesn't
matter
whether
I'm
running
on
gke
or
any
other
cloud
provider.
A
A
A
Strangely
enough,
it
runs
on
kubernetes,
it
can
talk
directly
to
the
kubernetes
api
and
talk
directly
to
the
resources
and
the
c
RDS
and
like
basically
use
all
the
same
things
that
you
would
be
familiar
with
when
dealing
with
kubernetes
resources,
which
is
really
cool.
So
it
can
then
be
like
hey
from
that
fleet
can
I
have
a
game
server
and
it
can
be
like
yeah
I've
got
plenty,
it's
totally
cool.
A
He
was
one
grab
the
IP
import,
much
like
it
did
before
and
push
it
across,
which
is
like,
like
any
extension
to
kubernetes,
just
kind
of
really
cool,
because
then
all
the
crew
Bonetti
stuff
all
works
the
same
way,
which
is
pretty
sweet.
We
have
a
whole
host
of
other
features
as
well
I
kind
of
want
to
highlight
the
fact
that
this
is
all
just
vanilla
kubernetes,
so
it
works
across
cloud
providers
if
you
want
to
run
it
on
Prem
anything
that
supports
our
host
port
for
Network,
topologies,
you're
good
to
go.
A
We've
got
people
running
it
across
both,
which
is
also
really
sweet.
We
have
a
whole
host
of
fleet
and
node.
Auto
scaling
I'll
talk
about
that
a
bit,
so
we've
got
some
basic
strategies
for
Fleet.
Auto
scaling
within
a
cluster
that'll
get
expanded
out,
but
we've
also
got
it
integrated
now
with
the
kubernetes
cluster
autoscaler,
and
that's
actually
got
some
really
nice
hooks
into
it
as
well,
so
that
you
can
actually
mark
which
pods
don't
like
don't
kill
these
nodes,
they
hit
players
on
them
and
that
works
really.
A
We've
got
a
bunch
of
well:
we've
got
some
local
development
tools
as
well,
so
if
you're
working
locally
or
you're,
probably
working
on
Windows,
if
you're
working
in
the
gameís
environment,
so
you
don't
have
to
spin
up
a
whole
menu
cube
environment.
You
can
just
see
what's
happening
with
that,
that
sdk
tooling,
that
you're
great
with
your
game
server
as
well,
and
we
have
a
whole
we're
about
on
top
of
that
as
well,
so
all
sorts
of
other
fun
stuff
coming.
So
we
won't
expand
on
fleet
order
scaling.
A
We
want
fleet
order
scaling
to
be
the
basic
building
block
of
our
order
scaling
like
kind
of
thing
we
want.
You
expand
your
fleet
and
then
your
cluster
adjusts
very
much
like
you
would
do
normally
inside
kubernetes,
so
we've
got
some
fun
stuff
coming
there
statistic,
collection
and
display,
so
we
want
to
have
you
know,
really
standard
statistics
around
like
how
many
games
are
you
running?
What's
your
scale
like
what
does
it
look
like
all
that
kind
of
stuff,
but
we
also
want
to
do
very
specific
game
server,
specific
kind
of
statistics.
A
We
want
to
make
that
easy
things
like
knowing
your
CPU
tick
rate
right.
That's
really
important
for
game
servers,
because
that
dictates
like
how
fast
your
simulation
can
go
and
if
you
have
hitching
and
your
CPU
profile,
that's
bad
experience
for
players,
but
this
is
sort
of
like
statistics
that
we
all
need
is
building
game
servers
right
now
we
don't
have
great
commercial
engine
integration.
So
if
using
like
unity,
your
unreal,
we
want
to
have
some
out-of-the-box
stuff.
People
are
already
doing
it
just
using
the
REST
API,
and
that
works
great.
A
A
Basically
we're
waiting
on
that
to
come
through
and
it
looks
like
that's
moving
at
a
really
rapid
pace
and
I'm
really
excited
about
it.
For
those
of
you
who
don't
work
in
games
industry,
everyone
uses
Windows.
That
can
be
a
bit
surprising
for
people
who
don't
work
in
the
games
industry.
So,
even
though
things
might
necessarily
get
hosted
on
Linux
and
there's
definitely
progression,
windows
use
cases,
the
development
environments
for
us
game
developers,
windows
so
having
a
short
iteration
loop
for
Windows
developers
is
really
really
important
for
us
as
well.
A
So
anything
we
do
make
my
shorter
multi
cluster
is
also
really
fun
and
one
of
the
reasons
I'm
at
Q
Khan,
so
we're
looking
at
how
do
we
set
up
and
orchestrate
fleets
across
multiple
clusters
and
allocate
across
multiple
clusters?
I
have
some
really
cool
ideas
from
some
of
the
talks
I've
been
today,
which
is
very
good,
but
we
actually
have
tools
coming
in
the
next
release
for
just
standard
things.
A
So
we've
got
five
minutes
for
questions
at
the
end,
which
is
great,
I,
didn't
mention
at
the
beginning,
but
I've
also
left
a
pile
of
my
business
cards
and
some
stickers
down
here
in
the
corner.
So
if
you
think
I
might
have
questions
later,
please
come
up
and
grab
some,
but
without
before
we
get
stuck
into
questions
Akana,
you
start
dev
redirects
to
the
github
repo,
so
you
can
check
that
out
and
you
can
check
it
out
on
Twitter.
A
You
can
also
reach
out
to
me
on
Twitter
and
I
twitch
stream
at
Kanye's
development
pretty
regularly.
So
if
you
want
to
see
more
how
controllers
work
and
how
to
test
them-
and
things
like
that,
you
might
find
that
interesting
and
I'm
wearing
this
t-shirt.
That
says
the
Google
cloud
platform
talk
the
podcast,
so
I
feel
like
if
I
didn't
mention
it.
That
would
be
sad,
but
we
talk
about
all
sorts
of
stuff
from
game
stuff
to
cloud
stuff,
to
kubernetes
stuff,
to
machine
learning
and
also
some
other
things
in
between
awesome.
A
A
We're
looking
at
the
community
honestly,
the
SDKs
are
really
simple.
Like
it's
like
five,
eight
five,
we
have
like
five
API
calls.
It's
like:
hey,
I'm,
ready,
hey
I'm,
not
let
me
know
when
a
configuration
changes.
I
would
like
to
put
a
label
of
like
with
this
prefix
on
my
things
and
like
that's
about
it.
A
Yep
yep,
so
the
question
was
just
to
reiterate
a
question:
we're
talking
about
whether
we're
working
with
either
unreal
or
unity.
So,
yes,
we
do
have
the
strategic
alliance
with
unity.
The
thing
with
this
project
is,
we
are
about
hosting
and
scaling.
We
don't
care
about
communication
protocols,
so
if
you
want
to
use
WebSockets
or
UDP
packets
or
the
HTTP
connections
that
do
not
care
as
long
as
you've
got
a
thing
that
does
in-memory
simulation,
it's
a
good
platform
for
you.
So
in
the
back.
A
How
do
we
do
it?
Nosy
neighbor
issues
generally
I
push
that
down
into
the
providers
is
the
short
answer.
The
the
things
we've
been
seeing
is
usually
that
yeah
that
tends
to
become
so.
The
issue
was
like:
how
do
we
do
with
noisy
neighbors,
and
we
were
talking
previously
for
people
who
aren't
from
the
game
servers,
CPU,
click-through
rate
and
keeping
that
stable
is
really
really
really
really
really
important.
A
Usually
that
really
comes
down
to
the
underlying
VM
infrastructure.
More
than
anything
else,
we
do
advocate
that
people
set
CPU
limits
or
requests,
and
it
and
test
that
accordingly
inside
their
clusters
and
that's
also
a
nice
way
of
doing
it.
So
if
you
set
a
PUD
level,
CPU
limits
or
requests,
then
that
actually
impacts
the
scheduler
as
well.
So
then
you
can
actually
be
like
okay.
You
know
this
one
needs.
A
So
again
that
comes
down
to
load,
testing
and
like
how
your
cloud
provider
and
the
network
provide
that
gives
you
and
that
kind
of
stuff,
and
so
we
kind
of
offshoot
to
them
and
how
they
do
that.
It
depends
on
your
cloud
provider,
how
much
network
bandwidth
you
have
and
how
much
that
impacts
things
inside
that
VM
and
then
like
what
size
do
you
use
and
then
all
that
kind
of
fun
stuff.
So
the
questionnaire
and
network
bandwidth
in
there.
A
Not
probably
so
the
question
is
like
transactional
games.
You
can
probably
get
away
with
a
lot
of
just
RPC
type
services
for
things
like
that,
and
so
I
would
just
use
regular
web
tech,
and
it's
probably
fine
things
like
WebSockets
G
RPC
can
also
help
stuff.
It
was
a
question
over
in
the
far
right,
I
didn't
see.
A
So
in
theory,
it
could,
under
this
model
on
kubernetes
I,
would
very
much
advise
against
it.
I
really
like
the
kubernetes
container
model,
which
is
you
like.
You
have
a
you,
have
a
container
and,
like
everyone
connects
to
that,
and
then
once
it's
done,
it
shuts
down
and
goes
away.
I've
definitely
seen
people
who
have
processes
and
it
supports
multiple
games,
I
think
it's
much
cleaner
to
work
it
this
way
and
a
little
less
risky
over
there.
Oh.
A
That's
a
really
good
question:
how
do
you
handle
node
maintenance?
So
mostly
what
I
actually
tell
people
to
do
is
use
node
selectors
under
tents
to
handle
it.
So
if
you've
got
a
game
that
cycles
through,
you
could
spin
up
a
new
node
pool
and
then
you
type
you
want
and
then
specify
they
when
you
and
then
change
your
configuration
basically
to
spin
it
up
so
that
they,
you
can
actually
do
a
rolling
update
with
the
fleet
with
the
new
node,
selector
or
tane
demanding,
which
way
you
wanna
do
it.
So
it
all
goes.
A
Is
the
node?
No,
no
pool
wait
for
all
the
game
service
to
disappear,
and
then
you
can
kill
the
other
note
pool
and
you
can
just
switch
that's
the
way
we've
been.
We've
been
talking
about
doing
it
yeah,
because
people
have
run
into
that
too.
It's
a
fun,
that's
a
fun
problem,
but
because
we
have
the
node
selectors
in
notates
that
actually
isn't
too
bad
those
herbs
in
the
middle
I'm
kind
of
going
backwards,
upwards,
yeah.
A
A
So
yeah,
that's
a
good
question.
So
how
does
the
container
port
get
exposed?
So
what
we
do
when
you
set
up
a
game
server?
It
fires
up
a
pod
and
it
exposes
that
as
a
host
port
so
that
we
have
a
network
basically
night-work
isolation
between
each
of
the
game.
Servers
so
right
now
that
routes
to
IP
tables
adds
about
half
a
milliseconds
of
latency.
That's
time,
that's
about
half
a
millisecond
to
latency,
but
there's
some
new
stuff
coming.