►
From YouTube: The Open Source Revolution: How Kubernetes is Changing the Games Industry - Dominic Green
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Open Source Revolution: How Kubernetes is Changing the Games Industry - Dominic Green, Netspeak Games
Traditional multiplayer game development involves dozens if not hundreds of engineers over several years building custom made backends - often from scratch. Companies have recently moved to GCP or AWS but most games infrastructure runs on metal - often using multiple providers. Cost, latency & security are all cited as the primary deciding factors when architecting a modern online game; but a new vector is emerging - speed. Specifically, speed of development; how fast can you FIND the game to make is the new race that modern games companies are finding themselves in & the open-source ecosystem is going to be their new best friend. Dominic Green will run through how Netspeak Games has taken advantage of cloud-native technologies and built out their games infrastructure on top of Kubernetes, Agones & Open Source Software - building an MMO that supports 100,000s of concurrent players.
https://sched.co/ek9u
A
A
So
I'm
head
of
platform,
a
small
game
studio
out
of
london,
england
called
net
speed
games
and
we've
built
and
launched
a
mobile
mmo
in
just
six
months,
I'm
the
head
of
platform
and
netspeak.
Now,
whilst
that
sounds
impressive,
it
basically
means
I'm
a
team
of
one.
The
only
platform
or
infra
engineer.
A
So
we're
going
to
talk
about
why
speed
matters
especially
for
free
to
play
online
games,
we're
going
to
look
at
the
infrastructure
of
games
and
what
the
component
parts
are
that
make
it
up
and
then,
when
we
look
to
use
kubernetes
what
were
the
challenges
with
this?
And
thankfully
there
was
a
solution,
and
this
was
going
as
an
open
source
project
and
also
how
we
can
use
open
source
and
kubernetes
to
empower
others
within
the
studio
to
help
them
build
the
game
faster.
A
So
how
long
does
it
feel
take
to
build
a
game?
This
is
a
game
you've,
probably
all
heard
of
world
of
warcraft
released
by
blizzard,
and
it's
probably
got
about
13
million
subscribers,
which
paid
nine
dollars
a
month.
So
it's
got
good
revenue.
It
was
also
released
originally
as
a
boxed
game
and
you
paid
an
initial
fee
to
buy
the
game,
and
this
actually
took
around
four
to
five
years
to
build
and
get
out
to
launch
and
even
after
that
they
still
constantly
updated
it
with
patches
and
expansions.
A
A
It's
probably
a
little
bit
complex
got
different
roles,
it's
got
different
ways
of
playing
and
it's
got
a
lot
of
different
content.
However,
supercell
probably
took
around
six
months
to
build
this
game
again
after
the
launch.
They
probably
tweaked
it
and
made
improvements
after
seeing
what
their
players
did,
how
about
free-to-play
games
built
on
top
of
snapchat
well,
this
is
bitmoji
party,
and
this
is
quite
a
very
simple
game
and
you
play,
as
I
said,
against
other
people.
You
know
in
snapchat
as
a
filter.
A
This
probably
took
a
really
small
team
about
three
months
to
play.
The
infrastructure
is
all
there.
It's
part
of
the
snapchat
platform.
It
should
be
quite
easy
and
quick
to
build
with
this
in
mind,
why
does
it?
Why
does
speed
matter?
Well,
we
want
this
kind
of
reaction.
We're
gonna
make
sure
that
it's
fun
to
play
for
our
players,
especially
in
the
free-to-play
market,
so
the
quicker
we
get
into
our
hands
the
quicker
we
find
out
if
the
game's
fun
and
if
people
will
continue
to
so
what
takes
up
majority
of
the
time?
A
Well,
actually,
it's
building
content.
That's
why
we
have
so
many
artists
inside
the
studio
and
six
percent
of
the
times
actually
built
and
spent
building
content.
Here
you
can
see
rowan
jumping
for
joy.
He's
one
of
the
npcs
within
our
game
and
then
there's
a
small
amount
of
time
and
for
design,
production
and
kind
of
operations
of
the
company
and
then
25
of
times
actually
spent
on
tech.
This
includes
the
gameplay
all
the
gameplay
systems
and
things
that
build
the
game
included
within
this
25
is
the
hosting
the
infrastructure
and
the
platform.
A
This
is
where
I
fit
in
so
with
six
months
and
a
game
that
needs
to
be
built
and
a
new
platform,
and
what
were
my
ideas?
Well,
firstly,
we
don't
want
to
build
from
scratch,
and
the
main
idea
was
that
we
wanted
to
invest
in
any
open
source
technologies
that
we
could
and
standing
on
top
of
the
shoulders
of
giants,
and
also
we
want
to
try
and
make
it
easy
for
the
game
team
to
use
this
platform.
We
didn't
want
any
friction
for
them
when
they
were
building
out
the
game.
A
After
this,
we,
let's
look
at
the
requirements.
We've
heard
that
we've
got
a
single
engineer
and
six
months
there
are
the
two
constraints.
What
do
we
need
to
be
able
to
support?
Well,
thousands
of
concurrently
connected
users
doing
thousands
of
requests
per
second,
resulting
in
hundreds
of
gigabytes
of
data
that
needed
to
be
stored
about
the
players
and
what
they
were
doing.
A
On
top
of
this,
we
needed
to
scale
with
demand.
Games
often
are
busier
during
the
evenings
and
weekends,
and
much
much
quieter
during
the
day.
So
we
had
to
have
this.
Elastic
wants
scalability,
be
in
multiple
locations.
The
closer
our
clusters
are
to
our
players,
the
better
latency
they'll
get,
which
means
the
game
will
probably
be
more
enjoyable
for
them.
We
needed
to
make
sure
that
the
platform
was
transparent
to
the
team
and
the
fact
that
it
could
evolve
over
time.
We
didn't
really
want
to
lock
ourselves
into
any
choices
early
on
also.
A
We
had
this
idea
that,
if
we
don't
need
to
build
it,
if
we
don't
have
to
so,
let's
have
a
quick
look
behind
the
scenes,
the
infrastructure
and
the
various
components
that
make
up
an
online
game
first
off
the
core
of
it.
Is
this
idea
of
a
dedicated
game
server,
and
this
is
a
single
process
where
tens
to
hundreds
of
players
will
connect.
A
A
For
example,
say
my
character
was
running
around
in
the
world
and
dancing,
and
my
client
would
send
this
up
to
the
dedicated
game
server,
which
will
then
replicate
it
back
to
all
other
connected
players
behind
the
game.
Server
is
a
number
of
data
services
which
connect
into
data
stores
early
on.
We
made
sure
that
we
abstracted
the
data
store
away
from
the
game
server
and
allowed
the
gaming
play
programs
to
focus
on
just
calling
into
apis
rather
than
underlying
storage.
A
This
also
means
that
in
the
future,
if
we
needed
to,
we
could
switch
out
the
storage
for
other
systems
if
we
okay.
So
how
do
you
get
into
an
online
game?
The
first
step
is
authentication
you're
going
to
go
through
a
number
of
steps,
and
when
you
call
it
all
service,
we
verify
who
you
are
and
where
you're
logged
in
from
and
send
you
back
an
authentication
token.
We
actually
use
jwt
for
this
from
there.
You
call
into
a
matchmaker
passing
that
token,
so
that
we
can
identify
who
you
are.
A
The
matchmaker
then,
will
often
talk
to
a
lot
of
the
datas
in
the
backend
to
get
information
about
the
player,
what
their
experience
is,
which
things
they're
doing
at
that
time
and
eventually
pass
back
the
host
ip
and
port
number
for
the
dedicated
game
server
that
play
is
going
to
connect
to
so
when
we
actually
connect
into
a
game
server.
The
client
actually
uses
udp
and
it's
actually
transferring
data
backwards
and
forwards
30
times
a
second.
So
this
is
quite
a
lot
of
data.
A
A
So
with
all
this
in
mind,
why
kubernetes
and
we're
going
back
to
the
requirements
we
needed
to
support
thousands
of
concurrently
connected
users,
and
we
had
thousands
of
requests
per
seconds.
We
also
needed
to
scale
up
and
down,
based
on
the
demand
that
we
were
seeing,
we've
seen
that
kubernetes
can
easily
deal
with
this
and
even
start
to
scale
up
to
thousands
of
nodes,
especially
in
lines
of
business
applications.
So
hopefully
we
could
take
this
and
apply
it
to
a
game.
A
Communities
also
runs
anywhere.
We
can
deploy
it
into
any
cloud
provider
or
we
can
even
use
bare
metal
with
this.
It
also
allows
to
standardize
the
technologies
that
we're
using
both
for
the
dedicated
game
server
and
those
supporting
services.
In
the
background,
if
all
of
these
are
running
in
kubernetes,
we
have
a
standard
way
of
doing.
The
majority
of
things
along
with
this
kubernetes
has
a
huge
community
of
thousands
of
engineers
that,
if
we
run
into
any
problems,
would
more
than
likely
be
happy
to
help
out.
A
So
was
it
just
that
easy?
Did
we
just
take
our
game
server,
put
it
into
a
container
and
run
it
in
kubernetes.
Job
done
dom
takes
a
holiday
for
the
remaining
three
months.
Unfortunately,
not
the
first
problem
we
came
into
was
containerizing
unreal.
Unreal
was
actually
first
built
in
95
as
an
engine
for
a
first
person
shooter
and
the
version
we're
using
unreal
four
was
actually
rebuilt
in
2005..
A
Thankfully,
it
can
be
cross-compared
from
windows
into
linux,
and
so
we
actually
were
able
to
get
it
into
a
container.
However,
try
running
it
as
root,
or
even
the
user,
unreal
and
you're
in
for
some
series.
So
from
here
we
try
to
run
our
game
services
deployment.
This
is
the
first
thing
we
looked
at
and
deployments
allowed
us
to
scale
up
and
down
easy.
The
other
thing
we
needed
to
do
was
because
we
were
doing
udp.
A
A
A
However,
the
promise
of
kubernetes
was
more
that
we
can
run
many
many
containers
on
the
same
node.
However,
this
starts
to
become
a
problem.
If
two
containers
on
the
same
node
are
trying
to
map
the
same
host
port,
we
end
up
having
collision.
So
it's
not
just
that
easy
to
multiple
containers,
mapping
to
the
same
host
port.
We
need
to
think
of
another
way
of
doing
it
or
some
kind
of
clever
infrastructure
between
to
make
sure
the
ports
mapped
to
different
ports
that
are
exposed
to
the
internet.
A
Next,
what
about
scaling
scaling?
The
pods
like
scaling?
Up's,
quite
easy.
You
just
add
more
pods.
This
isn't
going
to
affect
anyone.
That's
connected
into
a
dedicated
game
server.
However,
when
you
come
to
scale
down,
how
do
we
guarantee
that
we're
not
scaling
down
a
a
pod?
That's
got
players
connected.
A
This
is
definitely
one
of
the
bigger
issues.
It
also
is
an
issue
if
we
wanted
to
upgrade
our
game
server
if
we
deployed
a
new
image
version.
This
would
also
result
in
all
the
containers
being
restarted
into
that
new
version.
Thankfully,
open
source
came
to
our
rescue
and
there's
a
project
out.
There
called
the
goners.
You
can
go
to
against
dev
to
find
out
more
about
it
and
it
allows
us
to
host
run
and
scale
dedicated
servers
on
top
of
kubernetes.
A
This
project
was
actually
built
by
ubisoft
and
google
to
help
ubisoft
actually
scale
their
own
game
servers
on
top
of
kubernetes.
So
what
does
agonist
give
us?
Well,
it
gives
us
a
number
of
custom
resource
definitions
which
make
game
serves
a
first
class
citizen
of
kubernetes
and
accessed
via
cubecuddle.
A
Here
you
can
see
the
yaml.
I
know
everyone
loves
young.
Also,
it
wouldn't
be
a
kubernetes
presentation
without
a
bit,
and
here
all
we've
done
is
specify
the
image
we
want
our
game
server
to
be,
and
this
will
pull
it
down
from
the
container
registry
and
at
the
top,
we've
actually
said
that
we
want
to
map
the
container
port,
and
here
we
have
two.
A
A
What
else
does
agonist
gives
us?
It
gives
us
the
concept
of
fleets
again.
This
is
another
custom
resource
definition,
which
is
basically
a
collection
of
game
servers.
It
helps
with
allocation
of
users
to
the
dedicated
game
server
and
we'll
see
that
in
a
moment,
and
also
it's
got
different
ways
of
being
able
to
scale.
A
This
allows
us
to
say
that
we
want
a
maximum
of
10
game
servers
in
this
fleet
and
we
always
want
to
have
say
two
two
game
servers
that
are
warm
ready
to
accept
players
and
each
time
the
game
servers
move
through
different
stages.
This
would
allow
us
to
spin
up
more
and
keep
more
warm
for
the
players
coming
in
afterwards.
A
So
fleets
are
a
collection
of
game
servers,
and
this
will
help
us
scale
up.
As
we've
seen,
we
can
distribute
different
ways
over
nodes.
However,
it
will
also
help
us
when
scaling
down
and
making
sure
our
players
aren't
disconnected
from
their
dedicated
game
servers
to
see
this,
we
have
to
actually
dig
into
a
little
bit
more
of
the
life
cycle
of
what
happens
to
our
dedicated
game
servers.
A
A
At
this
point,
the
container
is
running.
It
goes
into
a
scheduled
state
from
here
it's
over
to
the
game,
server
itself,
the
game
server.
At
this
point
is
responsible
for
putting
and
telling
the
gamers
when
it
is
ready
to
start
accepting
players
and
the
reason
why
this
is
is
you
could
actually
be
loading
the
number
of
assets
in
the
back
hydrating,
some
kind
of
state
from
a
database
or
waiting
for
some
long
computation
to
happen
before
you
want
to
accept
players.
A
A
As
you
said
earlier,
players
come
in
through
a
matchmaker,
so
the
first
thing
the
client
will
do
is
auth
go
into
the
matchmaker
and
say
I
want
to
find
a
server
to
connect
to.
At
this
point,
our
matchmaker
actually
calls
into
a
number
of
backend
services,
gets
information
about
the
player
and
then
asks
the
ogona's
apis
internally
to
allocate
it
a
game
server.
A
What
this
does
is
agonis
will
actually
talk
to
the
fleet
and
say
tell
me
which
10,
which
game
server
to
allocate.
This
will
then
put
the
game
server
into
an
allocated
state
and
return
the
ip
address
of
the
host
and
the
port
number
where
the
player
should
connect
to.
Whilst
it's
in
the
allocated
state,
this
is
kind
of
one
of
the
most
important
parts
for
scaling
down
and
we'll
see
why,
in
a
minute,
but
this
effectively
saying
that
either
a
player
is
about
to
connect
or
has
connected.
A
Finally,
the
after
play
is
disconnected
and
the
game
server
is
no
longer
needed
anymore.
We
need
to
scale
it
down,
and
this
is
done
by
the
game.
Server
itself
saying
I've
got
no
more
players
left
I've
cleaned
up
saved.
All
the
data
that
I
needed
to.
I
can
actually
shut
down
in
doing
so.
It
calls
out
by
the
sdk
again
signaling
to
a
gun
is
to
clear
it
up
and
the
the
pod
will
be
placed
by
another
one
which
will
go
into
ready
state
to
accept
more
so
from
here.
A
Let's
have
a
look:
what
happens
if
we
scale
up
the
fleet
and
set
the
number
of
replicas
to
six?
You
can
see
that
there
are
a
number
of
different
pods
running
there.
These
are
actually
game
servers
and
so
which
pulled
them
up
to
on
the
host
machine
and
how
long
they've
been
running
so
great
we've
scaled
up
and
what
about?
If
we
try
and
scale
down
all
the
way
to
zero?
Well,
this
happens.
A
This
is
the
fact
that
we've
actually
said
via
cube,
cuddle
and
scale
of
fleet
up
to
six
replica
down
to
zero
replicas.
However,
it's
taken
all
of
them
down,
apart
from
the
game
server
that
was
allocated.
This
is
very
important,
and
this
is
why,
again,
this
helps
with
scaling
up
and
down
game
servers.
It
knows
that
players
are
connected
to
this
server,
so
won't
destroy
it
or
move
it
from
the
machine
that
it's
on
doing
so
would
obviously
end
the
session
for
the
player
and
result
in
an
unhappy
experience.
A
So
we
don't
only
have
to
run
a
single
fleet
within
our
cluster
and
what
we
actually
do
is
allow
devs
to
run
a
separate
fleet.
All
our
devs
inside
netspeak
can
have
a
fleet
of
their
own
and
at
any
point
in
time
they
can
actually
deploy
into
this
fleet,
so
they
can
run
against
the
same
infrastructure
as
our
players
are,
and
what's
there
talking
to
the
same
data
services,
effectively
testing
a
new
version
of
the
game
in
production
alongside
other
running
customers.
A
What
so,
what
does?
What
does
it
give
us?
Well,
we've
seen
this:
the
ability
to
map
container
ports
to
host
ports.
This
is
in
a
dynamic
way,
it's
protected
as
from
players,
whilst
we're
scaling
down
or
upgrading
the
version
of
the
game
server,
it's
giving
us
multiple
fleets
so
that
our
developers
can
run
in
the
same
infrastructure
as
our
consumers,
and
it
gives
us
the
ability
to
auto
scale
up
and
down
and
keep
a
number
of
warm
servers
for
us
from
here.
A
A
We
have
gameplay
programs
and
artists
who
aren't
familiar
with
infrastructure
and
especially
not
familiar
with
kubernetes,
so
we
wanted
to
make
the
experience
as
transparent
as
possible
for
them,
so
they
could
keep
up
this
quick
cadence
a
bit.
The
first
step
in
this
journey
is
making
sure
that
they
could
automatically
build
out
a
client
and
server
whenever
they
wanted,
and
so
we're
actually
using
github
actions
to
this,
and
they
can
signify
which
fleet
they
want
to
have
the
client
connect
to.
A
They
can
signal
what
branch
it
comes
from
and
they
can
do
the
same
with
the
server
and
once
they
do
this
for
the
client,
it's
actually
uploaded
and
they
get
a
notification
on
their
device
and
they
can
automatically
then
download
that
client
and
connect
into
a
game
server.
However,
we
still
haven't
deployed
the
game.
Server
into
coupon
deploying
into
kubernetes
was
one
of
the
areas
which
I
was
basically
a
real
hedge
scratcher
for
a
while,
and
then
I
thought
about
what
some
of
the
technologies
are
out
there
gitops.
A
It
was
the
answer
for
us
effectively
allowing
us
to
take
advantage
of
gear,
but
more
specifically
take
advantage
of
the
fact
that
github
and
gitlab
have
a
user-friendly
ui,
so
the
artist
can
easily
go
into
the
ui
change
the
version
number
and
not
have
to
worry
about
what
happens
with
the
infrastructure.
In
the
background.
A
This
allows
him
to
really
speed
up
and
iterate
fast
on
building
out
the
game,
so
we've
got
githubs
for
deployment.
We've
got
an
automated
build
system
that
pushes
out
clients.
We
also
have
a
supporting
cast
of
other
things
which
help
the
game.
Engineers
and
artists
ensure
that
the
game
is
being
built
in
the
correct
way.
We
use
things
like
c
advisor
to
ensure
that
we
have
the
correct,
metrics
and
cpu
memory
utilization,
which
goes
into
prometheus
and
eventually
displayed
to
them
in
grafana.
A
A
This
has
also
given
us
the
ability
to
support
thousands
of
connected
users
on
multiple
instances
of
unreal.
It's
allow
us
to
do
it
in
such
a
way.
That's
low
friction
for
the
game
team
and
they
then
have
a
self-service
way
of
deploying
and
monitoring
their
game.
Hopefully
this
actually
speeds
them
up
and
doesn't
get
in
the
way
of
what
they're
trying
to
achieve.
A
So
what
have
we
learned?
Kubernetes
gives
us
some
kind
of
standardization,
especially
with
containers
we
run
and
build
our
dedicated
game
server
out.
In
the
same
way,
we
do
our
matchmaker
and
all
our
data
services.
So
it's
one
standard
approach
to
doing
infrastructure,
we've
utilized
open
source
to
make
sure
that
we
can
move
fast.
We
didn't
have
to
build.
We
didn't
build
anything
that
we
didn't
have
to.
Also
the
open
source
community
adds
to
our
team.
A
They
really
allowed
us
to
move
fast
and
can
help
us
whenever
problems
occur
rather
than
using
something
that's
homegrown
and
custom.
Also
empowering
our
people
using
the
platform
inside
the
studio
is
really
important
if
it
gets
in.
If
kubernetes
gets
in
the
way
of
us
building
the
game,
we
won't
be
able
to
release
it
in
the
time
frame
we
need
so
what
we
looked
at
today.
We've
talked
through.
Why
speed
matters
in
the
games,
industry,
it's
about
getting
into
the
hands
of
the
users
and
finding
what
the
fun
experience
is.