►
From YouTube: Production-scale Containerized Game Platform Practice in Bytedance - Chenyu Jiang & Viktor Farcic
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Production-scale Containerized Game Platform Practice in Bytedance - Chenyu Jiang, ByteDance, Inc & Viktor Farcic, Upbound
Classical games servers are running on physical machines or virtual machines to provide services to players. However, packaging game servers as in containers is quickly gaining traction across the tech landscape because of container's isolated runtime paradigm, cost efficiency and elasticity. In Bytedance, games is one of the major vertical domains and we need a mature games-centric platform to serve games from both in-house games studios and agents of game manufacturers globally. In this talk, a Bytedance's practice will be shared in establishing a Kubernetes based Game platform. It leverages multiple CNCF open source frameworks: Crossplane, KubeVela, Agones to address challenges and requirements for games to go cloud-native, such as game servers and dependency resource deployment in multi-cloud/multi-region, game servers orchestration and stateful games service autoscaling.
A
A
A
My
base
is
in
mountain
view,
united
states
and
my
basic
interest
and
working
area
is
a
application
orchestration
and
the
kubernetes
scheduler,
authorism
and
richter
is
the
advocate
in
upper
bound,
who
is
the
maintainer
of
cross
plan,
one
of
the
greatest
components
we're
leveraging
to
manage
different
cloud.
Provisional
results.
Today's
talk
will
be
over
three
parts.
On
the
first
part,
I
will
give
an
overview
for
the
by
dance
and
the
game
platform
in
badass
company,
because
there
is
various
type
of
games
formations
and
there
are
many
aspects
of
the
games
like
the
security
etc.
A
A
Then
victor
will
give
a
brief
demonstration
in
the
third
part
for
the
game's
application
deployment
in
multi-cloud
area.
So,
let's
talk
about
some
background
so
far,
founded
in
2012
battle's
mission
is
to
inspire
creatively
and
enrich
life
with
the
suit
of
more
than
a
dozen
car
products,
including
tick
tock
hellos
and
results,
as
well
as
the
platform
specific
to
china's
market,
including
total
doing
and
the
chigwa
vitamins
has
made
it
easier
to
and
make
fun
for
people
and
to
connect
with
consume
and
create
content
for
the
games
area.
A
We
hold
a
big
heart
for
numerous,
as
egress
to
provide
again
to
millions
of
the
players
around
the
world.
Some
of
names
are
in
flavor,
with
the
asian
tea
market.
Like
a
mobile
legend
and
a
one
piece
of
the
games,
imbalance
are
using
different
infrastructure
to
host
their
service
in
platform
for
near
years
as
the
game
studios
growing
rapidly,
and
the
players
become
more
and
more.
There
are
more
and
more
problems
and
challenges
that
for
the
traditional
deployment
of
the
game
server,
it's
obvious
that
the
skills
bring
some
of
the
following
challenges.
A
A
A
A
Therefore,
we
are
seeking
a
way
for
free
a
part
of
the
operation
efforts
its
nature
for
us
to
take
an
investigation
into
cloud-native
way
of
deployment.
The
most
benefits
are
listed
as
the
feature
below
the
first
is
directly
from
the
money.
In
the
old
day,
we
used
to
release
the
virtual
machine
instance
manually,
and
it's
hard
for
management
and
cost
a
lot
of
weight
with
the
virtual
machine
using
the
containers.
You
can
use
the
kubernetes
easily
to
control
the
scale
so
that
we
can
reduce
the
cost,
as
well
as
the
maintenance
of
highly
availability.
A
The
second
is
that
we
can
deploy
the
application
stateless
with
some
template.
The
kubernetes
will
take
all
over
the
red
things
and
we
don't
need
to
consider
the
configurations
for
the
each
can
serve.
The
third
is
that
kubernetes
will
take
all
the
jobs,
so
we
don't
need
to
think
about
manually
for
tolerance,
and
it
will
make
the
architecture
simpler.
A
The
latest
and
the
most
important
is
that
you
can
easily
construct
the
application
and
we
can
provide
the
flexibility
for
the
customization.
Then,
let's
talk
about
the
practice
and
balance
the
optimized
results
of
the
workload
is
shown
as
the
follow
in
old
days.
The
developer
needs
to
come
up
with
the
sra
team
together
and
put
many
effort
in
the
deployment
and
the
daily
operations
like
the
configurations,
should
be
rendered
and
sent
to
each
of
the
new
virtual
machine.
A
Then
we
also
need
design
a
lot
of
self
discovery
mechanisms.
We
also
need
to
launch
a
resource
in
advance
and
it
will
cost
a
little
of
time
and
money
waste.
For
now,
the
developer
can
only
focusing
on
the
template
of
the
application
and
upload
the
image
the
system
will
automatically
take
over
the
other
things,
such
as
the
service
discovery.
A
Then
the
system
will
set
up
a
trigger
defined
by
dsi
team
to
distribute
the
application
into
different
kubernetes
servers
in
the
cloud
provider.
Then
this
application
would
be
rendered
automatically
by
the
cable
like
controller
into
the
real
kubernetes
object,
including
the
agonist
game
servers
workload
to
host
the
game.
The
cross-land
object
to
host
different
as
results.
We
also
use
the
open
cruise
manager
to
inject
the
system
container
like
the
file,
bits,
etc.
A
A
In
agonist,
we
can
host
a
fleet
of
one
port
of
auto
scaling
of
the
ready
game
server.
When
the
player
wants
to
have
a
server
to
play
with
and
send
a
session
connection
request
to
the
backend
platform,
it
will
send
a
location
request
to
the
kubernetes
api
server,
which
is
watched
by
the
controller
of
agonist
platform.
A
A
We
can
let
the
user
define
their
own
components
as
well
as
they
reached,
and
let
the
si
teams
define
the
operation
strategy,
for
example,
if
the
user
would
like
to
define
the
game,
server
application
with
the
game
and
one
rds
and
a
one
load
balancer,
they
can
just
define
the
application
in
one
yama
files
and
the
controller
will
automatically
pull
up
the
game
server
and
its
corresponding
results
in
the
target
cluster.
We
can
set
up
the
strategy
that,
after
all,
the
results
ready
the
service
is
exposed
automatically
to
the
player.
A
A
Some
game
server
may
not
need
the
fire
bits
and
some
may
need
have
a
bi
logs
agent,
since
the
user
would
not
want
to
just
do
be
disturbed
by
the
operation.
Like
the
system
containers
update,
we
just
use
the
side
car
set
in
open
crews
to
manage
them
so
that
we
can
do
hot
update
or
in
place
update
without
the
disturbing
of
the
user,
to
provide
a
better
experience.
A
As
the
practice
you
can
see
the
conclusion.
It's
obvious
that
the
cloud
native
brings
a
lot
of
the
advantage
for
games
platform.
First,
we
can
just
split
a
great
big
game
server
into
different
parts
and
host
them
in
different
containers
to
provide
high
availability
for
the
efficiency
for
the
service,
also
for
preventing
the
cost
binding
for
single
provider.
A
It's
naturally
fought
again.
You
choose
different
providers
and
in
current
state
we
can
just
use
them
as
a
normal
resource
in
kubernetes,
with
greater
efficiency
by
cross-frame
for
the
complicated
application.
The
oem
model
is
a
great
model
for
us
to
adapt
into
in
the
future.
We
will
put
more
effort
in
this
part,
including
the
oem
runtime
management
and
the
call
server
for
leveraging
best
cloud
resource
to
be
a
general
view.
It's
become
a
change
for
game
platform
to
go
cloud
native
in
nowadays.
A
B
We
do
not
have
time
to
go
through
the
details
and
explanation
of
the
whole
system
that
byte
density
is
building,
which
is
a
pity
because
they
are
building
some
really
cool
things,
and
it
would
be
very
useful
to
go
through
the
whole
picture.
But,
as
I
said,
we
do
not
have
time
for
that.
So
right
now
we
are
going
to
focus
on
cross
plane
or,
to
be
more
precise,
we
are
going
to
go
through
a
practical
demo
of
how
crosstrain
works
and
how
it
fits
into
the
system
by
students
is
building.
B
B
So
it's
just
as
much
about
to
managing
resources
at
scale
as
shifting
left
and
enabling
everybody
in
an
organization
to
be
able
to
manage
their
own
things
without
having
to
spend
months
or
years
going
deep
into
kubernetes
into
cloud
and
so
on
and
so
forth.
So
here's
the
scenario,
a
simple
one.
I
want
to
run
my
applications
in
two
clusters
and
those
clusters
might
be
in
different
providers,
let's
say
aws
and
co
just
to
spice
it
up,
and
once
we
create
those
clusters,
we
need
to
manage
some
applications
in
those
clusters
and
corresponding
services.
B
There
will
be
a
back-end
application
and
a
database
that
that
application
is
using.
Let's
start
with
clusters.
How
would
the
definition
for
managing
a
cluster?
Creating
updating
and
so
on
and
so
forth,
in
aws,
to
be
more
precise,
eks
look
like
the
definition
could
be
as
simple
as
this.
I
want
to
claim
a
cluster
and
that
cluster
has
a
name,
has
some
labels
that
identify
which
type
of
a
cluster
I
want
in
this
case
that
would
be
aws
and
eks
and
some
parameters
the
size
of
the
nodes
should
be
medium.
B
Now
you
might
be
confused
and
say
hey.
What
is
that
thing
cluster
claim?
Well,
that's
something
that
operator
in
a
company
created
and
exposed
as
a
service.
So
this
is
a
custom
resource
definition
with
the
controller
called
cluster
claim
and
that
cluster
claim
can
have
many
different
implementations.
B
I
will
show
those
implementations
of
how
I
came
to
those
applications
later
for
now.
Just
think
of
this
as
a
very
simple
and
easy
way
for
everybody
to
consume
a
service,
even
though,
what's
behind
that
service
is
potentially
complex,
and
if
I
would
like
to
get
that
clustered
the
cluster
defined
in
that
yaml.
All
I
have
to
do
is
execute
cubecattle,
say:
hey.
The
namespace
should
be
this
one.
I
want
to
apply
whatever
is
defined
in
that
file
and
go
now.
B
B
Labels
that
say
hey
this
cluster
should
be
in
civil
and
should
be
ck
or
civil
kubernetes
cluster.
There
are
a
few
parameters
they're
different
than
the
other
clusters,
simply
because
in
this
provider
we
might
need
the
smaller
size
of
nodes
and
less
number
of
nodes,
but
other
than
that
the
definition
is
exactly
exactly
the
same,
or
at
least
the
structure
of
the
service
is
the
same,
and
I
can
confirm
that
we
are
talking
about
the
same
definition,
even
though
the
providers
are
completely
different
can
be
done
by
diffing.
B
Those
two
files
you
can
see
here
that
the
name
is
different
id
is
different.
The
labels
are
different
and
some
parameters
are
different,
but
both
are
based
on
exactly
same
custom.
Resource
definition
created
automatically,
but
we
are
getting
there
so
to
create
that
cluster.
The
process
is
the
same,
keep
cattle.
This
is
the
namespace
I
want
to
apply
whatever
is
defined
there
go
now
to
get
to
the
point
that
we
have
services
like
this
custom
resources
with
corresponding
controllers.
B
We
need
to
do
two
things.
We
need
to
create
a
definition
and
then
implementations
of
those
definitions
which
we
call
compositions.
So
the
definition
itself
is
something
like
this.
We
have
open
api
schema
that
says:
hey
whomever
wants
to
work
with
those
compositions,
they
should
use
this
schema
and
the
schema
has
some
properties
like
id
and
parameters
and
within
parameters.
We
have
version,
node,
size,
minimum
number
of
nodes
and
so
on
and
so
forth.
B
A
simple
definition
open
api
schema
that
defines
what
will
be
the
interface
through,
which
others
can
consume
the
service,
and
then
we
can
have
as
many
implementations
of
that
schema
as
we
need.
In
this
case
we
are
using
eks
and
civo,
but
I
also
have
implementation
for
gke
for
azure
and
so
on
and
so
forth,
and
each
of
those
implementations
is
what
we
call
a
composition
and
that
composition
implements
all
the
details.
B
Everything
that
is
required
for
somebody
to
manage
a
cluster
without
really
going
into
details
of
all
the
services
required
for
eks,
for
example,
requires
10
to
15
or
20
different
resources
to
be
combined
together.
But
all
that
is
an
implementation
detail.
It
is
in
hands
of
an
operator
who
creates
the
service
and
not
for
the
consumer,
and
the
same
goes
for
civo,
which
is
yet
another
composition
and
it
has
different
implementation,
but
it
is
exposed
as
the
same
service
consumable
by
everybody,
and
the
important
part
here
is
that
you
are
in
charge
of
creating
compositions.
B
You
are
in
charge
of
defining
what
it
means
to
have
something
in
your
organization.
This
is
not
vendor
opinionated
solution.
This
is
a
tool
that
enables
you
to
create
your
own
opinions
and
based
on
those
opinions,
services
and
combine
those
services
into
an
internal
developer
platform.
Now,
let's
take
a
look
at
what
we
got.
B
When
the
cluster
is
up
and
running,
because
remember
it's
not
only
about
creating
and
managing
clusters,
it
is
about
creating
and
managing
production-ready
clusters,
and
that's
not
only
infrastructure,
but
also
services
that
are
running
inside
those
clusters.
And
then
we
have
a
number
of
releases,
which
is
a
number
of
helm,
charts
that
will
be
applied
to
that
cluster,
and
we
have
a
node
group
and
a
cluster
and
security
group,
and
we
have
subnets
and
a
vpc
and
route
table,
an
internet
gateway
and
all
those
things
are
aws
resources.
B
And
then
we
have
sibo
kubernetes,
which
is
in
a
completely
different
provider.
But
that's
how
we
create
cluster
in
ceo,
unlike
in
aws,
there
are
differences,
but
all
those
things
are
implementation,
details
which
enhance
of
sres
or
devops
or
operators
and
not
consumers.
From
the
end
user
perspective,
all
those
details
might
be
relevant
and
we
can
say,
hey
just
show
me
what's
happening
with
my
clusters
and
in
that
case
we
can
just
list
all
the
objects
that
were
created,
which
is
cluster
claims,
and
here
we
can
see
that
there
are
two
clusters.
B
None
of
those
is
ready
because
I
didn't
give
it
sufficient
time.
So,
let's
do
just
that.
Let's
wait
for
20
minutes
or
something
like
that.
Until
aws
creates
the
cluster
see
what
takes
a
minute
or
two
and
then
we
can
proceed.
I
will
not
bother
you
waiting.
I
do
not
expect
you
to
wait.
So
let
me
fast
forward
to
the
end
of
the
process
and
I'm
back
here
we
go.
Two
clusters
are
created.
You
can
see
that
by
the
column
ready
set
to
true
so
clusters
are
done.
B
We
can
see
that
in
case
of
eks
control,
plane
and
node
pool
are
active
in
case
of
cevo.
I
made
a
mistake.
I'm
not
populating
those
fields.
So
shame
on
me,
anyways.
If
you
ignore
that,
I
made
a
mistake.
You
can
see
here
that
both
clusters
are
ready
and
that's
what
really
matters
and
now
we
can
use
it
to
deploy
applications
and
services
and
a
few
other
things.
B
But
before
we
do
that,
let
me
retrieve
cube
config,
which
crosstalk
is
stored
in
a
secret
after
it
created
the
cluster,
and
then
I
can
use
that
cube
config
to
do
some
funky
stuff,
so
cube
cattle,
etc,
etc,
etc,
and
the
outcome
goes
to
cubeconfig,
eks
yaml,
and
now
we
can
use
one
of
those
two
clusters
and
to
be
on
the
safe
side.
Let's
retrieve
the
number
of
nodes
and
there
we
go.
There
are
three
nodes,
that's
my
case
cluster.
So
obviously
it
exists.
It
was
created
by
crossman.
B
It
is
managed
by
crossplane
with
drift
detection,
reconciliation
and
so
on
and
so
forth,
and
now
I
can
use
it.
I
can
deploy
my
application
inside
of
that
cluster
or
any
other
cluster.
In
this
case,
I'm
using
only
two,
but
normally
you
would
need
to
imagine
dozens
or
hundreds
of
clusters
being
managed
like
this.
But
before
I
proceed,
let
me
retrieve
cube
config
of
the
second
cluster
as
well,
so
that
we
can
use
both
the
second
one
is
in
cevo
and
there
we
go.
B
B
B
And
then
I
will
apply
that
manifest
to
the
civil
cluster.
And
after
that,
I
will
retrieve
all
the
resources
inside
of
that
specific
cluster
in
the
production
namespace
and
see
what
I
got
and
I
got
and
remember
this
is
a
silly
demo.
I
go
to
deployment
the
replica
set
and
so
on
and
so
forth.
Now,
let's
move
on,
let's
make
it
more
interesting.
B
Let's
say
that
we
want
not
simply
an
application
there,
but
application
with
all
corresponding
kubernetes
resources
plus
a
database,
and
that
database
in
this
case
could
be,
let's
say,
rds
in
aws
and
all
that
should
be
connected
and
glued
together.
How
would
they
do
that
without
going
into
details?
How
aws
works,
how
kubernetes
works?
How
would
they
do
that
as
a
developer
as
a
consumer
of
a
service?
The
definition
could
be
like
this
now.
B
The
first
part,
the
first
of
the
two
resources
defined
here
is
almost
exactly
the
same
as
the
one
I
just
applied
to
sivo.
It
is
an
application
claim.
Everything
is
the
same
except
matching
labels.
This
time
the
type
says
backend
db,
so
this
same
definition
will
use
a
different
composition
that
will
know
how
to
apply
and
which
resources
to
create
when
working
with
back-end
applications
connected
to
a
database,
and
then
we
have
a
second
definition
over
there.
B
That
says-
and
I
would
also
like
to
claim
an
sql
database
and
that
database
through
the
matching
labels
should
be
running
in
aws
and
should
be
postgres
again.
I
can
have-
and
I
do
have
different
implementations
database
in
azure
database
in
google
database
in
aws,
mysql,
postgres
and
so
on
and
so
forth.
But
all
those
things
are
implementation
details.
The
end
user
just
needs
to
specify
labels
and
say
pro:
this
is
the
provider.
This
is
the
type
of
database.
B
Go
I
mean
it's
not
go.
You
need
a
couple
of
parameters
like
this
is
the
version
of
the
database.
I
want
this
size
of
database
again.
I
do
not
know
the
real
sizes
in
aws,
but
I
will
say
small
and
you
figure
it
out
and
the
namespace
is
production
and
finally
create
a
secret
called
silly
demo
and
put
the
authentication
to
the
database
there
so
that
the
application
can
use
it.
So.
B
That
manifest
with
cube
cattle,
etc,
etc,
etc.
You
know
the
command
and
then
two
new
resources
were
created
this
time
in
the
eks
cluster,
not
in
civil,
and
let's
take
a
look
at
what
we
got.
If
I
list
all
the
resources
plus
ingresses
in
the
production
namespace
of
that
specific
cluster,
we
can
see
that
we
got
the
service
and
the
deployment
and
deployment
created.
Replica
set
and
replicas
have
created
pods
and
we
have
ingress.
There
is
nothing
really
special
here,
except
the
status
of
the
pod.
That
says,
create
container
config
error.
B
It
is
failing
big
time,
but
that's
normal.
It
is
failing
because
this
composition
knows
that
it
should
attach
a
secret
with
the
authentication
to
the
database,
and
it
cannot
attach
a
secret
because
secret
doesn't
exist
and
it
doesn't
exist
because
the
database
was
not
created
yet,
but
when
crosstalk
creates
the
database,
it
will
create
a
secret,
and
this
pod
will
automatically
work
because
the
secret
will
be
there
and
it
will
be
able
to
install
ct
itself
now.
B
To
see
the
statuses
of
those
specific
resources,
application
claims
and
sql
claims,
we
can
just
list
those
two
and
then
we
see
that
application
claim
is
ready
and
the
database
is
not
yet
ready,
but
it
will
be
ready
soon.
It
takes
five
minutes,
I
think,
to
create
an
rds
instance
in
aws.
So
let
me
fast
forward
to
the
end
of
the
process.
A
B
If
I
list
all
the
managed
resources,
I
can
see
a
bunch
of
resources.
Over
there,
a
new
security
group
was
created,
internet
gateway,
vpc,
subnets
objects,
no
objects
are
from
different
claims,
db,
subnet
group,
and
what
matters
the
most
rds
instance-
and
it
is
ready,
so
my
application
should
now
get
the
secret
it
needs
and
connect
to
the
database.
So
let
me
list
the
resources
in
the
production
namespace
one
more
time
and.
B
The
pod
is
up
and
running,
it
got
the
secret,
it
needed
it
connected
to
the
database
and
both
of
them
were
created
through
compositions
and
we
can
live
happily
ever
after.
Thank
you
so
much
for
watching.
I
hope
that
this
was
useful.
If
you
have
any
questions
and
if
you
have
any
time
remaining,
we
would
be
really
happy
to
answer
any
inquiries
you
might
have
or
even
better
come
to
upbound
booth
and
we
can
chat
more
over.
There
ask
anything-
and
I
will
answer.
Thank
you
so
much.