►
Description
Keynote: Tencent: Kubernetes in the Billions – Yunong Xiao, Director of Engineering, Principal Architect, Tencent Cloud
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
At Tencent, our business touches everything from gaming, social media, payments, to cloud computing. We’d like to share our story of how K8s is broadly used at Tencent, taking care of our infrastructure and platform needs while freeing us to focus on our strategic and business imperatives. We’ll go over a few use cases and their architecture on K8s: 1) The Tencent GPU computing platform uses K8s to manage our large GPU fleet, allowing us to unify resource scheduling and improve cluster utilization 2) The WeChat payments platform utilizing K8s to enable customers to send over 1 billion daily transactions 3) Building the Tencent Kubernetes Engine (TKE), which provides container-centric, highly scalable and high-performance container management services.
https://sched.co/NuHc
A
The
gaming
which
everyone
is
playing
at
home,
the
cloud
to
everything
else
right
advertisement,
and
it's
very
broad
and
all
of
these-
will
these
business
groups
kind
of
start
off
alone
and
independently
and
evolve
with
their
own
architectures
and
over
time
they
grow
and
they
become
really
big
revenue.
Streams
of
big
streams
of
business
and
big
bring
lots
of
revenue
to
$0.10
right,
and
so
you
know,
as
everyone's
talked
about
today,
which
everyone's
trying
to
move
on
to
the
cloud.
A
What's
that
one
technology
that
can
really
help
standardize
our
infrastructure
at
$0.10
onto
the
cloud
and
it
was
in
the
end,
it's
really
containers
and
kubernetes,
and
this
is
not
a
chakra
for
everyone
right
and
so
today,
what
I
really
wanted
to
talk
to
you
is
how
we're
leveraging
and
building
kubernetes
at
$0.10
a
and
our
history
of
kubernetes.
So
we're
going
to
talk
about
this
through
kind
of
two
different
lenses.
What
first
walk
through
the
container
history
at
Tencent
I?
A
Think
this
might
surprise
some
people
and
then
later
on,
we'll
talk
about
some
of
our
use
cases
using
containers.
So
let's
go
so
history
of
containers,
a
ten-cent.
Many
of
you
probably
don't
know.
We
started
really
early
in
2009.
Actually
but
I
didn't
know.
So
I
did
some
research
with
the
system
called
tor
cup,
and
that
was
our
initial
container
platform
attention
in
2009
and
we're
around
2013.
A
So
torque
I'll
write
a
little
bit
on
the
history
in
2009.
Actually
at
$0.10
there
was
no
virtualization.
Everyone
was
on
bare
metal
and
using
physical
machines,
and
you
know
the
one
of
the
reasons
why
we
wanted
to
do.
Virtualization
was
to
improve
resource
utilization,
increase
the
vulvar
productivity
and
reliability
and
specifically
around
resource
utilization.
It
was
shockingly
low
at
that
time.
Right.
Cp
utilization
was
hovering
around
10%
as
an
average
across
the
entire
fleet.
So
that's
really
really
low
utilization
obviously
terms
the
cost.
A
That
was
not
was
not
ideal
and
we
faced
some
challenges
right.
None
of
these
new
technologies
like
kubernetes
or
doctor,
or
swarm
or
Maysles,
and
these
things
existed.
So
when
we
were
thinking
about
building,
you
know
a
virtualization,
the
engine
of
virtualization
stack
internally.
You
know
how
to
how
do
we
go
about
building
this
right?
A
So
in
terms
of
the
technology
stack,
we
in
2007
cgroups
was
born,
and
so
by
the
time
2009
rolled
around,
we
were
thinking
about
this
project.
You
know
we
really
went
all
in
on
our
socialization
or
we
skip
our
virtualization
was
then
at
a
time
and
kind
of
we
loved.
All
the
virtues
of
always
virtualization,
with
the
fact
that
thin
and
light
expect
to
start
and
again
I
try
to
improve
utilization
right
in
terms
of
the
orchestration
system.
A
Not
none
of
these
technologies
that
we
love
to
use
today
existed,
and
so
we
actually
found
an
open
source
framework
called
condor.
Now
it's
almost
HT
condor
and
that
was
a
open
source
orchestration
system
developed
out
of
the
University
of
Wisconsin
with
about
20
years
of
experience.
So
that's
what
we're
really
leveraged
at
the
time
for
our
H
for
tour
cuts
back
I
just
want
to
say
tortoise,
stands
for
$0.10
Orca
or
could
as
a
killer
whale.
We
were
into
whales
and
containers
before
doctor
was
so
you
know.
A
That's
fine
in
terms
of
the
improvements
are
the
one
thing
that
I'll
talk
about
here.
If
you
only
have
time
is
we
really
made
everything
elastic
every
single
resource,
so
in
terms
of
CPU
IO
disk
memory
network,
everything
was
elastic
even
in
2009.
I
was
one
of
the
founding
principles
of
torque
ax,
and
the
reason
is
that
you
know
we
wanted
to
save
cost,
but
we
also
wanted
to
improve
early
in
utilization
right
so
being
able
to
burst
a
CPU
or
burst
past
your
memory
limit.
A
If
there
was
free
resources
on
that
box,
it
was
really
handy,
make
things
much
more
reliable.
You
weren't
going
going
around
and
killing
containers
that
exceeded
the
limits
and
at
the
time
you
know,
went
Orca
was
running.
We
actually
had
some
really
key
internal
customers
like
WeChat
and
soso,
which
is
our
search
product
at
the
time
and
things
like
QQ
zone.
So
even
then
you
know
we
were
landing
a
lot
of
our
critical
businesses
on
top
of
containers
so
fast
forward
into
2013.
A
You
know,
containers
were
going
a
lot
on
tour
cut
and
people
were
liking
it
and
we
thought
it
would
be
a
good
idea
to
move
our
big
data
platform
as
well
on
top
of
torque
right.
So
this
is
where
I
DuPont
Orca
came
along
hot,
and
the
thing
that
was
really
important
for
us
for
Big
Data,
obviously
showing
the
sign
of
the
times
in
2013,
was
having
100%
compatibility
with
Hadoop
so
that
our
customers,
our
engineers,
did
not
have
to
change
a
single
line
of
code.
A
Everything's
going
on
fine
and
then
2014
rolls
around
and
2014
was
when
kubernetes
was
first
released.
So
0.1
and
we
were
watching
at
the
time-
is
we're
very
interested
in
open
source.
But
0.12
didn't
quite
have
the
features
they
didn't
have
any
of
the
features.
So
we
wanted
right.
So
we
waited
a
year
until
2015
when
the
kubernetes
turned
1.0
and
that
was
kind
of
an
inflection
and
it's
hurting
employee
for
$0.10
of
them
for
those
containers
back
at
the
time,
because
what
torco
was
really
great
and
we
learned
a
lot
of
lessons
from
it.
A
A
So
things
like
alerting
a
monitor,
and
all
of
that
is
internal,
and
so,
as
we're
moving
on
to
the
cloud
and
trying
to
build
a
cloud
stack
that
our
private
cloud
customers
can
use,
you
really
can't
rely
on
internal
systems
if
you're
trying
to
build
a
non-prime
cloud
solution
and
also
we
want
to
embrace
open
source
right.
You
know,
there's
no
need
to
build
a
lot
of
stuff
on
your
own
if
you
can
just
use
open
source
components.
A
So
all
of
this
meant
that
you
know
our
decision
in
2015
was
to
build
something
called
Gaia
stack,
which
I
hope
of
some
of
you're
familiar
with,
and
we
really
built
Gaia
stuck
on
top
of
kubernetes
and
leveraging
all
of
the
lessons
that
we
learned
from
torque
ax.
So
things
like
performance
right.
You
know
we
still
had
the
really
great
elastic
resource
management
that
we
built
in
on
top
of
kubernetes.
A
This
is
a
recurring
theme.
Actually,
everyone's
buildings
to
these
set
based
just
shoot
your
file
systems,
and
we
did
that
too,
to
increase
performance,
reliability
and
again
for
CNI.
For
the
network
fabric,
we
built
an
open
source,
plugin
called
galaxy,
which
is
our
own
high-performance
Network
fabric,
which
really
increased
on
performance
right
from
the
reliability
angle.
You
know,
especially
in
2015
at
the
time.
Some
of
these
open
source
components
didn't
quite
have
the
reliability
characteristics
that
we
need
it
for
core
business
and
so
things
like
the
docker
registry.
A
We
build
our
own
highly
available
docker
registry
that
could
scale
to
10
cents
needs.
We
put
hundreds
of
additional
systems
and
cluster
know
metrics
by
default
into
the
stack
and
it's
really
mental.
We
were
able
to
really
monitor
every
single
angle
of
the
other
technology
and
output
of
the
system
to
improve
reliability,
and
a
really
key
point
here
is
also
we
built.
You
know
we
modified.
A
We
made
changes
to
kubernetes
so
that
there
were
no
single
points
of
failure
anywhere
and
we
really
made
it
extremely
highly
available,
which
is
really
important
and
so
and
also
some
of
the
features
that
our
developers
really
want
to
include
things
like.
You
know:
CIC
and
native
DevOps
integration
right
with
Jenkins,
GPU,
virtualization
I,
think
we
were
one
of
the
first
or
only
people
to
folks
to
to
really
build
GPU.
The
worst
virtualization
right
into
kubernetes
and
things
like
beam
will
do
advanced
canary
deployment.
A
Highly
skilled
container
platform,
also
unproven
Ares
that
you
know
had
a
lot
of
usage
and
a
lot
of
dodging
attention
scale,
and
so
what
we
decided
ultimately
was
to
actually
unify
the
two
platforms,
so
so
today,
TKE
powers,
both
internal
and
external
customers.
So
it's
the
same.
It's
the
same
codebase,
it's
the
same
platform,
the
powers,
both
our
cloud
customers
and
private
cloud
customers,
as
well
as
all
of
our
internal
customers,
and
you
know
getting
that
leverage
and
that
kind
of
dogfooding
really
helps
us
push
the
path
and
forwards.
A
So
we
just
talked
a
little
bit
about
the
history
and
then-
and
you
know,
we
just
talked
about
this
right
ten
years
of
a
container
experience.
You
know
it's
also
fully
compatible
with
kubernetes
and
unify
cloud
platform.
So
we
just
talked
a
little
bit
about
the
history.
Next
I
wanted
to
walk
through
some
of
the
use
cases
internally,
a
ten-cent
that
is
using
kubernetes
and
TKE
today.
A
So
we'll
talk
a
little
bit
about
gaming
and
its
different
workload,
which
is
you
know
very
different
custom
kubernetes,
we're
closed
that
doesn't
really
quite
fit
within
the
mold
of
some
of
these
CRTs.
That
exist
today
we
also
talked
about
WeChat
pay
and
its
challenges
around
stability,
right
and
scaling.
And,
lastly,
we'll
talk
about
advertising
and
it's
big
data
requirements
that
places
some
some
different
requests
on
our
on
our
kubernetes
engine,
aunty
K.
So
gaming
gaming
is
huge
attention
as
I'm
sure.
A
A
lot
of
people
know
you
know
all
the
most
of
the
popular
games
are
either
built
or
made
or
licensed
by
$0.10,
and
there
are
some
interesting
architectural
challenges
from
a
gaming
perspective,
because
we
acquire
a
lot
of
companies
and
we,
you
know,
with
with
the
hundreds
of
thousands
and
long
tail
games.
A
lot
of
these
services
tend
to
be
pretty
monolithic
right.
If
they're
big
heavy
services,
they
require
a
lot
of
resources.
A
They
spin
up
independently
and
a
lot
of
them
are
black
boxes
that,
because,
through
mergers
and
acquisitions,
that
we
can't
really
change,
we
can't
change
the
configuration
the
require
static,
IPS
and
most
of
them
are
stateful.
So
all
of
these
things
really
don't
lend
itself
really
well
to
working
within
the
kubernetes
mode
right,
and
so
how
could
we
solve
some
of
this
right?
A
Well,
in
terms
of
the
large
monolithic
architectures,
we
could
solve
that
using
the
the
excellent
dependency
management
spot
along
by
Daka
that
we
support
as
part
of
kubernetes,
something
that
was
extremely
important
for
gaming
is
have
static
IPS
for
a
lot
of
these
services.
So
when
they
crash
right
when
they
come
back
up,
they
have
the
same
IP,
because
we
can't
change
the
underlying
architecture
of
some
of
these
third-party
licensed
games
and
the
thing
that
I
think
some
other
folks
talked
about
today,
which
is
really
around
custom
resource
management.
On
kubernetes.
A
You
know
this
is
actually
very
interesting.
It's
a
recurring
theme.
We
also
built
something
called
ta
PP
write,
which
really
helps
to
manage
things
like
being
able
to
really
bash
the
play
outside
of
the
spatial
set.
Cid
that's
offered,
so
we
can
deploy
much
more
quickly
and
also
being
able
to
just
you
know
when
a
service
dies
to
not
have
it
be
automatic
cleanup
because
they're
stateful
and
you
want
to
wait
for
an
operator
to
go
and
restart
them,
so
you're
not
losing
state.
A
So
all
these
all
these
changes
that
we're
making
to
the
kubernetes
engine
will
really
allow
us
to
land
this
gaming
workload,
which
is
our
critical
revenue,
driving
use
cases
on
top
of
kubernetes.
So
next
I
want
to
talk
about.
We
shall
pay
right.
Everyone's
got
WeChat
in
their
pocket
and
it's
really
got
massive
scale
right.
You
know
a
billion,
a
billion
monthly
active
users,
a
building
financial
transactions
a
day
right
through
Chinese,
New
Year,
there's
a
huge
skill,
as
with
everyone
sending
red
packets
and
what's
really
important
here,
would
reach
a
pay.
A
Is
the
stability
angle
and
the
scale
right?
You
know:
how
do
you
support
the
sheer
size
of
customers
and
really
critically?
How
do
you
maintain
the
reliability,
instability
and
again,
I
think
this
is
where
the
ten
years
of
experience
that
we've
had
offered
in
containers
in
production
really
come
into
play,
starting
all
the
way
since
work,
alright,
so
building
the
kubernetes
engine
and
and
making
improvements
to
make
sure
that
there
are,
it
is
H
a
and
there
are
no
single
point
points
of
failure
anywhere
right.
A
Having
hundreds
of
additional
Bolton
system
metrics
means
that
we
can
operate
and
debug
and
troubleshoot
much
more
quickly
and
and
be
more
stability
to
the
engine
right.
A
really
critical
thing
that
we
build
into
our
engine
is
the
ability
to
do
live
upgrades.
So
you
know
we
can
upgrade
the
underlying
platform
without
inconveniencing
or
disrupting
the
tenants
that
live
on
top
right
and
things
like
isolation
between
networks
and
being
able
to
control
the
amount
of
bandwidth
us
that's
utilized
between
tenants.
Again.
A
All
of
this
helps
to
make
sure
they
make
Tencent
kubernetes
a
really
stable
product
for
internal
and
are
very
critical
use
cases
like
WeChat
pay,
and
so
the
last
use
case
I
want
to
talk
about
with
advertising
Tencent
advertising,
and
this
is
really
different
right.
This
is
kind
of
more
of
a
Big
Data
workload
in
which
a
machine-learning
workload,
so
you
know
a
lot
of
our
engineers.
Our
air
engineers
want
to
use
things
like
cube
flow.
They
want
to
run
big
data
and
that
really
want
you
want
to
be
leveraging
these
really
expensive.
A
Gpus,
and
traditionally
you
couldn't
do
that
in
a
virtualized
way.
On
top
of
kubernetes
right,
there
are
some
challenges
around
working
with
big
data
with
small
files
into
the
file
systems.
So
we
made
some
changes
right.
We
improved
the
small
file
on
capability
of
our
staff
file
system
that
really
increased
performance
by
about
10x
for
big
data
workloads,
because
we
have
a
folk
kubernetes
in
ableton
hundred
percent
palpable
kubernetes.
A
You
could
just
drop
your
machine,
turning
workloads,
I'm
hopeful
on
to
Tk,
and
they
would
just
run
right
and
the
GPU
virtualization
here
I
think,
is
really
important.
This
is
actually
a
really
awesome
part
of
TKE.
You
know
we're
able
to
bring
GPUs
online
virtualized
them
and
really
go
all
the
way
down
to
a
tenth
of
a
GPU
and
then
that
out
to
our
customers,
so
where
traditionally,
you
have
to
buy
these
expensive
GPUs
and
they
sit
around
on
racks.
A
So
today
we
talked
about
a
couple
things,
one,
which
is
our
story
or
history
of
containers,
the
$0.10
a
decade
of
production
experience
starting
all
the
way
from
2009,
where
we,
you
know
2
years
removed
from
the
launch
of
cgroups.
We
do
in
OS
virtualization.
All
the
way
to
today
in
2019,
where
we
have
TKE,
which
is
our
unified
cloud
container
platform,
and
what's
really
important
here
is
T
key
is
for
us
a
unified
solution
right.
A
We
are
using
it
for
internal
key
internal
customers
like
WeChat,
like
advertisement
and
like
gaming,
and
when
you
use
it
on
top
of
tenzin
cloud
you're
using
the
exact
same
code
base
and
the
exact
same
stack
for
our
customers,
and
so
we
think
this
is
really
important
for
us
for
$0.10.
If
you
will
do
things
both
internally
and
externally.
Only
by
doing
this
can
we
stay
close
to
the
customer
on
cloud.
A
A
None
of
this
would
have
been
possible,
I
think
to
get
this
kind
of
adoption
without
the
open
source
without
the
open
source
efforts
of
the
cloud
native
foundation,
right,
I,
think
having
kubernetes
having
dr
having
all
these
things
being
the
open
source
is
really
what
helps
to
drive
adoption.
You
know
free
and
open,
and
it
I
left
everything
crossed
broad.
You
know
geopolitical
boundaries
and
makes
things
free
and
easy
to
use
right
and
we
talked
about
being
100%
compatible
with
kubernetes
and
I.
A
Think
that's
really
what's
driving
the
success
of
virtualization
containers
and
kubernetes
forward,
and
you
know
I,
just
wonder
what
I
really
wanted
to
get
across
was
that
open
source?
Is
this
great
idea
and
nowhere
do
I
see
it
more
embodied
than
the
cloud
native
foundation
and
really
pushing
container
and
cloud
technology
forward?
Thank
you
very
much.