►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Model and Operate Datacenter by Kubernetes at eBay - Xiaogang Xin & Tommy Mei, eBay
Kubernetes provides amazing model abstraction of your application. Kubernetes also runs a group of controllers that take care of routine tasks to ensure the desired state of the application matches the observed state. However, would you think Kubernetes can even model your datacenter in large scale and also operate the datacenter through controllers? This is true in eBay. We are facing large scale clusters running on tens of thousands machines. We want a central service, a unified interface to operate both infrastructure and application.We are leveraging on Kubernetes CRD to model our infrastructure, such as, Rack, L2Domain, NetworkScope, Asset, Flavor, etc. And we also use Kubernetes CRD to model K8sCluster as well. In this talk, we will share how we model datacenter from 3 layers: - Fleet management - Configuration management - K8S cluster management
To learn more click here: https://sched.co/FuKD
A
Here
on
mission
dolmens
ago,
whose
home
teens
I
am
older
than
a
woman
seen
in
the
what
he
should
call
to
Anna,
so
it
says
all
the
Kawai
light
kingdom.
So
many
minutes
away,
your
fate
hung
called
on
TV,
integral
secret
into
Malibu
destroyer
in
eighteen,
the
top.
So
it
says
all
the
cocoon
cinnamon
okay,
what
kind
of
wayside
so
you
want!
My
I
should
prefer
a
human
diet,
yeah,
okay,.
A
C
B
You,
my
name,
is
Tommy
and
senior
manager
of
the
set
operation
in
eBay
I'm
running
like
the
platform
reliability
team
in
eBay,
and
we
managed
that
the
hinges
and
a
thousand
server
in
eBay
to
serve
like
the
thousands
application
in
eBay
for
the
global
platform.
So
shark
is
major
for
like
the
developing
the
communities
in
eBay,
that
is,
we
run
into
data
center
fleet
to
serve
the
traffic.
So
our
the
purpose
is
gonna,
be
keep
this
large-scale
the
data
center
to
serve
the
traffic
and
keep
that
reliable
and
a
scalable
okay.
B
B
A
C
B
Also
that
we
were
showing
about
how
we
designed
the
kubernetes
for
eBay
and
the
later
I
will
share
some
like
experience
from
like
the
operations
perspective
that
how
the
kubernetes
magic
help
us
to
manage
those
kind
of
large-scale
the
data
center
in
eBay.
Okay,
let
me
in
like
the
hub
the
room
counter
like
the
shotgun
to
give
us
the
definition.
I
will
come
back
later.
Yes,
thank
you.
Thank
you.
Thank.
A
You
Tommy,
so
this
is
our
agenda
for
today.
So
first
I
will
talk
a
little
bit
about
eBay
the
business
and
they
are
fleet
and
how
kubernetes
manage
our
fleet
in
magic,
kubernetes
concept
of
a
model
and
a
controller
is
the
most
impressive
for
us
so
that
we
just
think
to
use
kubernetes
model.
Our
data
center
by
these
models,
Tommy
will
show
you
how
easy
it
is
to
operate
her
Tennison
data
center
and
cluster
in
large
skill.
A
Okay,
at
the
end,
we
will
try
to
use
five
minutes
for
Q&A,
but
I'm
not
sure
whether
it
is
enough.
We
can
have
a
further
discussion
throughout
another,
our
other
channels.
Okay,
this
is
about
eBay.
Ebay
is
internet
e-commerce
company
founded
at
1995
in
California
till
q3.
This
year
we
have
177,
meaning
active
buyers
all
over
the
world.
A
A
A
A
A
A
Infrastructure
should
play
magic.
How
this
is
my
session
right?
My
presentation,
it's
a
long
way
for
us
to
manage
a
massive
infrastructure
at
the
very
beginning,
eBay
most
almost
all
the
applications
in
eBay
miranium
terminals
start
form
on
Tuesday
routine.
We
build
our
first
cloud
platform
based
on
VMware
virtualization
technology,
but
two
years
later,
OpenStack
is
the
hottest
open
source
cloud
platform
in
the
world,
and
then
we
build
our
second-generation
cloud
on
top
of
OpenStack,
but
until
2015
we
found
that
our
business
is
running
on
three
different
cloud
platforms
with
three
different
technologies.
A
A
Remember
what
I
said
municipal
right
infrastructure
should
play
magic,
but
how
I
think
you
have
the
answer?
That
is
kubernetes!
That's
why
all
of
you
are
here-
and
we
are
here
with
three
years
at
effort.
We
run
almost
all
the
workloads
and
evening
on
kubernetes.
We
run
our
search
engine
on
kubernetes,
which
is
the
CPU
and
a
memory
intensive
application.
We
run
Hadoop
on
kubernetes,
which
use
CPU
and
disk
heavily.
A
A
Declarative
kubernetes
has
a
straightforward,
simple
architecture
like
this,
which,
in
which
user
can
declare
what
they
want,
or
we
can
call
it
a
whizbee
what
it
should
be:
kubernetes
api
and
a
bunch
of
controllers,
watch
these
models
and
convert
the
wispy
to
the
real
things,
such
as
container
whip
volume
and
a
controller
will
also
reflect
the
worry.
What
it
really
is
back
to
the
model
as
status
controller
is
a
loop
so
that
it
always
try
to
reconcile
the
whizbee
with
a
worry
in
case
of
the
reality
has
drift
with
their
expectation.
A
A
A
It's
amazing
that
we
have
a
kubernetes
to
model
your
applications,
so
that
engineer
can
transit
from
machine
centric
to
application.
Centric,
okay,
but
the
question
is:
have
you
ever
think
about
these?
Two
questions
in
your
mind:
what
about
the
kubernetes
cluster
itself?
What
about
the
fleet
kubernetes
is
running
on?
A
A
If
you
want
to
build
a
kubernetes
cluster
from
scratch,
where
you
should
start
from
first,
you
need
to
have
some
assets
on
some
racks,
the
availability
zone
and
the
region
in
your
data
center,
and
you
need
to
connect
your
assets
with
nano
device,
such
as
a
switch
right
after
that,
you
need
to
provisioning
to
install
operating
system
to
get
a
compute
node,
which
can
be
SSH
but
I'm.
Sure
that
when
you
have
thousands
of
machines
or
compute
knows
it's
not
a
good
idea
to
install
kubernetes
by
SSH
to
do
it
manually.
A
I,
don't
think
it's
a
good
idea.
You
need
a
configuration
management
system
to
help
you
after
you
get
thousands
of
or
tens
of
thousands
of
computing
nodes
with
the
configuration
management
layer
ready
it's
time
for
you
to
spin
up
a
kubernetes
cluster,
and
it's
really
easy
in
each
of
this
step.
Our
team
extract
the
models,
different
models
and
I
will
share
in
detail.
A
A
A
A
A
So
all
these
things
we
have
to
model
the
onboard
process
in
data
center.
So
by
these
models
we
can
use
kuba
city
or
cat
to
get
all
the
regions.
Availability
zones,
nano
work,
zones,
racks
and
computer
assets.
I
do
not
share
any
controller
here,
because
different
company
has
different
environment.
The
controller
logic
is
different,
but
it's
really
a
good
case
to
understand
here.
A
A
After
you
get
your
compute
a
set,
what
you
want
to
do
you
need
to
install
operating
system
on
your
asset.
You
might
have
different
an
operating
system
for
different
use
cases
eBay.
What
do
we
care
for
an
OS?
Is
the
kernel
version,
OS
version?
What
is
the
OS
tree?
Url
OS
image
depend
on
different
OS
management
technology.
A
Flavor
is
more
complex.
Flavor
defines
how
you
use
your
computer
SS
resources
such
as
a
CPU
memory
and
a
disk.
The
most
important
part
is
the
disk.
You
need
to
design
how
you
want
to
partition
your
disk,
for
example,
how
many
discs
you
want
for
a
BIOS
boot
root
disk?
What
is
the
file
system
you
want?
Do
you
want
a
VM
set
up
or
not?
That
is
a
flavor.
A
A
As
I
said,
if
you
have
a
large
cluster,
it's
not
a
wise
decision
to
install
kubernetes
directly
on
thousands
for
nose
right.
You
need
a
configuration
management
system.
We
use
salt
stack
tomorrow.
A
salt
cluster
is
easy.
First,
you
need
to
have
a
salt
master
on
which
you
should
install
soft
state
tree
write
an
eBay.
We
put
our
soft
state
tree
in
github,
so,
okay,
repo
with
the
revision
ID,
can
represent
a
soft
state.
Tree
pillar
is
also
important
because
pillar
is
a
class
2
white
configuration.
A
A
I'm
sure
you
are
familiar
with
kubernetes
deployment,
a
rolling
update
strategy.
We
want
to
do
the
similar
to
our
cluster.
We
want
to
class
the
level
change
by
a
rolling
update
way
per
our
experience.
Rack
Barack.
It's
a
good
practice
to
follow,
because
usually
your
application
uses
application,
use,
rack
level,
anti
affinity
for
resilience.
A
A
You
just
need
to
define
this
model
and
use
cube
city
or
create
a
cluster
to
use
one
cluster
provision.
Another
the
cluster
controller
will
do
similar
things
like
a
crew
Bob
for
you,
but
it's
more
easy
and
simple,
because
you
only
need
to
figure
out
to
choose
some
compute
nodes
and
the
labor
weight,
for
example,
as
a
kubernetes
master,
they
use
a
salt
to
install
the
HDD
API
server
controller
managers
scheduler
in
minutes
right.
It's
really
easy.
A
B
Introduced
a
lot
of
details
about
the
kubernetes
in
eBay
how
to
design
so
lots
of
details.
Maybe
you
guys
maybe
needed
some
time
to
absorb
that
so
right
now,
I
will
give
you
guys
some
easy
time
to
understand
how
this
print
magic
to
the
operation,
so
usually
that
you
know
recently,
we
have
like
the
holiday
season
is
coming
and
you
guys
maybe
just
have
a
good
w11
right.
Do
you
guys
have
a
double
11,
good,
okay,
some
scenes?
Okay,
some
of
us
check
it
ahead.
B
Okay,
so
that
is,
for
the
customer
is
good,
but
for
the
operation
that
phrase
be
big
challenges
so
one
day
morning
that
that
you
go
to
the
office
and
be
you
tell
you
hey
Tommy,
we
got
you
like
the
build
another,
maybe
ten
thousand
server
to
handle
the
traffic
I.
Usually
a
car
I
said
ten
thousand.
Are
you
kidding?
When
did
you
want
that
today?
I
say:
okay,
because
right
now,
why
did
that
say
so
easy,
because
I
had
a
kubernetes
in
eBay?
B
So
then
we
just
I
just
run
the
Coco
Cooper
City
out
get
computer
lists
to
see
how
many
server
I
have
because
every
single
will
be
defined
in
a
kubernetes
already.
So
they
were
list
to
show
how
many
so
do
we
have
in
a
server
status.
Then
I
just
go
there
to
say:
hey
Kay
I
need
to
like
the
computer,
node
llamo
fire
already
define
which
application
they
won't
be
and
which
server
I
will
be
at
and
the
provision
as
the
per
hour.
B
The
purpose
then
I
define
which
the
role
which
assets
what's
flavor,
what
OS
version,
which
part
well
isn't
then
llamo
flower
is
ready,
I,
just
clicked
kuba
control,
crate,
okay,
press
the
enter
things
going
and
then
I
just
stand
up
grab
some
coffee
relax
chat
with
you
guys
head
here.
Everything
is
funny
at
the
back
and
the
only
now
it's
come
back.
Okay,
Soviets
ready,
10,000,
maybe
20,000
already
be
there.
So
they're
gonna
be
easy.
B
You
know
nyet
you're,
looking
at
looking
at
what
kind
of
POS
I
want
to
provision
did
I
set
up
the
SSH,
you
ready
and
there's
the
code
ready,
because
every
scenes
can
show
the
back
kubernetes.
Okay.
Next
people
will
ask
okay
when
you
finish
the
server,
but
in
case
you
said
everything
is
can
do
by
the
saw
but
house
the
swordmaster
get
down.
B
Are
you
are
the
kubernetes
class
of
worker
here
are
out
control,
I
always
say
no,
because
even
the
sort
of
must
the
header
of
particle
Burnett
is
controller,
is
be
kubernetes,
control
dies
themselves,
so
it'll
come
that's
gotta
be
magic.
We
mention
that
that
is
Khmer.
You
can
even
manage
itself
function
by
community
selfs,
so
era.
Here
you
will
see
still.
Okay
still
I
was
sure
how
many
so
again
and
I
just
say.
Okay,
this
application
already
defined
is
called
saw
the
master.
B
So
so
the
master
tell
the
whole
application
kubernetes
itself
defined
them
itself
say
hey.
What's
the
like,
the
application
we
are
and
what's
the
pillow
securities
and
which,
like
the
repo
location,
is
and
I
still
just
go
to
the
coop
control
crate
this
command
to
recreate
all
the
solder
master
in
the
kubernetes
cluster,
then
so
minion
will
automatically
to
Flags
the
local
master
information
we
control.
So
again
you
were
having
the
fish
in
my
previous
coffee
when
I
gotta
go
back
to
my
seat,
even
the
class
the
saw
the
class
master
was
done.
B
I
even
can
easy
to
handle
that
so
that's
kind
of
be
magic.
We
have
in
the
eBay
for
the
community
by
the
Canaries,
okay
move
on
so
right
now
the
propagates,
like
you
know,
on
the
open
source
technology.
Even
a
kubernetes
version
upgrade
is
very
fast
every
three
months
that
the
new
version
was
released
in
a
communities.
So
we
want
to
have
a
new
feature.
We
want
to
fix
out
the
Box
in
previous
version
that
can't
be
challenges
that
we
should
be
upgrade
those
kind
of
the
whole
kubernetes
cluster
in
very
short
time.
B
So,
every
three
months
we
should
be
upgrade
those
come
cluster,
but
how
many
server
we
have.
Does
anyone
still
remember
the
number?
Yes,
someone
with
the
hand
yeah,
it
cannot
be.
We
have
hundreds
hours
and
the
server
just
Palmetto
I
also
have
200,000
for
VM.
So
you
can
imagine
that
if
we
want
to
upgrade
that
kind
of
the
skills
side
of
the
wall,
the
server
instruments,
actually
we
just
take
one
months
job
great
because
have
another
two
months,
shotguns
teamwork.
We
figure
out
how
to
get
the
new
version
to
integrate.
B
So
in
that
case,
how
can
we
do
that?
Ten
hundred
a
thousand
server
up
Quay
and
everything
is
running
out.
There
part
you
cannot
let
let
the
traffic
goes
down.
You
cannot
just
say:
okay,
I
shall
down
the
whole
side.
No
way
sounds
like
that.
You
you're
writing,
like
the
j-20,
the
jet
flat
on
the
under
the
10,000
meters
high,
and
it's
words,
change
the
engine
right
in
the
fly
so
kind
of
be.
How
can
we
change
the
energy
in
the
fly?
B
B
We
can't
do
the
rolling
upgrade
once
we
finish
the
testing
in
the
dev
environment,
so
we
can
gradually
just
take
one
rack
out
to
upgrade
them
and
then,
when
they
finish,
people
going
move
on
so
because
everything
is
gonna
be
defined
by
the
like
the
you
know:
module
right,
so
everything
is
in
the
spec
is
defined.
So
this
is
folder
like
the
kubernetes
core
component.
Also,
we
have
errands
so
tell
which
version
we
have,
which
the
report
we
want
to
use
and
how
to
upgrade
them,
which
application
won't
be
installed.
B
So
everything
was
being
defined
there.
So
I,
just
clear,
still
can
collect
the
coop
can
show
command,
create
those
upgrade.
So
hopefully
I
mean
the
hundred
thousand.
A
server
will
be
rolling
up
with
one
rack,
pan
one
rack
automatically
going
there,
so
we
no
need
to
human.
You
enjoyed
this
kind
of
the
process,
just
finish
that
so
that's
kind
of
us
how
easy
that
we
upgrade
those
okay,
let's
recap
so,
first
the
scene,
the
magic
of
the
kubernetes
to
the
eBay
is
canopy.
It's
simple.
It's
very
simple!
B
Everything
can
be
easy
to
define
and
we
define
the
module
and
the
controller
is
the
is
the
key
or
concept.
So
you
have
a
multi
define.
You
have
like
the
controller
to
control
the
module
to
going
to
reconsider
yeah
you
can
easily
T.
Also,
you
can
easily
extend
your
kubernetes
api
to
feeling
you
are
the
requirements.
Then
everything
goes
easy.
How
no
matter
how
large
scale
potatoes
ending
easy
to
manage.
So
right
now,
I
have
a
few
points.
What
you
highlights
with
you?
How
can
we
make
them?
B
Because
we
do
the
everything
former
beginning
from
data
center?
We
use
mojo
to
define
all
the
physical
layer,
your
region,
you
are
AC
your
server.
Your
networking
is
be
defined
in
kubernetes.
Then
later
the
config
configuration
management
layer
we
define
such
as
I
mentioned
before,
so
the
master
also
Dominion
and
the
sub
deployment
used
to
be
predefined
in
the
like
the
communities,
so
you
can
use
kubernetes
itself
module
to
describe
there
and
also
connects
infrastructure
itself
can
be
defined
by
itself,
so
this
kind
of
describe
yourself
and
that
you
can
start
off
by
yourself.
B
That's
kind
of
the
secret
then
pace
on
this.
We
are
ready
peel
the
whole
infrastructure
of
the
kubernetes
in
eBay.
Then
later,
no
matter
what
kind
of
application
you
can't
build
a
Kafka
application.
You
can
want
to
build
the
Alexi
search
application.
You
want
to
build
any
whatever
the
application
based
on
this
kind
of
the
type
mojo
and
the
controller.
Once
you
want
to
release,
you
can
release.
So
everything
is
easy,
so
that's
kind
of
be
magic
to
the
eBay
and
also
that's
gonna,
be
magic
to
the
kubernetes.
B
The
kubernetes
itself
manage
it
about
kubernetes,
that's
the
magic
okay!
Here
we
are
that
the
here
is
the
oh.
That's
all
for
the
author
is
sharing
the
last.
The
most
important
sees
we
are
hurry
if
you
have
interesting
to
learn
this
and
a
go,
the
adventure
with
us,
but
it
can't
have
this
this
our
email.
Thank
you.
It's
okay!
Everyone.
B
Maybe
we
have
liked
a
few
minutes
for
the
questions
L.
We
can
like,
if
you
guys
like
to
have
some
like
the
private
information
once
you
share
with
us.
Okay,
we
see
it
here,
welcome
to
you
here.
Finally,
how
much
if
you
know
CJ
can
arugula
Yahoo
shinshu?
Okay,
oh
my
case
the
shallow
waters,
oh
you're,
so
hot
windy,
you
got
you
so.
C
D
A
Ratibor
conservative,
so
job
bootstrap
cluster,
your
Titian
tendency,
don't
eat.
So
a
dig,
a
cluster,
almost
Sonya.
You
change
your
channel.
It's
your
Koopa
protein
insulin,
a
butcher
sized
onions.
Here
you
coop
up
each
open
agenda.
Dopey
engine
coop,
ADM,
tent
returns.
A
hundred
young
teen
hundred
Shambo
Tsukuba
trigger
show
job
Chancellor.
So
your
children
function,
oh
honey,
GRU,
tiga,
cluster,
so
bootstrap
cluster
rob
a
soil
model
to
teach
found.
Is
you
into
the
cluster
to
prevent?
A
A
C
The
whole
we
are
team
debate
rather
thin
sound
you
can
you
actually
saw
device
to
create
toy,
it's
equal
sweetums
in
there
you
go
today,
good
tourist,
you
don't
you
announcer,
don't
tell
which
I'll
go
to
us
home
yeah
knew
it
could
be
city
outer
you
come.
You
show
decision,
do
Carla,
no
good
job.
Super
tender,
don't
odometer
attainment.
Color
jelly
was
really
achieve
our
vertical
function
ground.
We
also
manage
it
counter.
A
So
then,
till
you
go
one
here
what
how
to
allow
me
to
patrol
so
when
we
always
so
sure
nice,
young,
so
to
you
and
should
the
tiered
one
he
Musa
Dagh,
there's
manual
to
manage
just
a
server
today,
Horace
one
in
the
fleet,
Kumud
on
a
whole
new
list.
No
Jerzy
LEC
attention
holy
sharon
suzhou,
conceivable
nearly
ship
platform.
Today,
tom
and
deunan
soda
I
know
what
tango
model
on
the
controller
fathers
of
Dhamma.
So
he
didn't
soda.
So
new
challenges
and
Ally
challenges
to
pay
holding
the
controller.
A
So
traditional
illogical
way,
limine
nichijou
operation
question
so
to
don't
wash
it
some
young
Trudeau
speaker
using
the
Eagle
Road
King
America
always
reconcile
the
joining.
Of
course.
The
day
you
can,
he
runs
out
you
choke
on
your
fascination,
so
Romeo
touchdown
holder.
So
naturally
we
see
a
flapping.
Today,
kubernetes
didn't
function
to
the
controller.
You
loop,
your
movements
at
it'd
be
either
Wisby
children,
TC
wanna,
don't
see
questions
sure
you're
dodging
the
will.
The
can
you
are
you
guys
all
just.
B
Ecology,
the
Sequoias
detail,
would
you
do
it
so
much
easier?
Come
come
hoarder.
She
says
yeah
suggesting
that's
what
kubernetes
is
super
simple.
Don't
hang
about
machines
to
support
our
content
goes
see
them.
You
can
sorta,
see
cuz.
I
send
up
when
he's
happy
she's
all
you
know
what
I
know
you
go.
We
can't
Anna.
You
can
notice
that
was
at
every
moment.
You
need
you
kubernetes,
Paloma
and
so
host
will
contact
you.
He
found
one.
That's
all
need
a
view,
no
matter
Jessica,
neither
protection
easy,
that's
a
face.
B
How
sounds
to
them
so
he
he
thought
you
really
start
to
chant
since
I
really
thought
for
the
changes
inside
the
tab.
Even
the
staging
money
me
me,
children,
hey,
please
how
wrong
me,
the
reproduces
or,
but
if
we
find
his
managing
Wow
fast
in
some
yeah,
no
one's
I
upgrade
warm
the
classical
woody
right.
It
has
can
in
fact
need
technical
that
the
stadium
at
the
conduct
most
of
us.
We
need
to
go
any
Wi-Fi
here,
Peggy
from
South
with
you
conflicted
to
do
no
connection.
B
B
You
took
a
pimp,
you
too
tender-hearted
a
so
tiny
Tyson
chopped
onions
law.
If
when
are
we
each
jamia
condense
and
then
come
to
layaway
Oh
easy
to
dance,
Tahoe
mean
your
vaio
nupedia
I
need
everyone
gets
all
a
be
a
fireman
Tila.
First
Arsenal
Maji
Yas
kuwait
zoo,
which
is
your
70.
The
you
need
to
have
one
Aswad.
You
can
reach
awesome
a
CI
CD
Nikki
photo
energy
centers
ah-ha-ha-ha.