►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building and Managing Kubernetes with Kubernetes - Xin Ma, eBay
Kubernetes as a declarative and portable system can be used to do many things in different ways. At eBay we built a fleet management system based on k8s. Everything(server, subnet, OS, package and state) is declarative and can be modeled as CRDs in k8s, or referred to as a commit id in git from the objects. By running various controllers on top of these CRD objects, we use k8s to manage k8s, and the entire eBay data center. - Our system provisions hosts the same way k8s creates and manages pods. - We build k8s clusters with Salt. each host has a set of states defined in its salt CRD object. controllers pull states from git based on commit ids to apply. - We build both schedulers and deployment transactions to manage the k8s clusters for both config deployments and upgrades. This declarative, highly scalable, auto healing, and cloud native system is what we think can unify eBay’s fleet.
https://sched.co/Nrm0
A
Morning,
everyone
welcome
to
my
session
I'm
shimmer
from
eBay,
and
it's
such
an
honor
to
be
back
in
Shanghai.
You
talk
about
these
cool
pockets
and
thank
you
for
Kenny.
Today,
I'm
gonna
talk
about
how
we
view
and
manage
cube
annuities
with
kubernetes,
which
is
basically
our
eBay
speak
management
system
and
the
way
we
run
our
private
cloud
with
kubernetes
and
get
ops.
A
So
this
is
the
agenda.
I'm
gonna
give
a
very
quick
introduction.
First
on
Cuba
near
these
deployments
and
after
that
I'm
gonna
be
introduce
our
fleet
management
system
based
on
kubernetes
that
views
and
runs
our
private
cloud
in
a
deeper
dive
of
that
system.
I'm
gonna
walk
through
the
way
we
viewed
and
managed
is
with
kubernetes.
A
Basically,
we
build
compute
providers
based
on
kubernetes
and
how
we
build
eBay's
Cuban
Aires
clusters,
with
our
salt
operators
and
controllers
and
in
the
end
I'm
gonna
share
our
get-ups
practices
to
run
our
feet
and
lots
of
our
cool
features
in
fly
like
humanity's
first
things.
First
eBay
is
one
of
the
earlier
adopter
of
Cuban
Ares.
We
started
things
2015,
we
run
an
internal
distribution
called
tested
I/o.
Within
the
past
few
years,
we've
moved
massive
production
workflows
into
containers
running
on
kubernetes.
As
of
now,
we
have
50
plus
production
clusters.
A
We've
got
multiple
V
pcs
for
different
environments.
We
have
both
flat
netbook
and
overlay
network
based
on
Obion,
and
we
have
multiple
2000
plus
sized
clusters
with
heavy
production
workflows.
For
now
we
have
roughly
160,000
pots
running
on
at
the
30,000
hosts.
Most
of
them
are
bare
metals,
because
we
only
run
by
metals
for
production.
A
We
have
so
many
different
kinds
of
workloads
running
on
kubernetes.
Now
we
have
our
Web
Services
for
sure
we
have
database
search
engines,
we
have
Hadoop
and
AI
and
machine
learning.
Well,
in
my
past,
European
cube
con
I
had
a
talk
about
our
practices
to
run
high-performance
workflows
and
our
performance
churning.
If
you
are
interested,
you
can
always
find
the
splice
online
and
the
presentation
as
well.
A
We
are
also
running
on
the
edge
we
have
around
Lake
20,
actually
15
edge
clusters
across
the
world,
and
we
are
running
our
envoy
edge
proxies
as
well
as
software
load
balancers.
On
top
of
cuban
aires
there,
where
we
do
a
lot
of
SSL
termination
web
caching
to
accelerate
our
remote
eBay
users
user
experience,
we
are
rapidly
growing
our
footprint
on
the
edge
as
well.
So
our
ultimate
goal
is
to
unify
our
fleet
with
kubernetes
and
we're
trying
our
best
to
get
there.
A
So
now,
after
getting
some
idea
on
eBay's
kubernetes
footprint,
you
might
wonder:
how
did
we
get
there
and
the
short
answer?
Is
we
use
few
binaries
to
make
it
all
happen,
and
that
also
that
is
our
fleet
management
system
compute
and
manage
events,
private
cloud
that
runs
kubernetes
with
kubernetes,
as
most
of
you
know
that
cuban
Ares
is
not
just
a
container
platform,
is
a
portable
system
that
can
be
used
to
do
many
different
things
in
many
different
ways.
A
We
use
it
to
manage
and
build
our
private
cloud,
that
is,
our
fleet
management
system
same
as
many
other
kubernetes
services.
Our
system
runs
with
API
server
and
its
city
cluster
in
eBay.
We
automated
this
process
so
that
you
can
spin
off
a
few
pots
and
create
up
an
API
server
in
each
city.
With
its
video
operator,
you
can
run
them
in
a
single
cluster
or
run
like
molecule
clusters,
as
the
HBase
set
up
once
you
have
the
API
set
up.
A
We
create
a
lot
of
series,
which
is
a
fun
perk,
which
is
the
modeling
and
we
model
almost
everything
and
every
anything
so
from
hardware
which
we
model
our
entire
data
center
from
racks
to
switches
to
V,
PC,
subnets,
brows
and
eventually
computer
assets
where
we
can
provision
hosts
from-
and
we
are
also
touching
based
on.
The
software
stack.
We
start
from
operating
system
and
we
model
compute
nodes
and
we
build
urban
areas
result.
A
A
We
write
like
provisioning
controllers
that
provision
compute
nodes
just
like
reading
kubernetes
is
creating
pods
and
the
notebook
controller
will
create
a
set
of
compute
nodes
with
this
exact
same
setup
of
config,
and
we
support
multiple
providers
and
we
have
a
building
hunger
on
provisioning
system
that
I'm
gonna
talk
about
later
and
then
we
have
a
very
powerful
salt
operator
that
can
build
cubanía
DS
with
salt.
We
can
install
salt
master
on
compute
node
from
a
git
commit
ID,
and
then
you
can
build
a
cluster
out
of
it.
A
We
have
solve
deployments
and
a
few
other
controllers
to
manage
a
set
of
salt
minions
so
that
you
can
run
multiple
Kuban,
Ares
masters
and
kubernetes
nodes.
We
have
many
other
controllers
as
well.
For
example,
like
rack
controllers,
which
takes
care
of
the
bootstrapping.
We
have
the
scheduler,
we
have
IPAN
to
do
IP
allocation
and,
of
course,
DNS
controllers
and
remediation
controllers,
that
takes
care
of
the
compute
nodes,
lifecycle
management
and
with
all
those
controllers
and
models.
We
have
many
functions
that
we
are
inspired
by
human
Ares.
A
A
Ok,
let's
get
to
the
architecture
view
of
a
single
availability
zones,
fleet
management
system
deployment,
together
with
their
models.
So
here
is
the
API
server
and
its
city.
Then
you
have
Pez
master,
which
is
a
set
of
many
many
controllers,
that's
wrong
running
on
top
of
this
API
server
and
then
what's
inside
that
API
server
from
the
left
side
you
can
see
there
are
a
lot
of
IAS
bits,
for
example,
hardware's
racks
and
then
from
the
right
side.
You
can
see
there
are
a
lot
of
humanities
bits,
for
example,
salt
and
kubernetes
notes.
A
The
fun
part
is
that
the
whole
fleet
management
system
is
also
running
inside
of
Canaries
cluster.
That
abuse,
so
you
can
build
a
new
availability
zone
from
external
cluster
that
managed
this
new
availability
zone,
give
you
the
new
clusters
and
then,
after
that
you
can
migrate
the
whole
thing
inside
that
availability
zone
so
that
it's
self-contained.
A
First,
let's
zoom
in
into
the
is
layer
to
see
how
we
viewed
and
managed
it
so
kubernetes
knees
compute
from
providers.
We
have
different
providers,
for
example
OpenStack
as
a
private
cloud
provider
and
GCP
as
a
public
cloud
provider,
and
then
our
fleet
management
system
needs
to
define
interfaces
from
our
controllers
to
work
with
those
providers.
For
example,
a
typical
example
is
a
provisioning
controller,
would
define
the
interfaces
to
create,
delete
and
reimage
computes
and
then
different
providers.
Clients
will
implement
those
interfaces.
A
So
we
do
like
kickstart
and
proceeding
and
it's
a
it's
a
Cuban
editing
system
and
it's
it's
a
provider
for
our
fleet
management
system
so
that
it
can
build
BAM,
arrows,
whizzing
itself
itself.
Content
is
highly
scale
and
once
we
do
that,
we
start
to
create
CRTs
for
all
the
data
center
bits
and
then
that
inspired
us
to
move
further
to
manage
and
orchestrate
the
feet,
which
means
that
we
can
model
the
entire
data
center
and
it
empowered
us
to
do
many.
Other
things,
for
example,
manage
the
compute
nodes
lifecycle,
for
example.
A
You
can
do
remediation,
you
can
do
cleanups
and-
and
you
can,
you
know-
serve
as
a
CMDB
as
well,
because
many
of
your
objects
are
already
in
humanities
and
and
eventually
after
we
have
the
metal
clusters
viewed
out.
We
have
we
still
miss
VMs,
because
not
everything
is
in
container
for
us.
We
still
have
windows
were
closed
and
how
do
you
deal
with
that?
And
and
that's
how
we
start
to
think-
why
not
just
the
beauty
em
provider
with
kubernetes
as
well,
so
that
we
can
take
vm's
from
those
communities.
A
A
So
this
will
send
the
whole
thing
as
a
payload
to
format.
Foreman
is
a
service
lifecycle
management
system
that
connects
a
provisioning
system
with
EFT,
fika,
TFTP
and
DHCP,
and
it
provides
a
pixie
and
HTTP
infrastructure
for
your
installation.
So
once
the
extent
oh
man,
oh
man,
is
gonna,
send
it
to
our
registration
system
that
populates
the
CMDB
and
then
our
fleet
management
system
can
kick
in
and
do
the
model
of
the
rack.
So
our
data
center
modeling
is
based
on
a
rack
unit
from
a
rack.
A
We
can
see
all
the
subnets
from
the
rack
and
then
the
routes
are
the
l2
domains,
the
IP
addresses
and
eventually
the
computer
assets.
So
this
is
the
it's
a
little
bit
smaller.
This
is
the
computer
asset
object.
You
can
see,
there's
BNC,
IPs
and
eth0
MAC
addresses
and
manufacturers
serial
numbers
all
kinds
of
stuff
so
that
you
can
use
later
to
provision
bare
metal
hosts.
A
So
once
the
whole
thing
is
modeled,
we
have
the
assets
and
subnets
right.
They
are
CRTs
and
objects
in
API
server.
We
can
provision
hosts
in
a
previous
wise.
We
modeled
for
man-woman
is
also
serving
here
as
TFTP
and
HTTP
service,
so
that
you
know,
unlike
any
traditional
provisioning
system,
where
you
send
like
a
massive
payload
into
an
API.
You
just
create
a
computer
object
in
API
server
and
it's
pretty
much
on
the
you
create
the
compute
node
with
OS
image
we
use
CentOS
atomic
and
then
you
specify
a
flavor.
A
It
also
automatically
generates
partitioning
bits
there
so
that
it
can
tell
how
do
I
partition.
The
disks,
for
example,
have
three
disk
I
need
to
setup.
Lvm
I
need
to
setup
empty
area,
so
the
whole
thing
you
can
model
from
CRT
and
our
controller
will
automatically
generate
and
populate
that.
So
basically,
controllers
work
works
in
order,
so
they
subscribe
to
different
fields
of
the
camp.
A
You
know,
and
it
also
puts
finalized
errs
into
the
computer
object
so
that
when
you
delete
the
compute
node,
for
example,
DNS
controller
will
see
ok,
I
put
on
a
specific
finaliter
for
DNS,
so
I'm
gonna
remove
DNS
as
well.
Ip
location
will
release
the
IP,
etc,
etc,
so
the
whole
provisioning
system.
It
works
like
this,
it's
completely
declarative
and
then
you
know
each
stage
you
can
track
from
the
status
and
you
know
of
the
computer
object.
A
So
now
it
pretty
much
covers
the
provisioning
bits
and
you
can
provision
bare-metal
hosts.
We
start
to
think
about
how
do
we
run
virtual
machines
right?
So,
let's
assume
you
have
the
Amero
provision
and
you
have
a
bare-metal
only
kubernetes
cluster.
Well,
we
jumped
a
little
bit,
but
we're
wondering
if
we
can
get
v
ends
for
our,
for
example,
windows
were
closed
right
to
let
windows
only
run
on
virtual
machines
running
on
top
of
cubanelles.
A
So
basically
VM
on
kubernetes
has
two
major
technology
versus
cada
container
is
super
popular
I'm,
pretty
sure
most
of
you
already
heard
of
that.
So
it's
pretty
much
running
native
and
secure
workflows
in
Cuban
Ares
and
then
there
is
a
pure
virtual
machine
solutions
like
Marin
Texas
Berkeley,
and
do
it
hats,
so
we
pretty
much
do
both.
So
we
used
cada
as
well
to
run
our
Kara.
A
Is
there
kara
is
around
time
for
us
for
our
secure
workflows
so
that
you
have
a
very
quick
virtual
machine
spin
off
for
a
specific,
secure
part
you
want
to
run
and
then
the
Kara
agent
inside
the
VM
will
create
the
container
and
parts
for
you.
We
use
Kara
for
secure
floats,
but
Kara
is
not
easy.
At
least
to
me.
It's
not
so
easy
to
be
used
as
a
VM
provider
to
get
VM
computes.
That's
why
we
explored
on
some
other
solutions
and
we
eventually
pick
it
up
worth
left.
So,
first,
let
is
a
compute.
A
A
A
By
embedding
vert
led
as
our
VM
provider,
we
can
build
a
kubernetes
cluster
on
top
of
a
kubernetes
cluster,
which
is
the
fun
part
right,
so
you
have
the
VM
as
your
compute
provider,
and
then
you
compute,
a
vm
q
binaries
cluster
on
top
of
the
Palmetto
humanities
cluster,
and
that's
super
useful
for
us,
because
we
want
to
use
a
lot
of
CI
CDs.
We
do
a
lot
of
end-to-end
testing
because
we
run
our
own
distribution
of
communities
and-
and
this
is
pretty
much
a
compute
node.
Actually
this
is
an
old
post
back
forward
pet.
A
You
can
only
see
the
difference
between
this
one
and
a
specific.
The
Amero
camp.
You
note
is
that
it's
way
shorter,
it
doesn't
have
all
the
flavor
bits.
Oh,
it
has
is
the
VM
flavor
from
our
kubernetes
clusters,
and
then
we
also
have
the
provider
equals
to
worth,
let
together
with
other
bits
like
SSH
keys
and
stuff,
and
it's
gonna
create
a
part
in
communities
cluster
like
this,
and
this
one
will
create
a
virtual
machine
for
us.
A
We
worked
on
a
few
areas
like
config,
Drive
and
cloud
in
it,
but
eventually
we
can
get
virtual
machines
from
cubanelles.
So
now
we
have
all
those
computes.
We
are
going
to
build
kubernetes
clusters,
so
we
build
kubernetes
with
salt,
since
we
are
a
Cuban
Aires
system.
We
start
with
model
and
see
are
these
right,
so
we
start
to
model
the
salt
stack.
So
we
have
salt,
mustard
and
salt
minions
that
oppose
objects
managed
by
salt
deployments,
which
create
an
August
straight
on
both
objects.
A
So
a
salt
master
will
take
a
commit
ID
from
assault,
be
a
triple
and
set
up
a
salt
master.
That's
later,
gonna
be
used
to
deploy
kubernetes,
and
then
we
have
different
other
salt
deployments,
for
example,
like
masters
salt
deployment,
to
run
salt
high
states
with
masters
roll,
and
we
have
the
cube
notes,
assault
departments
to
run
the
cube
nodes
so
that
you
can
run
cubelet
and
cube
cube
proxy,
and
then
we
have
other
features
just
dimension
like
so
transactions
that
takes
care
of
rolling
updates,
because
we
are
running
like
a
huge
cluster
3,000
notes.
A
You
cannot
run
the
updates
at
the
one
shot
everywhere.
You
have
to
stage
it
right,
so
we
support
rolling
updates.
We
created
this
feature
inspired
by
Cuban
Aries,
which
is
go
transactions.
It's
gonna
break
the
note
pools
computers
into
different
buckets
based
on
the
strategy
that
you
specify.
We
can
do
note
by
note
rack
by
rack
and
many
others.
You
can
even
plug
in
your
own
strategy
so
that
we
can
put
the
compute
nodes
into
buckets
and
then
run
them
stage
by
stage.
That's
our
transaction.
After
that
we
have
controllers.
A
We
have
a
lot
of
salt
controllers,
and
this
is
the
breakdown
of
that.
So
basically,
a
salt
deployment
is
a
one-to-one
mapping
to
a
node
pool.
If
you
remember,
the
node
pool
is
a
set
of
compute
nodes
that
we
provisioned,
that
has
the
same
spec
of
operating
system
and
and
host
config,
and
then
salt
master
deployments
will
take
a
specific
community
from
git
and
then
deploy
a
specific
salt
master.
That's
going
to
be
deploying
our
cuban
Ares.
This
is
the
first
one
here,
salt
master,
and
then
we
have
the
raw
equals
to
master
so
deployment.
A
Here.
That's
gonna
deploy
the
cumin
Aires
masters,
basically
they're
setting
the
raw
and
grains
and
pillars
on
the
specific
notes
and
around
salt
high
state
so
that
they
can
install
API
server
and
it's
City
one
by
one.
We
fix
a
tendency
for
the
HDD
cluster
and
from
the
cell
state,
and
it
also
has
the
cell
transactions
here
so
once
it
has
a
cell
transaction
defined.
A
The
salt
deployments
will
create
the
salt
minions
or
objects
one
by
one,
because
mostly
for
master
knows
we're
gonna
do
one
by
one
for
post,
setting
up
and
and
and
upgrades
and
then
you're
gonna
see
different
purposes
of
kubernetes
notes.
We
have
delegations,
we
have
different
workloads
so
that
you
can
set
up
the
remaining
kubernetes
notes.
So
on
the
right
side,
you're
gonna
see
there
are
a
lot
of
controllers.
A
We
have
different
operators
that
does
different
things,
for
example,
so
deployment
controller
will
create
the
soul
transactions
based
on
the
running
app
strategy,
and
we
have
auto
resume
controller.
That's
basically
doing
buckets
by
buckets
automatically
based
on
the
very
important
health
health
monitor
framework,
which
we
also
took
from
Cuban
Aries.
So
each
bucket
will
check
the
props
and
conditions
to
make
sure
that
is
ready
and
then
move
on
to
the
next
one.
A
So
this
is
an
example
of
the
salty
form,
an
object
that
we
have.
Oh
it's
a
little
bit
too
small.
Sorry,
you
have
pretty
much
the
key
triple
here.
There's
a
git
commit
ID,
and
then
you
have
the
salt
master,
set
up
bits
and
assault
minions
set
up
bits
so
that
you
can
know
specifically
from
get
commit
ID
on
which
human
Ares
version
we're
running
and
your
code
base
is
on
what
so.
This
is
again
very
declarative
and
how
we
run
our
cumin
areas
clusters.
A
So,
ok,
now
we
have
pretty
much
gone
through
the
functional
bits
of
our
fleet
management
system.
If
you
follow
through,
there
are
a
lot
of
objects
and
a
lot
of
series
and-
and
you
can
almost
tell
that
every
bit
can
be
putting
to
get
and
then
we
can
run
get
ops
like
out
of
the
box.
So
let's
take
a
wider
view
from
so
we
run
our
operating
system
and
we
build
our
own
operating
system
using
OS
3.
A
So
operating
system
can
be
viewed
from
a
Jason's
back
and
then
it's
gonna
build
a
tree
and
you
have
a
commit
ID
that
represents
the
whole
operating
systems
footprint.
And
then
we
have
a
provisioning
system
which
create
a
lot
of
series
and
we
have
a
lot
of
controllers
to
create
in
the
supervision
the
bare
metal
host.
And
then
you
have
salt
to
install
and
configure
binaries.
And
then,
if
you
checking
all
those
theories
in
Huget,
you
can
run
get
ops
and
of
course,
I
missed.
One
part
here,
which
is
a
computer
and
clustered
lifecycle
management.
A
We
managed
the
remediation
policies
from
compute
node
as
well,
so
pretty
much.
You
can
have
everything
check
in
to
get
so
that
you
have
the
whole
fleet
in
get
part
of.
That
is
a
series.
The
other
part
is
your
state,
your
code,
and
you
can
connect
things
together
with
2
or
3
p3
posts,
and
you
can
make
everything
to
be
in
get
so
that
our
way
of
doing
things
is
get
ops
for
DevOps.
A
So
you
have
a
PR
driven
operations
and
with
everything,
declarative
and
that's
the
way
we
build
and
scale
our
kubernetes
footprints
in
eBay,
so
that
you
can
build
consistent
and
highly
scalable
clusters
across
so
or
for.
Even
so,
that's
pretty
much
that
bit
off
spark
and
the
last
I'm
gonna
talk
about
our
features
inspired
by
human
Ares.
We
have
a
lot
of
features
inspired
by
human
Ares.
We
don't
have
much
time
left,
but
I'm
gonna
quickly
go
through
them.
So
scheduler
is
similar,
as
our
schedule
is
similar.
A
As
cube
schedule
is
gonna
pick
some
free
assets
based
on
your
selector
and
enter
plenty
of
the
entities
vectors.
It's
gonna
pick
the
pick
the
asset
and
we
talked
many
times
about
the
voting
updates
with
view
transactions
and
buckets
and
strategies
which
is
pluggable,
and
then
we
have
destruction
budgets
so
that
you
can
limit
from
a
specific
namespace
of
a
cluster
to
see
how
many
nodes
you
can
delete
reimage
within
a
day
within
an
hour.
A
That's
something
we
took
from
the
pod
disruption
budget
and
we
use
it
from
our
host
and
flower
upgrades
and
host
our
release
and
as
well
as
OS
upgrades,
and
then
we
have
the
health
monitor
framework,
which
is
also
inspired
by
cubanía
DS.
We
put
a
lot
of
props
for
our
compute
nodes
that
there
are
a
lot
of
conditions
and
then
the
computers
liveness
problem
ratings
prop
is
a
major
driven
point
for
our
imitation
controller,
with
which
we
can
manage
our
humanities
clusters,
life
cycle
and
then
pretty
much.
A
We
have
many
other
things
that
we
didn't
list
here,
but
thanks
to
cuban
aires,
that's
how
it's
possible
for
us
to
manage
our
feet
with
a
declarative
system
like
this.
You
make
it
our
our
life
easier.
So
last
is
the
summary.
So
pretty
much
everything
is
covered.
I
just
want
to
say
that
the
Whizzer
scale
and
the
kind
of
footprint
we
are
running
with
kubernetes.
We
need
a
lot
of
help,
so
we
have
a
great
china
team
here
in
Shanghai.
So
if
you
are
interested,
please
directly
engage
with
our
folks
here.
A
So
if
so,
a
few
years
ago
in
I,
think
the
Boston
opens
that
summit
with
it
mentioned
about
the
open-source
version,
but
I
think
later,
with
building
too
many
eBay
logics
onto
it.
So
there's
no
plan
to
open
source,
it
yeah,
the
major
driven
or
this
talk-
is
to
go
through
the
ideas
of
how
do
you
use
a
cuban
ad
system
to
manage
a
large
clusters
like
ebay
is
running
ones?
Yeah
I
mean
even
for
the
small
pieces
of
the
code,
some
small
components
like
the
temptation
controller
or
something
not
not
I'm,
aware
of
Assam.
A
Now
we
have
some
other
stuff
that
we
are
planning
on
open
source
right,
the
visualization
bits.
We
have
a
plug-in
that
we
can
visualize
the
whole
series
and
the
data
signal
that
we
are
trying
to
work
with
the
CN
CF
to
open
source
that
but
I
don't
have
any
specific
things.
Yet,
if
you're
interested
please
directly
connect
with
me,
although
I
can
answer
your
questions.