►
Description
Intro to Control Plane Engineering - Bob Wise, Samsung SDS
Summary: Building Kubernetes clusters with large numbers of nodes, high pod start rates, or with needs for high availability? Large, high-performance and reliable kubernetes clusters require engineering the control plane components for demands beyond the defaults. In this talk we will cover the relationship between the various components that make up the kubernetes control plane and how to design and size those components. Where are likely system bottlenecks, and what are approaches for managing those bottlenecks?
About Bob Wise
Bob is the Chief Cloud Technologist for Samsung SDS Research America, and opened the Seattle office for Samsung SDS as home base for the SDS Cloud Native Computing Team. Previously he was the CIO at MTN Satellite Communications, where he was responsible for product strategy and development, including global cloud computing and data center deployments. Prior to MTN, Bob was a VP Engineering in HP's public cloud effort, and led transition to Openstack. He made his way to HP via the acquisition of Seattle startup Melodeo, an early mover in cloud computing.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
we
have
a
team
here
in
Seattle
a
few
other
places
as
well
we're
centered
here
in
Seattle,
like
all
we
do,
especially
in
open
source
projects.
These
are
team
efforts,
so
there
were
a
few
folks
on
the
team
here
that
put
some
extra
effort
into
this.
I
wanted
to
make
sure
I
put
their
names
up
here
and
with
that
out,
I'll
keep
going
ok.
So
this
is
a
real,
honest-to-goodness
disclaimer,
yes,
I
work
for
a
very
large
company,
but
they
can't
keep
me
from
showing
you
grumpy
cat.
A
If
you
really
want
to
read
all
that,
come
talk
to
me
later,
I'm
sure
I
can
arrange
it.
Ok,
so
before
I
get
into
control,
plane,
engineering
I
think
it's
really
important
to
talk
about.
Why
we
care
why
you
might
care-
and
there
are
a
bunch
of
things
we
believe
some
of
these
things
that
we
believe
are
not
necessarily
relevant
to
this
discussion,
but
some
of
them
are,
of
course,
open
source
at
the
heart
of
invisible
infrastructure
were
deeply
involved.
A
The
notion
of
high
velocity
organizations
seeking
failures.
He
can
change
the
automation,
tooling,
that's
required.
This
is
all
I
would
say
the
motherhood
and
apple
pie
of
invisible,
open
source
infrastructure.
The
notion
of
container
orchestration,
as
you
know,
one
of
the
big
moves
that's
happening
now
is
is
important.
Our
business,
both
internal
to
Samsung
and
external,
is
around
this
last
bullet,
which
is
this
is
really
hard
stuff,
but
technically
and
organizationally
and
organizations
often
need
help
making
some
of
these
shifts
and
that's
what
we
do.
We
help
with
the
shift.
A
We
have
a
bunch
of
our
team
here,
I
just
wanted,
throw
the
pictures
up
in
case
you
see
some
of
the
folks
wandering
around
here
the
conference.
Please
stop
them
and
bother
them
with
whatever
questions
so
anyway,
all
right.
So
so
this
talk
there's
a
lot
of
great
talks
here,
a
pretty
wide
variety
of
talks
here,
some
things
at
a
more
advanced
level,
some
things
that
more
of
a
I'll
say
beginner
to
intermediate
level.
A
This
talk
is
designed
specifically
to
be
more
of
a
like
your
next
step
in
kubernetes,
but
it's
not
quite
a
beginner
talk
as
I'm.
I
make
some
assumptions
here.
I'm
actually
I'll
cover
a
couple
of
these,
but
you
know
I'm
gonna
assume
you
know
a
little
bit
about
kubernetes
kubernetes
fundamentals,
what
it
means
to
run
a
application
on
that.
Probably
if
you
know
about
these
things,
you've
played
with
it
somewhere,
probably
on
gke.
That's
a
really
easy
way
to
get
a
kubernetes
have
a
kubernetes
cluster
you
can
play
with
chances.
A
Are
you've
run
something
on
your
on
your
desktop.
There's
easy
ways
to
get
down
on
your
laptop,
but
fundamentally
you're,
here
or
you'll,
find
this
interesting
or
you
can
go
get
a
cup
of
coffee.
If
you
don't
care
because
you
want
to
run
kubernetes
in
different
places
under
your
own
control
and
it
possibly
different
scales
may
be
very
large
scales,
maybe
smaller
scales,
okay,
so
just
a
quick
baseline
here,
so
I'm
gonna
do
hand
waving.
There's
this
control
plane
thing,
but
this
is
really
the
basics.
I
have
a
three
node
empty
cluster
here.
A
I
ask
it
to
do
some
work.
Please,
you
know
manage
some
replicas
as
creamy
some
pods
and
the
system
will
do
that.
They'll
create
a
pod
with
multiple
containers
in
it
of
type
X
and
maybe
I
need
a
couple
of
type
Y.
This
is
all
the
stuff
I
said:
you're
gonna,
assume
I
would
assume
you
knew,
but
I
covered
here
quickly
anyway.
So
this
happens,
things
change
in
your
system,
a
node
dies,
so
automatic
healing
of
the
system
has
to
occur.
A
This
is
all
magic
that
the
control
plane
is
doing
and
when
you're
running
multiple
applications
in
production
environment,
whether
these
are
applications
or
micro
services,
kind
of
doesn't
matter.
You
have
a
bunch
of
different
kinds
of
things
that
I'll
get
mapped
onto
the
cluster
in
under
control,
the
control
control
of
the
control
plane.
So
they're
really
what
is
control
plane
engineering,
so
here
right,
magic
alert.
You
know
something
happens
here
that
seems
very
magical
and
there
is
a
lot
of
magic
here.
A
So
let
here,
let's
dig
into
the
let's
dig
into
that
piece:
a
little
bit
so
I'm
gonna
put
those
nodes
aside.
So
really
what
I
want
to
talk
about
is
what's
going
on
inside
the
control
plane
here,
so
basic
building
blocks?
What
are
the
building
blocks
inside
the
control
plane?
Not
that
many
of
them
some
of
the
interactions
are,
it
can
be
complicated,
but
at
this
level
these
are
the
main
things
you
need
to
know
about
API
server.
A
This
is
fundamental
system
architecture,
design
choice
around
how
everything
goes
through
an
API
server
scheduler,
which
is
doing
the
basics
of
like
what
what
pods
should
run
where
the
system
assistant
called
at
CD
distributed
database
system.
This
is
where
cluster
state
is
maintained
and
then
a
controller
manager
which
is
where
the
basic
control
loops
implementation
of
business
logic
system
logic
is
done.
So
those
are
the
those
are
the
main
pieces.
A
I'm
there
are
so
when
you
say,
control
plane
to
a
kubernetes
expert
will
probably
start
they
might
quibble
with
this,
because
I'm
gonna
kind
of
push
the
stuff.
That's
on
the
node,
like
cube
light
and
COO
proxy
I'm
gonna
push
that
out
I'm
just
going
to
concentrate
on
the
server
bits.
Those
are
probably
technically
part
of
the
control
plane,
but
that's
another
set
of
conversations.
I'm,
just
gonna
worry
about
the
server
bits
in
the
middle
right,
so
the
different
kinds
of
kubernetes
concepts,
there's
more
but
replica
sets
deployments.
A
All
of
these
things
are
implemented
in
the
controller
manager,
all
right
so
so,
if
you
want
to
so
those
are
the
pieces.
Here's
the
basic
relationship
between
those
control
plane
pieces
again,
everything's
going
through
the
API
server.
Api
server
talks
that
CD
nothing
else,
talks
that
CD
and
then
the
scheduler
and
the
controller
manager
talk
to
everything
else
through
the
API
server.
So,
as
you
can
see,
the
API
server
is
a
pretty
critical
piece
in
the
system.
A
A
A
A
You're
gonna
have
failures.
You're
gonna
have
node
failures
in
a
thousand
nodes,
so
you
need
something
to
handle
that.
So,
if
you're,
if
you
have
a
a
system,
that
kind
of
looks
like
that,
then
that
single,
like
everything
running
on
a
single
machine
pattern,
might
well
be
fine
if
you're
doing
some
kind
of
CI
jobs.
That
might
be
another
way.
A
Another
reason
you
would
want
to
do
work
that
way
if
you're
just
doing
some
development,
you
need
something
quick,
but
really
what
I'm
trying
to
talk
about
here
is
a
notion
of
like
how
would
you
work?
How
would
you
think
about
running
a
kubernetes
cluster
on
an
infrastructure,
amazon
bare
metal
or
GCE,
where
the
cluster
is
going
to
stay
up
and
running
for
a
long
period
of
time,
probably
used
by
multiple
applications,
multiple
users?
A
So
what
we
worry
about
with
that
case
is
single
server
performance
is
the
sink,
is
the
single
server
I
have?
Does
it
have
enough
horsepower
to
actually
run
all
of
the
control
plane?
Probably
the
most
important
thing
here
is
high
availability.
If
you
lose
your
entire
control
plan,
your
cluster
state,
everything
is
all
sitting
on
one
machine
chances
are
that
machine
is
gonna,
go
away
at
some
point,
you
got
to
worry
about
that.
That's
gonna
have
a
poor
effect
on
your
cluster
availability.
A
You
do
want
to
worry,
about
instance,
cost
perhaps
depending
on
the
size
of
the
cluster.
So
you
need
to
worry
about
that,
but
generally
we're
looking
at
fewer
larger
clusters.
The
topic
I'm
gonna
push
to
the
advanced
topic,
I'm
not
going
to
try
to
talk
about
today.
Although
it's
a
very
interesting
topic,
is
if
you're
running
in
multiple
availability
zones.
How
would
you
tackle
that
problem?
A
I
would
certainly
recommend
there
I
think
there
two
schools
of
thought
on
this
one
one
school
of
thought
is
encapsulated
in
kubernetes
project
called
uber
Nettie's,
which
is
a
pattern
for
running
multiple
clusters.
A
a
collection
of
compute,
a
per
availability
zone
availability
zone
might
be
and
I'm
using
that
in
the
most
generic
turnt
sense,
not
in
the
Amazon
availability
zone,
right
availability
zone
might
be
a
rack
or
it
could
be
a
data
center
with
its
with
a
single
generator
or
something.
A
There
is
another
school
of
thought
around.
How
do
you?
How
do
you
take?
How
do
you
spread
a
single
cluster
across
multiple
availability
zones
and
manage
that
you
could
actually
distribute?
You
could
have
built
a
single
cluster?
Have
a
control
plane
be
built
out
across
multiple
availability
zones,
as
opposed
to
going
with
a
more
federated
approach?
Again,
that's
a
topic
for
perhaps
the
next
cube
con,
so
not
gonna.
Try
to
tackle
that,
although
you
can
see,
there's
a
lot
of
interesting
possibilities
here.
Okay,
so
complications
mostly.
A
What
I
have
here
are
the
complications
with
regards
to
scalability
and
one
one
thing:
I'll
talk
a
little
bit
more
about
this
later,
because
this
is
not
really
a
solved
problem
in
the
community.
It's
something
that
we're
putting
some
effort
into
would
love
to
have
contribution
to
help
discussion
around
this.
The
complications
when
you're
building
this
kind
of
cluster
are
I
would
say
these
are
like
the
major
axes
of
you
know:
sort
of
performance
issues,
total
number
of
pods
in
the
system.
A
Pods
in
the
system
nodes
in
the
system,
the
the
scalability
issues
tend
to
be
more
related
to
number
of
pods.
Then
it
does
number
of
nodes
based
on
the
the
testing
we've
done,
which
means,
if
you
have
a
really
big
cluster
system,
but
if
relatively
small
number
of
nodes,
starting
on
it,
you're,
probably
not
going
to
have
scaling
issues.
A
Interestingly,
this
should
be
probably
to
be
in
a
surprise,
but
if
you're
using
I'll
say
the
most
basic
features
of
kubernetes
you're
gonna
find
that
your
cluster
scalability
is
a
lot
better
and
when
you
start
using,
if
you
start
using
a
lot
of
the
more
advanced
features,
then
the
control
plane
performance
starts
to
take
a
hit.
I
just
threw
a
couple
of
example.
Up
here,
like
a
replication
set,
is
lower
load,
a
stateful
controller.
A
A
So
what
we
want
to
do
here
is
come
up
with
kind
of
a
deployment
pattern
and
way
of
thinking
about
this.
We
want
to
try
to
cover
the
you
know
what
it,
what
is
it
I
really
need
in
a
longer
running
cluster,
so
within
this
I
think
the
important
concept
is
that
it's
okay,
if
the
control
plane
is
unavailable
for
short
periods
of
time,
the
cluster
should
still
be
able
to
keep
running
it
might
mean
you
can't
schedule
a
new
job,
for
example,
but
that's
probably
okay.
A
If
it's
available
for
short
periods,
you
can
spend
enormous
amounts
of
effort
trying
to
come
up
with
something
with
a
hundred
percent
availability,
but
again
like
the.
If
components
are
dying,
the
cluster
itself
can't
go
away
and
you
want
a
way
to
think
about
how
am
I
going
to
adapt
this
cluster
for
different
conditions.
I've
gone
through
a
bunch
of
the
complications
here,
complications
around
number
of
pods
and
what
kind
of
features
I'm
using.
A
So
this
means
that
today,
the
engineering
that
you
have
to
do
here
around
this
is
a
little
bit
custom
to
your
situation.
So
you
want
to
have
a
way
of
thinking
about
how
to
do
that.
So
this
last
bullet
sharing
configurations
of
performance
info.
One
of
the
things
that
that
we're
fostering
trying
to
foster
trying
to
build
some
tooling
and
make
some
contribution
around
is
a
way
for
members
of
the
community
to
grab
their
cluster
configuration.
A
What
they
have,
including
things
like
settings
of
the
cluster,
be
able
to
dump
that
and
capture
that
and
export
it
for
sharing,
and
also
to
be
able
to
take
performance
information
from
that
cluster
and
do
the
same.
And
the
goal
here
is
to
be
able
to
cross
a
like
a
wide
range
of
different
kinds
of
circumstances
be
able
to
start
comparing
notes
across
the
community,
because
this
is
a
bit
early
and
these
configurations
are
very
custom.
A
I
think
you
know
at
least
it
would
be
great
if
there
was
I'll
say
even
a
version
of
a
wizard
grade
like
a
new
community
members
should
be
able
to
come
along.
Say,
oh
well,
like
here's,
the
here's,
the
characteristics
of
the
Buster's
I'm
trying
to
build,
and
we
should
have
a
better
database
of
information
that
helps
guide
what
the
configuration
of
those
clusters
should
be.
A
A
A
A
In
fact,
if
you,
if
you
talk
to
the,
if
you
talk
to
the
at
CD
folks,
I,
don't
know
it's
I,
don't
know
if
my
child
was
here,
but
if
you
the
sed
developers,
are
here
and
if
you
go
and
ask
them
about
it,
they'll
say
three
nodes
or
five
nodes:
that's
kind
of
the
T,
once
you
get
beyond
five
nodes
and
you're,
probably
into
extreme
at
CD
territory.
Although
I
know
people
do
it
so
really
three
node
minimum
is
what
you
want.
So,
let's
we'll
start
building
our
cluster
out
a
little
bit.
A
Okay,
so
that's
great
I
got
I,
have
three
at
CD
things
here:
okay,
so
here's
another
another
pro
tip.
So
there's
a
lot
of
events
that
flow
through
the
system.
The
event
the
event
load
is
partition
'l
from
the
cluster
state,
but
they
both
live
in
at
CD.
So
just
as
a
kind
of
principle,
you
really
want
to
protect
that
cluster
state
database
as
much
as
you
can
best
practice
and
I'm
no
Prince
here
gke.
A
A
The
API
server
can
be
stressed
with
high
loads.
So
it's
a
relatively
Stratasys
like
standard
web
deployment
type
stuff
in
terms
of
proxy's.
It
does
a
whole
bunch
of
state
watching
a
net
CD.
It
turns
out
the
performance
issues
here,
a
little
bit
complicated
because
in
the
newer
versions
of
kubernetes,
there's
a
bunch
of
caching
that
goes
on
inside
the
API
server.
But
still
it's
pretty
easy
to
load
balance
you
do
noodle.
Do
you
do
need
a
load
balancer
though
so
here?
Let's
do
this
all
right,
I'm
running
multiple
API
servers!
A
Now
that's
great
okay,
so
scheduler
and
controller
manager
are
not
chargeable,
but
you
still
need
a
chase.
You
can't
run
multiple,
multiple
hots,
but
you
still
have
to
solve
this
problem,
so
you
need
something
that
kind
of
looks
like
this,
so
this
here
I'm
showing
scheduler
as
a
backup,
backup,
scheduler
or
backup
controller
manager
again
going
through
the
load
balancer
to
get
to
the
API
server.
A
A
A
A
If
you
have
them
use
them
be
happy,
but
I
wouldn't
recommend
going
and
buying
those
appliances
new.
If
you're
building
new
clusters,
there
are
some
other
approaches.
You
can
use
all
right.
Okay,
so
there's
a
magic
alert
here.
I
did
a
little
bit
of
hand
waving-
let's,
let's
dig
in
on
some
of
the
hand
waving
here.
Okay.
So
how
does
that
work?
A
There's
got
to
be
something
that
deals
with
heart,
beating
and
fail,
overs
and
so
forth
turns
out
that
that's
built
into
kubernetes
itself,
so
implementation
there's
an
implementation
of
leader
election
patterns
negotiated
among
the
various
components.
The
followers
keeps
track
of
state,
a
failure
relieve
lead
to
a
new
leader
election.
So
good
news
is,
it
is
pretty
magical
because
the
software
is
already
there
to
do
it.
You
don't
have
to
roll
in
anything
on
top
of
it.
Okay,
so
so
that's
kind
of
the
next
dive
down
into
the
pieces.
A
A
So
what
you're
seeing
here
is
I've
just
laid
out
all
of
these
components
in
pretty
much
the
same
way,
every
time
so
lay
down
at
CD
on
all
three
lay
down
at
CD
and
all
three
again
scheduler
and
controller
manager.
We
use
leader
election
to
deal
with
that
spread
the
API
server
out
magical
load,
balancer
and
I'm
happy,
so
this
I'd
say
for
kind
of
that
next
step
of
how
is
it
that
you
build
a
che
cluster
for
a
cluster?
That's
going
to
be
using
a
hangar
over
a
long
period
of
time.
This
is
your.
A
This
is
your
minimum
configuration
you
could
even
depending
on
the
usage
of
the
cluster.
You
could
even
simplify
this
once
by
going
back
to
non
split
at
CD,
but
let's
say
our
recommendation
is
that
this
is
a
minimum,
a
minimum
config
for
running
your
own,
your
own
control,
plane
all
right
once
again.
Magic
alert
a
little
bit
of
hand
waving
us
there,
although,
if
you
have
an
f5
the
hand
waving,
is
pretty
good
alright.
So
so,
let's
say
you've
got
a
bunch
of
hardware
and
you
really
should
be
able
to
do
this
right.
A
You
should
be
able
to
have
a
cluster
with
a
bunch
of
commodity
Hardware.
Lay
it
all
out
in
such
a
way
that
you
can
get
what
you
want.
It
turns
out
that
in
kubernetes
there's
a
there's
this
there's
this
cube
proxy
that
does
magic
with
IP
tables
so
that
you
can
create
a
notion
of
a
service
IP.
So
what
we're
recommending
here
just
use
that
same
capability
within
the
cluster,
you
can
put
a
bend
put
a
proxy
in
front
of
it.
A
Engine
acts
are
each
a
proxy
or
pry.
It's
actually
possible
that
those
are
probably
possibly
overkill.
Even
for
this,
it's
probably
possible
to
maybe
do
something
even
a
little
bit
more
simply
so
I'm
kind
of
constraining
the
problem
here,
a
little
bit
for
the
bare
metal
case,
because
you're
my
assumption
here
is
that
you're
building
a
cluster
bare
metal
on
premises
and
you
have
some
control
over
DNS,
and
so
the
approach
here
is
have
a
stable
I
have
some
kind
of
stable
IP.
A
That's
mapped
to
DNS
map
that
I
address
into
into
this
system
here
have
and
then
have
some
way
to
do
some
kind
of
virtual
IP
or
some
kind
of
takeover
of
that
IP
address
within
the
cluster.
Should
one
of
those
nodes
fail.
This
is
pretty
common.
This
is
a
pretty
common
approach.
I'm
used
for
a
great
many
things.
The
alternative
to
an
approach
like
this
is
just
use.
A
A
If
that
control
plane,
that
I
just
laid
out
is
not
quite
big
enough,
here's
we're
having
a
pattern,
a
deployment
pattern
that
you
can
use
or
something
like
this
gives
you
a
method
for
thinking
about
how
you
would
do
the
control
plane,
engineering
here
and
so
here's
an
example
of
how
you
could
do
it
right.
So,
instead
of
running
on
three
machines,
I've
now
I'm
now
running
nine
machines.
A
This
is
probably
overkill
for
most
folks,
although
I
guess
I
should
say
overkill
depending
on
what
kind
of
servers
you
have.
If
you
have
a
big
pile
of
really
small
servers,
maybe
maybe
a
pattern
like
this
work.
So
again
it's
back
to
trying
to
figure
out
how
to
customize
this
kind
of
system
for
your
particular
requirements.
A
So
that'd
be
one
way
here:
let's,
let's
go:
let's
go
even
a
little
bit
more
extreme
here,
I'm
gonna
I
want
to
run
more
API
servers
for
some
reason.
My
pattern
works
in
that
case.
Probably
three
since
I
have
to
run
a
master
election
system,
I'm
not
doing
load
balancing
across
the
scheduler
and
the
controller
manager.
Probably
three
is
plenty
and
then
it
might
be
that
for
my
configuration.
A
A
Cases
in
spreadsheets,
rather
than
with
automation,
tools
and
the
the
sort
of
goofy
thing
about
this,
is
that
what
I'm
building
here
is
a
system?
That's
designed
to
do
exactly
this
kind
of
work
for
us,
so
yeah
a
lot
of
shouldn't
the
system
be
doing
that
and
yes
and
the
system
you
would
want
to
be
using
for
this
as
gerber
Nettie's
itself.
A
Partly
this
is
eat
the
dog
food.
You
already
have
the
system
they're
running.
You
already
have
the
familiarity
with
how
it
works,
so
we
believe
I
think
a
lot
of
people
in
the
community
believe
that
the
notion
of
self
owes
hosting
kubernetes
is
the
right
one.
So
the
idea
here
is
it's
a
you
know
entirely.
You
know
system
entirely
running
of
kubernetes
objects.
You
have
this
common
operations
said
it's
gonna
make
scaling
like
this
pictures.
I
was
showing
scaling
and
H
a
easier.
A
The
tricky
part
here
is
that
bootstrapping,
a
cluster
from
I
mean
you.
Have
you
don't
have
the
cluster
from
go,
so
you
have
to
kind
of
get
the
cluster
running
and
then
hand
control
of
it
back
to
itself,
and
so
there's
some
tricky
bootstrapping
things
that
need
to
happen
there,
which
is
why
this
is
not.
You
know
just
a
done
deal
but
I
think.
We
certainly
believe
that
that
tricky
bootstrapping
is
worth
it,
especially
if
you're
running
a
big
cluster.
A
If
you're
really
running
really
small
clusters,
just
stamping
them
out
really
small,
with
a
single
node
I
mean.
Is
it
worth
the
complication?
Maybe
not
you
start
getting
into
places
where
you're
really
worrying
about
control
plane,
engineering?
This
is
definitely
one
of
the
places
you
want
to.
You
want
to
go
I'm.
Also.
Actually,
upgrading
and
updating
the
system
is
gonna,
be
a
lot
easier.
There's
one
of
the
things
kubernetes
is
really
great
at
is
doing,
updates
enough
updates
and
upgrades.
So
I
put
a
note
here.
This
is
the
core
OS
blog
post.
A
They
had
a
talk
today.
If
you
didn't
see
the
talk
and
you're
interested
in
this
I'd
strongly
recommend
you
find
it
online
I,
don't
know
when
the
videos
are
gonna,
be
posted,
but
I'm
sure
it'll
be
worthwhile.
I
know
that
that
session
was
standing
room
only
or
you
couldn't
I
was
seeing
pictures
of
people
not
even
being
able
to
get
in
there.
So
I'm.
Sorry,
oh
there
is
oh
okay,
great
the
the
comment
was
there's
another
talk
on
it
tomorrow.
A
So,
but
let
me
make
one
more
point
on
that,
which
is
we're
kind
of
part
of
the
way
there
already.
So,
if
you
so,
if
I
go
back
to
my
three
node
cluster
layout
here,
let
me
let
me
turn
this
down
a
little
bit
a
little
bit
easier.
So
this
is
the
way
we
run.
Our
clusters,
today
I,
don't
actually
have
a
very
good
feel
for
how
common
this
is
across
I'll,
say:
kubernetes
users
generally.
My
sense
is
that
this
is
fairly
common,
although
you
certainly
don't
have
to
run
your
brandies
this
way.
A
So
we
run
we
run
this
today
as
a
set
of
so
we
run
pods,
but
we
run
them
as
statically
scheduled
as
opposed
to
using
the
kubernetes
capability
to
do
the
scheduling
so
today,
for
us,
these
are
static.
Pods,
but
dynamic
pods
are
coming
I'm,
not
really
sure
how
long
that's
going
to
take
I'd
say
maybe
across
the
course
of
the
next
year.
A
Today,
sharing
this
kind
of
information
is
not
as
easy
as
we
would
like
for
it
to
be,
and
so
we're
putting
effort
into
a
couple
of
related
things.
One
is
we're
calling
cluster
config
dumping
I've
put
a
couple
of
a
couple
of
pull
requests
here
that
we've
been
working
on
and
the
other
is
so
there's
the
configuration
and
then
there's
performance
dumping.
The
notion
around
the
performance
dumping
here
is,
although
you
could
use
it,
for
you
could
use
it
for
any
workload.
A
Our
notion
here
is
that
you
really
do
want
to
be
able
to
take
the
cluster
that
you
have
run
one
of
the
standard
community
tests
on
that
cluster.
You
know
density
for
example,
and
then
grab
that
and
then
be
able
to
use
that
to
compare
your
configurations
with
other
folks
in
the
community.
So
this
is
early
work.
A
Some
of
the
cluster
performance
dumping
we've
been
doing
in
our
own
github
repo,
but
we've
been
working
on
this
in
the
with
scaling
sig
as
the
home.
In
fact
you
can,
if
you
go
and
look
at
the
YouTube
videos,
you
can
even
see
some
of
the
demo
stuff
we're
working
without
so
that's
work
in
progress.
If
you're
interested
in
that,
please
come
talk
to
me
or
come
attend
scaleability
sings
meetings,
Thursdays
9:00
a.m.
it's
on
the
it's
on
the
the
kubernetes
page.
Alright.
So
that
is
my
talk
for
today.
A
Here's
some
contact
info
all
of
the
work,
all
of
the
work
that
our
team
does
is
in
open
source,
so
we're
either
contributing
to
kubernetes
or
I
mean
other
projects
as
needed,
but
any
work
that
we're
doing
that
sort
of
outside
a
project.
We
also
do
and
is
on
our
github
and
our
github
repo.
It
is
available
in
the
public
and
feel
free
to
hit
me
on
kubernetes
slack
or
by
email.
A
A
A
A
A
Yeah
I
think
I
think
I
think
the
question
was:
what
do
we
think
about
kind
of
a
distributed
load?
Balancer,
that's
running
and
more
of
the
cluster,
so
that
the
load
balancing
capability
could
move,
be
moved
to
the
edges
and
I'd
have
to
think
about
that,
and
one
of
the
other
Samsung
folks
here
is
saying,
you
know-
might
be
more
complication
than
it's
worth
so
anyone
anyone
else.
Oh,
don't
be
shy.
I
have
a
whole
two
minutes,
all
right,
I
guess
we'll
call
it
a
wrap
thanks
very
much.