►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: VMware SIG – Michael Gasch & Steven Wong, VMware
Kubernetes allows using topology labels to affect the scheduler’s placement of pods. This is used to spread pods across availability zones, while still respecting resource access and availability concerns. When Kubernetes runs on vSphere, the hypervisor platform also supports an underlying tier of high availability and automated placement options, for both control plane and worker nodes. 2 levels of scheduling and resource management are active. Currently no automatic scheduling integration occurs, that is, Kubernetes is not aware of the underlying vSphere topology (sites, affinity groups, NUMA, etc.). This session will explain the options to gain better performance, resource optimization and availability through tuning of vSphere, and Kubernetes configuration and labeling. This is applicable to any K8s distribution running on the vSphere stack.
To learn more: https://sched.co/Grd6
A
If
you
came
to
the
deep
dive
on
to
Burnett
deep
dive
into
kubernetes
scheduling
you're
at
the
right
spot,
so
this
is
going
to
coach
you
on
running
kubernetes
on
top
of
vSphere.
It
could
be
useful
if
you're
running
in
a
VMware
distribution,
but
this
stuff
also
applies
if
you're
running
any
other
cougar
any's
distribution
on
top
of
vSphere.
B
A
You
already
read
this
abstract
if
you're
here,
I'm
gonna
skip
it,
but
the
only
reason
I
left
it
in
the
deck
is
so
that
if
somebody
downloads
the
the
the
deck
later
that
they
can
scan
that
at
the
beginning
and
know
whether
they
want
to
read
the
rest.
This
is
the
agenda
for
today,
we're
going
to
start
with
covering
kind
of
kubernetes
default
scheduling
how
it
works,
regardless
of
whether
you're
on
a
hypervisor
platform
or
not
moving
into
the
zones
feature
that
can
be
used
with
vSphere
to
get
some
valuable
and
enhanced
features.
A
Next,
we'll
go
on
to
Numa,
we'll
explain
what
it
is
when
we
get
there
non-uniform
memory,
architecture
Michael
is
going
to
pick
up
and
cover
kubernetes
default
resource
management
and
ways
of
extending
the
the
default
functionality
of
kubernetes
by
utilizing
vSphere
features
in
conjunction
with
the
built-in
features
of
kubernetes.
Finally,
I
left
it
here.
We've
delivered
this
a
month
ago
in
Cuba
and
Shanghai,
and
ran
into
the
time
limit,
so
I
left
it
in
the
deck.
If
you
download
it
later,
we
may
or
may
not
have
time
to
cover
highly
high
availability
options.
A
So,
let's
move
on
to
kubernetes
scheduling,
what
is
it?
What
exactly
does
the
scheduler
do?
Well
through
the
API
pods
are
created
and
they're
placed
in
a
queue,
there's
a
beta
feature
that
potentially
can
cause
things
placed
in
the
queue
to
jump
ahead
of
others,
but
you'd
have
to
enable
it
then
the
scheduler
after
they
pour
into
this
funnel.
The
scheduler
continuously
pulls
pods
out
of
the
queue,
looks
at
their
requirements
and
assigns
it
to
a
worker.
Node,
assuming
you
know
you
have
more
than
a
one
node
cluster
when
it
does
this
assignment.
A
A
Kubernetes
is
very
modular,
so
it
comes
with
some
default
predicates,
but
this
is
extensible,
so
some
of
them
are
available
off-the-shelf
open-source,
but
you
could
write
your
own.
So
this
initial
filter
stage.
If
you
look
at
that
picture,
you
see
the
X's
over
those
and
those
are
just
a
determination
by
one
of
these
predicates
that
there's
no
way
that
this
this
this
pod
is
going
to
run
on
this
node
afterward
you're
left
with
a
list
of
eligible
nodes,
and
in
this
case
it
would
be
those
orange
ones.
A
So
we've
got
six
possible
nodes
that
are
not
meaning,
they're,
not
impossible,
and
there
might
still
be
preferred
nodes
to
run
those
odd.
So
this
ranking
is
called
priorities
in
the
kubernetes
architecture.
This
once
again,
is
extensible,
so
there's
a
default
list.
An
example
of
this
is
zones
which
we'll
get
into
in
just
a
bit,
but
there
are
others
and
like
I
say
you
come
with
some
of
those
enabled
by
default.
Some
are
non
default
and
if
you
want
to,
you
can
write
your
own.
A
So
after
all
of
these
stages
of
eliminating
the
impossible,
ranking
them
and
jumping,
you
know
to
a
preferred
order
list.
Finally,
that
pod
is
assigned
to
a
node
and
is
going
to
get
kicked
off
and
running
now
I
will
say:
I'll
go
into
the.
This
is
a
deep
dive,
so
in
that
ranking,
if
you
have
a
very
large
cluster,
the
thing
doesn't
necessarily
evaluate
every
single
note.
You
know,
I'm
talking
like
say:
I
have
a
thousand
worker
nodes
on
my
kubernetes
the
system,
and
this
this
can
be
overridden.
A
But
by
default
there's
some
number
saying
that
if
there's
a
humongous
number
of
kubernetes
nodes
and
I've
got
hundreds
of
them,
I'm
going
to
just
take
a
percentage
of
these
and
rank
those
and
it's
a
latency
concern.
If
you
crawled
over
these,
it
just
isn't
worth
it
and
once
again
you
can
turn
off
that
behavior,
but
I'm
just
telling
you,
since
it's
a
deep
dive,
but
that
is
something
going
on
under
the
covers,
if
you're
running
massively
large
clusters,
so
there
are
scheduling
modifiers
that
can
affect
this.
A
The
first
one
is
a
node
selector,
and
this
is
based
on
key
value
labels
you
put
on
the
worker
nodes
themselves.
Some
of
these
are
automatically
created,
but
if
you
want
to,
you
can
annotate
this
with
your
own,
for
whatever
purposes
you
might
have,
pods
can
define
rules
based
on
these
node
labels
to
achieve
preferred
placements
called
affinity.
A
So
the
the
sum
total
list
of
the
elements
that
influence
these
pod
placements
are
labels,
unknowns
and
zones
as
an
example
of
this
taints
in
Toleration
x',
where
you'd
mark
nodes
with
arbitrary
labels,
which
could
correspond
to
a
specific
resource
or
whatever
you
like
that,
could
either
clue
in
the
kubernetes
scheduler
that
I
don't
want
to
go
to
this
node
or
a
Toleration
says
yeah.
You
know
all
things
aside:
I'm
willing
to
go
there.
A
There
are
admission
controllers
and
a
wide
varieties
are
available.
An
admission
controller
would
take
the
pod
spec
and
either
validate
it,
meaning
it's
going
to
scan
it
for
acceptable,
behavior
or
potentially
even
mutated.
We're
a
pod
spec
might
be
missing
material,
and
this
would
inject
the
default
behavior
and
if
I
use
zones,
I'll
turn
it
over
to
my
colleague,
Michael
yeah.
A
B
B
Yes,
yes,
yes,
the
device
plug-in
yeah
right
will
do
I
just
have
to
understand
it
first
yeah.
So
your
question
is,
if
I'm
using
special
devices
like
GPUs
or
FPGAs
in
my
cluster,
how
would
I
protect
those
nodes
from
pods
being
scheduled
on
on
the
nodes
that
are
Pro
yeah
but
spots
you,
the
pods,
using
that
kind
of
hardware
making
sure
they
land
on
that
one
and
protecting
the
nodes
from
parts
not
using
GPUs,
for
example,
to
trouble
the
node?
So
it's
cut,
you
could
do
both.
B
It's
it's
important
to
understand
that
in
vSphere
we're
kind
of
running
kubernetes
on
vSphere,
we
have
like
two
levels
of
scheduling
or
abstractions:
we've
got
the
vSphere
scheduling,
which
is
like
DRS
and
the
kernel.
It's
actually
multiple
layers
there,
and
on
top
of
that
we
have
the
kubernetes
scheduler
and
logic,
and
probably
you
have
other
systems
on
top
of
that
running.
So
we
have.
We
have
multiple
hierarchies
of
schedule
involved
when
running
kubernetes
on
vSphere.
B
What
you
can
do
to
define
what
a
zone
in
your
environment
is
because
every
busier
environment
is
different,
you're,
probably
not
running
Street
data
centers
like
an
AWS
or
or
Google
cloud,
so
you
could
run
like
three
hosts
which
make
up
a
zone
or
you
couldn't
run
three
recs
which
could
make
up
a
zone
or
you
could
have
three
data:
centers
modeling
a
zone.
So
we
implemented
that
and
we
1.12
to
have
like
an
arbitration
and
knowledge
sharing
between
the
kubernetes
scheduling
and
the
vSphere
scheduler.
A
And
ultimately,
the
zones
feature
is
to
spread
things
across
failure
zones
like
automatically
with
you
minimizing
the
operational
cost
of
caught
of
achieving
that
now
we're
moving
on
to
a
subject
called
Numa.
What
is
it
non-uniform
memory
architecture
when
you're
dealing
with
server
grade
x86
hardware,
multi
socket
those
things
are
laid
out
in
a
motherboard
and
between
the
sockets
you've
got
this
pipe
line
drawn
in
red
there
and
you
might
have
much
faster
access
to
the
memory
in
the
dims
near
the
socket,
as
opposed
to
the
one.
A
On
the
other
side,
the
data
a
workload
running
on
any
either
of
these
CPUs
can
still
get
to
the
memory
on
the
other
side.
But
in
this
case-
and
this
varies
by
motherboard
and
chip
family,
but
in
this
case
that
bottleneck
in
the
center,
there
is
41
gigabytes
a
second
and
there's
also
a
latency
penalty
when
you
cross
that.
So
what
you've
got
a
scenario
where,
when
these
workloads
are
laid
down
by
kubernetes,
you'd
really
like
to
have
an
outcome
where
things
that
can
fit
on
one
side
or
the
other
actually
end
up
running
there.
A
That
really
is
sometimes
highly
performant,
sometimes
not,
and
you
end
up
with
non-deterministic
behavior.
The
these
non-deterministic
scenarios
tend
to
be
pretty
ugly
when
you're
in
an
organization
that
has
to
sign
up
for
a
specific
service
level
agreement.
So
you
know
Linux
has
some
recognition
of
this
when
it
initially
allocates
a
thread.
It's
assigned
to
a
preferred
node
with
default
allocations
from
this,
but
there
are
some
popular
application.
A
That
would
carve
this
thing
up
in
that
drawing
before
like
split
many
of
these
servers
will
obviously
have
more
than
two
sockets,
but
if
you
were
to
carve
this
up
into
two
VMs
the
size
of
either
side,
you
might
be
better
off
now,
if
you
have
monster
workloads
that
in
fact
would
suck
up
that
whole
machine.
Maybe
you
don't
want
to
do
this,
but
in
many
cases
you'll
have
clusters
where
you
know
for
efficiency
in
your
data
center
you'll
be
buying
larger
density
machines.
A
Just
so
you
can
fit
more
CPU
cores
in
yet
you
don't
have
workloads
that
hog
an
entire
machine
at
once.
You
may
be
better
off
doing
this
in
a
way
that
would
allow
the
underlying
hypervisor.
To
give
you
a
hand
with
the
non-uniform
memory
architecture,
so
how
can
these
be
avoided?
Well,
if
you
didn't
have
this,
if
you're
running
pure
Linux,
you
can
wrap
an
application
in
this
Numa
act,
I'll
I'm,
not
sure,
I'm
pronouncing
it
right
to
interleave
memory,
that's
something
where
you're
telling
it
just
to
get
predictability.
A
A
Maybe
in
your
organization,
you've
standardized
on
something
else
and
there
are
non
Java
things
that
encounter
the
same
kind
of
issue.
There
are
active
talks
going
on
in
enhancing
these
kinds
of
Numa
behaviors
in
a
the
resource
management
working
group
and
get
the
deck
later
and
the
link
is
live
here.
If
you
want
to
read
more
about
it,
so
you
can
use
an
uma
where
hypervisor
to
help
you
right
now
without
waiting
for
these
enhancements
that
are
still
being
discussed.
You
know,
assuming
your
workload
fits,
you
make
a
VM
as
a
walled
garden
sized
right.
A
You
know
to
an
appropriate
size
to
respect
those
boundaries.
If
you
can't
there's
some
there's
some
things,
you
ought
to
do
when
you're
laying
out
these
VMs
and
laying
out
your
pods
to
cause
the
system
to
not
have
a
tendency
to
cross
these
boundaries.
So
don't
declare
things
with
larger
socket
counts
than
you
need
for
your
requirement
in
general.
Never
pick
a
VM
with
an
odd
number
of
CPUs.
It's
just
going
to
tend
to.
A
Maybe
it
won't
cause
an
uma
boundary,
but
even
if
it
doesn't,
it
could
tend
to
waste
resources,
but
where
it
does
cause
one.
Is
you
end
up
with
this
odd
number
CPU
that
appears
to
be
available
resource?
The
kubernetes
scheduler
might
think
it
can
throw
more
in
there
yet.
The
only
way
it's
ever
gonna
run
is
to
cross
the
boundary.
You
don't
want
to
be
crossing,
never
ever
compose
a
VM
larger
than
the
number
of
physical
cores,
and
if
you
do
that,
there
are
also
some
additional
enhancements.
Look
at
the
deck
later
rare.
A
B
B
This
is
one
of
them
most
like
top
three
that
have
faced
in
the
field
and
working
with
customers,
performance
issues
in
nuuma
so
and
we
make
it
pretty
easy
for
you,
as
Steve
said
as
long
as
you
stay
within
the
Neumann
boundary,
meaning
look
at
your
socket
design,
two
sockets
10
cores
each
you'll
probably
define
your
VMs
in
a
way
that
they
are
less
than
10
or
equal
to
10v
CPUs
there.
So
they
nicely
fit
into
a
socket.
That
means
vSphere.
B
You
express
CPUs
in
milli
course,
or
course,
but
then
they
are
broken
down
into
milli,
cores
and
Millie
cores.
They
are
a
relative
unit.
They
are
no
not
an
act
like
an
an
absolute
unit
that
you
specify,
meaning
they
can
vary
between
the
environments
where
you
run
the
kubernetes
vm.
The
workers
on
on
top
of
that.
The
two
cores
on
a
AWS
ec2
instance
are
definitely
different
and
two
cores
on
HTTP
instance,
because
they
were
just
relative.
Now
in
this
fear,
we
are
not
counting
chorus
like
internally
we're
using
megahertz
and
the
indie
hypervisor.
B
So
in
the
in
the
UI,
you
have
the
option
to
add
cores,
but
internally
they
are
mapped
to
a
megahertz
and
India
is
already
a
little
bit
of
difference
going
on
here,
which
I'm
gonna
touch
later
and
for
memories.
It's
not
a
big
deal,
because
we
also
work
with
bytes
so
kubernetes.
As
of
like
an
absolute
number
of
a
unit
in
here
that
you
specify-
and
there
are
several
other
like
resources
that
you
can
specify
like
already
GPUs
were
mentioned
or
FPGAs
or
any
other
special
devices.
B
Some
of
them
are
not
available
on
the
vSphere
platform
like,
for
example,
FPGAs
mapping,
FPGAs
through
or
very
specific
network
cards
that
we're
not
supporting
on
the
hypervisor.
So
that
could
be
an
issue
for
GPUs.
We
are
usually
pretty
good
because
we
work
closely
with
AMD
and
NVIDIA,
so
we
have
good
decent
GPU
support,
we're
also
supporting
the
NVIDIA
GPU
about
your
GPU,
meaning
you
cough
up
physical
CPU
into
multiple
virtual
GPUs,
and
you
spread
them
across
the
VMS.
B
So
in
kubernetes,
then
a
developer
or
a
user
or
whoever
can
define
requests
and
limits
for
each
resource
that
you
can
specify
in
a
pot
so
for
CPU
and
memory
or
your
GPUs.
Some
of
them
are
allow
different
numbers,
meaning
that
you
can
specify
different
number
of
requests
or
limits
for
CPU
and
memory
for
GPUs.
B
You
have
to
set
them
to
the
same,
because
the
scheduler
doesn't
support
over
commit
on
GPUs,
for
example,
and
those
requests
are
done,
internally
mapped
into
Linux
constructs
and
they
were
mapped
into
the
vSphere
hypervisor,
which
we'll
see
in
two
slides
but
generically
speaking.
You
have
some
options
to
control
the
resource
management
side
on
kubernetes,
but
you
have
to
also
make
sure
that
kubernetes
runs
on
a
shared
platform
which
is
vSphere
and,
as
we
already
said,
there's
a
two-level
scheduling
going
on
between
the
vSphere
platform
and
kubernetes.
B
Unfortunately,
there
are
often
different
goals
that
you
want
to
achieve
in
a
kubernetes
cluster,
and
this
is
a
from
a
talk
that
I
gave
in
Kubik
on
Copenhagen
earlier
this
year
going
deeper
into
that
matter.
But
just
by
having
this
controls
of
limits
and
requests,
they
are
not
always
like
represent
what
your
goals
are.
When
you're
running
kubernetes
cluster,
for
example,
you
could
your
priority
could
be
running
high
utilization
in
your
kubernetes
cluster,
or
you
want
to
be
very
fair
between
workloads
or
you
want
to
prioritize
some
of
the
workloads
in
a
community's
cluster.
B
So
you
have
to
really
think
about
how
your
model
resource
management
in
your
kubernetes
cluster
and
I'm
just
going
to
refer
here
to
the
talk
that
I
gave
at
cube
con
CPU
and
memory
when
it
comes
to
like
over-committing.
So
you
you
express
more
resources.
You
give
more
resources
out
to
your
developers,
then
you
actually
physically
have
environment.
B
Kubernetes
has
some
protection
mechanisms
when
it
comes
to
over-commitment,
but
they
are
different
between
CPU
and
memory,
because
those
are
the
resources
that
you
can
overcome
it
for
CPU.
That's
pretty
easy
because
CPU
is
a
compressible
unit,
meaning
if
you
overcome
it
in
CPU
and
there's
contention
on
that
note
on
the
Booker
VM,
the
kubernetes
node,
the
couplet,
which
starts
throttling.
Actually
it's
the
Linux
kernel
starting
to
throttle
your
your
workloads,
so
you
better
make
sure
that
requests
and
limits
you
get
them
right
for
CPUs,
especially
for
workloads,
are
critical.
B
B
Now,
as
we
already
stressed
and
I'm
already
I'm
stressing
this
running,
kubernetes
on
vSphere
is
kind
of
the
multi-level
scheduling
being
involved
here
and
on
that
slide,
you
basically
the
higher.
You
see
the
hierarchies
that
those
values
of
CPU
requests
and
limits
and
memories
are
being
translated
into
the
Linux
kernel,
which
is
the
couplet,
in
your
case,
the
worker,
the
worker
VM
running
on
vSphere,
and
then
underneath.
You
also
have
some
controls
on
the
ESXi
hypervisor
to
control
CPU
memory
behavior
from
the
hypervisor
side.
B
Unfortunately,
not
all
of
them
directly
translate
into
the
lower
level
primitives,
for
example,
CPU
requests
from
a
pot
spec
manifest
perspective,
which
says
I
request.
Two
CPUs
translates
into
CPU
shares.
Phares
is
not
a
absolute
value,
it's
just
a
relative
value,
and
if
your
host
gets
constrained,
the
number
of
shares
are
being
used
to
determine
whether
your
workload
is
getting
throttled
or
not.
B
So
if
you
specify
two
CPUs
as
a
request,
that
doesn't
mean
you
are
really
getting
two
CPUs
at
the
end
of
the
day,
if
you're
over
committing
on
the
note,
CPU
limits
are
translated
into
CPU
quota.
Now
the
downside
of
using
CPU
limits
is
that
you
will
always
get
throttled,
no
matter
whether
this
host
the
worker
is
actually
under
constraint
or
not
so
for
the
limits
you
have
to
make
sure
that
you
set
them
right.
B
If
you
set
them
too
low,
your
workloads
will
always
be
struggled,
which
is
not
good,
for
example,
for
ingress
or
api's.
If
you
set
them
too
high
chances
are
good
that
you
over-committing
your
node
and
then
the
node
gets
under
trouble
and
is
being
oversubscribed
in
ease.
Here
we
have
some
controls,
so
we
have
CPU
shares
as
well,
so
they
would
translate.
But
in
the
field
we
advise
our
customers
and
users
of
kubernetes
on
vSphere
to
use
CPU
reservation.
B
Cpu
reservations
are
a
guarantee
that
you
get
from
the
vSphere
hypervisor
and
so
for
critical,
VMs,
critical
workloads,
user-facing
workloads,
databases
whatsoever.
We
recommend
setting
100%
reservation
for
memory
and
CPU
on
that
VM.
Just
to
make
sure
the
hypervisor
will
always
give
you
the
resources
that
you
need
in
case.
You
need
them
and
you
can
still
overcome
it.
B
That
means
you
can
run
other
VMs
which
are
less
critical,
with
50%
reservation
or
20%,
or
no
reservation
at
all,
and
if
kubernetes
does
not,
if
that
worker,
like
the
production,
VM,
doesn't
need
all
the
resources
we
give
them
to
other
ones.
But
if
the
production
PM
needs
the
resources,
we
reclaim
them
back
via
the
hypervisor
kernel
and
give
them
to
the
production.
So
mapping
and
understanding
those
constructs
requests
and
shares
and
reservations
is
critical
to
like
a
stable
and
robust
kubernetes
environment,
not
just
for
the
masters,
but
also
for
their
father.
B
Your
workers
we're
still
good
in
time,
so
we
can
cover
DRS
and
AJ
and
a
little
bit
deeper
here
so
these
here
they
are.
As
for
those
of
you
not
familiar
with
DRS
DRS
is
a
distributed
resource,
scheduler,
meaning
it
looks
at
the
cluster
of
e-cigs
I
host
and
it
looks
for
balanced
resource
utilization
of
your
of
your
VMs
to
avoid
hot
spots
in
your
cluster,
especially
in
cases
when
a
hypervisor
is
over
committed
or
there's
a
lot
of
trouble
on
one
node
DRS
will
look
like
by
default
right
now.
B
B
The
question
is:
does
DRS
require
Visa
Visa?
No,
so
yes,
we
son
the
storage
solution
from
members
not
required
for
DRS.
The
IRS
requires
Enterprise
Plus,
as
I
saw
like
license.
So
it's
not
part
of
all
of
the
vSphere
editions,
but
it
has
no
requirement
from
other
products
that
we
have
right
and
so
DRS
is
a
kind
of,
as
it
says
in
the
name.
It's
a
disability
resource
scheduler.
You
could
think
think
of
like
kubernetes,
as
well
as
being
a
disability
resource
scheduler.
So
it
looks
at
your
Asics
I
host.
B
It
has
no
understanding
of
kubernetes
all
the
workloads
within
those
VMs.
It
just
looks
for
CPU
and
memory,
and
if
you
want
storage
and
network
contention
and
then
it
tries
to
balance
your
cluster,
so
this
is
super
helpful,
but
you
also
have
to
constrain
it,
meaning
in
DRS.
It
could
happen
that
a
VM
on
one
horse
is
being
migrated
to
a
VM
on
the
other
host.
B
Now
without
the
zone,
support
or
without
any
support
to
model
your
vSphere
topology,
be
it
racks
or
data
centers
to
the
kubernetes
constructs
like
zones
or
failure
domains
without
them,
the
VMS
could
migrate.
In
your
cluster
and
potentially
violating
an
affinity
or
anti
affinity
decision
that
a
cubed
is
scheduler
has
been
made.
On
top
of
this.
B
As
an
easy
example,
imagine
three
nodes
on
the
Left
are
part
of
one
zone
and
the
other
nodes
are
part
of
the
alisonj,
without
tying
those
things
together
like
in
forming
kubernetes,
that
it
has
different
topologies,
a
kubernetes
scheduler
could
schedule
a
pod
on
the
same
VM
or
on
different
hose
in
the
same
zone,
even
though
a
user
would
have
requested
an
anti
affinity
because,
from
a
kubernetes
perspective,
all
it
looks
like
for
the
virtual
notes,
which
is
a
VM.
In
that
case,
those
two
VMs
could
end
up
running
on
the
same
physics.
B
I
have
proviso
in
order
to
prevent
that
we
have
some
controls,
and
this
is
a
1
to
12.
We
have
the
topology
support
and
the
VCO
cloud
provider.
We
have
DRS
affinity
groups
where
you
can
model
like
those
hosts
or
part
of
one
group,
and
those
are
some
part
of
the
other
group,
and
then
you
can
also
specify
anti
affinity
and
affinity
between
the
VMs
in
your
cluster.
B
For
example,
you
want
to
make
sure
that
your
master
of
three
masters
5
mas
por
masters,
how
many
you
run
that
not
all
of
them
run
on
the
same
as
ex
I
host
based
on
the
arrest
rules,
you
can
actually
model
that
I
do
not
see
a
lot
of
customers
using
those
rules
because
they
think
that
kubernetes
will
figure
that
out.
But
if
you
don't
try
those
kind
of
two
domains
like
the
vSphere
domain
and
the
kubernetes
domain
together
by
informing
them,
they
are
the
cloud
provider
integration.
A
A
B
This
is
just
an
image
of
how
you
would
specify
a
anti
affinity,
rule
or
I,
find
it
a
finicky
rule
for
for
the
masters
here.
So
we've
got
different
different
roles
and
in
this
scenario
here
we
have
modeled
a
fault
domain,
which
is
basically
a
rack
or
your
zone
out
of
those
four
hosts.
They
go
in
host
vm
rules
and
then
you
can
model
your
VM
enter
affinity
rules
to
put
the
master
on
different
host
even
in
the
same
domain.
So
it's
pretty
flexible
it
can.
It
can
get
easily
confusing
and
complicated.
B
So
we
don't
advise
our
users
to
choose
the
UI
and
do
it
manually.
We
have
in
one
of
our
CL
eyes
or
like
SDKs.
We
have
support
for
setting
this
programmatically,
for
example,
as
a
post
deployment
job
and
you
provision
a
VM
that
after
the
provisioning
is
completed,
you
set
the
rules.
We
are
an
API
call
and
darkus
here
is
I
want
to
just
give
it
a
shout-out.
So
go.
Vc
is
a
go
binary.
B
Customers
who
use
Gogi
C
to
do
the
post
provisioning
tasks
to
make
sure
the
VM
lands
on
the
same
or
on
different
different
hosts.
For
example,
one
last
comment
on
some
things
to
watch
out.
The
zone
interface
implementations
actually
split
into
two
phases.
The
first
phase
we
implemented
in
kubernetes
1.12
make
sure
that
if
you
run
different
host
groups
like
you
have
different
vSphere
clusters,
modeling
your
zone,
topology,
be
the
data
center
or
be
at
a
separate
vSphere
cluster.
B
We
define
than
a
vSphere
cluster
or
data
center
as
a
separate
zone,
meaning
that
those
these
four
clusters
don't
know
each
other.
Now
we've
seen
customers-
or
we
have
many
customers
running
stretch
clusters
stretch
these
four
clusters
across
two
data
centers,
for
example,
where
a
vSphere
cluster
is
kind
of
a
large
vSphere
cluster
stretches
across
two
different
data
centers,
but
it's
still
the
same
vSphere
cluster.
So
we
don't
have
multiple
vSphere
clusters
in
that
game.
Right
now
that
implementation
that
we
did
in
kubernetes
1.12
does
not
automatically
support
that
topology.
B
We
have
some
workarounds
that
you
can
use
if
you're
running
stretch
clusters,
for
example
like
Metro
clusters
or
IV
plexus
or
whatever
in
the
environment,
so
just
like
use
that
link
or
the
number
of
the
issue
there
and
you
get
some
feedback
or
our
slack
Channel
4:40
p.m.
are
sick,
and
we
kind
of
give
you
some
help
in
that
case.
But
if
you
have
the
luxury
of
running
separate
vSphere
environments
like
separate
VC
for
clusters,
then
it's
easy
really
easy
to
model
the
disowns
implementation
as
we
did.
So
that
is
the
phase
one
implementation.
B
Another
issue
right
now
that
were
working
on
is
coming
from
that.
Having
separate
vSphere
clusters
for
your
environment,
you
might
want
to
have
storage,
dynamic
storage,
provisioning,
which
is
a
nice
feature
in
kubernetes.
If
you
want
to
work
with
with
persistent
volumes
now
right
now,
the
vSphere
cloud
provider
does
not
support
deploying
dynamic
storage
or
volumes
in
those
setups,
where
there's
no
shared
storage
amongst
the
clusters.
Just
to
repeat
that,
if
you
have
separate
vSphere
clusters
or
hosts,
it
could
also
be
single
hosts
and
they
don't
share
storage
amongst
them
in
between
them.
Like
there's.
B
No
NFS
shared
storage
or
there's
no
VMFS
stretched
across
those
nodes.
It's
right
now
it's
not
possible
to
use
dynamic
volume
provisioning
in
kubernetes,
so
we're
working
on
that.
That's
just
a
logic,
a
logic
constraint
that
we
have
in
a
cloud
provider
that
we
need
to
fix
and
we'll
hope
to
fix
it
in
the
next
in
the
next
portion.
B
So
those
are
the
two
major
issues
that
besides
performance
I,
see
in
the
field
and
we
just
want
to
make
sure
that
we
prevent
them
by
educating
or
our
user
base
and
there's
workarounds
for
for
all
of
them,
but
all
of
them
are
nice,
but
we're
working
on
on
that
dressing
that
all
right,
that's
it
from
my
side.
I
think
we
are
good
on
time.
I
think.
A
B
B
We
are
actually
we're
working
on
completely
revamping
the
way
storage
is
implemented
in
the
vSphere
cloud
provider
right
now,
and
you
might
be
also
aware
of
the
transitioning,
but
from
the
entry
cloud
provider
to
the
out
of
three
cloud
provider,
which
is
also
an
undertaking
that
we
are
working
and
being
involved
in
so
and
the
container
storage
interface
si
si
eventually
like
just
g8,
and
so
we
have
now
kind
of
more
flexibility
and
freedom
to
to
implement
that.
But
right
now,
on
your
question,
it's
not
supported
to
run
data
store
clusters
for
storage.
B
A
Last
questions,
otherwise,
if
we
do
run
over,
we
can
take
it
into
the
hallway,
but
I
think
we
have
time
for
one
more
if
anybody's
got
one
yeah:
okay,
yeah,
okay!
Let's
thank
you
for
coming
q.
Very
much,
oh
by
the
way.
Let
me
throw
up
one
other
thing
as
part
of
kubernetes
itself.
We
have
a
bi-weekly
vmware
cig
meeting
and
this
is
a
thing
where
we
have
developers
of
the
cloud
provider:
the
storage
plugins
as
well
as
users,
and
we
have
guest
speakers.
It's
a
zoom
meeting.
A
You
can
join
the
group
and
I
do
bite
you
to
do
that
if
you're
just
catching
up
now.
The
notes
from
the
past
meetings
are
a
rolling
Google
shared
doc
and
the
videos
are
uploaded
to
YouTube.
So
if
you're,
using
kubernetes
on
vSphere
by
all
means
join
the
group,
you
can.
We
also
have
a
channel
on
slack
and
a
mailing
list.
Once
again,
thanks
for
coming
yeah.