►
Description
Kubernetes as a Service in Your Own Data Center - Jared Rosoff, VMware
Google Container Engine is awesome, but how can you get something similar in your own data center? Over the past year, VMware has built an open-source private cloud platform called Photon that delivers kubernetes as a service on your hardware. We’ll discus the architecture of an enterprise grade kubernetes as a service platform and how we’re solving problems like storage, networking, multi-tenancy, security, high availability and resource management.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
A
So
from
from
a
technology
and
product
standpoint,
that's
a
lot
of
our
mission,
but,
but
also
you
know,
part
of
this
group
is
making
a
huge
investment
in
in
open-source
and
so
I
know,
which
either
may
be
thinking
going
into
this
like.
Oh,
this
is
like
proprietary
software,
but
you
know
the
the
vast
majority
and
things
are
going
to
talk
about
today
are
either
open
source
or
free.
Some
of
them
are
commercial
products
and
I'll.
A
Try
to
point
those
out,
but
there's
some
pretty
cool
technologies
that
you
can
go
download
off
github
and
play
with
today.
So
I'll
try
to
make
sure
it's
clear
where
those
things
are.
The
other
thing
is
like
I
deal
well
with
interruption,
so
you
know
feel
free
to
like
raise
your
hand.
Ask
questions
in
the
middle
of
this.
Like
I,
said
it's
more
fun
if
it's
an
interactive
discussion
and
although
you
know,
if
we
start
to
have
too
many
questions,
I
might
ask
to
hold
them
to
the
end.
So
with
that,
let's
get
into
it.
A
So
the
topic
here
is
kubernetes
is
a
service,
and
you
know
so
so
when
we
walk
around,
we
talk
to
organizations
like
we
have
a
kind
of
different
perspective
than
a
lot
of
the
you
know.
Other
talks
here,
you
know
we're
typically
not
talking
at
the
EM,
we're
to
the
to
the
engineering
team,
that's
building
software
applications
and
said
we're
talking
to
like
the
IT
and
Operations
teams
that
have
to
maintain
all
the
hardware,
infrastructure
and
services
that
go
and
support
those
development
teams.
A
And
you
know
we
get
this
question
a
lot
from
IT
administrators
of
you
know
how
do
I
actually
deliver
kubernetes
to
my
development
teams.
I've
got
you
know
a
dozen
teams
inside
my
business
that
are
asking
for
a
technology
like
kubernetes
in
order
to
you
know,
build
their
next
generation
of
applications,
and
you
know
I've
got
my
developers
over
here
asking
for
this.
I've
got
racks
of
servers
and
storage
and
networking
over
here
and
I
have
to
figure
out
like
how
do
I
get
from
a
rack
of
hardware
to
actually
delivering
kubernetes.
A
Now,
if
I'm
in
the
public
cloud,
there's
a
lot
of
great
options
right,
like
you
know,
gke
in
particular,
sort
of
set
a
bar
in
our
mind,
for
you
know
a
great
developer
experience
of
actually
getting
access
to
on-demand
kubernetes
clusters.
You
can
spin
them
up.
You
can
scale
them
up
or
down,
you
can
get
them
on
the
fly
and
you
really
don't
have
to
do
a
lot
of
administration,
but
the
story
is
kind
of
different
if
you're
doing
this
inside
of
your
own
data
center
right.
A
So
you
know
when
we
go
and
talk
to
people,
we
see
a
couple.
Things
like
like.
First
of
all,
there's
no
integrated
solution
for
doing
this,
like
kubernetes
itself,
is
a
great
piece
of
software,
but
it
requires
the
bare
minimum.
You
know
a
you
know:
a
Linux
operating
system
on
a
bunch
of
hosts
a
networking
fabric
and
a
storage
fabric
underpinning
it.
In
order
to
be
able
to
do
this-
and
you
know
so,
we
see
people
try
to
take.
A
You
know
technology
like
OpenStack
and
deploy
like
a
private
cloud
infrastructure
on
top
of
that
and
then
try
to
deploy
kubernetes
on
top
of
that,
and
in
all
of
these
cases
we
find
this
becomes
a
significant
integration
effort.
You
know
there's
an
integration
effort
both
that
the
sort
of
OpenStack
layer
like
getting
an
OpenStack
environment
up
and
running
can
be,
can
be
challenging
in
and
of
itself
and
then,
as
you
go,
and
actually
try
to
run
kubernetes
on
top
of
that
you're.
A
Looking
at
like
okay,
how
am
I
binding,
like
kubernetes
networking
abstractions
into
my
underlying
cloud
platform?
You
know
how
am
I
integrating
persistent
volumes
into
my
underlying
storage
fabric?
You
know
how
am
I
going
to
do
authentication
and
security
and
authorization
into
this
environment.
You
know-
and
these
are
like
pretty
substantial
undertakings.
A
You
know
the
the
patterns
we've
seen
ironically.
Is
that
like,
when
you
look
at
organizations
that
have
actually
been
successful,
doing
this
on
their
own
facilities?
It
typically
involves
one
of
a
couple
things.
It
either
involves
hiring
a
systems
integrator
to
come
in
and
actually
build
all
this
out,
and
that's
often,
like
you
know,
a
year
of
work
of
like
customized,
a
private
cloud
distribution
figuring
out
how
to
bind
kubernetes
or
whatever
container
technology
you're.
Trying
to
run
to
that
you
know
so
that's
expensive
and
time-consuming
the
other
path.
A
We
see
we'll
go
down
is
like
actually
go
higher,
the
committers
to
these
projects
and
I'm,
not
talking
about
just
kubernetes
I'm
talking
about
OpenStack
and
cinder
and
Neutron,
and,
like
all
the
the
underlying
cloud
platforms
and
and
yet
you
know
when
we
do
see
people
that
have
actually
built
this
out
in
their
own
data
center.
It
often
like
fails
to
live
up
to
the
kind
of
experience
that
our
developers
are
used
to
from
the
public
cloud,
so
in
terms
of
like
ease
of
use
and
integration
and
degree
of
automated
delivery
of
our
service.
A
So
you
know
with
that
mine.
We
built
this
product
called
photon
platform
right
so
photon
platform
is,
you
know,
a
enterprise
cloud
infrastructure
platform
that
you
can
run
in
your
own
data
center,
it's
open
source,
so
you
can
go
to
github.
You
can
pull
down.
Photon
controller
and
I'll
talk
more
about
the
components
of
this.
This
solution,
as
we
go
through
the
next
couple
slides,
but
it's
basically
like
a
it,
integrates
a
concepts
of
infrastructure-as-a-service
private
cloud
architecture,
but
it's
also
delivering
things
like
container
platforms
as
a
service
on
top
of
it.
A
So
you
know
walking
through
the
stack
like
photon.
You
know,
first
of
all,
there's
the
thing
you
deploy
on
bare
metal.
So
you
take
a
bunch
of
servers.
You
know
we
run
ESXi,
which
is
VMware
hypervisor.
You
know,
and
obviously
that
is
not
a
piece
of
open-source
software.
However,
you
can
download
a
free
version
of
ESXi
for
website
and
photon
actually
works
fine
with
a
free
ESXi.
We
also
have
integrated
nsx
in
V
Sam.
These
are
not
open-source
and
they're,
not
free
products
either,
but
they're
sort
of
optional
components
in
this
architecture.
A
So
you
can
use
this
with
it
without
those
things,
we've
actually
built
our
own
Linux
distribution,
photon
OS,
which
we
can
use
as
a
guest
inside
of
there,
which
becomes
our
container
runtime
and
our
sort
of
default
Linux
OS,
although,
like
any
virtualization
or
cloud
platform,
we
can
use
any
operating
system.
So
if
you
want
to
use
core
OS
or
you
want
to
run
Windows,
you
want
to
run
whatever
you
can
do
that
instead
photon
controller
then
is
like
the
control
plane
on
top
of
this,
and
the
purpose
of
this
whole
platform
is
to
deliver.
A
You
know:
API
driven
services
that
can
be
consumed
by
development
teams,
so
it's
kind
of
like
the
public
cloud.
Experience
of
you
know
getting
compute
network
storage.
You
know
clusters
on
demand
now,
as
we
look
at
actually
building
this
out,
there's
kind
of
two
sets
of
users
of
this
platform
that
have
very
different
needs.
So
so,
as
an
infrastructure
operator,
you
know
I
actually
own
the
physical
hardware,
the
computers,
the
power,
the
storage,
the
networking
gear
that
actually
makes
up
my
data
center
and
I
typically
have
organized
this
into.
A
You
know
racks
and
pods
and
availability
zones
and
regions.
You
know,
depending
on
how
large-scale
I'm
actually
operating,
and
so
so,
what
we
do
with
with
photon
is,
first
of
all,
we
virtualized
all
these
things,
so
we
virtualized
the
compute
network
and
storage
layers
of
this
physical
hardware
so
that
we
can
programmatically
create
you
know:
virtual
networks,
virtual
storage
and
virtual
compute
nodes.
A
What
we
then
do
is
we
layer
on
top
of
that
underlying
virtualization
capability,
a
distributed,
control,
plane,
and
so
the
control
plane
is
basically,
you
know
a
set
of
software
that
makes
all
of
those
individual
hypervisors
network,
switches
and
storage.
Arrays
appear
to
be
a
single
system,
and
you
know
the
the
control
plane
extends
a
set
of
restful
api.
Is
that
allow
clients
to
actually
consume
this
infrastructure,
and
so,
as
an
infrastructure
operator.
A
You
know
my
job
is
to
actually
operate
this
platform,
so
I'm
running
the
hardware
I'm
managing
the
the
software
bits
running
all
up
on
all
these
bare
metal
machines
and
really
I'm.
You
know,
through
this
platform,
I'm
managing
the
set
of
services
that
you
know
a
development
team
is
actually
able
to
consume
off.
Of
this
infrastructure,
now
the
consumers
of
this
infrastructure
are
the
application
developers.
So
you
know
within
an
enterprise.
You
know
we
have
many
different
engineering
teams
that
are
trying
to
consume
it.
A
You
might
have
you
know
your
your
engineering
team,
your
marketing
team
might
be
running
Paine's,
that's
using
compute
capacity,
you
might
have
a
data
team,
that's
running
analytics
right,
and
so
your
job,
like
as
an
infrastructure,
administrator
you're
both
like
operating
this
hardware,
but
your
job
is
really
like
your
customer.
Are
these
different
groups
inside
the
enterprise,
and
you
got
to
think
about
like
how
do
I
secure
each
of
those
groups.
How
do
I
make
sure
they
can
get
the
resources
they
need,
that
they
don't
overextend
their
consumption
of
the
Reese?
A
The
the
infrastructure
like
how
do
I
limit
you
know
my
Big
Data
team
is
always
like
trampling
all
over
all
my
web
workloads.
So
how
do
I
like
stop
them
from
you
know
using
too
much
capacity
from
the
underlying
architecture,
and
so
so
really
the
the
photon
platform
is
part
of
it
as
a
technology,
but
part
of
it's
also
like
a
mechanism
for
collaboration.
A
It's
a
mechanism
for
enabling
this
sort
of
service
delivery
model
from
my
infrastructure
team
to
my
application
team
and
giving
them
a
great
experience,
but
allowing
the
operations
team
to
manage
consumption
of
that
compute
infrastructure.
So
I
start
with
talking
about
this
control
plane.
So
there's
probably
like
a
whole
other
presentation.
We
could
do
just
on
this
control
plane
technology.
So
what
we
have
built
in
the
underlying
technology
photon
is
the
software
framework
called
Zenon.
So
this
is
open
source.
A
You
can
go
to
github,
slash,
VMware,
slash,
xenon
and
xenon
is
a
decentralized
control,
plane
fabric,
it's
nothing
unique
to
clouds
or
containers
or
kubernetes.
It's.
You
know
it's
sort
of
a
marrying
of
a
database
like
transaction
processing
system
and
an
application
server,
and
what
it
lets
me
do
is
basically
create
like
a
set
of
restful
resource
objects
that
have
like
JSON
documents
behind
them,
so
like
an
example
would
be
like
a
VM
or
a
network
network
segment
or
a
volume.
You
know
each
one
of
these.
A
A
It
uses
an
asynchronous
programming
model
so
so,
like
it's
kind
of
like
a
database
in
that
it
stores
these
JSON
objects.
For
you,
however,
it's
also
an
gramming
model
and
you
actually
write
little
service
controllers
associated
with
every
namespace
in
the
control
plane
fabric
that
allows
you
to
add
logic,
so,
in
other
words,
like
this
system,
like
I,
can
not
only
post
a
JSON
object
representing
a
VM
there's
also
a
program
that
runs
that
handles
inserts
updates
and
deletes
of
VM
objects
and
I
can
compose
these
things
into
larger
systems.
A
So
I
can
create
like
a
JSON
object
that
represents
a
kubernetes
cluster
and
when
I
create
one
of
these,
the
service
handler
for
it
is
gonna
call
back
into
the
control
plane
to
create
VMs
and
networks
and
disks.
You
know
those
things
have
service
controllers
that
handle
those
creation
events
and
talk
to
the
underlying
hypervisors.
So
this
is
in
and
of
itself.
A
This
kind
of
an
interesting
system-
you
know
it's
a
it's
in
many
ways:
its
database,
it
uses
view
stamped
replication,
it's
distributed
so,
as
I
add
more
nodes,
it
actually
partitions
my
data
so
that
you
know
each
node
actually
owns
a
subset
of
the
data.
It's
able
to
do
a
tunable
consistency,
so
we
can
get.
You
know
quorum,
reads
and
writes,
or
you
can
do
optimistic
reads
and
writes.
It
facilitates
things
like
leader
election
and
it
actually
includes
a
leucine
based
full
text
index
over
all
the
data.
A
So
all
of
these
objects
inside
the
control
plane
like
VMs
networks,
disks
and
things
that
we're
managing
you
can
actually
there's
actually
a
pretty
expressive.
Query
language
that
lets
you,
you
know,
write
queries
against
your
data
center
to
say,
like
find
me
all
of
the
network
network
segments
that
were
created
by
you
know
Bob
on
Thursday
that
have
more
than
this
many
IP
addresses
allocates
it
like
you
can
you
can
write
so
pretty
expressive
things
like
that,
you
know
and
it's
actually
packaged
as
an
embeddable
Java
Runtime.
A
So,
unlike
you
know,
it's
in
many
ways
it's
similar
to
something
like
etsy
d.
In
that,
like
you
run
a
bunch
of
them
and
you
store
data
in
it.
However,
instead
of
running
it
as
a
separate
process,
it's
actually
a
thing
that
you
link
in
to
a
Java
program,
and
so
your
binary
becomes
the
actual
store
and
a
node
process
that
that
runs
that
makes
up
this
thing,
like
I,
said:
there's
a
whole
nother
presentation
that
we
could
do
on
Zenon
I'm
not
going
to
do
that
today.
A
So
when
we
actually
go
and
run
this
when
you
look
at
like
what's
inside
this
control,
plane
node.
So,
first
of
all,
you
know
we
are
the
compute
fabric,
so
we
can't
presume
that
there
exists
a
computer
before
we
are
actually
do
our
work.
So
you
know
the
way
this
works.
Is
you
actually
start
off
with
your
hardware?
You
install
ESX
onto
a
bunch
of
machines,
and
then
you
start
starting
these
photon
controller,
guest
processes
or
VMs
on
these
underlying
hypervisors.
A
So
a
standard
production
deployment
would
be
three
nodes
because
it's
a
quorum
consensus
system,
so
you
want
to
be
able
to.
If
you
want
to
be
able
to
survive
a
single
failure,
you
have
to
run
at
least
three,
but
you
could
run
you
know
eleven
or
a
hundred,
depending
on
how
big
your
data
center
is
and
what's
going
on
inside
of
these
control
plane
nodes.
There's
a
bunch
of
micro
services,
backed
by
REST
API,
is
to
do
a
bunch
of
things.
Yeah
question
a.
A
Good
question:
the
question
was
all
the
nodes
run,
control
plant
components
are
just
a
few.
The
answer
is
just
a
few,
so
you
know
like
I
said
standard
standard
deployment
model
would
be
take
a
bunch
of
ESX
hosts.
You
would
run
three
control
plane
guests
on
three
of
the
VMS.
Those
things
network
together
form
your
control
plane.
You
could
run
on
more,
but
best
practice
is
not
all
of
them.
So
inside
of
this
thing,
there's
a
bunch
of
services.
A
So
so,
first
of
all,
there's
this
thing
called
the
API
front-end,
and
this
basically
facilitates
like
you
know,
a
rest
request
can
arrive
at
any
node
in
the
control
plane
and
that
will
handle
forwarding
that
requests
to
the
node
that
actually
owns
the
object.
You're
trying
to
talk
to
there's
things
like
the
the
housekeeper.
This
is
like
a
garbage
collector
for
your
data
center,
so
it
like
detects
things
that
are
not
being
used
by
any
resources
anymore
and
will
freeze
those
resources.
The
deployer
handles
things
like
migrations
and
updates
to
the
control
plane
itself.
A
So
what
you
end
up
with
and
what
would
focus
on
actual
delivers?
Is
this
model
where
I
can
take
this
pool
of
infrastructure
and
I
could
carve
it
up
into
multiple
tenants
and
then
each
of
those
tenants
can
start
creating
resources.
Resources
might
be
virtual
machines.
We
can
create
kubernetes
clusters,
which
are
really
just
aggregates
of
VMs
networks
and
disks.
A
So
with
that,
let's
dig
into
like
what's
actually
involved
in
kubernetes
as
a
service
and
particularly
what
we
see
from
like
an
enterprise
perspective
of
what
goes
on
here.
So
there's
sort
of
a
lot
of
different
problems
to
be
solved
to
deliver
kubernetes
as
a
service.
Some
of
them
are
technical.
Some
of
them
are
not
so,
for
example,
the
first
one
commercial
support
right.
One
of
the
things
we
found
is
that
you
know
whatever
solution.
You're
choosing
you
know
what
our
customers
found
was
that
they
had
to
piece
together.
A
You
know:
storage,
vendor
X
network,
vendor
Y
operating
system
vendor.
You
know,
kubernetes
vendor
all
these
different
things,
so
one
of
the
problems
we
solve
is
actually
giving
a
completely
commercially
supported
stack.
So
you
know
I.
Think
part
of
the
news
here
is
that,
when
you
use
photon
platform
commercially,
we
actually
you
know
at
VMware
will
take
a
call
with
any
problem
related
to
anything
from
the
hypervisor,
the
Linux
kernel,
the
kubernetes
distribution,
the
cloud
control
plane.
You
know
any
of
the
components
involved.
A
Next
thing
is
like:
how
do
you
actually
provide
like
an
on-demand
self-service
interface
right,
so
your
developers
want
to,
like
you
know,
get
a
kubernetes
cluster
on
demand.
You
know
it
needs
to
be
like
an
easy-to-use
interface.
You
can't
like
ask
them
to
like
specify
20,000
lines
or
chef
templates
to
do
that.
Those
things
have
to
spin
up
fast
and
be
responsive
in
order
for
it
to
really
be
useful.
There's
our
solve
problems
of
resource
management
right.
A
How
do
I
carve
up
the
available
CPU,
RAM
and
disk
inside
the
data
center
to
like
allocate
it
to
different
teams?
I
need
to
be
able
to
do
things
like
network
micro
segmentation,
so
you
know
the
idea
that,
like
my
data
center
is
one
gigantic,
you
know
work
address
space
and
that,
like
everything,
can
see,
everything
else
is
not
really
a
great
idea
for
a
lockdown
enterprise,
highly
secure
environment.
So
customers
want
to
be
able
to
do
things
like
carve
up.
A
You
know
at
the
minimum
level
of
like
by
tenants
so
like
each
cluster
could
gets
its
own
isolated
address
space,
but
we've
actually
taken
that
a
bit
further
and
able
to
dive
deeper
into
the
kubernetes
cluster
itself.
In
ordered
segments
networks.
They
also
do
storage
management
right,
so
the
kubernetes
has
persistent
volumes,
but
those
volumes
all
are
represented
by
like
actual
volumes
on
disk
somewhere
that
we
need
to
manage
all
that
state.
You
know,
if
I,
don't
we
have
multiple
tenants
right.
So
how
would
you
actually
administer
like
granting
somebody
access
to
the
system?
A
A
B
A
So
that
half
the
room
so
yeah
so
vSphere
has
this
model
of
like
data
stores.
Data
source
is
the
thing
that
can
like
manufacture
volumes
for
applications
to
use
so
V
San
is
a
thing
that
can
be
a
data
store
in
our
sorry.
V
sanik
can
be
a
datastore
for
vSphere
or
or
for
photon
platform,
in
other
words,
v
san,
basically,
a
software
that
aggregates
all
of
disks
on
all
the
hosts
in
your
photon
cluster
virtualizes
them.
Let's
see
dynamically
create
volumes.
There
are
other
things
that
can
be
data
stores
for
photon
as
well.
A
If
you
have
like
an
NFS
filer
that
can
be
a
data
store.
If
you
have
a
fiber
channel
storage
array,
that
can
be
a
data
store,
so
you
don't
have
to
use,
be
sin,
but
v
santa
is
a
supported
option
so
start
off
with
the
the
basics,
so
so
dynamic
provisioning.
So
you
know,
the
basic
interaction
here
is,
is
you
know,
obviously
relatively
simple
to
imagine
from
a
user
experience,
but
you've
got
some
developer?
Who
wants
to
go,
create
a
new
cluster,
so
we're
presuming?
A
They
already
have
access
to
photon
they've
already
got
the
command-line
tools
installed
an
account
provision,
and
so
basically
the
the
workflow
here
is
the
developer
says:
like
hey
photon,
create
me
a
kubernetes
cluster.
Now
photon
actually
knows
what
a
kubernetes
cluster
looks
like
so
what's
happening
inside
the
control
plane
is
the
control
planes
actually
creating
a
new
cluster
object
and
the
whole
bunch
of
stuff
is
going
on.
A
It's
like
a
big
Plinko
machine
inside
the
control
plane
where
it's
like,
creating
VMs
and
disks
and
CD
nodes
and
API
masters,
and
actually
provisioning
all
the
underlying
virtual
resources
to
run
this
cluster.
You
know
once
you've
actually
created
a
kubernetes
cluster.
The
developer
can
actually
use
coop
cuttle
to
talk
directly
to
the
cluster,
so
we
don't
sit
in
the
path
of
that
cluster
once
it's
created,
we're
just
involved
in
actually
provisioning
it
and
setting
it
up,
but
we
don't
stop
there
right.
A
So
you
know
one
of
the
things
we
see
in
a
lot
of
existing
techniques
for
provisioning
clusters
is
that
the
I,
as
is
kind
of
dumb
like
it,
doesn't
know
that
this
set
of
20
VMs
is
a
kubernetes
cluster.
It
just
knows
that
you
created
20
VMs,
so
photon
works
different,
so
photon
actually
knows
that
there's
a
kubernetes
cluster
and
that
these
are
the
VMS
that
make
it
up,
and
it
also
knows
what
a
healthy
kubernetes
cluster
looks
like.
A
So,
while
kubernetes
can
handle,
like
you
know,
if
a
couplet
node
goes
away,
you
know
kubernetes
gonna
reschedule
all
the
pods
that
were
running
there,
but
there's
nothing
inside
of
kubernetes.
That's
actually
going
to
replace
that
that
node
in
the
kubernetes
cluster
itself-
and
this
is
where
photon
controller
takes
over.
So
what
goes
on
is,
first
of
all,
the
the
control
plane
of
kubernetes,
a
photon
actually
knows
about
this
kubernetes
cluster
and
it's
health
checking
the
underlying
infrastructure.
A
You
know,
first
of
all,
kubernetes
is
going
to
reschedule
the
pods
that
were
there,
but
the
control
plane
of
photon
is
going
to
know
that
that
that
minion
has
gone
away
and
it
actually
automatically
replaces
that
node
and
rejoins
it
to
the
cluster.
So
these
types
of
operations,
like
the
loss
of
any
of
the
VMS
or
networks
or
disks
that
make
up
your
kubernetes
cluster,
like
these,
will
all
be
automatically
corrected
by
the
photon
control
plane
without
in
most
cases
without
any
human
intervention.
A
So
this
this,
like
right
out
of
the
gate
like
eliminates
a
lot
of
the
actual
administrative
load
on
an
Operations
team
for
managing
kubernetes.
Now
we
can
also
use
this
capability
to
do
dynamic
scaling.
So
you
can
imagine
that
your
developer
is
like
using
a
20
node
kubernetes
cluster,
maybe
they're
getting
ready
for
an
ad
campaign
or
something
and
want
to
scale
up
the
double
the
size
of
the
cluster
in
the
capacity.
So
what
they
can
do
is
they
can
actually
ask
the
control
plane
to
resize
the
cluster.
A
So,
where
I
had
like
a
20
node
cluster,
they
may
say
hey
change,
that
to
a
40
node
cluster.
So
what
the
control
plan
can
do,
because
we
know
about
the
existing
kubernetes
cluster,
its
existing
configuration.
We
can
actually
go
spin
up
a
whole
bunch
of
new
kubernetes
nodes
and
then
join
those
nodes
to
the
existing
cluster
right.
So
we
don't
need
to
tear
anything
down
in
order
to
start
up
a
bigger
one.
We
can
do
this
as
a
no
downtime
resize
of
the
cluster.
A
A
So
this
is
the
developer
portal.
So,
when
you,
when
you
log
in
like
when
you
use
photon
as
a
consumption
as
like
a
developer
using
this
thing,
you've
got
a
web
UI,
there's
a
CLI
and
there's
a
REST
API.
Obviously
the
web
UI
is
the
easiest
thing
to
show.
But
what
we're
looking
at
here
is.
We
have
a
tenant
view,
so
this
is
like
the
marketing
teams
view.
You
know
my
view
is
broken
down
into
projects
and
the
projects
are
the
things
that
own,
like
VMs
and
clusters,
etc.
A
A
I
just
have
to
give
it
a
name.
I
tell
it.
I
want
a
kubernetes
cluster.
You
know
we
do
support
other
types
of
clusters.
Besides
kubernetes
you
know,
I
have
to
configure
some
network
settings
like
I
can
assign
different.
You
know
DNS
servers
to
different
kubernetes
clusters.
I
can
give
them
different
address
ranges.
You
know,
I
have
to
give
us
some
information
about,
like
you
know
what
size
should
each
of
the
nodes
be,
should
they
be
16
CPUs
or
a
CPUs?
A
You
know-
and
it's
showing
me
here,
you're
allowing
me
to
choose
the
size
of
my
cluster
and
showing
me
the
impact
on
my
actual
quota,
like
how
much
of
my
allocation
of
datacenter
capacity
now
once
I've
done,
that
the
system
is
actually
now,
it's
often
actually
provisioning
that
cluster.
So
it's
got
to
create
the
VMS.
It's
got
to
launch
the
Kubla
software
on
those
nodes.
It's
got
to
start
the
API
masters
and
things
you
know
here.
I've
got
a
view
of
you
know.
A
I've
clicked
into
one
of
these
clusters
and
I
can
see
all
the
individual
VMs
that
that
make
up
this.
This
particular
kubernetes
cluster.
So
again
the
control
plane
knows
about
you
know
what
the
all
the
components
are
that
make
up
a
cluster.
So
now,
once
I
have
a
running
cluster,
you
know
we're
shipping,
you
know
basically
off-the-shelf
kubernetes
software,
so
we
have
not
like
forked,
kubernetes
and
maeda.
A
You
know
our
own
custom
distribution
of
it
we're
shipping
upstream
kubernetes,
but
we're
also
means
we're
also
shipping,
all
the
same
software
that
you
get
if
you
were
to
download
and
use
kubernetes
itself.
So
you
know
from
this
UI
we
can
actually
click
through
right
into
the
kubernetes
management
user
interface,
while
talking
a
bit
about
how
we
do
single
sign-on.
But
basically,
all
these
things
share
the
same
user
credentials
and
access
privileges.
A
You
know,
and
now
once
I've
set
up
the
cluster,
the
process
of
actually
deploying
a
pod
or
a
service
is
just
done
exactly
the
same
way.
I
would
do
what
if
I
had
set
up
kubernetes
myself,
so
this
guy.
This
thing
there's
like
uploading
yam
will
file
deploy
some
pods
and
again
we
didn't
build
any
of
this
software
which
you're
looking
at
now.
This
is
all
just
like
the
management
UI
that
with
kubernetes.
A
So
you
know
this
kind
of
like
shows
the
the
user
experience
of
actually
consuming
a
kubernetes
cluster,
but
in
a
moment
we're
gonna
flip
over
and
look
at.
You
know
what
an
administrator
sees
that
that's
actually
managing
this
cluster
I
might
skip
my
yeah
he's
resizing
it
okay.
So
if
I,
don't
administrator,
I
get
a
view
more
like
this,
where
I
can
see
all
the
different
tenants
I
can
create
new
tenants.
A
So
maybe
I
want
to
create
an
account
for
my
marketing
team
and
I
can
grant
user
access
to
individuals
now
photon
actually
integrates
with
my
corporate
directory
and
single
single
sign-on.
So
you
can.
When
you
deploy
photon,
you
would
actually
connect
it
to
like
Active
Directory
or
if
you
have
like
an
LDAP
directory
or
sam'l
or
OAuth,
and
we
can
reuse
all
of
that
authentication
and
authorization
logic
inside
of
this
cluster.
So
you
know
when
I
create
a
tenant
like
for
my
marketing
team.
I.
A
A
You
know
what
I'm
doing
as
an
administrator
is
I'm,
basically
managing
these
tenants
and
managing
the
resource
allocation
to
the
different
tenants,
as
well
as
taking
on
basically
the
hardware
lifecycle
management,
so
like
dealing
with
maintaining
the
machines,
doing
patching
and
upgrades
dealing
with
outages
and
stuff
like
that.
Okay.
So
let's
talk
a
bit
about
resource
reservations,
so
the
problem
is
like
how
do
I?
How
do
I
actually
allocate
resources
question
there.
A
Yeah
question
is:
like
cluster
upgrade:
how
do
you
actually
do
that?
So
there
is
not
actually
a
cluster
upgrade
feature
in
photon
right
now.
I
think
that
you
know,
in
our
view,
a
lot
of
that
stuff.
I
think
that
there
are
things
are
being
handled
in
the
lifecycle,
sig
of
kubernetes
itself
of
like
how
to
actually
do
a
kubernetes
live
upgrade,
and
we
we
basically
reuse
that
work
inside
of
this
platform.
A
Yes,
the
question
is
already
stuck
with
whatever
version
of
kubernetes,
so
the
system
can
actually
support
multiple
versions
of
kubernetes
simultaneously,
so
user
one
might
have
kubernetes
one
three
and
user
two
might
have
kubernetes
one
four.
So
when
you're
creating
a
new
cluster,
you
can
actually
create
a
new
version.
You
can
deploy
it
with
a
new
version
of
kubernetes,
so
what
you
can
do
today,
if
you
download
the
software
is
like
you
know.
A
If
kubernetes
1:5
comes
out,
you're
running
one,
for
you
can
create
a
new
one,
five
cluster,
and
then
you
could
basically
do
a
migration
of
your
workloads
out
of
the
old
cluster
into
the
new
one
and
then,
as
the
upgrade
process
of
kubernetes,
gets
more
feature-rich,
we
would
be
able
to
do
an
in-place
upgrade.
Our
target
is
to
be
able
to
do
an
in-place
upgrade,
but
it's
just
not
built
right
now.
A
So
from
a
resource
allocation
standpoint,
right
I've
got
lots
of
different
resources.
That
are,
you
know.
I
need
to
share.
I
need
to
share
CPU,
RAM
storage.
So
you,
this
concept
of
we
called
resource
tickets
and
a
resource
ticket
is
basically
like
an
allocation
of
capacity
to
a
team.
You
know
we
saw
this
in
the
view
earlier,
so
what
I
can
do
is
that
what
you
know
I
can
create
multiple
resource
tickets.
A
I
can
assign
them
to
a
team,
and
it
basically
is
just
like
specifies
like
how
many
CPUs,
how
much
memory
you
know
how
much
storage
is
this
tenant
allowed
to
use?
Those
things
are
all
additive,
so
I
can
create
a
new
resource
ticket
and
grant
it
to
a
team
to
give
them
more
capacity,
and
that
basically
sets
up
a
quota
of
how
much
aggregate,
CPU,
RAM
and
storage
capacity
team
can
use,
and
that's
shared
across
like
whether
I'm
using
kubernetes
or
just
straight
VMs
or
running
Cloud
Foundry.
A
A
So
today,
if
you
pull
down
photon
and
deploy
it,
what
you're
actually
getting
is
the
flannel
overlay
Network,
which
is
the
same
thing
you
get
if
you
just
used
coop,
ADM
or
whatever
mechanism
to
deploy
it,
but
our
nsx
team
has
been
doing
a
lot
of
very
cool
integrations
with
kubernetes,
which
will
be
available
very
soon.
I'll
talk
about
some
of
those
things.
So,
first
of
all
the
you
know,
I
assume
you
guys,
then
we're
at
coop
cons,
so
the
kubernetes
flannel
overlay.
This
isn't
news.
We
didn't
do
anything
new
here.
A
This
is
the
off-the-shelf
software,
but
what
we're
doing
what
it's
kind
of
an
eye
chart?
So
what
we've
done
is
we
integrated
NSX
with
kubernetes
so
with
NSX?
You
know
there's
couple
of
things
to
understand.
So,
first
of
all,
you
know
our
view
of
the
infrastructure.
Is
you
know
we
look
at
the
hypervisor
itself
and
then
we
have
to
look
at
the
guest
running
inside
that
hypervisor
and
then
we
look
at
the
pods
running
inside
of
those
guests
and
so
with
the
NSX
networking
model
for
kubernetes
pods.
You
basically
have
a
you
know.
A
We
do
an
IP
address
per
pod
and
there's
a
OBS
open,
V
switch
running
in
each
minion
VM,
and
so
basically,
what
happens
is
that
that
OBS
running
in
the
minion
handles
all
the
actual
pod
traffic
from
from
that
minion
and
handles?
Basically,
you
know
relaying
that
to
another
OBS
switch
or
switch
that's
running
in
the
hypervisor
itself,
so
the
hypervisor
basically
is
like
aggregating.
You
know
the
traffic
from
all
the
minions.
The
minions
are
aggregating
the
traffic
from
all
the
pods,
but
all
this
acts
as
like
an
integrated
network
fabric.
A
So
we're
able
to
do
you
know
a
you
know:
container
interfaces
per
per
prod
and
per
container
inside
of
that
we
can
do
distributed
firewall.
That
runs
essentially
inside
the
hypervisor,
and
this
basically
allows
us
to
implement
firewall
rules
and
security
rules
that
execute
in
a
distributed
fabric
across
all
the
hosts.
So
you
don't
have
to
like.
If
you
want
to
do
firewalls,
you
know
say
they
protect
like
to
pause
from
each
other.
A
You
don't
have
to
like,
send
all
that
traffic
up
to
some
central
firewall
to
get
filtered
and
then
back
out
to
the
other
pod.
Instead,
those
firewall
rules
and
those
security
rules
get
pushed
down
out
to
the
edges
in
all
the
individual
hosts
and
in
essence,
every
container
and
every
pod
has
like
a
little
virtual
firewall,
that's
sitting
in
front
of
it,
that's
managing
all
the
trash
back
in
and
out
of
it
running
short
on
time.
So
I'm
gonna
skip
over
some
of
the
details
here,
but
this
isn't
very
cool
stuff.
A
The
second
thing
is
that
you
know
so:
NSX
actually
manages
like
the
networking
layer
both
for
pods
and
and
VMs,
but
it,
but
it
also
has
done
a
deep
level
integration
with
kubernetes
itself,
so
within
kubernetes,
like
the
default
networking
model
is
flat
right,
so
the
entire
kubernetes
cluster,
every
node
needs
to
be
able
to
talk
to
each
other.
Every
pod
needs
to
be
able
to
talk
to
each
other,
but
for
a
highly
secure
environment.
This
often
isn't
good
enough.
A
So
what
we've
done
is
we've
actually
integrated
NSX
control
plane
with
the
kubernetes
control
plane
in
order
to
synchronize
the
network
segmentation
with
the
underlying
kubernetes
configuration.
So
what
the
way
this
works
right
now
is.
We
actually
basically
watch
the
set
of
namespaces
that
have
been
created
inside
the
kubernetes
cluster
and
create
network
segments
inside
of
nsx
for
each
namespace.
So
every
namespace
in
kubernetes
translates
to
a
separate
isolated
network
at
the
virtual
networking
layer,
and
this
means
that
you
know
first,
all
two
namespaces
like
don't
even
have
address
ability
to
each
others.
A
A
Looking
at
the
guy
who
wrote
all
this
to
make
sure
he's
nodding
at
me,
saying:
oh,
that's
right,
he's
nodding
so
I'm
good,
so
so
this
very
cool,
basically
we
were
able
to,
like
you
know,
do
very
tight
security
on
all
the
traffic
between
pods
inside
this
cluster
and
give
from
an
administrative
view.
You
can
create
corporate
wide
policies
like
these
people
are
allowed
to
talk
to
this
database.
A
Everyone
else
is
not
allowed
to
or
these
systems
are
allowed
to
talk
to
each
other.
These
are
not.
You
can
actually
centrally
administer
that
and
you
can
administer
it
not
only
within
the
kubernetes
cluster,
but
also
across
kubernetes
clusters
and
between
kubernetes
clusters
and
non
kubernetes
infrastructure.
Writing
on
photon
yeah
question.
B
So
I
was
just
saying
that
it's
not
the
implementation
of
Calico,
the
concept
of
micro
segmentation
predates
calico
I
mean
it's
existed
a
long
time
before
that
the
idea
is
very
simple.
Wherever
your
workload
goes,
your
firewall
policy
follows
it.
It
makes
it
very
easy
to
manage
your
firewall
policies
so
so
NSX
had
this
technology
for
a
long
time,
we've
been
doing
it
for
virtual
machines.
We've
just
extended
it
to
contain
us
to
the
added
difference
between
other
solutions
is
that
we
don't
try
to
apply
the
firewall
policies
just
below
the
part
inside
the
VMS.
B
We
don't
think
it
is
secure.
If
you
assume
containers
are
not
secure,
then
your
firewall
policies
just
below
a
pod
is
not
going
to
be
secure.
So
we
keep
all
the
policies
inside
the
hypervisor
so
that,
even
if
there's
a
container
break
out
your
firewall
policies,
you
know
they
des
cannot
touch
it.
So
that's
the
point
so.
A
The
the
next
area-
thatthat
I'm,
gonna,
talk
about
a
storage
management,
so
this
stuff
is
is
coming
soon
to
some
degree
in
photon,
but
I'm
gonna
talk
about
the
bulk
of
the
work
here
actually
happens
in
the
UPS.
Excuse
me
in
the
upstream
kubernetes
repo,
specifically
for
vSphere,
but
so
basically
you
know
when
we,
when
we
talk
about
storage
for
kubernetes,
you've
got
these
persistent
volumes
and
persistent
volume
claims,
and
you
know
your
underlying
infrastructure
has
to
provide
something
that
provides
that
persistent
volume
capability.
A
So
what
our
storage
team
has
done
has
actually
done
an
integration
of
basically
vSphere
storage
primitives
into
kubernetes.
You
can
actually
see
this
in
the
kubernetes
project.
There's
a
getting
started
guide
for
how
to
run
kubernetes
on
vSphere.
The
codebase
is
actually
in
the
cloud
provider
interface
in
kubernetes
itself,
so
the
bulk
of
the
work
is
actually
in
there.
A
The
thing
that
changes
when
we
moved
to
photon
is
they're
slightly
different
api's
for
photon
compared
with
the
vSphere,
but
but
but
most
of
the
work
has
actually
been
done,
but
basically,
what
this
allows
us
to
do
is
we
can
use
the
underlying
storage
fabric
of
a
visa
nor
any
datastore
to
create
volumes.
Those
volumes
become
persistent
volumes
as
far
as
kubernetes
is
concerned,
so
I
can
I.
A
Can
you
know,
as
a
storage
administrator
I'm
using
V
Sam,
to
manage
all
these
volumes
in
the
pool
of
storage
that
I've
got
and
then
I'm
exposing
that
to
the
developers
of
kubernetes?
So
you
can
create
different
flavors
of
persistent
volumes.
So
when
we
say
flavors,
we
have
a
feature
inside
of
our
software
called
storage
policy
based
management.
So
a
storage
policy
allows
me
to
define
like
data
protection
rules,
quality
of
service
rules.
I
can
define
whether
or
not
I
want
this
data
duplicated
or
not.
A
I
could
define
whether
I
wanted
to
encrypt
it
or
not,
and
so
I
can
create
storage
flavors,
an
administrator,
so
I
can
say
like
this
is
an
encrypted
highly
available.
You
know
raid
5
array
with
this.
Like
Quast
setting
on
the
storage
fabric
and
I'm,
going
to
call
this
like
disc
flavor,
high
performance
right
and
then
as
a
developer,
I
can
create
a
persistent
volume
of
flavor
high
performance
and
I'm
gonna.
Get
that
storage
policy
applied
to
that
volume.
A
You
know
that
you
know
manifests
itself
to
the
developers
just
different
types
of
persistent
volumes
they
can
create,
but
it
manifests
itself
to
the
operator,
as
essentially
the
management
of
the
storage
array
of
you
know,
servicing
the
actual
underlying
disks
that
may
fail
over
the
course
of
the
day
of
expanding
capacity
and
looking
at
utilization
and
things
like
this
again.
This
stuff
is
coming
soon
to
photon,
but
you
can
get
a
feel
for
what
this
whole
system
looks
like
by
playing
with
the
vSphere
integration
with
kubernetes,
that's
in
Coober
days
today.
A
So
the
next
piece
is
on
multi-tenancy
right.
So
you
know:
there's
a
underlying
identity
model
inside
a
photon
where
we
create
tenants,
tenants
can
create
multiple
projects.
Projects
can
contain
multiple
kubernetes
clusters.
Project
can
also
contain
things
like
virtual
machines
and
Cloud
Foundry
clusters
and
other
things
that
aren't
kubernetes.
A
Virtual
machines
they're
also
isolated
at
the
network
layer,
so,
like
my
kubernetes
cluster,
your
kubernetes
cluster
different
networks
and
we
can
extend
like
even
within
the
namespaces
in
those
clusters
or
different
networks,
but
we
also
have
this
centralized
Identity
Management
model
inside
of
here,
and
so
the
way
this
works
is
that
that
there
is
actually
an
identity
directory
inside
of
boats
on
platform
which
handles
all
of
the
you
know,
authentication
authorization
and
access
control
across
the
cluster.
So
this
is
based
on
a
source
project
called
light.
A
Wave
light
wave
is
is
basically
at
its
core
and
implementation
of
an
identity
directory
using
LDAP.
But
it's
also,
you
know
an
OAuth
server.
It's
a
sam'l
server,
it's
a
certificate
authority
and
implements
a
whole
bunch
of
different
protocols
for
sharing
that
Identity
and
Access
Management
data
across
different
things.
A
So
when
I
create
tenants,
I
can
actually
bind
tenants
to
different
trees
in
my
corporate
identity
directory
and
I
can
actually
bind
different
tenants
to
different
directories,
so
I
might
actually
have
completely
different
identity
directories
for
different
tenants
and
I
can
manage
all
of
that
centrally
through
this
system.
Now,
what's
interesting
is
is
that
you
know
this
means
that
I
can
basically
use
my
corporate
credentials
to
log
into
photon,
but
it
also
means
that
we
can
actually
integrate
the
authentication
authorization
provider
of
kubernetes
with
this
identity
directory
as
well.
A
So,
for
example,
there
is
an
OAuth
provider
inside
Lightwave,
and
what
we're
working
on
right
now
is
we're
actually
activating
the
OAuth
authorization
plug-in
and
inside
of
kubernetes,
so
that
when
I
create
a
kubernetes
cluster,
it
basically
inherits
the
authentication
and
authorization
rules
of
the
project
in
which
it
was
created.
So
if
I
have
a
project
and
I
want
to
give
five
people
access
to
it,
I
give
them
access
to
the
project.
I
create
the
kubernetes
cluster,
and
anybody
that
has
access
to
the
project
will
be
able
to
perform
API
commands
against
the
kubernetes
cluster.
A
If
I
remove
you
from
the
project,
not
only
can
you
no
longer
talk
two-photon,
but
you
also
can't
talk
to
the
kubernetes
cluster
anymore
either.
I
can
remember
putting
this
in
context.
You
know
our
goal
is
to
make
sure
that,
like
for
the
developers
up
at
the
top,
you
know
that
if
I'm
a
member
of
like
tenant
B,
you
know
I'm
able
to
API
commands
against
those
two
kubernetes
clusters
but
I'm
unable
to
talk
to
any
of
the
other
tenants
working
on
that
same
platform.
A
So
you
know
just
in
the
nick
of
time
the
end
so
I
said
you
know.
A
lot
of
these
things
are
open
source.
A
lot
of
the
technologies
we
talked
about
today
are
up
on
github.
You
know,
photon
controllers,
the
underline
control,
plane,
totally
open
source.
You
can
go
to
github,
VMware
photon
controller
photon
OS
is
our
Linux
distro.
This
is
what
we're
running
as
the
node
and
the
underlying
Linux
OS,
and
all
these
kubernetes
knows
that
we're
running
you
know,
that's
also
up
on
github,
you
know,
Lightwave
is
the
identity
and
directory
infrastructure.
A
That's
also
up
on
github.
You
can
pull
it
down.
Zenon
is
the
underlying
framework.
That's
the
control
plane
that
we
built
photon
platform
on
top
of
that's
also
open
source
and
up
on
github,
there's
more
things
up
there,
but
that
that's
all
I
got
today.
I
think
the
the
the
key
thing
is
I
hope
was
kind
of
like
an
interesting
journey
of
looking
at
you
know.
How
do
you
do
you
know
a
shared
infrastructure
pool
that
can
support
many
kubernetes
cluster
simultaneously?
A
A
So
question:
do
you
lose
anything
by
switching
out
the
OS?
No,
not
really
I
mean
the
you
know,
mihai's
a
lot
like
like
how
you'd
expect
like
on
like
ec2
right,
where
you've
got
different
a.m.
eyes.
You
can
launch
in
a
bun.
Am
I
or
Arella
am
I
you
know
I
think
when
we
launched
a
kubernetes
cluster,
the
default
image
that
we
use
is
the
photon
OS
image
and
a
lot
of
that.
A
lot
of
that
was
just
like.
A
We
had
to
pick
something,
and
this
was
an
option
that
was
you
know,
you
know,
could
be
included
in
the
platform
and
could
be
commercially
supported
by
VMware
without
having
to
include
you
know
any
third-party
relationships,
but
you
know
we
have
customers
that,
like
run
rail
on
top
of
it,
that
run
core
OS.
On
top
of
it
all
those
things
work
great.