►
From YouTube: Storage Provisioning for Kubernetes on Windows - Anusha Ragunathan & Jean Rouge, Docker
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Storage Provisioning for Kubernetes on Windows - Anusha Ragunathan & Jean Rouge, Docker
As support for Windows container workloads on Kubernetes heads to GA, we need robust storage interfaces for the many Windows apps that need to manage state. In this session, we will:
1. Start with the foundational constructs around persistent storage in Kubernetes and go over existing mechanisms in-tree that support stateful Windows workloads .
2. Dive into enhancements necessary in existing external storage provisioners to support dynamic volume provisioning for Windows workloads and demonstrate the enhancements through end2end examples with the open source targetd iscsi target as a reference for what is required in plugins for storage backends.
3. Wrap up with what the future of this space looks like in the context of the modern Container Storage Interface, delve into platform changes in Windows necessary to support CSI plugins and short-term workarounds to overcome this.
https://sched.co/MPfV
A
B
A
A
The
agenda
for
today
we're
gonna
start
with
talking
about
persistent
storage
in
kubernetes.
This
is
going
to
be
an
OS
agnostic
session
section
we'll
talk
about
how
kubernetes
handles
personal
storage
in
general.
After
that,
we
will
move
into
the
current
state
of
state
in
Windows.
In
this
section
we
will
talk
about
the
different
storage
interfaces
that
work
with
Windows
today,
and
we
will
also
have
a
couple
of
live
demos
where
a
stateful
applications
such
as
sequel
server,
can
be
used
to
provision
volumes
can
be
used
with
persistent
volumes
in
the
backend.
A
A
First,
let's
talk
about
persistent
storage
in
kubernetes.
You
might
have
seen
this
architecture
diagram
before,
but
just
to
recap,
on
the
top,
you
see
the
kubernetes
master
right
now.
This
is
a
linux
nerd,
the
kubernetes
master
manages
the
workers.
The
workers
can
be
a
mix
of
both
windows
workers
and
Linux
workers.
The
Windows
workers
need
to
be
2019
Windows
Server
2019.
At
this
point,
I'd
like
to
point
out
that
kubernetes
version
114
provided
GA
support
for
Windows
workers.
A
Now
at
bottom
of
the
diagram,
you
see
the
network
and
this
storage
fabric
that
are
required
for
the
cluster.
This
can
be
both
on
print
or
on
the
cloud.
Looking
at
the
master
components,
you
have
the
kubernetes
scheduler
the
API
server,
as
well
as
a
bunch
of
controllers
that
work
together.
We'll
talk
about
the
controllers
in
a
bit
and
the
bottom,
you
actually
see
the
components
that
are
part
of
a
worker
node.
This
is
typically
the
crew
blade,
the
coop
proxy
and
the
different
plugins,
the
C
and
ICSI
and
CRI
for
Network
runtime
and
storage.
A
A
Looking
at
the
components
that
are
specific
to
storage,
there
are
controllers
that
are
running
as
part
of
the
master
that
work
together
to
provide
persistent
storage
for
kubernetes.
A
controller
in
kubernetes
is
basically
a
software
loop
that
looks
at
the
current
state
of
the
cluster
and
tries
to
convert
that
with
the
desired
state
of
the
cluster.
A
Moving
on
to
our
specific
logic
in
the
work
we
have,
the
couplet
the
couplet
is,
does
a
lot
of
the
heavy
lifting
around
mounting
and
unmounting
volumes
on
worker
nods,
and
it
has
a
lot
of
OS,
specific
logic.
Container
runtimes
like
container
D
or
dr.
D,
that
has
a
lot
of
specific
logic
as
well,
and
finally,
flex
volume
and
CSI
plugins
have
node
specific
attached
attached
operations
as
well
as
mountain
mount
operations.
A
Now,
let's
look
at
some
storage
concepts
in
kubernetes
before
we
go
further.
These
are
basically
like
building
blocks
that
can
that
kubernetes
overall
uses
to
provide
persistent
storage
pause,
as
we
all
know,
are
basically
a
set
of
running
containers
that
represent
your
workload.
A
persistent
volume
claim
or
PVC
is
basically
a
request
from
the
user
to
access
a
piece
of
storage
in
your
cluster
and
just
like
how
fast
consume
CPU
and
memory
PVCs,
consume
storage
and
just
like
how
pods
can
specify
number
of
CPUs
and
amount
of
memory.
A
A
This
is
a
very
generic
diagram
of
how
volume
can
the
volumes
lifecycle
you
have
the
storage
service
on
top
and
the
part
consuming
it
in
the
bottom,
so
the
volume
gets
created.
It
gets
attached
to
a
specific
node
when
the
plop
part
gets
scheduled
on
a
specific
node.
Once
it
gets
attached
to
a
specific
node.
The
part
actually
will
be
able
to
mount
that
volume
and
start
using
it
one
when
the
part
is
done
with
it.
It
can
actually
the
the
volume
gets
unmounted
when
the
pod
gets
reschedule
to
another
node.
A
Let's
look
into
pv
provisioning.
The
provisioning
is
basically
fancy
word
for
creating
or
allocating
volumes,
and
there
are
two
types
of
provisioning
that
are
supported
in
cribben.
It
is
static
and
dynamic.
Static
is
basically
creation
of
these
precision
volumes
upfront,
you
have
a
pool
of
persistent
volumes
and
then,
when
the
user
makes
a
claim,
you
Matt
the
controller
sort
of
matches
the
claim
with
the
existing
PDS
that
are
available
and
the
PV
and
PVC
are
bound
from
then
on
and
then
the
part
can
basically
come
in
and
use
it
mounted
and
so
on.
A
The
other
way
of
provisioning
is
dynamic.
Provisioning
in
this,
it's
basically
a
on-demand
provisioning
of
Peavey's.
So
when
the
user
submits
a
claim,
is
when
the
TVs
get
provision
in
the
backend.
This
is
definitely
the
recommended
way,
because
it's
on-demand,
you
don't
have
to
create
it
ahead
of
time,
and
you
also
know
what
the
users
specifications
are.
You
don't
have
to
sort
of
guess
on
your
capacity
or
the
request
that
is
made
now
dynamic
provisioning
is
done
through
a
concept
of
storage
classes.
A
It's
basically,
our
storage
passes
are
nothing
but
tiers
of
storage
that
are
available
in
your
cluster,
so
storage
admin
comes
in
and
creates
different,
tiers,
typically
based
on
performance,
some
SLA
requirements,
and
so
on,
and
the
storage
classes
are
backed
by
always
backed
by
a
provisioner.
What
is
a
provisioner
provision
is
nothing
but
a
piece
of
software.
That's
constantly
listening
on
incoming
persistent
volume
claims,
and
if
there
is
a
match
to
itself,
then
it
basically
creates
the
position
volumes
in
the
backend
the
appropriately.
A
Let's
look
at
the
state
of
State
in
Windows,
which
is
essentially
what
the
stock
is
all
about
with
that
in
the
background,
so
to
understand
that
let's
look
at
what
is
what
are
the
different
storage
interfaces
available
today?
In
kubernetes
we
have
entry
plugins,
which
are
basically
plugins
that
are
available
as
part
of
kubernetes
tree
and
run
in
process
with
kubernetes
common
core
components.
A
This
is
basically
a
host
based,
exact
model,
a
approach
for
flex
volumes,
external
provisioners
are
live
out
of
tree
and
out
of
process
from
kubernetes
and
they
take
care
of
the
provisioning
aspect
of
it.
So
the
create
delete
of
the
volumes
is
handled
by
external
provisioners
and
finally,
CSI
is
again
out
of
tree
and
out
of
process
from
kubernetes,
and
it
managed
that
manages
the
entire
lifecycle
of
a
volume
search
create
delete
a
a
mount
and
mount,
and
there
are
more
customizations
that
are
coming
in
for
CSI
now,
quick
pro
and
pro's
and
con's
table.
A
The
advantage
of
entry
is
basically
you
don't
have
to
do
any
extra
installation.
The
disadvantage
is
that
it's
all
the
entry
plug-in
work
is
tied
to
kubernetes
releases.
So
if
you
have
a
bug
or
a
patch
release
that
you
want
to
actually
make
to
your
entry
plugins,
you
have
to
wait
until
the
next
Jeff
Kober
Nettie's
is
out.
The
future
is
superseded
by
CSI
external
provisioners.
The
advantage
is
that
you
can
highly
customize
the
provisioning
aspect
of
it,
but
the
disadvantage
is
that
it's
limited
customization
for
the
attachment
amount
operations,
flex
volume.
A
You
can
actually
customize
the
code
for
attach
and
mount
operations,
but
you
cannot
customize
it
much
for
provisioning
and
also
in
Flex
Walliams.
The
idea
is,
there
is
expectation
there
are
certain
assumptions
that
are
being
made
as
far
as
scripts
or
binaries
to
be
existent
in
your
nodes,
at
a
certain
location
and
so
on
and
so
forth.
So
the
portability
of
this
model
is
very
limited
there.
So
for
that
specific
reason,
flex
volume
is
actually
deprecated
in
window
in
Linux.
A
It's
still
used
in
Windows,
because
there
is
no
better
alternative
for
Windows
and
we
will
see
why,
in
a
bit
in
the
later
half
and
finally
CSI
the
advantage
is
that
it's
highly
customizable,
almost
all
of
the
operations
can
be
customized
and
it's
an
industry
standard,
so
CSI
can
be
implemented
on
in
other
container
orchestrators.
Like
me,
source
and
cloud
foundry
and
the
same
storage
plugin
can
be
used
across
all
container
orchestrators.
A
If
they
are
there
to
the
spec,
so
it's
an
industry
standard
which
is
which
is
a
great
thing,
but
the
disadvantage
is,
of
course,
there's
the
minor
disadvantage,
which
is
you
gotta,
install
kubernetes,
and
on
top
of
that,
you've
got
to
maintain,
install,
maintain
and
configure
your
CSI
plugins
on
top
of
it.
But
this
is
definitely
the
future.
A
Now,
what
are
some
of
the
considerations
specific
for
Windows?
As
far
as
writing?
A
plugin
verse
now
disk
the
device
enumeration
and
volumes
enumeration
interfaces
on
Windows
are
very
different
from
that
of
Linux
in
Linux,
all
the
devices
and
volumes
end
up
as
a
file
on
your
file
system,
but
on
Windows
they
are
all
object
based,
so
you
have
to
make
win32
calls
to
get.
The
enumeration
done
in
is
not
like
you
can
actually
files
off
of
your
file
system
so
as
I
fear
a
plugin
author,
you
have
to
make
that
take
that
into
account.
A
A
We
don't
have
support
for
NFS
and
it's
something
that
has
been
talked
about
for
the
future,
but
for
right
now
we
have
Samba
support
and
windows
also
ends
up,
have
having
to
heavily
depend
on
powershell
command,
let's
and
scripts
for
many
of
the
actions
that
need
to
be
customized
now,
what's
available
today
for
in
pre
storage
for
Windows,
so
Kubla
dot,
exe
on
windows
basically
supports
partitioning
and
formatting
using
NTFS.
It
also
provides
support
for
linking
host
volumes
into
the
container,
so
this
is
similar
to
there.
A
There
is
a
tool
called
mate
link
that
is
actually
being
used
for
this
specific.
Linking
this
is
similar
to
bind
mounting
on
Linux,
where
you
can
bind
want
something
on
the
from
the
local
host
onto
your
container
file
system.
Linking
samba
shares
to
the
containers
filesystem
is
also
very,
is
also
supported.
So
with
these
features,
it's
actually
possible
to
provide
some
pretty
good
entry
support.
A
So
today
we
actually
have
file
based
cloud
volumes
on
as
your
file
through
samba.
In
fact,
our
demo
is
going
to
be
that
block
based
Cloud
volumes.
Support
is
also
available
on
both
azure
disk
and
GCE
PD
on
AWS
EBS.
It's
a
work
in
progress
and
for
ice
cozy
entry
support
is
actually
work
in
progress.
It's
targeted
for
kubernetes
116.
A
A
quick
diagram
about
the
entry
plug-in
architecture,
so
here
we
see
that
the
PV
claim
actually
comes
into
the
API
server
and
the
it
gets
dispatched
from
the
controller
to
the
different
entry
plugins
that
are
available.
So
basically,
there
is
a
inspect
of
the
coming
incoming
claim
and
the
correct
entry
plug-in
is
basically
looked
up
and
dispatched
to
four
create
delete
and
attach
detach
volumes.
Those
components
reside
in
the
master
four
mount
and
unmount
the
public
takes
care
of
it.
A
A
Now
we
will
look
at
the
entry
storage
with
as
your
file
demo
here.
I
have
a
three
node
cluster
with
one
master
and
two
workers.
Both
workers
are
Windows
2019
hi.
This
is
on
as
your
IKS
service
I
have
I'm
using
a
KS
engine
and
applying
kubernetes
114,
and
we
the
part
that
I
am
going
to
be
using
for
I'm,
basically
going
to
dynamically
provisioned
volumes
and
the
part
I
am
going
to
be
using
to
show
that
it
can
actually
mount.
Those
pd's
is
a
sequel,
server.
Pod.
A
A
A
A
A
A
A
A
B
All
right
so
initial
shot
us
how
entry
plugins
work-
and
she
mentioned
also
that
our
external
provisioners
exists.
So
it's
not
a
prisoner's.
Basically,
of
course,
cute
containers
didn't
want
to
have
every
every
single
storage
plugin
that
anyone
is
ever
gonna
write
into
the
cube.
Codebase
that
would
be
bloated,
reviews
would
be
horrible,
and
so
they
come.
They
came
up
with
a
first
idea
to
try
and
make
it
possible
to
write
out
of
tree
storage
plugins,
and
so
that
was
external
provisioners,
so
baby.
B
Actually
does
that
so
the
entry
plug-in
at
this
point
delegates
created
and
delete
operations
to
the
external
provisioners
and
the
next
presenter
then
returns
a
PD
object,
and
that
goes
back
to
the
IP
I
master.
The
iPad
master
takes
care
of
the
attach
and
detach
operations,
so
the
PV
object
has
to
be
one
of
the
entry
ones
and
then
the
amount
in
unmount
volumes
can
be
delegated
to
a
flex.
B
So
I
have
a
very
simple
cube
cluster.
Thank
you.
A
very
simple
cube
cluster,
with
a
gesture
master
and
I'm.
A
Windows
worker
and
ahead
of
time,
I
deployed
an
external
perimeter.
So
if
we
go
look
at
that,
IQ
get
all
that's
the
solution.
It's
running
I
made
the
manifest
files
available
in
the
hub.
I'll
give
you
the
link
register.
If
you
want
to
go
look
at
it,
but
the
interesting
part
is
the
Suraj
class.
B
So
that's
a
really
simple
pot.
That's
running
Microsoft
server
core
does
nothing
just
waits
and
it
has
one
PVC
and
that
PVC
references,
a
PVC
that
ended
that
didn't
define
right
above.
If
I
can
get
my
SSH
to
work
there,
we
go.
That's
yeah
PVC
that
got
defined
before,
and
the
important
part
here
is
the
storage
class
that
we
define
again.
That's
gonna,
be,
in
my
view,
tab.
So
you
can
go
see
the
exact
details
and
if
we
did
play
this.
B
B
If
we
go
look
at
the
mounts
here,
we
can
see
that
we
mounted
this
nice
cozy
volume
onto
the
pod.
Thank
you
very
much,
so
this
shows
that
you
can
do
today
dynamic
provisioning,
a
nice
cozy
on
Windows,
using
a
slightly
modified
version
of
the
target,
D
provisioner
that
is
out
there
and
using
flexible
in
plugins.
If
you're
interested
in
the
exact
detail
of
demo,
sorry
I
went
a
little
fast,
but
this
github
repo
as
write
up.
That
really
explains
in
details.
How
you
can
do
the
same.
B
So
entry
plugins
that
was
pretty
much
what
started
story,
storage
and
cube.
Then
there
was
external
provisioners,
but
you
still
don't
have
that
much.
Flexibility
you're
still
restricted
to
exacta,
but
random,
binaries
being
in
your
workers
being
where
you
expect
them
and
having
weird
contract
to
fulfill.
So
that
was
not
really
satisfying,
and
so
the
Coronas
community
came
up
with
a
CSI
which
is
a
standard
that
defines
how
plugin
should
behave
in
the
future.
B
But
that's
really
just
as
I
support
for
now.
So
what
a
CSI
plugin
really
needs
to
do.
It
needs
to
be
able
to
register
itself
with
the
API
and
the
cube
layer,
the
worker
notes
in
me
and
then
on
the
worker
nodes.
It
needs
to
be
able
to
make
these
calls
to
the
workers
operating
system.
It
needs
to
be
able
to
mount
volumes
on
that
host
that
are
accessible
to
other
processes
in
the
same
house.
B
Typically,
the
processes
are
going
to
be
running
and
the
PO
that
you
care
about,
and
if
you
look
at
the
seat,
I
plugin
that
exists
today,
they
they
all
continued
rise,
because
you
just
want
cube
to
be
able
to
run
your
stuff,
so
they
all
continue
eyes
and
now
I'll
pretty
much
do
this
operation
the
same
way.
So
the
way
the
register
is
using
a
sidecar
that
Kubek
Ramirez
provides,
and
so
they
register
your
unique
socket
talking
to
a
sidecar
makes
its
call
to
the
US.
B
They
need
to
run
in
a
privileged
mode
and
to
be
able
to
mount
volumes
on
the
hosts
that
are
going
to
be
accessible
to
other
parts.
They
need
to.
They
use
one
propagation
on
docker
and
the
problem
with
Windows
is
that
you
don't
have
privileged
support
and
you
don't
have
mana
propagation.
So
you
can't
really
write
CSI
plugins
the
right
way
on
Windows
today,
and
that's
why
I
say.
B
Support
for
Windows
is
really
very
much
a
work
in
progress
and
Microsoft
tells
us
that
privileged
privileged
well
is
not
gonna,
be
supporting
Windows
before
2020
at
best
and
we're
still
trying
to
get
them
to
implement
Mon
propagation.
So
in
the
meantime
we
have
to
come
up
with
a
solution
for
that,
and
basically
the
two
main
ideas
we
have
today
is
either
saying.
Okay,
like
you,
can't
write
suicide
plug
in
the
windows
in
a
container.
It's
just
not
possible.
B
B
So
you
need
to
package,
it
distributes
elf,
and
the
lifecycle
of
the
process
is
going
to
be
in
different
from
cube
too,
so
you
can
have
to
somehow
make
sure
it
restarts
if
it
crashes,
you're
gonna,
have
to
in
all
the
logs
yourself.
That
doesn't
seem
like
a
great
solution,
so
the
second
option
is
to
somehow
still
run
containers.
So
that's
cool
because
I
mean
you
have
cube
there.
After
all,
so
that's
a
good
way
to
make
sure
everything
runs
and
to
deploy
things.
You
don't
have
to
reinvent
the
wheel
for
that.
B
B
That
comes
with
a
number
of
problems,
so
the
first
one
would
be
that,
of
course,
this
proxy
would
have
an
api
that
we
need
to
version
and
maintain
backward
compatibility
across
versions,
and
that
sounds
painful
and
there's
also
security
concerns.
You
want
to
make
sure
that
the
access
to
this
name
pipe
is
gonna,
be
restricted
to
only
the
CSI
pods
and
not
any
random
parts
can
talk
to
them.
B
That
can
be
solved
in
part
by
our
back
policies,
but
there's
still
more
potential
issues
to
think
about
there
and
the
second
option
to
say:
okay,
like
we're
still
gonna
run
in
containers,
so
that
we
can
leverage
what
Cube
knows
how
to
do.
It's
basically,
a
hack,
saying,
okay,
we're
gonna
change
container
D
to
be
able
to
pretend
that
we
were
in
a
container
to
run
with
all
the
semantics
and
the
external
appearance
of
a
container,
but
actually
just
processing
the
host.
B
So
that
would
be
a
hack
at
the
continuity
level
and
and
that
would
allow
us
to
use
all
the
goodies
that
come
with
cube
promise.
Of
course
it's
really
hacky
and
it
might
resolve
in
very
poor
user
experience,
because
you're
gonna
tell
you
hey
like
this
is
pretty
much
if
on
Windows
and
yeah
sure,
but
is
not
a
container
anymore.
So
we're
not
really
sure
what
to
do
there,
there's
no
real
consensus
yep.
So
if
you
have
an
opinion
and
you're
interested
in
this,
please
do
come
to
cig.
B
Windows
community
calls
or
join
the
discussion
there.
We
have
calls
every
Tuesdays
at
9:30
a.m.
PST
and
while
we're
in
the
subjective
community,
I'd
like
to
thank
deep
debris
from
docker,
has
been
working
hard
and
this
problem
set
aside
for
Windows
and
the
two
chairs
of
the
sig
Windows
interest
group,
Patrick
Lang
for
Microsoft
and
Michael
Michael
from
VMware,
and
you
know
in
summary,
you
can
use
today
entry
and
five
million
plugins.