►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes SIG Storage Deep Dive - Xing Yang, VMware & Mauricio Poppe, Google
Speakers: Xing Yang, Mauricio Poppe
Kubernetes SIG Storage is responsible for ensuring that different types of file and block storage are available wherever a container is scheduled, storage capacity management (container ephemeral storage usage, volume resizing, etc.), influencing scheduling of containers based on storage (data gravity, availability, etc.), and generic operations on storage (snapshotting, etc.). In this session, we will deep dive into some projects that SIG Storage is currently working on, provide an update on the current status, and discuss what might be coming in the future.
A
A
B
B
B
Sixth,
sword
has
two
co-chairs:
usad
Ali
from
Google
and
myself
are
the
co-chairs
and
we
also
have
Michelle
from
Google
and
Young
from
Red
Hat.
They
are
two
tech
leads.
There
are
also
many
other
contributors
in
six
storage,
so
we
have
more
than
5
000
members
in
the
sexology
slack
Channel.
We
also
have
several
other
channels.
We
have
a
channel
for
cozy
for
CSI
and
CSI
windows.
B
What
do
we
do
in
six?
Storage
is
defined
in
our
Charter,
so
six
storage
is
a
special
interest.
Group
focuses
on
providing
storage
to
Containers
running
in
a
kubernetes
cluster
in
a
sick
storage,
most
notable
features
include
percent
volume
claims,
persistent
volumes,
so
the
class
and
dynamic
provisioning.
B
We
have
volume
plugins
in
addition
to
persistent
volumes
that
persist
data
beyond
the
first
life
cycle.
We
also
have
iPhone
volumes
such
as
Secrets
config
Maps,
don't
worry
apis
empty
doors,
their
life
cycle
are
tied
with
the
parts
life
cycle
and
we
also
have
container
storage
interface
CSI
that
allows
the
sorti
vendors
to
write
a
plug-in
and
have
it
work
in
kubernetes
and
other
container
orchestration
systems.
B
B
B
To
use
this
feature
in
the
CSI
driver
spec,
you
need
to
specify
volume
lifecycle
modes
as
Admiral
and
in
the
Pod.
You
need
to
specify
the
water
type
as
CSI
and
specify
driver
name
and
volume
attributes
they
see.
The
driver
is
suitable
to
be
used
as
an
admiral
volume
if
it's
a
built
for
special
purpose
and
need
for
volume.
Custom
parameters,
like
a
driver
that
provides
secrets
to
a
pod,
for
example,
secret
source,
is
a
driver,
is
a
good
example
of
that.
B
Another
example
is
a
search
manager,
Cesar
driver,
a
CSA
driver
is
not
suitable
to
be
used
as
a
CSI
inline
afro
volume
if
it
needs
to
support
features
such
as
on
snapshotting
cloning
expansion.
If
a
provisioning
is
not
limited
to
the
local
node
or
if
you
need
some
volume
attributes
that
are
only
that
are
supposed
to
be
restricted
to
the
the
admins
on
it
like
the
parameters
in
a
storage
class
and
also,
if
you
want
to
persist
your
data
beyond
the
past
life
cycle,
you
that's
also
not
suitable.
For
this
feature,.
B
The
second
feature
move
the
qta
is
the
local
Admiral
storage
capacity
isolation.
This
feature
was
introduced
back
in
1.7
release,
so
it's
been
quite
a
while
now:
Parts
use,
Admiral
storage
for
Scratch
space
or
for
caching
or
logging
same
logs,
but
the
parts
could
be
addicted
because
other
parts
are
feeling
the
local
storage
space.
B
B
B
B
It
is
typical
for
a
storage
system
to
require
credentials
to
be
passed
in
through
a
CSI
driver
for
voting
operations.
That
includes
one
expansion.
Phone
expansion
can
happen
on
the
controller
side
or
the
North
side,
or
both
from
the
controller
side.
We
already
have
a
way
to
pass
in
the
secret
from
the
CSI
driver.
B
So
now
in
1.25,
we
also
added
this
feature
for
the
secretary
passed
in
to
CSI
driver
when
the
water
won't
expansion
happening
on
the
Node
side,
when
the
first
system
is
resized
and
the
third
feature
is
a
reconcile
default
storage
class
assignment.
This
feature
allows
the
existing
PVCs
without
storage
class
name
specified
to
be
updated
to
use
the
new
default
study
class
when
their
search
class
becomes
available,
and
the
fourth
feature
became
Alpha
in
1.25
is
a
object.
Storage,
API,
cozy.
B
Cozy
adds
a
new
kubernetes
apis
to
provision
buckets
and
also
allow
ports
to
access.
Those
buckets
cozy
component
includes
a
controller
manager
that
binds
the
Cozy
created
buckets
to
bucket
claims.
There
is
a
sidecar
that
watches
the
kubernetes
API
objects
and
cause
the
Cozy
driver
to
provision
or
delete
buckets,
and
the
Cozy
driver
implements
grpc
interfaces
communicates
with
the
storage
backend
to
provision
buckets
or
delete
buckets.
B
B
A
bucket
represents
a
physical
bucket
on
the
through
the
back
end,
and
the
bucket
claim
is
a
user's
request
for
a
bucket,
that's
in
the
user's
name,
space
and
the
bucket
class
that
specifies
the
type
of
the
bucket
that
you
want
to
provision
and
the
second
side
of
apis
or
bucket
access
and
bucket
access
class.
That
is,
for
the
Pod
to
get
access
to
the
bucket.
B
B
So
we
have
delegate
FS
group
to
CSI.
Driver
mode
is
targeting
GA
in
1.26
release,
so
in
the
in
the
part,
security
context,
you
know
the
fs
group
setting
that
you
must
specify
so
that
the
Voting
is
readable
and
writable
by
the
part.
If
the
Pod
runs
as
a
null
root
user
and
based
on
the
fs
group,
kubernetes
will
recursively
change
the
ownership
and
permissions
of
all
the
volumes
on
their
pod.
B
Unless,
if
the
fs
Secrets
change,
FS
group
change
policy
setting
specified
to
skip
it,
however,
not
every
CSA
driver
can
modify
the
only
ownership
of
missions,
for
example,
NFS
type
or
other
voting
type.
That
does
not
support
Unique
Style
permission
changes.
That's
why
we
introduce
this
feature
to
let
the
CSR
driver
to
opt
out.
B
So
there
are
three
different
settings
for
this
FS
group
policy
in
CSI
driver
spec
by
default.
It
is
a
real
ones
with
FS
type.
That
means
we'll
examine
that
at
long
time
and
determine
whether
we
need
to
apply
modification
or
not.
The
second
one
is
the
file.
That
means
always
apply
modification
and
the
the
third
one
is
none
that
allows
the
system
driver
to
opt
out
and
never
apply
modifications.
B
We
also
have
a
few
features
targeting
data
in
1.26
release.
The
first
one
is
the
control
volume
mode
conversion
between
source
and
Target,
so
we
have
a
use
case
for
backend
restore,
sometimes
the
backup
software
needs
to
change
the
volume
mode
from
file
system
to
block
to
support
efficient
backups.
However,
that
could
lead
to
a
potential
security
problem.
B
That's
why
we
added
this
feature
to
prevent
unauthorized,
only
mode
conversion,
and
we
also
have
a
reconcile
default
search
class
assignment
that
we
are
targeting
data
in
1.26
and
the
third
one
is
a
project
we
co-owned
with
Sig
apps
Auto,
remove
PVCs
created
by
safer
sets.
This
this
adds
an
option
to
allow
PVCs
create
a
device
that
was
such
to
be
removed
automatically,
and
also
we
have
a
non-creation
feature
targeting
beta
in
1.26.
B
This
allows
stable
workloads
to
fail
over
to
a
different
nodes
after
the
origin,
node
is
shut
down
now,
gracefully,
a
no
shutdown
is
a
graceful
only
if
a
cubelet
can
detect
that
and
cubelet
depends
on
system
D's
inhibitor
lock
to
detect
a
shutdown.
However,
not
every
system
supports
this
mechanism,
so
if
it's
not
supported
cubely
does
not
detect
that,
then
it
becomes
a
non-graceful,
no
shutter.
B
B
B
B
The
first
one
is
the
provision
volumes
from
Cross
namespace
snapshot
that
allows
the
PVCs
to
be
provisioned
from
a
volume
snapshot
in
a
different
namespace,
and
we
also
have
this
warning
group
and
the
volume
group
snapshot
feature
which
introduces
new
apis
to
manage
multiple
volumes
together
and
provide
a
warning
group
snapshot.
Api
to
take
a
snapshot
of
a
boarding
group.
B
B
B
A
Right,
thank
you,
okay.
So,
let's
first
look
at
kubernetes
windows
and
CSI
on
the
diagram
in
the
right.
We
see
a
cluster
that
has
two
in
two
node
pools,
a
Linux,
node
pool
and
a
Windows
node
pool.
The
idea
is
that
Windows
workloads
run
in
the
windows.
Notebook
cornetta
supports
three
Windows
server
operating
systems.
Those
are
the
long-term
servicing
Channel
2019
2022,
and
it
also
supports
the
semi-annual
channel.
The
long-term
servicing
channel
has
raised
us
every
two
to
three
years.
The
semi-annual
Channel
I
think
it's
being
discontinued.
A
We
also
see
in
a
coordinates
cluster
that
is
running
Windows.
A
Windows
node
pool
that
the
control
plane
components
are
still
running
in
Linux,
so
all
of
the
components
that
you
are
aware
of
the
API
server,
the
queue
scheduler,
the
Q
control
manager,
all
of
those
are
still
in
the
control
plane
and
in
CSI
we
have
three
sets
of
RPC
Services,
representing
the
three
components
that
CSI
drivers
need
to
implement
the
null
component
and
the
identity
component
and
the
controller
component.
A
So
the
controller
component
is
still
running
in
the
control
plane,
but
for
operations
that
need
to
run
in
the
node
component.
That's
where
we
need
support
for
Windows
too,
so
in
CSI
there
are
three
components
that
need
to
be
working
in
Windows
2.
Those
are
the
no
component
of
the
CSI
driver,
liveness
probe,
no
driver
register,
those
two
last
ones:
liveness
probe
and
no
driver
register
are
maintained
by
kubernetes
CSI
and
the
CSI
driver
is
maintained
by
the
storage
provider.
A
Also,
kubernetes
is
minimally.
It
doesn't
tell
you
how
you
should
deploy
Your
solution
into
the
cluster
and
a
usual
solution
to
deploy
the
node
component
into
the
cluster
is
to
use
a
demon
set
and
in
order
to
avoid
having
two
different
images
in
the
Linux
demon
set
and
in
the
windows
demon
set.
We
can
use
multi-arc
build
pipeline
where
we
create
images
for
both
operating
systems
and
tie
them
together
with
an
oci
image
manifest
that
way.
A
A
What
happens
when,
when
a
workload
requests
a
PVC
is
that
there
are
components
in
the
control
plane
that
will
receive
requests
from
CSI
sidecars.
So
these
are
calls
that
should
be
implemented
by
the
controller
component
of
the
CSI
driver,
the
one
that
is
create
volume
and
control,
publish
volume
in
the
red
rectangle
there
and,
as
I
mentioned,
those
are
still
happening
in
Linux.
A
So
there
are
two
operations
for
setup.
The
first
one
is
know
the
stage
volume
and
node
publish
volume,
so
I
think
it's
worth
mentioning
what
happens
in
Linux
so
that
we
can
understand
what
is
the
difference
with
Linux
or
with
Windows
3.
So
in
in
Linux,
what
happens
when
a
device
is
attached
to
a
node?
We
need
to
format
it,
and
this
happens
in
this
node
stage
volume
call.
A
The
CSI
driver
usually
uses
a
library
from
kubernetes
called
kubernetes
Mount
utils
that
has
utilities
to
execute
them,
mkfs
command
in
Linux,
and
also
to
well
that's
for
to
format
it
and
to
mount
it.
It
uses
the
mount
command
in
Windows.
We
don't
have
these
commands,
but
we
have
similar
Powershell
commands,
so
they
workloading
in
Linux
in
Windows
is
to
create
a
Windows
volume.
We
formatted
to
NTFS.
We
create
a
partition,
access
path
in
the
node
and
that's
called
the
global
mount.
A
The
next
step
is
not
polish
volume,
which
is
to
make
this
volume
available
to
the
pod
in
Linux.
This
is
done
with
a
Buy
Mount,
but
in
Windows,
because
there
is
no
support
for
it.
We
do
it
with
a
Sim
link
to
the
partition
access
path
and
this
is
called
the
Pod
mount.
That
is
for
the
setup
step,
and
at
that
point
the
workloads
can
start
using
the
PV.
A
Then
at
some
point
the
workload
will
will
be
out
of
the
cluster
and
we
will
go
through
the
reverse
systems,
which
are
to
tear
down
the
volume.
So
in
Linux
this
is
done
with
the
u-mount
command,
but
in
Windows
it's
done
by
reversing
the
steps
above
which
are
removing
the
Sim
Link
in
no
damp,
unpublished,
volume
and
in
node
unstageable.
We
remove
the
partition,
access
path.
A
Okay,
so
in
addition
in
Windows
these
operations,
the
volume
of
format
and
mount
operations
are
privileged
operations
and
they
can't
be
done
through
a
Windows
container.
There
is
an
asterisk
there,
because
it's
now
doable
I
will
talk
about
that
in
the
next
slide.
But
when
we
started
working
on
this,
there
was
no
support
to
do
that
and
as
a
workaround,
a
lot
of
components
in
kubernetes
that
run
in
Windows
usually
do
this
through
proxies,
and
we
also
have
a
project
called
CSI
proxy,
which
is
a
binary
that
runs
in
the
host.
A
A
Next
we
deploy
the
CSI
driver
and
the
CSI
driver
needs
to
communicate
with
the
CSI
proxy
binary
through
a
library
that
is
also
an
artifact
of
the
CSI
proxy
project.
So
we
have
multiple
versions
of
it,
because
it
could
be
possible
that
a
driver
used
used
the
beta
version,
while
while
we
were
still
working
on
making
this
P1
and
then
at
some
point,
it
became
B1.
And
now
we
need
also
need
to
manage
these
multiple
version
problem.
A
And
finally,
it
comes
time
to
those
operations,
as
I
mentioned
before
that
are
executed
from
the
cubelete.
So
the
cubelet
will
call
node
stage
volume
it
will
go
to
the
CSI
driver.
The
CSI
driver
will
try
to
format
amount
that
the
device
it
will
use
the
CSI
proxy
library,
and
it
will
talk
with
the
CSI
proxy
implementation
that
does
all
of
those
commands
that
I
mentioned
above.
A
So
it's
worth
noticing
that
this,
even
though
this
is
the
current
CSI
proxy
architecture,
we're
planning
on
modifying
this,
so
that
it's
easier-
and
this
is
where
I'll
talk
about
first
process
containers.
Next,
a
few
of
the
drawbacks
of
this
model
that
are
not
only
Forester
HR
for
most
of
the
proxy
components
in
in
kubernetes
is
that
we
have
an
additional
component
that
we
have
to
maintain,
which
is
CSI
proxy,
and
it's
difficult
to
maintain.
A
As
I
mentioned,
we
may
have
new
features
and
those
need
to
be
part
of
our
new
new
release,
so
that
eventually
becomes
hard
to
maintain,
and
there
is
also
a
tiny
increase
in
latency,
because
now
we
have
to
make
calls
from
the
CSL
driver
to
CSI
proxy
through
the
name
pipes,
all
those
calls
are
done
with
grpc,
and
there
is
this
product
above
serialization,
this
realization
and
so
on.
So
that's
a
tiny
bit
of
increasing
latency.
That's
a
small
drawback
of
this
design.
A
A
It's
actually
running
as
a
process
in
the
Windows
host,
and
one
of
the
caveats
is
that
there
is
no
file
system
isolation.
So
this
means
that
the
cluster
should
have
additional
policies,
so
that
workloads
cannot
become
a
host
process
spots
because
they
could
access
the
volumes
of
all
of
the
other
workload.
So
that's
something
to
keep
in
mind.
So,
in
addition,
if
now
that
we
we
know
of
this
feature,
if
a
CSI
driver
becomes
a
host
process
spot,
then
the
CSI
driver
itself
can
run
this
privileges,
storage
operations
it
doesn't
go.
A
Okay,
so
this
slide
is
about
people
that
have
already
implemented
Windows
support
in
their
CSI
drivers.
On
the
left,
we
have
the
current
implementation
and
on
the
right
we
have
the
new
implementation
we
wanted
to
have
to
reach
a
step
where
we
had
as
minimal
changes
as
possible.
So
as
we
can
see,
it's
just
a
it's
just
renaming
the
Imports
in
in
the
in
the
file
where
the
calls
are
used.
A
That
is
one
change,
and
the
other
change
is
that
the
CSI
driver
deployment
needs
to
have
additional
cost
process
related
spec
fields
and
in
in
this
drivers
we
also
Mount
the
name
pipes
in
in
the
in
the
pot.
So
all
of
those
fields
are
also
removed
because
we
no
longer
have
the
name
pipes,
and
that
should
be
so.
A
I
would
like
to
thank
Alexander
ding
from
Brown
University
for
the
work
that
he
already
did
in
CSI
proxy
to
make
this
work
and
I
also
encourage
everyone
who
has
a
side
driver
and
doesn't
have
a
Windows
implementation,
yet
to
add
window
support
to
their
assistant
driver
I.
Think
it's
now
a
great
time.
You
don't
have
to
support
the
an
additional
proxy
component,
so
I
would
also
like
to
thank
stick
windows
for
working
on
this,
because
they
enabled
us
to
remove
all
of
code
that
are,
that
is
a
needle
with
that
pass.
B
On
this
six
storage
homepage,
we
have
a
lot
of
informations
for
you
to
get
started.
There
are
some
video
recordings,
including
some
of
the
sessions
from
previous
coupons.
Some
are
like
one-on-one
sessions,
so
check
it
out,
and
we
have
bi-weekly
meetings
on
Thursdays,
your
mailing
list
and
slack
channel.
So,
if
you're
interested,
please
join
us
and
get
involved.
B
C
A
C
A
That's
our
question:
I
wrote
the
timeline
for
it
on
the
cap.
We
are
thinking
about
having
it
on
maintenance
mode
because
of
the
drawbacks
that
I
mentioned,
but
that's
still
open
in
the
cab.
So
if
the
community
wants
to
keep
it
open
for
features,
we
could
still
support
both.
But
my
take
right
now
is
to
keep
it
in
maintenance
mode
and
just
go
all
on
the
library
mode.