►
From YouTube: So, What If I Don’t Want My Persistent Storage To Be Yet Another Bindmount? -Deep Debroy & Feng Wang
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
So, What If I Don’t Want My Persistent Storage To Be Yet Another Bindmount? - Deep Debroy, Apple & Feng Wang, Databricks
Speakers: Feng Wang, Deep Debroy
Most CSI plugins assume that a mounted volume will be consumed directly on the host. For sandboxed runtimes like Kata Containers, this results in less efficient storage IO; there’s a tradeoff between ease of compatibility versus performant storage. So, what if getting the PVC to the container isn’t just a bind-mount away? There has been progress in Kata Containers and within the greater container ecosystem to allow for storage to be presented to the runtime in a more VM-friendly way that results in better IO performance as well as a better security profile. In this talk, we'll highlight the work done for direct storage assignment, as well as the challenges we’ve worked through with the node and storage communities, as well as KEPS to facilitate this pattern in CSI and Kubernetes. We will show how a well defined generic API can allow for efficient storage handling for all sandboxed runtimes.
A
A
A
So
here's
a
quick
introduction
to
our
talk.
First,
we'll
start
off
with
describing
how
the
current
life
cycle
and
flow
of
events
work
for
pods
that
mount
persistent
storage
we'll
go
over
some
of
the
key
assumptions
that
are
there
in
the
cubelet,
as
well
as
container
runtimes
around
how
volume
mounts
are
set
up
for
pods
that
do
Mount
persistent
storage.
Next,
we
are
going
to
go
over
some
of
the
typical
bind
mounts
that
take
place
as
part
of
surfacing
person
storage
to
pods.
A
Today,
after
that,
we
are
going
to
go
over
some
of
the
alternatives
to
the
standard
flow
for
mounting
personal
storage
into
pods.
We'll
look
at
this
alternative,
mainly
from
the
perspective
of
alternative
runtimes,
to
run
C
such
as
micro,
VMS,
specifically
Kata,
we'll
also
go
over
some
of
the
challenges
that
arise
when
some
of
the
base
assumptions
in
cubelet
and
the
container
runtime
are
no
longer
true.
A
Finally,
we'll
be
going
over
some
of
the
enhancements
that
we
have
implemented
in
CSI
plugins
to
basically
surface.
This
alternate
flow,
which
we
call
the
direct
assignment
model.
We
have
initially
prototyped
it
with
by
adding
a
fair
amount
of
awareness
around
kubernetes
and
Kata
runtime
within
the
CSI
plugin,
and
we'll
go
over
a
future
Direction
where
we're
exploring
a
cap
to
make
this
support
a
lot
more
generic,
so
that
the
CSI
plugins
do
not
need
to
be
aware
of.
The
fact
that
they
are
operating
within
a
kubernetes
framework.
A
So,
let's
start
all
the
way
at
the
beginning
and
look
at
how
a
part
comes
to
life
from
the
perspective
of
the
cubelet.
So
typically,
you
either
have
a
pod
being
submitted
by
Cube
cuddle
or
a
workload
controller
such
as
dateful
set
and
once
the
Pod
gets
submitted,
it
gets
created
in
the
API
server.
The
cube
scheduler
picks
up
the
Pod
and
picks
the
most
ideal
node
in
the
cluster
that
the
Pod
should
get
scheduled
to
once.
The
node
name
is
set
by
the
scheduler.
A
A
One
of
the
first
things
that
the
cubelet
starts
processing
is
the
volume
section
of
the
pod,
so
there
are
two
main
categories
of
volumes
that
pods
can
mount
in
kubernetes.
The
first
kind
are
the
inline
volumes
and
they
mainly
present
ephemeral
data
to
the
pods,
and
this
include
things
like
config
map
or
secret,
which
are
typically
powered
by
a
piece
of
code,
that's
already
within
the
cubelet
itself,
because
they're
so
common
and
pervasive.
However
ephemeral
volumes
may
also
be
backed
by
specialized
CSI
plugin.
A
Now
person
volumes
can
be
backed
by
a
variety
of
different
storage
backends,
so
it
could
be
say
EBS
in
case
of
the
AWS
cloud
or
gcpd
in
case
of
the
gcp
environment
or
your
own
San
environment,
in
case
of
on-prem
setup.
Now
the
cubelet
it's
impossible
for
it
to
know
how
exactly
to
discover
these
various
storage
backends
and
how
to
mount
the
volume
from
these
backends.
So
for
doing
that,
it
depends
on
plugins
associated
with
these
back-ends
that
are
shipped
by
individual
vendors
and
installed
on
specific
nodes,
and
these
are
called
CSI,
node
plugins.
A
So
by
utilizing
a
CSI,
node
plugin,
the
cubelet
is
able
to
discover
a
storage,
a
piece
of
storage,
that's
attached
to
a
node
and
then
instruct
that
node
plugin
to
get
the
volume
prepared
so
that
it
can
be
mounted
in
appropriate
location
from
which,
from
which
point
onwards,
the
guplet
can
start
processing.
It.
A
Now,
once
the
file
system
on
the
volume
has
been
mounted,
the
cubelet
executes
a
bunch
of
very
important
actions,
specifically
the
first
one
involves
looking
at.
The
f
is
group
specification
in
the
Pod
and
applying
it
based
on
what
is
called
the
fs
group
change
policy
and
what
is
a
FS
group.
F's
group
is
essentially
a
supplemental
group
ID
that
a
pod
May
specify
so
that
different
pods
that
run
with
different
user
IDs
can
share
the
data
on
the
same
persistent
volume
in
older
versions
of
kubernetes.
The
cubelet
would
sort
of
blindly
mount
a
volume.
A
Look
at
the
f
is
group
and
do
a
recursive
file
system
traversal
through
the
entire
volume
to
apply
the
group
ownership
in
more
recent
versions
of
kubernetes.
There's
the
new
FS
group
change
policy
called
onroot
mismatch
that
allows
this
process
to
be
a
lot
more
optimized,
and
so
in
this
mode
the
cubelet
just
looks
at
the
top
level
directory
sees
if
it's
owned
by
the
right
supplemental
group,
ID
that's
specified
as
f
is
group,
and
if
so
it
just
doesn't
it
skips
the
entire
file
system
reversal
next
up.
A
So
the
right
slnx
labeling
is
is
provided
to
the
files
and
directories
so
that
containers
within
the
Pod
can
read
the
files
and
directories.
If
a
Linux
label
is
not
specified,
cubelet
still
instructs
a
container
runtime
to
go
ahead
with
SL
Linux
labeling,
but
figure
out
the
SL
Linux
label
at
runtime
dynamically
allocated
by
the
container
runtime.
A
So
once
the
file
system
mounts
are
ready,
the
cubelet
moves
on
to
the
next
phase,
which
is
invoking
a
CRI
implementation,
typically
continued
to
your
Creo
and
which,
in
turn,
goes
on
to
invoke
a
lower
layer,
a
lower
level
container
runtime
such
as
rancy
or
Kata,
to
create
the
Pod
sandbox
environment,
which
could
be
just
a
bunch
of
namespaces
in
case
of
run
C
or
a
micro
VM
in
case
of
Kata,
pull
down
the
container
images
and
create
the
individual
containers
specified
within
the
pod.
As
part
of
the
first
step.
A
Now,
once
the
Pod
and
the
containers
are
up,
it's
not
like
the
job
around
file
system
operations
is
not
quite
done.
Throughout
the
pods
lifetime,
there
could
be
other
volume
management
operations
that
are
that
need
to
be
executed
by
the
cubelet.
So
specifically,
there
are
Prometheus
metrics
around
file
system
stats
for
the
volumes
that
are
mounted
and
the
cubelet
needs
to
basically
query
the
file
system,
stats
for
all
the
volumes
through
the
corresponding
CSI
node
plugin.
A
A
So
here's
kind
of
like
a
complete
picture
of
how
all
of
this
fits
in
together.
So
again,
just
to
recap,
at
the
very
left
we
have
the
cubelet
talking
to
a
CSI,
node
plugin
to
First,
discover
the
backing
storage
and
then
get
it
mounted
with
a
specific
file
system
onto
what's
called
a
global
Mount
path.
This
typically
happens
as
part
of
a
CSI
API
called
CSI
node
stage
volume.
A
Next,
This,
Global,
Mount
path
gets
bind
mounted
to
a
pod,
specific
Mount
path
as
part
of
a
CSI
node
publish
volume,
API
and
finally,
the
cubelet
works
with
the
CRI
container
runtime
and
a
lower
level.
Runtime
Handler
like
run
C
to
prepare
the
Pod
sandbox
environment
and
bind
Mount
the
Pod
path
into
the
Mount
namespace
of
of
the
spot
of
the
Pod
sandbox.
A
A
And
again,
to
recap:
one
of
the
things
we
saw
is
that
all
the
cubelet
assumed
that
all
the
file
systems
for
all
the
volumes
are
fully
mounted
before
it
went
on
to
the
container
bring
up
stage.
This
is
important
because
the
cubelet,
as
well
as
the
container
runtime,
makes
these
assumptions
because
it
needs
to
execute
the
following
three
actions
on
the
file
system:
Mount.
So
that
again
is
applying
the
fs
group
settings
making
sure
the
subpats
are
sanitized
and
probed
and
finally,
applying
the
appropriate
SL
Linux
labels.
B
Thank
you
deep
next
talk
about
how
do
we
mount
a
PV
in
a
macro
VM
based
around
time,
so
microvm
based
container
runtime
is
the
category
of
runtime
that
runs
the
container
workload
in
a
virtual
machine.
Typical
examples
include
color
containers
and
file
cracker
compared
to
traditional
container
runtime.
That's
based
on
Linux
namespace
and
C
group
such
as
round
C.
The
color
containers
provides
better
workload,
isolation
and
much
better
security,
which
is
really
important
for
use
cases
such
as
multi-tenant
workload
in
one
cluster.
B
It's
also
more
lightweight
compared
to
a
foreground
virtual
machine,
for
example.
Containers
has
made
optimizations
to
improve
the
startup
time
and
reduce
the
memory
footprint.
Color
containers
is
also
fully
combined
to
the
OSI
container
format
and
then
the
contained
the
cryo
interface.
So
users
don't
need
to
change
the
application
to
migrate
from
a
run
C
container
to
a
color
container.
B
So
a
quick
recap
of
cut
the
architecture
of
color
containers
color
contains
interact,
integrate
with
CI
contained
runtime
such
as
containerd
and
cryo,
through
a
stream
through
the
stream
interface.
So,
under
the
hood,
kalashim
input
translate
ocis
back
to
a
set
of
VMS
back
and
launch
a
virtual
machine
through
the
hypervisors,
such
as
Kimu
and
Cloud
hypervisor
and
inside
the
gas
via
there's
also
agent
called
Car
Agent.
The
calcium
can
interact
with
the
color
agents
through
a
GT
V
socket
socket.
We
stock
socket
using
tdrpc
protocol.
B
B
So
how
do
we
mount
a
volume
in
color
container?
A
traditional
way
is
very
similar
to
what
deep
just
described
the
CSI
will
prepare.
The
volume
will
bind
down
the
volume
to
the
third
part
mount
and
then
the
color
Sheen
will
do
the
magic.
It
will
actually
find
Mount
again.
The
per
pod
Mount
to
a
shelled
location
and
the
shell
directory
is
then
shelled
to
the
get
through
the
shell
file
system
called
word
ioffs.
B
B
B
So
we
actually
did
some
micro
Benchmark
using
the
fio,
as
you
can
see,
compiled
to
the
lbrk
with
spdk.
As
the
data
plan,
the
web
RFS
has
much
worse
performance
in
the
random
right
and
random.
Read
the
run.
The
sequential
right
is
similar.
The
sequential
read
is
actually
faster,
so
I
think
this
is
largely
due
to
the
caching
and
the
the
pre-fetching.
B
So
because
of
this
caveat,
we
explore
a
alternative
approach
which
is
to
delegate
the
PV
Mount
and
preparation
to
The
Container
runtime
inside
the
guest.
We
call
the
direct
assigned
storage,
so
the
main
difference
here
is
instead
of
riding
on
with
our
FS.
The
color
scheme
will
actually
attach
the
block
device
to
the
guest
through
the
without
BLK
and
then
inside
the
guest.
The
agent
will
handle
the
mount
and
and
develop
the
preparation.
B
So
here
is
a
much
more
detailed
view
of
the
sequence
so
to
when
create
a
volume,
the
correct
will
call
the
notepad
urine
and
then
the
CSI
driver
plug
instead
will
invoke
a
new
command
line.
We
Implement
in
color
runtime
for
the
direct
volume
at
and
pass
the
amount
information
to
the
color
runtime.
The
color
runtime
then
persisted
amount
of
information
on
the
disk.
So
when
the
calcium
is
ready
to
start
the
pot
sandbox,
it
will
read
this
amount
information
from
the
file.
B
So
the
amount
is
actually
simpler.
When
the
part
is
gone,
all
the
mount
will
be
destroyed
inside
the
guest
so
on
the
host.
Only
thing
left
is
basically
the
mount
info
file.
So
when
the
coupon
code
note
on
publish
it
says
that
drive
plugin
will
basically
just
invoke
the
direct
volume
remove
CLI
the
color
runtime
just
remove
the
not
the
info
file.
B
B
Similarly,
to
resize
the
volume
the
CSI
plugin
can
invoke
the
drag
volume
resize
and
then
the
cut
runtime
will
first
resize
the
block
device
through
the
hypervisor,
assuming
the
hypervisor
support
in
this
feature
and
then
send
the
to
the
RPC
request
to
the
car
agent
to
resize
the
volume
which
the
car
agent
just
refresh.
The
file
system
metadata.
B
B
So
now
we
are
inside
the
container
in
a
guest
VM.
We
can
do
a
amount.
We
can
show
all
the
amount
for
you
can
see
here.
There's
the
red
I
will
be
okay
volume,
which
is
Mount
to
Dev,
slash
VDC.
It
does
the
valid
BLK
device,
that's
attached
to
the
VM,
and
then
the
type
is
ext4.
That's
what
we
designed
a
native
file
system
and
then
there's
also
this
red
IO
FS
volume,
so
this
is
has
type
with
lfs.
So
this
is
showed
front
host.
B
B
So
the
current
approach,
it
requires
a
specific
CSI
implementation,
there's
no
change
in
Kubla,
CI,
OSI
or
css
specs
and
most
of
the
post-mod
configuration
works,
but
there
are
also
a
few
drawbacks
right
now.
For
example,
the
sub
pass
report
is
not
supported
right
now.
The
CSI
plugin
also
need
to
be
customized
and
need
to
be
aware
of
the
runtime
class.
A
Great
thanks
a
lot
thanks
for
the
wonderful
demo.
So
let's
look
at
how
the
feature
of
this
would
look
like.
So
we
are
basically
exploring
kubernetes
enhancement
proposal,
specifically
2857
Upstream,
with
mainly
six
storage,
with
a
little
bit
of
signaled
involvement
as
well
to
address
some
of
the
limitations.
Fang
just
went
over,
so
the
idea
there
is.
A
Can
we
enhance
the
cubelet,
the
runtime
class
and
the
CSI
spec
in
a
way
that
allows
first
of
all
the
cubelet
to
pass
down
all
the
necessary
parameters,
around
Postman
configuration
down
to
the
CSI
plugin
and
basically
not
have
the
CSI
plugin
need
to
do
a
cube,
API
server,
lookup
that'll
help
solve
one
of
the
negatives.
The
other
is
one
of
the
other.
Feedbacks
have
been
like
if
the
runtime
class-
that's
already
there,
to
specify
the
the
micro
VM
runtime
details.
A
Can
that
be
enhanced
to
have
fields
that
allows
the
runtime
to
basically
advertise
to
cubelet
that
hey
I
am
capable
of
doing
this
file
system
mounting
and
delegation
from
a
CSI
plugin.
So
the
cubelet
will
essentially
be
enhanced
to
look
up
and
match
the
capabilities
of
the
runtime
with
that
of
the
CSI
plugin,
and
if
all
the
capabilities
of
the
Pod
spec
match
the
capabilities
of
the
plugin
and
the
runtime
around
all
kinds
of
Mount
and
post
mount
configuration.
A
A
Pvcs
may
not
be
safe
in
such
scenarios,
especially
if
you,
if
you're
using
a
standard
file
system
like
xfs
or
exe4,
there
will
be
file
system
corruption,
because
the
same
file
system
will
get
mounted
by
multiple
Pods
at
the
same
time,
to
work
against
that.
What
we
are
recommending,
as
part
of
the
enhancement,
is
to
use
a
new
CSI
access
mode
that
has
been
recently
introduced
called
read,
write
one
spot
typically
in
our
experience.
Most
pods
do
not
have
Affinity
relationship,
and
this
should
just
work
out,
but
just
wanted
to
call
this
out.
A
The
other
interesting
scenario
to
consider
is
that
if
the
mounting
of
the
volume
requires
a
secret,
then
it's
not
recommended
to
use
this
approach,
because
the
mount
info.json
file
that
Fang
was
going
over.
That
contains
all
the
mount
options
that
are
necessary
and
we
do
not
want
a
secret
to
be
persisted
in
the
OS
disk.
A
A
We
mainly
looked
at
it
from
the
perspective
of
blog
based
PVS,
and
the
key
thing
we
avoided
is
mounting
the
file
system
in
the
host
and
avoided
all
the
associated
mind
mounts
and
file
system
projections.
Through
things
like
what
iofs
we
would
love
to
get
your
involvement
and
feedback
so
feel
free
to
reach
out
either
through
the
cap
process,
which
is
cap2857,
and
you
can
reach
us
in
the
six
story.
Slack
channel
in
kubernetes,
as
well
as
in
the
Kata
Community
Slack.
C
Awesome,
thanks
for
the
demo,
just
had
a
question
around
the
usage
of
block
storage,
which
is
file
storage,
so
the
in
the
demo
I'm
assuming
was
using
a
block
device
and
that's
how
we
were
avoiding
this
cubelet
interaction
of
map
by
mounting,
but
in
in
the
cap.
Is
that
a
proposal
to
allow
for
delegation
I
just
wanted
to
clarify
that
point
or
would
users
of
this
feature
still
have
to
use
block
devices.
A
So
in
the
cap
from
the
sink
storage,
Community
there's
a
bit
of
hesitance
to
open
it
up
to
network
file
like
NFS
scenarios
as
well
as
block
their
recommendation,
is
to
start
off
at
the
alpha
stage,
at
least
with
just
block
and
then
move
over
to
Shared
file
systems.
However,
with
the
approach
that
Feng
went
over,
there's
nothing
that
prevents
you
from
just
you
know
applying
the
same
mechanism
to
NFS
or
SMB
as
well.