►
Description
New Features in Kubernetes Storage - Brad Childs, Red Hat
"Two major features were recently added to kubernetes storage-- volume selectors and dynamic provisioning using StorageClasses.
The addition of persistent volume selectors greatly increases a user’s ability to manage and consume volumes. With pv-selector the user can still view storage as ‘cattle’, meaning fungible, but with specific attributes that may impact a running application.
With the addition of StorageClass as configuration parameters to dynamic provisioning in Kube 1.4, administrators can create multiple provisioners with different parameters. Users then request instances of that storage by name.
In this presentation we’ll go over use-cases and scenarios where pv-selector is helpful in managing volumes. Some of the scenarios covered are data in brownfield (legacy storage), various storage topologies, shared storage security, and storage with specific performance characteristics.
We’ll also demonstrate use cases for StorageClass, configuring StorageClasses and requesting storage by named StorageClass. Dynamic Provisioning of storage on both AWS and GCE will be covered.
About Brad Childs
Brad is a principal software engineer and the container storage development lead at Red Hat. He has experience with big data, storage, cloud and PaaS engineering.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
my
name
is
Brad
childs
I
work
at
Red,
Hat
I'm,
the
development
lead
for
storage
volume
enablement
on
kubernetes.
If
you
were
next
door
earlier,
you
would
have
seen
a
yawn
saffron
active
at
awk
on
external
storage,
provisioning
and,
in
you
know,
typical
kubernetes
fashion.
It
was
kind
of
scheduled
out
of
order,
and
you
know
we
did
the
deep
dive
there
and
we're
gonna
do
kind
of
the
general
overview
of
storage
features
here
and
hopefully,
like
this
system
becomes
consistent,
alright,
so
kind
of
a
brief
roadmap.
A
We
I
started
with
the
basic
volume
plug-in
framework
at
the
initial
you
know
launch
of
cube,
and
that
was
that's
a
very
specific
specification
of
the
volume
that
you're
gonna
use
in
a
pod
and
since
then,
we've
added
features
around
well:
we've
added
different
volume,
plugins
we've
added
dynamic
provisioning
and
in
alpha
we've
added
dynamic
provisioning,
a
beta
we've
added
volume,
selectors,
we've
added
security.
We
started
adding
external
provisioners
and
that's
pretty
much.
A
So
start
off
with
the
volume
the
volume
plug-in
framework
I.
This
is
the
most
basic
way
to
use,
storage
and
really
there's
no
abstraction
between
the
developer
and
the
backing
storage.
There's
quite
a
few
different
volumes.
I've
supported-
and
these
are
all
types
that
are
also
supported
in
the
per
system
as
a
persistent
volume.
But
you
will
you
would
specify
your
volume
properties
directly
in
your
pod
and
you
just
get
a
direct
connection
to
your
volume
with
persistent
volumes.
A
We
actually
have
an
abstraction
between
the
storage
and
the
user,
and
it's
enabled
an
administrative
role
in
a
developer
role.
So
we
actually
have
like
a
separation
of
concern.
The
administrator
will
create
persistent
volumes
in
the
you
know,
register
the
persistent
volumes
in
a
pool.
The
developer
will
create
claims
against
the
persistent
volume
and
this
this
level
of
abstraction
allows
the
developer.
A
Not
to
have
to
know
about
storage
and
also
facilitates
portability
between
pods
and
the
actual
infrastructure
they're
running
on,
so
you
don't
have
to
with
the
volume
framework,
if
you
have
your
NFS
server
specified
as
an
IP
with
an
export.
If
you
were
to
move
that
from
on-prem
to
cloud
you'd
end
up
having
to
have
the
exact
same
connection,
details
in
cloud
with
persistent
volume
in
a
claim,
you
actually
have
a
layer
of
abstraction
and
you
can.
The
claim
can
be
something
else
on
a
different.
A
You
know
on
a
different
cloud
provider,
so
we
have
the
concept
of
developer,
owned
assets
and
administrative
owned
assets
for
persistent
volumes,
I
with
persistent
volumes.
We
have
a
binding
controller.
This
wasn't
so
much
an
issue
with
just
regular
volumes.
The
cubelet
would
handle
the
mounting
the
storage,
but
now
that
we
have
two
different
objects
that
have
to
be
linked
together,
we
have
a
controller
that
maintains
the
life
cycle
between
that
the
the
volume
binding
controller's
responsible
for
binding
provisioning
and
recycling.
A
Persistent
volumes-
and
this
is
you
know,
pretty
much-
the
entire
lifecycle
of
a
persistent
volume
and
when
it's
doing
its
matching,
it's
done
primarily
on
size
and
access
mode
and
then
some
of
the
features
that
we've
added
recently
like
label,
selectors
and
storage
classes
has
increased.
You
know
what
it
will
actually
match
on.
A
So
here's
the
basic
flow
for
binding,
a
persistent
volume,
so
we
have
a
binding
controller.
This
is
part
of
the
cube
framework.
It
starts
as
a
you
know.
You
know
when
you
start
your
cube
controller,
the
binding
controller
starts
and
it
sits
there.
Listening
for
persistent
volumes,
the
first
step,
the
administrator
will
create
persistent
volume
objects
and
the
persistent
volume
object
is
the
actual
specification
and
details
of
the
backing
storage.
So
it's
NFS
it'll
be
an
IP
and
an
export
if
it's
cluster
it'll
be
an
IP
and
a
volume
name
I.
A
Just
depending
on
the
details
of
what
your
sword
you
use
like.
The
persistent
volume
spec
is
very
specific
to
that
backing
storage
type.
So
a
step,
one
of
the
admin
creates
the
persistent
volume
with
the
volume
details,
step.
2
a
developer
goes
and
creates
a
claim.
The
claim
has
the
basic
request
that
the
user
needs
for
storage
and
the
most
basic
scenario
they'll
ask
for
storage
of
a
certain
size,
no
specifying
an
access
type.
So
you
have
read
write
once
read.
A
A
In
storage
or
network
storage,
whereas
read
alight
at
once
tends
to
be
like
block
storage
and
then
I
after
the
after
step
number
two,
when
the
dev
creates
the
claim,
the
binding
controller
will
look
at
the
claim
and
it'll
search
through
the
persistent
volume
pool
and
find
something
that
matches
it
and
actually
bind
the
claim
to
the
persistent
volume
the
binding
is
done.
Is
a
two
transaction
I
write,
so
we
have
a
tight
linking
of
the
PV
and
the
PVC.
A
This
is
somewhat
unique
in
cube,
I,
you
really
in
a
h.a
environment,
you
want
to
keep
the
transactions
to
a
minimum.
Two
rights
is
pretty
luxurious,
but
it
was
essential
for
actually
linking
the
two
objects
together
and
we
start
off
by
writing
the
PVC
or
start
off
by
writing
the
the
PVC
to
the
PV
claim
raft.
A
So
the
PV
gets
a
stamp
of
the
PVC
that
claimed
it
and
then
the
the
PV
volume,
the
PVC
volume
name,
gets
its
to
the
PV
name
and
then
during
these
two
rights,
if
a
version
error
occurs,
we
know
that
some
other
object
was
or
some
other
controller
was
trying
to
write
the
object.
The
user
may
have
deleted
the
client.
They
may
have
believed
the
persistent
volume
we
have
roll
forward
scenarios
for
that.
We
also
write
annotations
during
some
of
this
stuff.
A
So
that
goes.
Let's
talk
about
the
new
features,
so
labels
and
selectors
is
a
feature
that
we
added
recently
it's
quite
similar
to
node
labels
and
node
selectors.
The
idea
is,
you
tag
your
labels
with,
or
you
tag
your
volumes.
You're
persistent
volumes
with
certain
labels.
Then
you
use
match
criteria
on
your
PVC
to
you
know
essentially
match
the
two
together.
I
admit,
I,
admit
is
supposed
to
communicate
the
values
and
the
ranges
to
the
user.
A
This
is
kind
of
a
cumbersome
user
experience,
we're
kind
of
in
the
process
of
thinking
of
how
to
fix
this.
It's
a
similar
issue
to
node,
but
if
you
really
have
two
distinct
users,
the
developer
isn't
supposed
to
see
the
persistent
volumes.
So
it's
a
little
hard
for
them
to
discover
what
labels
or
values
you
might
have.
It's
kind
of
on
the
admin
to
communicate
what
those
labels
are.
Typically,
a
lot
of
this
stuff
is
automated
and
scripted
so
the
end
user.
A
A
This
would
go
into
a
persistent
volume
pool
then
step
number
two:
the
user
will
create
a
claim:
they'll
have
a
selector
with
their
match
label
and
then
label
in
the
value
that
they're
looking
for
and
then
step
number
three.
The
binder
will
come
and
actually
search
the
persistent
volume
pool
for
a
matching
volume
and
then
it'll
go
through
this
same
two-step
process
to
bind
the
volume
with
the
claim.
So
nothing
new
here,
except
we
add,
like
new,
match
criteria
for
for
actually
binding
the
volume
in
the
claim.
A
A
A
A
So
the
next
feature
is
storage
class.
This
one
I
they
have
it
has
some
overlapping
function
with
labels
and
selectors.
A
lot
of
this
was
added
in
anticipation
of
you
know,
enhancing
our
dynamic
provisioning
framework.
I
storage
class
really
represents
a
set
of
properties
that
all
pb's
in
the
storage
class
belong
to.
I,
the
storage
class
properties,
don't
have
explicit
meaning
to
the
framework,
but
it's
something
that
the
admin
or
the
developer
kind
of
agrees
upon
has
value.
Typically,
it's
like
performance
characteristics.
A
It
could
be
security,
it
could
be
backup
policy.
It
could
be
really
anything
that
you
want
it
to
be.
Each
claim
can
have
a
single
storage
class,
each
volume
can
have
a
single
storage
class
and
it
really
facilitates
dynamic.
Provisioning
and
storage
classes
can
be
used
for
pre-created
or
dynamically
provisioned,
and
persistent
volumes
and
I'll
show
examples
of
both
of
those,
so
here's
the
yamo
for
it.
So
this
is
the
actual
object
you
would
create
when
doing
a
storage
class.
A
Pretty
simple
the
parameters
are
a
string
string,
so
anything
could
go
there.
The
name
is
important.
The
name
will
have
to
be
specified
in
the
claim,
and
also
the
persistent
volume-
and
you
can
see
here-
is
this
annotation.
That
has
that's
how
the
the
volume
matches
to
the
storage
class
and
then,
like
I,
said
the
parameters
anything
you
want.
They
really
have
meaning
probably
to
whoever's
provisioning
the
volume
should
it
be
dynamically
provisioned
or
statically
provisioned.
A
A
A
There
may
be
provision
or
configuration
specified
in
your
storage
class,
and
this
is
parameters
that
actually
go
to
the
provisioner
when
it's
creating
a
new
volume
of
that
particular
storage
class,
or
it
could
also
have
some
securities
security
information
supplied.
But
again
the
framework
doesn't
care
about
the
properties
so
much.
These
are
brilliant
meanings.
That's
communicated
between
the
developer
and
the
administrator.
A
The
flow
of
persistent
volumes
with
the
binder
and
claims
in
this
example:
it's
just
showing
statically
provisioned
persistent
volumes
belonging
to
a
storage
class,
there's
no
dynamic
provisioning,
yet
I'll
show
that
in
a
minute,
but
it's
the
it's
a
very
similar
flow.
The
admin
will
create
a
storage
class
they'll,
create
persistent
volumes
and
they'll
when
they
create
the
persistent
volume
so
associate
it
with
the
storage
class.
They
created
is
step
number
two
step
number
three:
the
user
will
create
a
claim
and
in
that
claim,
they'll
specify
the
storage
class
that
they
want.
A
Then
the
binder
comes
in
it'll
search,
the
persistent
volume
pool
for
a
volume
that
matches
on
size,
access
mode
and
storage
class,
so
those
three
characteristics
and
finally,
the
binder
will
go
through
the
two-step
process.
The
two
transaction
process
to
actually
bind
the
volume
to
the
claim
real,
simple
stuff
for
this
one,
but
this
gets
us
to
something
more
interesting,
which
is
a
dynamic
provisioning,
so
dynamic
provisioning
is
the
automatic
creation
of
volumes.
So
we
could
either
be
a
cloud
provider
creating
the
volumes.
It
could
be
a
standalone
application,
creating
the
volumes.
It
could
be.
A
Somebody
locked
in
a
cage,
the
crates,
the
volumes.
It's
it's
really
anything,
but
it's
really
done
at
the
at
the
time
that
the
claim
is
created
I
in
our
alpha
version
of
dynamic
provisioning.
We
had
a
single
provision
or
per
cluster,
and
it
was
really
only
supported
by
cloud
providers,
so
AWS
GCE
cinder.
A
We
had
provisioners
there
and
but
it
was,
you
could
only
have
one
provision
in
the
in
the
cluster,
but
with
storage
classes.
In
our
beta
release
of
dynamic
provisioning,
we
really
enabled
multiple
provisioners
per
cluster
and
with
multiple
provisioners
you
could
have
different
properties
for
your
provisioner.
If
you're
an
AWS,
it
allows
you
to
have
SSDs
versus
spinning
discs.
You
can
have
different,
you
could
it's.
You
know
special
specify
the
I,
ops
or
different
performance
characteristics
and
divide.
Your
persistent
volumes
of
by
there
was
different
properties.
A
Then
you
also
have
entry
or
auditory
provisioners.
So
there's
there's
a
distinction
there.
Most
of
the
entry
stuff
right
now
is
cloud
provider.
Our
entry
stuff
does
support
multiple
provisioners
per
cluster
entry
and
then
out
of
tree.
You
can
also
have
multiple
provisioners
per
cluster,
and
you
can
even
have
a
mix
of
these,
so
you
could
have
5
10,
100
storage
classes
per
cluster.
It
really
doesn't
matter
I
and
they
could
be
a
combination
of
in
and
out
a
tree.
A
A
So
it's
it's
really
up
to
the
administrator.
Make
sure
that
what's
being
provision
is
what's
specified
in
the
storage
class,
so
here's
a
list
of
our
entry
provision
or
types.
We
have
a
cinder
a
double
yes
GCE
as
your
vSphere
glossary
FS
are
B
D
and
Q
bite.
Oh
wait,
I'm,
not
sure
how
to
say
that.
But
all
this
stuff
is
baked
in
tree.
You
don't
have
to
do
anything
when
you
start
Q.
If
you
have
storage
of
this
type,
it's
already
there,
you
don't
have
to
stand
out
anything
else
up.
A
B
A
Name
and
secret
namespace:
maybe
you
have
multiple
Gluster
packing
storage
providers
and
one
slower
and
ones
faster
than
the
other
in
create
two
storage
classes,
and
they
were,
they
would
have
different
rest
URLs
and
then
different
users
and
different
different
secrets,
and
that's
pretty
much
goes
for
all
the
rest
of
them.
There's
other
parameters
you
can
tweak
with
you
know,
defendant.
A
So
the
entry
dynamic
provisioning
flow
quite
similar
to
the
other
stuff.
We
have
the
administrator,
creating
the
storage
class.
We
have
the
the
binder
in
this
this
scenario
just
sitting
there
listening
starting
up
as
part
of
the
cube
framework,
the
user
will
come
in
and
create
a
claim
and
they
will
specify
a
storage
class
in
their
claim.
A
So
out
of
tree
provisioners,
if
you
worried
eunsuk,
you
went
over
this
pretty
extensively,
they
can
run
as
a
stand-alone
application.
There's
no
there's
no
specification
of
what
they
have
to
be.
It
could
be
Java.
It
could
be
you
see
it
could
be
going.
It
could
be
shell
script,
basically
whatever
you
want,
but
it
needs
to
be
able
to
communicate
to
the
API
server.
A
It'll,
listen
for
events
from
the
API
server.
It's
meant
to
listen
for
new
claim
events
and
then,
of
course,
for
provision
storage
based
on
the
provision
or
type
that
recognizes
the
recent
changes.
We've
added
I
put
the
provision
or
type
in
the
claim
right.
So
it's
so
they
don't
actually
have
to
look
at
the
storage
class
to
know
if
they
support
a
particular
claim
before
we
had
to
do
like
a
double
reference.
You
have
to
go
and
monitor
the
storage
classes,
watch
for
new
storage
classes
watch
for
types
and
then
monitor
there.
A
Once
you
see
a
provision
or
type,
you
recognize
watch
for
storage
classes
coming
or
claims
coming
in
with
this
storage
class.
Of
that
name,
there's
out
of
three
provisioners
have
no
explicit
call
by
the
volume
binder
it's
up
to
the
external
provision
or
itself
to
listen
for
the
claims
and
to
create
storage
based
on
the
events
coming
in
the
binder
is
completely
out
of
the
picture
in
this,
and
we
have
a
pretty
solid
example
with
our
NFS
provisioner.
A
A
Yeah,
that's
a
good
question,
so
we've
talked
about
some
certification
potential
potentially
certifying
auditory.
Provisioners
truth
is
we
can't
really
control
it.
I
mean
if
it's
on
a
tree
and
a
storage
vendor
wants
to
write
their
provision
or
they're
allowed
to
I
think
every
likes
poorly
on
the
vendor.
If
it's,
if
it
breaks,
wait.
A
Yeah,
so
that's
I,
think
that's
probably
where
we'll
go
is
we'll
end
up.
If
you
want
to
be
listed
on
our
you
know,
external
provision
or
page
of
supported
provisioning
types.
You'd
have
to
go
through
like
a
series
of
checks,
but
we
don't
have
that
in
place.
Yet
this
stuff
is
I
mean
it
was
committed
like
two
days
ago,
so
cool
it's
pretty
new
yeah,
okay,
so
out
of
tree
provisioners
their
response
to
api
events
on
its
own
with
the
the
NFS
provisioner,
we
use
the
go
client
library
and
we
watch
the
api
server.
C
A
So
the
flow
for
this
same
step
admin
creates
a
storage
class.
They
as
a
second
step.
They
will
create
or
run
an
external
provisioner,
and
this
may
be
something
that
runs
on
a
stand-alone
node.
It
may
be
something
that
runs
as
part
of
the
cube
framework.
It
may
be
something
that's
a
completely
manual
process
that
sends
an
email
out.
It's
really
up
to
whoever
is
authoring,
the
external
provision
or
how
this
works.
It's
important
to
follow.
You
know
to
follow
the
API
mechanism
that
was
mentioned
in
the
on
stalk.
That's
how
you
know.
A
We
make
sure
that
the
binder
knows
that
this
value,
that's
a
volume,
that's
created
by
the
external
provisioner,
is
valid,
but
how
this
is
actually
done
is
really
up
to
the
administrator.
So
step
number
one
admin
creates
the
storage
class
step
number
two.
They
install
and
launch
a
provisioner
they'll
point
the
provisioner
back
to
the
cube
API
server.
A
If
step
number
three,
a
user
will
come
in
and
create
the
claim
that
new
claimed
event
will
come
to
the
provisioner,
who
is
sitting
there,
monitoring
the
API
server
for
new
claim
events,
and
then
the
provisioner
will
actually
create
the
physical
volume.
Maybe
that's
a
call
out
to
glass,
or
maybe
it's
a
call
out
to
a
cloud
provider.
A
Maybe
they
put
a
zip
disk
and
a
drive
they'll
do
something
that
creates
a
physical
volume
and
then
they'll
create
a
persistent
volume,
object
that
maps
to
that
that
physical
volume
and
then
as
step
number
six
during
the
creation
that
persistent
volume
object,
they'll
also
bind
it
to
the
claim.
So
it's
creating
the
persistent
volume
and
binding
the
claim
is
kind
of
the
same
kind
of
same
process.
A
So
here's
the
slightest
open
I
didn't
really
have
to
talk
about
because
it's
probably
too
small
for
you
guys
to
see,
but
this
is
the
the
yamo
for
a
fully
bound
volume
and
claim
we
have
a
couple
annotations
off
to
the
left.
These
are
added
by
the
binders,
so
we
know
that
the
binder
was
actually
the
one
completing
the
vine
between
the
claim
and
the
volume
and
the
user
didn't
just
specify
these
values
when
they
created
the
claim
and
if
they
did
there's
some
use
cases
where
that's
valid.
But
we
want
to
know
about
it.
A
So
if
we
have
to
do
a
recovery
step,
we
can
tell
if
it
was
the
user
that
created
the
hard
link
the
two
together
if
the
binder
actually
did-
and
we
have
to
do
some
kind
of
recover
if
there's
a
failure,
we
have
a
volume
name
here
in
the
claim
point
since,
since
the
persistent
volume
is
a
globally
scoped
thing,
the
the
volume
name
has
to
be
unique.
And
then
you
know
you
see
the
UID
of
the
persistent
volume
claim
mapped
over
here
to
the
persistent
volume.