►
Description
Kubernetes Storage Evolution: Enabling High Performance Distributed Datastores [A] - Erin A Boyd, Red Hat & Michelle Au, Google
This talk will focus on the recent changes & challenges in Kubernetes to address the need for consistent & secure access to local persistent storage and raw block storage.
About Michelle Au
Michelle Au is a software engineer at Google and is leading the local storage effort in Kubernetes. Prior to Google, she was at EMC, working on cluster management and cluster communication protocols. She received a BS in Electrical Engineering and Computer Science from UC Berkeley.
About Erin A Boyd
Erin Boyd joined Red Hat to work on Big Data & Emerging Technologies in 2013. Since that time she has worked on various Open Source projects with most recently contributing to the Kubernetes Storage SIG, Storage E2E testing, and Container Native Storage. Prior to working at Red Hat, Erin spent 13 years at IBM rocking the boat. Creating a homegrown Enterprise Services Bus for a federated analytic solution using Open Source. When she isn't working, she is out enjoying the Big Sky of Montana with her husband and 5 kids.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
there's
some
expected
knowledge.
We
assume
that
you
have,
but
you
know
what
they
say
about
when
you
assume
so
today
we're
gonna,
look
at
pods
labels,
nodes,
persistent
volume,
claims,
persistent
volumes,
storage
classes
and
stateful
sets
storage
classes
and
stateful
sets
being
the
most
new.
Api
objects
is,
or
are
you
guys
familiar
with
both
of
these?
A
Not
really
okay,
so
storage
classes
are
a
way
for
the
administrator
to
classify
the
storage
that
they
either
create
through
static
or
dynamic
provisioning.
This
allows
the
administrator
to
create
sets
of
volumes,
maybe
for
a
specific
zone
that
they're
dynamically
provisioned
in
or
maybe
a
specific
use
within
the
corporation.
You
might
Dinah
you
might
provision
volume
specifically
for
accounting
and
make
them
really
slow
and
then
stateful
sets
allow
us
to
create
an
identity
that
sticks
to
the
pod
and
anytime.
I,
see
that
word
within
documentation.
I
constantly
have
this
image
of
sticky
I.
A
So,
as
I
mentioned,
we're
going
to
be
talking
about
local,
persistent
volumes
and
raw
block
volumes
in
terms
of
stateful
distributed
workloads,
so
that
word
kind
of
gets
tossed
around
a
lot.
If
lot
of
the
keynotes
they
talk
about
distributed,
distributed,
distributed
and
kubernetes
being
stateless.
Well,
we
hate
to
burst
your
bubble,
but
you
know
Cassandra,
MongoDB
and
cluster
FS
is
other
distributed.
Applications
actually
needs
state,
they
replicate
the
shared
data
for
high
availability
and
fault
tolerance,
and
many
of
them
are
critical
infrastructure
applications
within
the
enterprise.
A
Many
of
these
that
utilize
data
analytics,
for
instance,
you
can
see
where
local
persistent
volumes
can
provide
data
locality
for
performance
and
data
gravity.
I
think
is
probably
the
biggest
issue
here
is
you
can
execute
on
where
the
data
is
today
and
last
high
performance
tuning.
So,
with
the
advent
in
the
maturity
of
raw
block
volumes,
we
will
now
be
able
to
tune
the
storage
more
appropriately
to
really
get
what
you
need
out
of
raw
block
devices.
A
And
so
there
are
some
other
features
in
addition
to
what
was
merged
that
were
have
been
merged,
since
1.6
I
believe
kubernetes
that
benefit
stateful
workloads.
Stateful
sets,
as
mentioned
in
the
assumptions,
are
ways
to
provide
that
stable
identity.
It's
a
normal
controller.
If
you
went
to
any
of
the
keynotes,
they
talked
about
the
cat
set
in
which
you
just
have
a
controller
that
is
intending
to
go
to
a
certain
state
and
constantly
execute
trying
to
do
that.
A
How
is
this
different,
I
think
the
common
question
I
get
is?
How
is
this
different
than
my
replica
set
or
my
deployment?
It's
the
stable
identity
and
the
stickiness
that
it
provides.
It's
not
necessarily
appropriate
if
you
don't
care
about
those
things,
it's
also
for
workloads
that
you
need
to
have
a
graceful,
startup
or
a
graceful
shutdown
in
which
things
have
to
happen
in
a
certain
order.
A
Zookeepers
a
perfect
example
of
that
as
each
one
of
the
nodes
come
up
with
in
zookeeper
they're,
given
a
unique
DNS
network
identity
that
then
defines
them
within
the
quorum.
The
next
feature
in
kubernetes
that
also
benefits
stateful
workloads
is
a
pod
disruption.
Budget
for
control,
disruption
and
I
have
to
kind
of
giggle
to
think
about
what
a
control
disruption
is
in
such
an
immature
system,
but
it's
actually
pretty
remarkable.
A
So
this
allows
you
to
say
if
I
have
I'm
running
bluster
and
I
have
a
clica
three,
that
if
I
am
doing
a
rolling
upgrade
we're
not
talking
about
kernel,
panics
or
anything
like
that,
that
I
need
to
at
least
have
one
of
those
running
at
all
times.
That
I
cannot
have
my
entire
replica
go
down
or,
of
course,
we
have
de
loss.
So
the
pod
disruption
budget
allows
us
to
say
what
we
can
tolerate
in
terms
of
the
the
replicated
pods
being
down
all
at
once.
A
The
next
feature
of
pod
affinity
in
ni
infinity,
those
being
the
same
things,
allow
us
to
co-locate
or
spread
pods
among
the
nodes.
You
can
see
where
there's
a
huge
advantage
in
pod
affinity
where,
if
I
have
two
pods
that
can
benefit
about
being
co-located
together.
This
feature
is
wonderful
for
local
storage
and
analytics
to
provide
that
support.
The
anti
infinity
really
kind
of
solves
the
noisy,
neighbor
problem.
A
Where
you
have
a
lot
of
noise,
either
network
bandwidth
or
cache
consumption
that
you
need
to
be
removed
from
other
pods
and
not
scheduled
in
between
that
so
similar
I
think
to
taints
and
toleration
x'.
If
you've
read
much
on
that,
but
providing
what
we
need
in
terms
of
the
stateful
set,
the
last
one
is
pod
priority
and
preemption,
because
everyone
of
course
has
the
most
important
application
that
runs
within
the
system
right.
A
B
And
on
okay,
there
we
go
so
even
with
all
these
new
features,
there
are
still
few
missing
pieces
to
the
puzzle,
namely
currently
stateful
sets
work
very
well
with
remote
storage,
but
not
so
well,
if
you
want
to
use
local
storage
for
better
performance
at
reduced
cost.
Currently,
the
only
way
in
kubernetes
to
use
local
storage
is
to
use
a
host
path
volume
and
they
have
a
lot
of
problems,
as
you
can
see,
on
the
right,
I
have
a
pod,
spec
and
I
have
to
specify
two
pieces
of
information.
B
The
first
piece
of
information
is
the
path
to
the
local
disk.
I
want
to
access,
and
the
second
piece
of
information
is
the
node
or
that
disk
resides.
So
you
can
see
the
first
problem.
This
pot
spec
is
not
portable.
It
is
specific
to
the
configuration
of
this
specific
node
and,
if
I
wanted
to
scale
my
application
to
hundreds
of
notes.
That
means
I
have
to
create
a
pod
spec
for
each
individual
note
that
I
want
to
scale
on
and
basically
manually
scheduled
and
manually
scale
my
application.
B
The
second
problem
is
that
host
path,
problem
or
host
pack
volumes
are
a
big
security
risk
because
you're
allowing
the
user's
pod
to
specify
any
path
in
the
system,
including
a
path
to
someone
else's
data
or
a
path
to
some
system,
critical
data.
For
that
reason,
many
administrators
disable
host
path
all
UMES
altogether,
and
you
wouldn't
be
able
to
run
your
workload
on
those
clusters
if
you
had
to
use
host
Beth
Williams,
so
the
way
that
we
decided
to
solve
all
these
issues.
B
Oh
sorry,
so
what
many
application
developers
have
had
to
do
today
is
to
either
manually
maintain
the
pod
specs
for
all
of
their
nodes,
which
makes
scaling
very
difficult
or
they
have
to
write
their
own
custom,
schedulers
or
operators
and
local
disk
reservation
and
lifecycle
managers
in
order
to
handle
all
this
it
to
automate
all
of
this
manual
process
and
that
developing
and
maintaining
all
that
extra
infrastructure
just
support
your
workload
to
kubernetes
is
a
big
engineering
effort.
In
addition,
you
wouldn't
be
able
to.
B
So
to
solve
all
these
problems
with
hose
path,
we
decided
that
the
best
way
is
to
extend
the
existing
persistent
volume
claim
and
persistent
volume
model
in
kubernetes
to
also
support
local
disks.
So,
to
give
a
little
bit
of
background,
persistent
volume
claims
and
persistent
volumes
basically
add
a
layer
of
indirection
in
between
the
users,
storage
requirements
and
the
actual
storage
available
in
the
cluster.
B
So
in
my
users,
pod
I'll
specify
a
persistent
volume
claim
and
inside
the
claim,
I
would
specify
some
generic
volume
requirements
that
I
need
for
my
application,
such
as
the
capacity
then
the
cluster
administrator's
job
is
to
create
persistent
volumes
for
these
specific
volumes
that
are
available
in
their
cluster.
In
the
case
of
local
storage,
that
information
would
include
the
node
where
the
local
storage
is,
and
also
the
paths
that
you
can
access
that
local
storage
on.
B
B
It
will
just
work
so
now,
on
the
administrator
side,
this
is
where
I
define
the
specific
storage
that
is
available
in
this
specific
cluster.
In
this
case,
I
have
to
specify
the
storage
class
that
this
persistent
volume
belongs
to
I
specified
that
this
volume
is
a
local
volume
and
you
can
access
that
disk.
A
certain
path
and
then
there's
this
third
new
argument
to
persistent
volumes
called
node
affinity.
B
So
don't
worry
about
the
really
long
field
names,
just
think
of
node
affinity
as
a
node
selector,
and
we
are
basically
saying
we
want
this
volume
to
only
be
accessible
on
nodes
that
match
the
label
and
key
matching
that
I
have
expressed
here.
So
in
this
particular
example,
I'm
saying
this
volume
can
only
be
accessed
on
a
node
that
has
a
label
with
the
key
kubernetes,
the
I/o,
slash
hostname,
with
the
value
node
1,
and
actually
because
we're
using
labels.
Here,
this
node
affinity
can
be
expanded
to
support
more
than
just
local
volumes.
B
It
can
express
any
arbitrary
topology
that
your
volume
might
have.
So
if
you
created
some
crazy
volume
that
is
only
accessible
on
three
nodes,
you
can
specify
all
three
notes
here:
you
can,
instead
of
using
hostname,
you
can,
you
know,
use
key
equals
rack
in
rack,
1
or
rack
two.
You
know
this
is
a
very
powerful
notion
that
can
be
supported
and
beyond
local
volumes.
B
So,
to
summarize
what
we
have
implemented
so
far,
we
first
introduced
this
feature
in
1:7
and
in
1:7.
What
you
are
able
to
do
is
you're
able
to
create
a
local,
persistent
volume
type
and
specify
a
node
affinity
for
it,
and
the
scheduler
is
smart
enough
to
read
that
node
affinity
on
your
persistent
volume
and
schedule
your
pod
to
the
correct
node
in
1:9.
We
made
further
improvements
to
the
scheduler
so
that
the
persistent
volume
claim
binding
also
happens
during
pod
scheduling.
B
So
what
happened
before
is
that
the
persistent
volume
binding
would
happen
independently
from
when
your
pod
is
being
scheduled.
So
it's
very
possible.
The
binder
could
have
chosen
a
volume
that
can't
actually
run
your
pod
or
it
could
have
chosen
a
volume
on
a
node
that
couldn't
actually
run
your
pod,
but
now
in
1:9
we
improved
that
a
lot
better.
So
now
the
volume
binding
decision
will
also
take
into
account
your
pods
CPU
and
memory
resources,
node
affinity
policies
and
pod,
anti
affinity
and
affinity,
node,
taints
and
selectors,
and
so
on
and
so
on.
B
B
A
So,
in
addition,
whoa
speaker
to
the
local
persistent
volumes,
you
know
it
might
be
that
you
need
to
keep
it
local.
It's
not
like
you
work
for
the
NSA
and
you
put
all
the
top-secret
data
out
on
an
s3
volume
for
the
world,
so
in
addition
to
having
local
the
the
creme
de
la
creme
is
really
having
local
raw.
So
a
lot
of
applications
can
actually
use.
Have
the
performance
benefits
to
do
this,
though?
It's
not
completely
included
in
the
1.9.
A
We
had
some
specific
goals
to
expose
rob
block
devices
not
as
mounts
but
actual
devices
within
kubernetes
enable
durable
access.
So
what
happens
when
the
lights
go
out,
to
make
sure
that
our
data
is
there
and
it's
secure,
provide
flexibility
for
users
and
vendors
to
support
all
storage
types
prior
to
1.9,
all
volumes
that
were
created
in
kubernetes,
whereas
file
system
was
always
applied
to
them,
even
if
they
were,
you
know,
started
off
as
a
block
device.
It
ended
up
as
a
file
system.
So
there
was
a
lot
of
work.
A
I,
don't
know
if
you
were
able
to
make
mitzi
heroes.
Talk
at
2:00
o'clock
talked
about
all
the
work
that
had
to
be
done
to
enable
that
to
happen
within
the
flow,
and
the
last
goal
really
was
to
break
github
right,
Michelle
yeah.
So
if
you
ever
wonder
how
exhausted
the
community
is
in
kubernetes
to
make
sure
that
we
get
things
right,
we
have
an
incredible
amount
of
conversation.
We
started
with
in
a
Google
Doc,
eventually
moved
it
to
a
PR
and
eventually
broke
github.
A
There
were
so
many
comments
within
it,
but
storage
is
nothing
new.
A
lot
of
the
problems
that
we're
solving,
though
unique
to
kubernetes,
are
still
important
and
so
feedback
is
always
requested.
So
I
wanted
to
just
quickly
go
through.
So
we
have
time
to
see
Michelle's
demo
what
these
API
changes.
We
had
to
do
for
block
to
be
able
to
support
that
so
mm-hmm.
A
Now
the
users
also
put
volume
mode
block.
This
allows
us
to
be
very
intentional
about
how
we
want
to
consume
the
data
in
not
and
to
make
sure
that
I've
asked
for
block
I
expect
block
to
be
considered
on
the
back
end.
The
other
changes
you'll
see
in
the
far
right
corner
is
the
volume
devices
if
you're
familiar
with
pods
and
setting
up
your
storage
before,
if
you're
using
a
file
system,
you
would
put
that
it's
a
volume
mount
if
you're
using
a
block
device.
A
B
So
I'm
going
to
demo
and
hopefully
show
you
how
easy
it
is
now
to
switch
between
remote
and
local
storage.
Now
that
you
can
use
both
of
them
with
the
same
persistent
volume
claim
objects
that
you
use
today,
I've
taken
the
basic
replicated
my
sequel
example
straight
from
the
kubernetes
website
and
I
will
go
ahead
and
load
up
my
cluster.
Now.
Let's
see
this
is
hard
because
I
can't
I
can
only
see.
B
B
Let's
do
that
all
right,
so
let's
take
a
look:
okay,
I've
oak,
an
you
see
it!
It's
like
that's,
okay,
it's
just
one
character
off
the
edge.
It
won't
hurt
anything
so
I
have
my
two
files
here:
the
first
file,
one
file
for
specifying
remote
storage
and
the
second
file
for
specifying
local
storage.
Let's
take
a
look
at
my
remote
file
of
my
stateful
set.
Can
you
see
the
colors
okay
now
on
the
back?
Oh
well,
I'll
upload
this
on
youtube.
B
Alright,
so
you
can
see
I've
named
my
stateful
set.
My
sequel
and
I
have
specified
three
replicas
for
the
stateful
set
and
if
we
look
at
the
bottom
of
the
screen
here,
this
is
where
I
define
my
volume
claims
in
the
staple
set,
and
you
notice
here
that
I
specified
my
storage
cost
name
to
be
standard
and
then
I've
specified
I
want
360
gigs,
so
standard
in
my
cluster
refers
to
a
GCE,
remote
storage
and
I've
already
deployed
this
cluster
or
sorry.
Alright,
I've
already
deployed
the
stateful
set.
So
let's
just
take
a
look
quickly.
B
Okay,
so
I
have
three
replicas
running
and
if
we
look
at
the
volumes
that
those
replicas
use
you'll
see
that
there
are
three
persistent
volume
claims
one
for
each
replicas.
You
can
see
that
these
persistent
volume
claims
use
the
storage
class
standard
and
if
we
look
at
the
details
of
one
of
the
volumes
that
the
claim
is
bound
to,
let's
take
a
look
pick.
One.
B
B
Okay,
here
we
go,
this
looks
more
like
a
DIF,
so
you
can
see
I've
made
some
cosmetic
changes.
I
just
changed
the
name
to
for
the
local
one
to
call
it
my
sequel,
faster
other
than
the
cosmetic
name
change.
The
most
important
change
I've
made
is
I
changed
a
storage
class
name
for
the
remote
spec,
I
use
a
storage
class
name
called
standard,
and
now
for
my
local
spec
I'm
using
the
storage
class
name
called
local
scuzzy,
that's
the
only
change
I've
made
to
this
stateful
set
between
the
remote
and
local
storage.
B
Okay
and
let's
go
ahead
and
create
it
so
you'll
see
that
the
safest
controller
has
created
the
first
pod
in
the
stateful
set
on
the
left,
and
it's
also
created
the
first
persisted
volume
claim,
which
immediately
got
bound
to
a
local
persistent
volume
on
the
system.
And
now
the
volumes
are
getting
mounted
in
the
first
pod
and
we're
waiting
for
them
to
become
running
now.
The
containers
are
running
so
the
stateful
second
chiller
creates
the
second
pod
and
the
second
persistent
volume
claim
which
gets
bound.
B
B
All
right
over
came
up
it
like
I,
didn't
even
see
it.
Okay,
it
brought
up
this
Oh.
So
even
it's
now
on
the
third
replica
now
and
we
created
the
third
persistent
volume
claim
and
Oh
everything's
up.
Okay,
that
was
fast
faster
than
I
could
talk.
So
if
we
look
at,
let's
summarize
all
this,
if
we
look
at
all
the
pods
on
the
system.
B
B
B
We'll
see
that
this
is
a
local
volume,
it
has
this
path,
mount
disks,
SSD
one
it's
in
the
storage
class
name,
local
scuzzy,
and
then
because
this
is
an
alpha
feature,
the
notification
it
field
has
to
be
specified
in
an
annotation
which
is
kind
of
a
pain,
but
it
will
become
filled
in
beta,
but
here
you
can
kind
of
make
it
out.
This
is
where
we
specify
the
label
kubernetes
the
I/o
host
name
in
values,
and
then
this
is
the
name
of
my
node.
B
So
there
you
can
see.
That's
so
that's
the
demo
there
you
can
see
how
easy
it
is
to
now
switch
between
local
and
remote
storage,
just
by
changing
the
storage
class.
This
is
a
huge
improvement
from
what
you
had
to
do
before,
which
is
you
have
to
use
a
completely
different
method.
You
couldn't
even
use
persistent
volume
claims.
You
have
to
write
your
own
schedulers,
your
own
operators,
your
own,
all
this
extra
infrastructure
just
to
use
local
storage,
and
now
you
no
longer
have
to
do
that.
B
Where's,
my
mouse
all
right
all
right
so
now.
I
would
also
encourage
you
to
try
it
out
yourself.
There's
a
user
guide
at
the
end
of
this
presentation
that
will
show
you
how
to
bring
up
and
enable
this
alpha
feature
and
to
create
some
local
volumes.
You
know
take
existing
stateful
set
examples.
You
have
or
any
home
charts.
You
may
have
just
change
the
storage
class
name
to
use
a
local
local
storage
class,
and
then
you
know
just
experiment
and
give
us
feedback.
A
All
right,
I
think
Michelle
already
gave
a
really
good
summary
about
why
local
persistent
volumes
are
important
and
the
work
that's
been
done
to
have
no
to
infinity
and
smarter
scheduling
and
also
how
we're
going
to
be
provisioning
wrong
block
volumes
as
part
of
the
API,
the
only
plugin
that
made
it
into
one,
not
nine,
is
for
fibre
channel,
but
there's
a
bunch
more
on
the
way
you
know
in
all
of
these
are
building
blocks
to
allow
us
to
fully
enable
stateful
distributed
workloads.
So
in
the
future,
there's
still
a
lot
of
work
to
do.
A
There
is
a
ground
work
already
set
forth,
as
Michelle
mentioned
as
part
of
the
topology
for
dynamically
provisioning
local
volumes.
We
also
need
to
have
dynamic
provisioners
for
Rob
lock
volumes
for
these
really
to
become
fully
utilizable
and
scalable
within
the
environment
and
then
the
plugins.
These
are
just
some
of
the
plugins
that
we're
looking
at
updating
with
this
support
and
then,
lastly,
the
CSI
interface
I,
don't
know
if
you
guys
have
heard
of
that,
it's
the
buzz
around
storage
also
will
need
to
be
updated
for
lot
of
devices.