►
From YouTube: Persistent Storage with Kubernetes in Production - Which Solution and Why? [I] - Cheryl Hung
Description
Persistent Storage with Kubernetes in Production - Which Solution and Why? [I] - Cheryl Hung, StorageOS
Persistent storage often seems like a confusing plethora of options, from local volumes, NFS, distributed storage like Ceph, cloud storage such as AWS’s EBS and S3, to volume plugins with Docker and Kubernetes integration. This talk compares and contrasts the most popular solutions, and lays out the eight principles for cloud native storage.
About Cheryl Hung
Cheryl is an ex-Google software engineer with a passion for developer tools, experience and community. She founded the Cloud Native London meetup and codes, writes and speaks about containers, storage and cloud computing.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
If
you
want
to
follow
along
on
your
laptop
I,
also
tweet
it
or
show
so
tell
me
a
little
bit
about
you
guys
so
who
here
is
new
ish
to
kubernetes,
like
half
problem
of
the
audience,
how
many
people
running
kubernetes
in
some
dev
or
test
environment,
but
not
in
production?
Yet
ok,
so
it's
2/3,
probably
how
many
people
running
just
the
stateless
parts
of
their
application
in
production
with
kubernetes?
A
Ok,
so
maybe
a
quarter
and
how
many
people
are
running
everything,
including
all
the
staple
parts
in
production.
Ok,
it's
about
a
quarter
as
well
cool,
so
the
object
of
this
talk
is
to
answer
the
following
three
questions.
So
why
is
state
so
tricky
in
the
first
place?
How
do
I
go
about
comparing
my
storage
options
and
then
what
storage
options
should
I
look
at
with
kubernetes
and
what
are
things
to
look
out
for
plus
one
anti
objective
so
should
I
use
a
database
or
message
queue
or
key
value
store
or
something
else,
my
application?
A
So
we're
just
going
to
talk
about
the
storage
layer
here
so
first
off?
Why
is
state
so
tricky
anyway?
Why
do
I
even
need
storage,
because
probably
everyone's
heard
that
your
container
should
be
stateless?
Your
infrastructure
should
be
stateless
and
immutable
and
you
get
some
really
nice
properties
from
them.
The
problem
is,
there
is
no
such
thing
as
a
stateless
architecture.
So,
assuming
that
you
want
to
do
something
useful
with
your
application,
you
need
to
store
data
somewhere,
which
means
you
need
to
pick
a
storage
solution.
A
So
if
you
either
need
to
think
of
at
yourself
or
you
have
outsourced
that
problem
to
one
of
the
card
providers
or
your
to
your
internal
ops
team,
so,
first
challenge
of
storage,
no
pet
storage:
this
is
the
pet
versus
cat
or
analogy
for
servers.
So
you
shouldn't
treat
your
servers
like
pets
that
you
name
and
you
lovingly
look
after
them
and
when
they
fall
ill,
you
nurse
them
back
to
health
again.
A
Instead,
you
should
treat
them
like
cattle,
so
you
number
them
and
when
they
fall
over
you
take
it
up
back,
you
shoot
it
and
you
plug
in
a
new
one.
So
we're
storage.
You
want
to
avoid
having
special
servers
that
are
your
storage
servers,
because
then
you've
created
pet
servers.
That
now
you
need
to
look
after.
A
Second
challenge
is
data
needs
to
follow,
so
your
containers
are
small
and
lightweight
and
fast
to
move
around
the
cluster.
Hopefully,
and
your
data
needs
to
follow
that
around
wherever
it
goes.
What
you'll
want
to
do
is
you
want
to
avoid
saying
these
containers,
or
these
pods
need
to
live
on
these
exact
nodes,
because
then
you've
tied
that
to
that
particular
node
and
now
you've
lost
the
portability
and
the
mobility?
A
The
third
challenge
is
humans
are
fallible.
So
if
you
are
relying
on
a
human
operator
to
fix
something
by
running
through
a
manual
playbook
by
typing
in
keyboard,
commands
then
they're
going
to
make
mistakes
eventually,
22
percent
of
data
center
outages
are
caused
by
human
error.
So
as
much
as
possible,
you
want
everything
to
be
driven
through
an
API,
not
by
human
typing,
on
a
keyboard.
A
A
What
do
I
mean
by
cloud
native,
so
cloud
native
means
horizontally
scalable.
So
you
add
resources
by
adding
more
nodes
to
a
cluster
rather
than
adding
more
resources
to
a
single
cluster
to
a
single
node.
There
should
be
no
single
point
of
failure,
and
if
there
is
a
failure,
then
the
system
needs
to
be
resilient
and
it
needs
to
self-heal,
which
means
that
everything
needs
to
be
driven
through
api's.
A
So,
let's
go
through
the
eight
principles
of
cloud
native
storage.
First
one
is
that
classically
storage
has
been
presented
to
a
operating
system
so
to
a
single
node,
which
means
that
you're
tied
to
a
specific
instance
of
that.
So
now
we
want
to
tie
storage
to
the
needs
of
the
pod,
not
to
the
instance
to
the
node
that
it's
running
on.
A
Third
principle
is:
if
you're
going
to
be
able
to
move
this
from
environment
to
environment,
those
resources
need
to
be
declared
and
composed,
as
part
of
your
orchestration
or
in
sociation,
and
in
order
to
do
that,
everything
needs
to
be
a
API
driven.
So
you
need
to
be
able
to
provision
your
store
and
API
manage
it
through
an
API
move
it
through
an
API
and
your
developers
need
to
be
able
to
consume
that
storage
via
an
API
as
well.
A
A
Sixth
principle
is
the
whole
goal
behind
for
this
cloud
native
thing
is
for
agility.
We
want
to
move
our
pods
around.
We
want
to
have
kubernetes
orchestrate
those
and
move
those
around
and
the
data
should
move
just
as
easily
and
if
we're
talking
about
distributed
environment
then
this
has
performance
implications.
A
So
for
certain
use
cases
you
need
to
think
about
how
you're
going
to
manage
that
and
similarly
you
need
to
think
about
availability,
reliability,
consistency
and
so
on.
So
those
are
the
eight
principles
of
cloud
native
storage,
and
hopefully
you
can
start
thinking
about
what
storage
features
and
characteristics
are
offered
by
different
platforms.
A
So
this
is
so
far
so
good.
The
problem
with
this
is
that
this
requires
the
admin
to
provision
these
volumes
in
advance,
which
is
not
how
we
think
of
other
resources
like
CPU
or
memory.
So
what
if
you
need
wants
to
just
create
those
volumes
on
demand,
so
the
alternative
is
called
dynamic,
provisioning
and
that's
done
with
storage
classes.
So,
instead
of
creating
the
admin
creating
those
volumes
directly,
the
administrator
will
create
a
couple
of
different
profiles
of
storage
that
they
want
to
provide.
So,
let's
say
fast,
storage
is
always
provisioned
on
SSD.
A
A
A
So
she
goes
to
the
kubernetes
documentation
for
storage
classes
and
goes
Wow.
Ok,
that's
already
quite
a
few
options.
Then
we've
got
to
worry
about
flex
volumes
and
CSI
that
contains
a
storage
interface.
That's
coming
later,
how
do
I
go
about
picking
what
I
need
so
there's
key
information
that
she
needs
to
think
about
for
her
problem
number
one.
What
is
my
use
case
number
two.
What
does
that
mean
for
my
performance
requirements?
A
Number
three,
how
my
developer
is
going
to
access
it
and
the
number
four?
Where
is
that
going
to
be
deployed
and
managed?
So
what
am
I
use
case?
There's?
Obviously,
a
ton
of
use
cases
out
there,
but
the
common
ones
include
storage
for
your
application,
binaries
storage
for
your
data
configuration
files
and
then
backups.
A
A
Configuration
data
is
generally
small,
but
it
doesn't
need
to
be
shared
across
multiple
instances
of
a
pod
and
then
finally
backups
you
don't
need
you
need
it
to
be
highly
available,
but
you
don't
expect
a
high
volume
of
transactions
so
you're
now
balancing
cost
versus
performance
and
you
might
need
to
think
about
backing
up
to
a
cloud
as
well
am
I
developed?
How
will
my
developers
accessing
storage?
A
A
And
finally,
where
is
the
storage
deployed
and
managed?
So
this
is
on
Pam
and
cloud
or
managed
as
well,
but
if
you're
thinking
about
kubernetes
there's
multiple
ways
that
your
storage
system
can
talk
to
your
applications,
so
one
is
directly
another
is
to
go
through
one
of
the
control
pane
interfaces
like
CSI
or
the
doc
of
volume,
driver
interface,
flex
volumes
or
native
volumes,
or
so
on.
There's
quite
a
few
different
frameworks
and
tools
out
there.
An
example
would
be
rex
way
which
allows
multiple
orchestrators
to
work
with
it.
A
A
So
let's
take
a
look
at
database
at
location
credentials.
First,
what
kind
of
data
is
this?
This
is
configured
configuration
data,
so
it's
small.
It's
shared
across
multiple
instances
and
kubernetes
provide
some
primitives
for
this
already
secrets
and
config
map.
So
this
is
already
tightly
integrated
with
kubernetes.
A
The
second
one
is
user,
uploaded
media,
so
these
are
large
blobs
of
data,
but
they're
not
frequently
written
to,
and
these
also
need
to
be
shared
across
multiple
instances
of
pods.
So
this
is
actually
a
very
good
use
case
for
a
shared
file
system.
There's
cloud
options
such
as
managed,
NFS
and
I
would
be
slightly
wary
about
AWS
ZFS
here,
I,
don't
know
if
anyone
has
any
experience
with
EFS.
Does
anyone
use
EFS
here
a
few
like
10
15
people,
so
there
can
be
some
performance
issues
with
it.
I
see
some
nodding
there.
A
This
is
also
a
good
for
object
store
if
your
application
can
be
written
to
use,
object
stores
and
what
about
on-prem.
So
in
a
non-home
world
there
are
some
distributed
file
system
options.
Cfs
is
good
one.
Cluster
FS
is
another
good
one,
they're,
both
open
source
products
from
Red,
Hat
I
put,
please
not
NFS,
so
who
here
is
using
NFS?
A
Shame
on
you,
so
the
reason
I
don't
like
NFS
is
so
NFS.
Is
your
your
classic
file
server
that
you
exports
over
the
network
to
number
of
clients?
But
of
course,
your
file
server
then
becomes
that
special
pet
server
that
you
need
to
look
after,
and
you
need
to
make
sure
that
that
is
always
special
on
that
server.
So
if
we're
in
a
communities
pod
cloud
native
world,
then
NFS
is
actually
not
a
very
good
option
for
that.
A
A
This
is
a
good
option
for
object,
store,
so
managed
objects,
store,
I'm
guessing
who
uses
s3,
probably
half
the
room
in
here
so
yeah.
This
is
actually
quite
a
good
option
for
s3.
There
are
specific
options
for
this
use
case
like
Glacia.
Aws
is
Glacia
for
long
term,
cold
storage.
As
long
as
you
don't
need
to
access
it
very
frequently
because
it
gets
expensive
if
you're
actually
accessing
it
frequently
unfair
options.
There's
also
actually
object
stores
which
are
s
to
be
compatible,
but
you
can
run
them
on
them.
A
Finally,
come
to
the
Postgres
database:
this
is
probably
the
one
that
is
the
most
difficult,
because
you
care
about
availability
and
latency
and
having
a
deterministic
of
performance.
The
developers
are
accessing
it
via
database
connector,
but
remember
what
we've
mentioned
earlier
about
databases
running
best
on
block
storage.
So
if
you
were
looking
for
a
Postgres
option,
then
you'd
want
to
look
for
something
that
runs
on
block
storage.
A
A
A
Okay,
but
a
quarter
so
I
put
limited
offerings
here
because
actually
managed
databases
are
really
convenient.
They
have
all
these
high
availability
features
it
sort
of
built
in
for
you
as
long
as
you
don't
need
any
kind
of
custom
configuration
or
you
don't
need
a
specific
version
of
a
database.
That's
not
offered
or
you
have
a
specific
database.
That's
not
offered
I
mean
Postgres
is
quite
widely
used,
but
my
DB,
if
your
favorite
Kyle
provider
doesn't
offer
it,
then
you
may
not
have
this
option?
A
A
A
Create
that
demon
said
then
I'm
expecting
to
see
just
one
pod,
so
that's
going
to
create
itself
in
a
little
bit
and
by
the
way,
if
you
haven't
used
quatre
Koda
before
it's
quite
neat,
it's
this
sort
of
in
Bowser,
tutorials
and
the
kubernetes
stocks
youtube.
So
it's
quite
fun
and
it
means
that
you
can
go
and
try
this
yourself
as
well
later.
If
you
want
to
run
through
it
again,.
A
So
it
will
discover
any
block,
storage
and
present
that,
through
the
storage
pore
and
create
you
can
create
virtual
volumes
from
that
storage
pore
and
mount
them
to
containers.
The
nice
thing
about
that
is
now
the
containers
or
the
pods
can
move
around
the
cluster
onto
different
nodes
and
it
they
don't
need
to
be
aware
of
whether
they're
accessing
local
storage
or
storage
on
another
node
in
the
cluster,
because
it's
all
managed
through
the
storage
pool
that
spans
the
entire
cluster.
A
A
A
A
A
A
A
A
Think
about
your
storage
needs
and
use
those
eight
principles
of
cloud
native
storage
to
understand
whether
storage
is
a
good
fit
in
this
model.
Ask
yourself:
what
is
your
use
case?
What
does
that
mean
for
my
performance
requirements,
how
my
developers
gonna
access
it
and
then
how?
Where
is
that
going
to
be
deployed
and
managed?
A
So
what's
coming
next,
so
CSI
I
mentioned
before
is
container
storage
interface.
It
was
launched
as
an
alpha
when
kubernetes
1.9
and
the
goal
is
to
have
an
interface
that
vendors
can
write,
compatible,
plug-ins
against
and
now
users
will
be
able
to
move
between
different
vendors
more
easily.
So
that's
going
to
be
coming
soon,
a
bunch
of
resources.
You
can
try
this
demo
out
yourself.
We
have
a
QuickStart
kubernetes
guide
and
also
we're
hiring.
A
A
So
we're
quite
different
from
vexman.
Isn't
that
let
me
come
out
to
this
diagram,
so
Rex
Way
fits
in
this
frameworks
and
tools,
section
it's
a
connector
between,
say
the
kubernetes
api
is
and
other
ones.
Storage.
Os
is
more
on
the
storage
systems.
Side
of
this
diagram
as
to
how
we
compare
with
port
works
I.
Consider
it
not
good
form
to
necessarily
talk
about
competitors
in
that
way.
So
I
might
just
avoid
asking
that
question.
That's
okay,
hey.
A
B
A
C
A
A
A
C
A
A
Yeah,
so
we
have
we're
going
to
have
some
more
tools
and
functionality
around
that
yeah,
since
we've
got
a
minute
and
actually
I
think
this
is
quite
interesting,
so
I'm
gonna
add
one
more
h2
slide,
which
is
past
this
about
how
storage
OS
works.
So
the
model
that
we
have
is
not
it's
a
sort
of
hybrid
master,
replicas
model,
so
the
entire
volume
is
replicated
on
two
different
nodes,
as
opposed
to
like
a
distributed
file
system
which
shards
the
data
across
nodes.
A
Instead,
the
whole
volume
is
replicated,
and
that
means
that,
for
example,
this
one
this
is
a
five
node
cluster,
with
one
master
volume
on
node
one
and
then
two
replicas
that
are
spread
onto
different
nodes.
That
means
that
if
node
one
goes
down,
then
we
can
just
instantly
switch
from
one
of
the
replicas
to
become
the
master,
so
there's
no
fail
over
time
compared
to
a
distributed
solution.
A
Where,
because
you've
shattered
that
data
across
different
nodes,
you
now
need
to
read
and
byte
to
reconstruct
it,
and
then
the
other
nice
thing
is
that
in
a
distributed
system
you
get
that
fan
out
of
light.
So
one
light
that
comes
in
generates
a
lot
of
lights
on
the
packet,
and
this
is
synchronous
replication.
So
one
write
that
comes
in
will
just
write
two
in
parallel
to
any
replicas
when
they
act
back,
then
that's
marked
as
us
as
a
success.
D
Of
a
call
and
then
question
you
had
a
comment
about
not
using
NFS
for
on
pram
and
duty.
You
said
that
your
reasoning
was
because
it's
more
of
a
pest
on
cattle
and
my
China
argument
is
that
first
of
all,
NFS
is
just
a
protocol
and
the
NFS
server
itself
can
be
distributed.
Software
define-
and
you
mentioned
that
self.
A
Mean
you
can
put
it
behind
DNS,
but
I
have
something
back
in
my
head
about
why,
anyway,
the
whole
point
about
this
is
not
that
there's
one
right
solution
or
one
wrong
solution.
So
there
are
better
things
for
different
use
cases.
Nfs
I'm,
saying
is
a
is
one
where
you
can.
You
can
distribute
NFS
to
instance,
if
you
add
on
other
products
on
top
of
it
so
base,
NFS
is
just
going
to
be
one
file
server
and
that's
a
pet
server,
not.
D
A
A
Yeah
sorry
I
should
have
mentioned
that.
So
if
the
masternode
goes
down
so
the
question
was:
does
it
do
the
regeneration
of
data
if,
if
one
of
them
goes
down
yeah
again
so
the
master
or
one
of
the
reps,
because
it's
lost,
then
it
will
just
rethink
a
new
replica
in
the
background
on
a
different
node
and
you
can
choose
between
zero
and
five
nodes,
which
gives
you
up
to
four
nodes
that
could
go
down
and
then
still
remain
serviceable
yeah.
A
E
A
E
E
C
B
A
So
the
question
was:
are
there
open
source
alternatives
to
storage
OS?
Not
with
this,
not
with
the
model
that
I
described
that
I
know
of
with
them
the
master,
replicas
so
open
source
options
right
now
that
are
distributed?
Who
would
probably
be
safe
for
Gluster,
but
those
are
distributed
by
shouting
across
nodes?
So
then
you
have
the
fan
outs
so
that
I,
don't
think.
There's
anything
that
I
know
of
this.
Do
any
similar.