►
From YouTube: Keynote: Debunking the Myth: Kubernetes Storage is Hard - Saad Ali, Senior Software Engineer, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Keynote: Debunking the Myth: Kubernetes Storage is Hard - Saad Ali, Senior Software Engineer, Google
We often hear “Storage on Kubernetes is hard” or “Don’t deploy stateful applications on Kubernetes!” Is there any truth to these statements? In this talk I separate fact from fiction.
Storage on Kubernetes can mean a lot of things. First I will start by separating three layers: 1) consuming block and file storage from a pod running in Kubernetes, 2) deploying a stateful app (like a database) on Kubernetes using storage and workload primitives, and 3) deploying a software defined storage system on top of Kubernetes to provide block or file storage. Each of these areas has a different set of challenges that are often conflated to make storage on Kubernetes appear much harder than it is.
After separating these layers, I will help you understand what the challenges are with each of them, and how you work around them to effectively deploy stateful applications on Kubernetes.
https://sched.co/MQhi
A
All
right
good
morning,
everyone
I'm
so
excited
to
be
here.
My
name
is
Sally.
I
am
an
engineer
at
Google
and
co-chair
of
sig
storage.
I'm
really
really
excited
to
be
here
in
Barcelona
Barcelona
and
see
all
of
you
today.
I
want
to
talk
about
a
topic
that
is
near
and
dear
to
my
heart,
kubernetes
storage.
A
A
But
time
and
time
again
the
question
arises:
what
about
stateful
workloads
is
kubernetes
a
good
place
to
run
stateful
workloads,
there
still
uncertainty,
and
the
fact
is
that
containers
are
inherently
stateless.
Anything
that
you
write
into
your
container
file
system
is
going
to
be
gone
as
soon
as
that
container
is
terminated,
but
for
your
stateless
apps,
you
need
your
bits
to
not
disappear
when
the
container
dies.
So
what
are
your
options?
If
you
go
looking
for
advice,
online,
you're
gonna
run
into
a
lot
of
misinformation,
specifically
you're
gonna
hear
a
couple
of
tropes.
A
So,
while
I'm
going
to
try
to
separate
fact
from
fiction,
I,
don't
want
to
sugarcoat
things.
I
like
this
quote
from
Tim
Hawkins
he's
one
of
the
best
engineers
I
know,
and
he
says
that
storage
is
hard,
maybe
the
hardest
of
infrastructure
problems,
and
the
fact
is
that
storage
independent
of
kubernetes
is
a
complicated
problem.
Storage
can
encompass
a
lot
of
things.
A
Everything
from
consuming
block
and
file
storage
from
a
pod
running
on
kubernetes,
to
deploying
a
database
on
kubernetes
using
kubernetes
primitives
to
deploying
a
software-defined
storage
system
on
top
of
KU
Burnett
and
much
much
more
part
of
the
problem
is
that
there
is
a
misunderstanding
of
what
kubernetes
is
responsible
for
what
it's
capable
of
here's
another
quote
that
I
like
from
the
great
Kelsey
Hightower.
Basically,
what
it
boils
down
to
is
sprinkling
kubernetes
on
hard
problems,
won't
make
them
go
away,
and
this
is
especially
true
for
storage.
A
A
A
A
A
So,
while
kubernetes
doesn't
explicitly
handle
each
of
these
problems,
it
does
introduce
abstraction
layers
that
actually
enable
you
to
handle
these
as
separate
problems,
and
in
fact
you
could
handle
them.
You
could
have
different
sub
teams
within
your
organization,
handle
them
a
cluster
administrator
versus
an
application
developer
prior
to
kubernetes.
All
of
these
problems
were
conflated
together
because
it
was
very
difficult
to
handle
them
independently.
A
For
example,
if
your
app
was
consuming
say
a
block
storage
from
a
software-defined
storage
system,
and
you
wanted
to
switch
it
over
to
a
block
system,
a
block
storage
from
an
external
appliance,
it
would
require
you
to
either
modify
your
app
or,
at
the
very
least,
the
deployment
scripts
for
your
app
that
are
responsible
for
provisioning
storage
and
attaching
it
and
mounting
it.
In
all
these
things
with
kubernetes,
we
have
new
abstractions
that
enable
you
to
handle
these
piece
by
piece.
A
So
let's
walk
through
this,
so
let's
say
you
have
a
stateful
app.
The
first
question
is:
where
do
you
persist?
The
bits
you
have
a
lot
of
options
here
are
some
of
the
places
your
applications
can
store
bits.
Each
one
of
these
categories
is
optimized
for
different
types
of
data,
different
access
patterns
and,
in
addition,
each
one
of
these
categories
has
a
lot
of
options
and,
of
course,
this
is
not
an
exhaustive
list.
A
This
is
the
most
challenging
part
you
have
to
you
have
to
you
have
to
choose
and
the
way
that
you
do,
that
is,
first
by
understanding
what
options
are
available
to
you
and
what
the
differences
between
them
are.
You
have
to
educate
yourself
next,
make
sure
to
understand
what
the
requirements
for
your
application
are.
What
are
the
what's
the
kind
of
data
that
you're
going
to
be
storing?
How
are
you
going
to
be
accessing
it?
Where
will
you
be
accessing
it
from
all?
A
A
A
So
now
you've
selected
your
storage.
The
next
step
is
how
do
you
actually
deploy
it?
First,
let's
divide
storage
into
managed
and
unmanaged
solutions.
A
managed
solution
is
one
run
by
somebody
else,
for
example,
a
cloud
provider
on
Google.
We
have
cloud
sequel
as
a
managed
database.
Gc
persistent
disks
has
managed
block
storage
with
managed
solutions.
You
don't
have
to
worry
about
how
you
deploy
or
manage
the
storage
system,
because
somebody
else
is
taking
care
of
that
for
you
and
it's
perfectly
okay
to
do
this
with
kubernetes.
A
If
you
don't
depend
on
a
managed
solution,
then
you're
responsible
for
deploying
and
managing
the
storage
yourself,
and
you
have
a
few
options
again.
You
may
choose
to
deploy
your
storage
within
kubernetes
or
outside
of
kubernetes,
and
this
is
one
of
the
keys
that
I
want
you
to
take
away
from.
This
talk
is
that
the
deployment
and
consumption
of
storage
are
independent
problems,
so
it's
perfectly
OK
to
deploy
your
storage
outside
of
kubernetes
while
you
consume
it
within
kubernetes
for
block
and
file.
A
For
example,
you
may
have
a
dedicated
external
storage
appliance
that
you're
using
for
your
storage
and
that's
ok
for
data
services.
You
may
have
a
legacy
database,
that's
already
running
on
some
VMs
and
it's
ok
to
leave
them
there.
You
don't
have
to
deploy
your
storage
on
kubernetes
in
order
to
use
it,
but
you
may
choose
to
deploy
your
storage
on
top
of
kubernetes
and
if
you
do,
then
it
becomes
just
another
stateful
application
that
you're
deploying
on
top
of
kubernetes.
A
Yesterday,
David
Zhu
and
yon
Scheffer
Nick
from
Red
Hat
gave
a
great
toriel
about
deploying
stateful
workloads
on
kubernetes
check
that
out,
if
you're
interested
in
the
details
of
exactly
how
you
deploy
stateful
applications.
But
the
key
here
is
that
when
you
deploy
storage
on
kubernetes,
it's
just
like
deploying
any
other
storage,
any
other
stateful
application
on
kubernetes.
A
Now
stateful
apps,
like
databases,
can
get
fairly
complicated
and
they
can
have
very
complicated
life
cycles.
For
example,
a
database
may
have
custom
scale
out
logic
that
replicates
the
data
and
may
have
special
steps
that
it
needs
when
it
needs
to
be
upgraded.
Kubernetes
doesn't
know
how
to
handle
that
brian
was
talking
about
how
kubernetes
is
not
the
end.
A
Kubernetes
gives
you
the
primitives
that
you
can
use
to
build
higher
level
functionality,
and
this
is
where
kubernetes
operators
come
in
kubernetes
operators
are
CR,
DS,
custom,
resource
definitions
and
controllers
that
manage
the
deployment
and
lifecycle
of
your
application.
So
if
you
have
a
sufficiently
complicated
application,
which
is
especially
true
for
a
lot
of
stateful
workloads,
what
you
want
to
do
is
look
for
an
operator
for
the
type
of
stateful
app
that
you're
deploying
to
make
it
easier
to
deploy
and
manage
that
application.
A
A
A
We
have
to
remember
that
you
can
choose
to
deploy
your
software-defined
storage
system
on
top
of
kubernetes
or
you
could
choose
to
deploy
it
outside
of
kubernetes
regardless.
You
can
consume
that
storage
within
kubernetes,
without
issue
guard
list
of
where
you
deploy
it
and
like
any
other
storage
or
stateful
application.
If
you
do
choose
to
deploy
your
software-defined
storage
on
top
of
kubernetes,
it's
just
another
stateful
app,
which
means
that
you're
probably
going
to
deploy
it
using
an
operator.
A
All
right,
so
one
Operator-
that
is
a
great
example
of
this-
is
rook
rook
handles
Ceph
as
well
as
a
handful
of
other
storage
systems.
Saif
is
a
software-defined
storage
system
that
lets
you
consume
local
disks
in
exposed
block
or
file
storage.
To
find
out
more
about
rook
check
out
a
talk
this
afternoon
by
jared
watts.
He
actually
has
a
number
of
talks
on
rook
and
you
can
find
out
all
about
all
about
rook
through
those
talks
so
once
you've
deployed
your
storage.
A
The
next
challenge
is:
how
do
you
actually
make
it
available
within
your
kubernetes
cluster?
Now
before
we
talk
about
integration,
let's
simplify
our
storage
options.
This
is
a
lot
to
take
in
we've
been
throwing
around
a
lot
of
buzzwords.
I
talked
about
a
lot
of
example,
store
example,
categories
before:
let's
divide
them
into
two
areas:
data
services
and
block
and
file
storage.
A
By
doing
so,
a
pattern
begins
to
emerge.
Your
stateful
applications
may
depend
on
data
services
or
directly
on
block
or
file
storage.
Your
data
services,
your
data
services
in
turn,
may
themselves
depend
on
another
data
service
or
they
may
depend
on
block
or
file,
but
eventually
somewhere.
Something
needs
to
write
to
disk.
A
Something
needs
to
be
able
to
persist.
Those
bits
to
a
physical,
medium
may
be
flash
or
a
spinning
disk,
and
that's
what
makes
block
and
file
the
turtle
at
the
bottom.
It's
where
the
rubber
meets
the
road
or
where
the
bits
land
on
the
disk.
So
with
that
clarification,
let's
move
on
your
storage
system
is
now
up
and
running.
It
might
be
running
in
kubernetes.
It
might
be
deployed
outside
of
kubernetes,
either
way
it's
okay.
Now,
how
do
you
actually
integrate
that
storage
into
your
kubernetes
cluster?
A
Let's
talk
about
data
services,
first
data
services,
whether
they're
managed
by
you
or
managed
by
somebody
else.
Your
app
has
to
logic
to
be
able
to
discover
the
API
endpoints.
It
has
to
be
able
to
negotiate
off.
You
have
to
have
basically
some
sort
of
SDK
within
your
application.
That
knows
how
to
interact
and
discover
this
storage
system
for
block
and
file.
This
is
where
kubernetes
really
begins
to
shine
for
a
block
and
file.
Integration
is
a
cluster
level.
Operation
kubernetes
supports
an
extension
mechanism
called
the
container
storage
interface.
A
Csi
enables
kubernetes
to
integrate
any
block
or
file
storage
system
with
kubernetes
and
to
integrate.
All
you
have
to
do
is
deploy
a
CSI
driver
on
your
kubernetes
cluster.
A
CSI
driver
is
just
another
kubernetes
workload,
so
it
doesn't
really
matter
where
your
storage
was
deployed.
As
long
as
you
have
it
deployed
somewhere,
your
kubernetes
cluster
can
access
it
you're
good
to
go
so.
A
Specifically
with
block
and
file,
this
also
highlights
another
advantage
of
kubernetes
portability.
You
can
actually
swap
out
your
block
and
file
storage
system
without
ever
having
to
modify
your
application
so,
for
example,
without
kubernetes,
if
you
decided
to
change
from
one
block
or
file
storage
to
another,
you
had
to
rewrite
your
app
or
change
your
deployment
scripts.
In
order
to
adapt
this
new
new
storage
system
with
kubernetes,
all
you
have
to
do
is
deploy
a
CSI
driver
and
create
a
new
storage
class.
A
Your
app
in
its
deployment
Yam
will
never
change
once
you've
deployed
your
application,
you
have
to
make
it
available
on
your
cluster,
so
this
is
the
last
step
how
our
stateful
applications
actually
going
to
use
the
storage
that
you've
deployed
now
that
it's
integrated
within
your
cluster
for
data
services,
whether
they're,
managed
or
unmanaged
either
your
administrator
has
to
provision
the
storage
or
you
will
have
to
do
it
yourself
as
a
application
developer.
But
your
app
has
to
take
responsibility
for
it.
A
Your
app
has
to
have
the
SDK
or
the
bits
that
are
responsible
for
being
able
to
read
and
write
to
that
data
service
for
block
and
file
storage.
It
becomes
much
easier
with
the
kubernetes
storage
API,
you
have
persistent
volume
claims,
persistent
volumes
and
what
they
allow
you
to
do
is
consume
block
and
file
storage
in
a
portable
manner.
A
Now
I'm
not
going
to
go
into
details
of
what
persistent
volume
and
persistent
volume
claims
and
storage
class
objects
do
or
how
they
work.
There's
a
great
talk
by
David
Zhu
this
afternoon,
storage,
101
I,
recommend
that
you
check
that
out
if
you're
interested
in
the
details.
But
let
me
talk
about
the
high
level
benefits
of
using
kuben
the
kubernetes
storage
API
versus
deploying
without
kubernetes.
If
you
deploy
without
kubernetes.
First,
you
have
to
manually
provision
a
volume
on
your
storage
system.
A
Then
you
have
to
figure
out
how
to
attach
that
volume
to
the
node
that
you
want
to
run
your
workload
on.
Then
you
have
to
mount
that
volume
into
the
container
in
order
to
make
it
available
inside
that
container,
and
we
heard
from
Spotify
and
others
how
your
cluster
may
go
down.
Things
are
necessarily
going
to
fail,
and
when
that
happens,
what
do
you
do?
You're
gonna
have
to
have
some
sort
of
monitoring.
A
That's
going
to
have
to
notice
that
this
your
nodes
are
down,
move
your
workloads
and
then
go
in
and
manually
move
your
storage
over.
This
is
unsustainable.
So
with
kubernetes
everything
is
automatic.
Stuartt
storages
automatically
intelligently
provisioned,
we
figure
out
where
you're
the
best
place
in
the
cluster
is
to
provision
not
just
your
workload,
but
also
your
storage.
Your
workload
is
intelligently
scheduled
based
on
that
storage.
A
So,
let's
wrap
things
up.
The
reality,
of
course,
is
that
storage
is
complicated
and
the
way
that
we
make
complicated
things
more
manageable
is
by
breaking
them
down
into
smaller
problems
for
storage.
We
are
able
to
break
break
it
down
into
four
pieces,
selecting
your
storage,
deploying
that
storage,
integrating
it
with
your
kubernetes
cluster
and
then
consuming
it.
While
kubernetes
doesn't
solve
everything,
it
does
introduce
new
abstractions
that
let
you
handle
these
problems
piece
by
piece.
It
allows
your
storage
administrators
to
focus
on
storage
deployment.
A
It
lets
your
cluster
administrators
figure
out
how
to
integrate
with
that
storage
system.
It
lets
your
application
developers
focus
on
consuming
that
storage
from
their
application.
The
separation
of
responsibilities
makes
kubernetes,
makes
storage
manageable
within
kubernetes,
and
that
is
all
thank
you
very
much
for
your
time.