►
From YouTube: What You Need to Know Before Using Local Persistent Volumes - Sebastien Guilloux, Elastic
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
What You Need to Know Before Using Local Persistent Volumes - Sebastien Guilloux, Elastic
As Kubernetes matures, it becomes easier to run distributed databases and other stateful workloads on top of it. In order to do so, one important thing to figure out is how to manage persistent data. Local PersistentVolumes, as opposed to network-attached ones, can be particularly tricky to operate at scale. In this talk we will explore various options to make use of local volumes, along with a few important gotchas to be aware of. Come learn about static vs. dynamic provisioning, storage-aware scheduling problems, host failures, upgrades, and why your Pods may stay stuck in a Pending state.
A
My
name
is
sebastian.
Most
people
call
me
seb,
so
you
can
just
do
that.
I'm
a
software
engineer
at
elastic,
where
I
mostly
work
on
our
kubernetes
operator
for
the
elastic
stack
that
we
call
eck
as
in
elastic
cloud
on
kubernetes,
so
on
the
agenda
today,
we'll
mostly
talk
about
local
volumes
and
basically
start
with
explaining
how
they
work
and
how
things
are
plugged
together
in
the
kubernetes
world,
then
we'll
mention
how
you
can
provision
those
local
volumes
in
different
ways.
A
So
let's
take
an
example
here
on
the
left
in
that
pink
box,
I'm
deploying
a
stateful
set
with
three
replicas
and
let's
imagine
this
is
an
elasticsearch
cluster,
but
it
could
really
be
any
sort
of
distributed
database
in
this
example
and
in
the
stateful
set.
I'm
I'm
specifying
that
I
want
100
gig
of
storage
per
pods,
so
the
stateful
side
controller
in
turn,
will
create
three
pods,
because
I
I
want
three
replicas.
So
the
first
part
is
gonna,
be
with
ordinal
number
zero.
A
All
you
know
number
one,
and
then
we
have
another
one,
the
last
one
with
ordinal
number
two,
and
because
I
requested
a
claim
of
100
gigabytes.
Each
of
this
part
is
going
to
be
associated
to
a
persistent
volume.
So
a
storage
unit
of
100
gigabytes
and
the
nice
part
about
the
stateful
set
and
the
persistent
volume
concept
is
that
we
have
a
direct
direct
one-to-one
relationship
between
the
pod
and
a
persistent
volume
here.
So,
for
example,
this
part,
my
es
cluster
0,
is
associated
to
this
system
volume
called
data,
my
es
cluster,
zero.
A
So
here
the
stateful
set
controller
would
just
recreate
that
missing
part
for
the
stateful
set
and
that
pod
will
automatically
be
associated
to
the
same
persistent
volume.
So
that's
a
very
nice
way
to
treat
stateful
workloads
because
whenever
you
need
to
like,
for
example,
upgrade
the
version
of
that
database
or
do
a
change
in
the
configuration
you'd,
basically
roll
out
the
pods
one
by
one
and
when
the
restart
those
pods
stay
bound
to
the
same
volume
and
stay
bound
to
the
same
data.
A
So
if
you
get
into
more
details,
you'll
know
kubernetes
is
all
about
general
resources
right.
So
let
let's
take
a
look
at
this
specifications
so
on
the
left
we
had
our
stateful
set
with
three
replicas.
That's
what
we
have
here.
It
has
a
name.
My
es
cluster,
yes
has
an
elastic
such,
but
the
game
could
be
anything
and
I
specify
a
volume
claim
template
so
that
each
of
the
pods
of
the
same
full
set
has
a
particular
persistent
volume
with
here
100
gigabytes.
A
In
turn,
the
stateful
set
controller
reads
that
resource
and
is
going
to
create
for
each
pod.
The
pod
itself,
of
course,
along
with
an
associated
persistent
volume
claim.
So
we
have
the
butt
the
pod
here
in
the
top,
and
the
persistent
volume
claim
here
at
the
bottom
and
really
the
claim
itself
is
not
really
the
volume.
Yet
it's
just.
It
just
expresses
the
desire
for
the
pods
to
acquire
a
storage
unit
which
is
going
to
be
the
persistent
volume.
A
So
at
this
point
we
have
a
pod.
We
have
a
claim,
and
the
pod
is
basically
waiting
for
that
claim
to
be
bound
to
a
real
verb,
and
you
can
see
that
there's
a
very
nice
naming
convention
in
in
in
all
that
system,
so
the
stateful
set
here
that
is
called
my
es
cluster
determines
the
name
of
the
bot,
so
the
first
part
is
going
to
be
my
es.
Cluster
0
and
the
persistent
volume
claim
is
going
to
reuse
that
name
of
the
card.
He
has
cluster
0
that
prepared
to
it's.
A
A
So
now
we
have
system
volume
on
one
hand,
system
volume
claim,
on
the
other
hand,
and
the
volume
was
created,
especially
for
that
claim,
and
the
next
step
is
for
another
controller
in
kubernetes,
which
is
called
the
pvc
controller
to
bind
the
volume
and
the
claim
together.
So
whenever
there's
a
claim,
that
controller
will
look
for
any
available
volume
that
can
match
that
claim,
and
in
our
case
here
in
this
example,
the
volume
was
created
just
to
match
the
claim.
A
So
it's
very
easy
for
that
controller
to
bind
the
two
together
and
again
here,
there's
a
relationship
between
those
two
that
we
can
inspect
in
the
yaml
file
like
here.
We
see
that
in
the
volume
we
have
a
reference
to
a
particular
claim.
So
that's
the
claim
that
was
bound
to
that
volume.
So
it
has
the
same
name
here
as
our
claim
on
the
left,
and
we
also
see
that
the
claim
itself
was
updated
with
the
name
of
the
volume
to
which
it
is.
A
So
there's
always
a
trade-off
between
the
choice
like
you
can
choose
whether
you
want
to
use
network
dash
volumes
or
local
volumes
and
there's
a
big
trade-off,
and
it's
a
lot
of
research
to
do
in
order
to
make
sure
you
understand
the
trade-off
here.
But
basically
it
all
comes
down
to
the
performance.
You
need
from
that
volume.
Of
course,
if
you
use
a
locally
attached,
nvme
ssd
volume
to
the
vm
you're
going
to
get
much
better
performance
than
if
you
mount
a
file
system
using
the
network.
A
Although
the
performance
of
those
network
dash
volumes
is
getting
better
and
better,
it's
also
possible
that
the
pre,
the
price
is
very
different,
like
you
may
get
a
cheaper
volume
by
using
the
local
hard
disk
directly,
rather
than
paying
for
the
units
of
storage
you
use
over
the
network.
But
again
this
depends
on
your
provider.
A
Another
big
difference
is
that
those
local
volume
are
usually
not
there
by
default.
When
you
deploy
kubernetes,
you
probably
have
to
provision
those
yourself
or
to
install
a
provisioner,
so
you
get
access
to
those
local
volumes
and
that's
what
we're
going
to
see
next,
but
really
the
main
difference
between
those
two
is
that
a
local
volume
is
really
bound
to
a
particular
host,
since
it
directly
matches
a
physical
disk
like
that
is
plugged
to
that
virtual
machine.
A
So
how
does
this
this
work
in
practice?
Well,
it's
just
using
the
affinity
mechanism
of
kubernetes
the
same.
You
would
use,
for
example,
to
stick
a
particular
pod
to
a
particular
subset
of
nodes
in
your
kubernetes
cluster.
So
here,
for
example,
in
this
local
system
volume,
we
see
that
there's
a
local
path
defined.
So
that's
really
directory
on
the
file
system
onto
which
this
volume
is
mounted
and
an
affinity.
And
here
this
affinity,
setting
I'm
using
a
gke
cluster
here
specifies
that
this
volume
can
only
exist
on
a
host
with
that
particular
name
here
right.
A
So,
whenever
the
scheduler
schedules,
a
new
pod
on
that
pod
is
associated,
a
persistent
volume
claim,
which
is
stealth,
is
bound
to
persistent
volume.
Then
the
pod
itself
can
only
be
scheduled
on
the
same
host
as
the
persistent
volume.
So
how
can
you
make
use
of
those
volumes
yourself?
You
have
to
provision
them
because
they
are
not
likely
not
just
there
by
default.
A
A
Then
what
I
call
static
provisioning
where
you
run
to
run
a
program
or
an
agent
on
each
host
that
will
automatically
discover
the
disks
that
are
available
on
a
given
host
and
create
the
corresponding
volumes
automatically.
So,
for
example,
if
you
have
a
virtual
machine
with
three
different
hard
disks
attached,
you
could
run
this
tool
to
automatically
create
three
different
volumes
of
the
same
size
as
the
disk
that's
static
provision
I
mean
in
both
those
cases,
manual,
provisioning
and
static
provisioning.
A
You
usually
create
the
volumes
of
the
size
you
want
in
advance
and
then
the
pods
and
stateful
sets
etc.
You
create
can
make
use
of
those
available
volumes,
whereas
the
third
category,
dynamic
provisioning,
allows
the
controller.
So
it's
usually
a
controller
like
you
run
an
act,
an
additional
controller
or
operator
in
your
kubernetes
cluster,
and
this
controller
is
responsible
for
automatically
creating
the
persistent
volumes
on
demand,
depending
on
the
claims
that
were
created.
A
So
let's
take
a
look
at
manual
provisioning
and
how
you
can
create
a
persistent
volume
yourself.
Well,
basically,
it's
just
a
yellow
file.
Like
you
just
create
this
persistent
volume
resource
you
give
it
a
name.
You
specify
its
capacity
and
you
sort
of
indicate
the
path
in
the
file
system
that
you
want
to
use
for
that.
A
So
you
need
to
be
careful
here
that
the
capacity
you
advertise
in
the
spec
technically
doesn't
have
to
match
the
real
capacity
you
have
on
the
file
system
like
you
could
very
well
declare
volume
of
a
headrest
storage
here,
but
behind
the
scenes
that
file
system
is
actually
one
terabyte
of
storage
and
there's
nothing.
That
is
going
to
block
you
from
doing
that
and
nothing
is
going
to
prevent
you
from,
for
example,
using
500
gigabytes
instead
of
the
advertise
100
gigabytes
of
storage,
because
that
file
system
behind
the
scenes
is
much
larger
than
that.
A
Nothing
is
going
to
check
that
size
to
be
careful,
and
of
course
you
want
to
define
the
node
affinity
to
make
sure
that
particular
volume
belongs
to
that
particular
host
and
basically
that's
it
just
apply
that
yaml
file
and
create
that
local
volume
and
this
volume
can
then
be
bound
to
a
claim.
As
soon
as
you
create
the
claim
that
matches
the
phone.
A
So,
as
you
see
that's
very
much
of
like
a
lot
of
manual
work
that
is
required
that
you
can
script,
of
course,
to
create
all
these
volumes
in
advance,
but
what's
maybe
smarter
to
do
if
you
know
that
you
want
to
run
a
particular
stateful
workload
on
your
kubernetes
cluster
is
to
automatically
create
the
volumes
that
correspond
to
the
disks.
You
have
on
each
virtual
machine
and
there's
a
very
useful
tool
maintained
by
the
kubernetes
special
interest
group,
the
storage
special
interest
group
which,
which
is
called
the
local
persistent
volume,
static,
provisioner
and
really.
A
This
is
some
sort
of
demon
set
that
turns
on
your
kubernetes
cluster,
that
is
going
to
inspect
on
each
host
the
partitions
and
disks
that
are
available
on
the
file
system
of
the
host
and,
for
example,
if
that
tool
sees
that
there
are
five
disks
that
it
could
use.
It's
going
to
automatically
create
five
different
persistent
volumes
matching
those
disks.
A
So
you
basically
just
install
the
tool,
and
after
a
few
seconds
you
have
all
the
persistent
volumes
that
are
available
matching
the
disks
you
have.
So
it's
very
useful
if
you
sized
your
machines
and
their
disks
according
to
the
workload
you
expect
to
have
in
the
future
and
finally,
maybe
the
more
advanced
way
to
provision
local
volume
is
to
use
dynamic
provisioning.
A
Lvm
is
actually
very
similar
in
nature,
except
that,
instead
of
provisioning
volumes
based
on
zfs,
it's
going
to
provision
volume
based
on
lvm
and
then
format
this
volume
as
you
desire,
so
both
allow
you
to
kind
of
the
same
thing
right
and
really,
with
this
dynamic
provisioning,
it's
more
about
creating
the
right
volume
of
the
right
size
automatically
whenever
a
user
requests
a
particular
volume
of
a
specific
size.
So
it's
much
more
dynamic
and
you
can
do
much
more
advanced
things
like,
for
example,
on
a
given
host.
A
A
All
right
so
now
that
you
know
a
bit
more
about
what
system
volumes
are
what
local
system
volumes
are,
especially
and
how
you
can
provision
them.
Let's
look
at
how
we
deal
with
operations
and
actually
you'll
understand
why
using
local
system
volumes
can
be
much
more
difficult
to
operate
than
network
attached
problems.
A
So
let's
take
the
host
failure
case.
So
we
have
the
same
example
here
with
a
stateful
set
with
three
replicas
and
say
the
host
here,
node
c,
that
is
holding
the
workload
of
second,
so
this
stud
number,
two
so
really
the
third
replica,
but
ordinal
number
two
suddenly
becomes
pending
like
the
pod,
cannot
be
started
on
that
host
again
and
the
reason
is
that
that
host
is
dead
like
it's
unavailable,
it's
disconnected
from
the
fleet
entirely.
Maybe
the
virtual
machine
was
like
completely
recycled
and
you
cannot
make
use
of
it
anymore.
A
A
So
usually,
when
you
use
stateful
set
for
distributed
workloads
distributed
database,
there's
some
kind
of
replication
system
where
the
different
members
of
that
distributed
workload
are
able
to
replicate
their
data
all
around.
So
here
chances
are,
if
you
configured
it
properly,
that
all
the
data
that
used
to
be
in
that
volume
is
also
replicated
somewhere
else.
On
another
volume
like
we
can
imagine,
we
have
several
chunks
of
data
there
that
are
replicated
to
at
least
another
member
of
the
system
and
that's
fine,
it's
very
useful.
A
Basically,
so
when
that
happens,
you'd
like
the
system
to
sort
of
recover
automatically
right,
that's
what
kubernetes
is
supposed
to
do
like
say
you
run
a
deployment
if
one
pod
is
killed,
kubernetes
is
going
to
recreate
another
one
automatically,
but
with
stateful
sets
and
especially
with
local
volumes.
It's
more
complicated,
you'd
really
want
that
part
to
be
recreated
automatically
on
that
node
d.
A
A
So
you
could
say:
well
that's
easy,
let's
just
delete
that
pod
and
then
kubernetes
is
going
to
recreate
it
again,
but
but
it
doesn't
work
because
the
body
is
going
to
be
recreated
bound
to
the
same
claim,
which
itself
is
bound
to
the
same
volume
and
that
volume
is
bound
to
particular
host.
So
in
the
end
the
part
is
going
to
be
scheduled
on
that
particular
host
that
doesn't
exist
anymore,
so
it
doesn't
work.
A
What
you
really
have
to
do
here
is
first
delete
the
claim
and
then
delete
the
part.
So
there's
no
pod
anymore
and
there's
no
claim
anymore.
So
the
stateful
set
controller
is
going
to
recreate
that
new
pod,
but
it's
also
going
to
recreate
a
new
claim
and
that
claim
is
going
to
be
bound
to
a
different
new
volume.
A
A
What
could
happen
is
that
you
delete
the
pvc.
Then
you
delete
the
pod.
The
backpop
gets
recreated
before
pvc
deletion
is
complete,
so
the
pod
will
still
be
waiting
and
pending,
because
it's
bound
to
that
pvc
that
you
deleted
afterwards,
so
you
may
need
to
delete
the
pod
twice
actually
to
make
it
work
again.
A
You
should
note
that
here
it's
important
to
distinguish
between
a
host
that
would
be
completely
dead
and
unrecoverable,
as
in
there's
no
chance
ever
that
the
same
pod
is
going
to
run
with
the
same
data
on
that
host.
It's
over.
It's
done,
not
possible,
in
which
case
you
really
want
to
recreate
it
somewhere
else
versus
a
case.
Where
there's
a
chance,
you
can
recover
the
data,
and
you
can
recreate
that.
A
A
What
people
tend
to
do,
because
it's
simpler,
is
to
treat
all
the
kunes
nodes
as
immutable,
so
whenever
you
want
to
upgrade
the
kubernetes
version,
rather
than
upgrading
the
kubelet
in
place
on
the
same
machine,
you'd
rather
spin
up
a
new
virtual
machine
and
decommission
the
old
virtual
machine
right.
So
the
way
people
usually
do
that
is
spin
up
that
new
vm
drain.
A
A
So
once
that's
done
once
you
have
no
more
pod
running
on
that
host.
You
would
have
again,
as
we've
seen
before,
to
delete
both
the
pvc
and
the
pods,
so
the
pod
was
already
ability,
but
you
also
need
to
do
the
pvc
to
make
sure
that
this
pod
can
be
recreated
somewhere
else
with
a
new
fresh
volume
and
then
relying
on
the
replication
achieved
by
the
application
itself.
You
sort
of
recover
your
workload
and
you
didn't
lose
availability
and
you
didn't
lose
data.
A
It's
possible
that
60
minutes
is
not
enough
to
migrate
all
that
data
over
the
network
to
the
other
parts
right
and
because
that
pod
disruption
budget
is
not
respected
anymore.
After
this
this
hour,
this
60
minutes,
then
the
upgrade
system
is
going
to
move
on
to
the
next
part,
and
this
is
where
you
can
lose.
Availability
of
the
data
or
even
worse,
use
the
data
entirely,
because
the
application
may
not
have
enough
time
to
recover
from
the
deleted
pods.
A
That's
very
dangerous,
so
what
you
should
rather
do
instead
and
the
safest
option
would
be
to
manually,
create
this
new
node
with
the
new
kubernetes
version
and
make
sure
that
you
migrate
your
workload
here
right.
So
you
trigger
data
migration,
like
change
the
affinity
rules
for
the
new
pods
to
leave
on
that
new
node
pool
and
then
finally,
since
there
should
be
no
pod
anymore
on
the
old
node
put,
you
would
delete
the
older
pool
right,
that's
much
more
custom
and
really,
when
you
use
local
volumes
with
a
lot
and
a
lot
of
data
chances.
A
But
this
is
some
sort
of
planned
maintenance
right,
like
you
know,
you're
going
to
remove
that
part,
and
it's
a
bit
sad
to
sort
of
rely
on
the
mechanism
to
of
of
failover
of
the
data
like
of
recovery
of
the
data,
for
something
you
know
is
going
to
happen
and
even
worse.
What
if,
at
the
same
time
with
this
planned
maintenance,
you
get
another
unplanned
disruption
like
another
host
dies,
for
example.
A
Here,
if
you
have
pod
zero
and
pod
one,
if
we
want
to
get
rid
of
pod
zero,
what
maybe
we'd
like
to
do
is
create
pod
number
two
migrate,
the
data
there
and
then
delete
pod
number
zero
and
that's
it
and
we're
left
with
part
number
one
and
two,
but
in
practice
that's
not
possible.
Stateful
sets
don't
work
like
that.
They
work
the
opposite
way.
A
So
I
haven't
found
a
good
solution
to
that
problem.
Yet
with
the
stateful
sets,
like
some
people
manage
pots
directly,
some
people
just
can't
do
that.
So
another
thing
you
could
do
is
increase
the
number
of
replicas.
So
here
we
suddenly
have
three
replicas,
which
is
nicer,
and
then
we
remove
that.0,
but
then
it's
going
to
be
recreated.
So
if
we
want
to
get
back
to
zero
replica,
we
have
to
like
scale
down
to
two
rep
sorry,
two
replicas
to
delete
that
pod
again.
A
A
A
A
A
So
first
the
pad
is
deleted.
Then
it's
recreated
on
the
same
host,
because
there's
still
this
constraint
constrained
that
it
needs
to
be
bound
to
the
same
volume.
But
what?
If,
in
the
middle
of
these
two
operations
that
are
not
atomic
another
pod
gets
scheduled
on
that
empty
slot
here,
and
that
can
really
happen.
Like
say
here,
this
host
has
16
gigs
of
ram
and
there's
a
new
part
that
gets
created
in
the
meantime,
with
two
gigs
of
ram
and
then,
when
the
stateful
set
controller
tries
to
recreate
that
number
one
that
has
been
updated.
A
So
there
are
several
ways
you
can
deal
with
that.
The
first
thing
is
to
give
higher
priority
to
the
pods
that
are
of
local
volume.
So
you
can
use
this
priority
class
name
thing.
You
can
also
work
with
stains
and
toleration
to
isolate
workloads
with
local
volumes
to
a
particular
subset
of
nodes.
So
you
know
other
parts
are
not
going
to
be
scheduled
there
and,
finally,
what
I
think
is
the
probably
the
simplest
solution
to
this
hole
is
to
sort
of
plan
for
your
workload
in
advance.
A
A
One
other
important
thing
is
the
awareness
of
storage
capacity
in
kubernetes,
and
so
when
you
use
dynamic
provisioning,
for
example,
it's
possible
that
a
pod
gets
scheduled
onto
a
node
that
doesn't
have
the
necessary
storage
space
available
for
computers
has
been
fixed,
starting
kundalinis,
1.21
right
and
before
that
version.
One
way
to
deal
with
that
again
is
to
have
this
fixed
ratio
between
ram
and
storage.
So
you
know
that
if
the
bot
can
be
scheduled
because
of
its
ram,
it
means
the
storage
is
also
available.
A
And
finally
another
maybe
concern
is
that
there's
no
enforcement
that
you
get
the
exact
volume
of
the
size
you
requested.
Kubernetes
is
going
to
do
its
best
trying
to
match
the
claim
size
with
the
volume
size
that
it
can
happen.
That,
for
example,
you
you
request
one
gigabyte
claim
and
the
only
available
volume
is
one
terabyte
and
kubernetes
is
going
to
do
the
match
here
and
a
larger
problem
is
that
the
scheduler
sort
of
ignores
this
priority
over
the
closest
possible
size
of
the
volume
across
the
different
nodes.
A
All
right,
I
hope
that,
with
this
presentation,
you
got
a
bit
more
intimate
knowledge
of
how
local
volumes
work
and
especially
the
things
you
need
to
pay
attention
to.
If
you
decide
to
use
them
in
production
and
likely
those
involve
a
bit
more
work
and
a
bit
more
thinking
than
what
you'd
have
to
do.
If
you
were
using
network
attachments,
thank
you
for
attending.