►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Improving Availability for Stateful Applications in Kubernetes - Michelle Au, Google
Over 40% of Kubernetes clusters run stateful applications. However, there are still challenges with deployment, especially with choosing a storage backend that is best suited to support your application across multiple failure domains. In this talk, we will discuss improvements that make it easier to run stateful applications across multiple failure domains for increased availability, and demonstrate how Kubernetes automatically recovers from failure scenarios at the node and zone layers. We will also discuss different categories of storage and their pros and cons in these failure scenarios. You will walk away with a better understanding of Kubernetes’ scheduling and failure recovery capabilities with respect to stateful workloads, and which types of storage backends are most appropriate to meet your stateful application’s availability, reliability, and performance goals.
https://sched.co/MPfh
A
Portability
and
self-healing
are
one
of
the
most
compelling
aspects
of
kubernetes
and,
as
you
learned
from
the
keynote
earlier
this
morning
and
the
storage
one-on-one
session
earlier
today
in
storage,
persistent
volume
claims
provide
that
portability
layer
for
kubernetes
storage.
However,
that
doesn't
mean
you
can
completely
forget
that
house
storage.
You
still
need
to
understand
what
storage
options
are
available
to
you,
what
are
their
various
trade-offs
and
what
is
best
for
your
application.
A
So
today,
I'm
going
to
try
to
answer
some
of
those
questions.
I'll,
first
give
an
overview
of
the
various
options
that
we
support
today
in
kubernetes
and
then
I'm
going
to
deep
dive
into
building
highly
available
stateful
applications,
specifically
focusing
on
how
to
achieve
failure
domain
spreading
with
storage
I'm
going
to
give
a
demo
of
various
real
storage
systems
that
are
in
kubernetes
today
and
demo,
how
they
behave
in
different
failure
scenarios
and
then
I'm,
going
to
end
with
a
discussion
on
pod
downtime
and
how
to
build
recovery
into
your
application.
A
So,
let's
dive
right
into
what
we
support
in
kubernetes
today,
there
are
a
lot
of
options
available
to
you.
We
have
over
15
in
tree
storage
drivers
and
over
35
CSI
drivers,
and
the
count
keeps
on
increasing
these
drivers
interface,
with
a
wide
architecture
of
different
storage
systems,
ranging
from
local
to
remote
storage
cloud
services
to
network
appliances,
software-defined
storage
systems
distributed
systems
hyper-converged
systems.
The
list
goes
on.
So
with
all
of
these
various
options
available
to
you,
how
do
you
go
about
picking
what
is
the
best
for
your
applications
in
your
environment?
A
A
So
here
are
some
characteristics
that
I
think
are
especially
important
for
building
a
highly
available
application.
First
is
accessibility.
This
is
about
how
close
your
application
needs
to
be
to
the
underlying
storage
back-end.
Does
they
need
to
be
in
the
same
node?
It
doesn't
need
to
be
in
the
same
rack
or
zone,
or
can
it
be
anywhere
in
the
cluster?
A
This
is
very
important,
because
this
will
determine
how
flexible
your
application
pods
can
be
placed
in
the
cluster,
especially
when
there's
a
lot
of
moving
parts-
and
you
might
have
different
node
failures
going
on
next
is
availability.
This
is
about
how
resilient
your
storage
system
is
to
different
cluster
failures.
Can
your
storage
system
survive
a
node
failure
or
a
rack
failure,
or
a
zone
failure?
Often
times
availability
is
very
close
or
very
similar
to
accessibility,
but
that's
not
always
the
case
and
I
will
provide
an
example
of
this
later.
A
Then
we
have
durability.
This
is
about
the
data
replication
factor
underneath
and
how
many
disks
can
fail
before
you've
completely
lost
your
data,
and
then
we
have
access
mode.
Accessmode
determines
how
many
nodes
can
simultaneously
mount
the
volume.
At
the
same
time-
and
this
is
important
because
it's
going
to
restrict
how
many
replicas
of
your
application-
you
can
have
running
concurrently.
A
So
now
I'm
going
to
go
into
some
examples
of
specific
storage
systems
and
where
they
fall
into
these
various
characteristics.
Let's
start
with
local
disks,
a
local
disk
is
directly
attached
to
a
node
and
it
generally
offers
the
best
performance
and
the
best
cost.
However,
it
has
the
worst
availability
and
durability
if
that
disk
or
that
node
fails,
then
any
pod
using
that
disk
is
also
unavailable.
A
It's
important
to
also
point
out
that
in
many
cloud
environments,
the
local
disk
offerings
are
not
durable
beyond
the
life
of
the
VM,
which
means
that
if
that
VM
gets
terminated
or
the
physical
host
has
an
error,
then
all
the
data
on
that
local
disk
is
completely
lost.
So
this
is
something
you
really
need
to
carefully
consider
if
you
want
to
use
local
disks
in
your
application.
A
Next
we
have
cloud
disks.
These
are
generally
block
devices
that
show
up
as
block
devices
in
your
VM,
but
underneath
are
actually
remotely
accessed
over
the
network.
Typically,
they
are
restricted
to
a
single
zone,
but
they
are
replicated
within
that
zone
so
compared
to
local
disks.
Cloud
disks
have
proved
durability
and
accessibility,
but
the
performance
may
not
be
as
good
and
the
cost
is
a
little
more.
Some
examples
of
cloud
disks
include
GCE,
PD,
AWS,
EBS
volumes
and
Azure
disks.
A
Next,
we
have
a
single
NFS
server.
Access
is
greatly
improved
because
you
can
mount
that
NFS
share
from
anywhere
in
the
cluster.
You
can
also
have
multiple
amounts
of
the
same
share
across
multiple
nodes.
At
the
same
time,
however,
this
single
server
remains
a
single
point
of
failure
in
your
system.
So
this
is
one
example
where
availability
and
accessibility
are
not
the
same
cost
compared
to
a
cloud
disk
or
let's
say,
durability
also
depends
on
what
the
underlying
block
device
is.
That's
backing
your
NFS
server.
If
say,
we
use
a
cloud
disk
to
back.
A
It
then
we'll
have
that
3x
durability
within
that
zone
and
cost
is
a
little
more
expensive,
because
now
we
have
to
pay
for
compute
and
network
resources
and
in
terms
of
performance.
Now
you
have
an
extra
overhead
of
some
Network
protocol
on
top
and
you
might
also
have
traffic
coming
across
various
zones.
A
A
A
A
A
A
Instead,
single
writer
volumes
are
better
suited
for
distributed
applications.
These
are
applications
that
shard
and
replicate
their
data
between
the
application
pots.
Some
examples
of
these
include
distributed.
Databases
like
Cassandra
and
just
to
name
a
few
and
these
applications
do
not
need
high
availability
at
the
storage
layer
because
they've
already
built
it
in
to
the
application
layer.
So
these
single
writer
volumes
are
great
for
these.
A
They
don't
have
to
be
globally
accessible
or
available,
so
you
can
choose
potentially
cheaper
and
more
performant
options
such
as
local
disks
or
cloud
disks
for
these
types
of
applications
and
in
kubernetes.
The
way
that
the
stateful
sets
are
the
way
to
that
best
support
distributed
applications.
The
main
difference
between
stateful
sets
and
deployments
are
that
two
main
things
one,
the
every
pod
in
a
stateful
set,
has
a
unique
and
stable
identity
and
then
second,
every
replicas
in
a
stateful
set
has
its
own
unique.
Persistent
volume
claim.
A
So
you
can
see
in
this
example
that
we
have
pot
zero
using
PVC
zero
pod,
one
using
PVC
one
and
pod
using
PVC.
If
any
of
those
replicas
dies,
it's
going
to
come
back
with
the
same
name
and
it's
going
to
come
back
with
the
same
storage.
And
in
this
case,
if
one
of
the
zones
goes
out,
the
application
is
still
able
to
serve
the
data
from
its
remaining
shards
and
you
can
bring
up
new
pods
to
replace
the
failed
one
and
potentially
new
disks
to
replace
to
failed
one
and
those.
A
All
right,
so
this
wasn't
always
easy
to
do
in
trooper
Nettie's,
especially
when
using
volume
types
that
were
not
globally
available
or
accessible
in
the
cluster
kubernetes
originally
assumed
that
you
can
access
volumes
from
any
node,
but
now,
with
the
volume
topology
feature,
we've
added
intelligence
into
the
scheduler
so
that
it
can
better
understand
these
more
constrained
volume
types.
There
are:
there's
no
user
configuration
that
you
need
to
use
this
feature.
A
A
Alright,
so
now
I'm
going
to
go
into
a
demo,
I'm
going
to
demo
a
app
I'm
going
to
demo
applications
that
are
not
highly
available,
and
the
reason
for
that
is.
I
want
to
be
able
to
show
you
how
some
real
storage
systems
behave
in
certain
scenarios
so
that
you
can
see
that
behavior.
You
know
without
the
application
layer
doing
redundancy
and
you
can
decide
if
that's
a
model
that
is
best
suited
for
your
application.
A
A
You'll
see
here
that
this
volume
is
has
a
type
of
a
local
volume.
It
has
a
path
to
that
disc
on
that
node,
and
it
has
this
node
affinity
field
filled
in
to
it,
and
you
can
see
this
node
affinity
is
basically
telling
kubernetes
that
any
pod
using
this
volume
can
only
be
scheduled
two
nodes
that
have
the
label
hostname
with
this
value.
A
A
You
can
see
here
that
this
type
is
a
cloud
disk.
It's
a
GCE,
persistent
disk
and
it
also
has
no
Definity,
but
instead
of
having
no
affinity
on
a
single
node,
it
has
no
affinity
in
a
single
zone,
and
here
it's
saying
the
zone
is
Europe
West
1c.
So
this
tells
the
scheduler
that
pods
using
this
type
of
all
or
this
volume,
can
only
be
scheduled
within
that
one
zone.
A
And
if
we
look
at
our
next
one,
which
is
our
multi
zone
disk,
you
will
see
that
this
is
also
a
GCE
persistent
disk.
It's
a
special
flavor
of
it
called
a
regional
PD
and
the
special
thing
about
it
is
that
it
can
be
accessed
from
two
zones.
So
if
you
have
a
node,
if
you
have
a
complete
failure
in
one
of
the
zones,
you
can
have
replacement
pods
come
up
in
a
second
zone
and,
lastly,
we
will
look
at
the.
A
A
B
A
That's
right,
the
local,
the
pod,
using
the
local
volume,
is
not
going
to
be
able
to
come
up
after
I've
failed
this
node,
because
the
disk
that
it
needs
is
on
that
node,
which
is
now
unavailable,
but
the
other
parts,
since
they
are
accessible
to
more
than
one
node,
aren't
going
to
be
able
to
recover.
So
let's
go
ahead
and
fail
that
node.
A
All
right,
so
you
can
see
that
all
the
pods
have
immediately
are
terminating
and
restarting
so,
while
we're
waiting
for
that,
I
just
wanted
to
point
out
the
way
I
failed.
This
node
is
by
using
a
node
tinct.
This
is
a
mechanism
in
kubernetes
to
apply
special
conditions
to
notes.
So
you
can
see
here.
We
have
this
cue,
cuddle,
taint
command
and
you
specify
what
node
you
want
to
taint
and
you
give
it
a
condition
and
I
I
just
made
up
some
condition
and
the
other
thing
you
can.
A
A
A
A
So
what
happened
there
was
that
when
the
first
time
I
felt
the
note
it
got
rescheduled
to
a
completely
other
zone.
So
when
I
filled
the
whole
whole
zone,
these
two
pods
already
they
did
not
have
to
restart
because
they
were
not
in
the
zone
that
I
failed
so
yeah.
Now
we
can
see,
let's
take
a
look
at
my
single
zone
pod
and
we
can
see
it
failed
scheduling
because
two
of
the
notes
that
it
could
run
on
have
tanks
on
them
and
the
remainder
of
the
remaining
four
nodes
cannot
access
that
single
zone
pod.
A
A
Alright,
so,
while
we're
on
the
topic
of
pod
failures,
now
I'm
going
to
talk
about
pod
downtime
and
why
it's
important
to
factor
this
into
any
applications
architecture,
the
total
time
that
a
pod
is
down
is
consists
of
the
time
to
detect
failure
and
the
time
it
takes
to
replace
that
pod
and
from
the
demo.
The
time
to
detect
failure
was
essentially
zero
because
I
just
I
manually
tainted
the
note
when
I
wanted
to
fail
it.
A
But
in
a
real
production
environment
there
has
to
be
something
that
is
monitoring
those
nodes
and
that
something
also
has
to,
after
some
threshold
period,
go
ahead
and
taint
that
node
as
healthy
in
order
to
trigger
all
the
affections
of
those
pods.
So
that's
going
to
add
some
time
to
your
downtime
and
then
to
bring
up
a
new
pod.
You
kind
of
saw
a
little
bit
of
this
in
the
demo.
Some
pods
were
faster
to
come
up
than
others.
A
The
single
rider
volume
pods
have
to
first
because
they're
only
accessible
on
a
single
node
in
order
to
bring
it
up
on
a
new
node.
You
need
to
detach
it
from
the
first
node
and
then
go,
and
we
attach
it
to
that
second
node.
So
you
might
have
noticed
that
the
NFS
pod
was
able
to
come
up
a
lot
faster
because
it
doesn't
have
to
go
it.
Can
it
can
mount
multiple
on
multiple
nodes
at
the
same
time,
and
it
doesn't
have
to
wait
for
this
detach
and
reattach.
A
So
all
of
this
factors
in
to
the
total
downtime
of
your
pod.
So
it's
very
important
to
build
this
in
when
you
are
figuring
out
what
your,
what
your
maximum
degraded
downtime
State
can
be,
and
there's
one
more
caveat
that
I
want
to
point
out,
especially
with
stateful
sets.
So
many
stateful
applications
require
exactly
once
semantics.
What
this
means
is
that
it
cannot
have
two
containers
write
to
the
same
volume.
Otherwise
it
can
cause
data
corruption
to
that
disk.
A
A
So
when
cubelet
is
not
responsive,
it's
not
able
to
delete
that
pot
object
and
then
the
stateful
set
controller
is
not
able
to
create
a
replacement
pod,
for
that,
until
cubelet
is
able
to
acknowledge
that
the
pod
is
actually
not
running
anymore.
So
a
common
way
to
solve
this
is
a
technique
called
node
fencing.
A
This
gives
the
this
technique
ensures
that
the
workload
containers
are
not
running
on
that
node
anymore,
in
bare
metal
environments.
This
is
often
achieved
by
using
IPMI
power
we
set
to
the
server
and
in
cloud
environments
something
similar
can
be
done
by
deleting
the
VM,
but
because
this
is
environment
specific.
There
is
no
common
solution
in
kubernetes,
many
kubernetes
providers
have
built
functionality
on
top
to
give
this
capability.
A
So
you
know
adding
all
of
these
factors
together
recovering
a
single
state
full
set
pod
can
take
on
the
order
of
minutes
or
even
hours.
If
you
don't
have
such
an
automated
solution
and
you
need
to
go
in
and
manually
covered
them.
So
for
that
reason,
it's
very
important
that
when
you're
designing
your
highly
available
application,
you
build
pot,
you
don't
depend
on
the
availability
of
a
single
pot,
because
that
single
pod
will
become
your
single
point
of
failure.
A
Right,
so,
in
summary,
while
the
stateful
workload
experience
is
not
completely
seamless,
we've
made
many
improvements
in
kubernetes
over
the
last
few
releases.
To
make
this
better,
we
have
features
like
volume,
topology,
pod,
anti
affinity
and
no
taints.
That
will
help
with
in
making
it
easier
to
deploy
and
manage
your
highly
available
applications.
A
We
also
covered
some
high-level
architectural
models
of
deploying
stateful
applications
using
pod
anti
affinity,
and
we
discussed
when
it's
appropriate
to
use
a
deployment
versus
a
stateful
set
and
also
consider
either
hat
when
you're
designing
applications
to
have
the
data
redundancy,
either
at
the
storage
layer
or
at
the
application
layer,
and
perhaps
most
importantly,
when
you
are
designing
an
application.
A
highly
available
application
be
sure
to
build
redundancy
into
your
system
and
account
for
downtime
of
your
individual
replicas.
So
doing
so
will
make
your
applications
more
resilient
in
the
long
run.
C
Hi
thanks
for
the
talk
there
are
some
cases
like
in
when
running
in
AWS,
where
some
of
these
strivers,
like
the
elastic
file
system
for
leverages
a
sin
alpha
since
a
month.
So
this
is
track.
Class
is
not
available,
so
you
have
to
be
playing
with
selectors
and
a
particular
pool
for
for
this.
This
some
other
way
to
play
one
with
MPs
in
this
case,
when
this
storage
tracks
are
not
available
for
your
provider.
Oh
sorry,.
C
Are,
for
example,
if
we
have
the
classes
in
AWS
and
some
of
these
drivers
like
the
last
e-file
system,
the
MPs
are
not
available
or
in
alpha
version,
which
really
earlier
states.
So
we
ended
having
separated
pool
with
an
MVS
mounted
and
the
pot
basically
moons
a
host
path
and
is
the
way
you
can
use
Network
file
system
with
oh
I,.
C
C
A
C
A
So
I
think
that
you
can
there's
an
entry
NFS
driver
that
you
can
use
to
just
generally
mount
NFS
shares
it
works
across
to
any
it
doesn't
actually
care.
What
is
what
that
backing
NFS
server?
Is
it
just
needs
an
IP
and
a
share
I'm,
not
sure
if
EFS
has
like
any
special
protocols
or
permissions
that
you
need
the
special
driver
for
I,
wouldn't.
A
A
A
Draft
area
for
yeah,
so
there
are
various
open
source,
scale-out,
NFS
solutions,
I
think
like
Stefan
cluster
are
some
popular
ones
where
they
basically
take
inventory
of
all
the
local
disks
and
then
present
this
you
know
like
scalable
NFS
layer
on
top
of
it.
So
that's
one
way
to
do
it.
The
other
common
way
I
see
people
do
do
this
in
kubernetes
is
to
deploy
like
a
single
NFS
server
using
some
cloud
disk.
That's
backing
it.
So
I,
like
I,
mentioned,
like
that
single
NFS
server,
is
a
single
point
of
failure.
A
If
you
use
something
that
is
a
single
zone
cloud
disk,
then
if
that
zone
goes
down
that
your
NFS
is
down,
if
you
use
like
a
replicated
cloud
disk
solution
like
regional
PD,
then
you
are
more
resilient
to
that
single
zone
failures.
So
now
you
can,
if
that
whole
zone
fails,
you
can
do
your
NFS
server
pod
can
scheduled
to
a
different
zone
and
and
that
Regional
PD
can
access
that.
A
D
So
I'm
wondering
about
this
potential
split
brain
situation,
then,
if
you're
not
worrying
about
someone
had
something
writing
to
volume,
but
you
are
worried
about
something
updating
a
database
through
yeah
over
TCP
connection
to
some
other
or
that
still
alive.
What
does
Courtin
help?
Can
you
use
that.
A
So
I
think,
if
you
don't
actually
care
about
the
volume
layer,
but
you
care
about
the
date,
the
application
layer
writing
to
a
database.
Then
that's
something
where
you
might
still
need
to
build
in
some
concurrency
in
your
application
so
like
having
some
sort
of
leader
election
or
distributed
locking
system.