►
From YouTube: Whatever Can Go Wrong, Will Go Wrong – Rook/Ceph and Storage Failures - Sagy Volkov, Red Hat
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Whatever Can Go Wrong, Will Go Wrong – Rook/Ceph and Storage Failures - Sagy Volkov, Red Hat
Imagine running a 200-node Kubernetes cluster, and suddenly you lost a node or even a ToR switch. What is the state of your persistent storage that your application relies on? How can you make sure your storage is always available? How can you time and plan how long it takes for your storage to get back to 100% resiliency? In this presentation we’ll go over the basics of storage demands (RPO/RTO), How different types of replications in Ceph impact our recovery time, and how components failure such as drive, node or cluster determine how long we are at risk. We'll include a live demo of a Rook/Ceph recovery process from a failed component. We'll show what components of Rook are recreated, how Ceph behaves during components/pods recreation, and what is the impact on the application while these failures occur (In our case the application will be MariaDB).
https://sched.co/ekFH
A
A
In
our
session
agenda,
I'll
talk
a
little
bit
about
kind
of
a
storage
introduction
I'll
explain
a
few
resiliency
terms
that
are
commonly
used
in
the
storage
industry,
we'll
talk
about
cipher
storage
provider
rook
as
a
storage,
orchestrator
or
manager
of
fan,
storage
and
we'll
talk
about
failures
supposed
to
be
a
live
demo.
It's
going
to
be
a
recorded
demo
and
I'll
leave
time
for
questions
in
the
end.
A
A
We
start
to
see
the
the
ability
to
have
servers
have
the
ability
to
run
some
sort
of
a
software
that
provides
storage
to
a
cluster
of
servers
or
set
of
servers
in
a
manner
that
it
was
still
a
stable,
resilient
and
redundant.
Software-Defined
storage
is
probably
the
most
common
storage
used
in
kubernetes.
Nowadays,
you
can
use
it
in
a
converge
manner
where
we
have
our
nodes,
running
the
software
defined
storage
and
our
pods,
and
you
can
also
use
it
in
non-converge
manner,
where
we
have
some
nodes
running
the
pods.
A
So,
let's
talk
about
a
few
terms
that
are
widely
used
in
the
storage
world.
Of
course,
they
are
also
used
in
a
lot
of
processes
in
the
technology
world
and
outside
of
the
technology.
World
availability
is
basically
your
uptime
in
the
storage
domain,
you're
gonna
hear
a
lot
about
the
number
of
nines
data
storage
solution
has
that's
basically.
A
99.99
something
nines
which
basically
impact
the
availability
of
your
storage,
the
number
of
nines
after
the
dot
six
nines
is
equivalent
to
30
once
31
seconds
of
downtime
in
one
year.
Five
nines
is
about
five
minutes.
Four
nines
is
about
one
hour
and
three
nines
about
nine
hours
of
downtime
in
one
year.
A
Durability
is
like
the
name.
Basically,
how
durable
is
your
storage?
How
good
is
it
is
in
in
in
keeping
the
data,
the
data
might
not
be
accessible
all
the
time,
but
it
is
still
intact.
So,
for
example,
you
lost
a
a
data
center.
The
data
still
resides
or
you
or
you
hope,
still
resides
on
your
storage
solution.
When
the
data
setter
comes
back
up,
the
data
is
still
there
reliability
and
resiliency.
A
Two
terms
that
we
are
going
to
be
focusing
more
here.
Basically
a
probability
you
know
of
how
good
you
design
your
storage
solution
and,
and
basically
it
gets
impact
by
the
ability
of
your
storage
solution
to
heal
itself.
So
again,
we
lost
a
storage
device,
or
maybe
we
lost
a
node
with
a
few
storage
devices
or
a
data
center
or
a
switch.
A
It's
the
ability
for
the
storage
solution
to
heal
itself
and
to
continue
and
provide
the
services,
the
data
services
that
it
needs
to
provide
in
this
diagram.
I'm
trying
to
kind
of
explain
these
terms
with
a
couple
of
processes.
Of
course,
we
want
availability
and
durability
all
the
time
throughout
all
the
processes.
A
If
the
process
ran
100
times
we're
supposed
to
see
in
the
end,
a
thousand
rows
in
the
database,
if
we're
seeing
that
that's
a
basically
100
availability,
if
we're
seeing
950
it's
basically
a
95
percent
availability,
of
course,
throughout
the
process,
once
we
insert
the
data,
we
want
the
data
to
be
durable.
We
always
want
to
be
able
to
query
it.
Some
in
some
cases
in
the
storage
domain,
there's
a
term
called
retention
policy.
Maybe
you
only
need
it
for
one
night?
A
Throughout
this
retention
time,
and
then
you
know
once
process
z
ends,
process
y
starts
to
read
input
from
process
z
and
and
and
run
as
some
other
processes
as
well.
Throughout
all
this
time,
reliability
and
resiliency
is
a
must
again.
You
lost
the
node,
you
lost
the
device,
you
lost
the
switch.
We
want
to
be
able
to
continue
and
run
process
z
and
run
process
y
and
complete
them
successfully.
A
You
had
a
problem,
we
talked
about
it,
you
lost
a
storage
device,
you
know
it's
usually
either
a
drive
or
flash
device
or
or
an
array
of
storage
or
switch
and
how
easy
your
storage
solution
is
actually
repairing
itself
yeah.
When
the
storage
defined
software
world,
you
can
sometimes
define
decides
between
having
two
copies
of
your
data
versus
three
copies
of
the
data.
A
For
example,
if
you
have
two
copies
of
your
data
and
for
some
reason
you
lost
some
devices
and
you
now
only
have
valid
one
copy
of
the
data
until
you
actually
repair
and
go
back
into
100
resiliency
100,
two
copies.
You
are
basically
exposing
yourself
to
data
unavailability.
If
you
lost
more
devices,
and
now
your
only
copy
that
exists
is
also
not
available.
A
A
A
Just
for
you
know,
example,
an
enterprise
grade
drive
have
about
800
000
hours
of
mtbf,
which
is
roughly
90
years,
sounds
a
lot,
but
then,
if
your
storage
solution
have
90
drives
in
that
example,
that
basically
means
that
you're
going
to
have
one
failure
every
year.
If
your
stored
solution
have
or
data
center
have
900
drives,
basically
means
that
you're
going
to
have
a
a
failure.
Every
five
weeks
in
one
of
the
drives
one
of
the
900
drives.
A
A
A
Some
applications
cannot
have
a
no
access,
for
example,
no
more
than
five
minutes
in
one
year
or
30
seconds
in
one
year.
Other
applications
are
in
lower
tiers
and
yeah.
It's
fine
for
them
to
not
have
access
to
the
data.
For
you
know
one
day
in
a
year.
Rpo
is
your
recovery
point
objective:
besides
counting
on
your
storage
solution,
to
maintain
all
the
copies
and
run
all
the
time
and
provide
adequate
performance,
you
should
always
thrive
to
do
also
run
backups
outside
of
this
storage
solution,
and
this
basically
impact
your
rpo.
A
How
many
backups
you
have
in
case
you
lost
everything
in
your
storage
solution
will
basically
impact
your
recovery
point
object
objective
if
you're
on
a
backup
every
one
hour.
That
means
that
your
rpo
is
of
one
hour
all
right.
So
now,
let's
continue
and
start
to
talk
about.
Seven
rook
ceph
is
a
unified
storage
system,
software
defined
storage.
A
It's
been
around
for
a
long
time.
It
provides
basically
the
ability
to
have
object
and
block
a
and
file
file
from
sffs,
which
is
basically
in
the
kubernetes
world,
will
be
the
equivalent
of
ada
rwx.
It's
a
shareable
size
file
system.
Rbd
is
a
a
a
pure
block
device
and
rgw
is
the
ability
to
interact
with
object
using
s3.
A
For
example,
there
are
a
few
components
for
for
ceph
and
the
cephmon
or
the
monitor
is
basically
the
the
coordination
or
took
the
coordinator
of
all
data
and
components
in
in
the
cluster.
It's
using
a
paxos
to
keep
itself
alive.
There's
minimum
of
three
of
these
processes
in
a
cluster,
and
you
can
go
up
to
seven.
The
manager
mainly
concentrate
on
on
real-time
metrics
and
other
management
functions
that
are
for
the
cfa
cluster.
A
There's
one
active
and
you
can
go
to
have
one
active
and
one
standby
sephora
is
the
easy,
are
very
important
processes.
These
are
basically
the
I
process
that
attached
into
a
a
a
device
that
can
provide
storage
like
an
lgd,
sdd
nvme,
any
type
of
a
block
device,
and
then
in
return,
the
ceph
cluster
aggregates.
A
All
these
devices
create
pools
out
of
them
and
able
to
provide
the
storage
capabilities
for
their
clients,
in
our
case
are
kubernetes
and
pvcs,
whether
it's
an
object,
a
block
or
or
file
a
couple
of
other
processes
that
are
not
showing
up
here
on
the
presentation.
Rgw
is
basically
what
allows
for
object.
Storage
access
usually
starts
with
one
process,
and
if
you
need
to
scale
up
to
you
know
millions
of
accesses
or
millions
of
object
accesses,
you
can
add
more
mds
is
the
ability
basically
to
access
sffs.
A
You
start
with
two
of
these
processes.
One
is
active
and
one
in
a
standby
mode,
and
now,
let's
talk
about
our
rook,
the
rook
is
a
basically
a
storage
manager
for
kubernetes.
In
our
case,
we
use
it
with
ceph.
I
like
to
call
it
the
orchestrator
of
all
things.
As
you
can
see
on
these
slides,
anything
blue
is
kind
of
more.
On
the
rook
level
we
have
the
rook
and
ceph
operator
support
that
runs
in
one
of
the
nodes.
A
We
have
discovery,
pods
that
runs
on
any
node
that
we
would
like
to
that
will
provide
storage
that
have
like
block
devices
that
cephei
the
ceph
cluster
is
going
to
to
use
you.
Can
you
configure?
You
can
configure
these
by
labels
and
things
and
things
like
that.
The
cfcsi
level
are
basically
the
attacher
and
detach
attach
pods
that
exist
on
the
in
every
nodes,
whether
it's
for
a
block
or
for
cfs,
and
then
you
have
the
seth
demons
that
rook
is
controlling.
A
Of
course,
each
of
these
things
are
are
basically
pods.
You
have
mon
pods
that
runs
on.
You
have
three
minimum
of
three
m1
pods,
so
on
the
next
node
that
we
don't
show
up
in
this
slide,
we
don't
necessarily
have
a
mon
pod
osd's
will
actually,
we
will
have
osds
wherever
we
will
have
storage
that
we
consume
on
these
nodes,
and
we
talked
about
rgw's.
We've
talked
about
the
manager
having
one
of
them
mds,
there's
a
pod.
A
So
let's
talk
about
the
rook
and
safe
resiliency
in
kubernetes,
as
discussed
previously,
every
ceph
process
is
basically
a
pod,
so
we
want
to
understand
and
what
happens
when
a
pod
fails
going
back
to
all
the
terminology
that
we
use.
Of
course
this
impacts
our
resiliency.
So
you
know
the
mons
are
very
important
processes.
That's
why
we
have
three
of
them
that
spread
over
three
nodes
or
more.
The
mds
which
again
provides
access
to
cfs.
A
A
If
you
have
two
data
centers,
you
have
two
kubernetes
clusters.
You
can
use
self
built
in
replication,
such
as
the
self-rbd
mirroring
to
basically
mirror
a
pv,
a
pv
in
a
certain
pool
from
one
safe
cluster
in
kubernetes
cluster,
a
to
a
rook
safe
cluster
in
kubernetes
cluster.
A
b
ceph
also
offer
the
same
capabilities
for
object,
storage.
It
is
ga
in
the
self
project.
A
A
What
happens
when
you
reboot
a
node
or
have
osds
osd
pods
on
it?
Remember
osd
is
the
process
that
consume
a
blocked
device
on
a
node
and
then
is
part
of
the
whole
cf
a
cluster
that
then
provides
a
storage
to
our
kubernetes
application.
We're
also
going
to
look
at
what
happens
when
we
lose
one
of
the
devices
that
the
nosd
pods
is
using
I've.
All
this
demo
is
running
on
aws,
very
small
clusters
like
three
masters.
A
It's
gonna
be
three
nodes
that
are
going
to
run
rook
and
safe
and
provide
the
storage
and
three
nodes
that
are
going
to
run
and
applications
that
will
consume
and
stress
the
storage.
I'm
using
a
a
project
named
sherlock,
which
I
started.
A
couple
of
months
ago,
it's
basically
a
project
that
aims
to
check
and
test
and
stress
performance
on
all
sorts
of
stored
solution
in
the
kubernetes
domain.
A
A
All
right
so,
let's
start
the
first
demo,
I've
kind
of
split
my
terminal
into
four
sections.
Hopefully
it's
clear
on
the
top
right
section.
I
have
a
ceph
command
called
zephosd3
that
is
constantly
running
using
a
watch.
It
basically
shows
the
ceph
cluster
osd3.
As
the
command
says.
We
can
see
that
we
have
three
nodes
that
provide
the
storage
on
each
of
the
nodes.
We
have
two
osds
that
consume
two
ssd
devices
on
them
and
provide
it
back
into
the
safe
cluster.
A
A
A
I
also
have
12
syspent
jobs,
suspect
pods
running
on
a
on
these
12
mysql
databases,
and
this
is
done,
of
course,
using
the
sherlock
github
project
that
I've
previously
mentioned.
I'm
just
going
to
pick
one
of
these.
You
know
one
of
these
nodes,
one
of
these
mysql
jobs
and
I'm
going
to
basically
constantly
monitor
the
logs
of
this
job.
A
What
I'm
going
to
do
is
I'm
going
to
pick
one
of
these
basically
osd
pods
and
I
will
delete
one
of
them
in
this
case.
I'm
just
going
to
pick
a
osd
3
and
what
is
interesting
to
monitor
is
to
look
at
the
top
left
of
what
happens
to
the
pod.
I
kill
it
if
a
new
one
comes
in
and
on
the
top
right,
since
we
are
looking
at
the
osd
3.
That's
osd
3
right
here.
A
What
is
a
what
is
happening
to
the
the
status
of
that
osd
component?
Is
it
staying
up
for
how
long
there's
another
command
here
from
ceph
that
basically
dumping
all
the
pg's
or
placement
groups
that
and
the
cf
cluster
has?
As
you
can
see
now
we
have
81
pages
and
they're
all
at
the
all
at
an
active
and
clean
if
seth
see
or
that
something
is
wrong
with
one
of
the
osds.
A
A
Cisband
job
was
running
and
that
is
of
course
acceptable.
There
was
some
kind
of
an
eye
operation
that
internally
ceph
was
doing.
This
database
might
have
been
reading
from
placement
groups.
That
was
on
the
osd
that
I
have
deleted.
So,
for
a
brief
second,
it
had
to
get
the
pages
from
a
different
osd.
That's
that's
the
job
of
the
sf
months
to
basically
tell
all
the
clients
to
get
these
pages
from
a
a
different
location,
and
that's
the
end
of
this
first
demo,
all
right.
A
A
And
I'm
going
to
reboot
this
node
and
on
the
bottom
left,
I'm
going
to
look
at
the
the
node
status,
since
this
is
aws.
Things
are
probably
you
know,
rebooting
so
fast.
I'm
not
sure
how
much
of
a
cube
cattle
is
actually
going
to
capture
that
the
nodes
are
being
down,
but,
as
you
can
see
on
the
bottom
on
the
top
right
ceph
does.
It
definitely
starts
to
see
some
rpgs
that
are
not
available.