►
From YouTube: Disaster Recovery of Stateful Applications in a Multi-Cl... Orit Wasserman & Shyamsundar Ranganathan
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Disaster Recovery of Stateful Applications in a Multi-Cluster Environment - Orit Wasserman & Shyamsundar Ranganathan, Red Hat
Have you ever wondered how to provide for disaster recovery of the state stored in your persistent volumes? What needs to happen to recover the workload on an alternate kubernetes cluster? How can the state be replicated and workloads recreated to use their replicated volumes? Our talk aims to elaborate on the various issues around recovering a workload and its state, in a multi-cluster and a multi region environment. We will demonstrate how replication capable storage systems, such as Ceph/Rook, instead of higher level tools, can be leveraged to provide disaster recovery of workloads across clusters. In addition this session will tease out features required in a multi-cluster control plane, to enable one-click disaster recovery solution for stateful workloads. Attendees will learn how to approach building disaster recovery solutions for their own clouds.
A
Hello
welcome
to
kubecon
2021.
Today's
topic
is
disaster
recovery,
the
ability
to
recover
from
a
full
data
center
loss,
making
sure
you
have
the
data
and
can
recover
your
application.
My
name
is
vassalman.
I'm
after
chief
data
foundation,
architecture
trader
and
with
me
today
is
my
colleague:
sean
ranks.
A
Then
we'll
have
a
short
demo
and
talk
about
the
future
work.
A
demo
will
be
based
on
ceph,
with
a
software
defined
storage,
completely
open
source
quite
mature.
The
idbi
unsafe
is
installing
the
software
on
standard
servers
using
third
disk
from
http
memory
and
using
standard
ip
network
to
get
a
real,
very
reliable,
highly
available,
very
scalable
distributed
storage
system
in
a
single
step
cluster
you
get
all
the
storage.
Need
you
get
cloud
object,
storage
with
redux
gateway,
block
storage
from
rbd
and
set
refresh
will
provide
you
with
disability
file
systems.
A
A
Let's
talk
about
disaster
recovery
disaster
recovery
is
making
sure
we
can
serve
our
customers
even
if
we
lose
a
data
center
or
e
or
a
cloud
do
a
gen,
and
sadly
this
can
happen
on
the
left.
For
example,
we
have
a
photo
of
ovh
one
of
the
largest
data
center
in
europe
that
was
completely
destroyed
by
fire
last
year.
A
In
most
cases,
our
disaster
recovery
site
should
be
found,
and
that
will
mean
high
network
latency.
In
that
case,
we
won't
be
able
to
use
synchronous
applications,
but
asynchronous,
which
means
we
don't
you
need
to
drive.
We
don't
wait
for
the
ride
to
sink,
and
that
will
mean
there
will
be
some
data
loss
in
case
of
recovery.
A
A
A
very
important
way
we
measure
the
glass
recovery,
our
rpo
and
rtu
recovery.
Point
objective
is
the
amount
of
acceptable
data
loss
in
case
of
recovery
from
disaster.
It's
measured
in
time
unit
becoming
a
time
objective
is
the
amount
of
time
the
duplication
can
be
done
and
before
it
will
impact
the
business
again
measured
in
a
minute.
A
A
A
A
A
That's
the
west
will
resync,
we
cluster
it
catch
up,
then
the
application
and
the
pv
will
be
down
in
class
to
east
making
sure
all
the
data
is
synchronized
to
the
source
system.
First
of
all,
it
will
take
only
the
latest
changes
and
then
we'll
make
the
purviews
priming
again
and
writeable
end
application
in
cluster
ways
without
any
data
loss.
A
First,
why
can't
we
use
backup,
so
backups
are
great,
very
important.
You
should
backup
your
data
always
very
important,
they're
bad,
let's
think
about
how
frequently
we
back
up
our
data
daily
in
extreme
cases
every
few
hours
this
means
rpo.
The
amount
of
data
we
can
lose
in
case
of
disaster
is
ours.
A
A
Then,
let's
think
about
the
amount
of
data
we
need
to
transfer
in
case
of
disaster.
So
let's
say
we
back
up
to
a
remote
object,
storage,
so
in
case
of
recovery,
we
need
to
read
full
backup
in
order
to
restore
the
data.
All
that
data.
That's
a
lot
if
we
use
incremental
backup.
In
that
case
we
can
use
less
data,
but
it
will
increase
rto.
The
amount
of
time
of
restoration,
also
incremental
backup
requires
us
to
be
able
to
detect
incremental
changes
to
the
snapshots
we
took
in
in
kubernetes.
A
Let's
think
about
kubernetes
recently
set
up
a
point
in
time
between.
We
may
lose
some
resources
state
of
changes,
but
do
we
have
to
they?
Don't
change
that
frequently
we
can
do
it
symbolously
or
maybe
in
a
declarative
source
like
a
github.
In
that
case
we
can
even
get
to
rp
or
we
won't
lose
any
data
in
case
of
disaster.
A
Luckily,
for
us,
storage
systems
do
have
your
application,
built
in
usually
it's
based
on
periodic
snapshots
and
only
that
only
transferred
incremental
snapshot.
The
delta
change
between
the
period
in
that
period,
that
will
allow
slower
appeal,
and
they
also
have
much
more
efficient
storage
traffic
and
separate
between
the
storage
traffic
and
the
user.
Io
and
recovery
is
almost
instantly
or
very
close,
because
we
already
have
the
latest
beta
in
our
recovery
site.
So
the
rto
from
soy
specific
is
very,
very
low.
A
A
What
we
did
in
self-csi,
we
extended
the
standard,
csi
api
and
added
a
new
resource
volume,
replication
volume.
When
we
create
a
volume
application,
we
basically
enable
application
for
that
volume
between
the
two
clusters.
When
we
delete
the
resource,
we
disable
the
application,
we
have
the
application
state,
meaning
if
the
volume
is
primary
or
not.
In
this
case,
the
volume
is
finally
in
this
primary
site
and
is
active
invitability
and
replicated
to
best.
The
data
source
will
point
to
the
pvc
that
is
being
replicated.
A
A
In
order
to
enhance
more
capabilities,
we
also
added
volume
application
class.
The
class
contains
secrets
we
need
in
order
to
destroy
system,
to
be
able
to
communicate.
We
can
define
as
replication
schedule
as
not
all
application.
I
need
the
same
schedule,
some
have
you
don't
change
the
data
much
and
we
can
add
higher
longer
periods
and,
of
course,
to
store
all
the
vendor
specific
parameters
needed
for
the
storage
application.
A
So
I
will
recover
and
look
so
already
have
a
validification
created
in
clusterings
and
we
define
and
that
previous
primary
in
clusters
now
that
we
we
know
cluster
is
held.
We
want
to
recover
to
plaster
waste,
we'll
create
a
new
volume
applications
resource
in
cluster
waste.
Mark
this
replication
state
primary
use
the
same
data
source,
the
same
ppc
as
we
did
in
class
to
east
the
site.
A
A
A
So
here
we
have
a
problem
the
previous
year
provision
dynamically.
When
we
want
recover,
for
example,
we
need
to
attach
previously
not
to
a
neodymically
provision
npvc,
but
to
ex
through
an
existing
pvp,
the
replicated
one.
So
we
are
actually
create
a
matching
pv
in
cluster
west
that
is
connected
to
the
volume
that
we
replicate
to
cluster
with
then,
when
we
recover,
we
want
use
the
data
source,
and
but
we
need
a
way
to
tell
kubernetes.
A
B
We
also
need
that
any
custom
resources
that
these
applications
use
their
operators
and
custom
resources
are
deployed
on
both
clusters
or
all
clusters,
and
we've
also
looked
at
storage
storage
has
to
be
set
up
across
these
clusters
in
an
equivalent
fashion,
such
that
the
volume
replication
class,
the
storage
class
everything
is
equal
and
that
the
application
can
be
redeployed
onto
a
target
cluster.
So
from
a
cross-cluster
action
perspective.
Cluster
configuration
is
something
that
we
want
to
maintain
across
these
clusters.
B
From
a
user
perspective
for
application,
recovery
and
relocation,
users
would
need
access
to
congruent
namespaces
in
these
clusters,
where
they
can
place
their
workloads.
They
would
also
need
a
declarative
copy
of
their
resources.
The
application
manifests
that
they
can
recreate
on
these
clusters
and,
finally,
they
would
need
to
reroute
their
global
traffic
manager
or
the
inbound
traffic
to
the
cluster.
That's
actually
running
that
workload
at
any
given
instant
further
than
this
across
these
clusters.
B
We
need
to
manage
multiple
kubernetes
clusters.
We
chose
the
open
cluster
management
project
which,
as
the
tagline
goes,
is
a
community
driven
project
focused
on
multi-cluster
and
multi-cloud
scenarios
for
communities,
apps
works
perfectly
for
us.
We
are
looking
at
multi-cluster
and
managing
apps
across
multiple
clusters.
B
We
leverage
open
cluster
management,
primarily
for
the
cluster
configuration
and,
more
importantly,
to
manage
the
application
life
cycle.
So
we
need
application,
manifests
coming
from
a
declarative
source
and
ocm
has
a
channel
crd,
which
is
basically
pointing
to
a
git
helm
or
an
object
store
for
the
declarative.
Copy
of
the
application
manifests
ocm
also
has
a
placement
rule
crd,
which
decides
where
an
application
which
clusters
an
application
should
be
placed
too,
and
so
that's
leveraged,
so
that
we
can
do
the
orchestration
during
disaster
recovery
by
by
scheduling
the
placement
rule
appropriately.
B
There
are
gaps.
Disaster
recovery.
Orchestration
is
not
a
part
of
it.
They
are
more
around
stateless
apps,
which
is
where
we
come
in
and
we'll
talk
about
what
we're
going
to
do
about
that
so
for
disaster
recovery
orchestration,
although
we
have
these
various
components,
what
does
it
mean
for
a
user
and
how
easy
or
complex
is
it?
B
So
as
it
stands,
it
actually
becomes
quite
complex
as
we
look
at
three
use
cases
of
deploy,
relocate
and
recover.
Let's
start
at
deploy,
so,
for
example,
the
user
would
have
to
deploy
their
application
resources
to
a
cluster.
Let's
say
east
in
this
particular
example:
we'd
have
to
create
volume,
replication
resources
as
primary
to
establish
volume
replication
for
every
pvc.
In
this
particular
name
space.
B
They
would
have
to
ensure
that
the
volume
replication
is
occurring
before
they
decide
to
back
up
the
pv
cluster
data
because
that's
what's
going
to
be
used
in
the
alternate
cluster,
as
orig
mentioned,
to
reattach
to
the
volume
in
the
storage
backend.
And
finally,
if
there's
a
new
pvc,
that's
created
by
the
application
in
the
namespace
they'll
have
to
repeat
the
protection
for
that
particular
pc,
not
too
bad,
but
let's
go
to
recover
and
see
how
it
starts
increasing
in
complexity.
B
Technically,
at
the
at
the
point
where
we've
covered
these
steps,
the
rto
goal
is
met.
We
have
recovered
the
application
in
some
time,
but
it
doesn't
end
there
when
east
cluster
recovers.
In
this
particular
case,
east
is
temporarily
offline.
Let's
say
we
would
have
to
ensure
that
the
application
resources
at
least
are
deleted.
B
The
volume
replication
resource
is
marked
as
secondary
so
that
it
starts
re-syncing
data
from
the
new
primary
on
west
and
then
once
resync
is
established,
we
can
delete
the
volume
replication
resource,
which
would
in
turn,
delete
the
pvc
resource
and
free
up
east
cluster
and,
of
course,
once
this
happens
as
usual.
We're
gonna
have
to
protect
any
new
pvcs
that
appear
on
west
as
part
of
the
application's.
B
B
We
need
to
wait
for
the
final
sync
of
data
to
be
reported
by
volume
replication
resource
because
relocate
is
when
both
clusters
are
active
and
recovery.
Point
objective
is
zero.
In
other
words,
we
need
all
the
data,
and
once
the
final
resync
is
complete,
is
detected,
we
can
get
rid
of
the
volume
replication
resource
etc
on
the
west
cluster,
but
then
we
can
start
bringing
up
the
application
on
the
east
cluster,
restoring
the
pvs
first
recreating
the
apps,
recreating
volume
replication
as
primary.
B
So
there
are
quite
a
few
steps
to
perform
these
actions,
and
so
what
we
did
next
was
to
create
a
dr
orchestrator
called
ramen.
What
raman
does,
as
per
its
tagline,
is
instant
cloud
native
workload,
recovery
and
relocation
across
communities
clusters.
Yes,
we
are
interested
in
recovery
and
relocation.
Cross-Cube
clusters,
that's
the
use
case.
We
talked
about
that's
what
random
helps
us
with.
B
B
It
defines
a
replication
schedule
which
basically
defines
the
recovery
point
objective
for
the
for
the
application
that
wants
to
use
this
particular
policy,
and
it
optionally
has
a
volume,
replication
class
selector
to
disambiguate.
In
case
there
are
multiple
volume,
replication
classes
with
the
same
schedule
on
the
on
the
clusters.
B
It
refers
to
the
dr
policy
so
that
it
knows
what
the
scheduling
interval
is
and
or
which
clusters
this
workload
can
be
placed
on.
It.
Auto
protects
pvcs
based
on
the
pvc
selector,
that's
provided
in
its
definition
and
its
spec,
and
it
provides
for
two
actions:
recovery
action
or
a
failover
action
which
moves
the
workload
to
the
failover
cluster
and
the
relocate
action
which
relocates
the
workload
to
the
preferred
cluster.
B
B
One
of
the
operators
is
running
on
the
ocm
hub
cluster,
which
is
the
multi-cluster
orchestration
plane
where
the
dr
policy
and
the
dr
placement
control
objects
are
created,
its
responsibilities,
obviously,
to
reconcile
the
dr
placement
control
object
on
the
managed
clusters
where
the
workload
should
be
running.
There
is
an
additional
volume,
replication
group
api
resource
that
is
present
and
managed
by
the
raman
cluster
operator.
B
We're
not
going
into
that
because
it's
auto
managed
by
raman
but
volume
replication
group
api
is
the
one
that
helps
detect
new
pvcs
that
are
created
and
are
when
pvcs
are
no
longer
in
use
so
that
a
final
sync
is
initiated
with
volume.
Replication-
and
you
know
all
in
cluster
activities
are
managed
by
the
ones
in
replication
group
api,
which
is
controlled
by
the
dia
placement
control
api
at
the
hub.
B
B
B
The
application
that's
been
deployed
is
a
busy
box
pod
in
the
busy
boxing
sample
name
space.
It
very
simply
echoes
the
current
timestamp
to
a
file
every
10
seconds,
and
the
file
is
actually
an
amount
point
which
is
backed
by
a
pvc
which
is
backed
by
cfrbd,
which
is
set
up
across
east
and
west
for
storage
replication.
B
B
It
basically
has
a
dr
policy,
which
is
the
policy
has
east
and
west
clusters
in
it.
It
is
going
to
reconcile
the
busy
box
placement
placement
rule
and
finally,
it
has
a
preferred
cluster
mentioned
in
it,
which
which
tells
it
which
cluster
to
prefer
and
it
was
east
and
that's
why
it's
deployed
it
on
east
and
it's
already
set
up
volume,
replication
and
other
such
requirements
for
the
pvc
in
use
and
the
user
doesn't
have
to
worry
about
those
things.
B
Now,
as
we
are,
writing
timestamps
to
the
file
and
we
want
to
demonstrate
recovery.
Let's
take
a
look.
Let's
tail
the
part
of
dzbox
parts
that
we
know
the
timestamps
as
we
come
back
to
the
hub
to
perform
a
recovery
to
the
west
cluster
and
we'll
see
what
kind
of
data
loss
we
encountered
so
to
perform
a
recovery
of
the
workload
on
the
best
cluster.
B
We
need
to
edit
the
dr
placement
control
and
we
need
to
provide
it
with
an
action
of
failover
and
we
need
to
give
it
a
failover
cluster
and
that
failover
cluster
in
this
particular
case
would
be
west.
So
the
workload
can
move
from
east
to
west
without
any
coordination
it
assumes
east
is
down,
although
east,
it's
really
not
down
in
the
example.
B
B
We
see
the
last
series
of
timestamps
that
have
been
dropped
on
the
east
cluster,
which
is
what
we
compare
in
the
west
cluster
and
on
the
west
cluster.
We
see
that
the
part
is
already
running.
So
let's
take
a
look
at
the
data
within
the
pod
and
see
what
data
we
lost,
because
this
was
a
recovery
operation,
failover
operation
that
we
performed
so
looking
at
out
file.
B
B
So
if
you
look
at
it
here,
we
got
3755
time
stamp.
We
lost
the
series
of
38
and
139,
which
was
about
a
minute
worth
of
data
that
lost
that
was
lost.
The
replication
schedule
that
we've
actually
set
up
for
this
demo
is
one
minute,
so
it
lost
about
a
minute
of
data
from
east
it
recovered.
It
started
running
again
at
39.13,
so
the
recovery
time
was
approximately
a
minute
and
20
seconds
give
or
take,
and
how
did
we
achieve
this?
B
The
next
part
of
the
demo
we're
going
to
do
a
relocate
where
we're
going
to
show
zero
data
laws.
So,
let's
start
tailing
the
busy
box
board
on
the
west
cluster
and,
in
this
particular
case,
we're
just
going
to
see
that
there's
zero
data
loss.
So
we
need
all
time
stamps
back
on
east
once
we
relocate
it
and
to
relocate
it.
We
go
ahead.
We
edit
the
dr
placement
control
object
again
this
time
around.
We
change
the
action
instead
of
a
failover,
we're
going
to
say,
relocate.
B
B
B
So
it
took
about
a
minute
and
30
seconds,
which
is
your
rto,
although
this
is
not
purely
recovery,
but
that
is
an
application
downtime
during
relocation.
And
how
did
we
do
this
again?
We
just
edited
the
dr
placement
control
changed
the
action
to
relocate.
It
had
a
preferred
cluster
and
moved
over
there.
B
B
B
B
Again,
there
is
work
going
on
around
volume
groups
and
volume
group
snapshots
in
the
data
protection
sig,
and
then
that
becomes
a
reality.
The
the
replication
of
this
these
set
of
pvcs
will
also
be
point
in
time
consistent
across
each
other,
providing
better
resilience
and
recovery
for
the
application
on
the
target
clusters.
B
We
also
want
to
move
towards
more
data
agnostic
replication
mechanisms,
for
example,
proposals
like
the
change
block
tracking
can
help
reduce
rpo
and
transfer
data
across
various
storage
systems.
They
do
not
have
to
be
in
the
same
storage
system
so
that
can
make
us
storage
agnostic
and
address
our
peers.
Instead
of
backups
using
backups
address
rpos
using
change
blocks
across
snapshots,
I
will
also
look
at
providing
more
arbitrary
replication
scheme
than
volume
replication,
so
that
data
can
be
transferred
across
peer
clusters
and
the
storage
systems
probably
don't
have
to
be
the
same.