►
Description
How do you backup OpenShift-hosted applications? What do you need to backup? Red Hat Principal Architect Annette Clewett joins us to discuss data protection and disaster recovery with Red Hat Advanced Cluster Management (ACM) and OpenShift Data Foundation (ODF) because even with the best of luck and planning, everyone needs backup and recovery mechanisms.
What is ODF? Red Hat OpenShift Data Foundation is a software-defined, container-native storage solution integrated with OpenShifts. When used with ACM, ODF provides a disaster recovery capability for OpenShift Data Platform (ODP) to failover an application from one OpenShift cluster to another and then, when ready, failback to the original primary cluster. We'll also take a look at Ramen, an open-cluster-management (OCM) placement extension that provides recovery and relocation services for workloads. So join us to learn more about data protection and disaster recovery for OpenShift with ODF and ACM.
Did you miss the previous episode about Backup and Disaster Recovery?
https://cloud.redhat.com/blog/ask-an-openshift-admin-office-hour-backup-and-disaster-recovery
Learn more about disaster recovery with ODF and ACM: https://red-hat-storage.github.io/ocs-training/training/ocs4/odf4-multisite-ramen.html
Learn more about Ramen:
https://github.com/RamenDR/ramen
Follow us:
Andrew Sullivan, https://twitter.com/practicalandrew
Jonathan Rickard, https://twitter.com/jrocktx1
Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
Openshift.com https://red.ht/33Fh6uV
OpenShift on github https://github.com/openshift/
OpenShift on redhat.com https://red.ht/2o4ccDk
#RedHat #OpenShift #ContainerPlatform
A
C
C
A
C
C
Yeah,
so
we've
been,
we've
been
married
for
almost
20
years
right
and
my
wife
is
like
my
birthday
is
what
I
prefer
like
you
know.
We
can.
We
can
tone
down
valentine's
day,
it's
okay,
yeah.
She.
She
doesn't
seem
to
mind
that
you
know
20
years
in
that's
awesome.
So
hello,
everyone
welcome
to
the
stream.
C
C
That's
what
we're
here
to
for
you
know,
ask
those
questions
feel
free
to
message
us
in
chat
on
whatever
platform
you
happen
to
be
watching
us
on
the
the
software
that
we
use
behind
the
screen
behind
the
scenes,
rather
restream
make
sure
that
we
get
those
regardless
of
wherever
you're
at
and
then
our
wonderful
producer
stephanie.
In
the
background,
she
does
a
great
job
of
making
sure
that
johnny
and
I
don't
don't
miss
questions
as
well.
C
B
Let
you
introduce
myself:
okay,
thank
you!
Yeah,
I'm
annette
clewett,
I'm
on
the
same
team
as
andrew
and
johnny.
I
I'm
in
the
hybrid
platform
business
unit
and
recently
I've
been
doing
a
lot
of
work
with
disaster
recovery
solutions
that
work
with
openshift
as
well
as
persistent
data
and
how
we
make
all
that
work
together
with
advanced
cluster
management.
C
Yeah
and
I
know
annette,
you
usually
have
some
of
the
most
popular
summit
sessions,
around
storage,
integration
and
all
that
other
stuff
with
openshift.
So
I
I'm
I'm
happier
here.
I'm
excited
to
talk
about
today's
topic.
There's
there's
some
really
interesting
and
really
cool
tools,
things
that
I
have
learned
a
lot
just
from
our
few
interactions
with
each
other
as
we
covered
this
topic.
So
it's
it's
something
that's
exciting
to
me,
so
our
hope,
nine.
I
see
your
today's
first
crazy
question.
Can
you
access
user
metrics
data
via
the
occli?
C
B
That
note,
we
want
to
just
answer
that
quickly,
though
just
so
I'll
be
showing
it,
but
essentially
it
connected
or
disconnected.
The
only
real
difference
is
whether
or
not
you've,
you
know
mirrored
the
images
that
are
needed
for
all
the
operators
and
the
the
custom
resources.
It's
really
no,
no
different
than
any
other
operator
or
or
capability.
C
C
C
I
should
say
and
dean
for
anybody
who
doesn't
watch
or
hasn't
seen,
dean's
blog,
he
did
a
phenomenal
follow-up
himself,
so
he
he
didn't
get
through
all
the
stuff
that
he
wanted
to
talk
about,
so
he
he
went
and
recorded
like
another
50
minute
video
or
something
that
talks
about
a
whole
bunch
of
stuff.
So
in
our
blog
post
we
link
over
to
his
blog
post
on
vegecate
dot,
co
dot.
Uk.
C
If
you
wanna
go
ahead
and
go
there
but
yeah
he
he
did
just
a
phenomenal
job.
I
know
dean's
anxious
to
come
back
and
talk
with
you
all
again
and
answer
those
you
know
our
hope.
Nine,
all
those
crazy,
crazy
questions,
so,
let's
kind
of
quickly
get
through
our
top
of
mind
topics
for
today.
So
that
way
we
can
focus
on
a
net
and
you
know
application
disaster
recovery.
C
C
So
we
see
rail
8,
but
we
don't
see
either
openshift
or
core
os.
So
I
had
a
couple
of
conversations
with
engineering
this
morning
I
had
a
couple
of
conversations
with
product
management
this
morning
about
what
what
does
all
of
this
mean
right.
So
openshift
is
affected,
but
two
varying
degrees
based
off
of
what's
inside
of
here.
C
So
if
we
scroll
up
here
to
which
one
is
this,
this
is
0
1
85.
So
we
can
see
here
openshift
container
platform
where
the
default
restricted
scc
is
used.
The
issue
is
not
exploitable,
so
open
shifts,
while
core
os
is
rel,
eight
based
and
therefore
it
would
have
you
know.
Obviously
the
same
kernel
openshift
is
not
affected
so
effectively.
What
I've
been
talking
with
product
management
about
and
we've
since
reached
out
to
the
product
security
team,
is
you
know,
can
we
still
get
openshift
listed
in
this?
C
C
Certainly
for
me
first
thing
I
look
at
is:
is
my
platform
affected
in
this
list
and
then,
if
it
is,
then
I
go
back
and
read
all
of
the
details
there
so
just
be
aware
for
zero
one,
eight
five,
that
one
is
not
or
does
not
affect
openshift,
but
it
does
affect
rel
for
the
other
one
four
zero.
Three
four
technically
openshift
is
affected,
but
to
a
lesser
degree
effectively.
Because
from
what
I
understand
we
don't
we
don't
encourage
or
we
don't.
I
should
say,
want
folks
to
do.
C
C
However,
I
did
see
this
morning
that
the
fix
for
this
has
been
incorporated
or
already
built
into
the
the
core
os
build
so
as
of
this
morning
or
maybe
late
last
night.
So
look
for
one
of
the
future
z
streams.
I
don't
know
if
it
will
be
next
week's
c
stream
or,
if
it'll
be
the
week
after
that,
but
in
one
of
the
soon
to
be
released
streams
that
should
be
fixed
inside
of
there
johnny.
We
have
a
question
in
chat.
A
C
C
Let's
see
the
next
one
that
we
wanted
to
talk
about
here
is
microshift,
so
microshift
was
and
if
we
look
at
the
blog
post
here
january
19th,
so
it
was
last
friday
was
when
they
published
this
and
it
made
a
bit
of
a
splash.
I
think
there
was
a
hacker
news
thread.
I
think
there
was
something
on
reddit
as
well.
That
talked
about
it.
C
A
A
C
C
I
haven't
had
a
chance
to
try
it
yet
I
only
have
one
raspberry
pi
and
it's
a
pi
3,
not
a
four,
so
I
haven't
had
a
chance
to
try
it,
but
it's
something
that
I'm
looking
forward
to
being
able
to
try
in
the
future
we'll
make
sure
to
post
all
of
those
links.
I'm
sorry,
I
haven't
been
paying
attention
and
posting
links
I'll
post.
This
micro
shift
one
first
into
twitch
here
and
then
I'll
post,
our
two
cves
just
after.
A
That
yeah,
I'm
really
excited
about
the
microsoft
thing.
This
is,
you
know
like
when
you
think
about,
like
a
small
binary
that
you
can
install
on.
You
know
an
iot
device
or
a
laptop,
or
something
like
that.
You
know
what
I
mean:
something,
not
not
quite
the
edge
data
center.
You
know
big
machine,
something
very
tiny
and
you
can
run
you
know
in
an
open
shift
api.
It's
pretty
awesome.
C
Yeah
yeah
and
they
walk
through
kind
of
this
blog
post-
is
really
in
depth
right.
They
they
walk
through.
What
is
micro
shifts
right,
how
it
communicates
what
it
deploys
all
of
the
different
components
that
are
associated
with
it
and
kind
of
why
it's
different
and
if
you
click
on
this
here
so
if
we
click
on
that
link,
it'll
take
us
over
to
the
github
repo
that
has
everything
that's
inside
of
microshift,
including
you
know
the
documentation
to
deploy
it
and
all
that
other
stuff.
So
it's
you
know
we
get
asked
a
lot
about.
C
C
So
I
know
lots
of
people
will
be
interested
in
that
yeah.
So
let
me
catch
up
on
chat
here.
Tiger
yeah,
microsoft
doesn't
support
all
api
types
cs.
That
is
correct,
so
it
will
be
a
subset
of
the
total
components
of
openshift.
It
won't
be
a
full
openshift
deployment
with
you
know
I
don't
know
precisely
what's
in
there,
but
it
may
not
have
metrics.
For
example,
metrics
may
be
a
an
add-on
day,
two,
depending
on
the
amount
of
resources
that
you
have
available
so
again,
look
through
the
docs.
You
know
check
out.
A
A
Yeah,
so
the
the
get
ops
operator
updated.
1.4.0,
like
you
said,
there's
a
lot
of
big
changes
that
are
coming
out
in
that
and
the
biggest
ones
are
obviously
like
the
helm
upgrade.
But
there's
subscriptions
are
now
a
resource
that
will
be
that'll.
Have
health
checks
against
it?
So
just
something
to
be
aware
of,
and
then
another
piece
of
this
is
it
looks
like
there's
some
rbac
stuff.
A
I
haven't
had
a
chance
to
mess
with
it,
but,
like
the
you
know
like
right
now,
you
have
to
kind
of
manually
implement
some
r
back.
So
that
way
you
can
have
cluster
level
management
at
some
of
your
lower
level,
argo
deployments,
and
so
it
looks
like
there's
a
lot
of
that
stuff
built
in
so
I'm
gonna
mess
around
with
that
talk
to
christian
and
those
guys
and
and
get
a
better
handle
of
how
that's
going
to
go
down
and
I'm
excited
about
it.
C
A
C
Oh
nice,
so
talking
about
secure
secrets
with
get
ups
and
all
that
yep
very
cool
I
may
have
to
tune
in.
If
I
have
the
opportunity,
hey
you
usually
that
thursday
afternoon
about
when
his
show
kicks
off,
is
when
I
want
to
sit
down
and
and
listen
to
our
show
in
order
to
create
the
blog
post
and
all
that,
and
I
can't.
C
Any
of
you
all
on
the
stream,
you
know
I
can't
even
talk
and
type
at
the
same
time
right
status
of
tecton
pipelines.
Should
I
use
tekton
pipelines
or
argo
cd
workflow.
I
don't
know
the
answer
to
that.
I
think
they
are
probably
slightly
different
use
cases,
but
it
is
outside
of
my
area
of
expertise,
so
techdon
pipelines
would
be
like
a
a
replacement
for
a
jenkins.
You
know
pipeline
or
something
like
that.
C
A
C
Oh
there
you
go
christian,
I
hope
leg
day
is
going
well.
If
anybody
doesn't
know,
usually
christian
listens
to
our
show,
while
he's
in
the
gym
and
for
whatever
reason
wednesdays
are
leg
day.
So
christian
I've
been
on
the
same
team
for
a
long
time
now
and
last
but
not
least,
devconf
so
devconf.
Sorry,
I
got
you
all
your
drinking
johnny,
so
devconf,
which
there
is
two
dev
comps,
there's
one
in
the
us
and
there
is
one
in
in
czechia.
C
So
the
one
happening
right
now
or
is
it
right
now
or
is
it
this
weekend?
Is
it
in
czechia?
So
I
know
christian
has
actually
gone
and
presented
at
it.
You
know
back
when
we
could
travel.
I
know
it
is
titled
dev
conf
like
implying
developer
conference,
but
there
actually
is
quite
a
bit
of
material
and
quite
a
bit
of
things
that
will
be
interesting
to
us
as
administrators.
C
So
I
will
I'm
still
sharing
so
devconf.
If
we
go
to
schedule-
and
I
happen
to
already
be
at
the
page,
schedule
is
live,
so
you
can
see,
there's
a
ton
of
stuff.
That's
all
here.
I
don't
know
if
this
is
a
paid
for
or
a
free
access
one,
but
look
through
the
session
catalog.
There
may
be
a
lot
of
things
that
are
interesting
to
you.
C
I
saw
that
there
was
several
get
ops
stuff
in
here
that
I
was
hoping
to
be
able
to
check
out
but
yeah,
it's
another
great
conference,
it's
another
one.
That
is,
I
think
this
one
is
hosted
by
red
hat
as
well.
It's
at
our
our
site
over
there
and
now
the
name
of
the
city
completely
escapes
me
brno,
brendo.
Thank
you.
Yep
yeah
and.
A
C
Yeah,
just
looking
here
on
on
my
screen,
I
see
daniel
daniel
does
he
always
has
great
presentations,
videos
on
the
engineering
team.
You
know
diane.
So
if,
if
you
are
on
the
kubernetes
slack
for
openshift,
you'll
see
the
dm
in
there
quite
a
bit
so
diana's
our
community
manager
and
stuff,
like
that.
So
there's
there's
quite
a
bit
of
a
red
hat
representation,
there's
also
quite
a
bit
outside
of
red
hat
as
well.
B
Okay,
yeah,
thank
you
andrew.
So,
first
off
there
was
a
question
in
the
chat
and
I'm
I'm
having
issues
figuring
out
how
to
answer
the
chat,
but
is,
is
red
hat
advanced
cluster
management
going
to
be
discussed
in
terms
of
disaster
recovery
and
the
answer
is
yes,
so
I
thought
what
I
do
first,
just
to
sort
of
start,
the
the
discussion
off
is
go
ahead
and
just
look
at
sort
of
the
the
different
scenarios.
B
B
B
B
That's
openshift
data
foundation
used
to
be
called
openshift
container
storage,
so
that
also
in
terms
of
deploying
the
the
storage,
the
openshift
data
foundation,
that
also
makes
use
of
the
topology
labels
and
and
lays
itself
out
across
the
azs
so
that
you
have
have
the
ability
to
protect
against
an
az
failure.
So
you
know
having
high
availability
is,
is
critical
right
because
you
want
to
be
able
to
survive
a
side
outage,
but
in
the
case
of
if
you're,
if
this
is
a
single
openshifts
right.
So
this
is
a
single
cluster.
B
B
So
it's
it's
about
the
kubernetes
resources
and
it's
also
about
the
persistent
data
in
the
case
of
the
solution,
with
with
red
hat
and
using
openshift
we're
gonna
we're
gonna
do
that
orchestration
with
advanced
cluster
management.
So
that's
currently
the
solution
that
that
is
actually
available.
As
of
today,
it
was
all
the
components
were
released
earlier
actually
late
last
year,
but
available
this
year,
so
all
the
components
are
available
and
in
it
it's
not
totally
clear
from
this
diagram,
but
I'm
going
to
do
a
demo.
B
So
you
you
get
the
idea
on
the
alternate
cluster
you.
You
are
not
actually
running
the
resources,
meaning
you're,
not
consuming.
You
know
cpu
memory
until
you
actually
need
to
use
the
application
on
the
alternate
cluster.
The
other
thing
that
may
not
be
clear
here
is
that
that
arrow
on
the
bottom
that
can
go
either
way,
so
you
could
have
some
applications
that
are
essentially
protected
on
the
side,
the
second
cluster
and
then
other
applications
that
are
protected
from
the
second
cluster
on
the
first
cluster.
B
B
C
B
B
C
B
Again,
yeah
and
this
this,
this
diagram,
you
know-
lacks
a
little
bit
of
that,
but
the
applications
in
order
to
for
this
this
solution
to
work
have
to
be
deployed
via
acm.
So
there's
lots
of
ways
you
can
deploy
applications.
You
can
use
a
get
ops,
you
can
use.
You
know,
there's
whatever
ways.
Acm
will
allow
you
to
deploy
applications
you
have
to.
It
has
to
be
controlled
as
well
via
acm.
C
B
A
B
B
You
know
doing
some
kind
of
backup
on
your
hub
cluster
or
having
some
way
of
making
sure
that
your
hub
cluster,
the
function
of
the
hub
cluster
has
to
be
available
right,
so
it's
possible,
it
could
even
live
in
one
of
the
other
clusters,
but
but
there's
you
know
right
now.
This
is
a
three
cluster
solution
of
which,
if
you
lost
your
hub
cluster,
you
become
sort
of
headless.
C
C
It
becomes
a
single
point
of
failure
and
even
you
know,
with
single
node
openshift
doing
an
update
means
that
everything
goes
down.
So
you
know
minimum.
You
probably
want
a
compact
cluster
three
nodes
and
you
know
just
be
again
cognizant
of
the
limitations
there
and
and
how
that
might
affect
availability,
yeah.
B
I
I
totally
agree
I
think,
for
for
pocs
and
and
things
like
that,
you
know
proof
of
concepts
where
you're
wanting
to
reduce
your
footprint
could
certainly
be
a
good
solution,
but
yeah,
you
know
you
you
this
whole
thing.
As
I
first
said
on
the
first
slide,
you
always
have
to
think
about
availability.
You
know
at
every
level
right
so
so
just
to
not
wanting
to
confuse
it
too
much.
So
this
is
clearly
an
asynchronous
solution
from
the
point
of
view
of
persistent
data.
B
C
B
Well,
this
solution,
I
think
of
it,
well
the
way
I
think
of
it,
if
you
think
of
it,
that
each
each
cluster
can
protect.
You
know
an
application
on
the
alternate
cluster.
It
is
sort
of
active
active
from
the
point
of
view
that
it's
it's
both
directions,
but
if
you're
asking
can
the
global
traffic
manager
or
a
geo
load
balancer
be
able
to
send
connections
to
both
applications
at
the
same
time?
B
No,
because
the
way
that
this
works
and
again
I'll
show
you
in
the
demo,
but
the
storage
is
only
essentially
promoted
on
one
side
or
one
cluster
at
a
time.
So
I
because
the
storage
is
only
promoted
and
available
on
one
cluster
at
a
time
I
can't
have
you
know
the
storage
essentially
being
being
replicated
and
available
at
the
same
time
on
both
clusters
on
a
per
application
basis.
B
Technically,
yes,
you
can
do
it.
The
the
asynchronous
replication
method
currently
is
using
something
called
suff,
ceph
mirroring
so
vsf
commands.
You
can
set
this
all
up
yourself
and
schedule
and
do
all
the
orchestration,
for
you
know
you
would
have
to
reinstall
the
acm
resources.
I
mean
the
excuse
me.
The
kubernetes
resources
you'd
have
to
make
sure
you
don't
have
any
security
context
problems.
B
B
Yeah
you're,
basically
down
in
the
weeds
of
ceph,
commands
on
a
per
volume
basis,
so
you're,
basically
having
to
do
all
of
the
orchestration
yourself
so
coming
in
the
future.
I
just
want
to
just
put
this
out
here:
it's
not
available
yet
we're
probably
looking
at
this
being
available
in
about
four
or
five
months.
So
one
one
of
the
you
know,
I
guess
issues
or
things
is
this
is
the
asynchronous
asynchronous
mirroring
of
the
data?
B
So
this
solution,
which
will
be
called
metro,
dr
multi-cluster,
for
for
red
hat,
will
use
acm
in
a
very
similar
way.
You
know
all
everything
about
installing
the
applications
and
the
automated
fill
over
all
that.
But
the
difference
is
this:
will
use
a
stretched,
storage
cluster
it'll,
be
from
openshift
data
foundation
and
if
you're
familiar
it'll
be
called
ceph,
it'll
be
a
stuff
cluster
and
it'll
have
definitely
the
ability
to
do
synchronous
smearing.
So
you
you
know
your
your
rpo
is
going
to
be.
B
So,
even
though
you
have
two
openshift
clusters-
and
you
know
they're
going
to
be
fine
once
you
fail
over
there's
no
quorum
issue
at
lcd,
there
would
be
a
quorum
issue
for
the
storage.
So
I
just
want
to
just
I'm
we're
not
going
to
go
over
this
anymore
today,
but
I
just
wanted
to
know
that
we
are
working
on
a
synchronous
solution
as
well.
C
Okay
and
just
to
rewind
a
bit
somebody
asked,
let
me
see
if
I
can
find
the
question
somebody
asked
if
this
is
like
is
the
whole
state
of
the
cluster
in
sync
between
the
two.
You
know
cluster
one
and
cluster
two,
and
the
answer
to
that
is
no,
because
they're
distinct
clusters,
they're
not
sharing
an
etcd
or
anything
like.
C
C
B
C
B
Yeah
yeah.
Thank
you.
This
is
three
to
four
months
down
the
road,
but
I
just
you
know
for
people
who
absolutely
need
a
synchronous
solution
which
you
know
may
be
true
or
then
you
know
this.
This
is
coming
and
uses
a
lot
of
the
same
custom,
resources
and
operators
in
terms
of
the
integration
and
automation
via
acm,
but
it
just
has
a
difference
in
the
in
the
storage
plane.
C
B
B
But
it's
not
it
you're
not
getting
as
much
alerting,
because
it's
an
external
cluster,
so
the
you
know
you'd
still
be
managing
the
ceph
sort
of
a
stuff
native
cluster
in
some
in
some
extent,
you'd
still
be
managing
it
outside
of
openshift.
There,
like
I
said,
there's
some
alerting
and
rook
is
definitely
still
the
rook
stuff
upstream
project.
Rook
is
still
doing
a
lot
of
the
orchestration,
but
at
the
level
of
just
you
know,
storage
classes
and
and
hooking
into
the
external
cluster.
B
B
A
External
stuff
storage:
is
there
any
when
you
have
that
third
site,
like
there
well,
not
necessarily
the
arbiter
side?
But
when
you
have
these
these
sites
you
have
do
you
have
to
have
like
an
increased
amount
of
storage
for
the
external
cluster,
or
is
it
like?
You
can
just
kind
of
go
same
same
and
then
it'll
it'll
balance
out
as
it's
failing
over.
A
B
A
B
So
if
you,
if
you
think
of
just
like
having
to
switch
from
the
cluster
one
to
cluster
two
and
let's
say
in
in
the
same
location
as
cluster
one,
you
have
your
your.
You
know
your
com,
your
two
replicas
for
the
odf
cluster
and
you
lose
those
you
still
have
two
replicas
available
on
the
other
site
and
because
the
arbor
node
is
not
on
either
of
those
sites.
B
The
arbiter
node
can
reach
consensus,
and
so,
therefore,
all
of
your
all
of
your
volumes
that
you
would
switch
over
and
use
on
cluster
two
once
they
became
available,
they
they
would
all
be.
They
they'd
be
too
replica.
Now
they
wouldn't
be
for
replica,
but
they
would
all
be
able
to
be
used
for
reading
and
writing.
C
C
A
B
Yeah
I
just
wanted
to
ask
stuff
if
she
could
cue
up
the
video
and
so
what
I'm
going
to
do
is
there's
no
audio,
I'm
going
to
talk,
which
means
I
may
lose
my
my
place
here,
but
I'm
going
to
try
to
talk
to
the
video.
I
also
have
the
same
video
recorded
on
youtube
with
my
audio.
So
if
you
can
go
ahead
and
start
it
stephanie,
I
don't.
Maybe
it's
already
started
there.
We
go
okay,
so
that's
gonna
start
talking,
so
we
have
two
managed
clusters.
B
The
application
is
currently
deployed
on
the
primary
cluster
and
underneath
we
have
openshift
data
foundation
and
on
ocp
we
also
have
a
third
cluster
running
advanced
cluster
management.
You
notice,
there's
openshift,
dr.
That
means
openshift
data
disaster
recovery
and
we'll
get
into
that.
So
the
process
that
happens
here
is
the
administrator
is
going
to
initiate
it
on
the
hub
cluster.
The
the
failover
after
that
the
metadata
for
the
alternate
cluster
pvs
will
be
put
into
the
the
secondary
cluster.
B
Then
we're
step
three
we're
going
to
go
ahead
and
demote.
The
storage
which
will
take
the
application
down.
Then
acm
is
going
to
redeploy.
The
app
on
the
secondary
cluster,
the
ios,
will
be
redirected
via
some
kind
of
geo
load
balancing,
and
then
we
delete
the
app
on
the
primary
cluster.
So
it's
no
longer
running
so
in
acm,
I've
already
imported
two
clusters:
I'm
going
to
rename
them
to
cluster
one
and
cluster.
B
Two
they're
amazon
one's
in
us
east
two
and
one
is
in
us
west,
two,
so
approximately
50,
60,
milliseconds
apart
and
what's
required,
is
you
need
to
you
need
to
connect
these
clusters,
so
I'm
going
to
connect
the
cluster
via
an
add-on
with
acm.
That
works
very
well
now
called
submariner
and
you
can
actually
deploy
it
from
acm
using
a
cluster
set,
and
we
can
see
that
the
connection
is
healthy.
B
B
On
the
second
cluster.
We
have
the
exact
same
thing.
The
dr
cluster
operator
comes
with
odf
and
we
have
submariner.
So
if
you're
looking
to
purchase
or
use
this
capability,
it
does
require
openshift
platform
plus
and
the
odf
advanced
sku
on
cluster
3
or
the
hub
cluster
go
to
all
projects
to
look
at
our
operators.
So
this
is
where
I
put
advanced
cluster
management.
B
We
also
have
a
new
operator
with
odf
for
multi-cluster
orchestration.
This
is
to
set
up
the
mirroring
between
the
the
storage
clusters
and
then
really
important
for
doing
the
orchestration.
We
have
the
openshift,
dr
hub
operator,
and
we're
going
to
look
at
the
custom
resources
or
apis
that
that
brings
so
in
terms
of
how
we're
going
to
test
this
or
do
the
failover.
I
have
installed
a
simple
application:
using
get
ops
called
rbd
loop,
really
simple
application.
B
B
So
the
next
thing
I
want
to
show
you
is
how
we're
going
to
fail
over
so
to
to
fail
over
we're
going
to
go
to
our
hub
cluster
and
we're
going
to
go
into
the
operator
operator,
dr
hub
operator.
So
here
we're
going
to
use
a
really
nifty
custom
resource
which
is
namespace
scoped.
So
this
is
on
a
per
application
basis.
B
So
it's
called
dr
placement
control.
So
this
is
a
brand
new
custom
resource
from
the
upstream
ramen
dr
project,
and
it
allows
us
to
set
the
action
that
we
want
to
do
for
this
application
and
before
I
do
that,
just
let
me
show
you.
The
rbd
loop
is
tied
into
a
grafana
dashboard,
and
so
it's
it's
able
to
show
you
that
you
know
what
cluster
is
currently
active
and
it
shows
you,
the
the
ios
and
we'll
see
some
more
information
on
the
dashboard
after
we
fail
over.
B
It
tells
us
that
it's
going
to
fail
over
to
cluster
two,
so
that
that
is
something
you
set
in
the
custom
resource,
which
is
what
what
is
my
failover
cluster
as
soon
as
I
save
the
changes
to
that
yaml.
The
actions
I
showed
in
the
flowchart
are
going
to
start
now
and
we
can
watch
it
in
a
couple
different
ways.
B
If
we
go
back
to
the
terminal,
we
can
see
that
if
we
now
look
for
the
resources
on
cluster
2
they're
already
there
and
if
we
go
back
to
our
dashboard,
we
can
see
it
just
switched
over.
The
green
is
under
the
failover
cluster.
So
let
me
explain
the
39
seconds,
so
the
39
seconds
that
shows
there
for
this
grafana
dashboard
is
the
amount
of
data
loss
or
the
time
of
the
data
loss
for
this
application.
B
And
again
it's
writing
4kb
every
second.
So
what
happened
was
the
last
snapshot
on
a
five
minute
interval
and
again
it's
settable
was
you
know,
39
seconds
before
the
failover,
so
I
I
didn't
have
that
data
on
the
alternate
cluster.
So
therefore
it's
lost.
I
basically
brought
the
application
up
on
the
snapshot
that
was
last
replicated.
B
So
one
once
I
save
that
we
can
go
back
to
our
terminal.
We
can
see
on
the
bottom
that
it's
terminating
the
resources
and
if
I
do
a
watch
to
look
on
back
on
cluster
one,
I
can
watch
right
now,
it's
not
quite
recreated,
yet
my
dashboard
doesn't
show
that
it's
switched
over
yet
and
just
while
this
is
happening
in
terms
of
relocate,
it
actually
does
a
sink
on
the
volume.
So
you
don't
actually
lose
any
data
on
a
relocate.
B
So
as
we
see
the
resources
come
in
there,
we're
going
to
basically
be
bringing
up
the
application
again
on
cluster.
One
and
as
soon
as
it's
running,
go
back
to
our
dashboard,
see
it
switches
over
and
that
40
seconds
in
this
case
doesn't
represent
data
loss
because
we
synced
the
the
volume
right
before
it
represents
application
outage
or
rto.
C
B
B
B
Yeah,
that's
fine,
I
mean
you
know,
I
mean
the
good
part
it
is.
It
is
an
active
project
and
I
know
the
the
the
contributors
and
maintainers
are
are
giving
are
briefing
the
cncf
data
protection
working
group
and
there's
a
lot
of
interest
you
know
for
for
other
companies,
but
they
might
change
out
the
replication
method,
but
to
use
the
orchestration.
B
B
You
you
could
do
it
there.
There
were.
There
are
some
new
custom
resources
for
volume.
Replication
like
you
know,
so
actually
you
might
be
able
to
move
up
some
of
that
to
doing
it
via
kubernetes
or
openshift,
but
again,
even
if
even
if
you
can
do
it
within
a
custom
resource,
it's
all
it's
going
to
be
per
volume
and
it's
going
to
be
all
manual.
C
Okay,
and
just
so
in
case,
anybody
doesn't
know
oadp,
so
my
understanding
and
and
that
I'll
rely
on
you
to
make
sure
that
I'm
right
here
odp
is.
It
is
a
downstream
of
valero
and
the
goal
being
to
provide
an
open
shift
api
to
kind
of
collect
all
of
the
objects.
Kubernetes
objects
that
make
up
an
application.
C
C
C
B
C
C
B
B
In
fact,
all
the
triggering
would
be
of
the
failover
would
be
from
that
third
location
in
the
future.
You
know
the
acm
team
is
looking
at
ways
of
backing
up
that
hub
being
able
to
restore
that
hub
having
on
the
hub,
so
there's
all
kinds
of
solutions,
because
you
can
see
that
you
know
the
hub
and
the
ability
to
to
trigger.
B
C
C
Yeah,
so
we've
only
got
a
couple
of
minutes
left.
I
I
think
I
told
you
all.
Well
you
the
the
folks
that
I'm
here
on
the
stream
with
that
I
do
have
a
hard
stop.
Today,
I'm
going
into
the
tower
for
the
first
time
in
two
and
a
half
years,
I'm
going
into
red
hat
tower,
so
I
do
unfortunately
have
a
relatively
hard
stop.
So
any
questions
that
you
all
have
please
go
ahead
and
send
them
in
now.
C
We'll
take
the
last
few
minutes
to
address
those
and
then,
while
you're
typing
those
up.
I
do
want
to
highlight
to
everyone
that
so
next
week
we
will
be
johnny
will
be
the
center
of
the
the
star
of
the
show
we'll
be
talking
about
validated
platf
validated
patterns
patterns,
eventually
I'll
stop
calling
it
platforms
johnny.
I
swear.
That's
okay,
yeah
validated
patterns
the
week
after
that,
which
is
february
9th.
We
will
be
talking
about
what
is
on
our
schedule.
C
C
So
next
three
weeks
are
really
busy
really
interesting
topics.
If
you
aren't
already
subscribed
on
whatever
platform
you're
watching
on.
I
would
definitely
encourage
you
to
do
that.
You
can
also
go
to
it's
red
dot,
ht,
slash,
live
stream,
so
that
will
take
you
to
the
live
streaming
landing
page
and
it
has
the
calendar
that
has
the
big.
You
know
the
big
live
streaming
schedule
on
it.
It
is
a
google
calendar.
C
A
And
andrew,
I
just
want
to
acknowledge
that
someone
had
mentioned
or
had
requested,
that
we
have
like
a
security
topic
going
over
like
how
to
you
know,
secure
clusters
covering
fips,
encrypting,
comms
secrets,
etc.
So
so
and
yeah
we
can
definitely
have
one
of
those
coming
up
like
there's.
I'm
sure
we've
got
like
you're
right
each
one
of
those
is
its
own
rabbit
hole
and
then
it
can
spin
off
to
its
own
right.
It
can
just
get
insane,
but
yeah
we'll
definitely
do
something.
C
Yeah,
so
actually
it's
probably
a
pretty
good
time
to
do
that,
because
there
was
just
some
shuffling
of
team
organization
on
the
product
management
side.
So
now
we
do
have
a
team
that
is
dedicated
specifically
full-time
to
nothing
but
security.
So
we
can
get
the
get
them
on
here
and
and
have
a
good
conversation
around
all
of
that,
so
the
compliance
operator
and
all
that
stuff
andrew
witch
tower.
Oh
yeah,
red
hat
tower
here
in
downtown
raleigh.
C
Yeah
tower
yeah,
so
I
I
was
telling
annette
before
we
started
that
I
had
to.
I
probably
spent
an
hour
digging
around
for
my
badge
last
night,
because
I
just
I
didn't
know
where
it
was.
I
dug
through
all
of
my
backpacks.
My
wife
makes
fun
of
me
because
I
have
too
many
backpacks,
you
know
and
it
did
finally
find
it,
but
all
right.
Well,
I
don't
see
any
other
questions
that
came
in
selwyn.
Thank
you,
johnny,
for
highlighting
the
topic
request.
C
If
there
are
any
other
things,
if
anything
comes
to
mind
for
anybody
who's
watching
this
after
it's
live,
please
don't
hesitate
to
reach
out.
You
can
reach
me
at
andrew.sullivan
redhat.com
or
on
twitter
at
practical
andrew.
All
one
word
if
you've
seen
me
chatting
in
twitch.
That's
that's
my
also
my
twitter
username,
also
through
johnny
under
the
bus,
so
johnny
j-o-n-n-y
at
red
hat,
no
h
and
j-rock
tx1
right
yep
got
it
right.
Yep!
Thank
you,
stephanie
for
throwing
up
our
information
on
the
screen
there.
C
So
by
all
means
you
know
please
reach
out
to
us
at
any
point
in
time.
We're
happy
to
field
those
questions.
We'll
include
a
net
in
those
as
well.
I
won't.
I
won't
ask
her
to
publish
her
information
publicly,
but
yeah
by
all
means.
Please
don't
hesitate
to
reach
out
to
us
and
we
will
address
those
questions.
C
A
C
I
don't
think
we've
scheduled
it
yet
we
were
waiting
for
oadp
to
go,
ga,
which
I
haven't
checked
on
that.
So
I
don't
know
when
precisely
that
will
be,
but
I
do
have
it
penciled
into
our
schedule
for
I
think
march,
but
it
is
pencil
then,
which
means
that
it's
likely
to
change
yeah.
So
when
we
realized
that
fips
has
to
be
implemented
at
install
yeah,
unfortunately
it
does.