►
Description
What’s your Recovery Time Objective and Recovery Point Objective? Do you have a disaster recovery plan for both OpenShift and the applications? Not sure what they are or how they're related? Let’s talk about what RPO and RTO are, their similarities and differences, and why you need to analyze your application priority to balance between resources and application availability.
In our last episode, we discussed high availability for OpenShift. Now we get to the big question: what happens if things go wrong? We’ll discuss disaster recovery scenarios and strategies, including some suggestions on what is, or isn’t, important to protect and recover.
About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Chris Short sit down to answer your questions about Red Hat OpenShift.
Follow us:
Andrew Sullivan https://twitter.com/practicalandrew
Chris Short https://twitter.com/chrisshort
Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
Openshift.com https://red.ht/33Fh6uV
OpenShift on github https://github.com/openshift/
OpenShift on redhat.com https://red.ht/2o4ccDk
#RedHat #OpenShift #ContainerPlatform
A
Good
morning,
good
afternoon,
good
evening,
welcome
to
another
edition
of
ask
an
open
shift.
Admin
say
that
five
times
fast,
I
am
chris
short
host
showrunner
cncf
ambassador
to
the
stars,
and
I
am
joined
by
the
one
and
only
the
always,
intelligent
and
insightful
andrew
sullivan.
How
are
you-
oh
man,
andrew
you,
should
tell
my
kids
that
yeah,
I
I'm
I'm
doing
this
for
your
kids.
I
realize
that
right
tell
them
to
go
to
twitch
now
and
start
from
the
beginning.
Yeah
there
you
go
they're
at
school,
so
maybe.
B
Although
it's
about
to
last
at
all
so
but
yeah
so
welcome
everyone
welcome
to
our
audience.
Thank
you
for
joining
us
today.
It
is
just
chris,
and
I
today
for
the
first
time
in
in
a
little
while,
it's
just
me
and
you
buddy
so
yeah.
I
guess
we'll
just
have
to
shoulder
that
burden
right,
make
like
atlas
and
carry.
A
B
B
Ultimately,
our
goal
is
to
answer
any
and
all
questions
that
you
have
around
openshift
chris
and
I
are
both
administrators
by
trade
or
we
were
at
least
a
little
while
ago,
and
that
is
where
much
of
our
expertise
comes
in
and
that's
why
this
is
ask
an
openshift
admin,
so
please
don't
hesitate
to
ask
us
any
questions,
anything
that
comes
to
mind
and
we
are
happy
to
do
whatever
we
can
to
get
those
answered.
If
that
means
we
can
answer
them
here
on
the
stream,
we'll
we'll
do
our
best.
B
If
we
can't,
if
it
is
something
that
we
is
outside
of
our
area
of
expertise
or
we
need
confirmation,
we
will
reach
back
into
those
resources
here
and
red
hat.
So
yep
engineers
love
hearing
from
us,
along
with
product
managers
and
everybody
else,
we'll
get
those
answers
and
we'll
follow
up
so
christian
hiked
mount
vesuvius.
Did
he
really.
B
A
B
In
the
absence
of
any
questions,
or
in
addition
to
any
questions
that
you
all
happen
to,
have,
we
also
have
a
topic.
We
try
and
focus
each
stream
on
something
that
we
can
use
to
help
educate
and
oftentimes
learn
ourselves
every
time.
I
do
one
of
these
streams.
I
learn.
I
learn
new
things
it
seems
like
so
this
is
certainly
no
exception
today,
where
we'll
be
talking
about
disaster
recovery,
so
this
is
kind
of
a
we
didn't
call
it
a
multi
series
or
a.
B
Series,
but
this
is
kind
of
the
follow-up
to
our
episode
or
stream
last
week,
where
we
had
christian
hernandez
on
to
talk
about
high
availability
with
openshift.
So
if
you
didn't
see
that
previous
episode,
you
can
find
it
across
all
of
the
streams.
So
all
the
platforms,
I
should
say,
youtube
both
the
openshift
and
red
hat
youtube
as
well
as
twitch,
there's,
also
a
blog
post
on
oh
gosh.
What
is
the
new?
B
It's
cloud.redhead.com,
blog
yeah
or
if
you
go
to
openshift.com
blog
it'll,
redirect
right,
but
every
week
we
have
a
blog
post
that
comes
out
that
that
we
link
all
of
the
information
as
well
as
to
timestamps,
for
when
we
talk
about
certain
things.
So
I'll
include
a
link
to
last
week's
episode
in
this
week's
blog
post
as
well.
B
We
also
have
I'm
going
to
start
calling
it
a
segment.
I
think
that
feels
appropriate
at
this
point,
because
we've
been
doing
it
basically,
since,
like
episode,
two
or
three,
what
I
call
the
top
of
mind
topics
yes,
so
these
are
things
that
they
have
happened
in
the
last
week
or
two
right,
usually
since
the
last
stream
things
that
have
come
up
that
I
feel
are
important
to
you
all
or
should
be
important
to
you
all
so
messages
that
we
want
to
get
out
things.
B
B
A
Yeah,
so
we
had
team
members
on
from
the
kubernetes
122
release
team,
which
is
the
first
kubernetes
release
in
the
new
release.
Cadence
of
three
kubernetes
releases
a
year
versus
the
traditional
four.
So
it
was
interesting
to
get
their
insight
in
a
not
only.
You
know
some
of
the
interesting
things
that
came
out
of
122
or
the
release
itself,
but
b
kind
of
like
how
the
sausage
is
actually
made
and
like
how
we
constantly
in
the
kubernetes
community,
try
to
improve
on
the
release
process
right.
A
So
the
release
team
works
with
a
release
sig
to
make
kubernetes
actually
go
out
the
door
right.
So
there
is
like
a
continuity
group
behind
it,
but
the
team
is
not
comprised
of
solely
sig
release
members.
It's
comprised
of
volunteers
from
across
the
spectrum
of
experience,
so
you
could
be
a
non-code
contributor
and
easily
participate
in
a
release.
You
know
they
need
to
send
emails.
A
They
need
to
send
coms
like
everybody
else,
so
you
know,
there's
that's
a
great
way
to
figure
out
how
to
get
involved
and
like
just
having
that
team
on
yesterday
gave
me
a
lot
of
energy
and
life
and
so
forth.
So
please
check
out
that
video,
and
you
know,
let
me
know
what
you
think
and
if
you
have
any
questions
about
the
release
process
or
anything
like
that
feel
free
to
hit
me
up
short
at
red
hat
or
chris
short
on
twitter,
with
two
s's
and
yeah.
A
A
Right
so,
but
this
also
means
now
that
when
we
quote
deprecate
something
that
does
not
mean
remove
in
kubernetes,
that
just
means
it's
going
to
be
removed
soon.
So
that's
a
good
delineation
to
kind
of
keep
in
your
head
right,
like
deprecated,
does
not
mean
removed,
but
now,
when
something
is
deprecated,
you
have
a
full
year
to
replace
or
upgrade
it
right,
because
sometimes
an
alpha
api
is
deprecated
and
the
beta
api
comes
in
or
the
actual
you
know,
v1
api,
the
production
grade.
A
Api
gets
upgraded
from
beta
to
just
generally
available,
so
like
those
little
subtle
changes
in
group
version
kind
really
really
matter,
especially
when,
like
oh,
we
just
updated
our
openshift
cluster
and
oh
there's
some
apps
that
are
not
working
for
some
reason.
Oh
well,
that
if
this
api
got,
you
know
deprecated
or
upgraded
or
whatever
so
understanding
that
process,
I
think,
is
very
important
right.
So
deprecation
does
not
mean
removal,
it's
deprecation
and
then
eventual
removal
three
releases
later,
which
equals
a
year
now
so
before
it
was
three
releases
later.
A
That
was
only
like
a
handful
of
months,
so
you
had
to
act
very
quickly
and
I'm
sure
you
know
openshift
will
adopt
a
similar
kind
of
cadence,
potentially
we'll
see
but
yeah
the
all
the
new
things
in
kubernetes
122
we
talked
about
yesterday
we
asked
everybody
the
favorite
topics
and
everything
else.
So
please
check
out
that
video.
I
think
it'll
be
very
insightful
for
everyone.
B
Yeah,
so
this
is
the
what's
new
in
1.22
blog
published
by
our
product
marketing
team
yeah,
so
karina
here
is
credited
as
the
author,
but
this
was
I
I
wouldn't
say
I
participated,
but
I
was
aware
of
the
draft
and
I
know
that
there
was
like
half
a
dozen
at
least
of
the
pms
who
all
contributed
into
this.
So
definitely
worth
the
read
if
you
haven't
seen
it
yet.
B
So
when
1.22
gets
incorporated
into
openshift,
so
4.8
is
1.21
based,
so
you
can
kind
of
guess
when
that
might
be
it
just
things
to
look
forward
to
things
to
be
aware
of
coming
up
the
other
thing
and
I'll
use
this
opportunity
to
let's
jump
back
to
the
blog
and
if
I
scroll
down
here
to
so
here's
last
week's
blog
post
around
the
stream.
Remember,
I
said
at
the
top
a
little
cheat
here
that
I
talk
about.
B
Sometimes,
if
you
select
the
category
openshift.tv,
all
of
these
show
up
and
what
I'm
looking
for
here
is
one
of
these
streams
and
the
top
of
mind
topics
I
talked
about,
and
I
should
so
I
use
this.
I
do
this
all
the
time
right.
I
dig
through
my
own
blog
posts
to
look
for
things
that
I
was
looking
for.
A
B
B
B
So
it's
a
great
way
as
chris
was
just
saying
with
1.22,
when
we
start
making
api
deprecations,
you
can
immediately
see
whether
or
not
those
are
being
used
and
whether
or
not
you
need
to
take
action
with
any
of
your
applications
or
anything
like
that.
Inside
of
the
cluster,
I
will
say
that
sometimes
those
are
operators
right,
our
red,
hat's
own
stuff
that
are
using
those
apis
which
that'll
get
fixed.
B
B
B
So
like,
if
I
do
oc
api
resources
see,
it
tells
us
all
of
the
apis
that
are
available,
but
it
doesn't
tell
us
whether
or
not
they're
in
use
right.
So
I
use
this
one
frequently
to
identify
like
a
crd,
that's
associated
with
a
specific
thing
so
like
I
can
do
oc
api
resources
and
grep
for
storage.
For
example,
I
can
see
all
of
the
things
that
are
associated
with
you
know:
storage
storages,
so
yeah,
we'll
we'll
have
to
dig
up
that
command.
I
don't
remember
what
it
is.
Let's
do
this.
A
A
B
B
B
Apparently,
I
need
to
use
the
full
object
name
wow,
so
I
will
copy
and
paste
that
guy
over
here
just
to
be
clear
on
it
anyways.
So
we
can
see
that
this
one
and
you
see
we'll
scroll
all
the
way
up
to
the
top
requests
and
current
hour
requests
request
and
last
24
hours.
Then
we
have
this
removed
in
release.
B
So
if
we
scroll
down
here,
we
can
see
this
ingresses
v1
beta1.extensions
is
removed
in
1.22.
So
this
would
be
an
indicator
that
there's
something
that
is
poking,
that
api
and
hey.
You
need
to
look
for
that
because
in
the
next
release
it's
going
to
be
removed
now
coming
down
here.
Here's
mutating
web
hook
configurations
v1
beta1,
so
it's
a
good
one
to
have
just
to
be
aware
of
again.
I
don't
think
that
there
is
a
way
to
tell
like
what
is
actually
poking
those
api
endpoints.
That
would
be
very
helpful.
B
B
B
A
B
All
right,
so
the
second
thing
I
wanted
to
bring
up
this
week
is
somebody
asked
because
we've
I've
shown
this
page
before
so
this
is
the
ci
release
page.
If
you've
never
seen
this.
I
use
this
cheat
sometimes
and
look
at
what's
coming
in
releases
right.
I
can
click
on
one
of
these
guys
and
I
can
come
down
and
see.
What's
changed
from
release
to
release,
I
can
get
an
idea
of
like
what
version
of
coros
is
being
used.
I
can
see
right
all
of
this
information
here.
B
However,
you'll
notice
that
we
call
this
for
stable
and
we
have
basically
every
open
shift
release
that's
listed
inside
of
here,
but
if
we
compare
that
to
all
right,
so
I'm
going
to
switch
over
to
console.redhat.com
now.
Okay.
So
if
we
compare
that
to
the
releases
over
here,
you
can
see
releases
with
4.8
we've
got
484
is
the
current
stable
channel
fast
is
485
and
candidate
is
486.,
but
I.
A
B
B
B
That
doesn't
always
mean
that
it
is
a
supported
deployment.
However,
right
so,
for
example,
you'll
notice,
there's
like
rc's
in
here
and
fcs
right
if
fc
is
feature
candidate,
rc
is
release
candidate
or
feature
complete,
maybe
feature
yeah,
something
like
that
anyways.
Obviously,
an
rc
is
not
a
supported
release,
even
though
technically
you
should
be
able
to
deploy
this
because
it's
all
green
over
here,
you
should
be
able
to
deploy
it
without
issue.
B
B
You
can
use
these
releases
just
be
aware
that
you
could
up
in
a
could
end
up
in
a
situation
where
you
have
deployed
an
unsupported
release
and
there
is
no
supported
method
to
get
to
a
supported
release.
So,
for
example,
I
could
deploy
rc3
today,
set
it
to
the
candidates
channel
and
then
update
to
like
4.8.6.
B
None
of
that
is
supported,
even
though
it's
all
here
in
this
quote,
unquote
stable,
so
apologies
for
the
or
or
hopefully
that
eliminates
confusion.
Apologies.
If
it
creates
confusion,
please
don't
hesitate
to
reach
out
so
going
back
to
what
I
was
saying
a
moment
ago
around
the
releases.
So
if
we
were
to
look
so
I
am
on
the
cincinnati
graph
data
github
page
and
our
hope,
nine.
I
see
your
comment
in
there
we'll
oh
yeah.
Let.
B
So
I'm
on
the
cincinnati
graph
data
github
repo,
so
this
is
where
the
clusters
pull
their
update
information
from.
So
if
you
were
to
go
into
your
cluster,
I
happen
to
have
a
cluster
here,
let's
close
that
guy
yep
and
I
come
down
here
to
cluster
settings
right,
so
the
data
that's
found
inside
of
here
these
values
come
from
what
we
find
over
here
in
these
various
locations.
So,
for
example,
if
I
go
to
channels
and
I
go
to
stable
4.8,
you
can
see
that
I
can
update
from
4.8.2
to
3
or
4..
B
B
If
we
scroll
all
the
way
down
here,
you
can
look
through
the
comments
and
read
through
them,
but
more
or
less
they're.
Continuing
to
see
this
or
they
continued
to
see
it
in
a
few
clusters
that
were
being
upgraded
from
4.7,
so
they
decided
to
go
ahead
and
block
it.
So
that
way
we
could
prevent
it
from
becoming
an
issue
for
anybody
and
this.
If
we
look
at
this
particular,
these
two
particular
comments,
we're
seeing
that
this
appears
to
be
fixed
in
4.8.5,
which
is
not
yet
unstable
right.
A
Our
hope
nine
is
so
they're
looking
for
the
tlsca
bundle.pim
for
the
api,
so
they
can
remove
the
insecure
skipped.
Cls
verifies,
you
know
flag
and
actually
get
their
money's
worth
from
the
ca
in
ocp,
so
thoughts
I
have
a
few
and
they
understand
that.
There's
documentation
how
to
replace
the
cda,
but
like
that
internal
ca
is.
B
I
don't
we'll
probably
have
to
reach
out
phone
a
friend
to
get
an
authoritative
answer
on
this.
So
my
understanding
is
that
replacing
the
ca
and-
and
I
guess
and
I'm
going
to
assume
that
this
is
specific
to
the
ingress
and
the
the
wild
card
certificate
and
and
the
api
certificate
sounds
like
they're.
Looking
at
just.
B
B
Dot
pem
placing
the
default
ingress
certificate
and
I'll
post
this
in
here,
although
I
know
our
hope,
9,
you
said
you,
you
were
aware
of
the
docs,
but
so
we'll
we
can
reach
out
and
we'll
ping.
Our
subject
matter.
Experts
on
this,
but
off
the
top
of
my
head,
and
you
can
see
here,
here's
replacing
the
trusted
ca
within
yeah.
So
that's
the
the
one
that
it's
trusting
from
a
proxy
adding.
B
A
A
B
B
A
B
So
this
was
an
interesting
one
that
came
from
we
we
talked
about
get
ops
last
week
with
christian
on
and
christian
also
has
get
ops
guide
to
the
galaxy.
His
live
stream,
which
is
every
other
week.
So
it
is
not
this
week.
It
is
next
week.
I
believe
yes
anyways,
you
can
see
the
the
red
hat
cop,
which
is
the
community
of
practice
inside
of
red
hat.
So
a
group
of
subject
matter,
experts
based
in
the
fields
very
passionate
people
who
are
are
interested
in
these
topics.
B
So
the
cop
has
this
git
ops
catalog,
which
just
has
a
huge
number
of
different
examples.
Right
things
you
can
use
and
you
can
scroll
all
the
way
down
here.
It
tells
you
right
not
officially
supported
by
red
hat
right.
It
is
cop,
not
official
red
hat,
so
it
is
red
hat
folks
that
are
contributing
to
it.
But
it
is
not
an
officially
supported
red
hat
thing
right
and
then
you
can
also
see
customers
are
encouraged
to
take
individual
items
of
interest
into
their
own
curated
catalog
and
maintain
it
but
anyways.
B
A
Yeah
I
mean
this
was
a
collective
effort
between
you
know:
christian,
the
get
offs
team,
the
the
red
hat
canada
folks
right,
like
it's
been
a
lot
of
work-
has
been
put
into
this
to
kind
of
give
you
example
ways
of
implementing
these
various
tools
in
in
a
git
ops
way
right.
So
using
openshift
get
ops.
These
should
kind
of
like
work
out
of
the
box.
A
You
know
once
you
fill
in
all
the
variables
and
everything
else
you
need
to
make
it
unique
for
your
environment,
so
yeah
I
it's
it's
not
officially
supported.
I
want
to
make
that
abundantly
clear,
but
it
is
a
great
way
to
get
a
look,
because
I
learned
by
example.
If
you
want
a
good
example,
this
is
a
great
place.
So
please
reference
it.
I
mean
you
can
see.
The
contributors
on
there
include
christian
include
andrew
block.
A
B
A
B
A
B
Probably
should
have
highlighted
this
before
it's
hard
to
see.
This
is
openshift
4.7
yeah,
and
if
we
come
down
here,
you
can
note
that
our
costs
right
coreos
now
is
now
based
on
8.4.
So
just
be
aware
in
case
you
encounter
any
issues.
I
would
think
you
wouldn't,
but
just
in
case
we
did
rebase
from
rel
8.3
to
8.4.
B
A
A
B
Yeah,
all
right,
I
am
going
to
stop
sharing
for
the
moment
so
disaster
recovery,
yeah
and
I
know
we're-
we've
got
about
30
minutes
left.
We
do
have
a
hard
stop
today,
so
we'll
get
through
as
much
of
this
as
we
can.
I've
got
a
massive
note
stock
on
this
topic
as
chris
island.
It
took
me
before
we
started
so
I'll.
Take
all
of
the
information
that
I
have
in
there
and
again
it'll
be
in
the
blog
post.
So
even
if
we
don't
get
to
it,
I'll
include
all
of
that.
B
We
just
need
to
tolerate
that
loss
in
capacity
or
some
loss
and
functionality.
Disaster
recovery,
in
my
opinion,
is
typically
it's
all
gone
right.
The
whole
cluster
is
now
gone.
So
how
do
I
now
bring
back
my
application?
How
do
I
now
recover
the
application's
functionality
and
that's
not
to
say
that
a
high
availability
event
can't
turn
into
a
disaster
recovery
event
right.
A
perfect
example
is,
like
your
storage
failed
technically
all
the
compute's
still
there.
B
So
that
way,
we
don't
just
go
off
the
rails
and
do
all
kinds
of
crazy,
mitigations
or
crazy
actions
that
don't
really
affect
the
things
that
we
want
to
protect
against
right.
So
hey.
I
want
to
protect
against
a
single
node
failure.
Okay,
does
that
mean
that
I
need
to
be
doing
things
like
excuse
me
that
I
need
to
have
you
know
500
replicas,
of
the
application
just
to
protect
against
a
single
node
failure?
Well,
maybe
maybe
not,
but
we
need
to
understand
what
we're
protecting
against.
B
If
so,
how
am
I
doing
that
if
my
open
shift
is
deployed
to
bare
metal?
Do
I
have
you
know
multiple
network
adapters
that
are
connected
to
multiple
switches?
Am
I
using
you
know
some
sort
of
lacp?
Am
I
using
or
other
bond
mechanism
in
order
to
provide
high
availability
for
those
ip
addresses
inside
of
the
cluster,
so
just
identifying?
B
What
all
of
that
is
taking
that
into
account,
making
sure
that
we
do
the
normal
set
of
stuff
associated
with
high
availability,
most
of
us
think
of
that,
especially
us
old
vert
admins,
like
me,
think
of
it
at
the
infrastructure
layer
right,
the
the
in
the
physical
layer
so
again,
redundant,
network
links,
redundance,
cpus,
even
rates,
and
you
know
error
correcting
ram.
All
those
things
to
protect
against
cosmic
rays
is
the
case.
B
B
B
If
all
of
my
nodes
are
in
one
rack,
and
all
of
them
are
connected
to
one
top
of
rack
switch
and
that
one
switch
fails
well,
no
matter
how
much
is
in
the
application,
if
everything
is
inside
of
there,
I
still
have
a
disaster
recovery
scenario,
so
that
brings
us
to
today
so
much
like
last
week,
where
I
want
to
start
with.
This
is
having
an
understanding
of
a
couple
of
different
things.
B
One
and
probably,
most
importantly,
is
two
three-letter
acronyms
that
I
hope
everybody
is
familiar
with
right,
so
recovery
time,
objective
rto,
which
is
for
lack
of
a
a
better
definition
or
more
terse
information
definition,
is
how
quickly
do
you
want
to
be
able
to
recover
your
application
right?
Your
cluster
bring
everything
back.
B
Recovery
point
objective
is
how
much
data
you're
comfortable
losing
as
a
result
of
that,
and
sometimes
we
have
to
temper
expectations
of
business
folks
right
because
they'll
always
say
rto
of
now
and
rpo
of
zero
right.
We
we
want
everything
to
always
be
up
and
we
want
to
never
lose
any
data
whatsoever.
Right
disaster
should
be
completely
and
utterly
transparent,
which
is
possible,
but
as
we
increase
the
or
excuse
me
as
we
decrease
the
rto
and
the
rpo,
the
complexity
goes
up
both
at
the
infrastructure
layer
and
at
the
application
layer.
B
So
we
have
to
strike
that
balance
again.
I
hope
none
of
this
is
news
for
those
of
us
who
have
been
doing
infrastructure
who
have
been
doing
ops
for
any
length
of
time.
We
have
probably
had
to
account
for
this
already
at
some
point
right,
kubernetes
cloud
native
applications
aren't
that
old.
You
know:
kubernetes
celebrated
its
sixth
birthday
this
year.
Chris,
I
don't
know
about
you.
I've
been
doing
this
more
than
six
years,
so
we
had
to
do
this.
You
know
previously,
and
it
certainly
still
applies
when
it
comes
to
our
openshift
clusters.
B
B
Can
I,
if
I,
if
I
am
planning
for
a
top
of
rack
switch
failure,
can
I
mitigate
that
through
some
mechanism?
Okay,
that
one's
pretty
easy
right,
multiple
switches,
yeah?
What
if
it's
a
rack
falling
through
the
floor?
What
if
it's
a
pdu
failing
and
I
lose
a
whole
row
of
racks?
What,
if
it's?
My
my
you
know
my
crack,
my
my
now,
I'm
gonna
forget
what
the
crack
stands
for.
It's
the
the
chilled
water
units.
B
What
if
my
chilled
water
unit
fails
and
now
my
data
center
is
running
at
140
degrees?
Been
there
done
that?
Yes,
me
too,
so
what
are
you
planning
for?
How
much
failure
are
you
planning
for,
and
the
next
step
is,
how
do
we
mitigate
that
against
that
and
to
be
fair,
some
of
these
things
will
get
lumped
together
right.
So
is
there
a
difference
in
your
disaster
recovery
plan?
If
it's
a
row
failing
or
the
entire
data
set
you're
failing,
maybe
there
is
maybe
there's
not.
B
I've
worked
with
customers
before
I
was
one
of
these
customers
who
we
would
spread
things
out
across
multiple
rack
rows,
so
some
things
were
in
the
same
row
across
multiple
racks.
Some
things
were
across
multiple
rack
rows
across
multiple
racks
in
each
one
of
those,
because
we
had
at
the
network
layer,
rack
row
level
redundancy
there
was
a
in
the
middle.
There
was
a
big
switch
and
in
each
rack
there
was
top
of
rack
switches,
so
we
could
tolerate
failure
at
different
levels,
same
thing
with
power
we
had
power
at
the
rack
row
level.
B
So
how
does
all
of
this
eventually
relate
to
openshift
and
our
applications,
and
I
think
it's
important
to
take
into
account
a
number
of
different
things
here.
So
something
happens.
Disaster
happens.
We
we
make
the
decision
right.
We
got
to
push
the
big
red
button.
That
says
you
know
disaster
recovery
plan
execute
so
what
needs
to
be
available
on
that
disaster
recovery
site?
What
do
I
need
to
have
there
so
and
forgive
me
if
I'm
starting
at
basics
here,
I'm
just
trying
to
level
such
right.
B
B
B
B
So
of
course,
that
meant
that
we
had
to
shut
down
basically
the
entire
data
center
yeah.
We
kept
up
core
routing
and
a
couple
of
other
things,
but
yeah
that
was
a
fun
one
and
also
that
the
so
that
the
guy
could
come
in
and
pull
the
thing
out
and
then
scrape
it
off
over
a
bucket
and
then
shove
it
five
minutes
worth
of
work.
A
B
My
my
least
favorite
one
was:
we
had
a
a
crack
that
was
adjacent
to
the
racks
and
the
the
condensation
tray
would
clog
and
overflow,
and
so
building
maintenance
is
solution
to
this.
So
it
was.
It
was
not
a
raised
floor,
so
it
was
a
concrete
floor.
Build
a
main
solution
to
this
was
to
take
some
l
channel
and
glue
it
to
the
floor
and
caulk
around
it.
So
that
way
you
got
basically
two
inches
of
leeway,
but
before
you
had
before
it.
A
So
yeah
I've
had
incidents
where
it's
like
I've
had
to
figure
out.
You
know
we're
gonna
do
snmp
traps
on
every
like
thermal
detection
like
every
thermometer
in
the
data
center.
I
want
to
know
each
thing's
point
and
it's
just
this
huge
page
of
temperatures
and
graphs
and
so
forth.
So
on,
and
you
know
thermal
dynamics,
it's
weird
how
they
work
in
the
sense
of
like
it'll,
be
seven
o'clock
and
that's
when
your
data
center
is
the
hottest
not
at
you
know
the
heat
of
the
day
kind
of
thing,
because
it
lags
so
yeah.
A
B
Yeah
our
hope,
9.
yeah
people
growing
up
in
a
vmworld
without
having
to
deal
with
hardware,
yeah
yep.
I
think
I
talked
last
week
about
network
admins
had
to
learn
about
all
of
this
back
in
the
late
2000s
of
you
know,
yeah
sure,
when
it's
just
a
bunch
of
desktops,
you
know
talking
to
exchange
or
something
like
that
and
you
reboot
a
router
and
they
lose
a
few
packets
when
it's
all
of
a
sudden,
you
know
thousands
of
virtual
machines
communicating
across
nfs
or
iscsi,
and
they
drop
a
few
packets.
B
That's
a
much
bigger
deal,
yeah
yeah,
so
I
don't
want
to.
I
don't
want
to
get
too
derailed
here,
but
the
important
thing
is.
We
want
to
make
sure
that
we
have
at
the
dr
site
all
of
the
hardware
requirements,
whether
they
be
literal
hardware
or
moving
into
something
that
is,
you
know,
pseudo
hardware,
a
new
virtual
environment
available
to
us
importantly,
and
sometimes
not
always
obvious
right.
B
Remember
things
like
if
you're
using
sri
ov
devices,
if
you're
using
gpu
devices,
if
you're
using
in
the
case
of
like
openshift
virtualization
right
making
sure
that
you
have
in
the
in
the
bios
or
efi
right,
you
have
the
virtualization
extensions
turned
on
learned
that
one
the
hard
way
too,
as
a
vmware
administrator
of
hey,
look
at
all
this
shiny,
dr
hardware,
that
I
now
have
to
go
in
and
I
have
to
reboot
each
one
of
them
open
the
bio
screen
set.
This
thing
wait
for
it
to
reboot.
A
B
So
what
I
mean
by
this
is,
if
you're
an
active
directory,
you
know
organization,
making
sure
that
you
have
domain
controllers
at
that,
dr
site,
so
you
can
do
things
like
authentication
authorization,
so
you
can
do
dns
right
because
usually
folks
tie
active
directory
and
dns
very
closely
together.
Dhcp
servers.
B
All
of
those
things
that
you
need
to
be
able
to
get
everything
up
and
running
where
this
gets
complex
is
at
the
next
step.
So
remember,
if
I'm-
and
this
is
where
I'm
trying
to
think
of
how
to
phrase
this-
the
way
that
I
want
there's
kind
of
two
approaches
here.
One
is:
I
have
an
existing
cluster
on
the
source
site
and
I'm
going
to
move
that
cluster
to
the
destination
site.
I
think
something
like
vmware
srm
or
literally
and
red
hat
virtualization
has
kind
of
the
same
thing
right.
B
So
this
brings
me
to
the
next
point,
which
is
okay.
Let's
say
that
I
am
right.
I
have
either
a
warm
or
a
cold
destination
cluster
does
it
have
to
be
exactly
the
same,
and
this
is
an
interesting
question.
This
is
one
that
I
don't
have
a
straightforward
answer
to
right,
so
my
source
cluster
is
running
on
you
know:
it's
got
three
control
plane
nodes.
I've
got
30
worker
nodes
and
you
know,
let's
say
altogether:
it's
300,
vcpus
or
300
cpus
worth
of
compute
resources
and
three
terabytes
of
ram
on
the
destination
side.
B
B
What's
going
to
happen
there?
Is
it
going
to
be
able
to
accommodate
that
if
I
have
now,
let's
take
it
to
the
next
step,
maybe
my
application
team
has
also
put
in
place
limits
with
the
assumption
that
we'll
have
say
20
of
those
instances
up,
so
they
have
its
size
so
that
I
need
20
instances
of
this
pod
to
hit
the
cpu
and
memory
needs
of
my
application,
but
my
destination
cluster
only
has
three
nodes
and
a
hard
anti-affinity
rule
that
is
preventing
me
from
deploying
any
more
pods
than
that
of
this
particular
application
instance.
B
B
B
This
is
one
of
those
areas
where,
even
if
it's
a
cloud
native
application,
the
apps
team,
as
far
as
they're
concerned
they've
done
everything
right.
Horizontal
scalability
sizing
everything
right.
We
don't
want
to
overwhelm
nodes.
We
want
to
give
them
the
possibility
of
rescheduling
of
descheduling
of
right
eviction.
All
this
other
stuff,
they've
done
everything
right.
We
could
have
just
broken
their
disaster
recovery
plan
all
because
we
used
new
servers,
yeah.
B
B
Exactly
so
yeah
christian,
I
see
a
couple
of
couple
of
messages
up
above
or
no
I'm
sorry.
I
hope
none
responding
to
christian,
so
yeah,
absolutely
being
cognizant
being
aware,
working
with
your
peers,
working
with
your
teammates,
I
used
to
tell
folks
you
know
when
I,
when
I
would
give
presentations
at
at
vmworld
and
other
conferences
of
you
know:
hey
you've
got
this.
You
know
nfs
based
storage
over
here.
You've
got
your
vmware
data
center
over
here.
There's
this
giant
black
hole
in
the
middle.
Also
called
a
network.
B
B
B
Maybe
you
incorporate
that
into
your
rto.
Like
hey,
you
know
our
dr
scenario.
Is
it
we
start
at
zero,
and
then
we
launch
everything
which
means
that
we
have
to
pull
the
basically
kick
off
a
whole
pipeline
process.
Pull
the
code
in
go
through
a
build,
go
through
a
full
test
suite,
let
it
build
the
containers,
push
the
containers
of
the
registry.
Let
it
now
deploy
you
know
using
deployments
or
replica
sets
whatever
it
happens,
to
be
the
application,
let
it
recreate
all
of
the
services
and
routes.
B
B
Of
threads
yeah
and
that's
not
like-
and
I
think
there's
also
you
know-
there's
a
lot
of
tests
and
a
lot
of
other
things
that
go
on
inside
of
there.
So
you
know,
do
you
need
jfrog
right
artifactory?
Do
you
have
an
artifactory
instance
that
is
holding
some
of
those
artifacts?
You
know
locally
and
jfrog
is
just
the
one
that
came
to
mind
because
I've
worked
with
them
before,
but
you
know
other
similar
things.
B
Another
thing
to
take
into
account:
hey
we're
using
a
a
a
registry
proxy
right,
basically
a
an
inline
caching
registry,
so
we're
used
to
a
build
process
taking
30
minutes
because
really
all
of
those
registry
or
all
of
those
source
containers
right.
The
the
base
images
are
cached
locally
at
the
dr
site.
Are
they
already
cached?
What
if
I
have
to
go
up
and
start
pulling
them?
Oh
no
docker,
just
throttled
us.
A
B
A
B
Very
much
and
and
all
of
this
is
you
know
me
indirectly
reiterating
the
point
that
I
think
it
was
both
peter
and
rhop9
have
made
here
in
the
chat
which
is
test
test
test
test
test.
You
know,
vmware
and
and
srm
did
a
really
cool
thing.
They've
been
doing
this
for
a
while.
Now
of
you
can
basically
test
your
dr
plan
in
a
bubble
and
yeah
and
not
know
this
yeah.
B
So
if
that's
an
option,
if
that's
something
that
you're
using
it's
a
great
way
to
test
and
validate
that,
I
used
to
when
I
worked
for
a
storage
vendor,
we
had
a
customer
that
they
had
two
sites,
one
in
one
in
atlanta
and
one.
It
was
also
in
georgia,
but
I
can't
think
of
the
city
now
and
effectively.
They
would
run
six
months
in
one
location
and
six
months
in
the
other
location
and
twice
a
year.
They
would
test
their
dr
because
they
would
literally
move
operations
to
the
other
side.
B
B
B
So
this
is
one
of
those
scenarios
could
be.
Aha
could
be
dr.
Do
you
need
to
update
something
external
to
include
or
to
point
to
the
new
site?
The
new
location,
alternatively,
is
the
destination
cluster?
Let's
say
it's
something
that's
already
running
or
something
that
you
set
up.
Does
it
need
to
assume
the
identity
right,
the
dns
names
of
the
old
one?
Maybe
it's
as
simple
as
setting
an
apps
domain
right.
So
that
way,
your
routes
have
the
right.
You
know
domain
name,
that's
being
used
for
them.
Something
like
that.
B
B
If
something
happens
in
one
right,
this
goes
bye-bye
I
seamlessly
or
almost
seamlessly
transition
over
to
the
second
one.
So
that
is
broadly
what
we
define
as
a
hot
dr
system.
A
warm
dr
system
would
be.
If
we
switch
back
over
here,
more
or
less
I
have
everything
running.
I
can
have
two
sets
of
infrastructure
right.
I
can
have
two
clusters
running
but
on
this
destination
cluster.
So
what
this
diagram
calls
ocp2,
I
don't
have
the
app
actually
deployed
right.
Data
is
replicating.
B
I've
got
all
of
the
components,
all
the
parts
and
pieces
replicating
everything
is
there
and
ready.
I
just
need
to
actually
deploy
the
app.
Let
it
scale
up
flip
everything
over
to
the
destination
side
right.
That
would
be
considered
a
warm
dr
type
of
scenario
so
cold,
dr
the
last
one
that
we
have
listed
here
so
cold
dr,
would
be
more
or
less
none
of
this
exists,
and
by
none
of
this
I
mean
open
shift
wise.
B
That's
not
to
say
that
you
don't
have
you
know
a
vsphere
or
a
rev
environment
or
an
openstack
environment
or
physical
hardware,
with
all
the
network
connections
and
everything
configured
already
there.
But
it's
I
need
you
know
step
one
to
my
disaster
recovery
plan
called
the
r
plan
is
deploy
openshift
right.
That
is
effectively
a
cold
airplane.
B
A
Would
it
be
possible
to
get
chat
available
for
the
videos
because
on
youtube,
which
seems
to
be
the
archive
that
can't
be
viewed?
Yes,
so
that
is
actually
a
feature
that
we're
enabling
now
thanks
to
youtube,
exposing
it
to
us.
But
you
can
always
go
to
our
discord
and
go
to
the
live
stream
channel
in
there
and
literally
every
chat
message.
That's
ever
happened
is
there
and
it's
awesome.
A
A
B
As
you
politely
reminded
me,
we
are
out
of
time.
So,
yes,
thank
you
to
everyone
who
joined
us
today.
My
apologies
for
not
being
able
to
get
all
the
way
through
our
topic,
we'll,
maybe
spend
a
little
bit
of
time
next
week.
I'll
definitely
put
everything
into
our
notes
for
this
week
be
sure
to
keep
an
eye
on
cloud.redhead.com
blog
for
the
blog
post.
B
Thank
you
again,
chris,
for
your
help.
Thank
you
to
our
audience.
Thank
you
for
joining
us.
If
you
have
any
questions,
if
there's
anything
we
didn't
get
to,
please
don't
hesitate
to
reach
out
andrew.sullivan
redhat.com
or
on
social
media
practical
andrew
all.
One
word
just
like
you've
seen
in
the
chat
across
the
various
streaming
platforms,
yep.
A
I'm
chris
short
on
twitter
and
short
at
redhat.com,
and
you
can
always
find
us
on
discord.
Absolutely
all
right,
good
good
chatting
with
you,
everybody
and
all
of
you
out
there
pro
gamer
move
on
the
chat
logs.
Thank
you
all
right.
Coming
up.
Next
is
openshift
commons
briefing
we're
going
to
be
talking
a
little
bit
about
security.
So
if
you're
interested
in
network
controls-
and
you
know
deep
packet
inspection-
please
join
us
until
next
week-
stay
safe
out.
There.