►
From YouTube: Plan to Fail: A Good Captain Doesn’t Sail Without Life Rafts - Steven Wong & Carlisia Campos, VMware
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Plan to Fail: A Good Captain Doesn’t Sail Without Life Rafts - Steven Wong & Carlisia Campos, VMware
Historically, formal disaster recovery (DR) plans were only feasible for large enterprises. They could afford to allocate time, resources and the cost of duplicating a datacenter infrastructure. With the popularity of public cloud and cloud native technologies, the cost and complexity of DR planning has been significantly reduced. This means every company, large and small, can engage in business continuity planning. Why is this important? These are some of the reasons: Machines and software fail - People make mistakes - Hackers prey on the vulnerable - Weather, fire, terrorism, more... - You lose customers when there are outages and data loss - Legal standards often require data retention This talk will focus on: - Items that need to be backup and why - some might surprise you - Why you need selective restore capability - Existing tooling to simplify and automate a DR strategy
https://sched.co/NroN
A
If
you
came
to
a
session
on
disaster
recovery
planning
for
kubernetes
you're
in
the
right
room
presenters
today,
myself,
I'm
Steve
Wong
from
Los
Angeles
I
work
for
VMware
I'm
working
on
the
vSphere
cloud
provider,
I've
been
active
in
the
storage
saying
in
other
aspects
of
kubernetes.
Since
2015
and
as
part
of
my
career
history,
I
joined
VMware
when
my
startup
Albemarle
was
acquired
and
I
worked
in
the
backup
field
for
about
11
years.
B
A
So
the
agenda
day
we're
going
to
start
with
why
you
need
a
recovery
plan.
If
I
can
convince
you
that
you
do
need
one,
then
we'll
get
into
the
elements
of
what
should
be
in
it,
going
into
details
of
what
you
need
to
backup
and
why
and
finally
we're
going
to
wrap
up
with
a
survey
of
existing
tools
that
will
help
you
do
backups
and
recoveries
when
running
on
kubernetes
and
we're
going
to
close
with
a
demo
of
one
of
those
tools.
A
First,
to
start
out
why
you
need
a
recovery
plan.
Well,
people
are
obviously
aware
that
hardware
is
never
perfect
and
machines
and
software
fail.
In
my
experience,
working
in
the
backup
field.
Actually,
people
like
to
blame
the
hardware,
but
the
actual
statistics
tend
to
tell
you
that
most
of
the
really
ugly
scenarios
are
based
on
human
error,
where
people
make
mistakes.
You've
also
got
exposure
to
weather
terrorism,
crime
earthquakes,
a
long
list
depending
on
your
industry.
Many
industries
are
also
faced
with
conducting
backup
simply
because
regulations
require
them
to
retain
data
for
certain
of
years.
A
So
the
same
thing
you
would
do
the
same.
Backups
that
satisfy
legal
requirements
for
data
retention
can
be
a
part
of
your
backup
recovery
plan.
There's
a
lot
of
elements
of
a
disaster
recovery
plan.
You
know
it
isn't
all
about
backups.
You
need
to
plan,
for
you
know,
in
the
event
of
a
physical
disaster,
you
need
to
have
made
prior
arrangements
for
replacement,
Hardware
or
a
replacement
public
cloud
hosting
so
that
you
can
attempt
to
stand
it
up
at
your
recovery
site.
A
You
also
need
planning
you
don't
wait
until
your
system
is
your
data
center
is
on
fire
and
then
decide.
What
do
we
need
to
get
this
thing
back
on
the
air?
A
plan
in
one
person's
head
isn't
good
enough
you're
going
to
have
to
have
people
who
can
carry
out
whatever
you've
listed
in
that
plan
and,
finally,
those
people
need
to
be
training
trained
and
they
need
to
practice.
A
It's
not
enough.
Just
to
make
the
plan
it
has
to
be
a
living
document.
I've
I've
heard
of
places
that
write
this
plan
have
the
disaster
five
years
later
and
discover
that
so
much
has
changed
that
it
no
longer
is
a
viable
plan
and
it's
not
enough
to
document
it
and
keep
it
on
the
data
set
or
infrastructure
you're
trying
to
replace
there's
a
few
ugly
stories
of
people
who
had
an
elaborate
back-up
plan.
But
then
they've
lost
power
lost
the
data
center
and
they
no
longer
have
access
to
it
it.
A
It
might
be
a
good
idea
to
actually
physically
print
this
out
with
multiple
copies,
so
then
you're
guaranteed
to
have
access
to
it
or
to
keep
it
on
standalone
tablets.
Tablets
are
perhaps
a
little
more
high-tech
than
printouts,
because
you
can
do
things
like
have
checklists
as
you
go
through
a
process
to
try
to
eliminate
mistakes,
and
that
is
one
thing
that
I
believe
the
airline
industry
does
now,
where
the
pilots,
with
most
commercial
airlines,
keep
plans
on
tablets
that
the
pilots
routinely
practice
with.
A
What
do
you
want
to
protect?
Well,
it's
Theory
all
data,
cuber
Nettie's
objects
are
stored
in
the
sed
database,
so
periodically
backing
up
the
sed
cluster
is
important.
So
that
you
can
recover
a
kubernetes
cluster
and
this
this
statement
I'm
not
making
it
up
it's
in
the
kubernetes
Doc's.
The
sed
database
also
was
used
to
hold
state
for
some
of
some
popular
Network
plugins.
Not
all
there
are.
You
know.
A
The
modern
trend
of
extending
kubernetes
with
CR
DS
is
generally
going
to
hold
the
state
associated
with
those
CR
DS
in
sed,
but
there
is
critical
state
that
isn't
in
at
CD,
so
at
CD
isn't
going
to
be
enough.
If
you
have
a
Total
Wipeout
of
a
data
center,
some
things
that
live
outside
at
CD,
persistent
volumes.
So
if
you're
running
containerized
databases,
they
use
persistent
volume
mouths,
the
content
of
those
volumes
is
often
is
generally
going
to
be
large
and
obviously
not
in
the
at
CD
key
value.
Store.
A
Kubernetes
has
widespread
use
of
certificates
and
key
pairs
certificate
authorities,
and
if
you
wipe
everything
out,
if
you
really
want
to
get
a
restore
and
have
remote
clients
attach
and
get
authenticated,
you're
going
to
have
to
restore
those
certificates.
So
it's
a
good
idea
to
keep
backups
of
those
available.
A
You've
got
if
you
have
C,
RDS
and
CI
a
plugins,
not
using
sed.
You
need
to
figure
out
where
that
state
lives
and
cover
it
often
you're
dependent
on
physical
networking
infrastructure
things
like
configuring,
DNS
rough
records
of
routing
I
assignments,
and
if
you
do
a
rebuild
from
scratch,
you
want
to
record
those
things
so
that
they
can
be
recovered.
A
Likewise,
if
you're
running
on
a
cloud,
a
public
cloud
or
private
infrastructure,
there's
very
likely
to
be
a
bunch
of
account
and
configuration
status
and
credentials
for
that
redundancy
alone
is
not
enough.
If
you,
there
are
many
whose
first
impression
is
that
hey
I'm
running
a
three
node
at
CD
cluster,
if
I
lose
one
of
those
nodes,
inherently
Etsy
didn't
go
down,
but,
like
I
said
earlier,
the
hardware
failure
isn't
the
only
thing
you're
protecting
against
in
a
human
error
scenario.
A
If
someone
deletes
an
object
and
discovers
this
five
minutes
later,
that
change
will
have
been
replicated
more
or
less
immediately
to
all
three
nodes
of
your
red
CD,
so
you're
not
going
to
be
able
to
roll
it
back.
You
know
simple,
mirroring
of
what
the
state
is
now
on.
Multiple
things
only
protects
against
a
subset
of
the
failures
you're
potentially
exposed
to
other
things.
To
consider
in
the
realities
of
your
stateful
apps
are
that
even
your
stateless
apps
is
that
these
things
are
based
on
container
images.
A
That's
now
gone,
and
even
if
you
did
have
a
list
of
what
images
are
supposed
to
be
in
there,
it
might
take
a
long
time
to
get
those
things
restored
and
even
if
you
pointed
to
docker
images
with
particularly
Buhl's,
if
you're
familiar
with
how
that
works,
there's
no
guarantee
that
the
one
you
had
before
is
actually
identical
to.
What's
out
there
with
the
same
label.
A
Right
now
there
are,
you
can
put
things
in
docker
hub
with
pretty
loose
labeling
you're
not
supposed
to
do
it,
but
labels
like
latest
and
whatever
and
end
up
with
something
reproducible.
So
I'd
suggest
you
that
there
are
container
image
registries
that
are
out
there
capable
of
replication
so
that
you
can
replicate
them
to
an
outside
failure
domain,
maybe
even
to
another
image
repository
in
a
cloud
or
at
a
distant
location
and
then
count
on
the
replication
in
the
opposite
direction,
to
bring
things
back.
A
The
way
they
were
originally
helm
can
helm
charts
and
maybe
other
binaries
fall
into
that
same
category.
It
may
not
even
be
a
bad
idea
to
backup
any
things
associated
with
your
hardware
like
I.
Don't
know
that
base
images
for
os's,
maybe
even
going
so
far
as
firmware
updates
in
case
you
have
replacement
Hardware
that
might
be
sitting
there
acquired
two
years
ago
and
not
have
been
getting
updates.
A
B
Well,
I'm
back
so
when
we
think
about
cognitive,
backup
solutions,
we
usually
think
people
usually
think
of
three
scenarios:
it's
either
disaster
recovery,
they're,
archival
or
migration
and
the
tools
there
are
available
today,
as
at
least
as
far
as
we
are
aware-
and
so
there
are
open
source
at
least
I
listed
on
this
right
at
the
following
slide
on
this
slide
are
tools
that
do
that:
backup,
persistent
volume,
I'm.
Sorry,
these
tools
only
backup
the
coronaries
objects
and,
as
you
would
expect
each
of
these
tools,
they
have
different
levels
of
different
sets
of
features.
B
Do
that,
though,
I
want
to
give
you
a
visual
of
a
very
high-level
architecture
of
the
lair
Oh?
Basically
you
when
you
want
to
backup
a
cluster,
you
will
start
a
little
on
that
cluster
and
you
set
up
an
app
three
buckets.
Usually
the
bucket
is
going
to
be
on
the
same
cluster
in
the
same
provider
as
that
cluster,
but
it
doesn't
have
to
it
can
be
in
a
different
provider.
It
can
be
on
your
local
machine.
B
It
can
be
anywhere
where
you
can
set
up
another
three
buckets
and
when
you're
going
to
restore
that
cluster
into
another
cluster,
you
install
velaro
on
the
second
cluster
and
you
create
a
restore
our
first
point,
velaro
to
the
same
as
three
buckets
right:
that's
how
it's
going
to
get
a
backup
and
and
that
at
three
buck
is
the
source
of
true
truth
for
the
cluster.
So
you
point
the
sacrum
custard
to
the
s3
bucket
and
you
create
a
restore
and
that
will
restore
the
custard.
B
B
All
right,
so
what
before
I
start
with
the
demo
itself,
I
want
to
show
you
what
I
have
going
on
here?
The
I
have
a
window
that
is
pointing
to
a
cluster
that
I
called
developments
and
I
have
another
window.
That
is
pointed
to
a
cluster
that
is
called
staging
and
you
can
track
by
the
bots
on
the
bottom
here.
The
barrage
highlights
you
can
see
as
I
move
through
development
staging
now.
How
are
these
clusters?
How
are
this
cluster
set
up?
They
were
set
up
on
my
local
computer
on
my
local
machine.
B
Eyes:
I'm
not
showing
oh
yeah,
it's
up
there.
Okay,
sorry,
it's
hard
to
see,
but
I
have
a
mini
container.
I,
have
a
kind
I'm
running
a
docker
container,
four
kinds
for
the
development
cluster
in
another
one
for
the
staging
cluster
and
on
the
development
caster.
If
I
do
cube,
control
get
name
space,
you
see
the
very
bottom:
one
is
valera,
so
that
means
the
Lehrer
is
installed
in
that
in
this
cluster.
Now,
if
I
go
to
the
my
staging
cluster,
then
I
do
get
name
space.
B
You
don't
see
the
Valero
namespace
it's
because
it's
not
installed
there
we're
going
to
stop
I'm
gonna
install
it
there
later.
So,
let's
just
deal
with
this
for
the
moment
when
I
keep
clearing
this,
let's
see
what
do
we
want
to
do?
First,
first
I'm
going
to
check
if
there
is
any
backup,
my
cluster,
no
there's
not
alright
do
I
want
to
cap.
Now,
no
I
don't
want
to
back
up
now.
I
want
to
do
something
interesting
before
we
create
a
backup.
B
B
B
B
B
He's
going
to
get
the
name
of
the
nodes,
okay
seems
to
be
up
I'm,
going
to
jump
up
here.
Then
I'm
going
to
do
a
curl,
because
I
know
it's
on
port
8000,
and
you
see
here
welcome
to
names.
That
means
okay,
my
server
is
up
and
running
and
I
can
keep
curly
and
you
see
on
the
bottom
handle
in
connection.
It's
all
good
great,
so
I
have
a
cluster
I
have
a
server
running
on
that
cluster
is
life.
It's
responding.
B
Yes,
so
I
can
take
less,
so
you
see
the
name
spot
running
right
there.
It's
ready.
It
looks
great
I'm
really
happy
with
this.
So
now,
what
do
I
want
to
do
now?
I
want
to
run
a
backup,
because
I
have
something
of
importance.
So
the
workflow
that
I'm
sort
of
demonstrating
here
is
I
have
a
development
environment.
B
And
let's
say
you
have
a
you:
have
a
group
of
developers
and
you're
building
an
app
and
those
developers
are
pushing
to
a
server
and
eventually
that
the
app
needs
to
be
moved
to
staging,
where
maybe
people
are
going
to
Thursday,
and
that
might
happen
once
today,
multiple
times
a
day
once
a
week.
But
the
point
is
that
you
just
take
things
that
is
poured
on
another
cluster
and
it's
just
gonna
work
and
I'm
going
to
show
you
how
to
do
a
bit
with
Alero
and
I'm
gonna
prove
that
it
really
does
work.
B
So,
let's
add
a
backup,
that's
how
that's
the
command
for
Valero
to
add
a
backup.
I
am
only
I'm
limiting
this
back
up
to
the
namespace
name
example,
which
was
the
namespace
that
was
created
with
that
part.
Now,
once
you
create
a
backup
with
bolero,
Valero
gives
a
very
helpful
message
saying:
well,
the
backup
request
was
sent.
We
don't
know
yet
at
this
point.
If
the
backup
worked,
it
could
have
failed
validation,
it
could
have
felt
upload
to
their
three
buckets,
but
so
we
have
a
few
commands
here.
B
One
is
describe
the
backup
itself
and
the
other
one
get
the
logs
only
for
the
backup
so
you're
not
going
to
see
the
entire
log
for
the
whole
cluster.
Just
for
the
what
happens
to
the
backup
so
I'm
going
to
show
both
of
these
commands
the
results
of
those
commands,
the
more
the
one
that
I
usually
go.
Look
at
first,
is
this
the
scribe
and
at
the
very
top
you
see,
phase
completed
and
awesome.
B
B
B
If
I
point
to
localhost,
it
won't
work
because
Mineo
is
inside
a
container.
My
cluster
is
inside
a
container,
so
I
point
to
my
local
host
in
many
ways
were
running
on
port
9000,
so
that
is
couldn't
work
and
oh,
this
is
cool,
so
I
have
an
option
to
put
push
an
image,
specify
the
image
that
I
want
to
push.
B
B
Misspelled
I
do:
how
does
it
know
that
remember
when
I
solve
Valero
I
configured
to
point
to
a
menial
instance?
That
is
the
same
me
know
that
my
first
cluster
was
pointing
to
so
basically
this
it's
pointing
to
the
same
as
3
buckets.
That's
how
he
knows
now
from
here
so
I,
just
just
to
make
sure
I
don't
have
engine
engine
X
example
namespace,
because
why
would
I
I
didn't
I,
didn't
install
the
app
right
and
I'm
not
going
to
start
the
app
either?
What
I'm
going
to
do
is
get
so
I.
B
Let
me
see
at
the
restore
so
I'm
going
to
run
the
script.
That's
going
to
create
a
restore
based
on
that
backup
that
I
have.
So
if
you
see
the
command
there
is
the
letter
restore
creates
it,
you
can
give
it
a
name
or
if
you
don't
give
it
a
name,
you'll
create
able
generate
a
name
and
I
say
from
Becca
acme
doubt
and
again
it's
saying:
oh,
the
request
was
sent
successfully.
So
that
was
the
request
to
do
the
restore.
B
We
don't
know
if
the
restore
works
now
so
I'm
going
to
do
a
describe
and
up
there
you
can
see
phase
completed.
That's
me:
a
okay
and
now
I'm
going
to
look
at
the
logs
I
know
it
worked
so
now
I'm
going
to
do
Garrett
names
face.
Oh
look
at
that.
My
app
is
there
looking
good
I'm
gonna
come
down
here
now
and
I'm
gonna
see.
If
I
can
Buddha
I
mean
it's
it's
running
its
life,
but
what
I
want
to
do
is
prove
it
to
you
and
I
need
to
here.
For
me.
B
And
I
have
a
different
scripts
for
this,
because
I
need
a
different
port
number,
because
I
didn't
turn
off
the
the
number
the
the
port
forwarding
for
my
other
colors
turn,
so
that
looks
again.
I'm
gonna
curl
in
this
one,
this
port
forward
that
I
just
created,
if
you
can
see
on
the
bottom,
is
a
b100.
B
B
A
B
B
C
A
A
You
can
participate
remotely
by
zoom
and
if
you
can't
make
it
because
the
time
zones
inconvenient,
the
these
are
recorded
and
uploaded
to
youtube
and
there's
the
github
link,
so
our
wrap
up
here
is
that
I
think
some
of
the
key
points
are
that
there
are
a
number
of
these
solutions,
but
things
like
backing
up
at
CD
with
the
built-in
backup
gives
you
like
all
or
none
it
isn't
selective.
So
you
can't
choose
individual
objects
or
namespaces.
Let's
just
suppose
you
did
one
of
those
Follet
cds
every
day.
A
So
you
know
some
of
these
solutions
work
on
a
more
granular
basis
and
that
often
is
important
for
some
of
your
disaster
use
cases.
It
also
can
back
up
more
than
n
CD
capturing
the
stateful
volumes.
If
you
go
back
in
this
slide
deck,
we
did
link
to
other
open
source
projects
that
do
parts
of
this,
so
I
don't
mean
to
overly
promote
it,
but
just
go
look
at
the
facts.
A
D
B
Okay,
so
let
me
address
the
first
question:
is:
if
you
you,
can
we
offer
native
support
for
AWS
Google
cloud
in
Azure,
and
there
are
plugins
for
other
boiler
providers
like
digital
ocean.
You
can.
We
have
this
all
documented
in
our
Valero
dot,
IO
website.
So
just
like,
how
do
you
do
that?
It
supported?
We
have
documentation
for
how
to
do
that.
Yeah.
A
B
Roller
supports
that
natively
right
and
the
way
the
way
it
works
is
that
we
use
the
provider
API
for
so
for
those
providers
that
we
support
in
the
plugins
that
we
also
have
they
use
the
providers
api
to
take
snapshots
and
that's
how
it
gets
done.
So,
as
your
second
question,
you
do
one
from
from
one
provider
to
another
provider.
You
can
using
rustic
and
we
have
support
for
rustic.
So
when
you
install
the
little
using
rustic
you
a
note
say
you
have
to
annotate
the
file
that
you
want
to
backup.
B
C
D
B
A
Carly
Xie
we
ran
out
of
time
and
as
a
courtesy
to
the
next
speaker,
actually
I
had
hope
to
catch
the
next
one
myself
we
can
maybe
take
questions
in
the
hall
or
since
this
next
session
seems
to
be
somewhat
related.
This
topic,
I'm
gonna,
hang
around
and
I'll
be
available
even
after
the
session
after
this.
Thank
you.