►
From YouTube: Disaster and Recovery (The Podlets, Ep 8)
Description
In this episode of The Podlets Podcast, we are talking about the very important topic of recovery from a disaster! A disaster can take many forms, from errors in software and hardware to natural disasters and acts of nature. That being said that are better and worse ways of preparing for and preventing the inevitable problems that arise with your data. The message here is that issues will arise but through careful precaution and the right kind of infrastructure, the damage to your business can be minimal. We discuss some of the different ways that people are backing things up to suit their individual needs, recovery time objectives and recovery point objectives, what high availability can offer your system and more! The team offers a bunch of great safety tips to keep things from falling through the cracks and we get into keeping things simple avoiding too much mutation of infrastructure and why testing your backups can make all the difference. We naturally look at this question with an added focus on Kubernetes and go through a few tools that are currently available. So for anyone wanting to ensure safe data and a safe business, this episode is for you!
For the show notes and transcript: https://thepodlets.io/episodes/008-disaster-and-recovery/
Feedback and episode suggestions:
https://twitter.com/thepodlets
https://github.com/vmware-tanzu/thepodlets/issues
info@thepodlets.io
Hosts
https://twitter.com/carlisia
https://twitter.com/bryanl
https://twitter.com/joshrosso
https://twitter.com/opowero
A
B
D
Yeah
sure
so,
I'm
based
in
the
UK
I
joined
VMware
as
part
of
the
hep
tol
acquisition,
which
stuck
which
adjoined
hep
till
we're
back
last
year
in
October.
So
the
acquisition
happened
pretty
quickly
for
me
before
that.
I
was
at
Red
Hat
and
working
on
some
of
their
cloud
management,
tooling,
and
a
bit
of
open
shift
as
well
and
before
that
I
worked
with
HP
and
visit
sue
and
I
kind
of
work
in
Enterprise
Management
a
lot.
So
you
know
things
like
desired.
D
State
and
automation
are
kind
of
things
that
have
followed
me
around
through
most
of
my
career.
So
you
know
coming
in
here
to
be
aware
and
working
in
the
cloud
major
applications
business.
You
know
it's
kind
of
a
good
fit
for
me.
I've
got
two
little
boys
I'm
a
mom
of
two
and
and
I'm
based
in
the
UK,
which
I
have
to
point
out,
has
got
currently
undergoing
a
heatwave.
So
we've
had
about
like
three
weeks
of
like
25
to
30
degrees,
which
is
warm
very
warm
for
us.
D
I
studied
sort
of
chemistry
in
university
and
then
I
went
on
to
do
a
PhD
in
cancer
research.
So
I
was
trying
to
figure
out
ways
where
we
could
predict
how
different
people
were
gonna
respond
to
radiation
treatments
in
there
with
a
view
to
tailoring
everybody's
treatment
to
make
it
unique
for
them.
Rather
than
sort
of
you
know,
giving
the
same
treatment
to
different
people
who
presented
with
the
same
disease
but
were
responding
very
very
differently
so
yeah
that
was
really
really
interesting
and.
D
A
D
Yes,
oh
I'm,
a
cloud
native
architect,
so
I
help
customers
predominately
focus
on
their
kubernetes
platforms
and
how
to
build
them
either
from
scratch
or
help
them
make
it
more
production-ready.
You
know,
depending
on
where
they
are
in
their
kubinashi's
journey.
So
it's
been.
It's
been
really
exciting,
part
of
being
part
of
happy
of
and
following
through
into
the
VMware
acquisition,
we're
getting
to
speak
to
customers
a
lot.
You
know
at
very
exciting
times
for
them.
D
A
Let
me
pick
up
on
that
thread
actually,
because
one
thing
that
I
love
about
this
group
for
me,
because
I
don't
get
to
do
that,
you
all
meet
customers
and
you
know
what
they
are
doing.
You
either
get
that
knowledge
for
his
hands.
What
would
you
say,
the
percentage
of
the
clients
that
you
see
are
doing
have
a
disaster
in
recovery
strategy
which,
by
the
way,
is
the
topic
of
today's
show
I'm.
B
D
I'll
started
things
I'm
kind
of
talking,
but
then
I'll
you
know
hand
it
over
to
my
colleagues
and
so
yeah
I
speak
to
customers
a
lot
and,
as
I
mentioned
earlier,
there
a
lot
of
them
are
alike
and
different
stages
of
their
journey
in
terms
of
automation,
in
terms
of
structures,
our
code,
in
terms
of
where
they
want
to
go
for
their
next
platform.
But
there
generally
is
in
the
room.
You
know
a
team
that
is
responsible
for
you
know,
backup
and
recovery
and.
D
So
the
leads
into
this
storage
team
really
because
you're
trying
to
backup
state
humbly
and-
and
so
when
we're
speaking
to
customers
will
have
the
automation.
People
in
room
will
have
the
developers
in
the
room
and
will
have
the
storage
people
in
and
they're
the
ones
that
are
primarily
out
of
those
three
sort
of
folks.
I
mentioned
they're,
the
ones
that
are
prime,
be
concerned
about
backup.
E
Definitely
I
think
it's
interesting
because
it's
almost
scary
and
how
many
of
our
customers
don't
actually
have
a
disaster
recovery
strategy
of
any
sort
and
I
think
it's
oftentimes,
just
based
on
the
maturity
of
the
platform.
You
know
a
lot
of
the
applications
and
such
they're
worried
about
downtime,
but
not
necessarily
like
it's
going
to
devastate
the
business
and
a
lot
of
these
apps.
You
know
and
I'm
not
trying.
E
Run
the
mission
for
the
collapse
on
things
like
kubernetes:
it's
just
a
lot
of
people
are
very
new
and
they're
just
kind
of
ramping
up
but
yeah.
It's
it's
a
really.
It's
a
really
complicated
thing
that
we
work
with
our
customers
on
and
there's
so
many
like
layers
to
this
I'm
sure
layers
that
we'll
get
into
there's
things
like
disaster
recovery
of
the
actual
platform
like
if
kubernetes
as
an
example,
goes
down
getting
it
back
up.
E
You
know
like
backing
up
its
datastore,
that
we
call
at
CD,
there's
obviously
like
the
applications
disaster
recovery
like
if
a
cluster
of
some
sort
goes
down
via
kubernetes
or
otherwise,
shifting
some
CI
system
and
redeploying
that
into
some
b
cluster,
to
bring
it
back
up
right
and
then
to
Olives
point.
When
it
she
said
it
all
comes
back
to
storage,
yeah
I
mean
that's
where
it
gets
extremely
complicated,
so
I'm
when
well,
at
least
in
my
mind,
it's
complicated
for
me.
E
I
should
say
you
know
when
you're
thinking
about
okay,
I'm
running
this
Postgres
as
a
service
thing
on
this
cluster.
You
know
it's
not
that
simple
to
just
move
the
app
from
cluster
and
cluster
B
anymore.
I
have
to
consider
like
what
about
the
actual
persistence
or
sorry
just
to
say,
storage
data.
What
do
I
do
with
the
data?
How
do
I
make
sure
I
don't
lose
that
and
then
that's
a
pretty
complicated
question
answer.
I.
D
Think
a
lot
of
the
storage,
you
know,
providers
and
Stampede.
You
know
cookies,
vendors
playing
in
that
storage
space.
How
kind
of
looking
at
you
know
novel
ways
to
solve
that
and
have
kind
of
you
know
adapted
their
current?
You
know
they're
thinking,
maybe
that
was
maybe
slightly
older
thinking
to
new
ways
of
interacting
with
kubernetes
cluster,
to
provide.
D
You
know
that
ongoing
replication
of
data
around
different
systems-
and
you
know
outside
of
the
kubernetes
cluster
and
then
allowing
it
to
like
you,
know,
be
ported
back
in
women
when
the
kubernetes
cluster,
if
we're
told
my
kubernetes
in
this
instance
as
a
platform
porting
that
data
backing
and
that
there's
a
lot
of
vendors
playing
in
that
space.
So
it's
kind
of
very
exciting
space,
really
to
see
how
you
know.
D
Different
people
are
figuring
out
how
to
backup
distributed
systems
in
a
reliable
manner,
because
different
people
want
different
levels
of
back,
and
you
know
because
of
the
the
micro-services
nature
of
the
cloud
native
architectures
that
we
put
on
we
deal
with.
You
know
different.
You
know
your
applications,
not
just
one
thing
anymore.
It's
it's
not
different
things
and
certain
parts
of
that
application
need
to
be
recovered
fairly
quickly
and
other
parts
may
be
not.
You
know,
don't
need
to
break
over
that
quickly.
D
So
it's
all
about
functionality,
ultimately,
that
your
end
customers
or
your
end
users,
see.
So
if
you,
if
you
think
about
it
visually,
is
like
a
like:
a
banking
application,
for
example,
where,
if
you're
looking
at
things
like
the
so
the
customers,
interacting
with
that
and
they
can
check
their
financial
details
and
they
can
check
the
current
status
of
their
account
and
there
are
two
different
services,
but
the
actual
service
to
transfer
money
into
their
account
is
down.
D
You
know
it's
still
a
pretty
functional
system
to
the
end-user,
but
in
the
background,
all
the
three
systems
are
in
place
to
recover
that
transfer
of
money
functionality,
but
it's
not
detrimental
to
your
business
if
that's
down
so
there'll
be
different
essays
and
different
objectives
in
terms
of
recovery,
in
terms
of
the
amount
of
time
that
you
can
take
for
you
to
restore.
So
all
of
that
has
to
be
factored
in
into
disaster
recovery
plans
and
and
it's
up
to
the
company
and
any
we
can
help
as
much
as
possible
for
them
to
figure
out.
D
You
know
which
bits
of
the
application
which
bits
of
your
business
needs
to
conform
to
certain
SLA
is
in
terms
of
recovery,
because
different
bills
will
have
different
different.
You
know
standards
and
different
times
in
and
around
that
space.
That's
what
I've
seen
so
it's
a
complicated
thing.
You
definitely.
C
Alright,
so
I
want
us
to
take
a
step
back
and
impact
this
term
disaster
recovery,
because
I
can
assure
you,
careers
and
fortunes
have
been
made
on
helping
people
get
this
rate
and
really
what
do
we
I
guess
before
we
get
super
deep
in
this?
What's
a
disaster
and
and
then
what's
a
recovery
for
that,
have
you
thought
about
that
at
a
fundamental
level.
D
So
like
just
for
me,
you
know
they
if
we
would
kind
of
take
it
kind
of
like
at
face
value,
so
like
a
physical
disaster,
they
could
be
physical
wounds
or
soft
base,
one
software-based
ones
the
physical
ones
can
be
like
you
know,
earthquakes
or
floodings
fires.
Things
like
that.
That's
happening
like
either
in
your
like
fair
in
your
region
or
can
be
fairly.
D
You
know
widespread
across
the
area
that
you're
in
or
software
sorry
disasters
like
you
know,
cyber
attacks
that
are
perhaps
specific
to
your
own
internal
systems,
like
your
system,
has
been
compromised.
So
that's
fairly
local
to
you.
So
there
is
two
different
design
strategies.
There
become
much
visible
physical
disaster.
You
have
to
have
a
recovery
plan
that
you
know
and
leave.
You
know
that
is
outside
of
that
physical
boundary
that
you
can
recover
your
system
from
so
where
that's
not
affected
by
that
physical
disaster
and
again
they're.
D
So
for
the
recovery
from
in
terms
of
software.
In
terms
of
you
know,
your
system
has
been
compromised
and
then
the
recovery
from
that
is
is
is
different.
You
know
I'm,
not
an
expert
on
like
cyber.
You
know
attacks
and
vulnerabilities,
but
you
know
the
system,
the
recovery
from
there
for
companies
try
and
recover
from
that
they
plan
for
it
as
much
as
possible.
So
they
have
a
you
know
they
down
their
systems
and
try
and
get
patches
and
fixes
to
them
as
quickly
as
possible
and
spin.
C
C
Let's
say
we're
just
going
to
say
in
a
cloud
not
on
on-premise
we're
running
or
running
workloads
and
let's
say
AWS
and
in
the
United
States,
and
we
can
take
care
of
local
diversity
by
running
in
east
and
west
regions,
and
then
and
and
also
we
can
take
care
of
local
diversity
by
running
in
availability
zones
within
a
region,
because
AWS
has
guaranteed
that
a
Z,
1
and
AZ
3
have
different.
Network
connections
are
not
in
the
same
building
and
and
things
like
that,
would
you
agree?
C
Asher
or
cool
cloud
or
digital
ocean
or
SoftLayer
or
or
or
packet
or
I,
could
probably,
if
I
thought
about
this
privately
twenty
more.
What
do
you
say?
Ibm,
IBM,
that's,
why
said
SoftLayer
they
all
they
all
practice
in
the
physical
diversity.
They
all
have
different
regions
that
you
can
deploy
software
and
whether
it
be
data
locality,
but
also
for
data
protection.
C
C
C
Actually,
oh
I'm
actually
saying
way
more
it
Matt.
So
before
you
even
think
about
things
when
it
comes
to
sass
to
recovery,
you
got
to
define
what
a
disaster
is.
Some
applications
can
actually
run
out
of
multiple
physical
locations,
and
you
know-
and
let's
go
back
to
my
AWS
example,
because
it's
everywhere
and
people
understand
everyone
understands
how
AWS
works
set
up
at
a
high
level.
Sometimes
people
are
running
things
out
of
us
ds1
and
us
too,
and
they
can
run
both
of
the
applications
and
the
reason
they
can
do.
C
That
is
because
the
data,
the
the
individual
transactions
of
whatever
they're
doing,
don't
need
to
talk
to
one
another.
They
can
actually
just
run
besides
out
of
places.
But
to
your
point,
when
you
talk
about
now,
you
have
the
issue
where
maybe
you're
doing
inventory
management,
because
you
have
a
large
store
and
you're
running
it
out
of
multiple
countries,
so
you're,
actually
you're
you're
in
the
EU
and
you're
somewhere
on
a
pack
as
well.
What
do
you
do
about
that?
C
Well,
there's
a
there's,
a
couple
of
ways
that
I
could
think
about
how
we
would
do
that.
We
could
actually
just
have
all
the
database
connections
go
back
to
one
single
main
service
and
then
what
we
could
do
with
that
main
service
is
that
we
could
have
it
replicated
in
the
rear
in
local
place,
and
then
we
can
have
a
replicated
in
a
remote
place
too.
C
So
if
the
local
place
blows
up
at
least
you
can
point
all
the
other
sites
back
to
this
one
and
that's
the
simplest
way
and
the
reason
I
wanted
to
bring
this
up
is
because
I
don't
like
acronyms
all
that
much.
But
disaster
recovery
has
two
of
my
favorite
ones
and
they
are
they're
called
RPO
and
RTO
and
and
really
what
it
comes
down
to
is
you
need
to
think
about
when
you
have
a
disaster,
no
matter
what
the
disaster
is
or
how
you
define
it,
you
have
RTO.
C
Basically
it's
the
time
that
you
can
be
down
before
there's
a
huge
issue,
and
then
you
have
something
called
RPO
which
is
without
going
in
all
the
names
is
how
much
data
how
far
you
can
go
since
your
last
backup
before
you
have
business
problems.
So,
just
thinking
about
those
things
is
how
we
should
think
about
our
back
disaster
recovery
and
it's
all
based
on
how
your
business
works
or
how
your
project
works
and
how
long
you
can
be
down
and
how
much
data
you
have.
So
that
was
my
spiel
yeah.
C
C
B
C
A
But
it
is,
it
is
true,
though,
and
as
Josh
was
saying
some
some
teams
of
companies
of
products
don't
have
like,
especially
companies
that
are
in
story.
They
Geordi
the
cognitive
journey.
They
don't
have
a
backup,
because
there
are
many
complicated
things
to
deal
with
and
backup
is
super
complicated
me
disaster
recovery
strategy.
Doing
that
is
not
trivial
but
shouldn't
it.
Like
start
with
that,
or
at
least
because
it
is
so
complex.
A
E
Yeah
and
I
should
I
should
I
think
I
over
simplified
too
much
right.
Disaster
recovery
could
theoretically
be
anything
I,
suppose
they're
going
back
to
it,
you're
saying
Brian
the
recovery
aspect
of
it
like
recovery
for
some
of
the
customers.
I
work
with
is
literally
to
stand
up
a
brand
new
cluster,
whatever
that
cluster
is
a
cluster,
that
is
their
platform
and
then.
E
E
C
C
What's
the
scope
at
any
given
time
when
you're
starting
an
application
and,
let's
say,
I'm,
actually
making
a
huge
assumption
that
you're
using
someone
else's
cloud
you're
using
public
cloud
whenever
you're
in
your
data
center
there's
a
there's
a
different
problem,
but
whenever
you're
using
public
cloud
think
about
what
you
already
have
all
the
major
public
clouds
have
a
durable
object,
storage
and
of
many
nines
of
durability
and
then
and
then
fewer
nines,
but
still
a
lot
of
nines
of
availability
and
the
the
canonical
example.
There
is
s3.
C
So
when
you're
designing
your
applications-
and
you
know
that
you're
going
to
you
don't
have
disaster
issues
realize
that
s3
is
almost
always
going
to
be
there.
Unless
you
know
it
was
2017,
it
goes
down
or
the
other
two
failures
that
it
had,
but
pretty
much
it
will
be
there.
So
think
about,
if
have
data,
think
about
how
do
I
get
that
data
into
s3
and
I'm,
not
and
I'm,
just
saying
you
can
use
it
for
storage,
it's
fairly
cheap
for
how
much
storage
you
can
get.
C
You
can
make
it
sure
it's
encrypted
and
using
I.
You
can
definitely
make
sure
that
people
who
have
the
right
privileges
can
see
it
and
the
same
goes
with
Azure
and
the
same
goes
with
Google.
So
that's
the
first
face.
The
second
phase
is
that
now
you're
gonna
say
well:
I
don't
want
to
use
I,
don't
want
to
use
I
want
to
use
a
relational
database.
Well,
once
again
use
your
cloud
provider.
C
All
the
major
cloud
providers
have
great
relational
databases
and
actually
key
value
stores
as
well,
and
the
neat
thing
about
them
is:
you
can
actually
set
them
up,
sometimes
to
run
in
a
whole
region.
You
can
set
them
up
to
do
automated
backups,
so
at
least
the
minimum
that
you
have.
You
actually
use
your
cloud
provider
for
what
it's
valuable
for
now,
if
you're,
not
using
a
cloud
provider
and
you're
doing
it
on
on-premise,
I'm
gonna
tell
you.
C
The
simple
answer
here
is
I
hope
you
have
a
little
bit
of
money
because
you're
going
to
have
to
pay
somebody
like
either
one
of
our
kubernetes
architects
or
you're,
going
to
play
somebody
else,
but
there's
no
easy
button
for
this
kind
of
solution
and
and
just
for
this
little
mini
rant.
I'm
gonna
leave
everyone
with
the
biggest
piece
of
advice,
the
best
piece
of
advice
that
I
can
ever
leave
you.
C
If
you're
running
relational
databases,
if
you
are
running
a
relational
database,
whether
it
be
Postgres,
my
sequel,
you
know
Aurora,
whatever
have
have
it
replicated,
but
here's
the
kicker
have
another
replica
replicas
that
delay
and
make
it
delay.
10
minutes,
15
minutes
not
much
longer
than
that,
because
what's
gonna
happen,
especially
in
a
young
company,
is
you're
going
to
have,
especially
if
you're
using
rails
or
something
like
that.
You're
gonna
have
somebody
who
is
going
to
have
access
to
production
because
you're
a
small
company
and
really
federated
this
out
yeah.
C
And
it's
gonna
happen
and
you're
gonna
panic.
If
you
have
it
in
a
replica
that
database
is
gone
in
a
replica,
you
have
a
10-minute
delay
replica.
You
got
ten
minutes
to
figure
it
out
before
world
ends,
and
hopefully
you
know
some
like
leads
to
the
master
database.
You're
gonna
know
pretty
quickly
and
you
can
just
cut
that
replica
out,
pull
that
other
one
over
I'm,
not
going
to
say
where
I
learned
this
trick,
we
had
to
employ
multiple
times
and
it
saved
our
butts
multiple
times.
So
that's
my
that's.
My
favorite
thing
this
year.
C
D
So
we
mentioned
Haiti
Wood,
Wood
Wood.
The
panel
think
that
I'm
like
there's
now
a
slightly
inverse
relationship
between
the
amount
of
AJ
that
you
architect,
four
versus
the
disaster
recovery
plan
that
you
have
implement
on
the
back
of
that.
So
more
you
are
taking
around
hey
J,
like
the
less
you
architect
or
plan
for
D
are
not
eliminating
either
of
them.
Yeah.
C
E
C
Also
realize
that
everybody's
not
preparing
a
public
cloud,
and
in
that
case
well,
you
can
still
back
your
stuff
up
to
public
cloud,
even
if
you're
not
ringing
public
cloud,
but
there's
still
people
out
there
who
are
running
big
tape,
raisin
and
I've,
seen
them
I
scene
tape.
Arrays
that
are
you
know
wider,
then
I
mean
I'm,
seeing
an
80
inch
wide
table
bigger
than
this
table
with
robotic
arms.
It
takes
their
wrists
big
and
you
had
to
make
sure
that
you
got
the
pics
right
or
that
particular
day
doing
your
rotation.
C
So
I
guess
what
I'm
saying
is
that
there's
a
balance
H
a
KN
high
availability,
if
you're
doing
it
you're
doing
it
in
a
truly
highly
available
way.
You
can't
miss
whole
class
disasters,
but
I'm
not
saying
that
you
will
not
have
disaster,
because
if
that
was
the
case,
we
wouldn't
be
having
a
discussion
right
now.
So
I'd
like
to
move
the
conversation
just
a
little
bit
to
more
cloud
native
you're
running
on
kubernetes.
What
should
you
think
about
for
disaster
recovery?
What
are
the
types
of
disasters
we
have?
How
could
we
be
yeah.
B
E
Where
we
write
these
manifests
that
describe
the
desired
state
of
our
application
and
how
it
should
run.
You
know
recommending
that
we
make
sure
to
keep
that
declarative
state
in
the
source
control
right.
Just
like
we
would
our
code
so
that
if
something
were
to
go
wrong,
it
is
somewhat
more
trivial
to
redeploy
the
application
should
we
need
to
recover
now
that
does
assume
we're
not
worried
about
like
data
and
things
like
that,
but
it.
D
D
So
the
idea
that
you
would
get
a
language
that
had
the
functionality
that
suited
the
needs
of
all
those
different
teams
and
the
storage
team
could
write
could
automate
their
piece
which
they
were
doing.
They
just
wasn't
interacted
with
any
of
the
other
teams,
and
the
network
people
would
alternate
theirs
and
the
application
installed
people
would
do
their
bit
and
then
the
sort
of
the
server
OS
people
would
do
their
bit
having
a
process
that
could
like
tie
those
teams
together
in
terms
of
a
language.
So
you
know
and
danceable
puppet
chef.
D
Those
kind
of
things
tend
to
try
to
unite
those
teams,
in
that
you
can
all
do
your
automation,
but
we
have
a
tool
that
can
kind
of
take
that
code
and
run
it
as
one
system
into
end,
and
at
the
end
of
that
you
get
an
up
and
running
system,
and
if
you
run
it
again,
you
get
other
systems,
that's
actually
same
as
the
previous
one.
If
you
run
it
again,
you
get
another
one,
so
you
know
reducing
the
time
to
build.
These
things
you
know
plays
very
plays
very
importantly,
into
this
space.
D
You
know
disaster
is
only
disaster.
If
it,
you
know
in
terms
of
time,
because
things
break
all
the
time
right.
So
it's
how
you,
how
you
have
that
affects
you
and
how
quickly
you
can
recover.
If
you
can
recover
in
like
seconds
in
minutes
and
it
hasn't
affected
your
business
at
all,
then
was
it
even
really
a
disaster?
You
know.
So
this
is
the
time
it
takes
you
to
recover
to
build
your
things
back
is
key
and
all
that
automation
and
then
leading
onto
kubernetes,
which
is
the
next
step.
D
A
So
that
makes
me
think
I
don't
completely
understand
because
I'm
not
out
there
architecting
people's
systems
I'm
very
focused
on
doing
you
know.
The
one
thing
that
I
do
is
building
this
backup
tool,
which
is
happens
to
be
for
Cobra.
Ladies,
so
it
makes
so
I,
don't
completely
get
the
limitations
and
use
cases,
but
less
so
my
question
is:
is
it
enough
to
have
your
the
the
declarations
of
how
your
infrastructure
should
be
in
source
control,
because
what
are
you
running?
A
Applications
on
the
platform
and
your
applications
are
interacting
with
a
platform
and
changing
the
state
of
the
platform.
Is
that
not
something
that
happens
and
how
I
mean,
of
course,
is
ideally
having
having
that
those
declarations
in
source
control,
of
course,
is
a
great
backup,
but
don't
you
also
want
a
backup,
the
state,
this
the
stages,
the
changes
to
state
as
they
keep
happening.
C
You've
just
described
at
the
database
log
yeah,
of
course,
and
and
that's
and
that
that
has
been
used
for
a
long
time.
That's
how
replication
works
literally,
you,
you
take
the
change
and
you
push
it
over
the
wire
and
it
gets
applied
to
the
remote
system.
So
the
problem
is:
is
that
there
isn't
just
one
way
to
do
this,
because
if
you
do
only
transaction
based,
if
you
only
do
the
changes,
you
need
a
good.
You
need
a
good
base
to
start
with,
because
youth
apply
those
changes
to
something.
So
how
do
you
get
that?
E
Yeah
I
think
you've
hit
a
fatal
flaw
to
Carly
said
like
what
what
that
simplified,
just
like
habit
and
source
control,
model
kind
of
falls
over
right
and
and
I
think
you
know
having
that
declarative
kind
of
stamped
out.
This
is
the
ideal
nature
of
the
world,
for
this
deployment
and
source
control
has
benefits
beyond
just
that
of
disaster,
recovering
scenarios
right
and
for
stateless
applications,
especially
like
we
talked
about
in
the
previous
podcast.
E
It
can
actually
be
all
you
need,
potentially,
which
is
which
is
so
great,
like
move
your
CI
system
over
to
cluster
B
boom,
your
backup,
and
rather
that's
that's,
really
neat,
but
a
lot
of
our
a
lot
of
our
customers.
We
work
with
once
we
get
them
to
a
point
where
they're
at
that
stage.
They
then
go
well
what
about
like
all
these
persistent
volumes
which,
by
the
way,
is
just
like
a
volume
on
a
computer?
That's
a
kubernetes
term,
but
like
how?
E
What
about
all
these,
like
parts
on
my
disks
that
I
don't
want
to
lose
it
I
lose
my
cluster
right
and
that
totally
feeds
into
by
tools
like
the
one
you
work
on.
Are
so
helpful
and
maybe
I
don't
know
now
be
a
good
time,
but
maybe
probably
see
you
can
expand
on
that
tool
like
what
it
tries
to
solve
for
ya.
A
So
but
I
want
to
back
up
a
little
dose.
So,
let's
put
aside
stateful
workloads
in
volumes
in
databases,
I
was
talking
about
the
infrastructure
itself.
The
state
of
the
infrastructure
mean
if
you're
running,
is
it
not
common
and
I,
don't
know
the
answer
to
this
I
might
be
completely
off.
Is
it
not
common
for
you
to
develop
a
cloud
native
application
that
is
changing
the
infrastructure
they
state
of
the
infrastructure,
or
is
this
something
that's
not
good
to
do.
C
C
You
have
things
like
Kashi
Corp
and
all
the
things
that
they're
doing
they
have
their
hosted
thing.
They
have
their
on-premise
thing,
they
have
their
local
thing,
so
people
are
looking
at
that
problem.
The
good
thing
is
that
that
problem
hasn't
been
solved,
I
guess
good
and
bad
at
the
same
time
because
it
hasn't
been
solved
so
so
we
can
solve
it
better.
But
the
bad
thing
is
that,
if
you're
looking
for
good
infrastructure
infrastructure
as
code
software,
that
has
not
been
solved
yet
I.
D
Think
if
we're
talking
about
containerized
applications,
I
think
if
there
was,
if
there
was
systems
that
interacted
or
factors
or
change
the
infrastructure,
they
would
be
separate
from
the
applications.
As
you
were
saying,
Brian
just
expanded
a
little
bit.
It
would
be
a
container
eyes
or
sandbox
processes
that
were
running.
You
know
separate
to
the
main
application,
so
you
know
just
expanding
it,
you're
separating
out
what's
actually
running
and
doing
function
and
in
terms
of
application
versus
maybe
systems
that
have
to
edit
that
infrastructure.
First
before
that
main
application
runs.
C
A
D
Like
yeah
I
mean
you
know,
if
you
have
like
a
process
that
runs
to
stop
something,
you
would
also
have
a
process
that
started
something
like
you
shouldn't
ever,
if
you're
trying
to
dine
your
applications
and
if
it
needs
to
interact
with
other
things
and
that
application
design
should
include
you
know
the
consideration
of
you
know
what
do
I
need
to
do
to
interact
with
the
with
the
infrastructure
and,
if
I'm
doing
something
you
know.
So
the
left
wise
I
have
to
do
the
opposite
and
equal
reaction
right
wise
to
have
an
effectively
clean
application.
E
I
think
it
I
think
it
maybe
even
fold
into
a
whole
other
topic
that
maybe
we
can
even
cover
on
another
podcast,
which
is
like
the
notion
of
the
concern
of
mutating
infrastructure
right
and
if
you
have
a
ton
of
hands
and
those
cookie
jars
and
they're,
like
changing
things
all
over
the
place
like
you're,
losing
that
potential
single
source
of
declarative
truth
even
right,
it
just
could
become
very
complicated
and
I.
Think
maybe
to
the
crux
of
your
original
point:
cuz,
you
know
they're
not
super
off.
E
D
Okay,
I
mean
I,
mean
you
know,
application
design,
you
know,
plays
a
very
important
role
now,
especially
in
terms
of
the
plan
a
deal
architectures
or
we
talking
a
lot
about
micro
services.
You
know
where
a
lot
of
companies
are
looking
to
me
architect
their
applications,
and
so
maybe
mistakes
that
were
made
in
the
past
or
maybe
not
mister
mistakes,
and
perhaps
a
strong
word,
but
maybe
things
that
were
allowed
in
the
past
perhaps
are
not
best
practices
going
forward.
A
And
so
Josh
asked
me
to
talk
a
little
bit
about
other
touch,
touch
up
on
it
quickly.
Well,
first
of
all,
I
would
love
to
have
a
whole
show
just
about
infrastructure
like
old
guitars,
maybe
that
there
will
be
two
episodes
even
but
so
Valera
doesn't
do
any
backup
of
if
the
infrastructure
itself,
but
so
it
works
at
a
the
kubernetes
level.
A
We
recommend
you
always
back
up
everything
in,
but
you
can
just
selectively
restores.
So
that
would
be,
if
you
don't
need
to
restore
a
whole
class
or
why
would
you
do
it?
You
can
just
do
part
parts
of
it
and
it's
super
simple
to
use
is
almost
why
we
do
not
have
a
backup,
because
this
is
ridiculously
simple.
You
do
it
through
a
command
line
and
we
have
a
scheduler.
A
So
one
thing
to
highlight
about
there
also
that
I
wanted
to
highlight
welcome
error
is
that
we
do
everything
to
look
through
the
kubernetes
api
and
that's
that's.
For
example,
one
reason
why
we
can
do
selective
backup
of
restores
and
the
as
you
can
backup
the
CD
completely
yourself,
but
you
have
to
back
up
the
whole
thing
and
if
you're
on
the
manager,
you
wouldn't
be
able
to
do
that
because
you
just
wouldn't
have
access
so
other
tools
like
use
the
backup
to
the
city
offers
or
your
service
provider.
A
D
I
just
want
to
say
Valero,
the
last
customer
I
was
I,
I
worked
on.
They
wanted
to
use
Valero
in
its
capacity
to
be
able
to
backup
a
whole
cluster
and
then
restore
that
whole
cluster
on
a
different
cloud
provider,
as
you
mentioned,
so
they
weren't
sort
of
using
it
as
well.
They
were
using
this
backup,
but
their
primary
function
was
that
they
wanted
to
properly.
You
know
the
cluster
as
it
was
on
a
brand
new
cloud.
C
A
I
mean
so
sorry,
so
one
thing
that
like
I
said
Valero
does
is
back
up
the
cluster
like
the
Hulk,
all
the
copper
Nerys
objects,
because
why
would
we
want
to
do
that?
That's
like
because
if
here
declare
it
like
someone
explained
to
everybody,
who's,
listening,
listening
and
including
myself,
because
some
people
that
some
people
bring
this
happens
and
they
say
well
I,
don't
need
to
backup
the
coronaries
objects
if
I
have,
if
all
of
that
is
declared
and
I
have
that
declaration
in
source
control.
So
if
something
happens,
I
can
just
do
it
again.
C
Untrue,
because
just
for
any
given
kubernetes
object
there
is
there
is
the
configuration
that
you
created.
So
let's
say:
if
your
great
an
appointment,
you
need
a
you
need
spec
replicas,
you
need
the
template,
spec,
you
need
labels
and
selectors,
but
if
you
actually
go
and
pull
down
that
object
afterwards,
what
you'll
see
is
there's
other
things
inside
of
that
object.
C
So
if
you
didn't
specify
any
replicas,
you
get
the
defaults
or
other
things
you
get
P
bolts
for
so
you
don't
want
to
have
a
lossy
backup
and
restore
because
then
you
get
into
yourself
into
a
place
where,
if
I
go
back
this
thing
up
and
then
I
restore
it
to
a
different
cluster,
you
actually
sit
out
see
if
it
works.
It'll
be
different.
So
just
keep
that
in
mind.
Whenever
you're,
when
you're
doing
that.
E
Yeah
I
double
down
on
that
I
think
I
think
it
just
comes
down
to
knowing
exactly
what
Brian
just
said,
because
there
certainly
are
times
where,
when
I'm
working
with
a
customer
they're,
just
such
a
simple
use
case
at
the
notion
of
redeploying
the
application
and
potentially
losing
some
of
those
factors
that
may
have
mutated
over
time
like
they
just
shrug
to
it
and
go
whatever
right.
But
it
is
so
awesome
that
tools
like
Valero
and
and
other
tools
are
kind
of
bridging
that
gap
and
I.
E
Think
all
of
you
need
like
not
only
just
backing
that
stuff
up
and
capturing
its
state
as
it
was
in
the
cluster,
but
providing
us
with
like
a
good
way
to
like
section
out
one
namespace
or
one
group
of
applications
right
and
just
move
those
potentially
over
and
so
on.
So
yeah,
just
it
just
kind
of
comes
to
knowing
like
what
exactly
are
you
gonna
have
to
solve
for
and
then
how
complex
your
solution
should
be.
C
C
A
Is
so
true,
so
true,
I
mean
I'm
in
this
backup.
World
I
have
I,
have
only
been
in
this
backup
world
for
a
minute,
but
I
mean
I've
needed
to
backup
things
before
I.
Don't
think,
I've
learned
this
concept
after
coming
here,
I
think
I've
known
this
concept,
they
just
became
stronger
in
my
mind,
so
I
always
tell
people.
If
you
haven't
done
every
story,
you
don't
have.
If
you
you
don't
have
a
backup.
B
E
Its
luster
to
just
like
losing
an
important
component
right
and,
unlike
one
person
on
the
team,
will
say
once
a
week
or
once
a
month
depending
on
their
tolerance,
just
chooses
something
from
that
list
and
does
it
not
in
production,
but
does
it
right
and
it
gets
it
gets,
gives
you
the
opportunity
to
test
everything
em
to
end.
Like
did
your
alerting
fire
off
when
you
did
restore
to
your
points,
right
was
the
backup
valid?
C
Have
one
small
follow-up
on
that
also,
what
are
you
doing
when
you're
doing
backups?
What
and
no
matter
how
you're
doing
you
know
think
about
your
strategy
and
then
how
long
you
keep
data
I
mean
whether
it's
due
to
regulation
or
just
physical
space,
and
it
cost
money.
So
you
so
you
just
don't
back
up
yesterday
and
then
you
back
up
again.
C
A
D
A
And
there
were
definitely
vary
from
product
to
product
company
to
company.
So
this
is
what
my
it
might
be.
The
last
idea
or
question
of
the
show,
as
we
are
getting
into
this,
you
know
idea
of
ephemeral
clusters
and
the
automation,
and
we
can
really
good
at
automating
things
and
bringing
things
back
up.
Are
we
okay?
Is
it
possible
that
we
get
to
a
point
where
we
don't
even
talk
about
disasters
anymore?
Just
like?
Oh,
you
just
have
to
row.
A
You
know
bring
up
this
cluster
or
this
system
and
doesn't
even
matter.
Why
he's
okay,
something
caused
that
so
we're
not
going
to
talk
about
these
assets,
because
what
I'm
thinking
that
is
in
the
past
and
I'm
like
a
long
long
time
ago,
or
maybe
not
so
long
time
ago,
we
when
I
was
working
with
applications
and
there
was
a
disaster
as
a
disaster,
because
it
felt
like
a
disaster
like
somebody
had
going
manually
and
find
out
what
happened
and
what
to
fix
and
fix
it
manually
with
complete
chaos
and
strikes.
A
Now,
if
they
just
like
keep
rolling
and
Zelda
Maydays
have
thing
comes
down,
you
bring
it
back
up.
You
know
what
I
mean
is
like.
Maybe
you
won't
matter
why,
and
are
we
going
to
talk
about
this
in
terms
of
it
was
a
disaster
like
it
doesn't
even
matter
what
what
caused
it?
Maybe
it
was
a
point
like
a
disaster.
A
C
Think
we're
gonna
get
to
place
and
I,
don't
know
whether
we're
five
years
away
or
ten
years
away
or
20
years
away
or
a
place
where
you
won't
have
the
same
classic
disasters
that
we
have
now
and
think
about.
Where
we've
come
over
the
past
20
years
over
the
past
20
years,
we
basically
look
at
Hardware
in
Iraq,
is
replaceable
I
can
think
about
1988,
1999
and
2000.
We
rack
a
whole
bunch
of
servers
and
that
sir,
will
be
special
and
now
at
these
scales
we
don't
care
about
that
anymore.
C
We
just
when
a
server
goes
away.
We
have
50
more
look
just
like
it
and
what
we're
going
to
find
and
what
we
and
the
reason
we
were
able
to
do
that
across
large
platforms,
because
of
Linux
and
now
with
kubernetes
and
Cooper
Nettie's
keeps
of
going
in
the
same
trajectory
we're
going
to
basically
codify
these
patterns.
That
base
that
makes
hardware
lost,
not
a
thing.
We
don't
really
care.
C
Let's
model
docker
spanner
and
we're
going
to
get
to
the
place
where
this
kind
of
technology
is
what
is
available
for
everyone
and
we're
going
to
see
that
we're
not
going
to
have
these
kinds
of
disasters
that
we're
having
now
more
I,
think
what
we'll
have
now
is
bigger,
distributed
systems
things
where
we
have
timing
issues
and
things
like
that
and
later
election
issues,
but
I
think
the
old
school.
The
old
school
stuff
can't
be
fazed
out
over
the
least
of
the
next
computing
generation.
C
D
I
think
it's
just
it's
maybe
more
around.
You
know
architectures
these
there's
and
applications
and
designers
and
infrastructure,
architects
they're.
You
know
in
the
container
space
and
with
kubernetes
orchestrating
and
maintaining
your
desired
state,
it's
kind
of
more.
If
you
know
you're
thinking
of
that
things
will
fail
and
that's
okay,
because
it'll
be
put
back
to
the
way
it
was
before.
D
And
so
you
know
this
is
the
concept
of
something
stopping
you
know
in
mid
run
is
not
so
scary
anymore
because
it
would
be
get
put
back
to
its
state,
and
you
know,
maybe
you
might
need
to
investigate
if
you
keep
stopping
and
starting
in
kubernetes
keeps
bringing
it
back.
So
the
system
is
actually
still
fully
functional
and
talk
in
terms
of
end
users,
but
as
you
as
the
operator
might
need
to
investigate
wide
and
so,
but
they
actually
end.
The
endpoint
is
still
that
your
application
is
still
up
and
running.