►
From YouTube: How to Backup and Restore Your Kubernetes Cluster - Annette Clewett & Dylan Murray, Red Hat
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How to Backup and Restore Your Kubernetes Cluster - Annette Clewett & Dylan Murray, Red Hat
Operating Kubernetes clusters introduces many new practices, but does not change the need to be able to backup and recover your applications and data. Yet traditional methods of server backup work poorly with Kubernetes clusters. How can you make sure your cluster is protected? How can persistent data get saved in a manner so there is minimal or no corruption to the application if recovery is required? In this session we will explore how to use open-source disaster recovery tools you can use today such as Velero and Restic. We’ll also discuss how to use the Noobaa S3 API to reliably save and store backups for all resources including snapshots housed in Rook-Ceph. To prove this is not just smoke and mirrors, we will demonstrate in a live Kubernetes cluster deleting everything in a namespace and then continue on to show complete recovery of all resources and data.
https://sched.co/UaZN
A
Hello,
yes,
thank
you
for
coming
this
afternoon.
We
want
to
talk
about
something
that
I
think
at
least
everyone
I
talked
to
is
kubernetes
is
great
and
lots
of
good
solutions
going
on.
But
how
can
you
have
a
serious
platform
if
you
have
no
backup
and
recovery,
so
I
my
background
I'm
from
Red
Hat,
and
that
clue
and
I
have
been
working
with
kubernetes
and
and
storage
solutions
for
the
last
couple
years,
so
let
Dylan
introduce
himself
hi.
B
A
So
what
happened
was
I
thought
this
was
sort
of
a
serious
thing
that
we
should
solve.
So
I
had
been
working
a
lot
with
rook
stuff
and
with
new
bow,
which
you'll
hear
more
about
which
is
a
multi
cloud
gateway
for
for
s3
service
and
so
I
contacted
Dylan's,
Matt,
I
didn't
even
know
Dylan
at
the
time
I
contacted
Dylan's
manager
because
I
knew
about
the
effort
had
been
working
with
them.
A
We
know
these
technologies
should
work
together,
but
we
didn't
really
know
we
get
accepted.
So
after
that
we
decided
to
do
a
proof
of
concept
and
also
I.
Think
by
the
end
of
the
session.
You'll
see
that
we
were
able
to
prove
that
our
hypothesis
is
accurate
and
you
might
find
it
an
interesting
solution
for
you
all
right.
Well,
I'll
pass
it
over
to
Dylan.
A
B
B
It
operates
on
the
community's
resources
itself
and
then
also
the
persistent
volume
data.
So
Valero
extends
the
communities
API
and
it's
runs
as
a
conveyance
controller,
and
it's
watching
for
the
creation
of
backup
and
restore
objects
and
wouldn't
actually
detects
the
creation
of
these
resources.
It
runs
the
reconciliation
process
based
on
the
specified
criteria,
to
do
the
backup
or
restore
workflow.
B
Obviously
it's
not
that
simple,
but
falero
is
going
to
do
some
of
the
heavy
lifting
of
removing
the
status
of
resource
and
removing
some
metadata
fields
that
would
otherwise
not
allow
you
to
replay
the
object
on
the
cluster.
Then
for
some
customization
you
from
the
application
front,
you
can
customize
using
hooks.
So
a
perfect
example
of
one
you
want
to
use
a
hook
would
be
say:
your
pod
was
a
Postgres
database
and
the
post-crisis
database
obviously
puts
a
lock
when
it's
running.
B
You
may
have
the
requirement
that
priority
backup,
you're
gonna,
have
a
hook
on
the
pod.
That's
actually
going
to
run
a
file
system.
Freeze
then
customization
on
a
resource
specific
case.
I
can
tell
you
how
my
team
uses
this
is.
You
know
open
shift
routes.
They
have
sub
domains
that
change
based
on
where
the
clusters
are,
where
the
actual
resources
being
replayed
on,
so
our
plugin
will
actually
go
and
do
a
subdomain
swap
on
the
route
every
one
time.
B
So,
on
the
persistent
volume
front,
you
kind
of
have
two
ways
to
do
this.
If
you
have
volumes
that
are
provided
by
a
cloud
provider,
so
think
abs
is
your
GCP
port
work
situation,
then
Valero
is
actually
going
to
use
their
native
snapshot
API,
and
so
in
that
case,
the
precision
volume
data
will
actually
be
stored
on
the
specified
cloud
provider.
B
So
talk
a
little
bit
more
about
rustic.
How
actually
works
from
the
user
perspective
is
any
volumes
that
the
user
wants
to
back
up
with
rustic
you're
gonna
find
the
Associated
pods,
and
now
those
pods
you're
gonna,
add
a
specific
annotation
and
the
annotation
is
going
to
include
the
list
of
volumes
that
you
want
to
backup.
At
runtime
and
then
when
velaro
is
actually
running
through
the
backup
and
it
finds
this
annotation,
it
goes
and
creates
a
pod
volume,
backup
resource
and
that's
another
custom
resource.
B
That's
part
of
the
valero
installation
and
when
rustic
sees
the
creation
of
a
pod
volume,
backup,
it's
simply
gonna
go
find
the
volume
perform
the
filesystem
back
up
and
store
the
data
into
object.
Storage
then
I'm
restore
it's
pretty
much
the
opposite.
If
it
finds
that
a
restore
isn't
associated
pod
volume
back
up
flower
crates,
the
pod
volume
restore
and
rustic
is
going
to
run
a
restore
on
that
person's
volume
data.
B
So
I
mention
object,
storage,
every
velaro
installation
needs
an
Associated,
backup,
storage
location
and
this
backup
storage
location
is
some
form
of
object.
Storage,
most
typically
it's
an
s3
compatible
endpoint.
That
falero
is
talking
to.
So
in
our
case
for
this
demo,
you're
gonna
see
later
we
use
nuba,
but
you
can
also
use
AWS
s3
Mineo,
which
is
an
open
source
solution
to
stand
up
an
s3
bucket.
B
Then
there
is
also
a
zero
blob
storage,
Google,
Cloud
storage,
I've
been
a
bunch
of
community
maintained
providers
and
then
the
persistent
volume
data,
if
it's
using
rustic,
will
actually
be
stored
in
this
object.
Storage
if
it's
from
the
cloud
provider,
as
I
mentioned
before
it'll
just
be
stored
on
that
then.
Lastly,
the
as
I
mentioned
it's
extending
the
kameez
API.
B
A
So
when
we
were
looking
at
putting
this
solution
together,
I
had
done
a
lot
of
work
with
with
rooks
F
and
we
needed
storage,
especially
for
the
applications
that
we
were
going
to
test
with
this
we
needed
storage.
We
also
thought
at
the
time
that
we
would
look
at
the
container
storage
interface
herself,
so
I'll
go
into
that
a
little
more,
but
rook
is
essentially
a
storage
operator.
A
In
particular,
it
has
multiple
storage
types
that
it
can
orchestrate
the
one
that
I'm
most
familiar
with
this
stuff.
So
it's
an
open
source
project.
It
started
about
three
years
ago
is
currently
a
CNC
F
project
incubation
project
as
of
about
a
year
ago,
and
it
works.
You
know
within
the
kubernetes
framework,
you
know,
there's
an
operator
there's
a
cluster
custom
resource,
so
it
in
terms
of
working
with
kubernetes,
it's
a
very
good
fit.
A
A
So,
with
an
operator
orchestrating
stuff,
you
basically
turn
the
knobs
a
particular
way
via
the
the
cluster
CR,
and
then
you
deploy
South
containerized
and
those
knobs
are
already
decided,
so
the
the
user,
for
the
most
part
doesn't
have
to
learn
all
the
complexities
of
stuff
and
how
to
tune
it.
The
the
idea
is
that
you
you
tune
it
ahead
of
time,
and
then
you
deploy
it
that
way
and
some
things
not
all
things,
but
some
things.
If
you
try
to
change
it,
it's
gonna
get
changed
back.
A
So
that's
you
know
it's
always
watching
for
what
the
desire
to
say
it
is
and
like
I
said:
if
the
desired
state
is
not
there,
it
would
move
it
back
to
that.
So
just
a
sort
of
an
example
there
in
terms
of
the
architecture
we
have
on
on
the
left,
you
know
the
clusters,
the
objects
or
the
things
that
we
want
to
do
in
terms
of
Stan,
shading
it
and
then
we're
gonna,
give
it
it
to
to
kubernetes.
Then
we're
gonna
get
resources.
Out
of
that.
A
A
Part
of
that
is
because
and
I
think
this
is
going
to
change
over
time,
but
right
now,
if
you
snapshot
your
snapshotting
into
SEF,
you
still
need
to
then
mount
that
volume
somewhere
and
get
the
data
out
and
copy
it
somewhere
else
other
than
so
that,
if
you
need
to
restore
you,
you
have
the
data.
So
by
doing
what
Dylan
went
over
using
rustic,
we
essentially
moved
the
data
and
the
volume
directly
into
into
our
backup
target
without
going
through
SEF.
A
So
here's
just
say
what
the
custom
resource
definition
could
look
like
for
the
cluster.
C
are
some
important
things.
There
are
the
sufferers,
and,
what's
really
nice
about
you
know
using
this
type
of
storage
operator?
Is
that
as
soon
as
you
go
from
one
version
ACEF
to
the
other,
so
you
literally
go
in
and
edit
the
the
cluster
custom
resource
you
go
from
one
version
and
stuff
to
the
other,
and
you
have
essentially
upgraded
stuff
it'll
do
a
rolling
update.
This
can
be
done.
A
Even
when
you
have
you
know
application
iOS
going
on
it'll
do
a
rolling
upgrade.
You
still
probably
do
it
during
a
maintenance
window.
If
you
had
large
rook
self
environment,
so
there
the
other
thing
there
use
all
notes.
This
is
a
good
I
guess
thing
to
do
in
a
POC.
You
have
to
be
careful,
but
essentially
what
that
does
is
that
if
you've
created
your
cluster
and
you
if
you're
familiar
with
kubernetes,
you
add
another
node.
A
If
that
node
has
storage
available,
it
immediately
gets
seen
by
the
by
the
operator,
and
it
adds
it
to
the
soft
clustering.
So,
like
I
said,
you
have
to
be
careful
because
sometimes
you
don't
want
storage,
add
it
to
the
stuff
cluster,
but
it'll
just
sort
of
gobble
up
everything.
So
so
what
is
new
boom
nuba
might
be.
New
nuba
might
be
new
to
some
of
you.
It's
recently
an
open
source
project,
and
if
you
search,
we
have
some
references
at
the
end.
A
Numa
core
and
you'll
see
the
the
github
and
all
of
the
information
there.
So
what
it
is
at
you
know,
sort
of
a
simple
idea:
is
it
sits?
It's
a
multi-cloud
gateway
that
is
doing
object,
service,
s3,
object
service
and
it's
it's
a
relatively
sort
of
simple
idea,
but
it's
it's
quite
powerful,
which
is
that
you
can
have
back.
What's
called
backing
store,
Anubha,
you
can
have
different
backing
store
across
multiple
clouds
and
even
multiple
types
of
storage.
So
you
could
even
have,
in
the
case
of
the
demo,
we're
going
to
show.
A
We
just
have
the
the
Nuba
backing
store.
Different
storage
devices
on
the
machines
who
are
using.
So
we,
the
rook
stuff,
is
using
some
of
the
storage
devices
and
nuba
is
using
the
other
ones.
So
it
could
be
storage
devices.
It
could
be
EBS
volumes,
it
could
be.
You
know
in
any
of
the
cloud
storage
types,
so
it's
very
flexible
if
you
add
to
that
now
that
you
can
have
bucket
classes.
These
are
our
kubernetes
resources,
so
you
can
have
a
bucket
class
similar
to
a
stores
class
that
has
a
policy
in
it.
A
That
says
you
know
for
for
this.
This.
If
you
specify
creating
a
bucket
using
this
bucket
class,
then
you
will
get
this
kind
of
policy
and
the
policy
might
be
you
know.
I
want
to
do
mirroring
across
these.
These
different
clouds
or
I
want
to
spread
the
data
in
a
particular
way.
So
the
idea
of
a
multi
cloud
gateway
in
this
case
is
truly.
That
is
that
you
have
your
your
your
data.
Your
objects
spread
across
multiple
clouds,
as
well
as
on
Prem,
and
this
is
all
done
via
the
the
the
bucket
class
definition.
A
So
what
we
want
to
do
now
sort
of
transition
into
our
demo
because
want
to
show
how
this
is
real
and
we
do
have
plans.
It's
not.
We
don't
have
a
link
here.
We
have
plans
of
uploading
this
to
YouTube
and
having
sort
of
how
we
put
that
put
it
together,
as
well
as
just
showing
you
how
it
works,
but
right
now
what
I
want
to
do
is
just
go
through
quickly.
A
A
A
Using
a
very
simple
thing,
like
cube
kettle,
delete
namespace
we're
going
to
basically
remove
all
of
the
resources
from
open
stuff
for
that
namespace
and
we'll
we'll
validate
that
it's
gone
once
it's
gone,
going
to
go
ahead
and
do
a
restore
and
the
restore
is
and
and
I
didn't
mention
it
and
the
prior
thing.
But
but
the
target
for
the
the
the
backup
is
Anubha.
So
velaro
is
configured
to
use
an
s3
bucket.
A
A
B
I
know
better
than
I
do
alive
all
right,
so
you
know.
Obviously
the
primary
tool
being
used
in
this
demo
is
Valero,
but
let's
go
ahead
and
show
you
what
that
actually
looks
like
on
the
cluster.
So
the
Valero
namespace
consists
of
a
rustic
daemon
set
and
a
Valero
deployment.
So
rustic
is
running
on
every
node
in
the
cluster
and
the
Valero
is
just
running
in
a
single
pod
and
then
let's
also
show
you
what
the
might
sorry
the
WordPress
application
looks
like,
which
is
back
on
my
sequel.
So
we
have
two
pods.
B
B
So
it
would
stump
out
the
PD
definition.
You
can
see
the
annotation
provision
by
Brooke,
stuff,
okay,
and
so
now,
let's
show
you
the
Nuba
console-
and
you
know,
as
Annette
just
mentioned,
Valera
was
installed
with
a
backup
storage
location
that
points
to
an
s3
bucket
on
nuba
and
we'll
show
you
that
in
a
second.
So
this
is
the
Nuba
console
and
in
here
we
can
go
and
click
on
the
buckets
tab
and
we
have
the
bucket
name
Valero,
and
you
can
see
that
it's
got
zero
objects
inside
of
it.
So
it's
completely
empty.
B
B
And
now
that
it's
published
we're,
actually,
you
know
we've
stored
some
new
data
in
my
sequel.
So
what
we're
expecting
to
see
here
is
after
we
create
it
back
up
and
delete
the
namespace
and
restore.
We
should
see
that
article
there,
otherwise
everything
we're
saying,
isn't
true
right.
So
the
next
thing
that
we
need
to
do
is
since
we
have
purses
and
volumes
that
were
going
to
back
up
using
rustic.
The
first
thing
we
have
to
do
is
annotate
those
pots
right,
so
we
have
two
pods
with
two
purses
and
volumes.
B
So
that's
one
annotation
on
each
pod,
you'll
see
the
annotation
looks
like
it's
back
up
top
Valero
dot,
io
/,
backup
volumes,
and
then
it
takes
in
a
comma-separated
list.
So
since
I
only
have
one
volume
here
that
I'm
trying
to
back
up,
you
can
see
it's
my
sequel,
persistent
storage
I'll
go
ahead
and
annotate
that
my
sequel,
pod
and
now,
let's
go
ahead
and
at
ap
WordPress
pod.
B
You
can
see
the
name
of
this
volume
is
just
WordPress,
persistent
storage,
so
now
that
both
pods
are
annotated,
there's
nothing
left
to
do
other
than
creating
the
Valero
backup
co.
So
to
do
this,
as
I
mentioned
before
you
can
use,
you
know,
cube
cuddle
and
create
your
own
backup
definition,
but
I
really
like
using
the
Vallejo
CLI.
It's
a
lot
simpler
and
it
allows
me
to
include
this
flag
include
namespaces.
The
thing
might
be
cut
off
a
little,
the
top,
but
I'm
gonna
specify
just
one
namespace.
B
My
sequel
WordPress
then
we're
again
given
a
name
which
in
this
case,
will
be
WordPress
cute,
comm
all
right,
so
the
backups
crated.
We
can
view
the
backup
using
the
CLI
describe
it,
and
this
is
actually
going
to
essentially
list
the
backup
resource
in
a
human,
readable
format,
which
is
why
it's
pretty
beneficial
to
use.
The
bloggers
feel
I
in
this
case,
because
it
can
actually
talk
to
the
s3
bucket
grab
logs
and
all
that
kind
of
stuff.
B
B
So
now
that
we
describe
it,
you
can
see
that
the
rustic
backups
have
completed
and
the
overall
backup
itself
has
completed
and
succeeded
with
no
warning
snores,
so
the
backup
is
done.
Let's
now
simulate
Oh
before
we
do
that,
let's
take
a
look
at
the
actual
bucket
right.
Let's
make
sure
that
all
that
backup
data
was
stored
there
and
since
we
use
rustic,
we
should
expect
to
see
all
the
rustic
snapshot
data
in
the
bucket
as
well.
B
B
Alright,
so
we've
verified
that
our
backup
is
good.
All
this
data
is
stored
in
nuba.
There's
only
one
more
thing
to
do
and
that's
to
destroy
everything.
So
let's
go
ahead
and
probably
make
them
as
common
disaster
scenario,
someone
fat,
fingering
and
deleting
your
namespace,
because
I've
done
that
multiple
times
so,
let's
bring
it
up,
run
a
cube,
cuddle,
delete,
namespace.
B
You
can
see
how
to
new
Resik
restores
at
the
bottom,
and
the
phase
itself
is
in
progress.
I
can
actually
list
the
restores
that
exist
in
the
Bolero
namespace
you're
gonna,
see
because
it
took
me,
two
tries
to
get
the
video,
but
you
can
see
the
most
recent
one
is
21
seconds.
So
that's
when
we're
gonna
use.
B
And
I'll
specify
the
details
flags
to
get
a
little
more
information
about
the
actual
rustic
restores
themselves.
You
can,
you
can
see
their
names
and
refer
back
to
the
Blair
objects,
but
now
that's
completed.
Let's
go
ahead
and
check
out
our
pods
that
are
running
in
the
WordPress
namespace
and
you
can
see
they're
back.
B
And
you
can
see,
we
have
to
persist
volume
claims
that
are
bound
to
set
volumes.
So
if
the
rustic
restore
was
successful,
we
should
expect
to
see
that
article
that
was
already
created.
Otherwise
it
would
have
brought
up
the
blank
install
of
WordPress,
so
we're
gonna,
refresh
WordPress
and
you
can
see
the
articles
still
there.
So
everything
worked
and
now
let's
go
ahead
and
prove
that
you
know
the
database
is
actually
able
to
accept
new
transactions
that
we
didn't
secure
anything
up
in
the
application
state.
B
B
Done
no
errors,
although
the
home
page
and
there's
our
second
article,
so
you
can
see
that
we
took
the
entire
application,
destroyed
its
same
space
and
restored
it
from
a
previous
backup
taken
in
bolero.
So,
lastly,
here
is
a
list
of
resources
and
links.
If
you
are
interested
in
any
of
these
projects,
we
have
some
of
the
community
information
for
each
open-source
project
and
a
list
of
websites
and
their
github.
So
yeah.
A
B
Yeah
sure
so
Valero
is
gonna
skip
the
restoring
of
any
resource
that
already
exists
in
the
namespace.
So
if
I
back
back
up
a
namespace,
that
has
say
five
deployments
and
ideally
three
of
them,
that's
what
you're
asking
about
on
the
restore
it's
not
going
to
overwrite
the
to
running
deployments.
It's
only
going
to
recreate
the
three
deleted
deployments,
yeah
sure.
B
Yes,
great
question
yeah,
so
actually
that
that
brings
up
a
he
asked.
If
there's
any
scheduling
of
backups
that
you
can
restore
from
with
Valero
and
yeah
I
didn't
get
into
all
the
specific
details
of
Buffalo
can
do.
But
scheduling
is
a
big
one,
so
you
can
schedule
a
backup
to
run
every
two
hours
every
three
hours
you
don't
have
to
actually
write
your
own
code,
automate
that
yourself
yeah
great
question,
yeah,
see
three
over
here.
I'll
start
in
the
front
yeah
you
yep.
B
B
So
did
you
catch
that
yeah
so,
most
recently,
actually
literally
last
week,
I
back
up
15
namespaces,
with
a
total
of
80
gigabytes
of
data
and
I
used
rustic
for
the
backup,
so
rustic
also
exposes
the
incremental
backups
and
restores
so
in
that
case
the
initial
backup
would
have
taken
a
long
time
how
my
team
gets
around.
That
is
we
take
multiple
backups,
then,
at
the
last
time,
when
I
actually
want
to
switch
the
application
over,
we
run
a
restore
first
before
taking
the
app
down.
B
A
B
B
Yeah
a
great
question,
so
writing
a
plugin
is
super
easy
it.
It's
essentially
two
functions
that
you
specify
the
list
of
resources.
The
plug-in
target
is
going
to
operate
on
then
the
specific
backup,
a
restore
functionality
of
how
you're
gonna
mutate
the
object.
Now
you
asked
about
filesystem
freezing
of
a
database.
That
is
not
where
a
plug-in
will
go.
That
would
go
into
a
hook
and
a
hook
right
now.
Currently,
velaro
you
can
expose
a
command
or
a
script
that
gets
run
prior
to
the
backup
operation
for
plugins.
B
That's
for
resource
specific
operations,
so
say
like
secrets:
I
wanted
to
mutate
a
specific
feel
in
the
secret
during
backup
time
you
would
list
hey.
This
is
going
to
be.
This
plug-in
runs
for
these
resources
and
it
does
the
specific
behavior
of
a
backup
time,
and
so
you
asked
if
there
was
any
community
written
plugins
there
there
are,
but
they
are
pretty
I
would
say:
I've
seen
them
kind
of
specialized
for
customer
specific
use
cases
not
necessarily
generalized
since
falero
supports
all
the
kubernetes
resources
out
of
the
box.
B
B
So
you're
asking
if
I
can
back
up
the
EBS
data
into
the
s3
bucket
itself
yep,
so
you
can
use
rustic
on
EBS
volumes.
You
don't
have
to
use
the
cloud
native
snapshot
tool
in
order
to
do
that.
You
just
have
to
annotate
the
pod.
That's
gonna
be
include
in
the
backup
and
that's
gonna
tell
Valero,
even
though
it's
an
EBS
volume
don't
use
the
cloud
snapshot,
use
rustic.
B
A
B
B
Yeah
so
yeah,
that's
actually
a
great
question
and
preventing
application.
Downtime
has
not
been
an
easy
problem
for
my
team
to
solve,
in
this
case
the
best
way
that
we
found
to
do
it
is
so
say
the
backup
takes
two
hours
and
since
rustic
allows
for
incremental
backups.
What
you
want
to
do
is
what
we've
done
is
say
two
hours
a
lot
elapses
take
another
backup
that
might
take
ten
minutes,
bring
the
application
down,
run
a
final
backup
and
that
will
probably
take
a
minute.
B
B
B
B
Yeah
so
it's
kind
of
a
couple
reasons:
one
I
don't
have
any
experience
with
CSI,
so
we
explored
it
briefly
for
using
this,
but
it's
as
a
net
mentioned
earlier.
One
of
the
drawbacks
is
if
we
were
to
use
CSI
force,
F
right.
That
doesn't
actually
give
you
data
protection
if
the
stuff
cluster
goes
down,
so
you
want
to
take
a
backup
and
store
the
data
somewhere
else
with
CSI
it's
going
to
store
the
CSI
snapshot
in
SEF,
so
we
didn't
really
feel
like
that
was
actually
providing
data
protection
for
this
time.
A
And
actually,
as
we're
looking,
you
know
having
snapshot
capability
for
CSI,
most
the
strategies
I've
seen
for
backup
and
restore
snapshot,
creative
all,
create
the
volume,
take
the
data
out
and
put
it
somewhere
else,
because
so
that
it
actually
is
protected.
So
snapshot
is
good,
but
it
doesn't
give
you
data
protection.
B
B
B
Yep
great
questions
whoo
he's
asking
how
does
bolero
manage
failures,
so
it
kind
of
separates
errors
and
into
like
hard
warning
or
hard
errors
and
like
soft
warnings.
So
an
example
of
a
soft
warning
would
be
hey.
I
can
restore
this
object
because
it
already
exists
in
the
cluster
and
a
hard
error.
Like
say
the
rustic
restore
actually
failed.
The
restore
is
going
to
go
into
an
error
State
on
the
Status
yeah.
B
The
backups
can
completely
stop
if
it's
using
rustic.
You
will
at
least
have
all
that
snapshot
data
preserved
in
the
backup.
So
the
next
time
you
ran
the
backup
it
wouldn't
have
to
go
and
redo
all
of
that
snapshot
data.
But
the
back
of
itself
is
gonna,
stop
right
there
and
go
into
an
error.
State
yeah.
B
If
the
cluster
running
the
bolero
soul
is
just
completely
wiped
away.
Well,
the
the
purpose
here
is
that
you're
using
velaro
to
store
it
off
cluster
right.
So
in
this
case,
I'm,
probably
storing
my
data
in
Amazon
or
an
s3
bucket
off
cluster
I
know
in
this
case,
nuba
was
running
on
the
cluster,
and
but
we
were
using
storage
that
was
on
the
actual
node
disk
themselves.
So
if
a
cluster
died,
but
the
machines
were
okay,
we
would
have
had
that
data
stored
still.
B
I,
don't
believe
so
I
think
that
what
would
happen
is
Valero
on
restore
it's
gonna,
say
hey.
This
namespace
already
exists
and
it
won't
recreate
anything.
So
if
there
are
objects
that
don't
exist,
that
namespace
it
will
restore
those,
but
it's
not
gonna
force
overwrite
any
objects
in
the
cluster
that
might
be
a
configuration,
parameter,
I'm,
not
sure
yeah.