►
From YouTube: Disaster Recovery for your Kubernetes Clusters [I] - Andy Goldstein & Steve Kriss, Heptio
Description
Disaster Recovery for your Kubernetes Clusters [I] - Andy Goldstein & Steve Kriss, Heptio
It’s 3am. Your pager is beeping. Your Kubernetes cluster is down. Don’t panic - we’ve got you covered. In this talk, we’ll describe a variety of disaster scenarios you may encounter. We’ll arm you with the knowledge you need to overcome them. Whether you’re a systems administrator, application developer, or end user, after this talk you’ll walk away with a thorough understanding of Kubernetes disaster recovery, including:
A disaster recovery overview
- Strategies for Kubernetes
- Comparisons to federation and high availability
- Which components to back up vs recreating from scratch
How to minimize your time to recovery
- Automate cluster creation and infrastructure configuration
- Back up and quickly restore your cluster applications, workloads, and persistent volumes using tools such as Heptio Ark
How to handle specific disaster scenarios
- Losing nodes
- Recovering from bad configuration updates
- Cloud provider outages
About Andy Goldstein
Andy Goldstein is an engineer at Heptio where he works on tooling to make operating Kubernetes clusters easier, and he also contributes to Kubernetes. Prior to his current role, Andy worked on Kubernetes and OpenShift at Red Hat. Andy lives in Rockville, MD, with his wife, two children, and two noisy cats.
About Steve Kriss
Steve Kriss is a systems engineer at Heptio working on building tools and products to help Kubernetes users be successful, and has been a contributor to upstream Kubernetes as well as a member of the Kubernetes release team in the past. Steve recently relocated to Seattle from New York and is still trying to find a good bagel.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello
and
welcome
I'm
Andy
Goldstein
I'm
Steve
Chris,
and
we
are
here
to
talk
to
you
about
disaster
recovery
for
your
kubernetes
clusters,
a
little
bit
about
ourselves.
I'm,
a
staff
systems
engineer
at
hep,
tio
I
have
been
programming
for
a
long
time.
I
started
with
Commodore
64
basic
and
I'm.
Now
up
using
go
and
I've
been
a
kubernetes
contributor,
since
2014
and
I
am
the
hub,
do
work
team,
lead
and.
B
I'm
Steeve
Chris
I'm,
a
Senior
Systems
Engineer
at
hep,
Tia
I
work
on
hep
do
arc
with
Andy
and
I've,
been
in
the
past,
a
contributor
to
upstream
kubernetes
and
also
a
member
of
the
release
team
and
in
a
past
life
I
was
an
enterprise
IT
engineer,
so
I
certainly
have
some
experience
with
the
challenges
of
designing,
implementing
and
testing
dr
strategies.
Alright,.
A
Can
I
get
a
show
of
hands?
How
many
of
you
manage
clusters
in
production?
Alright,
and
how
many
of
you
have
a
disaster
recovery
strategy,
excellent
and
how
many
of
you
have
actually
used
it
to
recover
from
something?
Ok,
a
few
of
you.
So
let's
talk
about
what
is
it
that
you
want
in
your
IT
infrastructure?
What
you
want
is
a
collection
of
servers
and
services
and
applications
that
are
all
running
perfectly
well.
A
B
B
Typically,
and
if
we
ever
had
a
disaster
and
the
server
went
down,
we
do
a
full
restore
from
our
backup
bring
up
a
new
server
that
was
essentially
identical
to
the
old
one,
and
this
would
enable
us
to
restore
the
service
for
our
application
in
the
new
world
with
kubernetes
things,
look
a
little
bit
different
than
that,
and
so
now,
when
you're
running
a
kubernetes
cluster,
you
don't
just
have
one
server.
You
have
one
or
more
masters
which
are
running
the
kubernetes
control
plane.
B
You
have
many
nodes
which
are
running
again,
some
kubernetes
components
as
well
as
all
of
your
containerized
workloads,
and
then
you
also
have
a
net
CD
cluster
which
may
be
running
in
or
outside
of
your
kubernetes
cluster.
But
this
is
actually
storing
all
your
your
kubernetes
state
information,
and
so,
let's
take
a
look
at
what's
inside
each
of
those
a
little
bit
more.
So
within
the
master,
we
have
first
of
all
the
kubernetes
api
server,
and
this
is
the
entry
point
for
creating
or
fetching
information
about
kubernetes
state.
B
We
have
a
scheduler
which
is
responsible
for
deciding
which
nodes
pods
should
run
on.
We
have
a
controller
manager
which
is
going
to
be
running
the
core
control
loops
to
constantly
push
the
state
of
the
kubernetes
cluster
towards
the
desired
state,
and
then
we
have
NCD,
which
is
our
persistent
store
of
state
information
kubernetes,
and
then
we
also
potentially
have
some
CNI
pods
and
a
cube
proxy,
which
are
going
to
help
with
all
the
the
networking
and
communication
concerns
and
then
on
the
nodes.
B
We
also
have
cube
proxy
and
the
CNI
pods
for
networking
beyond
that,
we
have
all
of
your
containerized
workloads
in
the
form
of
pods,
and
so,
as
we
start
to
think
about
how
to
design
a
dr
strategy
for
for
this
new
kubernetes
environment.
We
really
need
to
think
about.
Where
is
this
state
within
this
environment
which
components
are
stateful
and
which
are
stateless?
B
B
And
the
second
is
in
your
persistent
volumes
for
your
applications.
So
if
you
have
workloads
that
are
using
volumes
to
store
persistent
data,
we
obviously
have
a
lot
of
state
here,
and
so
these
are
really
the
key
components
that
we
need
to
focus
on
and
make
sure
that
we
have
robust
backup
strategies
so
that
we
can
restore
this
data
in
the
case
of
a
failure.
B
If
we
think
about
the
masters
and
the
nodes
themselves
that
are
running
the
core,
kubernetes
components,
they're,
really,
basically
stateless,
and
so
this
means
that
if
we
as
long
as
we
can
quickly
bring
up
new
versions
of
those
in
the
case
of
a
failure,
we
don't
really
need
to
restore
from
an
exact
copy
of
the
previous
version
of
them.
We
can
spin
up
a
new
cluster
and
as
long
as
we
can
restore
our
@cd
data
and
our
persistent
volume
data
we'll
be
able
to
restore
service
to
our
applications.
B
A
Let's
talk
about
master
and
no
disaster
recovery,
like
Steve,
said
they're,
basically
stateless.
You
may
have
some
of
these
that
are
unhealthy.
Maybe
they're,
running
okay
for
the
most
part,
but
you're
getting
some
alerts.
You've
got
a
disc
that
is
flaky
a
network
card.
That's
not
performing
correctly!
So
there's
some
tools
you
can
use
to
take
these
out
of
service
cube
control
has
a
couple
of
features:
Gordon
and
Drain.
That
you'll
probably
want
to
add
to
your
toolbox.
A
Gordon
allows
you
to
mark
a
No
is
unschedulable,
so
any
new
pods
that
are
created
will
not
be
assigned
to
whatever
node
you've.
Cordoned
and
drain
goes
one
step
further
and
will
actually
evict
any
pods
that
are
running
on
a
node
that
you're
trying
to
take
out
of
service
and
once
you've
done
that,
as
Steve
said,
you
want
to
very
quickly
be
able
to
provision
a
replacement,
master
or
node.
A
So
how
do
you
do
that
automate
it
now,
unfortunately,
we
are
not
going
to
be
able
to
tell
you
the
one
and
only
one
way
to
automate
recovering
Andry
provisioning,
a
master
or
a
node
or
a
cluster,
because
you
all
have
opinions.
You
have
IT
departments
who
say
we're
using
ansible
or
we're
using
chef
where
we're
using
puppet.
So
we
can't
tell
you
what
to
use,
but
we
are
strongly
encouraging
you
to
automate
the
creation
of
masters,
nodes
and
clusters.
A
If
you
do
lose
them,
you
have
to
regenerate
them
or
get
new
ones,
and
you
potentially
could
have
an
outage
in
that
situation
and
I
do
want
to
highlight
that
recovering
the
masters
and
the
nodes
is
not
really.
The
crux
of
the
disaster
recovery
problem.
Stateful
data
is
what
we're
really
here
about.
So
let's
talk
about
EDD
CD.
First,
there's
a
few
different
ways
that
you
can
approach:
disaster
recovery
for
ED
CD.
The
first
two
are
similar
at
the
block
level.
A
You
could
take
a
backup
of
the
partition
or
the
disk
where
the
@
CD
data
directory
resides.
This
is
where
all
of
your
EDD
State
is
and
with
something
like
this.
If
you
lose
one
of
your
members,
if
you
have
a
highly
available
etsy
D
cluster-
and
you
definitely
should,
if
you
restore
from
either
the
block
device
or
at
the
file
system
level,
you
just
take
a
backup
of
the
directory
and
restore
it.
A
When
your
member
comes
back
online,
it
will
get
a
delta
of
the
data
that
happened
in
the
cluster
since
it
was
offline,
so
the
the
surviving
members
will
send
a
snapshot
of
what
it
needs
to
catch
up,
and
so
your
cluster
can
become
whole
again.
Another
option
is
to
use
sed
control.
They
have
a
great
feature
as
part
of
the
exit
III
API,
to
be
able
to
take
a
snapshot
of
your
@cd
database
and
at
some
point
in
the
future,
restore
it.
That's
someone.
A
You've
got
to
be
a
little
bit
careful
with,
though,
because
if
you
do
a
snapshot
and
restore
when
you
restore
it
ends
up
creating
a
brand
new
at
CD
cluster.
So
this
effectively
means
you
will
have
an
outage
if
you
go
this
route,
but
it's
a
good
tool
to
have
in
the
event
that
you
have
a
total
total
outage.
You
can
certainly
recover
some
of
your
@cd
state.
A
This
way,
assuming
you
have
backups
and
then
the
fourth
option
here,
which
is
our
favorite,
is
using
kubernetes
itself
to
get
the
information
out
of
the
API
server
about
what's
running
in
the
cluster.
So
the
API
machinery
special
interest
group
spent
a
lot
of
time
building
a
discovery
mechanism,
so
you
can
go
to
your
queue,
brandies,
api
server
as
a
client,
and
you
can
say
what
are
all
of
the
api
groups
that
exist
and
within
each
API
group.
What
are
all
other
resources
that
exist.
A
So
you
can
look
at
the
core
API
and
you
can
see
that
there
are
pods
and
deployments
and
our
pods
and
secrets,
etc.
You
can
go
to
the
apps
API
and
see
all
the
deployments,
and
this
is
something
it's
very
easy
to
write.
Just
a
loop.
You
can
iterate
through
everything
and
say,
tell
me
all
the
data
that
you
have
so
what
about
persistent
volumes,
because,
presumably,
if
you've
got
stateful
workloads
in
a
kubernetes
cluster,
you
probably
are
using
persistent
volumes
for
that.
A
A
That
right
now
allows
you
to
say:
take
a
snapshot
of
my
PV
there's
a
proposed
a
set
of
api's
to
do
that,
but
they're
not
available
yet
so
if
you've
got
kubernetes
in
production
and
you've
got
persistent
volumes,
maybe
you've
got
some
tooling
that
you've
written
to
do
it,
but
unfortunately
you
can't
rely
on
kubernetes
for
that
and
there's
other
volume
types
as
well
beyond.
Just
the
cloud
provider
ones
NFS
volumes,
anything
that
can
come
in
from
a
flex
volume.
So
how
do
you
back
those
up
again?
B
So
I'd
like
to
talk
to
you
now
about
an
open
source
tool,
we
built
called
hep
do
arc
and
its
purpose
is
to
help
with
backing
up
some
of
that
stateful
data
that
we've
talked
about
within
your
kubernetes
cluster.
So
what
exactly
does
hep
do
arc
do?
So
it
has
two
core
features
and
the
first
one
is
that
it
enables
you
to
backup
and
restore
your
kubernetes
api
objects.
B
Now
Andy
just
talked
to
us
about
some
of
the
different
options
for
backing
those
up
in
terms
of
what's
in
at
CD,
and
we
use
in
our
kubernetes
discovery
API
for
accessing
all
of
that
information
and
creating
backups
of
it
as
well
as
restoring
it
in
the
case
of
a
disaster,
and
we
do
this
for
a
few
reasons
and
Andy
started
to
talk
about
some
of
the
pros
and
cons
there.
But
you
know
one
of
the
reasons
we
think
the
discovery
API
makes.
B
A
lot
of
sense
is
that,
if
you're
running
in
a
managed,
kubernetes
provider,
you
may
not
have
access
to
the
underlying
at
CD
cluster,
and
so
using
at
CD
CTL
to
take
snapshots,
may
not
even
be
a
feasible
option
for
you
additionally
arc
and
using
the
discovery.
Api
gives
you
a
lot
of
fine-grained
control
over
what
types
of
resources
you
backup.
So
with
that
CD
backups,
it's
really
kind
of
all
or
nothing,
and
if
you
want
to
restore
your
cluster,
you
basically
have
to
restore
this
state
of
the
entire
cluster.
B
If
using
the
API,
though
you
have
all
the
controls
that
it
provides
you
in
terms
of
filtering
by
namespace
filtering
by
resource
types
filtering
by
label
selectors,
and
so
we
enable
this
all
through
arc.
And,
additionally,
if
you
are
backing
up
at
CD
directly,
you
you
don't
get
the
benefit
of
being
able
to
capture
all
the
information
that
is
stored
for
extension,
API
mechanisms.
So
if
you
have
an
extension,
API
server
as
part
of
your
cluster
odds
are
that
the
data
to
support
that
is
actually
stored
in
a
separate
at
CD
cluster.
B
This
is
the
recommendation
for
for
how
to
design
these
extension
mechanisms,
and
so,
if
you're
backing
up
at
CD
you're
not
going
to
capture
that
information.
But
if
you
use
the
discovery
API,
this
will
actually
aggregate
all
of
that
information
about
extension,
API
servers,
and
so
you
can
just
back
that
information
up
directly
into
your
your
arc,
backup
or
whatever
other
backup
mechanism
you
have,
and
so
we
believe
that
the
the
discovery
API
makes
a
lot
of
sense
here
for
accessing
that
information.
B
So
our
ques
is
the
discovery
API
to
to
pull
that
all
out
of
your
cluster
and
it
creates
a
tarball
that
stores
all
this
information
and
places
the
backup
in
the
object,
storage
system
of
your
choice.
Now.
The
second
big
feature
that
arc
has
is
that
it
will
actually
backup
and
restore
your
persistent
volumes
for
you,
assuming
you're
on
one
of
the
supported
cloud
provider
platforms
and
as
Andy
mentioned
a
minute
ago.
We
we
use
the
snapshot.
Api
is
that
the
cloud
providers
offer
for
taking
backups
of
volumes.
B
Arq
out-of-the-box
supports
the
three
major
public
clouds,
but,
as
we'll
see
in
a
minute,
we
also
have
an
easy
way
to
extend
the
functionality
of
arc
to
support
the
platform
of
your
choice.
So
as
long
as
there's
an
API
for
you
to
take
backups
arc
can
easily
integrate
with
that
now,
beyond
those
two
big
features,
we
have
another
of
a
number
of
other
features
that
make
it
really
easy
for
you
to
use.
So
we
support
scheduled
backups
so,
rather
than
having
to
go
manually,
create
a
backup.
B
You
can
simply
configure
the
information
you'd
like
to
back
up
through
our
set
of
schedule
and
have
those
run
on
an
automated
basis
over
time.
Additionally,
as
I
mentioned
a
minute
ago,
we
support
complex
filtering
both
when
you
take
a
backup
of
information
as
well
as
when
you
do
a
restore
of
that
information
back
into
a
cluster.
So
you
can
filter
based
on
the
namespaces.
B
Additionally,
we
give
you
the
ability
to
restore
in
two
different
namespaces
than
you
backed
up
from,
and
so
this
is
really
useful
for
use
cases
where
maybe
you
have
an
existing
namespace
and
you
want
to
create
a
clone
of
it,
maybe
for
testing
purposes
so
that
you
can
fiddle
with
some
configuration-
or
maybe
you
have
other
use
cases
that
require
you
to
change
the
namespace
arc
makes
that
really
easy
to
do
now.
We
also
design
dart
to
be
very
extensible.
B
The
core
our
code
base
doesn't
require
you
to
recompile
or
maintain
your
own
component
container
images.
The
second
major
category
of
plugins
is
what
are
called
item
actions
and
we
support
these
on
both
backup
and
restore,
and
so
these
are
little
bits
of
functionality
that
run
as
each
item
is
being
backed
up
or
restored.
B
They're
different
from
hooks
in
that
they're,
not
actually
scripts,
that
are
being
executed
within
your
pods
they're
being
run
by
the
arcs
server
and
they
allow
you
to
potentially
call
out
to
external
systems
to
take
certain
actions
or
they
allow
you
to
actually
mutate
the
item
that
you're
backing
up
or
restoring.
So
if
you
need
to
added
some
annotations
to
items
as
you're
backing
them
up,
add
labels-
or
maybe
you
want
to
actually
modify
the
spec
as
you're,
restoring
your
backup
into
a
new
cluster.
A
On
our
script
here,
so
this
is
all
alive.
So
first
thing
we're
gonna
do
is
show
you
what
namespaces
we
have
and
we
have
typical
ones.
You'd
see
default,
cube
public
cube
system.
We
also
have
hep-c
Oh
arc,
which
is
where
arc
is
running
and
we
are
using
rook
for
dynamic
provisioning
for
persistent
volumes
today.
A
So
we're
gonna
start
by
deploying
a
simple
nginx
application
and
you'll
see
that
this
creates
a
namespace,
a
PVC,
a
deployment
and
a
service.
So
if
we
take
a
look
at
what
the
PVC
looks
like
this
is
a
rook
block,
storage
class
and
it
is
bound-
and
we
are
going
to
be
storing
the
logs
for
nginx
in
this
persistent
volume.
So
here
is
the
PV.
Similarly,
you'll
see
that
it's
bound
to
nginx
logs,
and
if
we
take
a
look
at
the
deployment,
we
want
one
replicas
and
we
have
one
running,
and
here
is
the
pot.
A
A
A
Yet
and
now
let's
actually
look
at
this
access
log,
so
we're
gonna
exec
into
the
container
and
take
a
look
at
that
file
and
pretty
vanilla
access
log
we've
got
the
initial
request
that
we
made
and
then
the
10
after
so
let's
create
a
backup.
It's
this
simple,
you
just
say
arc
backup
create,
give
it
a
name
and
then
whatever
filters
you
want
in
this
case
we're
only
going
to
select
the
nginx
namespace,
and
it's
done
so.
We
have
an
arc
backup.
The
data
is
available
in
object.
A
Storage
for
this
demo
we're
using
Mineo
deployed
into
the
cluster,
but
in
a
real
world
scenario,
you
probably
would
want
to
have
your
data
backed
up
outside
of
your
cluster.
So
it's
time
for
a
disaster.
We're
gonna
go
ahead
and
delete
the
nginx
namespace.
This
will
delete
all
of
the
components
that
we
just
deployed,
including
the
persistent
volume
that
was
dynamically
provisioned.
A
One
of
the
great
things
about
arc
is
that
it
can
walk
from
the
pod
to
the
persistent
volume
claim
to
the
persistent
volume,
to
figure
out
that
there
is
a
relationship
between
3
and
make
sure
that
we
backup
everything
that
we
need
to
be
backing
up
alrighty.
So
our
name
space
has
been
deleted
here.
I
will
prove
that
to
you,
so
you
can
see.
We
do
not
have
an
engine
x,
namespace
anymore,
and
just
to
show
you
that
there's
no
longer
a
PV
anymore.
That
is
gone.
A
We
can't
find
it
so,
let's
go
ahead
and
use
Ark
to
restore
the
backup
that
we
just
took,
and
while
this
is
happening,
I
will
say
that
when
the
backup
was
going,
what
Steve
described
about
doing
an
FS
freeze
before
and
after
the
snapshot
was
taken
was
exactly
what
we
had
or
do
today.
So
our
restore
is
done,
let's
go
ahead
and
take
a
look.
We
have
a
PVC,
it's
bound
using
the
rook
block
storage
class
again
and
we
have
PV
similarly.
A
So
this
is
just
going
to
show
everything
that
we
had
before,
but
the
the
individual
names
for
anything
that
has
a
generated
name
like
the
pods,
for
example.
This
has
a
different
name
than
it
did
before,
and
everything
is
running
fantastic.
Let's
go
to
the
service.
This
is
a
different
IP
than
we
had
for
and
we'll
go
ahead
and
take
a
look
at
that
file
system
and
again,
this
is
the
log
file
system
from
the
persistent
volume
that
are
stored
still
has
about
a
kilobyte.
That
is
wonderful.
A
It's
all
of
our
data,
so
we
have
not
lost
anything
and
just
to
show
that
we
can
augment
it
we'll
go
ahead
and
curl
it
another
time
and
take
a
look
at
the
file
size
and
the
file
one
more
time.
So
you'll
see
that
1045
has
gone
up
to
1140
and
if
we
take
a
look
at
the
file
one
last
time,
you'll
see
that
we
had
a
series
of
requests
from
20
past
the
hour
and
then
the
last
one
from
23
minutes.
So
our
backup
was
successful.
A
B
Thanks
Andy
for
the
demo,
so
I'd
like
to
say
please
come
join
us
in
the
art
community,
so
Andy
and
I.
Obviously,
both
work
at
hep
do
but
Ark
is
completely
open-source.
We
have
a
number
of
external
contributors
who
have
been
working
on
Ark
since
the
initial
release
and
so
we'd
love
to
have
you
come
join
us
whether
it's
to
provide
feedback
on
arc,
whether
it's
to
provide
real-world
use
cases
that
you're
using
it
for
or
whether
you'd
like
to
add
features
yourself.
B
A
Really
are
looking
for
your
input.
We
have
so
many
ideas
about
backup
and
recovery,
but
I'm
sure
you
have
more
and
specific
needs,
so
please
do
come
and
find
us
whether
it's
today
or
next
week
next
month.
We
would
appreciate
the
input
and
at
this
point,
if
anyone
has
any
questions,
we
would
be
happy
to
answer
them.
D
A
A
A
E
A
A
really
good
idea-
and
it's
actually
in
line,
was
something
we
are
planning
on
doing,
which
is
in
addition
to
the
pod
hooks
that
Steve
mentioned.
We
do
want
to
have
just
overall
backup
level
hooks
so
when
it
starts
send
up
a
web
hook
when
it
finishes
what
it
was
successful
or
a
failure
send
out
a
notification
as
well
yeah
I.
B
E
A
good
pattern
for
now
cool.
Thank
you,
okay,
so
great
presentation
in
the
multi
classic
cig,
we're
looking
at
disaster
recovery,
use
cases
and
trying
to
figure
out
how
to
do
that,
and
it's
just
curious
whether
you'd
seen
anybody
do
this
particular
scenario
where
you
have
like
a
primary
cluster
and
a
secondary
cluster
and
something
that
monitors.
If
the
primary
cluster
goes
down
and
then
uses
you
know
arc
to.
Basically,
you
know
launch
what
was
running
on
the
primary
on
the
secondary.
A
That's
a
very
good
question:
I,
don't
think!
We've
heard
any
specific
requests
around
that,
but
one
of
the
things
that
is
on
our
roadmap
is
being
able
to
take
a
backup
that
was
in
say
one
region
if
you're
on
a
cloud
and
be
able
to
migrate
that
over
to
a
different
region
and
restore
over
there,
so
that
potentially
could
play
into
what
you're
looking
to
do.
But
I
think
art
could
definitely
fit
into
that
picture
in
terms
of
monitoring
and
automating
moving
the
data
from
cluster
cluster
as
needed
and.
B
I
would
I
would
also
add
to
that
that,
because,
because
Ark
uses
CR
G's,
you
always
have
the
option
to
write
a
layer
yourself
on
top
of
Ark.
That's
monitoring
the
health
of
your
primary
cluster,
and
if
there
is
a
disaster,
you
can
write
code
to
basically
create
a
restore
CR
D
in
your
secondary
cluster
and
automatically
restore
objects
into
that.
So
that's
that's
something
you
can
very
easily
do
around
arc
school.
Thanks
is
very
helpful.
F
Okay,
I'm
wondering
like
what
are
preconditions
that
might
cause
arc
to
fail.
Like
I
asked
a
question
before
it
gave
me
an
idea.
What,
if
let's
say
you
toured
that
a
namespace
or
you
lost
the
namespace
that
you
had
backed
up
and
you
wanted
to
restore
it,
but
for
some
reason
somebody
brought
that
namespace
up
and
maybe
had
some
of
the
resources
created
already.
What
would
happen
if
I
did
a
restore
sure.
A
So
what
art
does
right
now
is
it
will
try
to
create
every
single
object
that
you've
specified
as
part
of
the
restore
and
if
it
encounters
any
conflicts
it
logs
it
as
a
warning
right
or
it
puts
it
in
the
status
as
a
warning
right
now,
so
it's
very
visible
when
you
see
that
the
restore
has
completed
it'll,
tell
you
if
there
were
any
errors
which
would
be
catastrophic,
it'll
tell
you
if
there
any
warnings
such
as
there
was
a
conflict
at
the
present.
We
just
record
that
fact.
A
So
if
there's
something
that's
already
pre-existing
in
the
namespace,
we
won't
touch
it
in
the
future.
What
we'd
like
to
do
is
make
it
pluggable
so
that
you
can
say
on
a
conflict.
Here
is
custom
logic
to
run,
to
make
the
decision.
Do
I
patch,
what's
in
there
already
with
what
I
have
do,
I
accept?
What's
already
there
or
do
I
replace?
What's
there,
what
came
from
the
backup
are.
A
Yes,
the
one
that
I
can
think
of
off
the
top
of
my
head
load,
balancer
services,
those
depend
on
the
UID
of
the
service
and
that's
not
a
field.
That
is
something
you
can
mutate
or
set
it's
set
by
the
API
server
itself.
So
if
you
have
a
load,
balancer,
API
service,
tied
to
say
an
amazon
ELB,
if
you
take
a
backup
and
you
do
a
restore
you're
gonna
get
a
different
one.
Unfortunately,
and
hopefully
we
can
work
with
the
community
to
see
if
we
can
solve
that.
How.
A
A
We'll
go
as
fast
as
the
API
server
will.
Let
us
right
now
we
aren't
setting
the
QPS
on
the
client,
so
I
believe
the
default
is
about
five
requests,
a
second
which
certainly
could
be
slow.
If
you
have
a
lot
of
data,
we
do
plan
to
make
that
configurable
and
then,
as
far
as
the
PV
snapshots,
go
it's
as
fast
as
your
cloud
provider
or
whatever
you're
using
can
do
the
snapshots
over
here.
So.
C
Just
to
tie
in
to
the
last
two
comments,
it'll
be
great
to
see
it
run
faster,
actually
because
it
takes
a
couple
hours
to
do.
A
restore
something
event
of
a
major
outage.
They'll
be
pretty
difficult
for
us,
and
you
know
our
at
CD
size
about
850,
Meg's,
two
to
3,000
pods
and
a
whole
ton
of
config
maps
that
helm
leaves
behind
and
the
other
thing
is.
So
what
are
we
doing
about
load
balancers?
Because
that's
a
major
impediment
to
restore
I.
A
A
C
I,
so
my
questions
related
to
previous
ones.
Also,
how
well
does
it
interface
with
other
things
that
are
managing
resources
like
if
I
do
a
home,
deploy
it's
kind
of
keeping
track
of
what
resources
are
part
of
the
chart
and
something
horrible
happens?
Everything
goes
away,
I
restore
using
arc.
If
I
do
another
helm
deploy,
will
it
pick
up
correctly
or
will
try
to
start
a
whole
new
thing.
A
I'm
not
intimately
familiar
with
helm,
but
the
way
that
our
backups
and
restores
generally
work
is
we
backup
the
majority
of
the
object
we
may
strip
off
status,
for
example,
and
then
most
of
our
objects.
We
restore,
as
is
there's
a
couple
of
exceptions
here
and
there.
So
if
there
are
certain
pieces
of
data
that
you
need
that
we're
accidentally
stripping
off
or
not
on
purpose,
then
police
file
an
issue
if
you
find
problems
and
will
correct
them.
Okay,.
D
Here
so
to
answer
the
question
earlier
for
load,
balancers,
there's
actually
an
open
PR
to
set
the
load,
balancer
name,
which
cloud
providers
use
to
look
up
load
balancers.
So
then
they
will
be
out
here,
they'd,
be
able
to
restore
then
and
not
rely
on
the
UID
for
a
name,
that's
open.
So
if
you
want
to
go
comment
on
that,
PR
we're
trying
to
figure
out
a
good
way,
but
also
for
a
question
for
arc.