►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Beyond Storage Management - Andrew Large & Yinan Li, Google
Kubernetes added alpha support for persistent volume snapshotting in 1.12 through the Container Storage Interface (CSI). While having some limitations, this feature is critical to stateful workloads and serves as one of the building blocks for developing advanced, enterprise-grade capabilities around data protection. This talk provides an overview of standard enterprise data protection policies and practices and discusses how those might map into Kubernetes. We’ll discuss the full scope of what data protection might look like in Kubernetes and considerations that go into building an enterprise-grade data protection solution, placing the volume snapshot work in a larger context, and propose some explicit potential future standards activities.
https://sched.co/UaWQ
A
And
my
chick
all
right,
it
works
so
I'm
Andrew,
so
we're
gonna
be
covering
three
topic
areas
today:
a
kind
of
general
overview
data
protection,
kind
of
mapping
that
into
kubernetes
and
then
I'm
going
to
hand
it
to
you,
nan
who's,
going
to
deep
dive
on
a
number
of
sort
of
specific
and
considerations,
and-
and
things
like
that,
we
will
ask-
will
leave
time
at
the
end
for
questions
and
in
order
to
get
through
this.
This
lives
we're
going
to
ask
that
you
kind
of
hold
those
questions
to
the
end.
A
A
So
the
basic
idea
when
we
talk
about
data
protection
and
in
particular,
sort
of
the
disaster
recovery
kind
of
mode
of
that
is
this
idea
that
you're
saving
a
point-in-time
view
of
the
system
that
you
could
then
use
in
some
fashion
at
a
later
date
right.
So,
there's
a
lot
of
reasons
why
you
might
want
to
use
it,
you
might
want
to
use
it
for
some
sort
of
failure.
Recovery.
You
might
want
to
use
it
to
make
copies
of
parts
of
your
system.
A
What's
what
production
looks
like
so
generally,
when
we
talk
about
state
in
this
case,
there's
sort
of
two
kinds
of
state
and
try
not
to
map
it
too
much
to
kubernetes
at
this
point,
but
the
idea
is:
there's
state
that
represents
kind
of
the
construction
of
your
system
and
in
a
VM
environment.
This
might
be.
What's
the
you
know,
the
kind
of
the
OS
setup?
It
might
be
the
software
that
you
have
installed
your
network
configurations,
there's
a
variety
of
kind
of
things
and
then
there's
the
actual
sort
of
application
data
that
you've
stored,
persistently.
A
There
are
two
key
principles
we
generally
talk
about.
When
we
talk
about
this
one
is
it's
called
every
point
objective
the
others
recovery
time,
objective,
they're,
both
measured
in
time,
and
they
reflect
two
different
attributes
involved
in
the
backup
and
recovery.
So
the
recovery
point
objective:
is
this
idea
that
capturing
some
state
and
then,
if
I,
have
to
use
it
later?
A
How
old
is
that
state
by
the
time
I
use
it,
and
so
you
know
in
general,
for
both
of
these
lower
is
better
so
that
if
I,
if
I
use
it
and
I
can
say,
I've
got
some
kind
of
disaster
and
I
want
to
recover
from
that
disaster.
I
don't
want
my
last
backup
to
have
been
six
months
ago
right,
so
that's
RPO
and
then
RTO
is
a
measure
of
okay.
A
Given
that
I've
got
this
data,
no
matter
how
old
it
is,
how
long
does
it
take
to
recover,
and
so
there's
a
set
of
processes
and
whatever
and
and
part
of
the
issue
there
is.
How
available
is
that
data
to
me
that
I
want
to
recover
from
so
if
I've
stored,
that
you
know
vault
on
tape,
someplace
and
I've
got
to
have
a
guy
go
grab
the
tape
and
come
out
and
undo
it
or
that's
that'll,
give
you
a
large
RTO.
A
A
Generally
speaking,
when
you
have
this
kind
of
journaling
rights
or
things
like
that,
you
know
you.
There
is
a
kind
of
metadata
operation
where
you
can
say
hey
for
all
of
the
blocks
that
are
currently
active,
now
just
kind
of
make
a
record
of
those
that
are
active
at
this
point
in
time,
and
then
any
subsequent
rights
will
create
new
blocks
and
so
I
get
this
notion
of
a
snapshot
of
a
of
a
metadata
view
of
what
the
data
looked
like
at
a
point
in
time.
A
So,
generally,
right,
they're
really
quick
to
recover
from
relatively
quick
used
to
have
to
worry
about
work,
load,
stuff
and
things
like
that
use
this
space
in
the
actual
storage,
not
initially,
but
as
that
data
gets
modified
over
time,
you'll
start
to
use
space
and
then
you've
got
the
garbage
collection.
Cleanup
issue
things
like
that,
but
for
a
lot
of
the
scenarios
we
talked
about,
it
has
relatively
low
applicability,
I'm
the
first
one
being
disaster,
I
mean
if
I
lose.
A
My
storage
I
lose
my
snapshots
along
with
with
my
primary
storm,
so
it
tends
to
be
more
useful
for
things
like
I
am
going
to
do
an
application
upgrade
and
it
is
possible.
This
application
is
going
to
corrupt
my
data.
So
let
me
take
a
snapshot
before
I
do
the
upgrade
and
then,
if
it
did
corrupt,
my
data
I
can
just
roll
back
to
the
snapshot.
A
So
backup
is
a
little
bit
different,
where
you're
actually
taking
that
data
and
copying
it
into
different
media
right
back
up
can
be
incremental.
So
you
do
this
a
you
can
think
of
it.
The
first
time
you
should
take
a
full
copy
of
everything
and
then
subsequent
times
you
figure
out
just
what
blocks
have
changed
and
those
are
the
only
additional
blocks
you
need
to
store.
A
And
finally,
the
last
I'm
not
going
to
talk
too
much
about
it,
but
this
idea
of
having
the
same
kind
of
storage
and
just
sort
of
replicating
between
there,
and
so
this
can
provide
a
kind
of
high
availability
sort
of
scenario
where
your
downtime
is
a
lot
less.
But
of
course
it's
also
really
expensive.
I
mean
it's
using
primary
storage
in
both
locations,
etc.
A
So
when
we
talk
about
data
protection
policy,
of
course,
the
main
thing
we're
looking
at
is
minimizing
RTO
RPO,
but
also
minimizing
cost
and
so
a
common
approach.
There
is
to
have
this
kind
of
mix
of
snapshots
and
backups.
Where
you
you
keep
a
certain
number
of
snapshots.
Bounded
by
you,
know,
cost
considerations,
and
you
know
the
storage
overhead
and
that
solves
a
certain
set
of
problems,
and
then
you
use
backup
to
kind
of
fill
the
gap.
There
is
this
kind
of
often
quoted
three
to
one
rule.
A
A
So
in
that,
in
that
typical
environment,
you
often
have
a
couple
of
different
roles
involved.
You've
got
sort
of
what
we
call
the
infrastructure,
administrators
or
cluster
admin
or
whatever
you
want
to
call
it,
and
these
are
the
folks
that
are
generally
you
know
they
have
that
responsibility
to
sort
of
put
the
whole
thing
together,
don't
necessarily
know
what
applications
are
really
running
or
whatever,
but
they
are
also
often
handed
the
responsibility
for
executing
backups
and
things
like
that.
A
And
then
you
have
a
different
group
of
folks
who
are
usually
more
experts
in
a
particular
set
of
domains,
they're
responsible
for
particular
applications,
and
but
they
don't
have,
maybe
the
keys
to
the
kingdom
to
have
access
to
all
the
infrastructure
and
don't
have
necessarily
the
all
of
the
tools
necessary
to
be
able
to
do
the
backups
so
that
that's
a
point
that
we'll
come
back
to
later
so
so
now
trying
to
map
this
to
kubernetes.
There's
a
couple
things
we'll
talk
about.
You
know
what
what
is
it
that
we're
talking
about
scope
wise?
A
So
if
you
think
of
that
state
that
we're
trying
to
capture
in
coop
or
not
kubernetes
effectively,
it
breaks
down
into
two
categories:
there's
the
configuration,
the
resources
that
are
Annette,
CD
things
like
that,
the
the
stuff
that
describes
the
the
overall
system
and
then
there's
the
data,
that's
in
the
persistent
volumes
that
you've
got
so
we
talked
about
the
config
side,
there's
a
couple
of
different
strategies,
so
one
strategy
is
to
say:
look.
My
config
is
entirely
driven
by
stuff
that
I've
checked
into
my
source
code
control
and
I.
A
Just
build
my
systems
with
that,
and
we
kind
of
call
that
git
office
or
something
like
that.
Then
there's
the
complete
opposite
extreme,
which
is
man
I,
just
do
backup
and
recovery
on
that
I.
Don't
know
what
how
that
stuff
got
in
there
I'm
using
clique,
ops,
I'm
doing
what
I've
created
a
working
cluster
and
I
just
want
to
be
able
to
recover
to
that
all
right,
and
so
that's
the
sort
of
backup
recovery
model
and
then
there's
kind
of
a
hybrid
model
which
is
like
you
know.
A
For
my
cluster
level
resources
me
the
infrastructure
team.
We
put
that
together.
We
understand
that
that's
all
driven
from
our
source
code
control
whatever,
but
when
it
comes
to
the
applications,
you
know
they're
they're,
a
grab-bag,
some
of
them.
You
know
somebody
installed
helm,
charts.
Somebody,
you
know
installed
something
off
of
a
GUI.
Somebody
actually
has
this
full
thing,
and
so
maybe
there's
a
hybrid,
maybe
I
handle
configuring.
A
One
mechanism
and
I
handle
the
sort
of
application
config
bits
in
another
and
then
finally,
for
the
data
you
know
you're
talking
about
there's
for
persistent
volumes,
so
the
work
that's
been
done
to
date
on
this,
the
volume
snapshot
work.
Is
this
idea
of
being
able
store,
snapshots
in
the
local
cluster
storage
pool
and
then
volume
backups,
which
is
something
we're
starting
to
talk
about,
which
is
this
idea
of?
A
A
The
basic
idea
here
is:
there's
a
there's:
an
extra
a
couple
of
CR
DS
involve
where
you
can
ask
for
a
snapshots,
there's
a
claim
and
content
side
very
similar
to
the
way
PB
CPV
czar,
and
then
there
there
are
new
interfaces
added
to
the
CSI
spec
and
then
there's
a
new
CSI
sidecar,
that's
responsible
for
taking
those
those
kubernetes
constructs
and
plumbing
them
down
through
into
the
CSI
driver.
So
this
is
something
that
has
just
recently
undergone
considerable
surgery.
A
I
think
we've
got
sort
of
the
architecture
of
it
right
and,
and
there
might
be
some
implementation
tuneups,
but
but
I
think
at
this
point
this
is,
you
know
pretty
much
well
committed
feature,
which
is
this
volume
snapshots.
So
then
there's
a
number
of
other
things.
We're
sort
of
talking
about
that.
Aren't
there
yet.
But
one
of
the
things
is,
you
know
the
way
that
snapshots
work
today
is
PBC's,
have
a
data
source.
So
when
you
create
a
PVC,
you
say:
oh
yes
and
populate
it
from
this
snapshot
or
this
other
volume.
A
So,
if
you
can
imagine
with
volume,
backups
you're
now
talking
about
things
like,
oh,
yes,
and
where
do
I
store
this
backup
and
also
of
course,
then
how
do
I
import
a
backup
into
a
system
from
that
remote
location.
So
when
you
add-
and
then
there
are
other
considerations
there
too-
that
you
know
and
we'll
talk
about
some
of
these-
but
you
know
snapshots
is
inherently
a
primary
storage
responsibility.
A
You
know
multi
volume
operations
consistently,
so
a
single
point
in
time.
Capture
across
multiple
volumes
obviously
would
apply
to
both
snapshots
and
backups
potentially
and
then
the
last
topic,
which
again
you
know
I'll,
spend
some
time
on.
Is
this
idea
of
app
Kayson
consistent?
So
everything
we've
talked
about
so
far
has
been
sort
of
storage
level,
but
you
know
some
applications
can
do
what
we
call
crash
consistent,
which
is
at
any
point
in
time.
B
Right
thanks
Andrew,
so
now
we'll
walk
you
through
some
of
the
considerations.
I'll
go
into
building
a
production,
great
data
protection
system,
so
first
off
always
wanted
backups
I've
already
talked
about.
Like
you
know,
existing
says:
I
want
snapshot
now,
so
some
thing
about
wall
and
backups.
So
the
existing
says
one
software
support
is
for
managing
storage,
driven
wanna
snap
shots.
That's
your
list
or
you
know,
in
a
Latino
store
system.
That's
attached
to
your
cluster!
So
it's
for
info
cluster
data
protection
semantics.
So
but
for
backups,
it's
Troy!
B
You
know
it's
not
really
clear
like
how
backup
relate
assessment
takes.
Can
capture
actually
fall
into
the
system
and
particularly
it's
missing
several
key
pieces
of
information
that
are
necessary
for
supporting
wall
and
backups.
One
is
the
target
location
so,
basically,
like
you
know,
wearing
one
of
your
backup
to
be
to
go,
and
it's
also
meeting
a
dismantles
for
a
global
ID
that
can
uniquely
identify
a
wallet
backup
even
across
clusters.
It
also
doesn't
have
a
defined
import
and
export
flow
for
wallet
backups.
B
So
without
that,
it's
really
impossible,
or
you
know
to
actually
use
a
warning
backup
in
your
cluster
that
is
different
from
the
one
it
was
orange
or
cheddar
and
generate
it.
So
it
also
like
any
mission,
it's
part
of
a
tightly
coupled
with
my
storage.
So
when
it
comes
to
about
implementing
walling
backups,
there
could
be.
You
know
several
backup
model
now
people
might
want
to
use.
B
So
one
is
to
still
you
know
for
a
while
and
backups
through
the
pursue
the
primary
storage
system,
if
they
support
that
and
another
alternative
model
would
be
to
allow
people
to
have
to
use
separate
or
third-party
backup
providers
or
twos.
So
the
benefit
of
this
approach
is
that
it
potentially
allows
for
portable
backups
that
you
can
actually
use
across
different
storage
systems
or
even
between
you
know,
different
cloud
providers.
B
So
the
next
consideration
is
layer
administration.
So
typically,
data
production
or
storage
management
is
a
concern
of
infrastructure
at
administrator,
who
may
not
really
know
how
to
orchestrate
application
snapshot
or
backups.
Because
there's
you
know,
there's
applicant
domain
knowledge
there
that
they
probably
don't
have.
So
there
are
a
several
approaches
to
this
problem.
One
is
okay,
so
for
system
it
means
which
is
so.
They
just
rely
on
hooks
that
never
taught
by
application
operators.
B
Okay,
you
just
so
all
they
care
about,
like
you
know,
is
how
to
reliable,
run
these
hooks,
but
they
don't
really
care
about.
You
know
what
hope
do
or
what
or
how
to
how
to
like.
You
know
how
to
actually
run
them.
So
a
second
approach
is
to
treat
application,
backup
and
recovery
as
a
separate
problem
form
from
cluster
level,
backup
and
recovery,
and
for
awhile
I've
taken
an
operator
separate
tools
for
doing
application
levels.
You
know
data
protection,
so
the
cert
option
would
be
to
provide
some
mechanism.
B
So
when
you
do
cluster
level
backup
the
system
for
cluster
level,
backup
can
actually
automatically
orchestrate
application
backup,
as
in
the
same
process,
for
example,
the
system
can
actually
like
an
automatic
automatically
detect.
All
the
application
is
running
eating
the
cluster
and
then
figure
out.
What's
the
right
way
to
backup
each
individual
application
then
call
these
in
a
respective
mechanism
when
it's
time
to
back
up
these
things.
B
So
we've
talked
a
lot
about,
like
you,
know,
application,
backup
and
application
recovery.
So
a
key
to
these
things.
This
application
consistent
see.
So
the
goal
is
to
infer
that
your
entire
application
state
is
recoverable
and
the
recover
state
is
consistent
and
makes
sense
a
post
recovery.
So
this
when
it
comes
to
achieving
application,
consistency,
there's
typically
some
kind
of
flash
or
acquiescence
tab
before
we
can
simply
capture
one
of
snapshots
or
wanted
backups
and
after
your
Dow
is
your
one
in
snapshot
or
backups
there.
B
You
run
Kriya
step
to
unfreeze
your
application,
so
you
know
to
resume
their
normal
operations.
So
this
is
general
requireth.
Then,
when
your
application
in
your
opera
can
be
warmer,
you
don't
have
crushed
consistency
or
your
application
actor
managers.
You
know
a
bunch
of
different
volumes,
a
bunch
of
autumns
so
to
implement
application
consistency
in
kubernetes,
so
a
common
approach
would
be
to
define
quiesce
and
uncurse
commands
as
hooks
that
you
call
before
and
after
you,
you
do
the
warm
level
snapshot
or
backups.
B
So
there
could
be
many
kinds
of
hooks
and
some
hooks
are
pretty
generic.
Like
you
know,
FRS
freeze
and
some
hooks
can
be
pretty
application.
Specific,
like,
for
example,
for
MongoDB.
You
might
want
to
call
FS
dot,
F
synced
out
a
PDF
sync,
a
locked
actually
lock
your
database
and
flash
all
the
rights
before
you
can.
B
So
the
first
step
you
already
do
in
our
case
would
be
to
run
dbfs
sync
lock
to
flash
authorize.
Then
lock
your
database
and
well.
The
database
is
locked.
You
take
a
data,
wonnum
snapshot.
Basically,
if
you
use
that
use
the
MongoDB
for
system
level,
backup
semester,
you
are
making
an
exact
copy
of
of
all
the
data
files
and
after
that
you
run
FS
f,
sync
on
lock
to
unlock
your
database.
So
then
you
resume
the
normal
operations
of
your
MongoDB
instance.
B
So
you,
you
may
be
aware
that
a
lot
of
databases,
proper
databases
like
my
sequel,
Postgres
and
MongoDB.
They
all
provide
some
kind
of
data
replication
through
some
form
of
master
slave
architecture.
So
the
idea
is
that
writes
a
replica
date
all
the
way
from
the
master
to
auto
replicas,
so
tall,
Aitor,
asynchronously
or
synchronously
well,
even
even
though
most
most
likely
will
be
asynchronous.
So
in
that
case
you
it's
undesirable
to
actually
pick
the
master
replicas
for
backup,
because
that
means
dung
time
for
right,
because
well
the
master
is
freezed.
B
First
of
all,
you
find
and
pick
a
secondary
replicas
or
slave
replicas
and
take
that
replicas
offline
and
flush
and
clears
that
one-
and
you
know
well
then,
to
the
wall
in
level,
snapshot
or
backups,
and
then
are
you
on
quiesce
that
replica
afterwards,
and
so
when,
when
it
comes
to
recovery,
you
use
the
rub.
You
use
the
backup
you
took
from
the
second
or
replicas
and
use
that
to
restore
both
the
master
and
all
the
slaves
aerobic
house.
B
So
we've
been
talking
a
lot
about
what
do
we
could
call
storage,
mediated
application
back
up
man
because
you
know
sort
of
sort
of
like
an
relies
on
phone
system
level
or
volume,
level,
snapshot
or
backups
that
are
your
Lea
driven
by
the
primary
store
system.
So
but
there's
an
alternative
approach
that
we
call
application,
mediated,
backup
and
recovery
approach.
So
in
this
some
of
the
benefits
of
this
approach
include
like
data
portability,
and
so
because,
most
for
most
of
some
idea,
it's
actually
pretty
simple.
So
it's
a
you
know.
B
Instead
of
using
doing
volume,
level,
snapshot
and
backups,
rather,
you
just
use
application,
specific
tools
and
methods
like
dump
or
my
sequel
dump,
to
do
the
backup.
So
in
our
case,
data
generated
in
this
way
is
your
lay
logic.
So
they're
not
really
apply
to
particular
system.
There
are
agnostic
to
the
storage
system,
so
it
means
the
data
generated.
B
Is
this
way
they
are
portable,
so
the
you
can
actually
really
move
the
data
between
you
know
different
storage
systems,
even
between
different
different
cloud
providers,
and
also
most
of
these,
who
allow
you
to
run
them
well,
your
application
is
running.
For
example,
you
can
actually
run
a
single
town
wall.
Your
database
is
still
taking
rights
and
you
can
use
MongoDB
dump
to
dump
your
database
while
your
your
database
is
still
serving.
You
know,
requests.
B
So,
in
our
case,
that
means
there
won't
be
downtime
for
your
application,
even
though
your
application
going
to
have
a
single
replica,
but
there
might
be
some
performance
now
penalty
involved.
So
the
example
oxygen
flow
in
this
case
will
be
like
fine
I
pick
a
secondary,
replicas
I
still
doubt.
That
idea
applies
because
you
want
to
have
minimum
impact
to
your
running
application,
and
you
run
the
two
against
that
replica
to
perform
some
kind
of
data
dump
and
then
upon
completing
the
data
dump.
B
B
So
here's
example
that
either
it
looks
straight
like
you
know
how
application
mediated
back
that
mediated,
backup.
Look
like
I'm
using
MongoDB
s
as
one
example
here.
So
assuming
you
have
a
MongoDB
replica
set
with
one
master
replicas
and
two
secondary
replicas,
and
so
the
system
would
be
able
to
find
a
secondary
replicas.
That
has
the
least
replication
lag
and
one
Dom
inside
the
container
and
Dowell
generates
a
data
dump
somewhere
in
a
you
know,
in
a
storage
and
then
after
that,
you
can
actually
move
your
Dom
to
some
remote
storage
for
backup
purposes.
B
So,
to
summarize
data
protection
on
communities,
it's
actually
a
multi
persona
problem
as
so
it
you
know
like
Andrew,
mentioned,
there's
a
different
kind
of
roles
involved
here,
like
system
administrator
cluster
remains
or
application
operators,
so
they're
all
involved
in
this
system
so
and
also
data
protection
non-community.
It
has
a
lot
of
potential
use
cases,
for
example,
most
currently
you
can
use
that
for
disaster
recovery,
or
you
can
actually
use
that
when
you
want
to
upgrade
your
stay
for
applications.
B
So,
basically,
like
you
know
before
you
do
the
upgrades,
you
take
a
snapshot
and
you
can
actually
safely
rolled
out
back
once
you
know
if
the
after
it
fails
for
whatever
reason
and
somehow
our
use
cases
also
include
like
migration
from
one
storage
system
to
another
or
from
one
cloud
provider
to
another
and
yeah
so
and
like
what
we
were.
You
know.
The
point
we
were
trying
to
make
is
the
storage
management
incriminates
goes
beyond
simple
warning.
B
Snapshots
has
a
lot
of
things
in
was
especially
when
you
come
to
application,
backup
and
recovery,
or
you
know
quite
oh
I
feel
different
concerns
that
you
have
to
take
into
consideration
and
yeah.
So
I
have
gone
through.
You
know
some
of
the
considerations
that
go
into
building
a
such
a
production-grade
protection
data
protection
system
for
kubernetes,
all
right
so
where's
that
so
any
questions.
C
So
I
think
the
craziest
part
of
trying
to
get
this
going
is
the
flush
part
so
having
have
an
application,
regardless
of
what
it's
built
in,
have
a
the
ability
to
say
yes,
I
am
completely
flushed
and
I
will
no
longer
do
these
volatile,
writes
and
I
was
wondering
if
you
have
like
a
specification
for
what
a
flush
would
be,
because,
basically
is
it's
going
to
be
per
application?
This
is
your.
If
you
do
have
such
a
flush
call
to
to
a
running
service,
it
will
be
very
arbitrary.
C
A
Yeah
it's
an
interesting
question
and,
and
part
of
this
is,
is
you
know
the
the
windows
world
has
a
kind
of
cooperative
involvement
here
with
the
VSS
right
and
we
don't
have
something
like
that
yet
and
kubernetes
orb
linux,
but
that
might
be
an
interesting
thing.
We
haven't
actually
dived
deeply
into
that.
So
what
we've
at
this
point
pretty
much
said,
is
yeah.
A
It's
kind
of
an
application-specific
problem,
which
is
the
applications
got
to
have
some
way
to
do
it,
and
there
are
some
filesystem
level
things
like
the
FS,
freeze
or
stuff
like
that,
but
yeah.
There
are
definitely
edge
considerations
there
and
one
of
the
things
you
end
up
having
to
do
is
say
things
like
what,
if
I
also
I
want
to
put
a
timeout
on
this
hook
and
what
happens
if
it
exceeds
my
timeout?
Do
I
bail
on
the
on
the
backup?
Do
I
figure
it's
good
enough
and
I'll
move
forward
and
yeah?
D
D
A
So
so,
with
respect
to
integrity
there,
there
are
a
number
of
approaches
for
like
right,
only
or
right
once
kinds
of
scenarios
that
people
put
in
there's.
Also
you
want
to
generally
put
you
know,
you
separate,
say
metadata
about
hashes
or
something
of
the
content
and
yeah.
We
don't
talk
about
a
lot
of
that,
but
I
know
there
are
several
backup
providers
that
focus
on
things
like
that
to
try
to
provide
an
integrity
loop.
A
As
far
as
access
to
that
you
know
to
some
extent
that's
a
question
of
of
what
the
access
to
this
to
whatever
your
secondary
storage
is
I
mean
in
a
lot
of
cases.
We
presume
that's
likely
to
be
sort
of
cloud
provider,
object,
storage,
and
so
it's
you
know
following
best
practices
for
that
kind
of
thing,
but
if
it's
a
physical
security
issue
and
you're
storing
your
data
on
tape
in
the
same
underneath
the
same
physical
security
constraints,
then
yeah,
if
somebody
gets
one
they'll
get
the
other
but
I
agree.
It's
a
consideration.
E
A
So
adopting
alpha
features
is
painful
right
because
you've
got
to
start
up
the
API
server
in
a
way
that
releases
the
Alpha
case.
So
I
would
say
to
the
extent
that
I've
seen
this,
it
is
only
customers
who
are
playing
around
with
it,
not
any
serious
workloads
on
it,
and
it's
actually
been
more
storage,
vendors
and
folks,
like
that
who
are
trying
to
build
implementations
that
that
have
have
exercised
it.
E
A
F
A
G
B
For
example,
like
you
know,
one
one
thing
is
like:
how
did
how
will
be
the
standard
way
of
like
you
know?
What
defining
the
hooks
would
be
a
standard
way
of
defining
a
a
API
for
requesting
discovery
of
a
safe
secretary
up
slave
replica
to
use
the
kind
of
scene
can
now
all
you
know,
be
standardized
server,
some
API
that
we
are
working
out
on
and
will
probably
also
work
with
the
you
know
anything
in
the
future.
We
all
know
thank.