►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Protecting Stateful Workloads With CSI Snapshot - Xing Yang, Huawei & Jing Xu, Google
How do we make sure that data in a Kubernetes cluster is safe and reliable so that one can rest at ease at night? Snapshot is a point-in-time copy of a volume. It is the basic building block of backup and restore, data migration, replication, and many other advanced applications built on top of it. Snapshot is a sought-after feature in Kubernetes and it is finally here to support CSI! In this session, Jing and Xing will discuss about the snapshot support in Container Storage Interface (CSI), the standardized API design to represent snapshot objects in Kubernetes, and the controller to handle snapshot provisioning and binding. This talk will also cover restoring of volumes from snapshots, and how this can be applied to backup and restore, data migration, and many more advanced features to ensure data availability and integrity for stateful workloads in the Kubernetes environment.
To learn more click here: https://sched.co/FuKd
A
B
Everyone,
my
name,
is
Ching
I
work
for
kubernetes
storage,
lifecycle
team
at
Google
and
she
and
I
are
called
it.
The
CSS
snapshot
project
here
we
are
presenting
our
project
that
I
saw
Harwood
was
using
was
a
cooker
thinking,
Jana
it
agitation
of
the
CSS
snapshot
project
when
you
immunize
your
attention,
knowing
that
our
unit
area
20
high
thing
to
use
crowd
in
some
way.
B
B
A
B
B
I
think
we
just
covered
a
motivation
of
this
project.
Now
this
is
the
agenda
for
the
rest
of
the
talk.
First,
we're
going
to
have
some
backgrounds
on
Cuban
artists,
precision,
volumes
and
stylish
packing
and
then
introduce
our
CSI
contains
direct
interface,
followed
by
a
detailed
explanation
of
RCS,
a
snapshot,
design,
I
last
work
ever
to
date
or
demos.
B
So
enquiries
nominee
will
spread
into
towards
users
ends
sistema
beans
for
user
I
think
they
need
to
use
storage,
so
they
make
some
requests,
and
this
will
help
to
allocate
a
storage
and
the
inquiry
yes
is
about.
Ap
is
so.
There
are
two
ideas
to
represent
for
users
and
for
enemies.
For
you
decide
persistent
bottom
claim:
PVC
is
used
to
specify
what
kind
of
storage
and
what
type
and
what's
the
size,
the
user,
T's
right
and
then
Amin
can
based
on
those
requirements
to
allocate
storage.
B
Now,
when
user
create
a
PVC
system
of
AOPA,
some
storage
class
supervision,
the
volume
automatically
so
the
mean
does
not
need
to
do
anything
and
that
this
diagram
shows
the
volume
lifecycle
like
it
includes
created
each
attached
bottom
to
note
mount
amount,
and
there
are
several
controllers-
are
used
to
manage
dit
master
site.
We
have
two
controllers:
PV
controller
is
for
watch
like
AP
object,
creation
deletion
and
also
for
binding
the
PVC
and
PV,
and
the
storage
plug-in
will
provide
provisioning
interface.
B
We
also
have
attach
attach
controller
like
to
manage,
attach
want
him
to
know
detach
and
on
every
node.
There
is
a
note
agent,
a
couplet
water
manager
responsible
for
mounting
and
mounting
the
volume.
To
the
note,
oh,
these
kind
of
controller
follow
the
same
pattern.
So
it's
running
a
loop.
It
will
watch
the
creation
division
of
AP
objects.
Catheters
are
state
like
what
user
won't
stay
to
be
and
compare
with
the
current
state
and
make
actions
to
make
current
towards
to
the
desired.
B
A
Yeah
Jing,
actually,
we
are
not
alone.
Other
seals
also
have
similar
challenges.
In
addition,
every
CEO
wants
to
support
as
many
storage
systems
as
possible,
but
without
CSI
each
CEO
has
his
own
way
to
write
a
plug-in.
So
that
means
the
search
vendor
has
to
write
different
plug-ins
for
different
CEOs.
A
A
A
Here
we
have
three
different
types
of
deployments:
they
are
all
headless
plugins.
The
top
one
has
the
controller,
plugging
and
note
plug-in
separate,
but
they
are
all
running
under
worker
node
and
we
also
have
controller
and
node
plug-in
together.
That
is
also
running
on
the
worker
node
and
the
one
on
the
right
hand,
side.
That
only
has
the
note
plug-in
it
does
not
have
the
controller
plug-in.
So
an
example
of
that
would
be
the
NFS
plug-in.
It
only
implemented
node
functions,
but
not
the
controller
side
of
functions.
A
A
There
are
three
snapshot:
are
PCs
Chris
nap,
shot
the
lease
nap
shot
and
the
least
snapshots?
If
a
plug-in
supports
create
delete
snapshot,
controller
capabilities,
then
the
plug-in
must
implement
great
snapshot,
the
list
nap
shot
and
quit
volume
from
snapshot,
and
if
the
plug-in
supports,
least
snapshots
capability,
then
it
must
implement
lease
the
snapshots
function.
A
So
if
that
is
supported,
then
the
plug-in
must
implement
crit
volume
for
another
volume
in
the
Kris
snapshot
request
we
pass
in
the
source,
volume
ID
and
the
name
name
must
be
unique,
that
is
to
support
idempotency
and
it
has
secrets
and
parameters.
Parameters
are
opaque
to
the
CIO's.
Only
the
plug-in
knows
how
to
pass
them
and
in
a
Krishna
shell
response,
we
have
the
size,
so
that
tells
the
CEO.
A
When
you
want
to
create
a
wall
in
front
snap
shot
your
size
must
be
equal
to
or
greater
than
the
size,
and
we
have
a
snapshot
ID.
This
snapshot
ID
is
generated
by
the
plugin.
This
is
different
from
the
name
in
the
create
a
snapshot
request
that
one
is
generated
by
the
CEO.
So
when
you
call
create
snapshot,
you
give
the
name,
and
after
the
snapshot
is
created,
you
get
a
snapshot
ID
from
plug-in.
Then
you
use
the
snapshot
ID
to
call
these
snapshots
and
a
daily
snapshot.
A
We
also
have
a
creation,
timestamp
and
also
a
boolean
ready
to
use,
so
Chris
snapshot
can
have
two
steps.
Some
stored
systems
support
uploading
snapshot
after
the
snapshots
cut,
so
the
crate
snapshot
call
is
synchronous
and
blocking
until
the
snapshots
cut
that
point
it
returns
and
depending
on
whether
the
plug-in
will
upload
a
snapshot
of
not
if
it
doesn't
need
to
upload
snapshot,
it's
going
to
return
and
instead
ready
to
use
to
true
and
then
you're
done.
A
But
if
you
need
to
upload
snapshot,
then
the
ready
to
use
flag
need
to
set
your
force
and
upload
snapshot
will
continue
in
the
background
until
it
become
true.
So
seal
should
call
great
snapshot
repeatedly
to
find
out
whether
this
ready
to
use
flag
has
become
true,
and
at
that
point,
when
this
ready
to
use
is
true,
you
can
create
a
wall
in
front
a
snapshot.
A
A
So
Jean
talked
about
the
PBC
in
the
user
space
and
the
PV
in
admin
space
earlier
and
also
is
a
storage
class
for
dynamic
provisioning
for
rolling
snapshots.
We
have
similar
concepts.
Volume
snapshot
is
a
request
by
the
by
the
user
for
a
snapshot
and
volumes.
National
content
is
the
snapshot
created
on
the
storage
system
by
the
admin.
There
is
a
PI
D,
rational,
binding
between
the
warring
snapshot
and
wall
in
snapshot
content.
A
So
when
the
snapshot
is
ready,
it
is,
it
is
a
bound
and
we
also
have
a
warning
snatch
our
class
for
dynamic
provisioning
because
they've
all
you
may
have
multiple
snapshots
and
each
snapshot
may
have
a
different
request.
So
we
use
this
different
snapshot
class,
not
reusing
the
story
class
for
the
volume,
all
this
boring
snapshot.
Any
objects
are
CR
DS
that
makes
kubernetes
pluggable
and
extensible,
and
the
series
will
be
installed
automatically
by
the
external
snapshot
or
static
our
container.
A
So
this
is
the
volume
snapshot
ap
object--.
In
kubernetes
we
have
a
spec
and
status.
A
spec
describes
the
desired
characteristics
you
want
to
have
in
the
warning
snapshot
and
the
status
describes.
The
actual
state
of
the
volume
snapshot
in
the
spec
we
have
the
source,
which
is
the
PVC,
and
we
also
have
the
snapshot,
content,
name
and
the
snapshot
class
name
and
in
a
status
we
have
the
creation
times
time.
There
is
the
size
and
ready.
A
We
also
have
a
whole
new
snapshot.
Content
API
object.
So
in
the
contents
back
we
have
a
reference
to
the
warning,
snapshot,
object
and
also
we
have
a
volume
snapshot
source
which
is
the
CSI
source
and
Bowden
snapshot
feature
only
supports
CSI
drivers.
So
when
we
do
the
restore,
we
need
to
specify
this
a
datasource
object.
So
we
just
added
this
new
API
object
data
source
that
can
be
used
for
snapshot
in
and
also
restore
volume
from
snapshot
and
in
the
future.
We
can
also
support
PVC
type
for
cloning.
B
A
Yeah
sure
I'm
going
to
give
some
examples.
So
here
we
have
an
example
for
volume
snapshot
class:
you
can
create
a
default
class.
Specifying
is
T
for
the
class
to
true,
and
then
you
need
to
set
the
snapshot
to
your
driver
name
and
also
under
parameters.
You
can
have
the
opaque
parameters
for
your
driver.
There
are
secret,
there
are
reserved
keys
for
secret,
a
secret
name
and
secret
name
says
if
you
plug
in
needs
to
use
a
secret
to
create
a
delete
snapshot.
A
You
can
use
that
and
we
also
have
a
Yama
file
for
their
volume
snapshot
and
you
specify
the
snapshot,
class
name
and
the
source,
and
also,
if
you
are
not
specifying
the
snapshot
class
name,
the
default
class
will
be
used
so
to
create
a
volume
snapshot.
The
PBC
has
to
be
inbound
state,
otherwise
it
will
fail.
A
B
It
looks
like
snapshot,
that's
quite
easy
to
use,
let's
look
at
how
to
manage
a
CSS
snapshot
in
communities
and
deployment.
I
just
want
to
check
like
came
here
as
clearly.
It's
ok,
the
background.
Ok,
so
we
already
see
this
diagram
right
and
we
have
volume
lifecycle
and
we
have
controllers
now
we
see
si
since
es
is
out
of
tree
and
we
need
some
external
components
to
communicate
with
auto
presets
as
driver.
So
we
have
external,
provisional
external
attacher
and
also
the
CS
park
in
rot
will
communicate
with
the
node
agent
q8
volume
manager.
B
Now
for
snapshot
right.
We
have
snapshot
lifecycle
management
and
it
is
controlled
by
external
snapshot
ur,
and
there
are
a
list
of
things
that
charters
need
to
do
similar
to
a
PV
controller
in
each
watched
creation,
deletion,
events
of
snapshots
and
take
actions
and
also
point
snapshot
and
snapshot
content
objects.
It
also
need
to
make
sure
to
recover
from
the
previous
state
when
it
fails
when
the
controller
is
restart
and
also
we
are
adding
features
like
to
have
some
protection,
for
example,
to
prevent
deleting
the
snapshot.
Well,
you
have.
B
A
volume
is
being
restored
from
snapshot
or
it
prevented
eating
the
volume,
while
a
snapshot
is
taken
from
the
volume
and
also
it
supports
the
lesion
policy.
We
have
two
policies,
delete
and
retain
retain
means
when
your
30
leaked
avoiding
snapshot
object,
but
underlying
snapshot
will
be
kept.
It
is
still
there
and
for
deployment.
So
on.
Every
note
will
have
a
team
inside
to
run
CSS
driver
as
container
and
also
the
driver
restroom,
and
they
communicate
with
the
cumulative
volume
managers
through
the
sockets
and
for
controller.
B
B
The
controller
can
run
any
note
or
it
can
run
a
master
like
she
mentioned.
The
controller
will
watch
the
API
objects
through
API
server
and
now
CSS
snapshot
is
available
as
alpha
feature,
since
our
community
is
version,
one
point
12
and
we
have
a
handful
of
CSI
drivers
ready,
suppose
nap
shot
and
we
hope
the
list
will
cut
him
longer
and
if
anyone
interest
in
implementing
snapshot
suppose
I
shot
your
CSF
plugin
right,
it's
very
single.
B
You
only
need
to
implement
three
controller
costs,
our
pcs,
a
great
snapshot
release
after
all,
a
snapshot
so
that
your
plugging
have
those
capabilities
and
next
step
we're
going
to
have
a
long
list
of
features
we
want
to
implement
on
top
of
the
current
framework,
for
example,
topology
support,
so
that
you
can
specify
where
you
want.
You,
like
your
snapshots
located
at
and
currently
the
user
has
to
manually,
prepared
application
like
to
freeze
application
before
taking
snapshots
to
have
the
consistency
snapshot.
B
B
So
kubernetes
dock
has
this
tutorial
right.
It's
very
easy
to
install
WordPress.
It
includes
all
the
llamo
file
you
can
just
copy
it,
and
here
in
my
class,
are
already
settled
out.
So
I
have
stable
start
running
the
my
skew
and
I
have
a
deployment
running
in
a
while
or
WordPress,
and
the
service
is
running.
B
So
it's
really
exciting
rights
to
be
to
have
a
cube.
Come
here,
so
I'm
very
sad
to
publish
this
post
now,
let's
check
it.
Ok,
it's
here
because
my
post
is
so
important.
I
want
to
make
sure
I
don't
lose
it
I
take
a
snapshot.
This
is
my
snapshot.
Yama
file
is
very
simple.
It's
just
include
the
PVC
and
QVC
name
right,
which
is
the
word
press,
Q,
PVC
name.
B
Now
you
just
use
one
line
or
one
command
line
to
crease
that
shot
and
you
can
check
right
by
described,
snapshot,
API
object
and
there
are
some
information
you
can
get
from
here.
Sorry,
the
typo
humorous
always
happen
and
you
can
see
right
in
the
status
the
creation
time
is
set.
That
means
the
snapshot
is
cut,
but
reddy's
force
means
is
still
in
uploading.
It's
not
ready
to
be
used
and
then
I'm
going
to
do
something
wrong.
B
So
I'll
leave
some
bad
comments
because
I
plan
to
do
something
like
not
very
good,
so
here
we
can
check
the
bottom
snapshot.
The
ready
state
is
becomes
true.
It
means
uploading,
finish
and
I
plan
to
delete
the
volume.
The
mask
your
bottom
and
that
means
I'm
going
to
lose
my
great
post
and
after
I
deleted
I
can
check
the
status
of
the
PVC
and
you
will
see
the
PVC
status
will
becomes
terminating,
so
it
does
not
delete
it
completely
because,
actually
a
part,
the
mask
your
parts,
you'll
use
it.
B
So
there
is
a
protection,
so
I
then
go
ahead
to
delete
my
skill
stay
for
said.
So
in
this
way
the
part
safe
assets
will
be
gone
and
also
the
PVC
will
be
eventually
deleted
and
now
it's
time
for
me
to
recover
right.
So
I
recreate
my
mask.
You
see
for
set.
So
here
the
part
mask
your
party's
already
gone
so
I
just
use
queue.
Category
of
my
skills
before
set
the
civil
side
is
create.
B
The
arrow
is
just
showing
the
service
already
exist,
because
I
didn't
delete
the
service,
and
now
the
mask
your
part
is
trying
to
start
in
order
to
make
this
WordPress
work.
I
have
to
restart
WordPress,
so
here
I
just
delete
the
part
word
price
and
then,
since
it
is
under
a
deployment,
deployment
controller
will
automatically
restart.
Will
press
part,
let's
see
the
mask
you'll
volume
red
is,
this
is
a
new
autumn
is
already
created
and
bond
and
the
word
price
part
is
restarted.
B
My
skew
is
takes
a
little
bit
longer
time
to
start,
but
that's
ok.
Now,
let's
check
the
new
like
my
steak,
was
that
my
skew
stiff
was
that
in
order
to
restore
I,
do
add
a
few
lines
here.
The
data
source
and
the
low
volume
claim
template
so
Tina
source
means
the
volume
will
be
created
from
this
particular
snapshot.
Your
mother,
who
is
no
longer
empty
volume,.
B
A
Okay,
so
I'm
going
to
have
a
nginx
app
running
with
the
volumes
provisioned
by
open
SD,
SCSI
plugin,
with
the
LVN
back-end
and
I'm,
going
to
take
a
snapshot
and
then
upload
a
snapshot
from
the
local
PM
block.
Storage
to
the
s3
object
store
in
it
up.
Yes,
and
after
that,
I
will
restore
volume
from
the
snapshot.
A
A
A
A
A
A
B
Okay,
that's
our
two
small
demos
and
so
still
like
us,
starting
phase
so
I'm
sure
his
new
features.
We
are
going
to
support
more
companies
use
cases.
There
are
some
useful
links
here
like
we
have
a
blog.
We
we
list
our
external
snapshot
to
repo.
We
have
bakery
meeting,
you
can
check
our
talk
or
time
if
you
want
to
join
and
some
documentation-
and
this
is
definitely
a
collaborative
work.
So
if
you
are
interested,
please
join
us
and
those
are
the
contact
information
now
any
I
think
that's
for
our
talk.
We
can
have
questions.
B
Decision
to
auto
trade
in
a
CI
side
switch
out
of
trade
in
a
moment,
Sina
featured
always
a
CIT
channel
show
that
over
time,
mac
whirter
CSI,
so
you
to
hold
the
trading
Jupiter
in
trailer
na
modules.
Icsi
snapshot
amateurs
out
of
tree,
so
they
thought
one
per
controller
or
KPI
CRT
in
a
queen
is
in
that
of
teachers,
usually
exam
like
accessibility.
C
Hi
everyone
you're
here
now
walk
your
hand,
technique.
It's
tarnished
Orion
excellent.
Is
here
tinkly.
You
may
well
make
it
a
shoe
so
much
so
they
get
international
saying
that
y'all
gonna
shoot
you
and
me
pal,
Yoga,
Nidra
honey.
You
know
what
metalcore
I'm
sure
you
know
what
shuto
generally
I'd
remind
circle:
basically
eternal
ress,
my
rear
turret,
or
maybe
the
people
turquoise.
Yet
saboteur
comes!
Oh
now.
Young
people
enjoy
yourself,
RPG,
Java,
so
yeah
I
meanyeah
a
hundred
on
charging.
So
I
return.
Shinagawa.
You
may
slip
for
the
what
you
know.
C
B
Your
feature
like
you
can
like
how
long
you
want
to
retain
how
long
you
can
you
want
to
keep
your
snapshot
you
can
specify
after
that
period
time
we
can
automatically
delete
your
snapshot,
etc.
So
those
are
the
features
we
are
considering
and
like
policy
based
management
right,
you
can't
define
how
often,
how
frequent
and
how
many
snapshot
you
want
to
keep.
B
A
A
D
A
Yeah,
so
basically
that
just
means
it's,
it's
a
more
ready
right
so
before
that,
if
you
are
saying
it's
a
0.3,
it's
not
ready
when
domain
is
GA,
so
that
has
the
snapshot
feature
it
has
you
know
the
crate
volume
kill
a
volume
attachment
and
the
snapshot
feature
and
also
know
the
cloning.
One
was
just
added:
expand
we're
still
waiting
for
that,
one
that
PR
I
think
it's
getting
close
marketing
expand
watering
feature
might
get
in
we're
still
waiting
for
that.
One.
Okay,.
D
So
you
me
CSI,
one
point
is
also
includes
several
in
several
static.
Are
containers
like
this
bike,
just
a
spike?
Okay,
okay,
okay,
thank
you.
Okay.
Another
question
is
I'm,
not
sure
I
don't
understand,
because
currently
we
use
PDA
controller
to
manages
our
PV
and
the
PVA
say:
object
aperture,
either
similar
my
Sims
I.
To
for
the
slab
short
we
have
no
such
can
such
controllers
slab
shorter,
oh,
is
the
controller
yo-yo
mean,
is
a
entry
in
the
in
submarines
way
we
have
a
slap
turn
out
of
tray.
D
D
E
B
E
When
you
change
I
see
so
donít,
actually
Hoyo
don't
achieve
high,
so
2009
he
homie
out
what
a
Roberta
Roberta
no
corner
now
Roberta
the
corner
and
watch
a
little
Ronnie
Yahoo
sound
to
know
much
about
you
know:
suji
should
even
go
wait.
Wait
should
take
a
nice
man
she's
a
goner.
No,
no
anyway,
you
can.
When
teachers
watch
it
go
quite
away.
You
hope
your
trip
Japan,
but
just
remember
Nazis
or
Kannamma.
You
walk
with
it
on
an
open
manner.
Quite
on
your
journey.
Tijuana
I.
B
Don't
think
we
discussed
that
yet,
but
like
as
a
user
I
the
way
I
hear
your
feedback
and
if
you
have
some
use
cases
we
definitely
will
consider
to
support.
If
is
like
a
very
useful
use
case,
so
you
mean,
like
a
you,
have
several
versions
of
snapshots
and
then
so
right
now
you
can
just
manually.
You
can
do
that.
You
just
pick
okay,
which
version
you
want
to
restore
back
right.
B
E
B
B
I
think
that's
definitely
like
a
good
direction
to
automate
right,
but
we
need
to
build
up
like
all.
The
features
right
now
is
very
basic.
Well,
you
can
see
so
all
those
advanced
features
like
conic
code
see
the
then
he
says
Duchamp
Adonis
watch
on
the
sure,
Oh
mio
roadmap
potential.
She
done
a
CSI
or
a
matter
ABN.
You
know
you
see
a
Sun
et
al
economy
or
one
chance
to
address
number
lukewarm
away
his
hand
out
when
the
meeting
are
ahora.
When
you
shitty,
okay,
II
will
make
a
two-story.
Ok,
listen.
F
You
seems
you
guys
want
to
do
something
in
data
retention
area.
I
mean
this
is
the
data
retention
is
a
is
a
full
stack
right
underlying
devices,
maybe
object,
storage
or
traditional
block
storage
and
the
the
snapshot,
a
layer
or
your
CSI
layer,
but
the
upper
layer
for
the
backup
software
for
the
date
of
the
retention
for
backup.
Our
archive
I
think
is
a
very
specific
domain
for
for
some
today,
traditional
error
like
Veritas
right,
like
EMC,
backup
software,
so
we
kuba
need
he
need
his
avail.
B
So
right
now,
I
think
the
place
at
first
built
the
very
basic
functionalities
and
also
based
on
the
requirements
and
Bo
some
a
little
bit
more
robust
features,
but
I
don't
think
we
can
replace
like
the
whole
stack
or
anything.
So
it
definitely
needs
some
like
it's
not
a
full
replacement
for
like
they
have
backup
or
anything.
But
we
can
try
to
provide
some
like
common
use
case
right
and
so
have
some
features,
and
so
user
can
kind
of
customize
and
also
have
their
own
solutions
for
their
type
data.
G
A
H
B
B
I,
don't
think
it's
a
big
difference
like
some
use
case.
It
may
need
external
volume,
like
external
storage,
some
need
the
local
sorry,
but
the
whole
like
frame
where
you
can
see
it's
not
like
very
different
from
local
or
remote.
So
I
don't
think
it's
weak
right.
Now
we
don't
support
but
doesn't
mean
we
cannot
see
the
future
yeah.
I
B
I
Know
the
2002,
so
you
purchase
your
janitzio
hot
on
Dominica's
home
and
you
meet
a
guy.
He
gets
me
on
the
Champa
what
you
thought,
the
density,
CSIC,
quick,
Howard,
auto
so
see
what
the
ones
in
who
comes
anyway
minutes.
How
do
those
CSI
can
even
cover
Honda
Tony?
She
country
warming
trend
on
the
universal
kiss,
keep
it
is
stupid
and
simple
Sewell
just
mission,
so
does
the
ecosystem
can
then
go
to
autism
baggage
auntie?
Does
the
recovery
requires
the
ancestor.
A
Disaster
recovery,
shindig
easy,
creates
a
new
drivers.
Next,
one
seeking
create
solo,
show
open,
STS,
plugging
eating
creature.
Failover
la
childrenís
me
show
me
a
PV
PV
C
level,
and
usually
you
need
to
make
a
you
go
to
and
then
she's
having
should
appear,
you
see,
I
mean
you
travel
click,
one
Valley,
I'm
sure
to
use
your
cleats
or
dollar.
She
single
tell.
I
Wasn't
easy
disaster
recovery
in
approaches
that
you
hundreds
and
she
told
me
to
shoot
you,
sir
watch
ing
doctor
to
the
Kobayashi,
Maru
and
other
applications
chameleon
and
to
talk
so
what
I
wanna
do
to
saw
it
on
CSI
disaster
recovery,
that
should
you
should
shoot
that
CSI
Tamiya,
a
sewer,
hydrological
kubernetes,
oh
yeah,
babe
or
haunting
La
Russa
tada
Orchestrator.
You
put
me
behind
like
what
do
I
win.
This
jig
went.
B
B
Then
she
says
she
has
an
early
discussion,
so
she
thought
out
to
show
Samia
I
get
some
do
hi
Mayo.
He
goes
England
email
me
can
take
a
snapshot
alpha
feature
to
see
our
country
today
are
a
whole
woman.
Also
the
canal
she's,
a
common
use
case,
police,
recover
number,
10,
Buddha
wrong,
say
since
Ida
harm,
yo,
seh,
even
functions
features
way.
We
want
to
like
an
omission:
siento
no
Malcolm
advanced,
oh
hi,
tomatoes
or
taller.
The
number
Cinco
de.
A
K
J
J
Basically
is
when
you,
your
volume
is
deleted
and
you
want
to
recover
it
from
the
snapshot.
Thus,
not
recovering
may
take
some
time.
So
how
long
this
cubelet
will
keep
city
will
will
wait
for
the
response
from
the
daemon
or
at
what
point
it
will
take
the
decision
like
whether
it
is
have
you
recovered
or
not.
How
do
we
side
that
or
the
hold
is
the
CSI
besides
from
this,
so.
A
Right
now,
if
you
as
you,
can
see
in
our
demos
right
now,
it's
still
a
manual
process
because
we
have
not
make
all
of
the
restore
process
automatically
yet
so
right
now.
Actually
you
have
to
go
there
and
change
your
llamo
file
choose
specify
that
you
are
actually
creative
warning
from
a
snapshot
source
right.
So
that's
a
it's
not
automatic
yet
yeah,
so
you
do
have
to
yeah.
J
A
snapshot
so
what
I
mean
is
like
it
will
be
in
a
creating
state
or
it
is
in
a
bound
state
right
when
we
create
a
volume
when
we
create
a
PVC
so
for
the
snapshot
to
be
recovered,
it
has
to
be
completely
moot
complete
data
from
the
original
snapshot
to
the
volume
that
we
are
creating.
So
until
we
finish
this
step,
it
might
take
suppose
we
have
100
gig
of
data
and
it
will
take
us
out
of
time
right.
So
how
does
the
vendors
or
the
CSI
will
take
care
of
this?
A
K
A
K
A
K
A
K
You
can
organize,
yes,
someone
you're
sure
to
treat
a
man.
Yo
yo,
don't
show
your
window,
so
he
cannot
also
coincided
Shakeology
be
a
13
colonies
should
convert
and
then
turning
ago
organize.
Yes,
someone
may
be
repaid
on
IG
213
senator
own
people,
gosh
uncle
Yabba.
Now
children
13,
she
sure
knew
she
died
around
says
that
driver
to
gender,
32,
nuts
and
true
down
tutorial
forgiving
Julian.
A
Whooping
Herschel
with
your
number
vengeance.
What
you
think,
then,
since
I
she's
a
mean,
can
you
call
me
or
you
can
make
a
support
group
it
type
session?
How
who
enjoys
as
you're
sure
so
your
first
on
CSI
I,
sent
a
co-driver
triage,
should
focus
on
provisioning.
Yahoo
highways
here
can
concatenate
on
advanced
function
named
Duncan
Ohio
Levi
this
year,
project
Jitsu
sure.