►
Description
Kubernetes Data Protection Workgroup Bi-Weekly Meeting - 12 January 2022
Meeting Notes/Agenda: -
Find out more about the K8s DP WG here:
https://github.com/kubernetes/community/tree/master/wg-data-protection
Moderator: Xiangqian Yu (Google)
B
All
right
today
is
wednesday
january
12,
20
2022..
This
is
the
kubernetes
data
protection
working
group.
Today
there
are
two
main
ninja
agendas:
ones
that
we're
going
we're
going
to
continue
the
discussion
that
nice
presentation
shine
gave
last
year
last
year,
months
back
after
that,
we
will
discuss
a
couple
of
open
issues
shyam,
and
so
you
are
already
online.
Let
me
know
whenever
you
are
ready,
I
think
she
has
already
made
you
the
presenter.
A
D
D
D
A
D
C
C
So
I
just
wanted
to
reiterate
or
repeat
that
we
would
replicate
either
pvcs
or
volume
groups
and
the
grouping
really
comes
from
volume
groups.
That's
that's
in
flight.
They
kept
the
proposal
is
in
flight
and
we
don't.
I
don't
see
us
doing
replication
groups
as
such
as
something
separate
from
volume
groups,
but
does
that
conflict
in
any
way?
Based
on
the
discussions
last
time
we
met.
A
C
Instead
of
only
considering
pvcs,
we
should
probably
you
know,
out
of
the
gate,
have
something
for
volume
groups
and-
and
I
was
just
trying
to
repeat
the
notes
from
the
last
time
saying.
If
you
have
volume
groups,
we
expect
volume
groups,
grouping
of
volumes
to
come
from
the
volume
group
enhancement
proposal
and
and
and
not.
A
Right,
that's
another
proposal.
I
think
you
probably,
I
think
we
still
need.
We
need
to
have
individual
volume
replication,
so
I
think
we
will
need
both
anyway.
So
can
you
just
continue
with
whatever
you
discussed
in
the
last
meeting,
so
the
voting
group
replication?
That's
the
there's,
a
tab
that
I'm
working
on,
but
that
replication
part
is
kind
of
like
the
second
part
of
that
whole
thing
right,
because
we're
trying
to
address
the
group
snapshot
first,
exactly
correct,
so
yeah.
Why
don't
we
just
go
through
this
one?
First.
C
Right
so
we
talked
about
the
various
the
common
goals
in
terms
of
storage
in
terms
of
pairing
storage
systems
for
mirroring
and
talked
about
roles
across
these
storage
systems,
for
individual
volumes
being
primary
and
secondary
and
and
replication
of
data
happening
from
the
primary
to
the
secondary,
or
at
least
the
primary
is
writable
and
the
secondary
is
the
copy
of
data
snapshot
based
copy
of
data
or
other.
However,
the
storage
system
decides
to
do
it.
C
C
C
In
other
words,
if
we
request
for
a
volume
or
an
image
to
be
mirrored
replicated,
the
setup
can
be
done
from
the
east
storage
cluster
onto
the
best
storage
cluster.
It
doesn't
have
to
be
done
via
cube
on
west,
and
then
you
know,
and
then
you
come
back
on
the
east
cluster
and
redo
the
storage
pairing
as
such.
C
The
final
assumption,
which
is
something
we
we
have,
but
we
don't
want
to
really
take
it
forward,
is
that
it
assumes
that
the
volumes
are
accessible
via
the
same
csi
volume
handle
across
these
two
peer
storage
systems.
So
we're
basically
talking
about
what
we've
done
to
set
the
stage
for
what
can
happen
down
the
line.
C
So
what
we,
what
we
added
here,
is
a
csi
sidecar
to
do
to
to
watch
and
reconcile
volume,
replication,
crs
cluster
resources,
volume,
replication
cluster
resources,
pretty
much
point
to
a
pvc
and
hence
via
the
sidecar,
talk
to
extended
grpcs
on
the
storage
provider
to
enable
disable
volume
replication
for
that
particular
volume.
The,
as
I
said,
the
storage
is
it's
assumed
that
the
storage
is
already
set
up
or
appeared
for
replicating
images.
So
that's
out
of
scope
of
what
volume
replication
does
the
very
simply.
C
That
are
vendor-specific
and
so
on.
The
volume
replication
itself
points
to
a
replication
class
and
then
points
to
the
data
source.
In
this
case
pvc
and
it's
a
data
source
because
later
on,
when
volume
groups
come
in
can
point
to
a
volume
group
and
say
I
need
volume
replication
for
the
entire
volume
group
and
not
per
pvc
as
such,
and
it
has
a
replication
state
which,
which
is
flips
between
primary
and
secondary,
as
the
need
case
may
be.
G
C
The
second
second
observation
or
need
there
was
the
storage
systems
are
already
appeared.
So
any
the
question
is
usually
the
the
data
that
the
data
planes
appeared
and
the
control
plane
is
not
but
going
to
csi.
If,
if
the
control
plane,
information
of
the
participating
clusters
is
available
with
the
csi
plug-in
it
can,
it
can
proxy.
Instead
of
the
user
to
create
the
required
volumes
on
the
secondary
end,
they
have
no
representation
in
kubernetes.
C
So
when,
when
a
primary
volume
is
garbage
collected
on
on
the
primary
kubernetes
instance
again,
the
csr
plugin
can
actually
proxy
for
the
control
plane
of
the
secondary
and
garbage
collected
on.
The
secondary
makes
it
much
easier
to
deal
with
makes
simpler
for
the
user
to
deal
with
pvcs
and
and
pods
using
those
pvcs
on
a
cluster
rather
than
you
know,
actually
having
to
handle
these
across
clusters.
G
C
A
C
That
is
correct.
We
would
actually
want
to
remove
that
requirement.
We
can
so
what
we
instead
want
to
do
is
so
there
are
two
problems
that
is
problem
a
as
a
matter
of
fact.
Even
for
ceph.
We
had
to
do
some
level
of
handling
mapping
across
clusters
to
ensure
that
we
mapped
the
mirrored
copy
on
the
on
the
remote
class
or
the
secondary
cluster,
but
barring
that
it
also
means
that
a
claustroscope
resource
like
the
pv,
has
to
be
backed
up
and
restored
onto
the
secondary
cluster,
which
a
user
cannot
do
by
themselves.
C
So
it's
not
controlled
by
the
user
any
longer
the
cluster
admin
or
you
know,
has
to
step
in
to
backup
and
restore
the
pd.
So
that's
also
not
good,
so
we
don't
want
to
go
down
that
path,
but
that's
where
we
started.
So
that's
why
it's
represented
as
such.
What
we
instead
want
to
do
is
when
the
volume
replication
is
initially
created
on
the
east
cluster,
there
has
to
be
some
cookie
information
that
goes
from
east
to
west,
so
the
volume
replication
is
created
on
the
east
cluster
it
and
it
establishes
the
volume
for
replication.
C
It
spits
out
just
like
the
csi
volume
handle,
let's
just
call
it
a
csi
replication
handle
splits
out
the
csr
replication
handle,
which
is
what
we
use
to
create
the
volume
replication
resource
on
the
remote
end
and
create
a
pvc
from
volume
replication
as
the
data
source,
in
which
case
the
user
can
take
the
replication
handle
and
reuse
it.
On
the
other
cluster,
three
deploying
the
pvc
and
the
workload.
G
G
The
current
design
involves
like
some
administrator
backing
up
the
pv,
and
you
don't
like
that,
because
you
would
rather
have
a
self-service
kind
of
a
model
and
doesn't
that
get
you
back
to
the
point
where
the
user
should
be
creating
a
pvc
on
the
secondary.
So
it's
self-service
and
not
administrator,
controlled.
A
G
Okay,
okay,
so
so
so
they
can
create
their
own
pvc
and
point
it
at
some
object
which
points
back
to
the
original
cluster
implicitly
and
then
and
out,
and
that
pvc
will
pop
up
with
the
the
correct
data
from
from
the
mirror
relationship
right.
Okay.
But
but
you
wait
until
it's
time
until
after
the
disaster
has
happened,
and
you
want
to
do
the
failover
and
then
you
go
and
create
that
pvc
pointed
at
the
relationship
and
set
it
all
up.
C
Right
so
the
next
slide
is
just
about
ordering
in
the
pv
based
model
in
the
in
the
wall.
Rep
cookie
based
model
you'll
create
the
volume
replication
resource
first,
but
as
long
as
your
pvc
destination
is
updated
to
user
data,
source
volume,
replication
it'll
either
fail
to
get
provisioned
on
ease
on
west
till
the
volume
replication
resource
is
actually
created.
C
C
We
have
the
constraint,
because
if
you
create
the
pvc,
we,
if
you
don't
create
the
pv
with
the
required
claim
ref
for
the
pvc,
the
pvc
will
just
go
through
dynamic
provisioning
and
not
get
the
required
data,
because
the
storage
system
wouldn't
know
where
to
get
the
data
from
so
in
the
pv
based
model,
the
pv
has
to
be
restored
first,
which
is
another
sort
of
not
a
good
thing
from
a
from
a
design
perspective
to
actually
depend
on
pvs
and
csr.
Volume
handles
to
reattach
storage
on
the
west
cluster.
C
C
C
Moving
on
the
next
slide,
which
is
about
failover
again
part
of
it,
is
a
repeat
but
we'll
just
go
through
this
quickly
step.
One
is
east
becomes
unavailable,
which
is
a
disaster,
and
so
now
we
need
to
move
it
to
west.
So
the
first
thing:
here's
where
the
ordering
is
important.
We
first
need
to
restore
the
pv
cluster
data
onto
the
best
cluster
so
that
the
claim
refs
point
back
to
the
pvcs
that
were
created
step.
Three.
C
C
But
because
this
is
a
failover,
it
would
have
gotten
into
a
split
brain
issue
as
in
it
would
have
received.
It
may
have
received
some
ios,
which
are
out
of
sync
from
western,
so
well.
Storage
decides
whether
it
needs
to
reset
the
volume
from
a
last
known,
good
snapshot
or
just
continue
resyncing
from
where
the
current
primary
is.
C
C
This
is
important
so
that
we
can
ensure
the
pvc
is
not
in
use
before
we
flip
the
volume
replica
volume,
replication
resources
also
change
to
secondary,
but
before
we
act
on
that
and
tell
storage
to
change
it
to
secondary,
we
need
to
be
sure
that
the
the
pods
are
not
in
there
are
no
parts
using
the
pvcs
and
that
the
pvc
is
deleted.
So
there
cannot
be
an
out
of
order,
part
creation
that
will
use
the
pvc
as
we
are
marking
it.
C
C
G
G
I
mean
I,
ideally
it
has
the
the
latest
copy
right,
but
the
reason
I
bring
it
up
is
because,
in
my
experience
like
people
that
are
serious
about
disaster
protection
exercise,
their
disaster
recovery
plans
regularly,
even
when
there's
not
disasters,
because
you
never
want
to
be
in
a
situation
where
you
try
it
after
the
disaster
happens
and
find
out,
it
doesn't
work
right.
So
so
it's
important
as
far
as
I'm
concerned
that
there's
a
way
to
initiate
a
failover.
G
Just
when
there's
no
disaster
to
exercise
the
work
so
make
sure
that
it's
all
you
know
all
the
machine
is
working
basically,
and
so
so,
in
the
version
of
this
that
we've
implemented,
like
the
pvcs
just
hang
around,
we
don't
delete
them
and
create
them.
So
I'm
trying
to
understand
what
the
advantage
of
of
doing
it
this
way
is
where
you're,
creating
and
deleting
pvcs
every
time
you
want
to
fail
over
and
fail
back.
C
G
E
E
Okay,
is
that
realistic,
not
on
their
own
scenarios,
because
imagine
if
you
have
a
network
outage
or
something
user
may
not
even
reach
the
cluster
or
you
know,
if
you
I
mean
previously,
you
said
you
don't
want
the
cost,
the
users
to
manage
pvcs
on
the
west
cluster,
because
there
was
a
too
much
of
a
manual
process
now,
but
they
have
to
deal
with
the
apps
on
the
east
cluster.
Now.
C
C
In
a
regular
case,
I
mean
in
a
normal
scenario.
What
will
happen
is
if
the
app
continues
to
run
on
east.
There
are
a
few
things
that
will
happen
right.
I
mean
okay,
the
volume
still
noted
as
primary
okay,
so
technically,
the
storage
system
would
actually
allow
ios
to
the
to
the
storage
right
to
storage
right,
but
who
is
actually
using
the
app
I
mean,
where
are
other
applications
that
are
using
the
capabilities
of
the
application.
E
I
guess
the
issue
is
this:
you
know
like
once
I
guess
you
can
protect
the
new
primary
on
the
cluster
on
the
west
cluster
by
breaking
the
penal
relationship
right,
but
once
the
east
cluster
comes
back
up,
obviously
io
would
still
go
to
the
pvs
there.
This
wouldn't
affect
the
new
primary,
but
you
know
it
can
affect
the
old
primary
and
I
guess
I
don't
know
based
on
maybe
the
type
of
storage
backend
you
have,
that
can
have
implications
on
racing.
C
Yeah
the
load
balancer
should
not.
I
mean
it
should
be
managed
to
not
send
traffic
east
we
pre-fail
over
the
west.
We
we,
I
think,
10
12,
slides
ahead.
We
said
that
we're
not
dealing
with
a
load
balancer
or
traffic
redirection
in
in
this
set
of
slides,
because
it's
sort
of
external
to
storage
as
such,
but
yes,
the
load
balancer,
will
have
to
divert
traffic
to
west
and
not
discontinue
using
east.
C
A
C
Right,
so
yes,
in
a
dear
scenario,
east
is
not
reachable.
So
if
we
go
back,
one
slide
right.
So
this
is
the
dr
scenario
value.
So
we
we
even
the
the
line
item
five
where
east
is
available,
that
that's
when
we
really
can
start
garbage
collecting
resources
and
healing
easter,
secondary
and
so
on,
at
step
four,
you
actually
have
the
application
recovered
on
the
west
cluster.
C
C
So,
disaster
recovery
testing
we're
not
necessarily
switching
off
east
cluster
and
moving
the
workloads
and
as
failing
over
the
workloads
to
the
west
cluster.
If
you
are,
if
you
are
testing
it
from
an
ability
for
the
application
to
run
and
use
the
data,
then
the
relocate
can
be
performed
as
the
disaster
recovery.
C
As
the
as
the
dr
testing
during
the
dr
testing
phase,
because
you
don't
want
to
lose
any
data
during
vr
testing
phase,
you
need
to
ensure
that
there
are
no
further
ios
on
east
and
that
the
pods
and
pdc's
are
worn
down
before
you
actually
move
the
parts
and
pvcs
to
the
west
cluster
for
testing.
So
it's
a
real,
real.
C
G
Oh
no,
it
shouldn't
oh
yeah.
That
seems
pretty
clear.
You
know
dr
testing
can
involve
downtime
because
you're
you
have
to
take
things
down,
do
the
final
copy
of
the
data
so
that
the
secondary
is
fully
in
sync
with
the
primary
and
then
bring
up
your
app,
which
can
be
shrunk
to
as
few
as
you
know,
a
couple
of
seconds
of
downtime
if
you're
fast,
but
it's
unavoidable,
some
amount
of
downtime.
G
I
I
like
I
like
the
way
you're
thinking
about
you
know
relocating.
Instead
of
failing
back
and
being
able
to
move
back
and
forth
pretty
freely,
I
I'm
still
scratching
my
head
about
the
pvc
creation
and
deletion
and
the
way
that
the
the
the
selection
of
the
location
of
the
secondary.
So
when
it
sounds
like
in
this
design,
when
I,
when
I'm
on
the
east
cluster-
and
I
create
my
pvc
and
it's
of
a
type,
that's
supposed
to
get
replicated
automatically
a
secondary,
will
also
get
created.
G
I
mean
in
all
of
those
scenarios
like
like
in
in
just
a
one
or
just
a
one-way.
You
know
peer-to-peer
replication.
You
could
have
multiple
options
for
places
to
replicate
to,
even
within
the
same
site
right
you
could
have
multiple
devices
or
multiple
pools
of
disks,
some
of
which
are
fast,
some
of
which
are
slow.
You
know
all
of
those
all
the
storage
class
type
decisions
where
you
know
you
would
like
to
give
people
control
over
where
their
data
ends
up
for
performance,
characteristics,
reliability,
characteristics,
etc.
G
You
you
want
to
make
those
decisions
on
both
the
primary
side
and
the
secondary
side
right.
So
so
I'm
just
I'm
trying
to
because
this
that
it's
implicit
here
and
the
pvc
is
created
after
the
fact.
So
by
the
time
the
pvc
is
created
on
the
secondary,
the
data
already
is
where
it
is,
there's
no
scheduling
decision
to
be
made
for
the
for
that
volume.
It's
just
give
me
the
volume
where
it
is
now,
so
the
scheduling
decision
was
made
all
the
way
up
front
when
you
created
the
original
thing.
C
Correct
right
so
so
this
is
where
I
see
the
replication
class
playing
in
where
I'm
gonna
have
to
extrapolate
a
scenario
here
from
what
you're
saying
so
you
could
technically
replicate
from
east
to
west,
where
the
volume
on
east
is
on
fast
pool
and
volume
on
west
is
on
either
slow
or
fast
pool.
You
know
that.
C
Choice,
so
there
are
so
I
I
would
look
at
it.
I
would
off
hand
look
at
it
as
volume
replication
class
parameter.
So
so,
when
you
establish
it,
it
says
it.
It
kind
of
says:
replicate
this
to
slow.
On
the
other
side
volume
replication
class,
a
and
volume
replication
class
b
is
replicated
too
fast.
On
the
other
side,.
G
Okay,
yeah,
I
mean
so.
This
is
a
that
that
could
work,
that
this
is
another
situation
where
we,
when
we
designed
something
similar,
we
opted
to
just
create
pvcs
on
both
sides
so
that
the
ordinary
storage
class
scheduling,
other
algorithms
could
run
on
both
sides
to
pick
the
locations
for
the
two
volumes
and
then
once
the
two
volumes
existed,
then
we
establish
the
relationship,
but
but
if
you
want
to
encode
it
all
on
the
primary
side
and
then
do
it
implicitly,
it
could
also
work.
E
G
E
I
think
the
main
difference
is
that,
let's
say
in
the
trident
scenario
application
owner
decides.
You
know
where
to
pitch.
You
know
cluster
to
fade
over
to
in
this
scenario.
It's
the
same
thing.
Instead
of
just
deploying
the
application
you
first
deploy
the
vr
and
based
on
where
the
vr
gets
deployed.
That's
how
you
know
the
application
gets
deployed
there.
So
it's
you
know
one
extra
step.
You
have
to
deploy
the
vr
first
and
then,
once
the
vr
is
set
up
the
infrastructure
for
the
application
that
application
can
get
scheduled.
E
C
C
G
C
G
C
G
At
some
point,
yes,
so
so
the
idea
is
the
pvc
can
be
created
with
no
replication
relationship
and
it
can
just
be
an
ordinary
volume
and
then,
after
the
volume
is
created,
you
can
decide.
Oh,
I
want
to
mirror
that
one
and
you
create
the
necessary
objects
and
the
you
create
the
secondary
pvc
and
the
necessary
objects
to
tell
trident
to
do
to
do
its
thing
and
then
now
you
have
a
pv,
the
pvc.
G
G
So
you
have
to
basically
wait
now,
if
you,
if
your
app
is
a
stateful
set,
it
would
get
trickier
because
because
in
order
for
the
in
order
for
the
the
secondary
to
have
a
relationship,
you
do
have
to
create
the
a
different
object
before
you
create
the
secondary
pvc
so
that
it
knows
not
to
create
an
empty
pvc
but
to
create
a
mirror.
Pvc
on
the
secondary
and
a
state,
a
stateful
set
it
wouldn't.
C
Okay
and
now
I'll
segue
into
volume
groups-
okay,
the
way
I
see
volume
groups
playing
here
is
that,
instead
of
the
volume
replication
data
source
being
a
pvc,
if
the
volume
replication
data
source
is
a
volume
group,
the
one
that
gene
is
working
on
the
volume
group
specification
that
for
snapshots
that's
being
worked
on
right
and
right.
So
so
now
what
happens
is
newer
volumes?
Pvcs
could
be
created
added
to
the
volume
group
removed
from
the
volume
group.
C
What
not,
but
the
volume
replication
acts
on
the
volume
group,
just
like
a
snapshot,
would
act
on
the
volume
group
and
snapshot
all
pvcs.
So
it
would
again
from
again
assuming
that
the
control
plane
communication
happens
from
the
storage
yeast
storage
west,
create
appropriate,
mirror
replication
copies
on
west,
based
on
replication
class
parameters.
C
C
So
when
the
volume
replication
is
restored
on
are
created
on
west,
it
will
have
to
promote
or
demote
the
volumes
based
on
primary
or
secondary
roles
that
we
assigned
to
volume
replication
resource
on
west,
pointing
to
the
volume
group
which
will
come
in
when
the
app
is
deployed.
Just
like
the
pvc
comes
in
when
the
app
is
deployed.
A
C
A
Well
so
this
is
basically
like
in
the
the
current
the
cap
for
the
one
group
where
we
do
have
apis
for
you
to
add
a
volume
to
your
group.
So
let's
say
user.
Does
that
or
are
you
saying
once
they
start
this
replication?
They
can't
do
that
anymore.
I
guess
I'm
trying
to
figure
out
like
what's
the,
what
are
the
steps.
C
G
A
H
E
E
C
As
a
label,
so
we
we
did
not
implement
the
cookie,
so
we
are
transferring
pvs.
We
are
transferring
pvs
across
clusters,
so
with
the
assumption
that
the
csi
volume
handle
is
the
is
immutable
across
these
two
clusters,
and
so
it
will
be
able
to
map
back
to
the
original
volume.
So
that's
what
we're
doing
currently,
but
that's,
not
user
serviceable,
simply
because
it's
a
cluster
scoped
resource
and
definitely
not
true
for
all
storage
providers,
that
across
east
west
or
across
different
storage
clusters,
the
volume
handle
is
readily
reusable.
A
A
A
A
H
A
Right,
I
think,
probably
because
of
holidays,
I
haven't
I've,
not
seen
any
update
on
grant.
I
just
I
just
pinned
him.
I
think
I
have
not
got
a
response
yet,
but
I
I
was,
I
will
see
if
he
can.
He
said
he
will
get
back
to
this.
You
know
you
see
here.
I
just
said
he's
hoping
to
come
looking
into
this
again,
so
I
assume
now
after
the
holiday.
He
should
have
time
to
look
into
this.
So
let
me
I'll
ping
him
again
make
sure
that
sure
thanks
so.
A
A
F
Since
we
have
a
minute
left,
I
just
wanted
to
point
out
to
ben
about
the
model
that
sean
presented
right
which,
where
we
don't
create
the
pvs
on
the
other
side,
while
this
talk
was
primarily
for
async
replication,
but
when
you
apply
something
similar
for
you
know
for
sync
replication,
where
the
storage
is
stretched,
then
you
know
there
are
two
kubernetes
clusters
with
a
stretched:
storage.
Back-End
then
the
model
is
you
know
you
don't
want
to
create
pvs
on
the
other
side.
For
that
case,
then
it
will
pvc.