►
Description
Kubernetes Data Protection Group Bi-Weekly Meeting - 01 December 2021
Meeting Notes/Agenda: -
Find out more about the K8s DP WG here:
https://github.com/kubernetes/community/tree/master/wg-data-protection
Moderator: Xiangqian Yu (Google)
A
B
D
D
D
A
D
D
E
B
F
F
We
work
on
communities
in
openshift
and
stuff
and
also
with
ces.
We
had
brief
introduction
to
the
disaster
recovery
topic,
using
storage,
assisted
volume
replication
about
a
month
back
now,
maybe
three
weeks
back.
This
is
a
follow-up
to
that
to
try
and
present
what
we've
done
and
take
the
discussion
forward.
So
so,
with
that
the
biggest
disclaimer
is
this
is
just
presented
as
to
serve
as
a
discussion.
F
This
is
not
presented
as
a
solution
that
we
should
go
ahead
with
and
do
and
things
like
that,
so
so
we're
going
to
split
it
into
three
parts:
the
first
parts,
the
problem
which
kind
of
revisits
what
we
already
talked
about
and
what
we
understand
as
commonality
across
storage
systems
in
the
volume
replication
space,
both
synchronous
and
asynchronous,
and
and
present
a
sketch
of
the
solution
that
that
we
are
developing
presented
at
kubecon,
so
that
we
can
then
make
it
make
it
something
cross
storage.
If
which
is
the
desire?
F
The
whole
desire
of
talking
to
the
working
group-
let's
go
ahead,
so
I'll,
run
through
this
a
little
quick
because
most
of
this
is
understood
or
already
discussed,
but
please
do
interrupt
like
I
said
this
is
to
serve
as
a
discussion.
So
do
interrupt
us
if
there
are
questions
or
comments
and
so
on.
So
why
volume
replication?
F
This
still
leaves
out
the
application
resources.
This
discussion-
we
do
have
some
thoughts
on
that,
but
more
later
on
that
it
does
leave
out
the
application
resources,
like
the
part
secrets,
config
maps.
What
not
it
kind
of
leaves
out
that
you
actually
need
an
alternate
humanities
cluster
which
needs
to
be
ready
or
created
so
that
the
workload
can
be
moved
to
an
alternate
topology
segment
from
a
recovery
perspective
so
again
quickly.
How's
data
replicated.
F
What's
what's
storage
based
data
replication,
we
do
have
a
slew
of
tools
with
backup,
but,
like
I
said
yes,
they
can
be
leveraged,
but
the
intention
here
is
leverage
storage,
vendor
supported
volume,
replication
features,
for
example
like
rbd,
mirroring
incest
and
snap
mirror,
and
that
app
and
various
other
such
technologies
that
exist
primarily
because
it
helps
that
there
are
various
reasons.
Why
storage
renders
provide
this?
F
Probably
in
this
group
we
can
add
10
mobile
as
to
why
this
is
being
done,
but
in
general
it
offloads
replication
to
the
storage
system,
delta
transfer
efficiencies
are
inbuilt
into
storage
systems
based
on
change,
blocks
or
snapshots
and
journals.
What
not?
It's
basically
a
feature
of
the
storage
system,
how
they
optimize
their
transfers
and
also
it's
historically
readily
available
feature
by
some
storage
vendors,
so
how
to
make
it
cloud
nato
how
to
leverage
that
in
cube
and
provide
it
to
users.
F
F
F
There
are
two
or
more
communities
clusters
with
volumes
being
provisioned
on
a
local
storage
system,
and
the
storage
system
are
paired
for
replication
and
and
some
or
all
volumes
are
paired
for
replication
right.
So
all
volumes
are
not
necessarily
replicated
only
some
are
replicated
based
on
which
volumes
are
chosen
for
application,
so
shamelessly
pulling
out
a
slide
from
the
talk.
F
D
F
Which
basically
brings
us
to
the
gaps
that
we
have
in
this
area
is
that
we
do
not
have
any
apis
to
manage
the
replication
desire
or
state
per
pvc
and
later
per
volume
group.
When
we
have
volume
groups
in
play
and
the
desire
really
is
to
enable
users
to
manage
replication
for
their
volumes
and
not
not
not
a
system
administrator
or
a
cluster
administrator
dealing
with
this
we'll
touch
on
that
a
little
later,
because
that
defines
how
we
want
to
design
the
aps
when
we
get
there.
F
F
So
if
you
just
reapply
the
application
to
an
alternate
cluster
dynamic
provisioning
will
kick
in
and
deploy
a
new
volume,
whereas
we
really
wanted
to
attach
the
pvc
to
to
a
replicated
copy
of
the
data,
not
not
not
get
into
dynamic
provisioning.
So
that's
another
gap,
but
of
course
this
has
certain
solutions
with
the
data
source
and
other
such
enhancements
that
are
already
provided
but
which
can
be
leveraged,
but
it's
a
gap.
F
All
right
with
that,
moving
on
to
our
understanding
of
how
different
story
what's
com,
what's
common
across
storage
systems
that
provide
such
replication
and
hence
probably
provide
a
stepping
stone
to
design
an
api
or
set
up
the
needs
for
such
an
api
again,
disclaimers
is
landscapes.
Why
there
are
many
storage
vendors,
so
so
what
we
talk
here
is
probably
not
going
to
cover
everybody.
We
hope
to
at
least
cover
most,
but
we
also
want
to
be
educated
on
what
we're
missing
so
that
we
can
design
this
better.
F
F
Also,
some
commonality
that
we
notice
across
systems
is
that
the
the
volumes
individually
are
assigned
roles
primarily
something
called
a
primary
and
a
secondary
or
multiple
secondaries,
with
the
primary
being
the
active
source
for
rights
and
io,
and
is
the
source
of
replication
that's
replicated
to
the
secondaries
and
which
is
the
target
for
the
replication
and
replication
can
be
two-way
or
n-way,
but
with
only
one
volume
designated
as
the
primary.
It's
not
an
active
active
scenario
that
the
volume
is
being
used
in
two
sites.
G
G
Then
you
have
problems
that
that
are
unavoidably
bleed
through
into
the
into
the
api
or
the
user
interface,
and
that
that's
usually
one
of
the
big
sticking
points
is
that
there's
some
some
storage
systems,
that
literally
do
it
on
a
per
volume
basis,
and
you
have
fine
grain
control
and
other
ones
that
insist
on
pretending
to
have.
You
know
individual
volume
pairings,
but
in
fact
it's
done
at
some
coarser
granularity
and
you
can't
fail
over
one
volume
without
failing
over
all
of
the
volumes.
F
F
G
Yeah
I
mean
you've
got
to
be
careful,
though,
to
have
different
types
of
volume:
groups
like
there's
volume
groups
for
the
purposes
of
replication,
there's
volume
groups
for
the
purposes
of
consistency,
there's
volume
groups
for
the
purposes
of
asset
management.
I
mean
there's
all
kinds
of
different
ways
to
slice
and
dice
your
volumes,
and
I
would
just
be
careful
about
assuming
that
every
volume
group
is
a
unit
of
replication.
But,
yes,
you
could
have
a
type
of
volume
group
which
was
specifically
for
replication
and
design
your
apis
around
it
and
get
things
to
work.
G
It's
just
more
complicated
then,
because
you
probably
have
to
do
both
that
and
something
for
the
per
you
know
individual
volumes
that
are
not
grouped,
and
then
you
have
to
have
a
way
of
communicating
to
the
user.
Like
sometimes
you
have
to
use
groups,
and
sometimes
you
don't
have
to
use
groups
and
it's
a
detail
of
the
implementation
and
it
it
just.
It
makes
everything
more
complicated
and
ugly,
it's
not
insurmountable,
but
it's
makes
your
job
harder,
yeah,
so
ben
actually.
C
I
think
volunteers
will
complicate
things
mainly
going
back
to
your
comment.
Then
the
first
comment
that
a
lot
of
storage
systems
they
do
replication,
you
know-
or
at
least
they
establish
appealing
relationship.
Let's
say
at
the
storage
server
level,
storage,
virtual
machine
level,
and
if
volume
groups
span
different
storage,
backends
or
storage,
endpoints
or
peering
endpoints,
it
gets
even
uglier.
H
So
ben
sezzala
here,
hey
so
nice
to
meet
you
everybody
here.
So
one
question
is
like
you
know
in
the
recent
I
guess
in
the
last
decade
vmware
has
these
v-walls
all
enterprise
storage
systems
have
kind
of
adopted,
v-walls
to
some
degree
or
the
other.
So
I'm
wondering
if
v
walls
kind
of
made
it
forced
everybody
to
kind
of
actually
have
this
per
volume.
Replication
topology.
I
do
agree
that
open
source,
like
soft
storage
software,
probably
doesn't
have
those
features.
But
I'm
wondering
if
enterprise
software
has
these
features.
G
G
A
G
I
mean
yeah
netapp
has
snapmere
that
replicates
individual
volumes,
but
like
our
trident
csi,
implementation
has
a
mode
where
you
put
multiple
volumes
in
the
same
flex
flaw,
and
so
in
that
situation
we
can't
mirror
the
individual
volumes.
You
can
only
mirror
the
whole
flex
fall,
which
is
as
the
de
facto
pool
of
volumes,
and
so
we
could
just
say:
well,
we
don't
support
replication.
In
that
case,
maybe
that's
good
enough,
but
I'm
sure
there
will
be
other
cases
where
someone
will
say.
H
A
Have
capabilities
right?
So
that's
what
csi
has
have
a
lot
of
capabilities.
There
is
really
just
one
or
two
things
that
are
really
required
for
every
driver
right
only
the
month
amount.
Everything
else
is
optional,
including
creative
volume
is
optional,
so
those
are
all
controlled
by
capabilities.
So
we
can
do
the
same
thing
here.
H
G
We
do
right,
I
don't
want
to
discourage
any
particular
approach.
I'm
just
saying
that,
like
having
tried
to
do
it
in
a
way
that
pleases
everyone,
you
end
up
with
an
unavoidably
complicated
user
interface,
because
you
know
you
can't
set
the
expectations
properly
up
front
on
what
will
work
and
what
won't
work.
H
E
F
F
F
H
Other
question
I
have
is:
when
you
have
a
group
of
volumes
you
want
to
replicate,
let's
say
individually,
also,
if
they're
not
consistently
kind
of
snapshotted
together,
atomically
then
there's
a
problem
with
dr
because,
for
example,
cassandra
has
like
let's
say:
10
10
different
volumes.
If
they
replicate
individually,
that's
not
going
to
help,
you
need
to
kind
of
make
them
all
atomically
snapshot
and
send
the
data.
F
Absolutely
I'll
just
tell
you
this.
This
is
why
I
just
put
it
later
per
volume
group
here.
My
intention
of
thought
was
that
replication
is
per
pvc
or
per
volume
group.
These
are
the
two.
Pvc
is
not
a
grouping,
but
these
are
the
two
levels
of
abstraction
that
we
have.
We
have
pvcs
we're
getting
to
volume
groups
because
even
now,
snapshots
are
not
the
same.
F
Cassandra
example
right
and
the
snapshots
are
not
going
to
be
consistent
across
these
these
cassandra
volumes
because
we
don't
have
a
group
that
we
can
snapshot
against
and-
and
I
know,
jim's
working
on
the
proposal
for
the
volume
groups.
My
answer
is:
yes,
we
absolutely
need
groups.
I
was
literally
thinking
when
we
have
volume
groups.
We
extend
slash,
use
that
as
a
replication
boundary.
F
D
So
I
might
I
have
a
quick
question
here.
I
want
to
take
a
step
back.
When
will
the
replication
relationship
be
broken?
I
mean
in
the
dr
case,
your
original
cluster
is
completely
gone
where
the
api
operations,
like
deleting
the
let's
say
we
expose
some
api
where
there's
cause
relationship
whatever
it
is.
D
F
D
F
Let's
continue
on
rolls,
which
we
talked
about
primary
and
multiple
secondaries,
and
this
this
probably
answers
the
next
question,
so
the
other
operation
that
there
are
two
other
operations
that
are
supported
right
in
this
failover
and
fail
back
and
failover,
is
typically
when
the
server
is
down
where
a
chosen
secondary
volume.
If
you're
just
talking
about
two-way
replication,
the
second
volume
is
forced
to
become
a
primary
which
has
to
happen
at
the
secondary
at
the
peer
storage
cluster.
D
I
Yeah,
yes
and
yeah,
it
can
force
a
secondary
to
become
a
primary
and
make
sure
that
if,
if
the
old
primary
is
running,
it
still
won't
be
able
to
write
the
data
and
that's
one
option
most.
I
think
storage
systems
do
that
and
if
not,
then
we
may
extend
one
api
to
to
do.
Some
fencing
actually
tell
the
first
cluster.
You
cannot
try
like
in
the
kubernetes
level
to
make
sure
that
your
v
is
not
taxes.
F
The
synchronous
cases
definitely
needs
not
definitely,
at
least
in
my
experience
for
storage
systems
needs
fencing.
You
don't
want
to
corrupt
the
data,
because
there
could
be
a
spurious
node
which
is
still
alive
in
the
primary
cluster
accessing
this
endpoint.
So
you
need
to
very
clearly
break
the
relationship
and
or
define
the
replica
relationship
before
starting
to
use
it,
but
that
has
to
happen
on
the
surviving
end,
not
on
the
so
you
won't
you
won't.
D
God
the
interesting
thing
is
that
it's
really
hard
to
tell
and
without
human
or
manual
intervention,
because
if
it's
just
unreachable,
it
is
possible.
There's
a
temporary
network
issues
whatever
in
the
primary
storage
system,
where
the
associated
stuff
is
not
there
and
after
five
minutes
it
came
back,
then
what
would
happen.
The
original
association
is
still
there
in
the
original
cluster
right.
G
Right,
but
if
you
break
it
on
the
secondary,
it
would
block
any
additional
ropes.
So,
basically,
once
they
come
back
in
sync,
they
would
tell
it
that
the
surviving
one
would
tell
the
original
one
hey.
I
cut
you
off,
we've
diverged
now
and
I'm
the
new
primary
and
you
have
to
discard
all
your
old
data
or
something
like
that.
I
D
G
F
F
If
I
assume
a
different,
if
a
volume
is
marked
for
a
different
role
like
secondary,
it
starts
resyncing
from
the
known
primary
and
if
the
known
primary
is
divergent
because
it
was
force
promoted,
it
probably
doesn't
rethink
and
from
the
last
non-good
snapshot
or.
However,
the
storage
system
implements
it,
which
is
what
ben
was
also
alluding
to.
If
I'm
not
wrong.
I
Yeah
and
another
comment:
when
we
decide
the
cluster
is
down
it's
not
by
the
storage
layer.
It
should
be
decided
much
in
the
darker
layers
and
it
could
be
a
human
that
decide
that
your
public
primary
site
is
down.
It
could
be
some
automatic
care,
so
some
software
detection,
but
it
will
be
in
a
higher
level.
It
won't
be
in
the
storage
support
system,
doesn't
really
describe
decide.
I
F
Also,
the
storage
I
mean
this
is
probably
getting
into
the
details,
but
the
storage
system
could
also
report
on
availability.
The
cube
cluster
could
still
be
available,
but
the
storage
cluster
has
issues
for
whatever
reasons
and
and
that
could
be
another
disaster
scenario
that
that
requires
failover.
H
Also,
the
most
common
thing
people
do
is
not
failover.
They
do
dr
testing,
which
is
kind
of
not
breaking
that
application,
but
just
trying
to
see
if
the
other
side
works
and
they
do
kind
of
set
up
a
test
bubble
and
try
to
run
their
workloads
on
the
other
side
to
see
if
it's
all
coming
up,
I'm
sure
you
may
have
covered
this
another
difference.
Let's
try
to
share
that.
F
G
We
were
talking
about
the
scenario
where,
like
the
whole
site,
you
know
the
kubernetes
cluster
and
its
storage
are
all
gone,
so
you're
going
to
new
kubernetes
cluster
with
new
storage
that
has
replicated
copies
of
your
volumes,
but
they're
different
volumes
right
there.
So
it's
not
the
same
pv.
It's
a
different
pv
in
a
different
cluster.
With
with
the
correct
information
for
the
replica.
J
F
That
was
that's
that's
why
I
referred
to
that
point.
That
was
about
external
application,
sorry,
external
clients,
to
applications
running
the
cube
cluster.
But
the
question
was
clarified,
stating
it's
about
connectivity
to
the
storage
system,
not
external
connectivity
to
the
applications.
Am
I
right.
C
C
If
it's
external
to
the
kubernetes
cluster,
then
you
need
something
like
a
you
know:
global
load
balancers
to
make
it
seamless,
which
is
not
always
possible
right,
but
even
with
the
easiest
simplest
case
scenario
where
applications
are
running
the
same
cluster,
let's
down
cluster
east
and
now,
if
you're
doing
failover
to
cluster
west,
now
you're
spinning
up
a
whole
bunch
of
new
pods
pvcs
and
pvs,
and
you
know
that
kind
of
goes
back.
You
know
whether
these
pvc
pvs
are
creating
advanced
or
not.
Obviously,
you
cannot
run
pause
in
standby
because.
C
D
C
C
C
I
guess
for
software
defined
storage
is
possible,
but
again
that
should
be
okay.
You
know,
as
long
as
you
know,
pvcs
and
pvs
look
the
sds
instance
on
the
destination.
Cluster
can
work
with
the
new
pvcs
and
pvs
yeah.
But
look
do
you
have
a
working
prototype
look
or
this
is
more
like
a
just
thinking
about
look.
I
just
want
to
look.
For
example
like
do
you
have
an
implementation
or
a
prototype
based
on
ceph
or
something
else
or.
F
Yes,
we
do
so,
let's,
let's,
let's,
let's
go
down
a
few
more
slides,
okay,
so
I
think
I
think
we
understand
failover.
We
talked
about
recovery
a
bit.
There
are
going
to
be
use
cases
where
the
entire
cluster
is
gone
and
and
the
peering
relationship
ship
has
to
be
reestablished,
and
then
there
is
the
failback.
You
know
get
it
back
to
the
original
cluster,
where
it
involves
more
coordination.
When
you
need,
you
need
to
mark
the
current
primary
secondary
and
then
move
it
back
to
the
alternate
cluster.
F
So,
let's
quickly
jump
in
here,
there
are
some
caveats
to
what
we,
what
the
working
prototype
that
we
have.
Let's
call
it
that
it
does
assume
to
our
environment,
and
what
I
mean
is
that
you
can
actually
flip
roles
and
the
replication
direction
would
change
it
does
it
does
not
deal
with
pairing,
pre-created
volumes
or
pairing
post,
creating
a
volume
on
the
target.
It
deals
with
the
storage
system
capable
of
creating
its
its
its
pair
when
its
replication
enabled
on
the
local
instance
and
the
unfortunate
last
part
of
it
is.
F
And
so
a
pvc
so
user
deploys
the
part,
deploys
the
pvc
and
creates
a
volume
replication
resource
for
pointing
to
the
pvc.
The
storage
systems
are
already
replication
appeared,
so
this
particular
image
goes
down
to
the
csr
drivers
and
then
reaches
the
storage
control
plane
to
start
the
replication
process
and
also
designate
the
volume
as
primary
on
the
east
right
here.
Also,
we
also
extended
this
with
a
volume
replication
class.
F
This
helps
us
carry
secrets.
This
helps
us
carry
vendor-specific
parameters,
helps
us
carry
replication
schedules
instead
of
giving
the
user
the
flexibility
of
saying
replicate
every
minute
every
three
minutes
or
if
it's
asynchronous
replication,
it's
primarily
focused
on
asynchronous
replication.
Initially
now
coming
to
the
more
interesting
parts
is
when
east
goes
down,
the
process
of
recovery
is
to
to.
Basically
you
know
you,
you
create
recreate
the
pvc
point.
The
volume
replication
back
to
the
pdc
and
set
it
as
primary
east
is
still
primary,
but
that's
fine.
F
So
we
we
are
thinking
of
using
data
source
as
an
alternate
to
this
whole
thing
we'll
come
to
that
later.
But
assuming
you
allow
pv
copies
for
the
moment,
you
copy
the
pv
you
you
recreate
the
pvc,
so
it
will
bind
back
to
the
same
pv.
You
create
a
volume
replication
pointing
to
the
pvc
and
flip
its
role
to
primary
or
secondary.
However,
you
want
to
manage
it,
and
then
you
deploy
the
parts
the
ordering
would
be
would
be
that
way.
Hey.
H
F
F
F
H
F
No,
no,
no,
you
didn't
so
all
the
way
up.
I
said
we're
not
yet
dealing
with
the
application
resources
like
parts
pdc's,
conflict
maps
in
this
talk,
but
but
having
said
that
we
are
this
whole
solution.
So
so
let's
go
back
so
copying.
This
pv
is
bad
right.
So
what
what
we
probably
really
want
to
do
is
we.
F
We
really
want
something
like
a
replication
handle
being
reported,
as
in
as
a
transferable
replication
handle
being
reported
back
in
the
volume
replication
resource
which
hence
the
user,
can
take
and
recreate
the
volume
replication
resource
on
the
on
the
secondary
cluster.
Instead
of
pointing
to
pvc
that
could
instead
have
a
source
of
the
replication
handled.
F
G
But
there
are,
there
are
sequencing
concerns
when
you're
trying
to
restore
something
depending
on
exactly
how
the
pv
the
destination
pvc
needs
to
get
created
and
and
when
relative
to
everything
else.
That
should
happen
so
yeah.
There
are
other
solutions
that
do
it,
but
you
have
to
have
an
opinion
about
timing
and
sequencing.
F
Yes,
yes,
definitely
so,
like
I
said
so
back
in
this
sequence.
This
is
in
this
in
this
setup,
where
we're
actually
transferring
tvs,
the
sequencing
is
obviously
strict.
I
mean
the
pv
is
restored
first
and
then
the
pvc
is
restored
are
created
based
on
the
application
resources
so
that
it
can
bind
to
the
pv
and
later
on.
F
It
can
create
at
this
particular
point,
even
if
the
volume
replication
is
not
created,
the
volume
is
actually
secondary
on
rest
until
the
volume
replication
resource
comes
in
and
changes
its
state
to
primary,
that's
not
going
to
change
so
technically.
The
storage
driver
would
disallow
mounts
maps
or
usage
of
that
volume.
F
All
the
node
stage
requests
would
probably
should
fail,
even
if
a
part
comes
in
out
of
order
from
the
volume
replication
resource
and
when
the
volume
replication
resource
comes
in.
That's
what
trickles
down
to
the
storage
layer
to
either
force
promote
this
to
a
primary
or
or
whatnot.
So,
yes,
there
is
a
strict.
G
G
K
If
I
understand
this
proposal,
I
think
maybe
what
you
guys
are
saying
is
what
whatever
ben
is
saying
here
and
the
application
logic.
That's
that's
not.
In
the
scope
of
this
proposal,
you
are
assuming
that
a
higher
level
layer
will
coordinate
some
of
that,
the
sequencing
etc.
But
you
are
kind
of
abstract
away.
The
definition
of
volume
replication
is
that
is
that
kind
of
true,
so
that
it
can
be
agnostic,
and
you
know,
and
it's
like
kubernetes
standard.
F
Yes,
that
is
that
is
true.
We,
but
to
ben's
point.
The
pv
copy
model
is
very
prescriptive
on
order
right,
we
cannot
break
this
order
right,
whereas
what
we
really
want
to
do
is
sorry.
This
is
not
moving
oops,
okay,
so,
whereas
what
we
really
probably
want
to
do
is
always
create
a
pvc
using
volume
replication
as
its
data
source,
so
that
these
ordering
constraints
are
not
there.
So
the
moment
the
pvc
is
created,
there
is
a
populator,
and
you
know
the
source
is
a
volume
replication
resource.
F
A
D
G
A
So
I
guess
that
was
my
question.
I
was
reading
this
one
right,
so
basically,
this
is
a
bug
report
saying
that
the
great
snapshot
is
being
retrying
without
the
the
back
off,
but
the
arrow
that
I
see
in
this
in
this
logs
this
one
should
not
be
there
anymore
in
if
you
test
latest
code.
So
that's
why
I
was
asking.
A
A
D
A
I'm
thinking
maybe
xiang.
We
can
talk
about
the
soft
lines
because,
for
you
know,
if
it's
dynamic
provisioning,
we
actually
call
chris
snapchat
multiple
times
to
get
the
status
of
the
snapshot,
but
then
we
ended
up
calling
chris
snapchat
more
time,
so
maybe
wondering
if
csi
driver
supports
list
snapshot
with
a
snapshot
id.
We
can
actually
use
that
to
get
the
particular
snapshot
status
in
that
instead
of
calling
kristaps
snapshot.
I
don't
know
if
that
would
help.
So
so
shall
maybe
we
can
talk
about
that.
A
L
A
L
A
L
A
B
D
L
L
You
just
try
to
create
a
like
the
snapshot
which,
like
does
not
have
the
right
permissions
or
like
some
missing
permissions,
some
some
error
scenarios
in
few
of
those.
We
were
able
to
replicate
that
the
retries
are
constantly
happening.
If
you
go
and
see
the
control
plane
locks
the
retry
time
stamps
that
all
the
griffiths
mentioned
at
the
top,
they
were
very
close
to
each
other.
It
was
not
waiting
and
there
was
no
terminal
state.
So
I
had
a
cluster
in
which
I
had
a
failed
snapshot.