►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Storage on Kubernetes - Learning From Failures - Hemant Kumar & Jan Šafránek, Red Hat
Using persistent storage with Kubernetes has been continuously improved with each release, but getting where we are was not easy. In this talk, we are going to cover a series of war stories and failure scenarios. We will talk about bugs (or design) that resulted in data loss, file system corruption, or storage simply refusing to come up. The limitations of storage subsystems, both what it can and can not do, will also be discussed These failures have led to numerous enhancements in Kubernetes. We will review the lessons these failures have provided, and discuss how they have been vital to improving our handling of the storage subsystem.
https://sched.co/UadX
A
They're
going
to
tell
you
a
few
stories:
kubernetes
has
lost
data
of
some
users.
How
component
is
exposed
things
that
should
not
be
exposed?
How
kubernetes
corrupted
data,
or
even
how
kubernetes
did
not
make
data
available
to
post
due
to
some
attach
and
detach
issues,
and,
of
course,
we
will
focus
on
how
we
fixed
it
and
what
we
learned
from
these
failures.
So
you
don't
need
to
repeat
the
same
mistakes
we
did
in
the
end.
A
We
will
talk
also
about
the
open
issues
we
have
right
now
and
how
we,
how
are
we
going
to
tackle
them?
So
the
first
thing
that
users
usually
layer,
learn
the
hard
way
is
that
they
should
not
mess
up
with
persistent
volumes
and
persistent
volume
claims.
In
this
case,
a
user
tried
to
migrate
some
data
from
a
distinct
cluster
to
production
cluster.
A
They
literally
took
Yama
files
of
the
persistent
volumes
and
persistent
volume
claims
and
they
restore
them
on
the
new
cluster
and
they
will
store
the
persistent
volumes
first
and
what
happened,
and
that
was
surprising
to
them
that
kubernetes
deleted
all
the
data
on
their
storage
back-end
and
they
deleted.
Also,
the
processing
volume
object
so
why
it
happened.
This
is
the
picture
of
the
original
testing
master.
A
They
have
a
PVC
bound
to
a
pv
that
points
to
some
storage
in
the
storage
back-end,
and
then
they
brought
the
persistent
volume,
object
and
restored
it
in
the
new
cluster
and
the
persistent
volume
was
bound
and
it
was
bound
to
a
persistent
organ
crime
that
did
not
exist
at
that
time.
They
didn't
restore
it
yet.
So
what
kubernetes
does
to
bound
volumes
that
are
bound
to
a
claim
that
doesn't
exist?
It
executes
the
reclaim
policy.
A
So
in
this
case
the
return
policy
was
delete
and
kubernetes
deleted
the
data
because
to
the
kubernetes
it
looked
exactly
same
like
there
was
a
claim
there
was
a
TV,
but
somebody
has
deleted
the
crime.
It's
exactly
the
same
situation.
An
API
server
and
kubernetes
has
no
way
how
to
distinguish
this
failed
migration
or
use
the
deleting
persistent
volume
claim
and
how
we
did
fix.
How
did
we
fix
it?
We
did
not
fix
it
at
all
and
we
will
not
fix
it.
This
is
how
kubernetes
works.
If
there
is
persistent
volume
claim.
A
A
You
should
it
use
some
dedicated
tools.
For
example,
yesterday
there
was
a
talk
about
Valera
from
happy.
Oh,
that
can
migrate
objects
between
clusters
and
just
in
case
you
want
to
play
with
persistent
volumes.
Please
use
retain
reclaim
policy,
so
kubernetes
will
not
delete
the
data
and
you
can
restore
them
later
and
in
this
particular
case,
if
you
restore
llamó
files
from
different
cluster,
you
should
sanitize
them.
A
In
this
case,
in
the
persistent
volume
there
is
UID
of
the
claim
that
is,
it
is
bound
to,
but
you
IDs
are
assigned
by
kubernetes
and
they
are
random.
So
do
you,
the
persistent
vol-plane
can't
exist
basically,
and
you
should
also
clean
up
some
binding
annotation
on
PV
PV,
C
2,
so
kubernetes
finishes
the
binding
first,
the
view
ID
and
everything
works,
and
then
it
doesn't
really
matter.
If
you
restore
PV
first
PVC.
First,
it
will
work
what
we
learned
here,
that
we
should
educate
our
users,
so
here.
B
A
Now
you
know
don't
play
with
your
TVs
and
we
know
that
our
documentation
is
not
perfect
and
everybody
in
this
room
can
speak
English.
So
it's
not
hard
to
update
the
documentation.
You
don't
need
to
write
any
code,
you
don't
need
to
pass
any
everybody
can
contribute,
and
every
contribution
is
welcome.
A
Why
did
it
happen?
It's
because
kubernetes
api
server
offers
no
referential
integrity.
You
can't
tell
it
to
not
to
delete
some
object
while
it's
used
by
different
objects.
So
when
user
deleted
the
claim
that
was
used
by
a
pod,
we
already
know
what
happens
to
persistent
volumes
that
are
bound
to
non
existing
claims.
The
reclaim
policy
is
executed,
but
the
pot
is
still
running.
A
This
error
is
not
that
serious,
because
user
ask
for
the
data
to
be
deleted.
They
deleted
the
claim.
Just
the
order
was
not
right.
The
pod
was
still
running.
How
did
we
fix
it?
We
introduced
finalizes
finalizer.
Our
component
is
a
cooperative
concept.
It's
a
field
in
every
API
object
in
the
API
server
and
API
server
will
not
delete
object.
That
has
a
finalizer.
So
in
this
case,
we
put
a
finalizer
into
every
persistent
foreign
claim
object
in
the
cluster,
using
a
new
admission
plugin.
A
So
when
user
tries
to
delete
the
object,
kubernetes
does
not
delete
the
object.
It
only
marks
it
as
terminating
and
everything
works.
Nothing
is
deleted
only
after
the
user
deletes
the
pod.
Our
newly
introduced
storage
in
use
protection
controller
sees
that
there
is
a
claim
that
is
not
used
by
any
port
and
its
size
safe
to
delete.
So
it
removes
the
finalizar
kubernetes.
Api
server
finishes
termination
of
the
object
and
the
reclaim
policy
is
executed,
and
now
the
stuff
is
deleted
in
the
right
order.
A
Another
case
when
user
restored
some
data
was
quite
surprising
because,
basically
they
restarted
cubelet
and
they
lost
data
on
the
aerobic
volumes.
What
happened?
The
queue
blood
got
offline.
In
this
case
it
was
due
to
a
crash,
but
it
doesn't
really
matter.
It
could
be
like,
for
example,
cubed
update
and,
at
the
same
time,
something
deleted.
A
pot
on
API
server
and
the
newly
restarted
couplet
basically
wiped
out
all
the
data
on
the
volume
of
the
pot
that
was
just
deleted.
A
But
in
this
case
the
volumes
were
not
mounted
yet.
So
what
super
did
it?
Recursively
removed
all
the
data
in
the
garbage
in
the
trash
directory,
but
there
was
still
volume
mounted
there
and
it
removed
all
the
data
from
the
volume
we
fixed
it
at
first,
we
looked
at
all
the
places
were
cubed
recursively
delete
data
to
make
sure
it
never
happens
again,
and
we
add
that
we
added
a
special
check
while
removing
later
that
we
never
cloth
across
a
file
system
boundary.
A
So
we
accidentally
don't
enter
a
mounted
volume
and
don't
remove
the
data
there.
This
is
what
this
was
just
a
safety
check.
The
real
fix
was
in
two
that
we
introduced
reconstruction.
You
may
have
noticed
that
in
Varley
Pew
blood
directory,
there
are
some
crazy
directories
with
weird
names
and
that's
how
we
keep
a
state
and
from
names
of
these
directories.
Cubed
can
reconstruct
what
volume
was
is
mounted
where
and
also
how
to
clean
the
volume
up.
A
So
when
nuclear
blood
starts
and
the
pot
is
not
in
the
API
server
at
least
cubed
can
restore
the
caches.
It
knows
what
volumes
belong
to
which
pot
and
it
can
unmount
them
cleanly
and
then
delete
all
the
garbage
left
by
the
pot.
What
we
learned
here
was
that
you
should
run
some
tests
for
this
behavior.
So
now
we
have
so-called
disruptive
tests
that
run
about
kill,
cubelet
delete
the
pot
from
kubernetes
api
server
start
new
Coubertin.
A
C
A
Okay,
so
we're
under
the
Optive
test
that
tests
exactly
this
scenario,
so
we
have
our
safety
check,
never
crossfire
system
boundary.
When
removing
beta.
We
had
the
reconstruction
before
the
bug
is
gone
until
it
happened
again.
So
how
was
it
possible?
The
scenario
was
completely
the
same
like
before.
A
A
The
safety
check
we
use
to
never
cross
file
system
boundary.
It
can't
work
because
this
root
disk
doesn't
present
a
file
system
boundary.
It's
the
still
the
same
root
disk
as
we
use
for
cached
data
in
cubed
and
also
the
reconstruction
didn't
work,
because
in
this
spot
that
was
deleted.
The
user
used
so-called
subpaths
feature
where
cubelet
doesn't
give
the
whole
volume
to
a
pod,
but
just
a
subdirectory,
so
one
we
once
we
knew
that
there
is
something
wrong
with
that
path.
A
A
A
Why
did
this
happen
and
it
happened
because
a
sibling
that
was
completely
innocent
in
the
container
this
evening
was
evaluated
on
the
host,
and
it
was
not
that
innocent
anymore
last
year
me
and
Michelle,
or
from
Google.
We
had
a
whole
session
just
about
this
security
issue,
how
we
fixed
it
and
including
exploit.
So
you
can
look
at
it.
Online
I
will
not
cover
details
here,
but
would
you
learn?
D
All
right
so
John
has
talked
about
various
ways:
communities
lost
managed
to
lose
data,
so,
in
spite
of
our
best
efforts,
we
have
in
certain
cases
we
have
noticed
that
communities
could
cause
file,
system,
corruption
and
I'll
try
to
cover
the
cases
where
this
could
happen,
and
this
is
a
story
of
two
bugs
that
happened
two
years
apart.
So
this
is
a
stack
trace
from
a
real
bug.
Action
you
can
see.
The
XFS
metadata
is
corrupt,
and
this
was
reported
against
kubernetes,
1.10
and
volume
type
was
fibre
channel
and
was
repaired
reported
on
November
2017.
D
So
a
while
back-
and
this
is
another
stack
trace
from
a
real
bug
from
D
message-
kernel
log
you
can
see
metadata
is
corrupt
again,
and
this
is
kubernetes
version
114,
so
pretty
recent
and
it
is
chef,
RB
d,
where
CS
broke
and
was
reported
in
August
2009
t.
So
what
happened
here?
The
problem
is
basically
boils
down
to
the
sometimes
same
volume
could
be
temporarily
mounted
on
more
than
one
node
at
once
and
typically
block
devices
with
file
systems,
don't
like
it
and
it
could
cause
file
system
corruption.
D
How
do
we
fix
it?
And
you
could
say
that
storage
provider
should
fix
it
if
you,
if
you
are
probably
using
GC,
EBS
and
EBS,
doesn't
allow
it,
but
in
some
cases,
depending
on
what
storage
type
you
are
using,
it
may
not
be
possible.
Another
thing
that
kubernetes
could
itself
do
is
Coonans
try
to
be
helpful
as
much
helpful
as
it
can
and
it
has
a
concept
of
access
modes
and
kubernetes
could
enforce
the
access
modes
associated
with
persistent
volumes.
D
So
those
of
you
who
are
not
familiar
with
access
modes
I'll
just
give
a
brief
intro.
What
are
access
modes-
and
these
are
the
three
prominent
access
modes.
Readwrite
ones
means
a
volume
could
be
mounted
as
readwrite
on.
Only
one
node
readwrite
managed
could
be
mounted
as
readwrite
on
many
nodes.
It
only
many
means
it
could
be
mounted
as
read-only
on
many
nodes,
and
this
is
a
PVC
spec.
D
You
can
see
access
mode
is
specified
here
and
if
you
are
using
dynamic
provisioning,
then
kubernetes
will
try
to
bring
a
PV
that
matches
your
requested
access
mode.
The
thing
is
that
kubernetes
did
not
enforce
access
mode
at
all
until
version
1.7
1.8.
So
you
could.
It's
a
Wild
West,
you
could
have
a
volume
and
you
could
use
it
at
two
nodes
at
the
same
time
and
and
cause
problems
like
this.
So
how
did
we
fix
it?
D
We
started
enforcing
access
modes
in
kubernetes
and
so
that
you
could
not
attach
a
volume
that
has
read
heard
once
on
multiple
nodes
at
the
same
time,
but
you
may
notice
that
this
box
that
we
talked
in
the
beginning
was
for
1.10
and
what
at
14?
So
if
this
fix
was
made
in
1.78,
how
come
these
bugs
are
still
present?
That's
surprising,
so
we
need
to
learn
that
access
mode
enforcement
in
kubernetes.
D
What
are
these
limitations
is,
and
the
first
thing
is
that
it
only
works
from
volume,
types
that
are
attachable
and
I'll
talk
about
what
attachable
means
in
a
minute,
and
second
thing
is:
it
does
not
prevent
two
parts
from
using
same
volume
on
the
same
node.
Third
thing
is
it's
based
on
the
cast
volume
state
in
controller
manager,
so
all
attach
D
goes
through
a
control.
Plane
call,
but
that
state
is
cached.
D
Here
is
a
few
examples
of
attachable
volume
types,
EBS
open
stacks
energy
series
for
disk.
If
you
are
familiar
with
CSI,
any
CSI
volume
that
has
published
and
published
controller
capability
is
considered
attachable
by
kubernetes.
Here
are
the
few
examples
of
volume
types
which
are
not
attached,
but
like
ice
cassis,
fr
BD,
fibre,
channel
and
typically
CSI
volume
that
do
not
have
publish
and
publish
controller
capability.
These
are
an
attachable
volume
types
and,
and
these
are
they
typically
could
be
attached
or
published
to
a
node
directly.
They
don't
go
through
a
control
plane
call.
D
So,
as
you
can
imagine
that
for
for
the
bugs
that
happens,
fr
BD
fibre
channel
because
the
volume
is
not
published
through
a
control
plane,
they
could
be
directly
placed
on
a
node
the
access
modes
weren't
in
force.
That's
why
this
bugs
happened
so
how
we
fixed
up
how
we
fix
this
was
we
implemented
a
dummy
attached.
The
diet
interface
for
that
is
basically
a
no
op
for
I,
scuzzy,
fiber
channel
and
self
RBD
for
in
in
kubernetes
for
entry
volume
types.
D
This
would
basically
turn
non
attachable
volume
types
into
attachable
and
it
will
ensure
that
the
volume
is
made
available
through
a
control
plane
call
to
a
node
not
directly,
and
it
will
enforce
the
access
mode.
So
that
fix
the
bug
for
the
most
part,
but
it
still
did
not.
So
what
are
the
recommendation
for
CSI
drivers?
You
may
have
seen
that,
though,
the
bug
that
happened
in
1:14
was
it
reported
against
fr,
BD
CSI
driver
so
for
CSI
drivers.
D
Our
recommendation
is,
whenever
possible,
implement
control,
plane
based
enforcing
in
the
volume
itself,
so
that
it's
more
robust
and
it
pushes
the
problem
back
to
storage
providers.
It
may
not
be
possible
if
you
are
running
off-the-shelf
storage
solution.
There's
another
alternative
is
like
kubernetes
ships
external
attach,
a
sidecar
that
you
could
use.
So,
even
though
your
volume
is
not
does
not
have
publish
and
publish
capability,
the
CSI
calls
will
still
go
through
and
access
modes
will
be
still
enforced.
D
So
you
can
use
this
to
get
around
some
of
the
problems
and
you
should
not
disable
attach
attach
from
CSI
driver
object
for
to
make
it
work,
and
the
last
thing
is
that
I
will
say:
is
that
prefer
use
of
stateful
set
over
deployment
for
workloads
that
use
storage,
that's
kind
of
obvious,
but
a
stateful
set
has
stronger
guarantees
about
pod
placement
like
it,
you
cannot
have
two
pods
running
at
the
same
time.
That
use
the
same
thing.
So
that's
another
recommendation
that
we
would
make
alright.
D
So
next
section
that
we
want
to
talk
about
is
like
AWS
attached
at
issues
so
who
here
runs
AWS
kubernetes
on
AWS,
either
yie
case
or
okay,
quite
a
few
number,
so
being
a
largest
cloud
provider,
it
is
not
always
best
tuned
to
run,
kubernetes
I
would
say,
and,
as
we
have
been
doing
this
for
a
long
time,
we
had
some
starting
problems
and
and-
and
you
know,
teething
problems.
The
first
issue
that
we
noticed
was
that
sometimes
a
volume
will
stuck
in
attaching
and
detaching
state
forever
and
kubernetes.
It
was
very
hard
to
reproduce.
D
What
we
found
out
was
that
AWS
does
not
like
to
use
the
device
name.
Any
of
us
is
in
unique
in
the
sense
that
when
you
publish
a
volume
or
attached
volume
to
a
node,
you
have
to
give
a
device
name
generally,
all
other
drivers
go
and
do
that
and
that
device
name
must
be
free.
Another
thing
is
a
race
condition
like
if
you
recently
use
release
the
device
and
we
were
trying
to
use
it
again,
it
could
cause
the
volume
to
be
stuck
in
a
touching
State.
D
D
Another
thing
was
sometimes
like
saris
trying
to
be
helpful
and
they
would
force
detach
a
volume
from
a
node
on
AWS
to
you
know,
night
we
just
make
the
workload
go,
make
the
pod
run
or
whatever.
But
what
happens
is
that
just
makes
problems
worse?
We
found
out,
and
any
other
volume
that
tries
to
attach
on
the
node
will
also
get
stuck
in
attaching
state
forever.
So
in
the
end
we
this
disk
or
is
there
actually
an
entry?
D
We
started,
we
started
painting
the
node
if
the
attach
times
out
in
AWS
and
you
have
to
reboot
the
node
delete
the
node
object
to
make
it
work
and
kind
of
solves
some
of
the
pain.
There
is
another
problem
that
in
particular,
AWS
cloud
wire
we
have
observed
is
that
it's
API
is
our
eventual
consistent
in
particular
immutable
API
calls
like
describe
volume
describe
instance,
so
some
some
examples
are
like
we
detach
a
volume
and
we
call
describe
volume
and
a
tabular
says
it's
attached.
D
We
attach
a
volume
and
we
say
describe
volume
and
describe
all
returns
is
detached
so
and
sometimes
in
worst
case.
It
can
go
back
in
time.
Kubernetes,
unfortunately
well
at
least
initially
was
not
written
to
deal
with
this
kind
of
problems
so
how
we
fixed
it
and
it
benefits
other
cloud
providers
and
volume
types
to
is.
We
introduced
uncertain
state
in
an
in
controller
manager
where
certain
volumes,
when
we
cannot
be
sure
if
it
is
attached,
will
mark
it
uncertain
still.
D
A
So
this
was
about
the
issues
that
we
fixed.
You
also
have
couple
of
issues
that
we
did
not
fix
yet,
and
we
are
working
on
them.
The
first
one
that
we
would
like
to
fix
as
soon
as
possible
is
our
FS
group.
Fs
group
is
a
feature
of
kubernetes
when
date
on
volume
can
be
owned
by
a
random,
UID
and
GID,
and
also
a
poet
runs
as
different
random,
UID,
orgy
ID,
and
so
in
order
to
make
the
data
on
the
volume
available
to
the
pot,
we
have
a
called.
A
So
yesterday
we
had
a
fruit
of
fruitful
discussion
here
on
cube
con,
how
to
fix
it.
So
currently
it
looks
like
we
will
introduce
new
API
field
where
you
could
change
the
own
blood
would
change
the
ownership
just
once
and
not
with
every
pot
a
start.
We
will
see
how
far
it
goes
and
if
we
can
pass
API
review.
If
that.
D
Another
problem
that
those
who
are
running
kubernetes
on,
like
maybe
vmware
or
somewhere
else
might
have
seen
more,
is
like
kubernetes,
actually
does
not
detach
volume
from
shutdown
notes.
So
what
happens
is
like?
If
you
shut
down
a
node,
it
does
a
weak
parts
generally,
not
always
it
there's
a
there's
other
logic
built
in,
but
it
evict
the
pod.
D
But
you
will
see
that
the
the
new
parts
on
the
new
nodes
replace
nulls
do
not
are
not
able
to
come
up,
because
the
persistent
volume
is
still
attached
to
the
old
node
and
they
are
not
able
to
come
up
and
the
the
reason
for
this
is
that
the
parts
of
shutdown
not
knows
typically
do
not
get
deleted.
They
stay
in
unknown
state
because
kubilay
cubelet
is
not
there.
So
it's
not
able
to
signal
the
weather.
Pod
is
still
running
or
not
so
so.
D
D
If
you
are
running
in
a
cloud
provider
like
AWS
GCE
GC,
then
if
you
run
your
cluster
in
auto
scaling,
group
or
gke
generally
shutting
down,
the
node
will
cause
the
node
to
be
deleted
and
replaced,
and
in
that
case
volumes
are
automatically
detached
from
related
note
and
everything
works
perfect
for
better
bare
metal
clusters
or
cloud
providers
that
don't
allow
easy
replacement
of
a
node.
It's
a
bigger
problem,
because
in
that
case
you
cannot
just
do
that.
So
our
current
solution
is
to
write
external
controller
that
can
monitor
for
shutdown
nodes.
D
A
Another
issue
is
that
we
know
about
is
empty
dear
if
you
have
a
pot
that
uses
empty
there.
All
these
empty
directories
are
on
the
same
drive.
So
if
there
is
a
malicious
pot
that
uses
the
empty
DEP
too
much,
it
can
trash
IO
for
the
other
users.
We
currently
don't
have
any
solution
for
that.
You
don't
even
have
a
design.
So
if
you
are
affected
by
this,
please
come
to
us
and
we
would
welcome
any
contribution
here.
D
This
is
always
issues.
It's
looking
like
a
pattern
like
so
this
is,
if
you
are
using
encrypted
EBS
volumes,
you
might
notice
that
sometimes
the
volumes
come
with
nonzero
data
and
that
causes
the
cubular
to
detect
a
random
filesystem
signature
on
them
on
the
volume
and
cubelet
is
not
designed
to
overwrite
file
systems.
So
if
you
are
requested,
ext4
filesystem
and
encrypted
volume
says:
I
have
ms-dos
partition,
that
kubernetes
will
not
delete
that
ms-dos
partition
and
and
it
would
rather
fail
the
mount
operation.
D
Basically,
this
is
something
that
we
are
working
with
Amazon
and
trying
to
fix
it,
but
it's
a
pain,
so
yeah.
In
summary,
I
would
say
that
fixing
bugs
is
I,
don't
never-ending
process.
Six
storage
has
been
involved
in
this
effort
and
we
welcome
more
contribution
from
folks
who
are
here,
and
we
have
learned
from
our
failures
and
one
other
thing
that
yon
was
saying
is
like.
We
have
started
running
our
matrix
of
tests,
so
we
have
like
sub
paths
with
local
volume,
gcpd,
plus
a
path
plus
all
the
volume
tests.
D
D
Looking
in
the
talk,
it
sounds
like
we
lost
data
quite
a
bit,
but
we
do
not
lose
data
all
the
time.
It's
it's
sometime
and
then
sometimes
user
interaction
causes
it
and
we're
still
amazed
by
user
creativity
how
users
use
persistent
volumes.
We
did
not
expect
a
user
to
use
local
volume.
The
root
disk
has
local
volume
and
sub
Park,
and
that
is
another
good,
but
we
did
not
expect
that
there
was
a
bug,
so
we
fixed
it
and
we
try
to
ensure
that
this
does
not
happen.
So
that's
pretty
much
our
talk.
B
Hi
I
see
you
recommended
using
stateful
set
over
deployment.
What
we
have
seen
is
in
Stateville
set
whenever
a
node
goes
down.
The
problem
is,
it
doesn't
do
a
failover
and
if
node
went
down
permanently
it's
fine,
but
if
network
partition
and
all
how
do
we
detect
those
kind
of
scenario?
And
we
make
sure
that
app
is
not
affected
by
this
failure?
B
D
So
straight
full
set.
The
question
was
we
recommended
you,
stateful
set
our
deployment
and
the
when
estate
when
a
node
goes
down,
the
replacement
parts
are
not
created
when
you're
using
stateful
set,
and
the
reason
for
that
is
because
the
part
on
the
that
that
is
backed
up
by
stateful
set
on
the
shutdown
node
or
the
node
that
is
down
like
crash
or
whatever
is
still
actually
around
and
still
stateful
set,
will
not
create
a
replacement
part
until
the
old
part
is
deleted.
D
So
our
current
recommendation
is
like
is
until
we
solve
this
problem
for
both
stateful
sat
and
diplomas
to
force,
delete
the
part
and-
and
you
could
detect
like
if
you're
running
in
cloud
provider.
As
I
said,
the
node
such
nodes
will
have
a
shutdown
taint,
so
you
could
pretty
much
find
out
right
external
controller
and
list
all
the
all
the
nodes
that
have
shutdown
taint,
that
have
parts
and
in
unknown
State
and
you
could
delete
them
and
that
will
cause
the
replacement
parts
to
be
created.
I
hope
that
you.
A
Well,
we
run
most
of
our
tests
on
AWS.
So
that's
what
we
tested
the
most
and
we
noticed
these
issues
like
they
don't
happen.
Always
they
are
super
hard
to
reproduce,
but
they
sometimes
happen,
but
we
haven't
noticed
these
issues
on
GCE
yet,
but
maybe
that's
because
we
don't
run
this
the
test
that
that
much
so
we
don't
know
on.
D
C
A
We
have
several
layers
of
tests.
We
run
some
tests
for
every
PR.
We
have
flow
tests
that
run
every
I,
don't
know
one
hour.
There
are
some
scale
tests
like
you
can't
say
how
long
does
it
take,
for
example,
for
every
pool
requires
the
tests.
Take
I,
don't
know
30
minutes,
maybe
the
slow
tests
again
like
30
minutes
and
I
have
no
clue
how
the
stands.
A
D
F
Hey
I
have
a
question
regarding
the
access
modes,
so
I
have
been
trying
different
storage
providers
starting
from
1
point
1
4
in
1
point
1,
4
I
could
able
to
bye
and
mount
my
volume
to
multiple
parts
on
the
same
node.
It
different
storage
providers
behave
differently
on
1.16
I'm,
seeing
a
different
behavior.
Now
Luke
was
working
for
me
where
I
cut
multiple
attacks
on
the
same
node.
Now
it
is
broken,
looks
like
kubernetes.
Upstream
code
has
changed
in
that
way.
F
D
D
C
D
It
depends
your
so
yeah
so
readwrite
once
it
depends
like,
for
example,
a
luke
deployment
that
uses
CSI
and
if
they,
if
they
disable,
attach
required
in
CSI
driver,
then,
as
I
said,
the
attach
detach
they
will
not.
They
will
not
have
attached
to
start
and
and
volume
could
be
directly
mounted
on
a
node
and
in
that
case,
readwrite,
once
access
mode
will
not
be
enforced.