►
From YouTube: Five Great Ways to Lose Data on Kubernetes (And How to Avoid Them) - Robert Hodges, Altinity LTD
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Five Great Ways to Lose Data on Kubernetes (And How to Avoid Them) - Robert Hodges, Altinity LTD
Databases and stateful apps are gravitating rapidly to Kubernetes, so the sorrows of accidental data loss cannot be far behind. As long-time database engineers and authors of the ClickHouse Kubernetes operator, our team has seen lots of imaginative ways to lose data. We also learned how to prevent them. The talk starts with standard high availability/disaster recovery models used in DBMS and demonstrates that classic ways to lose data are still alive and well on Kubernetes. We'll then pivot to Kubernetes-specific disasters-in-waiting, such as the PV that wasn't, affinity afflictions, and the dreaded fat fingers of fate. The talk will help instill a healthy sense of paranoia and give listeners tools to ensure their experiences with cloud-native data will be happy ones.
https://sched.co/Zet6
A
Hello,
everyone-
this
is
robert
hodges
and
I'm
happy
to
welcome
you
to
my
talk
on
five
great
ways
to
lose
data
on
kubernetes
and
how
to
avoid
them.
It's
an
honor
to
be
presenting
at
kubecon
europe
2020,
where
this
is
a
great
conference,
a
lot
of
really
wonderful
talks,
so
we're
just
totally
happy
that
to
be
selected,
the
topics
that
I'll
be
talking
about
are
things
that
my
company
and
and
my
colleagues
have
dealt
with
from
day
to
day
over
the
last
couple
years.
A
So
we
hope
that
you
will
enjoy
the
information
we're
about
to
share
with
you
speaking
of
my
company.
Let's
do
some
introductions,
so
my
name
again
is
robert
hodges.
I
am
the
alternative,
ceo
and
but
more
relevantly.
For
this
talk,
I
have
a
background
long
background
in
data
I
started
with
my
first
database
management
system
in
1983..
A
A
My
company
altenity,
is
an
enterprise
provider
for
click
house,
so
we
provide
services
and
software
for
the
click
house.
Data
warehouse,
it's
a
very
popular,
very
fast,
open
source,
sql
data
warehouse
again
most
relevantly.
For
this
talk
we
are
the
implementers
of
the
click
house
kubernetes
operator
and
have
made
a
major
commitment
to
operating
data
warehouses
in
a
cloud
native
fashion
on
kubernetes,
and
that
work
is
the
source
of
a
lot
of
the
information
we're
going
to
provide
in
this
talk
today.
A
A
In
the
example
I'm
showing
you
right
now,
we
see
a
typical
table
from
a
from
a
data
warehouse
containing
sales
data,
so
data
warehouses
are
used
to
answer
questions
that
often
go
to
company
strategy
and
tactics
such
as
which
products
have
the
best
gross
margins
over
time.
If
you
can
figure
that
out,
you
know
which
products
make
the
most
money
and
the
ones
that
you
want
to
focus
on
selling.
A
Similarly,
we
want
to
define
what
we
mean
by
the
word.
Lose
well
in
the
context
of
data.
Lose
is
actually
a
spectrum,
it's
not
a
single
thing.
So
when
we
lose
data,
we
can
talk
about
losing
it
temporarily,
so
it
can
be
just
gone
for
a
while,
but
can
come
back
completely
all
the
way
to
a
state
where
it
is
completely
gone
forever.
You
will
never
see
it
again.
You
will
feel
awful
and
it's
there's
it's
just
vaporized.
A
So
I
call
this.
The
data
loss,
arrow
of
evil
and
what's
kind
of
interesting
in
in
when
dealing
with
data,
is
the
value
depending
on
what
kind
of
data
you
have.
The
loss
may
be
more
or
less
severe
at
different
points
in
the
spectrum,
so,
for
example,
if
you're
running
my
sequel
and
using
it
to
operate
a
an
e-commerce
site,
if
my
sequel
is
even
avail
unavailable
and
the
data
is
not
not
accessible,
it
means
you
can't
sell
anything
so
you're,
basically
at
a
standstill
for
data
warehouses,
that's
not
such
a
big
deal.
A
They
often
offer
many
of
them,
particularly
if
they're,
batch,
oriented
or
doing
answering
strategic
questions
may
run
sort
of
on
a
nine
to
five
basis
so
having
it
down
for
a
while
is
not
such
a
big
deal.
On
the
other
hand,
losing
some
big
chunk
of
data
means
that
you
lose
vision
on
what's
the
state
of
the
business,
so
that
can
be
a
very
serious
problem
in
both
cases
and
and
for
other
databases
as
well.
A
A
A
In
this
particular
example,
I'm
going
to
use
helm
so
helm
is
a
popular
tool
to
create
deployments
in
kubernetes,
bring
them
up.
You
can
upgrade
them.
You
can
take
them
away.
In
this
particular
example,
I'm
going
to
create
a
namespace,
my
sql,
using
cubecuddle.
I'm
then
going
to
run
a
single
line,
helm
install
command
to
install
the
a
my
sql
server,
which
will
then
pop
up
and
run
and
a
few
minutes
after
even
a
few
seconds
after
I
run
this
command.
A
A
A
So
my
sql,
as
you
know,
if
you've
used
have
experience
directly
with
mysql
yourself,
has
very
good
built-in
replication
and
so
replication
enables
you
to
have
live
copies.
You
have
a
primary,
my
sequel
server
that
would
for
your
e-commerce
site.
That
would
be
where
your
transactions
run.
It
has
storage
attached
to
it,
and
then
you
have
my
sql
replication
enabled
and
it
goes
it
as
soon
as
a
transaction
commits
in
the
primary
it.
It
immediately
transfers
it
to
the
replica
it's
very
fast,
often
a
second
or
less,
to
move
transactions
across.
A
The
other
kind
of
replica
is
a
static
copy
which
is
represented
by
a
backup,
so
you
can
take
those
replicas,
and
this
is
again
a
very
common
way
that
this
is
handled
in
my
sql
from
time
to
time.
You'll
run
a
backup
of
the
of
the
my
of
the
mysql
replica
and
then
you'll
store
that
back
up
in
another
location.
In
case,
you
need
to
make
more
replicas
at
a
later
time,
so
static,
copies
in
live
and
live
replicas.
A
So
the
issue,
though,
when
you
come
to
kubernetes,
is
how
do
you
actually
implement
this
in
kubernetes
because
we're
talking,
at
the
very
least,
a
complex
deployment
with
a
couple
of
different
types
of
servers
running
in
separate
pods?
We
have
backups
and
in
fact,
as
we
get
to
other
types
of
databases,
these
these
may
be
far
more
complex.
A
A
The
answer
that's
evolved
over
the
last
few
years
is
something
called
kubernetes
operators,
so
the
way
that
a
kubernete
what
kubernetes
operator
does
is
it
allows
you
to
create
what
are
called
custom
resource
definitions.
So,
instead
of
thinking
of
the
database
as
a
collection
of
deployments,
stateful
sets
services
pods.
You
instead
have
a
single,
a
new
resource
which
represents
the
entire
structure
of
the
database.
A
And
and
so
what
happens,
is
you
load
this
custom
resource
definition
into
kubernetes?
It
is
kubernetes
api
recognized
that
it's
handled
by
a
particular
operator
type
it
dispatches
it
to
the
operator,
which
then
looks
at
it
and
decides
what
changes
it
needs
to
make
in
kubernetes
to
implement
the
database.
So
what
it
then
does
is
from
there.
A
A
You
can
see
from
this
that,
in
fact,
what
this
does
is
it
it
is
much
simpler
than,
for
example,
having
to
do
the
deployments
directly
and
do
all
the
low-level
resources.
We
have
a
single,
very
simple
resource
file.
This
is
actually
a
real
resource
file
that
you
could
load,
and
it
says,
let's
have
a
cluster
named
ch01,
actually
with
a
configuration
called
replicated
two
shards
two
replicas
and
then
point
to
zookeeper,
which
is
used
to
keep
to
keep
track
of
of
state
between
the
replicas.
A
So
this
is
a
huge
step
forward
and
is
really
one
of
the
ways
that
we
can
implement
replication
and
backups
in
databases
on
kubernetes
in
a
relatively
straightforward
manner,
so
that
was
the
first.
So
replicas
are
one
of
the
first
solutions
to
to
losing
data.
Let's
look
at
some
of
the
other
ways
that
we
can.
We
can
lose
data,
so
the
next
one
is
what
I
call
blast
radius
blues
and
let's
talk
about
this
term.
It's
one
of
my
favorite
terms.
A
A
So,
for
example,
the
failure
of
network
attached
storage
or
a
failure
of
the
network
itself,
which
can
either
cause
data
loss
or
unavailability
beyond
that,
you
can
think
of
a
failure
at
the
level
of
a
kubernetes
cluster
so
that
things
within
that
cluster
become
unavailable
or
unusable,
and
then,
of
course,
a
failure
within
an
entire
region.
So,
for
example,
you
can
have
failures
in
you.
There
have
been
historically
failures
in
things
like
amazon,
that
cost
entire
regions
to
lose
access
to
critical
services
and
even
data
across
across
a
number
of
data
centers.
A
So
the
really
key
thing
with
with
to
avoid
blast
radius
problems
is
to
get
distance
between
your
replicas
and
for
that
there's
an
incredibly
helpful
concept
in
in
kubernetes
called
affinity
which
basically
gives
us
the
ability
to
move
pods
around,
as
well
as
associated
resources
like
storage
into
different
locations,
so
that
they
are
far
enough
away
that
if,
if
one
thing
fails,
it
doesn't
take
all
your
data
down
with
it.
So
here's
here's
a
simple
example,
a
very
simple
example
that
illustrates
affinity
as
well
as
it's
it's.
It's
opposite,
which
is
anti-affinity.
A
So
in
this
particular
picture.
We
have
a
couple
of
zookeeper
pods
zookeeper
is
a
very
popular
distributed
directory
service
that
maintains
consensus
across
a
number
of
nodes.
It's
used
for
things
like
leader
election
as
well,
as
you
know,
holding
distributed
logs.
Anything
where
a
series
of
processes
need
to
agree
on
what
the
state
of
things
is.
It's
really
important,
with
zookeeper
to
move
the
to
have
the
nodes
separated
so
that
a
failure
will
not
take
all
of
them
down.
A
In
this
particular
case,
what
we
can
do
is
we
can
use
anti-affinity
to
keep
two
zookeepers
from
showing
up
on
the
same
node
in
this
case
host
116
on
the
left.
Similarly,
we
can
use
affinity
to
drive
the
that
that
zookeeper
one
instance
to
a
host
that
has
a
particular
that
has
particular
characteristics
that
interest
us.
One,
common
use
of
affinity
is
to
drive
the
pods
basically
to
separate
the
pods
across
availability
zones.
A
So,
let's
first
look
at
anti-affinity,
so
this
is
the
this
is
pod
anti-affinity,
which
drives
the
basically
it's
going
to
drive
the
the
pods
apart
and
keep
them
from
landing
on
this
being
scheduled
on
the
same
hosts.
So
what
you
see
here
is
a
part
of
a
pod
definition
and
under
the
spec
we
have
affinity
rules
and
we
have
a
rule
for
pod
anti-affinity
and
we
say
that
this
is
a
rule
that
is
going
to
be
used
during
scheduling.
A
We
don't
want
to
have
two
pods
from
the
same
data
warehouse
on
the
same
host
as
you
can
see
from
looking
at
this
syntax
and
even
hearing
me
read
it
it's
a
little
bit
non-intuitive
and
we'll
get
to
that,
because
that
actually
turns
out
to
be
an
issue.
But
if
you
get
this
right,
what
this
will
do
is
is
drive
the
pods
off
the
you
will
never
get
two
pods
scheduled
on
the
same
host
now.
A
So
what
this
is
going
to
do
is
any
pod
that
contains
this
is
going
to
want
to
land
in
in
us
west
2a
when
it
gets
scheduled.
One
of
the
things
that's
kind
of
interesting-
I
should
note
here
about
about
affinity-
is
particularly
availability
zones.
Is
we
have
found
in
our
work
that
it's
best
if
you're
really
concerned
about
distribution
across
azs?
It's
best
to
be
very
explicit
about
it,
as
we
show
here
there
are
sort
of
tricks.
A
The
result
when
you
get
it
right
is
that
you
get
distance
between
your
replicas.
So
in
this
illustration,
right
here
we're
showing
a
couple
of
shards
on
the
data
warehouse,
that's
click
house,
and
then
we
have
three
zookeeper
pods.
So
that's
a
zookeeper
ensemble
we
have
the
replicas
neatly
spread
across
the
availability
zones,
us
west
2,
a
b
and
c
and
and
and
so
that
that
spreads
the
data.
A
The
other
thing
that
I
want
to
draw
your
attention
to
is
the
backups.
So
if
you
take
backups,
a
really
smart
thing
to
do
is
stick
them
in
object.
Storage.
This
is
common.
Just
because
object,
storage
is
convenient
and
relatively
inexpensive,
but
the
other
thing
about
object
storage
is
that
it
is
itself
replicated
and
can
be
quite
distant
from
the
systems
that
you're
backing
up.
A
So
the
topology
that
we're
showing
right
here
with
the
distance
between
replicas
is
actually
quite
common
in
kubernetes
and
and
is
very
easy
to
implement,
with
things
like
cops
with
aussie
ks
and
other
types
of
managed
kubernetes.
So
so
definitely
something
that
you
should
use
to
protect
data
when
we're
talking
about
blast
protection
across
regions
or
kubernetes,
in
other
words,
sort
of
wider
out
in
these
concentric
circles,
dealing
with
things
like
the
loss
of
a
kubernetes
cluster,
the
loss
of
a
region.
A
This
is
also
this
is
no
longer
a
data
problem,
but
more
of
a
question
of
losing
it
resources
entirely.
So
it's
much
more
of
an
exercise
for
the
reader
that
doesn't
just
include
data
but
includes
things
like
hey.
How
do
we
handle
networking?
How
do
we
handle
dns
things
like
that?
So
that's
really
beyond
the
scope
of
this
talk,
although
it's
something
that
people
commonly
do
so.
What
we'll
do
is
we'll
leave
that
here
and
let
you
work
on
that
yourself.
A
Meanwhile,
what
I'm
going
to
do
is
proceed
to
another
kind
of
data
loss
which
is
directly
related
to
kubernetes
itself,
and
that's
what
I
call
affinity
afflictions.
So
affinity
rules
are
great
and
but
one
thing
you
should
do
is
you
should
definitely
check
that
they're
really
doing
what
you
think
they're
doing.
Here's
an
example.
A
So
we
have
a
zookeeper
ensemble,
which
I
set
up
as
an
experiment,
and
what
I've
run
is
a
cube
cuddle
command
to
show
where
the
pods
are
running,
and
the
first
thing
that
I
notice
is
even
though
this
this
this
zookeeper
was
supposedly
properly
set
up
and
and
is
something
I
expect
to
run
in
production.
I
notice
that
two
of
the
pods
are
running
on
the
same
host.
So
that's
a
little
bit
spooky.
A
The
other
thing
that
I
notice
is.
If
I
want
to
go
check
where
those
hosts
are
in
in
which
azs
I
can
see
that
they're
spread
out
across
two
az's,
so
that's
good.
What
we
can
do
is
take
these
two
things
together
and
draw
a
picture
of
how
the
how
the
zookeepers
are
actually
distributed,
and
what
we
see
is.
We've
got
two
on
one
host
one
on
another,
and
the
fact
is
that
the
the
distribution
here
is
essentially
random.
A
It's
sort
of
luck
of
the
draw
that
we
got
two
on
one
machine
and
and
one
on
another,
the
fact
that
they
were
just
if
they
had
come
up
as
they
could
have
come
up
all
on
different
machines
or
we
could
have
even
have
had
all
pods
on
on
a
single
machine
and
what
was
actually
happening
here
in
this
particular
case.
Is
it
just
weren't
any
affinity
rules,
and
this
is
so
so
one
of
the
things
that
you
do
want
to
check
for
is.
A
As
I
mentioned
before,
the
syntax
for
affinity
rules
is
non-intuitive.
I
think
that's
a
kind
of
a
kind
word,
so
it's
pretty
easy
to
to
set
things
up
that
look
correct,
but
then
actually
behave
surprisingly,
when
they're
deployed
so
again,
double
checking
the
affinity
rules
and
as
well
as
the
the
actual
implementation
where,
where
things
where
the
pods
actually
land
is
super
important
for
using
them
correctly
and
protecting
your
data.
A
Let's
move
off
affinity
rules
to
another
one
of
my
favorites,
and
this
is
a
problem
that
I
call
the
persistent
volume
that
wasn't-
and
this
is
surprisingly
common.
I've
probably
hit
this
at
least
a
dozen
times
in
different
forms.
Let
me
talk
about
a
couple
of
common
ways
that
this
can
occur
before
we
go
anywhere.
It's
important
to
recognize
that
that
ins
in
kubernetes,
ephemeral
storage
is
not
a
bad
thing.
It
is
a
feature
not
a
bug.
A
That's
in
fact,
one
of
the
things
that
makes
kubernetes
so
powerful
is
you
can
have
different
kinds
of
storage
and
we're
going
to
take
advantage
of
that
in
in
in
just
a
couple
of
minutes.
But
this
is
a
picture
of
two
kubernetes
nodes
and
one
I
call
the
bad
pod.
I
has
a
bad
pod
running
on
it
and
it
has
no.
It
has
no
block
storage.
It's
just
writing
to
the
ephemeral
file
system
in
the
pod
itself,
like
overlay
fs,
the
other
one
is
a
good
pod
and
that's
actually
got
nicely
allocated
block
storage.
A
Now,
what's
interesting
is
as
far
as
the
applications
are
concerned,
they
can't
tell
the
difference
what
kind
of
you
know
what
kind
of
storage
you
have
the
pods
work
in
both
cases,
the
answer
api
calls
so
on
and
so
forth.
Everything
looks
good.
What's
even
more
interested
is
the
pods
themselves
cannot
necessarily
tell
whether
they
have
truly
persistent
storage
because
they
just
see
it
as
a
mounted
in
the
file
system.
A
So
in
kubernetes
to
say
something
is
bad
and
to
say
something
is
good.
Well,
this
is
in
the
eye
of
the
beholder
if
it's
a
web
server.
Of
course
it's
it's
it's
fine.
We
don't
really
care
whether
the
storage
is
ephemeral
or
not.
If
it's
a
database,
of
course,
that
could
be
quite
a
that
could
be
quite
a
serious
problem.
A
A
So
that
I
can
make
sure
that
make
sure
that
we
really
see
what
we
expect
and
when
we,
when
we
look
at
the
pvc,
we
see
that
it
has
a
storage
class
of
cops,
ssd
117,
that's
the
the
storage
class
for
cops,
running
kubernetes
1.17.
That
looks
pretty
good.
Moreover,
under
volume
we
see
there's
a
real
pv
out
there,
so
persistent
volume
has
been
allocated.
We
could
actually
check
that
using
a
get
pv
command
and-
and
look
at
that.
A
A
A
Everything
looks
copacetic,
however,
that's
not
necessarily
that's
not
to
say
that
we're
fully
protected
so
because
persistent
volumes
are
only
good
if
you
actually
use
them.
Here's
an
example
of
a
bug
we
hit
with
zookeeper.
It
fortunately
did
not
lead
to
a
down
system
because
we
detected
it
in
time
and
were
able
to
to
correct
it.
A
But
what
happened
was
due
to
a
misconfiguration
problem
in
our
zookeepers
we
had
the
ebs
storage
mounted
as
var
lib
zk
data,
but
we
accidentally
set
the
configuration
file
so
that
zookeeper
wrote
its
snapshots
to
data
and
it's
logged
to
data
log
in
ephemeral,
storage
and
the
problem.
Is
you
of
course,
can't
tell
that
this
is
happening
until
you
actually
restart
the
zookeeper
node
now
with
zookeeper?
There's
another
interesting
variation
that
it's
not
even
enough
to
restart
zookeeper,
to
see
this
because
zookeeper
replicates
automatically.
A
So
in
this
particular
case,
we
had
an
ensemble
of
three
zookeepers
when
individual
pods
restarted,
they
would
then
connect
back
to
the
other.
Two
pods
get
the
data
back,
and
so
it
would
be
transparent
that
we
had
lost
data.
The
the
zookeeper
replication
behavior
was
hiding
this.
It
was
only
at
a
certain
point
that
we
that
we
actually
accidentally
noticed
that
the
storage
was
not
there
and
realized.
A
We
had
a
real
problem,
so
so
you
really
want
to
be
very
careful
about
this,
and
this
is
an
example
of
an
area
where
the
answer
that
you
know
to
solving
this
kind
of
problem
is
testing
and
a
lot
of
testing.
So,
for
example,
just
be
super
paranoid
check
that
kubernetes
resource
definitions
are
out
there,
as
I,
as
I
showed
before,
look
at
file
system
mounts
make
sure
that
the
mounted
file
system
matches
where
you
exp
is.
A
Is
that
the
you
know
that
the
pvs
are
mounted
exactly
where
you
expect
them
to
and
you're
writing
to
them,
kill
pods,
that's
the
best
way
of.
If,
if
they're
just
single
databases
they,
if
they
lose
their
data,
then
then
you'll
detect
that
you
can
kill
nodes.
You
can
kill
and
restart
all
pods
in
a
replicated
database.
A
Deleting
volumes
is
a
good
thing,
so
take
the
volume
away
if
the
database
is
truly
dependent
on
it
and
it's
pointing
to
it
and
writing
to
it.
The
database
should
stop
working
and
finally
test
with
huge
amounts
of
data.
Many
database
problems
are
revealed
only
when
you
have
a
large
amount
of
data.
In
this
particular
case,
if
you
add
a
lot
of
data,
it
will
fill
the
local
file
system
and
then
you'll
quickly
see
that
you've
got
a
problem
because
you're
not
even
writing
to
the
to
the
to
the
block
storage.
A
So
with
that
we're
we've
shown
four
ways:
you
can
lose
data.
Here's
the
last
one,
which
I
call
fat
fingers
of
fate,
and
this
is
probably
the
saddest
one,
because
really,
if
you
think
about
it,
the
best
way
to
lose
data
is
to
do
it
yourself,
and
the
virtue
of
kubernetes,
of
course,
that
we
can
create
things.
Has
this
mirror
image
that
we
can
or
sort
of
as
equal
and
opposite
effect,
that
we
can
take
them
away
just
as
quickly
as
we
can
create
them?
A
So
this
single
line
command
will
wipe
out
that
mysql
database
that
I
created
as
an
example
at
the
beginning.
So
that's
it
data's
gone.
So
one
of
the
things
that
given
how
easy
it
is
to
destroy
things
accidentally,
one
of
the
things
that's
worth
thinking
about
is
how
to
prevent
is
how
to
protect
your
persistent
volumes.
So
they
don't
just
evaporate
and
it
turns
out
that
we
can
do
this
through
what
are
called
reclaim
policies.
A
So
we
we
list
the
pvs
here,
and
I
show
this
example
you
can
see
the
policy
is
delete.
That
means
that
when
the
the
persistent
volume
claim
that
caused
the
storage
to
be
created
goes
away,
the
storage
just
vaporizes,
so
that's
bad.
What
we
can
do
if
we
have
existing
pvs
is.
We
can
fix
this
by
changing
the
reclaim
policy
to
be
retained,
and
this
patch
command
shows
how
to
do
that.
A
Another
possibility,
if
we
want
to
do
this
in
a
general
way,
is
to
create
a
new
storage
class,
so
the
kubernetes
is
really
flexible,
and
this
is
a
great
example
of
where
that
flexibility
really
works.
For
you.
I've
created
in
this
definition
a
storage
class
which
is
exactly
the
same
as
the
normal
cop
storage
class,
except
that
it
by
default,
uses
retain
as
the
as
the
reclaim
policy.
So
any
any
persistent
volumes
allocated
with
this
will
have
retain
as
their
policy.
A
At
that
point,
if
I
destroy
my
sql
by
accident,
I
can
reclaim
the
storage
just
by
making
a
persistent
volume
claim
that
matches
the
the
volume
that
we're
trying
to
attach
to
in
every
respect,
and
this
is
something
where
you
have
to
be
quite
careful
about
it.
So,
for
example,
you
must
have
the
volume
name
stuck
in
your
persistent
volume,
otherwise
it
won't
match
and
it'll
just
stay
in
a
pending
state.
A
But
if
you
do
this
right,
you
can
recover
storage
quite
nicely
and
in
fact,
here's
the
process
that
you
go
through
to
reclaim
is
you've
blown
it
away
by
accident.
Just
go
in
remove
the
claim
ref
from
the
pv
that
says:
hey
it's
it's
now
free
to
be
reallocated,
go
ahead
and
run
that
definition
again
to
create
a
new
persistent
volume,
and
then
our
persistent
volume
claim
and
then
with
helm.
It
actually
has
a
feature
which
allows
it
to
reattach
to
existing
storage
and
you
get
your
database
back
and
that
solves
the
problem.
A
So,
with
this
we've
covered
five
ways
to
lose.
Data
talked
about
solutions.
I
just
like
to
summarize
by
thinking
not
so
much
about
the
ways
to
lose
data,
but
about
the
things
you
want
to
do
to
solve
those
problems.
So
the
core
of
of
solving
problems
with
data
loss
in
kubernetes
is
just
to
be
exceedingly
paranoid.
This
is
something
that's
true
of
data
in
general
and
then
beyond
that,
there
are
five
specific
things
that
you
want
to
do.
A
First
of
all,
have
replicas
of
data,
always
extra
copies
are
always
good,
separate
them
by
distance
use,
affinity
rules
so
that
you
can
lean
on
the
kubernetes
capabilities
that
allow
you
to
spread
pods
across.
You
know,
sort
of
a
process
across
availability
zones.
This
is
a
great
feature
and
keep
them
from
clustering
on
single
nodes,
use,
reclaim
policies
to
protect
your
storage
and
then
finally,
just
test
the
daylights
out
of
everything
if
you're
in
data.
A
So
that's
it.
I
hope
you've
enjoyed
the
talk.
It's
been
really
fun,
putting
this
together
and
and
and
presenting
it.
I
just
want
to
say
as
a
final
note,
we
are
hiring
like
any
self-respecting
startup.
If
you
like
this
talk
or
think
you
can
do
it
better,
please
give
us
a
call
we'd
love
to
talk
to
you.
Our
own
operator
is
is
shown
here
in
the
github
reference,
so
check
it
out.
There's
many
other
great
operators.
Percona
has
a
great
one.