►
From YouTube: Ceph Month 2021: 5 more ways to break your ceph cluster
Description
Presented by: Wout van Heeswijk
Full schedule: https://pad.ceph.com/p/ceph-month-june-2021
A
A
A
One
case
we
had
a
script
that
was
run
in
front
of
the
automation,
so
automating
the
automation
tool
and
through
all
of
that,
some
checks
were
missing
and
it
didn't
check
for
the
amount
of
monitors
that
was
set
and
it
just
didn't
fill
out
that
variable
overruled
the
existing
configuration
applied.
It
said
everything
is
okay,
yes
to
all
and
started
removing
the
monitors,
and
this
tool
is
also
pretty
thorough,
so
it
also
removed
all
the
monitor
databases
and
everything
there
was
no
trace
of
them
anywhere.
A
A
A
Another
one
is
the
size
is
two
was
the
previous
one,
but
we
kind
of
revised
that
to
the
min
size
is
one
we,
we
still
recommend
that
any.
In
any
case,
you
always
run
with
sizes.
Three,
if
you
value
your
data
or
do
something
with
the
ratio
coding,
but
we'd
rather
not
see
sizes
two,
and
if
we
see
size
is
two,
then
please,
please
don't
go
with
min
size
is
one.
A
A
We
also
found
an
interesting
case,
a
couple
of
interesting
cases
where
people
didn't
complete
the
update
in
the
chef
documentation,
there's
an
update
guide,
but
it's
it's
not
specific
to
a
release.
It's
a
generic
update
guide,
especially
with
the
upgrades
to
nautilus
and
the
messenger
version
version
2.
A
Yes,
we
saw
that
that
there
was
some
discrepancies
and
some
parts
of
the
upgrade
were
never
executed
and
miraculously
the
cluster
will
survive
for
a
some
time,
but
then
it
will
break
and
it
will
break
badly
for
people
and
it's
very
hard
to
troubleshoot.
If
you
don't
know
what
you're
looking
for.
So,
what
you're
looking
for
is
something
that
looks
like
this
continuously
for
a
long
time
so
and
the
solution
is
to
finish
the
update
finish,
all
the
steps
for
updating
to
nautilus
or
octopus.
I've
seen
a
cluster
that
survived
into
octopus.
A
A
Was
we
took
a
while
to
figure
this
one
out,
so
we
had
a
customer
that
was
having
problems
with
with
their
rgw
environment
and
after
a
while,
we
we
saw
that
the
ip
addresses
in
the
service
map
for
the
rtw
service
only
it
showed
three
rgws
and
the
customer
said.
I
have
nine
rgws
and
we
saw
the
ip
addresses
in
the
map
updating
all
the
time
now.
A
A
A
We
have
one
bonus,
one
which
is
blindly
trusting
the
pg
auto
scaler.
So
I
think
a
lot
of
this
has
to
do
with
chef
becoming
much
easier
to
use.
So
we
see
much
more
installs
and
you
don't
have
to
understand
or
know
as
much
as
you
used
to,
but
we've
seen
that
that
clusters
are
installed,
we're
all
with
all
the
defaults
but
they're
reasonably
large,
and
then
they
start
ingesting
data.
A
And
that's
when
the
pg
autoscaler
comes
in
and
says
right
now.
We
need
some
more
placement
groups
here
and
then
it
starts
splitting
placement
groups
and
then
you
start
ingesting
more
data
and
then
it
says,
oh
then,
you
need
some
more
placement
groups
so,
and
this
was
a
cluster
that
was
in
the
almost
a
petabyte.
A
So
we
also
looked
at
the
original
ten.
That
widow
did.
I,
I
urge
you
all
if
you
haven't
seen
that
one
also
look
at
the
original
10
ways
to
break
yourself
cluster,
but
from
the
original
10
there's
number
six
that
we
don't
no
longer
consider
a
way
to
break
yourself
cluster
because
it's
been
superseded
by
blue
store
exit.
We
don't
use
xfs
that
much
anymore.
A
C
I
have
a
question:
is
dan
here,
hey
dan
yeah
hi,
so
we've
luckily
never
had
to
restore
mons
from
the
osds
like
we've
never
lost
all
our
mons.
I'm
just
curious.
If
you
I
know
there
is
a
documented
procedure
somewhere
on
the
seth
docks
yeah
like
this.
Is
that
what
you
followed
is
it?
Does
that
work
still
work
yeah
that
works
okay,.
A
It
takes
a
little,
it
takes
a
little
time,
but
it
works,
but
it
it
copies.
The
data
of
the
database
over
to
from
the
osd's
adds
it
to
the
database,
then
so
this
the
lucky
we
were
very
lucky.
This
was
a
very
young
cluster,
so
there
was
not
too
much
data
to
scrape,
but
I
think
if
your
cluster
grows
and
your
your
usage,
you
use
it
much
longer
much
more
changes,
then
that
process
will
also
take
much
longer.
C
A
No,
no
yeah,
you
can
back
them
up,
but
the
data
is
is
always
you're,
always
behind
enough
that
it
doesn't
make
sense.
We've
also
seen
I
could
have
included
that
one
too
we've
we've
seen
that
somebody
had
a
virtual
install
and
they
had
some
interesting
data
to
them
on
a
virtual
chef
install
and
somebody
was
updating
and
then
the
updates
didn't
go
very
well
and
he
just
rolled
back
the
virtual
machine,
all
different
states.
A
Can
also
encounter
yeah.
That
was
the
sound
I
made
as
well.
Oh
so
yeah
there's
no
real
backup
strategy
there.
What
we,
what
we
try
to
do
is
just
monitor
your
mons
very
well,
make
sure
that
you
have
enough
of
them
better
on
maybe
two
different
hardware
fenders,
so
that
you
are
unlikely
to
encounter
the
same
problem
on
on
all
of
the
monitors
but
yeah.