►
From YouTube: Ceph Day Germany - 10 ways to break your Ceph cluster
Description
Ceph Day Germany
February 7, 2018
Wido den Hollander, 42on
A
A
A
Detected
levels
which
happen
I'm
not
trying
to
discredit
somebody
were
saying,
was
to
be
doing
it,
but
I'm
not
trying
to
learn
new
something.
So
you
don't
make
the
same
mistakes
every
different
system.
There's
no
reporter
in
these
I
just
picked
an
abyss
to
the
things
I
saw,
which
were
said
funny
cases
to
wait.
World
okay,
which
is
you
can
do
so.
Let's
get
started,
I've
got
like
30
minutes,
I.
Think.
A
A
Now
let
okay
so
I
located,
and
it
turned
out
that
they
had
their
backs
enacting
the
crush
madman
sites
up
straight
the
right
location
of
looking
at
depression,
that
there
was
a
mistake
in
the
crash
that
was
a
one-liner,
so
he
Co
is
making
juice
leaders
and
hosts,
although
he
had
the
racks
in
there.
What
the
servers
in
their
course
was
balancing
over
different
posts
instead
of
over
X
well,
eventually
need
to
collect
two
or
three
hours
to
get
the
power
back
on.
A
The
rack
can
get
all
of
hunting
again,
but
they
never
tested
it,
so
they
just
issued
because
they
have
the
racks
in
the
car
that
it
was
okay
to
was
running
well.
It
was
running
uphill,
like
a
machine
interact,
so
nobody
I
was
lost
in
this
case,
but
at
Boston
some
serious
downturn.
Afterwards,
we
had
to
migrate
roughly
500
petabytes
of
a
that
feedback
field
in
that
system,
which
took
quite
some
time
to
go
from
host
event.
So
do
these
deciliters
is
good
before
you
deploy.
Instead
of
assuming
you
will
be.
A
Ok,
they're
done
just
fine
and
yes,
these
things
do
happen,
so
testament
test
of
you,
testing
them
and
no,
we
were
slight
decommissioning.
A
host
all
set
has
all
kind
of
condoms
and
it
does
recovery
when
most
goes
down,
but
the
best
way
to
decommission
a
system
is
to
mark
the
Oh
sneezes,
Alice
and
the
enemy
in
this
case.
A
Well,
he
thought
he
knew
set
well,
so
we
turned
on
the
machine
and
after
a
few
minutes
that
actually
started,
and
it
was
working
fine,
but
a
few
hours
later
to
be
back
for
a
second
is
fail
to
the
system
of
monuments.
Replicas
replicas
later
a
lot
of
examples
with
this
another
disk
fails
and
then
he
was
taken
roughly
her
7
terabytes
of
data
on
the
side,
the
best
bosses
that
was
corrupted,
we
had
to
abandon
assistant
who
liked
it
and
try
again
so
this
video
you
can
have
over
three
replicas.
A
If
you
decrease
in
the
system
do
not
turn
off
the
system,
Marquis
of
Jesus
I
was
way
too
early
to
move
away
from
a
system
and
then
remove
it
from
your
cluster.
Don't
just
shut
it
down
whenever
recovery
ticket,
although
it
seems
to
work,
fine,
you're,
running
integrated
state
anger
just
endanger
your
data.
So
please
do
not
do
it.
Yes,
mark
a
data
set.
A
That
was
fire
flying
hammer
in
fuel
and
the
partition
was
filling
up
on
the
system
because
it
was
doing
a
back
view
and
monitors
database
keep
growing
because
they
keep
a
human
history,
audio
steamapps
and
he
kept
probably
rarely
made
growing
and
the
ending
in
this
case
and
then
example
dust
it
stuff
happens
in
the
real
world,
it
is
pretty
Google.
An
SSD
class
has
had
law
class,
so
he
issued
there
were
lots
of
the
monitors
instead
of
the
actual
database
of
all
class.
So
he's
not
removing
those
files
from
the
monitor
tonight.
A
Car
has
come
on
would
that
make
while
the
whole
system
was
lost,
although
he
can
rebuild
the
monitor
database
using
DoD's,
it's
quite
a
difficult,
painful
task
and
he
makes
me
system
was
abandoned.
What
am
I
stress
here
is
that
if
your
system
is
not
being
in
fact
still
for
a
very
long
time,
the
monitor
database
will
keep
growing.
It
comes
better
balloonist
and
there
are
patches
out
there
to
make
it
even
better,
but
to
keep
it
short.
The
monitors
history
keep
a
longer
easier.
A
The
system
just
a
few
days
ago,
I
have
a
system
which
is
doing
it
back,
though
going
from
class
or
to
boost,
or
it
has
been
back
then
for
eight
days
now,
and
one
of
the
databases,
crew
for
the
two
gigabytes
for
45
bucks.
We
sell
planning
their
system
this
year.
There's
enough
space
available
in
your
monitors
and
as
her
system
grows,
the
monitor
databases
will
grow
with
them.
So
honorable
service,
doesn't
that
wasn't
pretty
small
plan
for
having
more
Haggai
in
this
monitor
databases?
A
So
if
you
can
sustain
that
very
long
backfield,
which
might
be
happening
where
you
have
you
migrating
from
passports,
mostly
something
to
keep
in
mind,
they
might
be
statistic
space.
This
is
not
talked
about
which
is
coming
back
though
we
moved
it.
One
part
right
here:
networks,
that's
where
it's
my
niche
is
enough:
space
removing.
A
A
If
you
just
hide
from
who
the
person
missing
a
thought,
the
pool
was
empty,
it
wasn't
used
either
the
double
check
of
didn't
ask
his
colleagues
and
if
you're,
just
just
12
terabytes,
we
can
imagine
for
somebody
so
they're
much
could
be
a
lot
of
vital
information.
So
what
I
recommend
is
that
you
always
said
that
note
that
these
back
on
there's
a
flag
to
the
central
pool
preventing
it
to
be
deleted
and
previously
luminous
there
was
a
stereo
setting
one
allah-allah
me.
A
It
was
set
to
true
by
the
consulting
and
joel
and
it's
now
all
smooth.
So
you
have
to
go
through
a
lot
of
Hoops
removing
the
pool,
but
just
just
just
be
cautious.
If
your
removable
chemical,
you
get
somebody
else
also
to
get
it
see,
you
want
to
remove
this
one.
Yes,
so
if
you
get
a
cool
gooey
bike,
open
attic,
you
just
have
a
second
person.
A
Yes,
but
I
think
that,
because
the
DG's
are
expiring
idea
tools,
so
any
existing
client
will
just
be
talking
to
say
feces
even
notice
needed.
It
won't
mean
fines
which
are
starting
up
and
then
result.
We
accept
renamed
my
but
I
still
recommend
that
you
do
a
double
check
their
colleagues
and
yes,
it's
not
happen,
so
be
cautious.
A
Well,
setting
you
know
how
to
flag
for
a
long
time.
Does
everybody
know
what
to
know
how
flag
means
or
who
doesn't
know
what
no
I
was
like
these
few
message.
Let
me
explain
anyway:
if,
if
there
was,
he
goes
down
after
five
depended
more
ten
minutes
depending
on
the
version
of
secular
money
and
honesty
will
be
marked,
as
cuts
means
the
thing
that
starts
to
migrate
away
from
the
dirty
or
dusty.
It's
not
participating
in
different
lives
with
anymore,
so
Thanksgiving
and
recovery
kicks
it
to
fix
it,
but
some
people
have
these
usually
scrubbing.
A
So
what
they
did
was
they
said
that
those
Iraqis
use
about
black
oysters
the
system
into
the
health
warning
and
if
you
said
to
know
how
to
back
it
goes
to
health
warning
as
well,
but
these
techniques
were
used
for
having
a
system
running
a
warning
station.
So
nobody
noticed
that
you
know
out
of
cycle
says
that
there's
something
called
minimal
size
percent,
which
is
the
amount
of
the
replica
the
online
for
a
placement
for
a
mr.
al.
A
Although
there
are
many
would-be
replicas
minimal
size,
the
sets
of
one
so
over
time
on
this
fails
actually
never
started.
A
second
disability
actually
never
started
a
third
disc
fails
and
there
you
go
so
don't
get
used
to
running
the
system
in
health
warning
us.
In
addition,
it
will
size
she
resented
you
on
any
system,
but
just
don't
get
it
used
to
health
board
and
I
see
a
lot
of
systems
out
there,
which
digests
permanently
could
help
warn
and
people
are
just.
A
A
Welcome
I'm
TX
a
test
if
you're
still
using
possibly
are
still
using
if
you're
using
class
or
probably
a
lot
of
you
are
using
pod
story,
there's
something
called
no
barrier
and
people
who
are
just
seeing
the
problems.
They
start
to
believe,
and
they
see
somebody
saying
on
some
random
page
saying
that
using
a
no
barrier
option
easy
with
the
fergus
approvement,
so
people
start
applying
that
no
barrier
option
well
right,
very
there's,
not
something!
That's
a
safeguard
for
data
and
mustard.
This
case
was
it.
A
It
happened,
was
matter
but
no
barrier
and
there
was
a
ground
failure
in
the
datacenter.
So
there
wasn't
real
power
failure.
Power
fishnet
stayed
up
with
all
the
crimes
failed,
so
in
that
case
all
power
went
out
for
all
sisters
and
the
whole
system
just
crashed.
Well,
he
came
back
with
all
kinds
of
defects,
investor
file
systems
and
we're
never
able
to
recover
the
system.
It
was
just
way.
A
A
It
smart
people
actually
anymore
right
back,
cashing
on
their
good
seller
without
ever
doing
that,
every
backup
unit
they
don't
have
what
the
right
that
I
see
does,
but
it's
just
a
black
state
right
back
turn
on
some
trouble
refused
to
the
neighborhood
without
a
battery
backup
unit
do
and
again
here
we
go
the
power
failures,
although
you
seem
to
happen
as
one
datacentres.
If
you
happen
in
this
case,
there
was
only
one
act
which
were
done,
so
no
data
was
actually
lost
because
you
still
had
replicas
other
decks.
A
It
was
fine,
but
be
cautious
if
you
have
right
back,
enabled
also
mixture.
If
you
can
push
over
at
the
battery,
is
it
sometimes
happens
that
seems
to
talk
about
the
power
so
ii
never
replace
their
regularly
or
like
every
two
or
three
years
or
something
so
do
not
anymore
right
back
action
unless
you
have
a
better
right
there,
while
creating
20
placement
groups.
This
more
did
somebody
now
already
said.
A
Luminous
has
the
new
feature
called
EG
overdose,
I
think
it
was
tandem,
told
it,
but
eg
overdose
that
are
again
seem
to
be
creating
too
many
Bee
Gees,
but
in
the
earlier
days
of
set
and
meaning
those
systems
are
still
running
average
people.
Even
Firefly
systems
are
running
out
there.
My
configure
the
system
for
customer
whether
Dante
places
groups
and
they
need
some
additional
pulses
and
pessimistic.
Accomplish
my
configuration
occasion,
rules
I
get
wrinkles
when
client
and.
A
Rabbits,
they
keep
happening
starting
the
overseas
again.
They
started
as
soon
CP
human,
a
memory
just
snowball
effect
of
the
cluster
going
down
going
down,
and
eventually
you
start
using
memory
and
while
babysitting
at
court
week,
we
got
the
system
up
and
running
again.
My
warning
here
is
be
cautious.
Replacement
groups
are
very
happy
at
going
towards
mimic.
We
might
have
eg
virginity
to
each
other.
A
Model
doing
stuff
for
us,
but
if
you
still
running
something
with
a
hammer
or
jewel
or
you
know,
good
luminous
check
the
amount
of
you
G's,
you
have
their
number
three
and
you
will
run
into
most
of
the
time
PPG
issues.
If
you
have
a
serious
issue
to
be
rebooted
or
you
are
powerless
with
a
network
problem,
then
it's
time
to
review
those
needs
and
they
can
consume.
B
A
A
This
fact,
after
this
keyboard
yeah
there
are
all
kinds
of
things
that
you
can
do
to
refer
to
place
in
the
group
and
say
use
the
old
version
ignore
history,
but
eventually
you
will
corrupt
data
and
I
have
customers
saying
to
me.
Otherwise.
My
my
use
case
won't
wear
a
corset
because
it
becomes
too
expensive
and
I
said
well,
yeah,
that's
it!
Then
you
should
use
something
else.
A
A
A
Have
seen
people
using
USB
sticks
day,
they
use
McDowell
server,
which
has
SD
parts
in
there.
They
figured
life
fits
on
that
so
a
gun.
Well,
the
SD
card
works
out
pretty
quickly.
If
you
look
at
it
technically,
the
moment
are
synchronized
a
database
constantly
with
each
other.
All
synchronize
rights,
you'll
burn
through
the
SD
card.
Pretty
quickly
so
use
reliable
right,
like
I,
said,
use
an
intensity.
A
Well,
the
bigger
t
system
becomes
I
all
recommend
using
a
big
recipe,
just
to
make
sure
that
the
database
of
amount
of
bits
out
there
that
help
to
monitor
really
wasteful.
If
outdated.
The
moment
you
create
the
backup
so
there's
no
way
to
back
up
onto
databases
for
subsystems,
I,
never
recommend,
instead
of
having
three
monitors
and
use
different
brands
of
for
hardware,
so
that
you
have
a
year
at
risk
of
spreading.
A
The
risk
suggestions
again
set
up
a
recipe
behind
them,
but
they
say
all
good
things
go
to
11,
and
this
is
something
which,
when
I
happened
to
meet
that
night,
I
assisted.
It
was
actually
somebody
before
I
came
into
the
office
needed
to
some
hydration
digger.
He
didn't
set
before
he
guessing
and
reset
that
it
might
see
the
issue
or
the
key.
We
were
gonna,
be
having
a
new
rule
to
the
system
and
the
ability,
without
looking
at
the
key,
very
well
and
by
accident,
he
removed
the
right
fit
throughout
the
volume
school.
A
If
the
credentials,
and
although
the
new
voting
system
people
was
added
to
the
list,
70
starters
say:
there's
no
person
to
write
on
the
system
and
that
causes
some
real
issues
with
our
media,
the
PM's
about
2000,
and
there
was
a
lot
of
uncles.
We
get
from
customers
saying,
but
at
the
end
was
done.
It
wasn't
not
only
to
be
honest
and
what
we
had
corrupted
easy
for
journals
as
the
best
process
with
refusing
to
manage
all
kinds
of
stuff.
A
I
was
doing
notes
at
work
of
all
day
with
experience
so
because
we
are
updating
these
and
if
there's
a
good,
related
here's
what
we
got
JEP
before
it
comes
becomes
visible
to
all
clients.
So
if
your
system
goes
down
behind
OpenStack
or
something
the
IRS
simply
block,
that's
fine
the
excess
weight,
but
in
this
case
the
grey
house,
they've
already
start
spitting
arrows
to
get
on
top
saying
permission
denied
permission
denied
permission
denied
so
because
of
any
piece
double
check.
The
double
test
authority
rejected
a
victim
because
it
might
great
birth
system.
A
A
D
A
So
far
in
this
case
so
yeah
the
month,
which
you're
a
student
of
history
of
scales
with
the
amount
of
policy,
so
the
Google
Columbus,
let's
say
150
BG's
roasty,
sometimes
like
about
200
only
that
system
I
expect
within
reasonable
timeframe.
But
this
customer
he
has
a
except
opposed,
is
running
and
they
were
designed
to
run
with
800
G's
and
he
wanted
to
test
something
in
addition
to
what
we
had
running
his
application.
A
D
A
Was
asking
some
of
them
to
be
splitting
a
feature
is
increasing,
is
emerging
is
something
which
is
on
the
horizon.
It's
age
that
was
working
I
saw
one
I'm,
not
sure
yet,
but
first
what
we
did
to
testicles.
They
didn't
contain
any
data.
So
all
my
main
concern
was
getting
a
system
up
and
running
again,
so
add
additional
CPUs
and
a
memory
to
get
the
smoke
running.
A
We
set
that
down
flag
said
that
the
lungs
where
I
was
shooting
heartbeat
met,
values
of
everywhere
X
made
sure
it
just
called
them
and
then
eventually
waited
to
it.
Please
take
again
I
remove
those
additional
pools,
go
see,
strip
the
Bee
Gees
away
so
lemony.
It
was
not
even
increasing
a
PG
number
for
the
existing
pools,
but.
A
I
happens
in
the
movie
geez.
What
I
find
is
there's
very
difficult
for
people
to
understand
the
refugee
is,
although
you
senator
pool
it,
has
an
influence,
austere
white,
so
we
think
that
they
should
use
the
same
million
for
every
pool.
So
it's
still
a
very
difficult
topic
revenues.
You
actually
understand
a.