►
Description
Failing Better - When Not To Ceph and Lessons Learned - Lars Marowsky-Brée, SUSE
Talks on where and how to utilize Ceph successfully abound; and rightly so, since Ceph is a fascinating and very flexible SDS project. Let's talk about the rest.
What lessons can we learn from the problems we have encountered in the field, where Ceph even may have ultimately failed? Or where the behaviour of Ceph was counter-intuitive to user expectations? And if so, was Ceph suboptimal or the expectations off?
Drawing from several years of being the engineering escalation point for projects at SUSE and community feedback, this session will discuss anti-patterns of Ceph, hopefully improving our success rate by better understanding our failures.
About Lars Marowsky-Brée
SUSE
Distinguished Engineer
Berlin Area, Germany
Twitter Tweet
Lars works as the architect for Ceph & software-defined-storage at SUSE. He is a SUSE Distinguished Engineer and represents SUSE on The Ceph Foundation board.
His speaking experience includes various Linux Foundation events, Ceph Days, OLS, linux.conf.au, Linux Kongress, SUSECon, and others. Previous notable projects include Linux HA and Pacemaker.
Lars holds a master of science degree from the University of Liverpool. He lives in Berlin.
A
A
Yells
at
cloud
talk,
so
a
marketing
said
that
would
not
be
a
good
title
so
that
it's
not
what
the
session
is
right,
but
instead
I
would
like
to
talk
about
the
limits
and
the
constraints
of
Ceph,
so
we
can
have
a
better
understanding
of
where
staff
actually
does
make
sense
and
how
that
translates
to
you
using
it
in
reality
and
also
some
some
patterns
that
we
have
noticed
with.
Customers
share
some
war
stories
where
we
have
seen
people
slightly
deviate
from
house.
A
F
should
be
used,
and
why
I'm
talking
about
this,
so
my
name
is
las.
Most
people,
I
I'm,
working
at
Sousa
and
I'm
kind.
As
the
reasons
he's
got
into
the
Ceph
parts,
I
mean
the
actual
reason,
that's
obviously
sage,
but
at
Sousa
that
was
probably
my
fault
and
I
also
represent
Sousa
on
the
set
foundation
boards.
These
days
and
ultimately
many
of
the
escalations
at
customers
or
many
of
the
high-level
questions
end
up
on
my
desk,
so
I
have
basically
seen
everything
in
the
architecture
where
people
propose.
A
This
is
what
we
could
do
with
self,
or
this
is
how
it
went
terribly
wrong
and
somebody
needed
somebody
to
come
down
on
the
customer
while
an
actual
engineer
went
in
and
fix
the
problem.
So
all
of
these
are
might
end
up
on
my
table
eventually
so
yeah.
So
it
might
seem
odd
to
come
to
a
cephalic
on
and
say.
Well,
you
know
these
are
where
self
may
not
be
the
best
fit
or
where
things
went
wrong,
but
everybody
keeps
telling
you
why
we're
safe
makes
sense.
A
So
maybe
we
should
look
at
the
other
side
as
well.
So,
at
the
end
of
the
day
nothing
I'm
saying
here
is
going
to
go
into
no,
that's
a
no-go
area
for
Saif
right.
That's
not
what
this
is.
All
these
issues
were
overcome
eventually
and
they
were
solved,
but
they
are
simply
different
use
cases
where
you
know
you
have
to
invest
less
or
more
effort
to
make.
A
You
know
get
a
good
return
on
investment
on
set
so,
and
let's
start
this
with
this
slide
right,
when
you
really
need
self,
nothing
else
will
do
right.
There
is
no
other
open
source
project
that
can
handle
that
hyper
scale
scenario
when
you
like,
when
you're,
like
CERN
or
anyone
else,
there's
like
hundreds
of
petabytes
worth
of
storage,
a
massively
parallel
client
workload
and
all
the
things
that
we're
going
to
talk
about
in
a
bit
and
then
there's
just
nothing
else.
Right
chef
is
a
great
solution.
A
The
best,
if
that's
what
you
need
right
so
I'm,
not
here
saying
self,
is
a
bad
idea.
That
would
be
self-defeating
and
I
like
my
bread
on
my
table,
and
so
that's
that's,
that's
definitely
not
what
I
want
so
stuff
is
great,
but
let's
look
at
why
self
is
great
and
where
it
is
great.
So
sage
hinted
at
that
in
his
keynote
this
morning.
A
So
chef
has
been
designed
with
a
few
things
in
mind,
but
before
we
look
at
that,
look
at
what
what
was
there
before
software-defined
storage
right,
the
things
that
weren't
my
bread
on
my
table
before
I
went
to
chef.
We're
literally
the
first
ones
right
h8
clusters
to
notes,
replicated
storage
network
attached,
storage,
exporting
storage
to
a
bunch
of
other
nodes
in
the
cluster
you
know
cause.
Then
maybe
you
had
a
cluster
concurring
filesystem
on
top
of
the
storage
area
network
and,
let's
be
clear,
storage
area
network.
A
It's
nothing
else,
but
a
very
expensive
network
attached
storage
device
with
fibre
channel
interfaces
right,
that's
all
they.
Those
fancy
boxes
are
so
they
sort
of
equivalent
in
a
way
and
then
ultimately,
true,
scale-out,
storage,
right
and
there
it's
like
different
tiers,
and
when
we
talk
about
true
scale
out
storage,
it
sometimes
feels
like
we
have
forgotten
that
all
these
other
approaches
also
exist.
That's
that's
a
bit
weird.
A
So
what
are
the
design
goals
for
safe,
like
it's
literally
targeting
large
hyper
scale,
storage
systems?
It
was
designed
also
at
a
time
where
hard
drives
were
the
most
prominent
feature
of
storage
devices,
and
that's
fine
right.
It
also
values,
correctness
and
a
particular
interpretation
of
consistency.
Over
performance
correctness
is
really
important.
A
Nobody
likes
it
when
the
storage
layer
gets
a
connect
correctness
and
consistency
properties
wrong
right.
That's
it's!
Just
not!
The
storage
losing
data-
it's
not
a
good
place
to
be,
but
there
are
other
consistency
models
that
in
some
cases
can
work
better
like
when
you
look
at
how
we
replicate
object.
Storage,
that's
eventually
consistent
right!
So
when
we
replicate
s3
with
routers
Gateway,
it's
not
immediately
as
tightly
consistent
as
a
local
radio,
slicer
right,
and
so
sometimes
people
who
refer
to
other
consistency
models
as
cutting
corners
and
I.
Don't
think
that's
what
it
is!
A
It's
just
different
approach.
It's
different
kinds
of
guarantees,
your
storage
layer
provides
and
safe
Rados
provides
some
of
the
highest
guarantees
that
you're
used
to
right.
So
they
basically
provides
the
same
semantics
at
a
local.
Correct.
Local
filesystem
would
provide
on
a
single
note
right
and
that
that's
great,
because
those
are
the
semantics.
Your
application
is
used
to
that's
what
you
expect
from
a
ffs
so
that
you
can
just
take
your
regular
filesystem
workload
and
put
it
on
top
of
CFS.
A
There
have
been
other
approaches
that
allow
you
to
scale
better,
because,
honestly,
those
POSIX
filesystem
semantics
memory
map
coherence,
kind
of
tricky
to
scale
we'll
talk
about
that
in
a
second
as
well.
So
really
so,
this
is
basically
where
it
coming
from.
We
were
specifically
targeting
the
use
case
that
very
few
people
of
us
have
at
home
right
so
and
there's
another
trade-off
here.
So
centralization
versus
distribution
centralization
is
very
efficient.
It
also
doesn't
scale
distributed.
Modes
of
decision-making
are
less
efficient
but
scale
so
which
one
do
you
want
right
there?
A
It's
kind
of
tricky
safe,
avoids
most
coordination
for
the
direct
data
paths.
That's
what
crushes
about
right.
Each
client
node
can
compute.
Okay.
Is
that
OSD?
Has
that
block
I'm
not
going
to
give
you
the
fifths,
walk
through
of
crush
at
the
end
of
the
day,
but
that's
basically
the
idea
right.
So
you
can
directly
connect
to
the
OSD
that
holds
your
data,
and
so
there
you
don't
have
to
go
through
it.
A
You
know
single
gatekeeper
or
anything
so
surface
what
we
would
call
a
client
cluster
protocol
where
that's
what
traditional,
NFS
or
I
scuzzy,
basically
client-server
protocols,
right,
which
perhaps
some
some
features
they're
parallel
NFS,
of
course,
is
fully
different
already.
We
can
see
that
even
in
the
safe
architecture-
that's
not
quite
true
right.
Not
everything
is
fully
scale.
A
On
the
Mons,
because
that's
something
you
really
want
very
tightly
coordinated
and
just
on
cue
jawar
walks
in
the
Mons
use
Paxos,
because
that
data
needs
to
be
strongly
consistent
and
it's
actually
centralized
the
maps
are
centralized
right
and
the
metadata
server
force.
F
of
s
is
also
kind
of
centralized.
Because
that's
you
know
the
you
get
ordering
guarantees
that
you
have
to
satisfy
if
you
want
to
be
opposed
accompli
and
file
system,
and
so
that's
not
fully
distributed
easier.we
with
a
multi
MDS.
A
What
actually
happens
is
that
certain
parts
are
scaled
out
and
delegated,
but
ultimately
there
is
a
single
source
of
truth.
Rbd
mirroring
somewhat
familiar
when
you
enable
that,
supposedly
as
in
Kronos
replication
mechanism,
you
have
to
then
make
sure
that
the
data
you
replicate
is
ordered
so
that
you
actually
end
up
with
a
consistent
copy
at
the
other
side,
and
that's
where
that
performance.
It
comes
from
right
and
then
in
some
other
cases
we
have
chosen
to
have
bitwise
identical
mappings
or
copies
of
data
on
several
nodes.
A
Where
is
this
logically
equivalent,
for
example,
when
I
a
pet
when
I
have
like
records
I'm
keeping
track
of
customers
buying?
You
know
customer
a
buys
product.
One
and
customer
me
buys
product
to
those
two
transactions
at
the
end
of
the
day
it
doesn't
matter
in
which
order
I,
safe
them
right,
but
if
you
save
them
to
a
file
or
Rados
object,
you
have
to
have
that
in
the
same
order.
A
A
If
you
had
different
app
an
append
semantics,
you
could
just
keep
okay,
there's
a
new
transaction
appended
to
that
log,
and
you
wouldn't
have
to
you
know,
make
sure
that
they're
all
bitwise
identical
and
that's
why
a
local
system
like
NFS
and
NFS
server,
where
everything
goes
through,
that
one
gateway
can
or
local
node,
mounting
a
local
files
and
they
have
all
the
stadium
memories
I,
don't
need
to
coordinate
with
anyone
else
right.
They
are
perfectly
omniscient
about.
A
What's
going
on
with
all
other
clients,
because
they
are
all
as
a
client
and
NFS
gets
away
with
that
by
not
actually
being
POSIX
compliant
in
for
write
sake,
they
cut
corners,
say
they
claim
one
thing
and
then
deliver
another.
That
sometimes
is
a
bit
tricky,
but
we've
all
gotten
so
used
to
NFS
being
broken
that
it
just
works
in
practice.
A
It's
true
like
a
lab.
Would
so
and
that's
also
where
it
comes
from.
When
we
look
at
you
know,
customers
go
well.
My
system
is
freezing
when
an
error
happens
and
recovery
is
slow.
Well,
yeah
chef,
you
know
you
have
by
default
all
copies,
even
with
replicas
erasure
coding
doesn't
matter.
All
copies
are
owned
as
a
note
right.
They
all
go
through
that
slow,
relatively
slow
network
interface
right
and
locally.
When
I
pull
the
hard
drive.
The
controller
tells
me
your
hard
drive
has
gone
when
an
OST
goes
down.
A
It's
not
answering.
Is
it
coming
back?
I,
don't
know,
let's
wait
another
second,
so
you
have
in
distributed
system.
Safe,
is
not
alone
in
this
right.
Every
distributed
system
has
this
pacemaker
at
the
same
problem.
We
have
to
wait
until
we
can
declare
note
down,
and
then
we
have
to
reach
consensus
about
that
node
being
down,
and
then
we
have
to
do
recovery,
maybe
fence
it
depending
on
what
kind
of
guarantees
you
are
satisfying.
A
So
you
pull
a
drive,
and
this
is
our
node
and
you,
you
know
your
I
Oh
stops
for
a
second
or
two
or
depending
on
our
the
long
so
distributed
systems
and
real-time
are
not
perfectly
compatible
under
all
circumstances.
So
there
are
different.
You
know
different
use
cases
and
about
that
Nefer
network
traffic.
When
you
look
at
erasure
coding,
we
have
one
model:
that's
called
locally
redundant
codes,
local
redundant,
erasure
codes.
A
Maybe
we
need
to
look
at
that
in
some
more
detail
because
we
set
volume
we
could
just
like
make
it
all
one
big
world
and
if
they're
always
d
actually
scaled
well,
we
wouldn't
actually
need
to
have
so
many
different
strats.
So
we'll
see.
But
you
know
this
is
something
that
actually
matters
in
practice
right
so
take
away.
A
If
the
anti
node
goes
down,
then
you
have
system
in
your
problem
and
then
there's
another
design,
choice
that
sometimes
come
up
when
people
compares
against
other
storage
solutions
out
there,
SEF
replicates
and
recovers
on
the
back
end
right,
especially
the
front
end
replication.
So
the
client
only
ever
talks
to
the
primary
OST.
A
Some
other
solutions
do
this
in
the
front
ends
the
client
actually
talks
to
multiple
copies
of
the
data,
it's
very
difficult
to
determine
which
one
is
a
better
solution,
because
they
have
very
different,
true
trade-offs,
like
in
our
case
it
means
we
get
to
use
different.
We
get
to
use
a
beefier
back-end
network,
that's
dedicated
to
all
this,
so
it
helps
with
throughput
if
you
have
proper
sizing.
A
However,
if
you
have
a
single
threaded
workload
on
the
other
work
with
a
single
client,
that
client
can
load
the
lands
and
get
press
a
possibly
better
performance,
it
all
evens
out
when
the
network
gets
saturated,
then
you're
celebrating
your
network
and
you
know
all
clients
sort
of
gets
the
same.
Then
you
know
again
different
trade-offs
right.
So,
if
you're,
comparing
against
the
solution
like
that,
then
keep
in
mind
that
this
is
why
safe
does
it.
This
simplifies
it
because
safe
the
primary
OSD
knows
that
it
has
a
good
copy
of
the
data
right.
A
The
same
algorithm
and
be
pointed
is
the
right
server,
but
when
you
have
few
notes
yeah
or
a
few
drives,
then
pseudo
random
isn't
always
perfectly
balanced,
especially
when
you
then
have
large
objects.
So
at
one
point
this
all
relies
on
the
idea
that
our
higher-level
objects
and
workloads
actually
striped
at
the
Rados
level.
Right
at
one
point,
we
had
a
customer
uploading.
A
100
gigabyte
object
with
Rados
put,
which
you
know
works
but
crush
does
not
does
not
agree.
Right
crush
does
not
like
that
right.
A
A
If
you
haven't
been
in
there,
there's
a
lot
of
work
happening
here
with
with
both
PG,
auto
scaling
and
balancer.
That
makes
us
better,
but
especially
when
you
have
small
environments
and
can
still
be
a
problem
which
brings
us
basically
to
to
how
people
start
with
trying
self
right.
Well,
I
did
a
four
node
POC
and
SEF
underperformed,
and
you
know
because
I
just
was
trying
it
with
my
Windows
clients,
and
you
know
and
yeah.
That's
that's
not
the
best
use
case.
A
What
said
right,
it
works
all
the
functions
are
there,
but
you
get
basically
none
of
the
benefits
right
yeah.
Your
workload
will
typically
have
low
concurrency.
You
have
few
storage
devices,
especially
if
you
have
hard
drives
that
you
grab
from
somewhere,
because
if
it's
a
POC
proof
of
concept
implementation,
you
have
bad
data
balancing.
You
have
an
impact
of
co-locating
all
the
services
on
those
three
for
two
notes.
Even
you
have
so
fairly
thick
layers
of
Ceph
that
eat
into
that
and
little
of
the
gains.
A
So
you
see
all
the
pain
and
little
of
the
benefits
right.
So
basically,
what
this
means
as
economy
of
scale
you're,
not
getting
it
right.
This
is
all
of
a
simplification.
It's
not
like.
We
did
a
formal
study,
but
you
have
a
high
entry
fee
when
you
want
to
deploy
surf
right.
So
you,
your
first
set
cost
is
like
this
and
then
every
other
node
is
like.
So
it
makes
not
a
lot
of
difference
whether
you
deploy
a
hundred
nodes
or
five
right,
yeah
a
little
bit
of
an
oversimplification.
A
So
once
you
have
all
the
hardware
in
place,
which
is
the
hard
part
that
other
people
do,
I
only
do
the
software
at
that
point
in
time
you
run
your
automated
installer
and
whether
you
point
that
at
100
node
or
five
nodes,
it's
really
not
it's
a
runtime
changes,
but
the
effect
is
the
same
right.
So
safe
is
one
of
those
cases
that's
inverse
to
selling
razorblades
right.
It's
you
know
you
buy
a
razor
and
then
the
other
blades
are
more
expensive
right.
The
first
shot
is
not
free
with
Seth
right.
A
So
that's
just
something
to
keep
in
mind
when
you're
managing
expectations.
When
you
think
that
five
node
cluster
will
scale
linearly
to
a
hundred
nodes
right,
you
went
through
those,
oh
man,
all
the
trouble
with
those
five
nodes
and
I'm
never
going
to
do
that
20
times
as
much.
It's
not
true
right.
It
may
be
same
amount
of
work
or
twice
as
much
so
I.
Don't
know
it's
very
hard
to
say
exactly,
but
it's
you
know
it's
F
at
scale.
It's
actually
quite
easy
to
operate
so
I
bring
me.
A
Let
me
bring
out
my
next
topic
right,
big
data
right
surface
of
all
about
big
data.
Let's
be
real.
Your
big
data
is
not
big
right.
So,
if
you
look
at
what
kind
of
capacity
can
hold
on
a
single
node
with
redundancy
included
these
days,
you're
looking
at
something
half
a
petabyte
worth
of
data,
that's
something
you
can
easily
fit
into
a
single
chassis
right.
It's
no
problem.
A
Ssds
get
similarly
dense
and
cheap
and,
of
course,
they're
trade-offs
about
the
network
bandwidth
you
have
and
the
scalability
of
it
all,
but
on
big
data
right.
Occasionally
somebody
comes
up
and
says:
well,
I've
picked
a
turd
because
I
have
you
know
this
unnamed
in-memory
database,
which
is
really
a
lot
of
data
and
I
go
like
well.
How
much
data
do
you
have
and
I
go
20
terabytes
and
go
like
here's,
your
hard
drive?
So
it's
not!
That's
not
it's
not
a
good
use
case
for
self
right.
A
Nobody
is
well
served
by
trying
to
build
a
safe
cluster
that
serves
all
of
20
terabytes.
It's
a
different
solutions
out
there
that
serves
us
better
right.
Maybe
there's
still
a
use
case
because
you
want
all
the
scalability.
This
is
just
how
you
start
get
its
growth
trajectory
of
all
the
protocols
you
want
and
your
data
is
growing
right
and
that's
when
you
typically
will
see
a
slide
like
this
right.
Mobile
data,
IOT,
transactional
data,
medical
emails,
videos
right
and
then
exponential
growth
right
and
that's
true.
Data
is
growing.
A
My
own
data,
I've
noticed,
is
growing
more
like
linearly
and
not
exponentially.
At
least
the
data
I
have
access
to
not
to
one
everybody
keeps
about
me.
So
I,
don't
know
how
that
adds
up
and
if
you
are
like,
if
you
have
a
growing
customer
base,
that's
growing
linearly
and
for
each
of
them
their
data
is
growing
linearly.
A
The
compound
effect
may
as
well
be
exponential
right
that
that's
probably
true,
but
I
got
some
statistics,
so
these
statistics
actually
go
back
to
all
since
ninety
eighty,
and
also
the
pricing
statistics
turns
out
that
basically
price
per
capacity
has
been
dropping
at
steadily
28:30
percentage
since
then,
so
it's
almost
like
Moore's
law
capacity
is
getting
cheaper
and,
at
the
same
point
in
time,
drives,
are
getting
about
that
much
larger
right.
So
unless
your
data
is
growing
faster
than
30
percent
year-over-year
roughly
it
will
not
outgrow
that
chassis.
A
You
can
currently
fit
it
in
right.
It's
just
something
to
keep
in
mind,
and
maybe
your
data
is
growing
faster.
We've
seen
that
many
customers
have
said,
especially
when
you're
keeping
medical
data
life
of
patient
plus
40
years.
Yes,
you
need
a
scale
out
solution
right
so
but
then,
okay,
okay,
your
data
is
growing
all
this
and
then
you
want
your
workload
to
scale
workload.
Considerations.
Does
it,
though,
if
you
just
take
a
traditional
workload
and
throw
it
at
SEF,
probably
not
going
to
perform
very
well
single
threaded
workloads
are
terrible.
A
You
know
millions
of
objects
and
a
flat
indexable.
This
table
data
structure
do
not
scale
well,
sometimes
I
do
if
you
avoid
operations
that
actually
require
the
system
to
sequentially
enumerate,
all
of
them
consistently
POSIX
file
system,
semantics,
don't
scale
legacy
workloads
that
struggle
with
a
different
guarantees
of
object.
So
when
you
have
you,
when
you
have
a
dr
concept
for
yourself,
cluster,
first
shout-out
to
you,
you're
awesome,
but
RB,
DCFS
and
others
gateway
have
different
consistency.
A
Guarantees
when
you
replicate
them
right,
so
you
have
to
your
workload,
has
to
cope
with
the
fact
that
different
data
pools
are
probably
not
at
the
same
same
level
of
consistency.
That's
already
challenged.
So
perhaps
your
workload
fits
on
a
single
note.
In
that
case,
maybe
reconsider
you
know.
Maybe
a
different
solution
is
best
and
again
start
area
networks
or
network
attached.
Storage
can
be
great.
If
you
have
you
know,
single
node
can
hold
your
data
and
you
have
different
worker
nodes.
A
You
can
scale
that
out
fairly
well
to
say
sixteen
hundreds
of
maybe
even
server
to
client
if
that
server
provides
sufficient
bandwidth
and
make
no
mistake
a
single
chassis
these
days
with
a
syn
layer,
storage
system,
you
can
get
400
gigabit
of
bandwidth
out
of
a
single
server.
I
mean
have
seen
the
benchmarks
and
I've.
Seen
that
happen
in
practice
is
that
you
have
a
low
protocol
overhead
that
server
can
deliver
like
40
gigabit
gigabyte
per
second
right.
That's
pretty
amazing,
quadruple
bonded
100,
gigabit
NICs,
pretty
fast.
So
but
yeah.
A
You
want
to
build
that
scalable
network
attached
storage
system
with
Seth,
because
Seth
I
scuzzy
NFS
sifts.
It's
all
there
right
and
that's
true,
but
gateways
right.
So
with
these.
These
are
not
cluster
where
client
protocols
for
the
most
part.
Some
of
that
it's
changing
parallel
NFS
sip
scale-out
is
causing
moldy
parsing.
But
then
you
know
it's
all
there,
but
the
least
you
get
is
an
additional
network,
hop
which
impacts
your
individual
I/o
request.
Latency
throughput
is
fine.
A
You
know
through
port
eventually
will
scale
up,
but
you
have
set
additional
latency
hop
and
especially
when
that
goes
to
the
MDS
for
metadata
operations.
Keep
that
in
mind,
and
it
negates
quite
a
bit
of
safe,
smart
right.
So,
yes,
we
can
do
all
that
and
they
use
cases
where
that
makes
sense,
because
surf
is
cheap
to
scale,
and
you
know
you
can
build
a
nice
cozy
gateway
that
has
hundreds
of
petabytes
of
storage
behind
it.
You
know-
and
maybe
you
have,
that
separated
we
had
different
clients
and
then
it
works
fine
and
scales.
A
But
you
know
just
keep
that
in
mind:
hyper-converged
infrastructure,
yeah,
so
SEF
does
all
the
protocols
I'm
not
going
to
go
into
the
security
considerations
too
much
because
I
don't
want
to
annoy
CPU
Wenders
when
you
call
OK'd
multi-tenancy
on
your
environment,
but
chef
does
not
really
do
data
locality
right
safe
is
you
know
the
data
is
distributed,
see
you
to
randomly
in
your
cluster.
Ratos
classes
could
be
a
way
around
it,
but
it's
tricky
on
so
it
basically
you
get.
A
A
So
first,
there
are
several
places
where
you
could
be
right:
local
to
the
workload
like
client
side.
From
our
cluster
perspective,
there
could
be
a
cached
here.
I
have
words
about
cash
tearing.
You
could
be
cashing
at
sawasdee
levels.
Oh
Steve
has
an
in
a
memory
cache
right
or
even
below
the
OSD
level.
With
you
know,
DM
cache
built
into
album
into
LVM
volume,
set
volume,
sorry
be
cash
or
the
Intel
I
care
solution
and
on
that's
all
fine.
You
know
that
all
helps
well.
A
That's
first
look
some
of
those
questions
right,
so
people
app
caches
because
I
want
better
performance
and
you
know
they
add
a
cache
and
then
they
run
a
random
workload
against
it,
and
it's
not
faster
well
yeah,
because
you're
not
actually
accessing
any
data
twice.
You
know
it.
You
know
you're
not
getting
any
reap
benefit
from
that,
and
not
even
over
rights.
Don't
get
combined
because
you're
only
hitting
each
block
wants
also
inverse
I
benchmark
everything.
It
was
great,
but
my
actual
workload.
A
A
Yeah,
so
those
are
all
variations
of
basically
benchmarking
cash,
it's
wrong
because
eventually
they
will
fill
up
an
overflow
and
initially
they're
empty,
and
then
rights
are
fast.
If
you
have
a
write-back
cache
and
REITs
are
just
as
fast
as
before,
and
eventually
when
you
keep
throwing
random
data
at
your
cache,
writing
eventually
will
slow
down
because
it
has
to
flush,
and
at
that
point
you
have
both
data
coming
in
and
data
being
pulled
out.
Cash
eviction
is
one
of
the
three
hardware
problems
in
computer
science.
So
eventually
there's
some
overhead.
A
So
at
best
at
that
point
it
wouldn't
slow
you
down,
but
in
reality
it
will
a
little
and
when
you
read
only
once,
caching
also
doesn't
provide
a
huge
advantage
on
on
read
patterns
unless
your
workload
fits
into
the
cache
for
which
you
have
two
appropriately
sized
and
right
caches
provide
a
huge
benefit
for
bursty
workloads,
but
not
so
much
for
sustained.
You
know
max
throughput
and
let's,
let's
now
go
quickly
off
on
the
tangent
on
cash
tearing,
because
cash
tearing
is
actually
useful.
A
If
that's
what
you
need,
it
does
not
magically
make
your
classic
go
faster
because
first,
the
objects
are
actively
physically
migrated
from
one
tier
to
the
other
right.
They've
migrated
over
the
network
to
the
cash
tier
and
eventually
back,
you
know,
made
some
points.
They
keep
in
both
places
when
they're
unmodified
copies.
But
you
know
that
that
cash
eviction
is
very
tunable.
There's
a
huge
amount
of
complexity,
and
is
it
cost-efficient
still
is.
A
Maybe
you
get
benefits
from
just
making
a
back-end
layer
faster
because,
like
when
you
have
like
three
replication
on
the
front
where
this
ten
plus
four
in
the
back
with
either
Asia
coding
and
you
use
a
much
more
expensive,
front-end
tier
you
have
to
buy
a
lot
more
set.
So
sometimes
the
price
economy
doesn't
work
out
and
does
it
perform
right?
Your
front-end
tier
has
three
wet
three-way
right
amplification
and
your
back-end
does
not
my
ten
plus
four.
You
get
40
percent
right
overhead
on
the
network
and
on
the
storage
devices.
A
So,
depending
on
the
workload,
maybe
caching
isn't
actually
providing
the
benefit
you're
saying
it
does
and
about
those
right
caches.
What
is
it
your
ability
and
availability
model
right?
3R
a
replication
is
not
as
durable
as
ten
plus
four
you
know,
and
when
you
do
something
like
position,
client-side
caching,
which
are
B
D,
can
all
do
if
say,
client
node
dies.
You
have
lost
everything,
that's
still
in
cash
right.
Your
copy
may
still
be
consistent,
but
it's
also
not
up-to-date.
A
If
your
workload
is
fine
with
that,
that's
great,
if
it's
not,
you
know,
that's
just
something
that
you
have
to
keep
in
mind
right.
Everybody
goes
on
about
having
you
CC
memory,
and
then
they
all
write
it
to
that
disk
word
passes
through
the
buffer
cache
on
the
disk
device,
which
is
not
protected
at
all.
Things
like
that
writes
a
durability
model
and
availability
model
of
the
cache
are
really
important.
A
And
just
closing
it
out
on
benchmarks:
when
you
go
for
maximum
throughput
benchmarking,
you
get
terrible
latency
and
that's
expected
right
so
because
you're
trying
to
cram
as
much
through
as
possible,
you're
piling
on
requests
on
requests,
and
that's
just
like
that.
You're
trying
you
know
going
in
that,
you
will
not
be
able
to
service
all
of
those
requests.
So
when
you
want
to
actually
benchmark
for
performance
for
latency,
you
wouldn't
probably
only
benchmark
it
like
30,
40
percent,
maximum
throughput
right
and
also
you
have
to
account
for
that.
A
When
you
get
the
last
one
should
be
a
no-brainer,
you
have
to
make
sure
that
you
can
sustain
your
bandwidth
while
I
notice
down.
So
that
brings
us
to
hardware
notes
being
down
right.
Monocultures
are
cheap,
but
terrible,
right
or
cheaper
right.
It's
easy
right!
You
buy
that
one
SKU,
the
one
piece
of
hardware
20
times
everywhere,
and
it's
easy
right.
It's
easy
to
order!
It's
easy
to
set
up
all
the
systems
of
100%
equal
to
1
and
on
the
especially
terrifying
line.
A
A
Together
with
a
when
there
was
a
NIC
firmware,
but
those
were
not
pleasant
weeks.
The
customer
was
annoyed,
we
were
annoyed.
Everyone
was
annoyed.
Eventually
it
was
fixed
which
it
was
a
great
experience.
But
you
know,
monocultures
are
not
very
resilient
and
just
hard
drives
hard
drive
firmware.
It's
just
the
same.
We
had
to
from
an
incident
where
you
would
write,
data
and
drive
would
go,
I
wrote
the
data
fine
and
then
you
would
go,
read
it
and
would
go
Ino
dater,
which
is
called
a
week
right.
A
You
can
the
drive
notices
when
it
tries
to
read
that
the
data
block
was
basically
corrupted
and
refuses
to
give
you
a
bad
data
fare
and
there
was
a
firmer
back,
so
it
affected
all
the
drives
at
that
customer
deployment
took
a
while
to
settle
that
with
a
drive
vendor
who
shall
remain
unnamed
and
yeah,
and
some
intermediate
system
integrator
had
only
one
skew
for
that
2
terabyte
drive.
We
were
like,
we
want
you
to
terabyte
drives
but
not
said
vendor
and
they
were
like.
A
Well,
we
have
qualified
3
vendors
for
that
2
terabyte
drive
model
this
hours
q4
during
a
2
terabyte
drive.
We
cannot
guarantee
you
are
not
getting
that
vendor
again
we're
like
you
know.
This
is
the
kind
of
thing
that
you
have
to
deal
with,
because
you're
buying
commodity
off-the-shelf
hardware.
This
is
not
set
up
for
somebody
going
in
and
saying
I
want
this
particular
drive
model
which
that
firmware
right,
so
it's
and
frankly,
don't
even
bother
because
it's
not
sustainable.
A
You
know
when
your
self
cluster
is
hopefully
in
operation
for
like
multiple
years
right
because
of
certainty
grows,
that
we
have
established
that
your
data
has
right
your
big
data,
it's
growing,
so
you're
not
going
to
be
buying
the
same
hardware
next
year,
right,
you're,
just
not
so
just
go
in,
accept
things
that
you're
buying
from
multiple
vendors
right
and
save
yourself.
That
pain
because
buying
cheap
can
also
mean
buying
twice.
A
Software-Defined
storage
uses
commodity
off-the-shelf,
hardware
and
consumer
hardware
does
lie
right:
the
the
most
famous
things
as
a
consumer
hard
drives
lying
about
their
right,
caches,
being
disabled
and
because
then
the
bench
well,
if
they
actually
did
set
the
benchmarks,
would
be
bad.
So
they
lie
about
it
right.
So
you
want
to
do
some
minimal
qualifications
of
the
hardware
that
you
buy
to
make
sure
that
this
is
actually
not
happening
and
that
actually
happens
with
SSD
drives.
A
Ssd
drive
sometimes
have
memory
in
front
of
them
that
can
sustain
the
write
for
like
60
seconds
before
they
fall
back
to
flashing
to
the
cheapo
beckoned
flash
right,
because
flash
has
different
quality
levels
and
different
speed
levels,
and
it's
much
cheaper
to
only
have
so
much
of
that-
and
you
know
they
they're,
basically
hybrid
SSDs,
and
initially
we
also
had
SSD
drives.
That
would
be
really
fast
for
the
first
two
weeks
and
then
degrade
as
a
we're.
A
Leveling
kicked
in
right
so
so
that
all
unfortunately
happens,
and
so
you
don't
want
to
actually
buy
the
cheapest
consumer-grade
hardware
and
again,
that's
not
a
problem
when
you
buy
for
production
environment,
but
this
is
typically
how
people
build
their
pilots
and
then
run
into
all
these
things
without
knowing
that
they're
going
to
run
into
all
these
things
and
then
walk
away,
unto
us
fire
right.
So
just
something
to
be
aware
of,
and
yes
this
is
this:
a
favorite
of
mine.
A
Have
you
ever
looked
at
some
of
those
wreck,
tables
or
wreck
diagrams
that
people
draw
like
all
the
OSC's
right.
You
have
a
rack
full
of
OSTs,
and
then
you
have
the
requisite
of
services
right
all
the
overheads
that
you
don't
scale
so
much
because
you
know
three
months
are
probably
going
to
sustain
you're.
You
know
through
several
racks
of
always
T's
and
there's
this
one
power
supply
going
into
that
rack,
where
all
your
monster
and
all
your
MDS
and
all
your
radars
gateways
so
but
otherwise
the
racks
would
be
different.
A
No,
they
shouldn't
you
know
or
maybe
have
have
have
a
Mon
in
one
and
Dorados
gateway
in
the
next
and
alternate
between
that.
You
know.
Maybe
you
can
do
something
like
that,
but
if
you
have
yeah
like
I
thought
about
actually
drawing
a
picture
here,
but
it's
probably
not
necessary.
It's
just
a
bad
idea.
If
you
correlate
your
failure,
domains
like
that
in
general
ignore
the
throughput
you
get
from
self,
because
throughput
is
easy
to
get
or
total
ions.
You
can
just
buy
more
hardware.
A
You
can
all
just
just
add
more
nodes
at
more
storage
devices.
Right,
that's
easy!
That's
easy
to
scale!
We
can
scale
that
really
well,
but
you
will
not
easily
be
able
to
scale
as
latency
unless
we
get
crimson
to
you
and
latency
goes
down
magically,
so
people
buy
40
gigabit
because
it's
faster
than
25
true,
but
you
know
it's
the
latency.
The
signaling
rate
is
lower
40
gigabit
zone
on
25
gigabit,
so
I
don't
think
anyone
should
be
buying
that
anymore.
Everybody
should
be
buying
25
or
50
gigabytes
these
days.
A
Sometimes
people
disagree,
but
it's
just
because
the
latency
is
one
thing.
That's
really
hard
to
fix
because
that's
basically
static
and
then
people
compare
a
local
SSD
performance
to
the
ones
that
they
get
outside
distributed
system
and
notice
that
it's
much
slower
yeah,
because
you
have
to
go
through
the
network
multiple
times
right
again
to
focus
on
the
latency.
That's
really
that's
a
one
thing.
That's
really
hard
to
fix
and
also
for
an
old
pricing
right,
I'm
going
to
get
shot
so
per
node.
A
Pricing
is
hard
because
it
encourages
ideas
that
you
should
have
fewer
bigger
nodes.
You
shouldn't
write
you
you
should
have.
You
know
reasonable
notes,
but
you
should
not
make
them.
You
know
you
should
not
have
like
a
petabyte
of
capacity
on
one
node,
just
split
it
right
talk
to
your
vendor
about
discounts,
because
that
also
means.
So
you
have
more
bandwidth
per
storage
device.
You
have
less
impact
on
Alta
juice,
you're
losing
less
of
your
capacity
and
remember.
Every
Spindler
you
lose
is
part
of
the
bandwidth
that
you're
losing
and
it's
also
less
bandwidth.
A
You
have
to
recall
s
capacity.
You
have
to
recover
that
node
really
does
not
come
back
right.
If
a
one
petabyte
node
goes
down,
you
have
to
recover
that
right,
so
make
sure
that
not
a
single
failure
is
a
significant
part
of
your
availability
and
it's
also
cheaper
to
scale.
You
can't
call
scale
and
small
increments
right,
that's
also
quite
useful.
Buying
another
of
those
petabyte
nodes
is
probably
quite
expensive.
A
Buying
a
200,
terabyte
notes,
much
less
so
right
and
at
the
other
end
of
the
scale
when
I
say
light
white
nodes,
I,
don't
mean
flimsy
okay,
so
sometimes
people
want
to
absolutely
use
self
and
so
have
big
data.
So
and
they
go
this,
this
happened
right
well,
we
are
now
using
one
terabyte
hard
drives
and
two
of
them
one
or
two
of
them
per
node.
So
we
can
meet
the
minimum
requirement
for
failure,
domains.
A
Here's
a
bigger,
hard
drive
so
because
that
that
does
not
give
you
economy
of
scale.
This
is
not
cost-efficient.
Unless
you
do
this,
because
you
have
this
hyper-converged
environment,
where
you
just
put
that
into
the
slot
where
it
makes
sense,
that's
fine
and
you
have
set
drive
bay
anyway.
So
use
it.
That's
fine!
But
don't
do
this
if
you're
explicitly
building
a
safe
cluster
right
unless
you
your
demo,
that's
what
we
do
with
our
Raspberry
Pi,
it's
a
booze
and
that's
fine,
it's
great
for
demo!
Right
and
here
my
Joelle
quote.
A
Sometimes
people
run
safe
on
public
cloud,
because
sometimes
a
pricing
model
from
the
public
cloud,
vendors
for
long-term
storage
or
persistent
cluster
wide
storage
is
strange
and
it's
actually
cheaper
to
buy
three
times
as
much
ephemeral
storage
attached
to
your
instance
and
it
is
to
buy
zaire
distributed
storage
service
which
is
basically
performing
a
beat
arsh
on
their
pricing
model
and
this
works.
It
can
be
great,
it's
all
what
we
all
do
for
testing
and
after
all,
but
from
a
performance
point
of
view.
A
That's
pretty
terrible
because
you
don't
have
the
transparency
into
what's
actually
going
on
right.
You
don't
know
what
the
hypervisors
are
doing,
why
your
network
traffic
is
going
I,
it's
it's
just
I've
had
to
support
this.
We
made
it
work.
It
was
a
fun
experience.
I
learned
a
lot
just
think
about
whether
it's
really
what
you
want
to
do
so
almost
on
time,
closing
out
again
going
back
to
this
right.
So
when
you
have
said
workloads
that
you
need
surface
of
one
things
that
can
do
that
right,
there's
nothing
else.
A
It
can,
but
there
are
other
solutions
out
there
that
can
do
other
things
well
right
and
we
are
taking
self
to
close
that
gap,
but,
depending
on
the
time
of
day
and
which
here
it
is,
you
know,
the
trade-offs
may
not
swing
always
toward
self.
We
hope
that
you
have
that
hyper
scale.
Use
case
with
you
know
that
exponential
data
grows
because
that
is
lovely
like
I
said:
I
like
bread,
even
if
I
baked
it
myself
so.
A
And
this
probably,
what
I
would
like
to
leave
you
with
so
open
source
F
is
infinitely
flexible
right.
You
can
do
many,
many
things
with
it.
It
has
all
the
knobs.
You
can
make
all
the
choices
you
can
adapt
it
to
your
environment.
The
downside
is,
you
have
to
make
all
the
choices
somebody
has
to
make
all
those
choices
for
you
right
so
plan
for
learning
when
you
deploy
a
system
plan
to
adjust
what
you're
doing
and
remember
that
there's
no
such
thing
as
a
free
lunch
or,
in
this
case
a
free
dinner
right.
A
A
Why
did
you
spell
my
name
wrong,
I
own?
It
too,
you
know
you
actually
just
asked
me
weird
about
the
the
dildo
yeah
okay
yeah,
we'll
talk
about
that
over
dinner.
Sorry,
okay,
anything
else,
if
not
and
enjoy
the
birds
of
a
feather
sessions
which
are
also
happening
and
enjoy
dinner
and
see
you
tomorrow
morning,
at
9:00.