►
Description
Sage Weil discusses the Ceph distributed storage system at the October 2011 meeting of the UNIX Users Association (UUASC) hosted at Media Temple in Culver City California.
Sage's abstract is below.
Since the presentation Inktank www.inktank.com was cofounded by Sage to handle professional services support of Ceph.
More inforamation on UUASC can be found at http://uuasc.org .
Abstract
As the size and performance requirements of storage systems have
increased, file system designers have looked to new architectures to
facilitate system scalability. Ceph is a fully open source distributed
object store, network block device, and file system designed for
reliability, performance, and scalability from terabytes to exabytes.
Ceph's architecture consists of two main components: an object storage
layer, and a distributed file system that is constructed on top of this
object store. The object store provides a generic, scalable storage
platform with support for snapshots and distributed computation. This
storage backend is used to provide a simple network block device (RBD)
with thin provisioning and snapshots, or an S3 or Swift compatible RESTful
object storage interface. It also forms the basis for a distributed file
system, managed by a distributed metadata server cluster, which similarly
provides advanced features like per-directory granularity snapshots, and a
recursive accounting feature that provides a convenient view of how much
data is stored beneath any directory in the system.
This talk will describe the Ceph architecture and then focus on the
current status and future of the project. This will include a discussion
of Ceph's relationship with btrfs, the file system and RBD clients in the
Linux kernel, RBD support for virtual block devices in Qemu/KVM and
libvirt, and current engineering challenges.
Sage Weil designed Ceph as part of his PhD research in Storage Systems at
the University of California, Santa Cruz. Since graduating, he has
continued to refine the system with the goal of providing a stable next
generation distributed file system for Linux.
A
B
B
Behind
this
work,
but
I
found
that,
by
solving
a
problem
specific
to
their
issues,
ended
up
building
something
much
more
general
and
useful
for
really
often.
What
is
that?
It's
a
saleable
store
system
anywhere
from
one
node
to
thousands
of
nodes,
gigabytes
exabytes
system
is
designed
to
be
reliable,
but
no
single
points
of
failure.
All
data
in
the
system
is
replicated
across
multiple
nodes
because
we,
both
in
this
large
failure,
is
a
norm
rather
than
the
exception.
B
B
B
B
B
B
Designed
to
be
flexible,
designed
to
be
deployed
on
commodity
hardware,
hardware
with
incremental
scalability,
so
you
add
first
note
that
you
need
a
modulus
modular
server
component.
So
the
story
system
is
just
a
series
of
demons
running
on
all
these
servers,
their
local
storage
guys
different
ways.
You
can
talk
to
your
storage
and
we
try
to
avoid
legacy
design
independence.
So
we.
B
That
are
sort
of
made
sense
thirty
years
ago
and
lessons
today,
when
you're
talking
about
systems
better
distributed
across,
want
to
avoid
any
kind
of
manual
work
partition.
You
don't
want
to
happen,
have
an
administrator!
That's
telling
the
system
to
move
data
from
one
server
to
another,
and
this
isn't
to
deal
to
all
that
work
for
you
automatically
and.
B
Having
passive
servers
that
assistant
servers
that
sit
on
the
network
waiting
for
commands,
but
don't
actually
doing
useful
work,
while
the
sitting
idle
all
of
our
source
systems
these
days,
our
computer's
memory
use,
they
can
actually
exploit
that
at
a
high
level.
I'm
going
to
talk
about
I'm
gonna
talk
about
the
objects,
torrid
abstractions,
that
sort
of
form
the
basis
of
the
system
sort
of
the
design
question
of
where?
Where
do
you
decide.
B
Assuming
you
have
this
distributed
cluster
over
hundreds
of
notes,
how
do
you
decide
where
it
is
how
we
make
that
that
cluster
offensiveness
self
managing
how
the
nodes
actually
work
together
to
its
behavior
ceiling
and,
finally,
how
we
build
a
POSIX
top
of
that
so
start
out?
What
are
topics
but
I
think.
B
B
B
The
object,
storage,
the
cloud
I
guess
into
the
separate
tools,
so
each
pools
a
separate
namespace,
a
collection
of
objects
essentially
and
have
some
basic
properties
like
the
level
of
replication
that
we
want
to
those
objects,
whether
it's
two
times
three
times
four
times
whatever.
What
kind
of
allocation
policy
we
want?
D
B
B
B
B
So
one
of
the
basic
questions
when
you're
building
a
distributed
system
like
this
a
tiny
decide
where
put
your
data,
so
the
simplest
approach
is
just
to
remember.
We
put
it
I
sort
data
a
on
those
be
in
file,
see
a
problem
that
approaches
that
the
system,
once
you
do,
that
it
can't
move
to
data
somewhere
else,
because
then
you
won't
build
assignment.
If
you.
B
B
It
would
tell
you
go
over
here
and
then
you
go
to
that
location
and
you'd.
Read
the
data,
that's
better,
because
the
system
can
move
the
data
around
the
balance
data
to
recover
from
failures
and
deal
with
all
the
things
that
have
any
clutter,
but
it
built
that
still
sucks,
because
the
system
has
to
maintain
this
mapping
it.
These
are
the
names
of
the
objects
in
the
other
location
for
their
sort
and,
as
you
start
to
build,
a
system
has
billion
divided
by
cross
under
the
nodes
that
index
that
directories
become
targets
cables.
B
So
a
third
approach
a
little
bit
more
tricky
to
calculate
a
pseudorandom
mapping.
So
the
idea
here
is
that
you
take
the
name
of
the
object
and
the
set
of
nodes
that
comprise
the
cluster
and
you
feed
that
into
some
magic
function
and
out
spits
out
the
current
location
of
the
data
for
the
location
where
that
data
should
be
stored
and.
B
Approach
of
that,
you
don't
have
to
store
it
rectory
at
all,
you
just
give
it
the
name
of
whatever
it
is
called
or
would
be
called
if
you're
gonna
store
it,
and
let
me
know
the
question
is
about
where
Jesus
algorithm
that
we
used
to
do
this.
It's
called
crust.
So
the
key
things
about
press
to
remember
our
first
of
all:
that's
functional,
so
you
give
it
some
input.
B
B
B
And
the
other
key
feature
that
distinguishes
crushed
from
similar
algorithms
isn't
happening,
but
a
lot
of
these
misses
you.
What
makes
press
different
is
that
it
has
features
that
allow
you
to
have
a
reliability
a
bit
more
elegantly,
so
the
Indians
that
you
describe
the
cluster
in
terms
of
a
physical
architecture.
So
in
practice
you
have
you
know
a
data
center.
We
have
row.
B
So
you
might
say
I'm
gonna
figure
out
the
codes
I
want
all
them
to
be
in
the
same
row
so
that
when
I
replicate
them
across
multiple
nodes,
I,
don't
saturate
my
core
switches,
but
I
want
them
to
be
separated
across
different
racks,
so
that,
if
I
do
the
single
power
circuit
I
only
take
out
a
single
rep
code.
My
that
and.
B
B
B
A
more
detailed
picture
of
how
Christ
actually
puts
data
in
the
cluster,
so
what
the
system
will
do
typically
is
take
up
on
a
file
or
our
block
device,
whatever
it
is,
and
it'll
strike
whatever
will
be
London
by
default
topics,
our
format,
but
you
can.
You
can
specify
whatever
put
writer
parameters
in
there
and.
B
Into
what
we
call
the
placement
group,
or
just
a
sort
of
arbitrary
collection
of
objects,
kinda,
like
a
start,
you're
used
to
talking
about
to
search
a
database
or
something
like
that,
and
it
means
placing
group
or
collection
of
objects,
is
mapped
to
a
set
of
those
these
that
are
going
to
replicate
all
those
objects
I'm
using
this-
and
this
gives
you
a
pseudo
random,
uniform
histories
and
data
across
class.
So
the
key
things
that
are
great
there
is
that
first,
the
fast
calculation
in
order
to
look
up
data,
we.
B
That
says
where
the
data
is,
or
we
just
calculate
the
location,
it's
reliable,
because
we
can
specify
that
our
replicas
are
separated
across
the
other
good
means
racks
easily,
and
it's
tables
that
when
we
every
move
device,
a
small
number
of
objects
move
around
in
order
to
redistribute
things
you
believe
up
on
those
and
sort
of
maximally
utilize.
Our
available.
B
C
B
Especially
today,
things
are
typically
a
little
bit
different.
Things
are
moving
work
on
senesce,
so
people
have
servers
on
the
network
that
sort
of
deployed
in
terms
of
rack
mount
devices.
There's
computers
with
you
can
typically
we
use
higher-level
protocols
that
guys
got
the
best
bits
of
right
beat
them
so
the
situation.
It
seems
a
little
bit
now.
We're
talking
to
you,
know,
general-purpose
computers
on
the
network,
but
these
storage
servers
are
still
passive.
Even
though
our
sorcerers
have
CPU
memory,
the
running
arbitrary
software,
they
can
do
all
kinds
of
crazy
things.
D
B
B
Collaborate
with
their
peers,
so
Tucker
the
network
to
read,
write
data-
true,
it's
like
any
other,
but
they'll,
also
talk
with
their
peers,
who
are
disadvantaged,
useful
goody,
useful
work
like
replicating
our
data,
all
data
is
replicated
in
time
to
consistently
apply
updates.
So
if
you
write
to
an
object,
it'll
write
all
cups
of
the
object
consistently
and
make
sure
that
those
copies
are
in
sync:
it's
even
a
second
of
failures,
there's
any
kind
of
failing
the
system
and
also
work
to
migrate
data
to
ecology.
You
guys
need
this.
B
B
C
B
D
B
B
Process
of
Kult
hearing
so
whenever
there
is
need
even
starts
up
or
whenever
there's
a
quality
change.
Whenever
there's
a
map
update,
they
go
through
this
identical
process.
We
look
at
all
the
data
that
they
currently
have
and
recalculate
its
mapping
to
make
sure
that
it
should
still
be
stored
where
it
is
or
there
where,
where
it
should
go,
and
then
they
coordinate
with
each
other
taxi
move.
They
did
it
to
the
correct
so.
B
B
B
B
C
B
D
B
Something
throughout
never
restated,
verify
the
checksum
and
make
sure
it's
actually
what
it
expected
find
there
and
handle
the
local
devices.
Much
like
the
FSSA
setup
pool
the
storage.
It
will
replicate
internally.
First,
though,
it
has
transparent
when
compression
on
that
server
they're
cute
game,
but
beyond
that
sort
of
basic
functionality,
stuff.
B
Butter
vessels
internal
transaction
subsystem
in
order
to
manage
the
way
that
the
historic
diamond,
convinced
native
gist.
So
we
always
make
sure
that
the
storage
patina
that
this
storm
locally
is
coherent
with
other
copies.
They
can
take
advantage
of
that
we
hook
into
our
the
copy-on-write
infrastructure
so
that
when
the
snaps
off
happen
in
the
sort
of
higher
level
distributed
file
system-
or
you
had
the
object
interface,
we
can
clone
Bob
isn't
about
actually
copying
data.
B
C
B
B
C
B
B
D
B
The
object,
if
it's
set
to
a
or
some
value
then
proceed
with
the
operation,
but
if
it's
different
than
before,
so
you
can
do
sort
of
a
compare
and
swap
atomic
thing
in
the
objects
or
definite
for
each
interface.
If
you
snap
that
permanent,
so
your
operations
have
a
context
associated
with
them
that
informed
how
the
copy-on-write
stuff
happens,.
B
Notify
it's
sort
of
like
a
lightweight
locking
mechanism
and
communications
infrastructure
that
uses
the
optic
communications
channel
key
value
in
case
a
stork
you
can
die
and
an
object,
and
one
of
the
nice
things
is
that
the
objects
are
gigantic.
Samples,
of
course,
doesn't
service
the
other
source
making
new
it
sort
of
beyond
that.
We
have
this.
Plugin
infrastructure
defined
your
own
classes,
so
that
some
of
the
pieces
base
levels
the
system
influences
your
basic
interface
of
profits.
You
could
read:
read.
B
C
B
Dynamically
load
classes
into
the
cluster
that
do
build
new
functionality
on
top
of
those
basic
methods.
So,
for
example,
you
could
link
an
encrypted
library
and
build
a
class
that
would
have
calculate
a
sha-1
of
a
of
a
non-factor
returning,
just
result-oriented
link
in
an
image
library
and
how
to
rotate
an
image
to
resize
it
and
then
save
3.
So
without.
B
So
the
basic
golden
is
that
you
can
move
the
computation
to
your
data.
So
typically
in
these
systems
you
know
so
you're,
building,
clicker
or
something
you've
got
a
massive
deflection
images.
You
have
to
do
something
to
the
right,
so
you
want
to
resize
the
layer,
images
or
remove
redeye
or
something
you
can
do
that
on
the
sword
again,
without
actually
incurring
the
network
and.
B
That's
used
for
the
file
system
actually
uses
this
functionality
as
well,
so
it
has
a
class
that
stores
directories
and
objects,
and
so
we
can
send
a
command
to
the
object.
So
that
says,
remove
this
one
directory.
Entry
from
this
object
never
commits
up
the
director
content
and
the
other
thing
star
will
do
that
reports
that
it
I'm
gonna
read
the
entire
directory
of
network
making
small
scenes
and
then
write
an
etiquette
see.
One
of
the
other
pieces
is
there's
this
data,
a
restful,
object,
storage,
Great,
Escape
way.
B
B
B
Basically
puts
a
virtual
disk
image
enlist
in
the
cloud
and
cluster
and
the
single
that's
going
to
just
strike
over
all
1,000
servers.
100
service
are
bigger.
Buffers
do
really
cool
things.
Instead
of
having
your
personal
machine
tied
to
the
personal
machine
host
local
disk,
they
evoke
doesn't
fail
and
it's
stored
in
the
cloud
and
that's
replicated
minutes
high-performance
minutes
always
available.
You
can
do
few
things
like
migrate
by
migration
number
some
scenes
between
coasts,
for
example.
B
A
B
That's
that's
what
the
flaring
is
are
things
that
something
that
we're
working
on
right
now
I
mean
that's
basic
idea.
There
is
that
you
have
a
virtual
machine
image.
That's
you
know
who
to
ten
ten.
That's
your
goal,
image
for
this
class
of
persons
unit,
whatever
you
want
to
create
a
new
virtual
machine
image.
You.
E
C
B
A
B
A
B
I
should
call
the
pony
that's
a
tellurian
Larry
without
backing
over.
The
idea
is
to
do
to
do
this.
That's
one
use
case
where
you
have
a
goal
dimension.
You
want
to
create
another.
You
cases
me
that
you
have,
and
even
if
that's
throw
it
in
this
particular
pool,
the
cluster
is
stored
over
here
and
you
want
to
migrate
it
to
the
other
pool
the
cluster
or
somewhere
over
there,
and
so.
B
A
B
So
if
there's
needed
community
and
support
via
the
base
library
that
links
directly
into
the
commune
binary
source
code,
so
you
can
launch
numbers
to
Universal
machine
with
the
right
the
lessons
and
the
right
magic
string
for
your
image
and
it'll
talk
directly
to
the
source,
but
without
any
internal
support
without
any
so
distinctive
over
any
other
distance.
That's
my
boss!
The
Linux
kernel
driver,
so
you
can
echo
a
magic
string
to
the
Sisyphus
file
or
use
the
command
line
tool
for
you
to
map
an
image
and
it
shows
up
as
a
regular.
B
B
B
B
Like
a
large
distributed,
coherent
cache
of
the
base
with
your
innovative
operations
are
efficient
and
then
they're
eventually
backing
that
again
replicated
and
realize
it's
a
dynamic
cluster.
So
you
might
have
one
minute:
the
server
big
buses
them
or
you
might
have
100.
If
you
have
a
huge
really
whatever
whatever
it's
the
start
of
new
team
in
from
the
11th
of
them
in
the
cluster,
and
you
can
you
want
to
shut
him
down,
you
come
to
export.
B
And
then
it
will
issue
of
capability
back
to
the
client
essentially
says
you,
client
are
allowed
to
read
the
data
in
this
file
and
from
that
point
forward,
whenever
the
client
reads
or
writes
data,
it
just
talks
directly
to
the
ovens
or
it
knows
what
object
files
will
be
stored
in
is
basically
the
basically
inode
number
six
name,
and
so
we
can
calculate
the
location
of
the
data
go
directly
to
the
source
node.
That
has
that.
B
B
B
B
Could
have
a
static,
subtree
protection,
which
is
but
people
who
manage
farms
and
an
investor
groups
are
used
to
you
know
slash
home,
slash
all
the
A's
are
on
the
server
and
all
the
bees
on
this
other
area.
Don't
see
it
over
there
and
then,
when
you
get
too
many
Z's,
because
that's
a
buckle
that
are
you
split
them
up?
You
man,
my
my
husband
or
another
server,
look
there's
that
so.
B
B
Basically,
randomly
scattering
all
a
file
across
across
the
cluster,
and
that
gives
you
really
good
load
the
balance
right.
You
know
that
I
need
to
sort
of
optimized.
If
you
have
one
one
correctly,
that's
getting
a
lot
of
traffic,
that's
distributed
across
whole
cluster.
What
else
good,
but,
on
the
other
hand,
you're
just
storing
locality,
because
you
basically
completely.
B
C
D
B
So,
in
this
case,
before
clan
320
x
axis
before
300,
its
ism
of
doing
something
in
Africa
was
running
some
workload
and
it
was
balanced
across
closer
and
I'm
300.
Everything
stopped
shut
down
the
old
job
and
start
a
new
job,
which
was
the
whole
bunch
of
processes
all
creating
a
mess
of
files
and
directories.
B
B
C
B
B
So
one
of
the
other
interesting
things
that
the
Buster,
the
cousin
we've
implemented
a
sort
of
the
metadata
hierarchy
from
scratch.
What
to
do
into
some
interesting
things.
So
one
of
the
things
that
we
maintain
recursive
counts
of
files
and
directories
and
file
sizes
across
the
entire
harvest.
So
there's
our.
B
Directory,
if
that's
defined
as
a
summation
of
all
file
size
in
that
directory
and
nested,
underneath
it
in
the
hierarchy
and
by
default,
when
you
do
an
LS
FAL,
instead
of
giving
you
a
meaningless
number
for
the
side
of
the
directory,
you
do
that
recursively!
You
can
do
an
LS
and
essentially
get
that
what
you
would
get
from
it
to
you.
B
B
B
It
also
gives
you
fine-grained
snapped.
Oh
so,
sort
of
one
of
the
challenges
with
having
a
petabyte
skill
file
system.
That's
distributed
across
hundreds
of
members
is
that
you're
plugging
to
storing
quaza
different
types
of
data
in
there
and
you're
not
going
to
have
sort
of
the
same
data
retention
factor
policies
for
all
that
either
so
having
a
global
snap
set
of
entire
file
system
doesn't
necessarily
make
a
lot
of
sense,
and
so
what
we
give
you
is
a
the
ability
to
snap
some
arbitrary
Sudbury's,
sub
trees
or
sub
directory
system.
Then.
B
B
B
So
they're
never
amazing
about
clients.
I
have
a
farm
in
the
server.
The
goal
is
to
give
popping
semantics
with
strong
consistently.
So
the
standard
is
essentially
that
processes
that
are
interacting
on
different
hosts
with
the
file
system
should
have.
The
exact
thing
result
that
that
they're
on
the
same
machine
in
Arkansas
so.
B
B
B
Already
broke
the
model
it'll
do
a
mouth
test,
except
give
it
the
IP
address
in
it.
Talk
to
all
minutes,
there's
also
a
user
space
flight
interpretation.
So
there's
a
few
client
data
leaves
the
keys
interface
also
works
to
finances.
There's
a
lip
step,
library
that
you
can
link
into
applications.
That'll
talk
directly
to
the
sources.
Then
there.
B
F
E
C
B
A
D
A
B
Oh
and
oak
lab
use
cases
actually
where
we
implement
these
days,
even
though
already
you've
had
one
of
the
simplest
things
you
can
do
with
this
system
by
antiquated
way
of
plugging
story,
that's
what
most
people
want
and
that's
what
they
need
for
building
their
own
private
clouds
and
spending
up
all
these
virtual
machines.
Virtual
machine
just
wants
a
virtual
disc
Finnegan.
You
know
NTFS
or
degree
or
whatever
you're
going
through
together.
B
B
B
C
B
Testing
seven
tests,
except
for
a
long
time
actually-
and
somebody
in
this
company,
asked
them
for
virtual
machine
and
he's
like
well
he's
asking
me
to
buy
you
soon
to
it,
because
he
wanted
to
try
out
this
and
see
concepts,
and
so
he
created
it
and
not
realizing
that
the
person
was
just
asking
the
wrong
person.
Your
connection
was
like
production.
B
B
C
B
B
B
C
B
E
B
B
A
B
C
A
A
C
B
F
B
Mean
if
you
can,
you
can
wait
those,
and
that
is
that,
essentially,
is
the
relative
amount
of
data.
Again.
I
know
that
we
have
one
and
one
month
to
the
tune
of
get
twice
as
much
data
as
one
but
sense
of
what
that
means.
We
get
its
Pi
Sigma
Theta,
and
then
it
gets
quite
a
typo
because
it
has
twice
as
much
is.
B
A
B
A
B
F
A
C
B
A
So
they
let
you
that
much
memory,
you
look
at
the
file
systems
of
all
the
cavity
back,
and
so
it's
got
a
pretty
small
footprint.
It
just
maybe
has
a
lot
of
sockets
open
a
way
you
could
have
to
connect
you
to
other
no
school.
That's
what
doc
do
you
map
updates
or
like
as
you
scale
up,
do
you
have
pauses
where
people
are
trying
to
sync
up
the
ladies,
especially.
F
B
B
If
you
have
a
client
that
says,
I
have
math
100
and
the
server
has
math
100
did
they
agree
and
they'll
they'll
happily
read
write
data
and
it's
only
when
the
client
has
about
100
in
the
server
set
by
of
200.
Here,
here's
the
dip
so
to
bring
up
to
date.
You
should
actually
be
flying
on
the
server
and
that's
when
you
sort
of
get
somewhere
else.
They.
A
F
B
C
B
B
So
we
built
it
on
the
infrastructure
to
do
this,
so
that
when
you
make
a
change,
we
can
make
a
graceful.
The
truth
is
that,
most
of
the
time
now,
when
you
make
a
change,
we
don't
bother
a
lot
of
work
to
actually
do
it
and
are
that
many
users
with
big
clusters
that
need
to
read
online
and
so
were
searching,
but.
B
Well,
we
have
blogs,
it's
all
the
demon
side
of
this
logging.
Well,
we
can
turn
way
out
the
way
down
the
current
way
up.
It
gets
done
lots
of
crap
the
blog
and
it
goes
really
slow.
So
we'll
do
we'll
do
the
other
blogs
use,
run
things
through
background,
sometimes
which
is
a
sort
of
a
runtime
checker
that
make
sure
you
don't
access
items
like
memory
and
stuff
like
that.
B
Usually
sometimes,
but
it's
usually
fire
fast,
the
better
pressure
crashes
on
correctly.
So
that's
this
stuff
isn't
doing
anything
too.
Weird
I
mean
it's
just
reading
our
any
files,
but
it
still
manages
to
work
out
all
the
bugs
that
other
people
don't
hit.
So
we've
had
lots
of
bugs
and
ext3
these
before.
Actually
because
we
use
headers
and
most
people
don't
change
very
much.
They
like
set
a
security
label
and
then
they're
gone.
They
never
touch
it
again,
but
we're
constantly
setting
big
ones
as
following.
B
B
For
everything,
GCC
everything
we
et
Malik,
which
is
a
replacement
for
Malik,
came
out
of
Google.
That
is
way
more
efficient,
professor
nice
things
that
has
a
deep
profiling
tool.
So
you
can
put
the
Swift
and
it'll
keep
track
of
where
all
your
memory
allocation
sites
are
going
and
I
think
it
helped
it
done,
and
you
can
look
at
this
nice
directory
where.
D
D
B
C
D
D
B
B
B
Where
you
open
a
body
part
of
it,
it's
everything
cartridges,
video
feed,
that's
actually
a
very
sort
of
simple
and
actually
not
actually
correct
picture
of
things.
What
would
actually
happen
is
the
user
with
CD
into
a
directory,
and
it
would
it'll
send
a
request
minute
server
to
like
look
up.
You
know,
plop
phooo,
DV,
/,
suppose,
look
up,
+
vu
and
look
at
the
end
of
the
minute
server
and
get
a
capability
back
on
a
client
and
I
capability
basically
says
that
I
don't
have
the
reference
to
that.
B
B
E
B
Usually
your
writing
abilities
directly,
so
they're
getting
totally
different
processes
and,
depending
on
what
your
workload
is.
Usually,
if
we
drive
my
files,
you
don't
actually
they're
anything
until
you're
done
so
it
can
buffer
it
to
your
heads
and
ten
minutes,
first
of
all,
efficiently
in
treatment
over
the
network,
whereas.
B
A
E
G
D
C
B
This
much
work
to
be
done
and
this
many
servers
to
do
it.
You
either
so
one
of
the
interesting
things
when
I
was
doing
some
of
the
initial
metadata
server
work
and
figuring
out
how
to
make
the
load
balancer
work.
Is
that
trying
to
measure
how
well
I
was
doing
the
sort
of
the
obvious
thing
is
another
total
of
clusters,
but
but
it
actually
turns
out
that
you
can
get
the
highest
group
of
numbers.
B
If
you
take,
you
know,
whatever
the
did,
you
take
a
small
amount
of
the
work
load
that
working
set
and
you
distribute
it
across
all
nodes
and
let
them
operate
it
like
a
maximum
efficiency
and
then
make
everyone,
but
you
can
have
insane
throughput
numbers,
but
it's
totally
unfair,
because
some
of
your
work
is
going
crazy
fast.
So
that's
one
crazy
expose,
but
if
you
actually
evenly.
B
E
B
H
B
C
C
H
B
H
H
D
B
Can
liken
you're
gonna
run
it.
However,
you
want
I've.
Never
seen
in
the
thing
to
keep
in
mind
is
that
the
SS
DS
Disney
disc,
as
much
did
you
can
give
them
and
the
metadata
service
just
need
RAM
though
they're
young
US
air.
They
want
to
put
it
on
the
thing
note,
then
you
don't
usually
want,
if
you
have
a
say,
a
hundred
no
cluster.
If
I
don't
want
to
have
like
a
three
or
four
depending.
B
A
B
D
C
B
For
is,
there's
a
sort
of
fundamental
problem
with
I'm,
assuming
you're,
using
Linux
out
of
the
statutes
or
some
flares.
But
if
you
have
a
local
file
system
mounted,
that's
a
sorry
network
file
system
mounted
on
the
current
node.
That's
talking
to
a
server,
that's
on
the
same
note
and
you're
having
to
have
a
heavy
workload.
This
is
the
content
lock
because
when
it
gets
into
a
low
memory
situation,
the
colonel
says:
oh
I,
have
all
this
dirty
meta
data
that
I
need
to
write
out
to
the
server.
C
B
C
B
D
B
D
B
B
B
B
That's
that's
key
is
that
the
replication
is
decluttered
to
determine,
if
you
believe
so,
if
you
have
each
node
has
say
100
placement
groups
I
need
to
those
persons
get
it
on
another
note.
If
that
node
fails,
then
all
the
copies
of
those
placement
groups
are
randomly
scattered
on
100
other
nodes
and
when
we
replicate
those
to
new
locations,
those
new
rusticles
will
be
on
a
hundred
other
node,
and
so
you
basically
have
the
whole
cluster
going
all
the
all
in
parallel
recovering
that
data,
so
it'll
take
one
100.
B
A
D
A
G
On
a
small
scale,
would
there
be
any
practical
reason
if
you
were
at
real
limited
needs
and
you
had
one
file
server
to
run
this
on
one
file
server
rather
than
just
running
bed
with
butter,
efest,
music
NFS?
What
would
your
main
games
be
back?
I
really
any
reliability
games,
we're
teaching
them
out
on
that.
No
it'll
just
complicate.
B
B
C
A
B
C
A
D
G
Question
what
kind
of
disabilities
do
you
have
in
the
sense
of
rate
file
system
of
three
disks
and
others?
One
disk
I
know
I'm,
okay,
I
know
if
I
there's
another
discount
screwed,
so
what
kind
of
disability
do
we
have
when
we
have
lot
a
large
cluster
of
water
systems
to
know
how
vulnerable
we
are?
There's.
B
G
A
B
B
B
D
C
B
A
A
G
B
G
B
B
Trying
to
copy
you
know
a
terabyte
here,
whatever
from
other
places
or
you
have
under
other
locations
that
are
cutting
110
gigs
and
now
that
actually
go
way
faster.
So
it
actually
turns
into
you
a
reliable
thing.
When
you
start
the
numbers,
your
probability
of
data
loss
goes
up.
If
you
have
a
single,
no
that's
a
bottleneck
for
recovery.
That's.
B
G
C
G
B
Others
by
definition,
it's
it's
a
network
file
system,
so
you
have
to
talk
to
the
network,
so
it's
gonna
be
slowed
down
a
little
bit
and
by
definition
all
the
data
is
replicated
in
piles
to
maybe
hopefully
three
and
so
you're
going
to
use
three
times
as
much
to
space.
It
doesn't
do
parity
by
definition.
B
C
B
A
B
A
B
B
D
B
B
B
G
B
E
B
G
B
G
A
B
Think
when
sort
of
that
the
goal
is
that
when
you're
building
system,
they
think
they're,
so
big
thinking
and
affordable,
roughhousing
copy,
and
so
that's
why
you
build
redundancy
at
this
stage
of
development.
You
do
one
about
it,
because
you're
sort
of
together
is
going
to
be
popular
Billy
or
not.
Failure,
but
eventually
point
they.
Would
you
know
that
system
to
be
more
than
fulfilling
its.