►
From YouTube: Distributed storage and compute with Ceph's librados
Description
Presenter(s): Sage Weil
URL: https://lca2014.linux.org.au/schedule/30079/view_talk
The Ceph distributed storage system sports object, block, and file interfaces to a single storage cluster. The core infrastructure supporting these use cases is a distributed object storage and compute platform called RADOS (reliable autonomic distributed object store). The low-level librados API exports a conceptually simple yet powerful interface for storing and processing large amounts of data and is well-suited for backing web-scale applications and data analytics. In features a rich object model (bytes, attributes, and sorted keys/values), efficient key/value storage (using leveldb on the backend), atomic transactions (including efficient compare-and-swap semantics), object cloning and other primitives for supporting snapshots, simple inter-client communication and coordination (ala Zookeeper), and the ability to extend the object interface using arbitrary code executed on the storage node as part of the IO pipeline. Librados classes are a plugin API that allows computation to be performed on the storage node, avoiding expensive read/modify/write cycles, while still leveraging the replication, atomicity, and high-availability provided by RADOS.
This talk will include a brief overview of the Ceph architecture, and then focus on librados, the API and how it is used, the security model, and some examples of RADOS classes implementing interesting functionality (like cls_lua, which embeds a LUA interpreter for arbitrary code execution).
http://lca2014.linux.org.au - http://www.linux.org.au
CC BY-SA - http://creativecommons.org/licenses/by-sa/4.0/legalcode.txt
A
The
various
ways
you
can
use
it
today,
I'm
going
to
try
something
a
little
bit
different
and
I'm,
going
to
talk
specifically
about
the
object
storage
later
in
SEF
called
burritos
and
how
you
can
write
applications
that
talk
directly
to
that
storage
layer.
So
I'm
going
to
begin
by
setting
the
stage
a
little
bit
and
try
to
motivate
things
by
giving
a
sob
story
about
someone
trying
to
scale
out
a
really
simple
web
application
dealing
with
a
storage
part
of
that
I'll.
Very
briefly,
talk
about.
A
What's
a
phase
with
some
basic
features
of
the
architecture
are
and
then
I'll
ship
to
looking
at
liber8
is
specifically
the
interface
that
used
to
talk
directly
to
the
internal
object,
storage
layer,
I'm,
all
the
various
crazy
things
it
can
do
and
I'll
try
to
finish
up
with
an
email
use
case
that
sort
of
demonstrates
how
you
might
build
an
application.
That's
designed
specifically
to
use
that
storage
back
end
I
feel
free
to
interrupt
me
anyway,
anytime
I'm,
and
ask
questions
I'm.
A
So
imagine
you're
you're
still
in
college
you're
living
in
a
dorm
room
and
you
decide
to
write
some
simple
web
application.
You
know
maybe
you're
scouring
image,
ops,
images
off
the
way
up
and
overlaying
cute
quotes
on
top
of
them.
Something
like
that.
But
you
have
you:
have
a
single
server
you're
writing
much
files
to
a
directory,
maybe
they're
jpg,
something
like
that
life
is
simple
primarily
because
you
don't
have
any
users
you
send
an
email
to
the
canvas.
Email
list
suddenly
have
some
users,
so
you
move
over
this
everything
over
to
some
dedicated
servers.
A
Maybe
you
get
some
something
from
the
planet,
something
like
that
and
you
realize
you
have
to
centralize
your
storage.
So
you
have
a
number
of
web
front-ends
and
everything's
NFS
mounting
some
file
server.
Maybe
it's
just
a
linux
box.
Maybe
it's
a
net
up
whatever,
but
life
is
pretty
simple.
You
know
to
change
your
application
at
all.
A
Your
site
begins
to
grow,
people
actually
start
to
use
it,
and
you
realize
you
have
to
scale
out.
So
usually
this
means
you
buy
a
bigger,
more
expensive
file,
server
from
some
name-brand
vendor
max
out
your
credit
cards
that
works
for
a
while.
But
at
some
point
you
realize
that
and
the
cost
of
these
appliances
tends
to
be
nonlinear
and
the
amount
of
work
you
can
actually
get
them
to
do.
I
mean
solution.
A
Eventually
you
realize
that
you
need
to
scale
out,
so
the
usual
approach
is
to
hash
the
name
of
the
file
content
or
the
file
name,
or
something
like
that
and
use
that
to
distribute
across
some
number
of
servers.
So
you
have
to
content
hash
modulo
the
number
of
servers
and
not
the
whole
bunch
of
them
behind
thinks
everything's.
Everything's.
Fine,
your
site
continues
to
grow
until
you
start
to
deal
with
sort
of
the
various
problems
that
handled
just
by
having
to
manage
all
these
machines.
So
your
files
service
start
to
fill
up.
A
Your
directories
get
really
big.
That
might
be
a
problem
just
for
administrators
who
do
LS
dash
l.
It
might
be
that
the
file
systems,
what
they're
running,
don't
handle
large
directories,
although
that
tends
not
to
be
a
case
these
days.
So
the
usual
approach
here
is
that,
instead
of
hashing
across
file
servers,
you
hash
across
a
bunch
of
shards
will
call
them
small
and
directories.
A
You
have
lots
of
these
shards
stored
on
each
server,
which
means
that
when
you
add
new
back
in
source
servers,
you
can
sort
of
my
great
these
chunks
of
your
data
set
independently
to
other
other
machines,
probably
you're,
using
some
combination
of
our
synced
and
SCP.
Maybe
you're
doing
some
trickery
to
try
to
maintain
a
consistent
view
of
things.
Even
though
your
web
application
is
live
and
it's
reading
and
writing
data.
A
So,
in
particular,
if
you're
the
person
who's
writing
this
web
application.
You
don't
want
to
have
to
think
about
any
of
these
details.
Well,
you
really
want
is
some
sort
of
distributed
magical,
distributed,
store
system?
That's
going
to
sit
behind
your
web
application
and
it's
going
to
deal
with
all
these
nitty
gritty
dirty
details
of
scaling
things
starting
your
data
across
multiple
servers.
Rebalancing
when
you
add
new
servers,
replicating
migrating
healing
when
they're
failures,
all
that
good
stuff
and
in
exchange
for
that
usually
you're
willing
to
give
up
some
flex.
A
So,
even
though
you're
writing
to
a
file
system,
you
don't
need
to
rename
files.
You
don't
need.
Hierarchal
directories,
you're,
actually,
probably
fine
with
the
flat
or
flattest
namespace
and
like
would
be
much
simpler,
took
this
approach
now
turns
out.
It
is
2014
and
there
are
a
number
of
different
solutions
that
all
sort
of
fit
this
bill
today,
I'm
going
to
talk
about
Seth,
because
that's
when
I
know
all
of
that,
but
there
are
other
so
I'll
review
some
of
those
towards
the
end.
So
at
a
very
high
level,
stuff
is
a
distributed
source
system.
A
We
say
it's
distributed
because
it's
designed
to
scale
out
to
a
very
large
scale,
so
tens
of
servers
to
tens
of
thousands
of
servers,
terabytes
exabytes,
so
very
large
I'm.
It's
designed
from
the
ground
up
to
be
fault,
tolerant,
so
there's
no
single
point
of
failure
and
you
can
deploy
it
on
commodity
components
that
may
or
may
not
be
very
reliable,
and
it's
designed
to
be
self
managing
and
self-healing
so
that
when
you
know
disks
fail
and
hosts
go
down
and
so
forth,
the
administrator
doesn't
have
to
directly
intervene.
A
The
system
will
sort
of
deal
with
that
on
its
own
without
too
much
fuss.
Some
very
basic
architectural
features
set
the
design
around
a
number
of
smart
storage
demons.
The
basic
motivation
here
being
that
centralized
coordination
of
a
large
number
of
very
dumb
devices
is
a
lot
of
work
and
doesn't
tend
to
scale
very
well,
and
so
staff
takes
sort
of
an
alternative
approach
where
we
have
a
number
of
what
we
like
to
think
are
very
smart
devices.
A
A
Dip
only
not
yet
so.
The
other
key
feature
is
that
an
SF
is
built
around
a
flexible
placement,
algorithm
called
crush.
The
basic
idea
here
is
that
the
system
has
some
awareness
of
the
hardware
infrastructure
across
the
switch
its
deployed,
so
you
have
disks
and
hosts
hosts
and
racks
racks
and
rose,
and
then
you
have
a
hash
algorithm.
A
That's
far
enough
to
distribute
data
across
that
infrastructure
based
on
some
basic
policies,
so
you
might
say
that
I
want
three
rub
because
of
a
real
bright
object,
but
I
want
them
separated
across
hat
across
racks
or
maybe
across
datacenters.
Whatever
your
policy
might
be
in
the
system
is
smart
enough
to
do
that,
using
a
hash
placement
that
will
avoid
a
many
days.
Sir.
Yes,.
A
So
the
question
is:
what
happens
if
you
have
a
very
hot
shard?
Can
you
intervene
to
the
easy
answer?
Is
that
normally
don't
have
to
because
your
hash
hashing
objects
randomly
across
shards
and
then
the
shards
across
servers,
and
so
your
workload
tends
to
be
very
distributed?
If
you
had
like
one
object,
that's
extremely
hot
or
you're
very
unlikely
and
lucky.
Then
you
would
have
a
problem
and
there
are.
There
are
certain
instances
where
you
can.
You
can
tell
the
system
to
push
load
off
a
particular
server
that
happens
I'm.
A
So
at
a
very
high
level.
Sesh
is
built
around
what
we
call
ratos.
This
is
a
distributed,
object,
storage
layer,
that's
responsible
for
actually
storing
all
the
data
in
the
system.
This
is
the
part
that
scales
from
tens
of
service
at
tens
of
thousands
of
servers
and
make
sure
that
all
the
data
object
store.
The
system
are
replicated
across
multiple
nodes
that
they're
distributed
properly,
that
when
you
add
a
rack
of
storage,
things
sort
of
balanced
out
when
those
failed
thinks
it
replicated.
All
those
sort
of
details
are
handled
by
distributed,
object,
storage
layer.
A
So
from
the
perspective
of
somebody,
that's
using
it
aptly
named
liberate
us.
They
can
just
say
right.
This
object
into
the
object,
store
and
everything
is
sort
of
handled
by
the
rate
of
distributed,
object,
storage,
layer
so
liberate
us
is
the
client-side
library
that's
used
to
access
this
distributed,
object,
store,
I
mean
I
can
bind
it
to
applications
in
C
and
C++.
It's
just
a
shared
library
and
their
wrappers
for
various
languages,
but
it's
Python
or
relaying
or
PHP
or
whatever.
A
But
on
top
of
this
sort
of
basic
object,
storage
service
distributed
highly
reliable,
highly
available
service.
We
built
a
number
of
applications,
so
one
of
those
is
greatest
gateway
which
gives
you
a
restful.
A
higher-level
restful
object,
storage
interface
and
that's
compatible
with
Amazon
s3
and
OpenStack
Swift,
so
I
think
HTTP
gets
inputs.
That
sort
of
thing
there's
also
the
greatest
block
device,
which
is
a
shared,
a
virtual
disk,
that's
backed
by
objects
across
the
across
the
source,
server
across
the
stores,
cluster
and
that's
supported
by
modern
Linux
kernels
and
by
kbm
and
Zen.
A
So
you
can
a
virtual
machines
with
their
virtual
disks
back
by
back
by
the
stores,
cluster
and
the
sports
things
like
snapshots
and
so
forth.
Then.
Finally,
there's
Seth
FS,
which
is
a
full-blown
POSIX
distributed
file
system
that
distributes
all
its
data
across
the
sorcerers
and
the
basic
basic
idea
here
is
that,
because
we
built
this
reliable
and
highly
available
object
store
building
these
higher
level
services
as
much
easier,
because
we
don't
worry
about
replication
and
data
distribution
and
so
forth.
A
That's
all
pushed
down
into
the
lower
levels,
and
it
turns
out
that
in
the
beginning,
radius
is
actually
pretty
simple.
Don't
want
that
many
things
you
could
do
with
objects,
you
just
read
and
write
them
and
it
would
distribute
them
and
replicate
them.
But
as
we
build
these
high-level
services,
we
realize
that
there
are
certain
capabilities
that
were
really
useful
for
building
sort
of
coherent.
A
You
know
snapshot
able
highly
consistent
services
that
could
be
pushed
down
to
the
lower
levels,
though
it
may
call
that
work
much
better
and
the
end
result
is
something
that
we
think
is
pretty
interesting
and
I'd
like
to
think
that
would
be
useful
for
other
applications
that
need
sort
of
scale-out
storage.
So
what
is
liberate
us
then?
A
Labret
dose
is
a
native
library
for
accessing
this
distributed
objects
or
burritos.
It's
just
a
shared
library
with
bindings
and
all
these
different
languages.
It
gives
you
again
a
direct
data
path
of
all
the
storage
nodes,
so
it
speaks
the
native
step
protocol
on
the
backend
and
the
client
talks
directly
to
the
storage
server.
That
has
the
data
that
it
needs
and
it
hides
all
the
details
of
data
distribution,
migration,
replication
and
node
failure
that
your
application
doesn't
have
to
worry
about
any
of
that.
That's
all
handled
behind
the
scenes.
A
So
the
core
of
all
this
is
that
the
basic
data
ball
that
we
supports
so
you're
storing
objects,
object,
is
sort
of
a
lumpy
unit
of
data,
and
that
has
some
name
that
you
can
choose.
You
know
it
can
be
foo
some
string
of
digits
and
there's
some
data
associated
with
that
object.
That's
an
opaque,
byte
array,
so
it
looks
sorta
like
a
file.
It
can
be
anywhere
from
by
stick
gigabytes
and
you
have
bike
granularity
access
to
that.
So
you
can
truncate
it.
You
can
override
a
random
extent
to
it.
A
A
Later
objects
exist
in
the
cluster
inside
of
an
abstraction
called
a
pool
which
is
just
a
logical
collection
of
lots
of
objects
within
a
sort
of
a
separate,
independent,
namespace
I'm.
You
can
name
your
pools
whatever
you
want,
but
the
main
thing
is
that
pools
have
some
basic
properties
associated
with
them.
That's
the
replication
and
data
placement
policy,
so
you
might
say
that
this
particular
pool
has
three
replicas
and
it's
distributed
across
different
racks.
My
cluster,
that's
other
pool
only
has
two
replicas
distributed
across
these
hosts.
A
Maybe
this
other
pool
is
stored
using
a
racial
coding
separated
across
host
using
this
particular
scheme
or
whatever
the
erasure
coding.
Isn't
there.
Yet
it's
coming
in
the
fire
fire
release,
which
is
coming
out
in
February,
so
sort
of
getting
down
to
the
nitty-gritty.
Here
this
is
a
simple
hello
world
type
program,
so
I'm
going
to
use
the
C++
interface
because
that's
tends
to
be
my
language
of
choice,
at
least
in
user
space.
So
you
include
the
header
file.
Of
course
you
just
install
the
backs
and
so
forth
and
you'd
create
a
cluster
handle.
A
A
That's
installed
an
tcf
or
you
can
add
some
additional
calls
to
specify
which
server
you're
going
to
connect
to
and
so
forth,
metal
that
will
go
out
to
authenticate,
discover
the
topology,
the
cluster
and
come
back,
and
then
you
create
what
we
call
an
eye
of
context,
which
is
essentially
sort
of
like
a
filehandle,
that's
associated
with
a
particular
pool.
This
is
a
thing
that
you
use
to
do
actual
io
operations
against
the
cluster.
A
In
this
case,
we're
going
to
talk
to
a
pool
called
my
pool
next,
we'll
create
a
buffer
put
hello
world
in
it
and
call
the
the
right
function
to
write
to
an
object,
called
my
object
and
we'll
put
that
data
in
that
object.
Now
go
out
to
the
cluster.
Tell
it
to
write
it'll
get
applied
to
all
the
replicas
will
get
any
college
and
back
when
that's
that's
safe
and
sound.
We
can
then
go
set
an
attribute
on
the
object
that
that's
what
particular
version
is.
Yes,.
B
A
We
want
to
be
very
careful
that
we
actually
know
that
the
thing
that
you
thought
was
written
was
actually
written,
and
your
code
does
not
to
worry
about
the
case
where
it
actually
wasn't
and
their
other
systems
that
are
designed.
When
you
can
deal
with
those
sorts
of
ambiguities,
then
you
probably
want
a
different
type
of
system
than
SEF,
so
the
next
operation
will
just
set
an
attribute
on
that
object
again.
That
will
go
out
and
do
it
reply
and
of
course
you
can
read
that
data
back.
A
You
know
we
tell
it
to
read
from
my
object
again
into
a
different
buffer.
Some
number
of
bytes
at
offset
0,
and
we
assert
that
it
we
actually
got
the
right
data
back,
so
nothing
too
too
crazy.
Here
its
interface
probably
looks
pretty
similar
to
any
other
sort
of
storage
interface
that
you
would
use
any
other
environment.
So
the
first
thing
that
sort
of
distinguishes
liberate
us
from
other
systems.
A
That
makes
it
interesting,
I
think,
is
the
idea
of
having
object
transactions,
and
this
is
basically
the
ability
to
group
a
number
of
operations
on
a
single
object
into
a
single
request
that
shipped
out
to
the
cluster
and
applied
atomically
to
that
object
and
ensures
that
that
entire
transaction
is
implied
apply
it
exactly
once
so.
As
a
simple
example:
this
we
can
rewrite
our
hello
world
example
as
an
atomic
creation
transaction.
So
we
pair
all
our
buffers
we're
going
to
write
hello
world
and
set
this
attribute
we'd
acquire
declare.
A
This
object,
write
operation
structure,
which
is
essentially
a
description
of
the
transaction
that
we're
going
to
apply,
and
we
simply
call
methods
to
sort
of
build
up
that
transaction
in
memory.
All
the
different
operations
that
we're
going
to
do
so
first
is
going
to
be
an
exclusive
creation
that
will
succeed
if
there's
no
object
and
create
it.
If
the
object
already
exists,
it'll
air
out,
then
you
exist,
then
we'll
write,
all
the
data
and
then
set
the
edge
agree.
A
And
finally,
we
call
the
operate
method,
which
actually
says:
ok,
take
this
transaction,
submit
it
to
the
cluster
and
actually
apply
it
and
that'll
other
succeed
in
its
entirety,
or
it
will
fail.
So
that's
all
well
and
good,
but
it
turns
out
there's
there's
more,
you
can
do
here.
You
can
also
have
transaction
conditional,
so
you'll
notice
that
that
create
event
would
abort
if
the
object
already
exists.
A
So
you
can
actually
have
steps
in
your
transaction
that,
if
they
fail
they'll
make
the
whole
thing
abort
so
typically
on
the
strategy
here
is
to
create
guard
operations
early
on
that
assert
that
some
condition
is
true.
For
example,
they
might
verify
that
an
extended
attribute
has
a
particular
value
that
the
object
version
is
some
particular
version.
Something
like
that,
and
that
means
that
you
can
do
the
equivalent
of
atomic
compare-and-swap
operations
from
the
clients
perspective
that
are
enforced
by
the
storage
system.
A
So
you
can
have
lots
of
clients
in
your
system,
coordinating
implicitly
via
the
storage
system
without
having
to
have
sort
of
some
sort
of
walking
or
coordination
layer
that
you
build
yourself
on
top
of
things.
So
again,
moving
on
from
our
hello
world
example
save
instead
of
saying
world,
we
actually
want
to
say
perth,
and
so
we
can
prepare
a
transaction
that
has
the
new
data
that
we're
going
to
store
in
the
object,
the
expected
old
by
the
attribute
and
the
new
attribute
and
prepare
transaction
that
the
very
first
step.
A
It
asserts
that
the
version
attribute
that's
already
on
the
object,
has
a
specific
value.
That's
what
we
expect
the
one
that
we
set
before
and
assuming
that's
true,
then
the
transaction
will
continue,
will
replace
the
data
with
some
new
content
and
set
the
version
to
version
two
and
when
you
submit
this
to
the
cluster
it'll
go
it'll.
Do
that
validation?
If
it's
true,
it
will
apply
the
transaction
to
all
right
because
and
then
return
to
you.
A
So
the
nice
thing
here
is
that
you
could
have
a
similar
transaction,
this
being
sent
from
every
other
city
in
the
country,
I'm
trying
to
choose
a
different
City,
and
only
one
of
these
is
actually
going
to
succeed,
because
only
one
of
them
is
going
to
be
the
first
one
there
and
it's
going
to
change
them
version.
1
dispersions,
you!
A
So
the
second
interesting
thing
you
can
do
with
labor
liberate
us
that's
a
bit
unusual
is
that
these
objects
can
store
not
just
bite
data
like
a
file,
but
they
can
also
store
key
and
value
data.
So
each
object
can
soar
this
key
value
day.
That's
completely
independent
of
the
file
data,
so
you
can
actually
of
an
object
that
has
all
these
different
types
of
data
stored
sort
of
encapsulated
within
that
logical
unit
and
the
main
thing
with
the
key
value
stuff,
is
that
it's
gives
you
a
random
access,
insertion,
deletion
and
range
query.
A
So
key
value,
you
have
a
table
of
names
and
I
nodes
or
whatever
you
want
to
store
there.
So
this
is
really
good
for
structured
data
and
we
use
this
internally
with
a
number
of
other
stuff
services.
So,
for
example,
the
rate
of
skate
way
which
is
providing
this
s3
like
Swift
like
higher
level
restful
interface,
has
to
maintain
an
index
on
the
user,
buckets
that
indicate
what
objects
exist
in
that
bucket,
so
they
can
go
back.
A
You
can
list
them
later,
and
so
we
store
all
that
index
data
in
a
key
value
object,
and
so,
when
you're
actually
creating
an
object,
you
just
send
an
operation
that
says
insert
this
new
value
into
the
index
and
the
client
side
doesn't
have
to
know
how
big
the
index
was
or
what
offset
to
write
to
any
of
that
set.
The
storage
system
sort
of
manages
all
to
that
low-level
detail
for
you.
Similarly,
the
the
set
distributed
file
system
uses
these
key
value
optics
or
directories.
A
The
third
unusual
or
interesting
thing
that
that
liberate
asst
does
this
gives
you
sort
of
a
very
powerful,
although
somewhat
difficult,
to
use
infrastructure
to
support
snapshots,
so
the
snapshots
and
staff
can
be
either
it's
sort
of
a
high
course
pool
granularity
or
they
can
actually
be
as
narrow
as
a
individual
optic
granularity,
and
the
motivation
here
is
that
we
want
to
support
snapshots
on
smaller
pieces
of
data
in
the
system,
so,
for
example,
for
the
rightist
block
device.
You
have
a
number
of
disk
images.
A
Rbd
allows
you
to
create
snapshots
on
a
per
image
basis,
and
that
applies
only
to
the
objects
that
are
storing
data
for
that
particular
data
storage
that
disk
image.
Similarly,
for
the
ceph
file
system,
it
has
the
ability
to
create
snapshots
for
any
directory
or
subdirectory
in
the
system,
and
we
use
this
capability
to
apply
home
to
the
objects
that
are
storing
file
data
for
files
contain
within
those
directories,
so
that
the
cost
is
that
there's
some
coordination
with
the
cloud
client
code.
A
A
So
if
you
think
of
things
in
sort
of
the
high-level
object-oriented
sense,
where
you
have
some
piece
of
data-
and
you
have
another
number
of
methods
that
manipulate
that
data,
you
can
think
of
raid
us
as
implementing
some
very
basic
methods
like
read
and
write
extent
set
after
view,
delete
things
like
that
greatest
classes
allow
you
to
implement
new
code
new
object
methods
that
are
based
on
the
existing
functionality,
compile
that
couldn't
actually
embedded
into
the
storage
system
and
execute
it
there.
So
there's
sort
of
a
simple
plugin
api
to
lets.
A
You
write
these
new
methods.
The
admin
will
build
it
into
a
shared
object,
employed
on
all
your
object
servers,
and
then
the
clients
can
invoke
that
code.
I'm
using
the
rate
of
say,
p
I,
so
this
sort
of
comes
in
two
flavors.
You
can
have
read
side
methods
that
will,
you
know,
send
a
request,
method,
invocation
off
to
the
source
clusters.
They'll,
do
some
processing
and
return
the
result
and
then
there's
the
right
side
methods
where
you
send
a
request.
It'll,
maybe
do
a
read,
modify
write.
A
Do
some
computation
validation,
generate
a
transaction
that's
and
consistently
and
safely
apply
to
all
the
replicas
on
that
object.
So
a
simple
example
of
a
classmate
they
do
mine
implement
and
this
one
computes
an
md5
sum
on
the
object.
Contents.
So
there's
this
is
a
simple
scene
method
that
you
implement.
There's
this
context,
argument
that
you
can
mostly
ignore
it's
used
for
internal
tracking.
The
important
thing
here
is
that
there's
a
buffer,
that's
being
passed
into
your
function
and
there's
a
buffer,
that's
being
passed
out
of
it.
A
So
this
is
a
data
that
effectively
the
arguments
that
are
being
passed
from
the
client.
This
is
the
returned
value.
The
return
data
is
being
returned
for
the
read
side
methods.
So,
in
this
case,
we
we
stat
the
object
figure
out
how
big
it
is,
the
app
the
object
doesn't
exist
or
we
get
an
error.
We
just
returned
immediately
and
we
read
all
the
data
for
the
object
into
some
in
memory
buffer,
and
then
we
call
the
md5
library
to
calculate
the
digest,
which
is
128
bits
of
gibberish,
append
that
to
the
output
buffer.
A
That's
going
to
be
passed
back
to
the
client
and
then
return
success,
and
in
order
to
invoke
this
from
some
client
code
on
the
other
side
of
the
network,
will
make
this
example
a
bit
more
interesting.
By
doing
a
compound
operation,
we're
actually
going
to
be
performing
multiple
read
operations
in
the
same
sort
of
single
request.
So
we
declare
an
object:
read
operation,
just
sort
of
a
compound
transaction.
First,
we're
going
to
stop
the
object
figure
out
how
big
it
is,
what
its
modification
time
is
and
then
we're
going
to
invoke
the
class
method.
A
A
Yes,
yes
and
it's
the
administrators
job
to
make
sure
that
actually
happens
so
and
then
you
know
we
call
the
operate
method
which
gives
the
transaction
to
the
library
it
goes
and
performs
both
of
these
operations
and
comes
back
and
returns,
and
at
that
point
we
can
assert
that
we
have.
They
got
an
error
or
we
actually
got
no
hundred
twenty
eight
bits
of
stuff
back
and
printed
out.
A
So
this
is
a
pretty
simple
example.
You
can
imagine
their
allotted
from
possibilities
of
things
that
you
can
do
with
this
so
sort
of
going
back
to
a
motivating
example
from
storing
images
in
the
source
cluster,
with
each
each
object
as
a
big
jpeg
or
something
you
can
imagine
a
class,
that's
actually
doing
an
image.
Manipulation
might
be
rotating
an
image
and
rewriting
it
might
be
generating
thumbnails
and
returning
it.
It
could
be
applying
some
filter
to
it.
A
You
can
imagine
if
you're
you're
streaming
out
structured
data
and
some
logs
type
format,
you
could
have
a
class
that
is
going
through
and
filtering
the
records
and
looking
for
some
matching
pattern
and
only
returning
the
results.
So
instead
of
shipping,
the
haystack
over
the
wire
you're,
only
shipping,
the
needle
that
you
find
the
rate
of
block
device
and
rate
of
skate
wing
have
their
own
classes
that
they
use
mostly
to
enforce
sort
of
a
clean
interface
to
the
objects
that
are
storing
metadata
about
disk
images
and
so
forth.
So
they
can
do
argument.
A
So
so
all
that
said
so,
as
I
said,
that
the
beginning
rattus
began
as
a
relatively
simple
objects
toward
just
storing
lots
of
data
to
store
file
data.
First,
a
file
system
I
mean
as
we
develop
these
other
services.
The
rightist
block
device
and
Rios
gateway
realized
that
it
was
helpful
to
push
some
of
this
capability
down
into
the
storage
layer.
A
For
the
sort
of
last
bit,
I'd
like
to
look
at
sort
of
an
example
of
a
type
of
application
that
might
be
able
to
take
advantage
of
some
of
this
functionality
to
have
sort
of
a
scale-out
storage
back
end.
That's
hopefully
little
bit
more
efficient
in
and
attractive
and
what
the
current
solutions
are.
So
one
of
my
sort
of
pet
peeves
is
the
way
that
a
lot
of
service
advisors
deal
with
email,
so
it
turns
out
the
email
hosting
tends
to
be
a
pain.
The
pain
is
about
for
a
lot
of
hosting
providers.
A
Typical
solutions
tend
to
use
tools
like
postfix
and
courier
and
dub
cot
for
imap
and
so
forth,
but
it
turns
out.
Usually
people
are
storing
things
in
maildir
format,
which
is
not
the
most
efficient
thing
in
the
world.
They
tend
to
use
NFS
file
servers
and
it
tends
to
hammer
them
very
hard
and
performance
tends
to
suck
and
it's
expensive
and
file
servers
fail,
and
it's
not
very
not
very
nice
maildir
in
particular,
is
very
inefficient,
especially
when
you're
trying
to
list
the
contents
of
a
mailbox.
A
A
On
the
other
hand,
stuff
is
sort
of
an
attractive
back
in
from
high
level,
because
it
already
gives
you
sort
of
reliability.
Scale
abilities,
you
can
just
add
service
as
you
need
them.
Everything's
replicated
has
a
simple
consistency
model,
so
you
have
to
worry
about
these
weird
issues
where
things
appear
and
then
disappear
that
sort
of
thing.
So
the
question
is:
could
we
make
this
work?
A
So
the
nice
thing
is
that
the
data
model
for
email
is
relatively
simple
right.
So
you
have
you
have
messages
that
contain
some
sort
of
raw
data,
just
a
bunch
of
headers
in
which
a
body
so
that
clearly
maps
onto
just
the
the
byte
array,
that's
associated
with
an
object.
So
I
was
pretty
straightforward.
Sometimes
messages
have
some
mind
data
structure
associate
with
them.
They
have
some
structure,
so
it
might
be
that
you
want
to
you
know,
index
some
of
that
and
sort
in
attributes.
So
that's
easy
to
access
things
later.
Maybe
that's
a
win.
A
Maybe
not!
I'm
actually
don't
know
enough
about
mine
to
know
if
that's
even
at
a
difficult
problem,
but
clearly
you
can
easily
map
a
mail
message
to
an
object.
That's
pretty
simple:
you
also
have
mail
boxes
which
are
simply
collections
of
lots
of
email
messages,
so
you
have
references
and
then
we
unboxed
all
the
different
email
messages
that
it
contains
that'd
be
very
easy
to
store
as
keys
and
values
in
the
mailbox
object.
A
A
One
of
the
easy
optimizations
that
you
can
do
here
is
that
usually
things
like
I
met
will
do
list
objects
or
list
messages
and
get
back
only
the
headers,
but
not
the
message
content.
So
you
could
easily
just
take
all
of
the
message
headers
associated
with
the
messages
and
shove
them
in
the
mailbox,
as
the
value
associated
with
those
key
value
pairs.
So
a
little
bit
of
extra
storage
there.
But
when
you
list
the
contents
of
the
message,
it's
all
on
one
object.
It's
all
in
one
place
on
disk.
A
You
can
list
the
context
very
quickly
and
of
course,
just
around
things
out.
You
have
users
in
your
mail
system
and
the
up,
mailboxes
and
so
you'd
have
some
object
per
user,
that
maps
to
all
the
different
mailboxes
that
they
have
and,
of
course
now
a
box
is
mapped
in
the
messages
so
relatively
straightforward
and
it
sort
of
conveniently
maps
onto
both
the
the
data
portion
of
objects
and
the
key
value
capabilities
that
rattus
provides
so
sort
of
a
simple
example.
A
The
first
thing
you
want
to
do
here
is
actually
deliver
mail,
so
you
have
a
message.
Come
in
over
f
smtp,
you
generate
some
unique
message:
ID,
maybe
just
use
the
one.
That's
in
the
existing
headers.
You
write
the
message
object,
and
so
the
last
thing
you
need
to
do
is
add
a
reference
to
that
message
from
the
mailbox
object.
So
you
need
to
add
a
key
value
pair
referencing
the
message,
but
you
also
need
to
update
the
message
count
for
the
mailbox
so
that
you
know
that
the
size
increased.
A
It
turns
out
that
you
want
to
do
both
of
those
things
atomically
so
that
things
don't
get
out
of
whack
and
so
that's
a
perfect
opportunity
to
use
a
Raiders
class.
So
you
can
in
vote
a
mecca
and
the
method
that
says
add
message
and
it
would
write
the
key,
read
the
old
size
and
per
minute
and
write
a
new
size
all
in
some
nice
convenient
atomic
transaction.
So
that's
all
well
and
good.
But
what
happens
when
you
have
an
IMAP
client,
that's
open
and
it's
idling,
and
it's
waiting
for
you
coming
incoming
messages.
A
When
you
have
that
message
come
in
over
the
wire,
you
want
to
know
that
you
want
to
make
sure
that
the
I'm
at
client
can
notify
that
something
has
arrived.
So
it
turns
out
that
liberate
us
also
as
a
whiz
being
feature
that
we
call
watch
notify.
So
the
basic
idea
here
is
that
clients
can
establish
an
interest
in
an
object
which
call
a
watch,
so
they
send
a
request
that
says:
I
want
to
subscribe
to.
A
Essentially,
they
come
out
over
that
object
and
then
clients
can
also
send
a
notify
message.
That's
distributed
to
all
watchers
on
that
particular
object.
So
you
can
imagine
in
this
email
case
you
might
have
a
bunch
of
imap
clients
that
have
where
the
server
process
has
that
mailbox
open.
So
they
subscribe
to
wach
notifications
on
those
mailboxes
and
then
when
a
message
is
delivered,
or
maybe
when
you
delete
a
message
from
the
mailbox,
something
like
that
I
notify
request
is
sent
that's
distributed
to
all
the
current
Watchers,
so
that
process
gets
notified.
A
That
there's
been
a
change.
They
all
acknowledge
back
that
that
they
actually
process
that
and
the
person
who
sent
they
notify
finds
out
that.
Yes,
everybody
has
heard
that
I
made
this
change
and
things
things
can
continue.
So
that'd
be
kind
of
nice,
because
I
think
currently
I'm
app
servers
they'll
sit
there
and
pull
on
a
Melder
looking
in
the
new
folder
to
see
if
there's
been
a
change,
although
look
at
the
directory
and
time
or
some
some
kludgy
thing
that
doesn't
work
over
NFS
and
all
those
problems
would
go
away.
A
When
you
have
sort
of
a
1st
level
support
in
the
object
store
for
the
capability
that
you
really
need
it,
so
I
just
told
you
that
liber8
O's
can
do
everything
under
the
Sun.
There
are
a
lot
of
things
that
actually
can't
do
so.
I
want
to
sort
of
make
it
clear
what
where
it
might
be
a
good
fit
and
where
it
might
not
be
so
librettist
doesn't
stripe
large
objects
for
you.
A
So
if
you
want
to
store
a
big
one,
terabyte
thing,
you
want
to
stripe
that
over
lots
of
liberators
objects
and
not
sorted
one
big
bitrate
of
objects,
because
the
rate
of
subject
is
all
stored
on
one
sort
of
set
of
servers
as
one
sort
of
unit.
This
is
exactly
what
the
writers
block
device
and
the
raiders
gateway
in
the
stuff
filesystem.
A
Do
they
stripe
over
lots
of
lots
of
rate
of
subjects
you
can't
rename
objects
and
that
sort
of
antithetical
to
the
way
that
rate
us
works,
because
it
uses
the
object
name
to
figure
out
which
server
is
going
to
be
stored
on?
And
so,
if
you
rename
it,
then
you
have
some
weird
Multi
multi
server.
Communication
that
doesn't
work,
you
can't
do
multi-object
transactions,
so
we're
not
a
database,
so
we're
not
really
trying
to
be
so.
A
If
you
have
two
objects
that
are
sort
of
the
system
that
need
to
be
updated
in
tandem,
then
you
need
to
do
the
coordination
about
doing
that
in
a
safe
way
on
your
own.
So
usually
people
use
a
two-phase,
commit
or
intent
log
sort
of
tried
and
true
strategies
there
and
that's
exactly
what
the
file
system
in
Rideau
skate.
We
do.
They
have.
The
rooms
are
mechanism
for
or
keeping
things
consistent
rate.
Us
doesn't
do
secondary
object
in
indexes,
so
you
can
only
find
an
object
in
the
system
based
on
its
name.
A
You
can't
send
a
query
to
the
system
that
says,
give
me
all
objects
that
have
attributes
Lou
equals
bar.
They
won't
do
anything
like
that,
although
their
other
systems
that
do
so
that's
a
requirement,
you
can
look
at
those
and
you
can
only
list
objects
by
pool
in
sort
of
a
complete,
incoherent
way.
So
you
can,
you
can
iterate
all
the
objects
in
the
system,
but
they're
returned
and
hash
order,
which
is
generally
not
very
useful.
A
C
A
So
the
question
is:
how
efficient
is
it
it's
not
something
that
we
optimized
for
at
all.
That
said,
it's
not
inefficient.
It's
just
inherently
expensive,
because
you're
literally
new
Meridian
every
object
in
the
system,
so
you'll
send
a
request.
You'll
get
back
the
first
thousand
entries
you
send
a
request
to
get
the
next
thousand
and
it'll.
C
A
A
The
question
is,
can
you
do
wildcard
searches
for
objects
and
the
answer
is
actually
you
can?
You
can
say
list
objects
only
with
this
prefix,
but
at
the
same
time
it's
not
efficient.
So
it'll
it'll
actually
list
all
objects
and
we'll
just
only
return,
someone
that
match.
So
it's
not
that
they
go
in
yeah.
A
So,
just
to
put
a
little
rewrite
of
sort
of
in
perspective,
with
respect
to
other
object
stores
and
other
sort
of
scale
out
open
source
solutions
rate.
Us
is
a
strongly
consistent
object
store.
So
it's
a
CP
system
sort
of
in
the
cap
context
on
the
favors
consistency
of
our
availability,
so
their
number
of
systems
that
sort
of
take
different
design
positions
within
that
overall
space,
so
Swift
is
by
the
one
that
you're
most
familiar
with.
That's
part
of
OpenStack.
That's
a
distributed.
A
Object
store,
that's
restful,
so
it
is
generally
an
AP
system,
and
so
it's
typically
last
rider
wins
uses
a
restful
api
I'm.
The
data
modeler
model
is
much
simpler,
so
you're
storing
big
objects
using
a
get
or
put
sort
of
an
atomic
thing.
You
can
associate
metadata
with
them,
but
you
don't
know
sort
of
all
this
weird
key
value
stuff
that
you
can
surround
side
of
objects.
A
Gluster
FS
is
I,
think
you
best
describe
it
as
a
CP
system,
although
I
think
it's
sort
of
fuzzy
I'm,
depending
on
what
you're
trying
to
do
I
mean
it's
specifically
a
file
based
model.
So
it
looks
like
a
file
system.
Everything
is
a
file,
so
your
application
has
to
be
coded
to
store
everything
in
a
byte
array,
so
that
may
or
may
not
be
appropriate
for
your
use
cases.
A
Sort
of
shifting
to
the
other
side
of
more
the
database
side
of
things
no
sequel
world
react
is
another
AP
system
that
sort
of
faithfully
implements
the
dynamo
design
that
came
out
of
Amazon
a
couple
years
back.
So
that
means,
when
you
have
divergent
replicas,
you
have
flexible
ways
to
resolve
those
conflicts
in
your
application.
You
can
do
things
like
two
different
versions
of
a
shopping
cart
and
they
both
add
something
you're
just
going
to
stick
them
both
in
there
and
Marge
that
incoherency
in
a
in
a
application
specific
way.
A
It
has
a
simple
key
value:
data
model,
I'm
generally
targeted
towards
small
Olympics
I.
Think,
although
you
can
store
like
Jason
structured
data
in
the
value
and
it
will
generate
secondary
indexes,
can
do
things
like
mapreduce
queries
across
it
within
some
constraints.
So,
there's
that
and
then
sort
of
moving
up
in
terms
of
complexity,
there's
Cassandra,
which
is
also
an
80
system,
but
it
has
much
much
more
complicated
data
model.
So
it
looks
a
lot
more
like
a
database,
you
have
rows
and
columns.
You
have
this
extra
time
dimension
and
it
also
has
secondary
indexes.
A
So
you
can
do
these
sort
of
sequel
like
queries
across
the
data.
That's
stored,
stored
cassandra,
so
sort
of
wrapping
everything
up
in
conclusion,
I
think
that
the
main
point
is
that
usually
the
first
thing
that
people
reach
for
at
least
people
of
my
generation
reach
for
when
they're
building
a
scalable
storage
application
as
a
file
system,
but
file
systems
are
typically
poor
matches
for
scale
out
applications
so
usually,
depending
on
how
are
you
doing?
A
You're
gonna
have
to
deal
with
all
the
sort
of
ad
hoc
starting
to
make
sure
that
you
distribute
across
lots
of
servers.
It's
sort
of
annoying
to
reinvent
the
wheel.
The
file
abstraction
is
often
overly
simple
for
the
kind
of
data
that
you're
really
in
a
store
and,
at
the
same
time,
the
other
complexity
of
the
file
systems
offer.
In
terms
of
directories
and
renames
and
so
forth,
aren't
really
important
or
useful
for
most
scale
out
applications
where
you're
just
shoveling
image
data
into
a
storage
system.
A
On
the
flip
side,
something
like
liberate
O's
gives
you
transparent,
scaling
replication
very
soon,
we'll
have
a
researcher
coding
just
also
very
exciting.
You
have
a
richer
object
data
model,
so
you
can
store
bite
some
objects,
attributes
on
this
key
value
data.
You
have
it's
a
rich
API
with
these
sort
of
transaction
things,
snapshots
watching,
notify
and
most
interestingly
I
think
you
can
extend
that
functionality
using
this
class
mechanism
that
lets
you
sort
of
push
your
own
code
into
the
storage
layer.
So
you
can.
A
You
can
push
as
much
of
the
computation
in
your
distributed
system
to
the
actual
storage
nodes
close
to
the
data
and
use
that
as
a
fundamental
building
block
for
your
system.
So
that's
it.
I
have
a
couple
of
URLs
here.
Naturally,
the
best
reference
for
for
coding
to
a
new
API
is
to
look
at
the
header
file
that
finds
interface.
In
our
case,
we
have
doxygen
comments
for
most
of
methods.
The
c++
c
1
is
very
well-documented.
The
c++
one
is
slightly
us
documented.
A
A
Python,
yes,
so
the!
If
you
look
at
all
the
languages,
the
C++
one
is
the
most
expensive.
It's
the
one
that
has
support
for
all
these
transactions
and
so
forth,
because
that's
the
one
that
we
use
internally
for
ratos
gateway
and
RVD
and
the
python
bindings
are
after
the
sea.
Bindings
of
python
bindings
are
probably
the
most
complete
they're
used
by
some
Parvati
rapper
libraries
and
the
barber
tea
and
so
forth
that
are
used
by
OpenStack
and
some
other
things.
There.
A
For
the
the
transactions,
yet
in
an
interface,
although
be
pretty
straightforward,
add
it
just
we're
sort
of
waiting
for
a
user
to
ask
for
it
and
they'll
do
it?
The
Erlang
bindings
I
think
are
pretty
complete
their
sites
in
production.
We
have
customers
that
are
building
Dropbox
things
and
they're
using
Erlang
I,
don't
know
what
state
the
PHP
ones
are
in
yeah.
B
A
A
It
is
you
have
to
implemented
in
C,
so
the
question
is:
what's
the
execution
context
for
the
classes
that
implement
that
are
run
on
the
storage
nodes?
So
you
it's
written,
you
have
to
write
your
code
to
a
C
API
and
those
those
functions
are
invoked
directly
in
the
eye
opapp.
So,
instead
of
calling
into
an
internal
function,
that's
doing
a
right
or
something
like
that.
A
A
Yeah
yeah,
so
it's
it's
yeah,
so
you
can
mess
things
up,
yeah,
so
you're
running
you're
running
in
the
same
context
as
the
sort
of
server.
So
you
don't
want
to
deploy
bad
code.
If
you're
worried
about
you
know
sandboxing
things
more
tightly,
then
you
could
write
a
class
that
implements
a
sandbox
and
then
runs
something
in
a
sandbox,
for
example,
in
a
Louis
script,
or
something
like
that
yeah.
The
question
is
whether
the
execution
overhead
is
minimal
and
yes
exactly
that's
that's.
The
reason
is
somehow
I.
C
A
The
question
is:
if
there
are
restrictions
on
how
much
memory
you
can
use
encouraging
your
in
sandbox
I
mean
currently,
there
are
no
restrictions.
So
if
you
want
to
use
a
lot
of
memory,
that's
fine.
Your
your
OSD
process
will
just
get
really
big,
and
hopefully
your
post
as
enough
memory
to
deal
with
that,
if
not
a
little
crash
or
something.
A
So
I
think
I
think
if
you
need
an
additional
level
of
protection,
then
you
can
build
in
whatever
sandbox
you
want.
So
originally,
when
we
were
designing
this,
we
were
considering
just
building
a
Python
interpreter
in
and
actually
using
that
as
a
sandbox,
but
in
the
end
it
seemed
useful
to
be
able
to
also
implement
native
methods
that
run
a
higher
priority,
and
you
can
always
implement
that
sandbox
as
a
class
to
sort
of
layer
it
layered.
A
So
that
the
question
is
what
happens
if
you
have
a
walk
on
an
object
and
you're
on
an
eye
on
contacts
and
if
it
gets
interrupted?
The
short
answer
is
there
is
no.
There
is
no
locking
so
the
requests
that
are
generated
by
the
client
and
sent
to
the
cluster
or
sort
of
discrete
requests
that
execute
atomically
in
isolation,
and
so
there's
no
sort
of
stateful
walk
that
you
can
take.
That
said,
there
is
a
radis
class
called
lock
that
implements
a
cooperative
locking
scheme,
as
the
rate
of
the
class
underscore
lock
is
out
here.
A
A
Yes,
so
there
is
in
the
capi
there's
a
rate
of
lock
and
that,
as
X
goes,
are
just
wrappers
around
calling
the
lock
class.
So
it's
purely
cooperative
locking.
So
it's
similar
to
POSIX
locks
where
the
system
will
keep
track
of
who
holds
locks
and
you
can
break
them
by
just
removing
their
entry,
but
it's
totally
cooperative.
So
they
only
enforce
anything.
If
you
actually
check
to
see
if
you
hold
a
lock
or
not
so
just
to
sort
of
clarify
what
the
way
that's
usually
used
is
with
the
block
device.
A
For
example,
you
can,
you
can
take
a
lock
on
a
device
and
then
you
use
it
and
if
somebody
else
comes
along
and
they
want
to
lock
it
they'll
try
to
take
the
lock
they'll
fail,
they'll,
break
your
luck
and
blacklist
your
identifier,
your
sort
of
your
cluster
ID,
and
so
you
will
no
longer
be
able
to
talk
to
the
cluster.
So
it'll
fence
you
out
and
then
they'll
break
your
lock
and
then
take
over.
C
A
A
Right
yeah,
so
the
question
is
whether
there's
blocking
if
you
have
simultaneous,
read
and
write
requests.
The
short
answer
is
that
that's
essentially
see
realized
at
the
server
side,
so
the
request
will
come
in
they'll,
get
into
Q,
the
read
will
happen
and
then
the
right
weapon,
or
vice
versa,
so
there's
sort
of
an
implicit
orderings.
A
Right,
it
could
be
that
it
could
be
that
you
send
a
request
and
it
takes
a
long
time
because
the
cluster
is
busy
doing
another
operation
in
front
of
it.
That's
slow
or
maybe
there's
some
recovery
going
on,
and
it
has
to
get
fetch
the
right
copy
that
object
or
something
like
that,
but
in
general,
there's
no
sort
of
there's
no
steak
fulness
in
the
request
that
you're
doing
so,
unlike
a
traditional
file
handle
or
something
sort
of
an
atomic
atomic
unit
of
operation.
Yes,.
A
The
short
answer
is
we
haven't
done
any
performance
comparisons
on
size
of
s
recently,
so
most
of
the
performance
testing
I
did
was
back
when
I
was
working
actively
on
sephis
specifically,
and
now
it's
mostly
around
making
sure
that
the
metadata
pass
were
optimized,
so
that
you
would
actually
wait
for
disk
I/o
when
you're
doing
things
like
creations,
so
I,
don't
I,
don't
have
any
sort
of
good
numbers.
I
think
the
generally
it's
just
going
to
depend
on
how
fast
your
your
radis
notes
are
so
on
clients,
I
general.
A
You
can
saturate
the
network
interface,
it's
you
know
it's
shipping,
the
pages
directly
to
the
ethernet
or
whatever
the
network
stack
without
copying
them
and
so
forth.
So
it's
really
just
the
speedy
or
network
interface.
That's
limiting
you
in
the
client
side
on
the
server
side.
It's
just
how
bad
how
fast
your
disks
are.
The
fact
that
has
to
replicate
it
on
multiple
servers
so
and
there's
some
of
processing
overhead.
It's
writing
it
into
a
file
system,
and
so
there's
some
overhead
there.
So
it'll
it'll
slow
down,
but
there's.
C
A
Right,
so
the
question
is
whether
the
translation
from
the
POSIX
file
interface
to
the
object
interfaces
is
significant
and
I
think--not
particularly
on
that
on
the
server
side.
So
all
this
really
happening
is
when
you
do
a
POSIX
right,
it's
going
directly
to
the
page
cache
when
it
does
right
back.
Those
pages
are
being
passed
to
the
ceph
driver.
It's
calculating
the
striping
and
just
mapping
those
pages
to
a
particular
object,
and
it's
just
shipping
them
over
the
network
to
some
storage
server
and
then
the
source
server
is
handing.
B
A
Yeah,
so
the
question
is:
what
do
we
do
for
multiple
regions,
high
latency
links
running
a
single
cluster
across
multiple
countries?
So
we
generally
don't
do
anything
special
for
that,
so
that
I
mean
feature
or
whatever
property
of
rate.
Us
is
that
all
rights
are
synchronous,
so
it'll
right
to
all
right
because
of
the
object
once
all
the
acknowledgments
are
gathered
and
reply
back
to
the
client.
So
you
can
spread
a
cluster
across
a
wide
geography
or
and
very
slow
links,
and
it
just
means
that
those
rights
going
to
be
really
slow.
A
And
if
you
can
tolerate
that-
and
you
don't
care,
that's
fine,
and
if
you
can't,
then
that's
not
a
good
idea.
That
said,
there
are
lots
of
users
that
distribute
stuff
clusters
across
multiple
data
centers
in
I,
guess
small
countries.
Then,
for
example,
typical
European
countries
have
only
a
few
milliseconds
to
get
from
one
border
to
the
other,
and
so
they
can
actually
just
distribute
the
cluster
around
the
country,
and
that
does
those
latency
levels
are
acceptable
for
those
particular.
These
cases.
B
A
Question
is
what
about
network
splits
and
so
stuff
is
designed
to
deal
with
that
gracefully.
So
there
are
collection,
nodes,
a
subset
of
the
cluster
that
are
called
monitors
that
run
paxos,
essentially,
and
so
there's
majority
of
voting
essentially,
and
so,
whichever
part
of
the
cluster
you
happen
to
be
in
that
can
access,
the
majority
of
those
monitors
is
the
one
that
has
the
consistent
view
and
can
continue
reading
and
writing.
And
then
the
other
people
who
are
on
the
smaller
partition
can't
do
anything
yeah.