►
From YouTube: New Features for Ceph with Cinder and Beyond
Description
Speaker: Josh Durgin, Inktank
Ceph is an open source distributed object store, network block device, and file system. Ceph can be used for object storage through its S3 and Swift APIs. It can also provide storage for network block devices, with the thin provisioning and copy-on-write cloning features necessary to support large-scale virtualization.
Since the Folsom release, Cinder makes block storage for backing VMs a first class feature in OpenStack. Block devices can be created from images stored in Glance, and with Ceph's RBD behind both, new VMs can be created faster while using less space. In the latest Ceph 'Bobtail' release, you can start many VMs instantly by cloning from templates. Also, on the storage backend side, you will see increased I/O performance due to improved threading.
This session will cover an intro to Ceph, the current status of Ceph and Grizzly, the latest features of the Ceph Bobtail release and also the technical implications and the advantages of block storage within OpenStack.
For more summit videos, visit: http://www.openstack.org/summit/portland-2013/session-videos/
A
Everybody
I
think
it's
about
time
to
get
started
so
before
we
started,
I
want
to
get
a
show
of
hands
how
many
you
have
used
stuff
before
all
right,
pretty
good
number
I
still
hope
so.
I
want
to
go
over
a
bit
of
background
about
stuff.
First,
for
those
of
you
haven't
used
it
before
so
stuff
was
originally
created
by
a
sage.
A
Vial
and
sage
is
just
a
brilliant
guy
he's
been
doing
interesting
things
all
his
life
when
he
was
in
high
school
he
created
web
ring
may
have
heard
of
this,
which
was
bought
by
Geocities.
Then
he
went
to
college.
He
found
in
dreamhost
I
just
thought.
Well,
he
was
at
DreamHost
yeah.
He
encountered
these
large
like
sands,
that
you
use
to
store
all
these
files
and
I
had
lots
of
interesting
problems
managing
them,
and
he
was
very
interested
in
storage,
so
he
decided
to
pursue
a
PhD
and
the
result
of
that
PhD
was
SEF.
A
So
SEF
is
a
distributed.
Storage
system,
which
was
designed
to
scale
as
white
as
possible
to
petabytes
in
extra
bytes
and
beyond
and
sages
whole
philosophy
and
designing
stuff,
is
to
make
it
very
easy
to
manage.
So
instead
of
having
one
large
box
like
might
have
that
many
things
fail
inside
of
it
that
you
can't
access.
You
want
it
to
have
system
that
you
could
fully
interact
inside
of
and
I
mean
I
meant
making
it
all
software
based
so
stuff
is
fully
software
based
his
runs
on
commodity
hardware
and
it's
very
fun.
A
It
tries
to
make
make
everything
it
much
easier
for
you
by
automatically
having
multiple
copies
of
data,
making
everything
consistent
and
trying
to
make
the
life
of
an
admin
and
the
user
much
easier.
So
it
really
helps
bring
down
your
cost
in
time
and
money.
Stuff
is
also
I
tried
to
be
very
flexible,
so
it
didn't.
A
Although
originally
we
started
as
a
file
system
to
create
a
final
system,
I,
probably
compliant
file
system,
it
was
architected,
it
was
architected
in
a
way
so
that
it
could
be
bass
fillet
or
for
developing
many
other
kinds
of
things.
So
today
we
have
block
storage
and
object.
Storage
I
built
directly
on
top
of
stuff
and
stuff
file
system
is
in
fact,
I
have
a
lesser
part.
So
far,
less
developed
and
so
stuff
stuff
is
all
open
source.
It's
all
licensed
under
the
LGPL,
there's
nothing
secret
about
it
at
all.
A
Everything's
nothing
get
up,
and
you
can't
go
down
loaded
today,
so
I
want
to
go
over
a
bit
of
stuff
architecture
asset
and
how
stuff
works
internally
and
then
go
over
some
recent
changes
in
stuff.
In
the
last
six
months.
What's
happened?
What's
going
to
happen
in
the
next
release
and
maybe
what's
going
possible
features
in
the
future,
so,
like
I
said
before,
stuff
is
trying
to
be
a
very
generalized
layer
at
the
lowest
level,
and
this
lowest
level
is
is
a
strongly
consistent.
A
Reliable
object,
store
called
Vito's,
and
this
basically
means
that
you're
storing
objects
in
the
object
store,
but
also
supplies
more
than
just
a
simple
reads
and
writes:
it
has
transactions
that
you
can
add
arbitrary
operations
to
so
it's
a
very
rich
environment
in
which
to
develop
other
applications.
So
you
can
see
there's
several
different
layers.
Above
the
basic
rias
level,
you
can
have
the
Rios
gateway,
which
is
an
HTTP
REST
API
that
provides
a
swift
and
s3
interface
to
the
underlying
rate
of
storage.
A
There
is
a
staff
block
device
for
RBD,
which
is
a
virtual
block
device,
striped
over
objects
in
the
cluster.
And
finally,
there
is
that
stuff
filesystem,
which
is
apply
this
compliant
file
system
and
which
has
been
opportunity
in
the
Linux
kernel
since
2,
6,
3,
2
or
3
4,
and
has
also
has
a
fused
client.
If
you
don't
want
to
install
a
new
chrome,
wide
role.
A
So
how
does
stuff
work
internally?
Well
at
low
level?
All
staff
needs
is
a
disk
and
a
file
system.
So
in
SEF
there
are
three
kinds
of
storage
servers.
There
are
the
most
basic
is
the
OSD
or
object
storage,
and
even
just
this
sits
on
top
of
any
kind
of
regular
Linux
file
system
could
be
butter,
FS,
x,
FS
or
exp,
4
or
even
ZFS.
More
recently,
L
n
needs
is
extended,
attribute
support
and
these
OSDs
form
the
basis
of
the
cluster.
They
store
all
the
data
in
n
cells.
A
A
The
OS
DS
will
automatically
me
replicate
anything
that
go
that's
missing
or
anything
that
needs
to
be
have
more
copies
made
of
it.
The
three
M's
you
see
there
are
called
the
monitor
servers
and
the
monitor
is
basically
provide
a
record
of
cluster
state,
so
they
run
the
packs
with
algorithm,
which
makes
that
make
sure
that
they're
consistent
with
each
other,
and
they
can't
have
a
switch
planes
inherent
scenario
they
keep
track
of,
like,
for
example,
which
OCS
are
up
or
which
oversees
are
down
and
they're,
not
in
a
data
path.
A
They
just
provide
this
map,
of
which
OS
knees
are
up
and
down
right
now
or
tomorrow,
there's
up
and
down
right
now
to
the
clients
and
to
their
OS
teams.
So
they
can
tell
where
the
whether
it
needs
to
go
so
generally
and
the
OS
G's
run
on
top
of
one
disk.
You
could
essentially
run
them
on
top
of
evade
if
you
have
a
very
large
storage
array
or
a
very
large
server
that
has
that
say,
70
disks,
maybe
you
don't
have
enough
memory
or
enough
processor
power
to
support.
A
A
Basically,
a
notice
to
use
responsible
for
managing
all
the
storage,
all
the
objects.
There
is
no
nothing.
Nothing
stored
is
stored
on
the
monitors
everything
stored
on
the
OS
keys
and
they
all
talk
to
each
other
in
a
peer-to-peer
fashion
so
that
they
notice
when
other
Aziza
fails
and
they
report
that
up
to
the
monitors
and
then
the
monitors
can
update
that
cluster
map,
which
then
gets
propagated
via
got
a
gossip
protocol
to
the
rest
of
the
cluster.
So
the
question
now
is:
what's
happening
all
the
time.
A
So
that
that's
a
basically
the
state
of
the
cluster,
but
how
do
you
actually
access
the
cluster?
How
do
you
find
out
where
your
data
is
well,
it's
several
different
approaches
that
you
can
take
to
finding
where,
where
data
is
stored
among
a
group
of
servers,
the
most
basic
is
you
read
that
write?
It
write
it
where
your
data
somewhere
some
set
of
servers
and
then
store
in
some
other
service
where
that
set
of
servers
was
and
I'm
mapping
from,
say
the
object
name
to
set
of
servers
and
that's
okay,
but
it
doesn't
really
scale.
A
It
adds
another
layer
in
the
data
path
where
you
have
to
keep
looking
up
where
you're,
where
data
is
which
isn't
really
necessary.
So
a
common
solution
to
this
is
using
something
like
hashing
or
charting,
which
is
a
which
lets
the
client
I
say,
is
hashtag
licked
and
figure
out
exactly
which
server
it's
goes
to
and
it
needs
to
find
the
data
on.
But
if
you
just
do
simple
hashing,
when
you
take
out
when
a
server
goes
down
or
when
you
add
more
capacity,
a
whole
bunch
of
data
needs
to
be
reshuffled.
A
So
SEF
uses
us
a
bit
of
slightly
smarter
algorithm
called
crush,
which,
and
so
basically
I
was
locates.
Data
is
by
taking
in
the
hash
of
the
object
name
and
then
splitting
that
the
number
of
placement
groups
RPGs,
which
are
basically
just
shards
of
data
to
make
managing
cluster,
stayed
easier
and
inputs,
and
it
takes
that
placement
group.
A
The
existing
cluster
state,
which
is
maintained
by
the
monitors
which
oversees
are
up,
which
ones
are
down
and
a
set
of
rules
describing
what
what
higher
cluster
is
in
physically,
like
if
there
are
different
racks
or
different
rows
or
different
hosts.
That
devices
live
on
and
add.
The
output
of
the
crusher
algorithm
is
then
which
servers
and
store
that
data.
A
A
So
if
a
single
OSD
goes
down,
for
example,
you
see
the
yellow
box
and
then
the
green
block
here,
but
they
were
previously
located
on
it.
We've
our
be
replicated
to
other
OS
DS
automatically,
but
since
crushes
are
stable
algorithm
the
same,
if
the
same
OC
comes
back
up
again,
the
data
will
come
what
will
be
a
store
again
in
the
same
place
again
when
it
comes
back
up.
A
So
another
nice
thing
about
the
crush
is
because
they
can.
You
can
model
your
database,
your
topology
of
your
of
your
storage.
You
can
have
custom
rules
that
define
exactly
like
what
kind
of
placement
you
want
policy.
You
want
for
your
your
data,
so
you
could
have
say
a
set
of
fast
servers
and
a
set
of
slow
servers
and
have
the
first
I
replica,
be
n.
I've
set
a
fast
servers
since
reads
always
search
from
the
first
replica
and
the
rest
of
the
replicas
be
from
a
set
of
slow
servers
that
you
don't
care.
A
So
that's,
basically
how
a
client
can
talk
to
the
cluster
and
there's
several
different
ways
that
they
can
do
this.
So
the
lowest
level
API
is
called
librettos,
which
is
a
user
space
library
which
has
bindings
in
C,
C++
and
Java,
and
basically
the
Britos
talks
to
the
monitors
in
a
toxic
I,
guess
the
cluster
state,
and
then
it
can
talk
directly
to
all
of
the
OSD
in
parallel,
if
it
once
so.
A
A
So
the
Neos
gateways
are
stateless
service
that
coordinate
among
themselves
if
they,
when
they're
caching
authentication
information,
and
they
also
provide
multi-tenancy
since
the
native
readers
doesn't
have
both
MZ
directly.
The
readers
gateway
provides
that
and
through
the
idea
in
this,
a
thin
HTTP
API
is
of
the
access,
keys
and
tokens
and
the
users,
and
also
tracks
all
that
usage
information
for
all
that.
A
A
So
a
block
device
since
is
basically
striped
over
a
bunch
of
objects
in
the
cluster,
and
it
has
the
same
benefits
of
lebretia.
So
it
can
access
all
these
servers
in
parallel
and
do
all
kinds
of
reads
and
writes
in
parallel
without
any
kind
of
serialization
at
any
kind
of
central
authority,
because
it
can
because
of
crush
Lee,
knows
exactly
where
the
data
needs
to
go
and
is
calculated
on
the
fly.
A
Since
this
is
all
a
shared
storage-
and
this
means
that
if
you
actually
have
a
virtual
machine
that
is
booted
off
of
an
outrageous
block
device,
it
doesn't
have
to
depend
on
the
local
local
server.
We
have
diskless
compute
hosts
or
you
can
even
migrate
a
virtual
machine
from
one
one
node
to
another.
While
it's
running
off
of
RVD.
A
There's
also
a
kernel
module,
which
has
been
upstream
since
2
6
3
7,
which
allows
you
at
I'm
directly
map
a
radius
block
device
to
break
your
Linux
block
device.
So
you
can
just
have
like
dev
RBD,
zero
show
up
and
I
mounted
and
I
have
a
file
system
on
top
of
that
directly.
If
you
want
you're
not
doing
a
virtualization.
A
A
They
also
have
support
size,
efficient
snapshots
and
cloning,
so
you
can
take
a
snapshot
at
any
time.
It'll
it'll
be
consistent
with
all
everything.
That's
that's
there
at
that
time.
If
you,
even
while
running
a
virtual
machine,
if
you
make
sure
that
your
application
is
a
good
safe
state,
you
can
take
a
snapshot
at
while
it's
still
running.
A
It's
not
if
you
have
multiple
virtual
machines
running
basically
they're
all
accessing
the
cluster
and
through
the
same
and
different
paths,
so
they're
all
going
to
different
ways
to
use,
because
the
data
is
placed
randomly
across
the
cluster.
There's
no
single
bottleneck
and
you
can
scale
out
through
a
arbitrary
size.
A
But
how
do
you
it's
been
up
so
many
virtual
machines?
We
have
a
sudden
a
sudden
need
there
sudden
spike
in
demand,
and
you
want
to
create
new,
a
bunch
of
say
us
100
new
observers.
How
do
you
do
that?
Well,
if
you
have
the
web
servers
backed
by
various
block
device,
you
can
just
create
a
simple
clone.
So
if
you
have,
if
you've
taken
a
snapshot
of
an
existing
web
server,
that's
ready
to
run
the
penis
to
ID.
A
You
use
a
simple
clone
which
uses
no
extra
space
and
it's
an
instantaneous
copy
of
the
of
that
block
device.
So
then,
when
you
it's
a
copy
and
write
clone.
So
when
the
virtual
machine
starts
using
the
new
clone
and
they
start
writing
to
it,
they'll
copy
the
data
from
the
parent
and
then
from
a
parent
object
to
the
child.
The
clone
object
that
you
just
created
but
for
reads:
they're
still
going
to
fall
back
to
the
parent.
If
the
object
and
the
child
doesn't
exist.
A
So
one
of
the
things
that
that
used
to
be
it
can
be
very
slow
in
ANOVA
is
actually
creating
new
virtual
machines
by
copying
all
the
data
out
of
glance
onto
a
local
disk
and
then
creating
a
new
copy
of
that
file
to
boot
impression
machine
off
of
so
in
that
and
in
the
false
from
release.
We
add
chain
changed
this
a
bit
to
allow
glance
to
be
backed
by
our
BD
and
also
cinder
to
be
backed
by
our
BD.
A
So
you
can
have
your
virtual
machine
templates
in
a
by
this
written
stuff,
as
well
as
your
volumes
and
stuff,
and
if
you
created
that,
if
you
need
admin,
there's
an
API
called
added
to
create
at
a
volume,
permanent
image,
and
yet
these
are
both
back.
If
Manson
and
cinder
are
both
backed
by
stuff.
This
miss,
like
new
volume,
would
just
be
a
clone,
so
I
mean
instantaneous
copy,
with
no
Electra
space
used.
A
Number
of
things
improvement
there
at
the
in
the
OST
level,
how
it
interacts
with
the
M
file
system
and
with
its
a
it's
Journal,
it
can
be
on
a
separate
device
for
faster
rice
was
performance
was
improved
quite
a
bit
by
rearranging
how
the
locks
worked
and
making
them
more
find
range,
unless
course,
so,
a
single,
the
the
I,
absolutely
a
single
lowest.
He
could
get
out
of
a
single
underlying
file
system
increased
from
6,000
to
22,000.
A
Also,
we
structured
how
the
OSD
interprets
updates
to
the
state
of
the
cluster.
So
the
placement
group
concept
that
I
was
mentioning
earlier
was
a
the
unit
of
recovery
and
stuff.
So
whenever
the
state
changes,
our
placement
group
may
need
to
be.
We
replicated
to
a
different
esteem,
but
many
of
many
updates
to
the
map
only
effects
a
shouldn't
subset
of
placement
groups,
so
in
bobtail
we
made
on
the
map
handling
happened
on
a
per
face
McGrew
basis.
A
So
it's
no
longer
the
entire
OS
beat
going
through
the
list
about
all
the
placement
groups
at
once,
but
each
one
can
go
through
and
update
itself
to
a
new
map
according
independently,
so
that
just
improves
general
quality
of
service.
When
you
have
lots
of
event,
events
happening
in
your
cluster
like
no
is
dying
or
new
nodes
being
added.
A
In
addition
to
help.
So,
in
addition
to
the
map
handling,
adding
a
priority
system
to
the
way
that
messages
are
our
process
from
the
OSD
increased
the
ability
of
clients
to
continue
IO
while
recovery
is
happening,
while
more
data
is
being
rapidly
replicated
throughout
the
cluster.
So
basically,
the
old
system
could
allows
clients
to
be
stars
by
high-priority
recovery
operations.
But
now
because
we
have,
the
priority
system
has
been
reworked
to
be
based
on
weights
and
recovery
operations
can
be
placed
lower
than
then
client
operations
without
making
them
totally
starved.
A
Because
it's
a
now
weight-based
achilles,
we
change
dynamically.
So
if
you
have
like
a
certain
subset
of
objects
that
you
want
to
recover
more
efficiently
or
more
quickly,
for
example,
and
now,
if
we,
if
you
try
to
access
a
certain
objects
and
that
hasn't
been
recovered,
yet
it'll
be
huge
for
recovery.
Next.
A
It
yeah
it
was
I
was
integrated
in
syndra
with
balsam
and
the
ability
to
copy
an
image
from
glance.
That's
a
non
raw
that
is
it's
like
you
get
hurt
you
code
to
or
some
other
format
it's
now
I
think
they
can
have
be
converted
into
a
raw
format,
image
which
is
suitable
for
actually
used
by
a
virtual
machine
on
bar
BD
in
grizzly,
in
on
the
Rios
gateway
side.
A
So
what's
coming
up
in
the
next
stable
release,
cuttlefish!
Well
we're
on
a
three-month
mini
cycle
now,
so
this
release
is
coming
up
in
a
couple
weeks
now
couple
major
features:
her
are
in
it.
The
main
one
for
the
block
device
is
incremental
backup.
So
you
can
have
it
it's
a
it's
kind
of
like
the
ZFS
concept
of
send/receive
snapshots.
You
can
send
and
receive
RVE
snapshots.
Now
you
can
export
at
the
difference
between
one
snapshot.
A
A
Because,
on
the
OSD
side,
this
support
added
for
encryption
for
data
at
rest,
so
Anna
Choi
Steve,
can
have
its
underlying
file
system
on
top
of
ID
encrypted
block
device,
and
the
keys
right
now
are
currently
managed
by
the
monitors,
because
that's
the
simplest
plate
place
to
put
them.
That
might
be
a
point
of
change
in
the
future.
If
there's
a
other
key
management
services
that
come
up
for
the
readers
gateway,
there's
a
new
rest
API
for
managing
the
base
gateway.
A
Most
of
those
number
of
those
things
and
performance
improvements
came
from
just
removing
some
of
the
metadata.
That's
chef
stories
about
an
object
from
the
owner
underlying
local
file
system
to
a
level
DB
instance,
which
is
there's
a
level
DB
instance
for
each
OSD,
there's
plenty
more
than
or
two,
but
those
are
just
the
main
things
that
are
kind
of
important
for
OpenStack
users.
A
So,
there's
what's
next
for
that.
After
that,
though,
because
this
place
is
almost
out
well,
the
the
next
release
is
called
dumpling
will
be
out
in
August,
but
and
there's
a
couple
of
features
that
are
proposed
so
far,
there's
to
your
application
for
the
greatest
gateway
so
that
it
can
be
aware
of
multi
sites,
multiple
sites
that
don't
have
to
be
necessarily
always
in
sync,
but
can
be
synchronized
asynchronously
and
there's
also
a
general
but
rest
management
API
for
the
Swift
cluster
engine
I'm,
not
just
the
meanest
gateway,
but
the
whole
stuff
system.
A
So
you
don't
have
to
go
through
the
command
line.
You
can
just
go
through
the
HTTP
API
we're
having
a
virtual
staff
developer's
event
on
May
6th.
If
anyone
wants
to
propose
new
prints,
they
can
go
on
our
onsets
icon
and
there's
a
new
wiki
there.
Anyone
can
move
prints
and
come
to
the
staff
developer
summit
and
discuss
any
new
features,
they're
interested
in
or
interest
in
working
on.
We
welcome
any
contributions.
Anyone
has.
That
would
be
great
and
I
like
to
take
any
questions.
Anyone
else.
A
Question
is:
is
it
possible,
when
you're
using
block
storage,
to
kind
of
control
where
the
data
is
goes,
so
you
can
move
it
closer
to
where
the
virtual
machine
is
actually
running
and
that
is
possible
in
certain
setups?
So,
basically,
SEF
has
a
concept
of
a
storage
pool,
and
so
you
can
have,
for
example,
our
dream
house.
Does
this
in
their
dream,
computer
architecture,
which
is
available
online?
You
can
see
that
they
have
different
storage
like
pods
of
computing
clusters
and
that
they
have
an
Associated
storage
pool
with
each
other
compute
compute
cluster.
A
A
Questions
can
you
have
block
storage
and
object
storage
in
the
same
cluster
at
the
same
time,
and
he
absolutely
can
that's
like
one
of
the
big
benefits
of
stuff
is
that
you
can
use
the
same
hardware
in
the
same
cluster
to
serve
as
many
needs.
So
you
can
have
the
block,
storage
and
the
object,
storage
and
in
the
same
cluster
they
can
use
separate
pools
if
you
wanted
to
differentiate
them
in
some
way.
I
have
more
SSDs
for
one
and
say
that
for
the
other,
but
they
can
all
be
the
same.
Cluster
no
problem.
A
A
A
Do
we
recommend
the
that
userspace
client
or
the
colonel
plant
for
the
file
system
at
at
the
moment?
It
depends
on
your
needs
and
I
think
more
than
like
the
performance
difference
between
them.
Isn't
that
great
right
now,
because
the
file
system
is
relatively
unstable,
still
billions
more
QA,
but
if
you
have
the
stuff
used
as
my
future
to
get
started
with,
because
you
don't
need
that
kernel
module
and
you
don't
need
special
support
for
them
right.
There.
A
A
A
So
what
are
the
wear?
Performance
measuring
tools
are
what
are
tuning
tools
available
for
looking
at
stuff
there's
a
couple
different
ways.
You
can
look
at
this
now,
basically
at
each
level
of
stuff
like
at
the
staff,
client
and
the
OSC
level,
there
is
something
as
an
admin
socket
which
you
can
use
to
query
information
about
latency
or
other
perform
statistics.
I
got
the
last
n
requests
in
general.
You
can
look
at
lots
of
things
like
a
block
device,
utilization
and
general
I/o
stats
on
the
iOS
Keys
themselves
for
the
at
the
client
side.
Also.
A
If,
for
example,
you
could
look
at
the
using
these
Gavin's
like
a
paper
access
again,
you
can
look
at
that
how
efficient
the
cache
is
being
used
and
whether
it's
actually
benefitting
you
or
not,
not
for
your
workload.
So
there's
a
number
of
things
there
already,
although
they
could
certainly
be
improved
in
the
future.
There's
always
room
for
improvement
in.
A
A
Frances
between
Swift
API
and
what
the
radius
gave
me
supports.
I,
don't
remember
all
the
differences
off
the
top
of
my
head
I
think
one
of
them
might
be
a
tree
versioning
if
I'm,
not
sure,
if
that's
supported
now
or
not
yeah
I
think
there's
a
turtle
documentation
somewhere.
That
lists
out
exactly
which,
which
calls
are
right
over
there.
A
So,
what's
the
upgrade
process
like
since
we're
on
this
release
cycle
I'm
generally
we're
very
careful
about
eckrich
compatibility
and
rolling
upgrades
so
pretty
much
all
our
crazy.
We
test
and
I
think
that,
in
particular
between
the
stable
releases
right
pretty
extensively
I'm
looking
for
a
possible
causes
for
problems,
and
we
were
pretty
careful
to
make
sure
that
rolling
upgrades
are
always
possible.
A
Between
there
is
one
everybody's
gave
a
block
twice
more
reliable
or
longer
history.
They
were
kind
of
started
that
similar
times
I'd,
say
they're,
both
at
a
similar
level
of
reliability,
because
they're,
both
really
depending
on
the
core,
reduce
a
break
store
for
all
the
fancy.
Functionality
like
Raiders
itself,
is
handling
all
the
replication
and
all
the
consistencies
semantics
widows,
like
the
block
device
and
the
Gateway
are
really
kind
of
much
thinner.
Wrappers
around
that.
A
Come
from
and
reads
come
from
only
the
primary
copy,
and
this
is
especially
useful
for
cache
efficiency
because
it
on
the
underlying
on
the
OSD
either
stored
as
files.
So
when
you're
doing
rereading
the
same
object,
for
example,
that
files
going
to
be
in
the
page
cache
on
the
OSD.
So
if
you
had
a
spread
across
to
multiple
replicas
and
you
had
reads
hitting
each
of
those
replicas,
your
cache
efficiency
would
be
divided
by
the
limit.
Because
you
have.
A
So
easy
the
question:
it's
about
network
traffic
with
reads,
and
it's
true
that
it
reads
to
a
single
iolite.
So
if
everything
everyone's
trying
to
read
a
single
object
on
that
the
OSE
level
you're
going
to
the
same
OSD,
so
that's
gonna
be
a.
It
could
be
a
bottleneck
if
you
have
a
very
small
type
to
that
OSD.
But
in
general
we
don't
find
that
reading
the
same
outrage
from
many
places
happens
very
often,
usually
there's
either
there's
some
strafing
going
on.
A
A
A
A
We
don't
have
to
do
as
many
basically
F
syncs
to
the
data
disc
I
get
the
same.
Consistency
guarantees
that
we
do
any
best,
so
we
can
provide
a
theoretical
performance
boost.
In
that
case,
you
know,
a
butterface
also
provides
some
snapshotting
internally,
so
ACEF
snapshots.
Couldn't
you
can
use
the
butter
fest
knife
shots?
If
butter
efest
is
used,
there
stays
a
little
more
space.
A
Miss
butter
risk
stable.
It's
not
the
recommended
choice.
Right
now,
exercise
is
recommended
because
it's
generally
more
stable
and
so
there's
still
some
performance
problems
with
whatever
fest
over
time
as
it
gets
more
fragmented,
so
Express
is
still
recommended
as
being
more
stable
and
less
likely
to
fragment.
A
How
does
us
three
different
tenants
in
OpenStack
and
right
now,
I'm
at
the
pool
level,
but
in
safe?
The
pool
level
is
the
level
of
other
authentication,
but
there's
no
pools
also
mapped
directly
to
placement
groups
which
consume
more
memory
and
the
OSD
and
so
creating
more
pools
for
every
single
tenant.
You
have
becomes
unscalable
because
you
have
to
use
up
more
memory
to
Frederick
manage
all
those
placement
groups
so
right
now,
I'm
OpenStack
uses
a
single
pool
for
all
tenants
in
the
future.