►
From YouTube: Turned Up to 11- Low-Latency Ceph Block Storage
Description
Ceph continues to be the leading block-storage-of-choice for Openstack VM deployments. Latency-sensitive workloads such as databases remain an emerging Ceph use case - latency guarantees typically a challenge in shared VM storage environments as compute/networked-storage resources cease to scale as VM density increases. With growing flash footprint on clients, flash-based read/write caching has the potential to improve storage latency by reducing IO path dependency on the network. We'll introduc
A
A
Well,
will
eventually
be
three
speakers,
but
Jason's
going
to
kick
it
off,
and
the
two
Intel
gentlemen
will
be
joining
us
after
another
talk
that
they
will
be
sprinting
here
from
so
they're
back
to
back,
but
in
the
meantime,
Jason
from
Red
Hat
is
going
to
talk
about
some
of
the
the
block
storage
stuff
and
give
us
a
rundown
on
that
world.
So
Jason
take
it
away
all.
B
B
So
what
our
BD
is
is
a
block
device,
abstraction
layer,
that's
built
on
top
of
Rados
and
the
way
it
works
is
you
have
hundreds,
thousands,
tens
of
thousands
of
small
objects
within
the
radios
cluster
and
that's
a
stripe.
So
and
then
your
image
be
it.
You
know
terabytes
petabytes,
whatever
you
want.
That's
broken
up
and
striped
over
these
very
small
by
default
for
megabyte
objects
in
the
radios
cluster.
Some
of
the
highlights
that
we
have
with
our
IBD
it's
it's
broad
integration,
I'm
specifically
talking
about
OpenStack,
it's
probably
integrated
into
cinder
Nova
and
glance.
B
I've
heard
some
rumblings
about
even
pulling
it
into
ironic
to
offer
a
device
driver,
RBD
device
block
driver
on
bare
metal
stanley
provision,
so
anything
you
boot
up,
you're
not
instantly
allocated
that
space
out
of
your
cluster.
If
you
want
to
boot
up,
you
know
100
gigabyte
image,
you
don't
take
up
a
hundred
gigabytes
in
your
cluster.
It
takes
up
practically
zero
bytes
in
your
cluster,
just
a
little
metadata.
It's
only
when
you
actually
start
filling
it
up.
Do
you
actually
start
using
that
data
snapshot
functionality?
B
If
you're
here
earlier,
Greg
did
a
very
comprehensive
detailed
talk
about
the
internal
structures
of
how
snapshots
work
within
stuff,
but
within
our
BD
we
abstract
all
that
away,
and
you
can
just
snapshot,
delete
rollback
snapshots
of
your
images
and
even
on
top
of
that
from
your
snapshots,
you
can
then
create
clones
which
are
basically
ways
of
I
created.
An
image
I
can
create
that
image
and
call
it.
B
My
golden
image
snapshot
it
and
now
I
can
clone
off
new
images
from
that
snapshot
of
the
parent
image
and
those
clones
are
copy-on-write
thinly
provisioned
clones
and
since
this
is
OpenStack
I
always
like
to
point
out,
I
mean
obviously
I
think
cinder
and
OpenStack.
They
they
widely
use
a
RBD,
and
we
love
that
and
as
you
as
the
use
case,
users
of
our
BD
inside
of
OpenStack.
If
you
have
any
problems,
questions
or
concerns
or
any
improvement
advice,
I
mean.
Obviously
we
would
love
to
hear
from
it
because
we
want
to
make.
B
B
So
before
I
start
talking
about
some
of
the
problems
we're
trying
to
solve,
with
with
within
our
media
and
latencies,
I
wanted
to
go
over
a
high
level
for
those
that
don't
quite
understand
how
this
RBD
architecture
works.
So
we
have
two
different
ways
to
access
our
BD.
We
have
the
the
care
BD
driver,
that's
just
a
kernel
driver
that
provides
a
block
device
to
your
operating
system.
We
also
have
Lib
our
BD
and
that's
in
terms
of
OpenStack
use
cases.
B
So,
in
the
case
of
Kimio
your
operating
system,
your
ego,
your
VM,
has
exposed
a
block
device
within
it.
That
is
whatever
size
you've
configured
that
image
to
be
so
from
the
point
of
view
of
our
BD
libero
bt.
It
gets
requests
from
Khemu,
saying,
please
read
or
write
from
this
sector
this
many
sectors,
and
we
translate
that
just
internally,
just
quick.
B
And
then
diving
down
and
about
again
how
that
control
that
IO
flow
works
within
for
IO
request.
So
in
this
case
a
client,
let's
say
Kim
you,
it
creates
an
a
a
over
quest
in
asynchronous
IO
request
via
the
Lib
RBD
API,
and
that
request,
as
a
hinted
at
is
just
a
sector
number
of
sectors
of
data
that
you
want
to
read
that
comes
off
the
commute,
IO
dispatch
thread
that
actually
gets
then
in
queued
into
the
Lib
RBD
IO
dispatch
thread.
B
The
read
write,
truncate
discard,
for
example,
liber8,
owes
its
the
abstraction
layer
that
actually
handles
talking
to
the
OSD,
so
liber8
O's
takes
those
requests
against
very
specific
objects
in
the
backing.
Cluster
sends
those
off
to
the
correct
that
you
know
the
primary
PG,
whatever
OSD,
that
is,
it
just
uses
the
quick
crush
map
lookup
to
locate
the
right
place
to
go
and
then,
finally,
once
all
the
OSD
s
have
done
their
work
for
all
those
operations,
you'll
get
liber8
owes
the
OSD
s
complete,
asynchronously
back
to
liberators
liberators.
B
Asynchronously
completes
back
to
Lib,
RB
d
and
live
RB
d
can
bubble
that
completion
back
up
to
complete
the
AO
request
from
Kenya,
but
just
visually
because
words
sometimes
don't
express
it
just
to
visually,
show
how
that
that
works.
Just
in
terms
of
IO
file,
you
got
cheer
in
this
case.
Alright,
the
write
request
comes
down,
asynchronously
queues,
it
gets
popped
from
The
Dispatch
thread
that
can
get
translated
into
one
or
more
object.
Operations
which
then
gets
sent
off
in
parallel
to
liberate
owes
liberate
us
works
sends
out
in
parallel
to
the
clusters.
B
The
striping
that
I
showed
here
I
showed
how
an
example
of
you
know
one
eye
operation
translated
into
three
different
object
operations
when
you're
doing
normal,
I
get
small
small
IO
your.
What
you're
really
going
to
get
into
Lib
are
biddies
you're
going
to
get
you
know,
4
K
reads:
you
know,
read
one
four
K
sector,
you
know
at
the
specific
offset
and
what
happens
is
realistically
that's
going
to
properly
align
within
one
backing
object.
So
it's
not
going
to
translate
into
anything
more
than
just
one
operation
that
gets
sent
off
at
a
time.
B
B
So
one
of
the
things
with
one
of
the
design
features
of
of
Ceph
is
that
you
can
only
only
the
primary
PG
can
modify
the
object
that
you
want
to
that.
You
want
to
manipulate.
So
what
happens
is
if
you
have
a
lot
of
I/o
that
keeps
hammering
the
same
object,
you're,
going
to
have
a
lot
of
operations
that
keep
hammering
the
same
object
within
the
same
PG
and
all
that
work
basically
has
to
happen.
B
B
B
Eventually,
just
get
a
better
performance
so,
but
that
that
in-memory
cache
it
has
some
some
constraints
and
caveats.
First
and
foremost,
it
really
only
helps
you
for
sequential
workloads
in
terms
of
consolidating
down
those
and
merging
those
I/o
requests.
If
you
truly
randomly
going
all
over
all
over
your
your
backing
image,
there's
nothing
you
can
do.
Those
are
basically
to
be
one-for-one
requests
back
to
the
back.
B
You
know
SDS
number,
two,
it's
constrained
by
the
physical
memory,
so
you
have
RBD,
you
know
being
used
in
key
moves
that
you're
trying
to
stack
multiple
hypervisors
on
a
multiple
VMs
on
a
host,
and
you
don't
want
to
have
to
dedicate-
and
you
can't
dedicate
gigabytes
of
memory
for
for
an
in-memory
cache
to
help
relieve
some
of
this.
Some
of
this
pressure.
B
So
right
now
it's
by
default.
Unless
you
have
a
ride
at
your
configuration,
it's
defaults
to
at
a
maximum
24
megabytes
of
write
back
cache,
but
it
tries
to
keep
it
around
sixteen
megabytes
of
in-use
memory
and,
finally,
because
all
the
writes
are
we
don't
once
the
writes
go
into
the
cache?
We
don't
track
the
initial
order
that
they
come
in,
so
any
flesh
requests
that
come
in
from
your
application
that
actually
has
to
flush
all
the
dirty
objects
out
of
the
cache.
B
B
That's
right
slightly
modified
il
flow
with
the
in
memory
cache
and
what
happens
is
basically,
if
stain,
all
the
way,
the
top,
but
instead
of
breaking
that
up
into
per
object
requests.
What
we
have
is
a
per
object,
cache
those
requests,
populates
the
per
object,
cache
and
eventually,
and
that,
if
you're
doing
right
back
that
completes
your
your
whole
IO
operation
back
to
your
clients.
B
B
Let's
try
this
hi,
so
here's
we're
out
today
and
actually
this
is
this-
is
a
graph.
It's
on
onset.
Calm
are
available
from
South
calm,
and
this
is
from.
This
is
a
benchmark
that
was
performed
by
Intel,
and
this
is
actually
against
an
older
releases
against
the
hammer
release
of
Ceph
RBD
specifically,
so
this
is
for
K,
random,
read
workloads,
and
what
you
have
here
is
the
constraint
on
the
bottom
is
basically
constraining
the
client
workload
to
only
utilize
certain
number
of
threads
on
the
client
core.
B
But
what
this
shows
you
is
on
average,
on
on
this
half
of
the
graph
on
the
left,
half
of
the
graph
you
can
see
as
you
increase
the
cores.
Our
throughput
goes
up,
not
changing
their
client.
So
what
that
tells
me,
you
know
it's
a
developer
already
developer.
Typically,
that
tells
us
we
have
work
to
do
in
terms
of
how
much
we're
hitting
and
utilizing
the
CPU
like
we
want
to.
We,
we
have
performance
gains
that
we
can
still
get
out
of
Lib
RBD
to
reduce
the
CPU
impact
and
get
us.
B
You
know
back
up
to
where
you
start
seeing
it
like
40
cores
where
it's
the
CPU
is
no
longer
the
bottleneck,
but
you
can
see
on
the
on
the
alternate
access
the
latency
access
I
mean
on
average.
It
does
pretty
well
even
under
these
high
ups
high
ups
workloads.
You
know
two
millisecond
average
response
times,
but
the
really
what
we're
trying
to
get
gained
here
and
what
we're
eventually
gonna
start
talking
about
where
we
want
to
go
forward
in
a
live.
Our
BD
is
how
to
tackle
the
tail
latencies
of
our
BD.
B
So
here
you
have
an
example
of
as
AI
helps
increase
specifically
I
think
this
is
just
increasing
the
queue
depth.
The
queue
depth
is
increasing,
to
increase
the
the
ieps
count
and
what
you
can
see
is
yeah
again
the
the
average
latency
x'
stay
pretty
negligible,
oh,
but
the
queue
depth
the
tail
latencies
really
start
to
climb
and
already
you're.
B
Here
you
know
you're,
probably
already
at
20,
50
milliseconds
of
tail
latency
and
obviously,
as
the
cluster
is
getting
to
the
point
where
it's
maxed
out
you're
getting
to
four
and
a
half
seconds
of
latency,
which
is
pretty
unacceptable.
So
the
goal
is:
how
can
we
reduce
these
Layton
sees
for
for
certain
workloads
and
now
that
my
partner
in
crime
has
joined
me
he'll,
take
over
from
here
and
start
talking
about
what
he's
been
doing
or
what
they've
been
doing
at
Intel
to
help
address
this
scenario.
C
Yeah
sorry
guys
I
actually
had
a
had
an
overlapping
presentation
that
I
just
just
ran
out
of
so
just
just
finished,
talking
about
SP,
DK
and
I
sale
for
those
who
you
know
want
to
hear
it,
but
just
happened
so
anyhow.
So
just
to
add,
add
to
what
Jason
said.
You
know.
The
one
thing
I
wanted
to
point
out
was
so
we
we
actually
wanted
to
see
the
tail
latency
behavior
with
with
the
fastest
kind
of
back-end
that's
available
today
right.
C
So
this
is
this
cluster
is
a
5
node
cluster
with
4,
you
know
stared
out
state-of-the-art
nvm
ease,
so
you
have
an
all
nvme
self
back-end
right
and
with
the
fastest
we
see
the
the
tail
latency
being.
You
know
what
it
is
and
that's
kind
of
serves
as
the
as
one
of
the
motivation
right
for
us
to
to
do
some
optimization.
So.
C
So
that's
where
client-side
caching
comes
in,
as
you
know,
as
an
attractive
option.
Basically,
you
know
VM
iOS,
which
are
basically
tenth,
lend
themselves
very
well
to
in
the
locality
based
optimization,
so
explore
io
locality
to
cache
the
VM
iOS
on
the
node
and
reduce
IO
path.
Dependence
on
the
network-
or
you
know,
are
basically
the
two
two
key
reasons
why
why
we
are
doing
this
right,
which
which
helps
with
the
the
long
tail
latency
problem,
because
you
I
mean
for
for
the
right
use
cases
for
the
right
performance.
C
You
know
you
basically
kind
of
either
either
maintaining
a
log
journal
of
some
sort:
locally,
ie,
caching,
caching,
all
the
writes
locally,
which
and
then
you
can
and
and
also
serving
your
leaves
out
of
local
cache,
which
which
basically
leaves
on
the
network
network
for
your.
You
know,
network
bandwidth
available
for
for
your
rights
and
that
basically
helps
both.
You
know
both
the
RB
d,
as
well
as
the
radios
gateway
workload.
So
you
could,
you
could
do
have
a
block,
cache
or
also
a
you
know.
C
C
This
is
kind
of
a
recap
of
where
we
are
today
with
the
with
the
in-memory
cache
and
in
some
of
this
Jason
already
talked
to
you
know
the
debris
Ram
based
caching
is,
you
know,
kind
of
has
limits
in
terms
of
when
you,
when
you
reboot
the
node,
you
don't
have
there's
there's
no
warm-up
I
mean
you
need
to
wait
for
the
cache
to
warm
up
to
get
any
benefit
and
the
cache
ability
is
really.
You
know
limited
by
how
much
DRAM
you
have
available
and
as
the
system
ages
it
becomes
less
and
less
available.
C
So
there
is
no
order
right
back
support,
so
you
cannot
do
a
femoral
with
itself
today,
right
you,
you
have
the
object,
cache
or
implementation.
Today
is
it's
pretty
complex
because
it's
actually
shared
by
lebar,
BD
and
CFS,
so
so,
basically,
that
the
many
bought
single
there
are
bottlenecks
that
that
we
are
trying
to.
You
know
get
around
by
maybe
they're,
basically
thinking
that
this
cache,
as
you
know,
as
it's
basically
a
new
cache,
and
that's
that's
where
this
crash
consistent
right
back
caching
extension
comes
in
it's
a
it's
a
blueprint.
C
We
have
in
the
works
right
now,
which
is
extending
the
the
DRAM
based
caching
to
to
use
solid-state
media,
where
you
can
have
a
larger
SSD
footprint,
which
is
also
persistent.
So
so
you
can
get
get
a
nice
warm
up
kind
of
behavior
across
reboots
you,
you
have
ordered,
write
back
cache,
which
and
again
this
is
actually
where
Batum
from
the
blueprint.
C
That's
why
so
you
know
wordy,
but
basically
the
the
point
is
that
you
have
an
LV,
a
base,
read
cash
and
a
an
ordered,
write
back
cache
which
we
would
say
which
is
basically
a
journal
that
you
maintain
on
the
on
the
compute
node
on
it
on
a
an
SSD
and
it
it
also
has
a
colonel.
You
know:
caching
plug-in
interface.
C
So
so
we
have
been
working
on
this
blueprint
for
the
last
couple
of
months
now.
Actually
Jason
had
had
a
great
starting
point
for
us
last
year
that
that
we
started
outside
from
so
ve,
so
before
I
jump
in
and
talk
about
what
we
have
done
so
far,
but
this
is
this
is
actually
an
a
proof
point
for,
for
this
cache
gets
us
right
on,
and
so
we
did
an
early.
You
know
POC
at
Intel
that
the
small
team
that
we
have
we
actually
were
able
to
get
get
about.
C
C
You
know
standard
spinning,
media,
nvme
kind
of
combinations,
so
so
in
that
sense
these
are
kind
of
the
worst
case
results
right
so
with
with
spinning
backends
you're,
probably
going
to
see
a
lot
lot
better
throughput
and
latency,
and
we
also
compared
it
to
the
current
cash
sharing,
I
mean
which,
which
is
being
reworked
of
course,
but
we
also
had
had
a
you
know.
I
just
want
to
add
that,
as
a
reference
point
kind
of
to
say
that
on
the
cluster
side,
we
we
have.
C
B
Mean
I
think
we
just
cost
I
mean
I,
think
speed
through
some
of
this,
especially
for
time,
but
look
the
way
that
the
design
is
shaping
up
right
now
is
be
kind
of
actually
like
two
separate
caches,
a
read-only
cache
that
can
be
used
for
immutable
data,
immutable
metadata.
So
that's
that's
the
are
due
to
RGB
case
rgw
cases
and
your
RBD
snapshot
situations
like
a
parent
image.
B
That's
your
golden
image
that
you're
pulling
from
and
then
also
the
crash,
consistent
right
back
case,
which
uses
which
would
be
implemented
basically
as
just
a
pending
journal
that
just
journals
all
the
changes
that
you're
doing
so
that
your
eye
out
can
hit
the
journal
and
instantly
complete
back
to
ki
mu.
And
then
you
have
a
background
right
back
process.
That's
a
thread!
That's
pushing
data
as
fast
as
it
can
back
to
the
cluster,
but
avoiding
all
those
spiky
potential
for
tail
latencies.
B
Oh
yeah,
that's
so
yeah!
That's
that's
the
important
point
about
the
the
read
cache
is
that
how
it
has
to
be
designed
is
it
actually
has
to
be
shared
potentially
between
multiple
clients.
You
could
have
multiple
rgw
is
on
the
same
host.
She
doesn't
make
sense
to
cache
the
same
object
and
number
of
times.
It'd
be
great.
If
you
could
reuse
that
cache
data
and
have
a
consistent
hot
object
policy,
that's
shared
between
all
the
rgw
instances
that
are
running
and
then
similar
with
the
RBD
parent
image.
That's
that's.
B
C
C
It's
actually
a
file
based,
cashing
frame
of
Jason,
originally
started
it
has
today
it
has
a
you
know,
read
as
in
right
through
and
right
right,
back
mode
support
for
the
right
back
case.
We
have
a
you
know.
We
basically
have
a
time-based
flush,
so
we
maintain
all
the
blocks
that
are
returned
out.
You
know
are
basically
appended
appended
to
it
to
log,
and
you
know
you
act
act,
the
client
and
we
we
basically
have
a
time-based
RPO,
which
is
recovery,
point
objective,
which
is
basically
your
tolerance
on
how
much
data
you
can
afford
to.
C
Let's
say
loose
in
case
you,
the
node
or
the
cache
cache
device
fails.
So
so,
basically
that
that's
the
time-base
flush
that
we
have
implemented
we
and
for
the
eviction
policy.
We
have
a
very
simple
LRU
based
policy
for
for
now,
and
we
are
working
on.
You
know,
implementing
something
like
two
q
type
policies.
C
Again.
You
know
that
the
two
other
bullets
basically
tell
you,
you
know
kind
of
the
pretty
much
the
details
of
the
class
names
that
that
a
part
of
the
PR.
So
we
have
a
file
image
cache
which
implements
you
know
the
ordered
right
back
journal.
We
we
have
and
then
also
the
read-only
object
store,
which
is
which
is
a
read-only
part
of
the
cache,
as
well
as
the
policy.
So
that's
the
in
encapsulating
class
and
and
today
we
we-
you
know
if
you
officially
pull
this
down
and
try
to
you.
C
You
know
you're
not
going
to
see
performance
today,
but
because
we
are
working
on
it,
but
this
actually
has
configurable
options
for
right,
but
you
know
right
back
or
read-only
cache.
You
can
actually
size
it
size,
the
cache
statically.
The
one
thing
to
note
is
because
this
is
a
sparse
file
based
implementation.
C
Rather
the
the
reason
it
a
sparse
file
base.
Inflammations
was
so
that
you
can
easily
do
the
you
know.
One
is
to
one
our
beauty
volume
size,
I
really
want
him
to
to
cash
rate
ratios
right
and
then
right
now
I
mean
we
are
working
on
making.
All
of
these.
You
know
more
things
configurable
with
the
with
the
cash.
D
F
So
a
couple
of
quick
questions
on
the
blueprint
that
you
shared
for
the
caching
stuff.
So
first
you
showed
it
for
the
RBD.
Do
other
gateways
like
the
POSIX
one
cephus
would
be
able
to
leave
the
benefits
of
that
and
the
second
one
in
your
diagram.
You
showed
SSD
being
used
on
the
client-side
I
poured
the
caching
can
some
flavor
of
NVRAM
like
a
pea
mem
on
the
client
side
on
be
also
use
or
is
SSD
absolutely
mandatory.
F
C
C
F
C
C
So
again,
this
kind
of
summarizes.
You
know
how
how
the
I/o
flow
is
being
implemented
today
right,
so
you
basically
have
a
a
it
right
back
thread,
that
is,
that
is
flushing
out
those
general
entries
on
a
on.
Basically
when,
when
there
is
a
time
passed,
a
max
interval
which
is
configurable,
which
is
basically
how
many,
how
long
offered
data
loss
can
you
sustain
without
losing
consistency,
so
the
so?
The
key
here
is
that
the
ordered
write
back
basically
lets
you
lets
you
boot
or
migrate,
a
VM
to
to
other
node.
C
You
know
with
guaranteeing
consistency
that
that's
the
basic
requirement,
but
but
your
data
could
be
little
stale
right.
So
so
that's
why
the
that's?
What
the
last
bullet,
basically,
where
it
says,
backlog
time,
past
maximum
interval?
That's
that's
configurable
interval
that
you
can
sustain,
or
you
are
comfortable,
losing
a
few
future
data
data,
bytes
for
and
then
the
second,
the
second
bullet,
to
add
to
what
the
right
back
thread
can
do
you?
C
You
know
we
are
actually
working
on
a
shared
cache
policy,
so
basically
a
daemon
that
implements
a
shared
cache
policy,
which
is
you
know
which,
which
basically
serves
as
a
mechanism
to
you,
know,
to
communicate
between
between
multiple
clients,
because
that
the
cache
allocation
we,
you
know,
we
want
it
to
be
more
cooperative
in
the
sense
that
you
know,
if
you
actually
have
a
cache
or
as
a
service
or
na
is
a
central
entity
that
is
doing
all
the
block.
Caching,
for
you
that
could
become
a
you
know,
basically
hotspot
or
single
point
of
failure.
C
Let's
see
anything
else,
you
want
to
add
to
that
yeah
and,
like
the
last
bullet,
says
here
right.
There's
we
actually
what
we've
been
working
on
this
design
to
be
to
have
those
no
hearts
part
or
bottlenecks.
So
that's
why
the
all
the
I/o
is
basically
directly
to
a
client
specific
cache
file.
You
know
only
the
control
messages
are
basically
being
passed
with
between,
let's
say,
an
RBE
client,
which
is
a
virtual
machine
and
and
the
daemon
which
which
had
owns
the
you
know,
sort
of
the
cache
policy,
and
we
do
we
do
intend
doing.
C
C
Yeah,
so
so
what
we?
What
we
have
today
as
part
of
the
PR
for
the
request,
actually,
you
know,
works
functionally.
We
still
have
several
performance
improvements
and
functionalities
to
add
so
right,
coalescing
is
going
to
be
a
big
big
performance
improvement.
The
shared
cache
is
not
part
of
the
PR,
yet
it's
only
read
read
only
cache
per
VM
today,
which
you
are
working
on
and
then
the
dynamic
cache
sizing
is
a
is
a
big
will
be
a
big
feature
up
with
a
with
some
sort
of
a
cooperative
allocation
scheme
that
is
weighted.
C
So
so
you
could
assign
you
know,
weight
to
VM
and
depending
on
that,
so
so
basically,
one
one
scheme
that
that
we
had
thought
about
was:
you
could
have
a
you
know
fixed
slots
as
allocation
units.
So
so
so
you,
basically
you
you
deal
cache
as
a
cache
slots
at
let's
say
64
Meghan
size
and
then,
depending
on
how
important
a
VM
is,
let's
say
how
what
is
the
weight
of
the
VM
you?
You
basically
are
saying
it
gets
more
slots
and
then
the
VMS
cooperatively
manage
the
cache
space
in
the
sense
that
you
know.
C
If
a
VM
is
not
using
it,
it
basically
kind
of
talks
to
the
other
VMs
and
figures
out
that
it
needs
to
give
Erb
some
slots
to
the
to
the
general
pool
or
or
if
they're
more
available.
It
can
reclaim
some
right.
So
it's
a
it's
a
more
peer-to-peer
cooperative
kind
of
kind
of
an
allocation
scheme
so
that
that's
the
we're
calling
that
a
better
store
or
the
cache
allocator,
which
is
actually
heavily
work
in
progress
right
now.
So.
C
B
B
One
of
the
big
bottlenecks
of
it
like
with
this,
though
you
can
potentially,
if
you're,
writing
it
persistently
now
to
a
journal.
You
can
selectively
ignore
those
flush
requests.
You
just
journal
the
flush
request
and
now
on
disk
you
have
your
boundaries.
You
say
anything
between
that
flush
request
and
that
flush
request
that
I
just
read
a
chunk
off
disk
I'm
free
to
reorder
free
to
cut.
You
know,
coalesce
and
send
off
to
the
OSD
instill.
B
G
B
B
B
B
H
I
just
want
to
understand
a
little
better.
What
what
F
sync
means
for
the
write-back
cache,
so
is
it
flushing
to
the
local
client-side
journal,
or
is
it
going
to
flush
to
the
OSD
and
if
it
doesn't
flush
to
the
OS
DS?
Is
there
a
way
that
an
application
can
make
sure
that
data
actually
kind
of
gets
out
there
so
that
what
do
you
think
for.
B
You
per
flush
yeah,
so
there
would
be
have
to
be
a
configurable
policy
but
I
think
you're,
like
you're,
going
to
see
you,
but
if
you're
you
for
like,
if
you
actually
have
like
a
database
workload,
that's
like
right,
flash
right,
flash
right
flush.
You
know
you're
not
going
to
see
much
gain
out
of
something
like
this.
Unless
you
can
selectively
ignore
those
flush,
berries
too.
G
B
And
you're,
not
abandoning
those
flushes
and
you're
keeping
everything
ordered
and
consistent
if
you're
familiar
with
RBD
marion's
features
the
same
way
how
it
we
keep
everything
consistent,
it
might
be
delayed,
but
everything
is
respectful
of
the
of
the
boundaries
of
the
flush.
So,
no
matter
where
you
finish
replaying,
it's
still
consistent.
It
might
be
missing
data,
but
it's
not
inconsistent.
I
B
So
it's
not
necessarily
anything
in
this
flow
here.
That's
that's
happening.
It's
really.
What
happens
is
a
request
goes
to
the
OS
DS
and,
as
OSDs
might
be
so
hammered
or
the
network
is
so
backlogged
or
something
like
that
you
just
it
just
takes
longer
for
those
operations
to
take
place.
So
that's
one
of
the
nice
things
about
doing
it
locally.
As
you
take
all
that
out
of
the
equation
you
you
can
basically
smooth
out
and
get
rid
of
all
those
spiky
tail
agencies
that
you
just
get
on
some
random
I/o
I.
I
B
I,
unfortunately,
have
an
easy
answer:
the
full
view
into
the
stack
of
requests
was
hung.
Where
and
what
process
it
could
have
just
been
that
you
know
and
again
this
is
totally
presumptuous,
but
you
know
if
you
had,
everything
has
to
go
through
the
primary
PG,
but
if
that
primary
PG
was
getting
hit
so
high,
you
know
it
just
had
way
larger
backlog.
Then
you
know
the
99%
of
the
other
cases
that
it
just
took
longer
than
whatever
I
mean
yeah.
C
C
So
you
know
down
under
the
OSD,
take
the
rocks,
TV
layer,
and
so
on
so
I
mean
there
are
many
latency
adders
along
the
way
and
though
the
actually,
the
the
chart
was
basically
at
at
various
queue
depth,
so
the
the
for
second
latency
was
actually
at
64,
so
so
that
that
kind
of
cute',
after
your
clock,
effective
clustered
queue,
depth
is
probably
times
60
or
so,
and
so
so
that
basically
puts
a
lot
of
pressure
and
that
yeah.
So
quite
a
few
latency
addresses
on
the
way.
J
Thanks
a
lot
for
the
president
presentation
is
really
great
stuff
that
will
hopefully
address
many
of
the
issues
that
we
have
with
with
RVD,
where
we
have
users
that
suffer
from
to
large
Layton
sees
the
question
I
have
is
like,
because
it
respects
right,
burials
and
flushes.
There
would
be
no
issue
with
multi
attached
volumes
either
right
that
should
all
work.
J
B
But
and-
and
you
wanted
actually
the
best
performance
out
of
like
vehicle
cache,
you
would
have
to
basically
not
you
flushes.
The
bears
are
coming.
Local
application
get
recorded
locally
to
the
journal,
but
not
necessary.
That's
otherwise,
you're
installing
all
your
io
out
here
right,
really
no
better
than
you're,
just
taking
whatever
hit
that
the
OS
DS
and
the
network
are
providing.
So
in
a
workload
like
that,
I
would
say
it
is
not
a
good
fit
on
a
shared
RB
d.
B
So
I
mean
they're
they're,
separate
problems
that
are
solving
so
blue
stories.
You
know
a
great
reimagining
of
how
to
how
to
store
data
on
on
the
OSD
is
to
take
all
the
the
double
bit:
duffle,
jerk,
double
journaling
and
double
write,
penalties
off
and
basically
laugh
step
to
control
its
own
destiny.
You
know
directly
controlling
device,
as
opposed
to
event
having
a
layered
file
system,
on
top
sure.
B
B
K
Right
I
mean
everything
happens
at
a
cost.
To
do
something
else,
so
I
mean
that
question
is
you
know
where
everyone's,
obviously,
you
try
to
squeeze
sell
for
as
much
as
it's
capable
of
doing
I
understand
that
what's
what's
the
general
end
goal,
what's
the
customer
drivers
behind
this
who's
pushing
the
hardest?
For
you
know,
turning
this
into
a
performance
platform,
I.
C
H
C
G
You
one
question:
if
you
have
external
cash
external
from
chemo
protests
Rather's-
and
there
is
some
bad
consequences,
for
example,
see
me
crashing
host,
you
know
when
I
or
stalls
or
system
become
and
responsible
was
still
alive
on
the
normal
system.
With
the
in-memory
cache
you
terminate
chemo,
you
kind
get
rid
of
of
unwritten
data,
but
if
you
have
persistent
cache
on
the
host
and
separate
process-
and
you
restart
guests
on
the
new
host,
how
you
protect
guests,
which
start
doing
some
work
again
from
outdated,
writes
coming
from
the
cache,
which
is
a.