►
Description
Geographical Redundancy with rbd-mirror: Best Practices, Performance Recommendations, and Pitfalls - Florian Haas, City Network
rbd-mirror, introduced in the Jewel release, is a means of asynchronously replicating RADOS block device (RBD) content to a remote Ceph cluster. That's all fair and good, but how do I use it? How exactly does the rbd-mirror daemon work, what's the difference between one-way and two-way mirroring, what authentication considerations apply, and how do I deploy it in an automated fashion? How is mirroring related to RBD journaling, and how does that affect my RBD performance? And how do I integrate my mirrored devices into a cloud platform like OpenStack, so I can achieve true site-to-site redundancy and disaster recovery capability for persistent volumes?
This talk gives a run-down of the ins and outs of RBD mirroring, suggests best practices to deploy it, outlines performance considerations, and highlights pitfalls to avoid along the way.
Slides for this talk are at https://fghaas.github.io/cephalocon2019-rbdmirror/
About Florian Haas
City Network
VP Education
Austria
Websitehttps://fghaas.github.io/
Florian runs the Education business unit at City Network, and helps people learn to use, understand, and deploy complex technology. He has worked exclusively with open source software since about 2002, and has been heavily involved in OpenStack and Ceph since early 2012, and in Open edX since 2015. He co-founded hastexo, an independent professional services company, and served there as CEO and Principal Consultant until its acquisition by City Network in October 2017.
A
Rvd
mirror
is
a
set
feature
that
we've
had
for
three
years
since
the
jewel
release
as
a
means
of
asynchronously
replicating
Radice
block
device
content
to
remote
staff
cluster.
That's
all
fair
and
good
I
hear
you
cry,
but
how
do
I
use
the
damn
thing?
How
exactly
does
the
RVD
mirror
demon
work?
What's
the
difference
between
one-way
and
two-way,
mirroring?
What
authentication
considerations
apply?
How
do
I
deploy
RVD
mirror
either
manually
or
in
an
automated
fashion?
How
is
mirroring
related
to
our
BD
journaling?
How
does
that
affect
our
BD
performance?
A
And
how
do
I
integrate
my
mirrored
devices
into
a
cloud
platform
like
OpenStack,
so
I
can
achieve
true
site-to-site
redundancy
and
is
after
recovery
capability
for
persistent
volumes?
Well,
it's
great
you
ask
because
that's
what
this
talk
is
about
and
I'm
about,
to
give
you
a
rundown
of
the
ins
and
outs
of
RVD,
mirroring
I'll
suggest
some
best
practices
to
deploy
it.
I'll
outline
performance
considerations
and
I'll
try
to
highlight
some
of
the
traps
and
pitfalls
that
you
might
encounter
along
the
way.
My
name
is
Lorien.
A
I've
worked
with
SEF
since
about
2012,
and
my
twitter
handle
is
on
this
slide
and
it's
also
on
the
bottom
right.
The
bottom
right
corner
of
every
other
slide
in
this
talk.
So
if
you
have
any
questions
about
this,
then
please
feel
free
to
tweet
them,
including
my
handle
and,
if
time
permits,
then
I'll
be
happy
to
get
to
your
question
during
the
talk
and,
if
not
then
I'll
reply
to
your
question
on
Twitter
afterward.
A
So
as
an
application
writes
data
to
assess
cluster,
it
always
talks
to
a
single
OSD,
which
we
talk,
which
we
call
the
primary
OSD
for
every
bit
of
data
that
it
writes
and
that
primary
OSD
then
takes
care
of
replicating
that
right
to
the
other,
to
the
non
primary
OSD.
So
consider
the
flying
balls
here,
representing
I/o
in
flight.
A
So
as
long
as
it's
not
completed,
meaning
as
long
as
it
hasn't
turned
green
and
has
been
reported
as
such
from
all
the
OSDs,
the
application
itself
considers
it
not
completed
at
all
and
it
remains
blue,
and
only
when
we
receive
an
acknowledgement
from
all
of
our
OS
DS.
We
consider
the
right
operation
complete
and
the
application
turns
green
itself.
So
that
is
what
we
refer
to
as
synchronous
replication.
A
Synchronous
replication
is
inherently
latency
critical,
because
if
just
one
OSD
is
slow
or
a
merely
slow
to
get
to,
then
that
can
potentially
hold
up.
The
entire
application
basically
contributes
to
latency
for
the
entire
application,
which
is
why,
with
very
few
and
very
clearly
delineated
exceptions,
you
really
shouldn't
do
something
like
this.
So
this
approach
is
known
as
a
stretch
cluster,
and
it
means
that
you're
operating
a
single
SEF
cluster
that
is
stretched
across
several
physical
locations
or
sites.
A
Now
this
is
something
that
you
should
really
only
try
if
you
have
full
control
over
your
site
to
site
links,
and
you
can
ensure
that
your
latency
States
within
very
reasonable
limits,
which
normally
is
only
true
for
SEF
clusters,
where
all
of
the
nodes
are
within
the
same
city
and
also
where
you
have
dedicated
fiber
connections
between
them.
However,
building
these
stretch
clusters
becomes
impossible
once
you're
having
to
deal
with
C.
That
is
not
see
the
programming
language
but
see
the
speed
of
light.
A
These
scientists
and
academics
are
not
in
the
audience,
will
forgive
me
this
little
cheap
joke,
because,
obviously,
in
fact,
we're
not
talking
about
the
speed
of
light
in
a
vacuum,
but
in
fiber-optic
cables,
where
it
is
more
on
the
order
of
two
hundred
4190
point
five
kilometers
per
second.
But
you
need
to
consider
that
the
typical
Ethernet
round-trip
time
that
we
have
in
a
local
set
cluster
is
on
the
order
of
about
100,
microseconds
or
0.1
milliseconds.
A
Now
those
point
1
milliseconds
point
1
light
milliseconds
in
fiber,
that
is
to
say
the
distance
that
light
travels
in
a
glass
fiber
at
that
time
is
just
20.
Kilometers
or
a
round-trip
of
link,
that's
ten
kilometers
one
way
and
then,
of
course,
his
assuming
a
theoretically
perfect
environment,
where
there's
no
additional
latency
introduced
by
anything
other
than
that
of
light
traveling
in
a
straight
line,
fiber-optic
link,
but
you
never
have
between
the
two
locations.
A
But
generally
you
can't
ever
assume
any
land
like
latency
outside
the
ten
kilometer
radius,
and
that
also
means
that
if
your
cluster
sites
are
just
a
hundred
kilometers
apart,
which
is
which
doesn't
quite
cover
the
distance
from
here
to
the
French
border,
your
round-trip
time
increases
tenfold
versus
a
local
Ethernet
LAN
and
for
a
thousand
kilometers.
That's
roughly
the
distance
from
here
to
Munich
or
from
here
to
Lisbon.
A
It's
a
hundred
fold
so
clearly
building
stretch
clusters
is
not
an
option
if
you
want
to
replicate
your
Ceph
data
on
a
Continental,
let
alone
on
an
intercontinental
scale.
So
that's
why,
for
long
distance
replication,
we
have
to
take
a
completely
different
approach,
and
that
is
we're
not
doing
the
replication
at
the
low
radar
level
from
OSD
to
OSD,
but
instead
we're
doing
at
one
level
up
at
one
of
the
abstraction
layers.
A
On
top
of
Rados,
and
in
this
case
I'm
talking
about
the
abstraction
layer,
that's
most
important
for
virtualization
use
cases,
namely
RBD
and
importantly,
we
need
to
be
talking
about
replication.
That
is
asynchronous,
so
we
don't
have
to
worry
about
Network
latency
anymore.
So
what
we'd
like
to
get
is
an
RBD
image
whose
changes
are
asynchronously
applied
in
a
remote
location
and
to
get
that
capability.
We
need
to
track
and
record
those
changes
in
the
first
place
and
to
do
that,
we
use
an
RB
D
feature
called
journaling.
A
Now
our
BG
journaling
has
absolutely
nothing
to
do
with
OSD
journaling,
which
is
an
implementation
detail
of
the
OSD
file
store
implementation.
This
is
purely
this
purely
has
to
deal
with
Rattus
block
device
has
nothing
to
do
with
OSD
implementation
specifics.
So
the
purpose
of
this
OSD
at
this
I'm
sorry,
the
purpose
of
the
RBD
journal,
is
to
record
all
writes
that
includes
data
and
metadata
changes
to
an
RB
daemon
to
a
journal
of
Rados
objects,
preserving
a
consistent
point
in
time
stream
that
can
be
mirrored
to
other
sites.
A
The
journal
changes
to
a
different
RBD
image
when
we
get
to
that
in
a
moment,
and
it's
important
to
understand
that
as
of
today,
our
BD
journaling
is
a
Lib
RBD
only
feature
because
client-side
journaling
support
has
yet
to
land
in
the
RVD
ko
kernel
driver
and
we're
expecting
that
to
happen
within
the
next.
Shall
we
say
2
to
3
kernel
releases.
A
So
as
of
today,
it's
Lib
RBD
only
journaling
is
not
enabled
by
default
on
our
BD
images,
except
so
it's
in
general,
either
you
or
something
acting
on
your
behalf
must
enable
it
some
automation
whatever
and
the
RBD
utility
allows
you
to
do
that
with
our
BD
feature,
enable
in
principle.
You
could
also
make
this
a
default.
A
Of
course
there
is
the
RBD
image
features
flag
where
you
could
include
the
bit
for
the
journaling
feature,
but
that's
really
not
recommended
for
reasons
I
will
get
to
so
generally,
our
BD
feature
enable
followed
by
an
image
spec,
so
pool
slash
image
journaling
enables
this,
and
you
can
also
do
this
on
image
creation
or
image
import
with
the
image
feature.
Flag
and
you'll
need
to
specify
the
journaling
feature
itself
and
also
the
exclusive
lock
feature
that
it
depends
on.
A
It
should
be
noted
that,
at
this
stage
here
that
image
features
a
setting
image
feature
overrides
the
default
feature
set.
So
with
the
with
the
commands
shown
here.
What
you'll
get
for
the
images
is
just
journaling
an
exclusive
lock,
whereas
what
you
probably
really
want
is
image
feature:
layering
exclusive,
lock,
object,
move
fast
if
deep,
flatten
and
journaling,
but
I've
admitted
that
on
the
slide
here
just
for
brevity,
okay.
A
Just
as
easy,
we
can
use
our
BD
feature
disabled
to
turn
journaling
off
and
then,
if
we
do
our
B
the
info
again,
we
verify
that's
in
fact
off
and
if
we
list
our
radars
objects
again,
it's
gonna
happen
in
a
second,
then
you'll
notice
that
the
journal
objects
will
now
be
gone.
So
we
still
have
the
data
objects,
the
object
map
objects
and
that,
but
the
journal
objects
are
gone.
A
Okay,
so
our
BD
mirror
now
is
a
daemon
that
watches
journal
updates
in
journal
enabled
our
biddies
and
replays
the
journal
in
the
cluster
that
it
runs.
In
so
looks
roughly
like
this,
we've
got
to
set
clusters
which
here
I've
named
left
and
right
in
both
safe
clusters.
I
have
an
RB
d
image
and
when
properly
configured
an
RB
D
mirror
daemon
on
the
right
cluster.
In
this
example,
is
responsible
for
subscribing
to
the
journal
updates
that
are
generated
for
the
RB
d
image
in
the
left.
A
A
What
we
cannot
do
is
one
thing
which
is
reverse
the
direction
of
replication
so
replicate
into
the
left
cluster,
as
opposed
to
out
of
the
left
cluster,
because
the
left
site
in
this
graphic
here
doesn't
have
an
RB
d,
mirror
daemon
of
its
own,
and
so
therefore
this
is
the
mode
that
we
call
one-way,
mirroring
and
to
be
able
to
reverse
the
replication.
We
need
to
weigh
mirroring,
in
which
case
we
have
our
BD
mirror
demons
in
both
clusters.
A
Now
we
now
know
that
we
need
a
running
our
BD
mirror
that
can
watch
journals
on
one
cluster,
where
you
play
them
on
the
other,
and
that
means
that
that
Damon
needs
six
identities
with
very
specific
capabilities
on
both
clusters,
and
the
first
thing
that
we'll
need
to
do
in
order
to
get
our
BD
mirroring
enabled
is
create
those
on
pre
Nautilus
clusters
that
is
on
luminous
and
and
mimic.
We
have
to
do
this
either
manually
or
some
deployment.
Automation
has
to
do
it
on
our
behalf
and
Nautilus.
A
A
So
that's
why
I'm
talking
about
pre
Nautilus
releases
in
this
talk
at
length,
so
this
needs
to
happen
on
both
of
or
rather
all
of,
the
clusters
that
are
involved
in
replication.
We
we
create
here
a
client,
our
beady
mirror
some
some
unique
ID
user
or
suffix
identity,
and
we
give
it
the
Mon
profile,
our
beady
mirror
and
OSD
profile.
A
Rbd
capabilities
and
that
ensures
that
the
that
that
that
that
instance
can
participate
in
mirror
pure
discovery
and
mutual
authentication,
that's
what
the
amount
capabilities
for,
among
other
things,
and
they
can
read
from
and
write
to
our
BD
images
in
all
pools.
That's
what
the
OSD
capability
is
for
and
now
this
is
relatively
easy
because
we
have
capability
profiles
now
in
earlier
releases.
This
would
have
been
more
difficult
because
we'd
have
to
like
set
them
up
with
individual
capabilities.
Now
we
have
the
cap
profiles
great.
So
that's
relatively
easy
and
relatively
simple.
A
What
is
a
little
bit
trickier
is
enabling
the
RBD
mirror
instances
to
find
these
authentication
credentials,
which
largely
means
getting
the
keyring
locations
right.
So
first
on
this
gets
a
bit
complicated.
So,
first,
on
whatever
nodes
you
are
going
to
be
running,
our
BD
mirror
the
daemon
is
going
to
have
to
be
able
to
access
the
local
cluster
and
therefore
it
will
need
to
find
its
keyring
in
the
default
location.
A
A
So
if
you
want
to
call
your
RBD
mirror,
daemon
RBD,
mirror
or
foo
something,
then
the
proper
appearing
location
for
the
daemon
to
access
the
local
cluster
would
be
at
CCF
SEF
for
the
local
cluster
named
client
for
the
type
RBD
mirror
foo
for
the
ID
and
then
dot
keyring.
So
far
so
simple,
but
the
RBD
mirror
daemon
also
needs
to
access
the
remote
cluster,
and
this
is
the
part
where
it
gets
a
little
bit
complicated
because
first
you're
gonna
have
to
tell
the
RBD
mirror
where
to
find
the
remote
cluster.
A
That's
relatively
simple
because
all
it
needs
to
find
is
the
Mon
hosts,
and
so
you
can
give
you
our
beady
mirror
node
a
relatively
bare-bones
SEF
client
configuration
by
dropping
a
configuration
file
that
is
now
not
named
safecom,
but
remote
cluster
named
conf
into
at
CSF.
So,
for
example,
if
you
want
to
connect
to
the
right
cluster
from
the
left,
you
might
call
this
Etsy
SEF
right
conf
and
you
put
in
the
Mon
hosts
for
that
and
then
it
needs
to
needs
to
find
the
correct
hearing
for
connecting
to
that
cluster.
A
And
this
follows
the
same
naming
convention
as
before.
But
now
we
don't
use
the
default
cluster
name
of
SEF,
but
a
different
one.
So,
for
example,
it
might
be
Etsy,
SEF,
writes,
client,
RBD,
mirror
foo
keyring,
and
then
we
have
to
do
that
in
Reverse
on
the
other
side
of
the
replication
link
as
well.
So
that
sounds
complicated
and
it
is.
But
it's
really
quite
logical,
but
good
news
is
that
most
of
this
you
actually
don't
have
to
do
in
Nautilus
anymore.
A
But
if
you
want
to
do
this
with
luminescent
mimic,
that's
what
you,
what
you
need
to
do
if
you're
setting
up
our
BD
mirror
manually
or
you're
troubleshooting,
what
your
deployment
automation
setup
for
you
or
your
troubleshooting
RBD,
mirror
authentication
issues
in
general,
then
this
is
what
you
can
do
to
do,
that
you
should
always
be
able
to
contact
the
local
cluster
and
the
remote
cluster.
So
if
you
do
SEF
ID
RBD
mirror
dot,
unique
ID
s
or
status,
then
you're
connecting
to
your
local
cluster
as
our
BD
mirror,
and
that
should
return
a
result.
A
And
then,
if
you
do
SEF
cluster
right,
ID
something-or-other,
yes,
you
should
be
able
to
see
the
remote
cluster
status
and
the
same
should
be
true
in
Reverse.
Now,
with
all
those
prerequisites
squared
away,
we
can
discuss
what
needs
to
happen
in
order
to
actually
start
replicating,
and
this
depends
somewhat
on
your
deployment
automation
facility.
It
might
be
automated
away
to
various
degrees,
but
it
still
absolutely
doesn't
hurt
to
know
what's
happening
under
the
cover.
A
So
the
first
thing
that
we
need
is,
we
need
a
way
to
reliably
start
the
RBD
mirror
binary
and,
if
you're
running
on
a
conventional
that
to
say
a
non
container
driven
deployment
scheme,
the
RBD
mirror
binary
is
normally
started
from
system
D
and,
as
such,
the
package
deploys
a
system
to
unit
file
and
this.
How
is
how
you
enable
or
start
an
RB
D
mirror
service?
That's
managed
by
system
D,
relatively
straightforward,
and
something
that
you
probably
never
want
to
do
manually
is
run
or
BD
mirror
in
the
foreground.
A
If
you
have
to
this
is
how
you
do
it.
However,
this
is
also
how
application
containers
invoke
our
BD
mirrors.
So
if
you're
using
the
docker
container
approach
that
SEF
ansible
uses,
if
you
so
choose,
if
you
tell
it
to
or
you
are
using
rook,
this
is
how
it
would
invoke
by
and
large.
This
is
how
it
invokes
RB
d
mirror
as
a
foreground
daemon,
and
then
you
need
to
enable
it.
So,
even
if
you
have
our
BD
mirrors
running
our
beti
mirror
demons
running
on
all
your
sites.
A
A
What
has
to
happen
for
every
image
that
you
want
mirrored
is,
it
has
to
have
journaling
enabled,
and
it
also
has
to
have
mirroring
explicitly
enabled
there's
an
RB
D
mirror
image
enable
command
for
that
and
then
you'll
need
to
connect.
Our
BD
mirror
instances
to
one
another
and
we
refer
collectively
to
our
BD
mirror
instances
that
are
meant
to
talk
to
each
other
as
peers,
but
I
would
strongly
caution
against
using
the
term
peering
for
this
process,
because
all
you're
gonna
cause
is
a
royal
amount
of
confusion.
A
If
you're
talking
to
someone
who
is
not
yet
familiar
with
our
BD
mirror,
but
who
familiar
with
the
concept
of
PGP
replacement
group
hearing,
which
is
a
completely
different
thing.
Okay,
so
just
don't
call
it
peering
say
connecting
peers
instead
and
for
connecting
peers.
You've
got
basically
two
options:
you
can
do
everything
from
one
host
or
you
can
first
shell
into
one
and
then
the
other,
whichever
you
prefer.
So
you
do
RBD
mirror
pool
peer
ads
and
then
for
a
pool.
A
You
add
the
peer
and
then,
if
you
want
to
a
mirroring,
you
also
do
that
in
Reverse.
So
you
can
either
connect
to
the
other
cluster
from
your
SEF
client
or
you
can
just
shell
into
one
and
then
shell
into
the
other,
and
then
here's
a
that
screencast
loops.
So
that's
an
image
with
mirroring
enabled
right.
A
You
do
RBD
info
and
once
you
do
have
both
journaling
and
and
mirroring
enabled,
then
you
see
not
only
the
features
not
only
the
journal
ID,
but
also
the
mirroring
state
and
where
whether
we're
mirroring
or
not,
and
whether
we're
a
mirroring
primary
or
not,
and
also
you
have
our
BD
mirror
status.
That
gives
you
a
little
more
information
about
the
status
of
replications,
such
as
the
current
state
that
the
journal
is
at
as
its
as
its
replaying
and
then
I
mentioned
before
we
can
demote
and
promote
images.
A
So
once
the
clusters
are
properly
connected,
replication
is
active,
then
you
can
influence
the
direction
of
your
replication.
So
if
you
have
an
image,
that's
currently
primary
and
you
want
to
demote
it
to
the
non
primary
status.
You
do
that
with
our
BD
mirror
image
demote
or
you
can
also
do
that
for
all
images
in
a
pool
with
our
BD
mirror,
pool
demote
and
the
reverse
is
called
promotion
and
that's
our
BD
mirror
image,
promote,
pool,
slash
image
or
promote
all
of
the
images
in
a
pool
with
our
BD
mirror
pool
promote.
A
Now
an
image
must
be
demoted
on
one
side
before
it
can
be
promoted
on
the
other,
obviously,
except
in
the
case
that
it's
peer
is
down
or
unreachable,
so
something
actually
knocked
out
your
primary
cluster.
You
now
need
to
promote
all
of
your
mirrored
images
on
the
other
site.
In
that
case,
you
can
force
promotion
with
the
force
option.
A
It
does
also
help
you
with
with
keyring
exchange
and
remote
cluster
client
configuration,
so
it
does
give
you
the
tools
to
fully
automate
a
cluster
pair,
but
only
if
you
sought
to
set
two
different
clustering
names
from
the
get-go
and
if
one
of
your
cluster
is
already
employed
and
uses
the
name
SEF,
there's
some
manual
intervention,
that's
required
before
you
can
then
manage
everything
else
with
with
regards
to
our
BD,
mirroring
with
staff,
ansible
and,
of
course,
promotion.
Demotion
of
individual
images
are
completely
outside
a
fanciful
scope.
A
There's
something
that
very
recently
happened
in
juju
LAN.
So
in
juju
there
is
now
a
safe
RBD
mirror
charm.
It's
brand
spanking
new
that
can
automate
the
entire
BD
mirror
lifecycle.
That
includes
the
daemon
deployment.
Key
exchange
with
juju
offered
you
to
consume
failover
and
failback,
so
a
promotion
demotion
with
juju
run
action
and
so
forth.
It's
basically
meant
to
be
sort
of
a
full
lifecycle
management
tool
for
safe
RVD
mirror
and
looks
pretty
neat
deepsea
currently
supports
no
RBD
mirror
automation
facilities
whatever.
A
However,
if
you
happen
to
be
on
Suzanne
price
storage,
which
is
deployed
with
deep
sea,
then
you're
supported
RBD,
mirroring
deployment,
automation
facility
in
Suzanne,
price
storage,
6
and
later
is
just
the
post,
Nautilus
or
Nautilus
plus
set
dashboard,
where
you
can
do
all
of
that
from
the
dashboard
and
with
Nautilus.
You
also
don't
have
to
worry
about
with
the
stateless
RBD
mirror
demons,
there's
a
lot
less
to
worry
about
with
key
exchange
and
blah
blah.
So
basically
you
from
SES
6
forward
as
I
understand
it.
A
You
don't
really
need
deep
sea
to
manage
RBD
mirror
because
set
by
itself,
and
the
dashboard
can
essentially
do
it
quite
nicely
themselves.
And
finally,
your
rook
is
also
capable
of
managing
our
beady
mirror
instances
and
configuring
them,
but
it
does
currently
rely
on
a
cluster
administrator
running
the
RBD
mirror
commands
for
for
peer
to
peer
connection
and
pool
and
image
actions.
A
So
now
about
the
performance
implications
of
our
bde,
mirroring
I've,
good
news
and
I
have
bad
news.
So
the
good
news
is
our
BD.
Mirroring
really
doesn't
have
a
major
impact
on
an
images
performance
right,
it's
barely
measurable.
The
bad
news
is
journaling,
unfortunately,
absolutely
does,
and
you
can't
have
mirrored
volumes
without
journaling
right.
So
you
have
to
take
this
into
consideration.
So
what
I'm?
Talking
about
sorry?
Okay?
So
what
I'm
talking
about
next?
The
the
data
that
I've
compiled
for
the
following
slides
are
from
our
BD
bench
runs.
A
They
were
all
run
against
a
single
one,
gig
RBD
image,
using
the
default
order
of
22.
So
a
four
megabyte
object,
size
which
I
ran
on
an
all
SSD
blues,
all
blue
store
or
luminous
cluster,
which
used
the
same
pool
for
the
journal
objects
as
the
data
objects.
You
also
have
the
option
of
splitting
those
basically
write.
Your
journal
objects
into
a
different
pool.
I
didn't
do
that
here,
and
these
benchmarks
were
run
with
the
default.
A
setting
of
16
I
owe
3.
A
They
were
run
in
sequential
and
random,
read
and
write
mode
with
the
I/o
size,
whoops,
sorry
with
the
I/o
sizes,
varying
up.
No,
no,
no
sorry
there.
We
hope,
with
the
I/o
sizes
varying
from
4k
to
4,
make
it's
important
to
understand.
None
of
the
absolute
numbers
in
these
results
should
be
considered
dependable
benchmarks
and
you
shouldn't
compare
them
to
your
own
cluster
and
they
might
differ
wildly
based
on
what
hardware
you
use,
what
configuration
you
use,
etc.
My
point
is
simply
to
show
the
impact
that
journaling
has
in
relative
terms
versus
running
non-journal
RPD.
A
So
again,
good
news
first,
with
with
with
with
reads
well
I,
should
perhaps
explain
here.
What
do
you
see
on
these
charts?
All
of
these
charts.
We
have
the
I/o
size
on
the
x
axis
that
goes
on
a
scale
of
powers
of
2,
from
4k
to
4
meg
and
the
y
axis,
which
is
linear
that
has
either
bytes
per
second.
If
we're
dealing
with
throughput
or
I
ops.
A
If
it's
I
ups
that
we're
looking
at
the
black
line
is
always
what
we
get
without
journaling,
the
red
line
is
with
journaling
enabled
okay
and
so
I
promise
good
news
first.
So
here's
some
good
news,
there's
very
little
impact
that
RBD
journaling
has
on
image
reads
right.
So
if
you
have
a
read
heavy
workload,
then
that
shouldn't
expect
to
suffer
very
badly
when
RBD
journaling
is
enabled-
and
that
is
true
for
sequential
reads
what
we
see
here.
A
Okay,
now
generally,
if
we're
writing
our
data
and
our
journal
objects
to
the
same
pool
which
we
are
in
this
case,
then
we're
sort
of
assuming
our
throughput
to
be
roughly
half,
because
we're
writing
everything
twice.
That's
what
you
see
on
the
right-hand
side
of
the
of
the
chart
is
so
it's
true
for
rather
large
IO
sizes,
from
128
K
to
4.
Meg
is
a
roundabout
that
territory,
but
for
small
writes.
The
overhead
is
really
quite
astounding
right,
you
might
be
dealing
with
with
an
overhead
of
a
factor
of
10
for
for
4k
writes.
A
Now
there
is
something
you
can
do
to
address
this.
You
can-
and
this
is
word
from
this
very
conference
so
which
is
there-
are
some
Lib
RBD
client-side
options
that
you
can
set
that
determine
under
what
circumstances
the
RBD
Journal
is
being
flushed
and
all
these
all
these
are
beady
journal,
object
flush
parameters
default
to
zero,
which
means
essentially
flush
the
journal
after
every
write
and,
of
course,
when
we're
dealing
with
a
great
deal
or
a
great
number
of
small
writes
that
has
a
pretty
significant
impact
and
in
pre
Nautilus.
A
This
would
be
something
that
you
need
to
set
on
every
single
client,
whereas
from
Nautilus
onward
you
can
set
it
centrally
with
cluster
with
with
cluster
side
configuration,
because
that
also
propagates
to
the
clients.
That,
however,
is
something
I've
yet
to
test
okay
same
pattern:
expectedly
minutes
manifests
itself
for
random,
write,
throughput,
sequential
write,
I,
ops,
we're
also
seeing
that
9/10
reduction
for
for
small
writes,
that's
on
the
left
of
the
chart
and
again
for
random
writes.
A
It
looks
just
a
little
less
than
what
you'd
intuitively
expect,
but
it's
so
quite
significant
for
for
small
writes
just
so.
You
believe
me
that
it
really
it's
really
journaling
that
makes
the
difference
and
and
not
mirroring
what
follows
are
some
benchmarks
with
mirroring
enabled
those
were
taken
on
different
machines?
There
were
there
were
virtual
machines
even
with
file
store.
The
absolute
numbers
are
basically
completely
junk,
but
the
relative
comparison
versus
non
journaled
and
a
journals,
but
unmarred
devices
should
still
be
useful.
A
I'll
go
through
these
really
quickly
because
you
will
catch
the
gist
really
fast.
Just
look
at
how
close
the
the
blue
and
the
and
the
and
the
red
line
are
right,
so
black
is
still
without
journaling.
Blue
is
Wis,
mirroring
and
red
is
still
with
journaling,
but
with
no
mirroring
right
and
so,
like
I
said,
mirroring
and
journaling
doesn't
really
make
much
of
a
difference.
So
reads
nothing
at
all,
but
then
for
writes
here
again
is
the
rude
awakening
and
again,
black
is
no
journaling.
A
A
You
have
really
good
reasons
for
enabling
journaling
on
your
devices,
such
as
selectively,
mirroring
individual
volumes,
and
you
have
weighed
those
expected
benefits
against
the
performance
implications
that
I
just
talked
about
then
do
that,
but
don't
under
any
circumstances
enable
it
globally
or
just
on
a
whim,
because
chances
are
that
your
customers
are
gonna
hate
you
for
it.
Okay,
again,
I
need
to
reiterate,
luminous
and
and
and
my
hardware,
and
what
have
you,
but
my
slides
include
the
backup
slides
here.
If
you
look
them
up,
they
include
the
benchmark
strips.
A
So
if
you
want
to
replicate
this,
then
you're
certainly
welcome
to
do
so
and
then
finally,
I
have
some
thoughts
on
the
integration
of
our
beady
mirror
with
cloud
applications
and
I
only
have
a
few
minutes
left
and
I.
Consider
myself
qualified
to
only
talk
about
OpenStack
rather
than
any
other
cloud
platform.
So
that's
what
I'm
gonna
be
talking
about
so
first
up
in
OpenStack.
A
The
definition
is
that
any
persistent
data
needs
to
go
into
a
volume
and
therefore
everything
that
is
not
in
a
volume
is
by
definition,
ephemeral
and
in
so
that
means
the
only
user-generated
data.
That's
persistent
is
in
cinder
right
and
in
cinder,
which
is
the
block
storage
of
service
in
OpenStack
and
in
center
we
have
the
concept
of
replication,
which
you
have
to
define
by
adding
this
replication
device
stanza
to
your
cinder,
back-end
configuration
and
that's
totally
supported
for
the
RB
driver.
A
So
you
can
totally
have
a
cinder
interface
with
our
BD
mirroring
and
our
BD
mirror
managed
volumes
which
is,
admittedly,
a
bit
under
documented.
But
generally
it
works
in
such
a
way
that
your
OpenStack
environment
can
use
a
primary
and
a
standby
set
cluster
to
talk
to,
and
it
can
even
failover
volumes
between
one
or
the
other.
Does
this
get
me
full
automated
disaster
recovery
capability?
A
So
just
having
a
mirrored
volume
named
volume,
some
UUID
in
a
pool
isn't
going
to
help
you
a
whole
lot
if
you're
lacking
all
the
metadata,
that's
related
to
that
volume
such
as
what
OpenStack
tenant
doesn't
belong
to.
What
of
what
VMs?
Does
it
attached
you
and
so
on?
So
do
I
really
have
no
way
of
doing
this.
Well,
I
think
we
do,
but
to
be
fair.
A
What
follows
is
also
hand
waving
because
I've
not
implemented
this
in
production,
but
it's
the
only
way
that
I
can
imagine
for
this
to
work,
but
you'd
have
to
do
it
from
the
start
from
where
you
first
stand
up
your
OpenStack
environment,
and
you
have
to
get
it
right
immediately,
because
there's
practically
no
way
to
add
this
as
an
afterthought
right.
So
this
is
I.
Maintain
your
checklist
for
actually
building
dr
capability
into
your
OpenStack
environment.
A
In
the
first
place,
and
you
replicate
between
those
with
Galera
you're
gonna
need
OpenStack
API
services
in
all
those
locations
and
you're
also
going
to
need
availability
zones
by
sites
for
those
three
locations
for
Nova
senior
and
Neutron
you're
gonna
need
compute
notes
and
network
gateway,
Notes
Plus,
independent
provider
up
links
in
all
those
locations,
plus
a
dynamic
routing
protocol
like
BGP,
so
you
can
arbitrarily
announce.
Routable
IP
addresses
either
here
there
or
there
you're
going
to
need
a
management,
tenant
and
storage
network
layer.
A
3
connectivity
between
all
locations,
Nova
glance
and
cinder
in
all
those
locations
must
be
wired
up
to
their
local,
safe
cluster.
And
then
you
have
to
set
up
2-way
pool
level
replication
for
your
Nova
glance
and
syndra
pools,
and
you
have
to
all
in
Abel
journaling
on
all
RVD
images
created
in
those
pools
with
the
performance
implications
that
I
previously
discussed.
A
Ok
I'll
tackle
my
Twitter
very
promptly
for
your
questions,
but
before
I
do
that
as
pretty
much
all
my
presentations,
this
deck
is
Creative
Commons
licensed
creative
commons
attribution/share-alike.
This
is
where
you
find
them
again
and
finally
I'd
like
to
add
a
few
acknowledgments
for
some
people
who
kindly
provided
feedback
and
input
on
portions
of
this
talk.
So
how
we
doing
on
the
question
list.
There
yeah.