►
Description
Presented by: Danny Abukalam | SoftIron
Benchmarking Ceph has always been a complex task - there are lots of tools but many have drawbacks and are written for more general-purpose use. For Ceph we need to benchmark Librados, RBD, CephFS, and RGW and each of these protocols has unique challenges and typical deployment scenarios. Not only that, Ceph works better at scale and so we need to ensure that we can build a benchmarking system that will also scale and be able to generate an adequate load at large scale.
A
Three
three
Mark
Nelson,
two
others,
okay,
okay,
interesting
so
so
I
think
benchmarking
can
be
painful
for
a
few
reasons.
Firstly,
you
need
to
run
tests
many
times
and
consistently
consistently.
The
tests
you
need
to
run
need
to
be
a
minimum
length
of
time.
You
there's
loads
of
variables,
loads
and
loads
of
variables,
and
so
you
try
and
change
one
variable
at
a
time.
A
But
then
you
deduce
results
and
realize
that,
even
though
you
were
only
changing
one
variable,
there
was
actually
loads
of
variables
of
a
play
and
and
what
you
thought
was
true
was
not
true
and
you
just
yeah
give
up.
So
so
it's
kind
of
painful,
because
you
want
good
tools
and
you
don't
always
have
you
don't
always
have.
Sometimes
your
tools
get
in
the
way
so
for
for
staff.
A
A
We
have
massively
varying
workloads,
so
for
each
one
of
these
protocols
they
all
have
different
characteristics.
So
we
need
to
take
different
approaches
and
measure
them
independently
distributed
systems
need
distributed
benchmarks.
So
we
need
a
benchmarking.
Tooling,
that's
going
to
scale
with
our
distributed
system
right
and
also-
and
this
is
a
big
one-
workloads
can
be
invisible
to
operators
because
most
people
that
are
deploying
and
operating
SEF
aren't
actually
using
themselves
they're
presenting
it
to
a
set
of
customers,
and
those
customers
are
using
it
in
whatever
way.
Who
knows?
A
Please
don't
do
that,
and
so
sometimes
you
know
basically
you're
predicting
the
workloads
that
you
need
to
optimize
for
can
be
a
challenge.
If
you
don't
know
what
those
are.
And
finally,
this
is
this
is
great
as
well
is
that
Seth's
background
work
can
also
get
in
the
way.
So
SF
does
a
whole
bunch
of
background
work,
that's
semi-visible
or
sometimes
invisible,
and
this
can
have
a
performance
impact
if
you're
not
aware
of
it.
A
So
you
need
to
kind
of
be
aware
so
up
until
now,
there's
there's
a
whole
bunch
of
different
tools
for
benchmarking
stuff,
but
a
big
one
for
a
while
was
cause
bench
and
maybe
some
people
how
many
people
are
still
using
cause
bench
today
or
have
used
cost
bench
in
the
past
right.
Okay,
so
start
with
have
used
in
the
past
and
you're
still
using
today,
two
people
kind
of
one
and
a
half
okay.
A
Well,
so
we
use
cause
bench
for
a
while
as
well
and
so
did
our
customers,
and
we
thought
oh
yeah,
cause
bench
great
yeah,
but
actually
cos
bench
had
a
bunch
of
issues.
The
first
one
is
the
the
Java
native
interface
is
pretty
expensive,
and
so,
if
you're
doing
it
for
anything
other
than
S3
it's
it's
not
great.
Amazon
has
a
pure
Java
implementation
of
S3
clients,
so
it's
cheap
to
run
S3
benchmarks
from
jvm,
but
for
anything
else
like
liberators
live
RBD
cause
bench
has
to
Traverse.
A
A
It
uses
this
thing
called
osgi
and
the
idea
with
ohgi
is
that
everything's,
an
independent
bundle
rather
than
a
monolithic
application,
so
cost
bench
was
originally
targeting
lots
of
different
object
protocols
and
that's
probably
why
they
did
that.
But
the
result
is
it's
incredibly
fragile
structure,
that's
difficult
to
navigate
and
even
more
difficult
to
debug
when
things
go
wrong,
so
yeah
the
workflow
is
also
very
manual.
A
You
submit
these
XML
jobs
and
you
pick
them
up
and
there's
no
Builder
install
systems,
so
the
binaries
are
in
a
repo
and
then
you
kind
of
just
have
to
there's
no
way
it
doesn't
really
tell
you
how
to
build
it
from
Source,
so
there's
no
other
way
to
install
it.
So,
basically
inspire
all
these
issues.
Lots
of
people
were
using
cause
bench.
B
A
There
was
a
value
to
that
because
it
was
a
benchmark
in
the
sense
of
everyone's
using
the
same
thing.
So
even
if
it's
a
bit
not
so
good
everyone's
having
the
bit,
not
so
good
experience,
so
there's
value
in
that
right,
and
so
that's
why
we
kind
of
tried
to
stick
with
it,
and
so
we
did
a
whole
bunch
of
work.
Trying
to
make
this
more
pleasant.
We
got
a
building
with
Maven.
A
We
want
a
tool,
that's
simple
and
lightweight
easy
to
read
easy
to
run
easy
debug.
We
want
it
to
be
linearly
scalable
like
surface,
so
we
don't
want
the
benchmarking
tool
to
get
in
the
way
or
to
be
the
Benchmark
the
tool
itself.
We
want
to
Benchmark
staff.
We
want
to
Benchmark
all
the
set
protocols.
We
want
something
that's
designed
for
saf.
A
A
So
what
what
did
we
do
so
cybenchpin
and
go
so
it's
almost
free
to
call
out
two
steps:
libraries,
it's
both
a
demon
and
a
CLI
tool.
So
when
you
install
it,
a
cybance
driver,
Daemon
listens
for
work
on
a
port,
and
you
can
then
use
the
CLI
tool
to
generate
work
for
the
demons
with
localhost's
default.
Even
but
then
you
can
just
provided
other
workers
as
well.
A
A
A
It
does
bits
or
bytes,
because
networking
people
like
bits
and
then
storage
people
like
bytes
and
then
they
have
arguments
about
which
is
the
right
thing
to
format.
So
we
just
do
both
and
then
it
has
ramp
time
like
cos,
bench
does
as
well,
so
you
can
say
don't
actually
record
the
first
X
seconds
or
the
last
X
seconds,
so
you
try
and
get
a
representative
result
and
then
finally,
it
focuses
on
actual
benchmarking
and
not
orchestration,
because
there's
an
orchestration
piece
so
with
sidebench.
A
So
yeah
here
we
can
see
yeah.
So
here
you
can
see
that
the
different
protocols
that
we
we
support
with
sidebench
to
start
with,
so
you
can
see
cyberge
talks
directly
to
the
monitor
pool
and
then
each
worker
you
have
a
worker.
Basically
for
liberatos
you
give
it
a
Seth
pool,
you
give
it
a
SEF
key,
you
give
it
a
monitor,
address
and
then
that's
really
all.
A
A
We
don't
need
to
provide
a
pool,
so
we
just
provide
a
key
and
a
monitor
address
and
then
it
mounts
the
file
system
and
then
each
worker
thread
reads
and
writes
its
own
files
into
that
directory.
Wireless
gateway
Works
differently.
So
you
provide
an
S3
access,
secret,
key
and
S3
bucket,
and
then
this
actually
doesn't
even
need
to
be
a
router's
Gateway.
It
could
be
any
HTTP
endpoint,
so
you
could
put
your
load
balances,
Benchmark
load
balances.
You
could
Benchmark
non-s3
endpoint
as
well.
Theoretically,
I
mean
I.
Guess
that
would
work.
I!
A
Think
I
think
you
can
and
then
finally
you
have
the
ability
to
Benchmark
native
block
and
file
as
well.
So
you
can
just
point
it
at
a
block
device
or
out
of
folder
and
and
you'll
have
to
config
my
Mount
or
map
these
manually.
But
you
can
then
use
to
Benchmark
things
like
iSCSI
or
NFS
and
SMB
as
well.
A
So
some
of
the
other
cool
things
that
cybanks
can
currently
do
it
can
do
bandwidth
limiting.
So
we
have
customers
that
give
us
requirements
such
as
hey.
We
need
to
have
100
milliseconds
of
response
time
when
doing
30,
gigabytes
of
traffic
or
whatever
I
just
made
those
numbers
up,
but
it's
a
good
way
of
getting
latency
numbers
at
a
certain
bandwidth.
So
you
can
say:
I
don't
want
to
max
out
the
bandwidth
on
every
driver.
A
I
want
to
hit
a
certain
bandwidth
limit
and
then,
after
that,
what
I'd
like
to
do
is
kind
of
optimize
for
latency.
So
that's
pretty
good.
It's
a
good
way
to
make
sure
the
workers
aren't
maxing
out
their
pipe
every
time
it
has
a
this
slice
generator
so
by
default.
If
you
don't
use
this,
it
generates
random
data.
But
if
you
want
to
measure
for
Compassion
or
the
duplication,
you
can't
you
can't
do
that
with
random
data.
A
B
A
Are
like
50
reads:
50
rides
or
30
70
or
whatever,
and
then
it
also
has
support
for
writing
out
all
the
individual
stats
from
each
worker.
So
this
can
be
used
for
trying
to
understand
what's
happening.
If
the
numbers
don't
make
sense
or
you
just
want
to
debug
exactly
what
happened
in
a
workload
or
if
you
want
to
run
statistical
analysis
over
the
data
that
obviously
the
file
is
huge
because
you're
getting
a
far
adjacent
file
of
like
every
single
thing
that
every
driver
has
done
every
thread.
A
A
So
what's
bench
master,
so
benchmaster
is
a
little
python
Tool.
That
is
a
wrapper
for
both
cause
bench
and
sidebench,
and
and
it's
for
running
a
series
of
benchmarks
rather
than
just
a
single
one,
and
it
allows
us
to
provide
a
set
of
options
to
sweep
over,
so
I
can
say,
run
a
workload
for
1K,
4K
and
16k
and
I'll
go
and
do
all
three
of
those,
and
it
also
writes
out
all
the
the
workload
results
to
Google
Sheets.
A
C
A
C
A
So
that's
that's
the
demon
and
then,
if
I
do
you
know
sidebench
Ice
Age,
you
can
see
that
I
have
a
little
command
line,
utility
for
running
S3,
red
or
Surface
RBD
block
file
and
a
bunch
of
options.
We've
also
got
a
man
page,
which
I'm
a
huge
fan
of
I,
like
Man
pages,
so
kind
of,
goes
in
and
gives
you
some
info
on
various
things
and
the
other
thing
we
have.
A
A
So
you
can
see
it's
telling
me
the
list
of
drivers
which
in
this
case
is
just
the
Local
Host
note
that
I'm
on
and
if
I
had
more
nodes,
I'd
pass
them
as
as
on
the
command
line,
as
the
the
workers
and
I
can
see.
The
you
know,
object,
failures,
verification
failures,
it's
all
fine
I
can
see
the
right
and
the
read
and
they're
kind
of
separated.
A
You
can
see
it's
creating
a
Json
report,
writes
and
then
there's
a
prepare
phase
and
a
read
face
and
at
the
end
it
tells
me
hey
here's
the
kind
of
results
and
it
does
it
breaks
out
for
per
per
Target
and
per
driver
as
well
for
the
obviously,
with
rados
you're
only
going
to
have
one
monitorium.
It
doesn't
really
matter
so
you
know
for
RBD.
This
is
going
to
look
very
similar.
A
Basically,
it's
exactly
the
same
command
except
I'm,
using
RBD
instead,
and
what
you
see
here
is
the
same
thing,
but
what's
happening,
is
it's
actually
creating
loads
of
RBD
images
from
each
worker
thread?
So
in
this
case,
I
only
have
one
thread:
One
driver
but
I
have
32
cores
on
this
on
this
node.
So
it's
going
to
spin
up
32
RBD
images
right
read
to
them,
but
that's
all
configurable.
So
this
just
defaults.
A
A
A
A
A
A
A
This
is
why
everyone
else
used
the
clicker
thing.
I
had
to
be
clever,
okay,
so
the
final
thing
is
ideas
for
the
future.
So
what
we
thought
would
be
cool
is
like
a
workload
generator,
so
you
could
basically
have
this
demon.
This
is
like
a
dream
idea,
pipe
dream
idea,
but,
as
you
can
tell,
we've
made
this
particularly
for
our
own
use
cases,
so
it's
very
much
what
we
needed.
So
it
might
not
be
what
you
need
and
that's
fine,
but
if
it
is
then
that's
great
as
well.
A
So
one
of
the
things
we
quite
like
to
do
is
have
a
workload
generator
where
you
could
kind
of
sit
and
watch
how
safe
cluster
behaves
over
time
and
then
spit
out
a
set
of
side
bench
benchmarks
that
will
give
you
a
representative
kind
of
this
is
how
you've
used
this
cluster
over
the
last
month
or
whatever,
and
here's
a
bunch
of
benchmarks
that
you
could
run
to
help
you
optimize
and
get
to
a
solution.
That's
going
to
make
sense.
A
Also,
meta
operations
like
snapshots
or
omaps
will
be
able
to
like
create,
create
and
delete
images
or
some
other
things
in
Seth,
which
isn't
necessarily
just
reads
or
writes,
would
be
cool
to
add
kubernetes
support.
So
we
have
a
lot
of
customers
using
our
SEF
as
kind
of
CSI
persistent
volumes
and
having
something
that's
friendly,
for
that
would
be
interesting
and
yeah
I'm
interested
in
hearing
your
guys
kind
of
questions
towards
feedback
or
whatever
you
think
might
be
cool
as
well
and
yeah.
That's
it
that's.
Basically,
it.
D
Danny
great,
oh
sorry,
Danny
great
talk.
Thank
you
actually
on
that
last
slide.
One
of
the
things
that
I
think
would
be
interesting
for
me
for
us
would
be
the
ability
to
profile
a
future
event
like
what
would
happen
if
my
load
went
up
50
what
would
happen
if
I
added
50
osds?
What
would
happen
if
I
decreased
by
50
osds
like
Benchmark
against
that
and
see
what
your
performance
would
be,
because
there's
tuning
considerations
for
those
events
to
plan
ahead,
yeah.
C
E
Denny
I'm
Mark
yeah,
so
I'm
I'm
I'm
super
impressed
that
you
did
a
live
presentation,
good
job,
so
I
I
definitely
want
to
talk
to
you
later
about
all
this.
But
first
question
for
you
is:
if
I
want
to
just
use
the
individual
pieces
that
you
have
for
like
doing
block
or
doing
S3
testing.
Do
you
need
the
the
demon
to
do
that?
Or
can
you
just
run
it
like
individually
on
a
client
like
you'd,
run
fio.
A
You
need
so
the
fundamental
architecture
of
sidebench
is
you
need
to
have
a
the
way
it
works?
Is
the
demon
listens
on
a
pawn
and
then
side
bench
itself?
Is
both
a
demon
and
a
cell
at
all,
so
you
can
run
it
from
any
of
the
Demons
or
just
runs
high
bench,
but
it
you
start
the
server
on
on
on
wherever
you're
benchmarking
from
and
that
listens.
And
then
you
talk
to
that
server
over
the
side.
Bench
port
and
say:
send
this
chop
out,
send
that
job
out.
A
E
F
It
looks
fantastic,
the
kind
of
having
used
cause
bench
and
seeing
this
this
is
like
night
and
day
that
just
looks
absolutely
fantastic
and
I
like
the
fio
style
output,
I'm
just
curious
about,
like
some
of
the
findings.
You've
got
like
a
million
candle
power
flashlight
now
to
look
at
performance
with
the
with
the
side
bench.
What
were
some
of
the
things
that
surprised
you,
for
you
know
different
customer
deployments.
A
A
A
A
B
I'm,
just
thinking
about,
if
I
really
have
a
large
cluster
and
I
really
want
to
saturate
the
network
and
I
really
want
to
figure
out
what
is
really
the
next
button
like
I
will
hit,
so
I
have
to
take
it
into
account.
From
my
experience
when
I
did
this
benchmarking
test,
I
have
to
also
to
think
about
what
could
be
the
bottleneck
of
of
the
client.
A
Sidebench,
yes,
yeah,
absolutely,
and
that's
that's
exactly
the
process
that
we've
gone
through
of
like
okay.
We
have
this,
you
know
42u
42,
osds,
the
nvme
and
this
cross
connect
and
where
is
the
bottleneck
and
doing
it
over
and
over
again?
And
you
know
and
then
surprising
ourselves
like.
Oh,
you
didn't
think
about
this
or
whatever
right
so
yeah.
B
For
example,
entry,
if
you
generate
percent
three
objects,
the
test
and
so
on.
That's
also
sometimes
a
bottleneck
even
even
delays.
The
execution
of
your
program
as
well
I
mean
I.
I
really
did
this
on
soft
classes
with
I
think
next
to
three
thousand
overseas
wow
so,
and
this
really
20
50
clients
to
re-saturate
the
Clusters
and
it
was
not
possible
so
yeah.
A
You
yeah
we.
We
found
that
that
it
was
scary
how
many
drivers
we
would
need
to
properly
drive
a
large
OSD
cluster.
We
expect
we
almost
needed
as
many
drivers
to
as
we
as
we
needed,
OST
nodes
and
so
you'd
end
up
with
like
two
or
three
racks,
and
then
you
know,
half
of
it
is
drivers
and
half
of
its
OST
nodes.
So
yeah
you
need
a
lot
of
horsepower.
A
That's
the
whole
point.
The
whole
point
is
I
mean
I
ran
in
a
very
kind
of
caveman
way,
but
the
whole
point
is
to
have
loads
of
of
drivers
and
and
that's
the
kind
of
why
it's
a
demon
rather
than
a
command
line
tool,
is
that
you,
you
install
a
demon
on,
like
let's
say,
100
servers,
and
then
you
provide
those
100
service
to
SCI
bench,
the
the
command
line
tool,
and
it
will
it's
not
doing
the
chronic
tools
not
doing
anything.
A
A
It
doesn't
actually
have
many
dependencies
and
I
think
we've
packaged
it
for
Fedora
and
Debian
at
the
moment.
So
if
you
go
on
cybench.io,
there's
some
instructions
for
a
few
different
distributions
where
it's
packaged,
but
it's
not
an
upstream
package
upholsteries.
Yet
because
you
know
who
has
time
for
that,
yeah.
A
A
It
doesn't,
and
so
you
have
it's
something
you
really
have
to
be
aware
of,
which
is
why
we
removed
the
delete
thing
because
it
was
like
we
don't
want
to
worry
about
it.
We
want
to
try
and
remove
as
many
background
tasks
as
possible
there's.
Actually,
if
you
look
at
the
website,
there's
a
best
practices
page
of
how
to
run
side
bench
which
I
think
part
of
it
does
tell
you
like.
Please
don't
you
know
here's
some
things.
A
You
need
to
be
aware
of
before
running
The
Benchmark,
to
avoid
kicking
off
background
tasks
and
stuff,
but
then
sometimes
you
want
those
background
tasks
depending
on
what
you're
doing
so
trying
to
just
forget
about
it
isn't
is
never
going
to
be
an
approach.
That's
going
to
work.
You
want
to
think
about.
Okay
am
I
going
to
end
up
with
a
bunch
of
deletes,
and
that's
probably
what's
going
to
happen
in
real
life
right,
so
it's
yeah
so
trying
to
get
something
representative,
yeah,
yeah,
100,
cool,
okay.
Well,
thank
you
very
much.