►
Description
Exploring the Performance Limits of CephFS in Nautilus - Manoj Pillai, Red Hat
Ceph, with its broad adoption and its integration with various platforms, is expected to handle a diverse set of workloads well. Storage workloads in the real world come in many different flavors: caching-friendly and not, data and metadata intensive, latency-sensitive and throughput-oriented, single and multi-threaded, mmap-based, aio-based.
In this talk, Manoj Pillai will present performance results and analysis for CephFS in the Nautilus release using a broad spectrum of tests covering the above cases. The evaluation will include workloads that distributed file systems generally have trouble with, and will provide results from comparable solutions where appropriate. The goal is to establish the current state of performance of CephFS, which should be useful to users, as well as to developers working to enhance CephFS performance.
About Manoj Pillai
Senior Principal Software Engineer, Red Hat
Manoj Pillai is part of the Performance and Scale Engineering Group at Red Hat. His focus is on storage performance, particularly distributed storage systems. He has presented his work at a number of conferences including Vault, Open Source Summit and FOSDEM.
A
Thanks
for
coming,
I
think
they
saved
the
best
for
last.
Just
joking
thanks
for
staying
late,
my
name
is
Manoj
Pillai
I
work
for
the
Red
Hat
performance
engineering
team
and
the
talk
is
on
exploring
the
performance
limits
of
surface
in
Nautilus
right
and
just
just
as
a
background.
Until
recently,
I
was
working
with
the
other
Red
Hat
Software
Defined
storage
project
cluster
moved
pretty
much
full-time
to
safe
recently
and
in
the
process.
Something
that
I
wanted
to
do
was
some
of
the
performance
issues
that
we
had
faced
in
the
in
the
last
few
years.
A
Some
of
the
approaches
that
work
well
in
sort
of
identifying
bottlenecks,
and
things
like
that
as
the
first
step
moving
to
say
if
I
wanted
to
apply
some
of
those
here
and
see
what
comes
out
and
sort
of
engage
with
the
community,
starting
with
that
right.
So
so,
for
instance,
last
year
at
FOSDEM
we
did
a
talk
on
optimizing
cluster
for
nvme
storage
and
and
you've
heard
about
crimson
and
sea
star
and
and
some
of
those
problems
are
there
and
self
as
well
and
and
some
of
those
issues
in.
A
In
that
context,
we
saw
that
it
was
one
of
the
big
bottlenecks
that
were
not
solved
was
the
RPC
layer
bottleneck,
so
the
ability
of
the
RPC
layer
to
keep
up
with
nvme
right,
so
the
nvme
kind
of
levels
of
performance?
That
was
one
issue.
One
of
the
other
things
that
users
frequently
report
when
working
with
distributed
file
systems
is
not
so
much
the
you
know,
pumping
of
I
ops
into
the
system.
It's
it's
more
around
everyday
admin
activities
like
you
know.
A
So
you
know
III
wanted
to
sort
of
look
at
some
of
those
things,
and
basically
this
is
more
of
a
sort
of
what
are
the
areas
of
improvement
kind
of
talks.
So
it's
a
little
bit
more
developer,
centric,
not
so
much
of
realistic
user
centric
scenarios
and
it
helps
to
identify
the
bottlenecks.
I
have
seen
if
you
are
working
with
simpler
configurations
right.
A
So
if
you
have
a
simple
configuration
you
know
so
one
way
of
looking
at
the
efficiency
of
a
system
is
of
a
software-defined
storage
system
is
what
is
the
hardware
capable
of
delivering
versus?
What
is
the
software
running
on
top
of
that?
Actually
delivering
to
you
right
so
and
those
kinds
of
back-of-the-envelope
calculations
are
much
easier
to
do
if
you
have,
if
you're
working
with
simpler
configuration.
A
So
if
you,
if
you
want
to
focus
on
the
software
limits,
you
want
to
run
on
hardware,
that
is,
that
is
high-end
like
so,
for
instance,
if
you
are
running
on
spinners
and
you
know,
a
spinner
is
limited
to
150
I
offs
or
something,
and
so
you
will
not
detect
problems
that
are
there
in
your
software
layer
when
you're
doing
that.
So
essentially
we
are
trying
to
look
at
a
lot
of
what
we
will
do,
not
all
of
it.
A
So
we
have
F
I/o,
which
does
large
file
sequential
and
random
I/o
testing
and
what
you
can
get
out
of
that.
This
actually
does
not
stress
the
metadata
part
of
cephus.
So
much,
but
it
is
whatever
you
find
here,
are
probably
common
across
other
protocols
in
in
SEF
the
block,
and
so
there
is
a
small
file
tool.
Fio
is
good
for
large
file,
sequential
I/o,
I
ops,
that
kind
of
testing.
A
You
know
data
sets
and
the
performance
that
we
that
we
see
there,
and
there
is
something
that
I
was
developing
as
part
of
this
exercise
dimmer,
which
is
just
a
harness
for
running
these
benchmarks
and
when
they
are
running
collecting
all
the
stats
and
and
the
system
information,
and
all
that,
if
you
want
more
details
on
how
some
of
you
want
to
browse
through
more
details
on
how
some
of
these
tests
are
written
and
things
like
that,
you
can
refer
to
that.
The
link
is
over
there,
and
so,
like
I,
said.
A
As
far
as
the
hardware
configuration
is
concerned,
I'm
focusing
on
you
know
fast
nvme
drives
25,
Gigabit,
Ethernet
and
fairly
high-end
high-end
CPUs
and
the
the
hardness
that
I
was
talking
about
it's
basically,
mostly
self
independent.
You
know
it
works
on
it
on
a
standard
mount
and
by
substitute,
if
you
unmount,
safe,
mount
create
a
XFS
filesystem
on
the
only
on
the
local
nvme
drive
mounted
and
run
the
same
test.
You
get
a
sense
of
you
can
actually
measure
instead
of
relying
on
manufacturer
specs
what
the
device,
what
the
hardware
is
actually
capable
of
right.
A
So
when
you
run
those
kinds
of
tests
on
the
nvme
hardware
that
we
have,
a
single
drive
is
capable
of
on
a
4k
random
right.
You
know:
400
ki
ops
doubled
that
on
4k
random
read.
So
there
is
a
lot
of
performance
potential
here
right,
so
in
sequential
right
in
the
slightly
north
of
2
gigabytes
per
second
and
sequential,
read
slightly
about
5
gigabytes
per.
A
Second,
is
what
I
measured
on
some
of
these
same
tests
that
I'm
running
here
right-
and
you
know
this
is
just
just
to
give
a
sense-
I'm
not
going
to
spend
time
on
this
or
the
for
the
f
io
testing.
What
are
the
job
files
that
we
that
I'm,
using
it's
basically
sync
io
engine
for
the
sequential,
so
these
tests
have
got
a
sequential
right
phase
which
basically
creates
the
files,
and
then
you
have
the
sequential
read
and
the
random
breed
and
the
random
write
which
are
operating
on
those
files
and
and
files.
A
Different
jobs
are
connected
by
the
file
name
format,
so
that
the
one
next
test
reads
from
the
files
that
are
there
previously
right
and
for
the
random
I/o
case.
You
have
direct
I/o
based
tests
based
on
lis,
bio
block,
size
of
4k,
like
I
said,
and
for
the
small
file
data
sets.
You
have
the
initial
phase.
A
The
data
set
is
created
with
the
small
file
create
file,
create
command
operation,
which
is
multi-threaded
right,
so
you
can
increase
the
number
of
threads,
and
even
when
you
have
high
latency
in
your
in
your
configuration,
you
can
still
get
good
throughput
by
increasing
the
number
of
thread,
the
number
of
threads
right.
So
once
that
is
done,
the
the
read
faces
again
from
the
small
file
command
itself.
It's
multi-threaded
again
so,
but
but
some
of
the
commands
that
I
run
after
that.
A
A
These
commands
are
single-threaded,
which
means
that
you
know
the
performance
or
the
time
taken
to
complete
is
primarily
dictated
by
latency
and
distributed
file.
Systems
have
got
you
know.
The
network
latency
is
is
inherent
in
the
file
system,
so
because
of
that
they
tend
to
run
slower
on
distributed
file
systems.
By
the
same
time,
when
you
are
talking
about
scale
out
storage
systems,
people
tend
to
take
put
a
lot
of
data
into
these
into
these
solutions.
A
Right
just
because
of
the
nature
of
the
of
the
solution,
you
you
want
to
scale
out
to
beyond
what
a
single
file
system,
a
single
local
file
system
is
capable
of
doing
and,
and
the
time
taken
for
most
of
these
depends
on
the
size
of
the
data
set
is
itself
right,
so
both
of
those
factors
and
like
I
said
in
the
past,
we
have
seen
some
users
complaining
and
about
about
performance
of
this
and
I
wanted
to
get
a
sense
of.
What
do
you
expect
from
self
FS
in
this?
For
these
kind
of
tests.
A
So
a
very
simple
test
here,
where
you
have
you
know
three
surf
FS
three
SEF
nodes,
one
nvme
on
each
replica,
three
with
a
single
client
mount
not
co-located
with
the
SEF
nodes.
This
is
the
SEF
kernel,
client
by
the
way
and
I'm
looking
at
here
what
you
can
get
from
that
single
mount.
So
so,
if
you
have
a
single
surface
mount,
what
is
the
sequential
I/o
performance
that
I
can
get
from
it
right?
So
what
is
the
upper
limit?
A
And
how
does
that
compare
to
what
the
hardware
is
capable
of
providing
and
and
what
maybe
not
an
apples
to
apples
comparison,
but
some
other
distributed
file
system
is
capable
of
providing
right,
so
the
blue
bar
is
there
is
what
you
get
out
of
surface,
which
is
slightly
above
what
you
would
get
with
a
10,
Gigabit,
Ethernet
or
or
in
that
range
right.
So
if
you,
if
you
run
this
or
with
10
Gigabit
Ethernet,
you
would
believe
that
surface
is
basically
or
so.
A
In
that
case,
surface
would
probably
you
know,
give
you
what
the
hardware
is
capable
of
giving,
because
that's
that's
what
a
10
gig
Ethernet
is
giving,
but
in
this
case,
since
we
are
running
on
25,
you
got
internet.
You
can
see
that
there's
a
gap
between
what
you
could
provide
versus
what
you,
what
you're
get,
what
the
hardware
could
provide
versus
what
you
are
getting
right
and
again
when
I
say
what
the
hardware
can
deliver.
A
It's
difficult,
obviously,
for
a
distributed
file
system
to
match
latency
of
an
nvme
try
right,
but
potentially
at
least
things
like
AI,
ops
and
throughput,
which
can
increase
with
by
just
by
pumping
more
load
into
the
system
right
so
I'm,
trying
to
see
whether
you
can
match
that
or
not
so
with
some
tuning.
So
so
this
is
with
the
default
configuration
with
some
tuning.
A
If
you,
if
you
increase
cephus,
has
got
read
a
head
size
parameter,
which
is
by
default,
set
to
8
megabytes,
and
if
you
are
really
aggressive
with
that,
you
can
basically
get
the
read
bar
to
match
or
slightly
exceed
what
the
kernel
NFS
is
providing.
So,
in
the
in
the
case,
this
is
not
an
apples
to
apples
comparison
in
the
sense
that,
in
the
case
of
kernel
NFS,
there
is
again
it's
going
over
the
network.
A
We
are
getting
somewhat
more
than
half
of
that.
Similarly,
on
the
read
test,
because
it's
the
client
is,
is
separate.
Basically,
the
client
network
is
25,
Gigabit
Ethernet,
so
that
limits
the
performance
that
you
can
get
from
the
system.
So
it's
about
upward
of
2.5
gigabytes
per
second,
is
what
you
could
potentially
get
close
to.
3
gigabytes
per
second-
and
this
is
what
we
are
actually
getting.
A
A
So
slightly
different
test,
so
so
clearly,
so
in
this
case
you
know
you're
not
able
to
deliver
what
the
hardware
is
capable
of
delivering,
but
so
I
want
to
get
a
sense
of
what
whether
we
can
get
a
sense
of
the
bottleneck
is
on
the
client
side
or
whether
it's
on
the
on
the
SEF
node
side
right,
so
variation
of
the
test
with
one
two
and
three
clients.
So
in
this
case,
I
don't
have
enough
systems
to
do
put
all
the
clients
on
separate
or
no
not
co-located,
with
the
SEF
nodes.
A
So
these
mounts
are
actually
co-located
on
the
SEF
nodes,
so
we
are
varying
the
number
of
clients
and
also
going
from
three
SEF
nodes
to
six
F
nodes
or
three
OSD
to
600
s,
trees
right
and
some
of
these.
These
tests,
I
am
seeing
that
there
is
a
lot
of
run
to
run
variation
in
these.
So
on
the
later
tests,
I
sort
of
I
started
putting
the
ranges
that
we
are
observed
in
some
of
these.
A
In
some
of
these
cases,
you
have
a
large
large
variation
that
you
can
see
right
so,
but
basically,
on
the
sequential
write
test
in
a
number
of
scenarios
you
know
cephus
is
able
to
deliver
about
3/4
of
what
the
hardware
is
capable
of
delivering
up
to
75%
roughly
and
you
know
which
bit.
So,
if
you
look
at
the
one
client,
six
SEF
nodes
again
with
more
with
with
some
more
aggressive,
more
number
of
jobs.
A
So
there
appears
to
be
a
purr
on
you
know
there
is.
There
is
some
limitation
on
the
client
side
as
well,
and
there
is
someone
D
on
the
OST
side
as
well
and
and
on
the
sequential
read
test,
similarly
I'm
varying
the
number
of
clients,
winning
the
number
of
OS
DS
and
and
seeing
the
what
I
can
get
and
on
the
first
row
at
least,
and
seeing
really
good
scalability
as
you
go
from
one
client
to
three
clients,
it's
almost
linear,
scalability
right.
A
So
so
that
means
that
you
know
so
at
least
partly
the
bottleneck
is
on
the
client
side,
probably
on
on
the
how
much
the
client
can
pump
I
over
the
network
and-
and
we
have
seen
a
number
of
talks
today
about
some
of
these
bottlenecks
and
what
the
developers
are
doing
to
fix
this
again,
the
picture
is
a
little
bit
muddled
because
of
so
much
of
their
of
the
run
to
run
variation.
But
basically
you
can
get
a
lot
of
I
ops.
It's
is
on.
A
A
The
random
right
performance
is
the
one
that,
where
I
think
the
gap
shows
up
more
most
in
terms
of
so
for,
for
instance,
from
a
three
OS
d
group
I
am
able
to
get
about
fourteen
to
fifteen
ki
ops
max
right
compared
to
work
to
400k
that
each
OS
D
is
capable
of
delivering.
So
these
are
bottlenecks
in
the
especially
the
AI
ops
bottlenecks.
A
You
must
have
heard
multiple
talks
where
people
are
aware
of
these
bottlenecks
and
and
and
they're
working
on
that
and
like
I
said,
these
are
not
tests
that
stress
the
surface
path,
specifically
right,
but
some
of
these
small
file
tests
they
are
they,
they
stress
the
surface,
metadata
paths
more.
So
it's
a
combination
of
surface
metadata
and
end
data
paths
and
what
we
are
seeing
here
so
again.
So
in
this
comparison
again
it's
difficult
to
get
an
apples
to
apples
comparison,
because
these
solutions
are
quite
different,
but
I
have
included
some
of
these
results.
A
Just
so
you
get
a
sense
of
you
know
what
you
know.
Just
a
ballpark
figure
of
you
know
is
it
doing
well?
Is
it
not
doing
well
and
in
this
particular
graph,
I've
normalized
the
results
so
that
you
get
a
sense
of
how
these
solutions
are
doing
relative
to
each
other
and
and
basically
it's
F
F
F
S
on
a
single
mount
versus
called
NFS
version,
3
version
4
and
X
F
s
local
file
system?
So
you
would
expect
that
x,
FS
local
file
system
will
outperform
by
a
lot
and
that's
okay.
A
A
A
The
concern
for
cephus
might
be
in
the
in
the
in
the
create
and
in
the
RM
commands
right.
So
sorry,
those
those
are
the
two
where,
like
I
said
some
partly
what
I'm
trying
to
do
is
trying
to
figure
out
where
to
focus
or
some
of
the
come
up
with
some
of
the
focus
areas
for
driving
performance
improvements,
and
those
are
two
things
that
look
like
you
know
where
you
see
a
big
difference
between
the
solutions.
A
A
Okay,
sometimes
even
if
it's
a
high
latency,
you
can
increase
the
number
of
or
the
degree
of
concurrency
in
the
workload
and
still
get
and
still
see
whether
you
can
get
good
performance
or
good
throughput
or
not
right,
and
what
happens
in
this
case
when
you,
when
you
do
that
on
safe
Affairs,
is
that
you
seem
to
hit
a
plateau
fairly
quickly
right.
So
in
this
case
we
are
going
with
so
here
again,
I'm
switched
to
throughput
on
the
y-axis
instead
of
time.
A
So
higher
is
better
in
this
case,
so
we
are
starting
with
the
four
threads
small
file
create
command.
That
was
there
in
the
previous
graph
and
what
I'm
looking
at
is,
as
you
increase
the
number
of
threads
right,
so
how
much
you
can
go
in
terms
of
the
megabytes
per
second
that
you
can
get
here
right
and
you
know
with
read:
you
can
get
about
five
hundred
some-odd
megabytes
per
second
on
a
single
mount
before
you
hit
some
kind
of
a
limit
on
right.
A
It's
much
lower,
it's
about
128
megabytes
per
second
or
125
to
130
megabytes
per
second.
So
the
question
is:
what
is
it
that
is
limiting
the
performance
here
right?
So
if
you
take
a
look
at
sort
of
some
of
the
stats
that
come
in
as
part
of
these
runs,
if
you
look
at
the
address,
F
node
a
proc
view,
so
this
is
with
one
MDS
right
so
and
the
MDS
is
co-located
with
the
Ceph
OSD
node.
A
So
if
you
see
on
that
particular
node,
you
see
that
both
of
them
seem
to
be
working
equally
hard
right.
So
the
Ceph
OSD
%,
HCP
utilization
and
the
70s
but
percentage
utilization
is,
is
almost
the
same
on
the
create
phase
right.
But
when
you
look
at
a
thread
level
view
of
what
is
happening,
there
doesn't
seem
to
be
an
obvious
bottleneck.
As
far
as
you
know,
one
thread
is,
is
maxed
out
or
anything
like
that.
A
On
the
read
on
the
small
file
read
phase,
it
seems
to
be
mostly
stressing
this
F
MDF
MDS
and
again
on
the
safe
node
on
the
thread
level
view
of
what
is
happening
there.
There
isn't
any
obvious
saturation
of
any
of
the
threads
and
and
in
some
of
these
threads,
particularly
on
the
random
I/o
workload
that
I
ran
DeMayo
tests.
A
The
messenger
thread
I
have
seen,
go
up
to
the
high
90s
right,
so
that
is
probably
one
of
the
contributing
factors
for
the
for
the
performance
issues
and
also
the
the
load
distribution
between
the
messenger
threads
was
was
not
either
right.
So,
as
the
obvious
thing
to
try
here
is
to
increase
the
number
of
MDS
and
see
what
that
has.
A
What,
if
impact
that
has-
and
in
this
case,
in
going
from
one
MDS
to
two
MDS,
resulted
in
immediate
improvement
in
performance,
I
saw
almost
double
the
throughput,
with
two
active
MDS
as
compared
to
one
but
and
and
the
tests
were
showing
a
little
bit
of
instability,
and
that
has
to
do
with
these
the
tool
that
I'm
using,
but
it's
definitely
the
MDS.
That
was
the
bottleneck
in
the
earlier
test
right,
but
that's
still
a
concern
here,
because
if
you
look
at
the
maximum
throughput,
you
can
get
from
a
single
mount
or
with
one
MDS.
A
It's
like
I,
said
it's
about
125
to
130
megabytes
per
second
right.
So
if
you
wanted
one
gigabytes
per
second
of
small
file
create
throughput,
you
would
need
to
have
if,
if
the
slide
that
I
just
showed
you
about
multiple
active
MDS
that
that
holds
true,
you
would
still
need
to
have
eight
MDS
s,
which
seems
to
be
a
lot.
So
this
is.
This
is
not
just
about
the
the
way
you
configure
safe,
but
we
probably
something
that
we
need
to
look
at
there.
So
maybe
there's
some
tuning
that
I'm,
not
aware
of
that.
A
So
again,
in
the
previous
project,
which
was
a
fuse
based
file
system,
we
are
running
into
issues
like
when
you,
when
you
ran
pro
one
you
when
you
ran
workloads
like
Postgres
PG
bench
right,
so
there
were
problems
with
the
solution
not
being
able
to
reuse
the
cache
effectively
right.
So,
for
instance,
the
fuse
does
some
things
like
auto
invalidation
of
caches
and
and
because
of
that,
the
performance
on
some
of
those
workloads
we're
not
that
good.
A
So
one
of
the
things
that
I
wanted
to
do,
as
part
of
that
is-
and
probably
others
have
done-
this
as
well-
is
look
at
the
caching
effectiveness
of
the
solution
right.
So,
for
instance,
if
you
are
doing
reads
and
writes,
is
the
is
the
client
able
to
effectively
cache
the
data
or
not
or
is
it
invalidating
it
in
some
situations?
And
things
like
that
and
on
both
the
kernel
and
the
fused
client
I
was
seeing
that
that
is
not
the
case.
So
there
is
not
a
whole
lot
to
report
there.
A
I
mean
I
I,
don't
see
any
problems,
the
ffs
fuse
is
anybody
using
surface
views
for
any
actual
performance
kind
of
use
case
show
of
hands,
know
so
yeah.
The
caching
effectiveness
is
good,
but
there
are
a
bunch
of
on
pretty
much
all
of
these
workloads.
What
surface
fuels
could
deliver
was
was
fairly
poor
in
terms
of
performance,
especially
compared
to
the
safe
kernel.
Client.
A
There
is
another
investigation
that
is
under
progress.
I
don't
have
that
completely
wrapped
up.
Basically,
we
were
talking
to
the
chemo
guys
earlier
in
the
context
of
one
of
the
project,
and
they
have
one
chemo
thread:
doing
lib,
a
IO
and
basically
submitting
a
IOT's,
and
the
argument
was
that
since
it's
Lib
a
IO,
you
don't
need
to
have
more
threads
right
and-
and
we
saw
one
other
case
in
which
that
ran
into
a
problem.
A
You
know
we
were
seeing
that
that
particular
mode
or
oddly
by
IO,
maybe
the
implementation
of
Libya-
was
not
efficient
enough,
that
that
was
not
the
case.
So
if
you
had,
if
you're
doing
one
thread,
doing
all
the
submissions
versus
multiple
threads,
there
was
a
big
performance
difference
and
in
the
case
of
surface
I,
did
see
a
drop
when
you
have
a
single
thread.
Doing
lib
AIO
compared
to
multiple
threads,
but
I
need
to
get
some
of
the
I
I
need
to
dig
a
little
bit
more
before
I
wrap
that
up.
A
A
I
would
probably
want
to
focus
on
the
MDS
and
and
its
scalability
in
the
performance
limits
that
you
can
get
from
MDS
and
whether
there
are
any
potential
tunings
that
we
can
investigate
or
potential
fixes
that
we
can
look
at
to
basically
be
able
to
increase
the
the
throughput
data
that
we
can
get
from
a
single
MDS
on
some
of
these
workloads.
So
that
would
be
one
of
the
focus
areas
going
into
going
forward.
Questions.