►
Description
CephFS as a Scalable Filer - Rafael Lopez & Brett Milford, Monash University
Monash University's eResearch Centre adopted Ceph with the introduction of Openstack over 6 years ago and quickly grew to love Ceph's design, stability, flexibility and community. Ceph now serves as the backbone for not only Openstack storage, but also general online file and object storage for Researchers. This talk will dive into our Ceph journey, experiences and challenges and in particular present how we are making use of CephFS as an all purpose filer for our research community.
About Brett Milford
Research Devops Engineer, Monash University
Brett is an experienced cloud operator, working with Openstack for several years. After moving from Univerisity of Queensland to Monash, he has helped develop and innovate Monash's cloud environment.
About Rafael Lopez
Monash University
Research Devops Engineer
Rafael has developed and administered storage solutions for the past 6 years, working with various enterprise and open source technologies.
For the past couple of years he has been part of Monash University's eResearch centre, maintaining and developing the Ceph environment and other storage systems used by Researchers.
A
So
Brett's
going
to
talk
to
you
a
little
bit
about
what
we
do
in
general,
at
Monash
and
and
our
organisation,
and
also
go
through
run
down
throughout,
do
a
rundown
of
our
environment,
our
safe
environment,
and
how
we've
set
everything
up
and
I'll,
be
back
a
little
bit
later
to
deep
dive
into
some
performance
comparisons
and
look
at
it
a
little
bit
deeper
and
I.
Guess
what
we
want
you
to
take
away
from.
A
B
B
Thanks
so
I'll
give
you
a
brief
overview
of
University.
Firstly,
we're
a
relatively
large
University
in
Australia,
we're
part
of
what's
called
a
group
of
eight
University,
so
basically
they're
the
largest
grossing
research
and
teaching
universities
in
Australia.
We
have
approximately
70,
are
probably
greater
than
70,000
students
at
the
moment,
actually
greater
than
44,000
research
higher
degree
students,
their
PhD
students,
that
sort
of
thing
and
the
250
million
in
research
income
was
the
2016
figure,
but
I
think
it's
actually
substantially
more
now,
but
we're
largely
a
relatively
young
university.
B
Only
I
think
it
was
like
1950s,
surely
like
less
than
a
hundred
years
old.
So
major
disciplines
are
mostly
the
kind
of
more
technical
fields.
So
a
lot
of
medicine
and
pharmacy
research,
information
technology
and
business
disciplines
that
we
work
with
so
the
mana
actually
Research
Centre
is
like
a
specific,
I
suppose
organisation
within
the
universe.
That
is
focused
primarily
on
a
research
infrastructure.
So
our
goal
is
to
basically
provide
high
performance,
compute
and
scalable
storage
and
information
technology
services.
B
So
that
includes
collections
and
in
archival
services
for
research
data.
That's
correct,
that's
generated
in
Australia.
We
also
have
from
an
infrastructure
perspective,
which
is
interesting.
The
Monash
Erie
research
network,
which
is
a
hundred
gig
Ethernet
fabric
that
spends
data
centers
and
also
onto
our
campuses,
and
basically
allows
us
to
have
high
throughput
low
latency
applications,
just
like
just
about
throughout
the
university
and
across
across
our
research
infrastructure.
B
So
we're,
broadly
speaking,
our
infrastructure
is
sort
of
separated
into
two
sections.
The
cloud
cloud
basically
underpins
all
of
those
major
platform:
major
infrastructure
platforms.
I
just
told
you
about
so
it
underpins
massive.
It
underpins
CVL,
it's
also
the
research
cloud
itself,
and
this
is
kind
of
important
because
it's
probably
the
primary
way
well
one
of
the
primary
ways
which
we
engage
with
our
researchers.
B
We
currently
have
331
projects
running
on
that
cloud,
853
directs
users
so
that
those
people
who
are
logging
in
so
we
run
an
OpenStack
cloud
and
those
are
direct
users
that
are
actually
logging
in,
and
you
know,
building
VMs
and
doing
images
and
stuff
like
that.
We
also
have
about
a
petabyte
of
Ross
Swift
storage
and
to
petabyte
of
Ceph,
storage
and
I.
Suppose
it's
kind
of
interesting
to
note
that's
kind
of
where
our
journey
with
SEFs
started
and
that'll
kind
of
get
it
that
story
I'll
get
told
later
on.
B
B
It
has
about
400
active
NFS
connections,
800
active
SMB
connections.
We
also
have
a
hundred
350
terabytes
of
used
object,
storage
as
well
I.
Suppose.
The
key
thing
to
note
here
is
this:
infrastructure
is
used
to
service
both
a
HPC
clusters,
as
well
as
data
instrument,
data
ingestion
and
data
processing
pipelines,
as
well
as
just
used
as
a
general
purpose
file.
So
there's
a
high
degree
of
flexibility,
that's
sort
of
required
from
this
system.
B
So
why
don't
we
pick
SEF?
Basically,
as
I
sort
of
mentioned
just
earlier,
we
gained
a
lot
of
experience
from
using
SEF
with
OpenStack,
as
I
think
many
people
can
probably
attest
to
having
the
same
experience.
We
generally
liked
the
idea
of
not
having
the
vendor
lock-in
or
the
and
having
being
out
being
able
to
buy
our
own
hardware
and
have
the
open
source
software
on
top
also
the
flexibility
as
I
sort
of
noted
in
our
vast
infrastructure.
B
We
have
a
lot
of
diverse
requirements,
so
the
flexibility
of
SEF
allows
us
to
sort
of
support
those
unforeseen
circumstances
as
they
arise,
unforeseen
sort
of
requests
from
researchers
and
whatnot.
So,
for
example,
we've
been
able
to
use
the
object
storage
of
our
object
interface
to
our
SEF
cluster,
for
use
with
an
in-house
app
which
is
designed
to
ingest
data
off
instruments
and
then
stage
that
data
into
hi
performed
like
the
high
performance
compute
for
for
processing,
so
I'll
drill
down
a
bit
on
our
main
SEF
cluster,
just
that
on
its
own.
B
This
is
our
primary,
so
we
have
to
self
clusters
for
the
two
two
zones
of
our
cloud.
This
is
our
third
and
primary
SEF
cluster,
which
consists
of
around
a
hundred
servers
around
2000
OS
DS,
and
that
that
bare
metal
infrastructure
is
also
encapsulate.
S--,
the
control,
plane,
I,
suppose
the
Mons
MTS's
and
radars
gateway
services,
and
it
also
encapsulate
a
presentation
layout,
which
we
use
to
export
NFS
and
SMB.
B
So
the
need
for
file
based
storage-
basically
it
drills
down
to
this.
The
common
applications
that
our
researchers
use
and
require
will
need
often
need
file
based
storage.
Almost
zero
of
the
research
groups
in
Australia
have
the
manpower
or
capacity
for
application
development
to
be
able
to
develop
an
app
for
a
Doss
or
develop
an
app
for
object
or
develop
an
app
for
any
of
the
other
easier
to
use
interfaces
of
SEF
or
native
interfaces
of
Seth.
So
we're
required
to
buy
it
to
we're
required
to
provide
file
based
storage.
B
The
decision
to
go
with
Saif
came
around
a
time
when
we
had
a
large
number
of
of
essentially
like
physical
machines
that
were
just
doing
NFS
and
SMB
placed
throughout
the
place
throughout
the
university.
There
are
essentially
just
really
large
nodes
for
for
data
storage
at
the
time,
and
we
had
to
make
that
call
Seth
fests,
we
felt
wasn't
quiet
production-ready,
so
the
initial
solution
we
came
up
with
was
to
build
virtual
filers
with
Saif
and
RBD.
B
So
this
is
our
initial
attempt
at
providing
Saif
provided
story
providing
NFS
and
SMB
from
our
safe
cluster,
essentially
its
KVM,
with
our
biddies
presented
to
sorry
presented
into
the
cave
and
presented
into
the
virtual
machine,
they're
formatted
with
ZFS
and
exported
with
NFS
and
SMB.
As
you
can
see,
we
also
use
flash
storage
devices
from
the
hypervisor
to
boost
ZFS
performance.
We
have
authentication
services
provided
by
an
external
Active
Directory
and
we
perform
backups
of
this.
This
architecture
of
the
FS
on
the
RBD
made
is
made
using
TSM.
B
B
It's
been
easy
to
manage
shares
through
ZFS
datasets
and
the
compression
that
ZFS
has
provided
has
been
significant
bad
attributes
of
it.
It
hasn't
really
been
genuinely
scalable.
It's
not
really
H
a
there's
management
overhead
in
creating
creating
replicas
of
this
architecture
and
creating
our
biddies
setting
up
ZFS
and
all
that
sort
of
stuff,
and
it's
kind
of
hard
to
deal
with
large
allocations.
So
we're
talking
about
allocations
of
yeah,
suppose
fifty
hundred
two
hundred
terabytes
or
more.
B
So
SEPA
'this,
basically
we
settle
it
well.
We
wanted
to
use
this
because
sort
of
offered
a
lot
of
potential
improvements
on
our
current
setup,
basically
easier
to
use
and
manage
with
this
with
a
sort
of
single
file
space
easier
to
scale
the
backend,
we
can
just
add
more
set
of
s,
pools
or
OSDs
easier
to
scale
the
front-end.
We
can
just
add
more
NFS,
SMB
gateways,
no
block
devices
or
anything
like
that
to
deal
with
provides
flexibility,
so
we
can
plug
in
different
back-end
pools.
B
We
can
pin
directories
to
particular
MTS's
for
various
performance
profiles
and
in
the
future.
Potentially
later,
multiple
FS
is
per
cluster
yeah.
It
also
has
high
availability
characteristics
by
default.
There's
also,
potentially
performance
gains
to
be
from
from
using
the
native
SEF
mount
options,
and
it
also
has
application
to
parallel
file
system
applications
so
that
it
could
potentially
basically
be
used
directly
with
our
HPC
cluster
in
the
future.
B
So
this
is
what
this
architecture
looks
like
sort
of
roughly
working
from
the
left,
we
have
essentially
two
NFS
files:
utilizing
CFS
Ganesha
with
a
SEF
FS
al
back-end
set
for
sale,
back-end,
it's
not
H
a
so
clients
have
to
connect
directly
to
a
particular
server.
The
SMB
filers
are
slightly
different.
B
B
So
I'll
preface
this
by
saying
that
we
utilize
Red
Hat
Enterprise
SEF
for
all
of
our
storage.
It's
presented
a
couple
of
issues
in
terms
of
packaging,
basically
ctv
CT,
DB
and
Ganesha
were
not
easily
available
to
us.
We
had
to
build
Ganesha
from
source.
We
had
to
get
CT
TB
from
Santos,
repos
and
samba.
We
all
we
actually
get
directly
from
Red
Hat.
B
We
provide
native
service
access.
Sorry,
one
of
the
other
issues
we've
also
confronted
confronted
with
this
setup
is:
how
can
we
provides
native
CFS
access
whilst
and
14
forcing
quotas?
We
actually
developed,
developed
a
script,
to
monitor
quota
levels
and
to
a
low
dose
based
nut.
We
also
tested
forcibly
disconnecting
client
our
service
clients,
but
we
never
actually
implemented
that.
B
How
do
we
backup
a
large
file
else
file
system?
Basically,
we
just
try
to
paralyze
the
job
as
much
as
we
can
and
yeah
there's
a
couple
of
more
challenges
that
we
sort
of
hope
to
be
able
to
solve
with
this
FS
solution
in
the
future.
So
I'll
just
pass
over
to
Raph.
Now,
to
give
you
a
rundown
of
the
performances
of
this
setup.
A
Comparisons
that
we
that
we've
made
from
our
old
solution
and
our
new
surface
back
filers,
because
I
found
it
an
interesting
exercise
to
go
through
and
see
if
it
actually
stacks
up,
I
guess
I'll
just
glance
over
this
slide.
It's
just
details
on
on
the
on
our
environment.
Basically,
all
of
the
all
of
the
cluster
stuff
is
Red,
Hat
and
and
all
of
our
front
ends
and
that's
based
on
luminous,
which
is
12.2
date
on
the
client
side
with
the
client
I
set
up
was
actually
it's
actually.
A
A
typical
typical
client
that
we
would
have
like
an
actual
user
would
build
an
instance
in
the
cloud
I'm
using
Ubuntu,
so
that
I
did
exactly
the
same
thing.
I
built
the
Ubuntu
server
in
our
prime
cloud
for
these
tests,
all
the
mount
options
are
all
defaults
as
well,
so
for
these
tests,
so
I've
really
done
very
little
tuning.
It's
it's
almost
bog
standard
everything.
A
A
Then
I
want
to
look
at
specifically
at
some
native
surface
comparisons
and
then,
after
that,
I
want
to
look
at
SMB
and
NFS
again,
but
using
a
kernel
mounted
file
system
instead
of
using
the
SIF
for
sale,
for
example
or
safe
use.
And
finally,
the
the
last
series
of
tests
is
just
looking
at
it
straight
up.
Nfs
versus
safe
use
to
kind
of
think
about
whether
it's
actually
even
worth
using
safe
use
at
all.
Or
is
there
any
benefit
to
doing
that?.
A
Okay,
so
first
up,
we've
got
the
NFS
and
we've
got
virtual
file
versus
FS
right
bandwidth
in
FIO.
It's
clearly,
we've
got
a
win
for
surf
the
red
line.
There
is
force
FS
and
the
blue
line
is
for
virtual
file.
Is
the
bandwidth
is
almost
double.
I
should
also
point
out
actually
that
I'm
running
these
tests
off
the
same
data
pool.
So
these
are
the
same
OS
DS
that
are
supporting
these
two
solutions.
A
A
Now
we're
looking
at
latency
it's
a
little
bit
spiky
at
the
beginning
and
I
haven't
quite
worked
out
why
that
is
but
I
think
it's
something
to
do
with
the
NFS
client
itself,
and
the
reason
for
that
is
that
the
patterns
actually
the
same
for
both
the
virtual
file
and
surface
and
you'll
see
a
bit
later
on
the
when
I
use
this
and
be
that
it's
it's
actually
not
there,
but
in
terms
of
the
average
latency.
Anyway,
it's
a
it's
a
clear
win
for
surface
again.
It's
actually
better
in
both
read
and
write
latency.
A
The
read
performance
is
actually
a
clear:
we
enforce
FS,
it's
like
it's
significantly
better.
You
can
see
it's
over
200
mega.
Second,
better.
The
right.
The
right
of
the
hundred
gig
file
is
actually
about
the
same,
so
almost
150
mega.
Second,
so
so
far,
I
mean
it's
looking
pretty
healthy,
like
sever.
Fest
is
looking
quite
good,
so
this
is
actually
the
first.
This
is
the
first
test
where
the
virtual
file
was
actually
better,
and
this
is
a
I
guess.
It's
a
high.
It's
a
metadata
intensive
task.
A
Writing
lots
of
little
small
files
because
we're
we're
doing
an
ATAR
and
and
untie
so
the
blue.
The
blue
graph
is
actually
the
tiring
thousand
files
and
the
red
light.
The
red
block
is
untiring,
that
same
tart,
but
into
the
thousand
files
again
so
for
the
entire
test,
I
mean
it's
over
three
times
slower
for
a
surface,
and
the
virtual
filer
performs
pretty
well.
A
And
this
is
another
sort
of
real-world
tests
where
I've
pretty
much
stuck
a
hundred
thousand
files
in
a
directory
and
I
ran
an
LS
and
an
LS
dash
L.
You
can
see
that
the
again
the
surface
is
significantly
slower.
It's
actually
five
times
slower
for
the
LS
and
three
times
slower
for
the
LS
dash
L
on
a
hundred
thousand
files.
A
A
Having
said
that,
though,
for
the
read
bandwidth
surface
still
outperforms
the
virtual
file
solution
and
quite
significantly
so
I
mean
I,
think
it's
I
think
in
terms
of
averages.
I
don't
have
the
number
there,
but
it's
I
think
it's
about
double
and
I
have
no
idea
what
that
drop-off
is
right.
At
the
end.
A
So,
moving
on
to
the
lights
in
graph,
this
is
what
I
mentioned
earlier,
like
there's
that
spiky
period
sort
of
for
the
first
ten
seconds
isn't
there
for
the
when
I'd
mounted
with
SMB,
but
anyway,
there's
no
clear.
We
know
here
in
this
test.
For
writing.
Surface
is
a
little
bit
slower
than
the
virtual
filer
and
for
reading
it's
a
little
bit
faster,
but
in
any
case
the
latencies
they're.
You
know
they're
quite
healthy.
There
there's
nothing
there
above
five
milliseconds
so
for
network
file
storage
that
that's
certainly
plenty
for
us.
A
So
this
is
this:
is
the
untapped
our
entire
test
again
over
SMB,
with
a
virtual
with
the
virtual
filer,
the
ZFS
virtual
file?
Oh,
it's
it's
actually
quite
good
and
over
surface
it's
it's
actually
pretty
woeful.
So
what
we're
looking
at
here
is
basically
two
seconds
versus
104
seconds:
virtual
file
versus
the
CFS
and
for
untiring,
so
writing
lots
of
little
files.
We
have
just
under
5
seconds
versus
about
220
seconds,
so
significantly
significantly
slower
similar
story
for
the
LS
test.
A
We
we
can
see
like
there's
a
huge
discrepancy
between
the
virtual
file
and
CFS
I,
actually
only
used
10,000
files
in
this
test
instead
of
a
hundred
thousand,
because
Sam
is
a
little
bit
bad
at
doing
this
anyway,
but
you
can
see
like
there's,
there's
a
huge
difference
and
I
guess.
Look
at
this
point.
I
was
I
started
to
think
a
lot
about
like
what's
going
on
with
the
metadata
of
CFS
and
I,
wanted
to
investigate
that
a
little
more
so
for
the
next
series
of
tests.
A
A
So
I
didn't
I,
didn't
bother
with
the
fao
or
streaming
test,
because
we've
already
seen
that
it's
it's
already
quite
good,
so
we
went
straight
to
the
entire
test,
tyrant
artist.
So,
on
the
left
hand,
side
you
can
see,
ZFS
is
pretty
healthy
fractions
of
a
second
to
complete
those
operations.
Colonel
mounted
is
not
too
bad,
certainly
for
tiring,
it's
very
fast
and
for
untiring,
it's
just
under
two
seconds
for
a
thousand
files.
A
The
LS
test
100,000
files
in
a
directory,
so
this
one's
pretty
interesting,
so
the
virtual
file
R,
is
actually
you
could
argue
that
it's
a
little
bit
slower
than
the
kernel
mounted
surface.
It
is
actually
faster
for
the
LS,
but
for
the
LS
dash
L,
which
I
believe
actually
stats
every
single
file
in
the
directory.
It
is
a
little
bit
slower
than
the
kernel.
A
Mount
fuse
again
is
really
struggling
with
LS
dash
L,
stating
all
those
files
now
one
thing
I
haven't
mentioned
with
these
tests
is
like
the
effects
of
caching,
but
particularly
for
this
test.
I
made
a
point
to
try
to
to
try
to
drop
all
the
caches,
I
guess
past
this
F
cluster
I
couldn't
really
control
the
caches
on
the
Ceph
cluster,
because
it's
our
production
cluster,
so
I
guess
I
didn't
want
to
impact
like
our
production,
workloads
and
so
forth.
So
all
of
the
OSD
caching
is
there
the
metadata
server.
A
A
Okay,
so
now
what
I'm
going
to
look
at
is
doing
the
same
set
of
the
same
real-world
metadata
type
tests,
but
this
time
instead
of
using
our
production
setup.
Well,
the
production
setup
is
actually
on
the
right.
So
we
have
the
fuse
back,
SMB
mount
and
in
the
middle
we
actually
have
the
same
thing,
but
we
have
the
kernel
mounted
file
system,
that's
being
exported,
so.
B
A
That
makes
sense,
but
it's
basically
doing
a
comparison
of
SMB
/
CFS
using
kernel
and
fuse,
and
it
looks
very
similar
to
that
like
the
native
bounce
anyway.
So
the
virtual
file
is
the
the
fastest.
That's
with
the
RB
days
and
set
FS
on
top
kennel
mounted
is,
is
a
I
guess
acceptable,
which
you
forget
that
the
whole
thing-
and
it's
significantly
slower
doing
it
over
so
using
fuse.
A
I
think
I'm,
probably
gonna,
run
out
of
time
if
I
don't
sort
of
if
I
go
through
all
the
figures,
so
you
can
have
a
look
at
that
on
the
slides
later
doing.
The
LS
test,
with
10,000
files,
similar
sort
of
story,
but
actually
the
the
kernel
mounted
SMB
export
is.
It
comes
really
close
to
the
virtual
filer.
So
that's
nice
to
see
and
again
surf
users
is
really
struggling.
A
So
the
same
same
sort
of
story
with
NFS,
but
it's
not
quite
as
pronounced.
So
when
we
export
NFS
using
a
kernel
mounted
surface,
it
is
better
than
using
the
fossil
and
I
did
use
Ganesha
for
both
of
those
exports,
one
using
the
VFS
standard
for
Sol
and
the
safe
for
sale
for
the
that's
the
one
on
them
on
the
far
right,
so
using
our
biddies
and
virtual
file
Oh
still
it's
still
the
fastest
for
these
kind
of
metadata
intensive
tasks.
A
A
A
On
NFS
and
just
use
NFS
so
for
this,
for
this
test
series
actually
did
do
the
DD
test
and
you
can
see
that
there's
I
mean
on
this.
If
you
look
at
this
slide
alone
and
there's
there's
no
reason
to
use
confused
because
exporting
using
using
the
the
NFS
for
sale,
you
get
better
performance,
both
read
and
write.
A
A
And
I
got
that
so,
which
was
like
three
and
a
half
seconds
so
in
terms
of
a
what's
the
reason
for
that
I
have
to
the
only
way
I
can
explain.
It
is
that
it
must
have
something
to
do
with
the
current
workload.
That's
happening
on
our
cluster
at
the
time
of
the
test,
so
the
these
tests
were
all
done
on
our
production
cluster
there's,
depending
on
the
time
of
day
there
could
be
workloads
from
clients
that
could
be
workloads
from
backup.
A
Overall,
it
you
know
obviously
fuses
is
was
better
on
in
that
test
and
finally,
just
doing
the
LS
test
again,
the
facade
perform
better
in
this
test
and
I've
done.
I've
done
these
solicitous,
like
dozens
of
times
now
and
I,
get
very
similar
results
for
fuse
doing
an
LS
dash
L
on
on
a
directory
that
many
files
in
it.
B
A
Based
network
storage,
we
think
it
makes
a
good
base
for
for
general
purpose,
network
file,
storage
and
it's
it
does
look
like
it's
good
enough
to
actually
replace
doing
stuff
with
our
biddies,
especially
for
streaming
like
the
streaming
tests
are
very
healthy.
They
actually
will
outperform
using
our
BD
over
like
a
virtual
filer
stuff
stuff.
Where
you
use.
A
You
probably
want
to
look
at
mounting
native
kernel
if
you
can
but
again
probably
take
a
lot
of
this
with
a
grain
of
salt,
because
you
know
those
are
less
tests,
take
a
long
time
and
so
forth.
But
when
you
start
putting
caching
into
the
into
the
equation
as
well
like
I
mean
it
certainly
makes
it
much
more
acceptable
and
changes
changes.
Everything
and
that's
that's
more
a
real-world
representation
of
what
you
do
like
you,
wouldn't
be
dropping
caches.
You
know
all
the
time.
A
A
A
So
just
a
few
other
issues,
we've
noticed
with
surface
and
again
take
this
with
a
grain
of
salt
because
we're
using
a
specific
version-
that's
probably
a
little
bit
older
that
one
you
get
upstream,
but
we
have
run
into
some
like
red,
dirt,
loops
and
case
chunk
loops
with
NFS
Ganesha,
and
have
seen
quite
a
lot
of
commits
upstream
that
fixed
these
or
they
labeled
that
they
fix
issues
with
these
things.
So
I
think
that
a
lot
of
this
stuff
is
probably
solved
upstream
already.
A
We
do.
We
have
noticed
some
random
times
where
the
metadata
becomes
unresponsive,
long
at
least
times
even
on
small
directories
and
I.
Think,
that's
probably
also
to
do
with
these
bugs
that
we've
seen
in
Ganesha
if
you've
got
applications,
or
you
know,
scenarios
where
you
need
to
use
like
a
hundred
thousand
files.
A
A
A
Smb
client
I
actually
just
used
mount
safes,
yeah
so
again,
grain
of
salt
right.
So,
if
you're
using
SMB,
you're,
typically
using
for
Windows.
In
our
case
it
isn't
because
a
lot
of
our
researchers
they
use
my
own
thoughts.
It's
because
we
don't
do
it's
it's
easier
to
to
manage
on
their
side
like
we
don't
have
to
add
a
piece
to
their
export
and
so
forth,
so
they
can
mount
it
wherever
they
like.
That's
why
they
use
it
even
on
Linux.
A
So
the
question
is:
why
didn't
I
use
direct
IO?
The
reason
for
that
is
what
I
mentioned
earlier,
that
I
don't
I
like
to
test
things
a
little
bit
closer
to
what
they
are
in
real
in
you
know,
in
the
real
scenario
where
99%
of
the
time
you
won't
use
direct
IO
anyway,
so
I
wanted
to
kind
of
see
how
it
performs.
You
know
a
real
type
of
environment.
A
So
the
question
was:
what's
the
virtualization
solution,
we're
building,
we
build
our
VMs
on
and
that's
KVM
and
that's
just
the
KVM,
that's
shipped
with
Red
Hat
7
yep.
We
are
thinking
about
OpenStack
OpenStack,
a
fiying,
some
of
our
front-end
services,
but
I'm
not
sure
how
we'll
go
with
that.
I
mean
there's
other
stuff,
like
you'll,
see
talks
at
this
conference
on
kubernetes
like
doing
NFS
clusters
with
kubernetes
and
so
forth,
so
that
might
be
a
better
a
better
way
to
go
anyway.