►
From YouTube: Affordable NVMe Performance on Ceph & Ceph on NVMe - True... - Wido den Hollander & Piotr Dalek
Description
Affordable NVMe Performance on Ceph & Ceph on NVMe - True, Unbiased Story to Fast Ceph - Wido den Hollander, 42on & Piotr Dalek, OVH
About Wido den Hollander
42on
Ceph Trainer and Consultant
Netherlands
Websitehttps://42on.com/
Wido den Hollander has been a part of the Ceph Community since 2010. In 2012 he started his company 42on providing Ceph professional services. Since 2012 he works as a full-time Ceph Trainer and Consultant helping organizations with designing, implementing and running Ceph. Bulk storage, high performance, cheap, expensive, he has seen all these variants of Ceph clusters. In these years he has also contributed various patches and features to Ceph and the eco-system surrounding it.
About Piotr Dalek
OVH
Software Engineer
Software engineer with primary focus on systems performance and efficiency. For last two years working at OVH as a software engineer, assisting the storage team and providing Ceph expertise and patches to the Ceph community. Previously working as a software engineer in Fujitsu, working on Ceph-based Eternus CD10000 storage appliance and improving Ceph in the meantime. Authored the several notable improvements to Ceph, including but not limited to: - faster recovery (vastly reduced time required to start recovery after OSD crash) - prioritized recovery (instructing Ceph to recover particular dataset first) - numerous performance improvments, too many to list here
A
Hello,
everyone
I
would
like
to
start
that
presentation
about
having
had
true
and
biased
its
story,
affordable,
reliable
and
fast,
safe
on
and
be
aiming
drives
together.
I
am
with
build
and
calendar,
and
we
are
going
to
talk
about
affordable
cell
phone
from
the
perspective
of
large
cloud
provider
and
also
from
the
cold
Sutent
for
different,
smaller
and
larger
companies.
A
A
B
A
So
the
agenda
for
today
is
the
first
of
all
is
itself
right
tool
for
you,
because
it
might
not
be
right
tool
for
you
or
it
might
be.
It
depends
on
your
Q's
case.
Also,
if
you
decided
to
use
it
then
plan
ahead,
define
KPIs
for
itself
cluster
on
it.
In
BMI
storage,
talk
to
vendors
and
benchmark
your
cluster
to
get
the
most
performance,
you
will
need
also
to
tweak
it
a
bit.
A
B
There
there's
a
big
thing:
I
I
see
happening
in
the
field
that
people
deploy
safe
because
they
think
chefs
should
solve
all
my
problems,
but
before
you
start
implementing
self
think
about,
is
it
a
right
tool
for
you
and
one
of
the
things
is
we
saw
it
with
a
survey
from
last
year,
I
just
mentioned
to
piata,
saying
62%
of
the
people
say
we
want
the
focus
to
be
on
performance
with
self.
If
I
ask
somebody
here
in
the
audience
saying,
do
you
want
more
performance?
A
To
get
in
the
best
of
your
money,
you
should
plan
ahead.
You,
you
can't
make
rushed
decisions,
you
can
just
go
and
rush
through
the
shops
and
get
it
the
any
CPU
and
nvme
drives
on
the
market
and
call
it
a
day.
You
won't
get
the
best
of
your
money
or
your
hardware.
At
the
same
time,
if
you
do
it
that
way,
you
should
plan
ahead.
You
should
consider
also
all
options,
even
those
that
are
unrealistic
at
the
time.
A
For
example,
lately
AMD
has
a
came
back
into
play
and
they
have
shown
that
they
are
capable
of
producing
processor
that
are
able
to
provide
enough
of
performance
to
use
for
set
on
nvme.
There
are
also
other
nvme
vendors
and
just
enter
or
Samsung.
So
this
look
around.
Maybe
you
have
some
other
options
just
waiting
for
you
to
check
them
out.
You
need
to
specify
your
load
load,
meaning
a
number
of
clients
how
they
are
going
to
use
your
cluster.
What's
the
the
capacity
of
your
cluster,
because
they
did
is
also
related.
A
If
you
need
more
capacity,
you
might
need
to
spend
some
more
on
endgame
drives.
You
might
need
to
get
larger
drives
or
more
of
these,
so
speaking
of
capacity,
this
also
related
relates
to
the
performance
of
your
cluster,
because
if
you
get
too
much
of
capacity
per
node,
you
might
get
not
really
best
performance,
because
you
will
simply
overwhelm
your
machine
with
the
request.
A
Speaking
of
the
KPIs
key
performance
indicators
for
each
cluster,
you
need
to
be
certain
that
you
will
get
them
wrong,
but
what
you
should
consider
you
should
consider,
for
example,
I
of
latency
Bank.
Do
it
your
clients
will
produce
to
the
cluster,
how
many
clients
you
are
going
to
serve
or
what's
the
recovery
impact
when
you,
when
the
drive,
fails
what
the
impact
for
customers
for
users?
What's
the
SLA?
What's
the
SLA
for
the
cluster,
how
much
downtime
you
can
accept
and
of
course,
the
real
you
variable
capacity
yeah.
B
I
think
that
in
each
recovery,
impact
is
something
which
is
usually
underestimated.
Is
we
always
assume?
Systems
are
online
and
Vimy
has
infinite
performance,
so
we
can
use
everything
we
want
and
a
be
realistic
planning
ahead,
do
testing
and
actually
test
those
recovery
situations
to
see
if
it's
still
unto
par
with
the
performance.
You
expect
you
in
production,
otherwise
it
will
be
a
very
bad
day
or
a
bad
weekend
for
you.
If
the
drive
start
failing
if
stuff
just
goes
down
and
performance,
it
horrible
and
customers
start
complaining
or
inside
or
external
customers,.
A
So
when
defining
KPI,
you
will
get
them
wrong,
as
I
said,
so
what
can
go
wrong
in?
In
specifically,
for
example,
we
have
picked
240,
random,
read/write
jobs,
and
we
couldn't
achieve
that
because
we
are,
we
were
CPU
bound
and
we
couldn't
imagine
that
at
the
time
so
picking
too
much
eye
of
this
one
of
the
issues,
other
issues
like
under
estimating
what
customers
and
clients
can
do
for.
A
You
like
you,
are
announcing
new
offer
like
we
have
an
nvme
storage
now
and
what
happens
the
customers
will
go,
come
and
just
do
a
lot
of
benchmarks.
Killing
your
storage,
because
hey
it's
nvme
storage
I
can
do
everything
right
now
right!
No,
so
customer's
needs
to
be
aware
of
this
in
some
way
and
you
in
particular
to
one
issue
that
can
go
in
the
way.
Is
that
sometimes
managers
don't
listen
to
you?
It
doesn't
matter.
A
What's
your
seniority,
level
or
role
in
the
company,
they
are
not
listening
to
you
and
they're,
just
requiring
you
to
put
more
capacity
or
something
else
in
the
cluster
than
you
know
it
will
handle
resulting
in
classes
that
are
less
than
optimal
in
their
performance.
You
need
to
be
aware
of
this
situation
and
be
prepared
for
this,
and
you
need
to
be
to
be
assertive
also.
If
you
are
already
talking
to
vendors,
they
might
provide
some
benchmark
numbers.
You
need
to
make
sure
they
are
valid.
A
You
need
to
benchmark
your
hardware
once
you
get
it,
so
you
know
that
it's
up
to
spec
and
you
are
getting
what
you
have
ordered
and
even
if
you
assume
you
have
a
desired
type
of
IRS,
you
are
still
getting
clients
complaining
for
performances.
Why?
Because,
if
forget
about
latencies,
you
are
getting
300,000
area,
but
who
cares
when
you
get
to
100
milliseconds
of
latency
period,.
A
B
Yeah,
so
latency
is
something
you
should
focus
on
it.
We
stated
it
a
couple
of
times
and
verify
those
claims
from
vendors,
of
course,
Intel
AMD
Samsung.
They
all
create
beautiful
products
and
they
have
beautiful
benchmarks.
So
we
have
a
few
of
those
coming
up.
They
claim
some
numbers
but
verify
those
claims
for
yourself.
B
Just
don't
read
the
white
paper
because
there's
always
a
small
asterisk
down
down
somewhere,
stating
all
the
things
which
they
did
and
they
try
to
cut
corners
on
those
things
so
be
realistic,
verify
their
claims
before
you
go
into
production,
so
verify
those
claims
four
times
so
actually
before
you
do
something
with
it.
This.
B
And
this
is
one
of
the
worst
things
which
you
can
do.
What
Seth,
who
who
is
running
replica
two
with
a
safe
cluster,
I,
see
what
USA
I
know
you
are
as
he's
not
a
hand,
okay
who
doesn't
dare
to
put
up
his
hand,
but
it's
still
running
where
Africa
there
we
are
there.
We
are
yeah,
it's
it's
dangerous.
It's
dangerous,
but
you'll,
see
that
vendors
do
this
because
it
boosts
their
their
benchmarks.
So
we
have
some
benchmarks
later
on
showing
the
difference
between
one
two
and
three
times:
replication
and
yeah.
B
B
It
should
be
reliable
right
and
I
had
a
talk
before
the
conference
saying
it
ten
ways
to
break
yourself
cluster,
while
replica
two
was
one
of
them,
so
under
Newton
no
circumstance
use
two
times:
application
always
use
three
times
replication
and
the
other
thing
you'll
see
is
that
people
will
stop.
Focusing
on
bandwidth
with
nvme:
suddenly
we
start
putting
a
hundred
gig
networking.
It's
an
individual
machine
because
yeah
those
envy
amis
in
that
system
it
can.
They
can
saturate
the
the
network.
But
latency
is
what
matters.
B
What
I
see
in
production
with
systems
they
might
be
doing
tens
or
over
100
ki
ups
on
a
system,
but
it's
not
reaching
ten
gigabits
at
all.
So
if
you
look
at
the
vendors
again,
they
they
create
these
beautiful
white
papers
with
hundred
gig
networking
you'll
be
spending
a
lot
of
money
on
networking,
while
the
stock
switches
you
take
are
more
than
enough
ten.
Twenty
five
gigabits
per
node
is
usually
sufficient
and
just
tune
down
the
back
fields
to
prevent
saturating
the
bandwidth.
B
Instead
of
going
all
the
way
crazy
we're
trying
to
do
recovery
within
twenty
minutes,
you
can
do
so,
but
you'll
be
spending
a
lot
of
money
on
network,
while
I
recommend
just
buy
a
few
more
machines,
you
get
more
capacity
and
you
get
more
performance
because
we
are
a
scale
out
system,
so
be
cautious
with
spending
too
much
money
on
bandwidth
in
a
network.
Other.
A
Things
they
invent
of
still
seeing
for
paying
are
like
I
think's
passed
with
the
caching
drives.
They
are
saying
that
you
should
use
caching
drive
avian
in
front
of
NPM
in
drives
or
using
in
science
of
configuration,
that's
totally
unrealistic
and
incapable
for
running
in
production,
and
that's
configuration
that's
guaranteed
to
trash
your
data
at
some
point,
but
they
are
using
it.
A
One
case
we
had
at
OVH
actually
was
when
we
were
almost
screwed,
because
we
had
an
nvme
Drive
tailor.
It
was
a
cascading
failure,
we
had
actually
two
nvme
drives
lost
and
they
were
hosting
the
same
set
of
data
in.
In
other
words,
we
lost
two
replicas
and
we
weren't
completely
screwed,
because
we
had
last
replicas
living
and.
B
B
A
For
a
replica
to-
and
this
is
this
is
the
graph
of
showing
the
performance
across
different
to
client
count
from
15
to
240
and,
as
you
can
see,
the
yellow
line
denotes.
The
replica
factor
of
3
and
white
line
is
a
replica
factor
of
2.
As
you
can
see,
it's
almost
50
thousand
Arabs
more
than
replica
factor
of
3,
and
if
you
go
to
replica
factor
of
1,
it's
absolutely
insane
over
350
50000.
A
But
who
cares
about
performance
when
you
don't
have
your
data
because
you
lost
it
important
thing
that
video
also
mentions
at
the
his
presentations
is
that
you
shouldn't
copy
vendor
country
just
because
it
asks
you
for
this
most
of
users.
Don't
know
most
of
the
test
options.
Some
users
actually
just
deployed
safe
class
using
existing
tools
and
don't
think
about
configuring
set,
because
why
should
you
you
shouldn't
actually
so
sometimes
vendor
comes
by
and
ask
you?
Oh,
why
don't
you
put
this
line
in
the
country?
It
will
make
your
set
cluster
fly.
B
Like
disabling
your
safe
X
and
now
you're
like
it's
already,
the
messenger
v2
supports
encryption
of
data
on
the
wire,
but
probably
the
vendors
will
tell
you
to
disable
all
that
stuff,
because
it
it
hits
your
performance
but
again
safety
of
data,
something
which
is
important
as
well.
So
while
nvme
can
provide
all
the
performance
again,
there's
a
trade-off
between
performance
and
safety
or
consistency
of
your
data.
A
A
So
if
you
are
putting
some
kind
of
drive
in
front
of
the
end
gaming
drives,
you
are
risking
that,
if
that
caching
drive
fails,
you
will
lose
all
nvm
ease
behind
this
caching
drive
and
on
the
other
way,
if
you
are
going
to
lose
nvme
Drive
the
rebalancing
process
and
then
recovery
will
cause
your
caching
drive
to
get
trashed
and
don't
give
and
you
won't
give
any
speed
benefits.
So
it's
in
case
of
nvm
is
it's
simply
useless.
B
B
I,
ops,
who
doesn't
want
a
million,
I
ups
from
just
five
servers,
and
then
you
start
looking
because
those
were
at
the
really
small
bottom
part
of
the
of
the
whitepaper
and
they
are
using
CPUs,
which
are
10k
per
CPU
and
they're.
Putting
to
you
of
those
CPUs
in
a
single
machine
and
I
did
a
calculation
that
their
five
machine
cluster,
which
they
did
with
the
benchmark,
was
roughly
while
nearing
200-300
thousand
euros
for
just
five
machines,
and
the
raw
capacity
was
tens
of
terabytes
that
was
about
it.
B
B
They
have
the
most
expensive
CPUs
they'll
go
for
the
fastest
and
most
expensive
and
VMI's,
and
the
most
expensive,
Network
and
they'll
present
you
brilliant
numbers,
while
using
replica
one
or
replica
two
but
be
realistic,
read
those
white
papers
and
translate
to
something
which
your
organization,
instead
of
just
buying
what
they
recommend
you
to
do.
We
have
a
few
systems
coming
up,
which
we
think
are
affordable
and
Vimy
systems
which
you
can
use
and.
A
Speaking
of
verifying
things,
we
were
asked
to
use
some
expensive
drive
in
9900
FTB
a4
for
metadata
storage,
claiming
that
it
will
improve
our
performance
so
considerably
that
we
are
going
to
shut
out
every
one
memory
moment.
Many
we
have
just
to
get
that
performance
and
we
actually
get
one
of
this
drive
and
tested
it
ourselves
and
the
best
we
got
is
eight
percent
increase.
In
one
case,
one
single
case
it
was,
it
was
said
because
we
weren't
IO
bound.
A
We
were
to
be
abundantly
having
faster
drives,
doesn't
help
in
this
case,
speaking
of
slower
drives,
we
had
the
case
of
nvme
Drive
that
had
very
weird
performance
characteristic
and
we
couldn't
tell
why
is
it
happening?
Instead
of
sticking
to
the
drive
we
tested
to
drive
directly
with
file
with
the
cue
death
of
one,
and
we
realized
that
there
is
some
technology
behind
it
to
improve
temporary
write
performance
so
for
something
for
about
one
minute?
A
Sorry,
three
minutes:
we
had
about
almost
300
two
megabytes
of
write
performance
and
it
was
write
performance
that
was
considerably
faster
than
reached,
which
is
weird
actually
quite
insane,
and
after
that
time
the
performance
dropped
below
read
speed.
So
we
just
discarded
the
drive
hand
to
went
with
another
one,
and
our
performance
issues
went
away.
A
What
do
you,
what
should
you
use
for?
Benchmarking
is
mainly
file,
because
the
radius
bench
is
good
for
smoke
testing
to
make
sure
your
performance
is
in
line
with
your
expectations,
but
it's
not
for
the
proper
testing,
because
one
important
thing
default
options
for
rate
events
are
four
megabytes
rights,
and
this
is
not
testing
your
nvme
drives
about
network
performance,
yeah.
B
So,
to
add
their
if
you
use
the
radius
benchmark,
use
it
with
a
single
threat
and
then
set
the
Oh
a
be
flag
to
4k,
so
that
you're
testing
with
a
4k
right
in
a
single
threat,
because
then
you're
testing,
the
latency
of
your
radio
system.
Writing,
because
otherwise
people
are
just
cranking
up
the
amount
of
threats
with
the
Rados
bench
and
still
using
the
default.
B
4
megabytes
and
they'll
be
telling
their
colleague
oh
we're,
reaching
a
hundred
gigs
of
bandwidth
right
now,
yeah,
but
the
latency
might
be
super
high,
but
you're
not
noticing
it
so
test
with
a
single
threat
with
a
4k
right
see
what
the
performance
is
you're
able
to
get,
and
nowadays
you
should
be
able
to
get
something
below
one
millisecond
for
a
4k
replication
right
with
three
times
replication.
So
a
4k
right
somewhere
around
point,
seven
point:
eight
milliseconds
on
nvme
width,
which
is
affordable.
A
How
to
run
a
fire?
Benchmarking
is
quite
important
because
you
should
make
sure
it's
at
least
10
minutes
with
some
warm-up,
because
one
minute
will
not
give
you
the
real
performance
characteristics.
There
are
some
issues
with
that
because,
like
there
are
some
caches
in
the
way
that
can
be
filled
very
fast
and
then
real
write
performance
is
taking
over.
A
Another
thing
is
that
you
should
repeat
the
the
testing
at
least
four
times,
because
one
time
it's
definitely
not
enough.
You
should
actually
discard
the
first
number
because
it
most
of
the
time
it's
inflated
for
some
reason
which
I
will
mention
later,
and
you
should
also
try
your
destination
workload,
but
also
different
workloads
like
a
different
cute
operation,
sizes
and
so
on,
and
so
on.
To
get
a
better
view
of
an
entire
spectrum
of
your
crafted
performance.
Yeah.
B
So
to
add
to
this,
because
it
sits
on
the
slide
here
saying:
precondition
your
RVD,
because
everything
inside
SEF
is
thin
provisioning.
So
if
you
create
an
RB
d
image,
the
actual
Rados
objects
do
not
exist.
So
when
you
start
writing
for
the
first
time
to
that
image,
it
will
need
to
create
all
those
objects
inside
radio.
So
every
time
you
write,
there's
an
overhead
there's,
a
small
pause
while
you're
writing
and
actually
the
OSD
is
creating
the
initial
object.
So
that's
allocating
something
or
a
rocks
DB
inside
blue
store.
B
A
Remember
also
to
use
multiple
client
machines,
not
virtual
machines,
not
clients
like
running
fire
processes
in
single
machine,
but
the
physical
machines.
This
is
why
this
is
a
5
nodes
of
running
the
benchmark.
This
is
these
are
the
clients
and
they
are
doing
over
5,000
jobs
right
now,
as
you
can
see,
they
they
are
they.
Their
CPU
usage
is
actually
next.
Its
maximum
cluster
could
easily
go
faster.
You
could
get
more
out
from
this
cluster
if
you
had
one
more
client.
B
If
you,
if
you
trust
it
to
do
it,
but
they
do
have
a
performance
impact
having
those
mitigations
for
the
CPUs
and
enable
because
yeah,
of
course
they
were
of
performance
and
cheating
and
some
points
and
a
few
things
use
recent
kernels
because,
especially
with
the
new
Xeon
CPUs
is
scaleable
Zandi
aim
the
epoch.
You
need
a
recent
kernel
which
has
the
proper
support
for
that
CPU
I've
seen
differences
when
I
was
testing
with
the
boon
to
1604,
because
I
was
testing
last
year.
B
There
was
a
huge
performance
boost
with
just
going
for
a
new
kernel,
so
especially
if
you're
using
new
CPUs
make
sure
you
have
a
recent
kernel
which
has
the
support
for
that
CPU
and
especially
when
you
run
with
Santos,
it's
it's
sometimes
difficult
to
see
what
they
actually
back
ported
to
their
kernel,
so
verify
with
some
different
kernels
to
see.
If
you
get
a
better
performance
because
it
might
just
be
due
to
support
inside
the
CPU
and
I,
don't
think
yeah
the
sea
state
pinning
that's
it.
B
This
is
something
which
is
very
unknown
to
people
make
sure
you
disable
all
the
power
saving
for
the
CPU.
Have
it
running
on
the
highest
clock,
speed
at
all
the
time
and
pin
it
to
see
state
1
or
C
state
0
I'm,
not
gonna
into
the
C
states
themselves,
but
you'll
see
a
huge
improvement.
If
you
take
a
stock,
CPU
configuration
as
I
was
testing
with
super
Microsystems
I
had
a
two
and
a
half
to
three
milliseconds
right:
latency
for
a
4k
right
single
threat.
B
After
changing
CPU
States
doing
a
bit
of
Ceph
tweaking
and
having
the
power
saving
disabled
on
the
CPUs,
the
latency
dropped
below
one
millisecond
so
going
from
somewhere
around
3
milliseconds
to
below
1
milliseconds
by
just
going
for
the
CPU
tuning
and
a
bit
of
configuration
for
the
O's
DS,
but
not
not
that
much.
So
look
at
the
CPU
tuning.
That's
very
important
because
we
are
CPU
bound
with
nvme.
A
Speaking
of
the
mitigation
for
the
recent
flaws
here,
I
would
like
to
solve
some
impact
that
you
can
expect
from
disabling
the
hyper-threading
and
other
forms
of
SMT.
It's
around
20%
unloaded
cluster.
So
if
you
are
going
to
disable
the
hyper
threading,
this
is
a
water
is
going
to
cost
you
around
20
percent
of
performance.
In
both
my
event,
latency.
B
One
of
the
things
I
mention
about
with
the
CPU,
if
you
disable
all
the
logging
inside
the
OSD,
it
might
sound
weird
but
just
generating
the
logline
in
memory
is
compiling
that
block
line
and
maybe
even
keeping
a
memory
or
writing
into
a
file.
It
takes
time
for
the
CPU
and
it's
all
done
within
a
single
threat,
so
it
just
takes
lately.
It
eats
up
your
latency
disabling.
All.
B
So
again,
it's
a
trade-off
which
you
have
to
make
do
I
want
to
have
a
lower
latency
and
then
trade-off
not
having
any
logging
or
do
I
want
to
have
the
logs,
but
then
I'll
be
trading
in
a
bit
of
performance,
because
my
latency
will
go
up
because
the
CPU
is
busy
with
this,
especially
if
you
have
debug
OSD
on
20
or
something
you'll
kill
the
performance
of
that
OSD.
You'll
totally
kill
the
performance.
So
this
is
something
worth
trying
disabling
the
logging
on
do
STIs
and
you.
B
A
Thing
you
can
tweak
when
you
are
expecting
a
huge
loss
on
your
clusters.
You
can
tweak
the
OSD,
opt
rest
I'm
going
to
and
just
go
quickly
over
this,
but
the
general
idea
is
that
you
should
use
correct
number
of
the
OSD
obsess
and
it's
dependent
on
this
view.
You
are
using
actually
the
number
of
cores
you
have.
A
If
you
have
more
cores
you
can
you
can
use
more
or
diab
stress
if
you
have
less
threads
less
course,
you
should
use
less
course,
because
if
you
are
using
too
many
or
the
up
tress
the
context,
switching
will
clear
up
your
performance.
If
you
are
using
less
than
optimal
number,
your
say
for
SD
will
wait
for
available
or
the
up
tread
to
process
operation
and,
for
example,
on
one
specific
spiel
we
D
creates.
It
actually
decrease
it.
A
A
Speaking
of
latency,
it
didn't
change
much,
so
it's
mostly
related
to
our
hops.
Another
thing
you
can
try
to
change
is
the
async
up
tress,
because
on
by
default,
this
number
is
set
to
free,
but
this
number
is
actually
a
safe
bet.
You
should
keep
it
at
that
unless
you
have
a
slow
CPU
that
struggles
to
process
the
incoming
operations
and
you
are
getting
a
lot
of
weights
on
the
socket.
A
As
you
can
see
by
this,
we
decrease
number
of
oil
T
up
trades
and
to
one
and
performance
has
tanked.
We
increase
it
to
twelve
and
performance
is
both
also
bad.
So
three
is
the
same
bed.
You
shouldn't
change
it.
Unless
you
know
you
need
to
do
this,
you
can
also
try
using
my
make
instead
of
luminous,
because
luminosity
has
performance
decided
advantage
and
who
is
episode
in
a
second
to
how
big
this
advantage
is.
A
But
if
you
are
going
to
try
Miami,
you
can
be
tempted
to
try
a
naughty
lose
you
can,
but
in
our
observations
is
quite
new
and
you
might
need
to
do
some
proper
validation,
whether
it
is
stable
for
you
enough
and
so
on,
and
so
on.
When
I,
when
I
was
doing
the
testing,
it
wasn't
even
out
so
I
wasn't
sure.
A
B
A
Another
thing
when
you
are
picking
the
CPU,
you
should
take
care
of
if
the
CPU
frequency
trap,
because
some
people
believe
that
you
should
get
the
fastest
CPU
you
can
get
in
terms
of
CPU
frequency.
It's
not
true,
as
you
can
see
by
this
graph,
we
are
standing
between
a
one
and
half
and
four
gigahertz
in
all
cases
in
different
architectures,
but
we
are
going
up
in
terms
of
performance.
We
are
not
speeding
up,
but
we
are
going
up
to
make
this
even
more
striking
here
on
the
right.
A
You
can
see
the
graph
of
some
CPUs,
all
of
them
at
2
and
8
gigahertz,
and
they
have
a
totally
different
benchmark
scores.
So
why
is
this
happening?
Because
CPU
performance
doesn't
rely
on
the
CPU
frequency
alone?
Important
factors
are
also
architecture,
cache
sizes
and
their
performance.
Clock
speed
is
well
thermal
budget.
A
This
is
important
to
notice
that
the
thermal
budget
also
plays
a
key
role
in
CPU
performance,
because
if
you
are
going
over
the
general
budget,
it
will
down
talk
or
not
get
a
maximum
turbo
speed,
which
also
differs
between
processors
and
architectures,
and
also
software,
because
some
software
doesn't
utilize
some
instructions,
others
do
and
they
get
a
performance
boost
from
that
alone.
So
now
you
know
what
you
should
take
into
account
when
choosing
components.
A
B
B
A
It
should
also
keep
an
out
that
what
you're
expecting
load
is
because
if
you
are
expecting
a
little
number
of
little
number
of
clients,
but
you
need
to
latency
as
low
as
possible,
you
should
go
for
the
tested
CPU
as
you
can
afford.
But
if
you
are,
if
you
need
to
client
capacity,
you
should
think
about
getting
more
cores.
Instead,
this
is
just
trade
off.
You
need
to
judge
for
judge
for
yourself
if
you
also
get
a
fast
memory.
A
So
here
we
have
some
sample.
Size
stems
from
Intel,
based
on
like
Xeon
6152
and
a
terabyte
drives
so
per
first.
We
are
about
12,000
euros
and
we
have
24
terabytes
of
capacity
per
host,
so
total
cluster
consisting
of
3
nodes.
It's
about
35,000
euros
and
from
aimed
aside,
you
can
drop
to
23
thousand
euros
having
similar
capacity
and
another
sample
system
is
from
Super
Micro
to.
B
B
As
you
see,
these
systems
are
between,
let's
say
nine
and
eight
thousand
euros:
nine
and
a
half
eight
and
a
half
thousand
euros
per
system,
but
they
do
have
roughly
40
terabytes
of
raw
capacity
inside
the
chassis,
so
they'll
be
slower
than
the
previous
systems
you
saw,
but
many
of
their
customers,
which
I
work
with
that
come
from
spinning
disks
and
one
they
wanna
have
something
which
is
faster
than
spinning
discs
or
they
come
from
regular
SSDs
and
they
want
to
go
faster
than
regular
SSDs.
Well,
these
systems
worked
really
well.
B
They
will
be
CPU
bound.
That
I'll
tell
you,
because
you
can
get
in
faster
CPUs,
but
that
would
the
price
will
jump
and
then
I
will
say
just
buy
a
few
more
of
those
machines.
Instead
of
going
for
expensive,
CPUs
Saif
is
a
scale
out
system
and
with
nvme
we
tend
to
build
these
machines
with
super
fast
abuse,
where
I
would
say
just
scale
it
out.
B
Yes,
latency
doesn't
go,
go
down
with
scaling
out,
but
you
get
more
performance
in
general
because
if
you
scale
out
your
applications
as
well,
you
get
a
better
performance,
better
reliability
and
I
think
these
are
affordable
systems
and
within
a
year
time,
I've
seen
these
drop.
They
started
at
15k,
but
flash
memory
is
coming
down
and
they
drop
below
the
10k
in
just
the
beginning
of
this
year,
because
SSDs
are
getting
cheaper
and
I.
B
A
You
can
see
some
performance
benchmarks
of
the
first
two
systems.
We
mentioned
the
AMD
one
in
MD
and
in
tow
and
I
would
like
to
note
here,
because
it's
not
quite
obvious,
but
my
systems
right
now
are,
of
course,
slightly
slower
than
in
tow
it's
under
stable.
But
if
you
look
at
the
graph,
you
can
see
that
at
Q
definitely
16
at
240
clients.
The
AMD
system
was
a
capable
of
over
talking
the
Intel
system
in
terms
of
performance.
Why
is
it
happening?
A
A
A
For
example,
you
can
get
your
price
per
gigabyte
down
to
below
1
euro
per
gigabyte
on
a
in
this
system,
and
but
again
it
all
depends
on
what
you
are
targeting
for
if
you
are
targeting
for
performance
or
you
are
targeting
for
capacity,
it's
up
to
you
to
decide.
What's
the
important
factor
here,
yeah.
B
So
one
thing
to
notice:
with
this
morning
we
were
talking
about
Numa
at
the
keynotes.
The
benefit
with
the
AMD
system
says
it's
a
single
CPU
there's
no
Numa
issues
with
those
CPUs,
so
they
are
easier
to
run
less
configuration
required
on
with
the
AMD
systems
single
CPU.
Instead
of
having
duo
CPU,
so
they're
usually
bit
easier
for
the
admin
to
run
with
and.
A
You
probably
realized
by
now
that
there
is
something
wrong
with
this
graph
at
the
end
at
the
Intel
to
graph
it's
a
dip
in
the
price.
Why
is
it
happening?
Because
my
benchmarking
metals
had
a
flaw:
I
forgot
to
notice
that
the
first
run
had
to
be
discarded
and
some
of
the
performance
data
was
inflated
because
of
the
TC
Malik
was
well
cold
and
could
serve
memory
Rick
wrinkles
faster
than
later,
and
it's
simply
made
entire
benchmarking
looking
suspicious.
This
is
why
you
should
remember
to
benchmark
you
think.
B
B
B
Yeah,
so
for
the
for
the
audience
need
who's,
gonna
be
Rico.
Seeing
this
in
recording
the
slide
says:
debug,
OS,
d20
and
Java
is
mentioning
it.
Indeed,
if
you
set
it
to
0/0,
you
also
disable
the
in
memory
logging
of
the
daemon,
so
pupil
I'd
need
to
update
the
slide
saying
get
it
says:
0/0
itself
just
zero.
So
thank.
B
A
We
have
in
our
testing.
We
realized
that
at
one
machine
we
had
this
option
disabled
and
it
caused
a
lot
of
iron.
Q
storms
degrading
our
performance.
I
can't
exactly
explain
what's
going
on
here,
but
it's
related
to
the
how
memory
works
and
it
was
just
conflicting
with
other
devices
in
machine.
So
after
we
enabled
this,
the
performance
went
up.
B
A
A
This
is
a
fixed
a
bit
because,
if
you
are
writing
like
10
lines
in
a
one
go
they're
just
using
one
Cisco
and
to
the
performance
impact
of
writing
to
disk
it's
a
lower,
but
in
previous
version
the
performance
impact
of
writing
to
disk
is
quite
high.
In
any
case,
you
should
disable
logging
at
all,
not
just
to
memory
or
just
to
disk.
Okay,.
B
Okay,
so
the
question
is
how
much
you
pay
for
disabling
the
power.
Saving
I
did
some
testing,
but
just
going
into
the
ipmi
of
Super
Micro
and
those
ten
you
or
the
one
you
ten
env
meets.
They
consume
about
200
watts
of
power.
They
continuously
with
a
power
saving
disabled
and
with
the
power
saving
enabled
they
were
consuming
about
170
hundred
eighty
watts,
so
it
went
up
would
say,
10
15
percent
of
our
usage,
but
our
latency
drops
from
3
milliseconds
to
something
below
1
millisecond.
B
B
A
You
should
in
current
types
of
versions,
because
at
this
point
itself
isn't
able
to
utilize
purely
one
single
and
PME
drives
to
get
most
of
your
nvme
drives.
You
should
put
at
least
two
OS
on
single
and
vme,
but
you
need
to
be
also
o'er
of
the
implication
of
this,
because
if
your
drive
fails,
you
will
lose
to
Ortiz.
But
in
my
opinion
it's
a
good
trade-off.