►
From YouTube: 2018-JAN-04 :: Ceph Performance Weekly
Description
Weekly collaboration call of all community members working on Ceph performance.
http://ceph.com/performance
For full notes and video recording archive visit:
http://pad.ceph.com/p/performance_weekly
D
D
Looks
like
a
nice
review.
Okay,
there's
that
I
added
I
have
a
pull
request
that
at
some
minimal
tracing
to
blue
store
and
the
USD.
This
is
in
order
to
generate
a
trace
to
share
with
the
cash
physics
folks,
which
have
they're
building
a
like
a
SAS
type
thing
that
does
what
that
paper
that
we
sought
fast
several
years
ago,
Josh
and
Greg.
It's
the
one
that
does
a
cache
miss
reliability
curve.
Was
it
cash
MRC,
miss
rate
curve?
D
So
you
can
size
your
caches,
but
it's
you
know
all
improved
and
all
that
stuff.
So
there's
that
that
actually
was
interesting.
So
the
way
that
I
was
the
way
that
they're
setting
it
up
is
it
would
be
it's
like
a
you
set
up,
something
that
will
feed
traces
to
an
agent
that
feeds
it
back
to
their
like
SAS,
hosted
service
thing
or
whatever,
and
then
it
gives
you
back
all
that
all
the
nice
information,
which
is
nice,
because
licensing
issues
sort
of
go
away.
D
The
way
that
they're
their
preferred
way
of
collecting
traces
is
based
on
open
tracing
and
the
whole
conversation
led
me
to
so
they
want
to.
They
want
to
basically
have
these
probes,
where
they're
they're
sampling,
frequently
initially
to
build
up
the
initial
data
set,
and
then
they
sample
less
frequently
over
time
and
so
in
sort
of
the
steady-state
you're,
actually
sampling
at
a
very,
very
coarse
rate,
but
they're
looking
at
open
tracing
and
this
project
called
Jaeger,
which
is
the
is
in
the
CN
CF.
Now
it's
sort
of
the
new
version
of.
B
B
D
But
as
far
as
like
being
able
to
get
to
the
point
where
there's
something
that
is
attaching
to
roaming
process
and
like
collecting,
trace
data
and
sampling
it
and
doing
something
with
it,
I
don't
know
what
what
is
do
we
just
do.
We
do
anything,
and
so
just
using
LT,
T
and
G
to
do
that.
Or
is
there
like
some
library
that
were
Civic
type
of
I?
Don't
I,
don't
understand
networks
so,
but
it
kind
of
reopened
that
question
for
me:
I'm,
not
sure.
D
E
D
All
right
so
anyway,
if
that
pull
request
is
probably
nowhere,
but
it
was
enough
to
generate
a
trace
just
send
to
them
to
see
what
date
they
generate.
So
I
did
that,
let's
see,
there's
something
that
was
merged
with
caches
on
the
Barrelhead
I,
never
know
it.
Adams
pull
requests
actually
do
until
you
go,
read
them
or
JP
cache
improvement.
That's
a.
E
D
D
D
There's
a
change
to
the
radios
been
command
that,
like
I,
have
don't
care
about
mostly
so
it's
like
keepo
chatter.
So
that's
good,
reverting
a
proc
sighs.
So
it
sounds
like
the
takeaway.
Is
that
we're
stuck
with
an
order
in
list
size
function
for
the
lifespan
of
Ralph
707
because
of
Abyei
issues
with
the
build
tool
chain?
D
D
D
This
charted
up:
do
you
thing
where
I
don't
think
we're
going
to
merge
that
and
I've
forgot
about
this
one
from
a
long
time
ago,
Jean
Peng.
Did
it
a
thing
that
will
optionally
use
a
single
callback
for
both
op
commit
and
off
applied
instead
of
two
callbacks,
but
we
already
merged
the
thing
I
did
that
makes
the
unapplied
a
synchronous
call
back
I'm,
not
sure
that
that
one
makes
sense
or
is
going
to
be
helpful
anymore.
A
G
G
So,
as
a
result,
we
are
doing
also
a
synchronous
treats.
We
are
starting
the
odd
a
synchronous
machinery
also
for
cave
in
situations.
Our
data
are
actually
available
in
cache
in
Brewster
caches.
So
in
such
case,
it's
expected
to
have
a
performance
penalty
right,
I'm,
trying
to
address
that
with
they're,
getting
the
decision,
whether
to
go
synchronously
or
asynchronously
to
the
to
the
object,
store
implementation.
I've
pushed
a
branch
yesterday
on
my
github
at
the
moment.
It's
it
consists
changes
at
the
made
at
the
Brewster
liar.
G
G
To
have
the
hissing
machinery
started,
and
this
decision
is
made
in
Brewster
also
I.
Today,
I've
got
very
good
results
from
performance
results
from
mark
it
seems
we
got
some
some
performance
regression
in
in
writes
in
hungry
person,
pure
right
scenario,
which
excludes
the
possibility
that
the
bitter
I
of
threat
is
is
busy
with
handling
the
rich
completion.
Instead
of
doing,
instead
of
doing
the
right
related
things
rather.
A
Those
graphs
are
kind
of
screwed
up
actually
compared
to
mine
I'm,
not
sure
why
let
me
let
me
share
my
screen.
Okay,.
G
A
A
It's
it
appears
that
if
we
look
at
both
the
the
yellow
and
the
green
lines,
which
are
for
buffered,
basically
using
buffered
reason,
writes
in
blue
store
versus
disabling
those,
the
the
the
maroon
and
and
light
blue
lines
that
there's
a
much
bigger
effect
from
from
that
versus
using
your
branch.
Do
you
see
yep,
but
actually
the
the
reverse
is
true
at
small
sizes,
for
for
old
announcer,
the
reverse
is
true,
but
that
the
lines
cross
essentially
so
that
using
unbuffered
is
actually
faster
for
small
writes.
A
G
G
G
A
One
question
I
had
for
you
Retta
saw
is
in
this
random
read
case
when
we
have
your
branch
enabled
for
very
small
iOS,
there's
a
pretty
clear
trend
where
the
performance
drops
that
we
don't
see
well
specifically
with
unbuffered.
So
when
we
disable
buffered
I/o
in
blue
store
compared
to
the
master
branch.
Any
any
thoughts
on
on
that.
G
G
A
Yeah
agreed
agreed
I'm
going
to
try
to
rerun
some
of
these
tests
and
they
also
look
at
it
with
the
wall
clock
profiler
to
see
if
I
can
tell
if
there's
anything
interesting
going
on,
but
both
in
the
buffered
case
and
in
the
unbuffered
case,
especially
with
small
iOS.
It
looks
like
the
the
async
reed
PRS
is
slower.
G
A
A
C
G
G
A
A
When
one
thing
that's,
when
we
were
talking
to
stand-up
about
the
512
K
random,
write
drop,
that's
what
I'm
talking
about
right
there
that
that
point
where,
with
the
512
K
IO
size,
we
see
this.
This
regression
compared
to
vs.
file,
store,
there's
this
kind
of
drop
and
then
it
recovers
I'm
very
curious
if
that's
due
to
the
blob
size
being
the
same,
but
I'll
have
to
have
to
run
more
tests
and
try
to
diagnose.
What's
going
on
there.
A
But
anyway,
the
the
mixed
results,
I
think
are
probably
sort
of
similar
to
the
the
other
ones
for
sequential
mixed
IO
kind
of
are
the
the
sequential
read
as
dry.
It
kept
dropping
everything
down
a
little
bit,
but
your
branch
is
doing
better
than
than
other
things,
I
think
and
then
again
we're
faster
for
in
blue
store
for
random
I/o
again
with
this
camp
drop
at
512
K.
So
anyway,
that's
when
I
drop
at
8
K,
that's
below
the
minute
Alex
eyes
right,
because
the
male
looks
eyes
for
for
a
solid
state.
A
A
A
The
one
thing
I
will
say
is
during
this
testing
it
it
it
unfortunately
kind
of
late.
Then
the
game
became
really
apparent
that
the
amount
of
memory
in
the
these
notes
was
really
having
a
big
effect
on
on
the
performance
results,
because
each
of
these
knows
has
64
gigs
of
RAM
and
with
multiple
nodes.
That's
enough
that
even
fairly
large
RBD
volumes
have
a
significant
amount
of
cache
data
for
random
reads.
A
So
I
manually
had
to
go
through
and
inform
the
colonel
on
the
nodes
to
only
use
8
gigs
of
ram
that
it
was
really
the
only
way
to
to
restrict
the
amount
of
buffer
cache
that
file
store
could
use.
So
just
if
you're
doing
any
of
your
own
testing
like
this
there's,
there's
a
pretty
substantial
effect
from
cashing,
even
even
in
cases
where
you're
using
quite
large
volumes,
so
just
be
aware
of
it.
If
you're
doing
your
own
performance
testing
it
it.
You
know
it's
it.
A
G
D
A
Well,
we've
I
think,
there's
probably
still
plenty
of
investigation
to
do,
but
maybe
the
only
other
thing
I'll
say
is
that
there's
the
there's
really
wide
performance
wings
here
right
I
mean
we're
talking
about
in
the
sequential
read
case,
you
know
a
swing
of
like
almost
200
percent
and
some
of
the
other
cases
is
fairly
large
too.
So
we're
really
looking
at
important
critical
parts
of
the
the
the
read
and
write
path,
I
think
I,
guess
read
path
in
this
case,
but
it's
it's
the
right
thing
to
be
looking
at.
A
D
D
Peter
I,
just
rediscovered
your
CRC
cache
patch
that
looks
just
much
simpler,
I'm
tagging
that
one
for
QA
cool.
Thank
you
I'm,
trying
to
think
if
there
are
other
patches
that
are
sort
of
floating
around
that
I'm
that
we
want
to
look
at
that,
we
forgotten.
D
A
D
Sure
yeah,
one
of
the
one
of
the
things
that
came
up
last
night,
was
that
perhaps
we
should
it
might
be
a
little
bit
early
still,
but
at
some
point
make
a
DB
store
implementation
that
used
to
those
futures.
I,
don't
know
how
value
that'll
be
because
it'll
be
it'll,
never
block,
but
we
could
probably
make
a
mode
where
it'll
sort
of
do
a
synthetic
block
where
it
just
says.
Scientists
will
take
this
long.
They.
F
D
F
D
It
might
be,
it
might
be
premature
because
we
don't
have
any
of
the
other
stuff
ready
to
go.
It
might
be
that
there
were
still
better
off,
focusing
on
the
messengers
side
of
the
equation
and
some
of
the
other
infrastructure
but
I'm
at
some
point.
That's
I
think
it's
gonna
make
sense.
D
D
It
might
it
may
be
it's
just
useful
and
exercise
to
just
to
sort
of
lay
out
what
the
interface
would
look
like
for
things
like
cuz.
All
the
read
methods
basically
will
have
a
future
for
completion
like
getattr
and
exists
in
collection
lists.
No,
that's
so
maybe
that
makes
sense
so
that
when
we,
when
we
do
sit
down
and
so
I
think
about
how
the
OST
code
is
going
to
change,
we
actually
have
an
interface
to
line
up
against.
On
the
other
end,.
D
F
D
D
F
D
Mean
we
slowly
ripped
out
all
the
direct
read
operations,
because
everything
is
simpler
to
have
everything
go
through
a
transaction
and
in
practice
everything
should
go
through
transaction
so
because
I'm
I'm
the
reads
inside
a
inside
a
write
transaction.
The
question
is:
what
is
that?
What
would
that
accomplish
like
cuz?
Nobody
is
there
to
do
anything,
but
by
the
time
the
transaction
is
submitted.
You
already
know
what
you're
gonna
write.