►
From YouTube: 2017-APR-27 :: Ceph Performance Weekly
Description
Weekly collaboration call of all community members working on Ceph performance.
http://ceph.com/performance
For full notes and video recording archive visit:
http://pad.ceph.com/p/performance_weekly
A
A
B
A
All
right,
27.
A
A
For
now
we're
just
not
doing
it
at
all.
I
shouldn't
remember
a
conversation
with
Alan
where
we
decided
it
wasn't
even
worth
doing
at
this
stage.
But
I
can't
remember
exactly
what
the
reasoning
there
was
I
was
a
while
ago,
but
yeah
that
finding
to
revisit
it
data
up
to
throttle.
Is
there
I
think
it's
not.
The
good
news
is
not
having
a
big
performance
impact
I
just
backed
us,
because
it's
not
actually
really
throttling
anything
without
it
being
sort
of
tuned
carefully,
so
I'm
not
sure
that
it
actually
helps
is
doing
anything
useful.
A
So
I
need
to
sort
that
out.
So,
let's
see
so,
we
adjusted
the
throttle.
Cost
function
a
little
bit
so
that
it
works
better
for
small
angles
of
hard
disks
that
got
changed.
The
other
pull
request
that
cleans
up
strata
options
I
think
is
either
ready
to
merge
or
Mercia
already
I
can't
remember
if
it
emerged.
Yet.
A
Let's
see
there
was
a
fine
at
change
that
fix
encoding,
that
guy
merged,
that's
just
avoiding
some
wasted
CPU
when
doing
sparse
reads:
there's
a
blue
sword
change
that
makes
it
not
check
the
free
space
for
blue
FS
on
every
single
quit
cycle
that
showed
up
in
the
wall.
Clock
profile
that
got
merged
and
there's
a
set
of
s
changed
I,
don't
know
anything
about
that
merged.
A
There
was
a
really
ill
change
to
buffer
list
that
avoids
a
copy
that
got
merged
a
case
or
change.
I
got
merged.
Okay,
so
I
wasn't
really
used
by
anybody.
So
that
doesn't
really
matter
and
I'll.
Think
tiny,
optimization
to
pgic
see
that
changed
and
we
close
some
per
John
Pangos,
adding
some
perf
Kanaks.
We
close
that
I
just
decided
it
wasn't
for
a
while.
A
couple
of
things
are
still
in
flight.
There's
reducing
some
of
the
lock
acquisition
nice
attest
to
go
through
testing.
The
sync
right
change
still
needs
to
go
through
testing.
A
This
is
the
idea
here
is
to
reduce
the
latency
to
reassemble
context
switches
in
the
KB
sync
path.
Most
of
the
right
through
blue
FS,
through
rush
to
be
are
blocking
any
way
because
it's
compaction
thread
that's
doing
a
right
and
and
calling
us
Inc
or
it's
the
K
vsync
thread
where
it's
doing
it
right
and
then
calling
a
sync
on
it.
So
the
idea
is
to
just
do
a
direct
right
without
doing
context
which,
instead
of
scheduling
a
IO
having
it
wake
up
on
another
completion
thread,
then
wake
up
to
calling
for
it.
A
A
John,
paying
had
a
pull
request
that
was
combining
up,
commit
and
up
applied
that
actually
looked
pretty
good,
but
I
think
we
we
T
submitted
a
new
version
that
switches
around
so
that
the
OP
apply
just
happens
in
the
caller
synchronously,
since
blue
store
doesn't
actually
need
you
to
wait
at
all
and
neither
does
not
store
next
day
both
as
soon
as
you
queue
the
transaction.
Its
results
are
visible
through
the
cashing
in
the
object,
store,
implementation,
so
I
think
that'll
be
simpler.
A
That
means
that
the
OP
applied
callback
is
basically
only
used
for
file
store,
which
is
the
only
one
that
has
this
goofy.
You
have
to
wait
until
it
gets
journaled
and
then
apply
it
before
you
can
read
it
again.
So
that'll
eliminate
a
little
bit
of
PT,
lock
contention
because
will
be
one
less
callback
that
has
to
run.
A
Let's
see
the
blood
testing
needs
more
work,
failed
qi
again
excusing
the
crc
one
is
being
discussed,
and
I
have
a
poor
request
that
changes
around
the
epoch
tagging
on
post
ego,
so
you
must
just
hopefully
make
goes
to
use
block
less
often
and
that
I
rebase
that
yesterday
I
think
andrey
reviewed
it.
I
think
it
looks
good.
It
just
needs
to
be
tested
and
then
get
a
second
review
that
will
just
eliminate
sort
of
a
bunch
of
random
tail
agencies,
basically
in
USD
and
the
other
one.
A
A
A
B
A
A
A
That's
a
linear
combination
of
them
and
then
that
cost
function
is
then
applied
to
a
single
trottle.
So
in
reality
it's
it's
just
the
number
of
bytes
plus
some
multiple
times.
The
number
of
I/os
and
that
multiple
multiplier
changes,
if
it's
necessary
in
HT
t
because
sikhs,
are
more
expensive
on
our
disk,
and
so
it's
just
a
matter
of
picking
that
multiple
and
then
the
throttle
is
set
sort
of
in
units
of
megabytes
more
or
less.
A
And
so
the
idea
is
to
run
a
large
write
workload
and
then
adjust
the
throttle
so
that
it's
at
the
lowest
value.
That's
still
good
to
you,
almost
all
the
throughput.
So
we
have
the
shallowest
queue
that
still
lets
us
fully
consume
utilize
the
device-
and
that
seems
to
be
around
64
megabytes
roughly
at
least
some
hard
disk
I'm,
not
sure
it
matters
on
invading
and
then
for
small
I/o.
A
We
would
do
the
same
thing
where
we
leave
the
throttle
where
it
is
run,
small
EO
and
then
adjust
the
multiplier
so
that
effectively,
you
start
throttling.
The
small
is
with
larger
multipliers
and
again
get
to
the
point
where
it's
the
largest
value
and
that's
the
most
throttled
that
still
gets
you
most
of
the
true
put
of
the
device
and
that's
the
sort
of
the
shallowest
queue
depth,
I
guess
within
boost.
Or
so
that's
what
that's
a
bit!
B
A
A
The
the
fact
that
there
were
only
16
is
available.
Video
state
at
all
means
there's
a
ceiling
on
the
amount
of
batching
that
we
were
able
to
do.
Yeah.
No,
no
so
I
mean
in
theory.
If
you
had,
you
know
a
thousand
iOS
and
you
add
DP
was
infinitely
cheap
and
you
would
do
all
one
thousand
at
a
time
and
you'd
get
like
whatever
that
is
I'm
14x
per
second
or
whatever
or
or
4
Meg's
per
commit.
It
should
be.
You
know
what,
however
me
that
is
per
second.
B
Yeah
I,
guess
and
blend
blend
really
concerned
about
is
that
it's
just
going
to
ship
things
out
like
we're
going
to
try
this
with
different
IO
that
Vicki,
that's
or
other
things
we're
going
to
be
different
kind
of
points
at
which
we
that
which
we
see
that
transition
away
from
like
the
high
high
tail
latency
the
end
of
what
was
major.
Your
new
listing
at.
A
C
A
Not
offhand,
but
given
that
the
I
adept
with
only
16
I'm,
not
sure
that
the
numbers
mean
very
much
okay,
my
guess
my
best
guess
is
that
it
was
getting
into
one
of
those
cycles
where
it's
doing
one
commit
psycho
or
does
one
I
own.
The
next
cycle
is
of
15
and
its
footlong
back
and
forth,
and
maybe
based
on
these
straddles,
it
either
settles
into
that
sort
of
equilibrium,
state
or
an
equilibrium
state
where
it's
like.
A
A
A
A
B
B
Oh
sorry,
that
was
the
definition
you
had
another
multi
profile
and
have
there,
but
in
this
particular
one
this
is
just
doing
for
catering
two
nights
and
sisters
to
master,
maybe
to
the
fact.
But
in
the
email,
deceptive,
all
I've
got
a
bunch
more
information,
but
what's
kind
of
going
on
here,
I
mean.
B
A
Just
going
to
say
that
the
code
in
there
is
really
weird
I
understand
why
the
branch-
the
way
it
basically
it's
the
current
pointer
buffer
segment-
has
enough
for
this
whole
thing
that
we're
copying
out.
Then
it
does
a
shallow
copy,
which
is
what
we
want
it
just
as
a
reference
to
it.
But
if
it
extends
beyond
that
chunk,
then
it
does
a
full
copy
and
I
would
make
any
sense.
Why
I
would
do
that.
I
have
no
idea
why
it
could
I
need
to
go.
B
B
Okay,
so
this
other
one,
though
that's
pork,
a
random
right
in
that
one,
there
was
a
bunch
of
stuff
that
was
kind
of
interesting.
How
the
big
thing
well,
the
first
victim
is
in
there
and
thread
to
like
95
percent
of
the
time
spent
in
background
call
compaction
and,
like
I,
think
suddenly.
Some
percent
of
that
is
all
in
spent
doing
random
reads,
so
we're
just
set
sending
lots
and
lots
and
lots
of
times
doing
random
rates
and
compaction.
A
B
Let
me
get
somebody
does
open
right
now,
this
once
again,
I.
A
A
A
B
The
first
example
of
this
is
on
like
lines
like
I
guess:
thirty
fours
I
guess
kind
of
word
goes
down
to
you
can
see.
That's
that's
one
part
of
it,
but
then,
if
you
actually
go
to
some
other
lines,
you'll
see
you'll
see
more
there's
like
another
90%
of
spent.
Doing
this.
B
A
I
mean
that
it
kind
of
makes
sense
if
you're
getting
compaction,
you're
going
to
be
the
thing
that
you're
actually
consuming
all
the
time
is
reading
the
data.
That's
not
like
we're
doing
anything
wrong,
necessarily
unless
we
could
read
it
more
efficiently,
so
I'm
guessing
that
the
there
might
be
an
opportunity
to
do
prefetching
if
we
understood
what
the
workload
was,
but
you'll
notice.
This
first
case
on
line
31,
it's
a
random
access
file,
reader
and
I.
A
That's
not
what
I
thought
it
should
be,
but
so
everybody
who's
a
manager
on
this
call.
If
you
have
somebody
who
has
rocks
TV
background
or
experience
and
has
time
to
go,
do
this.
That
would
be
extremely
helpful
because
we
don't
have
anybody
in
Red
Hat
right
now,
so
it's
time
to
go
understand
what
Rock
City
is
doing
or
anybody
else
for
that
matter.
It's
good
to
get
to
this
so
be
helpful.
A
A
So
there
are
CB
is
spending
most
of
its
time
waiting
for
its
IO
to
reach
the
disk.
So
this
is
why
I
did
that?
Sync
blue
fsync
read
because
it's
in
weight,
/
io,
it's
sitting
there
waiting
to
be
woken
up
by
another
thread,
so
the
thought
was
that,
if
we're
not
doing
a
IO
we're
doing
a
synchronous
right,
that'll
be
one
less
context
which
maybe
they'll
be
a
bit
quicker.
So
after
we
sort
the
throttle
thing
out,
that
might
be
something
to
look
at.
Try
to
try
that
branch.
A
A
Yeah,
it
might
have
to
guess
I
would
say
my
guess
would
be
that
if
you
take
the
right
buffer
and
you
make
it
not
crazy
huge
and
now
it's
Turner
256
Meg's
and
you
go
back
down
to
like
eight
Meg's,
which
is
what
the
default
is
I.
Think
artistic
seem
like
something
small.
Then
this
will
go
a
down
because
that's
256
Meg's
of
tiny
key
value
pairs
that
have
to
be
sorted
in
memory.
That
turns
into
a
big
a
big
sorted
data
structure.
B
B
A
A
So
yeah
copy
isn't
going
to
change.
A
A
A
A
Wonder
if
we
should,
that
might
be
a
good
thing
to
keep
track
of,
as
if
we
can
find
multiple
cases
where
these
allocations
are
consuming
a
significant
amount
of
time,
and
there
are
small
so
line
2757
here,
there's
the
buffers
for
a
blob
for
shared
blob,
that's
usually
one
buffer
or
a
very
small
number
of
buffers.
So
that's
something
where
I'm
at
the
slab,
a
slob
collection
thing
that
Allan
prototyped
would
work
well
and
see
where
the
others
these
are
their
piles
will
bring
at.
A
Io
context,
that
might
be
a
good
one,
it's
a
list
of
AOS,
but
we
could
probably
make
it
a
vector
and
just
say
that,
because
the
trends
we
don't
really
care
how
much
memory
the
transplant
dice
consume.
It's
like
that.
It's
a
per
transaction
structure-
yeah,
that's
ephemeral,
they're
just
sliding
through.
So
we
could.
A
A
A
A
B
Allocate
and
deallocate
that
constantly
yay,
okay,
I'm
gonna
move
on
we've
get
2936,
and
this
is
just
Kansas
is
the
start
of
the
piddly
stuff
that
remains
at
in
in
GBC
threads
here,
Soto
there's
just
some
or
various
bits
going
on
this.
B
A
A
B
A
Yes,
I
think
this
is
getting
into
stuff.
Suppose
all
the
OSI
work
sure.
So
the
good
news
here
is
that
there's
a
bunch
of
time
here,
that's
spent
in
or
is
it
somewhere
here
we
should
see
the
PG
lock
up
that.
A
Have
four
interrupts?
It
doesn't
get
contended
okay!
Well,
regardless
the
there's,
a
pull
request
under
so
the
line
331
83
I
think
it
is.
A
B
C
A
A
E
B
A
A
B
A
B
A
B
And
farther
down
and
we've
got
lots
of
buffaloes
to
showing
them
up.
There
now
looks
whatever
but
farther
down
43-28.
We
have
a
prepare,
send
message
and
learning
code.
A
A
What
are
the
one
of
the
takeaways
here?
Your
some
notes,
the
things
to
look
at
putting
food
up
tracker
or
counters.
B
B
B
E
A
C
A
Yeah
so
you're
right
going
mapping
buffer,
rasa
CRC
map
and
that
it's
possible
that
we
don't
want
that.
Actually
we
added
that
when
we
were
trying
to
mitigate
the
CRC,
the
two
FERC's
that
happened
during
rites,
one
of
them
in
the
messenger
when
we
pull
stuff
up
the
wire
and
then
again
in
the
when
we
write
to
the
file
store
journal.
We
see
our
see
again,
and
this
is
on
an
arm
32.
A
Because
this
just
really
deep
call
chain
at
2618
is
pleasing
the
list
of
I/o
context,
which
I
think
also
has
a
list
inside
ot,
which
is
the
list
of
something
else,
and
then
the
buffer
list
inside
the
AIT,
which
does
the
buffer
list,
has
a
list
of
pointers
and
oh
I,
see
a
right,
but
it's
0.25
so
26:38.
So
that's
the
best
of
crc
one
right
there,
but
for
a
few
sample
points.
Let's
not
do
that,
but
yeah
I
think
we
could
kill.
A
A
B
Page
I'm
just
copying
out
the
form
megabyte
a
trace
and
I'll
throw
it
in
here.
We
don't
have
to
discuss
it
today,
but
I
died.
Looking
hey
I'll,
throw
it
in
the
dispatch
here.
That's
the
only
the
only
one
I
really
had
out
of
that
one.
That
was
that
that
really
stood
out
right
away
was
the
was
the
one
thing.
The
fish
I
was
happy
issue.
A
B
A
A
D
Whenever
we
have
a
hype,
you
know
I
all
our
node
extra
hassle,
he'll
just
get
killed
or
movies
and
then
in
a
automatically
shut
down.
There
are
several
s
DS
and
then
right
now
we
got
a
lot
of
instance
in
two
of
our
characters
will
run
across
there,
try
to
replicate
or
duplicate
their
aerials
as
one
who
why
the
excess
of
our
season
agenda.
D
A
B
A
C
A
D
A
D
It's
busy,
we
kind
of
merge
lots
of
abortions.
Why
is
charging
the
Downing's
from
wrong
Nellie's
and
they
I
also
went
from
how
my
team
from
exile-
and
it
was
getting
a
lot
of
improvement,
and
they
also
see
the
muscle
bath
as
well,
especially
for
their
home.
A
distribution
and
I
was
wondering
whether
we
can
have
a
talk,
maybe
in
Phoenix
before
meeting
I
said
yeah
sure.