►
From YouTube: 2017-APR-05 :: Ceph Performance Weekly
Description
Weekly collaboration call of all community members working on Ceph performance.
http://ceph.com/performance
For full notes and video recording archive visit:
http://pad.ceph.com/p/performance_weekly
A
B
A
Let's
see
la
there
are
some
new
pr's
this
week
is
a
couple
different
blue
star
ones.
The
signified
throttling
model.
It's
a
couple
of
different
commits,
but
I
think-
and
you
know
one
thing
that
does
is
try
to
increase
the
queue
depth
for
deferred
right
and
actually
in
contesto.
We
could
just
quickly
last
night,
it
looks
like
in
every
part
of
this
case
it
is
helping
or
small
synchronous
rights
and
random
lights.
A
So
Nick,
especially
for
smooth
things
with
your
looking
at
this
may
help
on
how
to
get
least
it
does
not
appear
to
be
helping
it
all
on
Indian.
So
that
is
something
I
think
they
just
plan
trying
to
look
at
it
more
today.
He
actually
mentioned
that.
He
thinks
that
there
may
be
an
issue
with
our
question
behavior
that
can
do
contribute
to
that.
Oh,
hey!
Take
your
here,
hey
you!
You
want
to
go
over.
Some
move
is
PR
three
hours.
You
can
go
through
them,
but
you
might
make
it
better.
C
Sure
yeah
for
this,
that
unify
for
early
model,
one
they're,
actually
two
fets
change
system
there,
one
or
just
targeted
at
improving
things
for
q-tip,
one
on
hard
disk
which
seem
to
be
helping,
and
then
there
two
patches
that
simplify
the
throttling
but
I
think
they're.
Still
it's
not
quite
right,
so
I
want
to
I
want
to
run
it
locally
with
envy
me
journal
hard
disk
data
and
make
sure
that
it's
doing
what
it
should
be
doing.
C
Excuse
me,
but
will
say
that's
fine,
there's
a
pull
request
that
tries
to
make
glue,
store,
unshare
blobs,
that
get
cloned
and
then
the
clone
gets
the
only
clone
gets
deleted,
which
should
help
for
ec
overwrite
workloads
that
needs
testing,
review
and
stuff.
It's
not
super
elegant
or
anything,
but
it
needs
did.
C
You
just
basically
choose
how
much
overhead,
basically
up
for
for
that
versus
a
throughput
type
workload,
and
that,
hopefully,
will
let
us
manage
tho
those
queue
depth
sort
of
automatically,
so
that
all
the
m
clock
and
qss
special
thing
will
actually
work
which
it's
looking
reasonably
promising.
But
we
haven't
actually
run
the
throttling
stuff
through
any
performance
tests.
Yet
to
see.
If
it
what
impact
it
has
on
the
overall
throughput,
so
let's
see
how
that
goes
is.
C
You
could
do
that,
but
I,
don't
know,
that'll
really
tell
you
a
lot,
because
I
mean
you
could
figure
out
what
the
latency
is
for
like
a
key,
tough
one,
but
that's
not
really
what
you're,
after,
like
the
fact
that
the
hard
disk
can
do
it,
you
know
an
eight
millisecond
right,
isn't
the
important
piece
of
information.
What
you
really
want
to
know
is
the
fact
that
there's
a
hard
disk,
for
example,
the
fact
that
there's
this
overhead
of
journaling
things
and
actually
committing
them
that
is
going
to
cost
you
eight
milliseconds.
C
A
C
Maybe
yeah
at
that,
it's
going
to
be
it.
We
can't
we
can't
get
too
clever,
I
think
trying
to
understand
device
because
it's
it's
it's
a
black
box
and
it
has
complicated,
weird
behavior
through
and
and
there's
going
to
be,
like
you
know,
compaction,
that's
going
on
in
the
background
and
there's
going
to
there's
all
this
stuff
going
on.
So
we
can't
I,
don't
think
we
can
get
too
sophisticated
because
it's
sort
of
doomed
to
fail
so
but
I
think
the
trick
is
trying
to
find
as
sort
of
like,
as
course,
a
model.
D
C
Well,
it's
not
going
to
help
with
Bluestar,
because
right
now
we
won't
take
advantage
of
that.
It's
probably
helpful
and
that
we
want
to
do
something
similar,
but
I
think
we
already
are.
We
have
a
very
limited
amount
of
right
back
that
we
do
in
boost,
or
it's
like
30
to
iOS
at
most
or
whatever.
It's
like
pretty
pretty
small.
C
C
Yeah
so
anyway,
there's
that,
let's
see
what
else
some
bunch
of
stuff
merged
some
boo,
fest,
optimizations,
very
small-
let's
see
the
those
theory
map
stuff
merged,
I-
think
Ashley
go
week
before
last,
which
means
that
we
can
get
up
once
you
sort
of
get
everything
up.
The
luminous
you'll
be
able
to
get
a
perfect
distribution
of
pg's
across
those
tees,
so
get
rid
of
all
that
utilization
variants,
which
will
help
a
lot
there's
some
crc,
stuff
or
powerpc.
That's
getting
close
I
think
keep
you
as
fixing
the
build
on
that.
C
Oh
I,
guess
you,
worship
areas
are
clean
up.
They
can
remember
in
any
way
the
arm
range
things
merged,
but
is
turned
off
by
default,
would
find
you
do
some
testing
and
then
make
sure
that
the
I
know
rocks
to
be
upstream
was
doing
some
work
to
make
range
key
removals
more
efficient.
I
want
to
wait
for
that.
There
are
a
couple
of
posts
from
right:
applause
fixing
stuff
the
bitmap
allocator,
the
first
one
is
merged.
C
It
just
rude,
reduces
the
amount
of
memory
that
it
uses
and
makes
there
I
think
there
if
your
reporter
do
preferences.
So
that's
good,
there's
another
one
that
changes
the
allocator
hint
behavior,
that
one
has
not
merged
that
needs
to
be
sort
of
cleaned
up
and
gotten
in
a
state
that
actually
make
sense
of
sort
of
a
proof
of
concept
and
then
there's
another
huge
branch
prior
to
slot
that
that's
down
here.
C
I
guess
the
optimized
processing
of
extent
that
we
could
some
at
one
needs
review
and
probably
would
break
Evans
pieces,
but
I
think
right.
Asafa
is
fixing
some
things
in
raid
0
skateway
first
is
trying
to
get
as
authentic
Asian
refactor
marched
their
first
then
come
back
to
it.
Hello,
lady,
oh
hi,
you're,
here
topic
on.
E
Yeah
I'm
here
so
the
out,
rework
husband
master
recently
also
some
features
for
swift,
suites
feature
parity
like
bulk
up
Oakland,
similar
stuff,
but
finally
got
some
chance
to
get
back
to
blue
star
sent
to
small
request.
Today.
One
is
definitely
a
dn.m,
because
it's
it
it's
the
one,
removing
blood
press
and
switching
to
plain
pointers
and
needs
a
lot
of
testing.
However,
the
first
one
is
pretty
small
it
gives
in
my
testing,
not
officially
yet
I
would
say
on
your
my
laptop
not
on
server.
E
E
C
Okay
and
I
think
the
last
one
that's
not
a
last
one,
but
the
next
one
that
sort
of
in
line
blue
stores
to
KD
Singh
thread.
This
was
originally
a
platform
in
tell
Igor
cleaned
it
up.
I
rebased
it
again,
but
then
mark
sauce,
I'm,
hang
so
I
didn't
it
make
sure
that's
not
actually
hanging,
but
that
just
makes
it
so
that
the
second
half
of
the
commit
thread
that
does
all
the
completions
is
in
a
second
threat,
so
actually
to
commit
threads
one
that
does
all
the
submission
I'm
one
of
those
completions.
C
So
it
want
to
verify
that
that
a
is
stable
and
view
that
it
actually
helps
those
things
pretty
good.
That's
gonna
help
I,
guess
so
and
then
once
that's
in
then
I
can
finally
go
back
to
the
blue
star.
Rtc
branch,
which
is
basically
trying
to
do
synchronous,
commits
in
this
in
the
queuing
path
when
possible
and
get
that
rebased
and
cleaned
up
and
stabilized
that'll,
be
good
for
like
option
journals.
Basically,
where
you
have
super
fast
right
and
with
bowling
team
I
think
that's
it
for
blue
store,
see
what
else
is
here?
C
There's
there's
a
bunch
of
already
may
work.
That's
going
on
already
made
by
the
way
in
a
sink,
messenger
is
enabled
in
the
default
builds
and
it
seems
to
be
pretty
stable
is
from
what
I
hear
how
I
X
guy
they
have
a.
They
have
a
production
cluster
with
a
test
customer,
that's
using
it
successfully
and
there's
sort
of
a
steady
stream
of
patches
from
the
mirantis
folks,
not
righteous
mellanox.
C
C
C
To
either
crc
calculations,
40
buffers
haven't
looked
at
that
yet
yeah
they're,
a
bunch
of
your
period,
I,
need
to
come,
go,
look
at
and
and
then
Isis
bottom
batch
that
I
this
them
hasn't
been
looked
at
carefully.
That
needs
to
be
through
view
and
testing
I.
Think
mostly
oh
I,
guess
the
big
one
probably
to
point
out
is
Zipkin.
C
Is
this
tracing
framework
that
we've
been
looking
up
for
first
freaking,
two
years
now,
there's
a
google
Summer
of
Code
student
who
rebase
can
clean
it
up
over
the
summer,
but
it
didn't
merge
over
the
summer.
Sadly,
so
it
needs
to
be
reversed
again.
I
have
at
least
a
ratos
portions
of
that
rebased
and
building
and
ready
to
go,
but
music
go
through
QA
and
then
once
that's
in
then
Jason
is
going
to
take
the
rbd
stuff
and
rebase
that,
because
it's
every
dias
been
refactored
like
twice
since
then
Todd
will
be
encouraging.
C
C
C
So
if
you
do
instead
of
doing
stuff
demon
and
then
demon
ad
and
then
a
command,
you
can
do
cess
demon,
perf
and
the
demon,
ID
and
it'll
do
a
every
second
of
a
spit
out
a
line
that
shows
you
the
interesting
metrics,
which
makes
it
easy
way
to
watch
a
demon
and
see
how
many
if's
is
doing
and
how
much
blue
FS
is
writing
to
its
wall
and
database
devices
and
and
so
on.
So
if
you
haven't
seen
that
edge
check
that
makes
the
perf
canter
is
much
easier
to
consume.
C
F
So
it
is
a
pull
request,
9430,
one
that
has
been
kind
of
languishing
clerical
and
I
was
wondering
like
if
we
could
come
up
with
a
way
to
make
some
progress
with
that.
My
problem
is
I,
don't
feel
like
the
people,
the
people
that
did
this
the
perfect
scale.
People
probably
aren't
the
right
people
to
be
doing
the
actual
commits,
because
we
don't
know
stuff
well
enough,
but
we
know
the
he
found
the
performance
problem
and
analyzed.
It
I
think
pretty
well
and
came
not
to
take
advantage
of
that,
and
definitely.
C
This
sounds
like
something
that
might
be
a
read-a-thon
Sally.
If
you
have
time
or
interest,
this
is
basically
I,
think
so
Joe
Mario
did
this.
Is
that
right
that's
round
and
he
was
using
perf
to
identify
cases
where
there
is
false.
There's
cash
lankan
tension
due
to
fall,
sharing,
yeah.
F
It's
a
tool
called
c
to
c
letter
c
number,
two
letter
C
of
it
it's
not
upstream,
but
it's
it's
almost
upstream
Chris,
but
it's
a
tool
that
can
can
show
you.
You
know
cache
line
conflicts
and
with
that
he
analyzed
the
stuffs,
and
you
know
the
OSD
and
basically
found
these
two
problems.
One
was
use
of
I/o
streams
on
the
iostream
c++
class,
which
is
currently
is
cache
line
unfriendly,
and
that
was
a
huge
hit
and
not
hard.
F
A
C
C
We
I
mean
the
changes
should
share
if
the
kleenex
is
code,
like
never
changes
yeah.
I
think
the
main
thing
is
that
we
want
to
just
separate
into
her
patches,
so
one
of
them
changes
the
alignment
for
the
logging
class
with
all
the
new
Texas.
Like
that's
a
pretty
simple
change.
It
just
makes
them
all.
64
pipelined
is
that
I
thought
the
cache
lines
were
128
bytes
not
switch
before
right
did
I
make
that
up
with.
F
Whatever
I
thought
it
was,
we
thought
it
was
64,
but
my
memory
kind
of
bad
that's
too
long
ago,
but
I
think.
But
you
feel
free
to
ask
Joe
and
he
like
he
can
explain
all
the
stuff
that
he
did
and
how
to
use
see
to
see
if
anybody
entrants
to
use
it-
and
he
did
do
upstream
mark
he
didn't
use
hammer
he
actually
built
and.
D
F
No,
no
it's
a
while
ago,
it's
like
six
months
ago
upstream,
but
it
was
still
yeah.
Okay,.
C
Well,
anyway,
that's
I
mean
yeah,
so
you
write
it
at
60
provides.
I
said
this
city
really
easy
to
fix
both
to
just
commit.
Basically
I
me
just
put
this
in
separate
commits
and
then
it's
just
aligning
mostly
mining
me
Texas,
giving
them
their
own
cache
line
when
their
intended
I'm
just
pretty
pretty
reasonable.
So,
okay,
good
I,
think
some
of
those
needs
to
take
a
look.
I
guess
the
question
is:
who
who
has
time
to
do
that?
C
C
That's
exactly
right:
okay,
both
of.
E
E
C
E
Try
to
the
dot
by
the
way
a
cache
line
on
the
newer
amd64
incarnations
is
still
a
64
bytes,
however
I'm
not
pretty
sure
but
I'm.
I
guess
I
read
somewhere
that
actually,
you
are
in
processor
x,
transfers
data
using
to
cache
lines
in
the
same
moment.
Basically
128
bytes,
but
still,
but
still
one
line
is
60
64
bytes
long,
okay,.
C
C
F
C
A
I
kind
of
mentioned
before
you
were
here
too
so
I
guess
Nick,
you
might
be
interested
in
at
least
an
attention
to
some
oats
are
trying
it
out.
Yeah,
probably.
C
D
C
Yeah,
I
know
if
sam,
I
think
it's
in
it's
in
much
better
shape
than
it
was
last
time
york
right,
I'm.
The
latest
test
set
marker
in
basically
for
queued
up
to
one.
It
was
the
same
latency
as
file
for
which
is
sort
of
what
you'd
expect
we're
basically
limited
by
the
speed
of
the
hard
disk
Deeks,
so
I
seem
to
make
sure
we
K
swear
the
journals
on
envy
me
and
the
rest
is
on
the
disc
make.
C
D
C
A
Maybe
oh
hey
Nick,
would
you
mind
I
think
maybe
you've
documented
this
before,
but
it
from
you
have
any
documentation
or
you
remember,
writing
any
point
you
could
use
and
let
the
nice,
I
remember
what
all
you're
doing
yeah.
D
B
D
There's
something
like
a
two
or
three
hundred
micro
second
delay
that
cpu
to
to
wake
up
and
because
the
linux
scheduler
is
not
power,
aware
that
the
running
Fred's
basically
hop
between
all,
of
course,
so
just
randomly
you're,
just
skin
massive
delay.
So
I
was
seeing
like
without
that.
I
could
pretty
any
hit
doing
like
a
cute
at
one,
maybe
about
four
five
hundred
I
ops.
But
then,
when
I
forced
the
cpu
frequency
and
sea
state
to
see
one
that
went
up
to
1600,
I
otra,
a
single
and
bread
and
a.
D
C
That's
good
to
know:
okay,.
C
I'll
mention
that
I'm
Matt's
team,
oh
I,
have
a
yeah
okay,
so
Matt's
team
is
looking
at
hearing
latency.
Finally,
so
there
they
have
a
closer
set
up
and
they're
going
to
look
at
sort
of
the
standard
scenarios
where
anos
he
goes
down,
come
suck
up,
spin
down
for
a
long
time,
come
suck
up
and
so
on
and
look
at
what
the
slow
requests
are
where
they
block
on
peering
and
then
so
we
can
figure
out
exactly
what
it
is
during
the
curing
process.
C
That's
blocking
them,
and
so
we
can
fix
it
so
that
we
can
eliminate
those
for
those
long
tails.
When
you
have
those
cases.
This
is
sort
of
a
long-standing
thing
that
people
have
complained
about
and
there
we
have
like
half
dozen
ideas
of
things
that
we
could
improve,
but
we're
going
to
do
a
bunch
of
testing
to
actually
figure
out
what
the
things
are
that
are
actually
making
the
most
impact.
So
we
can
deal
with
the
Logan
group
first.
So
that's
embargoes.
C
There
are
a
couple
of
simple
requests
for
changes
to
appearing
to
fix
a
few
things,
but
it's
unclear
whether
those
are
actually
addressing
the
things
that
are
causing
the
most
flow
request.
So
we
want
to
do
the
analysis
first.
This
is
this
is
basically
that
we
talked
about
this
on
the
list.
I
guess
that
we
can
ask
a
go
with
the
computer
at
OVH
about
this
also
searching
the
same
thing
so
hopefully
we'll
make
some
progress
there.
C
The
big
thing,
though,
actually
is
that
I
really
want
to
reschedule
this
meeting
if
I
should
have
talked
about
this
at
the
beginning.
But
I
would
very
much
like
to
move
this
meeting,
probably
just
an
hour
later,
unless
anybody
else
has
a
different
suggestion
or
request,
because
the
Ann
Arbor
folks
generally
can't
join
because
there's
a
conflicting,
Red,
Hat
internal
call,
that
is,
has
managers
from
like
a
million
different
groups
and
is
technically
impossible
to
move
so
well.
That's
that's
why
they
sent
me.
C
That's
the
other
planet,
but
Matt
Matt
complains
about
it
on
a
near
weekly
basis.
You
would
really
like
to
be
here
so-
and
this
is
also
this
is
also
8
a.m.
pacific
which
I
imagine
a
lot
of
people,
don't
like
that
much
either
so
I
guess
it'll
be
noon
eastern,
which
also
is
sort
of
a
convenient
time.
It'll
be
later
for
China.
C
C
C
C
C
C
C
F
C
C
C
C
F
A
G
C
Well,
I
guess
the
two
options
are
it's
either
same
time,
thursday
or
thursday.
One
hour
earlier,
I
guess
I've
sort
of
the
two
that
we'd
consider.
Maybe
we
can
just
that.
Maybe
we
can
follow
up
in
the
list.
Look
like
it,
I,
don't
know
what
surveymonkey
thing
or
whatever
just
figure
out
what
works
for
people.
C
Had
one
other
idea,
I
wanted
to
throw
out
for
people
who
are
interested
in
yeah
final
justice
on
meeting
move
will
be
well,
do
it
over
email,
so
we
get
some
final
input,
so
just
pay
attention
to
accept
avail.
Yeah
I'm.
The
last
thing
I
want
to
mention
is
the
new
d
encoder
stuff
changes,
the
encode
and
decode
functions,
and
so
they
use
different
types
and
these
pre-allocated
buffers.
So
there's
no
like
splits
across
buffer
segments.
C
They
have
to
worry
about,
which
is
nice,
but
most
of
the
code
still
uses
the
old
buffalo
stuff
and
it
could
get
converted,
but
it
hasn't
been.
It
will
probably
be
forever
before
that
happens,
but
the
one
case
where
that
concerns
me
is
that
in
the
append
case,
in
the
encode
case,
basically
every
buffer
list
object
in
memory
has
a
bunch
of
fields
that
are
there
just
for
handling
efficient
depends.
They
have
this
buffer
pointer
for
the
append
buffer
and
I
think
they
have.
C
A
I,
don't
remember,
I
don't
think
I
could
tell
easily
from
that
particular
one
says
missing
some
symbols
for
some
reason,
but
because.
C
That's
the
thing
book
or
if
we
can
figure
out
which
which
users
are
actually
abusing
the
peripherals
depends
like
we
can
just
optimize
those
ones
in
the
blue
saw
itself
it's
using
that
the
encode
dunk
code,
which
is
pre-allocated
fixed
size,
not
excited
about
whatever
precised.
What
do
you
offers?
So
just
overhead
should
be
very,
very
low
or
as
slow
as
week,
pretty
low.
B
C
No
problem
yeah,
it
occurred
to
me
like
immediately
after
the
call
last
week
that,
if
the,
if
we
use
the
the
few
more
effectively
use
the
appender
types
or
methods
or
whatever
it's
all,
it's
just
a
pointer
basically
is
all
that
class
is
just
wrapping
a
pointer.
That's
your
right
position,
then
you
can
avoid
all
those
function
calls
and
that
it
might
not
be
necessary
to
have
the
change
in
the
encoding
format
yeah.