►
From YouTube: 2018-FEB-15 :: Ceph Performance Weekly
Description
Weekly collaboration call of all community members working on Ceph performance.
http://ceph.com/performance
For full notes and video recording archive visit:
http://pad.ceph.com/p/performance_weekly
A
I'll,
maybe
just
give
folks,
maybe
a
minute
more
here
to
gather
and
then
we
can
get
started.
I
have
actually
been
on
pto
for
the
last
couple
weeks,
so
as
well,
I
haven't
been
here
and
I
have
to
apologize
a
little
bit.
I
I
did
not
make
it
very
well
through
the
the
backlog
of
performance,
pull
requests
that
have
been
updated
over
the
last
couple
of
weeks.
A
So
I'm
going
to
have
to
finish
that
up
today,
but
I've
got
some
of
it
done
here,
but
but
there's
a
whole
lot
of
new
stuff
that
I
don't
have
in
here.
So
maybe,
instead
of
specifically
going
through
these
pull
requests
right
now,
I
will
mention
some
of
the
things
that
have
been
going
on
over
the
last
couple
of
weeks,
while
I've
been
gone,
Retta
slavs
been
making
a
lot
of
progress
on
doing
some
performance
testing
on
the
read
side
and
uncovered
a
couple
of
different
issues.
A
The
the
big
one
that
that
he's
actually
already
got
a
PR
emerged
for
was
in
the
the
hot
path.
We
were
accessing
configuration
options
through
MD,
config
T,
and
it
turns
out
this
is
actually
really
slow.
I
was
actually
seeing
it
in
some
testing
that
I
was
doing
as
well,
while
I
was
in
PT.
Oh,
it's,
it's
pretty
bad,
so
he's
actually
got
a
PR
in
place
already
that
that
caches
or
makes
it
possible
to
cache
these
options,
and
it
helps
dramatically
it's
really
good.
A
A
A
So
the
good
news
here
is
that
it's
not
too
hard
to
fix
and
there's
actually
I've
got
a
PR
for
fixing
it
and
then
David's
got
another
PR
for
fixing
it.
That
works
just
a
little
bit
differently,
so
hopefully
soon
we'll
have
that
fixed
and
that
that
also
should
reduce
CPU
consumption
in
the
OSD,
at
least
by
a
little
bit.
A
There
was
using
vector
based
objects
rather
than
bufferless,
and
that
was
dramatically
faster
and
actually
really
easy
to
implement.
It
was
surprising,
just
kind
of
how
how
much
better
it
was
so
using
that
and
then
also
applying
some
of
these
other
performance
fixes
it
kind
of
gave
a
good
insight
into
what
the
the
bottlenecks
in
in
the
OSD
are,
and
the
the
really
big
thing
that
that
stuck
out
is
pg
log.
A
It's
it's
pretty
pretty
high
overhead
as
an
example
removing
in
say
the
the
replicated
back-end
the
call
to
log
operation,
commenting
just
commenting
that
out
and
then
fixing
the
asserts
that
that
get
thrown
because
of
it.
That
lowers
on
on
my
my
dev
box,
the
CPU
consumption
of
the
OSD,
when
using
an
in-memory
vector
based
in-memory,
store.
A
It
removes
the
the
CPU
consumption
from
like
five
cores
down
to
about
three
cores.
This
is
to
get
about
between
my
40
and
50,000.
Write
apps
for
four
K
random
writes,
so
it's
it's
really
significant.
Just
just
doing
PG
log
is
not
quite
half
of
the
CPU
consumption
and
also
when
you
look
at
with
blue
store,
PG
log
updates
are
really
significant
in
terms
of
how
much
work
they're,
making
the
K
V
store
do
so.
A
This
is
just
really
reinforced
for
me
personally.
That
kind
of,
in
addition
to
the
the
the
work
that
we're
looking
at
for
moving
over
to
something
like
C
star,
we
also
really
need
to
be
looking
at
PG
log
and
whether
or
not
we
can
do
any
of
it
better.
I
think
the
the
kind
of
idea
of
using
/
PG
ring
buffers
is
really
good,
at
least
on
something
like
nvme,
where
we
we
don't
care
as
much
about
contiguous
rights
as
we
do
about
just
lots
of
parallelism
and
reducing
the
amount
of.
A
Work
that
we
make
rocks
TBH
do
in
terms
of
kind
of
just
record-keeping
and
then
also
maybe
the
complexity
of
the
PG
log
code
itself.
So
anyway,
that
that
was
kind
of
where
I
wrapped
up
with
kind
of
that
that
project
there
were
some
other
things,
there's
a
fair
amount
of
encoding
and
decoding
overhead.
That
kind
of
crops
up
once
you
get
other
stuff
out
of
the
way
for
object,
info
tea
and
a
couple
of
other
data
types,
so
probably
improving.
A
That
may
be
using
the
same
mechanisms
that
we
did
in
blue
store
to
avoid
appends
to
the
to
the
buffers.
This
is
probably
good
and
then
PG
info.
There
were
some
in
overhead
with
that
as
well,
but
that's
that's
kind
of
getting
more
into
then
the
the
higher
hanging
fruit,
I
guess
so
yeah,
that's
that's
kind
of
what
I've
got
for
the
past
couple
of
weeks.
They've
been
going
on.
I,
don't
know:
Rattus
love,
and/or
Adam.
A
B
Guess
it
would
be
to
talk
about
some,
our
debugging
facilities.
We
have
in
OSD
that
are
affecting
them
the
main
path.
For
instance,
we
have
our
apps.
We
have
traction
over-over
mutexes,
it's
it's
responsible
for
Lok,
Lak,
Deb,
it's
responsible
for
for
counting
and
providing
a
lot
of
our
authors
with
methods.
Cult
is
locked
by
me.
B
B
B
B
B
What
is
interesting
is
when
I'm,
when
I'm
running
path.
What
what
is
interesting
is
the
difference
between
mutex
mutex,
sorry
come
on
/
mutex
and
the
p-fleet
call
to
lock
to
actually
lock
them
in
the
underlying
low-level
mutex
and
can
be
significant,
especially
in
especially
on
on
V
start
when
logged
up
is
enabled.
However,
even
after
this
implicit,
it's
still
not
zero.
A
A
A
A
A
B
A
A
B
B
Moreover,
it's
turned
by
default:
it
turn
it
student
on
almost
for
our
all
cases.
There
is
no
something
like
the
lock
DEP
to
to
disable
those
tracking
globally
nope,
it's
possible
only
by
passing
additional
additional
parameter
on
construction
that
it
seems
that
most
of
the
of
the
users
don't
do
that.
B
A
A
A
A
B
It's
quirky,
it
can
definitely
can
be
quickly
because
we
have
conditional
there
based
on
time,
so
it's
it
could
be
dad
that
profiling
effects
the
results
as
well.
However,
I
I
started
digging
their
way
in
with
perv
and
some
reduced
frequencies.
Something
frequency
and
still
some
overhead
is
clearly
visible.
There
are
some
play.
First
of
all,
those
guys
are
running
upper
locking
and
unlocking
mutexes
over
and
over.
There
is
a
call
on
track
table
called
gate,
duration,.
B
That
is
called
frequency
that
it's
called
very
very
frequently
and
unfortunately
takes
a
mutex
and
makes
some
comparisons
of
on
on
member
on
last
member
of
STD
vector.
So
memory
did
the
reference
there.
Also
they
operate.
The
comparison
operator
is
far
away
from
being
optimal.
I
have
some
putters
that
switched
from
STR
cmp
to
to
equality
operator
that
has
early
exit
based
on
size,
miss
math.
Unfortunately
we
are,
we
are
well
sink.
We
are
losing
the
the
information
about
string
size
because
of
taking
raw
because
of
taking
two
pointer
to
see
string.
B
A
B
Lock
DEP
is
disable
system.
I
had
I
had
some
profiling
traces
with
extremely
high
usage.
High
rate
of
CPU
burned,
CPU
cycles
burned
there
because
of
mutex
dance
and
the
mutex
dance
is
extremely
costly.
When
you
have
locked,
DEP
enabled
like
like
in
all
V,
starts
run
without
the
the
no
lock
depth
parameter.
B
B
C
A
A
A
A
B
By
the
way,
the
profiling
shows
that
on
the
right
path
we
have,
we
have
significant
significant
overhead
coming
from
front
end
not
on.
We
are
not
only
bound
by
memory
waiting,
which
is
typical.
We
saw
it
in
in
rocks
DB.
We
saw
it
in
HD
drink
right
path.
We
are
also
constrained
by
front-end
performance
client
of.
A
B
B
A
B
Definitely
see
states
were
disabled
during
that
states
that
the
test,
so,
of
course
it's
it's,
it
varies,
it
could
worry
with,
with
all
will
vary
with
process
of
not
even
microarchitecture,
even
with
with
model
here
one
processor,
one
CPU
has
eight
mega
megabytes
l3
cache
and
we
are
fitting
in
another.
One
have
another
one
has
half
of
it
and
we
are
lost.
A
A
I
think
we're
starting
to
get
there
now.
But
it's
it's
there's,
there's
so
much
this
going
on
right,
it's
hard
to
narrow
it
down.
Yeah.
B
A
Absolutely
absolutely
I
mean
a
lot
of
this.
You
know
in
the
the
hardest
days
or
even
hardest
days
with
with
you
know,
SD
journals
didn't
matter.
It's
now
that
we're
we're
really
starting
to
to
chase
these
high
performance
deployments
that
that
all
of
this
now
is
kind
of
starting
to
really
make
a
big
difference.
C
A
A
A
A
B
B
Okay
got
it
basically,
each
hour,
each
call
to
function
just
just
piece
of
code,
regardless
whether
it's
coming
from
from
shared
library
or
our
translation
unit,
that
is
part
of
stuff
itself,
has
to
go
through
PLT,
which
means
one
call
and
one
jump
both
of
before
they
are
under
ectopy.
There
are
and
direct
jumps
that
are
requiring
resources
from
from
grands
predictions
and
imposing
unnecessary,
unnecessary
pressure
on
front-end.
Maybe
maybe
cache
misses,
maybe
some
linking
something
like
that,
but
we
are
paying
the
cost,
but
is
the
reason?
Is
there
any
good
reason
for
that.
E
D
E
E
No
not
really
at
all
the
infrastructure
is
already
there.
It
just
needs
tweaking
with
the
global
options
to
disable
some
Global's
unable,
locales
and
add
dependencies
for
actually
the
pieces
that
are
used
for
a
short
libraries
just
to
point
them
to
be
position.
Independent
code,
just
not
think
nothing,
difficult,
okay,
I
was
gonna,
say
I
can.
A
A
Yeah
I
can
I
can
try
it
out
since
right
now.
This
this
version
of
pet
store,
I
guess
is-
is
pretty
good
at
uncovering
CPU
related
issues,
or
at
least
it
has
been
so
far
for
me,
are
you
testing
it?
We
just
with
like
blue
store
on
in
Sirte
or
something
yep.
E
A
A
A
A
This
is
just
the
stuff,
the
the
bass
one
that
I
applied.
Those
things
to
is
actually
based
on
the
new
store
revival,
branch
that
I
had
so
at
some
point.
This
needs
to
be
rebased
and
a
master
and
everything,
but
you
can
at
least
see
pet
store
in
there,
which
is
it's
not
it's
not
very
interesting.
It's
just
like
a
mem
store
really,
what's
with
some
some
tweaks.
A
A
D
A
A
B
A
A
All
right,
well,
yeah,
I,
don't
have
anything
else
this
week.
There's
anything
else,
guys.
B
Underscore
underscore
attributes
underscore
and
underscore
that
are
implemented
solely
in
DCC.
We
have
support
for
insulin,
but
they
provides
nice
functionality.
I
mean
here,
particularly
the
cult,
and
the
cult
attribute
allows
you
to
to
request
compiler
to
put
some
code
far
away
from
your
hot
path.
I.