►
From YouTube: 2018-May-31 :: Ceph Performance Weekly
Description
Weekly collaboration call of all community members working on Ceph performance.
http://ceph.com/performance
A
B
B
A
A
B
C
A
So
my
gosh
I
guess
isn't
here,
but
my
my
personal
take
on
this
is
I.
Don't
think
it
really
matters
that
much
I
mean
I
think
it
may
be
he's
gonna
marginally
help
in
situations
where
you
end
up
with
a
bunch
of
like
stale
stuff
in
a
high
level
that
you
don't
really
need
stuff
to
be
in
it
could
be
in
like
level
three,
even
though
it's
stuck
in
like
level
four
for
a
while.
So
you
get
some
additional
rate,
amplification,
I,
don't
I,
don't
know.
I
could
be
totally
wrong
by
kind
of
question
there.
A
D
A
C
D
C
C
C
A
One
thing
I
would
like
to
bring
up
with
the
the
my
PR
is
that
it
was
failing
these
ceftaz
tests
and
I
have
no
idea.
Why
and
I
still
have
no
idea
why
the
changes
I
made
actually
fix
that,
like
it's
now
passing
the
big
check
and
I,
don't
know
why
I
mean
I
know
I
fixed
a
bug
that
was
like
a
really
weird
bug
by
I
I
spotted
after
seeing
those
but
I
have
no
idea
why
I
fix
it.
Yeah.
C
The
atom
review
to
blow
the
fess
stuff
I've
changed
I
would
like
for
this
one
yet,
but
it
looks
like
he
liked
it,
that's
good!
That's
what
marketers
needs
QA
and
the
iOS
Rattler
Eric
extracted,
as
he
finished
extracting
I
guess,
planning
on
extracting
this
out
from
all
the
other
team,
plucks
stuff.
C
C
C
C
C
A
That's
just
me
making
the
assertion
that
someday,
we'll
be
able
to
say
here's
how
much
memory
the
OST
should
use
and
try
to
stick
to
it.
So
yeah
all
of
this
memory
like
work
fed
in
the
last
like
weeka
or
week,
or
two
that
I've
been
looking
at,
of
how
much
memory
we
use
and
then
also
kind
of
this
work
towards
dynamically
adjusting
memory,
use
or
memory
pool
assignments
based
on
priorities.
I
think
all
that
kind
of
confluence
into
this
storm
of
you
know
really.
A
What
we
want
is
to
say
yeah
try
to
stick
to
this
I,
don't
think
we
can
guarantee
it,
but
we
can
at
least
say
try
to
stick
to
this
memory
usage
kind
of
adjust
things
to
decrease
until
we're
close
to
that
memory,
usage
or
right
around
it,
and
then
you
know
do
our
best
I
think
we
can
do
that,
though
I
think
we
can.
We
can
we've
maybe
got
enough
data
to
be
able
to
say
you
know:
here's
what
we
want
to
target.
A
C
There's
gonna
be
a
point
where
we
add
a
configuration,
often
that's
like
listing
memory
and
then
everything
else
sort
of
is
subsumed
by
that,
but
we're
not
I,
don't
think
we're
there.
Yet
you
know,
don't
you
know
I'm
not
pieces
in
place.
The
one
thing
that
I
think
in
the
short-term
we
do
need
that
I
was
planning
on
doing
directly
on
top
of
your
pull
request
systems.
That
merges
is
just
like
adjusts,
based
on
the
observed
overhead
that
the
allocator
seems
to
be
adding
in
and
I'm
not
sure
how
to
do
that.
C
A
C
Didn't
I
tried
to
avoid
the
heat
profile
because
I
thought
that
was
an
optional
thing
that
we
wouldn't
to
turn
on,
probably,
and
so
the
interfaces
I
found
were
Mallen,
foe
and.
C
It's
basically
there's
a
mouse
stats,
I'm
out
TC
Malik
stats,
which
prints
everything
to
standard
out.
So
it's
absolutely
useless
and
there's
a
TC
male
info,
which
returns
this
mal
info
structure,
which
has
like
ten
fields
with
like
really
confusing
names.
And
then,
if
I
remember
correctly,
there's
this
comment
that
says
it's
only
for
arena
zero
and
they're
like
the
50
reais,
or
something
like
that.
So.
D
C
A
Option
that
we
have
is
given
that
all
ready
for
the
the
cache
balancing
we
do
this
on
a
really
coarse,
bein,
grace
a
course
based
course
basis
it.
It
might
not
be
terrible
to
just
grab
a
heap
profile
every
5
seconds
serve.
You
know
whatever
right
and
and
then
use
that
to
figure
out
if
we're
close,
I.
A
A
E
A
I'll
I'll
try
to
follow
up
with
Adam
you
tomorrow
or
on
Monday
if
he
said
any
progress
and
looking
at
this
stuff,
yeah.
Okay,
the
other
thing
that
would
be
really
useful
to
get
is
fragmentation.
Information
right.
How
fragmented
has
memory
become,
because
we
don't
even
know
that
yet
we
suspect,
but
yeah.
C
A
Also
want
to
design
an
interface
for
this
that
can
very
reasonably
take
into
account
scaling
factors
for
different
things
right
like
when
we
assign
priorities,
if,
if
allocating
50
megabytes
to
something
really
results
in
a
80
megabyte
allocation,
where
is
for
something
else,
Falcon
50
megabytes
results
in
a
55
megabyte
allegation.
The
priority
based
mechanism
for
for
dealing
with
this
stuff
should
be
able
to
kind
of
take
that
into
account
when,
when
assigning
memory
for
different
things,.
C
A
E
E
E
C
E
C
C
C
A
A
A
E
C
Okay,
well,
it
sounds
like
that's
the
next
step.
Well,
I
think
I
think
we
just
Monica.
We
want
to
get
marks
over
crossed,
tested
and
merged
yep
and
then,
on
top
of
that,
add
in
something
that
looks
at
these
T's
Malik
stats
and
based
on
the
disparity
between
what
it's
has
claimed
from
the
OS
and
what
it's
actually
allocated.
There's
a
30%
difference,
it'll
scale
the
boost
or
cache
size,
whatever
yep
based
on
that
I.
Think
that's
what
we
want.
C
Yeah
I
think
the
only
question
is
if
it's
above
some
threshold-
and
we
haven't
done
it
in
a
while-
should
we
do
like
the
automatically
do
the
TC
Malik
keep
free.
Whatever
thing
I
know
that
we
used
to
do
that
in
like
production
situations.
Where
remember,
we
got
out
of
control
and
we
would
have
to
like
poke
it
and
then
that
sort
of
seems
to
not
be
necessary
in
newer
versions
of
TC
Malik,
but
I
never
was
involved
in
those
support
cases
genome.
Do
you
need
to
remember
this
Craig
I
do.
E
E
C
A
Did
I
did
a
I
did
a
test
recently
with
the
the
rgw
case,
which
was
where
the
OST
was
using
the
most
memory,
and
in
that
case
it
was
about
10%.
It
was
like
six
and
a
half
for
seven
gigs
of
OS
s,
RSS
memory
usage
or
OST
RSS
memory
usage,
and
it
was
like
600
megabytes
of
of
you
know,
memory
that
could
be
freed
back
to
the
US.
A
A
One
thing
that
we
that
I
want
to
do
is
add,
like
PG
log
memory,
that's
pinned
into
the
pinned
memory
in
the
the
cache
balancer,
so
kind
of
insert,
including
things
like
that.
Like
some
of
the
memory
that's
allocated
is
going
to
be
reserved
for
PG
log
or
it's
going
to
be
reserved
like
for
other
things
like
right
now.
One
of
the
things
that's
reserved
is
memory
for
indexes
and
filters
in
rocks,
TB
mm-hmm.
A
So
so
there's
have
this
priority:
zero
level
of
reserved
space
for
things,
and
if
we
can't
hit
it,
then
we
actually
assert
right
now.
Initially
by
saying
you
know,
we
can't
support
this.
It
doesn't
for
or
rocks
B's
indexes
and
filters
at
that
point
it
will.
It
will
try
to,
but
it
won't
successfully.
A
It
might
make
sense
that
we
say
the
user
specifies.
I
can
only
have
one
gig
of
memory
for
the
OSD
and
they
specify
enough
PGS
per
OSD
that
they're
gonna
exceed
that
up
front.
We
might
actually,
in
that
case
one
has
a
certain
tell
them
you,
you
don't
have
enough
memory
assigned
to
do
this,
do
something
to
tell
them
yeah.
C
D
I've
just
got
one
small
thing:
yep
have
done.
Was
there
any
change
in
the
locking
during
scrubbing,
between
jaw
and
luminous,
only
asking
because
I
upgraded
a
cluster
to
luminous
and
when
I've
got
the
scrubbing
going
on
now
sort
of
a
radars
been
where
I
was
getting
about
one
or
two
millisecond
average
right
latency
I'm,
now
seeing
it
in
the
sort
of
30
millisecond
range
and
putting
in
scrub,
sleeps
and
stuff
like
that
seems
to
help
bring
it
down.
D
C
A
Do
you
know
which
lock
or
which
lock
sir
be
involved?
I.
D
Haven't
managed
to
look
into
that
deeply.
It
was
just
sort
of
one
of
the
last
clusters
aren't
graded
to
to
luminous,
which
is
where
we've
got
sort
of
probably
the
most
monitoring
one
I
just
son,
you
saw
we've
got
a
collect
Dijon,
which
just
does
a
radio
spends
every
60
seconds
and
something
noticed
that
went
from
sort
of
sitting
about
one
millisecond
up
to
about
thirty
post
upgrade
and.
D
So
if
I
do,
it
might
also
be
normal
scrub.
I,
don't
think
it
had
is
large
effect,
but
it
was.
It
was
still
there.
But
if,
if
I
run
like
a
Pheo
on
a
RBD
and
I
did
fork
a
sequential
writes,
though
it's
spending
a
lot
of
time
per
RPG
effectively
and
you
would
see
when
it
hits
one
of
the
affected
PG.
Suddenly
it
would
just
go
to
almost
like
0
io
for
a
couple
of
seconds
and
then,
as
it
gets
to
the
next
one
over
it,
then
spikes
back
up.
C
E
C
Yeah
there's
a
major
exchange
that
happened
after
luminous
fur
mimic
where,
if
you're
scrubbing
an
object
in
an
aisle
comes
along
instead
of
the
aisle
waiting
for
the
scrub
to
finish
the
iowa
preempts
crab
and
the
scrub
will
get
repeated
after
it
should
have
a
significantly
reduced
rias
latency.
But
we
didn't
actually
back
part
that
too
luminous,
though
I'd
be
kind
of
curious.
What
you
would
see
if
you
went
to
mimic
that
cluster,
but
since
you're
just
Ford
alumina.
That's
why?
Okay
for
a
while
yeah.
D
C
Go
ahead:
there's
the
I
think
you
can
do
clear
historic
ops
on
an
OSD
and
then
maybe
do
at
the
bench
and
get
some
some
of
those
sort
of
emails.
I
couldn't
rights
and
then
look
dump
them.
It
might
be,
there
might
be
a
clue
and
the
dump
the
store
cups
will
show.
Basically,
your
longtail
request,
whatever
that
might
tell
us
something:
okay,.
A
A
D
Okay,
I
mean
I,
assume
from
a
very
high
level.
What's
happening.
Is
that
this
process
of
7.2
K
disks
is
that
it's
probably
trying
to
read
like
a
size
of
data
putting
in
the
scrubber
know
maybe
four
mega,
something
which
is
probably
taken:
yeah,
40,
50
milliseconds
and
that's
just
basically
holding
all
the
ir
until
that
goes
through
I.
Don't
know
why
that
would
be
any
different
to
what
Jill
was
doing,
but.
C
B
D
Yeah
I
mean
it
sounds
like
pretty
where
it's
doing
now,
it
would
be
the
correct
what
you
would
expect
it.
We
learn
that
you
wouldn't
want
the
data,
possibly
changing
in
that
PG
y.
Whilst
it's
checking
it
I
guess
any
problem
is
the
dependent
on
the
amount
of
data
you
you
read:
you're
gonna.
Have
that
be
look
the
the
header
from
the
disk
for
that
amount
of
time.
D
C
Yeah
I
don't
have
a
good
answer:
I'm
guessing
that
that's
some
variations,
that
is
what's
happened
and
that
mimics
should
mitigate
it
by
pre-empting
that
letting
you
prep
those
operations
but
I
think
I,
don't
know
yeah.
If
you
can,
if
you
can
reproduce
on
another
on
another
environment
that
we
can
play
with,
that
would
be
okay.