►
From YouTube: 2020-05-21 :: Ceph Performance Meeting
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
B
A
All
right,
let's
see
what
else
do
we
got
here
this
optimizes,
the
lock
of
booster
rating
process
I,
should
probably
dig
into
that
I,
don't
know
that
anyone
else
is
going
to
and
it
and
I'm
scared
of
the
lock
locking
you
loose
door.
But
theoretically,
that
offers
a
nice
performance
improvement
if
it
actually
is
safe,
I
think
actually
I
can
I
can
punt
on
this
for
a
little
bit,
because
key
food
means
driving
through
pathology,
but
if
it
passes
tooth
ology
then
probably
should
actually
dig
in
and
see.
If
it
really
looks
safe
or
not.
A
A
Lots
of
stuff
with
no
movement
actually
didn't
make
it
through
all
of
them,
but
usually
the
stuff
at
the
end
doesn't
get
updated
anyway.
So
I'm
not
I,
don't
anticipate.
There
was
a
whole
lot
there
that
we
missed
I,
know
closed
PRS
for
performance.
That
I
saw
this
week,
but
it's
a
little
bit
of
a
slow
time
at
the
moment,
though,
not
terribly
surprising.
A
A
A
Than
me,
okay,
no
problem,
so
lots
of
users
are
reporting
that
they're
seeing
buffer
and
on
memory
growth,
a
mailing
list
and
I
think
Adam.
You
are
also
seeing
memory
usage
growth
during
tests
with
compression
enabled,
and
it
looked
to
me
like
in
the
work
that
you
just
did
yesterday
and
today
that
you
were
seeing
that
when
objects
are
pre
created
but
not
yet
fill,
then
the
own
words
memory.
A
C
This
is
incorrect,
though
I
guess
is
my
question
for
you.
Yes,
exactly,
that's
that's
the
issue
and
the
objects
start
with
very
low
overhead,
but
then
the
overhead
grows
very
quickly.
The
other
question
correlated
for
me
for
next
investigation
is:
why
do
we
need
about
one
megabyte
to
store
metadata
for
32
megabytes
object.
That's
like
crazy
stuff,
but
I'm
working
on
this
now.
A
A
A
Cool,
so
that's
very
excellent
work
Adam,
because
this
is
a
very,
very
good
to
find
out,
though
so
my
atom
you
have
you
have
your
proposed
fix,
that's
very
simple
and
very,
very
convenient
just
to
do.
The
trim
in
the
mempool
thread
I'm
a
little
nervous
about
doing
it.
There
that's
kind
of
what
we
used
to
do.
Is
we
trimmed
periodically
in
the
mempool
thread?
A
Potentially,
that's
maybe
why
we
didn't
used
to
see
it,
and
now
we
trim
every
time
a
new
thing
is
entered
into
the
O
node
cache
and
that
lets
us
trim
when
we
already
are
holding
the
lock
and
also
trim
in
the
different
T
POS
DTP
threads,
rather
than
in
a
single
thread
like
that.
The
the
reason
I'm
a
little
nervous
about
doing
the
trim
in
the
mempool
thread
is
one
I
think
that
the
way
that
your
PR
would
implement
it.
A
They
would
get
rid
of
it
eventually,
but
but
you
know
you
can
still
have
temporary
spikes
and
then
the
other
thing
I'm
a
little
bit
nervous
about
is
that
if
we
were
to
increase
the
frequency
of
that,
we
might
be
back
in
the
same
situation.
We
were
in
before,
where
we
are
grabbing
that
lock
and
doing
those
trims
very
often
and
it's
it
impacts
performance.
So
the.
C
First
thing:
I,
guess
that
what
you
say
is
actually
incorrect
because,
okay,
if
you
add
some
object
to
conquer
shart,
then
this
valley
marks
will
be
tested
and
that
dream
cannot
cure.
The
problem
stems
from
that.
There
was
no
traffic
on
Ahnold
cash
shots
and
then
they
just
stayed
big
and
I
mean
bigger
in
a
size
bar,
but
still
the
same
amount
of
objects.
So
that
is
the
case.
I
mean,
but
that
may
be
a
problem
with
efficiency
when
doing
periodic
and
trimming.
C
A
A
A
C
Does
that
make
sense?
Well,
yes,
it
makes
sense,
but
this
defeats
the
original
purpose
not
to
actually
touch
all
notes.
When
you
only
operate
on
data,
so
you
would
have
done
if
you
operating
on
like
shared
cache
data
cache
buffer
cache,
then
you
will
have
to
jump
suddenly.
Oh
no,
and
do
some
poking
there
I
guess.
C
A
Yeah
I
haven't
I,
haven't
looked
at
where
it
would
be
convenient
to
do
something
like
that.
They
you've
probably
looked
at
more
recently
than
I,
have
no,
but
if,
if
we
have
an
opportunity
to
do
something
along
those
line,
I
think
it
might
be
worth
crime
as
opposed
to
doing
in
the
men
pool
throughout
the
like
this.
But
we
you
can
see.
Okay,.
A
C
A
Okay,
so
the
other
topic
I
want
to
bring
up
today
is
this.
Is
this
is
good
good
information
I
think
so.
There's
been
a
lot
of
discussion
right
now
inside
Red
Hat
about
whether
to
run
multiple
OS
D's
on
one
device
and
if
we
should
be
building
tools
for
doing
partitioning
in
such
a
way
to
make
that
very
convenient
for
an
end
user
and
all
these
kinds
of
things.
So
one
question
that
came
up
is
whether
or
not
it's
really
beneficial
to
do
this
still
so
over
the
last
weekend,
I
ran
a
number
of
tests.
A
In
Nautilus
we
saw
a
very
significant
game
with
running
multiple
OS
DS
on
an
nvme
device,
two
of
these
and
one
nvv
device
and
all
the
tests.
It
was
better,
but
the
results
range
from
10%
better
up
to
around
one
case,
80%
better
for
random,
read
small
random
reads
in
octopus
in
master
it's
much
more
mixed.
A
Our
single
OST
case
is
much
faster
than
it
used
to
be
to
the
point
where
in
many
cases
now
you're
actually
in
these
tests
anyway,
it
was
faster
to
run
a
single
OST
on
an
nvme
device
than
it
was
to
run
multiple
hosts
he's
on
an
nvme
device,
and
there
are
a
couple
of
reasons
for
that.
I
think
the
cache
refactor
from
last
summer
I
think
played
a
big
role
in
some
of
this
and
then
bhajiya
Peng
had
a
very
good
PR
in
master.
A
That
is
very
simple,
but
it
we
do
notify
all
instead
of
notify
one
in
the
Shara
dock
work.
You
I
think
that's
a
large
part
of
the
reason
why
a
small
random
read
performance
continued
to
increase
and
master
the
point
now
where
it's
is
faster
with
one
OSD
vs.,
2
OS
T's.
We
previously
just
were
not
making
good
use
of
all
the
different
threads
we
weren't
breaking
them
all
up.
A
We
were
only
waiting
up
one,
so
it's
is
interesting,
I'm
getting
a
feedback
from
from
from
Intel
that
they're
not
seeing
the
same
kind
of
improvement
that
we
are,
though
we
do
need
to
do
some
more
investigation
into
why
there
they're,
seeing
regression
actually
with
both
octopus,
but
at
least
here
I'm
fairly,
confident
that
these
results
are
real.
So
the
good
news
is
that
I
think
we
have
made
some
some
real
improvements
here.
D
Okay,
good,
so
this
is
danny
abdul
kalam,
I've
just
joined
midway
through
this
call,
the
first
time
I
joined
so
I'm,
probably
missing
quite
a
lot
of
context,
but
these
test
results
look
really
interesting,
I
was
wondering:
is
there
a
somewhere
online
I
could
find
kind
of
documentation
or
information
about
the
process.
You
took
all
the
tools
you
used
to
run
these
tests
and
bench
punks.
A
Sure
I
can
I
can
tell
you
this
was
run
using
CBT,
so
that's
kind
of
our
general
test
framework
for
running
benchmarks
that
we
use
anyway
and
CBT
can
run
a
couple
of
different
benchmarks.
In
this
case
it
was
fi.
Oh
I,
already
familiar
with
a
file
at
all.
I
am.
A
Okay,
so
all
CBT
really
is
is
a
tool
that
then
goes
through
and
runs
a
combination
of
different
benchmarks
with
different
parameters,
so
you
defined
in
the
animal
and
of
what
what
different
parameters
you
want
to
run
with,
and
that
will
run
through
a
bunch
of
different
tests.
That
way,
you
know
if
I
already
can
do
some
of
this
itself,
but
DBT
is
kind
of
a
more
generic
way
to
do
this
across
different
benchmarks
around
multiple
benchmarks.
A
A
So
there's
two
different
things
you
can
do:
you
can
DVT
can
launch
and
run
cause
bench,
but
cause
bench
is
really
almost
its
own
framework
right,
it's
very
very
complex,
and
it
does
a
lot.
So
some
people
have
done
that
and
that
can
work,
although
I
think
a
lot
of
people
will
just
run
cause
bench
on
its
own
is
also
totally
fine.
I
also
wrote
a
benchmark
called
hot
sauce.
A
A
D
A
A
That
the
original
authors
have
moved
on
to
other
projects,
so
it's
you
know
I
think
I,
don't
know
how
much
maintenance
is
getting
these
days
so
yeah,
if
you
feel
free
to
you,
know,
try
CBT.
If
you,
if
you'd
like
you
know
it's
not
the
most
documented
tool
in
the
world,
but
but
you
know
people
are
actually
the
the
crimson
team
has
now
been
using
it
for
or
mainly
regression
testing.
So
it's
getting
a
little
bit
more
love
from
from
multiple
people
now,
so
hopefully
we
can
make
the
documentation,
look
better
cool
yeah.
A
A
E
Okay,
so
just
a
general
video,
very
you
for
everybody.
E
Yesterday
we
got
a
couple
of
complaints,
have
users
million
least
about
data
corruption
in
right,
ahead,
lock
of
frogs,
DB
and
also
the
well.
Both
cases
where
happened
after
upgrade
from
octopus
point
1
to
point
2
and
at
the
same
time,
neha
reported
a
similar
issue
in
iran,
master
and
finally,
I
managed
to
reproduce
at
local
as
well
and
after
some
investigation,
which
is
not
completed
yet
but
looks
like
two
recent
modification
triggered
this
issue.
E
E
E
A
E
As
I
said
that
these
are
preliminary
results,
but
I
can't
explain
why
I
am
getting
32k
of
zeros
instead
of
valid
data
in
the
local
III
checked.
If
we
pass
Ron
buffer-
and
it's
like
that,
but
it's
not
the
case,
but
definitely
I
can
see
multiple
running
iOS
for
each
reproduction.
So
III
think
that's
the
case,
but
I
haven't
proved
that
100%
yet
sure.
A
A
So
so
they're,
the
original
reason
why
we
disabled
blue
store,
buffered
I/o,
was
because
it
was
causing
significant
problems
with
rgw
performance,
where
we
were
actually
seeing
that
we
were
digging
into
swap,
oddly,
even
when
there
was
significant
memory
available,
and
we
didn't
really
understand
why
so
we
switched
back
Igor.
Do
you
think
we
should
revert
that
change
and
go
back
to
an
overnight
on
no.
E
A
E
C
This
is
true:
I
have
been
touching
a
feeling.
I
would
have
to
revise
it
because
I
don't
remem,
remember
correctly,
but
it
came
out
as
a
fix
for
problems.
I
had
when
actually
multiple
times
changing
logs
DB
sharding
on
the
same
volume.
Without
that
tree,
extending
it
almost
always
failed.
So
I'm
I
would
be
cautious
to
disable
that
that
part.
E
A
C
There
is
an
Inc
was
that
I
had
a
press
test,
that
changed
has
been
changing,
rocks,
DB
charting
schema,
and
the
problem
was
that
if
device
was
not
fully
realized,
two
zeros
just
reused
as
it
was
sometimes
after
restart
blue
store.
Basically,
Rob's
DB
couldn't
pick
up
it's
right
ahead
logs
and
continuing
ed
Aaron's
there.
So
that
was
the
reasoning,
but
never
I
could
repeat
the
problem.
If
I
had
always,
he
ought
in
all
tests
the
same
set
of
column
families.
E
C
The
original
reasoning
for
that
was
that
it
it
was
to
speed
up
to
or
meet
adjusting
size
with
each
right
to
new
right,
a
headlock,
so
we
had
a
larger
right,
a
headlock
and
just
appended
the
data,
never
bothering
to
update
blue
FS
log
that
contained
metadata
with
size,
so
that
that
way
we
were
a
bit
a
bit
faster.
That
way,
we
make
it
more
alike
for
as
its
what
in
normal
drugs
to
be
operation,
when
Rob's
DB
rotates
reuses
previous
right,
ahead
locks
and
just
appends
data
from
from
the
beginning.