►
From YouTube: Ceph Performance Meeting 2021-09-11
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
All
right,
let's
see,
I'm
just
gonna,
start
doing
this
and
maybe
they
they
join
when
we
actually
get
to
the
discussion
topics.
So,
okay,
let's
see
this
week
closed
pr's
there
was
a
prometheus
vr
that
just
lets
you
disable
the
cache.
I
think
that's
pretty
simple,
so
that
merged.
A
A
I'm
sure
it
was
basically
to
eliminate
some
kind
of
corner
case,
though
yeah
we
shouldn't
use
single
per
oh
no
transaction
for
such
an
upgrade.
When
oh,
who
knows.
Oh
map
list,
is
huge
yeah
high
memory,
consumption,
okay,
so
it's
basically
just
making
it
so
that
we
don't
end
up
consuming
tons
of
memory
when
there
are
like
huge,
oh
map
lists
in
a
particular
node.
A
Or
on
a
trend,
single
pronoun
and
transaction
okay,
so
splits
it
up:
okay,
yep!
So,
basically,
just
not
letting
you
blow
up
the
osd
when
things
are
awful,
all
right
in
the
mds
there's
a
pr
that
merged
that
switches,
the
mdl
mds
lock
to
a
fair
mutex
to
avoid
starvation
issues.
A
So,
okay,
that
looks
good.
A
bunch
of
updated
prs
this
one
from
adam.
We'll
talk
about
that.
I
think
a
little
bit
later
m
is
not
going
to
be
able
to
make
the
meeting,
but
the
gist
of
it
is.
Is
that
this
pr
does
direct,
I
o
for,
writes
and
then
buffered
io
for
reads,
and
so
the
the
concern
there
the
benefit
there
first,
is
that
it's
it's
kind
of
giving
us
the
best
of
both
worlds
in
terms
of
what
we've
seen
for
performance
potentially.
A
But
the
concern
is
that
this
could
introduce
consistency
issues
if
you're
trying
to
do
things
like
read
from
cash.
Well,
simultaneously,
you
have
direct,
I
o
rights
going
on,
so
that
would
need
to
be
very
carefully
audited.
I
think,
okay,
what
else.
A
Osd
compression
bypass
after
rgw
compression
looks
like
that
made
it
into
eric's
testing
branch,
so
maybe
that's
possibly
even
converge
if
it
passes
more
ongoing
work
with
this
ttl
cache
implementation
for
the
manager
module.
I
think
they
had
another
review.
A
A
Optimized
pg
removal,
pr
that
had
failed
tests
at
one
point
and
I
think
it
just
got
another
review.
I
don't
remember
who
looked
at
it
recently,
but
that
was
reviewed
again,
so
more
ongoing
work.
There.
A
The
ceph
messenger
header,
2
decoding
optimization,
that's
gotten,
updates
ilia
previously
had
reviewed
it.
I
think
there
were
test
failures
at
one
point,
not
sure
if
that's
still
the
case
or
not,
and
then
the
other
big
one
for
me,
the
one
I've
I've
been
excited
about
for
a
long
time
is
this
mds
remove
subtree
map
from
the
journal.
I
wasn't
sure
if
that
would
just
languish,
but
it
looks
like
zhang
has
updated
it
again,
so
really
yeah.
A
He
did
I'm
not
sure
exactly
what
he
did
there,
maybe
just
made
it
rebased
it
to
this
current,
but
but
he's
been
actively
periodically
keeping
that
up
to
date,
so
hopefully
we'll
be
able
to
get
that
in.
That
looks
like
a
really
good
pr
if
it
works,
and
then
that
was
it
for
updated
prs
that
I
have
is
there
anything
I
missed
or
anything
anyone
wants
to
discuss
related
to
those
prs.
A
All
right
moving
on
then
so
I've
presented
some
of
this
data
to
the
core
team
already,
but
I'll
share
it
here
as
well.
A
A
A
We
had
been
worried
that
there
were
some
m
clock.
Changes
for
qos
that
were
a
little
concerned
may
have
introduced
regression,
but
it
doesn't
really
seem
so
maybe
a
little
bit.
There
was
some
fluctuation
that
I
didn't
really
date
too
much
deeply
into,
but
it's
not
clear.
There
was
anything
other
than
just
random
variation,
the
real
big
source
of
regression.
Well,
there
are
two
that
we
saw
one
we
already
knew
about
and
fixed
that
was
a
7e
3
ece
online
16..
That
was
a
change
that
was
introduced
that
mark
hogan
actually
discovered.
A
First,
where
we
were,
we
ended
up
basically
doing
debug
builds
by
default
and
that
that
was
a
real
real
nasty
impact,
but
that
got
fixed,
so
that
was
no
longer
a
problem.
The
the
other
change
that
we
kind
of
suspected
and
also
kind
of
already
knew
would
not
potentially
cause
some
regression
in
this
kind
of
workload
was
when
we
switched
to
using
blue,
has
buffered
io
true
by
default
again,
we've
gone
back
and
forth
on
this
a
couple
of
times.
A
The
reason
that
we
switched
that
back
to
being
true
again,
is
that
for
some
reason
that
we
still
don't
totally
understand,
roxdb
will
not
read
blocks
that
you've
read
during
iteration,
like
iteration
of
our
collection
recently
from
cache.
For
some
reason,
it
seems
to
go
back
to
disk
again
when
we
have
this
kind
of
pattern.
It
may
be
that
we
did
something
like
deleting
something
that
caused
invalidation
of
the
cache.
I've
seen
that
in
rocks
tv
before,
where
potentially
like
you
do
a
compaction
and
the
entire
cache
is
invalidated.
A
It's
kind
of
weird
this,
maybe
something
like
that
is
going
on
or
maybe
just
their
behavior
is-
is
kind
of
broken,
but
what
we've
seen
is
that
roxdb
will
often
maybe
always
go
back
to
disk
when
you
reiterate
over
something
that
you
just
irrigated
over
rather
than
using
the
block
cache.
A
So
what
happens
is
that
the
buffer
cache
ends
up
doing
all
the
heavy
lifting
but
uplifting
it
will.
If
you
do
direct,
I
o
you'll
end
up
just
doing
all
of
your
reads
from
disk,
but
if
you
have
a
significant
amount
of
buffer
cache,
then
you'll
end
up
doing
those
those
reads
from
memory
instead.
A
So
that's
why
we
made
the
change,
because
people
were
noticing
this
in
for
for
real
deployments
during
you
know
any
kind
of
operation
that
would
require
collection,
listing
or
deletion
or
other
things,
and
it
was
bad.
We
also
were
sort
of
able
to
replicate
this
ourselves
through
a
benchmark.
I
wrote
called
omapbench,
that's
just
part
of
well,
it's
not
yet.
A
This
is
there's
a
pr4,
but
we've
merged
it
part
of
the
the
make
test
or
the
g
test
suite,
and
you
can
see
some
of
those
results
here:
lines
o
and
p
where,
with
buffered
io
we're
quite
fast
across
the
board
and
in
column
p.
When
we
have
direct
io
on,
you
can
see
that
oh
map
get
and
omap
remove
the
time
it
took
to
do
those
tests
increased
fairly
dramatically.
A
So
all
this
is
to
say
that
right
now
we're
in
kind
of
a
hard
place
where,
on
one
hand,
direct.
I
o
for
us
is
faster
for
things
like
rvd
4k,
random
reads,
but
it
potentially
is
significantly
slower
for
omap
operations
through
rocksdb
for
things
like
collection
listing.
A
The
good
news
here
is
that,
while,
if
you
go
back
to
the
the
other
spreadsheet
in
the
looking
at
the
bisection
results,
gabby
had
had
earlier
during
this
release
or
during
this
development
cycle
implemented
a
pr
that
removes
all
the
allocation
data
for
blue
store
out
of
rocks
tb,
and
that
was
a
significant
performance
when,
for
things
like
4k
random
rights.
In
that
case,
I
think
his
pr
is
probably
worth
about.
20
or
25
percent
improvement,
and
it
it
basically
makes
it
so
that
even.
B
A
You,
when
using
buffered
io
we're
actually
faster
than
we
were
when
we
started
quincy
in
those
results.
You
can
see
that,
in
fact,
by
switching
back
to
direct,
I
o
or
rbd
4k
random
rights,
it's
even
faster
we're
gaining
like
another
10
or
12
percent,
but
we
don't
quickly
need
it,
we're
still
faster
overall
than
we
were
when
we
started
quincy.
A
So
there
is
one
other
thing
in
here.
I
mentioned
adam's
pr
about
doing
direct.
I
o
rights
in
bluefs
and
buffered
reads
that
got
us
back
to
the
4k
random
write
performance,
we're
seeing
in
direct,
I
o
mode,
it's
possible
and
even
fairly
likely
that
we'll
see
the
kind
of
better
performance
in
the
omap
read
and
deletion
tests
with
that
pr,
maybe
it's
the
best
of
both
worlds.
A
The
downside
with
this
as
there's
some
discussion
in
the
pr
you
can
see,
but
the
the
real
downside
with
this
is
that
it
could
be
very
difficult
potentially
to
guarantee
consistency.
When
you're
doing
reads
from
the
linux
page
cache
while
simultaneously
doing
direct
io
rights,
we
will
need
to
be
extremely
careful.
I
think
if
we
go
down
that
path
and
make
sure
that
we're
not
introducing
any
kind
of
inconsistency
in
corner
cases,
but
just
for
the
sake
of
argument,
it
looks
like
there's
potential
there,
so
it
may
be
worth
further
investigation.
A
I
see
everyone
from
core
joint,
so
yeah
I'll
open
it
up
does
gabby.
I
I
want
to
specifically
mention
how
great
your
pr
is.
Looking
any
any
concerns
or
comments.
B
Okay,
okay,
can
you
hear
me
now?
Yes,
yeah
sorry,
my
mic
is
keep
jumping
between
different
definitions,
so
I
I
saw
in
in
in
the
doc
in
the
tables
that
the
commit
level
seems
very
far
away.
Did
you
just
take
compare
the
code
from
before
my
commit
and
after
there
was
a
a
big
difference
in
between
with
many
other
commits
coming
in
between.
A
A
Yeah,
it
and,
and
honestly
it's
it's
all
messy
right,
because
that
was
during
a
severe
regression
that
was
introduced
because
we
ended
up
doing
debug
builds
by
default.
B
A
Yeah,
I
would,
I
would
look
at
line
27
that
result
and
compare
that
to
probably
line
15.
It's
not
exactly
apples
to
apples,
but,
oh
sorry,
now
line
line
nine.
Actually,
that
would
probably
be
the
more
sorry
which
lines
nine
and
compare
that
to
line
27
approximately
it's
not
exact,
but
it's
it's.
You
know
very.
A
There
is
there's
a
lot
of
difference.
The
the
problem
that
we
have
right
now
is
that
we
don't
have
any
direct
comparison
right.
We
we
basically
before
line
10
when,
when
we
enabled
bluefish
buffer
diode
by
default,
we
are
seeing
fairly
consistently
around
66
to
68
000
iops.
A
For
for
this
particular
4k
random
write
test.
Once
we
did
that
we
were
dropping
down
to
like
58
to
61,
I
guess
and
then,
when
we
introduced
the
the
regression
that
made
us
compile
by
default
debug
by
default.
Of
course,
then
it
was
significantly
worse.
A
So
I
guess
really
the
significant
changes
over
master
that
we
saw
were,
if
you
explode.
This
debug
issue
was
when
we
disabled
bluefs
buffered
io,
and
when
we
introduced
your
pr.
So
those
are
kind
of
the
really
important
parameters
to
look
at
line.
27
is
the
average
of
those
three
runs
in
master
head.
As
of
like
a
week
ago,
roughly.
A
That's
just
whatever
the
state
of
master
is
with
your
pr.
Assuming
that
your
pr
does
that
then!
Yes,.
A
B
C
B
A
Yeah
exact
same
test
exact
same
configuration.
There
will
be
some
random
variation
involved,
but
the
bigger
thing
you're
right
is
that
I
mean
on
those
lines
it's
just
depot
built.
So
I
mean
you
can't
can't
directly
compare
right.
C
B
B
A
B
A
B
A
Oh
sorry,
10
is
buffered
out.
True
9
is
buffalo,
false.
C
B
A
B
Yeah,
so
by
by
disabling
buffer
io,
you
remove
a
bottleneck
which
was
part
of
the
work
needed
for
column,
family
b.
B
So
that
was
making
them
slower
with
without
buffer
io
the
if
the
io
was
just
slower
and
column
family.
The
work
was
affected
by
this.
When
you
remove
this
bottleneck,
then
you
get
improvement
for
column,
family
b
work,
but
removing
that
code
makes
even
better
of
improvement.
Okay,
so
I
take
it
20
with
buffered,
io
and
27.
D
A
Yes,
yes,
with
a
relatively
high
depth,
the
data
set
size
tuned
to
so
that
all
low
notes
fit
inside
the
cache.
B
And
24
to
27:
what's
the
difference
between
them.
A
Let's
see
24
27
27
is
just
the
average
of
those
three
separate
runs.
B
A
A
Yeah,
exactly
exactly
because
I
didn't
want
to
do
three
runs
for
every
single
commit
on
the
the
the
bisection
unless
I
needed
to
it
was
really
clear
this
time.
So
I
didn't
really
need
to
gather
more
samples.
A
Yes,
yes,
exactly
yeah,
your
pr
is
looking
great
and,
like
I
said
it's,
the
only
reason
right
now
in
master
head
that
we're
faster
than
at
the
quincy
launch.
So
kudos.
That's
that's
a
really
good
result.
A
B
There
are
two
which
were
still
investigating
one
of
them,
adam
thinks
might
be
unrelated
to
my
changes
and
maybe
in
his
domain,
the
other
one.
I
have
no
explanation,
but
I
was
able
to
see
that
with
my
changes,
disabled,
we
are
unable
to
get
these
failures
and,
with
my
changes
enabled
that
failure
happens
on
tautology
boxes.
I'm
still
unable
to
get
this
thing
to
fail
on
my
box.
Maybe
I'm
doing
something
different
in
the
theology,
but.
B
Environment
is
different,
but
this
thing
doesn't
fail
on
my
box,
which
is
very
annoying
again.
None
of
them
seems
to
be
it's
not
going
to
change
the
way
we
do
things,
there's
probably
some
uninitialized
something
somewhere.
So
it's
I
don't
expect
that
we'll
need
to
do
major
changes
once
we
find
the
root
cause.
A
B
It
might
be,
and
the
other
one
I
don't
know,
I'm
still
thinking-
maybe
I'm
doing
something
too
fancy
for
bluefs
the
way
my
file
is
stored,
I'm
doing
too
many
truncate
and
such
which
other
stones
seems
to
be
using.
Maybe
blue
effect
doesn't
like
truncation.
A
D
A
So
the
only
other
question
here
that
I
was
kind
of
thinking
about
is,
as
adam
was
talking
about.
You
know
his
pr
for
doing
direct
rights
and
buffer
greeds
is
this.
This
may
be
another.
Maybe
this
is
the
time
to
start
looking
at
iou
ring
again
and
potentially
running
iou
ring
in
buffered
mode,
seeing
how
that
does.
A
We
have
some
code
for
I
o?
U
ring,
but
I
don't
know
if
it's
even
working
at
this
point
this
that
whole
interface
has
gone
through
some
churn
in
the
last
year,
so
this
may
be
worth
reinvestigating
it
gabby.
Have
you
ever
looked
at
iou
ring.
C
I
wrote
our
during
application,
but
the
past,
not
not
in
the
reddit,
but
we
we
made
use
in
indigo
fire
urine.
But
that's
why
I
remember
a
lot
of
what
I
did
there.
A
We
so
so
ronan
we
have
code
that
came
in
from
an
external
contributor
that
kind
of
hijacks
our
aio.
A
Path
to
support
iou
ring,
and
it
I
mean
even
at
the
time
it
felt
a
little
hacky
to
me,
but
it
more
or
less
worked.
But
I
wonder
if
it
might
be
time
to
start
thinking
about
iou
ring
as
a
first
class.
You
know
kind
of
citizen
for
bluefest.
A
Yeah
and
think,
actually
it's
all
self-contained
in
the
blue
store
directory.
Let
me
look
well,
that's
not
still
contained,
that's
not
entirely
true,
but
the.
A
A
Somewhere
in
here
now,
I
thought
that
it
was
actually
a
super
file,
but
it
might
not
be
yeah.
Let
me
let
me
look
at
it.
Look
it
up
and
I
can.
I
can
send
it
to
you.
I
actually
don't
remember
exactly
what
the
state
of
it
is.
So
maybe
I'm
misremembering
something,
but
I
thought
that
we
we
supported
it
in
blue
fs.
A
A
A
Well,
anyway,
yeah
take
a
look.
Tell
me
what
you
think
you
you
probably
are
one
of
the
few
people.
That's
actually
you
know
done
anything
with
I
o
urine,
so
I'd
definitely
be
curious
to
see
what
your
thoughts
are.
A
Yeah,
I
doubt
anyone's
even
really
tested
it
much.
I
I
once
tested
it
a
long
time
ago,
and
that
was
it
that
was
even
before
centos
by
default,
supported
it.
So
you'd
have
to
use
custom
kernel
now.
Maybe
it's
different.
C
A
All
right,
I
think,
that's
all
I've
got
so.
Oh
I'll,
open
it
up
any
any
other
topics
that
people
want
to
talk
about
today.
A
B
A
A
We
we
sort
of
did
at
one
point
and
code
was
implemented
and
it
more
or
less
worked.
I
don't
know
a
year
or
two
ago,
but
it
wasn't
showing
any
particular
improvement.
There
was
very
little
difference
compared
to
using
the
aio
interface.
A
Theoretically,
it
may
have
shown
some
benefit
in
that
we
were
actually
at
the
time
able
to
make
io
submit
in
the
aio
path
stall,
which
you
know
it's
not
supposed.
Well,
it's
theoretically,
maybe
sort
of
not
supposed
to
do,
but
in
reality
you
can,
you
can
make.
I
have
submit
stall
in
the
ao
path
if
you
have
too
much
backed
up
io,
so
I
think
I
o
yearing
might
help
in
that
case,
but
in
reality
it
was
very.
It
was
really
similar.
A
It
didn't
show
a
whole
lot
of
performance
improvement
in
the
test
that
we
looked
at.
So
it's
just
kind
of
sat
there
for
like
two
years.
D
I
think
it
might
show
more
benefit
with
c
store
in
the
future,
rather
than
bootstore
more
likely
to
be
hitting
those
kinds
of
bottlenecks.
I'd
expect
with
much
faster
devices.
A
One
thing
I
was
curious
about
josh
is
whether
or
not
I
mean
it
looks
like
jen's
spent
a
little
bit
more
time,
thinking
about
buffered
io
in
the
I
o
urine
context
than
they
did
with
the
bayero.
I
was
wondering
if
bufferdio
may
actually
do
be
closer
to
the
direct.
I
o
level
of
performance
without
having
to
you
know,
do
this
kind
of
mixed
direct
rights.
You
know
buffered,
reads
kind
of
thing
that
that
I
was
proposing.
D
A
Don't
know
I
don't
know,
I
don't
actually
know
why
we're
we're
seeing
the
the
performance
regression
with
buffer
rights
that
we
do.
I
mean
it's
not
entirely
unexpected,
but
I
don't
know
if
it's
specifically
because
of
cisco's.
D
I
guess
my
intuition
is
that
it's
unlikely
to
be
the
cisco
related.
It's
more
likely
to
be
used
like
this
to
be
watch
tv
behavior
that
you
were
talking
about
well,.
A
That's
no.
I
was
thinking
more.
Why
do
we
see
blue
fs
writing
so
much
faster
with
direct
I
o
than
buffered
io?
That
was
the
question
that
I
have.
A
So
my
my
question
is
given
that
it
looks
like
jen's,
maybe
actually
thought
a
little
bit
more
about
buffered
io
in
the
I
o.
U
ring
context,
maybe
we
could
do
buffered
io
and
with
iou
ring
and
get
closer
to
direct
high
levels
of
right
performance,
while
still
having
reads
coming
from
page
cache
in
a
safe
way.
D
A
D
I
see
you,
that's
what
you
mean
yeah.
Might
I
I.
I
would
expect
that
the
more
related
to
going
through
multiple
hot
muscle
memory
in
the
kernel
in
the
buffer,
cache
yeah.
A
D
D
Yeah
exactly
these
are
much
higher
numbers
that
we
saw
even
a
few
years
ago,
yeah
speaking
of
the
landing
storage
deck
there's
just
paper
that
ran
across
a
few
weeks.
Back
that
was
very
interesting,
might
be
worth
discussing
in
a
future
foreign
maybe
about
we
are
connecting
linux.
Internals
big
have
much
lower
latency.
A
Yeah
that
would
be
the
23rd,
looks
good,
I'm
giving
a
talk
on
the
21st,
I
think
about
cephafest
performance
stuff,
so
I
might
be
maybe
cramming
and
reading
this
on
like
the
22nd,
but
I
think
I
can
do
that.
D
A
A
A
D
A
A
A
Oh,
let's
move
it
then:
let's,
let's
change
it
then
what
was
the
30th.