►
From YouTube: Ceph Performance Meeting 2023-03-16
Description
Join us weekly for the Ceph Performance meeting: https://ceph.io/en/community/meetups
Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contribute
What is Ceph: https://ceph.io/en/discover/
A
Hey
guys,
sorry
I'm,
a
little
late
or
I
was
going
over
so
hopefully
we'll
get
those
guys
in
a
little
bit
here.
B
Yeah
cool
I
I
think
there's
some
good
stuff
that
we
can
do
with
that,
both
just
for
optimizing.
The
Beast
front
end
itself,
but
also
for
performance
work
on
the
HTTP
3
front
end,
so
that
we
can
do
kind
of
apples
to
apples.
Comparisons
against
that.
A
B
Yeah
I
could
I
kind
of
hacked
that,
together
as
an
example
for
the
H3
front,
end
itself
but
realized
that
I
could
generalize
it
as
a
rgw,
rest,
API
and
I.
Think
it's
a
good
idea.
B
Oh,
that's
the
HTTP
3
quick
front
end.
Oh.
B
So
we've
we've
been
trying
to
do
some
performance
work
on
that
and
been
kind
of
tricky
to
sees
out
the
issues
there,
but
this
should
help.
A
Okay,
cool
well
yeah
and
I
noticed
there
was
some
stuff
regarding
HS
bench
in
there,
but
I
wasn't
paying
super
close
attention.
Is
there
anything
I
can
do
there.
That
would
make
it
easier
for
your
testing.
C
D
A
A
B
A
Right
what
what
happened
with
with
the
mini
IO
work
thing
did
that
actually
was
that
useful.
B
I
think
so,
I
know
that
Adam
and
our
team
has
been
using
that
okay
I've
seen
bug
reports
of
Upstream
users
that
are
using
it.
So,
okay.
A
Well,
in
any
event,
cool
really
neat
all
right,
let's
see
next
Igor.
This
is
your
PR
for.
A
E
It
speaks
that
I
advertised
a
while
ago,
which
makes
SSD
files
less
fragmented,
so
yeah,
it
looks
like
we
I
didn't
write,
rocks
DB
interface
properly
and
hence
we
get
when
it
flashes
SSD
file.
E
A
A
A
A
All
right,
maybe
I'll
I'll,
try
to
test
that
too.
A
Next
I
think
the
you
also
oh,
this
is
from
no
sorry.
This
is
from
Adam
harmonizing
log
read
and
write
modes.
I
saw
you
approved
that,
though
Igor
so
I,
don't
know
if
it's
actually
a
performance
thing,
but
I
figured
it.
If.
E
I,
go
on
think
it's
as
much
relation
to
Performance,
it's
mostly
to
fix,
but
you
know
an
issue
with
OSD
being
unaccessible
after
mkfs,
so
we
we
could
sometimes.
A
A
E
Looking
at
right
now,
well,
it
mostly
changes
buffer
pleats
for
lock.
A
A
Okay,
well,
moving
on
Casey
I
saw
that
you
merged
the
qat
batch
PR,
so
that's
exciting
or
anyone
that
can
use
it
I
suppose
next,
the
rocksdb
store
well
actually
igorithm
looks
like
two
of
yours:
were
both
the
apply
rocks,
DB
delete
range
threshold
on
the
Fly
and
also
don't
use
real
whole
space
iterator
for
prefixed
access
so
exciting
that
both
of
those
got
merged.
E
E
Pretty
significant
performance
impact
when
doing
full
scans
on
rocks
DB,
so
yeah.
A
I've
seen
it
too,
with
the
whole
Space
iterator,
when
we,
when
we
are
doing
anything
that
requires
you
know,
seeking
to
a
certain
place.
E
Prefixes,
which
belong
to
default
column
family,
while
still
having
a
huge
amount
of
dotted
for
instance,
sum
up
account
family
and
there
are
no
much
sense
to
scan
them.
C
A
Well
then,
at
the
last
one
I've
got
here,
which
is
close
by
the
bot
that
was
I
think
it
was
maybe
just
some
some
changes
that
Gabby
wanted
to
make
to.
You
know:
column
B
code
maintain
this
free
list
type
when,
when
in
that
mode,
okay
close
by
the
bot
hasn't
been
touched
in
quite
a
while,
so
we'll
see
if
he
reopens
it
or
not-
maybe
maybe
not
I
don't
get
the
sense.
It
was
really
that
big
of
a
deal
there
were
a
couple
of
updated.
A
Pr's
Matan
has
introduced
QA
level,
Crimson
performance
tests,
so
it'll
be
interesting
to
see
how
that
works,
and
we
have
rexdb
store,
use,
bounded,
iterators
and
arm
wrench
keys.
So
this
is
another
one
from
you,
Igor.
E
E
Right
and
this
might
cause
some-
or
this
might
improve
some
cases
indeed,
when
we
perform
bulky
that
remote
removals
so
yeah
using
this
generators
might
give
some
problems.
A
Cool
all
right:
next,
we
have
the
d4n
work
for
rgw
at
AC.
I
think
this
is
a
pretty
a
lot
of
comments
in
this
VR
right.
Pretty
big
yep
looks
like
it
I
think
it
just
got
more
updates
that
maybe
got
it
back
to
being
appliable.
A
All
right
and
then
the
last
one
I've
got
here
is
I
guess
this
is
my
PR.
This
is
for
enabling
TC
Malek
in
c-star,
so
Crimson,
basically,
basically
the
the
errors
that
we
are
we're,
seeing
the
leaks
that
we
were
seeing
previously.
We
we
think
that
they're
just
the
same
ones
that
we've
already
suppressed
in
valgrind,
and
we
have
to
do
the
same
thing
for
asan
I.
A
Think
that
Redick
said
that
he
tested
it
and-
and
it
worked
but
I
didn't
I
was
I
meant
to
ask
him
the
core
stand
up,
whether
or
not
that
was
with
TC
milk
enabled
if
he
got
to
pass
or
not
so
we'll
have
to
do
some
follow-up
on
that.
But
good
news
is
that
it's
being
worked
on
again
and
hopefully
we'll
we'll
get
Crimson
with
TC
Malek
in
general.
A
Here
so
that'll
be
a
really
big
win
both
for
performance
and
for
memory
usage,
especially
when
using
bluestore
with
Crimson
so
pretty
exciting
and.
A
Got
for
PRS
this
week
was
there
anything
I
missed
from
anybody.
A
A
Good
job
Igor
I
Joshua
had
mentioned
to
you
earlier
that
yeah
I
want
to
get
this
on
my
list
of
things
to
test
see
if
we
can
get
that
into
to
reach
before
before
we
we
release
it.
G
A
F
You
need
this
PR
in
into
tautology,
testing.
A
Yeah
we
should
get
into
toothology
testing
and
I
was
gonna.
Try
to
maybe
do
some
just
kind
of
one-off
testing
with
this
two
possibly,
but
it
the
the
cephalicons
coming
up
really
fast,
so
we'll
see
we're
we're
gonna
probably
be
going
there
early,
so
I
might
be
out
for
part
of
August
or
April.
F
What's
the
pr
number
just
so,
I
can
put
it
in
my
view,
for
testing
yeah.
A
One
second:
let
me
get
that
to
you.
It
is
oh
yeah.
A
A
Well,
let's
move
on
to
discussion
topics,
then.
First
order
of
business
blue
jeans
is
going
away
at
the
end
of
the
month.
I
guess
so.
We
we
discussed
this
in
the
the
self
leadership
meeting
yesterday.
I
think
the
plan
right
now
is
to
move
everything
over
to
Jitsu.
A
Instead,
that
was
what
Mike
advocated
so
I'm
guessing
that's
what
we'll
do
for
this
too,
but
we
at
least
have
a
little
bit
of
time
here
to
figure
it
out,
so
just
wanted
to
make
sure
everyone
knew
it's
not
hard
to
use.
It
works
pretty
well.
I
had
some
issues
with
Firefox
with
it
previously,
so
maybe
be
aware
of
that.
It
was
like
I
couldn't
hear
random
people,
but
in
I'm
I'm
using
Brave
now
and
it's
working
well
so.
A
C
A
A
All
right,
let's
see,
moving
on
then
wreath
performance,
so
I've
been
doing
a
lot
of
work
on
performance
over
the
last
year
or
two
about
a
year
or
two
week
or
two
Jesus.
Sorry,
so
the
gist
of
it
is
that
after
lots
and
lots
of
work,
trying
to
get
our
nvme
drives
working
right
again,
but
ended
up
requiring
firmware
upgrades
we're
now.
Our
our
test,
cluster
is
working
pretty
well
and
I
was
able
to
tease
out
some
some
interesting
differences.
A
So
Random
rights
is
is
the
interesting
case
which
it
usually
tends
to
be
in
stock
Quincy
or
mostly
stop
Quincy.
Let's
say
we're.
Seeing
about
750
000
random,
write
apps
on
that
cluster,
with
3x
replication
with
Reef,
we
see
around
800
000,
random,
right
iops.
So
it's
an
improvement,
but
if
we
apply
the
roxdb
tunings
from
the
article
last
fall
to
Quincy,
then
we
get
about
850,
000,
random,
right
apps,
so
even
though
reef
is
faster,
Quincy
is
actually
faster.
A
When
we
apply
the
same
tunings,
a
I
suspected,
it
might
be
due
to
the
roxdb
upgrade
so
I
I
recompiled
Reef,
with
the
old
version
of
roxdb
from
Quincy,
and
it
did
not
help-
or
at
least
it
didn't
help
much.
A
It
was
a
tiny
bit
like
maybe
maybe
like
810
versus
850,
for
for
Quincy,
with
the
tunings,
so
I
believe
we
may
actually
have
a
real
regression
buried
somewhere,
but
it
was
one
being
masked
by
the
the
drive
firmware
first
and
now
is
being
masked
by
the
the
roxyb
tuning
changes
that
we
made
so
I
can
probably
devise
a
bisect
to
go
back
and
figure
out
what
it
is.
A
It's
not
a
big
regression
right,
it's
like
I,
don't
know
six
or
seven
percent,
maybe
which,
whenever
it's
that
small,
it
can
be
tricky
to
track
down.
But
let
me
give
it
a
shot
we'll
see.
The
good
news,
though,
is
that
all
of
these
numbers,
otherwise
are
looking
really
good.
You
know
just
just
looking
at
Reef
on
its
own
with
that
those
those
tuning
changes
we're
we're
faster
than
we've
ever
been
before
for
small
random
rights.
A
Small
random
reads
were
faster
than
than
Quincy
when
I
looked
everything
else
was
faster
or
equivalent.
Rgw
is
looking
pretty
good.
It's
not
showing
some
of
the
same
things
we
had
to
track
down
in
Pacific
days.
A
It's
maybe
just
not
quite
as
efficient
as
Nautilus
was
but
Nautilus
was
was
kind
of
a
really
good
high
point,
and
it's
still
looking,
you
know
pretty
good.
So
anyway,
I'm
gonna
try
to
get
all
this
stuff
published
and
written
up,
but
maybe
maybe
not
until
next
week,
we'll
see-
and
that's
that's
really
all
I
had
guys
I'll
open
it
up
is
there
anything.
People
would
like
to
talk
about
this
week.
H
A
Yeah
and
about
4.5
million
random,
read
apps.
C
A
Oh,
and
also
for
IOD
F1,
like
sync
rights
about
0.4
milliseconds,.
A
Actually
I
did
test
Quincy
and
Quincy
was
pretty
pretty
pretty
similar,
so
I,
don't
think,
we've
actually
proved
it
dramatically.
Yeah
is,
it
was
yeah,
maybe
maybe
Quincy
or
maybe
Pacific
I
do
remember
a
while
ago,
maybe
a
year
or
two
ago,
I
did
tests
like
this
and
it
was
like,
maybe
0.8.
We
were
a
little
under
a
millisecond.
Now
we're
like
at
least
in
a
brand
new
cluster
right
brand
new
cluster,
we're
dramatically
under
a
millisecond.
A
Exactly
exactly
so,
you
know,
but
the
the
good
thing
right
is
that
it
means
that
the
stack
is
capable
of
it.
So
yeah,
absolutely
yeah,
we've
sufficiently
reduced
latency,
both
client-side
and
server
side
enough
to
to
get
comfortably
under
a
millisecond.
You
know
very
comfortably
under
a
millisecond,
so
yeah,
exciting.
E
A
E
Mark
you
mentioned
issues
with
when
we
meet
drives.
Is
this.
E
A
Yep
the
same
Samsung
drives
that
we're
seeing
with
the
the
allocator
strange
allocator
behavior
issues.
A
Me
Maybe,
you
know
I
the
behavior
I
was
seeing.
Is
that
one
of
the
OS
one
of
the
drives
all
the
drives
actually
were
a
little
bad,
but
one
of
the
drives
was
specifically
bad,
where
it
would
be
fine
for
a
little
bit
and
then
I'd
see
really
really
high
Q
wait
times
on
device
commits
I.
Guess
things
were
taking
a
long
time
and
just
you
know,
everything
was
was
slow
and
I.
Wonder
that
the
suspicion
I
had
previously
was
that
we
were
like
that.
A
E
Okay,
so
and
firmware
update,
resulted.
A
Seems
to
have,
but
my
question
is:
will
it
come
back,
maybe
I
only
temporarily
fixed?
It
is
this.
This
is
the
pm
893.
A
Yeah
Samsung
PM
893,
but
it's
like
a
generation
or
two
old,
now,
they're
they're,
very,
very
reasonably
priced
for
being
a
data
center
class
Drive.
A
When
I
was
doing
random,
I
o
to
the
whole
cluster
I
would
see
well,
okay,
random.
I
o
would
show
these
oscillating
periods
of
very
high
Q
wait
times
and
and
I
think
I
think
yeah
service
times
for
for
iOS,
but
even
even
when
I
was
doing
like
large
rights,
you
could
see
that
that
it
was
it
was
keeping
up,
but
it
was
that
you
could
see
that
the
few
wait
times
and
service
times
for
those
aisles
were
a
little
higher
than
the
other
ones.
A
G
Like
maybe
they
were
deferring
some
work
that
they
shouldn't
have
been,
but
I
know
like
I've,
seen
such
a
wide
variety
of
behaviors
in
terms
of
what
the
drive
does
under
fragmentation,
even
from
the
same
manufacturer
just
between
Generations
yeah,
once
the
drive
gets
fragmented.
It's
amazing
how
much
performance
Dives
eventually.
A
G
Yeah
I
mean
in
if
you
want
like
really
easy
in
fio.
Is
you
just
run
4K
random
rights
for
like
a
few
hours
just
to
fill
the
drive
and
then
watch
it
curve
and
see
what
happens
and
then
you
can
run
like
one
Meg
or
four
Meg
rights
and
then
watch
to
see
if
the
performance
goes
back
up
again
and
if
it
does,
it's
probably
like
you
can
kind
of
see
what
the
drive
is
doing
from
a
fragmentation
perspective
under
those
two
scenarios,
yeah.
A
G
A
A
C
C
A
Yeah
I
I,
don't
have
much
to
say:
I
was
just
gonna
say
the
good
news
is
that
now
we
know
to
look
for
that,
because
even
when
I
upgraded
the
firmware
on
that
OneDrive,
it
helped
a
lot
and
when
I
have
upgraded
the
firmware
and
all
the
other
drives,
it
actually
got
us
like
another
like
five
or
ten
percent.
So
we
we
know
now
to
be
paying
really
close
attention.
G
G
Seems
to
be
yeah,
especially
for
large
right
performance.
It
seems
to
make
the
biggest
difference.
G
I
did
leave
some
comments
in
the
pr,
though
so,
like
I,
don't
and,
and
the
submitter
did
say
that
they
have
to
update,
exist,
some
issues
with
what
they
did.
So
it's
it
by
no
means
in
a
shape
to
be
merged
right
now,
but
just
from
a
performance
perspective,
I
just
wondered
if
it
should
be
on
your
list.
A
G
For
this
discussion.
A
Did
you
have
numbers
like
comparison
numbers?
Is
that.
G
I
I
wrote
some
in
here
in
the
ticket
I
think
the
biggest
one
is
that
it
like
when
we're
doing
four
Meg
right
load
tests
on
a
fresh
cluster.
It
basically
kept
the
CPU
usage
path.
G
Yeah
I
only
tested
the
disabling.
The
work
keys,
I
have
issues
with
how
they're
doing
the
sector
stuff
so
I,
don't
know
in
the
end,
whether
they're
going
to
want
to
actually
just
break
that
up
or
even
just
abandon
the
sector
overrides,
because
once
I'm,
assuming
reef,
is
going
to
be
on
Centos,
nine.
All
right.
A
G
Want
to
see
because
I
think
7
plus
9
comes
with
scripts,
that
243
and
Crips
out
of
2.4
will
automatically
do
sector
size,
detection,
new
dmcrypt
devices,
so
I,
don't
know
I'm
kind
of
wondering
if
they
should
just
drop
that
part
and
just
focus
on
the
part
that.
A
A
Yeah
is
that
do
you
wanna?
Do
you
want
to
recommend
that
in
there
or
do
you
want
me
to
come
in
and.
G
A
G
E
G
E
B
E
Unrelated
to
previous
your
previous
issue,
so
it's
about
writing
SSD
files,
mostly
and
making
other
less
recommended.
A
Oh
actually,
I
do
have
one
other
thing
since
Alex
is
here
yesterday
in
on
slack,
we
were
talking
with
David
Orman
about
time
to
live
for,
watch,
TV
and
I'm
tempted
to
actually
we,
we
added
I,
think
a
six
hour
attempt
to
live,
maybe
for
Reef
as
part
of
the
other
Roxy
Beast
settings
that
we
changed
but
David
when
he
yeah
he
was
testing.
A
This
saw
that
they
didn't
need
it
at
all
anymore,
like
all
the
other
changes
are
like
completely
fixing
everything
and
so
I've
been
tempted
to
maybe
take
it
back
out
and
not
set
it
as
a
default.
Brief.
D
A
I
think
Reef's
gonna
be
a
good
release.
I'm
excited
all
right,
good
good,
so,
okay
I
think.
Maybe
we
just
let
it
stew
a
little
bit
more,
but
if,
if
everything's,
looking
good,
both
with
you
guys
and
with
and
with
David
orman's
group
and
and
we're
still
just
seeing
nice
behavior
like
like
this,
maybe
we
take
it
back
out
again.
A
A
It
I
mean
when
I
tested
it
with
nvme
drives
on
our
test
cluster.
It
didn't
actually
hurt
performance,
really
I
mean
there
was
like
one
case
for
reads
where
we
were
seeing
like
a
five
percent
drop
or
something,
but
it
was
offset
by
the
huge
you
know:
space
usage,
Improvement
that
you
know
and
write
a
publication
Improvement
actually
on
the
the
nvme
drives
right.
So
you
know
I
I,
don't
know
like
I.
G
A
C
A
I
could,
if
I
have
time
I
can
try
it
too
I.
Just
we'll
we'll
see
I'm
I'm
kind
of
running
around
like
a
chicken
with
my
head
cut
off
right
now,
trying
to
do
like
five
different
things
at
once
and
doing
them
all
kind
of
poorly.
So.
A
All
right,
yes,
very
good,
very
good.
That
was
really
the
last
thing
I
had
so
going
once
anyone
have
anything.
A
All
right
well
have
a
great
week.
Everybody
it's
been
fun
and
see
you
next
week,
yeah,
they
say.