►
From YouTube: Ceph Performance Meeting 2023-07-03
Description
Join us weekly for the Ceph Performance meeting: https://ceph.io/en/community/meetups
Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contribute
What is Ceph: https://ceph.io/en/discover/
A
B
B
A
Good
I
guess
is
a
little
more
relaxing
this
week
than
last
week.
So
that's
nice.
A
A
All
right
we're
starting
to
get
folks
from
Court
here.
So
let's
get
this
started.
Okay,
I
didn't
see
any
new
pull
requests
this
week.
If
I
missed
yours,
please
speak
up
and
edit,
but
for
now
I'll
go
on.
I
saw
two
close
pull
requests
this
week.
A
D
No,
no,
no,
no
I
think
that
we
it's
a
two-state,
merging,
rocks
the
pig
and
Dot
git
and
then
git
and,
according
to
my
notes,
I
will
double
check,
but
I
think
that
we
matched
to
the
to
a
brand.
That
is
simply
named
Reef.
A
D
That
over
right
exactly
yeah
instead
of
git,
it's
just
it's
just
Main,
okay.
However,
it's
a
I'm,
it's
a
basically
a
fix
up
for
for
the
version
of
rocksdb
that
we
are
using
in
May
in
every
event.
Now
in
Squid,
Till
There
is
no
big
difference.
D
Big
update
of
roxdb
I
will
prefer
to
keep
using
the
reef
Branch
even
in
Squid.
Thank
you.
A
D
Yeah,
basically,
basically,
we
sold
we
sold
the
upgrade
upgraded,
keep
up
with
Rose
dbb
major
releases,
I.
Think.
A
D
Thing
with
with
We've
outdated
drugs,
DB
yeah,
yeah.
A
D
I
guess
we
will
make
a
we
will
make
the
squid
Branch,
maybe
later
in
this,
like
I,
don't
know
winter
2020
free
spring
early
spring
24,
something
like
that.
A
D
D
Now,
even
even
now,
even
a
mess
with
with
Branch
naming.
D
D
Anyway,
anyway,
I
think
that
this
fix
up
is
also
beneficial
for
Reef
I
I'm.
Assuming
that
that
we
want
to
keep
billability
of
reef
on
on
on
new
Fedora.
Is
that
right,
Casey.
D
Maybe
not
neces
well.
Is
it
critical
to
have
it
in
the
reef
RC
on
or
on
the
main,
on
the
first
main
release
of
of
reef,
because
if
not,
maybe
we
could
backboard
the
backboard
light
after
after
some
after
some
more
staging.
E
So
I'll
just
create
a
Tracker
issue
for
it
and
yeah
we'll
handle
it
with
the
normal
backboard
press
process.
Whenever
you're
ready.
D
Cool,
well,
we
can,
we
can
we
can.
We
can
create
the
pr
now
and
and
click
the
button
later.
A
A
E
How
yeah
I
know
that
lib
format
made
some
breaking
changes
with
version
nine,
and
we
have
some
in
if
defs,
to
try
to
handle
that?
Oh
thanks,
I
think
for
Maine.
Ultimately,
we
want
to
just
bump
the
required
version
to
to
nine
or
higher
and
remove
all
that
backward
compat
stuff,
but
that
doesn't
solve
the
Pacific
and
Quincy
branches
which,
if
you
use
system
format
with
a
version.
Nine
then
I
know
that
we're
seeing
build
failures
there,
yeah
Snappy
I,
don't
have
a
strong
opinion.
We
might
just
consider
deprecating
it.
A
A
A
C
With
getting
that
read,
but
just
wanted
to
mention
it,
it
it's
working
and
for
compatibility,
I,
don't
know
if
we
don't
do
not
have
to
preserve
it.
D
Well,
if
you
want
to
deprecate
Snappy
this,
this
must
go
over
the
entire,
the
entire,
the
usual
deprecation
process,
and
we
and
riff
is
pretty
pretty
soon.
So
if
we
want
to
get
rid
of
Snappy,
no
no
much
time
to
this
to
make
the
the.
A
D
Yeah
yeah
for
keep
keeping
versus
the
removal
versus
depreciation.
Two
two
different
things:
yeah.
D
Make
a
dog
change
a
darker
suggestion
that
claim
that
we
are
going
to
because
we
are
deprecating
Snappy
with
with
Reef.
D
From
one
from
one
set
of
pictures
to
another,
okay.
E
D
E
Think
the
rgw
migration
will
be
as
easy
we'd
effectively
have
to
find
all
of
the
rgw
objects
that
are
compressed
with
Snappy
and
recompress
them
with
something
else.
D
Oh
man,
bigger
bigger
thing,
so
it's
not
so
for
sure
it's
not
just
a
car.
It's
seven
General,
basically,.
D
E
C
A
Oh,
my
sorry,
the
thing
at
the
end,
accidentally
a
threadzilla
and
somewhere
we've
got
the
comment
from
Google
saying
that
they
don't
they're,
not
gonna
they're,
not
going
to
allow
the
re-enabling
of
this
in
and
their
code
in
the
sappy
code.
D
So,
look
that
no
way
to
either
you
are
either
you
are
consuming
Snappy
from
our
Repository.
D
The
downside
I
see
is
that
basically
we
are
forking.
We
are
forming
another
another
component.
It
might
might
be
not
so
terribly
bad,
but
if
we
could
avoid
it.
A
Well,
that
would
be
the
the
thing,
though,
that
we
could
do
while
we
consider
deprecation
right,
as
this
would
be,
the
first
step
is
to
just
do
it
ourselves.
Yes,.
D
We
are,
we
aren't
living
in
IDL
World
likely.
We
will
need
to
to
make
our
make
some
tiny
Fork
of
of
Snappy,
like
we
did
with
drugs
in
the
format,
so
a
short
living
format
that
we
must
support
forever.
A
D
Have
we
checked
for
any
API
compliant
replacement
for
a
Snappy
I,
guess
that
we
aren't
the
very
sole
project
facing
this
issue?
Maybe
somebody
has
already
forked
Snappy
call
it
I,
don't
know
Snappy
old
generation
and
then-
and
they
say,
re-enabled
rtti.
There.
A
A
D
Well,
we
are
providing,
as
a
Adam
mentioned,
another
person
we
are
providing.
We
are
having
the
option
to
build
with
with
our
own.
It's
not
even
now,
I
think
am
I
correct,
I.
E
E
A
A
All
right,
let's
see
we
can
continue
on
with
the
pull
request
now.
I
guess
what
else
we
have
Igor
I
think
your
writable
file
allocate
merged
some
good
news
on
that
front
yeah.
It
looks
like
that
updated
I,
see
where
the
you
also
approved
the
first
part
of
the
elastic
shared
blob,
so
Adam's
work
on
elastic,
shared
bulbs
and
then
reviewed
part
three.
When
you
approve
your
friend
too,
as
well
yeah
yeah,
so
Adam
do
you
feel
like
or
do
we
have
Adam.
B
Does
that
I
think
I
saw
something
upstairs
about
Debian
looking
to
do
that
because
they
don't
like
it
breaking
everything
else
either.
So
it
might
be
that
Google
doesn't
want
to
enable
Upstream,
but
the
reason,
but
the
people
making
the
repos
might
enable
it.
Okay,.
A
Well,
yeah
Adam!
If
you
see,
if
you
saw
something
with
the
Debian
folks
talking
about
it,
then
yeah
we
can.
Maybe
we
can
get
that
added
into
a
Tracker
here
too,
for
us
on
our
side,
just
trying
to
figure
out
who's
doing
what?
Where.
A
All
right
so
back
to
the
elastic
shared
blobs,
PR
Adam.
Do
you
feel
like
with
Igor's
approval
on
the
first
two
parts
of
this?
Should
we
merge
those
first
or
are
you?
Do
you
want
to
do
this
all
in
kind
of
one
big
go.
C
I
would
want
to
go
with
one
big
go
as
specially
that
it
seems
now
we
want
to
have
it
a
bit
different
I,
don't
know
how
we
will
need
to
review
it,
but
it
seems
we
want
to
have
the
mandatory
part,
meaning
part
one
and
two
just
merged
with,
without
even
if
Def
and
then
drop,
if
def,
for
the
part
three
and
four
and
replace
them
for
just
runtime
conditional.
C
It
seems
that
we're
leaning
toward
that
direction,
and
unfortunately
it
might
mean
that
we
shouldn't
merge
it,
as
is
just
I,
will
need
to
modify
it
before
marriage,
just
at
least
drop
the
if
devs
for
part
one
and
a
part,
two
so
far
and
then
modify
it
part
three
and
four
for
some
just
wrapping
out
not
with
if
devs
but
with
just
run
timing,
so
that
that's
it
so
I'm
not
feeling
okay
with
merging
it.
As
is
now
okay,.
A
Okay,
well,
good
luck
on
that
and
it'll
be
I'll,
be
curious
to
see
how
that
all
works
in
the
end,
all
right.
Moving
on
so
I
have
a
very
old
PR
here
to
enable
TC
meloc
when
using
c-star
for
crimson,
and
it
broke
some
of
the
address.
Sanitizer
checks
that
were
being
done
and
radic
I
think
you
just
merged
the
pr
for
basically
yeah.
B
A
Thing
that
that
fix
makes
other
sanitizer
happy
again.
D
I
think
so
I
think
so
good
good,
I
I
think
there
were
some
trigger
match
conflict
after
after
that,
we
are
ready.
We
are
ready
to
go.
A
Okay-
okay,
let's
see
here,
oh
I,
think
it's
blocked
on
your
review.
Was
there.
D
Oops
I'm,
going
to
correct
myself
I
think
I
was
just
pointing
out
the
dependency
on
on
on
suppression,
essay
and
otherwise
we
would.
We
would
have
the
Crimson
Dash
radios
tested
red
again.
A
Deal
good
deal,
let's
see,
then
moving
on
Corey
Snyder's
PR
for
sitting
in
the
Rock
CB
iterator
bounds
for
collection
list,
I.
Think
Adam,
the
only
thing
there's
you
were
reaching
out
to
see
if,
if
he's
still
working
on
it
right.
A
And
Corey's
not
here
today,
so
I
guess
we'll
move
on
next.
Is
this
RBD
replace
image
context
config
with
image
config
proxy
there's
a
request
for
Iliad
to
re-review
that
again,
I
think
and
So,
since
neither
of
them
are
here
we'll
move
on
from
that?
Oh
and
that's
the
last
one
for
updated,
so
I,
don't
think!
There's
anything
else.
Interesting
in
here
for
today
was
there
anything
I
missed
from
anyone.
A
All
right:
well,
then,
if
not
maybe
the
only
discussion
topic
I
have
I
didn't
write
down
here
is
that
Paul
has
been
continuing
to
do
his
tests.
Looking
at
RBD
performance
in
Reef
versus
Quincy,
his
tests
are
rate
limited,
so
he's
trying
to
basically
look
at
a
very
given
consistent
workload
and
what
is
the
CPU
usage
and
what
is
the
the
underlying
physical?
A
I
o
that's
hitting
the
disks
and
in
his
tests
he
was
seeing
that
that
reef
is
looking
like
it's
maybe
less
efficient
than
Quincy,
both
in
terms
of
CPU
and
in
terms
of
the
amount
of
iOS
that
we're
doing
for
client
IO.
A
He
did
see
variations
when
he
increases
the
memory
Target
size,
the
CPU
usage
effect
goes
away.
Our
hypothesis
right
now
is
that
this
is
probably
the
roxdb
change
that
we
made.
The
roxdb
change
basically
makes
the
kvsync
thread
more
efficient,
but
the
trade-off
might
be.
We
might
be
seeing
evidence
that
the
trade-off
is
that
by
making
the
mem
table
smaller
so
that
the
KV
sync
thread
has
to
do
less
work
to
keep
each
mem
table
in
sorted
order.
A
We
might
be
seeing
that
proxdb
is
having
to
do
more
work
to
do
like
an
O
node
lookup
when
you
have
an
auto
cache
Miss.
So
it
might
be
that
as
long
as
your
own
ones
are
cached,
this
is
just
a
pure
win,
but
if
your
o
nodes
are
not
cached
are
not,
you
know
well
cached,
I,
guess,
then,
then
you
might
be
seeing
more
overhead
when
you
have
a
cache.
A
That
might
be
driving
CPU
usage
up
I'm,
trying
to
replicate
some
of
his
tests
right
now
on
Mako
he's
also
running
more
tests
himself.
A
His
new
test,
he's
going
to
run,
are
going
to
be
doing
very
low,
OSD
memory,
Target
tests,
so
that
o
nodes
are
not
cached
well
and
seeing,
if
that
makes
the
effect
more
exaggerated
and
he's
also,
his
tests
have
so
far
all
been
mixed
workloads,
random
reads
and
random
rights,
and
so
he's
going
to
look
at
just
a
pure,
random,
read
or
pure
and
right
workload
to
see
if
the
effect
is
more
or
less
present
in
either
of
those
or
just
always
present.
A
He
also
saw
that
the
effect
was
exaggerated
at
like
8K
I
o
sizes
and
less
exaggerated
at
14.
16K
I
have
no
idea
why
that
would
be
I
suspect
it's
just
an
artifact
of
something
happening
at
AK
versus
those
other
I
o
sizes,
but
that
that
was
one
small
interesting
bit
of
this
as
well.
C
I
have
to
tell
you
how
it
could
be.
It
could
be
a
pressure
from
caches.
It
could
a
larger
blog
data
cache
could
further
still
memory
from
oh
node
cache.
A
A
Well,
in
any
event,
hopefully
later
today
or
tomorrow,
I'll
have
I'll
have
some
tests
that
that,
hopefully,
at
least
to
some
degree,
replicate
what
Paul's
doing
it
will
be
exact.
A
He's
he's
letting
this
IO
age
for
for
quite
a
long
time,
so
I'm
I'm
right
now
just
trying
to
get
the
lay
of
the
land
and
do
some
quick
tests.
One
thing
I
do
notice,
though,
is
that
our
CPU
consumption
in
kind
of
a
repeated
test
like
this
after
you
do
other
I
o,
does
go
up
as
you'd
expect
since
there's
more
metadata
and
and
more
work
to
do
to
track
things.
So
I
do
Wonder
a
little
bit
if
we're
seeing
some
effect
of
the
Aging
impulse.
A
Tests
and
I
want
to
make
sure
that
the
Aging
actually
looked
exactly
the
same
between
reef
and
Quincy,
because
if
it
didn't,
maybe
maybe
we're
seeing
some
artifact
of
the
way
that
it
aged,
but
in
any
event
more
work
to
do
there.
So
that's,
oh
I
have
one
other
update,
so
that
was
one
and
the
other
one
is
that
I've
got
a
user
in
the
community.
A
That's
doing
some
high
performance
tests
with
just
a
bunch
of
nvmees
in
one
node
and
he
reported
really
good
results
and
he
actually
gave
me
access
to
do
some
work
on
that
node
and
I'm,
seeing
much
worse
results,
but
the
results
I'm
seeing
actually
mimic
what
I'm
now
seeing
on
official
analysis
as
well.
Basically,
once
you
get
past
like
three
or
four
nvme
drives
we're
seeing
scaling
slow
down
way
way
down
in
within
the
node.
A
If
you
just
do
like
a
single
cluster
test
on
that
node
and
it's
it's
interesting
because,
like
I
hit
about
a
a
limit
at
about
400
to
500,
000,
random,
read
iops
and
maybe
around
180
000
random,
write
apps,
not
counting
replication.
A
That's
really
similar
to
the
per
node
numbers
that
we're
getting
out
of
mako
when
we
do
like
a
big
10
node
test
and
get
like
four
and
a
half
million
apps.
Again,
it's
like
450
000
per
per
node.
It
makes
me
think
that
we're
hitting
some
limit
in
the
node
that
we
can't
easily
see
when
we
add
more
osds
the
scaling
just
kind
of
tapers
off
after
about
four.
A
So
that
is
something
that
needs
to
be
investigated
as
well,
and
I've
got
tests
also
going
simultaneously.
Looking
at
that,
but
anyway,
that's
that's
it
at
least
for
me.
B
Was
gonna
say:
I
found
the
Debian
Snappy
bug,
I,
don't
know
if
we
want
to
talk
to
people
in
Fedora
and
ask
them
if
they
can
or
we'll
do
that
or
send
to
us
nine
and
ask
if
they
can
or
we'll
do
the
same
thing,
because
you
know
debian's
doing
it
and
I
wouldn't
say
we
should
always
do
what
Debian
does.
But
it's
an
argument.
Yeah.
A
A
B
Sure
sure
you
want
me
to
make
well,
do
you
want
me
to
put
it
in
the
Seth
bug
tracker
or
do
you
want
me
to
go,
find
someone
in
Fedora
to
ask
them
to
do
it
or
yes,.
D
But
actually
it
might
be
even
worse,
above
because
we
also
want
to
be
aware
about.
We
want
to
track
the
problem
during
during
our
scraps.
So
if
it's
not
in
already
in
the
main
access
tracker,
I
would
kindly
ask
to
put
the
in
there
as
well
all.