►
From YouTube: Ceph Performance Meeting 2023--01-19
Description
Join us weekly for the Ceph Performance meeting: https://ceph.io/en/community/meetups
Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contrib...
What is Ceph: https://ceph.io/en/discover/
A
Hey
folks,
I
just
got
out
of
the
course
stand
up.
I
think
we'll
get
some
folks
here
soon
we're
just
talking
about
it
sounds
like
maybe
we've
got
the
build
issue
fixed
this
morning,
so
not
performance
related
exactly
but
good
news,
the
the
long
Saga
about
this.
We
were
running
on
like
luster
for
a
while,
just
because
it
was
like
what
we
what
was
set
up
years
upon
years
upon
years
ago,
and
we
moved
over
to
I
think
is
in
the
process
of
moving
it
over
to
stuff.
A
We
ended
up
with
a
self-related
problem
that
I
think
we've
got
fixed
now,
so
yes,
ktic
the
hands
of
joy,
but
it
sounds
like
radic
was
saying
that
he
submitted
something
this
morning
and
and
it
went
through
and
built
on
Central
estate,
which
was
the
the
issue
that
we
we've
been
having
so
anyway,
lots
of
good
stuff.
A
All
right,
oh
I,
see
Josh
you're,
adding
a
bunch
of
stuff
here,
very
good,
very
good.
Before
we
get
into
all
that,
I'll
quickly
go
through
pull
requests
and
I
almost
made
it
through
the
list.
I
got
really
close.
Okay,
let's
see
new
this
week,
Igor
you've
got
this.
This
very
interesting.
Looking
bounded
iterators
for
arm
range
keys,.
A
Any
anything
well
I
said
you
added
Corey
as
a
reviewer
for
it
so
Corey.
So
you
know
you've
been
tagged
any
anything
interesting
there.
You
are
to
to
watch
out
for
with
this.
B
Oh
well,
first
of
all,
I
purchased
to
reproduce
that
locally,
so
I
mean
the
pre-produce
ratio
locally,
with
rocks,
DBA
performance
degradation
and
it
was
a
map,
clear
function
which
called
Ram
range
keys
and
iterated
over
multiple
column,
families
and
column,
family
charts,
and
it
counted
up
to
1
million
piece
check.
If
it
needs
parents
delete
or
not-
and
this
definitely
triggered
the
issue.
B
B
We
effectively
stole
the
current
thread,
then
another
thread
from
the
same
as
D,
sharp
and
potentially,
if
client.
If
we
have
multiple
clients
which
says
well
PG
randomly,
we
can
cause
them
to
stroll
completely.
B
So
people
I
mean
that
again,
single
misbehave
in
threats
might
effectively
stall
that
the
the
whole
Blue
Store
actually
I,
know
the
way
how
to
avoid
this.
Just
to
share.
A
I
I
had
a
test
case
that
did
really
really
badly
when
we
had
completely
unlimited
range
delete
before
I
was
an
rgw
test
case.
I
was
just
looking
for
the
pr
where
I
had
the
stuff
there
for
it.
A
I
have
this
on
any
event.
I
can
find
it
again,
but
but
if
it's
useful
for
you,
we
could
try
running
through
those
old
test
cases
again
and
see
see
how
it
does.
If
we
allowed
for
range
delete
to
happen
with
fewer
elements.
B
Well,
I
have
one
thought
that
maybe
the
analysis,
the
root
cause
analysis,
wasn't
100
valid
in
that
time.
So
could
it
be
that
something
like
this
unbounded.
B
A
I
can
show
you,
though,
I
found
the
pr
that
where
I
was
testing
the
behavior
I
don't
know.
Maybe
this
could
have
been
more
related
to
to
some
other
aspect
of
this,
but
it
appeared
that,
for
whatever
reason
we
did
range
delete
with
a
small
number
of
keys
that
the
it
made
iteration
extremely
slow
in
general.
C
I'll
say
that
I've
sort
of
tested,
both
over
time
and
I,
don't
see.
I
haven't
seen
really
any
difference
from
my
side
between
a
delete,
range,
Tombstone
and
the
same
set
of
individual
tombstones
covering
the
same
contiguous
range,
but
in
either
case,
if
you
have
a
large
swath
of
a
contiguous
range
of
tombstones,
whether
it's
from
a
delete
range
or
individual
ones,
both
of
those
cause.
A
big
problem.
A
Corey,
did
you
notice
any
difference
that
you
had
like
say?
If
you,
you
did
small
delete
ranges
versus
individual
ones?
So,
if
you
had
like
say
a
delete
range
only
covering
a
couple
of
keys
versus
individual
keys,
what
would
that
show
any
difference?
A
C
Don't
I
don't
believe
so,
because
what
I
see
that
what
I've
seen
the
problem,
as
is
that
right
when
you
do
like
a
lower
bound
call
and
that
lower
bound
call
is
within
or
at
the
beginning
of
a
range
that
is
full
of
delete,
tombstones
or
delete
range
Tombstone.
It
first
has
to
do
a
binary
search
to
find
the
lowest
key
in
that
range
right,
and
then
it
has
to
go
back
and
check
whether
it's
deleted,
either
by
looking
at
individual
or
range
Tombstone.
C
If
it
finds
that
it
is,
it
has
to
do
that
binary
search
again,
and
so,
if
you
have
a
million
contiguous
tombstones
like
that,
it's
doing
a
million,
binary
searches
and
coming
back
and
looking
if
it's
deleted
until
it
finds
something
that
isn't
Okay.
So,
regardless
of
whether
they're,
multiple
tombstones
or
not,
it
ends
up
doing
the
same
thing.
A
I
wish
I
could
understand
why
we
saw
the
behavior.
We
did
back
when
this
other
PR
that
I
did
merged,
where
we
basically
only
used
Elite
range
when
we
had
large
numbers
of
keys.
A
C
I'm
struggling
to
remember
it
now,
though,
I
think
I
did
see
some
notes
in
rocks
to
be
changelogs
about
there
being
bugs
with
delete
range
in
the
past
and
prior
versions.
That
may
have
caused
some
performance
type
issues.
But
I'd
have
to
dig
that
back
up
to
see
what
that
was.
It
was.
A
A
B
Yeah
well,
I
mean
at
least
about
that
stuff
makes
sense
indeed
and
I
think
it
worth
backboarding
as
well,
and
meanwhile
we
might
want
to
proceed
with
using
bench
delete
unconditionally
in
main
background
the
if
works,
properly
yeah,
you
can
make
it
for
a
while.
A
Let's
see
moving
on,
it
looks
like
this
initial
commit,
adding
Arrow
flight
functionality
has
merged,
since
we
don't
have
Eric
PCD.
Is
there
anything
interesting
about
that
or
anything
new.
D
Well,
there's
a
lot
of
interest
from
our
team
and
aeroflight
and
Integrations,
but
we're
still
trying
to
find
kind
of
the
right
model
for
that
this.
This
PR
does
it
in
in
kind
of
a
weird
hacky
way.
D
D
I'm,
probably
not
the
best
person
to
summarize
this,
but
it's
a
fast
RPC
framework
for
low
latency
data
transfers.
Okay,.
A
A
This
is
this
isn't
related
at
all,
then,
to
the
like
Fast
S3
stuff,
that
we
were
talking
about
a
while
back.
D
Possibly,
the
idea
for
rgw
is
that
we
kind
of
use
this
as
a
side
channel
for
clients
to
request.
You
know,
or
you
know,
query
specific
parts
of
objects.
D
Yeah
I
I
think
there's
potential
for
Integrations
with
osds
also
so
that
rgw
could
like
describe
the
layout
of
data
and
and
have
it
fetch
things
directly,
but
it
definitely
needs
more
elaboration.
Yeah
that'd.
A
D
A
A
All
right:
well,
let's
see
next
I
think
this
is
just
get
closed.
It
was
a
pretty
old
PR
for
rewriting
the
hardware.
Docs
I
I
suspect
that
maybe
that's
been
superseded
at
this
point,
but
it
got
close
so
I
put
it
in
here.
Let's
see
updated
or
missing
part
of
this
one
I
don't
know
who
wrote
this.
Do
not
collection
list,
one
remove
collection.
Let
me
look
at
it
quick
race
on
okay.
C
E
A
Okay
sounds
like:
if
people
are
concerned
about
it,
we
should
maybe
maybe
either
either
request
changes
or
or
maybe
rejected,
so
it
doesn't
just
languish.
A
You
are,
maybe
maybe
just
put
a
comment
in
with
what
you
just
said.
Do
you
think
if
you
think
it's
reasonable,
if
we
we
just
have
an
option
for
it,
maybe
that's
maybe
that's
the
way
to
proceed.
B
E
Well,
the
argument
against
this
PR
would
be
also
that
we,
as
far
as
I,
remember
check
if
there
are
any
objects
in
the
collection
anyway.
So
even
if
we
skip
trying
to
list
any
object
on
that
level,
we
still
do
that
entire
access
to
rocksdb
in
Blue
store
anyway.
B
Well,
Adam
I
believe
this
PR
Works
in
a
different
way.
So
it
removes
the
check
from
the
the
check
for
empty
collections
like
empty
collection
in
at
blister
level,
and
we
still
have
it
address.
D1.
E
Okay,
I
might,
in
that
case
disregard
I,
might
have
misread
the
code
here.
C
If
Blue
Store
does
the
fact
that
there
are
still
objects
in
there
during
that
iteration
and
bubble
that
up
through
view
transaction
so
that
we
can
remove
the
one
in
the
OSD
layer
and
and
then
end
up
having
the
same
logic
for
retrying
the
PG
deletion.
When
that
is
the
case,
if
new
objects
had
been
added
instead
of
doing
the
assertion,
failure
that
way,
we
still
have
basically
the
same
failure
or
the
same
invariant
protection
at
the
Blue
Store
level.
But
we
avoid
doing
that
expensive,
iteration
twice.
B
Yeah,
this
button
can
make
sense,
just
not
sure
if
we
are
able
to
record
for
this
specific
function,
you
need
to
double
check
since
it's
default
in
a
transaction
menu.
B
E
B
Well,
definitely
OSD
code
for
PG
removal
doesn't
handle
that
at
the
moment,
but
it's
it
can
be
introduced.
What
is
labeled.
E
But
now
I'm
thinking,
if
maybe
there
is
a
way
that
we
could
store
in
a
Blue
Store
internals
some
kind
of
flag
if
the
collection
is
empty
like
a
because
the
problem
here
I
see,
is
that
we
do
iterate
over
entire
collection
in
submit
transaction
in
the
relay
here
is
a
poison
for
us.
If
the
same
delay
happens
when
we
list
collection,
we
will
be
fine
with
it.
A
Sounds
good
all
right
moving
on
then,
let's
see
next
Sam
reviewed
this
mcluck
PR
for
adding
the
ability
to
handle
high
priority
operations.
I
didn't
look
super
closely
at
it,
but
I
it
appeared.
Sam
did
a
pretty
in-depth
review
so
that
looks
good
and
moving
along.
Let's
see,
there's
been
I
think
a
little
bit
more
discussion
on
Corey's
PR
for
studying
your
xdb
iterator
balance
for
blue
star
collection
list,
or
was
there
anything
new
there.
C
A
All
right
good
deal
next
upgrading
rocksdb
to
the
latest
Facebook
release.
We
discussed
this
last
week.
I
just
put
a
note
in
there
that
we
can't
actually
adopt
Facebook's
Branch.
A
We
have
to
use
our
own
because
there's
still
that
fix
from
several
years
ago
that
we
we
still
decide
we
want
to
have
so
I
just
sent
the
author,
a
note
saying
that
we
still
need
to
do
that
and
if
he
could
update
the
pr
that
would
be
fantastic
and,
lastly,
for
updated
PR's
there
was
there's
this
heading
primary
balance.
Scores,
let's
see,
Laura
are
you
here
today,
I,
don't
see
Laura.
She
just
added
a
a
slight
suggestion
to
part
of
that
code.
F
I'm
here,
oh
okay,
good,
hey,
Josh,.
F
I
have
there
is
a
slight
problem
with
the
whole
calculation,
which
is
it's
very
easy
to
define
the
score
when
primary
Affinity
on
all
the
osds
is
one,
but
one
day
it's
it's
a
more
conceptual
problem
to
have
how
to
actually
Define
the
score
in
when
the
prime,
when
every
USB
is
different,
primary
Affinity
to
have
something
that
is
Meaningful
the
number
that
the
user
could
be
interpret
in
a
normal
way,
and
so
I
added
the
unit
test,
run
random
tests
and
that
I
could
run
them.
F
My
goal
was
that
I
would
have
went
in
all
kind
of
corner
cases
when
this
score
doesn't
make
a
lot
of
sense.
For
example,
if
we
have
replica
three
and
all
the
osds
have
primary
Affinity
of
0.2
or
something
you
can't
actually
meet
all
the
requirements
without
you.
Actually,
you
know
you're
breaking
the
the
rules
of
the
game,
because
you
give
more
primaries
than
what
the
primary
Affinity
asks
you
to
do
it
obviously
it's
possible
if
the
total
primary
acidity
is
too
low.
F
So
I
try
to
do
something
that
in
all
these
Corner
cases
at
least
zero
and
in
all
other
cases,
score
is
always
larger
than
one
and
has
meaningful
something
that
you
could
look
at
the
numbers
and
understand
something
out
of
these
I
still
struggle
with
this
actually
so
I
have
I
could
easily
explain
the
score
when
the
the
the
total,
the
primary
Affinity
is
high,
and
it
tends
to
be
more
and
more.
F
That's
got
it
abstract
number,
when
the
total
Prime
Affinity
of
all
those
is
becoming
low
and
reaching
near
the
the
one.
The
the
one
divided
by
a
replica
count
so
now
I
think
about
a
bit
of
different
approach
of
starting
the
the
score
again
when
I
have
the
primary
low
primary
Affinity
I'm
working
on
this,
but
I
still
need
to
worst
case
scenario.
F
Is
that
the
in
some
cases
the
score
would
be
smaller
than
one,
and
there
will
be
a
documentation
explaining
that
this
is
something
that
says
something
is
bad,
but
I
I,
probably
I
would
be
able
to
fix
it,
and
the
the
only
making
the
score
always
larger
than
one
is
simple.
The
the
complex
is
making
it
meaningful
so
that
you
could
look
at
it
and
understand
something
else:
it's
how
how
much
your
system
is
balanced
or
not
balanced.
The
the
goal
is
that
one
is
perfect.
F
1.1
means
10
degradation
in
read:
performance
1.2
meant
20
degradation.
That's
the
goal
I'm
working
on
this,
because
it's
it's
It's
Tricky,
when
the
primary
Affinity
numbers
are
random,
they
shouldn't
be
because
they
should
have
some
logic
behind
them,
but
I
don't
want
to
count
on
exactly
how
systems
are
set
in
the
field
and
say
that
this
is
the
illegal
configuration.
Well,
if
it's
accepted
by
says
it
should
be
handled
correctly,
though.
F
Currently
there
is
a
minor
tweaks
to
the
numbers
and
and
to
the
unit
test,
making
sure
that
the
hope
to
make
sure
that
actually
I
I'm
able
to
run
it
currently
I
have
something
like
I
failure
in
unit
test,
one
in
500
runs
I
want
to
make
it
never
fail.
Okay,
so
it's
still.
F
F
I
thought
that
I
had
the
formula
set
very
well
in
the
mathematics
were
good,
but
it
was
incorrect.
I'm
still
struggling
with
the
edge
cases
of
the
mass
over
there.
A
A
Good
luck!
Good!
Luck!
All
right!
Let's
see
I
think
that's
all
I
had
for
for
pulling
requests
this
week.
Guys
was
there
anything
I
missed
from
anyone.
A
All
right,
if
not
then
moving
on
Josh
right
before
I
I
I,
see
you've
got
a
lot.
So
I
want
to
speak
something
in
quick
before
before
I.
Let
you
go
yeah
for
sure.
So
Adam
has
been
working
on
all
this
work
for
making
snapshots
and
and
RBD
mirror
much
faster
and
and
his
results
are
amazing.
A
I've
I've
linked
the
spreadsheet
there
for
anyone
that
wants
to
take
a
look,
but
the
gist
of
it
is
that
his
new,
this
new
code
is
is,
is
really
really
good
and
probably
I
I
would
say,
maybe
obsolete's
the
need
for
my
object,
defragmentation
branch
that
I
have
worked
on
because
it
it's
so
fast
that
I
think
it
it
doesn't
even
need
it
Adam.
Is
there
anything
you
want
to
to
talk
about
with
with
your
work
other
than
the
fact
that
it's
amazing.
E
Well,
it's
the
code
itself
is
not
fast,
it's
actually
quite
slow,
but
the
modification
is
that,
during
a
cloning
of
object,
making
a
duplicate,
in
fact,
there
is
an
aggressive
procedure
that
tries
to
avoid
creation
of
any
extra
blobs.
It
tries
very
hard
to
keep
the
count
of
blobs
minimum.
E
So
when
faced
with
the
problem
with
constant
up
dreaming
and
creation
of
new
snapshots,
we
do
not
increase
amount
of
blobs
in
Object
Store.
That's
the
speed
up.
It's
completely
unrelated
Mark
to
your
work
on
copying
objects
from
place
to
place,
because
data
of
objects
will
still
be
fragmented.
It's.
E
A
Yeah
Adam
May
the
reason
I
I
said
it.
The
way
I
did
is
because
I
I
think
with
your
new
code.
It
appears
that
that
essentially,
the
only
benefit
from
my
version
now
is
the
fact
that
you're
getting
defragmentation
I
don't
think
my
my
code
will
in
any
way
help
improve
CPU
usage
at
this
point
after
after
your
work
is
applied.
A
Not
to
say
that
we
we
still
might
want
to
defragment
objects
on
clone
just
to
to
you
know
at
some
point
reduce
the
amount
of
fragmentation
that
exists,
but
I
think
I
think
with
your
latest
code,
assuming
there
are
no
problems
and
it
works
in
production.
The
way
it
does
now.
It's
looking
really
really
good.
E
A
Yep
yep
all
right,
so
that
was
that
was
all
I
wanted
to
bring
up
with
with
that
work,
exciting,
exciting
stuff,
so
Josh
I'm,
going
to
turn
it
over
to
you
looks
like
you've
got
really
good
updates
for
us
here.
G
Yeah
well,
I,
guess
we'll
see
whether
the
quality
actually
is
that
great,
but
I
did
want
to
give
an
update
since
I
brought
up
these
issues.
I
feel
like
it
was
two
months
ago
now
or
something
like
that,
with
a
break
in
between.
G
So
just
a
reminder
that
there's
two
things
that
we
saw
in
Pacific
once
we
were
upgrading
between
Donaldson
Pacific
that
were
puzzling,
and
this
is
mostly
in
the
rdd
side.
We
don't
have
a
lot
of
data
points
yet
for
our
rtw
workloads,
the
first
one
is
the
right
amplification
issue,
not
a
lot
to
say
here
we
haven't
been
putting
a
lot
of
focus
on
it.
I
did
for
your
thoughts.
Mark
I
did
quickly
look
at
some
block.
Trace
myself
and
nothing
was
obvious.
G
G
What
with
what,
when,
when
I
started
to
do
in
Prometheus,
was
we
have
the
ceph
reported,
how
many
rights
are
clients
doing
and
then,
of
course,
we
have
the
disk
rate
stats
from
every
single
one
of
the
nodes,
and
so
what
I
computed
was
how
many
disk
rates
are
happening
on
average
for
every
single
client
right
that
comes
from
Seth,
and
it's
pretty
clear
across
the
upgrade
threshold
that
it
jumps
from
it
depends
on
the
cluster.
G
It
jumps
from
two
to
three
OSD
writes
per
client
rate
up
to
three
to
four
per
OSD,
so
we're
3x
replicated.
So
this
is
jumping
from
six
six
to
nine
up
to
nine
to
twelve
essentially
writes
across
the
cluster
for
every
client
right
so
like
it
was
pretty
clear
that
basically,
it
looks
like
for
a
some
number
of
volumes.
There
is
one
extra
right
that
is
happening
for
every
client
right
that
comes
into
the
cluster.
G
That's
really!
As
far
as
we've
got
I
don't
know,
I
think
we
had
a
lot
of
questions
in
our
minds
about
like
did
object,
map
implementation
change
in
some
way.
Could
this
be
somehow
snap
mapper
related
we're
like
we're
just
not
right,
and
especially
since
we're
not
convinced
we're
actually
seeing
this
in
our
staging
clusters?
Our
staging
clusters
do
not
have
the
same
workloads
as
production
right,
so
that
makes
it.
A
G
The
problem
yeah
yeah,
exactly
yeah
yeah,
so
I'll
put
it
as
we're,
not
convinced.
We
see
it
in
staging
the
reason
why
we're
not
convinced
is,
unfortunately,
we
don't
have
the
Prometheus
history
to
be
able
to
run
the
same
level
of
analysis
because
we
only
realized.
This
was
a
issue
once
we
started
seeing
it
in
prod.
G
Okay,
we
have
one
staging
cluster
where
we
do
have
the
history
and
it's
actually
really
weird
because
it
looks
like
it
actually
has
the
amplification
for
like
three
days
and
then
it
drops
back
down
to
normal
again,
but
we
have
not
seen
that
in
fraud
and
where
it
dropped
off
in
staging
is
when
we
did
the
conversions
column
sharding
in
rocksdb
and
so
we're
like.
Oh
that's
weird.
Maybe
we
should
just
do
that.
G
So
we've
done
that
for
one
or
two
two
I
think
full
clusters
of
production,
and
it
did
not
make
a
difference
there.
So
Luke
I'd,
like
we
don't
know,
but
it's
like
our
single
staging
data
point
I
I,
take
in
almost
any
data
from
staging
clusters.
I
take
with
a
grain
of
salt,
so
I
wouldn't
put
too
much
weight
on
what
we
saw
there:
okay,
but
but
yeah
exactly
what
you
say.
G
We
don't
have
any
sort
of
isolated
case
where
we
know
we
can
reproduce
this
other
than
we
just
know
what's
happening
for
all
of
our
production
clusters
on
RVP,
when
I
do
the
same
analysis
of
disk
rights
per
client
right
for
our
rgw
implementations,
I,
don't
see
any
difference
between
Pacific
and
Nautilus,
but
we're
we're
not
nearly
as
advanced
there.
Yet
in
how
many
clusters
we've
upgraded.
So
I
still
need
to
see
some
across
that
boundary.
Okay,.
G
Yeah,
okay,
that's
a
good
point.
I
haven't
thought
of
overrides,
yet
we
were
trying
to
brainstorm
like
what
are
all
the
RBD
specific
things
and
so
like
Alex.
He
his
first
thought
was
okay.
Well,
let
me
just
go
and
disable
all
of
the
RBD
features
for
a
volume
staging
and
see
if
it
makes
a
difference.
But
again
it's
like.
Are
you
actually
seeing
this
in
staging?
We
don't
know
so
yeah,
oh
yeah,
so
anyways,
that's
where
that's
at
I
don't
feel
like.
We
quite
have
enough
data
here
to
like
write
a
ticket.
G
B
G
A
B
B
G
E
B
G
Okay,
all
right,
that's
interesting!
Because
I've,
my
my
thought
immediately
went
to
deferred
rights
because
I
know
there
were
changes
around
that
in
I,
think
Pacific
time
frame
and
but
at
first
I
thought
it
was
only
if
you
had
like
hybrid
installs
and
so
I
just
kind
of
discarded.
That
idea.
B
Well,
and
so
the
question
is,
what
are
your
main
devices?
Are
these
feeding
drives
or
SSD
for.
G
These
this
is
all
SSC
mixture
of
SATA
and
our
older
stuff
and
nvme
and
our
new
stuff,
but
we
see
the
same
the
same
amplification
in
both
cases.
E
If
both
are
drives
SSD,
then
the
defaults
we
use
for
different
rights
basically
delete
any
non-necessary
different
rights
actions.
It
means
that
if
we
overwrite
a
partial
allocation
unit,
then
we
do
different
right,
but
that's
that's
only
it.
If
a
device
is
Spinner,
then
we
also
use
different
rights
for
performance,
and
in
that
case
we
we
will
do
extra.
A
So
what
about
the
case,
if,
if
they're
on
flash,
with
only
a
single
partition
and
they're
doing
overwrites
to
RBD
what
would
be
the
cases
in
which
we'd
be
doing
deferred
rights.
E
E
But
that
brings
me
to
a
question
Joshua
if,
if
You
observe
some
performance
degradation
due
to
additional
rights
or
you
just
notice
that
there
is
more
iOS
to
the
device,
so.
G
We
definitely
noticed
the
degradation
are
older
clusters
at
first
we've
since
made
some
tweaks
to
the
this
configuration
Hardware
configuration
to
compensate,
but
we
definitely
saw
a
latency
increase
there
after
we've
compensated.
For
that
honestly,
the
performance
hasn't
been
as
the
difference
is
minor
and
actually
the
only
place.
I
generally
see
the
performance
difference
is
in
the
p50,
so
the
median
right
latency,
the
median
right
latency,
tends
to
go
up
with
Pacific.
G
G
G
G
So
we're
a
little
bit
concerned
that
maybe
something
will
happen
there,
but
yeah
like
we
aren't
seeing
now
that
we've
mitigated
the
older
cluster
issue,
mostly,
we
aren't
actually
seeing
a
major
end
effect
from
this
and
it
at
this
point
it
remains
mostly
a
curiosity
and
then
a
where
concern,
especially
if
we
do
see
it
in
the
rgw
side,
because
we
are
well
I
mean
we're
running
Spinners
in
our
GW
side
without
separated
wall.
G
Fairly
convinced
that
we
don't
because
data
was
kind
of
mixed
okay,
I've
gone
back
and
forth
on
that
a
few
times,
because
every
time
I
look
at
the
data,
it's
like
sometimes
I
decide.
Yes,
sometimes
I
know.
If
it
is
it's
not.
A
A
G
G
A
There
we
we
discussed
that
PR
or
those
PRS
I
think
quite
a
bit
several
years
ago
in
this
meeting,
and
there
were
definitely
performance
effects
early
on
and
Igor
I
think
you
did
quite
a
bit
of
work
to
do
to
mitigate
a
lot
of
that.
G
A
G
What
I'm
seeing
here
is
I'm
just
looking
to
see
if
I
can
share
this
now,
there's
some
stuff
I
don't
want
to
share
in
this
tab.
Okay,
what
I'm
seeing
is
so
that
earlier
calculation
I
was
talking
about
where
I'm
calculating
how
many
disk
rights
are
happening
per
client
right
across
the
upgrade
threshold
we
went
from
an
average
of
9.5
just
rights.
G
G
A
Is
that's
really
big,
okay,
so
yeah?
Definitely
for
for
where
we
should
yeah.
We
should
track
this
down.
B
We'll
say
it
again:
what
was
the
difference.
G
B
B
So
I
found
the
ticket
which
fixes
different,
which
fixes
than
expected
different
rights
for
Legacy
clusters
in
Pacific,
and
this
happened.
4.6
Pacific
release,
okay,.
A
I'm
trying
to
think,
if
there's
any
way,
Igor,
we
could
verify
whether
or
not
it
was
those
changes
to
support
Working
Man
outcome,
Spinners
that
that
might
have
caused
this
like
if
there's
any
way
on
there,
because
they're
running
in
vme
drives.
If
there's
any
way,
we
can.
B
Well,
fifth
of
all,
we
can
enforce
HDD
settings
for
the
drives
and
hence
it
once
applied.
It
should
work
similarly
to
spinning
drives
in
terms
of
before
tried
to
our
stuff.
So.
B
A
What
I'm
wondering
if
it
was
possible
that
any
of
the
changes
that
were
made
at
that
time
could
have
impacted
the
different
right
path
on
on
flash
drives.
G
B
G
So
so
the
example
I
just
gave
was
from
a
production
cluster
that
production
cluster
would
have
been.
Let
me
check
its
history
quickly.
Here,
I
was
going
to
say
it
was
deployed
on
Nautilus,
but
I
should
make
sure
that
it
wasn't
deployed
on
luminous.
A
B
Well,
I
need
to
run
through
all
these
changes
once
again
go
answer.
This
question
foreign.
A
Oh
sorry,
you
wanna
talk
about
you
found.
C
I
was
just
recalling
that
one
that
Igor
and
Adam
had
both
worked
on
where
there
was
a
bug
in
Pacific.
That
still
is
there,
since
we
haven't
had
a
release
yet,
but
we're
Boulder
clusters
that
still
have
the
64k
Min
allocation
size
that
have
been
upgraded
to
Pacific
aren't
different
yeah.
It's
based
upon
the
small
size,
but
I
don't
know
if
that
makes
sense.
A
G
A
Good
yeah,
I
I
was
pretty
sure.
Luminous
speed
had
a
4K
metallic
size
for
ssds,
but
you
know
one
point
earlier
on:
it
was
16k.
E
I
guess
one
more
thing
could
have
happened.
If
you
use
you
use
one
good
device
or
two
devices
for
separate
block
data
and
roxdb
data.
E
Are
you
able
to
distinguish
rights
to
that
came
from
block
data
rights
and
rights
that
go
to
roxdb,
write
ahead
log
when.
G
I,
look
at
the
Block
Trace
I'm,
pretty
sure
I
can
figure
out
which
is
which
is
based
off
of
right
patterns,
but
like
there's
no
way
for
I,
don't
think
there's
any
way
for
me
to
do
that
at
the
stat
level.
G
E
But
to
make
sure
when
you
refer
that
there
is
an
increase
of
right
access
to
the
device
you
talking
about
the
sum
of
block
data
rights
and
any
metadata
size.
Okay,.
G
B
B
B
B
Well,
it'll
be
great
if
you
share
please
well
these
zombies,
maybe
some
short
short.
A
G
Okay,
so
I
I
reset
the
counters.
In
the
last,
like
five
seconds,
there
were
three
thousand
deferred
rights,
so
like
600
per
second
roughly,
it
was
very
rough.
B
G
Yeah,
okay,
yeah!
So,
very
briefly,
if
I
look
at
this,
the
done
counter
sorry
I'm,
trying
to
find
one.
That
would
be
like
what
is
the
right
count
here
and
reading
this
correctly,
it
looks
like
the
majority
of
the
rights
that
were
done.
B
G
B
E
Yes,
that
that
could
be.
We
just
don't
really
test
that
that
configuration
I,
guess.
A
G
A
Yeah
interesting.
G
G
G
A
All
right,
well
any
anything
else
from
anyone
before
we
we
wrap
up
today.
A
Thanks
for
sticking
around
everyone,
I
know
we
went
a
little
over
all
right.
Well,
let's,
let's
wrap
it
up
and
give
everyone
next
week.
Then.