►
From YouTube: Ceph Performance Meeting 2021-10-07
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
All
right:
well,
it
does
look
like
we
are
getting
people
streaming
in
now
from
the
course.
So,
let's
get
this
show
on
the
road
all
right.
So
it's
been
about
two
weeks
since
we
did
the
pull
request
summary
here
since
last
week
we
discussed
the
excellent
paper
that
josh
found
for
us.
A
So
let's
see,
there's
actually
not
a
whole
lot.
The
only
new
pr
I
saw
in
the
last
two
weeks
was
actually
for
me.
This
was
just
an
updated
implementation
of
the
cash
age
binning
for
the
priority
cash.
A
I
had
thought
that
it
was
all
in
very
good
working
condition,
but
it
turns
out
that
during
a
run-through
technology
that
there
were
six
volts,
so
I
need
to
go
back
and
take
a
look
specifically
at
the
changes
to
the
lru
cache
implementation
for
the
roxdb
block
cache
something
I
changed
for
the
age
bidding
there
apparently
is
slightly
broken,
so
I'll
be
looking
at
that
at
some
point.
Coming
up,
I
saw
two
closed
prs
in
the
last
two
weeks.
One
was
for
prometheus
manager
module.
A
I
thought
that
something
like
this
already
merged,
but
apparently
this
is
new
offered
the
ability
to
disable
the
cache
kifu
merged
that
after
testing.
So
anyway,
that's
that's
there
and
then
the
only
other
close
one
was
the
old
version
of
the
cache
binning
that
I
had
submitted.
Probably
a
year
ago
ago.
I
finally
closed
that
in
favor
of
this
new
updated
rebased
version
of
the
same
code,
there
were
a
couple
of
updated
pr's.
A
Let's
see,
neha
reviewed
the
manager,
ttl
cash
implementation,
there's
some
additional
ongoing
discussion
and
updates.
I
think,
based
on
her
feedback
igor,
it
looks
like
there's
a
little
bit
more
review
on
your
pg
removal,
optimization
pr
any.
I
think
that
previously
had
been
failing
tests
is
that
are
there
any
new
updates
there.
C
A
Well,
is
it
messenger.
A
There's
an
optimization
regarding
header
decoding,
I
guess
iliad
reviews
reviewed
that,
and
there
were
some
additional
this
there
was
some
additional
discussion
and
updates.
I
believe,
but
I'm
not
totally
sure
what
status
of
that
is
beyond
that,
and
then
this
big
one
for
subtree
map
removal
from
the
journal-
I'm
very
excited
about
this,
but
it's
a
big,
complicated
pr
and
there
there
was
some
additional
review
and
update
and
rebasing
of
that
pr.
A
A
And
I
I
know
in
the
past
when
I've
talked
to
patrick
he's,
he's
mentioned
that
he
may
want
to
try
to
break
this
up
into
smaller
pieces
just
because
it
is
so
so
big
and
complicated.
So
we'll
see
what
happens
with
this,
but
I
am
still
very
excited
about
it.
A
No
movement,
lots
of
things
here-
sage
reiterated
that
he'll
he'll
do
a
final
review
on
the
bluefs,
fine,
green,
locking
from
adam,
which
is
really
good,
since
I
think
igor,
probably
you
and-
and
maybe
after
that
me
are
the
only
other
people
that
can
really
review
this,
and
I
know
I
I
very
much
believe
that
you
are
more
qualified
to
do
it
than
I
am,
but
you
know
if
necessary,
we
can
maybe
also
try
try
to
review
this,
but
but
sage
agreed
to
do
it
first.
A
So
that's
good!
Let's
see
what
else.
A
Not
a
whole
lot
of
other
things,
this
this
randomly
distributing
difference
to
multiple
mbss
needs
a
rebase.
Hopefully
this
ffs
team
could
take
a
look
at
that.
A
I
think
now
that
mark
cogan
is
back
he's
he's
working
on
this
shared
object,
cache
pr
that
for
me
updating
that
and
and
probably
expanding
it
to
be
even
more
useful.
So
that's
good.
A
I
don't
really
see
anything
else
here.
That's
worth
talking
about
this
week.
Did
I
miss
anything
folks,
any
anything
there
that
I
did
see
or
would
that
be
worth
talking
about.
A
All
right,
then,
moving
on.
I
only
have
two
fairly
brief
discussion
topics
for
this
week.
One
is
that,
with
the
the
the
downstream
testing
that
red
hat
does
there
was
a
discovery
that,
on
some
of
the
nodes
that
were
being
tested,
there
was
a
huge
amount
of
memory
that
was
utilized
on
the
nodes
that
didn't
appear
to
be
freeable
that
was
showing
up
in
the
proc
meme
info
per
cpu
category.
A
It
appears
that
we
may
be
hitting
an
upstream
bug
when
creating
and
destroying
many
many
c
groups
rapidly
we're
not
100
sure
on
this,
but
that's
kind
of
our
best
guess
right
now,
so
for
anyone,
that's
using
stuff
in
c
groups
specifically
session
containers,
this
may
be
relevant
for
you
so
check
out
the
upstream
kernel
bug
the
guy
that's
been
working
on
that
has
been
working
on
many
related
issues
as
well
in
the
kernel
regarding
seekers
specifically
to
support
their
usage
at
facebook.
A
So
definitely,
if
you're
you're
running
into
issues
with
containers,
you
see
groups
of
memory
usage,
especially
slow,
leaks
over
time
that
are
unexplainable.
This
may
be
what
you're
looking
for.
A
So
the
other
thing
that
I've
got
is
that
to
try
to
get
the
cache
age
binning
into
a
shape
where
we
can
merge
it.
I've
been
running
through
a
lot
of
tests
with
it
for
people
in
the
core
group.
You've
already
seen
all
this
for
other
people,
the
gist
of
it
is
that
the
cage
cash
age
binning
is
not
as
much
of
a
performance
win
as
I
was
really
hoping
it
would
be.
A
It
turns
out
that
that
when
we
do
age
bidding,
we
really
don't
actually
even
cache
omap
data
very
much.
The
only
cases
where
we
really
do
is
during
like
rgw
bucket
index
listing
for
for,
like
rgw
workloads.
If
we
ran
a
pure
omap
test,
we
probably
would
making
more,
but
that's
kind
of
rare
in
in
practice.
A
You
know
it
looks
like
bucket
index
listing
is
probably
the
the
major
common
use
case
where
we
would
see
a
lot
of
hot
o
nodes
in
the
cache,
and
that
use
case
typically
is
is
like
scanning
right
is
iterating
over
a
bunch
of
values.
So
I
I
guess
you
would
assume
that
read
ahead
would
be
far
more
important
than
something
like
storing
these
in
a
a
longer-term
lru
cache.
A
One
of
the
things
we
did
see,
however,
with
cash
bidding,
is
that
we're
doing
a
better
job
of
removing
old
o
nodes
and
because
we're
doing
that,
we
actually
see
less
fragmentation
when,
when
workloads
change
between
like
large
sequential
reads
or
rights
and
like
small
random,
reads
or
rights,
the
upside
here
is
that
it
seems
like
we.
We
actually
do
a
better
job
of
of
not
fragmenting
memory
as
significantly,
and
that
lets
us
have
more.
A
Cash,
doesn't
necessarily
always
turn
into
a
performance
gain,
and
you
know,
maybe
that's
because
of
other
bottlenecks.
Maybe
it's
because
caching,
the
kind
of
seldomly
used
or
older,
oh
no's,
doesn't
actually
matter
that
much,
but
in
the
event
it
appears
that
we
do
do
a
better
job
with
memory.
A
Fragmentation
with
this
enabled
and
adam
kind
of
convinced
me
that
that
maybe
I
was
being
pessimistic
and
it's
still
worth
actually
merging
this
so
and
in
the
event,
there
are
some
graphs
there
that
I've
actually
got
a
ton
more
data
beyond
what's
in
those
spreadsheet,
but
that
gives
you
kind
of
a
a
decent
high
level
overview
of.
Can
the
behavior
changes
from
page
binning
versus
the
current
master
behavior?
A
One
thing
to
add
there
is
that
the
age
binning
failed
qa
due
to
sig
faults
when
somewhere
in
the
there
are
rex
tv,
lru
cache,
probably
there's
just
a
bug
in
the
the
age
bidding
implementation
for
that
particular
cache,
so
I'll
just
have
to
go
back
and
figure
out
what
it
is,
but
you
know
otherwise
it
did
actually
get
through.
All
of
my
performance
tests
without
issues
so
seems
like
this
is,
is
somewhat
rare.
A
We'll
just
have
to
figure
out
what
it
is
before
we
merge
it
and
that's
all
I've
got
guys.
I
see
something:
that's
updated
in
the
chat
window.
E
Yeah,
so
I
was
just
wondering
if
maybe
I
was
curious
if
you're
looking
at
immediate
effects-
and
maybe
this
change
will
be
oh
age,
binning
thanks,
I
said
h-
I
was
thinking.
Maybe
it
would
be
more
of
a
long-term
effect
type
situation
where
if
we
merge
it
we'll
see
more
long-term
effects
or
you
know
since
you're
not
seeing
it
right
away.
Maybe
it
still
might
be
worth
it
long
term.
A
Yeah
yeah
that
was
actually
kind
of
the
intention
with
it
is
that
you
know,
rather
than
a
a
super
immediate
effect.
You
know
we
would
kind
of
gradually
migrate
cache
over
to
like
oh
map.
If
we
need
to
map
more
or
oh
note,
if
we
need
oh,
no
or
or
data,
if
we
need
data
more
and
it
does,
that,
I
mean
it's.
Actually,
you
know
if
you
look
at
like
the
memory
usage
graphs,
it's
actually
doing
a
fairly
good
job
with
that.
A
It's
just
that,
like
we're,
we're
so
dominated
by
performance,
drops
due
to
oh
node.
Cache
misses
that,
like
you
almost
always
want
to
give
everything
to
oh
node,
there
are
cases
where
data
cache
giving
it
to
data
cache
is
valuable
and
that's
like
the
times
when
this
is
actually
faster
are
when
you
have
like
a
mixed,
read,
write
workload
where
hitting
a
cached
read
lets
you
both
hit
the
o
node
from
cache
and
the
data
from
cache.
A
It
turns
out
that
that
oh
node
is
still
like
the
dominating
factor
in
all
of
this,
which
is
kind
of
unfortunate
and
sad,
but
but
it
is
what
we
it
is,
what
it
is
right
now.
I
still
think
cash
age
spinning
is,
is
yeah
kind
of
useful,
because
if
that
changes,
if
we
get
to
the
point
where
that's
no
longer
the
case,
then
then
the
age
meaning
should
really
dramatically
help,
and
there
are
cases
where
it
does
help.
It's
just
I
like
to
say
what
we're
we're
kind
of.
A
We
have
like
accidentally,
optimal
scenarios
right
now,
where,
where
we
end
up
like
devoting
a
large,
a
huge
percentage
of
the
cash
to
o
node-
and
you
know
it
seemed
to
help.
So
that's
what
we
did,
but
we
never
really
knew
why
or
if
that
really
was
optimal
it
turns
out.
It
is.
E
Yeah,
well,
it
could
still
help
in
those
cases
and,
like
you
said,
if
that
changes
in
the
future,
you
know
if
the
design
changes
in
the
future,
that
as
long
as
it's
not
causing
major
regressions,
I
think
it
makes
sense
if
it's
kind
of
keeping
things
relatively
the
same
things
could
change,
and
then
it
could
become
really
optimal.
A
The
ones
where
it
would
hurt
are
like
when
you've,
if
you've
done,
like
a
big
dump
of
data,
so
like
you've,
you've
done
a
fill
stage
like
you're,
just
like
pumping
stuff
full
of
data,
and
then
you
go
back
later
and
are
doing
like
little
random
reach
in
that
data
that
you
just
wrote
in
master
right
now.
We
almost
just
ignore
the
data
portion
of
those
rights
and
we
just
cache
the
nodes.
We've
all
always
are
like.
A
Like
oh
no's
are
coming
first,
always
with
cash
age
bidding,
it
will
say:
hey,
there's
a
bunch
of
big
rights
happening
like
four
megabyte
rights,
and
these
o
nodes
are
are
kind
of
small
relative
to
that
I'm
gonna
catch
the
data,
I'm
gonna,
assume
kind
of
in
the
future
that,
because
I'm
doing
big
rights
now,
I'm
probably
gonna
do
big
reads
coming
up
at
some
point,
and
so
it
will
favor
data
a
little
heavier
in
terms
of
like
a
byte
value
than
the
no
nodes.
A
But
if
later,
you
came
back
and
did
like
little
random
reads
from
those
you'd
actually
favor
want
to
have
the
oh
notes
cache
rather
than
the
data,
whereas,
if
you're
going
to
come
back
and
do
like
big
reads
from
that
data,
then
you'd
want
to
favor
data.
Those
are
kind
of
the
the
cases
where
the
existing
behavior
versus
the
age
being
behavior
different.
F
I
still
kind
of
hope
that
if
we
merge
it,
that
will
give
us
at
least
some
framework
to
try
to
improve
it
and
on
situations
that
we
see
that
could
be
improved
because
now
we're
testing
on
some
cases.
I
guess
when
other
people
will
test,
we
could
actually
adapt
better
to
have
a
better
aging
tables
for
different
different
type
of
caches.
A
Yeah,
I
agree
adam,
I
I
think
it
it.
It
gives
us
a
saner
way
of
dealing
with
all
of
this
than
what
we
have
now.
I
I
I
like
to
use
the
term,
but
I
feel
like
we're
like
accidentally
good
right
now.
It's
just
kind
of
because
oh
nodes
are
so
dominant
from
a
performance
perspective,
catching
them
that
that
we're
we
happen
to
be
good
because
we
favor
them,
but
if
we,
if
we
like
merged
igor's
pr
from
a
while
back,
that
makes
it
so
that
the
owner
data
structures
are
smaller.
A
It
might
shift
things
a
little
even
more
towards
wanting
to
cache
data
rewind
to
cache
omap.
So
like
the
more.
If
we
ever
do
that,
if
we,
if
we
merge
that
or
merge
other
things
like
that,
where
it
means
favoring
o
nodes
is,
is
less
important
that
the
cache
age
binding,
I
think,
will
even
you
know,
kind
of
start,
showcasing
more
benefit
automatically.
In
the
background.
A
I
suppose
the
counter
argument
to
all
of
this
is
that
if
we
start
really
putting
a
lot
more
resources
towards
crimson
and
secure
that
will
work,
you
know
completely
differently.
It's
not
clear
that
we
want
to
do
something
like
this
with
the
store.
I
don't
know
we
might
just.
That
would
be
a
much
bigger
conversation.
A
Maybe
you
don't
want
to
be
doing
this
kind
of
dance
of
of
trying
to
keep
memory
under
a
certain
boundary
and
reallocating
memory
for
different
caches
and
moving
things
around
and
evicting
gear
and
gaining
here,
maybe
in
that
kind
of
a
world
you'd,
rather
just
have
a
static
allocation
of
memory
for
different
things,
and
you
always
stay
within
under
a
threshold
because
you
you're
allocating
from
pre
allocated
pools
and
and
and
there
things
are
just
much
more
static.
F
A
A
I
think
it's
probably
a
better
design
than
all
of
this.
You
know
goofiness
that
that
we've
done
to
try
to
shore
up
our
existing
behavior
in
classic
right.
You
know
the
the
the
memory
auto
tuning
and
the
cash
balancing
and
the
age
bidding
all
this
stuff
is
really
just
trying
to
work
around
the
fact
that
we
make
a
ton
of
small
allocations
constantly.
E
Yeah
sort
of
seems
like
a
we
do
this,
because
this
existing
thing
happens
and
this
existing
thing
is
because
of
this
other
thing,
it's
sort
of
like
all
these
different
infrastructure
decisions
are
affecting
each
other
so
that
it's
a
little
bit
difficult
or
we're
constantly
doing.
Workarounds
almost,
but
I
mean
what
you
said
about
merging
this
pr
to
sort
of
like,
even
if
it
doesn't
make
a
huge
change
right
away
to
sort
of
set
the
ground
for
moving
into
creating
a
better
infrastructure.
That
is
more
sane.
E
A
A
I
I
think
it's
it's
more
or
less
mostly
fine,
but
the
fact
that
we
hit
a
sig
fault
in
testing
is
a
little
discouraging.
You
know
it's.
It's
I
like
to
think
of
this
is
fairly
clever.
The
way
that
this
is
working,
but
it's
definitely
touching
a
lot
of
stuff
that
you
know
introduce
bugs.
So
the
downside.
C
Yeah
in
our
pre
one
of
our
previous
discussions
about
having
multiple
pools
or
just
few
pools,
especially
with
reddish
gateway.
C
So
if,
if
you
would
have
better
hiddens
what
to
do
with
the
data
and
how
the
data
would
be
used,
so
this
could
be
something
that
we
could
leverage,
then
to
just
educate
the
changes
for
cache
behavior.
C
So
it
would
be
one
of
the
things
that
we
could
do.
Perhaps
if
you
had
some
something
like
this.
A
Way,
I
should
look,
I
don't
know
if
we
have
any
way
to
to
track
like
a
piece
of
data
in
the
cache,
what
what
pool
it's
tied
to,
but
that
would
be
also
really
interesting
from
a
display
perspective
right
like
if
you
could
say
that
okay,
you've
got
all
this
data
in
these
different
lru
caches.
A
You
know
this
is
data
versus
metadata
for
oh
nodes
versus
map
data,
but
then,
if
you
could
also
present
the
user
with
a
view
of
well
and
and
here's
the
breakdown
and
percentage
of
what
pools
the
this
kind
of
data
is
tied
to
like
this
particular
pool
is
getting
a
lot
of
cash
right
now
I
mean
that
could
be
kind
of
interesting
from
like
a
dashboard
perspective.
C
Yeah,
perhaps
not
only
from
this
site,
just
monitoring
and
seeing
or
what
is
done,
but
also
just
if
we
could
have
some
hint
what
the
data
is
about
or
just
is
this,
and
if
there
is
an
upload
of
a
large
object
as
it's
something
else.
This
is
a
tiny
thing,
so
I
don't
know
just
I
know
that
other
ecosystem
think
about
just
giving
down
hints
from
application
site
also
just
to
educate.
C
This
is
a
content
that
needs
to
be
cached,
because
I
would
read
it
once
again
soon
and
all
this
information
just
wouldn't
have
any
any
meaning
to
us,
because
we
cannot
deal
with
that.
Or
can
I
prepare
for
it
if
you
would
just
have
simply
the
the
additional
way,
the
actual
behavior?
C
A
All
right:
well
then,
I
don't
have
anything
else,
so
we
can
wrap
this
up
and
get
a
little
bit
of
our
time
back.
E
That's
good
I'll,
see
you
in
the
huddle
later.