►
From YouTube: 2019-04-18 :: Ceph Performance meeting
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
All
right,
well
I'll,
just
quickly
go
over
some
of
these
yars,
then
Adam.
Maybe
you
can
get
started
on
your
stuff
after
that,
all
right,
there's
two
new
ones
here,
the
first
one
is
kind
of
interesting:
it's
pulling
a
poll
events
from
user
space
rather
than
Colonel
and
he's
seeing
a
reasonably
decent
performance
improvement
from
it
in
the
in
the
async
messenger.
So
that
sounds
good
to
me.
I
I
can't
look
through
the
code
and
realized
I.
Don't
really
know
he
pulled
that.
A
A
A
A
Very
exciting
for
me
all
right,
so
that's
that's
new.
If
anyone
wants
to
look
at
and
and
give
their
thumbs
up
or
not,
obviously
it
needs
to
go
through
testing,
but
it
sounds
good
to
me
there's
this
other
one
for
optimized
check
utf-8,
which
is
apparently
something
this
I
think
this
is
one
of
those
like
someone
like
entreated
to
see
star
and
then
had
the
same
thing
for
us.
So
yeah
I
mean
seems
fine
me.
A
Who's
reviewing
it,
though,
so,
should
be
good
to
peers
closed
both
were
already
stuff
one
was
this
dispatch
delayed
requests
only
a
free
dinner
sex,
Jason
reviewed
that
and
then
the
other
one
was
this
live
already
new
default
right
around
cash
policy?
Apparently
the
issues
that
were
there
previously
must
have
gotten
fixed
and
that's
exciting.
That's
that's
good
news,
I'm
very
curious
to
see
how
that
does
and
then
two
that
were
updated.
The
new
IOU
ring
io
engine
I.
Think
I
may
try
to
take
a
test
drive
on
that
time.
A
A
Yeah
I
just
related
to
this
MDS
cache
memory
limit
one
where
we
had
a
meeting
earlier
this
week
to
talk
about
the
priority.
Cache
manager
I
think
have
explained
how
that
works
that
got
merged
to
earlier
this
week,
though,
work
should
now
start
up
on
doing
like
memory
auto
tuning
for
the
other
demons.
I
think.
Maybe
the
Mon
is
actually
going
to
get
get
it
first,
but
potentially
Mon
MDS,
maybe
rgw
I,
don't
know
if
it's
worthwhile
there
or
not,
but
but
at
least
for
the
mountain
MDS
I,
think
there's
work
plant
which.
C
Infield
touch
sheet,
and
here
are
some
configurations
in
the
columns
configurations
for
how
many
columns
prepared
and
how
large
is
a
right,
a
head
lock
for
each
column
and
definitely
there
is
consistent
improvements
for
performance,
although
small
like
a
maximum,
is
for
four
point.
Twenty
nine
percent
improvement
in
iOS
and.
C
C
C
C
Can
you
can
look
wrong?
25
are
split
or
parameters
for
column,
families
and
right
ahead,
lock.
So
three
dream
means
it
:
at
128,
megabytes
or
write
a
headlock
which
of
course
means
leaders
will
also
not
do
to
say
that
or
each
column
family.
The
requires
each
on
right
ahead
log
and
it's
it's
totally
separated
from
from
other
ones,
and
you
can
see
here
that
objects
objects
are
split
into
13.
Sharks
and
ohm.
C
B
C
Its
I
don't
believe
them.
They.
They
of
course
show
some
real
numbers,
but
I'm,
not
100%
sure,
if
actually
that
feature
in
fear
of
the
beans,
splitting
keys
and
values
was
enabled.
The
last
test,
which
has
not
finished
well,
I,
made
a
modification
that
looking
at
the
code
that
convinced
me
that
that
that
may
be
a
test
when
I
finally
got
that
feature
home,
but
I'm
waiting
configuration
from
Remington.
C
Mr.
Remington,
if
that
story-
because
it's
never
know
ATM,
it
may
be
stated
somewhere,
I-
wasn't
able
to
actually
find
a
manual
how
to
enable
this
feature
for
:
family
lore.
If
you
have
one
just
for
all
the
database,
it's
just
my
reading
from
code,
but
it
turns
up
a
bit
even
gives
even
more
more
numbers.
I
mean
the
hundred
percent
in
each
column
is
a
result
that
only
lasts
value
is
used
as
reference.
So
there
is
nothing
mean.
C
C
Okay,
so
that's
that's
the
wrong
above
this
I
hopes
per
second
I
hope
so
operations
per
second
and
there's
also
other
table
which
measures
it
started
wrong,
87,
which
measures
how
much
more
epu
we
consumed
to
actually
do
that
that
work.
This
is
obtained
by
just
Perth
that
are
starting
always
the
entire
OSD,
not
only
OSD
process,
I
mean
each
each
OSD
separately.
Of
course,.
C
C
A
It's
you
know
all
the
testing
I've
done
having
larger
right
ahead.
Log
buffers
usually
means
that
we
see
less
right
amplification
in
the
during
compaction,
I
suspect,
probably
because
we're
getting
the
tombstones
for
PG
log
updates
landing
within
the
same
buffer
and
so
before
it
ever
makes
it
into
level
zero.
It
can
just
be
eradicated,
which
is.
C
A
Don't
know
I
honestly
I
haven't
even
tried
any
of
this
kind
of
stuff
yet
with
like
separate
calm,
family
right
head
logs,
just
in
my
other
stuff
that
I
was
working
on
earlier
this
week,
I
just
started
looking
at
different,
like
cache
block
caches
for
different
column
families,
and
that
ended
up
helping
a
lot.
But
that
was
that's
unrelated
to
your
stuff,
I
I.
C
Pins,
how
did
it
may
may
impart,
because
I'm
actually
clueless
here
I
met
Blake
that
code
and
because
I
make
a
measurement
just
from
a
very
high
that
details?
Why
specific
change
gives
a
booth
is
totally
obscured
from
from
me,
I
I
tried
to
get
paraffin
Alice's
on
that,
but
it
seems
that
each
case
and
each
test
case
and
each
setting
sometimes
varies
very
much
from
the
point
where
it's
the
most
TPU
intense.
So
that's
makes
it
difficult
to
actually
pinpoint
linear
dependency
on
setting
what.
A
I
think
would
be
really
interesting
to
see
with
your
code
is
if
you
did
use
Rados
bench
or
something
whatever
it
doesn't
matter
to
do
a
bunch
of
small
Oh
map
rights.
So
do
like
you
know,
4k
or
I,
don't
know
whatever
you
want,
but
just
lots
and
lots
of
small
OMAP
rights.
You
fill
it
up
with
like
512
gigabytes
worth
and
then
watch
the
compaction
stats
in
the
OSD
log
or
you
know
whatever.
C
A
Cool
I
still
wonder
if
the
biggest
advantage
of
doing
something
like
this
would
be
to
have
the
kind
of
parallel
level
hierarchies
where,
instead
of
doing
a
bunch
of
having
tons
of
data
in
level
three
where
you
have
high
ramp,
liffe,
ocation
and
long
compactions.
If
now,
you've
got
more
data
in
level.
Two
multiple
level
twos
and
maybe
those
compactions
don't
take
as
long
I,
Matt,
I
think
on
customers,
sometimes
or
users
we've
seen
like
for
rgw,
like
long
compactions
in
practice,
if
I
remember
right,
yes,.
B
Well,
certainly,
compact
certainly
and
work
those
had
a
substantial
component
where
delete
component
of
deletes,
then
the
compaction
to
say
long
compactions,
but
yes,
basically,
compaction
tends
to
not
and
ends
up
dominating
workload
and
and
and
and
and
it
and
it
is
with
artisan
and-
and
it
is
frequently
the
case
that
you
know
it's.
A
B
It
sounds
like
tea
rocks
me.
This
is
a
question
for
Adam.
It's
done.
We
only
had
couple
days,
but
it
sounds
like
to
hear
it
sounds
like
it
sounds
like
my
team
they're,
the
interactions
with
with
our
mansion
was
a
little
bizarre.
I,
don't
think
I
get
the
photos.
I
haven't
gotten
that
good,
like
a
major.
If
he
rocks,
is
still
just
gonna
starting
with
isn't
so
you
did.
We
shouldn't
trust
the
evaluation
yet
or
do
you
feel
differently,
palette,
Adam,.
C
C
C
C
C
C
B
Ups
also
they're
they're
not
claiming
speed
on
this,
but
the
Oconee
Indian
loss
attraction,
data
movement
so
I'm
not
quite
sure
how
that
works,
at
least
in
the
CRX
report
that
they
send
as
initially
so
yeah
and
I'm
just
I'm
just
looking
to
learn
more
about
it.
But
if
you
don't
put
it
but
in
but
in
its
you
know,
there's
one
of
the
few
it's
one
of
the
it
seems
to
be
in
the
same
direction.
Is
that
it's
one
of
the
few
sort
of
scientifically
validated
provements
to
this
passive
system?
I.
C
B
C
The
really
good
good
part
is
that,
after
fixing
the
compilation
which
I'm
not
actually
amazed,
why
somebody
that
made
so
good
code
actually
left
defines
in
class
members
top
of
Abbott's
mysterious
for
me,
but
after
I
made
a
change
to
that,
it
really
works
without
hiccups,
so
it
maybe
it
was
even
used
somewhere
I'm
it.
It
definitely
looks
so
it's
too
good
to
be
just
completed
right
of
that
and
be
so
mature
that.
B
B
A
B
C
Mean
the
only
problem
might
be
that
I'm,
not
capturing
all
logs
and
all
data
that
might
be
useful,
so
I
will
have
to
rerun
some
test
to
to
obtain
really
information.
But
on
that
stage
my
my
path,
I
think,
will
be
to
ask
Remington
dragged
braixen
to
provide
some
configuration
or
column
families
or
another
settings
that
he
thinks
are
most
most
important.
C
A
Want
to
say
to
the
performance
may
not
actually
be
the
thing
that
really
matters
here
right
like
it's
nice
if
we
get
a
performance
gain,
but
the
bigger
improvement
might
be
that
if
there's
less
right
amplification
on
the
flash
device,
it
may
improve
how
long
the
device
lasts
for
users,
even
if
it
doesn't
make
it
faster.
If
you
know,
if
the
device
is
fast
enough,
it
can
just
absorb
the
rights,
and
you
know
the
same
speed
just
having
it
do
less.
Work
might
be
really
important
to
make
their
their
hardware
last
well.
B
D
B
Kinds
of
flash-
obviously,
here's
a
whole
lot
about
that,
and
maybe
that's
relevant,
but
you
know,
but
but
it
isn't
as
valuable
as
being
things
as
speeding
up
connections
and
for
at
least
for
the
click
for
the
users.
But
it's
for
they
can
challenges.
We
have,
or
at
least
in
rgw,
with
a
large
with
large
old
map
and.
A
B
It's
not
it's,
not
it's
not
gonna
Lille
to
there
a
satisfying
result
for
most
of
them,
I,
agree,
but
anything
that
we
can
do
to
maybe
lessen
the
pain
that
it
makes
sense.
I
mean
they're,
first-time
deployment,
it
could
be
usually
beneficial
and
if
it
doesn't,
if
it
isn't
drop-off
performance,
I
assume
it'll
be
part
of
some
solution,
but
that
morning
or
to
achieve
yeah,
what
about
I
mean
more
broadly?
B
Do
you
ever
see
in
any
testing
that
we
do,
and
this
goes
to
the
workload
top
point
you
just
raised
in
terms
of
forcing
confections
on
what
we
see
we
see
and
in
the
worst
case
scenarios?
Where
are
we
before
we've
aware
we
have
old
excessive?
We
have.
We
have
sometimes
it's
been
because
I've
been
sometimes
it's
for
nourishment
in
a
continuing
or
in
response
to
a
issues
in
Rodge
w.
They
should
really
have
been
there
like
lurking
that,
hopefully
we
didn't
know
in
women
when
we
were
improper.
B
If
we,
if
we
know
they
were
there
there,
there
have
been
folks
where
we
didn't
or
we
didn't
properly
clean
up.
All
all
all
the
key
read
all
the
key
sequences
for
you
know,
for
there
were
involved
in
sin,
multi-sensory,
condensation
data,
log
and
so
forth.
So
you
so
you
get
a
very
large
expansion
of
all
mapped
key
space
and
then
later
we
try
to
delete
it,
compacting
and,
and
so,
and
in
other
words,
in
some
of
the
in
various
real
workloads,
we
can't
we
get
operation.
B
Compaction
is
taking
so
long
that
that
OSD
stem
out
and
sort
of
tangential
to
the
to
the
simple
performance
topic
you
know,
we'd
like
to
make
progress
on
making
OSD
is
resilient,
so
that
sort
of
thing
doesn't
happen.
Yeah
and
I
wondered
if
you
know
and
I
wondered
if
you
defined
workloads
that
expose
that
know
and
allow
us
to
work
on.
You
know
know
how
to
make
I
know
how
to
make
the
system.
This
has
been
large.
You
know
degrade
gracefully
rather
rather
than
flap.
A
The
workloads
I've
seen
our
slowest
have,
like
you
said
Matt,
then
like
rgw,
deletes
those
suck,
but
but
on
our
Hardware
on
in
Sirte,
like
our
kind
of
community
test
Hardware
are
everything
is
fast
enough
that
the
OSD
isn't
very,
very
rarely
ever
timeout
I,
don't
think
I've
actually
seen
them
I'm
out
from
a
performance
workload,
maybe
in
a
very
long
time
in
the
early
days
like
new
store
blue
store,
we
could
do
it,
but
that
was
when
we
had
like
really
crappy
code
in
some
areas.
I
have
not
seen
anything.
B
B
B
B
D
D
It
sounds
like
more
of
a
discussion
about
adaptive
timeouts.
The
timeit's
were
deliberately
stupid
right,
there's,
nothing
clever
about
them.
It's
just
a
way
of
detecting
that
the
drive
has
gone
out
to
lunch.
It's
not
coming
back!
So,
if
they're,
showing
up
in
cases
where
the
OSD
isn't
dead,
multiply
the
timeouts
by
town,
sure
I
mean
they're
they're,
not
there
for
any
purpose
other
than
detecting
doubt.
That's.
B
Green,
so
I
think
we
do
that,
but
this
is
where
the
other
this
this
is
yeah.
This
is
this
is
a
thing:
I
don't
get
into
rental,
but
yeah
I'd
be
helpful
to
think
about
this,
and
if
it's
it,
if
they
should
just
be
ginormous,
then
we
should
just
make
it
so
that
there
are
job
by
default,
I
guess
in
almost
infinite.
But
if
that
doesn't
make
sense,
then
maybe
there's
some
need
for.
A
And
we
have
blocking
in
all
kinds
of
places
in
the
stack
right,
I
mean
even
at
the
LA
bio
level,
right
I
always
submit,
can
block
you
can
you
can
be
blocking
on
IO
submit
for
seconds
which
you
know
in
a
scenario
we
have
a
fifteen
second
timeout.
That's
that's
not
good,
though
it
might
be
that
the
more
we
get
rid
of
stuff
like
that
going
over
to
your
ring
going
over
to
more
asynchronous
models,
maybe
maybe
that
problem
kind
of
slowly
fades
away.
I,
don't
know.
A
B
D
Don't
I
don't
think
that
part's
that
that
complicated,
if
you're
submitting
enough
to
leads
to
Rocksteady,
eventually,
operations
that
should
be
fast,
will
block
on
the
progress
of
compaction,
which
will
turn
a
brief
operation
to
a
long
operation.
If
this
happens
in
a
way
that
fairly
uniform
across
the
OSB
is
they
will
all
pretty
much
die
at
the
same
time,
I
mean
this.
That
part's
not
here
is
the.
B
B
B
D
So
it
seems
like
there's
sort
of
two
basic
things:
one
advised
the
customer
to
increase
the
time
most
by
a
factor
of
100
or
whatever,
when,
if
you
see
this
and
two,
if,
as
we
do
this
sort
of
throttle
thing,
when
we
start
getting
a
feel
for
how
long
booster
actually
takes
to
do
things,
it
might
make
sense
to
feed
that
back
in
at
the
timeouts
as
well.
That
way,
we
know
we're
more.
It's
not
just
whatever
was
configured
it's
a
hundred
or
a
thousand
times
the
average
operation
length.
Something
like
that.
B
Yeah
but
I
mean
what
you
want
to
know
and
isn't
it
what
you
want
to
know
whether
or
not
whether
whether
or
not
the
the
shift
and
the
shift
in
late
until
they,
whatever
that
is
the
new
behavior.
The
current
the
current
observed
behavior
is
the
respondent
is
responding
in
a
natural
way
to
the
workload
and
for
the
hardware
that
we
hate.
You
know
that
essentially,
rather
that,
rather
than
you
know,
as
opposed
to
you
know
that
it's.
A
B
One
example:
there
there
been
more
than
one
but
one
one,
one
one,
not
naturalistic
one
necessarily:
we've
had
we
produce
which
we
predict.
We
have
we're
currently
relying
on
no
map
to
do
generate.
While
you
noted
to
a
generate
to
communicate
about
a
bunch
of
a
bunch
of
a
bunch
of
replication,
related
events
and
an
expectation
is
that
we're
going
to
be
continually
retiring
events,
you
know
when
the
ones
are
being
produced.
One
way
to
put
one
pathological
cases
would
be
allow
far
too
many
to
build
up
an
innocent.
C
B
My
knowledge
that
that's
that
that's
a
way
to
produce
some
behavior
that
we
have
seen
in
the
past
and
it
goes
back
I
got
you
know
you
seen
as
recently
as
some
of
yours
this
year,
yeah
that's!
It
has
been
anyway.
If
we
ever,
there
are
two.
If
there,
if
there
are
two
it's
almost
always
well,
we
didn't
have
data
log
trimming
than
the
data
log
trim,
the
data
log
segments.
B
You
know
for
all
the
old
map
for
all
for
all
that,
for
all
the
data
log
shards
on
a
particular
list,
he
got
extraordinarily
low
and
they're
all
they're.
All
the
keys
are
all
closed
because
they're
all
for
the
same
objects,
and
then
we
tried
that
we
try
to
try
to
rapidly
believe
them
in
any.
You
know
in
any
of
lis,
then
you
know
then
yeah
we're
using
Rilke
materials
commands.
Typically
then
yeah
it
can
promote.
That
has
been.
That
has
been
a
simple
case
that
that
reproduces
yeah.
E
Just
a
real,
quick
question
on
that
I'm
I'm
actually
dealing
with
a
best
three
workload
right
now
that
taking
far
longer
doing
lots
of
deletes
than
it
should
in
you
know,
I
haven't
looked
at
compaction
times.
Definitely
what
what
would
you
guys
consider
to
be
reasonable
quantities
of
compactions
and
compaction
times
when
we're
looking
at
your
system?
Yeah
I'm,
looking
at
one
right
here,
that's
over
a
second
in
length.
A
Yeah,
though
the
really
long
ones
I've
seen
have
usually
been
in
like
level
three,
the
higher
you
go,
the
more
ramp
there
is
no
longer.
The
whole
thing
takes
more
data
and
right
amp,
this
I
just
linked.
This
is
a
tool
that
I
wrote
a
while
back.
That
will
look
through
the
events,
the
the
compaction
events
and
then
you
can
kind
of
like
look
at
it
on
a
per
level
basis
or
look
at
the
overall
stats.
A
A
All
right
well,
and
if
you
don't
mind,
I
I
would
wanted
me
to
talk
about
the
cash
work
I
did.
Is
there
anything
else
for
the
tea
rocks
or
starting?
No
all
right.
Well,
then,
quickly,
here,
I'll
mention
that
last
weekend,
I
I
think
I
talked
a
little
bit
about
a
long-standing
problem.
We've
had
in
Blue
Star,
where
we
double
cash
Oh
nodes,
both
in
the
Blue
Star,
Oh,
node,
cache
and
also
in
the
rocks
DB
block
cache
they're
represented
in
different
ways,
but
the
you
know
the
overall
data
is
basically
the
same.
A
So
what
that
means
is
that
when
we
get
into
a
situation
where
we
don't
have
data
cache,
we
don't
have
the
own
cache
that
will
read
it
from
the
SSD
file
Roxie.
We
will
load
that
block
into
the
block
cache
and
then
also
it
will
get
read
into
the
blue,
storoe
node
cash
and
when
you're
not
doing
that,
if
you're,
just
if
you've,
never
had
to
read
anything
off
this,
that's
great.
You
end
up
only
really
having
data
in
the
annoyed
cache.
A
So
what
I
did
last
week
actually
on
Friday
I
wrote
a
basically
code
that
will
take
and
and
use
the
column
families
by
commenting
by
default
for
the
be
kV
data
we
already
had
that
built
in
already,
but
then
we'll
create
a
separate
block
cache
or
that
particular
column
family,
that's
limited
in
size,
using
the
same
priority,
cache
interface,
that
our
other
blue
store,
caches
use
now
and
and
then
that
will
be
treated
independently
of
the
rest
of
the
rocks.
Tv
block
cache
has
like
OMAP
entries
and
other
things
in
it,
and
so.
A
It
has
the
biggest
effect
when
you
have
kind
of
around
our
standard
low
steam
memory
target
settings,
maybe
three
to
six
gigabytes
of
of
an
OSD
memory
target
that
translates
into
like
between
maybe
two
and
four
and
a
half
gigabytes
of
cache.
It's
split
between
all
of
our
caches
and
yeah
with
different
data
set
sizes.
Once
you
get
big
enough,
it
has
a
fairly
substantial
effect.
A
Maybe
a
25%
improvement
for
small
random
write
workloads.
So
so,
basically,
you
know
smaller
item.
Write
workloads
against
a
large
dataset
is
where
this
kind
of
really
helps,
especially
if
you
have
enough
cash
that
you
can,
you
can
see
a
big
difference
when,
when
not
doing
the
double
caching,
so
that's
really.
It
I've
got
a
branch,
but
not
a
PR.
Yet,
although
now
that
we've
merged
the
priority
cache
manager,
I
should
be
able
to
make
one
fairly
quickly.
So.