►
From YouTube: 2019-06-27 :: Ceph Performance Meeting
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
A
Choruses
yeah
before
I
forget
David
was
just
asking
about
changing
the
debug
level
of
the
of
trim
and
boost
or
to
like
something
higher
than
20
so
go.
A
C
C
C
A
A
big
wait
so
I
actually
started
looking
at
this
morning's
I've
got
really.
Recent
wall
talk
profiles
and
a
lot
of
it
is
like
destroying
strings
an
event.
So
I'm
right
now,
just
like
switching
over
to
an
enum
and
like
that
referenced
one
from
like
when
you
register
in
the
OP
tracker
and
then
just
like,
losing
it.
A
C
C
A
C
I'll
see
how
much
that
helps:
okay,
okay,
so
there's
that
there's
not
a
pull
request
from
Igor.
It's
that
makes
the
you
know,
deletion
stuff
in
blue
store
faster,
and
this
one
surprised
me
a
little.
She
must
have
being
this
where
the
OST
is
spending
a
lot
of
time
deleting
PGs
up,
should
they
get
rebalanced
or
something,
but
it
basically
a
little
bit
more
clever
about
researching
when
it's
scanning
prefetching
and
pre
loading
stuff
into
the
cache.
They
don't
go
back
and
read
it
again
later
and
so
on.
C
So
it
makes
sense,
but
I've
reviewed
it.
A
closer
look
couple
things
merged,
adding
a
steno,
SD
or
SDHC
TT
variants
for
the
recovery
max
active
based
on
our
discussion
last
week
that
merged
the
open
question.
There
is
whether
I'm
turning
up
just
the
recovery
max
active.
C
A
C
That's
not
what
this
is
OST
recovery
max
active
is
a
member
of
in-flight
recovery
rights
essentially
messages
at
once.
So
if
it's
one,
then
there's
a
one
message
with
like
4
Meg's
or
whatever
going
over
the
network,
you
wait
until
you
get
a
reply
and
then
you
send
the
next
one.
If
it's
100,
you
got
literally
a
hundred
messages
and
four
mags.
So
four
hundred
Meg's
in
flight,
like
the
queue
depth
pipeline
or
whatever
of
the
recovery.
C
Max
backfills
is
basically
how
many
PGs
you're
spreading
it
across
yeah,
so
max
backfills
would
be
like
a
worth
pack
going
to
PGH
have
50
in
flight
operations,
as
opposed
to
one
fiji
with
102
the
operations.
But
the
only
real
difference
is
that,
if
you're
bottlenecked
on
the
source
like
if
you're
back
filling
to
the
scene
or
yeah
I
guess
it
does
actually
matter.
C
If
you
think,
if
you're
bottlenecked
on
the
read
side,
then
having
multiple
backfills
means
that
two
O's
geez
might
be
reading
and
writing
to
the
same
location
but
might
go
faster
versus
having
one
OS,
do
you
reading
and
writing
to
the
same
location
but
I'm
a
little
bit
I
think
there
are
cases
where
increasing
max
backfills
will
speed
it
up
that
most
of
the
time,
just
increasing
max
active
should
enough.
It.
A
Didn't
seem
to
work
that
way
when
I
was
testing
it.
Okay,
it
looked
the
the
behavior
seemed
to
be
that
we
could
get
with
with
max
active
bumped
way
up.
You
could
have
roughly
like
OSD,
divided
by
two
active
PJs
being
being
worked
on
in
the
cluster
at
the
same
time
through
that
was
with.
That
was
with
the
OC
max
sorry,
OST
right.
A
C
Is
that
I
forget
I
mean
that
the
goal
is
to
have?
Is
the
overall
throughput
of
recovery?
No
max
backfill
controls
how
many
PG
czar
recovering,
but
that's
orthogonal
to
the
throughput
or
is
most
survivor
right
so
that
we
don't
care
about
how
many
peaches
are
are
actually
recovering
like
we
actually
want
that
to
be
a
smaller
number
so
that
they
can
sleep
sooner,
but
we
want.
Is
the
recovery
throughput
to
be
as
high
as
possible?
So
that's
don't
pay
attention
to
the
PG
can't
pay
attention
to
throughput.
A
C
B
C
A
C
C
If
we
should
repeat
that
experiment
basically
and
crank
up
max
active,
really
high
and
see
how
high
the
recovery
throughput
goes.
That's
the
that's
the
thing
we
want
to
maximize.
We
don't
care
at
all
about
the
number
of
teaches
you
want
to
maximize
the
throughput
and
see
if
it's
really
necessary
to
increase
max
backfills
and
already
get
higher
throughput.
A
C
A
A
C
Higher
than
that
that
it
that,
if
it,
if
it's
a
significant
improvement
like
two
or
four
then
maybe.
A
C
C
That's
why
sure,
and
that's
going
to
mean
that
those
PG,
so
probably
what
we
really
should
do
is
do
that
experiment,
but
actually
measure
the
start
and
end
times
of
recovery
for
each
PG
and
then
and
then
look
at
how
much
slower
the
recovery
time
for
individual
PG
s
are,
or
actually
even
just
look
at
that
overall
recovery
time
for
the
whole
cluster,
because
maybe
TTS
take
a
little
bit
longer,
but
because
they
can
start
sooner.
The
overall
recovery
is
sooner.
A
So
sage,
the
my
talent
proposals,
rather
than
doing
all
of
this
work
and
like
trying
to
figure
out
optimal
values
for
these-
that
don't
actually
may
not
make
sense,
based
on
how
variable
different
in
DB
drugs
are.
Let's
just
make
the
like
buttom
attic
like
trying
to
auto-tune
this
based
on
client
IO
wall
clock
time.
Let's,
let's
do
it
that
way,
and
then
we
don't
have
to
try
to
like
figure
out
stuff.
We
just
have
it
doing
itself.
Yeah.
C
C
In
our
buddy
will
request
dental
performance
that
was
merged,
that's
good,
there's,
a
h,
ma
jumping
that
adds
a
new
up
create.
This
is
basically
a
blue
store,
related
optimization
that
allows
us
to
avoid
somewhat
cuts
in
the
cash
when
we
know
we're,
creating
an
object
that
doesn't
exist
and
that's
merged.
There
was
one
dog
that
fell
out
of
that,
but
I
think
that's
getting
fixed
now
so.
C
C
Roman
had
a
messenger
change,
it
was
sort
of
a
micro
benchmarks.
It's
unclear
whether
that's
make
a
have
an
effect
on
real
workloads.
So
that's
some
discussion
there
and
I'm
clear
whether
it's
worth
proceeding
with
and
then
your
cash
spinning,
which
I
guess
is
going
to
follow
after
your
that's
going
to
be
based
on
top
of
your
cash
reflect
remark.
Yeah.
A
C
D
We
talked
about
trying
to
set
up
a
sort
of
like
a
continuous
benchmarking
performance,
benchmarking,
force,
F
or
deep
development,
so
testing
against
things
that
are
emerging
in
release
branches
and
set
up
like
a
what
defines
a
an
actual
regression
and
I
think
there
are
some
machines
available
for
days.
You
would
involve
some
CBT
work
for
sure
integration
with
I
mean
ideally
integration
with
github,
so
that
we
would
be
able
to
publish
immediately
results
like
what
what
what
the
bench,
what
the
the
test
is
actually
spitting
out.
I
think
it's
very
interesting
work.
D
C
A
Know
mark,
do
you
know
that
so
sage
I
think
the
idea
would
be
that
on
the
for
the
stuff
like
Keith
who
in
and
Radek
want,
it
would
be
entirely
through
jenkins,
doing
the
basically
just
for
builds
and
then
the
stuff
and
thorgy
would
be
potentially
even
the
same.
It
could
be
even
the
same
test
that
I
run
through
T
beat
CBT
or
a
different
set
of
testing.
It
doesn't
matter,
but
they
would
be.
You
know
the
the
stuff.
That's
in
teeth,
ology
would
be
kind
of.
A
C
A
C
A
C
Yeah
early,
something
that's
simple
enough:
it'll
run
on
a
single
box,
a
multiple
of
STIs,
but
one
bucks,
no,
no
network
variation.
So
if
this
is
through
so
we're
talking
about
doing
mr.
Jenkins
and
it's
like
hooked
into
github,
all
of
the
current
plugins
have
like
a
one-liner,
basically
that
you
can
show
up
after
the
test
name.
You
know
something
like
currently.
It's
things
like
all
commits
are
signed.
C
D
C
D
D
D
Just
comment:
yep,
absolutely,
maybe
even
better
now,
I
think
that
but
yeah
sorry,
I
I
think
that
we
just
need
to
concentrate
on
getting
the
benchmarks
working
in
there
and
and
for
sure
I'll
bring
it
up
so
that
we
can
discuss
like
you
know.
What's
what
what
do
you
guys
think
you
think
is
best
thing.
A
We
can
add
attachments
to
like
I,
don't
know
if
you
can
do
that.
But
if
you
can
like
insert
attachments
into
the
comments
with
like
80
a
bunch
of
like
just
get
Wolcott
profiles,
we
could
have
automatically
generated
well-thought
profiles
for
every
single,
like
performance
tag
PR.
That
would
be
amazing,
yeah.
D
C
D
C
Yeah
there
are
like
they're
like
a
handful
of
I,
don't
know
two
three
four
tests
that
run
or
even
just
one
I
could
imagine
that
the
the
comment
would
be
like
a
little
table
that
has
the
the
test
name
and
then
with
a
delta
to
the
latest
master
and
the
latest
stable
branch
that
it's
your
based
on.
So
if
you
like
latest
master
latest
Nautilus,
you
know
plus
10%
minus
2%,
whatever
it
is,
tells.
C
A
C
A
I
was
thinking
if
we
can
do
like
different
tests
based
on
the
tags.
The
mistake
is
performance
in
our
BD
we
run
some
our
BD
tests.
Let's
take
performance
and
rgw,
we
can
do
different
tests
like
a
thing.
Yeah
that'd
be
pretty
cool,
I,
don't
know
that's
hard
to
do
or
not,
but
it
seems
like
that
would
be
something
that
hopefully
like
Jenkins,
github
integration
and
stuff
can
do.
C
If,
if
we
had
do
you
had
both
a
baseline
with
the
current
master
like
if
you
just
keep
the
keeps
track
with
the
latest
master
run
was
and
also
the
latest
for
each
stable
release
branch,
and
you
showed
both
then
also
it'll.
Just
let
you
sort
of
keep
a
running
tab
on
like
how
you're
doing
overall,
because,
as
the
development
cycle
progresses,
you'll
see
you're
like
plus
or
minus
1%
on
master,
and
you
know
plus
20%,
on
I'm,
not
Elissa.
Other
thing:
that's
nice
to
just
have
to
see
your
progress
over
the
over
the
courses.
A
C
A
D
A
Happier
Alfredo
Orlando
expressed
some
interest
in
possibly
also
hoping
of
working
on
the
like
data,
parsing
side
and
graphing
and
making
you
know
nice
nice.
You
know
pretty
results
from
it.
He's
done
some
of
that
before
I
think
doesn't
look
like
he's
here
today,
but
I
might
try
to
get
you
guys
hooked
together
too,
so
that
you
can
I.
A
Doesn't
work
for
he
does
know,
he's
from
Intel
right
can't
the
the
goal
on
that
we
had
for
CBT
output
data
was
to
have
like
a
hash
encoded
directory
structure
for
the
output
data,
so
that
you'd
be
able
to
have
that
kind
of
be
the
unique
key
that
defines
then
this
based
on
all
of
the
parameters
that
were
used
to
run
the
test,
and
then
you
could
build
some
kind
of
index
for
it
have
it.
You
know,
do
some
kind
of
query
against
your
your
set
of
output
directories.
A
B
Right
I've
got
it.
Yeah
I've
got
a
quick
note.
We
we
did
discuss
this
meeting
with
the
MOC
students
and
invited
them
to
periodically
drop
in
and
maybe
even
present,
some
of
their
stuff
or
ask
questions.
I
mean
I
know
you
know
we
are
trying
to
be
a
little
more
engaged
with
them.
I
know,
mark
has
been
working
with,
or
at
least
talking
or
a
little
bit,
but
it
was
just
sort
of
there's
been
a
lot
of
discussion
with
bu
and
northeastern
faculty.
B
Now,
what's
the
right
engage
model
with
the
students,
they
I
think
they're
a
little
bit
protective
of
them
on
one
hand
and,
on
the
other
hand,
Matt
and
I
were
stressing
if
they're
gonna
actually
be
doing
things,
and
you
know,
even
if
we
take
the
code
and
ultimately
get
it,
get
it
merged.
You
know
the
right
way
to
engage
is
with
our
upstream
meetings,
so
that
might
be
might
be
happening.
A
A
One
thing
I
did
why
I
mention
is
that
Intel
is
like
ready
to
ship
us
the
notes
for
the
new
officinalis
cluster
David.
Even
the
shipping
address
and
everything
so
they've
got
everything
they
need,
so
we
might
just
be
showing
up
at
any
random
time
here.
I,
don't
know
what
the
status
of
the
network
is.
Yeah
I
think
we
got
hardware
in,
hopefully
I
think
we
have
it
on.
Ok,.
B
And
that
that
node
configuration
with
it
with
just
the
swapping
of
TLC
for
qlc
is
probably
what
Intel
is
gonna
recommend
slash
provide
for
MOC
for
both
rgw
caching,
for,
like
the
read-only
s3
level,
caching
that
were
trying
to
get
merged
as
well
as
just
straight
RBD
storage,
Peter
destroyers
did
some
math
and
he
was
convinced
that
the
qlc
is
gonna,
be
as
good
as
far
as
data
retention
and
reliability
as
the
high-capacity
drives
he's
got.
You
know
he
almost
gave
this
man
that
wasn't
quite
sure
of
his
math,
but
but
anyway,
they're
gonna.
B
A
A
So
if
we
those
come
in
soon,
we're
gonna
have
a
ton
of
work
to
do
to
get
those
set
up,
and
then
we've
got
some
tests.
That
Intel
had
had
been
shown
a
lot
of
interest
in
running
so
testing
on
those
and
then
started
migrating
the
inserter
nodes
into
probably
due
to
thought.
We
can
lock
them
and
then
assign
them
for
Jenkins,
then
Alfredo.
Those
will
at
least
probably
at
least
someone
we
can
for
mystics
of
those
will
go
to
you
burn
for
yourself.
A
Sense
that
would
be
yep.
That
would
be.
The
goal
is
to
migrant
in
certain
ODEs,
probably
like
three
through
eight
to
those,
because
those
three
three
eight
probably
have
had
less
wear
on
the
the
nvme
dress,
one
is
going
to
be
ones
probably
get
die
soon,
because
that's
what
I
use
a
lot
but
but
we'll
keep,
maybe
one
or
two
of
them
just
around
four
or
like
one
off
testing
I.
Think
yeah,
maybe
like
six
of
them
for
Jenkins.
D
A
It
to
also
express
a
lot
of
interest
in
having
some
of
the
efficient
all's
nodes
potentially
participate
in
like
PR
testing,
so
you
may
get
a
couple
of
those
as
well.
We
have
ten
of
them,
which
will
will
have
to
figure
out
the
right
way
to
divide
them
up,
but
but
we
might
at
least
be
able
to
get
you
a
couple
of
those.