►
From YouTube: 2020-04- 23 :: Ceph Performance Meeting
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
All
right,
well,
I,
don't
know.
Maybe
this
is
already
today.
Maybe
everyone
else
is
actually
busy,
so
I
guess
I'll
get
started.
Let's
see,
I
keep
away
as
a
PR
this
week
for
CBT
they've
been
working
on
some
really
neat
testing
infrastructure
to
be
able
to
run
PRS
through
tests
and
then
look
at
whether
or
not
the
results
have
changed
and
then
mark
the
pass
or
fail.
And
so
in
this
case
we
were
seeing
a
lot
of
failures
because
of
just
random
variants
and
test
results.
A
So
this
is
just
a
good
PR
that
changes
that
variants
could
be
bigger
so
that
are
not
variously
bigger.
But
the
requirement
for
the
variants,
though,
that
there
are
fewer
false
positives
in
the
testing,
and
that
was
the
only
new
PR
I
noticed,
but
this
possible
or
something
else,
I
missed.
There
were
two
that
closed.
That
I
saw
one
is
you
used
full
strip
width
for
padding
and
pens
for
sales
journal,
I,
think
Jason
reviewed
that
and
then
also
there
was
another
one
that
merged
the
Keef
who
had
merged.
A
A
Let's
see
more
discussion
on
blue
sperm
in
Alex's
set
to
4k
there's
some
questions
on
what
to
do
about
testing
the
new
AVL
allocator
are
sorry
hybrid
allocator
rather,
and
so
the
thought
was
to
maybe
randomly
switch
allocators
and
then
run
them
through
tests.
Although
that's
going
to
be
deferred
until
later,
we
can.
We
can
figure
out
how
we
wanted
at
that
later
on,
but
get
merged.
For
now.
A
Let's
see
my
PR
for
pre
splitting
the
MDS
directories
into
its
multiple
dear
Frank's,
that
was
updated.
I
got
some
feedback
that
folks
weren't
happy
about
how
much
the
the
12
bytes
that
were
being
added
to
I,
know,
T
and
so
I
can
shrink
that
down
to
one
byte,
but
we
lose
a
little
bit
of
functionality.
A
Now
you
can
no
longer
actually
read
back
the
expected
files
etc
that
you
set
as
a
user
which
is
kind
of
unfortunate,
but
but
it
does
let
shrink
down
the
the
additions
they
I
know
downs,
just
a
single
byte
alternately.
We
could
also
just
store
the
expected
files
in
four
bytes
in
the
inode
and
then
calculate
the
bits
on
the
fly
that
might
be
some
more
CPU
overhead,
possibly,
but
that
could
be
done
as
well.
A
A
We
can
discuss
that
a
little
bit
lower,
but
he
did
some
really
good
investigation
work,
so
the
gist
of
it
is
that
we
can
improve
performance
without
actually
changing,
reducing
the
number
of
threads
or
gene
or
sorry
shards,
cheating
them
or
reserves
and
threads.
This
there's
a
simple
fix
that
that
can
fixes
the
bad
behavior.
A
C
A
A
And
nothing
huge
there
all
right
so
much
even
pings
PR
the
this
is
exciting.
Basically,
there
was
just
a
one-line
fix
that
that,
in
his
testing
dramatically
improved
the
read/write
performance
with
our
current
default
settings
from
I.
Remember
if
I
think
it
was
like
a
almost
like
a
30%
improvement
by
just
waking
up
doing
a
you
know,
notify
a
multiple
notify
instead
of
notify
one
and
the
in
queue
front
and
the
showed
up
for
cute
so
very
exciting
there.
That's
a
really
really
nice,
though
we
generally.
A
B
B
A
A
A
B
I
think
he's
enabled
it
and
it's
running
and
I
read
every
PR,
that's
tight
their
crimson
crimson,
pr's
or
performance
for
the
past
four
years,
and
that
was
kind
of
a
notification
back
from
that
job.
To
give
us
the
results-
and
they
could
have
comment
so
now,
Kieffer's
just
been
kind
of
manually,
pacing
them
out
of
the
deck
out
of
Dickens.
D
Yeah
I
guess
before
we
wrap
up
the
Jenkins
topic.
I
think
the
next
piece,
in
my
mind,
is
also
to
look
at
the
variability
of
results,
so
I
haven't
found
the
time
to
go
and
look
at
what
that
looks
like
for.
If
we
are
trying
to
trigger
master,
runs
across
PRS,
how
much
variability
are
we
seeing
that
that'll
at
least
tell
us
how
the
hardware
is
behaving
and
then
maybe
have
more
reasonable,
pass/fail
criteria?
D
I
know
we
are
trying
to
change
the
standard
deviation
and
do
some
minor
tweaks
just
to
make
those
tests
fast,
but
then
I
would
like
to
see
how
much
variability
exists,
at
least
for
master
pr's
like
master
baselines.
That's
that
part
is
something
that
I
want
to
take
a
look.
Maybe
whenever
I
have
time
but
yeah.
The
next
piece
is
about
hardware,
which
I
started
a
discussion
with
David
Galloway.
This
is
about
doing
long-running
performance
tests
using
the
pathology,
CBT
frame
bell
that
we
already
have.
D
The
idea
was
to
be
able
to
label
some
nodes
just
for
this
purpose
and
trigger
tests
using
the
existing
framework.
Now
the
first
thing
that
came
to
mind
was
to
repurpose
some
of
the
Smitty
machines
for
this.
If
at
all,
we
have
some
extra
and
I
think
David
Calloway
was
also
trying
to
figure
out
if
the
the
existing
insert
us
that
are
being
used
for
Jenkins
based
testing
can
be
repurposed
for
this
purpose.
D
A
So
we've
got
a
total
of
eight
notes:
okay
aidan
certain
notes.
One
of
them
has
an
nvme
drive,
that's
I
think
starting
to
fail,
but
that
doesn't
necessarily
mean
we
can't
make
use
of
it.
Oh,
my
my
thought
was
that
we
would
use
four
of
them
for
actually
six
of
them
in
the
long
run
for
Jenkins
and
then
two
of
them
available
for
basically
doing
like
one-off
tests
on
hardware.
That
also
has
hard
drives
in
it,
because
those
are
these.
There
are
nodes
of
the
only
machines
we
have.
A
We
could
rearrange
some
of
that,
depending
on
how
much
Jenkins
is
actually
being
used
on
the
inserter
nodes.
We
could
maybe
grab
like
one
of
them
from
Jenkins
grab
the
other
two
from
the
pool
that
are
still
being
used
by
people
for
I
get
those
people
moved
over
to
officinalis,
and
then
we
would
maybe
could
scrounge
up
like
three
of
them
for
Jenkins
and
three
of
them
for
technology,
but
it's
gonna
be
really
tight
and
have
the
the
downside
of
this
too.
Is
that
right
now,
if
the
ten
officinalis
nodes?
A
That's
a
really
nice
chunk
for
doing
kind
of
these.
These
bigger
investigations
like
for
the
IO
500,
where
the
guys
had
no
challenge
that
kind
of
thing,
so
just
I've
been
kind
of
thinking
about
how
how
to
divvy
this
stuff
up
in
a
way
that
we
would
allow
us
to
still
make
use
of
them
for
kind
of
those
bigger
jobs,
but
then
not
have
them
kind
of
an
depended
on
buy
a
lot
of
stuff.
A
At
the
same
time,
if
we
got
that
piece
into
thought,
Google
Summer
of
Code
project
I,
think
that
would
make
things
a
lot
better.
I
know
we're.
You
know
we
were
talking
about
yesterday,
not
kind
of
conflating
the
two,
but
if
we
had
that
so
that
we
can
make
sure
that
bigger
jobs
into
Thal
gr
not
stalling
out
I,
think
that
would
make
it
a
lot
more
palpable
to
just
stick
it
all
in
technology
and
then
be
able
to
kind
of
queue
them
up
for
different
purposes.
D
A
D
That's
is
just
like
the
the
tautology
project
that
we
have
it's
more
for
our
regular
development
testing,
where
people
are
like
multiple
people
are
trying
to
add
jobs
to
the
queue
and
they
are
not
finishing
or
they're,
not
you
know
finishing
in
time,
but
when
we
are,
when
we
have
these
separate
labels,
it's
not
even
going
to
be
a
person.
We
could
just
use
cron
jobs
and
stuff
to
run
these
long-running
tests
and
which,
if
their
labels
are
right,
there's
not
going
to
be
any
convention.
D
B
A
Sure
sure
I
you
could
do
that
I'm.
Just
saying
that
other
piece
where,
then
you
don't
have
things
waiting
things
free
up
right
like
you
could
you
could
basically
like
try
to
grab
those
resources
and,
like
you
know,
forcibly
lock
them
for
yourselves,
but
you
know
if
you
had
a
job
that
you
want
to
submit
that
used
all
ten
nodes
right
now,
if
you
just
wanted
to
do
that,
but
let
things
finish
you
couldn't
really
write,
because
something
else
with
a
smaller
job
will
come
in
and
steal
the
the
nodes
before
your
job
runs.
D
B
You
use
the
menu
looking
to
kind
of
avoid
that
problem.
If
you're
talking
about
trying
to
use
some
of
these
nodes
or
even
a
larger
portions
I'm,
just
locking
in
one
at
the
time
as
they
become
available
and
you're.
D
B
D
Other
way
to
achieve
what
you're,
trying
to
say,
Marc
is
to
follow
a
schedule.
So
if
there
is
a
schedule
or
a
cron
job
schedule,
which
we
know
when
want
to
run
these
long
running
tests,
we
also
know
when
they
get
over
roughly
so
whenever
those
resources
are
free.
That
is
the
time
when
you
should
be
able
to
lock
anything
from
that
pool.
A
B
Yeah
I
think
it's
also
worth
considering
what
kind
of
hardware
we
need
for
different
kinds
of
jobs
like
for
longer
running
tests
that
are
small-scale.
It
makes
sense
to
use
in
Sirte,
but
if
we
want
to
do
something
larger
scale,
if
they
aren't
enough
do
that,
they
would
need
to
go
with
a
probably
larger
pool
Smithy's.
For
that
purpose,
sure.
B
A
Yeah
I
mean
right
now
right
the
easiest
route
to
be
able
to
do
this
is
just
to
grab
spent
these.
If
we
have
a
small
pool
of
smoothies
that
we
can
grab
for
the
moment,
that's
that's
probably
the
fastest
answer,
simply
because
we
still
have
people
using
in
Sirte
for
development.
So
if
we
can,
if
we
can
just
grab
some
smoothies
for
the
you
know
next
couple
weeks
or
month
or
something
that
would
that
be
probably
the
way
to
just
immediately
get
this
going.
A
B
A
B
D
A
To
do
that,
you
have
to
kind
of
try
to
to
do
a
lot
of
things
that
are
irritating
but
helpful
right,
like
reboot
the
nose
before
you
run.
The
tests
redeploy
the
cluster
before
you
run
the
test.
If
you're
going
to
age
it
you
age
the
test,
the
exact
same
way,
every
single,
though
you
know
it's,
it's
kind
of
about
making.
Everything
is
exactly
similar
as
possible
between
two
separate
tests,
except
for
the
you
know,
variable
that
you're
changing
yeah.
D
A
D
Yeah
I,
I,
agree
and
I.
Think
I
did
this
exercise
like
a
couple
of
years
ago,
and
at
that
time
it
was
randoms
with
ease
there
was
getting
picked
and
also
there
was
other
variability
with
the
operating
system
and
other
things.
So
I
didn't
really
find
a
good
baseline.
So,
instead
of
just
redoing
the
whole
thing
again
and
being
disappointed,
I
just
want
to
fix
my
matrix
first
and
I
want
to
know
which
machine
we
are
going
to
be
using
for
this
and
then
try
to
do
the
baselining
again.
That
makes
more
sense
to
me.
D
A
D
D
A
It'd
be
interesting
to
see
I
see
really
weird
behavior
in
this
regard.
It
sometimes
our
our
performance
is
incredibly
consistent
depending
on
the
test.
I'm
running
I
can
get
very,
very
good,
consistent
results,
and
then
there
are
other
situations
where
just
wildly
varying
performance
a
lot
of
times
it.
It
really
depends
on
what
you're
testing
right.
A
D
A
D
D
The
other
thing
I
was
wondering,
is
I'm,
going
back
to
the
Jenkins,
stop
again
I
think.
Currently
we
have
a
couple
of
workloads
that
are
being
run
if
I'm
not
mistaken,
but
for
a
classical
stay
I
think
we'd
like
to
expand
it
a
bit
mark.
Do
you
have
any
thoughts
on
what
kind
of
what
those
workflows
should
look
like
for,
like,
let
us
say
core
peers
sure
I
mean.
A
In
terms
of
testing
the
OSD
in
a
really
fast
simplistic
fashion,
our
Reedy
with
FIO,
really
kind
of
is
bang
for
your
buck,
pretty
good.
It
doesn't
test
everything
by
any
means,
but
usually
from
what
I've
seen
the
results
are
fairly
consistent
because
our
BT
is,
you
know
pretty
straightforward
and
simple.
A
Certainly
we
want
like
oh
map,
test
sand
and
rocks
more
heavy
rocks
TV
stuff,
but,
but
you
know
starting
out
I'd
say
like
Justin
FIO
workload,
that
does
some
big
reads
and
writes
some
small
random
reads
and
writes
probably
a
mixed
workload
which
I
admit
I
end
up
testing
less
than
I
should
especially
now
looking
at
my
things.
It
is
results
from
his
tests,
though.
So
probably
you
know,
that
would
be
a
good
kind
of
core
test.
A
Just
do
do
big
reads:
big,
writes,
Winchell
and
then,
like
small
random,
read
small
random,
writes
and
a
small
random
mixed
workload.
That
would
that
would
be
a
really
good
just
basis
and
then
add
on
you
know,
map
heavy
tests,
either
energy,
w
or
Arenas
bench
should
be
one
tough
FS
workloads
are
actually
really
good
for
testing
mb/s.
But
that's
you
know
it's.
D
Mm-Hmm
yeah
I
guess
at
the
moment
Jenkins
just
uses
Rados
bench
as
it's
a
testing
framework,
so
we
could
just
start
off
with
like
adding
some
OMAP
heavy
workload
profiles
and
like
mixed
eat.
Some
rights
and
I
think
we
have
just
read
and
write
already
so
I
think
that
should
be
good
enough
and
we
also
don't
want
to
increase
the
amount
of
time
these
tests
take
for
on
poor
PR
pieces.
But
as
far
as
the
FI
of
stuff
course
I
think
that's
where
that
integration
is
already
there
in
the
truth,
ology
framework.
A
I
think
we
want
that
in
Jenkins
as
well,
because
we
can
run
fairly
short
FIO
tests
and
still
get
really
good
data
out
of
it.
I'm
a
/,
PR
basis.
I
think
we
want
that,
but
we
could
also
do
much
more
extensive
FIO
testing
with
tooth
ology,
like
looking
at
like
aging
there's
gonna,
be
the
big
thing.
I
think
it
will
walk
tooth
ology
for
in
that
case,
because
we
won't
want
to
create
anything
for
like
Jenkins
tests.
We'll
just
want
to
play
cluster
immediately,
get
some
results
out
of
it.
So
they
finish
fast
yeah.
A
B
B
A
B
I
think
like
to
me,
like
I,
said
it
seems
like
like
our
FIO
RVD,
and
it
has
eventually
be
kind
of
more
minimal,
realistic
workloads.
Right
then,
I.
A
Think
yeah
I
very
much
think
so,
and
then
we
can
decide
on
the
southwest
side
to
what
we
want
to
do.
The
so
the
FIO
benchmark
can
certainly
be
used
with
with
Sufis,
and
it
will
you
know
it
makeup.
You
can
create
a
parallel
workload
against
it,
but
it
won't
stress
the
MDS
very
heavily,
which
is
really
what
we
want
to
test.
I.
Think
I
mean
it's
good
to
have
it's
good,
to
be
able
to
look
at
FIO
and
stuff
of
us
and
do
comparisons
between
the
two.
A
If
there's
like
some
weird
regression
and
larger
sequential
writes
or
something
or
it
reads,
we
actually
did
see
that,
but
but
for
testing
the
MVS,
we
need
something
like
MD
test
or
or
maybe
small
file
or
or
something
yeah.
That's
certainly
something
more
specialized.
A
So
Neha
I
think
if
IO
is
a
good
way
to
start
on
the
technology
side,
if
we
can
get
that
make
just
miniaturist
using
the
normal
FIO
benchmark
and
then
yeah
I
mean
like
just
grab
like
one
tooth
ology
note
or
once
I
won
I
said
yeah
I'm.
Looking
for
one
of
our
our
fast-fish
nodes
busy
there
we
go
one
of
the
smithy
notes
and
just
yeah
I
mean
see,
see
how
it
does
over
time,
but
more
full
runs.
D
A
A
A
Frank,
if
you're,
here
or
listening,
you
have
any
preferences
regarding
different
tools
for
half,
stressing
the
MDS
I've
kind
of
been
using
MV
tests.
Just
because
that's
what
we
always
used
to
use
my
HPC
days,
I
know
that
there's
small
file
from
from
Ben
England
is
there
anything
else
you
guys
have
been
using.