►
From YouTube: Ceph Performance Meeting 2022-12-08
Description
Join us weekly for the Ceph Performance meeting: https://ceph.io/en/community/meetups
Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contrib...
What is Ceph: https://ceph.io/en/discover/
A
So
I
I,
finally,
for
the
first
time
in
a
while
made
it
through
all
of
the
pull
requests,
so
we've
actually
got
a
fairly
up-to-date
list
now,
and
luckily,
you
know
last
week
was
the
majority
of
the
work
since
it
had
been
a
while,
since
I
had
done
it
before,
but
but
there
were
still
a
couple.
I
had
admits
that
I
needed
to
get
in
so
hopefully,
hopefully
this
is
accurate.
Now
so
in
the
last
week,
I
had
nothing
particularly
new
that
I
saw
if
I
screwed
up.
A
Let
me
know
I
didn't
see
anything.
There
were
a
number
of
pull
requests
that
got
closed.
A
Those
were
all
by
the
bot
from
what
I
saw
and
there
was
an
updated
PR
here.
This
one
on
the
rgw
side
for
d4n
looks
like
that's
gotten.
Some
updates
in
case
he's
been
on
top
of
that
otherwise
lots
of
stuff.
From
last
week
that
hasn't
seen
Movement,
we
had
a
large
number
of
PR's
come
in
related
to
to
roxdb.
A
Well,
he's
not
larger
than
birth
just
a
few,
but
we
should
probably
decide
if
we
want
to
try
updating
for
the
most
recent
version
which
is
required
for
for
this
ball
compression
option.
A
B
A
Is
not
here
today
and
neither
is
Igor
I
did
want
to
talk
about
that,
one
to
see
what
kind
of
effects
that
might
have,
but
since
they're
not
here,
let's
move
on,
there
was
also
a
PR
from
Igor
about
faster
Blue,
faster
allocation
in
the
AVL
and
hybrid
allocators
I
glanced
at
that,
but
I
need
to
look
at
that
more
in
depth.
I
think
Igor
was
hoping
for
me
to
review
that.
A
So
that's
on
my
list,
yeah
otherwise
I
think
we
covered
a
lot
of
the
new
stuff
last
week,
so
any
anything
I
missed
that
anyone
has
either
knew
or
updated
or
anything
I
would
like
to
talk
about
regarding
pull
requests.
A
All
right!
Well
then,
let's
move
on
Luke
I,
see
that
you've
put
a
discussion
topic
in.
Do
you
want
to
take.
C
Over
yeah
I,
just
we
have
a
saf
cluster
where
we're
trying
to
optimize
for
performance.
Of
course.
So
it's
a
great
place
to
be
on
this
call.
So
we
in
in
our
last
cluster
we
had
we
used
raid,
five
and
and
now
we're
looking
to
see
what
rate
options
people
have
made
use
of
so
we're
in
the
kind
of
setup
where
we
have
say
200
boxes
and
each
have
24,
spinning
drives
and
one
SSD
metadata,
Drive
and
just
different
configurations.
C
You
know
we
started
playing
around
even
with
like
three
three
spinning
discs
grade
zero
clusters,
just
what
what
if
people
tried
as
far
as
different,
even
RAID,
0
raid
five
clusters
or
just
all
spinning
discs,
separate
and
and
what
are
the
implications
with
that
with
when
you
just
have
one
metadata,
Drive.
A
Sure
sure
so
it's
been
a
while,
since
I've
done
anything
kind
of
real
exotic
back
in
the
early
days
of
ink
tank.
We
we
looked
at
a
lot
of
different
things
like
this,
where
we
were
trying
different
raid
configurations
and
and
different
replication
levels
when
at
least
back
in
those
days
with
file
store.
We
definitely
saw
that
the
controller
cache
made
a
a
fairly
positive
difference
in
terms
of
performance,
especially
I
think,
with
the
way
that
file
store
did
journaling.
A
It
seemed
like
that
that
helped
quite
a
bit
in
terms
of
getting
fast
or
lower
latencies
with
blue
store.
It's
a
little
trickier
right
because
you've
got
your
your
essentially
during
the
right
hand,
log
typically
on
the
ndme
driver
and
the
SSD
drive,
and
we
actually
saw
early
on
that.
A
Sometimes
controller
cache
could
get
in
the
way
that
happened
on
on
some
earlier
HP
boxes
and
actually
I
think
we
might
have
seen
it
with
earlier
LSI
controllers
as
well,
and
I
haven't
looked
at
it
since
that
was
like
five
years
ago.
So
in
terms
of,
what's
out
there
now
I'm
afraid
I
I'm,
not
not
really
up
to
date
on
on,
you
know
what
what
the
state
of
the
art
is
in
terms
of
this
kind
of
stuff.
A
C
I
mean
our
workload:
I
mean
we
typically
don't
have
a
problem
with
our
ingest
and
write
speeds.
I
mean
that's
not
what
we're
necessarily
optimizing
for,
though
most
of
our
most
of
our
heavy
workloads
are
through
like
an
S3
and
most
of
our
data
is
like
say,
30
megabyte
per
k
files,
and
so
we
have
different
different
operations
that
are
reading
tens
of
thousands,
hundreds
of
thousands
of
these
rather
large
parquet
files
from
S3
that
you
know
we
have
other
workflows
but
I'd,
say
that's
our
most
demanding
one.
A
A
The
the
impact
of
having
to
do
reads
from
different
EC
chunks
on
on
secondaries,
rather
than
just
the
primaries
kind
of
rough
I
mean
it
if
you're
doing
like,
if
you're
looking
at
3x
replication
versus
EC,
it
can
be
actually
faster
to
do
EC
if
you're
doing
big
rights,
but
for
reads
it's
at
least
I've,
never
seen
it
actually
be
faster
to
do
EC
with
raids
versus
just
always
reading
from
the
primary.
So
you
know
that's
that's
something
to
consider
I!
A
Guess
whether
or
not
you
want
to
do
like
raid
under
an
OSD.
A
C
So
I
mean
we'll
we'll
test
a
bunch
of
these
configurations,
but
we're
thinking
both
raid
zero
testing
and
raid
five
testing
I.
Don't
you
know
raid
six
I,
don't
think
I
mean
just
the
extra
layer,
extra
parity,
you
know
disk
I,
don't
think
we're
gonna
need
depending
you
know,
so.
Yeah
grade
zero
and
rate
Five
are
mostly
what
we're
thinking.
A
Sure-
and
you
know
what
what
you
implement
too
I-
don't
know
what
what
vendor,
if
you're,
going
through
any
vendors
for
support,
but
they
may
have
their
own
opinions
on
what
they're
willing
to
support
as
well.
It's.
A
Oh
okay,
okay!
So
that's
like
Community
stuff,
so
yeah
I
mean
from
that
standpoint.
You
know
whatever
you
guys
are
comfortable
with.
You
can
try.
A
I'm
gonna
guess
that
at
some
point,
you're
going
to
be
limited
by
the
OSD
I
mean
I,
don't
know
how
fast
these
drives
are.
But
you
know
if
you're
creating
a
big
raid
under
the
OSD,
you
may
start
seeing
that
the
OSD
itself
could
be
a
bottleneck,
but
maybe
not
I
mean
I.
I
haven't
tried
it
in
a
long
time.
So
I
guess
I
can't
tell
you
you.
Can
you
can
get
pretty
fast
throughput
like
read
and
write
throughput
to
an
OSD
backed
by
an
nvme
drive
like
you
can
get.
A
You
know
two
three
gigabytes
per
second
per
OSD
pretty
pretty
reasonably.
But
having
said
that,
you
know
that's
a
different
device
than
a
raid
full
of
hard
drives.
That
may
or
may
not
have
other
bottlenecks
that
that
hurt
you.
A
A
C
And
so
what?
What
do
you
expect?
The
difference
between
EC
versus
just
replicate
I
mean
we've
only
for
for
definitely
for
our
S3
store.
We've
we've
only
our
last
cluster
in
this
class.
We're
only
thinking
of
EC,
but
like
is
there
a
big
performance
hit.
A
Well,
you
have
to
fetch
chunks
from
the
replicas
right
and
you're
breaking
down
the
up
the
instead
of
having
like
say
one
big
object:
I
I,
don't
remember
how
rgw
breaks
stuff
down
by
default,
probably
into
like
four
megabyte
chunks.
Kc
is
that
is
that
right.
A
So
instead
of
having
like
a
big
four
megabyte
object
that
you
can
read
from
just
from
the
primary
now
you're
reading
a
local
copy
from
the
primary,
but
then
you're
going
and
fetching
smaller
chunks
from
other
osds
to
recombine
back
into
the
object
that
you
can
give
to
rgw.
So
you
know
the
impact
there
depends
on
what
these
like
highest.
Latency
fetch
is
from
all
of
the
the
secondaries,
and
that
depends
on
network.
A
C
I
know
you
said
yeah
you
haven't
done
raid
testing
in
a
while,
but
if
we're
primarily
doing
those
kind
of
S3
reads
as
our
biggest
workflow
does,
that
we
want
to
make
the
stripe
size
if
we're
rating
like
as
big
as
possible,.
A
Like
in
terms
of
the
the
hardware
raid,
this
underneath
yeah
I
mean
that's
what
it
used
to
be
like
for
luster,
right
and
other
things
back
in
the
day,
if
you're
doing
big
streaming
reads,
you
wanted
to
like
both
make
the
the
chunk
size
big
and
you
want
to
make
the
the
the
kernel
setting
for
like
how
you
like
Max
sectors,
KB
I,
don't
know
if
you've
ever
look
at
that
inside.
A
What's
the
setting
match
sectors,
KB,
basically
like
how
much
you're
breaking
stuff
down
at
the
Block
device
layer
for
tuning
raid,
you
probably
want
to
be
like
looking
at
that
and
looking
at
what
the
max
setting
that
the
device
allows.
A
C
I
think
like,
if
you
look
at
the
or
maybe
maybe
what
it
is,
is
that
the
at
the
OS
layer
it
actually
detects
the
chip.
You
know
I
think
I've
seen
D
message
something
about
LSI,
so
maybe
the
exit,
somehow
that.
A
Way
it
used
to
be
like
a
the
2008
and
2008
were
the
cards
I
used
to
kind
of
remember
testing
on,
but
but
that
was
a
long
time
ago.
So
yeah
I
mean
the
I
guess.
The
the
answer
is
I
expect
that
if
you
do
EC
at
the
self
level,
you
should
expect
that
it's
probably
going
to
be
a
little
bit
slower
for
this
than
than
just
doing
replication.
Because
of
that
extra
work
I
mentioned,
and
if
you
do
raid,
you
know
below
the
OSD.
A
There's,
there's
probably
going
to
be
some
things
that
you'll
need
to
tune
there
that
that
you
know
are
are
specific
to
the
the
hardware
configuration
that
you're
running
in
terms
of
of
the
performance
of
that
raid
setup
I
mean
it
I
would
expect
it
should
be
like
a
big,
fast,
hard
drive
right.
It's
not
going
to
be
good
at
that
small
random.
I
o,
probably
not
not
as
good
as
the
the
disindividually
I
would
I
would
think,
but
the
controller
cache
is
going
to
have
an
effect
there.
A
So
you
know
lots
of
variables.
C
I
appreciate
that
so
so
we
will
be
playing
with
a
bunch
of
different
those
parameters.
There's
lots
of
lots
of
things
to
test
and
it
takes
a
little
while
to
build
up
a
whole
test
cluster
with
the
different
things.
So
when,
when
we're,
when
we're
doing
our
testing
and
we're
doing
a
bunch
of
reads-
and
you
know
ultimately
what
we
get
is,
we
seem
to
overload
the
rdws
and
start
getting
timeouts
I
mean.
Is
that
what
you
expect
when
you're
like
kind
of
stressing
S3,
you
know
I,
just
you
know,
I'll
load.
C
You
know
five
terabytes
worth
of
data
in
in
these
30
megabyte
chunks
and
then
try
and
read
it
as
fast
as
possible
and
basically,
the
more
the
more
workers
I
have
reading
it.
Eventually
I
just
start
getting
rgw
timeouts,
or
at
least
that's.
What's
read
from
like
you
know
the
API
at
the
at
the
application
layer
from
like
an
S3
read:
are
there?
A
D
A
D
I
think
so:
yeah
I
mean
a
client
shouldn't
time
out
unless
it
receives
no
data
at
all,
for
something
like
60
seconds
or
whatever
its
timeout
is
and
rgw
should
be
able
to
stream
data.
You.
C
I
see
from
our
Gabi
logs
is,
like
you
know,
say
I'm
doing
a
five
terabyte
read
that
I
I
start
getting
pretty
good.
What
looks
like
pretty
good
read
speeds
for
for
a
little
while
for
one
or
two
terabytes
and
the
latencies
look
good
and
then
all
of
a
sudden
I'm
getting
latencies
in
you
know
the
10
second
range
in
the
rgw
logs
and
then
I
start
getting
timeouts,
but
but
definitely
for
or
you
know,
you
know
say
that
took
five
minutes
to
read
five
terabytes
in
in
the
setup.
D
How
are
you
measuring
the
latencies?
Are
you
just
looking
at.
D
C
Well,
for
quite
a
while,
you
know
I'm
getting
a
second,
you
know
like
0.1.2
seconds
and
then
and
then
it
just
hits
some
kind
of
something
and
then
all
of
a
sudden
it
shoots
up
to
you,
know
five
seconds
ten
seconds
and
then
I
start
getting
at
the
same
time.
I
start
getting
at
the
application
layer
like
timeouts,
which,
what's
seen,
is
from
S3
perspective.
Timeouts.
C
B
A
There
was
a
comment
in
the
chat
window:
I,
don't
know
if
you
see
it.
A
In
terms
of
diagnosing
further
one
thing
that
might
be
good
is
to
try
to
see
kind
of
what,
if
you
can,
if
you
can
track
down,
what's
going
on
in
rgw
was
versus
what's
going
up
on
in
the
the
osds
like
you
know,
even
at
a
surface
level,
what
looks
busy
right
is
rgw
really
busy
are
the
OSD
is
really
busy
you
know
is:
is
there
can
you
cast
our
isolating
what's
what's
doing
work?
A
C
A
Those
yeah,
okay,
pretty
big
and
and
when
you
see
this
High
latency
start
kicking
in
what
kind
of
throughput
are
you
getting
aggregate.
C
So
we're
we're
reading
at
about
20
Gigabytes
per
second,
and
then
we
start.
That's
that's
you
know.
That's
our!
Our
aggregate
read
speed
from
our
kind
of
test,
read
cluster
our
workers,
and
that
goes
on
for
say
a
minute
or
two
at
that
kind
of
rate.
C
E
C
A
A
A
A
Might
be
worth
doing
and
we
have
seen
situations
sometimes
where
the
net,
like
the
switch,
looks
good
when
you're
doing
like
point-to-point
tests
and
then,
if
you
saturate
it
with
like
traffic
from
all
Nicks
to
all
other
Nicks,
then
the
switch
kind
of
falls
apart.
A
That
in
the
past,
has
been
primarily
due
to
like
weird
issues
with
the
bonding
configuration.
I.
Don't
know
if
you're
using
bonding
but.
C
We're
not
using
bonding,
but
we're
actually
thinking
about.
There's
there's
they're
dual
25
gigabits,
but
we
don't
have
the
other.
The
other
Port
lit,
so
something
we
could
play
around
with
is
is
is
bonding
that
and
having
the
225s
bonded,
but
we
haven't
played
with
that.
Yet.
A
Sure,
but
okay,
let's,
let's
just
assume
that
you
can
do
like
two
gigabytes
per
second
and
then
it's
reasonable
for
Nick.
So
with
20
of
those
things
you
should
be
able
to
do
like
up
to
like
40
gigabytes
per
second
would
be
kind
of
what
you'd
expect
the
limit
would
be.
That's
assuming
it's
working
right
that
shouldn't
be
limiting
you
right.
You
should
be
able
to
do
pretty
easily
all
right,
okay,
so
then
I
think,
and
did
you
say
you
said
three,
a
three
disk
grade
zero
behind
each
OSD
and
what
was
the
search
configuration?
C
What
was
the
question?
I
mean?
We
have
eight
eight
osds
replication
or
racial
coding.
Oh
all,
right,
shortcutting.
A
It
was
eraser
coding,
okay,
so
you're,
breaking
up
that
four
megabyte
object
into
multiple
smaller
objects
and
then
you're
requesting
to
raise
zero
to
service
those
I
wonder
a
little
bit
of
the
combination
of
VC
and
raid
is
maybe
not
the
way
to
go.
Maybe
you
do
Wonder,
try
one
or
the
other
and.
A
So
Maybe,
if
you,
if
you've,
got
a
test
cluster
to
just
play
on
and
and
you
can
create
different
pools
with
different,
you
know
stuff,
replication
or
racial
coding
factors.
Maybe
maybe
just
you
know,
don't
don't
redeploy
everything
yet
because
you've
already
got
the
the
three
disk
grade:
zero
ones,
but
just
try
a
couple
different
pools
with
different
different
configurations.
You
know
EC
or
replication
and
and
see
how
that
affects
your
results.
A
C
A
Path
is
that
what
you
said
yeah
exactly
right
like
if
well
that's,
not
true,
it
is
for
the
right
hand,
log
it's
not
for
for
database.
Oh
node
reads
right,
so
so
sorry,
I
I,
that
was
a
stupid
thing
to
say
it.
It
primarily
affects
your
right
path
in
terms
of
the
right
log
for
for
the
database
itself
is
going
to
be
writing
out
SST
files
or
reading
in
SST
files.
A
So
it
you're
going
to
expect
to
see
like
bigger,
bigger
reads,
which
you
may
actually
be
better
on
the
hard
drive.
If,
if
you're,
you
know,
only
have
one
one
device
for
24
hard
drives,
you're
just
gonna
have
to
to
see
it's.
It's
going
to
be
a
I,
don't
know
a
complicated
thing.
A
C
A
Yeah
tough,
to
say
with
just
one
for
24
hard
drives.
That's
a
good
chance
that
you're
going
to
be
overloading
that
thing,
but.
A
It
depends
that
depends
then,
on
how
much
you
have
and
how
much
cash
you
have
for
OSD.
A
A
Okay,
I
I
would
so
the
OSD
will
try
to
tune
self-tune
its
caches
based
on
the
Target
that
you
give
it
for
memory,
and
so
the
the
big
one
that
affects
performance
here
is
the
onode
cache
and
blue
store.
That's
basically
like
an
inode,
but
for
for
sefl
nodes.
A
And
if
you
don't
have
enough,
oh
node
cache
in
in
Blue
Store
to
be
able
to
fit
all
the
O
nodes
that
you're
working
on
there.
Then
it
will
it'll
fetch
from
roxdb
and
roxdb.
Maybe
we'll
have
to
read
an
SSD
file
from
disk
to
be
able
to
fetch
the
owner
that
you
need
so.
C
A
Yeah,
it's
probably
four
megabytes,
then
or
four
gigabytes.
I
have
four
megabytes,
but
if
you've
got
tons
of
memory.
B
A
A
Well,
it's
not
going
to
be
four
megabytes
if
you're
doing
EC
at
the
stuff
level
right
breaking
that
up
into
smaller
chunks.
Okay
is.
C
B
Despite
you
have
a
limited
amount
of
flash,
but
the
amount
of
memory
and
the
handling
for
the
larger
lsds
might
be
more
complicated
than
the
smaller
individual
osds.
So
you
have
a
lot
of
data
backed
by
one
OSD
and
then
try
to
just
suck
in
everything
from
this
large
OSD.
C
Okay,
we'll
try
that
I
mean
one
thing
that
just
to
notice
that
there's
a
change,
because
our
old
cluster
was
running
Nautilus
and
even
in
that
one
we
didn't
have
a
separate
metadata,
Drive
and
like
when
you
do
a
LS
block.
You
could
see
the
the
like
the
the
metadata
partition
on
each
drive,
but
when
I
don't
specify
metadata
drive
here
and
I
get
24
drives,
I,
don't
see
like
a
metadata
partition
on
each
Drive.
Is
that
a
change
with
Quincy
we're
doing
this
on
latest
Quincy?
A
Sorry
Luke
I
was
reading
Tyler's
response
in
the
the
chat
window.
Can
you
repeat
the
question.
C
So
if
you
don't
specify
metadata
drive
it,
just
it
just
uses
one
on
each
of
the
regular
drives
and
I
guess.
The
reason
I'm
asking
really
is
is
is
I'm
Nautilus
when
we
did
that
I
think
you
saw
in
lvm
like
a
a
volume
that
corresponded
to
the
metadata
on
each
drive,
but
I
don't
see
that
in
Quincy
when
I
do
that?
Is
that
just
a
difference
in
organization?
And
it's
just
it's
doing
it.
A
Good
question
I,
don't
know
the
answer
to
it:
it's
possible
that
they're,
organizing
things
differently,
I'm
guessing
novels,
might
not
have
been
using
stuff
idiom
which
Quincy
it
think
is,
but
this
is
more
kind
of
how
like
work
is
doing
things
and
I'm
I'm,
not
an
expert
there.
A
I
mean
it's:
if
you
had
200
boxes
with
24
osts
per
box
or
something
cluster
I,
guess
the
the
question
for
you
would
be
kind
of
what
you
know.
What
are
you
thinking
about
in
terms
of
how
you
you
manage
all
this
and
like
the
things
I've
seen
in
the
past,
have
been
sometimes
the
amount
of
data
that's
being
fed
into
the
manager
can
be
like
a
little
bit
overloading
you
might
have
to
turn
down
some
of
those
settings.
A
A
I
think
he
does,
though,
and
that
would
be
a
really
good
place
if,
if
Dan
is
there
to
talk
to
him
about
it,
because
they
have
very
very
large
deployments
at
CERN
and
and
he
can
probably
give
you
a
a
nice
description
of
what
they've
they've
had
to
do
to
kind
of
you
know
manage
at
that
scale.
B
I
think
the
the
size
with
forehand
and
a
little
bit
osds
is
just
a
medium-sized
cluster,
so
the
usual
deployment
for
some
common
environments.
So
it
shouldn't
shouldn't,
be
a
big
deal.
If
you
go
far
beyond
that,
so
say
two
thousand
dollars.
Something
like
this
might
be
more
communicated,
but
still
there
it
says:
okay,
so
yeah.
B
I
think
this
was
also
the
size
that
that
Kyle
tested
with
in
his
setup
and
separating
rjw
on
cluster
rjw
y's
into
different
zones,
and
this
was
a
deployment
I
think
tested
for
for
an
80
per
Stone
area
in
because
I
was
subclusters.
So
just
I
I
think
that's
reasonable.
B
I
know
just
it's
only
just
for
the
folks,
because
sokai
beta
did
some
testing
around
how
far
you
could
scale.
B
It
was
based
on
the
older
bits,
so
I
think
that
Nautilus
still
but
soda
was
some
kind
of
performance
boundary
that's
imposed
by
the
rgw
caching,
but
so
this
this
size
what
they
used
where
I
think
480
per
subcluster.
So
we
need
a
new
Scale
based
on
that,
so
it
should
be
a
reasonable
size.
Would
you
do.
B
All
right,
just
in
I'm,
sorry,
I'm,
sorry
just
I,
think
I,
remember
correctly
so
20
gigabits
per
second
of
where
something
that
was
the
upper
limit
that
you
could
get
out
of
such
a
set
of
drives,
and
just
even,
if
you
scale
it
beyond
the
number
of
rgws
could
scale.
But
but
you
don't
want
to
get
more
audio
will
not
get
more
performance
out
of
this
basic
setting.
You
know.
A
C
We
tried
different
amounts,
so
that's
that's
actually
reminds
me
the
question
like
what
is
the
optimal
number.
We
could
run
an
rgw
on
on
each
box.
We
could
do
I
think
for
the
20
node
test
cluster.
We
had
eight
or
ten
rgw
nodes
pill
located.
A
I,
don't
know
if
there
is
exactly
an
optimal
number
I
mean,
like
I,
can
typically
scale
pretty
well,
if
the,
if,
if
I'm,
seeing
that
there's
more
capacity
at
the
OSD
level,
throwing
more
rgw's
in
and
having
clients
talking
to
different
ones
generally
lets
me
scale
farther
when
I've
done.
Testing
like
this.
A
C
C
A
Casey,
do
you
do
you
have
any
opinions
I
mean
I've
I've
just
been
there?
Usually
I
can
get
pretty
good
scaling
with
multiple
rgw's
demons
and
clients
talking
to
them
individually.
D
Yeah,
the
that
eight
to
ten
number
sounds
perfectly
fine.
A
A
C
A
And
I
pushed
it
harder,
I've
been
able
to
get
rjwgs
more
than
that,
but
it
depends
on
the
particular
version
of
the
code
you're
using
too
how
much
when
you're,
getting
that
20
Gigabytes
per
second,
how
many
rgw
instances
do
you
have
running.
A
Is
eight
okay,
you
might
be
able
to
achieve
that
with
fewer,
possibly
I
think
I've
gotten
rgw
to
do
more
than
like
two
gigabytes
per
second
in
aggregate
before,
but
it's
it's
been
a
while
since
I
looked
at
it.
C
A
Have
you
really
interested
if
you
did
try
with
fewer
if
it
if
it
topped
out
lower
or
if
it
you
know,
if
you
had
more
issues
with
with
timeouts
I
mean
that
would
be
maybe
a
one
way
to
start
playing
with
and
seeing
you
know
what
happens
if
you
scale
that
bigger
or
smaller,
if
you've
got
20
nodes
to
work
with
I
mean
you
can
run
a
fairly
large
number
of
rgw
instances
that
you
won't
have
memory
issues.
A
B
C
Thanks
really
appreciate
all
the
comments
and
and
help
we'll
we'll
try
a
lot
of
these
different
tuning
parameters
and
different
configurations
and
love
to
report
back
in
a
future
meeting
on
what
seems
to
work
and
probably
more
questions.
A
Yeah
I
I,
don't
know
like
with
four
thousand
eight
hundred
I,
don't
know
if
that's
actually
a
how
big
of
a
cluster
that
is
in
reality,
these
days,
I
I,
don't
typically
get
to
test
out
that
scale.
So
you're
you're
you're
doing
things
that
people
do
do,
but
I
I
I,
don't
so,
oh
Laura.
How
big
is
the
is,
is
the
our
our
test?
Cluster
are
just
like
you
know,
tiny,
tiny,
OST,
one.
A
Okay,
that's
that's!
Basically,
we've
we've
partitioned
stuff
up.
That's
our
I
think
our
biggest
in-house
one
that
we
test
on
is
about
a
thousand.
So
you
know
4
800,
you're
you're.
This
is
the
the
problem
in
storage
all
the
time.
Oh
vendors
never
have
setups
as
big
as
the
the
ones
that
users
are
building
so
definitely
be
interested
in
hearing
hearing
what
you
run
into.
A
Yeah
no
problem,
no
problem
and
yeah
I
would
definitely
encourage
you
to
reach
out
if
Dan's
at
the
user
meeting
you
know.
Cern
is
a
wealth
of
knowledge
for
for
anything
like
this,
so
he
may
have
some
really
good
advice.
A
All
right,
the
only
other
thing
I
was
going
to
mention-
is
that
there's
more
work
ongoing
with
the
RBD
mirror
issue
that
that
we've
been
trying
to
to
get
past
Adam
started
working
on
a
new
version
of
his
shared
blob
code
rather
than
having
a
single
tracker
now
he's
trying
to
make
it
so
that
we
can
add
new
extents
to
an
existing
shared
blob
and
he's
got
code
that
works.
A
We
we
tested
it
out,
and
it's
it's,
maybe
not
quite
as
impressive
as
the
initial
version
of
the
the
one
shed
tracker
was
but
I
think
it's
actually
better
than
the
fixed
version
of
the
one
share
tracker.
So
in
the
chat
window,
I'll
put
a
link
to
the
latest
test
results.
That's
here
for
anyone
interested
I
also
tested
his
latest
version.
This
is
the
this
elastic
shared
blob,
Branch
overlaid
with
my
defrag
on
clone
pull
request,
and
so
the
original
is
the
Blue
Line
there.
A
His
work
is
the
yellow
line.
The
the
defray
gun
clone
work.
That
I
did
is
the
red
line,
and
then
the
combined
is
the
green
line
and
I
know
this
is
kind
of
hard
to
read.
It's
it's
really
kind
of
nasty,
maybe
I'll
I'll
actually
share
green
and
I
can
just
kind
of
point
it
out.
A
Okay,
there
we
go
so
anyone
see
that.
C
A
D
A
A
A
Yep
so
yeah,
that's
the
the
gist
of
it
is
really
that
the
combined
work,
where
we're
doing
both
the
defrag
and
the
atoms
elastic
shirt
blob,
looks
like
we
could
benefit
from
both
of
them
and
their
their
additive.
So
that's
kind
of
what
we're
hoping
for
is
that,
maybe
maybe
by
using
both
we
we
can.
We
can
kind
of
make
this
better.
A
The
the
thing
that
we're
still
worried
about,
though,
is
that
just
getting
snap
shots
and
clothing
faster
may
not
be
enough
to
make
RBD
mirror
faster,
we'll
we're
gonna,
hopefully
get
this
into
Paul's
hand
soon.
So
we
can
try
with
his
RBD
mirror
setup,
but
that's
that's
kind
of
the.
The
concern
is
that
this
might
not
be
enough.
We'll
see.
A
So
yeah
yeah,
hopefully-
and
that's
that's
the
only
other
thing
I
wanted
to
talk
about
today,
so.
E
Just
a
quick
note,
basically
I
raised
you
from
the
cursing,
if
you,
if
we
have
time,
of
course,.
E
Crc52
faster
for
arm
V8
I've
taken
a
look
on
the
implementation
in
rocksdb
and
actually
it's
free
available.
The
question
is
whether
we
really
pick
it
up.
I
started,
let's
start
from
our
good
old
old
friend,
from
similar
x86
investigation.
E
Here
is
the
point
where
we
are
determining
which
which
exact
implementation
should
be
taken
and
what
should
be
I,
guess:
I,
don't
know
rm2l
I,
guess
it
should
be
this
one.
Let
me
paste
link
to
the
fast
path,
but
I
think
I
think
it's
faster
for
arm
64.
E
E
So,
first
of
all,
we
need
to
have
the
auto
detection
at
make
level
fully
operational.
So
so
let
me
pinpoint
then,
to
an
exact
area
to
ensure
the.
B
E
E
And
also
another
thing
is
that,
let's
do
it
the
probability
of
of
get
aux
valve,
it's
also
checked
by
CRC
and
finally,
the
hard
kernel
needs
to
inform
us
is
to
set
this
beat
select
the
proper
CRC
implementation.
E
If
either,
if
any
of
of
those
is,
is
false,
it's
not
correctly
working,
then
we
are
calling
falling
back
at
a
slow
implementation,
but
just
to
ensure
I
understood
that
you
saw
those
those
issues
in
rocksdb,
because
okay,
we
have
at
least
two
places
that
are
interested
about
your
critical
collection
on
the
right
path.
First,
one
would
be
okay,
free
messenger
Bluster,
but
those
parts
of
blister
that
are
being
executed
by
the
TP
osdtp
threats
and
finally,
robsdb.
A
So
the
the
background
for
folks
on
the
call
we
got
I
got
an
email
from
Rama
over
at
Amper
Computing
trying
to
do
some
performance
testing
on
their
own
CPUs
and
we're
getting
more
performance
to
the
next
86.
We're
trying
to
figure
out
why
they
did
some
wall
clock
profiling
and
in
the
wall
clock
profile,
it
appeared
that
we
were
using
the
the
slow
cr3
flow,
CRC
32
path
rather
than
the
fast
path.
A
That's
the
the
background
here
just
give
me
one
second
Braddock
to
find
the
right
line.
Sure.
E
Mark
just
a
note
on
that,
it
might
be
misleading.
I
recall
from
x86
that
on
the
on
some
platforms,
it
was
it
was
in
line
in
a
way
and
in
profiling.
E
Somebody
could
get
suggested
that
fast
CRC
is
being
used
while
because
of
identical
right
now,
maybe
some
inlining,
maybe
maybe
other
compiler
optimization,
but
it
was
at
first
at
first
glance
it
was
looking
like
past
algorithms
was
selected,
but
actually
the
slower
one
was
was
in
use,
so
might
be,
might
be
worth
to
double
check,
okay
extent,
implementation.
D
A
E
That's
the
place
for
selection,
see
still
the
same,
have
irm
64,
CRC
and
also
the
same
way.
Crcp
model
runtime
check
so
basically
like,
like
the
regions
mentioned
before
the
same
conditions,
to
have
to
have
a
fast
CRC.
E
Anyway,
what's
it,
how
do
we
build
well,
what
was
the
source
of
the
packages
images,
whatever
I
used
in
this
particular
environment,
great.
A
It's
it's
safe,
Quincy
on
Ubuntu
installed
with
surfadium
in
containers;
okay!
So
that's
what
it
is!
It's
whatever
stuff
ADM
does
when
you
use
it
on
on
Ubuntu.
E
E
Well,
actually,
at
least
it's
worth
feeling
a
Tracker,
it's
not
it's
not
what's
this
information,
this
one
RM,
it's
writer,
rarely
containing
our
in
our
trackers
yeah.
Do
you
have
any
any
hardware
for
investigation.
A
I
want
to
say
that
I
thought
we
did
have
some
kind
of
Emperor
right
in
the
lab.
You
know
we
have
to
because.
A
Yeah
Dan
Mick
has
worked
on
it
before,
but
I,
don't
I,
don't
know
anything
about
the
other
than
that.
We
we
have
it.
E
No
idea
it
looks
like
we
will
need
to
go
to
the
main
main
increase.
The
task-
maybe
maybe
somebody
knows.
A
Yeah
I'll
I'll
reply
back
to
tarama,
though,
and
and
just
mention
that
we
we'd
like
to
to
create
a
tracker
for
the
CRC
issue.
So
we
can
at
least
get
it
get
into
the.
E
System,
well,
it
might
be
worth
providing
them
with
the
links.
Maybe
they
are
doing.
Okay
aw,
they
are
doing.
Custom
builds
I
doubt
that
some
realistics
for
us.
A
Yeah
they
I
mean
they
said
the
in
this
document.
I
see
now
that
the
the
Seth
Quincy
and
I'm
going
to
install
the
surf,
ADM
and
containers
so
I,
don't
think
they're
doing
a
custom,
build.
A
All
right,
well
anything
else,
guys.
A
If
not,
we
used
up
the
whole
hour.
So
thanks
for
sticking
around
and
have
a
have
a
good
week.