►
From YouTube: Ceph Science Working Group 2022-05-24
Description
Join us for Ceph Science Working Group meetings. We alternate the third and last Tuesday of each month at 14:00 UTC.: https://ceph.io/en/community/meetups
Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contribute/
What is Ceph: https://ceph.io/en/discover/
C
B
A
Yeah
at
least
on
our
clusters.
We
don't
change
the
default
tunables
for
for
scrubs
and
deep
scrubs,
but
by
default
it's
supposed
to
be
it's
supposed
to
be
doing
things
in
the
background,
however,
if
you
have,
I
mean,
if
you
have
billions
and
billions
of
small
objects,
then
maybe
there
are
cases
where
the
rock
cb
can
become
really
slow
reading
through
and
iterating
through
the
objects
in
the
rocks
db.
A
That's
why
I
was
asking
if
it's,
if
that's
related,
but
did
you
did
you
maybe
write
a
bunch
of
objects,
then
delete
a
bunch
of
objects,
then
write
more
objects
than
delete
like
is
there?
Is
there?
Is
it
possible
there's
quite
a
lot
of
deleted
objects
in
the
in
the
I.
B
A
A
You'll
get
those
warnings.
If,
if
seth
cannot
scrub
all
of
the
objects
every
day
and
cannot
deep
scrub
all
of
the
objects
every
week
but
those
those
defaults,
obviously
they
don't
make
sense
for
every
like
for
a
full
disc.
If
it's
like
a
16,
terabyte
disc,
it's
simply
impossible
to
deep
scrub,
all
of
the
objects
once
a
week.
So
yeah
you
should
you
can
just
there's
a
there's,
a
tunable
to
to
have
to
to
wait
longer
to
complain
to
complain
after
a
longer
amount
of
time.
Have
you
seen
these
these
options.
B
Yeah
we
we
had
this
email
conversation
with
pieteri.
I
think
pietre
sent
you.
A
B
D
D
B
D
Okay,
so
thing
for
watch
out
for
is
this
like
you're
using
any
raid
controllers
like
they
might
be
doing.
I
know
dell
calls
it
like
patrol
reads
where
it
does:
a
consistency
check
on
the
whole
disc,
like
that'll,
slow
down
that
whole
disc
and
I
o
or
scrubs,
or
anything
for
quite
a
while
on
large
discs.
A
B
A
Yeah,
I'm
thinking
I'm
thinking
about
creating
a
tracker
for
this,
so
that
I
mean,
if
we
do
the
math,
if
we
do
the
math
for
to
deep
scrub,
every
object
in
a
disk
in
one
week,
what
would
be
the
megabytes
per
second
that
would
be
needed
to
do
that.
I
guess
you
have
to
also
like
multiply
it
by
the
replication
count
as
well
to
count
the
amount
of
I
don't
know.
I
I
think
we
should.
A
We
can
justify
through,
like
the
data
rate,
to
see
how
how
often
it
would
take
to
deep
scrub
say
a
20
terabyte
disc,
which
is
becoming
the
normal.
Now
I
would
say,
20
terabyte
is
at
least
this
year
or
next
year
is
what
people
will
have
mostly
and
then
we
can
set
the
default
accordingly.
Maybe
the
default
should
be
two
weeks
instead
of
one
week
or
three
weeks,
but
then
yeah.
E
D
A
Because
we
had
a
long
ago,
we
had
a
cluster
that
was
six
terabyte
drives,
but
the
workload
was
such
that
deep
scrubs
couldn't
be
guaranteed
more
often
than
once
per
two
months,
but
you
would
never
have
calculated
that
from
the
six
terabyte
drives,
but
because
the
drives
were
so
busy
with
client,
I
o
they
could
not.
They
didn't
have
iops
for
for
scrubbing.
A
A
We
can
open
a
ticket
and
see
and
see
with
the
os
with
the
routers
team
if
they,
if
they,
if
they
have
a
better,
a
better
suggestion.
A
E
D
A
A
So
if
you
have
like,
if
each
osd
can
only
do
one
scrub
at
a
time,
then
like
three
replica,
oh,
a
three
replica
pool
can
be
scrubbed
twice
as
fast
as
a
four
plus
two
erasure
coated
pool,
because
the
four
plus
two
is
involving
it's
locking,
six
osds
and
the
replica
three
three
osds.
So
it's
also
the
it's
also
the
the
pg
size
not
count
yeah,
not
num,
not
pgm.
D
Yeah
see
what
she
thinks
about
it.
A
Okay,
but
I
I
mean
yoni
what
I
was.
What
I
was
saying
is
that
in
practice
you
just
increase
the
ratios,
that's
what
most
people
are
doing
now,
I
would
say
if
you
get
the
warning:
just
increase
the
ratio
and
increase
that
rate,
the
ratio
for
scrubbed
and
deep
scrubbed.
Until
until
you
stop
getting
the
warning,
voila.
A
B
B
B
E
Maybe
write
a
script
to
find
out
from
pg
dump
which
pieces
are
not
scrubbed
and
we
can
get
the
timestamp
right
when
were
they
last
scrub
from
the
pg
dump
and
run
a
deep
scrub
manually.
E
A
A
D
Other
topics-
that's
just
tossed
in
the
pad,
was
like
you
know,
there's
some
frequency
hit.
I
don't
know
if
anybody's
even
tried
that
one
yet
for
releases.
I
want
to
talk
about
it,
octopuses
and
then
the
life
here.
A
Yeah
there's
a
new
octopus
release
that
will
be
coming
out.
There's
there'll
be
there'll,
be
one
more
octopus.
Release
is.
D
C
A
A
A
B
Well,
the
most
biggest
reason,
I
think,
is
that
we
don't
have
time
to
do
it
as
we
are
bringing
bigger
and
bigger
clusters
production
after
after
the
peak
cluster
is
it's
coming
in
pacific.
I
think
we
will
run
in
pacific.
We
haven't
tested
quincy
yet,
but
after
that
is
in
production,
I
hope
that
we
can
upgrade
from
nautilus
to
pacific.
I
think
we
would
be
skipping
octopus.
A
D
A
If
you
have
old
clusters,
even
the
mon
db
won't
load.
I
guess.
A
Yeah
conversion
memory
is
vague.
I
know
that
we
changed
to
rocksdb,
but
I
thought
that
we
did
it
by.
I
thought
that
we
did
it
by
deleting
the
deleting
the
mondb
directory
and
redeploying
them
on,
like
resyncing
them
on
yeah,
but
maybe
if
people
had
ceph
deploy
or
stuff
adm,
it
happened
automatically.
A
D
D
D
D
D
A
D
A
D
D
I
think
what
I'll
end
up
doing
is
either
put
the
those
files
that
are
too
big
we're
using
the
liberators,
if
you
don't
recall
and
input
those
on
s,
the
gateway
or
staff
ffs,
and
then
all
the
other
data
sets
always
on
continuing
libraries.
A
D
Yeah
I'll
I'll,
be
there,
you
think
you're
making.
D
A
E
A
We
use
elasticsearch
for
the
for
s3
logs,
but
we
don't
use,
but
not
for
sep
itself.
Right
now,.
E
E
We
were
researching
between
low
key
from
tail
in
elasticsearch
and
even
greylock,
because
greylock
insight
of
greylog
is
implemented
natively
by
default.
But
graylog
has
some
issues
like
you
have
to
statically,
configure
everything
so,
but
out
of
the
whole
decision,
loki
and
prom
tail
were
chosen
since
grafana
is
used
already.
D
I've
been
thinking
about
streaming
like
seth,
logs
and
everything
else
in
the
loki
two.
The
nice
thing
is,
you
can
just
point
it
at
the
radio's
gateway,
s3
back
end
for
its
storage
benefits
for
all
the
logs.
A
So
this
this
lucky,
I'm
looking
at
that
loki
pr
was
that
did
that
make
it
into
to
quincy.
A
E
Yeah
from
a
user
standpoint,
I
thought
when
the
clusters
can
be
as
huge
as
possible
right,
we
could
have
logs
in
one
place.
We
can
do
a
pattern
based
searching
for
it
would
be
a
one-stop
search
right
for
the
logs
and
troubleshooting
could
be
much
more
easier
from
a
user
standpoint.
D
It
seems
like
that's
a
lot
of
the
way
this
features
and
stuff
have
been
going.
Lately
is
more
easy
usability
of
everything,
the
dashboard,
if
you're
going
to
do
low-key
and
logs
as
well.
D
A
C
Which
was
perfectly
fine
with
the
kernel
client,
the
cfs
until
recently,
when
the
performance
has
really
turned
abysmal
and
I'm
a
bit
perplexed,
because
other
clients
are
fine.
So
I
think
the
service
itself
is
basically
okay
and
I
see
lots
of
messages
about
sockets
put
one
in
the
chat.
C
C
E
A
A
Fixes,
like
that,
could
that
be
causing
this.
C
I
don't
think
there's
been
any
theft
related
upgrade
on
the
repos
that
are
available
for
it
in
quite
a
while.
So
it's
why
it's
suddenly
changed
it's
perplexing.
Oh.
B
B
C
C
A
And
didn't
make
any
difference
check.
Maybe
the
can
you
check
for
like
tcp
errors,
tcp
retransmits,
like
maybe
there's
something
wrong
on
the
network
interface?
Maybe
it's
not
really
to
power
at
all.
Maybe
it's
I've
seen
this
in
the
past.
A
A
D
I've
seen
something
in
one
of
our
custom
apps,
where
we
were
hitting
like
view
limits
for
open
files
sockets,
but
you
would
probably
see
a
error
more
regarding
that.
That
was
a
case
of
this.
C
C
D
All
right
anything
else,
we
want
to
talk
about
or.
D
D
Yeah
I
had
all
people
said
too:
they
would
work
for
them,
but
could
just
be
bad
luck
this
month
with
people
that
not
too
many.
D
Oh
sure,
if
you
want,
but
I
sent
it
to
the
stuff
users
and
then
I
also
send
it
to
like
a
private
mail
list
of
people
who
joined
so
as
long
as
your
email
is
in
the
sign
in
section
of
the
that
pad
I'll.
Add
you
to
the
private
list
so
that
you'll
get
a
separate
email.
That's
just
a
group
of
us
who've
been
to
these
before
and
usually.