►
From YouTube: Ceph Science Working Group 2020-11-25
Description
A
A
A
A
A
I
guess,
since
nobody's
speaking
up,
we've
all
had
a
great
past
two
months
and
no
outages
or
anything
that
a
change
very
welcome,
though
kinda
along
the
same
topic.
Anybody
hit
any
bugs.
That's
caused
them
headaches
or
you
know
not
necessarily
an
outage,
but
you
know
just
a
general
pain
to
have
in
the
cluster
for
clients.
A
B
This
is
this
is
with
nautilus
like
14
211.
I
think
we
had
to
re-enable
bluefs
buffered
io
that
helped
helped
a
lot
and
then
well
igor.
The
the
blue
store
genius
is
is
replaced,
is
is
preparing.
He
has
a
patch
for
pg
deletion,
and
I
guess
that
would
like
fix
the
underlying
problem,
but
we
need
we
need
buffered
io
in
the
meantime.
Otherwise
the
cluster
is
not
stable.
A
B
B
B
Of
it,
the
end
result
is
a
lot
like
how
file
store
was
when
we
used
to
run
that
the
deleting
and
merging
what
was
it?
The
like,
the
the
merging
that
happens
inside
the
the
file
store
directories
was
too
too
slow,
but
this
is
like
a
completely
different
cause,
but
the
effect
is
quite
similar,
the
osd
start
flapping,
and
then
we
had
to
pause.
The
rebalancing
like
set
no
rebalance
and
then
find
a
way
to
do
to
make
it
more
stable,
interesting.
B
A
A
A
D
According
to
the
box,
we
we
have
one
with
osd
memory
liquid
in
notions.
After
upgrading,
we
noticed
that
our
osd
osds
are
taking
more
and
more
memory,
and
we
have
continuously
flapping
this
in
our
cluster,
so
we
have
one
flap
per
hour
in
5000
osd,
so
I
talked
with
mark
nelson
from
performance
team
and
he
agreed
about
that.
D
D
B
They
became
like
so
large
that
all
the
osds
on
a
machine
went
out
of
memory
and
then
we
restarted
and
it
replayed
the
pg
logs
and
then
all
of
that
memory
went
from
being
in
pg
log
mempool
to
being
in
buffer
and
on
mempool,
and
then
we
shortened
the
pg
log
from
3
000
max
entries
to
500
max
entries-
and
we
haven't
seen
this
since
then,
and
but
I
have
no
idea
why
the
pg
log
got
so
large
for
us
that
that
particular
it
was
like
it
was.
It
was
like
one.
B
It
was
actually
immediately
after
we
upgraded
to
nautilus.
We
went
straight
to
14
to
11.
the
memory
usage
was
going
up
up
up
up
up
and
then
the
p
and
the
pg
logs
were
like
it
seemed.
They
were
never
freeing
their
memory
or
something
like
that,
and
then
we
rebooted
once
everything
and
then
there
were
and
then
we
haven't
had
it
happen
again
since
then.
D
B
D
D
D
D
D
B
Interesting
we
had.
We
had
one
one
issue
recently
where
so
we
we
normally
do
the
tls
termination
in
our
in
our
front
ends,
which
is
traffic.
This
load
balancing
this
load
balancer
traffic.
Anyway,
we
have
some
gateways
that
we
we
connect
to
directly
with
ssl,
and
we
had
an
issue
just
actually
this
week
connecting
to
those
and
they
were
giving
like
tls
errors.
I
wonder
if
this
is
the
same.
C
D
D
I
can
write
you
how
our
tests
are
when
we
test
it.
I
suppose,
tested
my
website
in
friday,
because
for
tomorrow
I
have
another
things
to
do,
but
it's
very
interesting
because
when
we
tested
when
test
upgrade,
when
we
test,
when
we
done
one
upgrade
when
where
we
have
unreplicated
pull
everything
goes
smoothly,
but
on
our
largest
cluster,
we
had
problems.
B
Oh,
that
was
me
well
it's
just
that
when
we
so
we
upgraded
to
from
luminous
to
nautilus
our
our
biggest
ffs
and
we
used
to
have
10,
active,
mdss
and
actually
upgrading
was
quite
painful
because
you
have
to
shrink
down
to
one
mds,
and
there
was
one
step
I
think
from
three
to
two.
We
actually
had
30
minutes
of
slowness
because,
like
re-exporting,
everything
that
was
pinned
to
mds2
or
mds3
to
move
it
down
to
two
was
was
not
transparent.
B
But
anyway
we
we
went
down
to
one
upgraded,
and
now
we
have
three
active
and
actually
like
the
the
first
one
is
doing
most
of
the
work
and
the
the
second
and
third
are
doing
very
little
work.
So
I'm
thinking
that
actually
just
one
active
mds
might
even
handle
the
load
anyway,
and
all
this
multi
mds
isn't
worth
the
trouble.
So
I
just
wonder
if
people
out
there
have
like
really
high
high
metadata
intensive
workloads
on
with
just
one
single
mds
or
if
people
really
do
think
that,
like
multi-mds,
is
needed.
C
B
B
And
what
the
other
reason
that
I
was
looking
at
this
recently
going
down
to
one
is
because
I
tried
I
wanted
to
scrub.
I
wanted
to
like
scrub
a
path
to
fix
some
metadata
that
had
gotten
weird.
Some
some
files
had
had
disappeared
or
something,
and
then
I
tried
to
scrub
the
path
in
it,
and
I
got
the
warning
that
scrub
is
not
supported
on
multi-mds
with
multi-active
mds,
and
then
I
saw
that
there's
a
so.
This
is
now.
This
has
now
been
just
merged
to
master,
and
I
guess
it
will
appear
in.
B
A
A
A
B
Not
I
mean
there's:
no,
we
don't
well
we're
just
starting
to
get
our
first
like
physics,
users
on
it
most
of
the
time
it
is
used
for
kind
of
infrastructure
stuff
like
all
of
our
linux
repositories
and
well,
we
use
it
for
it
is.
It
has
actually
the
home
directories
for
hpc,
but
but
those
guys
don't
really
hammer
it
too
hard.
They
they're
pretty.
They
go
pretty
easy
on
it.
B
B
D
B
D
D
D
D
D
D
D
D
D
D
Organization
of
data
during
upgrade
went
quite
smoothly.
We
didn't
notice
any
interruptions,
so
maybe
in
other
cluster
will
be
something,
but
in
our
test
cluster
we
didn't
sell
so
nothing.
We
will
be
seeing
vapor
production
and
the
productions
later,
but
we're
waiting
for
the
merge
and
then
we
we
we
did
it
to
our
production.
B
Can
I
ask
a
different
question
about
bluestore
there?
There
are
several
allocator
related
crashes
that
people
are
reporting,
especially
with
nvmes.
Do
you
see,
do
you
have
any
like
allocator
related
crashes
at
your
site?
Do
you
have
some
kind
of
custom
custom
blue
store
configuration
that
you
run
or
do
you
just
use
default
14
to
11
blue
store
configuration.
B
B
One
last
question:
maybe
I
can
ask
since
since
everyone's
quiet
we
run
our
rados
gateways
on
some
vms,
with
only
32
gigabytes
of
ram
and
sometimes
if
the
cluster
gets
very
slow
or
if
a
user
sends
a
lot
of
requests.
If
users
really
hammer
the
router's
gateways,
the
memory
usage
of
the
rouse
gateway
can
increase
to
be
a
lot
and
we're
trying
to
find
ways
to
limit
the
route.
D
D
B
B
D
B
B
B
No,
we
have,
we
have
say
eight
or
nine,
maybe
10,
I
don't
know
we
have
15
total.
We
take
randomly,
we
use
rat,
we
use
round
robin
dns
with
say
10
of
them
and
then
so
then
we
have
traffic,
which
is
like
h.a
proxy.
I
don't
know
if
you
know
chaffee
it's
like
aj
proxy
listening
on
port
80
and
443,
and
then
inside
we
route,
we
look
at
the
bucket
name,
we
route
to
specific
routers
gateways,
depending
on
the
bucket
name,
depending
on
like
a
regex
of
the
of
the
bucket
name.
B
A
C
B
D
C
A
A
A
Yeah
and
well,
I
finally
got
some
new
switch
infrastructure.
That's
more
40,
100,
gigabit
switches,
because
right
now
it's
all
just
10
gigabit
for
my
networking,
a
part
of
the
data
center,
and
I
always
thought
that
would
be
a
kind
of
a
pain
with
radius
gateway,
because
I
would
just
need
an
insane
amount
of
ports
going
to
each
host,
because
that
compute
cluster
will
pull
at.
You
know,
160
gigabits
or
whatever.
A
That's
why
we
stuck
with
the
liberators,
because
you
didn't
have
the
bottleneck
of
the
gateway
in
there
right,
yeah,
yeah
yeah,
but
now
that
we
have
the
the
infrastructure
to
support
it.
I
might
actually
start
going
down
that
path
of
finally
get
over
onto
using
radio's
gateway
and
off
of
librido,
so
that
I
can
finally
do
that
boost
door.
Switch
from
file
store
because
of
the
file
size
limit
that
we
have
that
blue
store
has
and
now
finally
realize
some
of
the
benefits
of
it.
B
Yeah,
I
think
I
think
our
our
friends
at
palsy,
super
in
australia
are
trying
to
do
multi
gigabytes,
multi-tens
of
gigabytes
per
second
through
router's
gateway
as
well
on
a
new
infrastructure
that
they're
just
there
that
they're,
I
think
procuring.
Now
I
don't
know
if
I'm
just
checking
who
made
it
who's
on
the
call
yeah,
it's
not
there.
B
A
A
A
A
B
The
the
the
main
place,
so
we
also
have
netapp
and
ceph,
and
we
use
netapp
for
our
oracle
databases,
so
we
haven't
yet
our
db
team
has
not
yet
had
to
the
guts
to
move
to
cef.
Yet
for
that
versus
ffs.
But,
to
be
honest,
I
mean
on
the
just
keeping
technical,
I'm
not
sure
how
we
would
back
up
a
cfs
cluster
that
was
running
all
of
our
oracles,
because
we
have
something
like
10,
petabytes
of
oracle
or
more
now.
This
is
that's.
B
B
B
A
C
A
A
All
right:
well,
thanks
everybody
for
joining
in
good
talk
the
next
one
I
guess
would
be
close
to
the
end
of
january,
whatever
the
the
fourth
wednesday
of
january
will
be
I'll,
send
out
the
usual
reminders
on
the
set
list
and
to
the
private
email
group
as
well,
otherwise
enjoy
the
holidays,
and
I
guess
yeah.
You
continue
to
be
safe.