►
From YouTube: Ceph Science Working Group 2021-03-24
Description
A
A
B
Well,
based
on
irg
channel
discussions
and
the
mds
issues
with
the
cfs's
kind
of
annoying
things,
at
least
it
sounds
like
people
are
having
issues
when
they're
they
bringing
their
mdss
back
on
on
a
cluster,
and
things
like
that.
I
think
that
someone
here
had
last
time
to
mention
about
the
same
kind
of
issues.
C
B
A
C
In
did
anyone
did
anyone
see
for
bugs
there's?
I
think
we
we're
planning
to
upgrade
to
14
to
18.
Soon
it
looks
pretty
good.
I
don't
know
if
anybody
had
real
big
problems
with
it.
There's
one
known
issue
with
the
with
the
interface
that
it
binds
to
for
some
people
it
binds
to
the
loopback
interface
because
of
some
some
commit
that
changed.
C
A
E
E
B
B
I
was
wishing
and
waiting
that
the
fifth
bug
that
I
was
preferring
already
on
etipad
would
have
some
got
some
fixes
because
it's
been
there
quite
a
long
time-
and
even
rafael
pointed
me
about
that
there
there
is
a
bug
fix
on
a
master
branch.
It's
not
point
and
click
solution
for
for
nautilus.
Yet
so
the
structure
for
the
user
tenant
is
different
and
it
would
require
some
coding.
And
frankly,
I
don't
have
a
time
for
for
making
that
code
change
for
backboarding
that
bug.
B
Well,
our
we
have
five
different
clusters
and
we
are,
we
are
not
running.
There
is
a
legacy,
client
interfaces
that
we
need
to
support
and
I'm
not
confident
going
to
like
master
release
or
octopus.
Yet
with
that
system
or
systems,
and
and
that's
why
I
was
asking
about
to
make
some
matching
different
versions.
So
I
have
every
anyone.
E
B
C
On
our
side,
our
plan
is
to
upgrade
to
the
last
nautilus
so
like
14
to
18,
because
that
will
let
us
immediately
do
some
moving
of
hardware
around
that
we
have.
We
have
some
new
machines
and
move
a
lot
of
pgs
around
and
then
from
there
yeah.
I
think
octopus
is
okay.
Now,
right,
I'm
not!
I
don't
have
big
fears
of
octopus.
There
were
some
scary
s3
things
in
octopus.
Like
objects
disappearing.
C
E
C
F
F
C
C
B
C
F
F
A
A
E
B
D
C
H
D
I
D
I
Bring
it
on
matthew,
I
I
saw
that
and
I
had
the
thumbs
up
the
whole
way,
because
we'd
really
like
to
keep
everything
really
simple,
and
so
we
can
see
what
the
osd's
are
doing
and
we
don't
use
containers.
Hardly
you
know
anywhere
in
our
infrastructure,
we're
scientists
and
we're
just
lots
of
data
like
for
probably
you
guys,
but
it
adds
another
layer
of
abstraction
which
complicates
management
from
our
perspective
and
I'm
you
know
I
was
depressed
this
morning
trying
to
use
self
deploy
on
a
alba
eight
lineup
spots
of
money.
J
I
I've
got
a
question
as
well,
so
with
the
new
version,
do
you
have
to
use
fadm,
or
can
you
control
it
by
some
other
means?
So
we
are
we're
using
no
tillers
on
on
manage
on
managed
linux
systems
and
we
don't
use
self-deploy
either.
So
it
we've
got
our
own
scripts
to
to
do
all
that
and
yeah.
If
he
had
to
change
too
massively
that
that
might
be
quite
quite
tricky
for
us.
E
E
B
I
can
try
condor
containerized
dev
cluster
with
our
fresh
eight
racks
set
up,
but
I'm
pretty
sure
that
we
don't
put
that
on
a
production,
because
we
we
we've
been
having
issues
during
the
years
and
debucking
containerized
environment
is
pretty
hard.
If,
if
there's
a
real
big
issues
on
a
hardware
level,
the
obstruction
layer
add
the
additional
additional
obstruction
layer
between
the
hardware
and
the
safe
cluster
will
will
keep
give
us,
maybe
a
too
much
complexity
with
the
scale
that
we
are
installing
the
systems
for
small
scale
like
15
node,
dev
cluster.
H
H
B
B
Because
I'm
pretty
sure
that
well,
we
are
running
bare
metal,
osds
and
we've
been
having
issues
with
the
memory
constraints
earlier,
for
example,
this
we
had
a
downtime
and
there
was
some
issues
with
that
with
a
pg
lock
that
grows
too
fast
or
or
too
much,
and
it
it
required
a
lot
of
memory
on
those
nodes.
So
that's
why
I'm
in
interested
about
the
content,
containerized
ost
memory.
A
Just
to
have
a
easy,
so
you're,
not
so
os
dependent.
So
though
you
can
have
everything's
all
all
your
libraries
are
there
there's
less
variables
involved
in
running
your
your
demons
yeah.
It
makes
troubleshooting
easier
for
the
developers
because
they
know
how
the
image
was
built.
It's
not
you
know,
somebody's
cluster
that
could
be
built
in
any
way
with
any
number
of
variables
involved.
B
B
F
F
C
A
A
C
B
E
I
have
to
admit
not
sitting
in
front
of
zoom
all
day,
like
I
spend
too
much
time
in
front
of
zoom
already,
so
I
think
from
that
point
of
view,
given
there's
no
need
to
have
everyone
together,
and
so
you
know
the
need
to
put
it
all.
In
one
day
I
think
actually
spreading
out
and
not
having
so
much
student
fatigue
in
a
day
is
quite
a
good
idea.
Actually
that
does
seem
like
quite
a
good
idea.
F
B
Month,
zooming,
for
example,
with
the
safe,
even
though
I
would
like
it,
but
I
would
add
something
more
to
that.
The
irc
channel
is
like
seeing
a
tread
and
well
f
is
not
single
thread.
We
have
a
lot
of
threads
right
and
what,
if,
along
that,
we
have
a
some
kind
of
low
dock
or
rocket
chat,
or
some
some,
this
custom
board
with
that
that.
B
B
C
C
Do
you
think
that
it
would
be
really
popular
to
have
people
submitting
talks,
or
should
it
be
like
each
day,
maybe
like
one
developer,
a
7s
developer
presents
what's
new
in
cfs,
and
maybe
one
or
two
users
present
some
some
related
talks
in
that
area
like
how
do
you
think
do
you
think
it
should
be
a
regular
call
for
papers,
or
should
it
be
like?
Maybe
the
board,
like
picks
like
like,
knows,
picks
known
people
and
invites
them
invited
talks?
Let's
say
what
do
you
guys
think?
A
B
Yeah
there
there
should
be
a
marketing
that
you,
you
should
put
your
car
for
that.
If
you
have
a
good
paper,
even
if
it's
not
the
best
paper
that
you
can
get,
but
it's
if
it's
still
something
like
if
you
are
dealing
with
a
genomic
area,
and
you
have
a
certain
genomic
pattern
that
you
need
to
feed
on
a
safe
cluster.
You
you
are
using
this
kind
of
speed
up
parameters
and
make
a
sensitive
data
check
your
way,
and
that
might
be
a
good,
but
it
it
will
go
on
on
outside
the
radar
quite
easily.
B
C
So
when
you
say
marketing,
do
you
think
that
there
should
be
like
voting
on
the
paper
on
the
on
the
proposed
topics,
because
I
think
normal
in
a
normal
cephalocon?
We
would
have
many
many
many
tracks
and
many
many
papers
or
talks.
Yes,
but
we
probably
won't
have
the
same
capacity
for
for
ceph
month
right.
E
B
A
A
C
C
E
I
think
having
some
sort
of
shared
focus
is
quite
useful.
I
think
I
mean
this
course
sort
of
grew
out
of
a
meeting
of
the
stuff
like
on,
if
I
remember
rightly
where
we
thought
it
might
be
useful
for
both
running
in
a
similar
sort
of
problem
space.
If
you
like
to
catch
up
from
time
to
time-
and
I
think
maybe
that
topical
focus
is
quite
useful-
that's
not
to
say
you
couldn't
have
other
similar
meetings
with
slightly
different
topical
focuses.
A
Yeah
I
agree
having
like
the
the
shared.
The
shared
topic
is
nice,
but
for
like
another
group
like
you
know,
if
they
wanted
to
split
off
and
do
like
a
cloud
users
where
they
focus
more
on
rbd
usage
or
I
don't
know
whatever,
we
just
would
have
to
figure
out
what
those
topics
are
and
see.
If
there's
a
community
enough
within
the
seth
community
to
want
to
get
together
every
once
in
a
while
and
discuss
this.
C
A
A
B
B
That's
that's
great
when,
when
you
have
a
lot
of
stuff
going
on,
but
it's
bit
annoying
that
if
you
have
up
up
map
manager
process
running
and
it
sees
that
well,
I
have
one
host
full
of
like
one
for
four
four
percent
of
pages
or
not
misplaced
it.
It
will
strike,
try
right
away,
put
them
all
in
a
queue
and
then
my
up
mapping,
pre-op
mapping
is
kind
of
wayne.
It's
it's!
It's
taking
them
on
a
backfield
state
again
quite
fast.
C
E
A
E
B
B
B
For
the
next
four
bullet
point
about
the
beast:
civic
wet
experience
in
our
side
we
haven't-
we
are
still
running
civic
web
on
us
on
our
certain
clusters,
mainly
because
it
was
there.
We
haven't
changed
that
and
we
didn't
see
the
on
that
user
load,
much
change
about
the
performance,
but
on
on
some
some
environments,
I
would
go
with
a
beast
if
possible,
because
it's.
E
C
F
F
E
F
C
C
Do
you
guys
force
https,
or
do
you
allow
http?
Personally,
we
keep
hdp
enabled
because
if
you
have,
if
you
have
applications
streaming
at
like,
if
you
want
to
use
real
stream
at
gigabytes
per
second,
you
can't,
I
don't
think
you
can
encrypt
that
fast,
like
I
find
that
there's
a
performance
there's
a
throughput
limit,
and
I
did
it
open
ssl
test
this
week
and
like
one
cpu,
can
encrypt
at
300
megabytes
per
second.
Only
so
am
I
am
I
like
speaking
nonsense
here
and
has
everyone
switched
to
secure
or.
E
Not
because
a
lot
of
our
use
cases,
people
are
sharing,
signed,
urls
and
external
sharing,
and
we
didn't
want
to
have
the
the
risk
of
people
using
the
wrong
protocol
by
mistake,
and
so
we
now
also
promote
as
well.
So
if
you
connect
for
a
while,
we
had
no
listener
on
hd
on
port
80
at
all,
and
now
we
have
remote
in
hd
proxy.
F
C
I
have
no
idea
when
I
google
around
I
found
that
there
was
a
website
called,
is
tls
fast
yet
or
something
like
that,
and
it
gave
me
some
open,
ssl
commands
to
run
and
it
seems
like
they've
focused
on
making
http
like
they've
focused
on
the
latency
aspect
like
now,
there's
like
there
were
that
you
used
to
have
to
retry
or
something
I
don't.
I
don't
know.
I
don't
know
how
this
works,
but
you
used
to
have
to
like
used
to
be
many
many
hops
or
like
many
back
and
forth,
to
actually
negotiate
the
connection.
C
And
now
it's
like
zero,
zero,
rtt
or
something
like
this.
But
anyway
they
focus
on
the
latency.
But
not
the
throughput
aspect,
and
I
mean
our
raz
gateways
are
running
10
gigabits.
They
have
10
gigabit
interfaces
and
I
yeah.
I
would
just
be
curious
to
know
if
people
can
saturate
a
with
https,
like
at
one
gigabyte
per
second
from
one
router's
gateway.
C
B
E
Yeah,
okay
and
we
spent
I,
I
spent
some
time
tuning
because
there's
hk
proxy
in
front
which
gives
us
some
reliability,
and
I
spent
a
bit
of
time
tuning
how
much
cpu
we
give
h
a
proxy
and
how
many
threads
we
give.
If
it
were
then,
and
the
number
of
random
people
that
kind
of
thing,
and
that
made
quite
a
difference
before
we.
B
E
Https
so
because
the
way
the
network
is
we
we
wanted
to
have
the
proxy
might
not
be
on
the
same
machine
as
their
ls
gateway.
It's
talking
to.
We
want
to
keep
that
traffic
encrypted,
though
the
tls
termination
is
done
by
the
civic
web
process,
and
at
that,
with
with
that
benchmarking,
we
were
basically
using
all
the
cpu
on
our
other
google.
E
C
C
C
C
Oh
sorry,
that's
me:
well,
we
had
an
incident
like
february
18th,
where
cefs
got
really
slow
for
everyone,
and
we
don't
really
know
why
and
by
slow
I
mean
like
it
took
10
seconds
to
create
a
file
and
then
we
well.
We
kicked
out
some
clients
and
we
didn't
very.
We
don't
really
have
very
good
monitoring
to
know
which
clients
are
doing
are
are
being
heavily
loaded,
but
the
whole
time.
We
noticed
that
the
the
mds
was
like
just
spinning
like
the
cpus
was
flat
140
140
percent.
C
So
I
guess
we've
known
for
a
long
time
that
the
mds
is
single
threaded,
but
I'm
just
wondering
if
other,
if
there's
anyone
else,
that's
running
like
big
sffs
that
notices
this
and
do
you
have
like.
Have
you
already
been
through
a
cycle
of
testing
and
tuning
and
getting
the
best
possible
mds
optimized
hardware,
and
then
second
thing
is
like?
C
B
E
I
J
J
J
I
think
it's
900
clients,
something
like
that
and
yeah
it
just
so
at
some
stage
I
I
wondered
if
he
should
have
multiple
mdss
and
I
just
noticed
the
load
goes
up
and
figured
well.
Actually,
I
will
deal
with
dive
if
there's
a
problem
but
yeah
it
just
works.
So,
okay,
I
I
did
notice
once
it
slowed
down
for
a
little
while
I
think
some
people
copied
a
lot
of
stuff
around
and
the
cluster
was
quite
busy,
but
otherwise
it's
it's
fine.
J
C
J
But
so
all
the
machines
inside
the
server
room
are
adjusted,
so
we've
got
general
compute
boxes,
multi-user
boxes,
remote,
desktops
and
so
on,
and
so
we
have
one
client
per
mount
and
that's
mounted
cfs
and
then
outside
the
server
room.
We
deliver
the
home
directories
via
samba
and
nfs
and
but
we
probably
get
rid
of
nfs.
If
I
get
around
to
it
that
yeah.
I
C
I
I
That
okay,
yeah
on
the
people
have
got
linux
boxes,
who
have
their
own
roots.
We've
got
a
handful
of
those
and
we
trust
them
with
this
fx
key.
But
it's
not
the
roots
of
x
key,
it's
just
their
home
directory
key
effectively,
and
so
they
can
access
it
from
the
cluster
and
they
can
access
it
from
their
their
own,
their
own
box
and
as
long
as
we
know,
they've
got
a
fairly
up-to-date,
client
and
they're
not
going
to
use
some
neolithic
version
of
seth.
C
I
Just
sharing
not
sharing
this
way,
yeah
you're,
not
showing
the
route,
we're
just
here's
the
key
for
your
directory
and
and
and
your
sub
directories
and
we've
had
somebody
who
wants
to
do
their
own
samba
server
for
their
own
group
and
well.
Why
not?
You
know
if
it's
if
it's
from
your
subdirectory
tree
you're
doing
much
else,
it
works
yeah,
but
it's
good.
H
G
J
Yeah,
it
just
uses
more
memory
than
it
is
supposed
to
so
we
do
see
that
we
for
nfs,
we
use
ganesha
nfs,
and
the
problem
seems
to
be
that
ganesha
does
some
caching
of
metadata
itself
and
so
it's
holding
on
onto
bits
and
when
the
mds
tells
ganesha
to
release
some
as
some
bits,
then
it
doesn't
respond
to
that
and
yeah.
We,
I
just
muted
that
particular
complaint,
though
our
the
machine
that
runs
the
mds
has
plenty
of
memory
and
it's
fun.
C
There
there
are
a
couple
of
like
buggy
cases
like
ganesha.
Does
that
and
also
if
a
user
mounts
ffs
and
then
mounts
and
then
lists
and
like
like
ls
the
directory
and
then
mounts
ffs
again.
On
top
of
it
then
like
for
some
reason,
those
caps,
those
bits
in
the
in
the
original,
the
outer
7s.
They
never
get
released
the
nds,
so
the
mds
will
ask
for
those
be
released,
but
they
never
get
released.
So
there's
weird
things
like
that
that
can
cause
this,
but
otherwise
yeah.
It's
like
it's.
C
It
means
that
the
client
is
usually
when
we
see
that
the
client
is
really
busy
and
they're
they're
do
they're
starting
a
lot
of
files
or
creating
a
lot
of
files,
and
the
memory
on
the
mds
is
too
low.
So
he's
asking
the
client
to
to
call
back
to
like
drop
some
of
its
cash,
but
that
client
is
kind
of
that.
Client
is
allowed
to
do
what
he
wants.
There's
no
throttle
on
the
client,
so
he
just
keeps
grabbing
more
caps,
but
this
is
all
actually
in
14
to
18.
C
C
C
He
slows
down
like
so
that
client
would
then
start
suffering
he
would
his
ls
would
would
slow
down
I'm
going
to
try
to
find
the
they
just
wrote
a
documentation
for
that
after
the
release
was
released,
so
I'll.
Try
to
I'll
I'll.
Add
that
to
the
minutes,
to
the
to
the
minutes
of
this
call,
as
soon
as
I
find
it
might
be,
quick.
G
G
G
G
Yeah
I
actually
tried
the
dml
kernel
as
well,
and
I
could
sort
of
replicate
problems.
It's
it's
really
only
some
directories
and
just
for
test
one
very
problematic
subdirectory.
I
just
moved
it
over
to
a
pool
which
is
not
sharing
snapshots
with
the
main
server
as
data
tool,
and
then
the
problems
immediately
went
away.
So
it's
definitely
got
something
to
do
with
just
the
number
of
snapshots,
I'm
having
on
the
pool.
How
many
snapshots
are
you
using
it's
currently
around
six
to
seven
hundred
snapshots.
H
G
So
I
was
doing
different
subdirectories
for
each
of
the
home
directory
subtrees
more
or
less
and
wanted
to
do
around
50
to
60
snapshots
for
all
the
home
directories
and
since
they
are
not
in
since
snapshots,
are
not
supported
in
some
volume
groups.
I
just
had
to
do
them
for
each
sub
volume
themselves
and
that
just
adds
up
that's
something
that
I'm
going
to
change
in
the
next
few
days
by
just
migrating
everything
into
a
a
tree
that
is
more
or
less
completely
ignoring
the
openstack.
I
J
J
So
the
way
I've
set
up
so
actually
the
most
difficult
bit
for
for
our
system
was
deciding
how
to
to
split
up
the
these
ffs
and
I
decided
to
at
the
top
level
I've
got
a
scratch
and
backed
up
directories
backed
up
everything
underneath
is
backed
up,
gets
shipped
off
every
night
and
we've
got
different
snapshot
schedules
at
those
top
levels.
So
the
backed
up
directories
get
snapshoted
every
night
for
two
weeks
in
this
clutch
one
every
night
for
one
week
and
that's
just
done
at
the
top
level,
and
it's
super
nice.
J
I
Yeah
we
were
a
similar
idea,
but
we've
been
doing
snapshotting
on
the
roots
of
the
file
system.
We
also
have
a
second
cluster
and
we
have
a
whole
lot
of
home
brewed
scripts
to
do
a
type
of
geo
replication
for
backups,
and
we
keep
snapshots
on
the
backup
cluster
for
about
a
month,
and
we
only
keep
a
very
limited
number
of
snapshots
in
the
primary
cluster.
I
I
I
I
B
I
Yeah
45
drives:
we've
released
a
sort
of
a
geo
replication
tool
as
well,
which
is
based
on
c.
I
think
what
it
calls
rsync.
It's
it's
been
rough
around
the
edges,
but
so
there
are
some
people
out
there
trying
to
do
something.
We
wouldn't
release
our
scripts
because
they
they
are
like
granny's
knitting.
You
know
they're
very
messy,
but
they.
C
I
said
I
think
I
would
like
to
use
multiple
file
systems
per
set
cluster.
To
avoid
these
kind
of.
I
would
like
to
give
big
users
their
own
dedicated
mds,
without
relying
on,
without
relying
on
like
directory,
pinning
subtree
pinning
because
subtree
printing
pinning
works
really
well,
but
it
brings
another
kind
of
category
of
problems
when
you
need
to
when
you
need
to
like
upgrade.
You
need
to
decrease
always
down
to
one
mds
due
to
the
upgrades-
and
this
is
always
painful.
I
I
I
We
use
eight
plus
two
encoding
and
the
raw
is
about
five
petabytes,
so
we
get
about
not
far
off
four
petabytes
usable
of
it.
There
was
some
guys
at
nasa
who
were
using
razer
encoding
at
south
barcelona
I
spoke
to,
and
I
think
they
were
using
eight
plus
two
as
well.
It
seems
a
good
balance.
We
would
use
eight
plus
three,
but
we
have
a
replicated
cluster
to
store
everything
on.
B
B
H
Yeah
we're
using
seven
plus
two
for
ratio
coding
and
we
did
try
to
compare
five
plus
two
six
plus
two
and
seven
plus
two
for
their
performance.
When
we
first
set
up
the
cluster
and
it's
they
all
seemed
pretty
similar,
and
then
we
only
had
we
have
ten
nodes
at
the
start.
So
that
limited
us
to
seven
plus
two.
A
C
Let's
say
that
cern
and
fermilab
are
working
together
to
come
to
a
decision
between
them,
because
I
mean
certain
fermilab
used
to
do
scientific
linux.
As
you
probably
know,
and
then
both
we
just
went
with
centos
eight
so
now,
like
together,
they'll
find
a
common
solution,
but
but
there's
some
kind
of
committee
of
like
50
people
or
100
people.
Even
I
don't
know
exactly
that,
are
all
putting
forward
all
the
different
use
cases.
C
C
Yeah
yeah
they
will,
they
will
so
the
details
they'll
be
worked
out,
but
they
will.
They
will
definitely
do
like
a
qa
thing
where
they'll
they'll
they
mirror
upstream
they
put
like
they
put
the
upstream
mirrors
into
like
the
next
next
week's
version,
and
you
can
have
a
few
qa
machines
getting
the
latest
rpms
and
then,
if
everything
looks
okay,
then
you
can
you
can
just
roll
ahead
next
week.
All
of
your
machines
will
get
it
or
like
our
actually,
they
use
ffs.
For
this.
C
They
put
all
these
yum
repos
instead
of
fs,
and
they
do
snapshots
with
hard
links.
They
have
this
tool
that
does
for
each
each
day
they
take
a
snapshot
like
a
hard
linked
snapshot
with
the
date
in
the
directory
name.
So
if,
if
people
want
to
have
the
state
of
the
young
repo
at
this
exact
date,
then
they
can
do
that.
C
J
A
A
Here
all
right
cool
next
one
may,
whatever
the
fourth
wednesday
of
may,
will
be
I'll,
send
out
my
usual
stuff.io
users
list
and
if
you
have
put
your
name
and
contact
information
into
the
the
pad,
I
send
also
a
private
email
to
the
group
along
with
a
calendar
event
in
it.
So
yeah
you're
not
on
it
and
you
want
to
be
on
it.
Add
your
information.