►
From YouTube: Ceph Science Working Group 2023-01-31
Description
Join us for Ceph Science Working Group meetings. We alternate the third and last Tuesday of each month at 14:00 UTC.: https://ceph.io/en/community/meetups
Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contrib...
What is Ceph: https://ceph.io/en/discover/
B
I
just
started
talking,
so
you
know,
oh
so,
I
I
think
I
sat
through
one
of
these
before
and
looked
in
the
background,
but
didn't
talk,
but
I'll
talk
this
time.
Garen
Atterbury
I
work
with
the
University
of
Nebraska,
Lincoln,
well,
University
of
Nebraska
system.
Our
research
computer
groups
called
the
Holland
Computing
Center.
We
have
a
few
ceph
clusters
of
various
shapes
and
sizes.
B
B
B
Yeah
we
went
to
the
set
of
this
for,
for
this
particular
cluster.
The
majority
of
the
data
on
the
17
raw
petabyte
one
is,
is
this
ffs
file
system,
then,
in
the
past,
when
it
was
hdfs,
we
only
ran
the
hdfs
file
system
component.
We
didn't
actually
do
mapreduce
and
the
Hadoop
things.
On
top
of
that,
that's
honestly
been
a
pleasant
experience
of
the
you
know.
The
the
file
system
comes
along
happy,
as
can
be.
I
was
planning
on
upgrading
to
17
at
some
point
just
haven't
gotten
there.
B
B
Largely
that's
it's
unused
or
underutilized.
I
should
say
not
not
a
lot
of
activity
on
that
project,
and
then
we
also
use
Seth
via
Rook.
We
have
a
new
kubernetes
kubernetes
clusters
that
we've
spun
up
for
various
reasons.
So
one
has
a
bunch
of
pure
nvme
nodes,
burning
Seth,
it's
actually
triply
replicated
I
know
a
lot
of
people
do
double
because
of
the
reliability
of
nvmes.
B
In
theory,
but
that's
that's
another
one
we
have
and
then
because
we've
had
success
with
stuff
I
guess
we're
planning
on
building
a
new,
roughly
five
to
six
petabyte
Seth
cluster.
That
will
largely
be
general
purpose:
storage
for
our
campus
and
our
researchers.
The
intent
is
to
provide
well.
My
intent
is
that
it
is
a
you
know,
the
start
of
a
storage
platform.
So
it's
you
know
we
have
a
chance
to
get
some
brand
new
hardware,
use
that
and
you
know
utilize
it
in
various
ways.
B
The
majority
will
be
as
ffs
again,
but
we
also
have
people
wanted
to
do
some
block
device
things
which
are
back
be
backed
by
it
as
well
as
likely
the
next
Cloud.
So
the
object
store
side,
so
yeah,
I,
guess
I
I'm
here,
just
because
I
thought
this
was
interesting.
We're
a
scientific
research,
Computing
Center,
all
the
stuff
we
do
is
research,
oriented
and-
and
our
success
with
Seth
in
the
past
few
years
is,
has
led
to
that
be
kind
of
coming.
B
Yeah,
yes,
and
no
so
the
CMS
workflows
are,
are
mixed.
The
majority
of
the
access
to
it
is
largely
bulk
data
storage.
We
we
do
have
in
the
past.
When
we
had
hdfs,
we
actually
had
a
mix.
The
the
data
nodes
also
were
the
worker
nodes
of
the
cluster.
So
it
was,
you
know,
by
servers
with
CPU,
good
CPUs
and
then
disks
in
them
too,
as
we
moved
to
Seth
the
the
servers
with
usds
are
separate.
They
don't
run
any
of
the
workflows
CMS.
B
A
B
A
A
B
C
As
far
as
the
detailed
technical
details
go,
I'm
not
going
to
try
and
pretend
I
know
exactly
how
it's
set
up.
Thomas
Bennett
has
all
of
those
details
and
he's
and
and
he's
on,
the
call
and
I'm,
not
sure.
If
you
you
know
him,
but
this
is
this
is
part
of
a
transitioning
process
of
Thomas
is,
is
now
moved
out
of
cereo
and
it's
moving
into
a
private
consultancy
and
hopefully
we'll
still
be
supporting
our
safe
cluster.
A
G
You
think
okay,
I
I,
think
I
could
save
one
word
or
two
words.
You
have
I'm
from
CSC
pieter
and
you
only
have
most
likely
told
you
about
our
clusters
already,
but
just
straight
to
hello,
you
and
they
I-
haven't
been
in
this
meeting
earlier
I'm,
basically
trying
to
take
that
Union,
better
and
other
guys
have
resources
and
capability
to
run.
Our
machines.
G
D
D
So
some
of
some
of
the
tools
they
understand
the
token
principle,
but
majority
majority
of
the
tools
that
our
customers
were
using
are
failing,
and
now
we've
been
developing
method
to
Auto
expire
keys
on
on
on
S3.
So
if
you
are
using
familiar
with
this
fifth
API,
you
can
there
expire
the
keys
quite
easily,
but
with
S3
they
are
like
forever.
If
you
have
a
key
and
secret
there,
people
are
tend
to
put
them
places
and
they
forget
that
they
have
leaked,
for
example,
the
keys.
D
F
F
A
D
So
most
of
them
worked
just
fine,
but
but
not
not
wide
enough
correct
or
different
tools,
so
that
was
that
was
a
good
intention
and
really
a
way.
It
should
done
still.
I
think
that
it
would
should
be
the
way
and
we
found
some
bugs
on
a
runner's
Gateway
internal
code.
Already
with
that,
and
we
have
made
made
a
progress
with
the
Naya
about
fixing
those
token
problems
on
the
back
end
or
the
right
Escape
by
side.
During
that.
D
D
A
D
D
D
H
D
D
F
D
B
D
B
D
A
D
E
A
Where
you
know,
if
we
Kickstart
using
in
place
the
only
destroy
the
OS
drives,
don't
touch
any
of
the
osds
and
then
when
it
comes
up
it
just
it
has
a
process
that
you
figured
out
to
bring
the
osds
up,
discover
them
with
stuff
volume,
bring
them
up,
and
you
know
it
takes
60
Minutes
of
host
or
something
I
think
I
saw
him
if
he
joined
the
call.
So.
F
D
J
F
I
I
would
just
a
quick
recap:
my
name's
Bruno
canning
I'm
working
welcome
Sanger
Institute
in
South
Cambridge.
Here
in
the
United.
We
are
one
of
the
largest
genome
sequencing
facilities
and
genomics
research
facilities,
world
we're
running
a
51
node
Steph
cluster
with
60
drives
storage
node,
it's
procured
about
five
six
years
ago
now.
I
We
have
other
data
storage
facilities
on
site
like
irons,
which
gives
us
metadata
annotation
of
our
data
as
well,
and
we
operate
compute
files
too.
So
our
surf
cluster.
It
was
running
a
bionic,
Ubuntu
18.04
and
we
upgraded
to
vocal,
which
is
20.04.
We
had
to
go
to
focal
because
we're
on
octopus
and
we
wanted
I,
think
I
think
we
are
required.
We
can't
go
straight
to
Quincy,
I.
Think
there
is
this:
changing
the
OSD
code
such
that
we
have
to
go
over
the
Civic
first
and
so
our
upgrade.
I
I
The
other's
degradation
in
the
data
redundancy
one
thing
I
did
notice
is
that
it's
perhaps
related
to
the
the
ceph
orchestrator
problem
observed
by
you
guys
at
CSC.
Is
that
when
it's
time
to
bring
the
osds
back
online,
if
you
set
a
note
up
and
what
it
can
do
is
set
for
them
patiently
bring
all
the
osds
up,
but
they
won't
actually
start
peering.
You
can
bring
up,
you
know
14
times,
16,
osds
and
they're
all
ready
to
go,
and
then
in
one
command.
I
You
are
you
the
no
upper
flag
and
you
don't
get.
You
obviously
get
a
huge
amount
of
peering
happening,
but
you
don't
get
the
state
where
there's
a
long
delay
between
the
first
OSD
coming
online
of
the
upgraded
hosts
and
the
last
one,
because
this
will
this
will
cause
backfilling
and
then,
as
more
rsds,
come
online
Seth
realizes
that?
Oh,
some
of
the
backfilling
it's
already
embarked
on
is
is
actually
not
not
required.
I
F
I
D
I
At
least
I
was
a
little
horrified
when
one
of
my
colleagues
suggested
that
I
could
take
an
entire
domain
off,
but
he
said
well,
look.
We
had
a
power
failure
in
the
data
center
some
time
ago
and
the
cluster
just
kept
on
kept
on
running
kept
on
serving
IO
and
then,
when
the
powers
restored
to
that
failure
domain,
you
know
it's
then
back
catches
up
backfills
and
you
know
it's
Health
occur
again.
So
you
know
what
it
is
scary
to
take
that
many
nodes
off
in
one
go.
B
You
have
a
question
since
talking
about
Crush
map
back
when
I
was
first
starting
with
Seth,
and
we
had
a
mix
of
node
sizes.
Some
were
small
40
terabytes,
some
were
800
terabytes
and
it
turned
out.
I
had
not
not
enough
of
each
type
to
do
what
I
was
expecting
to
do
and
I
had
all
sorts
of
balancing
problems
that
went
through
all
of
that
all
resolved
after
having
a
sufficient
quantity
to
meet
the
you
know
erase
your
coding
profile.
We
had
set,
of
course,
and
everything
seems
fine.
B
So
my
generic
question
is
whether
there
have
been
any
developments
there
or
are
there
any
cases
with
some
of
you
that
have
larger
clusters
and
perhaps
a
more
mixed
environment
where
there
are
still
challenges
as
far
as
keeping
this
utilization
balanced
across
the
large
things,
or
is
that
largely
solved?
And
you
know
happy
easy
times.
A
I
think,
even
with
our
art
festival,
you
know
we
had
six
eight
ten,
twelve,
whatever
14
terabyte
drives
on
at
once.
You
know
24
each
and
ever
since
you
know
they
started
putting
the
effort
into
the
manager
and
the
balancer
and
up
map
back
in
the
day
yeah.
It
was
a
pain,
keeping
things
balanced
across
a
cluster.
But
now
it's
pretty,
in
my
opinion,
it's
pretty
trivial.
A
G
D
D
J
J
G
J
B
One
you
listed
is
actually
the
one
that
I
used
and
it
was
recommended
to
from
people
at
CERN
on
some
of
their
early
days
with
dealing
with
balancing
issues.
But
when
I
had
16
nodes
that
had
40
terabytes
and
three
nodes
that
had
you
know
upwards
of
800
terabytes,
you
know
I
had
to
use
this
tool
in
order
to
get
anything
resembling
equal
space
utilization
and
it
it
got
out
of
hand
quickly.
I
use
this
Tool
too
many
times.
B
Essentially
in
my
I
had
you
know
revamped
smile
log,
and
it
was
like
wait.
How
do
I
undo?
All
of
this
now
took
a
while
to
actually
learn
how
all
the
things
were
pieced
together,
but
it
certainly
works,
and
you
know,
has
its
use
cases.
But
ever
since
we
had
we,
we
have
enough
nodes.
Now
that
are
eight
plus
three
Erasure
encoding.
You
know
that
the
balancer
can
figure
out
and
do
the
right
thing
with
the
ability
to
put
data
in
the
right
places.
D
E
B
Another
question
so,
when
I
introduced
the
Seth
clusters,
we
had
I
neglected
to
mention
the
really
old
original
ceph
cluster,
which
backs
a
openstack
instance,
which
is
still
running
Joule
in
the
case
of
you
know,
don't
touch
it,
don't
talk
about
it.
It
has
anyone
gone
the
whole
Jewel
all
the
way
up
to
latest
things
with
some
of
their
clusters
were
there
any
major
gotchas
between
versions
that
I
should
just
it's
not
worth
trying
her
plan
is
to
replace
it.
You
know
new
Greenfield
solution,
but
I'm
just
curious.
B
A
I
said:
do
it
my
cluster,
my
big
one
started
on
Hammer
I,
think
and
I've
ran
octopus.
Now
I've
hit
every
version.
I
prefer
to
just
hit
I
know
you
can
skip
some
versions
in
some
cases,
but
if
you
just
read
the
yeah,
if
you
read
the
release
notes
and
you
just
do
all
the
steps
you
know,
I've
never
had
a
problem
coming
all
the
way
from
Hammer
up
to
octopus.
B
All
right,
you
know,
I
had
gone
back
and
looked
at
some
of
the
you
know,
older
release,
notes
But,
as
time
has
gone
on,
finding
the
finding
documentation
that
is
correct
or
looks
correct
and
official
for
the
old
versions
has
become
more
challenging.
So
it's
like
this
really
applicable
still.
Is
it
a
is
something
yeah.
A
B
B
I
K
I
We
had
some
issues
with
the
first
deployment,
so
we
tore
our
cluster
down
and
we
built
it
completely
from
scratch.
I
think
that
was
the
Dual
release
and
they
certainly
got
as
far
as
luminous
I'm
a
bit
out
of
touch
with
them,
but
I
should
imagine,
I
mean
they're
running
it
as
a
production
system,
so
I'd.
Imagine
if
they're
not
on
the
latest
they'll
be
only
one
main
releases
behind
yeah,
Tom
Byrne
is
I.
Think
he's
principal
operator
of
Lancaster
he'll
certainly
know
once
more.
G
B
D
Well,
we
haven't
upgraded
the
nine
version
of
a
railover
events
yet,
but
there
is
a
one
big
thing
which
is
difference
between
teaming
and
bonding.
So
in
case
you
are
using
a
teaming
on
a
Centos
Rocky.
Whatever
eight
person,
the
nine
version
is
in
combat
with
income
of
it
all
with
that,
so
so
the
now
the
sit
they
are
shifting
back
to
bonding
bonding
will
break
your
teaming.
If
you
have
done
teaming
on
on
a.
B
D
D
F
A
K
F
K
F
K
K
K
K
F
F
F
A
I
Although
the
OSD
is
it's
running,
just
fine,
it
gets
marked
down
by
its
peers
and
then
I
think.
As
soon
as
one
of
our
osts
goes
down,
we're
actually
rebalancing
back
feeling.
Sorry,
then
the
osc
comes
back
and
says:
oh
yeah,
no
problem
I'm
still
here
and
then
yeah,
but
after
a
while,
what
can
happen?
Is
you
get
an
avalanche
failure?
You
know
eventually
the
OSD
will
be
marked
down
properly
and
you
get
an
avalanche.
Failure
of
the
next
OST
becomes
a
problem
in
the
next
one.