►
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
Ceph Storage at CERN
Speaker(s):
Dan van der Ster, Pablo Llopis Sanmillan
CERN
Abstract:
Ceph and its Reliable Autonomic Distributed Object Store (RADOS) offers a scale out storage solution for block storage (RBD), object storage (S3 and SWIFT) and filesystems (CephFS). The key technologies enabling Ceph include CRUSH, a mechanism for defining and implementing failure domains, and the mature Object Storage Daemons (OSDs), which provide a reliable storage backend via replication or erasure coding. CERN has employed Ceph for its on-prem cloud infrastructures since 2013. As of 2021, its storage group operates more than ten clusters totaling over 50 petabytes for cloud, Kubernetes, and HPC use-cases. This talk will introduce Ceph and its key concepts, and describe how CERN uses Ceph in practice. It will include recent highlights related to high-throughput particle physics data taking and SLURM storage optimization.
Bio(s):
Dan manages the Ceph storage at CERN in Geneva, Switzerland. He has participated actively in its community since 2013 and was one of the first to demonstrate its scalability up to multi-10s of petabytes. Dan is a regular speaker at Open Infrastructure events, previously acted as Academic Liaison to the original Ceph Advisory Board, and has a similar role in the current Ceph Board. Dan earned a PhD in Distributed Systems at the University of Victoria, Canada in 2008.
Pablo is a computer engineer at CERN, where he manages the IT department’s HPC service. He provides HPC support to both engineers of the Accelerator Technology Sector and to theoretical physicists. Pablo works on improving the performance of their HPC workloads, and on other projects such as the automation of operational tasks of the infrastructure. He holds a Ph.D. in computer science from University Carlos III of Madrid. In the past he has also collaborated with Argonne National Laboratory and IBM Research Zurich on HPC and cloud-related topics. His main areas of interest include high performance computing, storage systems, power efficiency, and distributed systems.Zoom:
Host of Seminar:
Alberto Chiusole
Storage and I/O Software Engineer, Data and Analytics Services
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
C
Thanks
thanks
alberto
thanks
for
the
invitation,
so
yeah
we're
we're
from
the
cern
I.t
department.
We
are
pablo
and
I
are
in
different
groups,
I'm
in
the
storage
group
and
pablo's
in
the
computing
group,
and
we
work
together
on
different
types
of
storage.
So
it's
pleasure
to
be
here
to
speak,
just
get
started
so
quickly
for
those
that
maybe
have
are
not
familiar
with
where,
with
with
cern.
C
This
is
cern
it's
in
it's
located
in
geneva,
switzerland,
like
kind
of
halfway
across
the
border
between
france
and
switzerland,
it's
27,
kilometer,
circumference,
underground,
ring
100
meters
under
the
ground,
colliding
particles
in
opposite
directions
to
discover
the
fundamental
fundamental
nature
of
matter
around
the
around
the
the
corners.
There
are
these
detectors
like
cms,
lhcb
atlas
alice?
C
These
are
large
cameras,
basically
detectors
taking
pictures
of
of
collisions.
This
is
what
the
tunnel
looks
like
underground,
superconducting
magnets
accelerate
and
bend
the
the
proton
beams
to
collide.
This
is
what
the
detectors
look
like.
So
there's
like
there's
a
scientist
there
there's
a
physicist
standing,
but
you
can
see
you
can
understand
the
scale
of
it,
and
this
is
what
it's
all
for
it's
for.
Taking
pictures
of
collisions,
this
is
a
picture
of
a
higgs
boson
candidate,
so
this
is
one
of
the
candidates.
C
That's
that's
responsible
for
for
for
mass
okay,
so
this
is
a.
This
was
a
nobel
prize
winning
picture
basically
now
to
seth.
So
this
outline
of
this
talk
so
I'll
cover
the
first
two
parts,
which
is
what
is
ceph
and
how
does
it
work
actually
we'll
go
into
a
bit
of
detail
there,
because
it's
quite
interesting,
I
think,
to
people
we'll
explain
the
stuff
use
cases
at
cern
and
then
we'll
explain
about
7s
for
hbc.
C
So
what
is
ceph
and
now
I'll
just
say
that
these
slides
are
credit
mostly
to
sage,
while
the
inventor
of
seth
he's
got
a
nice
deck
that
that
we
borrowed
liberally
from
so
the
buzzwords
of
what
is
seph.
So
you
might
hear
like
software
defined
storage,
unified
storage
system,
scalable,
future
of
storage,
linux,
of
storage,
okay,
but
what's
the
substance
behind
this
is
that
it
sets
an
open
source
software.
C
C
When
you
see
when
you
see
how
performs
later
seth's
also
scalable,
so
it's
elastic
storage
infrastructure,
you
can
grow
or
shrink.
You
can
add
the
add
hardware,
while
it's
while
it's
running
and
under
load,
you
can
scale
up
with
bigger
and
faster
hardware
or
out
with
capacity,
and
you
can
also
federate
multiple
clusters
across
sites
with
different
technologies
which
are
higher
level
which
I'll
get
into
a
bit
in
the
end.
In
one
picture,
this
is
the
unified
storage
system,
which
is
ceph.
C
It
has
a
object,
storage
object,
stores,
layer
named
rados
at
the
bottom,
which
is
accessed
by
a
library
called
librados
and
then
on
top
there's
three
main
use
cases,
object,
storage,
compatible
with
s3
and
swift
via
something
called
the
rados
gateway
block
storage
via
a
library
and
interfaces
known
as
rbd,
and
a
distributed
network
file
system
known
as
ffs,
so
ceph
is
also
a
big
community.
Seth
is
a
has
a
foundation
part
of
the
linux
foundation.
C
With
all
of
these
members
in
the
bottom
right,
you
can
see
many
like
labs
and
universities
and
then
there's
like
tons
of
companies,
few
tens
of
companies
involved
supporting
staff
and
supporting
staff
developers
and
helping
to
organize
conferences
and
meetups
and
all
of
those
things
that
go
into
open
source
communities.
These
days
I'll
explain
now
the
technologies
inside
ceph,
so
first
rados
ratos,
is
the
reliable
autonomic
distributed
object
store.
So
it
gives
that
common
object,
storage
layer
underneath
all
of
the
staff-based
services.
C
It's
it
gives
low-level
object,
storage,
it's
reliable
and
highly
available,
it's
scalable
and
it
manages
all
replication
and
erasure
coding
data
placement,
rebalancing
repairs.
All
of
this.
It's
strongly
consistent,
so
cp,
and
it
it's
having
this
reliable
air
lets.
You
simplify
all
the
higher
level
layers.
C
The
components
of
ceph
are
something
called
the
monitor,
which
is
essential
authority.
It's
it
serves
small
cluster
maps
to
coordinate
the
the
all
of
the
demons
involved,
something
called
the
manager
which
is
for
offloading
non-critical,
non-like
service,
critical,
but
still
useful
things
like
metrics
or
plugable
components
and
then
osd's
the
object,
storage
daemon.
C
So
how
does
stuff
look
so
first
I'll
show
I'll
remind
like
how
does
legacy
client
server
architecture
work?
You
have
an
application
at
the
top
that
talks
to
a
server
that
then
like
uses
back
ends
to
to
serve
your
requests.
If
the
if
the
server
is
down
okay,
you
have
things
like
you
can
have
like
ips
failing
over,
and
then
you
have
a
backup
server
that
then
at.
C
Is
recovering
and
and
and
keeping
the
the
application
highly
available?
This
is
not
how
ceph
works.
Ceph
has
application
interfacing
to
the
storage
through
something
called
liberatos
and
then
libreos
is
handling
all
of
the
intelligence
needed
to
access
an
entire
cluster
with
hundreds
or
thousands
of
demons
and
moving
parts
things
going
up
and
down
the
live.
The
library
is
still
simple,
but
what's
underneath
is
much
more
complex?
C
How
is
data
placed
in
us
in
a
rato's
cluster,
so
you
have
an
application
that
wants
to
store
an
object.
How
do
you?
How
do
you
know
where,
in
the
cluster
to
send
it
in
a
let's
say,
traditional
or
old-school,
clustered
storage
system?
It
would
ask
some
central
authority
where
it
should
store
it
and
that
central
authority,
so
it
could
look
up,
okay,
but
that's
slow,
and
then
it
would
also
be
hard
to
struggle
to
scale
to
trillions
of
objects.
C
So,
for
example,
it
can
get
a
state,
a
cluster
map
which
has
the
layout,
including
the
number
of
osds
and
then,
for
example,
if
the
topology
changes
in
real
time,
like
a
disk,
failing
or
host
failing,
the
the
application
gets
a
new
map
and
then
recalculates
a
new
place
to
send
the
object,
and
similarly
all
of
the
osd's
and
all
of
the
components
in
the
cluster
get
the
same
map.
At
the
same
time,
they
all
know
that
the
data
needs
to
move
to
a
different
place
in
case
of
a
failure.
C
That's
the
general
idea.
So
how
is
this
done
into
more
detail?
So
you
have
some
discs
or
some
files
or
some
films
or
some
pictures.
You
want
to
store
them.
Okay,
so
these
get.
Let's
say
one
film
file
gets
cut
into
several
objects:
those
objects.
You
want
to
logically
store
them
in
a
pool.
A
pool
is
a
thing
that
can
store
billions
or
trillions
of
objects,
okay,
petabytes
of
data
in
one
pool,
but
that's
a
logical
construct.
C
What
we
do
in
stuff
is
we
carve
a
single
pool
into
several
things
called
placement
groups,
so
placement
groups
are
basically
you
take
all
of
the
objects.
You
take
an
object
in
a
pool
and
you
do
a
hash
of
the
object
name
divided
by
or
modulo
the
number
of
placement
groups
in
that
pool,
and
that
gives
you
the
placement
group
id
where
you
should
store
it.
Okay
and
then
there's
some
magic
that
happens
and
which
maps
a
placement
group
to
an
osd
which
is
a
disk,
and
you
see
like,
for
example,
displacement
group.
C
What
1.0
is
stored
on
the
first
disk,
the
third
disk
and
the
fourth
disk?
Okay?
Now
how
do
you
actually
place
those
placement
groups?
That's
with
this
magic
component
and
stuff
called
crush.
Okay,
it's
a
pseudo,
random
placement
algorithm!
It's
a
it's
similar
to
consistent
hashing,
but
not
not
exactly.
The
inputs
are
the
topology
of
the
cluster.
So,
like
the
orc
hierarchy,
things
like
osds
are
in
a
host
they're
in
a
rack
they're
in
a
room.
C
Okay
and
then
the
administrator
can
say
they
can
define
rules
such
as
I
want
three
replicas
in
different
racks
only
on
ssds
or
I
want
six
plus
two
erasure
coding,
two
per
rack
only
on
hdds,
okay
and
this
crush
algorithm
is
outputting
a
list
of
or
an
ordered
list
of
osds.
For
that
placement
group,
it's
stable
if
the
topology
changes
like,
if
you
add
hosts
or
if
you
remove
disks,
fail,
there's
a
there's,
a
minimal
amount
of
data
movement
after
after
such
a
topology
change,
and
it
also
supports
varying
device
sizes.
C
C
Each
rada's
pool
must
be
durable
and
each
pg
must
be
durable
to
make
the
pools
durable.
So
seph
does
this
with
two
ways:
replication,
so
identical
copies
of
each
pg.
Here
this
is
the
picture
on
the
left.
If
you
have
two
objects,
data
and
my
object,
you
have
three
copies
of
each
okay.
You
just
store
them
exactly.
This
is
a
200
overhead,
but
it's
fast,
but
you
can
also
do
erasure
coding.
So
this
is
like
raid
over
the
network.
C
Each
object
is
split
into
chunks
and
then
you
compute
parity
or
other
other
k,
type,
shards
or
m
for
parity
or
redundancy.
Usually
we
have
something
like
50
overhead.
So
like
four
data,
stripes,
two
parity,
stripes
or
some
people
do
eight
and
three:
it's
it's
it's
it's.
You
can
use
arbitrary
k
and
m
and
stuff.
C
Of
course
they
have
implications
on
performance
and
availability,
but
in
general
like
to
conclude
on
rados,
it's
like
something
that
lets
an
operator
define
what
they
want
like
three
x,
replicated,
ssd
pools
or
erase
recording
with
eight
plus
three
on
hdds
or
three
x,
replicated
hdd
pools
and
then
magic
happens
and
everything
is
stored
down.
On
the
devices
and
that's
implementing
like
giving
you
a
platform
for
high-level
services,
now
I
go
into
those
services.
C
This
is
a
concept
that
has
users,
users,
create
buckets
and
users
store
objects
in
those
buckets,
and
then
there's
data
and
permissions.
They
have
access
keys
to
to
to
access
these
and
there's
lots
of
different
apis
that
can
be
used
to
delegate
access,
and
but
it's
important
to
keep
in
mind
that,
like
rados
objects,
are
not
the
same
as
the
rgw
objects
like
an
s3
object
can
be
very
big,
it
can
be
one
terabyte
and
that
gets
split
into
many
smaller,
smaller
radius
objects.
C
How
it
looks
like
inside
is
that,
for
example,
you
want
to
do
a
put
you
put
to
this
gateway,
which
itself
speaks
librettos,
so
it
it
has
some
pools
that
know
about
users
and
buckets
okay,
that's
step
one.
It
then
knows:
okay,
it
has
a
different
pool
for
bucket
indices.
These
are
like
lists
of
the
objects
in
a
bucket
okay,
and
then
it
has
data
pool
to
actually
store
the
objects
or
to
read
the
objects
and
then
also
update
the
index.
C
You
can
have
multiple
zones
in
a
cluster
or
they
can
be
in
separate
clusters,
and
you
can
configure
stuff
to
share
the
user
and
bucket
info
between
the
two
zones
and
similarly
so
these
this
could
be.
For
example,
you
can
have
zones
with
replication
or
expensive
fast,
and
you
can
have
cheap
ratio
coded
with
different
with
different
performance
qualities.
C
Similarly,
you
can
have
zone
groups
which
is
a
higher
level
concept
that
ensures
replication
between
zone
groups.
So,
for
example,
you
want
to
create
a
bucket,
you
want
to
make
sure
it's
always
replicated
between
two
data
centers.
You
can
have
this
all
part
of
one
realm,
so
it's
it's
like
it's
like
a
very
it's
very
flexible
federation
and
geo-replication
service.
All
built,
there's
lots
of
other
features.
C
I
won't
go
into
all
of
them
in
detail,
but
I'll
say
that,
like
our
users
use
the
lifecycle
management
a
lot,
so
that's
like
they
can
define
policies
for
cleaning
up
objects
like
expiring
them
after
a
certain
amount
of
time,
and
then
also
like
interesting,
is
there's
a
pub
sub
event
system.
So
you
can
set
up
kind
of
serverless
processes
based
on
sending
objects
to
buckets
getting
and
getting
getting.
Actions
triggered
afterwards
now
go
to
block
storage,
so
rbd
is
about
creating
virtual
block
devices,
so
virtual
disks
stored
in
rados.
C
It's
like
it's
made
for
things
like
kvm
qmu
running
in
openstack,
for
example,
or
linux,
hosts
that
you
want
to
attach
a
virtual
disk
to.
In
both
cases,
you
like
need
to
format
that
with
a
file
system,
normally
it's
analogous
to
ebs
in
aws
yeah.
It's
well
integrated
with
things
like
openstack
and
kubernetes
and
and
other
cloud
platforms.
C
Basically,
the
idea
is
it's
taking
one
large
disk
image,
maybe
10
terabytes
carving
it
into
tiny
pieces
again
and
sending
that
into
your
router's
quest
too.
It
supports
some
really
useful.
High
level
features
like
these.
Those
disk
images
can
be
snapshotted.
You
can
create
read-only
snapshots
point
in
time
consistent.
C
C
And
then
also
you
can
mirror
block
devices
between
different
pools.
So
when
you're
writing
to
when
you're
writing
data
to
like
cluster
a
you
can
actually
enable
like
a
kind
of
journaling,
it's
kind
of
like
a
redo
log,
which
is
which
is
stored
in
rados
as
well
and
then,
as
a
new
demon
called
the
rbd
mirror
demon,
which
is
highly
available,
also
replays
the
rights
to
your
images
on
cluster,
a
in
a
different
cluster
cluster
b,
so
that
can
be
for
for
disaster
recovery
purposes.
C
C
Lots
of
other
lots
of
other
goodies
there
now
come
to
steffs
cfs
is
distributed
network
file
system,
so
this
is
like
nfs
it's.
What
you
expect
important
thing
about
cefs
is
that
it's
really
like
strongly
consistent
and
coherent
across
many
nodes,
so
the
design
goal
is
to
be
like
the
consistency
of
a
local
disk
for
all
the
users
across
the
network.
C
Ceph
lets
you
scale
the
metadata
and
data
independently,
so
it's
a
separate
data
and
metadata
pool
and
there
are
clients-
it's
showing
here
case,
ffs,
there's
clients
built
into
linux
kernel
and
there's
also
a
fuse
clients
yeah
to
implement
ffs.
There's
a
new
daemon
needed
the
mds,
the
metadata
server.
C
Each
cluster
has
one
to
ten
of
these
oops.
I
should
go
back,
let's
say
10
plus
standbys,
and
then
they
can
fail
over.
So
we
have
this
happening.
Sometimes
network
outage,
you
have
the
mds
have
to
fail
over
to
another
one-
and
this
is
maybe
a
few
tens
of
seconds
depending
on
the
size
of
the
cluster
and
the
activity,
and
you
have-
and
you
have
a
the
cluster
back
online
again.
C
Here's
a
picture
of
the
metadata,
basically
like
a
kernel
client
will
be
looking
in
the
metadata
server.
The
directories
are
all
stored
in
a
metadata
pool.
This
is
also
journaled
for
this,
for
this
kind
of
like
easy
recovery
from
from
crashes,
and
then
data
objects
are
actually
stored
in
a
data
pool
which
can
be
replicated
or
ratio
coded
depending
on
your
requirements.
C
There's
a
scalable
name
space.
So
how
ceph
does
this?
Is
you
can
just
basically
each
if
you
have
several
mds's
so
here
in
this
picture
we
have
five
mds's.
Each
mds
is
responsible
for
a
different
part
of
the
tree
in
the
name
space,
the
directory
trees.
This
is
the
root,
and
then
this
is
like
slash
home
or
whatever.
C
The
by
default.
Stuff
is
actually
looking
at
the
the
temperature
of
different
subdirectories
and
like
rebalancing
in
real
time,
the
the
subdirectories
out
to
these
different
mdss
and
then
there's
different
heuristics
for
tuning
this,
and
it's
also
possible
to
if
you
know
your
workload
ahead
of
time,
to
pin
to
specific
mdss
so
like
in
us.
At
cern
we
use
on
our
largest
ffs.
We
have
four
active
mdss
and
we
pin
exactly
the
where
we
want
things
to
be
to
be
to
be
balanced.
C
Now
sevenvest
supports
snapshots.
Any
directory
can
have
a
snapshot.
You
just
cd
to
a
hidden
directory
called
dot
snap
and
you
and
you
mk,
dura
there
and
then
now
you
have
a
snapshot
of
the
parent.
You
can
also
remove
that
directory
with
arm
der
it's
efficient
fast,
but
then
it's
it's
the
delete.
You
have
to
pay
the
cost
of
the
deletion
time.
Basically
it
can.
It
does
a
snap
trimming
thing
to
the
users.
C
There's
also
some
neat
features
like
recursive,
accounting,
so
from
any
directory.
You
can
see
the
recursive
stats
like
the
number
of
bytes
stored
or
the
latest
change
time
or
the
number
of
entries
or
files
or
sub
directories
all
beneath
all
the
way.
So
you
can
use
this
to
very
quickly
like
discover
if,
like
if
subtrees
have
changed
like
without
iterating
through
all
the
files
and
going
through
the
whole
directory.
C
There's
lots
of
other
features,
of
course,
which
I
won't
go
into
in
detail.
You'll,
probably
hear
about
lazy
io
in
a
few
slides
from
pablo,
but
otherwise
yeah.
The
kernel
client
is
extremely
high
performance
like
compared
to
fuse
the
kernel.
Client
works
works
very
well
and
most
of
our
we
have
something
like
maybe
2.
000
users
like
most
are
using
kernel
client
by
now.
C
C
So
the
certain
computing
infrastructure
is
not
we're.
Not
a
supercomputer
center,
we're
more
of
a
high
throughput
scientific
computing
center.
We
have
batch
system
with
something
like
250
cpu
cores
with
hd
condor.
We
have
a
storage
system
called
eos
written
at
cern,
managing
around
500
petabytes
of
raw
storage,
and
then
the
cern
tape
archive
is
a
similarly
written,
as
cern
for
long-term
icar
archival
over
half
an
exabyte
of
tape
managed
by
cta,
but
we
also
have
like
the
traditional
it
infrastructure
needs
like.
C
C
We
have
small
mpi
clusters,
we
have
generally
an
open
infrastructure
of
things
which
are
like,
but
all
those
like
logo,
logo
soup.
Here
on
the
slide,
showing
all
the
kinds
of
technologies
that
we
use
and
they're
all.
Basically
they
can
be.
If
you
need
to
run
this
kind
of
stuff,
you
can
run
it
on
steph.
That's
why
we
that's
kind
of
why
we
have
seth
so
currently
in
production,
I'll
just
go
through
briefly
what
we
have
for
block
storage.
C
C
These
are
all
integrated
to
openstack
under
multiple
qos
types,
so
a
qs
type
is
basically
just
an
iops
throttle
and
also
availability
zones.
So
we
by
default
we
just
kind
of
spread
the
the
volumes
requests
across
different
clusters,
but
also
users
can
say
they
want
it
here.
They
want
it
here
for
for
different
availability
reasons.
C
But
then
we
have
several
other,
maybe
three
or
four
other
smaller
s
clusters
on
all
flash
targeted
for
specific
use.
Cases
like
some
hyper
converged
tests
for
a
databases
group
for
our
enterprise
group,
where
they
wanted
a
dedicated
set
of
s
and
also
for
hpc
in
general.
Our
experience
that
ceph
is
like
it
lives
up
to
the
hype
kind
of
it's
robust
and
performant.
C
We've
used
it
since
2013
data.
Aside
from
like
a
few
strange
incidents
and
amazing
bugs
it's
it's
quite
it's
quite
good.
Basically,
we've
had
like
100
uptime
aside
from
those
incidents,
never
any
data
loss
after
in
infrastructure
outages.
The
failure
recovery
is
basically
transparent.
It
just
comes
back
online,
basically
by
itself.
Okay,
sometimes
it
can
become
complicated,
but
it's
it's
usually
quite
okay
and
then
also
hardware
replacement
is
like
super
easy.
C
This
is
the
killer
feature
of
ceph
we've
gone
through
three
procurement
cycles,
so
we
keep
the
stairs
storage
and
clusters
like
living
forever.
They're
like
living
organisms
that
live
forever.
We
just
replace
the
hardware
we
never
transfer
bytes
around
ceph.
Does
it
all?
We
never
ask
users
to
transfer
bytes
around
ceph.
Does
it
for
them.
C
So
this
is
all
mostly
exposed
via
our
openstack
cloud.
Also
since
2013,
it's
hosting
90
of
the
computing
resources,
you
can
see
there
that
it's
like
we
have
over
290
000
cores
in
here
in
this
picture.
It
shows
here
we
go
like
images,
so
these
are
like
system
disks
stored
on
ceph
volumes.
These
are
these
are
rbd
images
on
ceph,
six,
thousands
close
to
seven
thousand
volumes
and
then
file
shares
these
are
cefs's,
so
these
are
users
that
create
their
own
stuff
by
passive
volume.
C
I
wanted
to
highlight
one
project
interesting
project
before
handing
it
off
to
pablo
one
more,
which
is
that,
like
eos,
is
our
large-scale
physics,
data
storage,
but
being
working
in
the
same
group.
We're
always
interested
to
like
see
if
there's
kind
of
synergies
between
cef
team
and
eos
team,
so
eos
was
developed
at
cern
for
physics
and
regular
system
managing
like
350
petabytes
at
the
moment,
close
to
500
petabytes.
C
If
you
account
for
all
the
newest
deliveries,
so
we
asked
ourselves
like,
is
it
feasible
and
even
useful
to
layer
eos
on
top
of
a
cef
or
7fs
backend?
This
would
give
us
like
the
best
of
both
worlds,
feature
like
eos,
which
has
all
the
features
that
the
scientists
need,
that
the
img
physicists
need,
plus
the
flexible
object,
storage
underneath
provided
by
ceph.
C
So
it
turns
out
this
can
be
done
and
we
had
a
we
did.
We
did
a
paper
about
it,
which
you
can
click
the
link
later
to
read
about
that,
but
basically
because
eos
is
built
upon
something
called
x-ray
d
and
it's
just
so-
it's
replicating
and
doing
race,
recording
itself
and
storing
metadata
and
something
called
quark
tv,
but
actually
underneath
it
stores
the
files
in
xfs
in
local
xfs.
So
it
uses
a
simple
like
inode
hash
naming
convention
and
why
don't
we
just
instead
of
using
xfs
use
ffs?
C
E
E
Well,
there's
several
reasons,
but
one
of
the
main
reasons
is
that
sooner
we're
already
running
this
whole
zoo
of
file
systems
right
we
have
eos,
we
have
cbmfs,
we
have
afs
and
we
have
cfs
and
we
actually
wanted
to
give
it
a
try
and
see
if
it
would
work,
because
we
didn't
have
a
desire
to
introduce
yet
another
file
system.
We
didn't
have
any
experience
with
lesser
gps
gpfs,
so
we
decided
to
give
a
test.
E
A
try,
so
also
part
of
the
reason
is
that
the
way
we
procure
hpc
clusters
at
cern
is
not
the
usual
way
of
procuring
it
from
some
vendor
and
getting
like
the
whole
hpc
cluster
delivered
installed
and
running.
What
we
do
is
that
we
place
what
the
certain
procurement
team
does
is
that
we
we
place
a
very
large
order
of
servers,
and
these
these
servers
will
be
destined
for
different
purposes.
E
For
instance,
for
the
htc
as
in
high
throughput
computing
cluster
or
for
storage,
and
then
this
will
be
a
very
large
order
and
a
subset
of
those
machines
will
then
be
intended
to
become
an
hpc
cluster
and
then
we
just
add
open
source
software
and
we
buy
a
low
latency
interconnect
such
as
infiniband
trying
to
connect
those
clusters
and
voila.
We
have
the
an
hpc
cluster.
This
is
not
a.
E
These
are
not
huge
hpc
clusters,
but
still
so
we
validated
cfs
as
a
multipurpose
file
system,
including
use
as
a
scratch
space
file
system
for
for
hvc
workloads,
and
actually
it's
been
working
quite
well.
We
have
accumulated
over
five
years
of
experience,
running
ffs
on
certain
it's
hpc
service.
In
production
we
use
slurm
and
together
we'll
set
yeah.
That's
fine,
you
can.
You
can
skip
thanks,
so
this
is
a
logical
overview
of
how
we
did
it
at
the
beginning.
So
I'm.
E
So
in
the
beginning
we
had,
as
you
can
see,
on
the
left,
the
hpc
cluster,
which
everything
everything
almost
everything
in
our
infrastructure
is
provisioned
in
an
openstack
environment,
so
even
the
htc
worker
nodes,
they're
bare
metal,
but
they're
provisioned
by
by
openstack
and
then
on
some
other
end
of
the
data
center.
We
have
this
this
cfs
storage
cluster
and
through
the
openstack
apis.
We
would
provide
a
scratch
space
that
the
worker
nodes
would
mount
and
we
would
access-
and
this
worked
well,
but
it's
next
time
please.
E
So
one
of
the
one
of
the
issues
is
that
this
was
the
this
ffs
cluster,
that
the
worker
nodes,
which
are
the
ceph
clients,
were
mounting
to
access
the
scratch
space.
This
was
in
a
shared
ffs
cluster.
When
I
say
shared,
I
mean
this
was
shared
with
other
it
services.
It
was
not
dedicated
for
for
hpc
as
a
result.
In
occasions,
sometimes
we
experience
contention.
E
We
would
basically
affect
each
other
another
issue,
probably
the
the
biggest
issue,
was
the
fact
that
the
cluster
was
not
very
close
in
the
network,
so
it
was
far
in
the
data
center
and
therefore
any
network
issue.
It
would
be
far
more
likely
that
we
would
experience
issues
when
there
was
some
kind
of
network
issue.
There
was,
of
course,
much
greater.
I
o
latency
as
a
result
of
this,
so
this
was
this
worked,
but
we
had
we
faced
those
issues.
E
Another
thing
is
that
we
relied
on
cepheus
mounts,
so
this
is
fuse's
file
system
in
user
space.
As
a
result,
well
fuse
is
known
for
being
not
super
performant
and
the
sephus
implementation.
In
addition,
well,
I'm
not
sure
if
this
was
the
use
implementation
of
fuse
itself,
but
from
time
to
time
experience
especially
with
when
facing
network
issues
we
were,
we
were
facing
stack
mounts,
stale
data
and
things
like
that
that
we
had
to
fight,
but
then
later
on,
we
evolved
into
next
step
next
slide.
Please
yeah!
E
We
transitioned
all
of
these
two
things.
So
first
we
transitioned
to
a
dedicated
ffs
cluster
and-
and
this-
and
this
proved
of
course,
as
expected,
this
provided
much
greater
performance
than
resiliency
and
also
that
the
fuse
mounts
as
long
as
soon
as
the
kernel
code
was
ready.
We
transitioned
to
kernel
mounts,
and
then
we
experienced
much
greater
performance,
also
much
greater
stability
and
improved
resiliency
for
network
failures,
and
actually
we
haven't
experienced
any
any
issues
with
stack,
mounts
or
stale
data.
E
E
And
finally,
when
I
say
we,
we
have
a
dedicated
cluster.
Actually,
what
we
did
was
that
we
didn't
procure
a
new
storage
cluster.
What
we
did
was
that
we
used
one
of
our
hpc
partitions
and
we
transformed
it
into
a
hyper-converged
cluster,
meaning
that
we
run
the
sef
osd
daemons
on
the
actual
compute
nodes.
So
the
compute
nodes
in
the
cluster
are
both
the
serving
as
the
storage
back-end
on
the
compute
nodes,
and
this
provided
many
benefits
next
slide.
Please.
E
E
We
also
tuned
a
replica
count,
which
obviously
also
has
an
impact
on
write
latency.
So
this
is
something
to
take
into
account
if
you're
using
it
for
a
scratch
space.
Maybe
you
don't
you
do
not
need
you
know
three
replicas
or
more.
If
you
lower
them,
you
will.
You
will
improve
that
in
our
case,
even
though
it
is
a
scratch
space,
all
the
slurm
services
are
also
relying
a
bunch
of
things
that
we've
built
on
top
also
are
relying
on
the
set
of
fast
mounts.
So
we
we
do
have.
E
E
So
if
you
have
more
than
one
nds
as
dan
already
explained,
all
the
the
hot
directories
will
be
migrated
across
mdss,
so
all
of
that
will
work
work.
Well.
However,
we
found
that
if
you
know
that
a
work
there's
going
to
be
a
very
intensive
workload
and
it's
going
to
happen
in
one
sub
directory,
you
can
pin
that
subdirectory
to
a
single
mds
and
usually
a
single
mds
can
handle.
E
E
Wants
to
be
a
very
posix,
it
wants
to
be
posix
basically.
So
if,
if
you
have
you
want
to
maintain
this,
this
posix
consistency
model
and
all
the
coherency.
So
if
you
have
a
single
shared
file
for
doing
collective
io,
which
is
common
in
certain
hvc
workloads,
maintaining
this
posix
consistency
means
that
seth
was
keeping
locks
locks
on
the
file,
so
lazy-
I
oh,
can
you?
Can
you
move
to
the
next
slide?
Please
lazy.
I
o,
when
you
activate
lazy,
io
and
ceph.
E
It
refers
to
a
mode
in
which
this,
these
posix
semantics
and
consistency
and
coherency
restrictions
are
dropped,
they're,
relaxed
and
then
basically,
this
allows
for
log-free
parallel
rights,
and
this
is
great
for,
if
you're
doing,
shirt,
file,
collective
io,
the
coherency,
of
course,
is
delegated
to
the
application.
So
if
you
write
overlapping
bits
of
the
same
file,
then
it's
up
to
you,
but
since
hbc
applications
usually
take
take
this
into
account.
E
E
E
So
now
I
want
to
go
into
the
some
of
the
limitations
that
we
found
with
with
this
approach,
especially
referring
to
to
hyper-converge.
This
is
not
just
ffs.
These
are
limitations
that
we
that
we
found
about
this
hyper-converged
architecture
and
then
also
introduce
some
of
the
solutions
that
we're
working
on.
E
It
has
excellent
scalability,
but
it
is
very
sensitive
to
this
this
kind
of
os
noise,
so
we
even
had
to
disable
any
kind
of
monitoring
that
we
were
running
even
lightweight
monitoring
tools,
because
we
could
see
big
performance
drops
when
running
open
qcd
on
our
worker
nodes
and
in
the
case
of
hyper
converge.
If
there's
a
very
high
intensive
application,
we
measured
the
impact
to
be
about
20
performance
drop,
which
is
obviously
not
non-negligible.
E
So
one
of
the
solutions
that
we're
that
we're
working
on
is
to
switch
switch
to
hyper
converge
burst
buffers
so
under
under
this
model.
What
would
occur
is
that
when,
when
a
user
runs
a
job
on
a
set
of
nodes,
only
those
set
of
nodes
would
be
in
the
same
set
of
nodes
that
are
running.
The
workload
would
be
involved
in
the
I
o,
so
there
will
be
no
interference
between
users.
E
E
So,
of
course
a
very
common
scenario
is
there's
a
security
vulnerability
in
the
kernel,
and
you
need
to
upgrade
the
kernel
and
then
read
the
notes.
So
what
we
usually
do
is
that
we
don't
shut
down
the
cluster.
We
do
this
transparently,
so
we
we
drain
machines
and
then
we
reboot
them
as
they
get
drained
and
the
way
this
usually
works
in
the
batch
system.
Is
you
if
you're
running
a
normal
bash
system?
E
They
have
the
notion
of
draining
a
node,
so
you
will
drain
a
set
of
nodes
and
when
the
job,
when
the
jobs
finish,
those
nodes
will
be
ready
and
you
will
be
able
to
reboot
them.
However,
in
a
storage
system,
you
can't
just
reboot
a
machine
when
it's
ready
because
it
could
be,
it
could
be
hosting
a
replica.
E
So
if
you,
you
can
shut
down
server
three
and
server
two,
because
one
replica
of
each
would
be
available,
but
you
wouldn't
be
able
to
to
shut
down
server
three
and
server
4,
because
then
the
the
green
data
would
be
unavailable
right
and
in
a
hybrid
converged
architecture.
You
have
both
problems
at
once.
You
can't
just
you
can't
just
tell
the
batch
system
to
drain
the
node
and
then
just
shut
off
and
reboot.
E
E
So
it
knows
how
to
handle
slurm
or
an
hd
condor,
and
it
knows
how
to
drain
a
machine
and
slurm
or
how
to
drain
a
machine
on
http
counter,
and
then
it
manages
the
life
cycle.
Basically,
this
is
what
we've
deployed
on
our
on
our
on
all
of
our
cluster,
to
manage
all
the
over
250k
course,
and
the
cool
thing
about
this.
E
This
tool
is
that
it
is
able
to
manage
the
several
concurrency
strategies
next
slide
next
slide,
please
so
as
a
result
of
being
able
to
manage
the
concurrency,
a
user-defined
concurrency
strategy
is
that
you
can.
You
can
do
orchestration
based
on
that,
so,
for
instance,
you
could
say
I
want
to
reboot.
Let's
say
I
want
to.
I
want
to
reboot
the
whole
cluster,
but
I
only
want
to
have
10
of
the
cluster
draining
and
rebooting
at
a
time
and
these
agents
they
will.
They
will
coordinate
this
by
themselves.
E
The
same
thing
happens:
if
you
have
restrictions
such
as
hyperconverged
file
system,
where
you
may
have
some
data
replication
strategies
where,
for
instance,
you
may
only
reboot
machines
from
the
same
row
at
the
same
time,
so
yeah
the
basically
this
this
tool
will
be
able
to
orchestrate
this
and
and
this
this
plugs
into
our
automation
tools
to
handle
all
these
cases
and-
and
that
was
it
well.
I
think
dan
will
provide
some
final
awards
and
then
we'll
be
happy
to
take
questions.
C
C
The
7th
in
portland
will
be
getting
together,
and
so,
if
you're
interested
in
hearing
more
about
steph
or
meeting
other
users
or
meeting
developers
involved
in
ceph,
that's
a
great
opportunity,
I
would
say-
and
otherwise
thanks
for
listening,
if
you
have
any
questions,
we're
happy
to
chat
now
also
feel
free
to
get
in
touch
with
us
via
email.
And,
if
you'd
like
to
find
out
about
sef,
then
ceph.io
is
the
website.
E
F
A
C
C
C
What
it
what
it's
doing
is
normally
like
in
fs,
even
if
you
use
the
the
thingy
to
lock
a
range,
it
doesn't
just
lock
a
range.
It's
locking
to
do
the
whole
consistency.
It's
it's
locking
an
entire
file.
So
the
client
gets
a
gets
a
capability
exclusive
right
capability
and
then
he's
able
with
exclusive
right
cap
capability,
a
client
can
write,
can
do
buffered,
io
and
then,
as
soon
as
another
client
wants
to
read
that
the
mds
revokes
that
exclusive
capability
now
with
lazy
io.
C
So,
okay,
if
you
have
many
writers,
none
of
them
get
the
exclusive
right.
So
they
end
up
like
passing
kind
of
like
passing
a
token
around
who
who
writes
they're
right
in
a
circle
so
like
it's,
really
slow
if
they're
all
running
to
the
same
file
with
lazy
io,
this
is
all
relaxed
and
they
can
all
just
do
buffer.
I
o,
and
they
can
all
just
write
and
flush,
flush,
flush
and
and
the
end
is
just
accepts
everything
and
they
just
write
directly
to
the
data
pools
and
there's
no
there's.
E
F
E
E
C
Yeah
I
mean
otherwise,
I
mean
outside
of
hpc.
We
have,
we
have
multi-petabyte
clusters
and
of
order
like
say
two
or
three
thousand
clients,
mostly
like
kubernetes
clients
or
other
just
generic,
like
linux
boxes,
doing
stuff
per
attached
to
a
single
cluster
and
there's
a
there
are
I've
heard
of
others
I
mean
there
are.
There
are
others,
of
course,
using
it
at
a
bigger
scale
than
us.
I've
heard
of
late
latest
I've
heard
of
someone
with
some
that
he
submitted
a
patch
set
to
make
this
even
more
efficient,
but
he
was
using.
D
G
C
I
can
take
that
one.
I
guess
so
each
object
when
you,
when
you
write
an
object,
an
object
is
normally
something
like
a
few
tens
of
megabytes
of
data.
Okay
at
most
now
attached
to
that
you
can
they.
There
is
a
key
value
store
attached
to
each
object,
so
you
can
you
can
in
general
this
is
called
the
omap
the
object
map
and
there
you
can
have
keys
of
say
tens
of
characters
and
values
of
maybe
a
few
kilobytes.
C
Something
like
this.
So
this
o
map,
this
object
map
is
used
internally
by
the
routers
gateway
to
do
the
bucket
indices,
for
example,
and
by
the
file
systems,
do
directories
so
a
directory
node
is
actually
just
an
empty
object
with
inodes
attached
to
it.
That's
how
directory
that's
how
the
file
system
structure
is
created
yeah,
so
omap
is
the
key
word
there,
and
the
other
restrictions
are,
as
I
described,
is
that
does
that
answer
the
question.
C
D
Okay,
all
right
well,
first
of
all,
thank
you
for
the
talk
that
was
interesting.
Well,
I've
seen
a
presentation
about
self
a
few
years
ago,
where
there
was
some
experimentation
related
to
using
a
hardware
that
was
really
object
oriented,
so
your
disk
was
actually
providing
a
key
value
interface,
as
opposed
to
block
storage.
Is
that
something
that
is
still
alive
and
something
that
ceph
is
using?
Are
you
using
a
file
system
behind
the
scene
to
store
the
data
that
you
store.
C
Yeah,
so
that
was,
I
forgot,
the
name
that
was
a
seagate
product.
Wasn't
it
let's
remember
yeah.
This
was
so.
This
is
not
actively
being
worked
on
as
far
as
I
know
so,
yeah
there
was
so
okay
inside
inside
the
osd
there's
an
abstract
implementation
of
an
object
store
under
inside
the
osd
in
the
in
the
early
days
of
ceph.
This
they
were
trying
to
use
btrfs
to
do
this,
to
take
advantage
of
snapshots
and
everything
like
that.
But
it
didn't
perform
well
enough.
C
So
then
they
just
implemented
a
simple
one
called
file
store,
which
used
xfs
to
store
the
all
of
the
objects.
But
this
eventually
doesn't
scale
either
because
you
have
to
put
everything
in
directories
and
you
have
to
like
split
the
directories
if
they
get
too
big
and
you
have
to
merge
the
directories
if
they
get
too
too
small,
and
it
just
becomes
like
something
that
you
can
have
like
steps
and
glitches
and
latency
increasing.
C
C
So
it
stores
it
implements
a
very
simple
fs
api,
so
that
rocksdb
can
run
on
top
of
the
the
block
devices
and
it's
using
that
so
that
the
osd
can
keep
track
of
which
objects
are
stored
locally
and
like
store
the
cluster
maps
and
things
like
that.
But
then
for
the
rest
of
the
actual
big
blobs
of
data.
C
It
just
has
very
simple:
allocators,
actually
they're
getting
progressively
more
and
more
complex
and
progressively
more
performant,
with
every
version
of
ceph
now,
but
but
they're
just
consuming
the
the
the
block
devices
raw
now
and
yeah.
That
thing
is
actually
quite
good.
It
can
do
compression
inside,
like
it
can
compress
the
objects
before
it
stores
them,
and
that
was
what
led
to
something
that
alberto
said.
C
G
F
Okay,
what
I
was
curious
about
is,
I
noticed.
I
think
one
thing
that's
really
compelling
about
stuff.
Is
this
idea
that
you
could
have
like
just
blobs
of
storage
and
then
you
can
compose
it
kind
of
dynamically
in
these
different
ways
with
these
different
interfaces
right,
but
I
noticed
in
in
at
least
in
your
deployment
you've.
It
seems
like
you've
partitioned
a
lot
of
these
into
separate
file
systems,
and
I'm
just
curious,
if,
like
I,
I
can
see
why
there
might
be
just
kind
of
operational
reasons
that
that
has
to
be
done.
F
C
So
like
we,
you
notice
that
we,
I
mentioned
qos
types
in
the
block
storage.
This
is
because,
with
a
very
few
handful
of
vms,
they
can
like
saturate
the
io,
the
iops
of
the
cluster.
If
you
have
a
spinning
disk
cluster,
so
we
want
to
sort
we
we
like
need
to
throttle
those
clients
that
they
don't
overwhelm
them.
We
can
have
thousands,
we
have
like
7
000
clients
now,
but
if
we
then
threw
us
ffs
on
there
on
that
same
cluster,
we
can't
throttle.
C
C
So,
in
the
end,
it's
because
of
those
like
inability
to
to
like
have
qos
guarantees
between
the
different
pools.
That's
what
makes
us
separate
right
now
also
for
operations.
Reasons
like
there
are.
There
are
pragmatic
reasons
related
to
when
you're
updating
things
that
that
often
like
put
you
in
the
direction
of
separating
things-
maybe
you
maybe
like
block
storage,
is
actually
very
simple.
It's
you
it's!
C
It's
very
rock
solid,
very
few
bugs
I
don't
remember
any
bugs
like
in
years,
so
you
can
upgrade
that
you
can
follow
the
latest
upstream
with
with
block
storage
clusters
or
pure
routers
clusters.
But
s3
is
like
a
little
bit
more
complex.
You
might
want
to
like
wait
a
little
bit
and
see
and
test
things
more
carefully
and
cfs
as
well
like
this.
It's
has
it's
so
complex,
a
posix
file
system
that
you
really
want
to
test.
C
The
thing
is
that
you
can
easily
like
in
the
end,
it's
not
a
big
problem
for
us,
because
you
can
so
readily
just
drain
nodes.
Move
data
around
there's,
never
actually
a
problem
where
oh
I've
got
too
many.
Like
free
bytes
in
this
cluster,
and
not
enough
this
other
one,
you
can
always
just
like
drain
nodes
and
then
add
them
to
the
other
cluster
move
hardware
around
anyway,
so
we
operate
14
or
so
or
15
clusters.
Now
and
it's
it's
it's
pretty.
It's
pretty.
Okay,
pretty
easy.
F
A
C
E
E
A
E
Yeah,
I
mean
just
to
expand
on
dance
comments.
We're
layered,
as
in
you
know,
just
just
like
the
infrastructure
layers,
so
you
have
the
the
openstack
guys
and
then
you
have
different
components
built
on
top.
So
if
you're,
if
you're
somebody
like
if
you're
running,
for
instance
like
me,
the
hpc
service,
you
do
get
users,
you
can
use
your
tickets
and
you
do
support
the
users
directly,
such
as
you
know,
the
physicists,
the
theorists,
the
engineers.
A
E
A
E
Fact
yeah,
I
was
just
going
to
ask
you
dan
thanks
for
moving
yes,
so
back,
then
it
was
the
mo.
We
were
mostly
using
set
fuse
mounts,
but
with
the
kernel
model,
things
are
a
lot
better,
a
lot
easier
and
in
fact,
there's
also.
You
can
use
libs
ffs,
which
is
a
user
space
library
and
actually
it
works
great
and
somebody
from
red
hats
mark
from
red
hat
and
me
we
contributed
code
to
ior.
A
G
E
Well,
that's
that
that
was
my
my
first
slides
yeah.
We
already
have
quite
a
number
of
file
systems
and
it
could
be.
It
could
have
also
maybe
been
a
choice
and
say,
let's,
let's
just
run
luster,
but
the
storage
team
was
already
running
like
a
set
of
file
systems
and
we
didn't
feel
like
supporting
yet
another
file
system,
adding
another
file
system
to
the
zoo,
yeah
of
all
systems.
C
There
were,
of
course,
yeah.
There
were,
of
course,
tests
like,
because
we've
had
open
af-s
for
so
long
and
like
open
afs
like
there
have
been
several
efforts
to
get
rid
of
open
afs
at
cern
and
one
of
those
was
actually
like
evaluating
luster
and
doing
some
tests
to
see
if
we
replaced
with.
Why
is
all
of
our
three
billion
files
in
open
afs,
with,
like
the
whole
worldwide,
like
lhc
computing
grid
accessing
with
luster?
C
And
I
guess
I
don't
know
this
was
before
my
time,
but
it
was
decided
to
stick
with
afs
and
then
I
suppose
the
other
chance
would
have
been
when,
like
cephas
sorry
cern,
has
always
been
writing
its
own
file
systems.
We
had
something
called
castor
before
certain.
Like
advanced
storage,
I
think
it
was
and
then
at
the
time
that
the
lhc
started,
castor
was
not
performed
enough,
so
we
needed
something
else.
C
So
then
we
used
x,
root
d,
which
is
a
storage
framework
written
it's
stanford
slack
and
we
used
x
or
d
to
build
to
build
a
simple.
Like
physics,
data
store
called
eos.
Okay,
so
that's
how
that
happened.
That
was
like,
probably
another
chance
where
luster
was
was
tried.
I
think
that
we
tried
gpfs
at
that
moment
as
well,
but
it
was
just
simpler
to
to
build
something
with
with
x-root
at
that
time.