►
From YouTube: Ceph Operations at CERN: Where Do We Go From Here? - Dan van der Ster & Teo Mouratidis, CERN
Description
Ceph Operations at CERN: Where Do We Go From Here? - Dan van der Ster & Teo Mouratidis, CERN
This talk will present a top down view on how Ceph is operated within the large scale research environment of CERN. Scientists at CERN use Ceph in an increasing variety of ways, from block storage for OpenStack to HPC filesystems to S3 object storage. Operating this ~20PB of infrastructure requires continuous measurement and performance tuning to keep the infrastructure running optimally. In this area, we will present our experience tuning and scaling RBD and CephFS, with the latter culminating in the first appearance of Ceph in the IO-500 list. On the operations side, we will present our approach to commissioning and decommissioning hardware, demonstrating some advanced features such as the Ceph balancer. We will conclude by presenting what is upcoming for storage in general at CERN, and present different scenarios how Ceph might play a role in that story.
About Teo Mouratidis
Storage Engineering Fellow, CERN
Teo is works in devops on CERN's storage team. While operating Ceph at large scale, Teo has been contributing to Ceph development in the areas on data balancing and RBD.
About Dan van der Ster
CERN
Storage Engineer
Dan manages the Ceph storage service at CERN in Geneva, Switzerland. He has participated actively in the Ceph community for more than 5 years, being one of the first to demonstrate Ceph's scalability up to multi-10's of petabytes. Dan has spoken at several Ceph Day's and OpenStack Summits, acted as Academic Liaison to the original Ceph Advisory Board, and now has a similar role on the Ceph Governing Board. Dan earned a PhD in Computer Engineering at the University of Victoria, Canada in 2008.
A
Ok,
it's
2
o'clock,
I!
Guess
we
start
the
afternoon
sessions
so
we're
from
CERN
I'm
Dan,
and
this
is
Te'o
he's
working
with
us
on
set
operations.
So
the
talk
today
is
basically
a
summary
of
the
set
operations
at
CERN.
Where
do
we
go
from
here?
I
mean?
How
do
we
come
up
with
names
for
slides?
That's
the
that's
the
point
there.
The
picture
there
is
a
is
a
proposed
collision
of
particles
from
the
new,
the
new,
much
larger
particle,
etc.
That's
planned
for
for
certain.
That's
the
one
of
the
promotional
pictures.
A
A
A
It
was
in
February
2013
that
we
first
wrote
this
kind
of
proposal.
Openstack
was
becoming
popular
and,
as
is
kind
of
common
at
CERN,
like
they
forgot
about
the
the
data
part,
the
storage
part
for
our
cloud,
so
they
said:
hey
gee.
Can
you
guys
do
something
for
data
and
then
Google
around
and
you
find
Saif
is
the
obvious
one
and
that's
how
it
kind
of
started.
A
This
was
our
first
self
cluster.
It's
still
in
production.
It
was
during
the
the
dumpling
days.
It
was.
Each
of
these
servers
is
a
J
baud
with
some
SSDs.
Actually,
we
didn't
have
SSDs
at
the
beginning,
and
we
had
to
like
prove
that
SSD
Journal's
was
was
interesting,
and
this
was
all
all
the
the
fun
days
when
we
were
learning
how
stuff
was
working,
this
one's
still
in
production.
A
So
we
started
with
a
300
terabytes
proof-of-concept.
Then
we
had
three
petabytes
in
production
that
fall
over
the
next
couple
of
years.
Things
were
growing
and
we
were
involved
in
the
erasure
coding,
development
and
also
like
for
physics,
data
use
cases.
We
noticed
that
Rados.
We
wanted
to
use
the
low-level
Rados
protocol
for
some
physics,
data
storage,
but
it
was.
It
was
not
appropriate
because
the
objects
can
be
too
large,
so
you
want
to
stripe
them
into
smaller
pieces.
A
So
this
is
why
we
wrote
this
radio
Stryper
library
that
people
have
maybe
seen
in
2016,
we
upgraded
one
of
our
big
clusters,
replacing
all
the
hardware
that
three
petabytes
that
I
showed
migrating
all
the
data
to
another
cluster,
with
with
twice
the
space
with
no
downtime.
So
this
was
the
first
time
that
we
could
actually
try
this.
This
idea
that
that
stuff
is
like
a
kind
of
organic,
always
living
data
cluster
that
you
can,
that
can
survive
through
multiple
generations
of
hardware.
A
A
This
is
our
clusters
that
we
have
today.
The
biggest
use
cases
are
still
for
OpenStack
cinder
and
glance.
We
have
six
petabytes
in
in
Geneva
in
our
main
data
center
one
and
a
half
petabytes
in
a
remote
data
center
in
Budapest,
that's
actually
being
decommissioned
this
year,
so
we'll
migrate.
All
this
data
back
and
then
we
have
a
small
hyper-converged
cluster
set
of
s.
We
have
three
clusters
again
like
one
petabyte
for
Manila
for
OpenStack
Manila
and
like
just
like
bog-standard
mass
use
cases.
A
We
have
a
small
test
cluster,
because
everyone
should
have
a
test
cluster
and
again
hyper-converged
clusters.
So
we
are
hopping,
average
set
of
s
is
actually
like
it's
an
OpenStack
cloud
cell
with
block
storage
and
SEPA
fests
they're,
all
in
the
same
nodes,
all
flash
for
core
and
we'll
get
more
into
that
in
later.
In
the
talk
we
have
a
five
petabyte
cluster,
which
is
a
buffer
in
front
of
our
tape
system,
so
users,
writing
writing
files
to
our
tape
or
recalling
from
tape,
goes
through
this
disk
buffer
and
then
yeah.
A
We
have
a
two
petabyte
s3
+,
swift
and
right
now
we're
running
all
luminous
or
mimic
we're
just
starting
to
mimic
upgrades
and
now,
after
hearing
of
the
exciting
news
about
Nautilus,
we'll,
probably
maybe
next
week
go
to
Nautilus,
you
see
the
rest
of
the
talk
is
gonna,
be
like
outlining
our
recent
notable
ops
experiences
like
migrations
and
up
map
stuff,
and
some
s3
authentication
tail
will
explain
that
part.
I
will
explain
some
of
our
highlight
some
of
our
interesting
use
cases
and
then
Te'o
will
explain.
B
Hello
I'm
tell
I'm
going
to
talk
to
you
about
our
recent
operations.
Experience
that
we
did
this
year
in
early
2018,
we
had
three
active
active
clusters
on
file
store,
six
petabytes
for
blob
snorts,
two
petabytes
for
s3
and
one
petabyte
force
FS.
Each
of
these
clusters
has
about
40
gigabytes
as
the
journal
per
HDD
and
the
goal
to
reduce
those
SSDs
as
they
rocks
to
be.
B
We
have
to
start
from
somewhere,
so
we
chose
the
clusters
that
are
least
sensitive
to
latency,
so
the
clients
won't
notice
the
impact
when
we
do
the
operations
with
all
the
stratton.
The
standard
procedure
which
is
to
drain
the
oil
is
this
related
to
one
SSD
using
suppose
D
out
we
wait
until
the
data
is
drained.
Then
we
start
to
zap
the
SSDs
and
the
eight
CDs
with
the
sub
volume
commands,
and
then
we
recreate
the
OS
this
with
a
safe
volume
batch
command
using
the
same
as
the
IDS.
B
The
current
luminous
implementation
doesn't
have
this
one
so
hard
to
backport
it
from
master,
something
that
we
didn't
see
one
with
the
operations.
That
is,
we
use
the
AdMob
balancer
to
balance
all
clusters,
but
when
we
started
doing
those
migrations,
the
app
maps
were
removed.
Once
we
run
the
surface
they
up
command,
so
our
data
became
imbalanced.
We
can
see
that
on
the
next
slide,
and
one
also
optional
and
but
dangerous
procedure
is
that
we
wanted
to
do
those
migrations
as
fast
as
possible.
So
what
we
did
is
that
we
skip
the
drain
part.
B
So
what
would
be
the
exactly
was
just
the
second
and
the
third
step.
We
stopped
immediately
the
design
SSDs
and
we
recreate
them
and
with
then
let
the
backfill
happen.
Don't
do
this
if
you,
if
you
don't
want
to
go
into
danger
zone,
these
are
some
numbers
from
the
file
so
to
blue
store
performance.
We
can
see
that
blue
sorb
increases
the
performance.
A
lot
on
identical
hardware
from
file
store,
the
Rattus
gateway
has
had
a
massive
metadata
performance
increase.
B
We
used
the
back
at
the
indices
in
SSD
rocks
to
be
which
is
much
faster
than
the
one
we
had
before,
which
was
level
to
be
on
HDD
before
we
had
those
metrics
that
we
can
see
on
the
top
of
or
boot
head
and
delete
for
the
rads
gateway.
And
now
you
can
see
that
we'll
have
about
83
thousands
of
operations
per
second
plus
minus
four
and
the
others
are
on
the
same
scale
and
this
metrics
before
with
file
store.
They
were
two
kilohertz,
it's
and
it's
it's
a
massive
performance
gain.
B
Also
our
surface
users
reported
almost
double
the
speed
up
for
read
performance.
The
load
average
on
our
file.
Storage
server
were
more
than
the
gusto
one.
So
this
means
that
the
Brewster
hosts
use
much
less
CPU
than
the
file
store
runs.
We
can
see
here
on
on
the
graph.
It's
a
yam
wrapper
single
for
the
Serapis
right
and
it
took
on
CFS
more
than
two
hours
and
now
with
the
blue.
So
once
it
takes
one
point
two:
this
is
a
heat
map
of
our
OSD
utilization.
B
B
The
blue
means
that
there
are
less
data
points
in
that
and
the
red
means
that
there
are
more
so
you
can
see
that
there
are
more
in
the
area
of
50%
and
when
we
start
do
the
migration
to
blue
stone.
You
can
see
here
that
our
data
immediately
became
balanced.
The
variation
just
spread
it
out
and
on
the
bottom,
you
can
see
the
file
store.
B
The
balance
of
module
and
up
map
is
particularly
interesting.
What
it
does
is
that
it
Maps
Fitzy's
to
underutilized
this
this,
so
we
can
have
data
balanced.
The
heuristics
used
by
the
Atma
balancer
have
been
tuned
in
recent
months.
What
you
do
is
that
you
remove
a
pub
address
which
don't
help
with
the
data
distribution
you
add,
or
remove
as
the
mappings
for
existing
up
map
entries
and
support,
new
trees
and
grass
trees
and
rules.
These
are
the
versions
that
have
all
those
garden
heuristics
for
the
mg
up
and
they
are
AdMob
balancer.
B
This
graph
here
shows
again
those
digitalization
after
the
file
so
to
boost
our
migration.
The
first
part
is
when
the
brewster
migration
finished.
These
are
the
the
Brian
so
Falls,
the
utilizations
that
we
have
and
when
we
enable
the
op-amp
balancer
within
two
days,
you
can
see
here
that
are
all
of
our
oils
days
rely
on
the
50
or
55
percent.
It
was
little
ization
more
fun,
with
admin
at
safe
day
in
Berlin
done
presented.
B
The
second
application
of
add
map
up
every
month
is
a
tool
to
force
with
this
to
be
active
and
clean
without
data
movement.
What
you
do
is
that
sometimes
we
want
to
run
some
commands
that
do
changes
in
the
class
alike.
Changing
the
the
grass
tree.
That
requires
data
movement,
and
you
don't
want
to
do
the
recovery,
because
it's
a
bit
heavy
for
the
client
I/o.
So
you
up
map
those
pieces
back
to
their
previous
place.
So
it
everything,
looks
okay
and
then
you
enable
the
a
power.
B
B
The
solution
is
to
synchronize
the
East
Auckland
s
l's,
with
the
Rattus
gate,
one
through
an
open
side.
Mr.
job,
what
we
do
is
we
synchronize
the
code
essence
and
the
quota
of
the
users
to
local
rubbish
gateway
users
and
we'll
let
the
the
our
our
clients
use
the
V
local
of
indication
which
is
master
faster
than
the
Keystone
one.
You
can
see
some
tails
on
this
following
link.
A
Hi
thanks
Jeff,
so
I'll
describe
sort
of
the
use
cases,
so
most
of
our
OpenStack
cloud
is
designed
for
just
bulk
storage
of
typical
VM
stores.
Okay,
I,
don't
know,
I
don't
want
to
go
to
detail,
but
we
limit
the
QoS
to
the
I
ups
to
100
or
500
I
ops
for
these
volumes
and
like
99.9%
of
the
users
that
we
have
are
totally
happy
with
this.
But
then,
of
course,
there's
always
the
the
remaining
users
that
are
late
have
latency
senses,
sensitive
applications
like
databases
or
like
new
messaging
infrastructures.
A
A
So
this
is
like
a
no-brainer.
We
just
use
our
aja
coding
in
this
cluster,
so
this
is
like
a
possible
new
architecture
for
our
cloud
to
have
all
flash
eraser
coated.
Hyper-Converged
squab
sells
another
interesting
thing:
is
this
HP
c
on
SEF
FS,
so
we're
lard
motes
turns
mostly
a
high
throughput
computing
lab,
so
we
have
like
multiple
hundred
thousand
CPU
cores
is
doing
batch
processing.
This
is
not
typical,
HPC
workloads,
but
we
have
some
applications
like
beam
simulations
or
quantum
chromo
fluid
dynamics
which
need
which,
which
need
high
memory
applications.
A
Also
doing
massive
message-passing
and
they
need
like
fast
POSIX
storage,
so
we've
been
using
set
fest
for
this
because
it
was
a
what
something
we
were.
We
were
familiar
with
already
operating,
we've
been
doing
this
for
two
years
already
and
last
year
we
got
into
this
new
IO
500
top
list,
which
is
which
is
a
list.
That's
announced
each
supercomputing,
so
we're
number
21.
A
They
don't
they
don't
see
any
slowdown,
a
set
of
s
and
so
they're
happy,
and
then,
moreover,
we
started
testing
this
new
feature,
which
is
lazy,
IO,
which
is
a
which
is
a
kind
of
relaxed,
POSIX
standard
that
self
implements
and
they
can
get
like
three
to
four
times
better
parallel
I/o
widths
ffs
than
their
current
performance.
So
it's
looking
quite
good
this.
This
study
is
being
wrapped
up
into
a
paper
at
the
at
the
suitors
OSI
European
supercomputing
conference
called
ISC
at
this
there's
a
what
there's
a
workshop
on
power
on
parallel
storage
performance.
A
So
currently
we
place
these.
The
MDS
is
just
somewhere
nearby
but
like.
If
we
in
our
HPC
s,
we
have
like
multiple
switches
and
routing
between
them.
So
if,
if
the
client
has
to
go
through
a
router
to
get
to
the
traffic,
it's
it's,
it
can
be
slow.
So
now
we
we,
we
try
to
do
like
different
zones
of
the
MD
of
the
of
the
Ceph
of
s
filesystem,
and
we
have
a
plan
to
actually
improve
this
even
further,
so
that
we
would.
A
We
would
have
like
a
fast
HPC
scratch
area,
something
like
a
burst
buffer.
What
we
would
do
with
that
is
we
would.
We
would
put
some
very
high
endurance,
expensive,
small
capacity,
flash
like
directly
on
the
nodes
and
then
with
SEPA
vests,
because
you
can
like
very
quickly
create
a
new
file
system
or
create
new,
create
directories
with
with
map
to
do
pools
you
and
with
crush
mapping.
You
can
like
locate
these
pools
to
the
specific
parts
of
the
cluster.
A
You
can
create
like
targeted,
fast
scratch
areas
for
the
applications,
so
what
we're
basically
hooking
into
slurm,
which
is
the
HPC
job
scheduler,
so
that
when
a
job
gets
scheduled
to
a
part
of
the
cluster,
you
create
a
set
of
s
there,
MDS
OSDs
and
the
directories
right
there,
and
then
they
can
do
their
scratch.
They
can
burn
through
those
that
flash
and
then
when
their
job
finishes
you
they
stage
their
data
out
somewhere
and
you
and
you
delete
the
data.
A
This
is
all
planned
to
try
to
get
a
better
score
and
that
IO
500
we're
also
we're
also
having
fun
trying
to
back
up
things.
We
have.
We
have
another
storage
system
at
certain
called
CERN
box,
which
is
like
a
sync
and
share
platform
with
with
accounts
for
everyone
at
our
lab
and
all
of
our
users
like
sixteen
thousand
users,
half
a
billion
files
and
three
petabytes
of
data
so
far
and
we're
trying
to
back
up
to
s
3
to
s
FS
3,
and
we
found
this
cool
tool
called
rustic
which,
which
is
really
nice.
A
A
So
we
like
it
but
running
this
kind
of
backup
system
for
one
user
is
very
easy,
but
when
we
have
16,000
users
to
backup,
we
basically
need
a
backup
scheduling
system,
so
we're
developing
a
backup
scheduling
system
that
coordinates
all
these
backups
resolves
errors,
schedules
them,
and
this
is
going
quite
well,
I
think
we're
backing
up
like
300
users.
Now
in
the
current
system,
this
is
the
architecture,
so
the
data
in
certain
box
is
stored
on
our
storage
system
called
a
OS
which
we
mount.
A
A
We're
also
using
object,
storage
and
our
s3
cluster
for
some
like
fun
physics,
analysis,
demos.
If
people
are
staying
for
the
coop
con
on
Tuesday,
one
of
our
colleagues
is
going
to
give
a
presentation,
so
basically
in
2012
CERN
and
some
people
had
certain
got
the
Nobel
Prize
for
discovering
the
Higgs
boson.
A
What
what
some
of
our
colleagues
have
done
is
they've
like
reproduced
this
whole
analysis
that
took
several
years
of
data
process
and
they've
collapsed
it
down
to
a
like
a
subset
of
the
data
and
they
will
like
live
in
real
time
with
a
jupiter
notebook,
auto-scaling
kubernetes
and
some
object.
Storage,
they'll,
they'll,
actually
redo
that
whole
analysis.
So
it
looks
like
this
there's
like
there's
like
70
terabytes
of
simulated
data
that
we
have
stored.
A
How
physics
analysis
always
works
is
that
you
simulate
what
should
happen,
and
then
you
observe
what
actually
does
happen.
The
observe
observation
is
just
taking
pictures
of
what
the
collisions
look
like
and
then
you'd
like
to
do
a
subtraction.
So
it's
hard
to
see,
but
like
this,
this
like
red
bump
here,
is
actually
like
a
discovery
of
the
Higgs
boson,
because
it's
not
in
the
simulation.
So
that's
what
they
do.
They
stimulate
a
bunch
of
data,
compare
it
with
the
real
data
and
then
do
it.
A
B
Cern
has
three
active
commuters
to
serve.
We
are
mostly
fishing
fixing
issues
fun
with
the
operational
scope,
some
requests
that
we
have
from
clients,
some
bugs
that
we
find
and
they
are
easy
to
fix
and
we
contributed
a
little
bit
with
the
mzr
balancer,
the
subvolume,
the
RTD
trust
and
the
client
table.
Sometimes
you
have
to
do
some
builds
custom
ones
with
hot
fixes,
we'll
find
some
problems
we
see
on
the
tracker.
B
What
back
with
the
back
filling
needs
to
scan
all
the
objects
in
in
Apogee?
It
will
have
to
check
its
objects
version
and
it
will
try
to
match
this
with
with
its
replicas
and
trying
to
do
that.
For
the
whole
disk
is
a
really
slow
operation.
What
we
could
do
is
we
could
create
some
data
structures
that
describe
the
underlying
data
of
the
pizza,
and
then
we
can
use
those
ones
to
navigate
and
the
check
which
segments
of
a
pizza
were
changed
during
OSD,
which
was
down.
B
B
One
of
those
solutions
are
myrtle
trees.
They
were
invented
by
1979
by
Ralph
Merkle.
They
are
used
by
block
chaining
get
the
tree
is
built
by
splitting
the
day
time
into
chunks,
passing
those
chunks
and
then
combining
those
classes.
Until
you
get
to
the
top
hash
mark,
approves
Mirko
proofs
say
to
us
that
the
attack
would
exist
in
entry
if
two
hostas
are
met.
So
if
you
has
the
same
objects,
they
it
means
that
they
should
produce
the
same
house.
B
So
what
you
do
instead,
is
you
compare
the
houses
instead
of
all
the
objects
in
them
on
the
disk?
This
is
a
sample
of
Merkle
tree
that
could
be
an
unsafe.
The
bottom
row
that
is
from
0
to
7
is
range
and
we
cut
into
8
tanks.
So
in
those
tanks
there
could
be
busy
objects
along
with
their
versions,
and
what
we
could
do
is
that
who
has
all
those
objects
and
versions,
and
then
we
combine
them
until
we
reach
the
top,
has
so
two
replicas.
B
In
this
particular
example,
we
could
see
that
the
resulting
table
has
of
two
different
races
different.
That's
why
it's
marked
with
red
the
ones
that
are
the
same
are
green
and,
as
we
go
down
the
tree,
we
found
out
that
only
3/8
of
an
axle
PG
has
to
be
recovered
instead.
So
this
would
save
a
lot
of
time
in
some
cases.
B
With
this
feature,
we
can
quickly
discover
which
ranges
we
can
skip
during
backfilling
and
make
it
faster.
The
trees
can
resume
concept
memory,
because
it's
just
the
tree,
it
doesn't
add
anything
else
like
the
Pizza
log,
which
has
to
increment
every
diamond
io
happens
and
currently
is
the
developing
proof
of
concept.
I
have
done
the
implementation,
the
Skip
brains,
functionality
is
done.
B
A
That's
the
end
of
the
talk,
I
mean
so
it's
just
another
talk
that
we've
given
in
the
long
series
of
Ceph
talks,
the
key
usage
that
certain
continues
to
grow.
It's
like
now
well-established
a
reliable
backstop
for
our
cloud
and
our
HPC
activities.
It's
kind
of
becoming
a
little
bit
boring
it's
so
reliable
and
well
established.
We
have
to
shake
things
up
a
little
bit,
I
think,
like
many
others.
I
want
to
thank
the
the
the
awesome
stuff
community
which
are
so
friendly,
knowledgeable
peers
that
we
learn
a
lot
from
and
everyone's
constructive
working
together.
A
It's
really
great,
and
we
know
that
it's
like
super
important
to
maintain
a
good
community
karma,
so
I
hope
that
you
guys
also
try
to
do
that
and
we
have
a
save
the
date
here,
because
we're
planning
an
event.
This
fall
in
September
at
CERN
to
host
a
set
day
where
we
want
to
focus
on
research
or
academic
and
nonprofit
Atkins
of
Seth
it'll
be
like
one
day
or
maybe
we
might
change
the
date
to
September
17th.
A
We
have
to
just
get
the
exact
room
booking
down,
so
keep
keep
an
eye
on
the
on
the
mailing
list
for
the
final
announcement.
It's
we're
trying
to
coordinate
it
very
close
to
an
event
which
is
the
14th
and
15th
of
September,
which
is
certain
open
days,
which
is
certain,
will
open
its
doors
to
the
public
and
you
can
come
and
visit
all
of
the
different
sites
get
tours
of
things
that
are
normally
closed,
so
yeah.
That's
why
we're
trying
to
do
this
at
the
same
thanks
a
lot.
Are
there
any
questions.
C
A
We
we
have
both.
Okay,
our
large
block
storage
cluster
has
to
crush
routes
like
one
in
one
room
with
ups
battery
power
and
one
crush
route
in
a
room
with
a
diesel
generator
power.
So
it's
two
different
qualities
of
service
and
we
used
to
in
fact
put
the
first
two
replicas
in
the
diesel
room
and
the
third
replica
in
the
main
room,
but
because
of
capacity
reasons
we
moved
that
third
replica
back
to
back
to
the
diesel
room
yeah.
Thank
you.
D
Just
a
quick,
just
good
question
about
the
s3
and
Keystone
integration:
I
read
your
podcast,
but
I've
seen
there
was
some
Petrus
from
BBC.
They
didn't
make
it
into
mimic,
I!
Think
yeah
I'm,
not
not
sure
about
this
I
you
aware
of
them
and
which
solution.
Do
you
often
and
do
you
think,
is
better
for
production
use
if
you're
running
mimic.
A
A
Yeah,
so
we
did
hear
about
that
so
at
the
OpenStack
summit
our
colleague
Jose
presented
this.
He
did
that
work
for
that,
and
he
pointed
us
out
so
yeah
as
soon
as
there's
some
like
proper
solution
for
this.
We
will
stop
synchronizing
the
kings
into
our
rattles
gateway,
okay,
but
but
for
now
it's
the
only
option:
okay,
yeah
thanks.