►
Description
Juggling Petabytes: Managing Ceph at Scale with Ceph-ansible - Matthew Vernon, Wellcome Sanger Institute
The Wellcome Sanger Institute has 18PB in its largest Ceph cluster. This talk will explain how the Sanger used Ceph to build and scale a reliable platform for scientific workflows, and enable secure data sharing via S3. And how they got 100GB/s read performance out of their cluster.
Matthew will outline the interesting aspects of the Sanger's Ceph setup, including how the team grew it from a small initial installation, automated deployment management and monitoring, and some of the issues they have encountered along the way. Matthew will also explore some of the good (and less good!) aspects of running Ceph at scale.
About mcv21
Wellcome Sanger Institute
Principal System Administrator
Matthew Vernon is a Principal System Administrator at the Wellcome Sanger Institute, and a member of the HPC team. As well as traditional HPC farms, the team supports an OpenStack platform, and 3 Ceph clusters, the largest of which has 18PB of raw capacity. Matthew's current work is largely around the management of the Ceph clusters. Matthew has been a Debian developer since 1999, and has a PhD in "spatial spread of farm animal diseases"; he has spoken at a number of scientific and technical conferences in Europe and the USA.
A
It's
a
bit
it's
a
bit
difficult
to
see,
but
I
think
it's
probably
best
to
take
questions
during
the
talk
rather
than
at
the
end.
There
will
be
some
time
at
the
end,
but
if
you
think
of
something
to
sort
of
wave
and
interrupt
me
and
I'll
deal
with
it
at
the
time
rather
than
we
get
to
the
end
and
I
realize
I
fail
to
explain
something
important
for
the
first
slide,
I'm.
A
A
Working
our
safe
cluster,
there's
sort
of
things
you
might
want
to
know
so.
I
start
working
a
bit
of
background
by
the
Institute
and
our
use
of
set
and
how
that's
grown
then
I'll
talk
about
how
we
deploy
safe
with
Stefan's
ball
I'll
talk
about
how
we
monitor
and
manage
it
and
then
finally,
talk
about
some
benchmarking
upgrades,
not
I'll
finish
with
some
problems.
We've
had
and
things
we
like
about
SEF,
so
the
sort
of
the
bad
bits
and
the
good
bits.
A
So
this
is
the
sangla
chooses
a
slightly
dated
now
aerial
photograph
of
the
campus.
The
institute
was
founded
in
1992
and
it's
really
specializes
in
gene
sequencing
and
genomic
analysis.
We
were
the
single
largest
contributor
to
the
sequencing
of
the
human
genome
and
we
continue
to
do
a
lot
of
both
gene
sequencing
and
analysis
of
gene
sequencing.
A
Typically,
that's
been
done
on
batch
based
clusters,
so
LSF
if
you're
familiar
with
LSF,
and
we
have
about
20,000
cores
available
in
that
batch
based
clusters
and
support
that
kind
of
work.
We
have
quite
a
lot
of
storage.
If
you
go.
If
you
talk
to
people
about
at
CERN,
they
will
describe
themselves
as
high
performance
computing,
we're
a
bit
more
highly
throughput
computing
rather
than
having
in
very
very
parallel
computer
obsolete
other
using
MPI,
or
something
like
that.
A
Typically,
where
our
scientists
are
reading
in
large
quantities
of
data
and
then
performing
some
rosalie
memory,
intensive
computation
on
those
data
and
then
writing
large
data
out
again
so
to
support
our
work.
We
have
quite
a
bit
of
lust.
We
have
about
15
petabytes
of
lustre
storage
lustre
as
a
fast
Network
file
system
and
about
20
petabytes
of
irods.
Irods
is
an
object
store
in
some
ways
like
Seth,
but
it
its
particular
strength,
as
it
enables
you
to
store
significant
quantities
of
metadata
along
with
your
objects.
A
So,
typically
a
sequence
file
will
be
stored
along
with
which
machine
it
was
sequenced
on
when
the
samples
came
in
dates
and
times,
and
that
sort
of
thing,
and
over
the
last
few
years
we've
been
looking
at
adding
OpenStack
to
our
computing
facilities.
We
now
have
about
10,000
cores
in
across
our
OpenStack
Suites.
A
We've
got
a
an
open
stack
instance
running
the
OpenStack
pike
release
and
one
running
the
Queen's
release
and
we're
in
the
process
of
migrating
our
users
from
one
to
the
other,
and
then
we're
already
started
to
look
Hertz
and
rocky
in
subsequent
releases,
and
we
started
off
with
SEF
as
a
way
of
providing
storage
to
OpenStack.
I.
Think
that's
quite
a
common
story
here
and
rather
than
having
one
set
cluster
per
OpenStack.
We
have
one
surf
cluster
and
all
of
our
open
stacks
are
just
from
SEFs
point
of
view
different
clients
to
that
OpenStack.
A
Finally,
we
also
when
we
started
with
staff,
thought
we'd
use
the
Raggles
gateway
future.
We
thought
our
scientists
might
want
to
use
s3
to
collaborate
and
share
data,
as
it
turns
out.
That's
been
wildly
popular
last
time.
I
checked
about
90%
of
our
safe
cluster
was
devoted
to
providing
storage
for
s3
through
the
Randalls
gateway,
and
that's
not
just
people
sharing
our
scientists
showing
objects
with
one
another,
but
they've
actually
built
entire
web
front-ends
on
the
top
of
that
s3
service.
So
we
have
a
thing
called
Jen
server.
A
If
you
want
to
get
and
sequence
analysis
done,
it
gives
you
a
web
front-end,
you
can
upload
sequences
and
manifolds
and
all
that
sort
of
thing,
and
it
does
it
for
you
and
really
that's
quite
a
thin
layer
on
top
of
s3.
So
my
team
is
informatics
support
we
manage
all
of
the
things.
I've
just
talked
about
all
the
scientific
compute
clusters,
the
storage,
all
that
sort
of
thing.
We
have
other
people
in
the
Institute
to
do
infrastructure
there.
So
networking
and
we
have
for
datacenter
halls
on-site.
A
So
we
have
a
specialist
team
that
handles
those
in
the
hardware
work
and
we
have
a
service
desk.
You
provide
that
sort
of
first-line
support
users.
I
mentioned
this
because
throughout
this
talk,
I
will
be
talking
about
how
we've
managed
to
offer
load
bits
of
running
safe
on
to
some
of
these
other
teams.
Basically
we're
busy
or
lazy.
Take
your
pick,
and
it's
been.
You
know,
for
the
things
like
hardware
work,
we
hand
off
to
the
data
center
team
because
it
frees
us
up
to
do
other
things.
So
that's
that's
who
we
are.
A
We
started
off
with,
and
all
of
our
set
clusters
are
on
identical
hardware,
if
you're
among
the
keynotes
earlier
from
Super
Micro,
they
talked
about
they're
for
you
storage
servers.
We
start
up
with
nine
of
those.
We
now
have
51
in
our
production
cluster
and
you
can
see
if
you
like
cars,
we're
here
and
specs
half
a
terabyte
of
RAM.
We
have
Mellanox
two
times
hundred
gigabit
networking
and
we
bond
those
together.
So
that
gives
you
one
in
one
faster
interface
and
when
we
run
the
Ceph,
private
and
public
cluster
networks
using
VLANs.
A
On
top
of
that
bond,
we
have
a
couple
of
small
SSDs
the
operating
system
and
then
the
main
storage
for
our
OS
DS
is
66
terabyte
SAS
drives,
and
then
we
have
two
two
terabyte
nvme
cards
for
in
file
store
the
journal
and
for
blue
store
blocked
ODB.
So
each
those
nvme
cards
is
providing
the
journal
for
30
OS
DS.
We
found
this.
We
grew
the
cluster
that
particularly
doing
a
significant
rebalance.
The
monsters
store
could
grow.
Quite
large.
A
I
think
the
largest
we
ever
saw
was
200
gigabytes
and
that
was
getting
embarrassingly
close
to
actually
filling
the
file
systems,
so
he
that
we
later
bought
some
dedicated
storage
for
our
monitor
stores
and
we
started
in
2016,
and
so
we
started
with
a
bunch
of
xenial.
We
are
now
running
Bionic
ever
except
our
production
cluster.
A
Give
you
a
night
of
timeline.
I
started
at
the
Sanger
2016
the
summer
2016,
when
we
had
basically
no
experience
of
Seth,
we'd
sort
of
set
up
a
toy
openstack
using
some
spare
hardware.
We
found
on
the
back
of
a
rack
and
then
later
in
2016,
we
bought
some
dedicated
hardware.
We
started
out
with
nine
of
those
super
mikro
boxes
in
three
racks,
three
in
the
bottom
of
each
rack
and
the
rest
of
those
racks
were
filled
with
the
OpenStack
compute
nodes
and
suchlike.
So
we
started
off
with
about
three
petabytes
of
raw
capacity.
A
Then
in
2017
we
essentially
took
an
extra
rack,
filled
it
up
with
SEF
nodes
and
put
those
adjacent
to
all
of
our
existing
racks.
So
they
were
still
the
same:
failure
to
mangle
using
the
same
switches
and
cross
cabling
the
networks
through
so
and
then
at
the
back
end
of
2017.
We
bought
another
rack,
and
this
was
our
proposal
for
future
expansion.
A
You
get
nine
four
you
boxes
in
Iraq,
along
with
the
switches
and
suchlike,
and
then
you
just
sort
of
plug
it
in
turn
it
on
and
away
you
go
so
as
of
now,
we
have
about
eighteen
petabytes
of
raw
capacity.
We're
used
about
40
percent
of
it.
Now
so
I
don't
see
us
buying
any
more
Hardware
this
year
and
that's
in
for
failure,
domains,
three
of
which
are
14
servers
and
one
that's
nine
servers.
A
A
We
could
be
quite
agile
about
how
we
may
change
to
our
production
cluster,
but
inevitably,
as
time
goes
by,
Steph
proved
itself
to
be
very
reliable,
so
people
started
building
websites
and
suchlike
on
the
top
of
our
production,
OpenStack
and
Seth,
and
that's
mint
they've
been
more
risk-averse
and
we
had
to
become
more
conservative.
So
we
bought
a
small
test
class.
That's
just
three
houses,
a
petabyte
of
raw
capacity,
and
that
lets
us
do
things
like
try
out
the
Luminess
upgrade
try
out
the
migration
to
blue
store.
A
That
sort
of
thing
in
an
environment
doesn't
disrupt
our
users,
our
last
bit
of
stuff.
Is
we
have
a
disaster
recovery
site
in
Slough
near
London,
and
we
have
again
three
hosts
one
petabyte
raw
capacity,
not
much
a
quick
show
of
hands
it,
who
is
at
least
passingly
familiar
with
ansible,
pretty
much
everyone
good?
So
I
won't
say
much
about
that
for
those
of
you
that
haven't
encountered
out
of
all
this
a
configuration
management
system,
it's
agentless,
there's
not
something
can
continually
running
on
the
target
server.
A
A
We
picked
surf
and
sport
at
Sanger
because
we
are
already
using
ansible
for
our
configuration
management,
and
so
we
were
going
to
essentially
we're
going
to
use
ansible
to
manage
our
surf
cluster,
and
then
we
discovered
that
someone
else
has
already
done
most
the
hard
work
for
us,
which
was
great
when
we
started
with
Stefan
swell
in
2016.
It
was
quite
a
young
project,
so
our
initial
deployment
was
basically
just.
We
took
a
branch
of
what
what
had
been
a
commit
on
the
master
branch
upstream
at
some
point
in
the
past.
A
Stefan's
wall
has
become
rather
more
mature.
In
the
meantime,
we
used
versions,
2.1
and
3.0
for
jewel
and
we're
now
basing
our
deployment
branch
of
3.2
for
our
luminous
clusters.
We
have
a
number
of
Sangha
specific
modifications
to
upstream
set
thanks
Paul,
partly
to
set
up
our
locals
our
local
things
so
that
bonding
I
alluded
to
earlier.
We
set
that
up
with
a
saying
of
prerequisites
role
along
with
and
configuring
our
local
DNS
and
email.
All
that
kind
of
thing.
A
Then
we
have
a
role
for
s3
user
management
because
we
don't
want
have
to
do
that.
The
helpdesk
have
a
thing
called
run
deck
which
gives
them
a
web
UI
to
administrative
tools
and
we've
written
a
playbook
that
that
essentially,
is
a
receiving
end
for
that,
so
the
run
deck
SSH
is
in
as
an
unprivileged
account
and
talks
to
our
server,
which
then
works
out.
A
What's
of
to
Ruggles
Gateway
admin
for
user
management
quota
adjustments
that
sort
of
thing
we
also
we
use,
let's
encrypt,
which
is
a
free
Ca
for
HTS
for
cheerless
certificates,
so
we
use
it
for
HTTP
for
are
rather
gateway
server.
Now,
let's
encrypt
is
free,
but
the
certificates
are
shortly
if
they
last
about
three
months,
so
you
have
to
automate
everything
to
do
with
that.
So
we
wrote
a
playbook.
A
We
burn
a
role
to
do
that,
using
seventh
of
all
it
automates,
generating
the
challenge
responses
getting
in
new
key
in
certificate
and
then
deploying
that
across
our
suite
of
servers.
If
they're
interested
in
that
might
want
think
mile
to
use
that
yourself,
does
it
get
hub
repository
with
the
URL
at
the
end
of
this
talk?
So
we'll
come
back
to
that.
A
A
Cluster
is
simply
a
matter
of
specifying
different
inventory
scripts
and
we
have
a
bunch
of
utility
functions
that
all
of
our
you
all
of
our
inventory
scripts
use,
which
means
we
can
reuse
our
code,
and
this
is
just
a
lot
of
the
broadest
outlines
or
sort
of
things
you
can
do
in
the
dynamic
inventory.
So
you
can
see
here
at
the
top.
We
specifies
our
hosts,
then
we
can
say
which
hosts
have
which
role.
So
here,
obviously
we're
specifying
the
monitor
nodes.
A
You
can
then
configure
particular
variables
you
want
for
the
services
you
need
so
here,
for
there
are
those
gateways
we're
specifying
the
interface
to
use
and
that's
one
VLAN
on
the
bonded
interface
I
talked
about
earlier.
Also
Stephan
support
lets
you
set
up,
pools
and
such
like.
So
here
we're
specifying
placement
group
number
and
the
name.
This
is
the
pool
use
for
the
data
for
the
rails
gateway
and
similarly,
you
can
specify
keys.
The
the
names
here
are
OpenStack
pools,
OpenStack
keys,
but
as
it
happens,
they
don't
need
to
be
open.
Snaps
specific.
A
A
We
have
a
few
bug
fixes
if
you
so
fanciful
upstream,
there's
odd
sort
of
picky
things,
we've
got
pushed
upstream
and
then
we
have
a
few
more
fixes
that
we
just
keep
locally
and
we
also
template
collecti,
because
we
use
quality
for
collecting
metrics
from
SEF
and
there's
a
set
plugin,
but
it
needs
to
be
told
individually
about
every
demon
on
every
host
which
is
tiresome
to
maintain.
So
we
do
that
with
Stefan's
ball
as
we
configure
each
node,
we
also
configure
collecti,
which
brings
me
nicely
onto
metrics
and
monitoring.
A
Hopefully
you
can
see
that
roughly
well
this.
This
is
our
dashboard.
This
is
a
graph
on
a
dashboard
we've
had
it
the
same
for
a
number
of
years
now,
when
I
first
produced
this
slide,
we've
used
about
5%
of
our
cluster.
Now
we've
used
40%,
and
this
is
what
we
use
as
an
at-a-glance
idea
of
the
state
of
our
cluster.
So
we've
got
at
the
top
here:
the
number
of
OS
DS
that
are
up,
and
in
and
3,060
that's
all
of
them
the
monitor
quorum,
how
much
space
we've
used
and
all
of
these
are
color-coded.
A
So
if
you
lose
one
or
two
OSDs
it'll
go
orange
because
that's
not
something
you
need
to
worry
about.
If
you,
if
you've
lost
more
than
the
hosts
worth,
these
will
start
to
go
red
because
that
that
that's
a
sign
that
you
need
to
stop
what
you're
doing
and
go
and
look
at
it.
We
then
have
the
throughput
of
the
cluster
on
the
left-hand
side
here,
the
client
throughput
on
the
right-hand
side
going
through
the
rivals,
gateway
and
then
a
bunch
of
load,
metrics,
so
CPU
the
load
network,
traffic
swap
and
then
at
the
bottom.
A
To
give
you
an
idea
of
the
volume
of
metrics
with
Joule
you
get
about
19,000
metrics
per
server.
Aluminous,
that's
gone
up
quite
a
lot.
A
server,
that's
just
running
OS
T's
in
luminance,
is
producing
about
25,000
metrics
and
for
one
that's
running
the
rails
gateway
as
well,
that's
more
like
28,000
metrics,
and
we
found
that
as
we
migrated
up
for
adoption
cluster
from
to
luminous,
that's
quite
a
significant
extra
volume
of
metrics
and
our
metric
service
starts
to
struggle
a
bit.
A
So
that's
perhaps
something
to
be
aware
of,
as
you
migrate
to
luminous,
that
the
sheer
volume
of
metrics
you
can
collect
from
a
safe
cluster
gets
pretty
big
if
you're
in
the
previous
talk
about
looking
after
your
self
cluster,
your
eyes
that
metrics
are
only
one
part
of
looking
at
the
health
of
a
cluster,
you
also
need
to
monitor
it.
Our
infrastructure
colleagues
running
a
Geass
set
up,
so
we
use
that
and
as
well
as
looking
at
server
uptime,
that
sort
of
thing
we
also
have
some
SEF
specific
monitoring.
A
So
Nagios
is
putting
objects
into
a
pool
and
then
get
again
just
and
also
putting
things
into
s3
and
retrieve
them
from
s3.
So
that
gives
us
not
only
or
all
the
server's
up,
but
the
cluster
is
functioning
in
good
order
and
that's
quite
useful
as
a
sort
of
early.
Often
if
the
cluster
is
becoming
stressed
and
response
time
decides
to
degrade,
naggy
us
will
give
us
a
warning
before
I
used
to
start
complaining,
which
is
nice
because
always
good.
A
If
your
users
come
to
you
and
say,
is
there
a
problem
with
surf
to
say
yes
and
we're
working
on
it
rather
than
to
sort
of
look
rabbit
in
the
headlights?
Oh
dear,
is
something
wrong.
The
last
bit
of
monitoring
is
we
have
a
script
that
runs
nightly
to
check
whether
there
are
any
OSDs
down
and
then
that
emails,
our
hardware
team,
who
can
then
manage
replacing
the
discs,
that's
something
we
do
not
need,
because
a
failed
disk
just
happens.
It's
not
something.
A
There's
it
needs
addressing
urgently
when
we
started
with
SEF
our
DC
team
colleagues
were
a
bit
wary
of.
Its
F
does
have
this
reputation
of
being
complicated
and
they
were
quite
unsure
as
to
how
to
deal
with
failing
disks
and
the
initial
thing.
I
sent
them
a
long
checklist
and
they
were
like
no.
This
is
too
complicated.
We
need
something
more
straightforward,
so
we
have
a
script
that
they
can
now
run,
and
this
is
the
sort
of
output
you
get
so
you're
writing
on
OSD,
and
it
tells
you
which
disk
gets
on.
A
It
looks
at
the
smart
status
of
the
drive
you
can
see
here.
The
drive
is
about
to
fail
if
the
smart
health
check
is
okay,
which
we
often
find
it
will
be
with
a
drive
umhum
when
the
drive
is
well
on
its
way
out
and
we've
seen
lots
of
medium
errors.
It
will
then
go
through
the
kernel
log
and
see
if
there
any
error
messages
there,
and
then
it
tells
you
what
you
need
to
know
if
you're
going
to
replace
that
drive.
A
So
the
serial
number
is
always
good
because
it's
not
always
entirely
reliable,
which
drive
bay
appear
at
switch
LUN.
Unfortunately,
these
super
micro
boxes,
which
can
be
a
bit
of
a
bore,
so
we
always
provide
the
serial
number
to
give
data
center.
Colleagues,
the
obvious
thing
to
check
pull
the
drive
out.
Is
it
the
right
serial
number?
Yes
great,
if
not
plug
it
back
in
several
sort
it
out?
Finally,
we
blink
the
light
on
the
drive
bay
and
tell
you
which
journal
partition.
A
The
journal
was
on
because
we're
going
to
rip
we're
going
to
rebuild
that
OSD
with
a
fresh
drive,
you
need
to
use
the
same
partition
and
then
we
put
up
this
little
ASCII
art
map
you
ever
open.
One,
those
for
you
super
micro
servers.
This
is
how
the
60
bays
are
laid
out
and
basically
x
marks.
The
spot
tells
you
which
drives
are
poor
now
I
understand
from
the
keynote
this
morning
that
eventually
in
Nautilus
or
maybe
octopus
SEF-
will
do
all
of
this.
A
For
you
in
the
meantime,
we've
written
a
bunch
of
scripts
that
do
it,
and
this
makes
our
DC
colleagues
life
much
easier.
They
know
which
drive
to
pull
there'll
be
a
light
on
it
and
they
can
check
the
serial
number
now.
A
failure
mode
you
may
have
seen
and
with
drives
are
the
failing
as
they
appear
to
vanish
from
the
house.
A
The
device
node
entry
disappears,
the
colon
says:
I
can't
find
the
drive
anymore
and
obviously
then
you
can't
turn
its
light
on,
because
there
isn't
a
device
know
to
associate
it
with
it
and
they'll
script.
In
that
case
lights
up
everything
else,
so
you
then
say
to
the
operator
well,
pull
out
the
drive
that
doesn't
have
a
light
on.
That's
probably
the
one
you
want.
All
of
this
stuff
is
quite
specific
to
these
super
micro
boxes,
though
the
bit
that
knows
about
the
the
drive
layout
is
a
function
you
could
pull
out
and
replace.
A
So
this
sort
of
code
is
also
available
on
github
and
they'll,
be
a
URL
at
the
end.
If
you
want
to
download
it
I,
don't
really
believe
benchmarks,
but
everyone
seems
to
like
them.
If
you
go
to
all
of
our
all
the
vendors
out
out
there,
they
will
have
benchmarks.
I
think
you
should
generally
chain
them
with
at
least
a
pinch
of
salt.
That
said,
we
did
some
benchmarking
over
Christmas
and
our
cluster
was
quiet.
A
This
was
when
we
are
still
running
jewel
and
file
store,
and
this
is
just
using
the
rattles
and
bench
utility
that
comes
with
SEF
running
it
on
some
of
the
cluster
nodes
themselves.
So
taking
advantage
of
the
available
networking
within
the
cluster,
and
you
can
see
with
the
R
and
Reed
benchmark,
we
got
just
over
a
hundred
gigabytes.
A
second
which
is
you
know,
that's
pretty
good,
going
similarly
sequential,
read
about
90
gigabytes
a
second
and
with
write
about
tanking
abouts,
a
second.
We
did
also
benchmark
through
the
s3
service
using
cold
bench.
A
A
But
having
done
all
of
this,
once
we
finished
with
our
luminous
and
blue
star
upgrade
I
imagine
we
will
run
these
again
and
maybe
I'll
come
back
to
the
next
safe
conference
and
say:
look
the
numbers
went
up,
so
that's
benchmarking,
I,
wouldn't
I,
wouldn't
get
too
hung
up
on
it.
Inevitably,
it's
going
to
be
an
artificial
environment,
so
the
migration
migrating
from
jewel
to
luminous,
there's
not
much
to
say
about
it.
Safe
and
small
has
a
rolling
upgrade
playbook,
which
does
all
of
the
things
in
the
release
notes.
A
So
you
don't
have
to
worry
about.
Did
I
remember
afterwards
to
set
the
right
flags,
because
the
PlayBook
just
does
it
for
you.
It
was
really
easy
and
we
did
it
on
our
test
clusters
first,
obviously,
but
when
it
came
to
the
production
cluster,
we
updated
our
inventory's
script
to
know
about
their
manager,
nodes,
rancer
fonts
ball,
with
the
rolling
upgrade
playbook.
When
had
a
cup
of
coffee
came
back,
it
was
all
done,
brilliant,
less
straightforward.
Is
the
migration
from
file
store
to
blue
store
because
essentially,
there's
no
two
ways
about
it.
A
You
have
to
take
all
of
the
data
on
the
given
OSD,
throw
it
away
and
rewrite
it,
and
we
do
that.
We
wrote
script
to
do
this.
It
runs
off
a
monitor
node.
Essentially
it
logs
in
it
waits
the
class
to
be
in
health.
Ok,
because
you
know
you
don't
start
mucking
around
your
OS
DS,
when
you
don't
have
to.
If
the
cluster
isn't
happy,
then
it
logs
on
to
a
net
to
a
target
machine
and
for
each
OSD
on
that
machine.
It
stops
it
marks
it
out.
A
The
cluster
wipes
the
file
system
and
then
remakes.
It's
using
serf
volume
LVM
create
to
make
a
blue
store
OSD
and
it
uses
the
same
hard
drive
for
data
and
nvme
partition
for
block
DB,
as
was
used
for
the
false
diversity,
and
this
is
a
graph
of
the
state
of
Ceph
objects.
As
that
migration
progresses
yeah,
you
can't
see
it
so.
This
orange
line
is
the
number
of
degraded
objects.
The
blue
line.
It's
a
number
of
misplaced
objects.
You
can
see
here
time
along
the
bottom.
A
This
is
where
it
starts
on
the
node,
so
you
get
an
enormous
spike
in
both
of
these
because
you've
just
taken
it
essentially
you've
taken
the
Machine.
Now
if
the
cluster
and
sold
safe
that
one's
gone
away,
and
then
you
have
make
a
whole
new
part
of
er
STIs
and
it
has
to
backfill
all
of
them
as
I
alluded
to
where
cluster
is
about
40
percent
full.
So
that
means
you're
writing
to
in
a
bit
terabytes
to
every
single
OSD,
and
that
just
takes
time
and
the
cluster
eventually
rebalances.
A
A
This
process
is
quite
IO
intensive
and,
if
you're
looking
at
this
graph
on
wondering
what
this
spike
was,
that
was
another
LSD
failing
in
an
unrelated
note,
we
found,
as
we've
moved
through
this
migration,
that
a
number
of
disks
have
failed,
which
disk
failure
happens.
You
just
swap
them
and
carry
on
to
give
you
an
idea
of
the
time
scale.
This
process
from
remaking
a
host
worth
of
SDS
the
cost
of
being
happy
again
takes
about
30
hours,
so
we
have
51
OSDs.
A
A
Unfortunately,
this
is
currently
stalled
because
our
cluster
has
been
stuck
in
health,
warned
for
a
while
with
a
large
OMAP
object.
If
you
run
luminous
or
if
you've
watched
the
Ceph
users
mailing
list
you'll
see
a
lot
of
people
talking
about
large
OMAP
objects,
we've
got
one
that
appears
to
be
associated
with
a
bucket
in
s3
that
no
longer
exists,
so
we're
still
working
with
canonical
to
try
and
figure
out.
A
Why
that
happened
and
basically
how
we
can
get
rid
of
it,
so
we
can
get
our
migration
going
again,
which
brings
me
on
to
the
bit
where
I
rant
I
well,
my
colleagues
will
tell
you:
I
could
have
filled
this
talk
with
rants,
but
I'm.
Just
just
a
few
sort
of
pain
points
we've
found
particularly
they've,
mostly
been
around
the
Randalls
gateway
service.
All
the
migrations
are
luminous.
A
So
when
you
migrate
to
luminous-
and
suddenly
you
get
this
thing
that
you
have
Haitians
on
your
pools-
and
you
can
do
that
fine,
but
from
a
loop
from
administering
a
luminous
cluster
this
this
application,
you
have
to
certain
the
pool-
is
just
essentially
an
arbitrary
string.
It's
not
very
useful
I,
don't
know
whether
that's
going
to
become
key
to
managing
the
cluster
and
lace
releases.
But
from
our
point
of
view
this
is
something
you
have
to
do
in
luminous.
A
Otherwise
your
clusters
and
health
worn
and
we
don't
seem
to
get
any
benefit
from
it
and
I've
alluded
to
large
OMAP
objects.
This
is
a
standard
thing,
you'll
get
when
you
upgrade
from
joules
luminous,
because
the
Ruggles
gateway
and
luminous
can
recharge.
Your
buckets,
but
if
it
hasn't
recharged,
your
buckets,
you
get
these
large
Omak
objects
which
corresponds
to
the
index
of
large
buckets
and,
as
I
say,
the
round
escape.
We
can
restart
them
for
you,
but
it
doesn't.
A
You
have
to
wait
for
users
to
put
objects
or
remove
objects
from
the
buckets
and
then
the
dynamic
recharging
will
come
long
as
they.
Oh
yes,
this
bucket.
It
needs
to
be
shutting
I'll
go
ahead
and
do
that
which
is
all
well
and
good,
but
if
you've
got
a
lot
of
large
buckets,
you've
just
done
this
migration
and
you
want
your
cluster
back
into
health.
A
Ok,
you
have
to
go
and
do
it
by
hand
which
means-
and
you
can't
just
say
to
the
Riedel
Skateway-
oh
that
bucket
rashard
it
now
you
have
to
work
out
how
many
shards
it
should
have,
which
is
not
hard.
You
find
the
calculation
the
source
code,
you
do
it
away.
You
go
it's
just
one
of
those
things
that
feels
like
it
could
have
been
done
better
and,
as
an
admins
point
of
view,
this
is
a
whole
pile
of
things.
I
had
to
do
by
hand
and
I.
A
A
Another
problem
we've
had
is
that
we
started
off
everything
through
a
replicated
because
we
were
new
to
safe
through
your
application
was
easy
in
luminous.
Erasure
coding
is
much
better.
It's
a
much
more
performance,
we've
done
some
benchmarking
on
our
test
cluster
and
we
can
get
equivalent
or
maybe
slightly
better
forms
out
of
erasure
coded
pools.
We
could
for
replicated
ones,
but
we
now
have
a
lot
of
data.
Several
petabytes
of
data
in
replicated
pools,
backing
s3
and
ok.
A
You
can
say
to
the
ruggles
gateways,
so
these
users
make
the
default
call
now
be
this
erasure,
coded
pool
and
as
new
things
that
are
bloated
to
your
cluster.
Great
they'll
be
put
in
the
erasure,
coded
pool,
but
there's
no
way
of
taking
objects
out
the
old
replicated
pool
and
migrating
them
and
okay.
A
You
can
do
it
by
hand,
but
you
have
to
make
sure
you
get
all
the
ACLs
right
and
you
need
to
coordinate
that
with
your
users,
who
might
be
doing
other
things
to
their
buckets,
and
we
really
want
the
sort
of
administrative
tool
here
to
say
both
how
much
stuff
will.
How
much
should
we
still
build
on
our
old
replicated
pools,
and
can
we
move
some
of
that
char
erasure
code?
A
It
pulls
automatically
rather
than
having
to
get
on
either
get
I'll
use
this
to
do
it
or
find
some
way
of
automating
it
and
be
sure
that
we'll
get
it
right
and-
and
that
brings
me
on
to
another
problem
again-
you'll
have
heard
in
the
keynote
North
us
will
be
able
to
shrink
pools.
But
at
the
moment
we
can't
we're
running
luminous,
we're
not
gonna,
be
in
Northwest
anytime
soon,
and
we
have
a
large
replicated
pool
for
the
Rather's
gateway
which
eventually
we're
going
to
want
to
shrink.
As
we
move
to
arranger
coding.
A
We
can't
do
that.
Similarly,
we
have
multiple
open
stacks,
we'll
try
a
new
release,
we'll
use
it
for
pilot
once
we're
happy,
we'll
start
moving
users
onto
it
and
that's
great.
As
we
move
users
on
to
a
new
OpenStack,
we
can
grow
the
pools
backing
that
OpenStack
great,
but
we're
also
migrating
users
off
the
old
OpenStack
we're
clearing
hypervisors
out
of
the
old
OpenStack,
and
we
can't
do
anything
to
shrink
that
pool.
A
We
have
to
wait
until
everything
is
off
and
then
we
can
blow
the
entire
lot
away
and
again,
they'll
be
that'll,
be
great
and
Nautilus,
but
realistically
we're
not
gonna
be
running
north
list
for
another
couple
of
years,
and
we
have
to
put
up
with
you
saw
things
in
the
meantime.
Our
final
pain
point
is
about
deployment.
I
said
we
use
safe
ansible,
which,
on
the
whole,
we're
pretty
happy
with,
but
its
weakness
is
provisioning,
OS
DS!
You
only
have
to
do
it
all
by
hand
so
specify
this
drive
this
nvme
partition.
A
This
drive
this
nvme
partition,
which
okay
you
can
do
that
and
in
dual
you
set.
You
specify
that
Andy
would
set
them
up
for
you
make
the
partitions
in
luminous.
It
doesn't
do
that
anymore.
It
wants
to
use
serf
volume,
LVM
batch,
which
is
fine.
If
what
you
want
to
do
is
to
take
all
of
your
nvme
cards
and
turn
them
into
one
LVM
volume,
but
we
don't.
We
want
to
keep
them
separate.
A
So
if
one
of
them
fails,
we
lose
half
of
machines
worth
of
IDs,
so
we're
back
to
having
to
do
it
by
hand,
and
this
is
just
fiddly,
and
this
is
all
the
thing
that
I
want
my
configuration
management
system
to
take
care
of,
for
me
not
have
to
do
myself.
So
that's
something
I
would
like
to
be
better
and
also
when
the
disk
fails,
you
replace
it.
Sometimes
it
will
come
back
as
a
different
device.
A
You've
probably
seen
this,
you
replace
the
disk,
it
was
SDA,
you
put
it,
you
put
another
drive
in
it's
s,
DCI
or
something,
and
then
Stefan
small
will
come
along
and
say
well,
I
can't
do
this
machine
because
we're
missing
SDA
and
those
sort
of
things
they're
just
it
feels
like
you
ought
to
be
able
to
automate
that
entirely
and
just
say.
Well,
there
are
six
to
your
spinning
disks
in
this
box.
A
Use
them
we're
not
there
yet,
lest
you
think,
I'm
going
to
route
for
my
remaining
time
and
we're
actually
pretty
happy
with
Seth
it
scales
out
really
well
the
last
time
we
wanted
to
expand
our
safe
cluster.
As
I
said,
we
took
nine
boxes,
put
them
in
Iraq
and
from
them
arriving
on
site,
so
literally
coming
through
the
door
of
a
datacenter
to
being
fully
integrated
in
the
classic
four
working
days,
which
is
pretty
good.
A
If
you
look
at
the,
if
you
compare
that
to
the
time
to
Commission
a
new
lustre
file
system
last
time,
we
did
that.
That
was
several
weeks
and
if
several
intensive
weeks,
whereas
this
it
really
was
plugged
them
in
till
our
phase
server,
which
we
used
for
the
operating
system
install,
this
is
what
they
are
they're
going
to
be
Steph
not
bodes
turn
them
on.
Let
them
install
run
so
thanks,
Paul
go
and
drink
coffee
for
several
days.
Come
back.
It's
all
done,
that's
really
good!
It's
really
makes
expansion
so
easy.
A
Also,
Seth
has
been
really
reliable.
Hardware
fails
all
the
time
and
safe
just
carries
on.
We
have
now
four
phaser
mains
and
we
lost
two
racks
at
one
point,
there
was
an
unfortunate
incident
from
the
DC.
The
circuit
breakers.
Two
of
our
racks
were
tripped.
Just
like
that
whole
thing
goes:
dark
safe,
then,
obviously,
wouldn't
write
any
data
because
it
had
lost
two
of
its
four
failing
domains,
but
when
we
got
the
power
restored
and
the
machines
came
back
up
and
they
sought
their
file
systems
out
safe,
just
sorted
itself
out,
we
had.
A
Maybe
you
know
a
couple
of
dozen
placement
groups
are
needed,
a
little
bit
of
hand-holding,
but
essentially
it
just
went
back
to
running
and
that
sort
of
robustness
has
been
really
good
for
us,
because,
inevitably
things
go
wrong
from
time
to
time.
The
network
fails
drives
fail,
power,
fails,
cooling,
fails.
All
of
these
things
happen,
and
sometimes
quite
excitingly,
but
Seth
just
keeps
going,
keep
servant
and
that's
something
as
an
admin
who
has
a
lot
of
things:
I'm
responsible
for
I,
really
like
about
self
it's
pretty
much
bulletproof.
It's
really
easy
to
expand.
A
Yeah,
it's
great
and
I.
Think
that's
about
all
I
was
going
to
say:
I
alluded
to
earlier,
our
self
ansible
playbooks
for
using
let's
encrypt
your
routers
gateway,
that's
github,
URL
for
that
our
SEF
disk
utilities,
another
github
URL
there
at
the
moment,
they're
still
our
jewel
and
file
store
bastes
scripts.
We
have
got
scripts
for
luminous
of
blue
store,
they're,
not
sure
they're
cool
ready
for
public
release.
If
you
want
an
advance
copy
drop
me
an
e-mail
log,
let
you
have
them
on
d.
If
it
breaks,
you
can
keep
all
the
pieces
basis.
A
We
use,
but
no
there's
a
there's
an
existing
collar.
It
was
a
question
about
collecting
modules.
There's
an
existing
Ceph
plugin
for
quality.
You
have
to
configure
it
about
every
single
demon
on
the
host
which
we
do
using
our
ansible
playbooks.
It's
a
template
file,
but
it's
just
the
straightforward.
You
need
to
get
the
right
version,
there's
a
different
version
for
luminous
than
for
jewel.
It's
not
compatible
they're,
not
into
compatible,
but
there
is
now
a
package
for
Ubuntu
Barnack
that
provides
the
luminous
quality
module.
A
A
How
do
we
manage
the
ansible
Playworks,
so
they're,
basically
they're
all
and
get
when
we
want
to
move
our
version?
Our
branch
to
a
new
upstream
version,
there's
a
bit
of
fiddly,
get
shenanigans
which
isn't
very
interesting.
Basically,
we
merge
the
previous
branch
we
were
working
off
with
the
new
upstream
is
one
merge,
and
then
we
merge
our
deployment
branch
on
that
on
top
of
the
result
of
that,
and
then
we
have
a
test
class
that
we
so
we
can
test
it
both.
Does
it
broadly
work,
but
the
main
thing
you'll
see
change.
A
If
you
update
safe
hands,
was
your
SEF
token
for
change?
So
when
you
run
SEF
install,
you
can
say,
don't
restart
any
demons
and
what
that
lets
you
do
is
you
can
run
it
then
look
at
how
SEF
confers
changed
and
say
well
are
these?
Am
I
okay
with
these
changes?
Am
I,
not
okay,
with
these
changes
and
if
I'm,
not
okay.
With
these
changes,
you
can
just
back
out
to
the
previous
version
and
nothing
has
been
restarted.
So
that's
all
fine.
So
that's
what
we
do.
A
We
test
it
in
our
test
cluster
when
we're
coming
to
do
it
on
the
production
cluster,
we
say:
don't
restart
anything
and
just
eyeball
against
F
tokens
to
make
sure
we
understand
all
the
changes
that
are
being
made
and
again
we
have
a
different
inventory
file
for
each
play
book
each
cluster.
The
lets
us
do
those
independently
of
each
other
question
here.
A
It's
a
question
about
sefa
fest:
do
we
use
it
short
answer?
No,
when
we
started
with
jewel,
it
was
still
a
bit
sort
of
technology.
Preview,
I,
understand
it's
much
more
robust
and
luminous
and
I.
Think
the
next
OpenStack
release
we'll
be
looking
at.
Can
we
use
Manila
two
provisions?
Effervesce
file
systems
for
clients
sort
of
transparently,
so
that's
sort
of
our
roadmap
for
surface
the
people?
Are
certain
have
been
using
it
quite
extensively
and
seem
quite
happy
with
it,
so
we'll
be
looking
at
it
with
on?
A
There's
a
question
about
bringing
a
manually
deployed
cluster
in
Stefan's
ball:
we've
not
tried
it.
There
is.
There
is
a
playbook
as
though,
as
well
as
their
main.
The
other
thing
you'll
run
all
the
time.
So
fan
support
comes
with
a
number
of
infrastructure
play
books,
one
of
which
is
taking
over
an
existing
cluster,
and
people
have
definitely
done
it.
I've
seen
and
threats
about
it,
I've
not
tried
it
because
we
started
off
using
so
fancy
balls.
So
we've
not
had
to
try
that,
but
it
is.
A
A
There's
a
question
about
how
large
our
team
is
and
how
we
split
the
work.
There
are
eight
of
us
in
the
team
that
manages
the
LSF
clusters,
the
open
stack
installs,
the
last
two
file
systems,
the
Arabs
file
systems,
safe
and
use
the
sort
of
second
or
third
line
UNIX
support,
which
what
we
try
and
do
is.
We
will
have
people
working
on
particular
projects
and
then
also
dealing
with
user
tickets.
A
The
theory
is
if
you're
I
mostly
do
safe,
but
while
I'm
on
when
I
get
the
ticket,
rotor
I
then
have
to
cover
all
of
our
systems.
So
I
spend
a
guess
about
three
quarters
of
my
time
on
safe
at
the
moment
and
that's
looking
after
all
three
clusters,
but
there'll
be
other
times
when
I
do
less
safe
and
more
LSF
say,
and
we
try
and
mix
it
around
the
bits
that
we
don't
just
have.
Particularly.
A
We
don't
want
a
system
that
only
one
person
knows
about
so,
for
instance,
I'm
here
in
Barcelona
and
I've
got
a
colleague
back
home
looking
after
the
upgrades
from
files,
thought
blue
store
and
he's
been
lazing
with
canonical
about
that
issue.
With
the
larger
map
object
that
I
talked
about,
so
it's
not
just
me,
but
I'm
primarily
do
safe.
Does
that
sort
of
answer
your
question.
A
A
A
Okay,
it's
a
question
about
replacing
our
STIs.
Do
we
replace
a
nasty
the
minute
it
fails?
Do
we
wait
for
a
few
OSDs
to
fail
at
once?
The
answer
is
in
theory.
We
replace
the
nurse
tea
as
soon
as
it
fails,
because
we
don't
have
to
take
the
hose
down.
You
can
pull
it
out
the
rack
open
it
up,
pull
the
drive
out,
replace
it
plugging
the
hard
drive
in
push
it
back
in
and
away.
You
go
in
practice
because
it's
another
team
doing
it.
A
It
tends
to
be
when
they
have
some
time,
they'll
go
and
do
them.
So
at
the
moment,
the
person
who
generally
does
them
is
off
sick.
So
we've
got
a
backlog
of
five
or
six
that
need
doing,
but
in
theory
no
it's
just
as
they
fail,
we'll
replace
them
and
if
you're
in
the
previous
talk
and
the
speaker
talked
about
you'll,
start
to
see
scrub
errors.
A
So
sometimes
when
we
we
start
getting
back
to
scrub
errors,
we'll
look
for
a
common
OSD
and
then
go
and
look
hard
at
that
one,
you
know:
was
it
smart
status?
Are
we
seeing
medium
errors
in
Cal,
Kern
log
and
then
we'll
actually
replace
it
before
it's
failed
completely?
It
started
school
scrub
errors.
Let's
get
rid
of
it
now,
because
replacing
an
OST
is
very
lightweight
it.