►
From YouTube: 200 Clusters vs 1 Admin - Bartosz Rabiega, OVH
Description
200 Clusters vs 1 Admin - Bartosz Rabiega, OVH
Bartosz will explain how and why multi ceph cluster ecosystem was developed at OVH.
Why is it good to have multiple ceph clusters?
How a single member of ceph as a service team at OVH can keep an eye and take care of 200 ceph clusters?
These questions and more will be addressed by Bartosz.
About Bartosz Rabiega
OVH
DevOps Engineer
IT enthusiast...
IT professional since 2011 as QA Specialist, System Analyst and Software Developer. Since 2016 working as DevOps engineer in Ceph as a Service team at OVH (Poland).
Develops and takes daily care of Ceph as a Service project providing 200 ceph clusters.
Passionate Python developer.
Hardware and Software enthusiast.
Likes to understand high and low level designs.
Data visualization - let's do more!
A
Hi,
my
name
is
bartosz
la
Vega
I
am
from
OVH
Poland
and
today
I'd
like
to
tell
you
a
story
about
SEF
at
OVH.
What
do
we
do
is
surf?
How
do
we
maintain
it?
How
do
we
manage
it?
How
do
we
deploy
it
and
also,
as
a
consequence,
how
is
it
possible
that
a
single
system
administrator
is
able
to
maintain
more
than
200
safe
clusters?
A
A
few
words
about
me
so
I've
been
working
for
OVH
since
2016
as
a
DevOps
engineer
and
I'm
a
part
of
safe
as
a
service
team.
What
we
are
doing
is
that
we
are
basically
providing
managed
safe
clusters
for
different
of
each
purposes,
for
internal
and
for
external
customers,
but
I
will
tell
you
more
about
it
in
a
moment.
In
the
past,
I
was
working
as
a
software
developer
as
I
think
you
a
specialist
and
also
as
a
system
analyst
now
I'm
a
DevOps
engineer.
A
A
We
have
26
data
centers
located
all
over
the
world
and,
what's
so
special
about
the
village
is
that
we
are
doing
water,
cooling
of
our
server
equipment,
and
let
me
tell
you
some
numbers
comparing
to
traditional
air
truth
data
centers.
We
are
able
to
save
up
to
40%
energy
thanks
to
the
air
crew.
Thanks
to
the
water
cooling,
we
have
13
terabit
network
capacity
at
the
moment,
and
all
of
that
is
behind
anti-ddos
protection
liar.
Our
current
server
capacity
is
about
1
million
dedicated
servers.
B
A
What
for
do
we
use
safe
at
OVH,
so
public,
so
eh
is
providing
a
service
called
public
cloud,
it's
based
on
of
OVH
dedicated
servers
and
OpenStack
technology,
and
we
needed
to
provide
a
block
storage
for
additional
discs
for
OpenStack,
so
additional
volumes
for
the
public
cloud.
Basically,
so
we
are
our
main
use
case
is
our
BD
force
F,
and
besides
that,
we
have
also
some
internal
teams
which
are
using
self
as
a
database
storage
liar.
Besides
that
we
also
have
a
product
for
our
external
customers.
A
A
A
So
this
cut
those
clusters
have
different
sizes,
starting
from
small
clusters,
free
or
SD
clusters
up
to
more
big
clusters
like
800
SD
clusters,
that's
our
biggest
cluster
at
the
moment,
I
think
and
just
to
sum
up,
we
have
more
than
20,000
OS,
DS
and
650
monitors.
So
that's
quite
a
lot,
and
the
picture
here
is.
C
A
A
Next,
we
wanted
to
have
maximise
performance
and
minimized
costs,
but
I
guess
everyone
wants
that.
Besides,
that
we
wanted
to
have
a
good
control
over
the
hardware
resources
and
also
we
wanted
to
have
an
easy
deployment
of
our
storage
solution.
Easy
gross,
easy
upgrades,
basically
easy
management
of
the
whole
storage
solution,
so
the
first
point
is
quite
easy
to
deliver
thanks
to
Seth,
it
has
no
single
point
of
failure
due
to
its
architecture.
Now,
the
only
thing
we
need
to
do
is
to
provide
no
single
point
of
failure
at
the
DC
topology.
A
So
this
is
a
rather
typical
setup
for
Seth.
We
have
three
different
costs
located
in
three
different
tracks
and
those
tracks
are
connected
together
through
top-of-rack
switches.
So
we
do
a
replication
factor
of
three
and
all
the
data
is
replicated
into
three
different
hosts
in
three
different
tracks.
So
the
no
single
point
of
failure
is
rather
simple.
In
our
case,
if
our
entire
rack
fails,
your
data
is
still
available
and
there
is
no
service
interruption,
so
we
can
cross
the
first
point
out
of
the
list.
A
A
So
flash
cash
provides
kind
of
a
software
driven
hybrid
drive,
so
it
allows
you
to
build
software
as
SSD
accelerated
drive
out
of
the
SSD
and
HDD
drive.
So
what
we
do
at
OVH
is
that
we
take
an
nvme
drive.
We
do
a
multiple
type
partitions
of
that
an
nvme
drive
and
we
take
an
entire
HDD
drive
and
we
combine
those
2d
vices
to
create
a
flash
cash
device.
A
Now
that
flash
gives
device
is
exposed
as
a
block
device
for
the
Linux
kernel
and
we
pass
that
block
device
as
a
data
block
device
for
our
OSD
and
we
take
a
second
partition
for
the
journaling
of
our
is
OSD,
because
we
are
still
using
file
store.
In
most
cases,
we
are
in
the
middle
of
migration
to
newer
versions
of
safe
and
booster
as
well.
A
So
so
we
are
aware
of
the
numbers
in
a
generic
cloud
based
workloads,
IO
workloads.
We
are
able
to
gain
50
percent
of
IO
performance
thanks
to
the
flash
cache
and
an
nvme
accelerated
HDD
drives,
and
it's
also
good
for
the
spiking
IO.
When
you
have
a
bursting
kayo,
it
will
probably
fit
into
caching
part
of
your
blog
device.
So
we
are,
you
are
handing
it
in
a
better
way
than
they
are
or
Denari
HDD
okay.
So
this
is
how
we've
back
in
2015,
maximized
costs
and
minimized.
A
Sorry
maximize
performance
and
minimize
costs.
Sorry
guys!
Ok,
now,
how
do
we?
How
do
we
do
a
proper
resource
separation
and
how
do
we
have
a
fine-grained
control
over
the
resource
allocation?
From
the
network
point
of
view?
This
is
quite
simple:
we
use
a
Mac
villain
technology
in
order
to
provide
different
networks
for
different
purposes,
so
we
have
a
two
network
interfaces
provided
by
macmillan
and
those
two
network
interfaces
like
are
connected
to
us
DS.
A
So
one
of
the
interface
provides
public
network,
it's
in
a
separate
villain,
while
the
second
one
provides
private
networking
for
SD
replication
and
it's
a
completely
separate
villain
and
all
of
those
interfaces
are
connected
into
a
single
40,
gigabit
network
card
and
interconnected
together.
So
this
was
for
the
network
part
now
how
we
have
a
fine-grained
control
over
the
RAM
allocation
or
CPU
allocation,
and
the
answer
also
is
quite
simple:
there
are
technologies
to
provide
that
and
that
technology
is
called
containers,
so
we
use
containers
back
in
2015.
A
A
Once
you
have
that
image,
you
only
put
a
set
configuration
there
with
some
basic
data
like
keyrings,
and
you
start
that
container
and
voila.
You
have
your
OSD
running
in
a
great
shortcut,
but
this
is
basically
how
it
works
and
you
can
do
the
same
for
the
upgrades.
You
just
prepare
self
image
system
image
with
pre-installed
newer
version
of
safe.
You
should
dump
the
container
and
you
start
the
container
with
newer
subversion,
passing
the
correct
block
devices
into
that
container.
So
our
whole
story
looks
like
this.
A
We
have
a
whole
SD
each
in
separate
container
and
we
pass
to
that
container
to
block
devices
one
for
the
journal
and
the
second
one
is
a
flash
cache
for
the
data
block
device
and
also
we
pass
to
network
interfaces
provided
by
Mac
villain
for
the
networking
for
the
public
network
and
for
the
private
network.
As
for
the
containerization
engines,
we
are
using
docker
and
legs
d.
Both
of
them
has
proven
to
be
a
solid
piece
of
technology.
A
We
are
not
using
kubernetes,
yet
we
are
still
evaluating
the
our
future
of
our
projects,
but
at
the
moment
we
are
using
docker
and
legs
D,
and
these
are
solid
technologies.
So,
at
the
coast
level
the
whole
story
looks
like
this.
We
have
multiple
containers
are
running
multiple,
as
these
like
each
container
has
a
single
SD
and
we
are
able
to
run
multiple
safe
clusters
in
multiple
sub
versions
on
a
single
host.
Now,
there's
a
huge
benefit
comes
from
that
we
are
able
to
do
the
upgrades
OSD
by
OSD.
A
So
we
have
our
design
goals
fulfilled
thanks
to
the
the
whole
set
up.
Now,
as
I
said,
we
have
26
data
centers
and
we
have
multiple
public
cloud
regions.
So,
typically,
when
you
have
to
provide
multiple
regions,
you
will
set
up
multiple
safe
clusters,
safe
cluster
per
each
region.
In
order
to
provide
the
storage
for
each
region,
while
this
is
a
good
approach,
but
it
has
its
drawbacks
and
what
we
are
doing
at
OVH,
that
is
that
we
are
running
multiple,
safe
clusters
per
each
region.
A
Let
me
tell
you
why,
and
the
answer
is
here
when
you
have
a
safe
failure,
I'm
not
saying
surface
in
perfect,
it's
perfect,
but
you
know
there
are
miss
configurations.
There
are
hardware
failures,
there
are
human
errors
and
you
really
don't
want
to
bring
down
your
entire
region.
So
when
you
have
a
set
failure
and
you
have
a
single
set
for
a
single
region,
then
at
the
end
you
will
have
your
entire
region
down
and
if
you
have,
for
example,
100,000
customers
there.
A
What
are
the
other
benefits
to
this
solution
is
that
we
are
able
to
upgrade
safe
cluster
I
said
cluster.
We
don't
have
to
have
with
the
whole
big
surf
cluster
at
once,
and
since
you
are
probably
aware,
there
are
versions
of
jewel
which
had
some
problems,
and
you
know
it's
always
easy
to
it's
always
good
to
go,
go
to
the
future
in
small
steps.
So
that's
what
we
do
by
upgrading
cluster
by
cluster
and
what
is
also
good
is
that
we
can
do
the
fine-tuning
for
each
cluster.
A
So,
for
example,
we
can
provide
a
safe
cluster
for
database
purposes
and
we
can
provide
a
difference
of
cluster
for
general
purpose
cloud
computing
so
that
database
workloads
from
the
database
cluster
will
be
slightly
different,
will
be
completely
different,
actually
from
a
general-purpose
crowd.
The
instances
due
to
the
fact
that
databases
need
load
needs
low
latency
and
they
do
a
lot
of
flushing.
So
we
can
tune
up
some
parameters
of
that
cluster
to
meet
the
storage
solution
requirements.
A
For
example,
if
you
have
a
single
safe
cluster,
there
are
plenty
of
objects
to
manage.
There
are
plenty
of
configuration
stuff
which
has
to
be
stored
somewhere.
So
if
you
have
a
single
safe
cluster,
for
example,
you
have
three
safe
users.
We
have
three
sub
poles
and
for
configuration
options
and
also
like
20
ACLs
access
control
lists.
It's
always
good
to
setup,
irulu,
firewall,
rule,
sorry
and
there's
also
some
maintenance
operations
to
do
on
that
single,
safe
cluster.
Now,
when
we
do
200
safe
cluster,
the
math
is
quite
easy.
A
You
will
have
like
30
objects,
fruity
configuration
objects
from
that
single
cell
cluster
multiplied
by
200,
which
makes
6000
configuration
objects,
I
would
say,
and
that's
a
huge
amount
of
information,
huge
amount
of
configuration
you
have
to
do
manually
and
it's
almost
it's
almost
impossible
to
do
it
manually
correctly.
There's
always
some
problem
with
that,
but
there's
something
you
can
do
about
it,
and
we
actually
did
something
to
solve
this
problem
at
OVH.
A
What
we
did
is,
basically,
we
implemented
ourselves
a
control
plane,
not
a
remote
control,
plane,
just
a
control,
plane
and
we've
implement
automation
to
the
rescue.
To
is
the
maintenance
burden
of
all
those
safe
clusters,
so
we
will
have
a
actually.
We
have
a
single
control
plane
which
is
connected
to
all
our
safe
clusters
and
all
our
dedicated
servers
and
that
single
control
plane
is
able
to
do
all
the
maintenance.
Do
all
the
configuration
now.
What
would
such
control
plane
manage?
A
Let
me
tell
you:
there
are
three
groups
of
configuration
staff
which
would
have
to
be
implemented
by
our
control
plane.
So,
first
of
all,
there
are
set
configuration
stuff,
so
these
are
users
desire,
ACLs.
These
are
pools
and
also
self
consumptions
pass
to
the
EDC
set
conf
and
all
those
configuration
stuff
is
related
to
single
cell
cluster,
so
we
will
have
multiple
of
them
now.
The
next
group
of
stuff
we
want
to
manage
is
the
amount
of
SDS
and
amount
of
monitors.
A
So
basically,
this
is
the
way
how
we
create
a
cluster,
how
we
grow
a
cluster,
how
we
upgrade
it.
So
these
are
the
containers
part.
Actually,
this
is
this
is
the
information
to
where
our
containers
and
how
to
have
to
create
them,
and
the
last
piece
of
this
functionality
we
want
to
have
in
our
control
plane
is
our
maintenance
operations.
So
we
want
to
have
the
automatic
maintenance
operations
like
placement
loop
based
on
group
growing
or
rebalancing
your
cluster
or
disc
replacements
stuff
like
that?
A
Okay.
So
how
do
we
implement
such
thing?
First
of
all,
we
need
to
put
all
those
features
into
the
control
plane
and
the
control
plane
will
be
able
to
maintain
newscasters
all
those
testers.
So
as
an
operator,
the
only
thing
I
have
to
do
is
to
tell
the
control
plane
to
do
something
for
me.
So,
for
example,
if
I
want
to
change
some
ACLs
I,
just
tell
the
control
plane
to
control
plane.
A
Does
this
cluster
named
a
and
it
required
it
requires
two
new
ACLs
and
the
control
plane
will
do
the
rest,
and
the
same
thing
applies
to
the
placement
group
growing
or
stuff
like
that,
so
how
to
implement
such
thing?
First
of
all,
we
need
a
source
of
truth
for
the
entire
system,
and
that
would
be
a
highly
availability
database.
We
will
put
all
the
saturated
configuration
options
into
the
database
and
also
we
will
put
all
the
set
clusters
related
stuff
into
that
database.
A
Like
always,
these
monitors,
racks
and
hosts
and
on
the
left
side
of
this
question,
a
question
there
will
be
an
operator
and
it
will
have
some
kind
of
an
API
to
manage
all
those
stuff
and
on
the
right
side,
there's
a
provisioner
which
will
be
able
to
translate
the
information
from
the
database
into
real
configuration
stuff.
So
the
provisioner
is
able
to
take
the
information
like
self
user
from
the
database
and
apply
it
on
the
safe
Gluster.
That's
the
main
idea.
B
A
So
we've
implemented
kind
of
a
application
in
Python
which
is
able
to
provide
the
operator
a
nice
way
to
manage
safe
clusters,
all
that
safe
clusters
in
our
infrastructure.
So
it's
written
in
Python.
As
for
the
database,
we
use
MySQL
database
aguillera
version,
which
is
highly
available
and
for
the
provisioner
we've
used
puppet.
A
So
puppet
is
able
to
fetch
the
information
from
our
database
through
some
provisioning
API
and
it's
able
to
apply
changes
on
the
infrastructure
on
safe
clusters
on
dedicated
servers,
whatever
it
needs
to
do,
and
we
are
happy
because
we
covered
all
the
self
management
operations
set.
Man.
Sorry,
not
safe
configuration
stop.
The
first
first
point
is
check.
B
A
A
We
had
similar
API,
similar
CLI
for
the
operator,
so
the
only
thing
the
operator
has
to
do
is
to
tell
the
control
plane
to
create
a
new
cluster
in
our
data
center
and
he
has
to
tell
the
size
in
OSD
count
just
just
to
know
how
big
the
new
cluster
should
be
or
in
the
same
way
we
can
grow
the
cluster.
So
we
can
tell
I
control
point:
please
grow
up
this
cluster
by
three
TOS
DS
and
that's
it.
A
A
A
We
need
a
remote
API
for
our
customers,
so
they
can
able
to
that.
So
they
are
able
to
manage
their
pools,
manage
the
users,
because
they
are
aware
what
kind
of
pools
users
ACS
they
need
and
that's
what
we
basically
did.
This
is
available
through
our
OVH
API
comm
website,
I,
think
yeah,
API,
OVH,
comm,
sorry
and
all
that
API
is
available
there.
So
if
you
buy
a
safe
cluster
from
us,
you'll
have
all
that
API
of
all
that
API
available
for
you
to
create
to
do
basically
figure
your
cluster.
A
A
Now
the
only
thing
which
is
missing
are
the
maintenance
operations
and
maintenance
operations
are
a
little
bit
harder
more
difficult
to
implement.
Since
there
are
no
ordinary
configuration
objects,
and
let
me
tell
you
an
example,
so
if
you
want
to
grow
placement
groups
of
your
pool,
you
would
like
to
do
some
kind
of
algorithm.
A
A
B
A
If
you
take
a
closer
look
on
that
diagram,
this
is
actually
a
kind
of
a
sequential
processing
and
it
can
be
displayed
like
that.
So
basically,
there
are
few
blocks
here.
The
blue
blocks
are
just
wait
for
class
tile
hype.
Okay
blocks,
let's
call
them
tasks
when
the
yellow
one
is
set
PG
and
it
was
orange,
but
it's
not
orange
here
nevermind.
A
A
Another
example
growing
upgrading
your
cluster,
so
if
you
upgrade
your
cluster,
a
serf
version
of
your
cluster,
you
would
probably
like
to
do
some
operations
in
parallel,
so
you
would
probably
upgrade
flag
by
rack.
As
I
said,
we
are
having
thanks
to
having
containers
we
are.
We
are
able
to
upgrade
OSD
by
OSD
and
that's
also
the
case.
We
can
upgrade
track
by
rack
by
2
OS
DS
by
2.
As
this,
for
example.
In
this
case,
we
will
update
the
head
self
cluster
by
free
OS.
A
This
at
once
realize
this
in
each
rack
and
how
we,
how
will
we
do
it?
We
will
once
again
check
for
cluster
help.
Ok
will
execute
three
tasks
in
parallel
to
upgrade
each
or
s
d3,
or
s
this
at
once,
and
then,
once
again,
we
will
wait
for
cluster
help,
ok
and
so
on.
Until
we
finish
with
all
the
racks
we
have-
and
this
is
more
complex
workflow.
So
as
a
result,
we
need
a
component
to
our
control
plane,
which
will
be
able
to
do
that
kind
of
processing.
A
We
need
to
be
able
to
do
a
parallel
processing
like
this
one.
We
need
to
be
able
to
do
the
sequential
processing
like
this
one,
and
also
we
would
like
to
have.
The
combined
processing
like
hardshell
is
sequential
and
partially
parallel,
and
it
turns
out
that
there
are
open-source
projects
which
allows
such
kind
of
a
processing
and
amongst
them
there's.
For
example,
Apache
airflow,
there's
a
salary
project
called
there's
a
sudden
project
implemented
in
Python
and
there's
also
a
Luigi
project
from
Spotify.
A
There
are
multiple
project
solutions
which
allows
you
to
do
such
kind
of
processing
and
at
OVH,
and
we
decided
to
implement
and
integrate
salary
into
our
control
plane.
So
once
again,
the
familiar
diagram
we've
implemented
the
workflow
processing
platform
component
in
Python
and
salary.
So
once
again,
the
operator
is
able
to
tell
the
control
plane
to
grow
PGS
and
the
workflow
processing
platform
will
do
the
rest
for
him.
That's
the
main
idea.
B
A
A
You
may
ask
yourselves
what
that
would
be,
and
let
me
tell
you
so
you
are
all
having
some
kind
of
a
monitoring
solution
for
your
clusters,
I'm
pretty
sure
of
that.
So
you
are
doing
monitoring
of
different
stuff
from
set
perspective.
You
did
the
monitoring
of
active
placement
groups
of
blocked
requests
stuff,
like
that.
A
That's
the
main
idea.
So
when
our
monitoring
solution
detects
the
problem,
its
able
to
tell
the
control
plane
that
there
is
a
problem,
and
now
the
control
plane
is
able
to
react
on
that
notification
and
apply
some
fixing
on
our
hosts
on
our
set
clusters
on
whatever
is
whenever
whatever
is
the
problem?
Actually,
and
let
me
tell
you
an
example:
this
is
the
way
how
we
remove
broken
disks
from
our
infrastructure,
so.
A
When
the
disk
is
failing
its,
it
can
fail
in
a
different
ways
and
in
most
cases
it
starts
to
have
some
IO
errors.
It
doesn't
fail
entirely.
It
has
just
errors,
so
our
monitoring
is
able
to
detect
those
errors
at
the
host
level.
So
the
monitoring
will
be
able
to
tell
the
control
plane
that
there's
a
problem
with
a
disk
and
then
the
control
plane
checks
that
there
is
an
OSD
hosted
on
that
disk
and
then
it's
able
to
start
workflow
processing
in
order
to
fix
the
problem.
A
So,
first
of
all,
it
sets
the
down
times
in
our
monitoring
solution,
because
when
we
are,
we
don't
want
to
be
waking
up
during
the
night.
If
some
disk
has
IO
errors,
if
we
can
fix
them
automatically.
So
that's
why
we
are
setting
down
times
now,
since
the
disk
is
still
working,
and
we
don't
want
to
affect
our
customers
too
much
by
the
performance
impact
of
the
huge
rebalance
caused
by
this
removal.
A
We
will
slowly
decrease
the
OSD
weight
up
2-0,
so
we
decreased
by
0.1,
I
guess
as
I
for
as
far
as
I
remember,
then
we
wait
for
class
to
help.
Okay
and
then
we
once
again
decrease
up
to
the
zero
once
that
is
done,
we
can
safely
remove
that
OSD
from
our
cluster
and
then
I
think.
The
next
thing
to
do
is
just
to
remove
the
container
from
our
host
and
notify
the
data
center
technicians
to
replace
that
disk,
and
that's
all
of
that
is
done
automatically.
A
A
Adding
a
single
OSD
into
a
big
cluster
can
cause
a
huge
performance
impact
due
to
the
fact
that
there's
a
lot
of
data
to
rebalance.
So
we
do
that
in
in
the
situations
when
there
are
multiple
as
these
two
to
replace
to
create
new
OS
this,
so,
basically
that's
how
it
works
and
I
don't
have
any
slide
for
that.
But
there's
another
thing:
we
do.
We
also
do
nvme
where
out
predictions.
A
So
all
those
SSD
and
VMI
devices
have
where
out
counters,
they
are
reported
from
smart
I,
think
and
we
are
doing
also
the
linear
regression
on
that
kind
of
matrix.
So
we
can
expect
some
kind
of
updates
in
the
future
when
the
device
will
be
failings.
So
again,
thanks
to
the
self
killing
capabilities,
we
are
able
to
react
on
such
kind
of
event
and
we
are
able
to
replace
the
MV
and
vme
drive
before
it
fails
and
pretty
much.
That's
it.
That's
how
we
are
at
ovh
able
to
maintain
more
than
200
self
class
stirs.
A
Even
if
we
had
five
hundreds
or
thousands
of
clusters,
we
will
be
able
to
maintain
them
thanks
to
the
our
control
plane.
Thanks
to
the
self-healing
capabilities
and
there's
a
lot
of
room
to
implement
more
features
like
that
in
our
control
plane
and
that's
basically
how
it
works.
So
we
love
separate,
OVH
and.
C
C
B
A
The
team,
sorry
yeah,
so
the
team,
which
was
starting
in
2015,
had
four
people.
Now
we
are
six
people,
but
we
are
also
doing
different
stuff,
like
capacity
planning
like
new
hardware
evaluation,
so
I'm
not
really
sure
how
much
say
how
much
time
does
it
take
to
implement
such
such
such
system.
But
in
our
case
it
took
like
three
years
because
we
were
constantly
adding
new
features
to
that
control,
plane
and,
as
I
said
in
2015
15
we've
started
only
with
the
basic
part
which
could
create
self
cluster
or
grow
it
or
upgrade
it.