►
Description
Keynote: Supporting Swiss Academia with Ceph & OpenStack - Jens-Christian Fischer, Team Lead, Infrastructure & Data, SWITCH
SWITCH is the Swiss National Research and Education Network (NREN) and has been operating a Ceph & OpenStack based IaaS called SWITCHengines for higher education in Switzerland since 2014. In this talk, we describe our setup, the use cases we support and the experiences we have with running multiple multi Petabyte Ceph clusters in production. For Science!
About Jens-Christian Fischer
SWITCH
Team Lead, Infrastructure and Data
Jens-Christian is the team lead of the "Infrastructure and Data" team at SWITCH that is responsible for developing, building and operating SWITCHengines. He has a background in software development and agile project management. He holds an MSc in IT from the University of Liverpool.
A
All
right,
so
thank
you
very
much
I'm
honored
to
be
here.
Sef
sage
wrote
me
an
email
a
couple
of
weeks
ago
say:
would
you
like
to
give
a
talk
on
the
academic
use
of
SEF
and
I
said
yes
and
I
thought
I
would
be
off
in
a
corner
somewhere
talking
to
the
academics.
The
ten
people
that
show
up
so
being
here
on
stages
was
a
bit
of
a
surprise,
but
I
see
what
we
can
do
about
that.
I
have
to
start
off
with
a
disclaimer
of
course,
switch
and
and
staff.
A
We
go
a
long
way
back
and
we
love
Saif.
Saif
has
been
really
good
to
us.
It
has
been
a
wonderful
tool,
so
you
should
take
everything.
I
say
with
a
grain
of
this
love
salt,
but
we're
here
for
science.
So
let's
have
some
real
hearts
repeating
and
I
want
to
go
from
from
hearts
to
something
else,
and
what
you
see
up
here
is
a
slice
of
a
mouse
brain
now
you
would
one
might
wonder
why
would
you
have
a
mouse
brain
on
stage
here,
but
this
is
for
science,
so
a
mouse
they.
A
What
we're
looking
at
is
the
image
of
a
laser
beam
shaping
imaging
microscope.
A
mouse
brain
contains
about
0.5
terabytes
of
data.
One
of
these
micro
microscopes
can
image
3
mouse
brains
per
week,
which
I
would
guess,
gives
you
around
120
mouse
brains
per
year,
which
is
60
terabyte
of
brain
data
that
needs
to
be
made
accessible
and
stored
for
research
and
for
future.
This
is
where
we
come
to
storage.
We
also
come
to
scientists-
and
this
is
probably
the
weirdest
office
sign
I've
ever
seen.
A
That
was
at
the
at
the
lab
at
the
imaging
lab
last
week.
So
what
were
they
thinking?
Right
switch
is
a
national
research
and
education
network,
as
we
call
it.
An
N,
R
n
Enron's
exist
to
provide
services
to
academic
infrastructures,
and
there
is
one
in
every
country
we
happen
to
be
the
Swiss
one.
We
are
bound
to
serve
the
academia,
the
public.
We
have
a
number
of
core
competencies
that
we
are
not
going
to
go
through
in
detail,
but
we
are
a
foundation
we're
a
public
foundation.
A
Our
customers
are
basically
the
universities
and
they
are
related
institutions,
research
institutions,
we
serve
all
of
them.
Equally,
we
have
a
other
institutions
that
are
loosely
connected
to
research
like
University
Hospitals,
for
example,
and
we
also
have
a
couple
of
commercial
customers.
One
of
them
is
the
Swiss
government,
because
we
are
by
an
happy
accident.
We
are
the
registry
of
the
CH
top-level
domain,
and
that
is
a
story
for
another
day.
A
It
happen
is
something
with
a
professor
that
registered
dot
CH
at
the
ICANN
many
many
years
ago,
and
we
were
sitting
with
the
top-level
domain
for
Switzerland,
where
it
wondered
what
he
should
do
with
it.
So
he
gave
it
to
us.
We
provide
a
lot
of
services
to
the
core
server.
Is
that
the
one
that
started
it
all
is
internet?
We
operate
two
and
a
half
thousand
kilometers
of
fiber
in
Switzerland
we
have
owned
the
optical
equipment
and
we
have.
A
A
We
are
part
of
the
academic
community,
so
we're
not
coming
in
as
a
vendor
trying
to
sell
stuff
to
the
universities.
We
see
ourselves
as
partners
and
luckily
the
universities
also
see
ourselves
sees
us
as
their
partners.
So
we
have
a
discussions
on
a
whole
different
level
than
just
trying.
Here's
here's
some
stuff
give
us
the
money
and
you'll
be
good.
We
also
work
internationally
in
tons
of
different
organizations
across
Europe
and
the
world.
A
My
team
infrastructure
and
data,
so
you
can
see,
is,
is
a
member
part
of
the
OpenStack
thing
and
there's
a
space
down
on
the
right
and
I
hear
from
our
legal
team
that
we
are
ready
to
sign
the
ACEF
foundation
agreement
and
will
part
be
part
of
the
staff
foundation
as
of
today.
If,
if
things
go
well,
we're
really
happy
about
that,
but
anyway
we
I'm
going
to
talk
to
you
about
where
we
come
from
and
what
we,
what
we
have
been
doing
in,
in
particular
with
staff
in
in
the
last
couple
of
years.
A
A
However,
that
sort
of
started
to
fade
away
in
the
early
tens
when
this
newfangled
saying
from
a
internet
bookseller
started
to
become
more
and
more
interesting
to
people.
So
we
started
to
look
at
ways
how
we
could
provide
services
on
a
more
self
served
based
model,
and
one
of
my
colleagues
proposed
that
the
switch
should
start
investing
in
cloud
and
said
and
done.
A
We
started
in
2012
with
building
a
first
small
cluster,
consisting
of
eight
machines,
see
if
we
could
get
this
up
and
running
with
some
some
open-source
software
that
we
found
on
the
Internet
basically
start
with
OpenStack,
which
was
a
guess
at
that
time.
There
were
other
cloud
providers,
cloud,
orchestration
providers
in
the
market
we
choose
OpenStack
and
for
storage.
Actually,
like
my
colleagues
from
CERN
said
yesterday,
there
was
only
one
choice
and
it
was
safe,
and
so
we
started
with
that.
A
What
you
see
up
here
is
a
slide
from
a
presentation
in
2013
that
we
gave,
which
describes
our
very
first
infrastructure
and
Poe.
A
very
proud
of
it,
64
servers
across
two
data:
centers
32,
compute
nodes
for
all
of
448
cores
and
32
storage,
nodes
divided
among
two
safe
clusters
with
all
of
1200
terabyte
raw
capacity,
and
we
thought
we've
got
a
set
up
here.
A
We
didn't,
but
you
learned
that
afterwards
we
are
hosted
in
universities
with
the
data
centers
we
operate
are
in
the
cellars
in
the
basements
of
these
two
universities,
one
in
Zurich
in
the
Swiss
German
or
in
the
german-speaking
part
of
the
country,
one
in
Lausanne
in
the
french-speaking
part
of
country.
That
also
takes
care
of
politics,
because
you
have
both
regions
equally
pleased,
and
it
also
is
great
for
geographic
diversity.
In
case
of
disaster
recovery,
things,
we've
been
growing
over
the
years,
we've
been
continuously
upgrading
our
clusters.
These
are
the
the
current
numbers.
A
I
think
we
have
around
400
servers,
I'm,
not
sure
we
have
three
thousand
seven
hundred
and
some
coarse
physical
cores
as
of
last
week.
We
cannot
County
hyper-threaded
cores
anymore
because
of
intel's
bug
of
the
the
year
and
we
have
a
couple
of
safe
clusters.
Currently,
we
have
around
10
petabytes
of
raw
capacity
on
spinning
rust
disks
and
we're
slowly
slowly
starting
to
build
out
our
nvme
disks
pools
and
we
plan
to
migrate
our
virtual
machine
workloads
to
nvm
ease
in
at
some
point
in
the
future.
A
Research
is
not
only
about
data
about
data,
but
also
about
compute.
So
we
have.
We
have
heard
requests
from
our
customers
for
graphical
processing
units
GPUs
for
machine
learning
and
so
on.
So
we
have
a
few
of
them
and
the
most
surprising
thing
to
me
when
I
put
together
these
numbers,
we
started
with
two
switches.
Now
we
have
26
switches,
they
proliferate
much
more
than
then
compute
hardware.
A
We
still
have
our
dual
jet
10
gig
up
links,
but
we
were
upgrading
them,
200,
gigabit,
up
links
in
the
near
future
and
the
last
thing
here.
What
we
also
provide
is
layer,
two
tunnels
into
campus
networks
that
allows
virtual
machines
that
run
on
our
cloud
infrastructure
to
be
part
of
a
campus
network,
which
is
something
that
the
university
or
the
IT
department
apartments
at
the
universities
like
quite
a
lot
right.
A
A
They
are
60
years
old,
it's
the
same
clusters,
they
have
been
upgraded
over
the
years
and
we
have
never
lost
data
on
them.
We
we
had
a
few
small
problems,
but
we
never
lost
data.
We
never
lost
access
to
that.
So
that's
a
one
of
the
reasons
why
we
love
this
infrastructure.
We've
had
a
couple
of
problems
and
I'd
like
to
share
a
couple
of
more
stories
to
make
this
less
academic
and
I.
Think
there
are
three
types
of
problems
you
can
have
with
the
staff
class.
A
One
has
easy
problems:
you
have
a
disk
guy
or
server
dies.
Everybody
knows
what
happens.
It
just
works
and
fine,
it's
a
bit
more
interesting.
If
it's,
which
dies
where
half
of
your
computer
storage
cluster
is
that
half
of
your
cluster
is
attached
to.
We
had
that
happen
with
our
old
switches
that
had
went
into
packet,
storm
mode
and
basically
stopped
responding
to
everything.
A
So
half
of
our
cluster
went
down
right,
half
of
our
safe
cluster
lost
connectivity
and
the
result
of
that
is
it
just
works
as
soon
as
somebody
got
out
to
the
data
center
and
kicked
the
switch
because
we
didn't
have
remote
hands
at
that
point,
but
easy
stuff.
That
is
easy,
peasy
things
they're
medium
problems.
A
Maybe
we
had
Mons
crashing
back
in
the
early
days,
I
think
it
was
months,
and
it
also
it
just
worked,
because
we
got
sage
on
IRC
and
he
helped
us
debug
and
recompile
the
software
at
night
at
hour
a
night
at
his
day.
So
thank
you
sage,
wonderful
support,
there's
also
other
things
we
I
remember
a
case
in
three
years
ago,
when
we
had
a
PG
unfound,
which
Google
was
very
unhelpful
with
and
the
safe
documentation
was
very
unhelpful,
wilt.
Well,
it
turns
out
it
just
works.
A
But
unfortunately,
it's
stuck
behind
this
lack
ransom,
bear
saying
and
I
couldn't
couldn't
find
out
what
it
was
that
the
dis
or
an
engineer
told
us
three
years
ago,
but
I
guess
you
could
you
could
do
a
grep
on
the
safe,
codebase
and
search
for
command-line
options
with
force
in
them,
and
then
there
are
hard
problems
and
that
think
the
hardest
problem
we
ever
encountered.
Words
was
the
clicker.
No,
we
had.
A
The
problem
got
increased
by
me
being
on
vacation,
which
meant
that
disk
up
management,
attention
and
management
didn't
quite
understand
how
distributed
storage
system
works,
so
they
sent
off
the
team
chasing
various
rabbit
holes.
One
of
them
was,
we
should
blame
the
running
blue
store
migration,
because,
if
you
have
problems
with
the
storage,
blue
store
must
be
the
culprit
that
didn't
help
either.
So
we
thought
we'd
ask
the
nice
people
at
CERN,
because
that
helped
last
time
saying
thank
you
for
trying
to
help
it
didn't
we
didn't
find
it
out.
A
It
took
us
around
two
weeks
to
get
to
a
state
where
we
could
reproduce
this
behavior
synthetically.
So
it
was.
It
was
very,
very,
very
difficult
to
diagnose
and
what
what
we
then
found
is
you
shouldn't
look
at
the
storage
cluster,
but
at
the
network
and
after
a
while,
our
engineers
discovered
that
after
a
while
TCP
connections
got
dropped
to
512
byte
fragments
every
200
milliseconds
and
they
would
never
recover.
A
Install
a
new
kernel
because
there
was
a
clearly
a
bug
in
the
TCP
stack
after
kernel
and
it
started
working
again.
You
never
learn
you
never
stop
learning.
That
was
a
that
was
quite
an
experience,
and
luckily
we
could
pass
on
that
knowledge
to
colleagues
at
another
university
a
week
later,
that
ran
into
the
same
kind
of
problems,
so
safe
is
very
stable
and
safe
is
also
very
versatile.
A
You
know
the
Swiss
soldiers
pocket
instrument
here
of
choice
safe,
sometimes
feel
a
bit
feels
a
bit
like
it
does
everything
it
has
raras
it
has
RB
de
and
so
on
its
force.
However,
at
some
points
in
life
there
might
be
times
where
a
certain
tool
like
a
real
hammer
or
a
real
saw
or
a
real
knife
or
better,
and
what
we
we
use
with
SF
for
a
lot
of
things.
We
also
have
started
to
look
at
the
other
storage
systems.
We
don't
feel
like
this
is
cheating
on
chef
unsafe.
A
A
A
A
Guess
the
the
the
one
of
the
problems
is
we
looked
at
our
switch
engines,
shiny
things,
shiny
red
fire,
truck
or
milk
truck
or
whatever
from
this
perspective,
and
people
actually
looked
at
it
and
saw
that
it
wasn't
that
big
as
we
thought
it
was,
and
we
haven't
proven
ourselves
yet
to
be
able
to
provide
all
of
this
capacity
to
them
again,
stability
and
availability.
So
what
we
did
is
we
ate
our
own
dog
food?
A
We
have
a
another
product
called
switch
drive,
which
is
a
sync
and
share
service
for
academia
that
is
built
on
on
another
open-source
tool.
Own
cloud,
we
put
that
on
switch
engines
put
that
in
production
before
we
went
into
production
with
switch
engines
itself
and
put
it
out
to
the
universities,
and
that
has
been
a
quite
a
success.
This
is
the
last
or
is
it
four
years
of
storage?
We
started
out
with
zero,
obviously,
but
we
don't
have
graphs
going
all
the
way
back.
When
this
graph
started.
A
We
have
26
terabyte
of
data
on
switch
drive,
and
now
we
are
at
240
terabyte
of
user
data,
we're
serving
30,000
users
off
of
Ceph
of
not
the
most
or
not
the
best
infrastructure.
We
run
virtualized
NFS
servers
with
virtualized
RBD
volumes
and
there
I
don't
know
50
or
60
RPG
volumes
in
that
cluster
of
things.
They
are
all
linked
with
siblings
and
and
various
weird
contraptions,
but
it
works
and
we
lost
data
once,
but
that
wasn't
staffs
fault,
but
own
class
fault
due
to
some
weird
things.
A
They
the
tool
that
they're
using
they
had
an
s3
interface
and
the
team
discovered
that
we
also
had
an
s3
capable
storage
pocket,
so
they
put
their
video
data
on
to
us
and
that
caused
the
first
denial
of
service
attack
when
one
of
the
universities
started
to
run
videos
on
all
their
digital
signage
signs
in
non-stop
for
over
and
over
and
over.
But
right
now
we
have
something
like
45
terabyte
of
video
data
that
we're
streaming
directly
from
Dorados
clusters.
So
the
uptake
is
coming.
A
Data
is
coming,
there's
a
bunch
of
our
users
and
I
want
to
point
out
one
of
them
in
particular.
There's
the
data
science
Swiss
data
science
center.
This,
the
the
Swiss
data
science
center,
is
a
project
funding
too
little
ow.
It
will
fund
processing
data
science
and
one
of
the
projects
that
its
funding
is
the
ACE
data,
the
Antarctic
circumnavigation
expedition,
which
is
delivering
added
value
to
Antarctica.