►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Rook Deployed Scalable NFS Clusters Exporting CephFS - Patrick Donnelly, Red Hat
Rook was developed as a storage provider for Kubernetes to automatically deploy and attach storage to pods. Significant effort within Rook has been devoted to integrating the open-source storage platform Ceph with Kubernetes. Ceph is a distributed storage system in broad use today that presents unified file, block, and object interfaces to applications. This talk will present completed work in the Ceph Nautilus release to dynamically create highly-available and scalable NFS server clusters that export the Ceph file system (CephFS) for use within Kubernetes or as a standalone appliance. CephFS provides applications with a friendly programmatic interface for creating shareable volumes. For each volume, Ceph and Rook cooperatively manage the details of dynamically deploying a cluster of NFS-Ganesha pods with minimal operator or user involvement.
https://sched.co/Nrpy
A
Hello,
everybody
I'm
patrick
connolly.
I
am
the
SEPA
fests
team
lead
at
Red
Hat.
My
colleague,
Jeff
Leighton,
couldn't
make
it
to
UConn
this
here,
so
he
won't
be
joining
me
to
give
this
talk.
So
it'll
just
have
me
today.
I
want
to
be
talking
about
the
new
technology.
We've
been
working
on
to
have
NFS
servers
deployed
with
the
rook.
A
Is
interesting
because
it
uses
the
clients
and
MDS
is
cooperatively
maintained,
a
distributed
cache
of
the
Dinos
and
and
the
directories
that
is
sort
of
a
difference
between
many
other
distributed
file
systems
where
clients
talk
directly
the
servers
and
the
server's
manage
all
the
state.
This
was
a
decision
made
early
on
to
allow
the
clients
to
to
have
direct
access
to
the
date
to
the
data
without
needing
to
go
through
any
type
of
gateway.
A
Sefa
fests
is
provides
full
failover
management
by
the
use
of,
for
example,
standby
metadata
servers.
It
also
provides
for
horizontal
scale
out
by
having
multiple
active
metadata
servers
served
in
a
cluster.
The
metadata
servers
do
not
maintain
any
state
they're
very
easy
to
containerize.
They
a
store
older
state
in
Rattus,
Rattus
being
a
distributed,
object,
data
storage
data
store,
which
is
the
main
underlying
component
of
SEF.
A
On
top
of
that,
the
metadata
servers
represent
the
the
metadata
hierarchy
and
the
present
that
to
the
clients
for
access
clients
directly
talked
to
the
object
store
when
they're
writing
reading
or
writing
to
the
the
file
objects.
In
order
to
maintain
consistency,
the
MDS
hands
out
capabilities
which
are
somewhere
to
leases
from
the
academic
concept.
They
hand
these
out
to
clients
which
allow
them
to
delegate
metadata
for
its
authority
to
clients.
A
A
Gone
a
great
have
done
a
lot
of
integration
with
Stefan
in
this
way,
and
seth
has
become
one
of
the
more
popular
storage
technologies
to
use
with
an
open
stack,
Community
Survey
done
in
the
past,
and
a
few
years
ago
we
found
that
its
ffs
was
the
primary
storage
provider
for
for
use
within
Manila,
and
the
reasons
for
this
I
think
are
fairly
straightforward.
Saif
is
a
open
source
storage
solution
and
is
easy
to
incorporate
into
your
OpenStack
clusters
and
then
also
it's
free.
So
if
I'm
talking
about
OpenStack,
then
why
what
why
not?
A
Why
am
I
here?
Why
are
we
talking
about
Cooper?
Why
should
I
be
talking
about
kubernetes,
so
it
turns
out
that
there
is
a
big
push
right
now,
as
we
all
know
that
to
manage
distributed
services
within
the
context
of
a
container
Orchestrator
in
this
case
kubernetes
and
there's
a
lot
of
attractive
reasons
for
doing
this,
containers
are
lightweight
and
easy
to
deploy
in
response
to
changing
application
needs.
Importantly,
especially
for
this
project,
it
allows
for
extensible
service
infrastructure,
so
I
can
deploy
services
in
a
response
to
the
changing
needs
of
the
application.
A
Containers
are
also
parallel
enough
to
be
are
lightweight
enough
to
allow
for
easy
parallelism.
I
can
deploy
a
lot
of
containers
in
response
to
sudden
to
those
changing
needs
of
the
application.
It's
a
much
easier
to
do.
Failover
with
with
in
a
container
orchestrating
device
environment
the
instead
of
having
a
dedicated
machine
that
is
there
to
take
over
when
a
service
fails.
I
can
just
spin
up
a
new
container
anywhere
in
my
infrastructure
to
take
over
that
role,
and
finally,
kubernetes
also
allows
us
to
do
fast.
A
Some
yamo
files
into
cube
cuddle
and
the
rook
or
agents
that
are
running
on
all
the
nodes
will
figure
out
where
it
can
host
object,
storage
devices
with
force
F,
attaching
an
object,
storage
device
to
available
disk
drives
or
SSDs
on
all
of
your
nodes
in
your
kubernetes
cluster
and
then
deploying
the
SEF
demons
as
needed
in
order
to
get
your
cluster
ready.
The
great
hallmark
success
of
this
is
that
it
allows
you
to
deploy
a
set
cluster
without
knowing
anything
about
SEF,
which
is
sort
of
store.
Eclis
been
a
very
difficult
thing
to
do.
A
A
A
The
primary
use
case
for
CFS
was
for
an
HPC
environment
where
we're
running
a
lot
of
high
performance
computing
jobs
that
need
access
to
fast
file
systems,
especially,
for
example,
for
scratch
space,
and
that
was
his
target
use
case
and
because
of
that,
it
was
operating
in
a
mostly
trusted
environment
and
the
the
application
could
be
trusted
to
to
access
the
file
system
in
a
safe
manner
in
a
in
today's
deployments
and
like
OpenStack
or
kubernetes,
and
how
you
may
want
to
serve
your
infrastructure
to
various
tenants.
That
may
not
be
an
assumption.
A
You
can
safely
make
and
there's
a
few
reasons
for
that
is
not
necessarily
easy
to
to
restrict
which
which
types
of
clients
are
accessing
your
filesystem,
you
may
have,
for
example,
if
I'm
running
with
an
open,
stack
and
I
have
virtual
machines
being
brought
up
by
my
customers
and
they're
running
older
kernels.
It
may
not
work
without
bugs
against
my
chef,
cluster
and
I
lack
control
over
what
types
of
clients
they're
using
to
talk
to
Seth.
A
Another
issue
is
security.
You
may
want
to
firewall
off
your
storage
cluster.
In
fact,
you
probably
do
to
prevent
clients
from
having
unrestricted
access
to
your
to
your
storage
network,
so
there
may
be
a
firewall.
Additionally,
you
may
want
to
introduce
some
kind
of
authentication
and
authorization
mechanisms,
for
example
to
Kerberos,
which
Ceph
does
not
even
support
and
force
clients
to
authenticate
through
those
mechanisms.
A
It's
open
source
software
license
under
the
LGPL.
What
makes
Ganesha
attractive
is
that
it
provides
a
plug-in
interface
for
multiple
different
kinds
of
exports
and
then
also
provides
support
for
exporting
different
types
of
file
systems.
The
most
obvious
and
usual
case
would
be
a
local
file
system
having
an
FSO
organ
I
should
export
the
something
in
the
local
file
system,
but
within
our
case
we
want
to
actually
export
this
ffs
file
system.
A
To
do
this,
Ganesha
has
these
backends
called
the
the
file
system,
abstraction
layer
and
that
abstraction
layer
for
staff
uses
lips
ffs
to
actually
talk
to
the
SEF
cluster,
so
any
incoming
NFS
requests
RPC
will
be
translated
into
a
corresponding
call
to
through
lips
ffs
correspondingly
Ganesha
also
uses
librettos,
which
is
a
library.
Diedrich
gives
direct
access
to
the
object,
storage
layer
to
store
its
various
state
that
it
needs
to
maintain
in
order
to
recover
from
a
failover
when
a
standby
condition
takes
over.
A
It
also
uses
rathaus
to
now
stores
configuration
files
instead
of
using
the
Etsy
namespace
within,
for
example,
can
is
filesystem,
and
for
these
reasons,
Ganesha
is
very
amenable
to
containerization.
We
can
store
all
of
our
are
all
the
file
system
stages
of
course
stored
in
cephus.
We
can
store
all
the
configuration
and
recovering
information
within
rathaus.
There's
no
need
for
a
writable
local
file
system.
A
So
up
to
now,
Ganesha
has
been
deployed
in
a
passive
active
fashion,
and
this
has
been
available
since
cefalu
back
in
August
2017,
you
have
won
NFS
server,
which
all
of
your
NFS
clients
talk
to
and
use
the
third-party
tools
pacemaker
chorusing
to
handle
the
failover
between
the
active
NFS
Ganesha
demon
and
the
standby
NFS
demon.
The
main
drawback
of
this
approach
is
that
it
scales
poorly
and
it
requires
a
lot
of
resources.
A
You
need
a
idle
NFS
Ganesha
server
available
to
take
over
when
the
active
fails
and
all
the
NFS
clients
are
talking
to
a
single,
active,
NFS
server
for
all
of
their
data
needs,
which
obviously
will
not
scale
very
well.
So
one
of
our
goals
was
to
allow
for
all
of
these
NFS
clients
to
talk
to
multiple
NFS
servers.
A
They
now
have
a
sessions
layer
and
add
it
on
to
for
the
NFS
clients
when
they're
talking
to
the
access
server,
which
provides
exactly
once
semantics.
So
you
don't
when
you're
retrying
an
operation,
it
doesn't
happen
multiple
times
and
now
also
a
recant
reclaim
complete
operation.
So
when
an
NFS
server
goes
down
goes
down
and
it
comes
back
up,
it
becames
a
reclaimed
process
where
all
the
NFS
clients
can
reestablish
whatever
state
they
had
with
the
prior
NFS
server.
A
A
So
after
an
NFS
server
restarts
or
a
server
takes
over
for
four
failed
and
a
best
server,
it
has
no
offense
date
because
it's
a
dozen
storage
state
anywhere
locally,
and
so
all
the
clients
have
to
notify
the
NFS
server
exactly
what
state
they
had.
That
includes
what
files
that
opened,
how
they
had
the
files
open,
any
kind
of
locks
they
had
and
because
you
don't
want
one
client
to
say,
I
had
I
want
I
I
want
to
write
to
this
file
and
another
client
which
is
not
reconnected
yet
has
write
access
to
that
file.
A
It
may
be
buffering
rights.
You
need
to
wait
a
little
bit
to
give
all
the
clients
that
a
chance
to
reconnect
to
the
server
and
that's
what
that's.
What
this
reclaim
session
period
is
is
to
lease
periods,
at
least
very
important,
of
corresponding
to
how
long
you
wait
for
a
client
to
say
they
are
still
holding
on
to
a
lease.
A
So,
logically,
this
can
be
organized
into
a
series
of
epics,
where
the,
where
an
epic
corresponds
to
the
lifetime
of
a
particular
instance
of
an
ephah
NFS
demon,
beginning
with
a
grace
period,
where
all
the
NFS
clients
can
reconnect
and
try
to
establish
whatever
state
they
had.
And
then
you
have
after
that
completes
the
NFS
server
transitions
into
the
normal
operation,
which
will
be
generally
a
very
long
time
that
should
be
in
a
normal
operation.
Fail.
Overs
are
not
that
common.
A
A
They
will
you
may
have
state
this
issue
to
one
client
of
one
NFS
server
and
during
a
failover
event,
another
NFS
client
of
the
other
NFS
server
may
try
to
acquire
that
state,
and
so
now
we
need
to
form
agreement
on
what
state
has
been
issued
and
how
to
reclaim
it
and
so
the
way
the
NFS
ganesha'
handles.
That
is
through
the
use
of
coordinating
grace
periods.
A
It
has
a
central
database
in
a
rathaus
object
which
keeps
track
of
what
the
current
grace
period
epoch
is
for
all
of
the
NFS
servers
so
that
they
can
form
consensus
on
on
when
the
grace
period
should
happen.
So
all
the
NFS
servers
together
in
the
cluster
will
enter
grace
period
together
whenever
there's
a
failover
event,
so
that
none
of
the
NFS
servers
accidently
issue
state
that
another
client
that
needs
to
reconnect
to
a
failed
NFS
server
needs
to
reclaim.
A
A
So
the
next
challenge
we
had
to
address
with
this
work
is
we
wanted
to
layer
NFS
over
a
cluster
file
system
and
because
both
of
the
protocols
force,
ffs
and
also
nfsv4,
one
are
stateful
and
they
use
a
lease
based
mechanism.
You
need
some
way
to
handle
the
case
where
an
NFS
server
fails
and
another
one.
A
Another
server
wants
to
come
in
to
acquire
that
state
and
you
need
a
way
to
make
that
failover
event
happen
faster
in
be
within
cephus,
namely
sefa
fest
has
its
own
timeouts
that
it
maintains
for
its
clients
whenever
a
client
fails
and
comes
back.
That
client
has
a
certain
amount
of
time
to
reestablish
its
session
with
SEPA
fest.
A
Likewise,
all
those
NFS
daemons,
all
the
NFS
clients
have
a
certain
amount
of
time,
the
corresponding
to
the
least
periods
to
to
reconnect
to
the
NFS
daemon
to
re-establish
their
state
and
there's
some
conflicts
that
are
there's
some
overlap
there
in
time,
which
can
cause
a
very
long
delay
for
the
NFS
clients
to
re-establish
their
state
and
be
able
to
continue
the
normal
operations
they
want
to
perform
and
and
how
long
it
takes
for
the
NFS
server
to
reacquire
the
stadium
means.
What's
in
in
cephus,.
A
So
one
of
the
new
features
in
Nautilus
was
that
we
allowed
a
client
to
reclaim
the
state
of
a
prior
session
when
a
cephus
client
dies
abruptly.
The
MDS
will
keep
its
session
around
for
several
minutes,
allowing
that
client
to
come
back
and
say
I'm
still
here
and
I
want
to
reclaim
all
of
these.
The
capabilities
that
I
had
all
the
file
locks
that
I
had
and
then
with
the
presumption
that
these
clients
are
not
actually
have
not
actually
failed
and
come
and
come
back
as
a
brand-new
client.
A
There
was
simply
a
network
partition
that
separated
them
from
them,
yes,
and
for
a
while.
This
was
fine
enough
for
current
for
a
kernel
client
which,
if
it
were
to
restart
it,
wouldn't
be
trying
to
get
a
new
session
with
them.
Yes,
who
would
create
I,
wouldn't
try
to
reclaim
the
old
session
with
him?
Yes,
because
all
the
applications
that
were
running
with
the
kernel
client
have
also
been
rebooted,
so
it
just
gets.
A
fresh
state
is
the
same
thing
with
SEF
views,
which
is
another
alternative
mounting
mechanism
for
CFS
with
NFS
Ganesha.
A
We
have
this
new
problem
where
the
NFS
demon
has
state
that
it's
issued
to
the
clients
and
if
the
daemon
fails
and
comes
back
those
it
needs
to
reacquire
all
that
state
that
it
had
in
the
prior
session.
So
now
in
arles,
cephus
supports
in
a
cephus
client
to
come
back
and
claim
all
the
state
that
it
had
in
a
prior
instance
of
the
client,
without
knowing
immediately
what
all
that
state
was
during
the
grace
period
for
the
NFS
server,
the
clients
will
come
in
and
say
what
state
they
had
with
the
benefit.
A
The
prior
instance
of
NFS
demon.
What
files
I
had
opened
correspondingly
this
the
Ganesha
server
will
talk
to
sefa
fest
and
reestablished
that's
the
state
that
it
had
from
the
prior
session
when
all
of
that
is
complete,
it
actually
completes
the
session
similar
to
the
reclaim
complete
operation
of
NFS
clients
send
to
the
NFS
server.
A
What
we
need
next
is
a
trivial
or
a
simple
way
to
deploy
those
NFS
Kaneesha
servers
in
an
active
active
configuration
dynamically
in
corresponding
to
whatever
kind
of
load
we
have
on
the
on
that
particular
export
of
the
ffs
file
system,
though,
that
the
NFS
clients
are
accessing
and
then
also
we
want
to
scale
in
another
direction
where
we
have
an
NFS
cluster
for
each
export
in
the
set
fest
file
system.
We
have
so
keep
in
mind.
A
This
ffs
can
be
a
very
large
file
system,
with
billions
of
AI
nodes
and
many
different
users
with
different
use
cases
and
applications,
and
so
we
want
to
be
able
to
have
an
NFS
cluster
front,
each
sub
tree
within
cephalus,
corresponding
to
the
application
need
and
all
of
those,
and
then
you've
achieved
a
few
things.
By
doing
that,
the
you
improve
performance,
because
now
you
are
isolating
certain
file
system.
A
Finally,
another
challenge
we
need
to
address
is
IP
main
great
migration
and
failover.
So
when
an
NFS
server
and
evitable
II
may
die
for
some.
For
some
reason,
we
need
a
way
to
spin
up
a
replacement
very
quickly
and
cheaply
and
also
we
need
a
way,
a
mechanism
to
migrate,
an
IP
address
from
the
prior
instance
of
the
NFS
server
to
the
new
instance,
and
these
are
historically
decision.
A
A
A
Wrote
in
that
and
now
staff
have
a
deep
integration
with
each
other
between
the
the
SEF
Nautilus
release,
which
was
just
released
about
two
months
ago
and
rook
itself.
So
now,
SEF
is
also
extendable
and
extensible
and
with
rook
in
that
it
now
is
able
to
deploy
services
in
response
to
to
the
changing
application
needs,
for
example,
creating
a
new
file
system.
A
By
talking
to
rook
and
saying
I,
you
know,
I
need
another
metadata
server,
please
spin
another
container
up,
and
this
has
all
been
abstracted
away
very
nicely
and
the
look
we
were
playing.
We
are
able
to
use
this
to
have
the
manager
daemon
and
deploy
NFS
clusters
in
response
to
what
the
user
specifies
via
the.
A
Furthermore,
we
can
scale
up
the
NFS
cluster
up
or
down
in
response
to
how
the
application
needs
change
and
have
this
whole
centrally
managed
from
a
manager
in
the
future.
We
also
want
to
use
this
too,
and
it's
sort
of
a
holy
grail
type
concept
within
cephalus.
We've
always
thought
it
would
be
very
nice
to
be
able
to
deploy
more
active
metadata
servers
in
response
to
the
need
to
change
changing
load
of
the
file
system.
A
A
Now
we
have
this
ability
to
create
volumes
and
sub
volumes
volumes
corresponding
to
separate
file
systems
which
multiple
set
file
systems
are
not
yet
supported
within
SEF
is
something
we're
planning
to
do
in
a
future
release.
The
more
common
thing
to
do
is
to
have
the
sub
volume
concept
where
you
separate,
or
you
dedicate
an
entire
directory
tree
to
a
volume
which
maybe
is
exported
to
a
particular
person
machine
or
a
set
of
containers
for
sharing,
and
then
this
allows
Manila
for
OpenStack
or
the
CSI
interface
for
kubernetes.
A
A
A
Much
very
turnkey
in
terms
of
actually
configuring
the
volume
and
then
setting
up
the
NFS
kanesha
cluster
so
to
in
version.
Fourteen
point
two
point:
two:
the
next
version
of
this
FS
Nautilus
are
sort
of
Sept
Nautilus
release
you'll,
be
able
to
create
these
sub
volumes
through
a
centralized
command
and
then
in
a
subsequent
future,
release
we're
planning
to
make
it
very
simple
to
actually
deploy
an
NFS
cluster
in
an
active-active
fashion
in
front
of
staff
with
rook
and
kubernetes,
just
with
a
simple
command
enabling
NFS
and
setting
various
configuration
options.
A
You
might
want
to
have,
for
example,
what
kind
of
an
effect
our
network
namespace.
You
want
to
attach
the
pods
to,
in
terms
of
say,
for
example,
if
I
have
OpenStack
virtual
machines
that
have
been
set
up
and
I
want
to
attach
my
NFS
kinesha
clusters
to
the
same
network,
namespace
that
that
those
that
those
tenants
are
in
then
I
can
configure
that
through
this,
this
command
and
then
finally
turn
it
on
by
just
setting
sharing
a
Festa.
True.
A
When
that
may
not
be
necessary
because
salt
sub
volumes
within
sefa,
that's
may
not
share
any
I
knows
so.
There
is
no
shared
state
between
sub
volumes.
It
would
be
better
for
the
NFS
server
clusters
to
be
isolated
from
each
other
or
not
enter
grace
periods
together,
because
there
would
be
no
conflicting
state
between
them
and
then.
Finally,
it
would
also
be
nice
to
use
this
for
SMB.