►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Rook Deep Dive: Ceph - Travis Nielsen & Sebastien Han, Red Hat
In this talk, a deep-dive will be presented for the Ceph storage provider. Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments. Rook is run today in many production environments, providing a stable storage platform for your data. The architecture and recent improvements will be show how Rook provides the Ceph management layer for production environments. Rook was accepted as the first storage project hosted by the Cloud Native Computing Foundation in January 2018.
https://sched.co/Zey5
B
A
A
Have
a
chance
to
go
check
that
out.
I'd
definitely
recommend
it.
We,
you
know
what
what's
happening
in
rook,
just
to
recap
what
they
might
have
already
said.
So
the
we
we've
been
a
cnc.
I
think
you've
been
competing
projects
since
september
of
2018
and
we're
now
going
through
the
final
phases
of
graduation.
A
The
voting
is
in
progress
as
of
our
recording
today
and
we're
really
hoping
fingers
crossed
that
the
voting
is
all
completed
and
we
can
announce
graduation
at
kubecon.
Now
we
hope
for
your
vote
there
as
well.
It's
just
been
a
great
journey
and
and
looking
forward
to
to
that
being
done,
our
latest
release
is
version
1.4,
so
check
it
out.
Try
it
out.
We'd
love
your
feedback.
We
always
value
the
feedback
of
the
community
to
make
the
product
better
and
better.
A
So
again,
what
is
rook
so
to
start
off.
Rook
is
an
open
source
project,
apache
2
license,
and
it
is
a
a
collection
of
storage
operators
for
kubernetes
for
multiple
storage
providers,
including
ceph
and
cassandra
nfs
and
a
few
others.
But
today
we're
going
to
focus
on
the
ceph
operator
for
this
deep
dive,
so
rook,
fundamentally,
its
job
is
to
automate
the
management
of
ceph
seth.
As
a
software
defined
storage
system.
Is
you
know
it's
it's
complex
and
it
well.
It
needs
to
be
deployed
configured
when
it's
time
to
upgrade.
A
A
A
So
what
does
this
look
like?
We've
got
a
few
diagrams
here
well
before
we
get
to
the
diagrams.
Actually,
let
me
talk
about
three
layers
of
the
system
that
are
just
important
and
helpful
to
understand
the
difference
so
rook
the
operator
has
an
operator
for
ceph
and
it
owns
the
management
of
ceph.
That
means
it
deploys
ceph.
It
manages
everything
about
kubernetes
integration
with
ceph.
A
A
A
Let's
see
if
I
can
point
with
my
mouse
here
so
the
the
rook
operator
is,
is
the
core.
The
core
component
in
in
the
system
of
managing
seth
the
operator
is
the
brains
that
decides
how
to
deploy
ceph.
You
know
it
stores
or
calls
all
the
kubernetes
apis
and
works
to
make
sure
seth
is
just
working
seamlessly.
A
So
a
comment
about
the
color
coding
here,
the
blue
pods,
are
what
we
consider
as
rook.
You
know
core
rook
pods,
like
we've,
got
discovery
demons
that
are
running
on
nodes
to
discover
what
devices
are
available,
but
most
things
are
not
actually
core
rook
pods.
These
green
pods
then
are
layer
two
with
these
csi
plug-in
csi
driver
for
stuff,
and
then
the
red
pods
are
going
to
be
all
of
the
ceph
demons.
A
So
steph
has
a
number
of
different
demons
that
have
their
responsibilities
to
provide
the
storage
and
now
you've
got
the
mons
that
are
the
brains
for
ceph.
You've
got
the
osds
that
actually
manage
the
individual
data
devices
where,
where
storage
is,
is
bound
on
on
an
actual
node
and
anyway,
we
don't
have
time
to
go
into
all
what
all
these
demons
are,
but
just
know
that
so
rook
is
deploying
these
pods
and
services
and
other
kubernetes
resources
to
manage
seth
and
and
the
csi
drive.
A
A
Okay
and
the
same
basic
pattern
happens
for
the
shared
file
system.
I
mean
you've
got
the
pvc
that
request
the
storage
from
this
ffs
storage
class.
Now
cfs
is
the
shared
file
system
and
then
the
csi
driver
for
cephas
is
the
one
that
provisions
the
volume
and
again
mounts
it
for
the
application
pod
and
the
third
type
of
storage
we've
got
here
is
object
with
the
s3
or
rest
endpoint,
and
we've
got
in
rook
a
bucket
claim
a
concept
very
similar
to
pvc.
But
it's
for
object,
storage.
A
Okay,
so
now
that
we've
deployed
rook
and
layer,
one
we've
got
the
csi
driver,
attaching
all
the
storage
to
your
pods
in
layer
two.
Now
it
comes
time
to
actually
write
data
to
seth
player
three.
So
when,
in
this
this
picture
the
layer,
one
and
two
are
really
out
of
the
picture.
We're
just
talking
about
your
application
now
needs
to
write
data.
A
You
know
the
the
client
just
talks
to
the
volume
or
the
s3
I
use
the
s3
client
like
it
would
any
other
targeting
s3
or
targeting
volume
and
underneath
the
covers
set.
The
best
kernel
driver
handles
the
connection
to
ceph
or
the
s3
client
handles
the
connection
to
the
rdw
endpoint.
For
that
object,
storage,
okay,
well
time
is
flying.
I
wish
we
had
even
more
time
to
dive
in
here.
So
what
does
it
take
to
get
started
with
rook?
A
A
That
gives
you
your
rbac
settings
gives
privileges
to
the
operator.
Basically,
so
you
create
the
operator,
and
then
you
define
how
you
want
rook
to
deploy
the
seth
cluster
in
the
cluster
yaml
on
the
right
here.
We've
got
a
snippet
that
we
don't
really
have
time
to
talk
about,
but
basically
we
tell
we
tell
this
tells
rook
how
to
deploy
ceph
in
your
cluster
once
you've
deployed
seth,
then
again
that
layer
1,
this
picture
kind
of
shows
the
layer
two
and
three,
where
you
define
the
storage
class.
A
A
A
Do
you
have
local
devices
or
do
you
have
something
else
so,
whether
you're
in
the
on
bare
metal
or
in
the
cloud
you
need
you
can
decide?
Well.
Do
I
want
to
provision
storage
that
sef
uses
provision
that
from
a
pv?
If
I'm
in
the
cloud
like,
maybe
I
want
to
to
back
ceph
by
an
ebs
or
you
know,
google
volume,
whatever
storage
provider
or
cloud
provider,
I'm
in
I
should
be
able
to
consume
its
storage
and
put
stuff
on
top
of
it,
and
we'll
talk
more
about
that
in
a
minute.
A
In
that
scenario,
or
if
I'm
in
my
data
center
and
and
I
just
have
raw
devices-
okay,
maybe
I
want
to
use
all
raw
devices.
Maybe
I
want
to
list
all
the
nodes
and
devices
individually,
because
I
don't
trust
rook
or
don't
want
rook
to
use
all
devices,
or
maybe
I
want
a
lot
more
flexibility
for
these
for
a
new
concept,
called
stef
drive
groups
and
roof
lets.
You
configure
those
drive
groups
in
our
latest
release.
A
And
again,
when
you're
setting
up
your
cluster
topology
your
you
got
to
think
about
your
failure
domains.
You
know.
Where
do
you
you
know?
If,
if
you
have
a
node
or
or
a
zone
whole
zone
go
down,
you
know.
How
can
you
keep
your
your
cluster
working?
You
want,
you
need
to
spread
the
different
root
components
and
set
components
across
those
failure
domains,
because
if
one
failure
domain
goes
down,
if
it's
set
up
properly,
your
storage
keeps
on
working
and
it
handles
that.
It's
designed
for
that
resiliency.
A
And
also,
you
know
as
you're
doing
that
you
know:
rook
is
as
flexible
as
possible
within
the
means
that
kubernetes
provides.
So
you
can
have
node
affinity.
You
know
you've
you're,
tainting
your
nodes
and
you
can
add
tolerations
to
rook.
You
can
pod
and
affinity
et
cetera,
whatever
options
for
placement
that
kubernetes
offers.
We
do
our
best
to
expose
that
through
book
and
one
more
so
back
to
this
topic
on
cloud
environment.
So
why
would
you
want
to
run
rook
and
ceph
in
a
cloud
environment?
You
might
ask,
because
the
cloud
provider
already
has
storage.
A
Well,
there's
a
few
important
concepts
here.
So
first,
you
know
having
a
consistent
storage
platform
wherever
kubernetes
is
deployed
is
a
powerful
concept.
Rook
gives
you
that
storage
abstraction
where
you
can
run
rook
everywhere,
kubernetes
runs
store.
Our
cloud
providers
do
have
shortcomings
that
we
hear
about.
You
know
whether
it's
having
storage
across
azs
or
there's
slow
failover
times
limitations
of
the
number
of
pvs
per
node
and
even
proof
characteristics
of
a
large.
You
want
the
perfect
characteristics
of
large
volumes,
not
small
volumes,
so
you
get
that
with
with
rook
and
then.
A
A
B
Yeah
thanks
travis,
so
now
that
we
are
familiar
with
what
rook
is
what
it
does
and
also
what
ceph
is
and
what
it
can
provide
to
us
in
terms
of
functionality.
Then,
let's
dive
into
some
of
the
key
features
that
rook
provides
to
you
all
right.
So
one
of
the
beauty
of
rook
is
that
everything
is
automated
and
that's
it's
it's
essentially,
it
is
really
what
the
goal
of
every
single
operator
is
is
is
to
take
all
of
that
operational
knowledge
and
just
just
put
it
into
a
logical
entity.
A
B
It
could
really
benefit
of
years
of
experience
of
managing
upgrades
and
actually
upgrades
is
probably
always
one
of
the
most
painful
and
difficult
thing
to
to
achieve
when
it
comes
to
software,
so
fortunately,
rook
is
here
for
you
and
rook
can
handle
all
of
that.
So
there
are
two
things
right.
The
first
one
is
upgrading
rook
on
its
own
and
it's
super
easy.
B
Obviously,
once
we
we
do
a
big
step
to
a
new
version
of
rook.
For
example,
we
we
just
released
1.4.
So
if
you
go
from
1.3
to
1.4,
then
there
are
steps
that
you
would
have
to
apply
so
to
using
using
the
the
upgrade
guide
so
that
crd
definitions
can
be
updated
as
well
as
necessary.
R
backs
to
to
have
rook
keep
keep
on
working
and
then
probably
the
most
interesting
piece
here
is
the
ceph
upgrade
on
its
own,
where
rook
will
really
be
handling
everything
and
again
it
is
really
simple.
B
The
only
thing
you
have
to
do
here
is
just
to
update
the
setcaster
custom
resource
and
change
its
image
by
a
new
one.
Rook
supports
pinpoints
updates,
as
well
as
major
upgrades
of
ceph
and
all
of
the
intricacies
and
all
of
the
details
that
you
have
to
all
the
things
you
would
be
doing
manually
per
se.
B
Rook
will
go
one
by
one
demon
by
demon
and
make
sure
that
they're
all
healthy
before
moving
to
the
next
one
as
part
of
its
own
reconciliation,
so
yeah
upgrades
are,
I
would
say
finally
made
really
simple
and
really
easy
to
do,
and
actually
we
just
have
really
good
feedback
from
the
community.
It's
really
super
straightforward
to
upgrade
rook
and
not
painful
at
all,
so
moving
on
now
to
the
csi
driver.
Obviously,
as
travis
mentioned,
we
can
deploy.
B
Rook
deploys
ceph
maintains
it
through
its
entire
life
cycle,
but
there
is
no
point
of
simply
deploying
storage
technology.
Also.
What
you
have
to
do
is
to
use
it
so
that
you
can
provide
persistent
storage
to
your
containers,
and
this
is
where
csi
and
particularly
sap
csi
plays
a
big
role
as
part
of
the
latest
version
of
rook.
We
also
introduced
and
released
at
the
same
time
a
three
dollar
version
of
the
csi
driver.
B
As
we
saw
earlier,
it
still
supports
dynamic
provisioning
for
all
the
access
methods
rwx
and
for
both
block
devices,
as
well
as
the
system,
but
what's
really
new
about
this
is
really
the
snapshots
and
cloning
functionality
that
are
actually
better
and
ready
to
be
consumed
as
some
kind
of
tech
preview.
Let's
say
also,
we
still
have
support
for
flex
drivers,
so
if
you're
still
using
it,
then
upgrade
will
be
supported
and
you
you
can
continue
on
using
it.
But
obviously
this
is
where
the
community
is
moving
communities
moving
out
of
flex
driver
for
csi.
B
So
it's
it's
really
highly
recommended
to
move,
but
to
csi,
but
still
we
we
support
flex,
all
right
x,
no
cluster
connection
that
was
probably
one
of
the
most
desired
feature
two
or
three
releases
ago,
and
essentially
what
what
we
do
here
is
that
it
is
not
always
about
green
fields.
Right,
not
everybody
has
moved
to
fully
capabilities
environments
yet,
and
maybe
some
of
you
will
never
do
because
you
already
have
a
brownfield
environment
and
your
subcluster
is
already
there.
Maybe
it's
serving
other
purposes.
Maybe
it's
connected
to
openstack.
B
Maybe
it's
connected
to
your
proximos
hypervisor,
whatever
it
might
be,
and
what
this
excellent
mode
allows
you
to
do
is
basically
start
this
consumer
producer
relationship
where
the
producer
is
the
external
tester
and
the
consumer
at
this
point
is
rook.
So
the
major
difference
here
is
that,
at
this
point
won't
be
managing
the
resources.
It
will
only
consume
them
and
the
beauty
of
that
is
that
it
is
really
simple.
So
the
only
thing
you
have
to
do
is
to
inject
a
couple
of
details
from
your
external
cluster
and
and
you're
ready
to
go.
B
We
have
already
discussed
that
briefly
in
one
of
the
diagrams
that
travis
showed
you
earlier,
but
essentially
it
is
really
similar
to
the
pvc
interface.
Where,
at
this
point
you
don't
do
block
devices
you
just
do
bucket
as
a
user,
you
don't
want
to
really
go
into
the
let's
say:
well,
the
pain
of
having
to
create
a
bucket
manually
having
to
create
a
user
manually.
The
only
thing
you
care
about
is
give
me
a
bucket.
B
B
B
First
off
we
maltus
first,
we
we
introduced
multis
during
the
1.3
cycle
and
we
we
actually
marked
it
as
experimental.
The
reason
why
we
marked
it
as
experimental
is
because
we
we
have,
we
had
half
of
the
picture
complete
and
what
I
mean
by
that
is
in
one
three.
We
were
already
able
to
consume
motors
by
basically
bootstrapping
the
entire
roof
cluster
using
dedicated
network
interfaces,
but
was
what
was
not
available
yet
is
the
connection
through
csi?
B
B
So
if
we
step
back
a
little
bit
like
in
malta's,
why
would
you
need
to
have
notice
if
you,
if
you're
using
a
burglar
environment,
and
then
you
really
want
to
take
advantage
of
all
the
network
interfaces
that
you're
that
that
basically
both
for
for
your
hardware,
then
this
is
where
multis
comes
into
play?
We
don't
really
want
to
use
the
host
networking
mode
because
it
has
some
security
implications,
since
we
expose
the
entire
network
stack
from
the
host
into
containers,
but
with
multis.
B
Still
we,
the
way
multis
is
supported,
is
through
the
whereabouts
ipam.
It
is
really
preferred
and
that's
the
the
essentially
the
only
one
we
support
right
now.
It's
probably
the
one
that
works
best
at
scale,
because
there
is
no
dhcp
involved
and
and
whereabouts
does
its
own
ip
management
internally.
B
So
it
is
a
more
than
a
year
old
and
of
really
active
development.
So
obviously
we
don't
have
the
full
picture
yet,
and
things
are
missing,
like
supporting
services,
we
still
cannot
create
services
that
are
backed
by
multis
networks,
basically,
but
this
is
coming
and
what
we
have
today
is
really
consumable.
So
we're
super
excited
about
this.
A
B
Is
a
feature
that
has
been
around
in
ceph
for
its
object
gateways
for
years
now,
and
it's
probably
one
of
the
most
desired
feature
that
community
has
been
asking
for
for
the
past
year
is
the
basically
ability
to
do
object
through
multi-site
either.
You
have
geographically
separated
data
center
between
regions.
Maybe-
and
the
idea
here
is
to
replicate
objects
between
rook
set
clusters
are
that
are
distanced
distant
to
to
each
other's
with
that?
B
B
So
it
is
marked
as
experimental,
because
we
only
have
the
controllers,
as
well
as
the
new
custom
resources
and
in
one
five
we
will
be
working
to
basically
get
all
the
pieces
together
to
get
a
complete,
a
complete
experience
and
mission
controller.
You
cannot
actually
validate
everything
through
the
api
validation
from
your
line
from
your
yaml
definitions.
Also,
you
don't
necessarily
want
to
do
all
of
your
cr
cr
spec
validation
through
the
controller,
so
through
the
operator,
because,
essentially
once
you
do
that
it
is
a
little
bit
too
late.
B
B
The
toolbox
the
toolbox
has
been
around
for
like
since,
since
ruka
actually
started,
it
is
a
little
deployment
that
allows
us
to
get
hands-on
access
to
the
step
environment.
You
just
bootstrap
it
like
a
normal
part
and
once
you
exec
into
it,
you
can
run
any
of
the
staff
comments.
Just
like
you
would
be
like
just
like
you
would
do
normally
and
in
1
4.
What
we
added
essentially
is
jobs
that
would
do
specific
actions
such
as
day
two
operations.
B
Again,
we
don't.
The
main
point
of
rook
is
to
make
storage
easy,
easily
consumable,
easily
configurable
and
easily
maintained
and
self-maintained
as
well
through
rook.
So
the
idea
here
is
to
have
no
manual
integration
and
we
have
jobs
that
will
do
specific
actions,
for
example,
if
if
a
disk
is
is
dead,
then
we
can
simply
remove
it,
and
for
that
we
would
have
a
specific
job
that
we
care
take
care.
All
of
that
will
take
care
of
that.
B
B
We
improved
our
external
mode,
as
mentioned
before.
This
was
really
one
of
the
best
features
that
we
delivered
in
one
three
and
we
have
been
building
on
top
of
what
we
had.
So
it's
even
more
stable,
even
more
robust
right
now.
We
added
the
support
for
gathering
external
metrics
from
the
external
cluster
through
the
theft
manager
promised
exporter.
B
So
it's
basically
getting
the
best
out
of
prometheus
onto
an
external
environment
which
is
which
is
kind
of
cool.
Also,
we
extended
the
external
support
to
the
staff,
object
store.
So
now
you
can
actually
connect
to
external
gateways
and
they
will
be
integrated
in
the
exact
exact
same
fashion
as
when
you
do
the
converged
mode.
So
you
just
pass
gateway,
iap
addresses
and
they
will
be
integrated
as
a
kubernetes
service.
So
that
is
also
a
really
nice
feature
and
much
more.
B
Unfortunately,
we
don't
have
much
time
to
discuss
into
many
details
all
the
good
things
all
the
good
contributions
we
received
in
that
wonderful
1
4
cycle,
but
some
of
the
few
last
items
worth
mentioning
encryption
for
osd
on
pvc,
so
essentially
just
doing
encryption
at
rest.
On
the
on
the
drive,
health
checks
and
liveness
probe
configuration,
rook
has
some
internal
health
checks,
like
checking.
B
B
Now,
not
only
you
can
clean
up
the
drive,
but
you
can
also
apply
specific
and
actually
more
powerful,
sanitizing
functionality
for
for
your
drives,
also
just
like
travis
mentioned
earlier,
we
that
there
is
a
new
self-drive
group
standard
to
define
osd
drives,
and
this
has
been
embedded
into
the
the
step
cluster
cr,
and
I
guess
with
that.
This
really
concludes
our
presentation.