►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Rook - Travis Nielsen, Red Hat & Alexander Trost, Cloudibility
The Rook operator implements custom resource definitions (CRDs) to express the desired state of storage providers for Kubernetes. This deep dive will review the framework Rook provides to integrate the storage providers with an operator and CRDs. As an example, details of the Ceph operator will be shown, including how it builds on the Rook framework and how Ceph’s specific orchestration needs are met. The Ceph mons require special handling to stay in quorum and handle failover. Ceph OSDs require several stages of discovery and provisioning before the daemons are started. The Ceph mgr runs an active and standby daemon for high availability. For object storage, Rook creates all the pools and starts the rgw daemons needed. For a shared file system, Rook creates the pools and starts the MDS daemon with a standby. These and other challenges with the Ceph daemons will be discussed.
To learn more: https://sched.co/GrdU
A
Welcome
to
the
rook
deep-dive
session,
I'm
travis
nielsen,
one
of
the
maintainer
Jean,
the
rook
project,
I've
been
with
the
project
since
its
I
was
originally
created
a
couple
years
ago
we
open
sourced
it
at
coop
con
Seattle
when
it
was
a
little
bit
smaller.
It's
amazing
to
see
how
much
it's
grown
and
the
project
is
grown
and
it's
great
to
be
a
part
of
it.
A
Earlier
this
year
I
joined
the
Red
Hat
team
got
to
work
with
the
the
Ceph
team,
so
all
right,
so
what
is
rook
so
to
get
an
idea
of
who's
in
the
audience
so
who
all
was
able
to
make
it
to
the
intro
to
rook
a
couple
of
days
ago?
So
a
lot
of
you,
that's
good
and
Jared
also
he's
here.
He
gave
it
and
who,
who
has
actually
run
rook
before
in
test
or
other
cluster.
All
right.
A
We
got
a
lot
of
users,
great
good,
to
see
all
of
you
here
and
who's
running
in
production,
okay.
Yes,
we
do
have
a
good
collection
of
production
or
near
production
users
tomorrow,
okay,
perfect.
Those
are
the
stories
I
love
to
hear
that
it's.
You
know
that
it's
working
for
you
that
the
production
is
ready
for
it,
and
you
know
that
you
know
earlier
this
year.
Well,
September.
We
graduated
to
be
an
CN
CF
incubation
project,
thanks
to
all
of
you,
the
community
and
knowing
that
it's
working
in
production.
A
So
what
so?
What
is
rook
as
I've
talked
to
people
in
the
booth?
The
last
few
days,
I've
kind
of
refined?
How
I'm
talking
about
it
and
sort
of
the
one-liner
I
came
up
with,
is
so
it's
open
source.
It's
and
it's
a
storage
couldn't
control
plane
and
it's
for
kubernetes,
so
buy
storage
control
plane.
What
that
means
is
it
provides
operators
and
CR
DS
to
orchestrate
or
otherwise
configure
and
manage
storage
backends,
so
we'll
get
into
more
what
that
means,
but
basically
think
about.
We
don't
provide
the
implementation
for
the
storage.
A
We
bring
the
storage
platforms
to
kubernetes
in
a
kubernetes
native
way.
So
I'll
go
briefly
over
this.
That
you've
already
heard
in
the
intro
section,
but
so
an
operator
like
you
heard
in
the
keynote
this
morning
the
operator
applies
desired
state
into
your
cluster.
So
you
say:
here's
my
desired
state.
The
operator
goes
and
makes
it
happen,
and
the
way
you
do
that
is
with
the
CR
DS.
A
Let's
see
our
DS
or
your
standard
kubernetes,
the
ml
file,
it
looks
like
any
other
kubernetes
type
and
but
you
know,
ruk
defines
those
CR
teas
that
are
specific
to
the
storage
providers,
and
so,
in
addition
to
that,
it
look
really
is
more
than
operators
and
and
CR
DS.
So
a
framework-
and
this
framework
is
it's
still
and
it's
in
its
youth,
but
we're
evolving
it
to
make
it
even
easier
for
storage
providers
to
join
the
kubernetes.
A
Today's
native
landscape,
so
testing
efforts
anyway.
All
these
things
there's
a
lot
of
very
specific
needs
for
each
storage
technology
and
among
those
you
know,
automated
deployment,
bootstrapping
configuration,
upgrading
migration
and
every
storage
platform
needs
to
do
these
if
you're
going
to
trust
them
with
your
data.
A
So
what
are
the
storage
providers
in
Rooke
today?
So
we
we
have
kind
of
what
I'm
thinking
of
as
two
categories
of
storage
providers.
We
have
those
that
provide
the
data
platform
itself.
You
know
block
object
and
file
storage,
so
SEF
Mineo
NFS
and
they
exempted
Jeff
s
fall
into
those
that
category.
The
databases
then
are
cockroach,
DB
and
Cassandra,
which
which
might
build
on
the
other
data
platforms.
A
They
need
to
be
always
running
so
that
your
data
is
available
and
safe.
So
the
the
general
orchestration
approach
we
took
for
rook
was
the
operator
it's
going.
It's
it's
very
familiar
with.
What's
F
needs
to
do
so.
Steph
needs
to
you
know,
creates
ethics
keys
and
generate
it.
We
need
to
create
crush
map.
We
need
to
do
other
things
to
you
know
with
placement
groups
and
if
you've
run
rook
but
not
SEF.
A
Hopefully
you
haven't
had
to
hear
about
any
of
those
terms
because
they're,
you
know
they're
difficult
to
manage
at
the
end
of
the
day.
So
rook
takes
care
of
those
those
for
you,
the
operator,
then
after
talks
to
staff
or
Wallace
talking
to
SEF
it's
creating
kubernetes
resources
like
deployments
to
manage
life
cycle
of
each
SEF
demon.
A
You
know
doing
things
like
creating
Anette
containers
to
generate
the
SEF
kampf
and
then
running
the
SEF
daemons
directly
with
with
a
version
of
SEF
that
that
you
want
to
run
and
I'll
talk
more
about
that
versioning
here
soon
and
then
the
last
thing
that
the
operator
does
is
it
really
manages
the
health
of
the
system
if
your
SEF
mons
go
down,
kubernetes
doesn't
really
know.
If
you
know,
if
they're
really
down
it
only
knows
if
the
pod
stops.
A
So
the
operator
does
extra
work
to
say:
hey
are
your
SEF
mons
in
Coram
and
if
they're
not
it'll,
take
action
to
start
up
a
new
deployment
and
get
a
new
man
running
okay.
So
that's
the
approach
start.
You
know,
configure
your
storage
in
a
very
storage
provider
specific
way
and
keep
it
healthy.
So
what
does
a
CRD
look
like?
So
in
our
release,
our
dot?
Nine
that
we
just
had
a
few
days
ago,
we
declared
the
SEF
CR
DS
as
v1,
so
we're
excited
about
that.
A
That
means
that
users
are
trusting
it
while
it
was
in
beta
in
production
even
and
we're
yeah.
We're
just
excited
that
that
that's
going
to
keep
rolling
out-
and
that
means
you
know
they'll
always
be
backward
compatible,
upgradeable
and-
and
we
believe
you
know
that
that
you
can
do
what
you
need
to
with
these
well
be
adding
new
features
that
doesn't
mean
they're
locked
in
place
that
will
be
backward
compatible,
always
so
just
to
look
at
a
few
of
these
properties
here
at
the
CRT
dated
or
host
path
here
on
the
right.
A
So
what
this
is
is
some
of
the
SEF
demons
require
persistence,
its
storage,
of
course,
if
you're,
if
your
node
reboots,
if
it
just
goes
down
momentarily
and
comes
back,
you
need
to
make
sure
that
data
is
persisted.
So
the
host
path
is
where
we
store,
you
know
basic
configuration
and
persistence
of
the
metadata.
A
It
can
hold
data
as
well,
but
the
data
is
generally
held
on
your
raw
block
devices
mounted
on
on
your
local
nodes,
the
Ceph
version.
Let
me
skip
over
that
one
that
I've
got
another
slide
for
that
settings
for
dashboard.
Do
you
want
the
Ceph
dashboard
enabled
or
not
and
network
yeah?
What
kind
of
network
are
you
running
in?
Do
you
need
host
network
access,
or
are
you
running
on
a
virtual
network
or
with
the
kubernetes
CNA,
so
the
the
versioning?
So
in
the
the
latest
release?
A
One
of
the
big
changes
we
made
is
that
rook
is
no
longer
tied
to
a
specific
version
of
SEF,
so
rook
was
tied
to
Stef
12.2,
which
was
luminous
before
this
release.
But
now
you
can
say:
I
want
to
run
rook
V
dot,
9
o
die
9,
but
I
want
to
run
SEF,
luminous
or
SEF
mimic
or
SEF
Nautilus,
and
so
the
version
of
SEF
is
independent
of
what
rook
is
running.
This
is
very
important
because
people
care
a
lot
about
their
data
plane.
A
They
need
to
know
it's
safe
and
they
need
the
admins
need
to
decide
when
they
want
up
creating
their
data
plane.
So
that's
the
purpose
of
that
separation,
so
you
can
control
when
to
upgrade
the
data
plane
and,
yes,
you
can
up
to
upgrade
independently
from
rook,
and
the
support
lifetime
really
then
depends
on
the
Ceph
project,
not
on
the
rook
project,
for
a
specific
version
of
stuff.
A
Alright.
So
what
are
the
differences?
It's
the
operator
actually
behaves
differently
between
each
of
these
versions
of
stuff,
so
aluminous
and
just
these
are
the
main
examples
right
now
and
over
time,
this
difference.
These
differences
could
grow
in
implementation.
But
so
right
now
with
luminous,
the
operator
will
start
up
the
SEF
dashboard
running
on
port
and
just
on
port
80.
It's
it's
a
read-only
dashboard.
It
doesn't
need
anything
special
or
less
special
in
mimic.
Then
now
it's
a
configurable,
more
feature-rich
can
dashboard
in
SEF.
A
It
needs
to
run
in
HTTP,
so
rook
will
generate
a
self-signed
cert
on
your
behalf
and
generate
an
admin
password
and
store
it
somewhere
safe
for
you,
so
that
you
can
have
a
working
dashboard
right
when
you
launch
and
then
in
Nautilus,
which
is
still
pre-release.
It's
you
know
it's
planned
to
be
released.
Maybe
the
next
couple
of
months
rook
will
allow
you
to
run
it.
A
A
The
upgrades
of
rook
are
mostly
automated.
Now
it's
it's
a
lot
easier
than
ODOT
eight
who's
who
in
the
past
has
upgraded
a
rook
release.
A
few
of
you,
probably
okay.
There's
we
have
a
big
long
guide.
A
lot
of
manual
steps.
Those
manual
steps
are
much
shorter.
Now,
there's
really
just
a
couple
of
manual
steps
and
I'll
I'll
get
into
a
demo
of
what
the
upgrades
look
like
now.
A
So
it's
it's
really
about
the
operator.
Managing
that
complexity
to
you,
because
or
for
you,
because
if
you
have
to
manage
any
anything
manually,
it's
it's
bound
to
fail
and
then
your
date
is
at
risk,
so
what
it
looked
like
to
upgrade
rook,
so
you
tell
with
this
command
at
the
bottom
of
the
screen.
Here
you
can
tell
rook
what
version
you
want
to
run.
It's
really
just
the
image
of
the
rook
operator.
A
That's
running
so
you
say:
coop
cuddle,
set
image
on
this
kubernetes
deployment
and
here's
the
new
image
version,
so
rook,
slash,
staff
vo9,
is
what
you
would
use
to
upgrade
from
ODOT
eight
currently
so
this
to
be
clear,
so
it
updates
the
rook
operator
and
the
related
orchestration
pieces,
but
it
does
not
update
the
data
path
so
after
the
upgrade
you'll
still
be
running
luminous,
just
like
you
were
running
before
or
not
on
a
date
and
okay,
so
the
Ceph
upgrade.
What
does
that
look
like
this?
A
If
the
data
plan
needs
to
be
upgraded
or
data
migration,
we
don't
currently
have
any
need
for
a
data
migration
between
these
versions
of
Ceph,
but
it's
possible
in
the
future,
depending
on
what
changes
you're
made
all
right.
So
what
do
I
want
to
do
with
the
demo
now
is,
first
of
all,
I'm
going
to
configure
luminous
and
then
update
that
from
luminous
to
mimic
and
we'll
watch
what
how
the
operator
deals
with
that.
A
A
Shouldn't
be
doing
window
management
during
a
demo,
all
right
so
Oh
to
do
a
demo
efficiently.
I
thought
I
better
keep
using
my
same
aliases,
I
always
use.
So
let
me
just
show
you
those
real
quick,
so
you're
not
totally
confused
about
what
I'm
doing
so
here.
My
alias
is
Kay
for
group
cuddle
KN
to
do
it
in
the
rook,
namespace
and
etc
a
command
to
look
at
the
operator
log
all
right.
So,
though,
that's
what
I'm
running
all
right!
A
So,
let's
start
the
cop,
the
operator
first
assignment
I
have
my
V
o.9
manifests
here,
so
I'm
gonna
say
create
the
operator
and
before
I
hit
go
on
that
I
wanted
to
show
down
here
what
resources
are
running.
So
no
pods
are
running
you
up
for
the
operator.
So
after
I
hit
go
on
that
it's
going
to
create
a
namespace
for
the
operator
and
then
it's
going
to
start
start
the
operator
deployment.
We
should
see
a
pod
any
moment
here
after
you
operator
itself
starts.
A
It
starts
up
a
couple
of
other
demons
that
need
to
run
on
each
node.
There's
the
agent
which
manages
mounting
the
storage
from
your
PB
sees
with
our
flex
driver
ORS
and
in
the
future
that
will
be
moving
to
CSI
the
discovery
pod.
This
one
discovers
hey
what
what
storage
devices
do
I
have
on
each
node
in
the
system,
make
sure
I
discover
it
and
then
the
operator
you
can
reconcile
what
it's
going
to
configure
on
each
node.
Okay,
so
the
operators
running
nothing
new
here
from
the
intro.
A
A
So
this
is
starting
up
the
SEF
cluster.
The
first
thing
it
does:
is
it
detects
what
version
of
Stef
you
want,
so
it
should
have
detected
luminous
and
it's
going
to
start
creating
sechs
Kies
store
them
in
kubernetes
secrets
and
any
moment
now
we
should
see
the
first
Mon
pod.
There
we
go
so
each
Mon
is
going
to
go
through.
A
So
after
it
gets
to
the
Mons,
then
it
will
start
up
the
ACEF
manager,
which
is
where
the
dashboard
and
the
Prometheus
modules
are
running
and
then
it'll
get
to
the
OS
DS.
So
with
the
OS
DS,
we
get
this
prepare
pod.
You
see
here
at
the
bottom,
so
this
one
it
goes
out
to
the
node
and
it
says.
Ok,
you
told
me
to
configure
these
devices
on
this
node
and
for
my
configuration
I
just
told
it
you
know
configure
all
devices
just
whatever
you
find
consume,
so
it
what
it
should
be
doing.
A
Is
it
found
three
devices,
SDA
SDB,
SDC
I
think
they
are
and
then
it
tells
F
hey
configure
each
of
these
devices
with
OS
DS
and
after
those
OS
T's
are
configured.
It
goes
back
to
the
operator
and
says:
ok,
I'm
done
now
go
ahead
and
start
a
pod
for
each
one
of
these
devices,
and
now
we
see
those
being
created
so
that
each
OSD
can
run
independently
and
if
one
pod
fails,
it
doesn't
affect
other
OSDs
running
on
on
the
same
node.
A
Alright,
so
we
have
the
basic
cluster
up
and
running
and
to
show
I
guess
we're
on
Luminess.
What
I
want
to
do
is
is
bring
up
the
dashboard
so
to
expose
the
dashboard
on
my
on
my
Mac
host
I've
just
got
to
open
up
a
node,
pour
it
into
the
mini
cubm.
So
let
me
I
need
to
create
a
dashboard
on
HTTP.
So
since
Luminess
is
on
HTTP
now
let
me
get
the
port
that
that's
on
and
the
mini
Q
by
P
and
then
I'll
go
to
my
browser.
A
Okay,
so
the
here's,
the
IP
and
the
port
the
dashboard
is
running
on
is
31
374
whoops,
that's
from
a
previous
run-through.
Okay!
So
we're
on
this
IP
with
31
374
same
port.
Okay,
so
here
is
the
the
Ceph
dashboard
running
on
luminous.
So
it's
you
know
read-only!
We
can't
do
a
lot
with
it
all
right,
it's
running
so
now,
let's
go
ahead
and
update
to
mimic
I,
don't
like
this
dashboard
I
want
the
latest
and
greatest
dashboard.
A
A
So
as
soon
as
I
save
this
CRD,
the
operator
is
going
to
wake
up
and
it's
going
to
say:
oh
you
want
luminous.
Now
I
mean
sorry.
You
had
luminous.
Now
you
want
mimic
I'm
going
to
go
through
and
update
each
of
your
demons
that
are
running
and
any
additional
configuration.
That's
that's
needed
for
that
version,
for
example
running
the
mimic
dashboard
as
a
secure
site
now.
So
let
me
save
this
and
we'll
see
what
starts
happening
on
the
bottom
with
those
pods.
A
Okay,
so
immediately
see
the
a
pod
that
says
I'm
going
to
detect
the
diversion
it's
already
done.
It
detected
mimic
and
now
it's
going
through
the
Mons.
The
moms
are
always
first
to
get
updated
one
mile
at
a
time
updated
to
mimic.
So,
while
that's
going
I'm
going
to
look
at,
what's
happened
with
with
them
on
so
describing
them
on.
A
Okay,
so
now
to
make
sure
we
have
mimic
running
now.
What
I
want
to
do
is
start
at
the
dashboard,
see
it's
on
HTTPS
now
so
I
have
a
separate
service.
I
need
to
create,
so
my
laptop
can
look
at
it
in
the
browser,
so
dashboard
HTTPS,
alright,
and
let
me
look
at
what
which
can
get
service
so
now
HTTPS.
If
I
tried
this,
try
this
old
port.
Now
it's
the
dashboard,
isn't
running
there
anymore,
but
now
I
need
to
look
at
this
secure
port.
A
Okay.
So
if
I
refresh
I
can't
get
the
the
dashboard
on
HTTP
anymore,
it's
just
not
there!
So
let
me
put
HTTPS
with
the
Newport
and
okay.
So
it's
a
self-signed
certificate.
So
we
get
this
big
red
warning:
don't
connect
to
the
site,
well,
I'm,
going
to
anyway
for
the
demo!
Thank
you
very
much.
Okay
and
here's.
The
mimic
dashboard
rook
generated
an
an
admin
password,
so
it's
kind
of
small.
So
this
when
it
generated
the
password
it
put
it
in
a
kubernetes
secret
and
I
need
to
go
get
that
secret.
A
A
A
A
So
now
one
more
thing
I
want
to
make
sure
I
show
a
couple
more
things:
we
have
a
the
toolbox
if
I
could
you're
right,
I'll
create
the
toolbox,
and
so
what
the
toolbox
pod
is
just
a
convenient
place
to
run.
Sef
commands
so
I'm
gonna
connect
it
to
this,
and
just
show
you
how
simple
it
is
to
look
at
this
F
status
so
connect
to
this
we're
inside
the
toolbox.
Now,
which
knows
how
to
talk
to
SEF,
so
I,
say:
hey
Steph,
give
me
the
status
and
well,
we
already
saw
in
the
dashboard.
A
We
had
the
Mons
and
everything
running
and
I've
got
an
error.
I
need
to
investigate
after
this,
but
the
cluster
is
all
running
all
right
great.
So
so
what
happens
with
persistence
now
on
the
host?
So
remember
how
we
put
the
persisted
configuration
in
barley
Brook.
So
let
me
just
connect
so
I'm
going
to
SSH
into
my
mini
qbm
and
just
show
you
basically
what's
you
know
what's
inside
the
host
path?
Okay,
so
if
I
go
into
Varela,
Brook
I
see
a
directory
for
each
of
the
mons.
A
Now,
I
only
have
one
note
here
which
isn't
a
terribly
interesting
cluster,
so
they're
all
on
the
same
node,
you
know
in
any
production
environment
the
mons
would
need
to
be
on
different
nodes
and
the
OS
DS
yeah,
whatever
devices
you
have
there.
So
the
what's
in
the
Mon
directory
is
not
terribly
interesting,
but
you
see
there's
a
data
directory
which
persists
the
important
Mon
data
and
same
thing
for
OS
DS.
A
So
these
are
a
little
more
interesting
where
the
osts
are
what
are
what's
holding
your
data
behind
the
scenes
and
I've
configured
blue
store
on
top
of
the
device,
so
it
created
partitions
for
for
the
data.
So
the
block
has
a
symlink
over
to
that
partition
for
the
data,
the
boost
or
database
and
the
blue
store
right
ahead,
log
and
other
basic
configure.
So,
that's
in
a
nutshell
where
this,
where
your
date
is
where
your
data
is
stored,
all
right.
So
let
me
go
back
to
the
slides
now
and
move
on.
A
So
there's
a
lot
of
options
in
the
in
the
CR
D
that
we
we're
running
out
of
time
on
already
talked
about,
but
Mons
at
the
end
of
the
day,
you
need
to
have
an
odd
number
of
them
and
you
need
to
control.
Are
they
on
separate
nodes,
so
so
usually
3
mons
as
a
default?
It's
tolerant
of
a
single
failure
place
them
on
unique
nodes.
A
A
It's
okay
delete,
deploy,
rooks,
F,
Mon
Hey,
alright,
so
you're
gonna
see
that
pod
go
away
now,
and
at
this
point
the
the
Mon
is
going
to
be
out
of
quorum
really
quickly.
The
operator
you
know,
does
its
health
checks
periodically
I
turned
the
the
frequency
down
for
the
demo,
so
within
the
next
15
and
30
seconds,
we
should
see
the
operator
noticing
the
the
Mon
is
gone
and
bring
up
a
new
one.
A
So
if
I
look
at
the
operator
log,
we
probably
see
that
yep
okay
is
not
in
quorum
and
the
it's
going
to
wait
a
little
bit.
Is
this
just
a
network
glitch?
Is
this
something
that
I
need
to
recover
by
default
operator?
We'll
wait,
wait
five
minutes
before
it
starts
up
anew
and
but
yeah
we
should
see
really
soon
another
one
starting
now
and-
and
here
it
is
so
decided,
yep
it's
dead.
A
A
Okay,
so
it's
coming,
it
should
refresh
itself
momentarily
all
right.
Let
me
come
back
to
this
in
just
a
minute
and
make
sure
that
that's
exceeded,
but
the
new
Mon
came
up.
The
old
one
was
taken
out
of
out
of
the
quorum,
so
OS
DS
there.
There
are
a
lot
of
ways
to
configure
your
storage
devices.
How
do
I
tell
work
to
go
consume?
You
know
my
my
device
is
on
my
heterogeneous
environment.
A
Everybody
has
very
specific
needs
for
how
they
run
it.
But
overall
you
know
what
Brooke
does
it
schedules
a
job
on
each
node
and
then
creates
a
deployment
for
each
OSD
when
new
new
nodes
and
devices
are
added,
so
the
clustered
rook
can
also
can
also
add
those
to
the
cluster.
Although
currently
it
does
require
an
operator
restart
to
pick
those
up.
A
Usually
all
right.
There
are
different
modes
for
selecting
OSDs,
so
the
first
mode
that
I
was
using
is
automatic
selection,
where,
on
the
right
here,
I
just
said,
use
all
nodes
use
all
devices.
Just
let
go
find
everything
and
configure
it.
For
me.
That's
easy,
but
I
might
not
want
work
to
do
everything.
So
the
next
option
is
well
the
kind
of
the
other
end
of
the
spectrum
be
as
prescriptive
as
you
want
to
be
so
don't
use
all
devices
or
nodes
set
those
defaults.
A
A
So
next
mode
is
filtering.
So
if
you
filter
the
devices,
you
can
use
a
regular
expression
here
with
the
device
filters.
So
any
right
now,
it'll
say
all
nodes,
use
a
device,
filter
and
and
go
go
set.
Those
those
weren't,
very
communities
native,
so
that
occur
Bernays
native
way
to
do
a
similar
thing
is
to
use
node
labels,
so
you
can
set
a
label
on
a
node.
A
So
the
next
option
we've
got
is
sort
of
a
performance
optimisation.
So
some
people
have
like
they've
got
a
high
performance,
they've
got
SSDs
or
nvm
ease,
just
one
or
two
of
those
on
a
node
and
then
a
whole
bunch
of
spinning
discs.
So
the
way
you
can
configure
that
is
you
tell
okay,
my
my
metadata
device.
Whoops.
Let
me
go
back
my
metadata
device.
Is
this
nvme?
Oh
one,
and
then
you
know
pick
up
all
the
other
devices
like
SD
star
and
and
put
the
data
on
those
but
yeah.
A
This
gives
me
a
higher
performance
cluster
with
Ceph.
Put
the
blue
store
right,
head
logon
database
on
one
high-performance
device,
put
the
put
the
data
somewhere
else
on
the
spinning
of
disks,
and
one
more
option
is:
if
you
really
have
high
performance
clusters,
you've
got
all
nvme
devices
which
would
pretty
pretty
nice
I
want.
One
of
those
someday
you
know,
envy
means
are
so
fast
that
they,
you
know
you
can
actually
put
multiple
OSDs
on
the
same
on
the
same
nvme
device.
A
A
A
A
A
It's
done
so
momentarily.
We
should
see
some
new
RBD
mirroring
pods
show
up
on
the
bottom.
Half
it
to
actually
go
marry
or
anything
are
be
marrying
is
about
replicating
your
data
to
another
cluster.
It
does
require
more
configuration
in
the
SEF
at
the
cellar,
so
you
under
the
tool
box-
and
you
tell
Seth-
which,
which
images
you
want
to
want
to
orchestrate
and.
A
Alright
any
moment
I'm
waiting
for
them
to
show
up
at
the
bottom
there,
and
now
we
can
look
at
the
operator
log
to
see
what
it's
doing
and
it's
anyway,
it's
it's
thinking
for
a
little
bit
alright,
but
it
always
stays
in
a
reconciliation
loop.
It
will
come
back
and
and
make
sure
it
happens.
Sometimes
it
just
takes
a
minute
or
two.
A
A
Set
that
fast
same
sort
of
thing,
there's
a
CRT
you
create
it
and,
and
the
operator
makes
it
happen,
I
wish
we
had
more
time
the
agent
I'll
just
keep
over
this.
So
how
to
get
involved.
You
know
go
to
our
github.
Now
we've
got
a
slack
with
an
active
community
love
to
hear
your
questions,
make
sure
your
issues
are
getting
fixed,
always
welcoming
new
contributions.