►
Description
Getting Started as a Rook-Ceph Developer - Blaine Gardner, SUSE
Kubernetes. We've all heard the name. It's the advanced container orchestration platform that lets you run your container applications at any scale. But can Kubernetes help make Ceph orchestration better? To answer that, let's have a look at Rook. Rook is the upstream project the Ceph community has been working with to make Ceph on Kubernetes a reality. We'll explore the benefits Rook gives Ceph, the challenges we have faced, the challenges we have yet to face, and our current vision for the future of Rook.
About Blaine Gardner
Ceph Containerized Storage Engineer, SUSE
I am a maintainer for the Rook upstream project's Ceph backend, and I work for SUSE as lead software engineer on the Enterprise Storage team's efforts to containerize Ceph and run it on Kubernetes. My go-to random fact is that beavers are nocturnal.
A
Right,
my
name
is
Blaine
Gardner
I
work
for
Sousa,
with
our
enterprise
storage
division,
I'm,
specifically
working
on
staff
containerization
and
that
primarily
means
rook
I'm,
one
of
the
maintainer
x'
for
rooks
F,
back-end
and
I
I
want
to
talk
about
what
it's
like
to
get
started
as
a
developer
on
rooks
stuff
backend,
because
I
know
it
can
be
daunting
and
I.
Don't
want
that
to
keep
anyone
out
from
you
know,
committing
and
contributing
where
they
would
like
to.
A
A
There
are
also
pods
which
are
our
our
unit
of
work
in
kubernetes,
and
they
consist
of
one
or
more
containers,
running
sequentially
or
side
by
side.
We
have
config
maps
and
secrets
which
pretty
simply
store
small
bits
of
data
persistently,
for
configuration
secrets
are
good,
for
you
know
things
that
you
don't
want
exposed
to
the
outside
world,
like
your
authentication
keys
and
things
like
that.
A
A
The
core
root
concepts
are
going
to
be
the
concept
of
an
operator
itself,
which
was
a
term
coined
by
core
OS
a
few
years
ago,
and
it's
essentially
just
an
application
that
runs
in
kubernetes
and
helps
the
administrator
to
operate
a
thing
which,
in
our
case,
is
going
to
be
set,
and
the
operator
effectively
just
augments
kubernetes
capabilities
to
better
to
better
understand
how
to
better
help.
Kubernetes
control
that
applications,
unique
domain
and
stuff
is
definitely
a
unique
domain.
A
There's
also
a
distinction
that
I
would
like
to
make
between
rook
and
between
rooks
and
what
I've
called
rooks
F
here.
So
rook
is
not
it
in
and
of
itself.
An
operator
rook
is
an
operator
framework
used
for
creating
for
creating
individual
operators,
and
Saif
is
one
of
these
operators.
There
are
other
other
entry
operators
as
well
in
the
upstream
project.
A
I'm
gonna
do
my
best
to
always
call
the
operator
the
operator
or
rook
SEF,
but
when
I
inevitably
slip
up
and
say,
rook
I'm
almost
always
mean
the
the
operator
itself
force
F
in
our
case,
there's
also
the
SEF
orchestrators
manager,
module,
and
that
is
in
short,
gives
us
an
interface,
an
API
and
a
CLI
for
communicating
with
an
Orchestrator,
whether
that's
ansible
or
deep-sea
or
rook,
and
because
we're
focusing
on
rook
there's
also
the
rook
Orchestrator
plugin,
which
will
translate
those
API
commands
or
those
CLI
commands
into
a
domain.
The
orchestrator
understands.
A
We
have
a
deployment
of
the
rooks
F
operator
and
that's
gonna
be
shown
in
blue,
and
at
this
point
we
also
create
our
custom
resource
definition
for
Seth
and
with
that
custom
resource
definition,
we
will
then
create
a
manifest
for
that
which
rook
will
pick
up
from
kubernetes
and
it
will
start
creating
our
SEF
cluster.
Our
starting
point
is
always
just
a
seed
cluster
of
a
monitor
and
a
manager,
and
once
that
is
created,
we're
also
going
to
create
a
a
toolbox
pod,
and
this
toolbox.
A
Pod
really
just
gives
us
easy
access
to
the
Ceph
CLI.
This
is
more
or
less
the
official
place
to
use
subsea,
SEF,
CLI
tools
and
we're
gonna
do
that
by
now
using
this
Orchestrator
module,
where
we
can
list
and
add
hosts,
we
can
add
monitors.
We
can
list
disks
in
the
cluster
and
add
those
disks,
and
we
can
also
add
other
Damons
as
we
see
fit.
A
There's
way
more
information
here
than
I
can
go
through
with
you
can
watch
Sebastian's
presentation
from
yesterday
or
there's
good
documentation
also
on
SEFs
website,
but
the
first
part
is
adding
more
monitors
because
we
want
a
better
cluster
than
just.
You
know,
a
single
monitor
and
so
rook
will
go
through
and
it
will
create
these
monitors.
It
will
place
them
on
separate
nodes
so
to
help
us
with
our
failure
domain,
and
these
monitors
are
run
as
deployments
from
there.
A
A
And
this
this
constitutes
our
core
self
cluster
here
and
I
think
this
is
a
good
time
to
mention
that
I
mentioned
the
off
keys
for
all
of
these
are
stored
as
kubernetes
secrets,
and
that
way
you
know,
malicious
users
wouldn't
just
have
arbitrary
access
to
those
and
from
here
it's
kind
of
up
more
up
to
the
user
to
decide
what
they
want
to
do
generally.
Creating
Damons
is
the
next
step.
These
may
be
one
or
it
may
be
many.
A
And
this
is
also
once
we
have
our
our
gateways
set
up,
whether
that's
in
FS
or
or
whatever
our
users
user
applications
can
then
make
requests
for
that
data
to
kubernetes,
which
then
communicates
with
rook
to
understand
how
those
applications
can
use
the
underlying
safe
storage.
So
rook
doesn't
sit
in
the
data
path
at
all.
A
A
So
the
big
long
path
here
at
the
top
has
some
of
our
example
manifests
for
creating
the
rook
operator
for
creating
SEF
clusters.
File,
block
objects,
storage-
these
are
pretty
straightforward.
The
command
directory
is
where
we
actually
have
our
entry
points
or
what
I've?
What
I
call
entry
points,
at
least
for
the
rook
operator
and
for
any
related
components,
primarily
the
related
component,
is
that
OSD,
preparation,
pod
and
package
contains
the
vast
majority
of
rooks
actual
code.
We
have.
The
API
is
more
or
less
just
definitions
for
the
structures
underlying
our
set
custom
resource.
A
We
also
have
daemon
Seth
and
code
under
the
daemon
umbrella
is
generally
dealing
with
those
related
components,
like
I
said,
primarily.
The
OSD
preparation
job
and
under
the
operator
umbrella
is
where
most
of
our
core
logic
lives,
and
so
these
are
things
that
are
are
only
run
within
the
context
of
the
operator.
These
aren't
there's
no
code
here
for
related
components
and
our
operator
whenever
it
wants
to.
A
Whenever
it
wants
to
run
SEF,
it
will
configure
a
deployment,
for
example,
to
run
a
pod
which
we'll
call
SEF
Mon,
make
FS
and
then
we'll
proceed
to
run
this
F
Mon
binary
afterwards
or
for
the
OSD
preparation
it
will,
instead
of
it,
will
call
one
of
the
end
points
defined
in
command,
and
so
that's
how
a
rook
can
call
code.
That
is
when,
within
itself
for
running
other
things
and
tests
is
the
high
level
directory
where
all
of
the
integration
tests
are
located.
A
Before
moving
on
too
far,
I
wanted
to
talk
about
community
involvement.
First
and
foremost,
the
Ceph
community
calendar
has
pretty
much
everything
as
far
as
upstream
coordination.
You
would
want
to
listen
in
on
or
you
know,
get
started
with.
Actually
talking
about
information,
the
key
items
you're
looking
for
here
are
bolded
the
community
meeting
the
orchestrator
weekly
meeting,
and
then
we
have
a
daily
ish
huddle
meeting.
A
A
But
now
actually
talking
about
once
we
understand
what
rook
is,
how
what's
the
actual
flow
for
development
and
we've
got
a
we're.
Gonna
start
with
our
we'll
set
up
a
virtual
cluster
or
maybe
a
real
cluster,
if
you're
feeling
particularly
bold
and
then
we're
going
to
install
kubernetes
on
it,
most
I
mean
a
lot
of
that
could
be
left
up
to
the
user
if
they
so
desired.
A
Once
those
first
two
steps
are
done,
you
generally
don't
have
to
revisit
those,
and
then
there
are
separate
workflows
for
rook
for
stuff
and
then
there
I
would
also
call
the
upgrade
workflow,
which
is
kind
of
a
combination
of
both
of
those
separate.
That's
also
something
that's
sufficiently
complicated
and
not
something
I
think
everyone
needs
to
do
right
off
the
bat
that
we're
gonna
skip
that
in
this
talk,
but
starting
with
the
starting
with
the
rook
workflow.
A
We
it
starts
with
building
whatever
changes
you've
made
to
rook
and
we'll
specify
that
we
only
want
to
build
the
Ceph
component.
So
that'll
save
us
a
little
bit
of
build
time
and
then
we're
going
to
push
that
container
to
all
of
our
nodes,
which
is
going
to
be
I
mean
if
you're,
using
the
docker
tools
at
least
you're,
going
to
docker
save
into
a
tarball
you'll
SCP
that
to
your
hosts.
And
then
you
will
docker
load
that
tar
blow
tar
ball
from
each
of
those
hosts.
A
Then
you
will
install
rook
with
whatever
manifests
you
you
need
to,
and
you
can
I
do
a
lot
of
manual
testing
myself.
You
can
also
run
integration
tests
individually
generally,
because
those
will
take
about
five
minutes
to
run
using
the
there's
a
binary,
that's
created
when
you
build
rook,
that's
called
integration,
and
then
you
can
pick
a
test
with
a
regular
expression
or
if
you
want
to
run
all
of
them,
that'll
take
30
40,
maybe
45
minutes.
A
A
A
For
for
practical
development,
I
spent
some
time
the
past
few
weeks
up
streaming
the
development
environment
that
I've
been
using
for
the
past
year.
I
guess
and
you
can
find
that
at
this
repository
here,
I've
called
it
dev,
rooks
F,
and
this
is
primarily
based
around
make
files
for
the
user
operation
and
libvirt
for
the
virtualization
of
the
cluster.
Behind
it.
A
There's
a
make
QuickStart
target,
which
at
least
on
my
laptop
gets
a
three
node
cluster,
going
with
kubernetes
ready
for
Rick
in
about
six
minutes,
and
this
also
has
security
policies
enabled
within
kubernetes,
primarily
because
it's
important
to
me
to
run
an
environment
that
is
closer
to
what
you
know.
Any
of
the
users
in
production
are
going
to
be
using.
A
A
I
do
want
to
introduce
sorry
wrong,
talking
point
just
a
brief
overview
here
of
the
targets
in
in
that
repository.
I
have
something
for
setting
configuration,
especially
because
of
those
security
policies.
It's
I
like
to
just
freeze
a
configuration
in
place
that
I'll
use
for
all
of
my
route
code
changes,
there's
a
target
for
building
and
pushing
automatically
or
building
and
then
pushing
automatically
also
installation
reinstallation,
as
well
as
building
and
replacing
the
operator
in
place
or
building
and
redoing
the
entire
cluster
or
iterating
on
configuration.
A
Changes
themselves
is
another
sort
of
sub
workflow
and
some
targets
just
to
aid
with
development
like
watching
logs,
and
things
like
that.
I
haven't
yet
needed
to
actually
iterate
on
SEF
code
changes,
so
none
of
these
targets
exist.
If
you're
excited
about
writing
these,
please
do
and
I
do
want
to
introduce
Sebastian
vogner,
who
actually
has
been
working
with
SEF
code
changes
and
I'll.
Let
him
introduce
himself.
B
B
My
development
environment
is
a
bit
ad
hoc
I'm
using
a
project
called
test,
Luke
Orchestrator,
which
its
main
purpose
is
to
to
test
the
route
Orchestrator
within
a
virtual
cluster,
and
by
coincidence
it's
also
working
great
as
a
development
environment.
It
contains
everything
that's
needed
to
start
up
a
Cuban,
a
test
cluster
with
openSUSE
cubic,
deploy,
Orchestrator,
deploy
a
safe
cluster
and
with
cube
checker,
it's
possible
to
on-the-fly,
upgrade
or
update
the
Python
files
or
binaries
within
the
safe,
safe
containers
by
using
a
local,
unsafe
registry.
B
C
C
It's
about
been
about
three
years
almost
since
we
first
created
the
repo
and
had
the
first
commit
so
I've
had
a
little
experience
with
developing
on
on
Brooke
and
there
are
all
sorts
of
ways
to
poke
it
to
build
it
to
run
it.
So
we
just
have
a
few
minutes
today
to
talk
about
some
of
those
and
another
time
we'll
have
to
talk
more
about
how
to
poke
it.
C
Even
more
so
we've
got
about
ten
slides
worth
on
this
one
slide
so
where
to
start
so,
first
of
all,
it's
really
important
to
have
an
environment
where,
where
we
can
quickly
develop
quickly,
iterate
because
building
these
containers
starting
to
burn
Eddie's
and
all
these
things,
it
takes
a
lot
of
time
if
you
try
and
bring
up
a
new
cluster
every
time.
So
the
first
point
I
want
to
make
is
that
you
know
95%
of
the
time
I'm
using
mini
cube
as
the
simplest
environment
to
get
kubernetes
running.
It's
a
single
node.
C
It's
really
fast
to
start.
You
know
the
first
time,
I
start
mini
cube,
it's
about
five
minutes
and
from
there
I
can
restart
it
really
quickly
too
so
I
bring
out
mini,
cube,
I'm
I'm,
using
a
Mac.
So
on
my
Mac
start
mini
cube,
I
use
a
virtual
box
for
my
VM
provider.
You
can
use
you
know
any
provider.
You
want
one
of
the
first
things.
I
do
then,
is
I,
create
or
attach
a
couple
of
devices
to
it
since
Seth
with
OS
DS
needs
devices
to
start
OS
DS.
C
Okay.
So
that's
the
first
point
find
a
simple
environment
that
works
for
you
most.
The
time
and
yeah
simple
is
good
the
so
when,
when
we're
testing
in
mini
cube,
now
testing
the
changes
most
the
time,
I'm
testing
changes
in
the
operator,
and
you
know
we
need
to
test
well.
How
is
it
starting
the
Mons?
How
is
it
starting
the
OSDs
and
when
it
does
that
we
want
to
start
it
in
this?
The
simplest
way?
That's
gonna!
C
So
these
manifest
files
in
in
the
rook
repo
have
the
test
suffix,
so
they're
called
cluster
test,
dot,
Yambol
or
object
store
test
llamo,
and
when
we
create
those
it
creates
the
minimal
optimal
environment
for
a
single
node
cluster.
There's
no
replication
of
the
data
there's
only
one
OSD,
one
Mon:
it
just
gets
the
cluster
up
and
running
okay.
C
So
so
that's
the
next
way
to
iterate
quickly
have
a
simple
deployment,
and
so
when
we
do
need
to
test
with
the
multi
node
environment
and
things
there,
we
do
have
a
couple
of
documented
ways
to
start
up
with
like
cube
ATM
clusters,
but
it
does
take
longer
and
on
my
Mac
honestly
when
I
start
up
three
VMs
it
just
16
gigs
isn't
enough
memory,
so
it
doesn't
work
well.
So
I
found
other
environments
that
that
work
for
demos,
where
we've
got
openshift,
bringing
up
multiple
nodes
and
that
just
works
with
me.
C
Fortunately,
somebody
else
has
solved
that
problem
of
bringing
up
a
cluster
where
I
can
run
that's
not
on
my
local
laptop,
because
the
local
laptop
really
won
one
VM
is
all
it
handles
well
and
then
last
point.
So
what
I'm
using
my
Mac
is
vs
code.
Vs
code
has
a
great
going
extension
since
most
of
Rooke
isn't
going
and
I
can
use
it
to
debug
the
unit
tests
and
do
basic
basic
development.
So
you
know
I
think
some
people
might
be
using
whatever
your
favorite
IDE
is.
C
It's
I
mean
just
code
you
can
use
whatever
you
want,
but
being
able
to
debug
the
unit
tests
is
is
a
key
thing,
just
as
a
last
point,
I
think
one
of
the
most
critical
parts
of
this
whole
development
process
is
saying.
Well,
how
do
we
know
it's
good
code?
How
do
we
know
it's
good
production
quality,
so
we
can
ship
something
that
we
know
works
so
there's
three
levels
of
testing,
just
to
mention
so
first
we've
got
to
test
it
manually.
C
I've
got
to
see
it
working
in
my
cluster,
so
second,
we've
got
unit
tests,
so
those
are
pretty
simple,
really
quick
to
iterate.
You
can
debug
a
test
or
run
a
test
in
seconds
at
least
and
see
it
in
the
your
vs
code,
editor
and
then
third,
we
have
these
integration
tests
that
you
can
run
on
a
real
kubernetes
cluster
and
see
you
know
it
all.
The
components
working
end
end
as
opposed
to
the
unit
tests,
which
are
only
really
mocking
everything.
C
There's
no
actual
kubernetes
running,
there's
yeah,
so
those
3
environments
are
really
critical
to
running
Brook.
Now
this
is
a
really
fast
session,
so
feel
free
to
ask
questions.
Yeah
we're
on
the
slack
next
slide
again:
yeah
join
the
slack
I'll
come
to
github.
Ask
questions,
love
to
answer
him
and
now
we'd
like
to
open
it
up
for
questions.
C
So
the
question
is:
if
we're
only
consuming
discs
for
OS
DS,
okay,
so
there
there
are
actually
two
modes
in
Rooke
today
we're
in
our
in
my
development
animals.
Actually
we
use
the
simple
mode
which
will
create
file
store,
OSDs,
just
in
a
directory,
so
that
doesn't
use
devices.
But
this
is
just
really
for
a
test
environment.
A
I
see:
okay,
okay,
so
there
there
seems
to
be
a
concern
that
the
OSDs
are
shown
having
discs,
but
the
the
metadata
servers.
The
the
monitors
are
are
shown
as
ephemeral,
and
so
the
monitors
do
actually
store
data
on
on
the
hosts,
and
so
the
monitors
are
like
planted
to
a
host
specifically
and
then
it
will
use
that
hosts
directory
if
a
monitor
or
host
were
to
fail.
It's
just
sort
of
like
in
real
life,
where
a
new
monitor
on
a
different
host
would
have
to
build
up
its
corpus,
its
Mon
map
and
everything
again.
A
I
mean
this
decision.
I
mean,
in
my
mind,
is
primarily
for
all
all
of
you
like,
especially
if
you're
looking
to
get
started
on
any
of
this
stuff.
So
please,
please
do
ask
questions
but
like
maybe
as
a
show
of
hands
like
who,
like
for
people
who
are
interested
in
doing
development.
Are
you
mostly
looking
for
like
development
on
on
rook,
like
by
a
show
of
hands,
or
like
mostly
development,
on
Ceph
sort
of
both.