►
Description
Object Bucket Provisioning in Rook-Ceph - Jonathan Cope & Jeff Vance, Red Hat
While Kubernetes internally supports a generalized API for managing file and block storage, S3 object storage is fundamentally lacking. Rook, a cloud native storage orchestrator, has brought several S3 object storage providers into the Kubernetes ecosystem, including Ceph-Object. What Rook lacks is a generalized Kubernetes S3 API for bucket provisioning. We are designing and implementing such an operator for Rook. This operator provides a generalized S3 bucket provisioning API for Kubernetes users via a set of Custom Resource Definitions. Through these CRDs, Ceph-Object consumers can utilize Kubernetes to provision and manage their Ceph-Object buckets. This presentation focuses on the design goals and use-cases for native Rook bucket provisioning, and some bucket CRD implementation details.
About Jonathan Cope
Senior Developer, Red Hat In.c
Jon has been working at Red Hat for 5 years and lives and works in Austin, Tx. He is a senior developer with a long-time focus on Kubernetes storage and recently on Rook-Ceph. His current project is designing and implementing object bucket provisioning.
About Jeff Vance
senior developer, Red Hat
Jeff has been working at Red Hat for 7 years and lives and works in Santa Cruz, Ca. He is a senior developer focusing on Kubernetes and Openshift storage. He is currently involved with various object stores and incorporating them into both Kubernetes and Rook-Ceph.
A
So
I'm
John
cope,
I
work
for
Red
Hat
in
the
CTO
office,
Jeff
and
I
work
closely
together
on
this
project.
We
authored
the
Brooks
F
bucket
provisioner,
which
is
the
primary
subject
of
this
talk
and
the
bucket
provisioner
API,
which
will
also
address
it's
a
library
that
we
use
to
implement
the
provision
or
it
handles
all
the
control,
the
control
loops
and
everything
that
should
make
provisioning
development
much
easier
for
most
object.
Vendors
and.
B
Doing
the
same
thing,
John
and
I
work
together
at
Red
Hat,
and
so
this
is
what
we're
gonna
cover.
I'm
gonna
do
about
the
first
three
bullets
and
John
will
do
the
last
three,
including
a
demo.
What
I'd
like
to
see
a
show
of
hands
is
how
many
people
here
have
hands-on
experience
using
kubernetes,
because
I
want
to
know
whether
I
have
to
explain
very
much
or
not?
Okay,
so
a
lot
of
people
and
how
about
hands
on
experience
using
rook
and
again,
most
people
do
so.
B
If
you
didn't
have
your
hand
up,
and
you
have
a
question
to
definitely
ask
it
at
the
end
or
grab
us
afterwards,
so
I'm.
That
means
I'm
going
to
go
over
the
next
slide
quicker,
but
I
actually
grabbed.
This
quote
for
kubernetes
out
of
a
Red
Hat
open
chef
description
where
they're
trying
to
differentiate
open
ship
from
kubernetes.
B
But
it's
a
good
description
and
almost
every
word
there
has
meaning
it's
open
source,
it's
popular,
there's,
a
very
vibrant
community,
around
kubernetes
there's
a
dozen
or
so
SIG's,
including
SIG's
storage,
which
we
have
people
in
the
room
that
participate
in
and
the
other
thing
that
matters
is
that
kubernetes
has
a
storage,
abstraction
layer
and
that's
really.
The
basis
of
what
this
presentation
is
is
how
we
key
off
the
well
understood,
storage,
abstraction
layer
of
kubernetes
and
bring
that
into
Rooke
south.
B
And
then
the
rook
stuff,
which
most
people
know
and
I,
think
an
Alexander
just
covered
this
slide.
But
the
thing
the
important
point
here
is
that
although
rooks
F
is
a
great
storage
Orchestrator
with
all
kinds
of
different
storage,
there
isn't
a
way
to
provision
a
CEFR
GW
bucket
in
rook
stuff,
and
so
it's
one
piece,
that's
missing.
You
can
do
all
kinds
of
things
related
objects.
You
can
provision
the
object
store.
You
can
provision
a
user
in
that
object,
store
the
author
of
that's
in
the
room,
but
you
can't
provision
the
bucket
yet
okay.
B
So
the
motivation
for
us
writing
a
bucket,
provisioner
and
choosing
Brooks
F
was
basically
like
I
said
it's
a
way
to
complete
the
storage
picture
with
a
simple
intuitive
control
plane
you,
you
have
a
concern.
You
have
a
consistent
way
of
provisioning
file,
storage,
block
storage
and
with
this
now
object
storage.
So
it
that's
our
motivation,
and
it's
also
was
a
way
for
us
to
test
and
showcase
the
bucket
provisioning
library
that
we
have,
which
we're
going
to
go
into
in
more
detail.
B
So
anybody
can
do
it.
You,
if
you
know
some
things
about
kubernetes
and
you
understand
the
object
store
that
you
want
a
provision
buckets
in.
You
can
write
a
bucket
provisioner
and
everyone
could
write
their
own
and
they'd
all
be
a
little
different
and
the
way
they
would
interact
with
the
with
the
pod,
which
is
the
working
unit
in
kubernetes,
would
be
a
little
different.
We
wanted
to
standardize
it.
B
We
wanted
a
contract
that
application
developer
could
count
on
so
that
they
could
move
their
application
if
they
wanted
to
from
s3
in
the
cloud
to
rook
seth
on-prem
it
would.
It
would
facilitate
portability
better
if
we
were
opinionated
about
what
the
names
of
certain
keys
should
be
for
endpoints
and
credentials,
and
so
we
have
a
library
that
does
that,
for
you
and
a
benefit
of
this
library
is
that
you
only
have
to
have
your
expertise
in
your
particular
object,
store
its
s3
or
its
edge
EFS
Mineo.
B
Well,
they're,
not
I,
guess
they're
now
called
min
IO.
Their
community
voted
for
a
little
bit
different
pronunciation
of
their
name,
but
you
know
the
object
store,
but
you
don't
have
to
know
kubernetes
when
you
use
our
library,
you
just
import.
It's
an
import
statement,
a
couple
of
import
statements.
You
have
a
library
and
you
focus
on
the
knowledge
you
have
about
your
object,
store
and
let
up
and
let
the
library
take
care
of
the
rest
in
that
definition,
to
kubernetes
that
I
skim
by
it
said
it
was
extensible.
B
B
Algol
I
didn't
I've
covered
some
of
this
already,
but
the
library
it's
opinionated,
but
it's
agnostic.
It's
opinionated
on
certain
names
that
will
help
application
portability,
but
it
doesn't
care
what
the
underlying
object
store
is,
so
it
it
can
be
used
for
any
kind
of
object,
store
and
the
last
bullet
is
just
a
lot
of
focus
in
kubernetes
is
dynamic,
provisioning
of
new
storage
Greenfield.
We
call
it,
but
we
want.
We
want
the
experience
to
be
the
same
for
users
of
the
library,
whether
they're
using
new
buckets
or
existing
buckets.
B
So
a
little
bit
more
detail
on
what
we
do.
The
kubernetes
has
the
concept
of
a
persistent
volume
in
a
persistent
volume
claim
have
any
of
you
use
that
in
the
room,
TVs
and
PVCs
about
a
third
of
the
hands
up,
so
Moe,
more
than
half
haven't
used
them
a
PVC
or
a
persistent
volume
claim
is
a
request
for
storage
and
the
idea
in
it's
an
abstraction
in
kubernetes.
The
idea
is
that
I
don't
really
necessarily
care
where
it
comes.
B
The
storage
comes
from
I
just
need
two
hundred
gigabytes
or
something
and
I
want
it
to
be
block
or
file.
The
application
probably
cares
about
that.
We
have
the
same
I
and
then,
when
you,
when
the
request
for
storage,
has
been
satisfied
by
kubernetes,
you
get
a
PV,
a
resource
that
describes
what
your
storage
is
and
we
have
the
same
concept
for
buckets.
B
B
Storage
classes
are
the
key
to
dynamic
provisioning
in
kubernetes
for
a
file
and
block
and
they're
also
the
key
for
provisioning
new
buckets
in
the
library
and
for
rook
Seth
and
where
we
differ
a
little
bit
from
what
traditional
kubernetes
PV
PVC
model
is.
Is
that
you
don't
mount
a
bucket
right?
You
you
get
credentials
and,
and
those
are
in
a
kubernetes
secret.
It's
a
native
resources
and
kubernetes
and
you
get
an
endpoint
information
and
those
are
in
a
native
config
map,
again
part
of
core
kubernetes,
so
your
pod
isn't
mounting
a
bucket.
B
It's
consuming
a
configuration
map,
a
config
map
which
has
endpoint
data
and
it's
consuming
a
secret
which
has
credentials
unique
for
your
access
to
that
bucket.
You
under
the
covers,
when
you've
written
a
provisioner
all
of
the
complexity
of
retry
logic,
consentual,
consistency,
error,
recovery
watching
for
events
that
happen
is
all
taken
care
of
by
the
library.
B
So
again,
as
the
provision
or
author
you're
not
worrying
about
that
stuff-
and
this
is
the
connection
point
between
the
library-
and
in
this
case
the
seth,
rgw
provisioner,
there's
four
functions
or
methods
or
api's
that
we
define
provision
is
creating.
A
new
bucket
delete
is
removing
that
bucket.
The
provisioner
itself
will
have
to
do
what
it
needs
to
do
for
its
own
object
store
when
it
creates
a
bucket.
It
will
have
to
create
the
physical
bucket
and
policy
users
ackles
whatever
it
is
around
that
bucket
delete,
says
undo.
B
All
those
things
the
library
will
take
will
remove
kubernetes
resources.
The
provisioner
removes
resources
specific
to
the
object
store
and
at
the
bottom
we
have
four
existing
buckets,
a
way
to
grant
access
to
a
bucket
through
the
law
method
and
take
that
access
away
and
now
John
will
give
you
the
next
layer
of
detail
thanks
Jeff.
A
So
current
implementations
of
provisioners,
using
this
bucket
library,
that
Jeff
was
describing,
we
ate,
of
course
Rick's
FRG
W,
which
we'll
be
demoing.
At
the
end
of
this
talk,
we
have
a
AWS
s3
iteration,
which
is
also
a
proof
of
concept
to
test
and
show
that
the
library
works
just
as
well
for
that
NUMA
is
in
progress,
has
one
in
progress
that
are
developing
it
themselves
and
then
in
the
future,
we'd
like
to
also
develop
one
for
Azure
and
for
Google
Cloud
storage
to
hit
all
the
the
major
cloud
providers.
A
So
a
little
more
information
about
how
the
provisioner
runs.
The
provisioner
is
a
kubernetes
controller
living
inside
the
library.
When
you
define
your
provisioner,
are
your
provisioning
method.
Rather,
the
library
will
call
that
provision
method
went
to
text
a
new
OBC
when
the
OPC
is
created.
The
library
controller
detects
that
creates
a
bucket
then
like
Jeff,
was
saying,
creates
a
secret
and
config
map
in
the
name
space
of
the
claim,
then
that
is
consumable
by
the
application
pod.
A
So
the
structure
of
the
OPC
is
pretty
simple
and
we
try
to
follow
kubernetes
conventions.
You
have
generate
bucket
name,
OOP
apologies
generate
bucket
name
behaves
like
generate
name
in
kubernetes.
You
get
a
randomized
suffix
at
the
end
of
the
name.
Bucket
name
allows
you
to
pick
a
static
bucket
name.
You
do
run
the
risk
of
bucket
name
collisions
in
the
object
store,
but
you
just
have
to
be
aware
of
that
and,
of
course,
the
storage
class
name.
The
storage
class
on
the
right
hand,
side
here,
key
things
to
note,
of
course,
the
provisioner.
A
This
is
a
value
passed
to
the
library
to
tell
it
that
it
is,
in
this
case
the
set
provision
rather
than
AWS
provisioner,
and
that's
how
it
differentiates
between
object.
Bucket
claims
parameters
are
specific
to
the
provision
or
so
in
the
case
of
SAP,
we
have
a
region
object,
store
named,
object,
store
namespace.
A
Once
you
create
an
OBC,
you
should
see
these
three
objects,
then
the
OB,
which
is
non
namespace
and
then
the
secret
in
the
config
map
in
the
same
namespace
as
the
OBC,
when
you're
ready
to
attach
this
to
a
pod
there's
a
few
methods.
You
can
do
it
if
you
are.
If
your
app
is
fine
with
these
environment
variable
names,
you
can
do
a
straight
environment
from
to
find
the
config
map
now
inject
those
keys
in
your
environment
as
is
or
if
your
app
expects
different
environmental
variable
names.
A
For
the
the
endpoints
and
the
authentication,
and
then
it
would
be
connecting
to
that
set
bucket
to
delete
buckets
it's
the
inverse
of
creating.
So
you
delete
the
OBC
the
controller
in
the
library
detects
that
deletes
the
bucket
cleans
up
the
API
objects
that
created
the
secret
and
the
config
map
in
the
object
bucket.
It
also
deletes
the
user
in
the
case
of
a
delete
policy.
Calls
delete,
like
I,
said
Leitz
the
bucket
in
the
case
of
routine.
It
just
revokes
the
credentials
and
deletes
the
user.
A
So
the
current
status
of
the
rooks
F
and
the
the
library
is
they're
both
event
alpha
state
AWS
s3
is
more
listing
where
it
is
for
the
moment
it
was
more
of
a
proof
of
concept.
We
have
a
number
of
PRS
and
issues
and
we're
open
to
any
kind
of
help
that
anyone
wants
to
provide
new,
is
ongoing
and
then
again
down
the
road
during
Google
storage
would
be
the
next
targets.
A
We
have
it
there's
a
few
restrictions
that
come
with
the
library,
it's
very
new,
and
so
it
doesn't
have
things
like
I've
been
recording
or
the
ability
to
cancel
buckets
in
progress.
But
those
are
things
we
intend
to
add.
One
key
thing:
we'd
like
to
expand
on
is
adding
a
a
library
defined
parameter
key
in
the
storage
class
for
reclaimed
policy
override.
That
would
give
us
the
ability
to
add
an
erase
function
being
that
we
clears
out
all
the
objects
but
saves
the
the
name
of
the
bucket.
A
We
know
that
that's
an
important
thing,
you
don't
have
to
lead
a
bucket
and
have
someone
else
grab
that
name
and
yeah.
A
few
other
features
down
the
road
that
we
addressing
issues
you
can
contribute,
will
have
the
links
here
and
will
will
distribute
the
the
slides
I'll
leave
that
up.
If
anyone
would
like
to
take
a
picture
of
it,
okay,
I'll
wait
for
the
cameras
to
go
down.
A
Okay,
alright
moving
on
and
then
links
you
can
find
you
know
kubernetes
and
rook
storage
of
these
links
and
our
we
have
a
set
of
examples
on
running
ponds.
We
have
a
simple
photo
application
pod
that
you
can
deploy
from
this
examples
and
blogs
repository
that
will
connect
to
the
bucket
and
allow
you
to
through
a
web
interface,
send
and
receive
photos.
A
A
B
A
So
is
that
a
little
bit
better
good,
excellent?
Okay,
so
I've
divided
my
terminal
on
here
using
T
MUX
on
the
top
panel.
This
is
a
live
log
from
the
rooks
F
operator.
I've
got
rooks
F
deployed
already.
Let
me
just
so.
You
can
see
Brooke
oops
I
got
apologies,
yeah
I'm
running
in
mini
cube
right
now,
locally
on
my
laptop
I
have
rooks
F
deployed
in
the
rooks
F
namespace
on
the
lower
left,
pane
Here
I
am
getting
a
I,
have
a
watch
on
config
maps
and
secrets
and
on
the
right
hand,
side.
A
A
A
A
A
B
A
B
A
A
B
A
A
A
So
you'll
see
here,
we've
got
the
bucket
host
and
bucket
name.
The
host
in
this
case
would
be
the
service
for
the
are
the
URL
rather
for
the
store
service
and
then
the
name
of
the
bucket
and
the
port,
the
region
and,
in
this
case,
we're
not
using
SSL
and
then
to
demo
our
to
share
the
values
in
the
secret.
A
A
A
So
you
see,
we
deleted
the
OBC,
it's
cleaned
up
the
bucket
the
config
nap
in
the
secret
before
it
did
that
it
attempted
to
really
delete
the
bucket
in
the
store.
So
if
it
fails
to
leave
in
the
bucket
it's
gonna
leave
the
API
objects
there
to
kind
of
indicate
that
something's
wrong
or
that
there's
been
a
hang-up,
and
that
way
we
don't
end
up
with
an
orphan
bucket
and
just
to
show
that
the
bucket
has
been
deleted.
We
no
longer
have
the
bucket
in
the
object
store.
C
Hi
so
I
work
with
the
monitoring
pieces
of
Cuban
and
s.
So
since
you
have
created
new
resources
on
Cuban
artists
and
you're,
deploying
through
rukh,
how
do
I
know
what
resource
is
consuming
so
just
for
an
example,
cuban
artists
gives
cubelet
metrics
for
PBS
and
pvcs
cuban
artists
gives
a
cumulative
or
PVCs
and
Peavey's,
and
so
have
you
integrated
some
metrics
for
your
buckets
as
well
there
or
do
you
have
any
plans
regarding
that
that.
A
D
D
D
A
D
A
Yeah,
no,
we
don't
set
the
limits
on
the
bucket.
That
is
something
that
you
can
do
and
rook
stuff
to
set
a
maximum
amount
of
size
or
objects.
It's
something
we
can
add
down
the
road.
It
would
be
a
self
specific
edition,
so
wouldn't
go
into
the
library.
Just
it'd
be
a
friend
she
at
that,
but
yeah.
That's
definitely
something
we
could
do.
E
A
Sense
yeah
now
I
understand
the
question
DeYoung
perspective
on
that
was
to
follow
a
kubernetes
pattern
of
allowing
it
to
operate
on
OBCs
and
Obie's
and
secrets
anything
that's
across
all
namespaces.
We
felt
that
that
was
a
common
pattern
in
kubernetes.
We
try
to
limit
exposure
to
the
object
store
by
giving
one
key
per
object,
claim
limiting
the
rights
of
that
key
so
that
they
cannot
create
or
delete
buckets.
They're
only
allowed
to
put
and
potentially
delete
objects
within
the
one
bucket
that
they
were
given
for
that
key.
A
So
we're
trying
to
address
at
the
s3
layer
rather
than
kubernetes
later,
oh
I'm,
starting
to
add
to
that
the
administrators
can
use
or
a
cluster
role
definitions
to
differentiate,
which
namespaces
should
even
be
allowed
to
create,
Obie's
or
and
access
secrets
and
names
based
on
things.
So
we
allow
at
the
kubernetes
layer
there's
a
bit
of
security
there,
any
any
more
questions.