►
Description
Building a Storage Cluster with Kubernetes [I] - Bassam Tabbara, Quantum Corp.
Modern software storage systems are inherently complex. They are composed of numerous distributed components, require careful balancing of resources, and have stringent performance requirements. If you're running your applications in a public cloud you're typically shielded from this complexity and can utilize managed storage services like EBS, S3 and EFS. If you're running on-premise, however, your choices are quite limited and typically result in using traditional big-iron storage systems.
In this talk we'll walkthrough how we've built a production-ready storage cluster using Kubernetes. Storage nodes run as pods and enumerate the available storage devices within the cluster. We'll explore how to optimize the network through CNI plugins to separate client and storage cluster traffic. We'll show how some of the features of Kubernetes including controllers/operators, third-party resources, resource management, and rolling upgrades can lead to more powerful and resilient storage clusters. We'll also walk through use cases where the storage cluster is dedicated (hyperscaled) or shared with other applications (hyperconverged).
About Bassam Tabbara
Bassam Tabbara is the CTO of Quantum Corporation, a world-class leader in storage. He is spearheading several storage projects including Rook (http://rook.io). Prior to Quantum, Bassam was the CTO and co-founder of Symform, a P2P storage startup acquired by Quantum. Prior to that he was at Microsoft where he lead a number of key initiatives as part of Microsoft Research, Azure, Windows Server, and Visual Studio.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
B
B
Typically,
when
you
you
know
when
kubernetes
first
started,
there
was
no
persistent
storage,
there's
nothing
outside
of
a
cluster.
So
this
is
a
camera
which
version
that,
but
essentially,
if
you
wanted
to
store
data
that
survived
a
pod
moving
between
nodes,
you're,
you're
kind
of
screwed,
all
storage
was
essentially
on
the
actual
node,
and
that
was
all
that
was
available.
B
You
heard
the
big
announcement
today
around
11.6
with
dynamic
provisioning,
where
we've
done
is
that
we've
with
kubernetes
can
now
manage
volumes
that
essentially
are
provided
by
external
storage
clusters.
So
if
you
think
about
what
that
means,
just
like
just
like
managing
the
network
or
managing
pods
managing
replication
and
all
those
things,
kubernetes
can
now
manage
the
creation
of
volumes
and
the
mapping
or
attaching
of
volumes
to
pods
right,
but
kubernetes
does
not
manage
the
other
side
of
the
the
actual
storage
cluster,
providing
these
volumes
right.
B
So
if
you're
in
a
public
cloud
environment
like
AWS
or
Google
cloud,
that's
typically
not
a
problem
because
you
have
persistent
disks
or
we
have
EBS
or
you
have.
You
know
the
your
cloud
providers,
storage,
that
with
dynamic
provisioning
kubernetes,
can
essentially
Mountain
EBS
volume,
attach
it
to
your
instance.
It
could
make
it
available
part
as
part
of
the
pod
yeah
and
the
same
with
Google
system
disk.
If
you're
not
running
in
a
public
cloud
environment,
you're
kind
of
out
of
luck,
you
essentially
have
to
run
a
storage
cluster
that
can
provide
volumes
to
Granny's.
B
Somehow
and
the
way
you
do.
That
is
it's
an
external
thing:
it's
not
part
of
the
kubernetes
environment.
So
if
you're
running
a
software-defined
stack
such
as
staff
or
Gluster,
you
essentially
have
to
run
a
full
deployment
of
self
and
Gluster
and/or
Gluster,
and
you
do
that
in
a
traditional
way
as
you
do
it
as
part,
you
typically
have
storage
admins
that
are
managing
it.
It
is
essentially
a
completely
external
to
kubernetes,
or
you
use
traditional
hardware,
whether
it's
NetApp,
filers
or
emc
boxes
or
whatever
your
enterprise
has
for
storage.
B
You
can
certainly
use
those,
but
it's
the
same
thing
there
completely
outside
this
environment,
required
with
a
require
some
admin
or
ticket
system
or
something
to
provision
storage
in
this
world,
and
it's
just
essentially
external
making
sense
so
far,
I
would
love
and
interactive
sessions.
You
feel
free
to
ask
questions
I'm
happy
to
happy
to
make
this
as
interactive
as
you
can.
B
So
if
we
did
that,
then
to
create
the
storage
cluster
I,
just
cube,
cuddle,
I
start
up
a
a
new
storage
cluster
like
I,
would
say
a
load,
balancer
or
like
I
would
say
Prometheus,
and
what?
If
that
storage
cluster,
could
actually
provide
volumes
through
dynamic
provisioning
to
other
pods
running
in
the
same
cluster?.
B
We
think
that
that's
what
we
have
to
do
to
solve
the
storage
problem
in
kubernetes.
We
think
that
that's
not
only
going
to
be
a
the
default
way
of
running
storage
in
kubernetes,
but
it's
going
to
unlock
scenarios
around
portability
across
clouds
and
the
ability
to
actually
reduce
the
management
and
the
reduce
the
surface
area
of
what
is
running
in
Kupa
days.
B
Now.
That
doesn't
mean
that
you
kind
of
have
to
run
in
a
converged
mode
like
that.
You
can
actually,
if
you
wanted
to
create
another
kubernetes
cluster,
just
for
storage
and
scale,
your
storage
independently
from
your
compute
and
your
applications.
But
there
is
no
reason
if
your
organization
is
settling
on
using
kubernetes.
As
the
you
know,
the
single
way
of
managing
and
running
applications
there's
no
reason
why
you
can't
go
in
your
storage
cluster,
even
a
dedicated
one,
in
the
same
way
that
you
run
your
applications.
B
B
B
So
what
is
rook
rook
is
essentially
battle-tested.
Software-Defined
storage
and
I,
say
battle-tested,
because
we
base
truck
on
the
version
of
stuff
embedded
stuff
that
we
make
part
of
their
raw
binary.
Stuff
is
awesome.
Ef
has
ten
years
of
production
deployments
on
top
of
it
and
runs
the
world
some
of
the
world's
largest
storage
clusters.
B
B
B
C
B
Right,
let's
see,
okay,
so
the
first
thing
we're
going
to
do
is
that
we're
going
to
teach
kubernetes
how
to
manage
storage
clusters
by
extending
kubernetes
and
the
way
that
you
can
extend
kubernetes
today
is
by
essentially
creating
an
operator
or
canoe
controller,
so
part
of
what
what
Rooke
does
is.
It
has
a
controller.
That's
designed
to
manage
storage
in
the
cluster
we've
designed
this
again
to
be
so
deeply
integrating
the
kubernetes
so
we're
using
third-party
resources
to
describe
every
aspect
of
the
storage
cluster.
B
That's
pretty
much
it.
The
rook
operator
is
now
running,
but
first
thing
that
there
happens
at
the
rook
operator
creates
is
sent
a
set
of
third-party
resources,
which
is
again
a
way
to
extend
the
kubernetes
api.
The
ones
we
create
are
for
storage
cluster
for
pools
and
a
number
of
others
that
that
essentially,
why.
C
B
C
B
B
And
so
that,
once
the
operator
is
running
that
you
only
have
to
do
this
once
in
a
cluster
once
the
operator
is
running
from
this
point
onwards
can
go
create
new
clusters,
you
can
create
pools,
you
can
do
all
sorts
of
things
with
clusters
and
and
we'll
do
that.
This
is
a
what
again
a
one-time
operation.
B
Now,
let's
actually
look
at
the
ammo
for
a
cluster.
So
again,
this
is
a
fairly
simple
description.
You'll
see.
The
kind
here
is
a
new
kind.
It's
a
real
cluster.
It's
not
so
real
cluster
is
a
third
party
resource
by
default.
Kubernetes
does
not
understand
rook
clusters,
but
we
just
define
them,
and
this
will.
B
All
right
we'll
do
this
idea
and
when
the
demo
fails
all
right
so
now
that
they
have
the
operator
running
and
all
the
third-party
resources
have
been
declared,
we're
going
to
go,
create
a
cluster,
and
we
do
that
by
creating
a
yellow
file
of
kind
or
cluster.
So
what
this
says
is
the
declarative,
declarative
statement.
B
It
says:
I
would
like
a
new
cluster,
a
new
cluster,
to
be
deployed
on
this
kubernetes
cluster,
and
you
can
also
set
you
can
set
the
namespace
that
you'd
like
the
cluster
we
deploy
it
on
and
a
bunch
of
metadata
parameters
that
are
essentially
define
the
cluster
right.
Now,
it's
very
simple,
which
version
of
image
to
use
a
Brooke
we're
just
using
latest
weather
to
use
devices
or
data
ders
and
a
bunch
of
device
filters.
This
is
well,
you
will
see
a
lot
more
here,
but
this
is.
B
B
What
happens
is
in
if
an
instance
of
the
cluster
is
created?
The
operator
is
watching
this.
There
are
party
resources.
It
sees
that.
Oh
somebody
wants
a
new
cluster
and
here
are
the
parameters
for
the
cluster,
so
the
operator
wakes
up
says
I'm
going
to
go,
deploy
a
new
cluster
and
what
the
operator
does
is.
It
actually
looks
at
the
size
of
the
cluster.
It
looks
at
the
number
of
nodes
in
the
cluster.
B
It
looks
at
storage
nodes,
it
finds
it
uses
a
number
of
heuristics
to
figure
out
what
the
right
size
of
cluster
to
deploy
and
in
this
case-
and
it
starts
with
creating
essentially
the
set
monitors,
bootstrapping
them,
creating
a
quorum
waiting
for
them
to
come
up.
Putting
the
networking
setting
everything
else
around
the
cluster.
B
There
we
go,
and
so
you'll
see
that
it
created
three
monitors.
This
is
a
step
term
if
you're
not
familiar
with
chef,
Steph
has
essentially
its
own
Paxos
role,
where
it
uses
it
for
essentially
maps
and
and
consistently
in
the
cluster.
So
this
rook
operator
created
three
monitors
in
a
three
node
cluster:
it
created
three
storage
nodes,
one
for
each
device
on
each
of
the
nodes
and
it
set
it
set
those
up
and
it
created
a
rook
API
server,
that's
also
running
as
part
of
the
cluster.
B
B
B
B
Yep
yeah,
so
the
other
thing
that
rook
does.
Is
it
essentially
abstracts
a
lot
of
the
complexities
of
stuff
around
not
that
they're
not
important
concepts,
but
we
think
that
we
can
expose
simpler
concepts
that
essentially
right
on
top
of
us
crash
maps
are
the
ones
here
you
mentioned,
are
a
good
example
of
that
super
powerful
constructs.
B
B
B
A
B
You
could
you'll
see
I
said
they
use
all
use
all
devices
equals
true.
This
essentially
is
a
flag
that
says
I'm
essentially,
I
have
a
dedicated
storage
cluster.
All
physical,
the
storage
devices
will
be
turned
into
a
rook
cluster.
Right,
so
will
will,
in
you
ate
all
your
devices
will
figure
out
the
layout
and
we
will
use
them
for
storage.
You
can
use.
A
C
B
So
surf
surf
is
really
good
at
all
of
that
stuff
has
internal
mechanisms
for
detecting
wine
storage.
Note
goes
away
what
was
both
temporarily
and
permanently
and
triggering
essentially
their
rebalancing
and
and
and
the
rear
application
of
data,
so
that
we
take
advantage
of
all
of
that.
What
at
the
rook
level
will
also
recognize
when
a
new
node
shows
up
in
a
kubernetes
cluster
and
then
add
it
to
stuff
and
manage
that
that
part
of
it
ok.
C
B
B
You
know
some
data
to
move,
but
it's
not
a
loss
and
it's
not
necessarily
affecting
a
single
pool
right.
It
might
be
affecting
different
pools,
but
if
you're
asking
how
do
I
if
I
want
to
decommission
this
set
of
notes
harder
and
how
do
I
know
which
order
to
do
them,
there
isn't
any
sports
for
this
right
now,
yep!
B
B
What
I'm
going
to
do
next
and
I'll
show
you
this
it's
a
this
one
has
a
little
bit
of
a
hack
until
we
get
our
volume
plugin,
but
I'll
show
you
so
I'm
going
to
do
two
things.
Now
that
I
have
a
physical
cluster
running
I'm
going
to
create
a
pool
and
a
pool
is
essentially
a
logical
construct
that
says:
I
want.
You
know
this
kind
of
replication
policy,
or
maybe
it's
a
racial
coding
and
I
can
select
devices
whether
it's
SSD
and
p.m.
et
cetera
kind
of
alike
and
I
can
I.
B
Can
its
I
could
use
this
pool
in
for
certain
applications
or
for
certain
workloads.
It
Maps
very
closely
to
the
concept
of
a
storage
class
in
kubernetes,
so
the
approach
we're
taking
is
that
a
storage
class
can
actually
reference
a
pool,
but
you
can
create
the
pool
directly
from
using
a
third
party
resource,
so
I'm
going
to
do
that
here.
The
top
part
is
kind
equals
root,
rook
pool,
so
that
creates
a
new
pool.
B
It's
set
to
replicate
and
right
now,
just
one
replicas
as
in
very,
very
durable
and
on
the
bottom
section
shows
a
storage
class,
and
this
is
a
this
is
an
existing
API
code
concept
as
part
of
kubernetes,
and
there
will
actually
point
the
storage
class
to
the
rook
pool
so
I'll.
Let's
actually
do
that.
B
So
if
you're
not
familiar
with
storage
classes,
they're
essentially
a
way
for
admins
to
design
a
policy
for
how
volumes
get
created
right
from
which
pools
which
discs
in
a
say,
AWS
environment.
It
could
specify
the
kind
of
EBS
volumes
you
want
mount
and
once
you
create
a
storage
class,
then
people
can
use
the
storage
class
application.
B
Can
actually
use
the
service
to
create
volumes
and
volume
claims
and
we'll
walk
through
that.
So
at
this
point,
I've
done
I
created
a
pool.
I
have
a
storage
class,
let's
go
ahead
and
go
create
an
application.
I'm
going
to
use.
Wordpress
WordPress
requires
persistent
storage
itself
for
its
art
for
its
content,
but
it
also
uses
MySQL.
Mysql
requires
persistent
volumes
to
store
the
data
to
be
reliable.
So
let
me
show
you
what
that
looks
like
and
then
just
quickly,
so
the
top
bar
creates
a
persistent
volume
claim
for
the
MySQL
database.
B
So
this
claim
essentially
matches
up
to
the
search
class
and
the
storage
class
matches
up
to
the
pool
and
the
pool
is
connected
to
the
cluster.
That's
essentially
the
chain
of
how
we
know
which
devices
end
up.
Storing
the
data
that's
sitting
underneath
the
spool,
and
it's
also
how
we
know
the
kind
of
replication
policy
or
what
are
using
a
ratio
coding
or
all
that
stuff
to
define
on
that
path.
But
it's
defined
in
different
forms
in
different
ways,
so
that
there's
a
nice
separation
of
concern
between
an
admin
and
an
application,
developer,
Lucas.
B
A
B
B
So
at
the
time
of
when
the
pot
is
launched
or
scheduled,
kübra
days
will
go
and
it
will
create
a
new
volume
and
I'll
do
so
against
the
storage
class
in
the
pools
and
everything
else.
And
then,
when
the
pod
is
scheduled
on
the
machine,
it's
running
as
a
part
of
docker,
then
it
will
mount
and
there
is
the
mount
that
volume
on
that
path.
All.
A
A
B
Volumes
have
been
created,
one
for
MySQL,
the
other
for
WordPress
20
gigabytes
each
now,
after
those
are
created,
the
WordPress
application
is
running,
and
so
the
MySQL
database
that's
supporting
it.
All
of
this
was
done
automatically
right.
All
the
volumes
are
created
everything
cluster,
all
of
it.
Let's
actually
start
it
up
and
see
it
running.
A
C
C
B
The
other.
The
other
reasons
that
people
look
at
this
in
these
environments
is
that
they,
you
know
at
least
in
Amazon
an
EBS
volume.
Is,
you
know,
essentially
within
one
availability
zone,
you
can't
say
detach
it
from
one
zone
and
attach
it
in
another
zone.
You
have
to
snapshot
and
move
it
right
by
running
a
software-defined
stack,
you
can
actually
create
a
cluster
that
is
across
multiple
zones
and
a
pod
could
fail
over
across
zones
without
needing
to
actually
do
anything
and
in
such
environments.
C
B
Question
right
now
we're
using
the
standard,
CFR
Biddy
plugin.
That's
part
of
kubernetes
that
big
big
statement
here
with
all
these
exports
and
magic
acts
in
it
is
to
work
around
a
set
of
issues
with
the
RVD
plugin
we're
creating
a
rook
plugin.
We
hope
we
hope
will
be
part
of
1.7
kubernetes
1.7.
That
makes
this
whole
experience
a
lot
better.
B
The
other
thing
I'll
say
is
that
rook
is
not
just
like
stuff:
it's
not
just
blog
storage,
so
you
can
actually
run
a
shared
file
system
here
as
well,
and
so
you
know
you
could
not.
You
can
go
through
a
path
of
not
just
RVD
that
sefa
'this
is
the
other
path
and
there
you
just
use
you
know
whatever
your
favorite
SP
client
is
from
or
swift
client.
B
C
B
Very
good
question
so
so
in
general,
I'll
say
at
the
managing
resources:
kubernetes,
there's
a
lot
of
facilities
for
managing
memory
and
CPU,
and
you
know
with
labels
and
selectors
and
all
policies.
You
can
set
network
traffic
very
different
story.
Typically
with
CNI.
You
have.
You
know
one
interface
that
you
set
on
a
volume
or
on
a
container
and
that's
all
you
get.
You
can
play
games
with
things
like
SRV,
io
and
others
at
the
host
level.
To
do
this,
you
can
play
games
with
v-line
colouring.
B
Essentially,
you
can
mount
multiple
networks
in
a
container
even
with
C
and
I
there
is.
There
are
ways
to
do
that
right
now,
we're
taking
the
simple
case.
It
is
not
the
fastest
case.
The
fastest
case
will
be
one
where
you
have
to
color
the
traffic
and
prioritize
it
in
one
of
the
ways
I
just
mentioned.
A
B
C
B
Have
it
on
adjust,
we
have
a
bunch
of
stuff
on
the
read
media,
helps
you
walk
through.
All
of
us.
I'll
put
this
exact
demo
on
there,
as
well,
so
just
to
wrap
up
kind
of
what
the
storage
operator
is
and
what
it's
doing.
Essentially,
it's
defining
desired
state.
This
is
what
I
want
to
happen
in
my
cluster
through
third-party
resources,
and
you
can
take
those
extensions
to
kubernetes
just
like
there
is
a
deployment
and
a
service
and
all
those
concepts
that
are
in
kubernetes.
B
We
now
added
a
cluster
or
storage
cluster
in
a
pool,
an
object,
storage
and
all
those
those
are
essentially
concepts
that
are
now
extending
kubernetes.
The
operator
tries
to
achieve
this
desired
state.
The
you
know
the
will
of
the
admin
the
desired
by
watching
both
what's
happening
in
a
network
what's
happening
in
the
cluster
and
watching
the
actual
desired
state,
and
it's
sitting
essentially
in
the
loop,
just
like
other
controllers,
trying
to
make
actual
state
approximate
desired
state.
That's
the
role,
and
it
knows
a
lot
about
the
storage
cluster
itself.
B
So
it's
doing
it
in
a
way.
That
is
very,
you
know,
unjust
understanding
of
the
fact
that
it's
a
stateful
data
and
needs
rebalancing
and
you
need
notes,
come
out
and
all
those
deficits,
and
then
the
last
part
I'll
say,
is
that
it's
not
on
a
data
path
can
be
offline
for
minutes,
not
everything
will
run,
but
when
it
comes
back
up,
everything
is
everything's
normal
and
it
can
run
in
an
H
a
pair
to
a
deployments
if
we
need
it
to.