►
Description
Torus: Focusing Storage for Kubernetes - Barak Michener, CoreOS, Inc.
"If Kubernetes can orchestrate computation across any cluster, on any cloud, how can we do the same for orchestrating storage? Further, can storage for Kubernetes be easily managed by Kubernetes? CoreOS is attempting to bring solid, free persistence across a cluster through the Torus project, starting as a prototype and being built publicly with the open source community. This talk will discuss:
* The needs of the community around container storage
* The goals and architecture of Torus
* How Torus integrates with Kubernetes FlexVolumes
* Managing Torus using Kubernetes
* What we’d like to see in the future for Kubernetes-native storage"
About Barak Michener
Barak Michener is a backend Go developer working on Torus for CoreOS and lead maintainer of Cayley, an open source graph database. Previously at Google he worked on Knowledge Graph using structured data to improve Google Search after some time focusing on music research and multi-model machine learning algorithms. He calls his alma mater of Berkeley home after years in NYC.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Alright,
let's
go
ahead
and
get
started:
I
guess:
yeah
hi
everyone,
I'm
barrack,
I
work
at
core
OS
and
I
work
on
Taurus,
which
you
may
have
heard
of
you
haven't
heard
of
until
now.
Well,
welcome
because
you're
going
to
hear
about
how
it's
built
and
what's
going
on
yeah,
it's
a
little
bit
about
this
talk
we're
going
to
talk
today
about
storage
in
general
storage
on
Cooper,
Nettie's
motivations.
What
do
we
have
going
on?
Scuba
name
is
where
we're
at.
A
A
We
want
innovate,
storage
with
Cooper,
Nettie's,
so
kind
of
a
weird
question,
but
who
here
is
to
put
a
cluster
the
hardware
away,
I
mean
I
know
how
Kelsey
I
did
the
hard
way,
but
about
the
hardware
way
good
lots
of
hands.
That's
actually
better
I
was
expecting
I
mean
how
many
people
pixie
boot
their
own
servers
and
like
that
closet
or
rack.
A
If
you
have
one
at
work
as
opposed
point
being
setting
up
storage
on
these
things,
which
you
can't
get
going
was
one
of
the
first
things
I
noticed
by
setting
up
my
server
that
wasn't
immediately
obvious
right,
because
it's
all
about
the
big
three
computer
networking
storage
in
any
sort
of
a
cluster
environment.
This
is
what
we
care
about
so
compute.
We
notice
this
is
Cuban
is
this
is
what
we're
all
here
for
is
how
do
we
distribute
compute
across
bunch
of
nodes
in
a
cluster?
A
Sed
is
a
lock
server
to
coordinate,
which
IP
is
go
where
so,
that's
kind
of
it.
Ok,
interesting
pattern.
What
about
storage
and
no,
not
everything
is
stateless.
Much
as
you
would
like
it
not
to
be
right,
so
you've
got
a
mess
of
options
when
it
comes
to
Q&A
is
this
is
just
from
the
docs,
the
various
you
know,
volume
plugins
we
have
baked
into
Cuban
a
as
it
is
today
and
well
so,
let's
you
know
take
this
moment
and
gotta.
Have
this
whole
daunting?
Let's,
let's
back
up,
what
is
it
we
need
right?
A
What
do
we
need
out
of
storage
on
cupid
is
well
if
we
go
way
back
to
the
basics
right.
This
is
something
you
probably
all
saw
in
CS
classes
or
books
on
CS
before
which
is,
as
you
know,
you
get
closer
and
closer
to
the
CPU
to
have
higher
bandwidth
low
latency,
but
you
can't
store
nearly
as
much
and
with
what
we
kind
of
want
right
behind
us
all.
A
It
is
after
system
and
after
local
storage
is
distributed,
mass
storage,
a
place
where
our
bits
can
live
and
in
perpetuity
that
may
or
may
not
be
on
our
local
host,
and
there
are
two
main
categories
for
this
right:
block,
storage
and
object:
storage
blocks
towards
being
sort
of
volumes
for
pods.
The
way
we
think
about
them
today,
available
anywhere
in
a
cluster,
would
be
nice
right
and
should
be
as
easy
as
host
path
or
empty
there,
which
are
built
in
like
the
things
we
see
in
all
the
examples
for
cooper.
A
Nate
is
how
you
run
applications
that
have
some
state.
They
usually
sit
in
a
hose
path
under
empty
there
and
wouldn't
be
great
if
we
kind
of
had
EDS
in
the
amazon
world
but
everywhere,
because,
like
a
lot
of
hands
with
showing,
we
want
to
run
this
on
bare
metal,
we
don't
have
the
wheels
of
amazon
behind
us
and
then
there's
object
storage.
A
Similarly,
application
is
built
to
understand
buckets
things
like
s3,
a
cloud
storage,
a
little
bit
of
HDFS,
which
we
have
dealt
with
before
if
you've
been
managing
servers
for
a
long
time,
which
usually
has
really
great
throughput
like
I
can
just
write
as
fast
as
I
can
blast
to
the
network,
but
you
know
limited
semantics
on
you
know
what
I
can
possibly
do
to
this
file.
It's
not
necessarily
always
arbitrary
lead
right.
A
Sometimes
there
are
penned
only
sometimes
you've
got
some
other
constraints,
but
importantly
also
gives
you
the
ability
to
have
multiple
nodes
in
a
cluster.
You
know
we
didn't
write
the
same
things
at
the
same
time,
which
is
really
handy.
A
lot
of
different,
especially
data
processing,
use
cases.
Yeah
so
wouldn't
be
great.
Also
if
this
was
easy
to
run
as
a
cooper
Nettie
is
you
know,
replication
controller
deployment.
Damon
said
anything
you
like.
We
already
do
this
for
cube
DNS,
it's
one
of
the
things
we
go.
A
You
know
tend
to
install
first
when
we
first
have
a
community's
cluster
running.
We
also
do
this
with
monitoring.
We
do
this
with.
You
know
the
API,
which
we
call
it
cube
proxy,
there's
a
handful
of
things
that
start
up
on
every
nail
that
once
you
get
going
and
if
it
were
an
open
community
platform
to
abstract
persistence
across
the
cluster
of
servers
using
at
CD
is
a
lock
server
following
the
rest
of
the
pattern
that
we
have
been
noticing.
So
what
we'd
really
really
want
right?
A
Some
sort
of
magical
file
pointer,
something
in
the
cluster
that
stored
bits
that
was
distributed.
Sharded
replicated
error
detected
error,
corrected
version
encrypted
and
available
everywhere.
That's
the
goal
right,
that's
the
thing
we
want
is
to
be
able
to
build
so,
and
we
want
it
to
be.
One
could
cuddle
away
right.
We
want
to
be
able
to
just
launch
this
upon.
Creation
of
you
know:
we've
pixie
down
machines,
we've
got
you
know
terminators
up
and
running.
Maybe
it's
self
hosted,
maybe
not
other
talks.
A
We've
been
giving
this
conference,
but
it
should
be
just
as
easy
to
go
to
create
dash
F
storage
and
make
it
work,
and
so
that's
why
we
create
Taurus
right
we're
trying
to
push
the
into
the
envelope
and
push
into
this
white
space
of
what
can
be
build.
That
is
this
easy
to
use
under
Cooper,
Nettie's
kind
of
a
missing.
A
Please
the
platform,
if
you
think
about
compute,
networking
storage,
all
based
for
consensus
reasons
on
entity
so
and
whatever
well
we're
giving
close
actually
in
the
sort
of
minimum
viable
Taurus,
it
was
distributed
and
sharded
and
replicated
in
there
and
detected
not
corrected
yet
versions
and
available
anywhere.
So
we're
you
know,
kind
of
building
up
the
things
that
we
kind
of
want
a
little.
So
now,
let's
talk
a
little
about
later
stores.
How
does
it
work
right?
It's
I'd
love
to
give
a
live
demo!
I'm
happy!
A
A
So,
as
with
many
architectures
for
other
for
storage
products
in
general,
there's
consensus
and
the
metadata
being
run
through
etsy
d,
and
then
we
draw
a
kind
of
a
very
hard
interface
here,
which
is
any
sort
of
metadata
or
agreement
upon.
The
state
of
the
world
needs
to
go
through.
This
go
interface
up
to
etsy
d,
and
this
kind
takes
place
of
a
metadata
storm
on
or
something
similar.
A
The
server
abstraction
is
just
kind
of
the
entry
point
for
main
we
create
a
torus
server,
it
can
be
a
client,
it
can
be
a
server
server
is
kind
of
a
misnomer,
and
this
is
the
thing
that
allows
you
to
open
one
of
these
magical
file.
Pointers,
that's
it.
It
is
a
way
look.
A
view
into
the
distributed
platform
is
also
where
we
can
track
monitor
it.
Heartbeats
this
heart
beats
up
to
X,
ed,
saying
I'm
alive.
A
I'm
here
I'm
using
whatever
resources,
I'm
using
I'm,
providing
whatever
resources
I'm
providing
and
below
that
it
needs
to
be
able
to
get
blocks
for
this
file.
That
is
just
handed
to
you
so
underneath
it
is
kind
of
a
distributor
layer
and
named
it
have
to
kind
like
a
distributor
cap
in
a
car
which
is
you
know
the
way
that
each
block
can
be.
You
know
fetched
peer
to
peer
over
gr,
pc
and
then
also
on
this
layer.
A
Keep
track
of
what
storage
lies
underneath
it,
which
is
just
a
key
value,
store
nothing
too
fancy
down
there
or
appears
as
one
and
then
also
then
replicates
to
GCS,
based
on
the
state
of
both
etsy
d,
the
metadata
coming
from
above
and
the
messages
is
receiving
from
its
other
peers.
That's
right!
So
yeah.
What
is
one
inside
one
of
these
files?
I
said
the
server
hands
you
a
file.
How
do
we
achieve
some
of
these
magical
file?
Pointer
aspect,
which
is
that
least,
you
know?
A
First
inside
of
a
file,
every
file
has
one
of
these.
I
know'd
struct
it
since
it's
a
pendulum,
a
view
of
the
world,
which
is
to
say
that
once
these
one
of
these
things
is
written,
it's
never
rewritten,
it's
always
only
ever
replaced
and
so
and
I
know
over.
The
file
of
the
span
of
its
life
will
be
replaced
by
multiple
previous
I
know,
so
that
creates
the
chain
there
and
then
also
keeping
track
of
which
blocks
are
in
this
file.
A
A
list
of
blocks
of
a
fixed
size
default
is
512k,
but
that's
actually
configurable
at
cluster
creation.
Time
and
above
each
of
these
blocks
is
from
stretchin.
If
you
can
see,
we
could
have
just
a
list
of
blocks
and
that
would
be
a
fully
cogent
file,
but
we
could
also
have
a
list
of
CRC's
for
each
of
these
blocks
and
we
can
check
those.
We
can
also
have
a
with
subsidies
blocks
having
a
error
correction
block.
We
can
also
have
each
of
these
things
have
in
a
caption
key.
A
We
could
also
have
an
arbitrary
abstraction
that,
to
the
consumer
looks
like
a
list
of
blocks,
but
below
gets,
checked
and
fetched,
possibly
from
other
servers
or
generally
looks
like
a
list
of
blocks,
and
so,
as
these
things
kind
of
progress,
you
know
we
have
v1
that
they
end
up
being
replaced
by
V
to
the
N
dote
upon
the
sinkhole
st.
being.
A
Why
are
my
favorite
sis
calls
because
that's
when
you
are
probably
lied
to
by
your
underlying
storage,
but
just
for
Speed
reasons,
but
that's
actually,
where,
like
we
try
and
have
a
contract
right,
is
to
say
that
my
F
sing,
something
it
actually
got
written
theoretically
to
the
underlying
storage
and
in
the
same
way,
that's
when
etsy
be
will
be
updated,
informed
of
a
new
version,
a
new
version
printed.
The
inode
is
no
longer
mutable.
A
At
that
point,
it's
fixed
and
sent
out
everywhere
and
the
way,
then
that
works,
is
that
if
you've
changed,
the
block
inside
of
and
I
know
it
saved
on
from
being
a
blue,
the
new
blocks
data
will
also
be
sent
out
as
soon
as
it
was
received.
Just
the
referenced
gets
changed.
We
sink
and
the
data
still
lives
out
there
until
finally,
the
rebalance
er
comes
along
and
sees
okay.
A
I
have
this
block,
which
is
no
longer
lives,
no
longer
reference
by
a
nod
to
and
that
second
data
block
there
is
ready
for
garbage
collection
once
every
node
has
checked
in
with
that
CD
and
knows
that
the
latest
version
is
too.
If
it
still
thinks
it's
one,
it
can
hold
on
to
future
versions.
That's
cool
because
it's
still
in
the
future
and
we
don't
know
to
delete
it
yet
and
we
shouldn't
because
it's
coming
from
the
future
coming
from
possibly
unwritten
state,
but
if
it
was
old,
we
know
it's
time
to
go
mmmm,
okay.
A
How
do
we
go
ahead
and
distribute
these
things
so
we've
built
with
a
file
abstraction?
We
can
keep
our
blocks
in
our
storage,
male
tanki
value
below
us.
How
do
we
go
about
actually
distributing
them
and
so
wait?
Tourists
visit
earth
consistent,
has
passion
for
rebalancing,
and
this
is
kind
of
a
fun
little
twist
of
a
kind
of
put
on
the
standard
model,
which
is
this.
A
If
you
assume
that
there's
a
pure
function,
the
hatch
ring
one
that
takes
any
reference
for
any
block
and
gives
you
a
full
permutation
of
all
the
nodes
in
the
cluster.
It
could
be,
you
know,
hundreds
possibly,
but
for
the
moment
where
it,
let's
look
at
four
and
with
a
replication
layers
to
that
means
that
the
first
two
are
red.
That's
where
the
block
should
live.
There's
on
those
first
two
and
should
always
live
on
the
top
n
of
the
resulting
list.
A
A
A
So
we
had
his
estate
and
we've
gone
ahead
and
added
the
new
node.
We've
changed
the
function.
Each
of
these
functions
as
a
version
h1
is
immutable.
Each
one
gets
updated
from
that
CD.
The
same
way
that
I
know
Dwight
and
note
for
looks
around
soon
as
it
has
its
block.
It's
always
checking
itself
and
says:
hey
I,
have
this
block
the
new
hash
function
says
I'm
ranked
3
in
this.
It
should
be
too
hey
just
to
have
it.
Oh
no
to
doesn't
have
it
over
a
peer-to-peer
here.
A
Put
a
note:
5,
please
have
a
copy.
35
gets
its
copy.
It's
like!
Oh
thank
you,
note
4
and
now
a
3
copies
habit,
and
then
we
come
back
around
again.
This
is
also
been
garbage
collection,
which
is
node,
5
Susan
says
it's
happy,
make
sure
the
no
2
has
it
no
to
check
node,
5
and
node
for
once
it's
finally
done
checking
itself
and
seeing
that
everyone
who
should
have
this
data
has.
A
It
is
free
to
delete
it
and
today
we've
kind
of
rebalanced,
our
cluster
and
changed
where
this
piece
of
data
lives
by
changing
appear
function.
Now
that
function
can
be-
and
this
is
kind
of
one
of
the
more
not
out
there
but
like
direction
or
ideas.
We're
thinking
about
is
being
able
to
have
combine
multiple
functions.
If
you
can
think
about
a
function,
that
kind
of
takes
two
of
these
rings
and
plates
another
ring
out
of
it
as
long
as
has
consistent,
you
know
way
of
going
about
it.
A
We
can
start
to
do
things
like
record
wariness
right.
We
could
say
only
things
that
have
this
rack
should
be
at
the
front
of
the
list
or
things
that
you
know
are
in
this
data
center
or
you
can
or
have
SSD.
Is
there
have
a
certain
speed,
there's
a
lot
of
different
parameters
you
might
want
to
have
on
these
things,
tags,
labels
and
cooper
Nettie's,
but
at
the
end
of
the
day,
you
could
have
functions
that
build
on
the
other
functions
and
compute
these
things,
which
would
be
really
cool
so
well.
A
A
Interestingly,
the
storage
nodes
identity.
It
gets
minted
when
you
create
the
data
directory
and
lives
with
that
data
directory.
The
note
the
fact
that
the
process
itself
is
running
or
hat
is
kind
of
arbitrary.
If
that
process
were
to
die.
If
you
would
read
with
machine
everything,
you
need
lives
in
the
day
directory.
So
as
soon
as
it
comes
back
up
and
sees,
it
just
presents
that
same
folder
for
lack
of
a
better
word,
either
stay
directory
or
a
physical
disk.
We
actually
have
a
PR
for
that
too.
A
This
will
also
happily
use
the
new
SED
controller
that
we
at
core
OS
we've
been
talking
about
this
past
week.
If
we
can
spin
up
and
manage
and
have
self
managed
at
cds,
that
kind
of
creates
our
metadata
story.
We
can
just
deploy
an
exedy
controller,
specifically
futurus
on
our
cluster
and
then
deploy
all
the
tourists
ads
that
connect
to
the
set
C
D,
and
then
we
connect
in
some
sense
it's
a
wit.
It's
a
service
designed
to
bring
a
host
path
or
some
other
local
item
per
node
into
the
distributed
world.
A
So
that's
its
job.
How
did
it
actually
connect
to
these
things
as
well
through
flex
volumes
which
I
I'll
get
back
to
this
one?
Second,
however
Taurus,
if
you
haven't
used
the
Flex
William
before
it's,
how
other
pods
can
use
tourists
or
any
other
arbitrary
binary
for
volumes?
It's
just
right
now,
exact
interface,
which
means
that
the
couplet
shells
out
runs
your
flex
volume
binary
in
it
lens
reflex.
A
William
volume
mount
with
some
options,
does
nothing
special
other
than
exec
with
the
appropriate
interface,
but
this
is
really
kind
of
a
pain
to
update,
because
if
we
have
this
binary
on
each
host,
we
have
to
actually
put
that
binary
on
each
host.
Somehow
and
hey.
Wouldn't
it
be
great
if
we
could
do
that
through
containers,
because
hey
we
delivered
by
night
hosts
all
the
time.
So
in
fact,
that's
a
little
bit
of
what
we're
thinking
about
I
see
a
lot
of
people
here
from
sink
storage
hi.
A
A
Am2
TCP
socket
a
unix
akkad,
something
that
can
grab
a
container
from
the
internet
and
then
provide
an
endpoint
by
which
mounts
can
happen
through
that
binary
that
lives
in
the
container
controlled
by
the
couplet,
which
will
actually
talk
to
the
socket
endpoint
and
send
the
commands
similar
instead
of
exacting
but
just
running
something
else,
and
then
I
forget
it
forgotten
to
15,
but
I've
dynamic,
provisioning,
4
flex
volumes.
If
it
doesn't
exist,
it
will,
which
is
awesome.
A
A
Space,
maybe
were
doing
it
wrong,
labor
billion
right,
but
we
need
at
least
a
straightforward
software-defined
storage
solution,
one
coupe
little
way
so
that
we
can,
even
if
we
have
better
hardware
for
like
your
database,
everything's
a
specific
new
hardware
that
you
pay
a
lot
of
money
for
there's
a
lot
of
little
stateful
services
that
could
greatly
benefit
from
having
generic
available
storage
on
every
cluster
that
wanted
it,
and
so
software-defined
storage
should
be
free
and
open
source.
I
see
no
reason
why
it
shouldn't
be
yeah.
It's
been
poked.
A
B
A
Obviously
we're
working
on
it,
that's
pry
my
team's
goal
and
we're
going
to
my
team,
which
is
good.
So,
hopefully
you
know
our
goal
is
in
the
moderate
future
to
start
saying
beta,
which
is
to
say
yeah,
I
love,
some,
perhaps
not
mission
critical,
but
did
you'd
like
to
see
persist
with,
but
as
for
trying
it
today,
you
can
obviously
just
try
it
and
see
if
anything
works
at
all,
which
is
where
we're
starting.
C
A
Is
presented
as
a
mount,
however,
mount
happens
and
it's
completely
up
to
the
flex
volume
binary
to
do
that.
So
I
can
use
a
block
store
inside
the
pot
as
well
yeah
or
it
could
use
a
like
a
fuse
driver
or
something
it
just
has
to
be
able
to
given
a
directory
mount
whatever.
It
means
to
into
that
directory
and.
C
A
Really
the
thing
before
we
test
a
little
more
is
to
do
more
dogfooding
internally
and
to
do
more
kind
of
torture
tests
instead
of
yeah
I
have
things
that
died
intentionally.
We
did
this
a
lot
with
that
CD.
That
was
really
helpful
for
making
sure
that
city
was
fairly
stable
and
furthermore,
because
of
that,
we
can
trust
that
to
be
a
lot
more
for
tours
as
I
might
a
datastore
so
setting
up
the
same
sort
of
thing,
futurus
and
then
yeah,
that's
kind
of
how
does.
A
A
A
Throughput
just
direct
blasting
bits,
you
know,
do
a
DD
to
a
block
device
or
just
copy
a
whole
image
up
pretty
much
saturates
gigabit
at
least
as
much
as
we
have
in
our
in
my
little
test
environment.
As
for
like
random,
small
I,
ops
inside
of
a
block
device,
that
could
be
better
improving
that
it's
about
a
hundred
ish
now,
which
isn't
great
200,
is
like
a
good
rest,
spinning
disk.
A
South,
because
listen
to
be
a
second
I
mentioned
that
because
and
the
reason
I
think
Seth
is
in
this-
is
guy
my
own
conjecture,
quantico
eating
everyone
else's
lunch
is
that
they
went
through
the
trouble
with
terrible
about
three
years
ago
of
actually
going
to
writing
a
taxes
implementation
inside
of
their
mom.
That's
something
that
things
like
luster
have
not
done,
and
so
you'll
notice,
I'm,
pretty
general
about
think
you
need.
A
Consistently
just
saying
practice:
isn't
it
okay,
just
implementing
factors
anyway,
so
having
some
sort
of
multi
host
highly
available,
but
strongly
consistent
metadata
store
is
something
that
every
distributed
system
like
this
needs.
In
my
opinion,
and
that's
why
something
like
this?
If
I
didn't
need
split-brain
yeah
I
was
methodical,
fine,
but
yeah,
so
having
something
like
that,
so
that
you
can
guarantee
which
side
of
a
partition
say
wins.
A
A
If
you
want
to
tell
you,
have
it
I
met
some
interesting
folks,
this
conference,
the
Rakaia
folks
out
the
door
but
have
been
repackaging,
got
a
staff
into
something
that
could
be
containerized,
which
I
think
is
actually
really
kind
of
cool
idea,
just
that
the
notion
being
that
it
should
be
easy.
It
should
be
open
source,
you
know,
and
that's
kind
of
where
we
should
go
with
it
period
and
as
force
F,
and
why
I'm
lost
my?
Why
this
not
surf?
A
Why
there's
not
Seth
if
I'm
a
customer
sell
it
to
me
if
you're
a
customer
solitude,
because
it's
way
easier
and
we're
focused
on
doing
kind
of
the
things
that
you
need
for
Cooper
nights
and
somewhere,
and
if
someone
else
is
able
to
achieve
that
great
you
know,
because
open
source
is
what
we're
all
here
about
right.
Dubonnet
is
is
open
source,
even
though
we
have
different
vendors,
doing
lots
of
different
things.
A
At
the
end
of
the
day
it
I
we
can
get
a
pretty
far
with
tourists,
because
EDD
we
know
works
as
a
consensus
store
and
the
rest.
We
can
it's
a
separable
problem
right.
Instead
of
trying
to
build
the
whole
world
trying
to
build
just
half
the
world,
I'm
still
a
hard
problem,
but
it's
at
least
you
know,
I
am
I'll,
give
it
the
old
college.
Try
so,
and
maybe
we
can
move
Fester's
or
as
well.
Maybe
we
can
do
other
cool
things
like
it's
amazing.