►
From YouTube: Building Blocks: How Raw Block PVs Changed the Way We Look at Storage - Jose A. Rivera & Rohan Gupta
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building Blocks: How Raw Block PVs Changed the Way We Look at Storage - Jose A. Rivera & Rohan Gupta, Red Hat
Raw block PersistentVolumes (PVs) allow applications to consume storage in a new way. In particular, Rook-Ceph now makes use of them to provide the backing store for its clustered storage in a more Kubernetes-like fashion and with improved security. Now we can rethink the notion of how we structure our storage clusters, moving the focus away from static nodes and basing them on more dynamic, resilient devices. This talk will go over how we incorporated raw block PVs, how the operator manages them, and how we can now define storage cluster. It will also include a demo of the resiliency of these new types of devices. By the end of the talk, you'll not only know how to use raw block PVs but also why and when to use them.
https://sched.co/UaWo
A
All
right
welcome
everyone.
My
name
is
Sally
I'm,
very
lucky
to
introduce
Rohan
and
Jose
from
Red
Hat
they're
going
to
be
talking
about
raw
block
volumes
today.
So
to
put
this
into
context
for
you,
if
you're
familiar
with
the
kubernetes
volume
subsystem,
what
you're
going
to
know
is
that
for
the
most
part,
the
system
allows
you
to
access
volumes
as
files
with
a
file
system
applied
to
it
and
you'll
have
a
directory
mounted
into
your
container,
and
when
you
read
and
write
files
to
that
directory,
you
know
it'll
be
available
to
you.
A
It'll
be
persisted
out.
There
are
some
subset
of
applications
that
don't
like
to
interact
with
volumes
through
a
file
system
interface,
and
they
want
to
be
able
to
have
raw
block
access
to
those
volumes,
and
so
this
new
feature
that
is
currently
in
beta
raw
block
volumes
is
what
they're
gonna
be
talking
about
today
and
they'll.
Tell
you
all
about
it.
Please
give
them
a
warm
welcome.
Thank
you.
B
B
Alright,
introductions,
I'm
Jose
over
there
on
the
over
there
on
the
left,
I've
been
in
and
around
storage
for
about
ten
years
currently
I
work
in
open
shift,
container
storage
or
OCS
I
participate
in
six
storage
and
I,
like
hitting
things
mostly
drums.
This
over
here
is
Rohan.
He
graduated
from
college
in
India
in
2018
he
did
a
google
Summer
of
Code
with
the
CNC
F
and
worked
on
adding
an
NFS
operator
to
rook.
B
Currently
he
also
works
with
OCS,
focusing
on
the
rook
upstream,
and
if
you
can't
tell
by
now,
he
loves
watching
anime
and
riding
motorbikes,
alright,
so
here's
a
quick
overview
of
what
we're
gonna
go
through
we're
at
the
agenda
right
now.
We're
gonna
go
over
some
of
the
basics
of
storage
and
kubernetes.
Do
another
quick
overview
of
raw
block
TVs,
especially
for
those
of
you
just
filtering
in
and
introduction
to
rook
and
rook
stuff.
B
Then
we'll
get
into
the
meat
of
it
of
how
OS
D's
war
war
have
been
provisioned
and
how
they're
being
can
be
provisioned
now
and
then
we're
gonna
have
a
nice
little
demo
at
the
end,
all
right
so
to
set
the
stage,
storage
and
kubernetes
your
basic
workflow
with
storage,
D
States
involves
persistent
volumes
or
Peavey's
persistent
volume
claims
or
PVCs
and
storage
classes.
Scs.
B
The
persistent
volume
represents
a
volume
of
storage
in
a
storage
subsystem
somewhere
different
storage
backends
defined
what
a
volume
represents,
but
for
the
purposes
of
an
application
running
kubernetes,
they
don't
entirely
care
what's
running
down
there
and
in
particular
applications
don't
interact
directly
with
PVS.
They
interact
with
persistent
volume,
claims
or
PVCs,
which
represents
a
request
to
use
a
piece
of
storage
that
is
available
in
kubernetes
in
particular.
B
Pbs
are
non
namespaced,
so
their
cluster
wide
but
PVCs
our
namespace,
and
then
you
have
start
classes
another
non
namespace
resource
that
basically
provides
a
point
of
access
where
PVCs
can
dynamically
provision
PBS
for
for
their
use.
This
is,
you
know,
a
relatively
recent
development,
whereas
previously
you'd
have
to
statically,
define
your
PBS
and
then
create
a
PVC
that
directly
referenced
that
pv.
Now.
Instead,
you
create
a
PVC
that
references,
the
storage
class
which
talks
to
the
storage
driver
underneath
and
creates
the
corresponding
PVC.
B
Here's
a
nice
little
graphic
of
that
where
the
developer
specifies
the
PVC
and
operations
or
infrared
ever
specifies
the
storage
class
that
the
PVC
is
submitted
to
it.
Instructs
the
storage
back-end
to
create
the
persistent
volume
at
the
end,
and
then
the
persistent
volume
is
mounted
by
the
application.
On
whatever
notes
note:
it's
running
raw
block
TVs.
So
again,
why
rob
block
TVs?
It
allows
kubernetes
to
present
storage
to
containers
without
having
a
formatted
file
system
on
top
of
them.
B
So
the
reason
for
this
is
that
many
applications,
like
databases
or
software-defined
storage,
can
Leverett
storage
directly
at
the
block
layer,
meaning
that
you
can
cut
out
a
lot
of
abstraction
and
indirection
that
file
system
present.
This
allows
this
typically
allows
for
more
consistent,
IO
performance
and
low
latency
access
to
your
storage,
and
you
can
find
a
link
to
more
about
how
to
use
that
down
there.
B
The
way
you
access
this
in
your
PBS
and
PVCs
is
through
a
new
field
called
volume
mode.
This
went
beta
and
kubernetes
1.13
and
it
specifies
whether
how
a
storage
will
be
accessed,
whether
with
a
file
system
or
astrov
lock.
As
mentioned
a
volume
mode
field,
is
present
in
both
TVs
and
PVCs,
and
then
you
can
just
leave
it
unspecified
or
specified
volume
mode
file
to
go
with
the
backwards-compatible
default.
B
Nord
related
access
modes
includes
things
like
read:
write
many
and
read:
write
once
that
controls,
how
many
pods
may
access
may
attach
a
PVC
at
any
given
time
and
whether
or
not
whether
or
not
how
many
of
those
pods
can
write
to
it,
where
they
intersect
is
that
certain
storage
drivers
that
that
allow
Rob
Locke
volumes
may
allow
various
different
access
modes
to
their
rob
block
storage
depending
on
what
the
technology
supports.
This
is
usually
a
limitation
of
whatever
technology
is
used
to
attach
the
storage
to
the
node.
B
The
depart
is
running
on
all
right
now:
rook
and
rook
Seth.
What
is
rook
we
have
rook
is
storage
operators
for
kubernetes.
It
helps
you
automate
various
aspects
of
your
of
maintaining
your
data
plane
from
deployment
to
bootstrapping
configuration
and
upgrade
rook
up.
Rook
is
a
collection
of
operators
which
are
a
piece
of
software
that
implements
the
operator
pattern
for
storage
solutions,
like
obviously
rook
stuff
here,
but
also
edge,
FS
and
Mineo
in
just
a
quick
overview.
B
Basically,
it
watches
for
changes
in
the
cluster
or
in
your
desired
state
configuration
and
applies
changes
to
the
cluster
to
make
the
cluster
state
match.
The
cluster
desired.
State
rooks
F
was
is,
is
the
attempt
to
bring
SEF
into
containers
to
give
you
resilient,
distributed,
storage
that
is
native
to
kubernetes?
In
particular,
it's
self-healing
can
recover
from
a
variety
of
error
conditions
and
because
it's
stuff,
it
is
highly
scalable-
can
run
on
commodity
hardware,
and
it's
fully
open
source
here
is
just
sort
of
a
quick
overview
of
what
a
stuff
cluster
looks
like
you.
B
B
The
part
we
care
about
here
is
that
SEF
is
divided
into
a
number
of
demons
that
produce
that
have
several
functions.
The
one
we
care
about
are
the
OS
DS
that
are
towards
the
end
in
the
top
corner
there.
This
stands
for
what
is
it?
Objects
or
objects,
service
demon
of
the
stage,
object,
storage,
demon
and
they
are
what
represented
the
storage
device
into
the
stuff
cluster.
B
The
way
this
works
is
that
traditionally
OS
DS
use
locally
attached
devices
to
your
kubernetes
nodes.
So
what
you
would
do
is
that
you
would
set
up,
you
know,
instances
or
physical
machines
with
storage
devices,
and
then
you
could
specify
how
those
storage
devices
would
be
used
by
rooks
F
to
build
your
SEF
cluster
at
its
most
rudimentary
configuration
using
devices
in
this
manner
is
really
easy.
Rook
in
Luke
implements
a
auto
discovery,
feature
that
looks
through
all
your
kubernetes
nodes
or
the
kubernetes
notes.
B
You
limit
it
to
and
finds
available
non
formatted,
empty
storage
devices
and
just
consumes
some
or
all
of
the
storage
devices.
So
at
its
most
basic
configuration,
you
can
just
tell
the
storage
subsystem
to
use
all
nodes
in
the
cluster
and
use
any
free
devices
it
finds
on
those
nodes.
There's
obviously
mostly
ideal,
if
you're,
using
a
purely,
if
you're,
using
a
kubernetes
cluster
purely
to
host
your
data
plane,
but
you
can
use
a
variety
of
node,
selectors
and
taint
in
toleration
x'
to
limit
where
the
rook
SEF
pods
will
run.
B
So
really
all
you
have
to
do
is
define
your
storage
nodes,
define
your
local
devices
and
then
rook
takes
care
of
the
rest.
It
will
prepare
the
storage
devices
for
consumption
by
SEF
and
then
start
the
OSD
pods
on
the
nodes
that
contain
those
storage
devices
on
the
upside.
These
things
are
easy
to
configure
and
for
especially
for
most
storage
administrators
coming
into
kubernetes
for
the
first
time.
B
This
is
very
familiar
to
them,
because
they're
very
much
used
to
attaching
physical,
physical
disk
to
physical
nodes
and
then
consuming
them,
especially
if
they're
coming
from
the
straight-set
world
and
coming
in
to
rook.
This
is
very
familiar
to
them
and,
in
particular,
disappoints
a
any
type
of
device
or
appliance
that
linux
support,
since
all
it
looks
for,
is
a
device
on
the
node,
on
the
other
hand,
especially
for
there's
especially
concerned
for
people
deploying
on
cloud
environments.
B
B
Sturge
plus
devices
were
designed
as
a
generic
rook
struct
and
as
UNF
you
if
you'll
remember,
rook,
is
a
collection
of
operators
for
multiple
storage
services,
so
the
storage
class
device
sets
are
not.
You
know,
are
fairly
extensible
and
feature-rich,
but
some
features
of
them
are
not
used
in
Brookes
F.
We
just
use
a
small
number.
We
just
basically
use
what
we
need
to
get
stuff
up
and
running.
B
So
the
first
thing
we
need
to
set
is
a
name
for
your
search
class
device
set.
This
is
used
in
part
to
generate
unique
and
consistent
PVC
names
for
the
PVCs.
That
will
be
consumed
by
the
OS
DS,
but
it
also
helps
internally
to
sort
of
to
keep
track
of
which
device
belongs
to
which
set,
because
if
you
can
see
on
the
second
line
there,
this
is
actually
a
list
of
storage
class
device
sets.
B
You
can
specify
more
than
one
set
per
storage
cluster
and
we
didn't
want
to
necessarily
go
with
a
straight
mat
with
a
width
map
to
to
set
this
up.
So
we
just
went
with
the
list
of
named
items
now
here
we
get
to
count
count
is
the
number
of
devices
in
this
set.
So
if
you
set
a
count
to
3
3,
OS
DS
will
be
spun
up
in
association
with
this
set.
You
could
also,
for
instance,
specify
three
different
sets
with
a
count
of
1
to
also
get
three
storage.
B
3Os
Dee's
running
in
your
cluster
and
I'll
explain
more
about
that
in
a
little
bit
all
right
portable.
This
is
the
big
feature
for
storage
class
device
set,
meaning
that
PVCs
are
allowed
to
move
between
nodes.
This
is,
in
particular,
an
important
configuration
for
Brooke
stuff,
because
this
is
how
you
trigger
the
new
behavior
that
I'll
that
will
cut
a
new
behavior
that
we
had
to
come
up
with
that
we'll
cover
in
in
a
little
bit
in
a
little
bit
here
and
then
volume
clang
templates.
B
This
is
just
a
list
of
PVC
templates
that
follow
it's.
It
just
imports
the
standard
spec.
So
if
you've
written
any
PVC
if
you've
written
any
PVC
ya
Mille,
this
should
look
very
familiar.
You
specified
the
amount
of
storage.
You
want
the
storage
class.
You
want
to
provisioned
against
volley
mode
block
as
a
requirement
here
and
then
any
access
modes
that
you
want
set
for
this
PVC.
B
B
The
pros
of
this,
you
can
offload
your
device
distribution
to
the
kubernetes
scheduler,
since
these
things
are
just
running
as
pods.
Now,
this
scheduler
will
take
care
of
finding
notes
with
appropriate
resources
and
scheduling
or
moving
pods
around
as
needed.
Incidentally,
device
migration
between
nodes
is
now
possible,
as
I
mentioned
earlier.
Stuff
is
a
self-healing
self-healing
storage
service,
meaning
that
there's
a
lot.
There
can
be
a
lot
of
a
lot
of
data
traffic
going
around
as
it
tries
to
rebalance
how
the
various
blocks
are
distributed
across
the
cluster.
B
So
no
additional
replication
no
need
to
move
anything
around,
and
this
works
with
just
any
Rock
raw
block
TV,
regardless
of
the
storage
driver
underneath
so
as
long
as
your
as
long
as
your
storage
vendor
supports
using
volume
mode
block
on
there
PBS
we
can
consume
it
and,
of
course
this
is
shiny
and
new.
So
it's
very
exciting,
at
least
for
me.
Unfortunately,
it's
a
bit
bulkier
to
deal
with
than
just
than
just
using
local
storage
devices.
B
The
device
support
is
not
as
robust
as
it
is
in
just
straight
Linux.
We
are
limited
only
to
what
we
are
limited
to
only
what
kubernetes
supports
and,
as
you
saw
earlier,
the
configuration
for
this
can
get
fairly
extensive
and
it's
not
even
different.
So
it's
natural
to
be
skeptical
of
these
things,
all
right
and
now,
I'm
gonna
turn
this
over
to
Rohan
to
go
over
some
of
the
bumps
in
the
road
that
we
found
along
the
way
to
implementing
this.
C
C
To
solve
this,
we
use
a
little
trick,
so
we
use
a
non
privileged
init
container
to
copy
the
device
as
the
devices
a
file
into
the
Linux
system,
so
that
file
was
copied
to
a
shared
directory
which
was
shared
between
the
inner
container
and
the
privileged
container
thanks
to
John
for
this.
So
if
I
look
at
the
right-hand
side
bottom,
we
have
two
volumes.
One
is
the
block
TV
that
is
set
one
def
0
and
next
we
have
in
in
today
a
directory.
C
C
This
was
because
when
chef
volume
prepares
the
LVM,
so
we
have
/dev
mounted
and
cuban
it
is
monster
block
device
as
loopback
device.
So
when
slash
Davis
motors,
we
have
the
original
device
of
the
node
there,
as
well
as
the
loopback
device.
So
when
we
use
a
chef
osg
command
to
start,
so
it
detects
like
that.
Iridium
is
having
two
Peavey's
linked
to
it.
One
is
the
loopback
device
and
one
is
the
actual
devices
there
to
simplify
this.
C
The
last
problem
which
we
faced
was
so
G's
we're
clustering
onto
a
same
node.
Suppose
we
way
if
we
spend
1000
series
and
we
had
three
nodes
so
like
five
to
six,
always
DS
we're
coming
out
on
the
same
node.
So
suppose
we
have
three
replicas
for
the
OS
days,
sewed
and
all
they
are
on
the
same
node
and
if
the
node
goes
down,
we
lose
our
data
so
which
was
which
is
undesired.
So
to
solve
this,
we
use
placement
affinities.
So
this
is
a
basic
placement
affinity
that
will
distribute
the
OECD's
around
nodes.
B
Well,
the
main
reason
we
have
opened
openshift
cluster
on
is
because
Red
Hat,
you
know
we,
we
have
a
lot
of
internal
automation
to
make
it
easy
to
set
up
an
AWS
cluster
and
run
openshift
and,
in
particular,
we're
going
to
be
using
a
an
aspect
of
open
shift
for
machine
management
which
makes
it
easy
to
tear
down
and
bring
notes
back
up
from
the
open
chef
console
cool.
So
if
you
have
T
is
the
font?
Okay
for
every
one
volts
in
the
back.
D
C
C
B
So
now
we're
accessing
the
OpenShift
machine
API
to
look
at
the
machines
that
we
have
set
up.
The
main
thing
to
note
for
machines
is
that
these
machines
are
controlled
by
what's
known
as
a
machine
set,
and
a
machine
set
basically
is
allows
you
to
just
like,
like
a
replica
set,
for
instance,
its
job
is
to
through
the
use
of
the
through
the
use
of
the
machine
set
controller
or
the
OpenShift
machine.
Api
api
operator
is
to
make
sure
that
there
are
always
enough
machines
to
satisfy
the
needs
of
the
machine
set.
B
So
what
we're
gonna
do
here
is
delete.
What
is
delete
one
of
the
machines?
That's
part
of
a
machine
set
so
that
it
goes
down
and
another
one
is
being
brought
up.
But
since
we
don't
want
to
wait
for
the
next
machine
to
be
brought
up,
we're
just
going
to
evict
all
the
pods
from
the
first
machine
we're
deleting
and
they
should
be
brought
up
elsewhere.
So.
C
C
Here
we
go
so
when
I
deleted
the
machine,
we
see
an
OSD
event
into
the
pending
state.
This
is
the
new
Z
that
is
coming
up,
Juan
Jose
that
went
off
it
vanished,
so
it
is
the
uzi
that
was
there
on
the
node
which
we
deleted
and
that
always
reminded
it
to
a
new
node,
which
is
one
7:18
in
this
case,
and
we
will
just
wait
for
it
to
come
up
here.
We
can
see
one
of
the
moon.
C
C
C
B
F
B
Why
would
you
arm
as
this
is
in
particular,
if,
for
some
reason,
you
have
a
mix
of
storage
devices
on
a
given
node
that
you
want
to
make
sure
and
you
want
to
make
sure
you
only
use
a
certain
type
of
storage
and
not
the
other?
This
comes
from
this
comes
from
the
early
days
of
how
you
run
Seth
and
in
particular
you
could
run
multiple
Seth
clusters
in
the
same
kubernetes
cluster.
So
you
could
have.
You
could
have
the
same
node,
providing
storage
for
different
stuff
clusters.
B
This
is
something
that
the
rook,
the
rook
project,
wanted
to
make
sure
that
they
that
they
supported,
but
then
also
it's
a
matter
of.
It
could
also
be
a
matter
of
security
policy
where
you
want
to
make
sure
that
there
is
no
like
you
can,
you
can
add
additional
storage
devices
to
the
node,
but
you
don't
want
them
necessarily
automatically
consumed.
So
you
want
to
manually
change
the
configuration
before
they're
consumed
I.
G
E
C
B
C
G
E
B
H
One
more
question:
so
when
you
are
running
with
kubernetes
I
mean
when
you're
running
surf
with
kubernetes,
if
I
OS
B
fails,
then
kubernetes
might
go
on,
might
try
to
restart
it
on
some
other
node
right,
but
the
device
may
not
be
the
same
like
now.
Here
I
could
have
diverse
PA,
but
there
it
could
be
diversity
II
right.
How
do
we
handle
those
kind
of
situations
so.
C
B
B
D
B
Can
but
they
don't
have
to
so
there's
the
distinction
between
now
that
now
you
have
to
make
the
distinction
between
the
PVCs
Seth
is
consuming
for
its
OS
DS
and
a
PV
and
the
PVCs
it
provides
for
applications.
These
are
two
completely
distinct
sets,
so
you
know
one
uses
the
underlying
storage
provider
like
EBS
or
or
Azure
discs
or
whatever
and
then
stuff
just
presents
set
volumes,
and
if
you're
using
RBD
volumes
you
can
consume
those
as
either
raw
block
or
file
volumes.
So.