►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Big Data Operations Using Kubernetes and Local Storage - Dan Norris, NetApp
Many companies run large, difficult to operate data-processing systems that are key to their business. Kubernetes local storage provides a way to keep running these types complex systems on the infrastructure they were provisioned on while providing a better way to operate in production. In this talk, we'll go over how to use Kubernetes local storage to run complex systems such as Cassandra reliably across multiple datacenters. We'll discuss how the local volume provisioner is used to prepare Persistent Volumes on hosts, how to use Storage Classes to segment workloads on the same host, as well as discussing potential failure modes and how we handle backups.
To learn more: https://sched.co/Grav
B
As
well
get
started,
I
think
it's
11
40s
so
good
morning
everybody
hope
everybody's
been
having
a
great
conference.
So
far
my
name
is
Dan.
Norris
I
am
a
senior
cloud
native
engineer
which
totally
a
title
we
made
up
over
at
NetApp
working
on
our
net
up
kubernetes
service
on
today,
I'm
here
to
talk
to
you
about
big
data
operations
using
kubernetes
and
local
storage,
so
kind
of
a
point
of
orders
to
go
through
the.
B
Give
you
just
enough
background
on
Cassandra,
specifically
just
enough,
so
you
can
know,
know
enough
to
be
dangerous
and
then
go
through
the
basics
of
local
storage
kind
of
give
an
overview
of
how
it
works.
How
you
consume
it
then
talk
about
what
deploying
Cassandra
and
communities
looks
like,
while
using
local
storage
and
also
go
through
the
operations
that
this
setup
allows
us
to
automate
in
a
code.
So
quick
background
again
what
I'm,
hoping
that
you
take
away
from
this
session,
is
basically
a
high
level
description
of
running
Cassandra.
B
So
with
that
in
mind,
I'll
talk
a
little
bit
about
Cassandra,
real,
quick
and
specifically
I
want
to
talk
to
you
about
the
Dana
bottle
and
kind
of
what
Cassandra
is
so
Cassandra,
I'm
kind
of
on,
like
the
cap,
theorem
transport,
more
the
AP
side.
So
it's
eventually
consistent
by
design.
The
goal
here
is
that
it's
every
node
or
it
should
be
tolerant
to
individual
node
failures,
so
data
ends
up
being
distributed
across
many
many
nodes,
and
one
of
the
interesting
parts
is
that
replication
is
configurable
per
data
center
or
region.
B
If
you
happen
to
be
working
in
the
cloud,
and
so
if
it
is
an
example
of
that
we,
whenever
we
do
a
data
right,
we
can
place
two
copies
of
that
data
in
u.s.
East
one.
If
we
have
me
running
a
table
us
and
three
copies
of
that
data
in
u.s.
west
two
gives
us
all
sorts
of
interesting
things,
especially
with
like
GDP,
are
now
to
actually
go
ahead
and
really
control
where
our
data
lives
in
the
world
or
within
these
clusters.
B
An
interesting
caveat
to
all
that
is
that
read
and
write
consistency
is
tunable,
so
you
even
on
a
per
read
or
a
per
write
basis
individually.
You
can
actually
decide
whether
or
not
you
want
to
get
cluster
poram
to
respond
back
to
you,
local
quorum.
So,
within
your
data
center
right,
you
just
care
about
one
node.
If
you
really
want
to
get
a
fast,
read
or
write
out
of
that,
and
so
to
put
this
kind
of
into
more
of
a
picture,
this
is
an
example
of
a
ten
node
Cassandra
cluster
running
in
two
different
regions.
B
Us
East
one
USS
two,
and
this
is
sort
of
just
what
the
data
path
might
look
like
on
a
right.
So
when
a
client
goes
ahead
and
issues
a
right,
it'll
hit
an
individual
node
which
may
or
may
not
be
the
actual
destination
for
that
data,
and
if
it's
not
the
destination
where
that
data
that
node
acts
as
a
coordinator,
so
that
node
goes
ahead
and
actually
issues
rights
to
the
nodes
where
that
data
is
supposed
to
live.
In
this
case
on
the
diagram,
node
five
is
acting
as
that
coordinator,
writing
out
two
nodes.
B
Three
and
four:
that's
on
the
left
hand,
side
of
this
diagram
and
concurrently.
So
once
it
actually
goes
through
that
process.
That's
when,
generally
speaking,
when
that
write
is
considered
to
be
successful
in
the
background,
it
goes
ahead
and
replicates
that
data
to
the
other
cluster,
since,
if
it's
configured
for
two
writes
into
US
East
one
and
three
into
US
West
to
it,
goes
ahead
and
issues
that
replication
request
on.
In
this
case
note
five.
B
On
that
right
hand,
side
is
acting
as
the
coordinator
replicating
three
copies
out
to
the
individual
nodes
where
that
data
is
supposed
to
live.
So
that's
a
very
high-level
overview
of
sort
of
that
write
process,
so
the
problems
with
Cassandra
sort
of
as
from
an
operational
or
sort
of
as
while
using
the
system
right
is
that
it
is
actually
very
difficult
to
operate.
It
requires
quite
a
bit
of
knowledge
in
terms
of
tuning
quite
a
bit
of
knowledge
in
terms
of
knowing
where
those
rough
edges
is.
B
It's
no
different
than
running
some
other
big
data
systems
like
Kafka
Hadoop
spark.
You
kind
of
you
always
have
to
sort
of
have
an
expert
in
play,
or
a
team
of
experts
to
really
get
this
to
scale
and
part
of
that,
at
least
from
the
perspective
or
looking
through.
The
lens
of
kubernetes
is
that
some
of
the
stuff
was
actually
built
for
a
pre
container
world
Cassandra
originated
at
Facebook
I
think
it
was
2008,
so
a
very
different
tech,
ecosystem
and
landscape.
B
One
of
the
more
interesting
parts
that
we
ended
up,
discovering
while
going
through
and
building
out
a
very
like
production-ready
deployed
of
this,
is
that
no
discovery
is
by
IP
Cassandra
identity,
sort
of
matters
by
IP
address,
and
usually
that's
okay.
But
for
example,
if
you
want
you
to
run
a
service
mesh
and
have
something
terminate,
connections
in
front
of
Cassandra,
that's
actually
really
relatively
difficult.
B
So
that
other
way,
let's
talk
a
little
bit
about
local
storage,
so
local
storage,
what
I
mean
by
that
is
actually
local,
persistent
volumes,
and
this
is
a
feature-
that's
been
entry
for
a
little
while
it's
actually
just
went
beta
in
112.
So
it's
you
know
in
kubernetes
speak.
That
means
it's
basically
good
to
go
and
the
api's
aren't
going
to
change.
B
So
what
it
allows
you
to
do
is
expose
directories
and
paths
on
your
individual
worker
nodes
as
persistent
volumes.
You
always
could
do
that
at
least
on
the
pod
level.
You
could
use
what's
called
a
host
path
which
would
actually
allow
you
to
mount
in
directories
on
a
host,
but
that
was
in
the
pod
or
deployment
or
staples
that
a
stateful
set
spec,
which
is
not
a
great
abstraction
right.
It
kind
of
it
really
ties
you
in
to
your
config,
for
that
particular
workload.
B
So
a
better
way
to
do
this
is
what
local
storage
allows
you
to
do,
which
is
to
clue
the
scheduler
in
to
tell
the
scheduler
and
let
it
worry
about
how
to
put
your
pod,
where
its
storage
is
rather
than
have
the
pod
trying
to
figure
out
and
slot
itself
in,
and
it
also
lets
you
hide
local
paths
from
the
pods.
The
pod
doesn't
need
to
know
anything
about
how
its
configured
all
it's
doing
is
consuming
storage,
just
like
it
would.
B
B
This
turns
all
of
your
nodes,
where
you
have
workloads
that
are
now
being
pinned
to
nodes
into
snowflakes
and
generally
that's
not
something
we
want
to
do
with
kubernetes.
We've
all
been
in
that
situation
as
developers
and
as
operators,
where
we've
built
systems
that
are
relatively
brittle
and
the
whole
point
here
is
cattle,
not
pets,
right
and
so
local
storage.
By
its
very
nature,
inherently
starts
to
break
that,
so,
if
you
are
going
to
start
using
local
storage,
just
be
very
cognizant
of
that
fact.
It's
just
something
that
you
want
to
keep
in
mind.
B
It's
not
a
bad
thing.
You
know
you
just
need
to
sort
of
prepare
for
it,
but
it
does
really
make
data
and
node
locality
come
into
play.
So
just
as
an
FYI
and
that's
not
usually
a
approach
you
really
want
to
take.
Unless
you
have
to
so
really
cease
network
storage,
if
you
can
that's
pretty
much,
all
I
wanted
to
get
out
of
that,
so
in
terms
of
using
local
storage
with
thus
ends
up.
Looking
like,
is
you
end
up?
Creating
a
storage
class?
B
Is
everybody
familiar
with
a
lot
of
these
abstractions
see
some
nods
okay,
cool
I'm,
gonna,
just
kind
of
go
through
what
this
looks
like
through
storage
classes,
persistent
volume,
just
kind
of
show
you
what
some
of
the
gambol
ends
up,
looking
like
to
give
you
an
idea.
So
this
is
an
example,
storage
class,
it's
all
very
dependent
on
what
you
want
to
name
it.
In
this
case,
we
have
a
storage
class
which
is
calling
it
local
Cassandra,
and
the
real
big
takeaway
here
is
the
provisioner.
B
So,
with
most
storage
classes
you
give
it,
you
tell
it
that
you
wanted
to
use
like
kubernetes,
do
like
EBS
right
or
something
that
clues
the
scheduler
and
to
know
that
you
were
going
to
dynamically,
create
this
volume
and
for
local
storage.
You
tell
it
to
use
this
special
provision,
which
is
the
null
provision
or
the
no
provisioner,
so
it
doesn't
do
that.
All
it
does
is
just
create
the
object
and
then
it's
good
to
go
other.
An
interesting
thing
to
keep
in
mind
with
local
storage.
B
B
So
here's
an
example
of
what
actually
consuming
this
as
a
local
volume
might
look
like
or
a
persistent
volumes.
So
this
is
just
a
very
basic
persistent
volume.
Spec
we're
just
calling
it
example
PV,
because
I'm
super
creative
naming
the
capacity
kind
of
doesn't
matter
just
because
it's
going
to
be
whatever
your
disk
capacity
actually
is.
The
important
takeaway
here
is
that
we
set
the
storage
class
name
to
local
Cassandra,
which
is
the
storage
class
that
we're
looking
at
before,
because
it
is
a
local
volume.
B
Typically,
this
is
sort
of
generated
for
you,
but
what
what
this
block
of
Hamel
is
saying
is
that
it
requires
the
pot
and
we're
if
it's
going
to
consume
this
storage.
It
has
to
be
on
the
particular
node
known,
as
example,
node
right,
that's
just
the
name
of
the
node
in
this
case.
So
that's
what
actually
locks
it
to
that
node
and
the
scheduler
knows
how
to
consume
that,
and
it's
pretty
much
good
to
go
so
note
affinity
is
how
that
takes
care
of
it.
You
can
also
use
anti
affinity
by
the
way.
B
This
is
a
trick
that
we
kind
of
figured
out
after
running,
say
Prometheus
and
Cassandra
on
the
same
node,
which
was
a
terrible
idea,
because,
if
you've
run
Prometheus
at
scale,
you
know
that
it
tends
to
thrash
the
disk
as
much
as
Cassandra
might
so
that
was
causing
all
sorts
of
interesting
issues.
So
what
you
can
do
there,
because
Cassandra
or
if
you're,
using
Cassandra
local
storage
right
and
you
want
to
bind
it
to
a
particular
node.
B
You
can
use
anti
affinity
in
the
specification
for
your
stateful
set
and
we're
going
to
account
what
that
looks
like
a
little
bit
to
prevent
that
from
happening.
So
you
can
tell
it
that
actually,
no
make
sure
that
when
we,
when
you
start
scheduling
these
workloads,
don't
actually
try
and
schedule
to
anything,
that's
running
Prometheus.
To
avoid
the
whole
situation
entirely.
My
recommendation
is
especially
for
these
type
kind
of
workloads.
If
you
are
going
through
and
provisioning,
these
kind
of
really
heavyweight
systems
that
pretty
much
are
consuming
the
whole
note.
B
B
So
this
is
an
example
of
a
persistent
volume
claim,
it's
kind,
as
persistent
volume
claim
we're
just
calling
it
Cassandra,
or
example,
and
reclaim
because
again
super
creative
from
the
naming
and
the
important
part
is
just
to
specify
the
storage
class
and
then
that
clue
is
the
scheduler
and
that
it
knows
to
not
provision
anything
and
some
of
the
magic
that
happens
behind
the
scenes
with
using
some
of
the
stuff
is
the
external
volume
provisioner.
This
is
a
out
of
tree
project.
B
It's
a
career,
Denny's
incubator
project
I
believe
it
started
out
of
like
a
NFS
sort
of
automated
system,
but
what
this
project
is
is
it
contains?
Most
of
the
volume
code,
that
is,
entry
and
it's
basically
to
split
out
eventually
I,
suspect,
they'll
end
up
that'll,
just
be
a
plug-in
that
you
install.
B
But
what
this
is
is
that
it's
a
daemon
set
that
you
can
have
automatically
create
persistent
volumes
from
these
directories
or
mount
pads
and
you
what
you
do
is
you
map
storage
classes
to
local
directories
and
it
automatically
creates
and
deletes
those
persistent
volumes
for
you,
it's
pretty
slick
how
they
recommend,
checking
it
out.
It's
pretty
easy.
You
install
it
just
on
github.
Examples
are
in
gke
right,
so
what's
ends
up.
Looking
like
from
the
object
point
of
view
are
the
things
that
we're
actually
going
to
build
to
run.
B
Cassandra
we're
gonna
use,
two
stateful
sets
one
for
seeds
and
one
for
nodes
and
I'll
go
into
a
little
bit
of
what
that
actually
means
later.
But
Cassandra
does
have
two
types
of
nodes
in
its
mind
and
then,
of
course,
all
of
these
services
that
we
want
to
run
to
expose
these
these
workloads
right
on
the
internet
or
make
them
actually
accessible.
B
So
this
is
an
example
of
what
this
might
look
like
from
a
very
high
level.
We
have
just
a
pure
Benes
cluster,
which
is
kind
of
a
big
box,
and
then
two
different
stateful
sets
one
for
nodes,
one
for
seeds
in
this
case
the
scale
just
set
that
to
two.
So
we
see
that
there
are
two
regular
Cassandra
pods
right
and
one
seed
pod.
What's
really
really
nice
about
this
being
able
to
split
those
out
it's
a
little
bit
of
overhead
in
terms
of
API.
You
have
to
go
through
and
actually
manage
these
things.
B
You
have
to
build
tooling
to
do
this
right,
but
it
allows
you
to
scale
up
and
down
and
be
very,
very
intentional
about
the
way
you
do
that.
So,
for
example,
if
you
need
to
update-
or
you
need
to
make
configuration
changes
right,
if
you're
using
stateful
sets,
you
can
partition
and
it
also
lets
you
be
very
careful
about
your
seeds.
So
it
gives
you
a
lot
more
options
that
way
to
kind
of
split
it
out
and
that's
something
that
we
found
was
very,
very
useful.
B
Of
course,
the
building
block
in
this
scenario
is
local
storage
and
the
one
of
the
other
caveats
of
local
storage
is,
you
have
to
go
through
and
allocate
that
store
or
those
directories
on
every
node
before
you
actually
try
and
schedule
anything
right.
Something
has
to
create
directories.
Otherwise,
it's
not
going
to
work,
so
you
can
use
daemon
set
to
do
it.
If
you're
so
inclined,
you
could
use
your
config
management.
B
You
can
SSH
in
as
a
normal
user
and
do
this
which
I'm
totally
guilty
of,
but
you
can
go
through,
and
so
the
provisioner
by
default,
expects
entries
and
FS
tab.
Basically,
so
it
expects
amounts.
So
you
need
to
have
your
local
directories
as
a
mount
for
it
to
actually
understand
that
exists,
so
it's
fun
because
you
have
to
go
through
and
actually
bind
mount
for.
What
feels
like
kind
of
no
reason
but
like
it
makes
sense.
B
So
in
this
example,
since
we've
been
using
optical
storage
Cassandra,
that's
actually
the
bind
mount
so
which
you
would
go
ahead
and
do
would
be
to
create
the
Optus
Android
directory
as
the
main
directory.
All
that
data
is
going
to
be
stored,
you
go
ahead
and
you
create
a
second
directory,
opt
local
storage
Cassandra,
which
is
actually
what
the
pods
are
gonna
write
to,
and
then
you
buy
it
and
bound
to
the
original
directory
to
the
local
storage
directory
and
you're
done.
B
That
actually
makes
it
available
for
the
local
volume
provisioner
to
actually
be
able
to
consume
it
and
in
terms
of
what
the
external
volume
provisioner
expects.
You
just
give
it
a
config
map
on
when
it
starts
up,
and
it's
able
to
have
that
mapping
between
storage
classes
and
local
directories,
so
we
can
create
all
those
volumes.
So
this
is
an
example.
It
just
really
stripped
down
one
of
what
this
might
look
like,
which
just
a
single
storage
class.
B
So
it's
the
bolded
term,
is
the
storage
class
that
we've
kind
of
been
working
with
in
this
example,
which
is
local
Cassandra,
and
in
it
we
specify
the
host
directory
that
we
want
to
create
local
volumes
for
so
opt
local
Cassandra,
same
exact
directory
from
the
commands.
I
showed
you
before
yup
specify
mount
directory,
because
the
provisioner
is
also
going
to
mount
that
path.
To
make
sure
that
it
exists
so
can
actually
make
it.
So
if
that
goes
away,
it'll
automatically
clean
up
and
delete
volumes.
Things
like
that,
so
typically
they're
the
same
thing.
B
Unless
you
want
to
confuse
yourself
which
I
don't
recommend
and
then
the
last
part
I
want
to
talk
about
in
this
config,
which
is
really
interesting,
is
that
entry,
the
container
that
ships
for
the
provisioner
actually
has
a
bunch
of
really
useful
scripts
in
it?
So
it
has
this
hook
for
a
block
cleaner
command,
which
you
can
run
shred
whenever
you
actually
delete
the
volume,
so
it'll
run
it
to
a
level
of
to
automatically
behind-the-scenes.
B
So
things
like
Cassandra
Gamal,
which
is
the
base
config
for
Cassandra
JVM,
got
options
which
is
to
configure
the
JVM,
because
you're
going
to
want
to
tune
that
the
other
config
files
that
you
may
need
there's
a
whole
slew
of
them,
which
I'm
not
gonna,
get
into.
Typically
what
I
tend
to
do
for
handling
this
is
just
to
jam
it
into
one
giant
config
map
and
use
like
sub
pads
to
mount
the
individual
files
into
the
pod.
But
it's
really
up
to
you
all
of
these
files
could
be
individual.
B
B
You
have
obviously
a
bunch
of
different
options,
but
what
I've
done
is
used
a
couple
of
init
containers
and
then
a
couple
of
other
side:
cars
to
really
surface
some
interesting
functionality
that,
along
with
Cassandra,
so
for
example,
what
I've
done
for
the
setup
is
use,
assist,
control
container.
It's
like
a
busy
box
to
go
through
and
make
sure
that
the
nodes
themselves
are
ready
to
go.
B
There's
increment
ideas,
there's
a
couple
of
different
options
that
you
can
kind
of
set.
If
you
use
the
give
it
the
right
security
context,
things
like
network
bandwidth,
making
sure
you're
not
constrained
things
like
max
open
files,
the
stuff
that
you
always
kind
of
have
to
do
if
you're
running
a
big
database
and
actually
hitting
or
meeting
more
options
that
the
default
kernel
settings
typically
give
you
and
then
the
other
init
container
that
I
used,
which
is
getting
back
to
the
TLS
cert,
is
a
initializer
for
the
java
key
store
java.
B
If
you've
had
to
deal
with,
it
actually
has
a
lot
of
its
own
bespoke
tooling
for
doing
really
common
things
like
working
with
TLS
certificates.
So
typically
it's
much
easier
to
take
all
of
that
and
not
have
to
like
check
in
like
a
binary
set
of
data.
You
can
just
take
your
TLS
start
as
like
a
PEM
file
turn
that
into
a
key
store
and
mount
it
and
I'll
show
you
what
it
looks
like
in
a
second,
so
the
other
containers
in
this
pod
or
the
actual
running
containers.
That
is
part
of
the
application.
B
You
need
a
container
for
Cassandra,
which
is
the
one
on
the
left.
You'll
probably
want
something
for
metrics
and
what
I've
done
typically
is
if
you're
using
Prometheus
there
is
a
JMX
exporter
out
there,
that's
active,
that's
pretty
good,
all
it
really!
It's!
You
can
give
it
a
config
file
and
you
can
kind
of
filter
out
metrics
that
way
and
it'll
just
expose
that
through
Prometheus
scraping
so
pretty
easy
and
then
also
there's
a
backup
sidecar.
So
the
sidecar
process
that
I
call
it
that,
but
there's
more
you
can
do
with
that.
B
B
What
that
ends
up
looking
like
is
it
pulls
the
TLS
secret
reads:
it
converts
it
to
the
jks
format.
Using
java
tooling
saves
that
in
an
empty
directory,
which
is
just
in
RAM
right
and
you
can
mount
that
into
Cassandra
map
that
using
docker
and
I'm,
not
the
persistent
volume
you're
good
to
go
so
what
this
kind
of
putting
it
all
together
right.
If
we
had
a
multi
region,
Cassandra
cluster
running
into
different
AWS
regions-
us
least
one
us
to
in
this
example
that
I
have
up
here.
B
If
you're
running
on
bare
metal,
if
you're
running
in
public
cloud,
that's
pretty
much
a
whole
talk
into
itself,
I'm
perfectly
happy
to
kind
of
dive
into
that
after
the
talk
so
now
that
that's
all
out
of
the
way,
let's
talk
operations,
so
I
entered
at
this
before
with
stateful
sets,
but
the
by
having
two
different
stateful
sets
the
way
you
can
actually
rule
out
updates
the
way
you
can
increase
the
size
of
your
cluster
right.
All
you
have
to
do
is
make
changes
to
the
state
of
all
set
itself.
B
So
if
you
increase
the
replicas
that
gets
you
more
Cassandra
nodes,
you
process
more
data.
You
have
to
sometimes
be
careful
with
that,
but
it's
a
pretty
easy
process
generally.
The
other
nice
part
about
using
stateful
sets
is
it's
anybody
work
with
partitions
in
stateful
sets.
I
know
you
have
okay,
so
there's
a
feature
that
not
too
many
people
use,
which
is
a
partition.
It's
actually
filled
in
a
stateful
set
and
what
it
allows
you
to
do
is
to
actually
control
and
draw
a
line
sort
of
through
doing
rolling
updates.
B
You
know
make
changes
to
the
way
that
clusters
laid
out
or
to
force
backups.
You
can
replace
all
that
with
gruber
names,
jobs
just
super
nice
and
we'll
show
you
an
example
of
that
right
now.
So,
of
course,
when
running
any
database,
you
probably
want
to
have
to
deal
with
taking
backups
at
some
point.
Whether
or
not
you
actually
restore
them
is
a
totally
different
story,
but
you're
gonna
want
to
have
them
somewhere.
B
So
in
this
example,
I
have
the
prettiest
cluster,
just
three
nodes
running
on
a
kubernetes
cluster,
with
the
backup
job
going.
So
we
issue
some
requests,
and
then
you
can
back
that
up
to
any
remote
storage
that
you
want.
S3
cloud
storage,
they're,
the
ones
who
had
nice
icons
by
the
way
there
are
others,
and
so
in
terms
of
the
pot
itself.
B
If
we
kind
of
do
a
little
bit
of
a
deep
dive
from
that
particular
high-level
view,
when
the
back
up
side
car
receives
a
request,
what
that
ends
up
doing
is
typically,
you
should
not
expose
certain
ports
on
Cassandra,
but
because
they're
basically
authenticated
it's
kind
of
like
a
management,
API
control
plane,
you
can
add
authentication,
but
it's
a
pain,
and
you
may
as
well
to
sort
of
expose
a
very
specific
service
to
do
some
of
these
things.
So
that's
what
the
back
up
side
card
does.
B
So
it's
it's
ready
to
go,
and
since
it's
that
volume
is
mapped
in
the
pod
and
the
sidecars
in
the
pod,
what
it
does
is
tore
up
that
snapshot
and
then
syncs
that
up
to
s3
whenever
it
gets
around
to
it
or
Google,
Cloud,
Storage
or
digital
ocean
spaces
really
doesn't
matter.
You
can
put
it
wherever
you
feel
like,
and
so
the
last
operational
bit
that
I
kind
of
wanted
to
talk
to
you
about-
and
this
is
very
Cassandra
specific
as
well-
is
syncing
seeds.
B
So
in
Cassandra
there,
as
I
mentioned
before,
there
are
two
different
types
of
notes:
there
are
seed
nodes
and
their
regular
nodes
and
a
seed
node
is
effectively
responsible
for
helping
to
boot,
grab
the
cluster,
so
they
kind
of
serve
as
kind
of
communication
gateway
slightly
to
tell
new
nodes
the
topology,
the
cluster
they,
for
all
intents
and
purposes,
that's
their
only
special
option.
So
you
just
want
to
make
sure
you
have
enough
of
them
around
just
to
survive,
node,
failures
and
case
things
restart
right.
B
But
if
you
think
about
the
way
that
kubernetes
handles
things
right,
when
you
restart
a
pod,
that
IP
is
going
to
change
so
at
some
point
you
need
to
have
a
new
synced
list
of
IPs.
So
one
way
to
do
that
is,
of
course,
to
run
a
job
on
some
predefined
interval
to
sync
all
of
that
up
as
an
FYI,
dynamic
reloading,
where
you
don't
have
to
actually
restart
Cassander
to
pick
that
up
is
coming
in
Cassandra
for
ATO
apparently.
B
B
What
I
had,
of
course,
I
have
to
say
the
obligatory,
we're,
also
hiring
if
you,
if
anything's
sounds
pretty
cool.
If
you
want
to
work
on
multi
region
who
are
in
Eddy's
as
a
service,
that's
effectively
what
my
team
does
at
NetApp.
So
if
you
want
to
drop
by
our
booth
more
than
happy
to
talk
about
it
and
I
think
we
have
a
little
bit
of
time
for
questions
a
couple
minutes.
So
if
you
got
any
questions
have
at
if
not
I'm
gonna
stick
around
after
the
talk.
B
B
Yes,
the
question
was:
have
I
done
any
performance
benchmarking
and
the
answer
is
no
not
in
this
case
it.
It
will
be
quicker,
but
it
depends
really
dependent
in
the
provider
like
if
you're
doing
your
latency
will
be
effectively
EBS
and
that's
not
going
to
be
any
different
for
any
other
workload,
typically
yeah.
If
your
key
exactly
your
local
disk,
like
if
you're
using
SSDs,
this
is
going
to
be
fast
if
you're
using
spinning
rust,
not
so
much
yeah.
B
Yeah
mm-hmm,
so
the
question
was:
have
I
experimented
with
the
local
L
family
controller.
No,
but
I
really
want
to,
because
that
is
much
much
easier,
especially
because
you
get
a
little
bit
of
isolation.
The
problem
with
local
Valley
or
there
are
many
interesting
caveats
to
local
volumes,
some
one
of
which
is
that
you
have
no
isolation
and
you
really
can't
even
specify
a
size.
B
So
LVM
helps
you
with
that
right
because
you
can
say
I
just
want
a
local
50,
gig
volume
and
once
it
exceeds
50
gigs,
that's
actually
a
partition,
so
yeah
I'm
super
interested
in
seeing
it
I'm
just
having
out
a
chance
to
play
with
it
there,
as
I.
Think
Ben
talked
a
little
bit
in
in
cig
storage
about
making
that
actually
the
default,
local
and
volume
provisioner
I,
think
they're
kind
of
trying
to
figure
out
what
the
future
of
that
looks
like
yeah,
no
problem.
Yes,.
B
B
Sure,
okay,
so
let
me
restate
the
question
is
so
it's
on
record.
The
question
was
to
your
pods,
when
using
local
storage
require
privileged
access,
and
yet
the
answer
to
that
is
no,
because
it
is
a
persistent
volume,
so
it's
being
kind
of
consumed
as
one
right.
It's
not
actually
mounting
a
host
path.
It's
kind
of
happening
behind
the
scenes.
Kubernetes
is
doing
that,
for
you.
B
Yes,
so
question
was
for
the
SIS
control,
Inuk
container
genie
its
privileges,
for
that
depends
on
what
you're
running
right.
If
you're
running
the
namespace
sets
of
sis
controls,
you
can
either
actually
use,
what's
built
into
kubernetes
to
do
that
instead
of
using
that
container
or
you
can
grant
it,
the
proper
security
context
do
that,
but
if
you're
running
any
of
the
non
namespaced
ones,
which
is
typically
most
of
the
interesting
ones,
it's
going
to
have
to
be
privileged,
which
effectively
makes
the
rest
of
the
pod
privileged
yeah.
B
So
kubernetes
has
an
abstraction
for
doing
some
of
that,
like
there
is
a
security
context
that
you
can
provide
in
a
pod
spec
right.
So
that's
what
you
can
do,
that's
what
you
can
set
to
make
that
happen.
All
right,
cool
I'll
be
around.
If
anybody
has
follow-up
questions,
I
really
appreciate
everybody
coming
out
here.