►
From YouTube: Kubernetes Persistent Data Challenges – AZ, Region and Multi-Clou... Chris Milsted & Patrick McFadin
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes Persistent Data Challenges – AZ, Region and Multi-Cloud Patterns - Chris Milsted, Ondat & Patrick McFadin, DataStax
In this talk we are going to explore the patterns for building resilient applications with data persistence requirements on Kubernetes. We will be discussing deploying stateful workloads like Cassandra in Kubernetes using operators, with demos and examples at each stage using native objects like PVCs, dynamic provisioning and the k8ssandra operator. We will also be providing a sneak peek of some of the multi-cluster challenges with Kubernetes and Kube-Fed. Four architectural patterns will be covered with worked examples: The first pattern will be within an AZ, how to work with PVs. The next pattern will be spanning three AZs, and how to embrace or work around PV/AZ affinity constraints. The next pattern is three AZs and two regions within a single provider. Here we start to see physical limitations such as the speed of light begin to impinge on our patterns. Lastly, we move to two providers and hybrid-cloud patterns, leveraging operators to deliver our sample Cassandra workload.
A
Good
afternoon
welcome
to
our
virtual
talk.
This
is
a
talk
on
kubernetes,
persistent
data
challenges,
availability
zones,
regions
and
multi-cloud
patterns.
So
what's
the
next
slide,
quick
hellos
from
us?
My
name
is
chris
milstead.
I'm
a
solution
architect
at
on
that.
So
working
with
customers
around
all
things,
data
and
kubernetes,
and
my
co-presenter
is.
B
Dr
mcfadden,
I
work
in
data
stacks.
I
work
in
developer
relations
and
developer
experience
and
I
work
in
the
apache
cassandra
project
and
a
few
others,
but
mostly
around
open
source
data
and
kubernetes.
I'm
also
writing
a
book
right
now
cloud
native
data
on
managing
cloud
native
data
on
kubernetes
available
at
a
bookstore
soon,
chris
very
soon.
Oh.
B
A
Brilliant
and
before
we
get
into
the
bulk
of
the
presentation,
just
a
quick
couple
of
thanks
to
alex
and
rags
alex
is
one
of
the
engineers
on
the
kate
sandra
project
and
has
been
invaluable
help
in
setting
up
a
lot
of
the
demos
and
stuff
we're
going
to
show
later
and
rags
actually
built
a
lot
of
the
multi-cloud
crazy
demo
that
we're
going
to
show
right
at
the
very
end.
So
with
that
onto
the
bulk
of
the
presentation-
and
I
believe
you
can
leave
this
off
patrick.
B
I
will
so
chris,
I
think
everyone's
going
to
ask
this
question.
Why
would
we
even
do
this?
So
I'm
going
to
walk
you
through
some
of
the
reasons
that
you
may
want
to
be
outside
of
one
data
center.
One
rack-
and
this
is
a
pretty
typical
question
because
it
does
add
complexity.
It's
very
simple,
let's,
let's
be
very
clear,
it's
very
simple
to
run
something
that's
running
on
your
laptop
and
that's
the
most
simple
way
to
do
it.
B
This
is
this
is
why
we're
using
multiple
az's,
multiple
multi-anything
is
things
around
failures
and
then
there's
more
relevant
reasons
for
contemporary
reasons,
I
would
say
which
I'll
get
into
in
a
minute
but
like
we'll
start
with
just
availability
zones,
so
availability
zones
are
built
by
clouds,
for
this
problem
is
to
mitigate
certain
types
of
failure.
So
I
have
a
little
chart
here
that
is
is
out
there.
B
You
can
it's
it's
nothing
new,
but
it's
about
you
know,
managing
your
your
failures
and
the
service
level
agreement,
the
slas
that
people
that
clouds
give
on
like
an
individual
vm.
Is
you
get
three
nines?
Okay,
that's
not
bad,
but
it's
not
great
either.
You
know
that's
still
minutes
or
hours
of
downtime
with
hardware
failures
and
when
we're
talking
about
cloud,
usually
one
piece
of
hardware
could
run
multiple
vms.
You
know
multiple
images,
multiple
containers,
but
that
you
know
you
get
a
five
on
the
end
of
that.
B
B
Pretty
much
halfway
around
the
world
you're
looking
at
hundreds
of
milliseconds
around
time
and
if
you're,
if
your
data
is
in
one
place
and
your
customers
are
in
another,
somebody's
gonna
lose
out,
and
so
getting
your
data
closer
to
customers
is
the
name
of
the
game.
Now,
if
you
have
global
customers,
you
need
global
data.
There's
also
a
lot
of
regulatory
requirements
that
you
have
to
manage.
B
Every
sovereign
data
protection
law
has
different
rules
if
you're
in
europe,
if
you're
in
north
america,
if
you're
in
australia,
like
any
of
these
any
of
these
governments
that
are
gonna
and
they're,
always
changing
by
the
way,
are
gonna
have
regulatory
requirements
on
privacy
and
things
like
that.
You
have
to
manage
that.
B
But
sometimes
you
have
when
you
have
customers
in
those
different
areas,
you
need
to
manage
the
data
that
goes
there
and
then
finally,
I
mean
this
is
my
favorite
part
is
maximum
up
time,
and
I
I
say
this,
like
you:
have
zero
percent
chance
of
a
100
uptime.
If
you're,
only
in
one
region,
one
region
can
fail,
and
if
you
just
do
a
quick
google
on
cloud
failures,
you
can
see
that
this
happens
quite
often
and
then
multi-cloud,
and
this
is
getting
into
like
where
people
really
get
freaked
out.
But
why
like?
B
Why
would
you
be
do
multi-cloud
and
it's
a
funny
thing,
because
sometimes
it's
something
that's
done
to
you
as
an
sra
or
an
operator.
Multi-Cloud
can
come
about
whenever,
for
instance,
if
you
have
an
acquisition
and
you
acquire
a
company
that
is
in
a
different
cloud
provider
than
you're
in
or
if
you're
migrating
to
a
new
provider,
let's
say
you're
going
from
amazon
to
google.
Well,
while
you're
in
the
middle
of
that
which
is
probably
going
to
be
three
or
four
years,
maybe
five.
B
B
I
I
don't
know,
no
one's
ever
figured
out
exactly
why,
but
it
happens
all
the
time-
and
I
know
a
lot
of
you
are
probably
shaking
your
head
and
like
yeah.
That's
that
group,
you
know-
and
you
know
multi-cloud
is
there's
other
reasons,
and
I
think
these
are
more
relevant
contemporary,
like
I
said
that
we're
at
the
point
now,
where
we're
talking
about
avoiding
lock-in,
we're
talking
about
you
know
spreading
your
bet
using
the
cost
basis
of
cloud
in
a
better
way.
B
So
you
know
this
is
where
we're
getting
into
this
world
of
hybrid
and
hybrid
is
like
when
we're
building
out.
This
is
probably
where
a
lot
of
you
are
now
where
we're
trying
to
save
money
in
the
cloud,
but
we
still
own
a
lot
of
data
centers
and
when
you're
migrating
from
your
physical
data
center
that
you
spent
billions
of
dollars
on
and
you're
moving
over
to
the
cloud
during
that
time
frame
again,
probably
measured
in
years.
B
A
We
probably
did
it
in
the
last
40
years
before,
and
I
think
two
of
those
trends
are
the
first
one
is
looking
at
the
data
problems.
You
know,
we've
moved
our
compute
to
the
data.
If
you
look
at
the
whole
kind
of
mapreduce
shift
we
had
and
how
we
do.
Did
you
know
data
transformation,
data,
information
data?
You
know,
I
need
information
from
my
data.
How
do
I
mine
it?
We
found
out
that
moving
data
sets
around
is
just
too
costly.
A
A
A
You
know
an
enterprise-grade,
ssd
nvme
drive
and
we
get
a
million
iops
out
of
that
without
batting
an
eyelid
and
the
gentleman
who
looks
after
a
lot
of
the
linux
kernel
components
here,
he's
got
to
set
up
with
a
couple
of
octane
cards,
so
two
opt-in
cards
and
he
can
get
10
million
iops
out
of
two
cards.
So
thinking
about
what
did
we
do
when
processors
got
that
much
that
much
more
powerful?
We
virtualized
them.
A
What
are
you
finding
when
you
put
data
in
kubernetes
they're,
finding
there's
a
few
things?
They've
got
to
fix,
but
actually
they're,
finding
that
all
the
benefits
they
were
getting
from
their
applications
and
the
standardization,
the
ci
cd
and
the
you
know,
clicking
a
button
and
putting
their
developers
closer
to
the
computer
and
giving
them
back
control
and
moving
faster.
You
know
applications
to
production
safely
in
a
day,
all
those
kinds
of
things
we're
looking
for
data
is
actually
complementing
that
as
well.
A
B
Yeah
the
the
thing
that
I
I
feel
like-
and
this
is
a
very
important
part
of
what
you
know
this
is
part
of
my
book
chris.
So
this
is
a
shameless
plug,
but
it's
this
concept
of
in
kubernetes,
where
we're
building
building
out
something
very
revolutionary
and
evolutionary
at
the
same
time
where
we
went
from
physical
servers
to
virtual
machines
to
containers
and
now
we're
orchestrating
containers
with
a
little
bit
of
magic
and
creating
virtual
data
centers
it's.
B
This
is
a
logical
progression
in
our
virtualization,
where,
instead
of
owning
a
data
center
and
and
managing
the
power
managing
the
network,
all
those
things
we
are
at
the
point
now,
where
we're
renting
compute
network
and
storage,
creating
our
own
virtual
data
centers
for
what
we
need
bespoke
and
the
reason
is
because
we
are
building
applications
that
have
a
certain
amount
of
input.
And
then
we
require
a
certain
amount
of
output
and
what
happens
inside
that
virtual
data
center.
We
can
be
very
specific
about
and
we
can
build
the
tools
that
we
need.
B
A
We're
not
saying
to
change
any
of
the
kubernetes
architectural
patterns
they're
already
out
there
there's
a
link
at
the
bottom
here
to
the
cncf
disaster
recovery
paper.
It's
very
good.
Broadly,
it's
don't
stretch
your
kubernetes
clusters
across
regions
start
doing
that
at
the
application
space.
As
patrick
said
earlier,
you
know
the
speed
of
light
is
finite
as
soon
as
you
leave
an
availability
zone,
if
you're
building
in
the
cloud
things
cost
money-
and
you
know
bandwidth-
is
not
infinite
and
all
those
other
fallacies.
A
So
at
some
point
you
know,
cap
theorem
will
pick
in
and
your
partition
or
availability
or
your
consistency.
One
of
these
things
will
have
to
give.
So
what
we're
saying
is
this
is
how
to
complement
with
the
data
layer,
but
don't
think
we're
saying
do
anything
different
at
the
kubernetes
layer.
So
we'll
just
be
very
clear
on
that
before
we
go
any
further.
So
yes,
on
to
the
next
slide,
so
now
we're
going
to
get
to
the
bulk
of
the
fun,
which
is
the
first
set
of
patterns.
A
A
So
it
would
like
to
have
a
single
data
store
and
if
the
pod
dies,
then
it
will
want
to
restart
and
want
to
reconnect
to
that
same
data
store.
On
the
other
side,
I've
got
cassandra
which
is
using
stateful
sets,
and
it's
you
know
we
can.
Cassandra
is
a
fantastic
application
and
we
can
stretch
that
halfway
across
the
world
as
we'll
see
in
the
next
demo,
but
in
this
one,
what
I'm
going
to
show
is
how
we
can
allow
cassandra
to
do
all
the
kind
of
storage
layer
intelligence.
A
So
the
important
thing
in
both
in
these
kind
of
two
demos,
on
the
left
hand
side
of
the
right
hand.
Side-
is
how
we
complement
with
the
csi,
so
the
container
storage
interface
plug-in
and
the
application.
So
making
sure
these
two
things
are
working
in
harmony,
so
there's
about
an
hour
and
45
from
cold
to
cold
start
to
end
of
this
demo.
We've
cut
it
down
to
about
eight
minutes,
so
I'm
going
to
try
and
narrate
it
as
we
go
along,
so
patrick,
let's
go
for
it.
A
So
first
step
we're
going
to
spin
up
our
cluster
I'm
using
aws,
because
I
can
use
eksctl
just
to
easily
spin
up
my
cluster,
I'm
using
i3en
instances.
Those
are
instances
with
local
ssds
and
I'm
using
that
because
I
want
to
get
you
know:
300
000
iops.
I
want
to
get
the
fastest
storage.
I
can,
and
you
can
hopefully
see
in
there
there's
some
two
here
in
each
of
the
1a1b1c
those
availability
zones.
So
hopefully
that
aligns
with
the
picture
that
you
saw
on
the
previous
page.
A
A
So
I've
installed
the
storage
os
is
the
underlying
technology
name,
the
on
that
product
onto
the
platform
and
you'll
see
that
that
component
is
coming
up
there.
So
when
it
installs
by
default,
we'll
get
a
storage
class
created,
which
is
just
give
me
I'll
call
it
dumb
storage,
but
just
give
me
straight:
here's
a
bit
of
storage
from
an
nvme
drive.
I
also
created
in
there
a
more
intelligent
storage
class
where
I'm
asking
the
storage
layer
to
do
replication
and
I've
told
it
to
also
encrypt
it.
So
we
can
do
per
volume
encryption.
A
I've
also
asked
it
to
replicate
it
between
the
three
azs
and
that's
the
one
I'm
going
to
use
for
prometheus
the
more
intelligent
one,
because
prometheus
doesn't
have
the
smarts
to
replicate
the
data,
but
for
cassandra
I'm
going
to
do
that.
So
we're
now
into
installing
the
actual
workloads,
so
certificate
manager's
been
going
on
in
the
background
there
and
then
I'm
using
customize
here
to
actually
build
and
deploy
the
kate
sandra
operator.
A
So
now
we're
going
to
try
and
put
prometheus
onto
the
box
so
we'll
go
for
the
fourth
way
of
installing
things
so
we'll
use
the
community
helm
charts.
So
we're
going
to
put
prometheus
onto
my
cluster
now
as
well.
So
what
we're
going
to
get
to
is
the
point
where
we
have
that
kind
of
traditional
workload
and
we
should
start
to
see
metrics
in
our
cluster
and
we
also
should
have
our
in
a
second
we're
actually
going
to
spin
up
our
actual
kate
sandra
sorry,
cassandra
cluster,
using
the
case
android
operator.
A
So
let's
just
go
and
apply
that
so
there
should
be
a
a
custom
resource
of
oh
I'm
going
too
quick.
There
we
go,
but
hopefully
the
important
thing
in
there
was
there's
one
a
one
b.
One
c,
so
I
told
cassandra
the
case
under
operator
to
deploy
a
cassandra
cluster
and
there
should
be
a
staple
set
in
rack.
One
rack,
two
rack
three
and
those
are
pinned
to
the
1a
1b
1c
availability
zones.
So
hopefully
in
the
bottom
right
you
can
see
there's
that
little
181
b1c.
A
So
I
also
do
a
little
trick
here.
I'm
going
to
actually
label
them
on
at
the
on
that
layer
to
do
fencing.
So
what
we
can
do
is
we
can
do
faster
failovers.
So
if
a
node
times
out,
rather
than
waiting
the
five
minutes
for
a
kubernetes
node
timeout
to
expire,
the
platform
can
actually
monitor
it
and
kill
the
pod
and
actually
start
a
faster
failover.
If
you
wanted
to
do
that,
so
just
a
little
trick
to
speed
up
the
stuff.
A
The
other
thing
here,
I'm
now
running
a
job.
So
I'm
running
no
sql
bench.
So
I'm
running
workload
now
and
it's
running
live
against
this
cassandra
cluster.
I've
now
got
running
on
these
components
and
you
see
top
right.
There
we
got
metrics
so
prometheus
is
running.
We've
got
metric
storage
for
prometheus
cassandra's
running.
A
You
can
see
my
nosql
bench
I'll
be
ticking
along
we'll
keep
going
back
to
the
nosql
bench
and
what
you
should
see
is
we
get
up
to
about
60
through
my
workload,
it's
not
running
a
hard
benchmark,
but
it's
just
running
background
load.
So
now
we're
going
to
do
something
nasty
we're
going
to
cordon
and
drain
a
node.
A
A
A
My
rack
2
instance
is
a
very
orange
there
very
pending.
It's
not
very
happy.
So,
let's
check
my
newest
girl
bench
benchmark
is
still
running
through.
So
even
though
cassandra
has
lost
a
node,
it's
not
using
any
smarts
at
the
csi
plug-in
it's
just
using
raw
swell.
It's
using
the
csi
plugin
to
do
smart
things
to
be
able
to
automate
access
to
the
nvme
drives,
but
it's
not
doing
any
replication
of
that
layer.
It's
doing
that
all
in
the
cassandra
layer.
A
A
If
we
have
a
look
in
a
set,
we
should
see
if
we
go
into
one
of
these.
We've
still
got
metrics
and
we've
got
all
the
historical
metrics
as
well.
You
can
go
back
and
look
at
the
time.
Stamps
and
you'll
see
that
that's
running
all
the
way
through
so
we'll
bring
that
node
back
into
the
cluster
as
well
and
now,
of
course,
because
I'm
evil
we're
gonna
go
for
the
big
test,
which
is
rather
than
turn
things
off
nicely.
A
A
So
I'm
going
to
go
into
my
aws
console
and
I'm
going
to
terminate
my
instance
and
of
course
it's
going
to
say
to
me:
are
you
sure,
because
you're
using
very
fast
local
nvme,
300
000
io
devices?
But
if
you
kill
them,
you
lose
them
and
like
yeah,
don't
worry,
we've
got
that
covered
we're
using
a
different
plug-in.
It's
fine!
So
we
kill
it
and
you
notice.
I've
lost
one
of
my
other
cassandra
nodes
now,
but
my
no
skill
bench
still
plugging
away
so
brilliant.
A
A
What
actually
happened
here?
It
couldn't
recover
to
the
the
second
node
in
that
availability
zone,
because
the
stargate
pod
was
running
and
I
discovered
later
on
in
the
cassandra
operator-
there's
actually
a
flag
you
have
to
set
if
you
want
to
allow
stargate
and
cassandra
to
actually
coexist
on
a
node.
So
what
actually
happened
here
was
kubernetes
did
its
stuff.
A
So
if
you
go
in
and
look
at
the
kubernetes
cluster
and
in
the
eks
you'll
see
we
get
it's
initializing
here,
so
it's
actually
building
a
seventh
node
to
replace
the
sixth
one
that
I
nastily
killed
so
taints
as
it
comes
up
and
then,
as
it
comes
up
notice
on
the
right
hand,
side
the
metrics.
There
is
a
small
gap
in
the
metrics
here,
because
we
killed
that
no,
we
killed
that
stateful
set
by
pulling
the
power
out,
but
there
we
go.
There's
another
storage.
Os
demon
set,
so
we
brought
storage.
A
The
demon
sets
running
on
there,
so
we
can
access
the
storage
over
the
network
now
and
there
we
go
a
cassandra
cluster
is
back
to
full
health.
We've
got
three
stateful
sets
in
three
azs
and
our
nosql
bench
is
running
along,
so
there
we
go,
hopefully
that
everyone
followed.
That,
as
I
said,
is
about
now
45
end
to
end
in
real
time.
You've
we've
compressed
it
to
eight
minutes
and
if
we
go
on
to
the
next
slide.
A
A
We
are
doing
clever
things
with
the
csi
plugins,
I'm
trying
to
get
the
fastest
storage
at
the
most
cost
effective
point
I
can
and
I'm
trying
to
make
sure
that
that
csi
plugin
is
doing
the
right
things
for
the
right
workload.
So
for
a
work
like
prometheus,
I'm
doing,
storage
level,
replication,
storage,
level,
encryption,
storage,
level,
topology
awareness,
all
of
those
components
in
there,
but
for
cassandra,
I'm
just
saying
hey,
I
will
automate
you
access
to
these
nvme
drives,
so
you
can
have
as
many
iops
as
you
want
to
do.
A
Your
back
end,
storage
and
what
we're
trying
to
say
is.
As
we
said,
you
know
you
can
get
to
no
to
load
down
time
using
a
single
region
by
taking
this
approach
even
using
it,
because
this
approach
as
well
because
we're
using
local
nvme
drives.
This
will
map
to
you,
know
your
eks
anywhere
or
you're
on
premise.
Anything
like
that.
Hopefully
you
can
see
as
well.
A
You
know
there's
nothing
special
about
running
this
in
the
cloud
we're
using
local
nvme,
so
you
could
take
this
and
run
this
in
your
data
center
today
as
well,
and
I
believe,
that's
the
end
of
the
first
kind
of
set
of
patterns
and
I'm
going
to
hand
back
to
patrick
now
to
take
us
into
the
multi-reed
multi-region
multi-provider
ones.
Now.
B
Yeah,
the
the
multi-multi
multi,
the
multiverse,
the
multi
multiverse
of
madness,
speaking
of
shameless
plugs
the
so
this
is
this
is
another
pattern
and
it
exploits
the
the
goodness
of
cassandra.
So
this
is
again
using
kate
sander,
which
is
a
project
to
help
this
built
for
running
cassandra
and
kubernetes,
without
pain,
batteries
included,
and
so
there's
some
really
cool
things
that
keep
sandra
can
do,
especially
with
the
latest
operator,
about
running
a
multi-cluster
multi-kubernetes
cluster.
So
if
you
have
multiple
kubernetes
clusters
across
regions
or
different
clouds,
it
doesn't
really
matter.
B
C
This
is
a
demo
about
cassandra
on
multiple
clusters,
so
the
first
thing
you're,
probably
going
to
want
to
do,
is
set
up
the
networking,
the
routing
peering
and
all
that
I
myself
am
not
a
networking
geek.
So
what
I
do
is
I
have
used
a
product
called
aviatrix
and
using
aviatrix
you
can
set
up.
You
know
the
vpcs
that
are
taking
the
pairing
fairly
easily.
C
It's
essentially
on
this.
You
know
particular
github,
the
rags
and
s
avx
multi-cloud
communities,
and
I
basically
forked
it
from
from
aviatrix
and
essentially
what
you
can
do
here
is
you
can
specify
the
different
cloud
accounts
and
once
you've
onboarded.
These
accounts,
which
I
haven't
talked
about.
You
set
up
the
controller
ip,
the
username
and
the
password,
and-
and
you
know
then
the
rest
of
the
terraform
scripts
use
this
to
you
know,
do
the
appropriate
networking,
let's
go
into
lens,
okay
and
and
look
at
these
different
clusters.
C
As
you
can
see
here,
I
have
a
bunch
of
different
clusters,
but
I've
added
them
to
the
hotbar
here:
the
aks
avs
abs,
cluster,
the
eks
avx
cluster
and
the
gke
avx
cluster
okay.
So
these
are
the
three
ones
so
now
that
we
have
seen
this
the
next
step,
you
know
once
I
have
to
set
up
the
clusters
and
on
on
the
three
public
clouds
there
used
to
be
a
manual
step.
C
If
you
wanted
to
do
a
multi-cluster
in
which
you
had
to
manually
inject
the
seeds
from
one
cluster
into
another,
you
don't
have
to
do
that
anymore.
Instead,
what
you
do
is
you,
you
have
the
concept
of
what
is
referred
to
as
a
control
plane,
and
then
you
have
the
concept
of
a
data
plane.
You
know
very
similar
to
you
know
what
the
community
services
provide.
The
control
planes
essentially
take
care
of
installing
cassandra
on
a
communities.
C
C
You
know,
basically
what
I
did
was
I
installed
the
kit
center
operator.
Okay
and
then
you
know
essentially
there's
the
concept
of
the
control
plane
and-
and
you
set
that
to
true,
so
you
can.
If
you,
if
you
look
at
you,
know
the
different
clusters,
you
will
see
that
one
of
them
the
aks,
is
set
up
in
as
a
control
plane,
and
you
know
the
remaining
two,
which
is
gke
and
eks,
are
set
up
as
data
planes.
C
So,
as
you
can
see
here,
there's
a
gk
cluster
right
and-
and
you
can
see
here,
that
it
has
the
kate
sandra
operator
right
and-
and
if
you
dig
through
this,
you
know
kate
sandra
operator,
you'll
be
able
to
see
that
this
is
operating
as
a
data.
Plane
and-
and
you
know,
aks-
is
the
control
plane.
So
that's
kind
of
how
how
I
set
that
up
so
to
install
the
kit
center
operand
on
the
control
plane.
C
So
you
take
the
gke
client
config
and
install
it
in
aks
and
likewise
you
know
take
what
is
an
eks
and-
and
you
know
let
the
control
plane
on
the
aks
know
that
you
know
eks
is,
you
know,
is
part
of
it
and
look
at
client
conflict.
You
can
see
that
you
know
these
two
clusters
have
been
set
up,
which
is
the
eks
cluster
and
the
gke
cluster
on
on
on
aks
to
install
the
cluster.
You
know
what
I
do
is
I
and
we
can
take
a
quick
look
at
this.
C
Essentially,
this
is
the
we
are
installing
a
kit,
sander
cluster.
You
can
specify
some
default
values
for
the
storage
class,
but,
as
you
know,
you
know,
the
storage
class
is
going
to
vary
in
in
the
different
clouds,
so
for
gke,
I'm
using
premium
rwo
for
eks,
I'm
using
gp2,
and
then
I'm
going
to
set
stargate,
which
is
our
unified
apis
for
rest,
sql,
grpc
and
you
know
graphql
and
so
on.
Now
that
we've
done
that,
let's
go
take
a
look
at
one
of
the
clusters
like.
C
Let's
take
a
look
at
gk,
let
me
take
a
look
at
the
parts
that
are
running
and
most
critically
the
cassandra
data
center
itself.
So
this
is
the
gke
cluster
and
you
know
if
you,
if
I
go
back
and
look
at
the
eks
you'll
find
something
similar
as
wow.
So
if
you
look
at
the
cassandra
radiation
data
center,
this
is
an
eks
and
essentially
what
happens
is
when
you
install
cage
center
operator
with
the
control
planes
and
the
data
planes
appropriately,
they
will
set
up
a
protocol
called
gossip
protocol.
C
So
you
need
to
be
able
to
enable
those
ports
to
be
able
to
talk
to
each
other
and-
and
you
know
once
they're
able
to
talk
to
each
other,
they
can
form
a
a
multi-cluster
and
one
of
the
critical
things
that
you
do
is
basically
make
sure
that
it's
one
giant
you
know
humongous
cluster
basically
is
like
a
node
tool
command.
So
so
what
you're
going
to
do
here?
B
So
that
was
a
a
really
quick
and
interesting
demo
about
how
we
can
run
a
multi-cloud,
and
there
were
some
really
important
takeaways
inside
of
that
and
the
first
one
really
and
rags
was
very
clear
about
how
he
did
it.
He
used
aviatrix
but
he's
getting
the
network
right
when
you're
transiting
from
one
region
or
from
one
cloud
to
another.
That's
that's
not
just
a
click
of
a
button,
and
so
we're
not
going
to
try
to
sugarcoat
that
and
getting
the
network
right
means
that
you
need
to
have
your
routes.
B
You
need
to
have
your
security
and
everything
else
right.
It's
it's
not
impossible!
We've
done
worse
things,
those
of
you
who've
ever
done.
Bgp
and
cisco.
Networking
know
that
this
is
probably
going
to
be
way
easier
than
that,
but
it
is
the
most
important
thing
in
this
case
and
keep
in
mind
mind
your
charges,
because
when
you
do
transit,
that's
when
the
charges
for
clouds
get
really
up
there.
B
B
That's
just
a
really
important
thing
to
take
note
of
because
when
you're,
when
you're
spanning
one
kubernetes
cluster
to
another,
they
need
to
be
cool
with
each
other
and
the
way
we
do
that
is
with
credentials
and
then,
finally,
with
the
cage
sandra
operator,
we
do
have
this
difference
between
a
control,
plane
and
a
data
plane,
and
that's
an
important
call
out,
because
it's
not
just
one
full
thing.
You
have
to
think
about
like
what
we're
trying
to
do
is
running
across
multiple
kubernetes
clusters,
we're
against
what
kubernetes
really
wants
to
do.
B
A
Yeah,
so
I
think
just
a
summary
of
it
all
so,
hopefully,
you've
seen
how
you
can
put
data
into
your
kubernetes
clusters
safely
and
how
you
can
take
these
patterns
and
how
you
can
build
applications
with
no
low
downtime
and
how
to
use
the
right.
I
mean
this
is
the
big
thing.
The
big
chunk
in
the
middle
here
is
complement
your
csi
provider.
A
With
your
application.
There's
got
to
be
that
contract
between
kind
of
the
infrastructure,
kind
of
operations
of
kubernetes
and
the
applications
consuming
it.
So
they've
got
to
know
how
to
design
their
application.
You
don't
want
them
to
have
like
nine
times
replication,
where
you've
got
three-way
replication
in
the
application
and
three-way
replication
at
the
storage
layer.
B
And
that's
that's
a
really
important
point.
Neither
of
us
are
here
saying
this
is
just
oh.
This
is
a
click,
a
button,
easy
and
it's
because
the
clouds
don't
get
along
with
each
other.
That's
okay,
but
it's!
It
should
be
something
that
we're
all
used
to
as
engineers
and
operators.
As
the
you
know,
there
are
the
hard
parts,
that's
why
we
still
get
paid
money.
If
it
was
a
click
of
a
button,
then
developers
would
do
this
all
day
right.
A
Indeed,
and-
and
I
think
you
know
in
summary-
we're
saying
you
know-
data
and
kubernetes
if
you're
going
to
be
successful
in
kubernetes,
you've
got
to
get
your
data
into
kubernetes,
so
I'm
expecting
over
the
next
year,
20
12
24
months,
every
person's
going
to
be
having
this
is
their
kind
of
next
next
leg
of
their
kubernetes
strategy.
So
you
know,
hopefully
there's
some
good
questions
for
us
after
this
and
please
hit
us
up
on
the
q,
a
tool
and
some
links
in
here
as
well.
All
right,
I
just
thank
you.